Existing literature reveals that major reasons of fatalities on construction sites are linked to mobility related issues such as unsafe human behaviors, difficult site conditions and workers striking against or being struck by the moving objects. In an attempt to reduce such fatalities, a system is proposed to enrich spatio-temporal trajectories of construction workers. Three semantic enrichment techniques are used to construct semantically enriched trajectories which are: (1) enrichment with semantic points that maps site location identification to trajectory points; (2) enrichment with semantic lines that relies on the speed based segmentation approach infers modes of transportation used in trajectory‘s episodes; and (3) enrichment with semantic region for mapping a complete trajectory on an actual construction site zone. Constructed semantic trajectories will help Health and Safety (H&S) managers in making improved decisions for monitoring and controlling site activities by understanding workers behaviors that ultimately attempts to contribute in reducing fatal accidents occurring on construction sites each year.