Existing studies reveal that unsafe worker movement behaviors are one of the major reasons of construction site fatalities resulting in serious collisions with site objects and machinery. For understanding worker movements in dynamic construction environments which involve moving and changing objects, a system named 'WoTAS' (Worker Trajectory Analysis System) is proposed. First, a real-time Bluetooth Low Energy (BLE) beacons-based data collection and trajectory pre-processing subsystem is built for extracting multifaceted trajectory characteristics and stay regions of the workers that will help in recognizing the important regions in the building for categorizing the worker movements. Second, to enable the desired semantic insights for better understanding the underlying meaningful worker movements using the contextual data repositories related to the building environment, an ontology-based 'STriDE' (Semantic Trajectories for Dynamic Environments) model is applied which tracks the evolution of moving and changing building objects and outputs semantic trajectories. For extracting insights from the semantic trajectories, the Hidden Markov Model (HMM) is used which is one of the probabilistic approaches present in the literature for describing the object behavior in time. Using the HMMs, a set of trajectories belonging to a stay region is analyzed by categorizing the worker movements into four different states. In the end, the output of the Viterbi algorithm is visualized using a BIM model for identifying the most probable high-risk locations involving sharp worker movements and rotations. The developed 'WoTAS' system will help safety managers in monitoring and controlling building activities remotely in dynamic environments by understanding the worker movements for improved safety management in day-to-day building operations. Eventually, understanding the worker movements will contribute towards reducing the chances of near-miss incidents on sites which have the potential to cause serious accidents.