.. _preproc_reference: Pre-processing: ``mtldp.preproc`` ================================= ``mtldp.preproc`` (repository ``mtldp-preproc``) turns raw inputs into the processed data described by ``mtldp.meta``: an OpenStreetMap file into a ``Network``, GPS points into map-matched points, and matched points into movement-level trajectories with performance indices. It depends on ``mtldp-meta-utils`` and on the Fast Map Matching library ``fmm``. The package has three sub-packages plus the command line tools of :ref:`preproc_cli`; the step-by-step data flow is described in :ref:`pipeline`. .. list-table:: :header-rows: 1 :widths: 24 76 * - Sub-package - Purpose * - ``build_network`` - Parse OSM XML, build nodes, segments, links, movements and arterials, derive the signalized network, apply manual overrides, export for map matching. * - ``map_match`` - Wrapper around ``fmm``: prepare the input, run the matcher, attach the results and the map attributes to every point. * - ``process_trajs`` - Cut matched points into trajectories, compute distances to the stop bar, stops, delays and the other performance indices; quality-control plots. * - ``process_spat`` - Placeholder; SPaT parsing is done by ``mtldp.utils.data_io`` and the ``parse_spat`` command. ``build_network`` ----------------- The public entry point is ``build_network(region_id, osm_file_path, city_id, logger_dir, logger_file_path, build_networkx, overwrite_json_path, arterial_file_path, overwrite_node_path, overwrite_movement_path, overwrite_segment_path, overwrite_link_path, signalized_link_search_threshold, signalized_link_min_entry)``, which returns a ``Network``. The modules it is built from, in the order they are called: ``overwrite_attribs.generate_overwrite_json`` Consolidates the override CSV files into ``network/overwrite.json``. ``node.csv`` provides ``controller_id`` per ``node_id``; ``movement.csv`` provides ``movement_index`` per ``movement_id``. ``link.csv`` and ``segment.csv`` are accepted but not parsed yet. The JSON is left untouched when it is newer than all CSV files, so edits made directly to it (for example by the review tool) are preserved. ``parse_osm.load_xml_map`` Parses the OSM XML into ``OsmWay`` and ``Node`` objects, records which ways start, end or pass through each node, and drops ways with fewer than two nodes and nodes referenced by no way. ``process_node.add_different_class_of_nodes`` Classifies nodes by degree: degree one becomes an ``EndNode``, degree above two becomes a ``SignalizedNode`` when tagged ``highway=traffic_signals`` and an ``UnSignalizedNode`` otherwise; degree-two nodes stay ordinary. ``process_way.split_traverse_intersection_way`` and ``update_way_details`` Splits every way at the intersections it passes through (children share the intersection node and get IDs ````), then extracts the attributes from the tags: speed limit (``maxspeed``, default 25 mph, converted to m/s), one-way flag (``oneway``, ``junction=roundabout`` or a single lane), lane counts (``lanes``, ``lanes:forward``, ``lanes:backward``; an unspecified two-way road is split evenly), lane use (``turn:lanes`` and directional variants) and name, and computes length, geometry and headings. ``process_segment.generate_segments_in_network`` and ``generate_segments_connections`` Creates one ``Segment`` per way and direction of travel (``0`` forward, ``1`` backward, the latter only for two-way roads) with reversed geometry for the backward direction, the per-direction lane attributes and a display geometry shifted to the right. Degree-two nodes with traffic become ``SegmentConnectionNode`` objects. Then the segments are connected: at connectors every upstream segment is joined to every downstream segment except its own reverse; at intersections the turn letter of every pair is derived from the cardinal directions and, when ambiguous, from the headings. ``process_link.init_links_in_network`` Walks from every intersection or end node through connector nodes to build ``Link`` objects (``_``) with the segment list, total length, harmonic-mean speed limit, geometry and heading. When two links share the same end nodes the straighter one is kept. ``process_movement.init_movements_in_network`` Enumerates one ``Movement`` per upstream segment and downstream turn at every intersection, with ``movement_index`` from the approach direction and turn (see :ref:`meta_reference`), and adds a sink movement ``_dest`` to every link. ``mtldp.utils.algs.build_link_networkx_graph`` Builds the NetworkX graph used for shortest paths. ``process_node.infer_node_name`` Names unnamed intersections ``/`` from the OSM ``name`` tags of the approaching segments. ``overwrite_attribs.overwrite_map_attributes`` Applies ``overwrite.json``: for each of ``nodes``, ``segments``, ``links`` and ``movements`` it sets the listed attributes on the matching element. Any attribute can be set this way; the CSV files only produce ``controller_id`` and ``movement_index``. ``process_arterial.init_arterial_in_network`` Reads ``raw_map/arterial.json``, a list of objects ``{"arterial_id": ..., "ref_node": ..., "details": {"": ["", ...]}}``, and builds one ``OnewayArterial`` per direction. Only the end links of a direction need to be listed: intermediate links are filled in along the shortest path. ``process_signalized_network.build_signalized_network`` Derives the ``SignalizedNetwork`` (:ref:`signalized_network`) with the two thresholds from the region configuration. Two more functions belong to the package: ``parse_osm.merge_signalized_intersections(input_filename, output_filename, merge_threshold=60)`` The ``merge_osm`` command. Groups ``highway=traffic_signals`` nodes that lie within the threshold of each other, moves the surviving node of each group to the group's mean position, rewrites the way references and removes the merged nodes. ``convert_for_mm.save_network_to_xml(network, bbox, output_file, directed=True)`` and ``save_network_to_shp(network, output_folder)`` Export the network for the map matcher. With ``directed=True`` every **segment** is written as a one-way OSM way whose node order follows the direction of travel, so the matcher sees a directed graph and returns segment IDs directly. The shapefile ``edges.shp`` has the fields ``osmid`` (segment ID), ``u`` and ``v`` (end node IDs) expected by ``fmm``. ``map_match`` ------------- ``FmmModule(map_folder, shapefile_folder, load_map=False, k=16, search_radius_in_m=5000, gps_error_in_m=100, restart_on_disconnect=True)`` Loads ``edges.shp`` into an ``fmm`` network and graph and prepares the *upper-bounded origin destination table* (UBODT), a pre-computed table of shortest paths between edges within 0.03 degrees that makes the matching fast. The table is written to ``/ubodt.txt`` and reused when ``load_map=True`` and the file exists; ``build_traffic_network`` deletes it so it is regenerated for a new network. ``k`` is the number of candidate edges per point, ``search_radius_in_m`` the candidate search radius and ``gps_error_in_m`` the assumed GPS noise; all three are converted to degrees internally. With ``restart_on_disconnect`` the matcher restarts after two consecutive points that cannot be connected through the network instead of discarding the whole trajectory (requires the patched ``fmm`` build). ``run(input_file, output_file, status_file)`` matches one file and appends the timestamped matching statistics to ``status_file`` (``points/mm_log.txt``). ``dump_trajs_for_fmm(points_df, fmm_input_file, attr='traj_id', network_bound=None, resort=True)`` Sorts the points by trajectory and time, assigns a dense integer ID per trajectory (``int_id``) and writes the ``fmm`` input: a semicolon-separated CSV with the columns ``id``, ``x``, ``y``, ``timestamp`` (GPS point mode). ``load_fmm_result_to_trajs(points_df, fmm_file)`` Reads the ``fmm`` output and adds four columns to the points: ``segment_id`` (matched edge), ``error`` (distance from the GPS point to the matched position, meters), ``offset`` (distance of the matched position from the start of the segment) and ``spdist`` (shortest-path distance since the previous point; ``-1`` marks a restart after a disconnection). Points that could not be matched are dropped. ``add_map_details.add_complete_map_info(df, network)`` Resolves every matched ``segment_id`` against the network and adds ``link_id``, ``upstream_intersection``, ``downstream_intersection`` and ``junction_id`` (the downstream intersection the vehicle is heading to). ``process_trajs`` ----------------- ``process_points_to_trajs(df, network, process='simple', maximum_away_per_lane=10, split_time_gap=-1, dummy_trajs=False)`` The core routine of ``process_trajs`` (``trajs_process.py``). It reads the ``POINT_COLUMNS`` (``trip_id``, ``link_id``, ``segment_id``, ``junction_id``, ``timestamp``, ``date``, ``tod``, ``latitude``, ``longitude``, ``speed``, ``error``) of the matched points and #. drops points without a matched segment or link; #. drops points whose matching ``error`` exceeds ``lane number × maximum_away_per_lane + 15`` meters for the segment they were matched to; #. walks the points of each trip in time order and starts a new trajectory every time the ``link_id`` changes. The boundary point is kept in both trajectories, so a trajectory runs from the entrance of its link to the first point on the next link; #. attaches the movement that joins the link to the next link (or the link's sink movement at the end of the trip), the ``movement_index``, the traversed ``segment_list``, the point lists and ``avg_error``; #. computes the metrics with ``simple_trajectory_processing`` or ``complete_trajectory_processing`` (below). With ``dummy_trajs=True``, when two consecutive links of a trip are not joined by a movement but a path exists between them, the trajectory takes the first movement of that path and one *dummy trajectory* without points (``points_num`` 0, ``avg_error`` -1) is emitted for every further link of the path, so that every trip is a feasible path in the network. ``split_time_gap`` is reserved for splitting trips at long gaps and is not implemented. ``filter.filter_points_by_bounding_box(points_df, bbox)`` Inclusive rectangular filter on ``latitude`` and ``longitude`` (used by ``split_points`` and ``match_points_to_map``). ``single_traj_proc`` ``simple_trajectory_processing(time_list, lat_list, lon_list, upstream_link)`` computes ``travel_time``, ``travel_distance`` (path length of the GPS trace), ``avg_speed`` and ``dis_diff`` (travel distance minus link length). ``complete_trajectory_processing(...)`` additionally computes the signed distance to the junction for every point (``calc_dist``) and the delay and stop indices (``calc_delay``); a trajectory that never comes within ``lane number × 10`` meters of the junction is flagged ``AwayJunction`` in ``comments``. ``calc_dist.get_travel_distance_to_junction(lat_list, lon_list, ref_lat, ref_lon)`` Projects the junction on the last leg of the trace with the law of cosines to find where the trace crosses the junction, and returns ``distance_list``, the cumulative distance rebased so that **0 is the junction, negative is upstream and positive is downstream**, together with ``min_dis_to_junction`` and a ``comments`` flag (``Cross``, ``NotReached``, ``OverCross`` or ``Only1Point``). ``trajs_segment`` and ``trajs_intervals`` ``trajectory_segmentation(time_ls, distance_ls, speed_ls, free_flow_vt, stop_vt)`` cuts the time series of one trajectory into ``STOP`` (below ``stop_vt``), ``FREE_FLOW`` (above ``free_flow_vt``) and ``TRANSITION`` intervals (``TrajectoryInterval`` objects), then merges intervals of the same type separated by small gaps and reclassifies stops shorter than 3.5 s as transitions. ``fetch_details_from_intervals`` serialises the stops for the ``stop_details`` column. ``calc_delay.get_trajectory_delay_stops(time_ls, distance_ls, speed_ls, link, delay_threshold=60)`` Runs the segmentation with a stop speed of 1.5 m/s and a free-flow speed of 80 % of the link speed limit and derives the performance indices defined in :ref:`pipeline_process_trajs`: ``free_v``, ``free_arrival_time``, ``arrival_time`` (interpolated with ``interpolate.get_traverse_x`` at distance 0), ``control_delay``, ``stop_nums``, ``stop_delay``, ``queue_dis``, ``service_level``, ``stop_details`` and ``split_failure``. ``trajs_quality.generate_and_plot_trajectory_metrics(config_path, date_pairs, exclude_weekends, junction_ids, output_filename, fig_width, fig_height, fig_format, dpi)`` The ``eva_trajs_qc`` command. For every date range it loads the signalized trajectories of each day, computes the number of trajectories and trips, vehicle miles and hours travelled, the total number of points and the average sampling interval, caches them as ``trajs_metric__.pkl`` in the trajectory cache directory and plots the ranges side by side in a three-by-two figure saved in the region's ``figures`` directory.