Pre-processing: mtldp.preproc

mtldp.preproc (repository mtldp-preproc) turns raw inputs into the processed data described by mtldp.meta: an OpenStreetMap file into a Network, GPS points into map-matched points, and matched points into movement-level trajectories with performance indices. It depends on mtldp-meta-utils and on the Fast Map Matching library fmm. The package has three sub-packages plus the command line tools of Command line tools of mtldp-preproc; the step-by-step data flow is described in The raw data pipeline.

Sub-package	Purpose
build_network	Parse OSM XML, build nodes, segments, links, movements and arterials, derive the signalized network, apply manual overrides, export for map matching.
map_match	Wrapper around fmm: prepare the input, run the matcher, attach the results and the map attributes to every point.
process_trajs	Cut matched points into trajectories, compute distances to the stop bar, stops, delays and the other performance indices; quality-control plots.
process_spat	Placeholder; SPaT parsing is done by mtldp.utils.data_io and the parse_spat command.

build_network

The public entry point is build_network(region_id, osm_file_path, city_id, logger_dir, logger_file_path, build_networkx, overwrite_json_path, arterial_file_path, overwrite_node_path, overwrite_movement_path, overwrite_segment_path, overwrite_link_path, signalized_link_search_threshold, signalized_link_min_entry), which returns a Network. The modules it is built from, in the order they are called:

overwrite_attribs.generate_overwrite_json

Consolidates the override CSV files into network/overwrite.json. node.csv provides controller_id per node_id; movement.csv provides movement_index per movement_id. link.csv and segment.csv are accepted but not parsed yet. The JSON is left untouched when it is newer than all CSV files, so edits made directly to it (for example by the review tool) are preserved.

parse_osm.load_xml_map

Parses the OSM XML into OsmWay and Node objects, records which ways start, end or pass through each node, and drops ways with fewer than two nodes and nodes referenced by no way.

process_node.add_different_class_of_nodes

Classifies nodes by degree: degree one becomes an EndNode, degree above two becomes a SignalizedNode when tagged highway=traffic_signals and an UnSignalizedNode otherwise; degree-two nodes stay ordinary.

process_way.split_traverse_intersection_way and update_way_details

Splits every way at the intersections it passes through (children share the intersection node and get IDs <way id><n>), then extracts the attributes from the tags: speed limit (maxspeed, default 25 mph, converted to m/s), one-way flag (oneway, junction=roundabout or a single lane), lane counts (lanes, lanes:forward, lanes:backward; an unspecified two-way road is split evenly), lane use (turn:lanes and directional variants) and name, and computes length, geometry and headings.

process_segment.generate_segments_in_network and generate_segments_connections

Creates one Segment per way and direction of travel (<way id>0 forward, <way id>1 backward, the latter only for two-way roads) with reversed geometry for the backward direction, the per-direction lane attributes and a display geometry shifted to the right. Degree-two nodes with traffic become SegmentConnectionNode objects. Then the segments are connected: at connectors every upstream segment is joined to every downstream segment except its own reverse; at intersections the turn letter of every pair is derived from the cardinal directions and, when ambiguous, from the headings.

process_link.init_links_in_network

Walks from every intersection or end node through connector nodes to build Link objects (<upstream node>_<downstream node>) with the segment list, total length, harmonic-mean speed limit, geometry and heading. When two links share the same end nodes the straighter one is kept.

process_movement.init_movements_in_network

Enumerates one Movement per upstream segment and downstream turn at every intersection, with movement_index from the approach direction and turn (see Data structures: mtldp.meta), and adds a sink movement <link id>_dest to every link.

mtldp.utils.algs.build_link_networkx_graph

Builds the NetworkX graph used for shortest paths.

process_node.infer_node_name

Names unnamed intersections <street of movements 1,2,5,6>/<street of movements 3,4,7,8> from the OSM name tags of the approaching segments.

overwrite_attribs.overwrite_map_attributes

Applies overwrite.json: for each of nodes, segments, links and movements it sets the listed attributes on the matching element. Any attribute can be set this way; the CSV files only produce controller_id and movement_index.

process_arterial.init_arterial_in_network

Reads raw_map/arterial.json, a list of objects {"arterial_id": ..., "ref_node": ..., "details": {"<direction>": ["<link id>", ...]}}, and builds one OnewayArterial per direction. Only the end links of a direction need to be listed: intermediate links are filled in along the shortest path.

process_signalized_network.build_signalized_network

Derives the SignalizedNetwork (SignalizedNetwork) with the two thresholds from the region configuration.

Two more functions belong to the package:

parse_osm.merge_signalized_intersections(input_filename, output_filename, merge_threshold=60)

The merge_osm command. Groups highway=traffic_signals nodes that lie within the threshold of each other, moves the surviving node of each group to the group’s mean position, rewrites the way references and removes the merged nodes.

convert_for_mm.save_network_to_xml(network, bbox, output_file, directed=True) and save_network_to_shp(network, output_folder)

Export the network for the map matcher. With directed=True every segment is written as a one-way OSM way whose node order follows the direction of travel, so the matcher sees a directed graph and returns segment IDs directly. The shapefile edges.shp has the fields osmid (segment ID), u and v (end node IDs) expected by fmm.

map_match

FmmModule(map_folder, shapefile_folder, load_map=False, k=16, search_radius_in_m=5000, gps_error_in_m=100, restart_on_disconnect=True)

Loads edges.shp into an fmm network and graph and prepares the upper-bounded origin destination table (UBODT), a pre-computed table of shortest paths between edges within 0.03 degrees that makes the matching fast. The table is written to <map_folder>/ubodt.txt and reused when load_map=True and the file exists; build_traffic_network deletes it so it is regenerated for a new network. k is the number of candidate edges per point, search_radius_in_m the candidate search radius and gps_error_in_m the assumed GPS noise; all three are converted to degrees internally. With restart_on_disconnect the matcher restarts after two consecutive points that cannot be connected through the network instead of discarding the whole trajectory (requires the patched fmm build). run(input_file, output_file, status_file) matches one file and appends the timestamped matching statistics to status_file (points/mm_log.txt).

dump_trajs_for_fmm(points_df, fmm_input_file, attr='traj_id', network_bound=None, resort=True)

Sorts the points by trajectory and time, assigns a dense integer ID per trajectory (int_id) and writes the fmm input: a semicolon-separated CSV with the columns id, x, y, timestamp (GPS point mode).

load_fmm_result_to_trajs(points_df, fmm_file)

Reads the fmm output and adds four columns to the points: segment_id (matched edge), error (distance from the GPS point to the matched position, meters), offset (distance of the matched position from the start of the segment) and spdist (shortest-path distance since the previous point; -1 marks a restart after a disconnection). Points that could not be matched are dropped.

add_map_details.add_complete_map_info(df, network)

Resolves every matched segment_id against the network and adds link_id, upstream_intersection, downstream_intersection and junction_id (the downstream intersection the vehicle is heading to).

process_trajs

process_points_to_trajs(df, network, process='simple', maximum_away_per_lane=10, split_time_gap=-1, dummy_trajs=False)

The core routine of process_trajs (trajs_process.py). It reads the POINT_COLUMNS (trip_id, link_id, segment_id, junction_id, timestamp, date, tod, latitude, longitude, speed, error) of the matched points and

1. drops points without a matched segment or link;

2. drops points whose matching error exceeds lane number × maximum_away_per_lane + 15 meters for the segment they were matched to;

3. walks the points of each trip in time order and starts a new trajectory every time the link_id changes. The boundary point is kept in both trajectories, so a trajectory runs from the entrance of its link to the first point on the next link;

4. attaches the movement that joins the link to the next link (or the link’s sink movement at the end of the trip), the movement_index, the traversed segment_list, the point lists and avg_error;

5. computes the metrics with simple_trajectory_processing or complete_trajectory_processing (below).

With dummy_trajs=True, when two consecutive links of a trip are not joined by a movement but a path exists between them, the trajectory takes the first movement of that path and one dummy trajectory without points (points_num 0, avg_error -1) is emitted for every further link of the path, so that every trip is a feasible path in the network. split_time_gap is reserved for splitting trips at long gaps and is not implemented.

filter.filter_points_by_bounding_box(points_df, bbox)

Inclusive rectangular filter on latitude and longitude (used by split_points and match_points_to_map).

single_traj_proc

simple_trajectory_processing(time_list, lat_list, lon_list, upstream_link) computes travel_time, travel_distance (path length of the GPS trace), avg_speed and dis_diff (travel distance minus link length). complete_trajectory_processing(...) additionally computes the signed distance to the junction for every point (calc_dist) and the delay and stop indices (calc_delay); a trajectory that never comes within lane number × 10 meters of the junction is flagged AwayJunction in comments.

calc_dist.get_travel_distance_to_junction(lat_list, lon_list, ref_lat, ref_lon)

Projects the junction on the last leg of the trace with the law of cosines to find where the trace crosses the junction, and returns distance_list, the cumulative distance rebased so that 0 is the junction, negative is upstream and positive is downstream, together with min_dis_to_junction and a comments flag (Cross, NotReached, OverCross or Only1Point).

trajs_segment and trajs_intervals

trajectory_segmentation(time_ls, distance_ls, speed_ls, free_flow_vt, stop_vt) cuts the time series of one trajectory into STOP (below stop_vt), FREE_FLOW (above free_flow_vt) and TRANSITION intervals (TrajectoryInterval objects), then merges intervals of the same type separated by small gaps and reclassifies stops shorter than 3.5 s as transitions. fetch_details_from_intervals serialises the stops for the stop_details column.

calc_delay.get_trajectory_delay_stops(time_ls, distance_ls, speed_ls, link, delay_threshold=60)

Runs the segmentation with a stop speed of 1.5 m/s and a free-flow speed of 80 % of the link speed limit and derives the performance indices defined in Stage 4: process_trajs: free_v, free_arrival_time, arrival_time (interpolated with interpolate.get_traverse_x at distance 0), control_delay, stop_nums, stop_delay, queue_dis, service_level, stop_details and split_failure.

trajs_quality.generate_and_plot_trajectory_metrics(config_path, date_pairs, exclude_weekends, junction_ids, output_filename, fig_width, fig_height, fig_format, dpi)

The eva_trajs_qc command. For every date range it loads the signalized trajectories of each day, computes the number of trajectories and trips, vehicle miles and hours travelled, the total number of points and the average sampling interval, caches them as trajs_metric_<start>_<end>.pkl in the trajectory cache directory and plots the ranges side by side in a three-by-two figure saved in the region’s figures directory.