Processing raw data

Processing raw data#

Typical scenarios#

  • Produce the processed data of a new region, offline or as part of a data production pipeline;

  • re-process a region after fixing its map, its overrides or its arterials;

  • update or test the algorithms of the pre-processing module;

  • add a data adapter for another trajectory data source.

The walkthrough below processes one region from scratch. Every command is described in detail in Command line tools of mtldp-meta-utils, Command line tools of mtldp-preproc and The raw data pipeline; the region and sub-region layout is described in Regions and sub-regions. mtldp-preproc and its dependencies must be installed (Preparation).

Step 1: Create the configuration file#

Choose a project directory and create the configuration in configs/ with the bounding box of the area of interest (lat_min,lon_min,lat_max,lon_max, the order shown by OpenStreetMap):

(mtldp) $ mkdir my_project && cd my_project
(mtldp) $ create_config_file -n oakland_co -c oakland -m preprocessing \
             -b 42.4249,-83.4558,42.6011,-83.1054
Configuration file generated at: `configs/oakland_co.json`

The options that are not given are asked interactively; the value in square brackets is the default and Enter accepts it. Open the file afterwards to fill in junction_id_list (the intersections of interest) once the network is built.

Step 2: Create the directory tree and download the map#

(mtldp) $ create_region -c configs/oakland_co.json --download-osm --subregion
Directory tree and template files generated.
Download OSM data from Overpass API based on bbox ...
There are 2 x 2 = 4 grids. Each grid will be downloaded as a separate file.
OSM data saved to raw_data/oakland_co/raw_map/map_0_0.osm
...

This creates the raw and processed trees of the region (see Regions and sub-regions for the layout), downloads the OpenStreetMap data of the bounding box and, when the box is larger than about 0.1 degree in either direction, splits it into tiles and creates one sub-region per tile under configs/oakland_co_subs/. A small region gets a single raw_map/map.osm and no sub-regions.

draw_region_bbox -c configs/oakland_co.json draws the tiles on a map to check them. Tiles can be redrawn by editing raw_map/map_bbox_dict.json and running create_sub_regions.

Without --download-osm the map has to be obtained manually (Preparation of OSM data) and saved as raw_map/map.osm.

Step 3: Prepare the map inputs#

Merge split signal nodes. Large intersections are often mapped with one traffic-signal node per carriageway. Merge them so that every intersection has exactly one node:

(mtldp) $ cd raw_data/oakland_co_subs/0_0/raw_map
(mtldp) $ mv map.osm map_raw.osm && merge_osm -i map_raw.osm -o map.osm -d 60

Arterials. Describe the corridors of interest in raw_map/arterial.json, one entry per arterial with the first and last link of each direction (intermediate links are filled in along the shortest path):

[
  {
    "arterial_id": "Woodward",
    "ref_node": "62590214",
    "details": {
      "N": ["62590180_62590214", "62590290_62590301"],
      "S": ["62590301_62590290", "62590214_62590180"]
    }
  }
]

Overrides. raw_map/overwrite/node.csv (node_id,controller_id) maps intersections to their signal controller, and raw_map/overwrite/movement.csv (movement_id,movement_index) corrects movement indices that were inferred wrongly from the geometry. Both files start empty and are usually filled after a first build, when the network has been reviewed.

Step 4: Build the traffic network#

(mtldp) $ build_traffic_network -c configs/oakland_co.json

All sub-regions are built in parallel. The results are in processed_data/oakland_co_subs/<tile>/network/: the network pickle, the files for the map matcher, the GeoJSON layers and the log map.log. Review the network (with mtldp-review, or by loading the GeoJSON in any GIS tool), fix the inputs of Step 3 and rebuild until the intersections of interest are correct. Then write the junction_id_list of each sub-region configuration.

To customise the build, call the library directly:

from mtldp.preproc.build_network import build_network
from mtldp.utils.config import ProdRegion
from mtldp.utils.data_io import dump_traffic_network_to_pickle

region = ProdRegion('configs/oakland_co_subs/0_0.json', load_network=False)
region.network = build_network(region_id=region.region_id,
                               city_id=region.city_id,
                               osm_file_path=region.input.osm_path,
                               arterial_file_path=region.input.arterial_path,
                               overwrite_node_path=region.input.overwrite_node_path,
                               overwrite_movement_path=region.input.overwrite_movement_path,
                               overwrite_segment_path=region.input.overwrite_segment_path,
                               overwrite_link_path=region.input.overwrite_link_path,
                               overwrite_json_path=region.output.overwrite_json_path,
                               logger_dir=region.output.network_dir,
                               signalized_link_search_threshold=region.search_link_length_threshold,
                               signalized_link_min_entry=region.min_link_entry_threshold)
# ... manipulate the network object ...
dump_traffic_network_to_pickle(region.network, region.output.network_pickle_path)

Step 5: Parse the SPaT data (optional)#

Prepare the seven CSV files described in Preparation of Signal Phase and Timing (SPaT) data in raw_spat/ of each sub-region that has signal timing data, then:

(mtldp) $ parse_spat -c configs/oakland_co_subs/0_0.json

The result is spat/spat.pickle.

Step 6: Split the raw trajectory points#

Point the command at the vendor delivery and give its format:

(mtldp) $ split_points -c configs/oakland_co.json -v SL_2025 -d /data/deliveries/2025-03

Each sub-region receives the points inside its bounding box, one file per local date in trajectories/raw_pts/. Run the command once per delivery; to start over, clear the directory with delete_processed_trajs -c configs/oakland_co.json -d raw_pts.

Step 7: Map matching#

(mtldp) $ match_points_to_map -c configs/oakland_co.json -s 2025-03-03 -e 2025-03-09

The first run of a sub-region computes the shortest-path table of the map matcher, which takes a while; later runs reuse it. The matched points of each date are written to trajectories/points/.

Step 8: Compute the trajectories and their indices#

(mtldp) $ process_trajs -c configs/oakland_co.json -s 2025-03-03 -e 2025-03-09 --process-mode signalized

This writes trajectories/trajectories/<date>_signalized_trajs.csv with one row per trajectory and the performance indices (travel time, control delay, stops, queue distance, …). Without --process-mode signalized the indices are computed on the whole network and written to <date>_trajs.csv.

Step 9: Check the result#

(mtldp) $ eva_trajs_qc -c configs/oakland_co_subs/0_0.json \
             -d 2025-03-03,2025-03-09@2025-06-02,2025-06-08

compares the daily number of trajectories, trips, miles and hours travelled between periods and saves the figure in the figures directory of the sub-region. The processed region is now ready to be used by applications (Building applications on processed data).

Tip

All commands accept --help. Passing the configuration of a single sub-region to any pipeline command processes that sub-region only.