Regions and sub-regions

Regions and sub-regions#

Everything in MTLDP is organised around a region: a geographic bounding box together with one JSON configuration file, one raw data directory and one processed data directory. All command line tools and the Region classes of mtldp.utils.config take the configuration file as their single entry point and derive every other path from it.

The configuration file#

A configuration file is created with create_config_file (see Command line tools of mtldp-meta-utils) and lives in a configs directory:

{
  "region_id": "oakland_co",
  "city_id": "oakland",
  "bbox": [-83.4558, 42.4249, -83.1054, 42.6011],
  "raw_dir": "raw_data",
  "processed_dir": "processed_data",
  "junction_id_list": ["62590214", "62590221"],
  "date_str_list": ["2025-03-03", "2025-03-04"],
  "timezone": "America/Detroit",
  "mode": "preprocessing",
  "search_link_length_threshold": 2000,
  "min_link_entry_threshold": 200
}

Key

Meaning

region_id

Unique name of the region. It is the name of the region directories and the region_name of the traffic network.

city_id

Name of the city the region belongs to.

bbox

Bounding box as [lon_min, lat_min, lon_max, lat_max]. All bounding boxes stored by MTLDP use this longitude-first order; only the --bbox command line option accepts the OpenStreetMap order lat_min,lon_min,lat_max,lon_max and converts it.

raw_dir / processed_dir

Roots of the raw and processed data. Relative paths are resolved against the project root (see below).

junction_id_list

Node IDs of the intersections of interest. Used to filter trajectories when loading them, by process_trajs --filter-junctions, by copy_regions and by the signalized_selected_junctions.json export.

date_str_list

Default list of dates (YYYY-MM-DD) an AppRegion loads when no dates are given.

timezone

IANA time zone used to derive local dates and times of day from UTC timestamps.

mode

preprocessing for regions that process raw data (ProdRegion), application for regions that only consume processed data (AppRegion).

search_link_length_threshold

Optional, default 2000 m. How far a signalized link is extended through unsignalized nodes when the signalized network is built (SignalizedNetwork).

min_link_entry_threshold

Optional, default 200 m. Minimum length kept for the entry of a signalized link; it is capped at half of the search threshold.

Project root and path resolution#

The location of the configuration file defines the project root, so that a whole project can be moved or mounted elsewhere without editing any file:

<project root>/
├── configs/
│   ├── oakland_co.json               master region
│   └── oakland_co_subs/              sub-regions of oakland_co
│       ├── 0_0.json
│       └── 0_1.json
├── raw_data/                         <raw_dir>
│   ├── oakland_co/
│   └── oakland_co_subs/
│       ├── 0_0/
│       └── 0_1/
└── processed_data/                   <processed_dir>
    ├── oakland_co/
    └── oakland_co_subs/
        ├── 0_0/
        └── 0_1/

  • For <root>/configs/<region>.json the project root is the parent of configs.

  • For <root>/configs/<master>_subs/<sub>.json the project root is the parent of configs as well (two levels up).

  • A relative configuration path such as configs/oakland_co.json is accepted only when the current working directory is the project root; otherwise an absolute path is required.

Region classes#

mtldp.utils.config exposes three classes.

Region

Base class. Parses the configuration, resolves raw_dir and processed_dir, builds the BoundingBox and provides convert_GPS_to_xy / convert_xy_to_GPS (local metric coordinates with the south-west corner of the bounding box as origin).

ProdRegion

Region in preprocessing mode. input is a RawPathManager and output a ProcessedPathManager (both described below). network holds the full Network loaded from network/traffic_network.pickle when it exists (load_network=False skips the load). This is the class used by all pipeline commands.

AppRegion

Region in application mode. It only knows the processed tree (ppm) and loads data through a ProcessedDataLoader (pdl). On construction it loads the signalized network into network and, unless load_spat=False, the SPaT pickle into spat. Trajectories are loaded on demand:

  • load_trajs_dict(date_str_list, junction_id_list, overwrite_buffer, load_points, load_row_num, signalized) returns a TrajectoryDict and caches each day as a pickle under trajectories/trajectories/cache/ so that the second load is fast;

  • load_trajs_df(date_str_list, junction_id_list, signalized) returns a single pandas.DataFrame of trajectory rows;

  • load_stop_bar_by_movement, load_lane_num_by_movement, load_lane_num_coef_by_movement, load_stop_sign_adjustment_by_movement and the per-period load_start_loss_by_movement, load_clock_shift_by_movement, load_penetration_rate_by_movement and load_manually_adjustment_by_movement read the JSON files of the calibration directory and return an empty dict when the file does not exist.

Both ProdRegion and AppRegion offer create_directory_structure_and_template_files(), which is what create_region calls.

Directory layout of a region#

The two path managers of mtldp.utils.config.PathManager define the layout. Every attribute that ends in dir is created as a directory and every attribute that ends in path is touched as an empty file by create_region.

Raw data (RawPathManager)#

<raw_dir>/<region_id>/
├── raw_map/
│   ├── map.osm                   OpenStreetMap XML of the region (input of build_traffic_network)
│   ├── map_bbox_dict.json        tiles of the region, written by the OSM downloader (see below)
│   ├── arterial.json             arterial definitions
│   └── overwrite/
│       ├── node.csv              manual attribute overrides, one file per element type
│       ├── link.csv
│       ├── segment.csv
│       └── movement.csv
├── raw_spat/                     the seven SPaT CSV files (optionally raw_spat/estimate/)
└── raw_trajs/                    raw GPS files delivered by the vendor (CSV or Parquet)

Processed data (ProcessedPathManager)#

<processed_dir>/<region_id>/
├── network/
│   ├── traffic_network.pickle             the Network object (loaded by ProdRegion / AppRegion)
│   ├── traffic_network.json               GeoJSON layers of the whole network
│   ├── signalized_traffic_network.json    GeoJSON layers of the signalized network
│   ├── signalized_arterial.json           …restricted to the arterials
│   ├── signalized_selected_junctions.json …restricted to junction_id_list
│   ├── filtered.osm                       directed OSM XML consumed by the map matcher
│   ├── shapefile/                         shapefile export of the network
│   ├── csv/                               tabular export of the network elements
│   ├── overwrite.json                     merged overrides actually applied to the network
│   ├── ubodt.txt                          shortest-path table cached by the map matcher
│   ├── raw.osm, static_net.json, bbx.json
│   └── map.log                            build log
├── trajectories/
│   ├── raw_pts/<date>_raw.csv             output of split_points
│   ├── points/<date>_matched.csv          output of match_points_to_map (Feather binary)
│   └── trajectories/
│       ├── <date>_trajs.csv               output of process_trajs (mode all)
│       ├── <date>_signalized_trajs.csv    output of process_trajs (mode signalized)
│       └── cache/<date>_trajs.pkl         pickle cache written by AppRegion.load_trajs_dict
├── spat/spat.pickle                       output of parse_spat
├── calibration/                           stop_bar.json, lane_num.json, lane_num_coef.json,
│   └── cache/                             stop_sign_adjustment.json and per date/TOD files
├── figures/pi/                            figure outputs of applications
├── tables/pi/, tables/spat_est/           table outputs of applications
└── ui/                                    figures and HTML pages of the review UI

ProcessedPathManager also provides path helpers for the application layer, e.g. pi_table_path(start, end, level, prefix, suffix) under tables/pi/<start>_<end>/, coord_diag_table_dir and green_diag_table_dir under tables/, and the ui_* helpers under ui/<start>_<end>/. Calibration files that depend on a period are named <date>_<tod range>_<kind>.json (for example 2025-03-03_7-9_start_loss.json).

Sub-regions#

Why regions are tiled#

The cost of the pipeline grows quickly with the size of the map: the Overpass API rejects or times out on large downloads, and the Fast Map Matching algorithm pre-computes a shortest-path table (UBODT) whose size and memory footprint grow with the number of edges. A county-scale region is therefore processed as a set of sub-regions: tiles of the master bounding box that are each a complete, independent region with their own map, network and trajectory files. The master region keeps the shared inputs (the full map, the tile list, the arterials and the overrides), and every pipeline command that receives the master configuration fans out over the tiles.

How the tiles are defined#

The tiling is decided when the map is downloaded (create_region --download-osm). The OSMDownloader of mtldp.utils.data_io splits the master bounding box into a grid whenever its latitude or longitude span exceeds 0.1 degrees: the number of rows is int(lat span / 0.1) and the number of columns int(lon span / 0.1), and the span is divided evenly. Each tile is downloaded separately, with a 200 m buffer (MAP_BBOX_EXTEND_METER) around it so that roads crossing a tile border are complete on both sides, and saved as raw_map/map_<row>_<col>.osm. When the box is small enough for a single tile the file is simply raw_map/map.osm.

In both cases the downloader writes raw_map/map_bbox_dict.json, which maps the tile ID to the unbuffered tile bounding box:

{
  "0_0": [-83.4558, 42.4249, -83.2806, 42.5130],
  "0_1": [-83.2806, 42.4249, -83.1054, 42.5130],
  "1_0": [-83.4558, 42.5130, -83.2806, 42.6011],
  "1_1": [-83.2806, 42.5130, -83.1054, 42.6011]
}

This file is the single source of truth for the sub-regions. It can also be written or edited by hand to define tiles of arbitrary shape, as long as the IDs are valid file names.

Where the sub-regions live#

Each tile becomes a region in its own right:

  • its configuration is configs/<master>_subs/<tile>.json, with region_id equal to the tile ID and city_id, mode, timezone and date_str_list inherited from the master. junction_id_list starts empty and is meant to be edited by hand;

  • its raw_dir is <master raw_dir>/<master>_subs and its processed_dir is <master processed_dir>/<master>_subs, so its data sits in raw_data/<master>_subs/<tile>/ and processed_data/<master>_subs/<tile>/ with exactly the layout described above;

  • its raw_map/map.osm is the tile’s map, cut from the master map with the same 200 m buffer.

Creating the sub-regions#

There are two ways to materialise the tiles:

  • create_region -c configs/<master>.json --download-osm --subregion does it right after the download, copying each map_<tile>.osm into the sub-region;

  • create_sub_regions -c configs/<master>.json (re)creates them from map_bbox_dict.json at any later time. It cuts each tile out of the master map.osm (so manual edits made to the master map are propagated), merges the master’s overrides and arterials into each tile, and can copy the processed trajectories with --copy-trajs. Existing tile configurations are kept unless --overwrite is given; --exclude skips tiles.

Fan-out rule of the commands#

Every command that accepts a configuration file resolves the list of regions to work on with the same rule (load_sub_regions_config_dir_and_bbox_dict in mtldp.utils.config.Region):

  1. read raw_map/map_bbox_dict.json of the given region; when the file does not exist the region is treated as its own single tile;

  2. when the dictionary has exactly one entry, the region is the tile: the command works on the configuration it was given;

  3. when it has several entries, the command iterates over configs/<region>_subs/<tile>.json for every tile in the dictionary.

Consequences of this rule:

  • passing the master configuration processes all sub-regions; passing a sub-region configuration processes that sub-region only (a sub-region has no map_bbox_dict.json of its own);

  • build_traffic_network builds the tiles in parallel, one process per tile, and accepts --exclude-sub-regions;

  • split_points reads every raw trajectory file once and writes the points into the tile whose bounding box contains them. Because the tiles do not overlap, a point belongs to exactly one tile, while the buffered maps guarantee that the network of each tile extends beyond its points;

  • match_points_to_map and process_trajs loop over the tiles sequentially, each tile with its own network and its own cached ubodt.txt;

  • delete_processed_trajs and draw_region_bbox also operate on all tiles.

A trip that crosses a tile border is split between the two tiles at the border. Trajectories are computed per movement, so only the movements whose approach straddles the border are affected; if this matters for an application, the tiles can be redrawn in map_bbox_dict.json so that the borders fall away from the intersections of interest.

Master region versus sub-regions in applications#

The master region itself does not get a network or trajectories when it is tiled: its processed directory only exists to hold the <master>_subs tree. Applications therefore open the sub-region configuration (AppRegion('configs/<master>_subs/<tile>.json')) that contains their intersections. To study a corridor that is spread over several tiles or several regions, copy_regions assembles a new, self-contained region from the source regions, merging their maps, overrides, arterials and processed trajectories for the requested junctions.