.. _regions: Regions and sub-regions ======================= Everything in MTLDP is organised around a **region**: a geographic bounding box together with one JSON configuration file, one raw data directory and one processed data directory. All command line tools and the ``Region`` classes of ``mtldp.utils.config`` take the configuration file as their single entry point and derive every other path from it. The configuration file ---------------------- A configuration file is created with ``create_config_file`` (see :ref:`meta_utils_cli`) and lives in a ``configs`` directory: .. code-block:: json { "region_id": "oakland_co", "city_id": "oakland", "bbox": [-83.4558, 42.4249, -83.1054, 42.6011], "raw_dir": "raw_data", "processed_dir": "processed_data", "junction_id_list": ["62590214", "62590221"], "date_str_list": ["2025-03-03", "2025-03-04"], "timezone": "America/Detroit", "mode": "preprocessing", "search_link_length_threshold": 2000, "min_link_entry_threshold": 200 } .. list-table:: :header-rows: 1 :widths: 30 70 * - Key - Meaning * - ``region_id`` - Unique name of the region. It is the name of the region directories and the ``region_name`` of the traffic network. * - ``city_id`` - Name of the city the region belongs to. * - ``bbox`` - Bounding box as ``[lon_min, lat_min, lon_max, lat_max]``. All bounding boxes stored by MTLDP use this longitude-first order; only the ``--bbox`` command line option accepts the OpenStreetMap order ``lat_min,lon_min,lat_max,lon_max`` and converts it. * - ``raw_dir`` / ``processed_dir`` - Roots of the raw and processed data. Relative paths are resolved against the *project root* (see below). * - ``junction_id_list`` - Node IDs of the intersections of interest. Used to filter trajectories when loading them, by ``process_trajs --filter-junctions``, by ``copy_regions`` and by the ``signalized_selected_junctions.json`` export. * - ``date_str_list`` - Default list of dates (``YYYY-MM-DD``) an ``AppRegion`` loads when no dates are given. * - ``timezone`` - IANA time zone used to derive local dates and times of day from UTC timestamps. * - ``mode`` - ``preprocessing`` for regions that process raw data (``ProdRegion``), ``application`` for regions that only consume processed data (``AppRegion``). * - ``search_link_length_threshold`` - Optional, default 2000 m. How far a signalized link is extended through unsignalized nodes when the signalized network is built (:ref:`signalized_network`). * - ``min_link_entry_threshold`` - Optional, default 200 m. Minimum length kept for the entry of a signalized link; it is capped at half of the search threshold. Project root and path resolution ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The location of the configuration file defines the project root, so that a whole project can be moved or mounted elsewhere without editing any file: .. code-block:: text / ├── configs/ │ ├── oakland_co.json master region │ └── oakland_co_subs/ sub-regions of oakland_co │ ├── 0_0.json │ └── 0_1.json ├── raw_data/ │ ├── oakland_co/ │ └── oakland_co_subs/ │ ├── 0_0/ │ └── 0_1/ └── processed_data/ ├── oakland_co/ └── oakland_co_subs/ ├── 0_0/ └── 0_1/ * For ``/configs/.json`` the project root is the parent of ``configs``. * For ``/configs/_subs/.json`` the project root is the parent of ``configs`` as well (two levels up). * A relative configuration path such as ``configs/oakland_co.json`` is accepted only when the current working directory is the project root; otherwise an absolute path is required. Region classes -------------- ``mtldp.utils.config`` exposes three classes. ``Region`` Base class. Parses the configuration, resolves ``raw_dir`` and ``processed_dir``, builds the ``BoundingBox`` and provides ``convert_GPS_to_xy`` / ``convert_xy_to_GPS`` (local metric coordinates with the south-west corner of the bounding box as origin). ``ProdRegion`` Region in ``preprocessing`` mode. ``input`` is a ``RawPathManager`` and ``output`` a ``ProcessedPathManager`` (both described below). ``network`` holds the full ``Network`` loaded from ``network/traffic_network.pickle`` when it exists (``load_network=False`` skips the load). This is the class used by all pipeline commands. ``AppRegion`` Region in ``application`` mode. It only knows the processed tree (``ppm``) and loads data through a ``ProcessedDataLoader`` (``pdl``). On construction it loads the **signalized** network into ``network`` and, unless ``load_spat=False``, the SPaT pickle into ``spat``. Trajectories are loaded on demand: * ``load_trajs_dict(date_str_list, junction_id_list, overwrite_buffer, load_points, load_row_num, signalized)`` returns a ``TrajectoryDict`` and caches each day as a pickle under ``trajectories/trajectories/cache/`` so that the second load is fast; * ``load_trajs_df(date_str_list, junction_id_list, signalized)`` returns a single ``pandas.DataFrame`` of trajectory rows; * ``load_stop_bar_by_movement``, ``load_lane_num_by_movement``, ``load_lane_num_coef_by_movement``, ``load_stop_sign_adjustment_by_movement`` and the per-period ``load_start_loss_by_movement``, ``load_clock_shift_by_movement``, ``load_penetration_rate_by_movement`` and ``load_manually_adjustment_by_movement`` read the JSON files of the ``calibration`` directory and return an empty dict when the file does not exist. Both ``ProdRegion`` and ``AppRegion`` offer ``create_directory_structure_and_template_files()``, which is what ``create_region`` calls. Directory layout of a region ---------------------------- The two path managers of ``mtldp.utils.config.PathManager`` define the layout. Every attribute that ends in ``dir`` is created as a directory and every attribute that ends in ``path`` is touched as an empty file by ``create_region``. Raw data (``RawPathManager``) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: text // ├── raw_map/ │ ├── map.osm OpenStreetMap XML of the region (input of build_traffic_network) │ ├── map_bbox_dict.json tiles of the region, written by the OSM downloader (see below) │ ├── arterial.json arterial definitions │ └── overwrite/ │ ├── node.csv manual attribute overrides, one file per element type │ ├── link.csv │ ├── segment.csv │ └── movement.csv ├── raw_spat/ the seven SPaT CSV files (optionally raw_spat/estimate/) └── raw_trajs/ raw GPS files delivered by the vendor (CSV or Parquet) Processed data (``ProcessedPathManager``) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: text // ├── network/ │ ├── traffic_network.pickle the Network object (loaded by ProdRegion / AppRegion) │ ├── traffic_network.json GeoJSON layers of the whole network │ ├── signalized_traffic_network.json GeoJSON layers of the signalized network │ ├── signalized_arterial.json …restricted to the arterials │ ├── signalized_selected_junctions.json …restricted to junction_id_list │ ├── filtered.osm directed OSM XML consumed by the map matcher │ ├── shapefile/ shapefile export of the network │ ├── csv/ tabular export of the network elements │ ├── overwrite.json merged overrides actually applied to the network │ ├── ubodt.txt shortest-path table cached by the map matcher │ ├── raw.osm, static_net.json, bbx.json │ └── map.log build log ├── trajectories/ │ ├── raw_pts/_raw.csv output of split_points │ ├── points/_matched.csv output of match_points_to_map (Feather binary) │ └── trajectories/ │ ├── _trajs.csv output of process_trajs (mode all) │ ├── _signalized_trajs.csv output of process_trajs (mode signalized) │ └── cache/_trajs.pkl pickle cache written by AppRegion.load_trajs_dict ├── spat/spat.pickle output of parse_spat ├── calibration/ stop_bar.json, lane_num.json, lane_num_coef.json, │ └── cache/ stop_sign_adjustment.json and per date/TOD files ├── figures/pi/ figure outputs of applications ├── tables/pi/, tables/spat_est/ table outputs of applications └── ui/ figures and HTML pages of the review UI ``ProcessedPathManager`` also provides path helpers for the application layer, e.g. ``pi_table_path(start, end, level, prefix, suffix)`` under ``tables/pi/_/``, ``coord_diag_table_dir`` and ``green_diag_table_dir`` under ``tables/``, and the ``ui_*`` helpers under ``ui/_/``. Calibration files that depend on a period are named ``__.json`` (for example ``2025-03-03_7-9_start_loss.json``). .. _sub_regions: Sub-regions ----------- Why regions are tiled ~~~~~~~~~~~~~~~~~~~~~ The cost of the pipeline grows quickly with the size of the map: the Overpass API rejects or times out on large downloads, and the Fast Map Matching algorithm pre-computes a shortest-path table (UBODT) whose size and memory footprint grow with the number of edges. A county-scale region is therefore processed as a set of **sub-regions**: tiles of the master bounding box that are each a complete, independent region with their own map, network and trajectory files. The master region keeps the shared inputs (the full map, the tile list, the arterials and the overrides), and every pipeline command that receives the master configuration fans out over the tiles. How the tiles are defined ~~~~~~~~~~~~~~~~~~~~~~~~~ The tiling is decided when the map is downloaded (``create_region --download-osm``). The ``OSMDownloader`` of ``mtldp.utils.data_io`` splits the master bounding box into a grid whenever its latitude or longitude span exceeds 0.1 degrees: the number of rows is ``int(lat span / 0.1)`` and the number of columns ``int(lon span / 0.1)``, and the span is divided evenly. Each tile is downloaded separately, with a 200 m buffer (``MAP_BBOX_EXTEND_METER``) around it so that roads crossing a tile border are complete on both sides, and saved as ``raw_map/map__.osm``. When the box is small enough for a single tile the file is simply ``raw_map/map.osm``. In both cases the downloader writes ``raw_map/map_bbox_dict.json``, which maps the tile ID to the *unbuffered* tile bounding box: .. code-block:: json { "0_0": [-83.4558, 42.4249, -83.2806, 42.5130], "0_1": [-83.2806, 42.4249, -83.1054, 42.5130], "1_0": [-83.4558, 42.5130, -83.2806, 42.6011], "1_1": [-83.2806, 42.5130, -83.1054, 42.6011] } This file is the single source of truth for the sub-regions. It can also be written or edited by hand to define tiles of arbitrary shape, as long as the IDs are valid file names. Where the sub-regions live ~~~~~~~~~~~~~~~~~~~~~~~~~~ Each tile becomes a region in its own right: * its configuration is ``configs/_subs/.json``, with ``region_id`` equal to the tile ID and ``city_id``, ``mode``, ``timezone`` and ``date_str_list`` inherited from the master. ``junction_id_list`` starts empty and is meant to be edited by hand; * its ``raw_dir`` is ``/_subs`` and its ``processed_dir`` is ``/_subs``, so its data sits in ``raw_data/_subs//`` and ``processed_data/_subs//`` with exactly the layout described above; * its ``raw_map/map.osm`` is the tile's map, cut from the master map with the same 200 m buffer. Creating the sub-regions ~~~~~~~~~~~~~~~~~~~~~~~~ There are two ways to materialise the tiles: * ``create_region -c configs/.json --download-osm --subregion`` does it right after the download, copying each ``map_.osm`` into the sub-region; * ``create_sub_regions -c configs/.json`` (re)creates them from ``map_bbox_dict.json`` at any later time. It cuts each tile out of the master ``map.osm`` (so manual edits made to the master map are propagated), merges the master's overrides and arterials into each tile, and can copy the processed trajectories with ``--copy-trajs``. Existing tile configurations are kept unless ``--overwrite`` is given; ``--exclude`` skips tiles. Fan-out rule of the commands ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Every command that accepts a configuration file resolves the list of regions to work on with the same rule (``load_sub_regions_config_dir_and_bbox_dict`` in ``mtldp.utils.config.Region``): #. read ``raw_map/map_bbox_dict.json`` of the given region; when the file does not exist the region is treated as its own single tile; #. when the dictionary has exactly one entry, the region **is** the tile: the command works on the configuration it was given; #. when it has several entries, the command iterates over ``configs/_subs/.json`` for every tile in the dictionary. Consequences of this rule: * passing the master configuration processes all sub-regions; passing a sub-region configuration processes that sub-region only (a sub-region has no ``map_bbox_dict.json`` of its own); * ``build_traffic_network`` builds the tiles in parallel, one process per tile, and accepts ``--exclude-sub-regions``; * ``split_points`` reads every raw trajectory file once and writes the points into the tile whose bounding box contains them. Because the tiles do not overlap, a point belongs to exactly one tile, while the buffered maps guarantee that the network of each tile extends beyond its points; * ``match_points_to_map`` and ``process_trajs`` loop over the tiles sequentially, each tile with its own network and its own cached ``ubodt.txt``; * ``delete_processed_trajs`` and ``draw_region_bbox`` also operate on all tiles. A trip that crosses a tile border is split between the two tiles at the border. Trajectories are computed per movement, so only the movements whose approach straddles the border are affected; if this matters for an application, the tiles can be redrawn in ``map_bbox_dict.json`` so that the borders fall away from the intersections of interest. Master region versus sub-regions in applications ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The master region itself does not get a network or trajectories when it is tiled: its processed directory only exists to hold the ``_subs`` tree. Applications therefore open the sub-region configuration (``AppRegion('configs/_subs/.json')``) that contains their intersections. To study a corridor that is spread over several tiles or several regions, ``copy_regions`` assembles a new, self-contained region from the source regions, merging their maps, overrides, arterials and processed trajectories for the requested junctions.