.. _preproc_cli: Command line tools of ``mtldp-preproc`` ======================================= Installing ``mtldp-preproc`` registers the console scripts below (the ``entry_points`` of the package). They implement the raw data processing pipeline described in :ref:`pipeline`. Every command accepts ``--help``. All commands that take ``--config`` accept the configuration of a **master region** and process each of its sub-regions in turn (or in parallel for ``build_traffic_network``); passing the configuration of a single sub-region processes only that sub-region. The two exceptions are ``merge_osm``, which works on plain files, and ``parse_spat``, which works on exactly the region it is given. ``merge_osm`` ------------- .. code-block:: bash (mtldp) $ merge_osm -i raw_data/my_region/raw_map/map_raw.osm \ -o raw_data/my_region/raw_map/map.osm -d 60 Merges signalized intersection nodes of an OpenStreetMap file that lie within a distance threshold of each other into a single node. Large intersections are often mapped as two or four separate ``highway=traffic_signals`` nodes (one per carriageway); MTLDP expects one node per intersection, otherwise the movements and the SPaT mapping cannot be attached to it. This command performs the merge automatically, which replaces the manual editing step in JOSM described in :ref:`download_osm`. .. list-table:: :header-rows: 1 :widths: 28 18 54 * - Option - Default - Description * - ``--input``, ``-i`` (required) - - Input OSM XML file. * - ``--output``, ``-o`` (required) - - Output OSM XML file. * - ``--threshold``, ``-d`` - ``60`` - Maximum distance in meters between two signalized nodes for them to be merged. ``build_traffic_network`` ------------------------- .. code-block:: bash (mtldp) $ build_traffic_network -c configs/my_region.json -e 0_3 Builds the MTLDP traffic network of every sub-region from its ``raw_map`` inputs and writes all derived network files into ``network/`` (see :ref:`pipeline_build_network` for the algorithm and the outputs). Sub-regions are built in parallel, one process per sub-region up to the number of CPU cores. Because the network changed, the cached ``ubodt.txt`` of the map matcher is deleted so that it is regenerated on the next ``match_points_to_map`` run. .. list-table:: :header-rows: 1 :widths: 28 18 54 * - Option - Default - Description * - ``--config``, ``-c`` (required) - - Configuration file of the master region. * - ``--exclude-sub-regions``, ``-e`` - none - Comma separated sub-region IDs to skip. The thresholds that control how far a signalized link is extended upstream and downstream are read from the configuration file: ``search_link_length_threshold`` (default 2000 m) and ``min_link_entry_threshold`` (default 200 m, at most half of the search threshold). ``parse_spat`` -------------- .. code-block:: bash (mtldp) $ parse_spat -c configs/my_region.json Reads the seven SPaT CSV files of ``raw_spat/`` (``version_history.csv``, ``network_mapping_data.csv``, ``tod_event_data.csv``, ``ring_structure_data.csv``, ``dial_split_data.csv``, ``program_week_data.csv`` and ``program_day_data.csv``; their content is described in :ref:`spat_prepare`), builds a ``HistoryRegionSPaT`` object and stores it as ``spat/spat.pickle``. With ``--estimate-spat`` the files are read from ``raw_spat/estimate/`` and the result is written next to the regular pickle with an ``_estimate`` suffix. .. list-table:: :header-rows: 1 :widths: 28 18 54 * - Option - Default - Description * - ``--config``, ``-c`` (required) - - Configuration file of the region whose SPaT is parsed. * - ``--estimate-spat`` / ``--no-estimate-spat`` - ``--no-estimate-spat`` - Parse the estimated SPaT instead of the agency SPaT. ``split_points`` ---------------- .. code-block:: bash (mtldp) $ split_points -c configs/my_region.json -v SL_2025 \ -d /data/raw_trajs/year=2025 Reads the raw GPS trajectory files delivered by the data vendor and redistributes the points into one CSV file **per sub-region and per local date** (``trajectories/raw_pts/_raw.csv``). Each raw file is read once; for each sub-region the points inside the sub-region bounding box are kept, the local ``date``, ``time`` and ``tod`` (time of day, in hours) columns are added using the region's time zone, and the rows are appended to the file of their date. .. list-table:: :header-rows: 1 :widths: 28 18 54 * - Option - Default - Description * - ``--config``, ``-c`` (required) - - Configuration file of the master region. * - ``--format-version``, ``-v`` - ``GM_2024_pre`` - Layout of the raw files: ``GM_2024_pre`` and ``GM_2025`` are CSV deliveries, ``SL_2025`` is a Parquet delivery. Any version starting with ``SL`` is read with the Parquet loader. * - ``--raw-traj-dir``, ``-d`` - ``raw_trajs/`` of the region - Absolute path of the directory holding the raw files. For CSV formats only the files directly inside the directory are read. For Parquet formats ``*.parquet`` files are discovered recursively, so Hive-partitioned trees (``year=…/month=…/day=…/hour=…``) are supported; parts are read one partition directory at a time and marker files (``_SUCCESS``, ``.*.crc``) are ignored. * - ``--bbox-expand``, ``-b`` - ``0`` - Expansion of the sub-region bounding box in meters. A warning is printed when it exceeds the built-in GPS buffer. ``match_points_to_map`` ----------------------- .. code-block:: bash (mtldp) $ match_points_to_map -c configs/my_region.json -s 2025-03-03 -e 2025-03-09 \ --exclude-weekend Runs the Fast Map Matching (FMM) algorithm on the daily point files of every sub-region and writes ``trajectories/points/_matched.csv`` with the matched road element and the map attributes of every point. Despite the ``.csv`` extension these intermediate files are stored in the binary Feather format for speed. See :ref:`pipeline_map_match` for the details. .. list-table:: :header-rows: 1 :widths: 28 18 54 * - Option - Default - Description * - ``--config``, ``-c`` (required) - - Configuration file of the master region. * - ``--start-date``, ``-s`` (required) - - First date to process (``YYYY-MM-DD``). * - ``--end-date``, ``-e`` - none - Last date to process (inclusive). Without it only the start date is processed. * - ``--exclude-weekend`` / ``--include-weekend`` - include - Skip Saturdays and Sundays of the date range. * - ``--exclude-weekday`` / ``--include-weekday`` - include - Skip Mondays to Fridays of the date range. * - ``--use-cache`` / ``--recompute`` - ``--use-cache`` - Reuse the upper-bounded origin destination table (``network/ubodt.txt``) computed by a previous run; ``--recompute`` rebuilds it. ``process_trajs`` ----------------- .. code-block:: bash (mtldp) $ process_trajs -c configs/my_region.json -s 2025-03-03 -e 2025-03-09 \ --process-mode signalized --filter-junctions Cuts the matched points of every sub-region into movement-level trajectories and computes their performance indices (travel time, control delay, number of stops, queue distance, …). The result is one CSV per date: ``trajectories/trajectories/_trajs.csv`` in ``all`` mode, or ``_signalized_trajs.csv`` in ``signalized`` mode. See :ref:`pipeline_process_trajs` for the columns and the definitions. .. list-table:: :header-rows: 1 :widths: 28 18 54 * - Option - Default - Description * - ``--config``, ``-c`` (required) - - Configuration file of the master region. * - ``--start-date``, ``-s`` (required) / ``--end-date``, ``-e`` - - Date range, as for ``match_points_to_map``. * - ``--exclude-weekend`` / ``--exclude-weekday`` - include both - Date filters, as for ``match_points_to_map``. * - ``--process-mode`` - ``all`` - ``all`` computes indices on the complete network; ``signalized`` keeps only the points on the signalized network (the links that enter signalized intersections, see :ref:`signalized_network`) and attributes each trajectory to its downstream signalized junction. * - ``--filter-junctions`` / ``--all-junctions`` - ``--all-junctions`` - Keep only the trajectories whose junction is listed in the ``junction_id_list`` of the sub-region. With an empty list the sub-region is skipped. * - ``--process-details`` / ``--simple-process`` - ``--process-details`` - *Complete* processing computes all performance indices; *simple* processing computes only travel time, average speed, travel distance and distance difference. * - ``--fill-dummy-trips`` / ``--no-dummy-trips`` - ``--fill-dummy-trips`` - Insert trajectories without points for the links a trip must have traversed between two matched links, so that every trip is a feasible path in the network. * - ``--max-away-per-lane`` - ``100`` (``all``) / ``10`` (``signalized``) - Maximum lateral distance of a point from the road centreline, per lane, in meters. Points further away are treated as noise and dropped. * - ``--parallel`` / ``--no-parallel`` - ``--parallel`` - Process the trips of a day in a process pool. The points are split into chunks of complete trips, one per worker. * - ``--cores`` - ``-1`` - Number of worker processes; ``-1`` uses all cores but two. ``eva_trajs_qc`` ---------------- .. code-block:: bash (mtldp) $ eva_trajs_qc -c configs/my_region.json \ -d 2025-03-03,2025-03-09@2025-06-02,2025-06-08 -o qc_march_vs_june Quality control of the processed trajectories. For each date range the daily trajectory metrics (number of trajectories, points, matched share, …) are computed, cached under the region's trajectory cache directory, and plotted side by side so that two or more periods can be compared day by day. .. list-table:: :header-rows: 1 :widths: 28 18 54 * - Option - Default - Description * - ``--config``, ``-c`` (required) - - Configuration file of the region. * - ``--date-pairs``, ``-d`` (required) - - ``@``-separated list of ``start,end`` date ranges. All ranges must have the same number of days and must start on a Monday so that weekdays line up. * - ``--exclude-weekends`` / ``--include-weekends`` - ``--exclude-weekends`` - Whether weekends are skipped when the missing daily metrics are generated. * - ``--junction-ids``, ``-j`` - all junctions - Comma separated junction IDs to include. * - ``--output``, ``-o`` - auto-generated - File name of the figure, without extension. * - ``--fig-width`` / ``--fig-height`` - ``18`` / ``9`` - Figure size in inches. * - ``--fig-format`` / ``--dpi`` - ``png`` / ``300`` - Output format and resolution. Maintenance scripts in ``tools/`` --------------------------------- Two stand-alone scripts live in the ``tools/`` directory of the ``mtldp-preproc`` repository. They are not installed as commands, need no ``mtldp`` import and are run with ``python tools/