Command line tools of mtldp-meta-utils

Installing mtldp-meta-utils registers the console scripts listed below (they are the entry_points of the package). All of them are built with Typer, so every command accepts --help and prints a formatted option table. Options that are marked prompt are asked interactively when they are not given on the command line.

Most commands take a region configuration file (see Regions and sub-regions). Whenever a command accepts the configuration of a master region, it automatically fans out over all of its sub-regions.

mtldp

(mtldp) $ mtldp

Prints the list of all available commands, in the order in which they are normally used. The commands of mtldp.preproc are listed only when that package is installed in the same environment.

create_config_file

(mtldp) $ create_config_file -n my_region -c ann_arbor -m preprocessing \
             -b 42.2149,-83.8283,42.3318,-83.6472

Creates the JSON configuration file of a region at <config-dir>/<region>.json. The command refuses to overwrite an existing file. Every option can be given on the command line or answered interactively.

Option	Default	Description
--region, -n (required, prompt)		Unique region ID. It becomes the file name of the configuration and the name of the region directories.
--city, -c (required, prompt)		City ID the region belongs to. It is stored in the network and used as a prefix in exported tables.
--config-dir, -d	configs	Directory in which the configuration file is written.
--mode, -m (prompt)	application	application or preprocessing. It decides which region class (AppRegion or ProdRegion) and which directory tree create_region builds.
--bbox, -b (prompt)		Bounding box of the region as lat_min,lon_min,lat_max,lon_max (the order used by OpenStreetMap). It is stored in the file as [lon_min, lat_min, lon_max, lat_max].
--start-date, -s / --end-date, -e (prompt)	empty	Inclusive date range (YYYY-MM-DD). It is expanded to the date_str_list entry of the configuration, which is the default date list of AppRegion.
--junction-ids, -j (prompt)	empty	Comma separated junction (node) IDs of interest, stored as junction_id_list.
--timezone, -t (prompt)	America/Detroit	IANA time zone used to convert timestamps into local date and time of day.
--raw-dir, -r (prompt)	raw_data	Root of the raw data, relative to the project root (the parent of configs).
--processed-dir, -p (prompt)	processed_data	Root of the processed data, relative to the project root.

The generated file looks like this:

{
  "region_id": "my_region",
  "city_id": "ann_arbor",
  "bbox": [-83.8283, 42.2149, -83.6472, 42.3318],
  "raw_dir": "raw_data",
  "junction_id_list": [],
  "date_str_list": [],
  "processed_dir": "processed_data",
  "timezone": "America/Detroit",
  "mode": "preprocessing"
}

create_region

(mtldp) $ create_region -c configs/my_region.json --download-osm --subregion

Creates the directory tree and the empty template files of a region from its configuration (the raw and processed trees for a preprocessing region, the processed tree only for an application region).

Option	Default	Description
--config, -c (required, prompt)		Configuration file of the region.
--download-osm	off	Download the OpenStreetMap data of the bounding box from the Overpass API into raw_map/map.osm. Only effective for preprocessing regions. A large bounding box is split into tiles (see Sub-regions) and raw_map/map_bbox_dict.json is written.
--subregion, -s	off	When the download produced more than one tile, also create one sub-region per tile: a configuration in configs/<region>_subs/, its directory tree, and a copy of the tile’s map as the sub-region’s map.osm.

create_sub_regions

(mtldp) $ create_sub_regions -c configs/my_region.json --exclude 0_3,1_3

Creates (or refreshes) the sub-regions of a master region from the tiles listed in raw_map/map_bbox_dict.json. For every tile the command

1. writes configs/<master>_subs/<sub_region>.json (city, mode, time zone and date_str_list are inherited from the master; junction_id_list starts empty);

2. creates the raw and processed directory trees of the sub-region under <raw_dir>/<master>_subs/ and <processed_dir>/<master>_subs/;

3. builds the sub-region’s map.osm by sorting and merging the master map with osmium and cutting it to the sub-region bounding box (plus the map buffer) with complete ways semantics, so roads crossing the tile border are kept whole;

4. merges the master’s overwrite/*.csv files and network/overwrite.json into the sub-region (entries are kept when they touch a junction of the sub-region’s junction_id_list, or all of them when that list is empty);

5. copies the arterials of arterial.json that touch the sub-region’s junctions;

6. optionally copies the processed trajectories of the sub-region’s junctions.

Existing sub-region configurations are reused, so manual edits such as junction_id_list are preserved unless --overwrite is given. Building the traffic network of each sub-region is a separate step (build_traffic_network).

Option	Default	Description
--master-config, -c (required)		Configuration file of the master region.
--exclude	empty	Comma separated sub-region IDs to skip.
--overwrite	off	Regenerate the configuration files of existing sub-regions from the master configuration.
--copy-trajs / --no-copy-trajs	--no-copy-trajs	Copy the processed trajectories of the master region into each sub-region, filtered by the sub-region’s junction_id_list.

Note

The map operations rely on the osmium command line tool (osmium-tool), which must be on the PATH.

copy_regions

(mtldp) $ copy_regions --source-configs configs/a.json,configs/b.json -n corridor_x \
             -j 1001,1002,1003 -b 42.30,-83.75,42.33,-83.70

Creates a new region from one or more existing source regions. Typical use: extract a corridor with a handful of junctions out of a large processed region, so that it can be re-built and analysed on its own. The command

1. copies the first source configuration to <target-config-dir>/<region>.json and replaces the region ID, bounding box and junction list;

2. creates the directory trees of the new region;

3. merges the source maps with osmium and cuts them to the new bounding box (plus buffer);

4. merges the overwrite CSV files and overwrite.json of the sources, keeping the entries related to the requested junctions;

5. merges the arterials of the sources that touch the requested junctions;

6. copies the processed trajectories of the requested junctions from every source, then de-duplicates them by traj_id and timestamp and sorts them.

The traffic network of the new region still has to be built with build_traffic_network.

Option	Default	Description
--source-configs (required)		Comma separated configuration files of the source regions.
--region, -n (required)		ID of the new region.
--target-config-dir, -d	configs	Directory of the new configuration file.
--bbox, -b (prompt)		Bounding box of the new region, lat_min,lon_min,lat_max,lon_max. Make it at least one block larger than the area of interest so that the approaches of the border junctions are complete.
--junction-ids, -j	empty	Comma separated junction IDs. Trajectories are always filtered by this list, so an empty list yields empty trajectory files.
--overwrite	off	Overwrite an existing target configuration.
--copy-trajs / --no-copy-trajs	--copy-trajs	Whether to copy the processed trajectories.

extract_osm_map

(mtldp) $ extract_osm_map --source-configs configs/a.json,configs/b.json \
             --target-config configs/c.json

Only the map part of copy_regions: sorts and merges the map.osm of the source regions with osmium and writes the part that lies inside the target region’s bounding box (expanded by the map buffer) to the target’s raw_map/map.osm. Ways that cross the border are kept complete.

Option	Default	Description
--source-configs (required)		Comma separated configuration files of the source regions.
--target-config (required)		Configuration file of the target region.

delete_processed_trajs

(mtldp) $ delete_processed_trajs -c configs/my_region.json -d points,trajectories

Deletes intermediate trajectory directories of a region and of all its sub-regions, then recreates them empty. The list of directories is printed first and nothing is deleted until the literal answer Yes is typed.

Option	Default	Description
--config, -c (required)		Configuration file of the (master) region.
--dir-types, -d (required)		Comma separated subset of raw_pts (output of split_points), points (output of match_points_to_map), trajectories (output of process_trajs, including its cache) and cache (the pickle cache of the trajectories only).

draw_region_bbox

(mtldp) $ draw_region_bbox -c configs/my_region.json

Draws the bounding box of the master region and of each of its sub-regions on an interactive Plotly map, which is a quick way to check the tiling before downloading data.

Option	Default	Description
--config, -c (required)		Configuration file of the master region.