.. _meta_utils_cli: Command line tools of ``mtldp-meta-utils`` ========================================== Installing ``mtldp-meta-utils`` registers the console scripts listed below (they are the ``entry_points`` of the package). All of them are built with `Typer `_, so every command accepts ``--help`` and prints a formatted option table. Options that are marked *prompt* are asked interactively when they are not given on the command line. Most commands take a region configuration file (see :ref:`regions`). Whenever a command accepts the configuration of a **master region**, it automatically fans out over all of its sub-regions. ``mtldp`` --------- .. code-block:: bash (mtldp) $ mtldp Prints the list of all available commands, in the order in which they are normally used. The commands of ``mtldp.preproc`` are listed only when that package is installed in the same environment. ``create_config_file`` ---------------------- .. code-block:: bash (mtldp) $ create_config_file -n my_region -c ann_arbor -m preprocessing \ -b 42.2149,-83.8283,42.3318,-83.6472 Creates the JSON configuration file of a region at ``/.json``. The command refuses to overwrite an existing file. Every option can be given on the command line or answered interactively. .. list-table:: :header-rows: 1 :widths: 28 18 54 * - Option - Default - Description * - ``--region``, ``-n`` (required, prompt) - - Unique region ID. It becomes the file name of the configuration and the name of the region directories. * - ``--city``, ``-c`` (required, prompt) - - City ID the region belongs to. It is stored in the network and used as a prefix in exported tables. * - ``--config-dir``, ``-d`` - ``configs`` - Directory in which the configuration file is written. * - ``--mode``, ``-m`` (prompt) - ``application`` - ``application`` or ``preprocessing``. It decides which region class (``AppRegion`` or ``ProdRegion``) and which directory tree ``create_region`` builds. * - ``--bbox``, ``-b`` (prompt) - - Bounding box of the region as ``lat_min,lon_min,lat_max,lon_max`` (the order used by OpenStreetMap). It is stored in the file as ``[lon_min, lat_min, lon_max, lat_max]``. * - ``--start-date``, ``-s`` / ``--end-date``, ``-e`` (prompt) - empty - Inclusive date range (``YYYY-MM-DD``). It is expanded to the ``date_str_list`` entry of the configuration, which is the default date list of ``AppRegion``. * - ``--junction-ids``, ``-j`` (prompt) - empty - Comma separated junction (node) IDs of interest, stored as ``junction_id_list``. * - ``--timezone``, ``-t`` (prompt) - ``America/Detroit`` - IANA time zone used to convert timestamps into local date and time of day. * - ``--raw-dir``, ``-r`` (prompt) - ``raw_data`` - Root of the raw data, relative to the project root (the parent of ``configs``). * - ``--processed-dir``, ``-p`` (prompt) - ``processed_data`` - Root of the processed data, relative to the project root. The generated file looks like this: .. code-block:: json { "region_id": "my_region", "city_id": "ann_arbor", "bbox": [-83.8283, 42.2149, -83.6472, 42.3318], "raw_dir": "raw_data", "junction_id_list": [], "date_str_list": [], "processed_dir": "processed_data", "timezone": "America/Detroit", "mode": "preprocessing" } ``create_region`` ----------------- .. code-block:: bash (mtldp) $ create_region -c configs/my_region.json --download-osm --subregion Creates the directory tree and the empty template files of a region from its configuration (the raw *and* processed trees for a ``preprocessing`` region, the processed tree only for an ``application`` region). .. list-table:: :header-rows: 1 :widths: 28 18 54 * - Option - Default - Description * - ``--config``, ``-c`` (required, prompt) - - Configuration file of the region. * - ``--download-osm`` - off - Download the OpenStreetMap data of the bounding box from the Overpass API into ``raw_map/map.osm``. Only effective for ``preprocessing`` regions. A large bounding box is split into tiles (see :ref:`sub_regions`) and ``raw_map/map_bbox_dict.json`` is written. * - ``--subregion``, ``-s`` - off - When the download produced more than one tile, also create one sub-region per tile: a configuration in ``configs/_subs/``, its directory tree, and a copy of the tile's map as the sub-region's ``map.osm``. ``create_sub_regions`` ---------------------- .. code-block:: bash (mtldp) $ create_sub_regions -c configs/my_region.json --exclude 0_3,1_3 Creates (or refreshes) the sub-regions of a master region from the tiles listed in ``raw_map/map_bbox_dict.json``. For every tile the command #. writes ``configs/_subs/.json`` (city, mode, time zone and ``date_str_list`` are inherited from the master; ``junction_id_list`` starts empty); #. creates the raw and processed directory trees of the sub-region under ``/_subs/`` and ``/_subs/``; #. builds the sub-region's ``map.osm`` by sorting and merging the master map with ``osmium`` and cutting it to the sub-region bounding box (plus the map buffer) with *complete ways* semantics, so roads crossing the tile border are kept whole; #. merges the master's ``overwrite/*.csv`` files and ``network/overwrite.json`` into the sub-region (entries are kept when they touch a junction of the sub-region's ``junction_id_list``, or all of them when that list is empty); #. copies the arterials of ``arterial.json`` that touch the sub-region's junctions; #. optionally copies the processed trajectories of the sub-region's junctions. Existing sub-region configurations are reused, so manual edits such as ``junction_id_list`` are preserved unless ``--overwrite`` is given. Building the traffic network of each sub-region is a separate step (``build_traffic_network``). .. list-table:: :header-rows: 1 :widths: 28 18 54 * - Option - Default - Description * - ``--master-config``, ``-c`` (required) - - Configuration file of the master region. * - ``--exclude`` - empty - Comma separated sub-region IDs to skip. * - ``--overwrite`` - off - Regenerate the configuration files of existing sub-regions from the master configuration. * - ``--copy-trajs`` / ``--no-copy-trajs`` - ``--no-copy-trajs`` - Copy the processed trajectories of the master region into each sub-region, filtered by the sub-region's ``junction_id_list``. .. note:: The map operations rely on the ``osmium`` command line tool (`osmium-tool `_), which must be on the ``PATH``. ``copy_regions`` ---------------- .. code-block:: bash (mtldp) $ copy_regions --source-configs configs/a.json,configs/b.json -n corridor_x \ -j 1001,1002,1003 -b 42.30,-83.75,42.33,-83.70 Creates a **new region** from one or more existing source regions. Typical use: extract a corridor with a handful of junctions out of a large processed region, so that it can be re-built and analysed on its own. The command #. copies the first source configuration to ``/.json`` and replaces the region ID, bounding box and junction list; #. creates the directory trees of the new region; #. merges the source maps with ``osmium`` and cuts them to the new bounding box (plus buffer); #. merges the overwrite CSV files and ``overwrite.json`` of the sources, keeping the entries related to the requested junctions; #. merges the arterials of the sources that touch the requested junctions; #. copies the processed trajectories of the requested junctions from every source, then de-duplicates them by ``traj_id`` and ``timestamp`` and sorts them. The traffic network of the new region still has to be built with ``build_traffic_network``. .. list-table:: :header-rows: 1 :widths: 28 18 54 * - Option - Default - Description * - ``--source-configs`` (required) - - Comma separated configuration files of the source regions. * - ``--region``, ``-n`` (required) - - ID of the new region. * - ``--target-config-dir``, ``-d`` - ``configs`` - Directory of the new configuration file. * - ``--bbox``, ``-b`` (prompt) - - Bounding box of the new region, ``lat_min,lon_min,lat_max,lon_max``. Make it at least one block larger than the area of interest so that the approaches of the border junctions are complete. * - ``--junction-ids``, ``-j`` - empty - Comma separated junction IDs. Trajectories are always filtered by this list, so an empty list yields empty trajectory files. * - ``--overwrite`` - off - Overwrite an existing target configuration. * - ``--copy-trajs`` / ``--no-copy-trajs`` - ``--copy-trajs`` - Whether to copy the processed trajectories. ``extract_osm_map`` ------------------- .. code-block:: bash (mtldp) $ extract_osm_map --source-configs configs/a.json,configs/b.json \ --target-config configs/c.json Only the map part of ``copy_regions``: sorts and merges the ``map.osm`` of the source regions with ``osmium`` and writes the part that lies inside the target region's bounding box (expanded by the map buffer) to the target's ``raw_map/map.osm``. Ways that cross the border are kept complete. .. list-table:: :header-rows: 1 :widths: 28 18 54 * - Option - Default - Description * - ``--source-configs`` (required) - - Comma separated configuration files of the source regions. * - ``--target-config`` (required) - - Configuration file of the target region. ``delete_processed_trajs`` -------------------------- .. code-block:: bash (mtldp) $ delete_processed_trajs -c configs/my_region.json -d points,trajectories Deletes intermediate trajectory directories of a region and of all its sub-regions, then recreates them empty. The list of directories is printed first and nothing is deleted until the literal answer ``Yes`` is typed. .. list-table:: :header-rows: 1 :widths: 28 18 54 * - Option - Default - Description * - ``--config``, ``-c`` (required) - - Configuration file of the (master) region. * - ``--dir-types``, ``-d`` (required) - - Comma separated subset of ``raw_pts`` (output of ``split_points``), ``points`` (output of ``match_points_to_map``), ``trajectories`` (output of ``process_trajs``, including its cache) and ``cache`` (the pickle cache of the trajectories only). ``draw_region_bbox`` -------------------- .. code-block:: bash (mtldp) $ draw_region_bbox -c configs/my_region.json Draws the bounding box of the master region and of each of its sub-regions on an interactive `Plotly `_ map, which is a quick way to check the tiling before downloading data. .. list-table:: :header-rows: 1 :widths: 28 18 54 * - Option - Default - Description * - ``--config``, ``-c`` (required) - - Configuration file of the master region.