.. _meta_reference:

Data structures: ``mtldp.meta``
===============================

``mtldp.meta`` (repository ``mtldp-meta-utils``) contains the data structures shared by every other
part of MTLDP. They are deliberately plain: they store attributes and references between elements
and offer only light helpers. Building, loading, converting and analysing them is the job of
``mtldp.utils`` and ``mtldp.preproc``.

There is no top-level ``mtldp.meta`` namespace to import from; the three sub-packages are imported
directly:

.. code-block:: python

   from mtldp.meta.TrafficNetwork import Network, SignalizedNetwork, Node, Link, Segment, Movement
   from mtldp.meta.Trajectory import Trajectory, TrajectoryDict, Trip
   from mtldp.meta.SPaT import HistoryRegionSPaT, HistoryControllerSPaT


Traffic network
---------------

Element hierarchy and identifiers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. figure:: ../_static/map_related/network.png
   :width: 70%
   :align: center
   :alt: Network elements

   Nodes, segments, links and movements of a network.

A ``Network`` is a set of dictionaries keyed by element ID:

.. list-table::
   :header-rows: 1
   :widths: 16 20 64

   * - Element
     - ID format
     - Meaning
   * - ``OsmWay``
     - OSM way ID
     - A raw OpenStreetMap way; the input from which segments are cut.
   * - ``Node``
     - OSM node ID
     - A point of the map. Typed as ``ordinary``, ``connector`` (segments meet without a turn),
       ``signalized``, ``unsignalized`` or ``end`` (boundary of the network).
   * - ``Segment``
     - ``<way id>0`` / ``<way id>1``
     - One direction of one OSM way with a constant lane count (``0`` is the forward direction of
       the way, ``1`` the backward direction). The finest topological element.
   * - ``Link``
     - ``<upstream node>_<downstream node>``
     - A directed stretch of road between two intersections or end nodes, made of one or more
       segments.
   * - ``Movement``
     - ``<upstream node>_<junction>_<downstream node>``
     - A turn at a junction from an upstream link to a downstream link. The artificial *sink*
       movement that terminates a link at the network boundary is ``<link id>_dest``.
   * - ``Arterial``
     - Arterial name
     - A named corridor, holding one ``OnewayArterial`` per direction of travel.

Every element carries a ``geometry`` (``Geometry``, two parallel lists ``lon`` and ``lat``) and a
``display_geometry`` shifted sideways so that the two directions of a road do not overlap when
plotted.

``Network`` and ``MetaNetwork``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``MetaNetwork`` is the container base class with the dictionaries ``nodes``, ``links``,
``segments``, ``movements`` and ``arterials``, plus ``networkx_graph``, a NetworkX graph of the
links used for shortest-path queries. Its helpers are ``add_node`` / ``add_link`` / ``add_segment`` /
``add_movement``, ``get_movement_from_links(upstream_link_id, downstream_link_id)`` and
``get_link_sink_movement(link_id)``.

``Network(region_name, city_id)`` adds the region metadata, the ``ways`` dictionary, the
``signalized_node_list``, ``unsignalized_node_list`` and ``end_node_list`` of node IDs, and the
``signalized_network`` attribute that holds the derived ``SignalizedNetwork`` described next.

.. _signalized_network:

``SignalizedNetwork``
~~~~~~~~~~~~~~~~~~~~~

The signalized network is the view of the network that signal-related applications work with:
only the signalized intersections are kept as nodes, and the links between them are rebuilt so
that consecutive signals become adjacent. It is derived during ``build_traffic_network`` by
``build_signalized_network`` (``mtldp.preproc``) with two parameters read from the region
configuration:

* ``search_link_length_threshold`` (default 2000 m): starting from each movement of a signalized
  node, the upstream and downstream links are extended through unsignalized and connector nodes
  until another signalized node is reached or this distance is exceeded. Links that belong to an
  arterial are followed without a distance limit so that arterials stay connected;
* ``min_link_entry_threshold`` (default 200 m): the minimum length kept when an extended link has
  to be cut because no signal was found within the search distance.

The result is a ``MetaNetwork`` with the same ID conventions (movement IDs are recomputed from the
new links, and every link keeps its ``sink_movement``), its own ``networkx_graph`` and the
arterials rebuilt on the reduced link set. ``AppRegion`` loads this view as its ``network``;
``process_trajs --process-mode signalized`` computes indices on it.

``Node`` and its subclasses
~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. list-table::
   :header-rows: 1
   :widths: 28 72

   * - Attribute
     - Meaning
   * - ``node_id``, ``name``, ``type``
     - Identifier, optional street-based name and the node type listed above.
   * - ``latitude``, ``longitude``
     - Position, parsed from the OSM attributes.
   * - ``osm_attrib``, ``osm_tags``
     - Raw OSM attributes and tags.
   * - ``upstream_segments``, ``downstream_segments``
     - Segments entering and leaving the node.
   * - ``upstream_links``, ``downstream_links``
     - Links entering and leaving the node.
   * - ``movement_list``
     - Movements through the node (``add_movement`` de-duplicates by ID).

Helpers: ``is_intersection()`` (signalized or unsignalized), ``is_ordinary_node()``,
``get_upstream_junctions()``. The subclasses ``SignalizedNode`` (with ``controller_id``, the key
into the SPaT data), ``UnSignalizedNode``, ``EndNode`` and ``SegmentConnectionNode`` pin the ``type``
and are created from a plain node with ``init_from_node``.

``Segment``
~~~~~~~~~~~

.. list-table::
   :header-rows: 1
   :widths: 28 72

   * - Attribute
     - Meaning
   * - ``segment_id``, ``osm_way``, ``osm_direction_flag``
     - Identifier, the source ``OsmWay`` and ``forward`` / ``backward``.
   * - ``belonged_link``
     - The ``Link`` the segment is part of.
   * - ``upstream_node``, ``downstream_node``, ``node_list``
     - End nodes and all nodes along the segment.
   * - ``upstream_segments``, ``downstream_segments``
     - Connected segments.
   * - ``downstream_directions_info``, ``downstream_directions``
     - Turn letter (``l``, ``s``, ``r``) to downstream segment ID, and the concatenated letters.
   * - ``length``, ``speed_limit``, ``lane_number``, ``lane_assignment``
     - Length in meters, speed limit in m/s, lane count and lane-use string
       (e.g. ``left|through;right``).
   * - ``from_heading``, ``to_heading``, ``from_direction``, ``to_direction``
     - Headings in degrees and cardinal directions (``N``, ``E``, ``S``, ``W``) at both ends.
   * - ``geometry``, ``display_geometry``
     - Polylines.

``Link``
~~~~~~~~

.. list-table::
   :header-rows: 1
   :widths: 28 72

   * - Attribute
     - Meaning
   * - ``link_id``, ``upstream_node``, ``downstream_node``
     - Identifier and end nodes.
   * - ``segment_list``, ``node_list``
     - Ordered segments and nodes.
   * - ``length``, ``speed_limit``, ``heading``, ``from_direction``
     - Length in meters, speed limit, heading and direction of origin.
   * - ``sink_movement``
     - The ``<link id>_dest`` movement.
   * - ``geometry``, ``display_geometry``
     - Polylines.

``to_dict()`` and ``to_df()`` export a link as a dictionary or a one-row ``DataFrame`` with the
references replaced by IDs.

``Movement``
~~~~~~~~~~~~

.. list-table::
   :header-rows: 1
   :widths: 28 72

   * - Attribute
     - Meaning
   * - ``movement_id``, ``node``
     - Identifier and the junction.
   * - ``upstream_link``, ``downstream_link``
     - Entering and exiting links (``downstream_link`` is ``None`` for a sink movement).
   * - ``direction``
     - Turn code: ``l`` left, ``s`` straight, ``r`` right, combinations such as ``ls``, and ``d``
       for a sink movement.
   * - ``movement_index``
     - Standard intersection movement number (the numbering used by signal engineers and by the
       SPaT ``network_mapping_data.csv``): 1 = eastbound left, 2 = westbound through, 3 =
       southbound left, 4 = northbound through, 5 = westbound left, 6 = eastbound through, 7 =
       northbound left, 8 = southbound through; 9 to 12 are the right turns and 13 to 16 are used
       for U-turns and special cases. ``-1`` for sink movements.
   * - ``geometry``, ``display_geometry``
     - Upstream link polyline followed by the downstream link polyline.

.. figure:: ../_static/spat_related/phase_movement_mapping.png
   :width: 55%
   :align: center
   :alt: Movement indices

   Movement indices at a four-leg intersection.

``Arterial``, ``OnewayArterial`` and ``NetworkLinkPath``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``NetworkLinkPath`` is an ordered chain of links in a network: ``link_list``, ``origin_link``,
``destination_link``, the concatenated ``geometry``, the total ``length``, ``distance_by_link``
(cumulative distance at the end of each link) and ``movement_list`` (the movement between each
pair of consecutive links, ``None`` when they are not connected, in which case ``connected`` is
``False``). It is initialised with ``init_from_link_list(link_id_list)`` or
``init_from_od(origin_link_id, destination_link_id)``; ``fill_shortest_path()`` fills gaps between
consecutive links with the shortest path of the ``networkx_graph``.

``Arterial(arterial_id)`` is a corridor with an optional ``ref_node`` and a dictionary ``oneways``
of direction letter to ``OnewayArterial``. ``OnewayArterial`` is a ``NetworkLinkPath`` with a
``direction``, a ``name`` (``"<arterial id> <direction>B"``) and a back-reference
``sup_arterial``. Arterials are defined in ``raw_map/arterial.json`` (see
:ref:`pipeline_build_network`).

``BoundingBox`` and ``Geometry``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``BoundingBox(lon_1, lat_1, lon_2, lat_2)`` normalises its corners into ``min_lon``, ``min_lat``,
``max_lon``, ``max_lat`` (rounded to 5 decimals). ``to_list()`` returns
``[min_lon, min_lat, max_lon, max_lat]``, ``get_expanded_bbox(meters)`` grows the box by a
distance in meters and ``split_into_grids(lat_num, lon_num)`` returns a grid of boxes.

``Geometry`` stores ``lon`` and ``lat`` lists. ``to_string()`` yields ``"lon lat;lon lat;…"``,
``to_coord_list(lat_ahead)`` a list of tuples, and ``+`` concatenates two polylines while merging
the shared joint point.

Constants and types
~~~~~~~~~~~~~~~~~~~

``mtldp.meta.TrafficNetwork.constants`` defines ``MPH_TO_METERS_PER_SEC`` (0.44704),
``MILE_TO_METER`` (1609.34), ``DISPLAY_LANE_INTERVAL`` (4) and ``SEGMENT_SHIFT`` (1.5) used for the
display geometry. ``types`` defines the ``Literal`` aliases ``Direction`` (``E W N S``),
``OsmDirection`` (``forward backward``), ``Turn`` (``l r s ls rs lr lsr``) and ``NodeType``.


Trajectory
----------

Vocabulary
~~~~~~~~~~

MTLDP distinguishes three levels of trajectory data:

* a **point** is one GPS record: ``timestamp``, ``latitude``, ``longitude``, ``speed``, ``error``
  and, after processing, ``distance`` (distance to the stop bar of the junction);
* a **trajectory** is the sequence of points of one vehicle on one movement, i.e. from the moment
  the vehicle enters the link that leads to a junction until it leaves the junction. Its ID is
  ``traj_id``;
* a **trip** (``trip_id``) is one journey of a vehicle: the chain of trajectories it produced on
  consecutive links.

``Trajectory``
~~~~~~~~~~~~~~

``Trajectory(traj_id, trip_id, points_df)`` holds the point table in ``points_df`` and any number
of trajectory-level attributes. Attributes are dynamic: setting ``trajectory.control_delay = 12.3``
registers ``control_delay`` in ``traj_attributes``, and ``TrajectoryDict.get_trajs_df()`` uses that
list to build a table. Helpers: ``get_points_df(attributes)``, ``get_points_attributes(*names)``
(one list per point column), ``set_points_attributes(**columns)``, ``get_traj_attributes(*names)``
and ``get_traj_attributes_dict()``; ``len(trajectory)`` is the number of points.

The canonical attribute names are exported by ``mtldp.meta.Trajectory``:

.. list-table::
   :header-rows: 1
   :widths: 30 70

   * - Constant
     - Attributes
   * - ``basic_trajs_attributes``
     - ``trip_id``, ``traj_id``, ``date``, ``tod``, ``points_num``, ``junction_id``, ``link_id``,
       ``segment_list``, ``movement_id``, ``movement_index``, ``timestamp``, ``avg_error``
   * - ``simple_trajs_attributes``
     - ``travel_time``, ``avg_speed``, ``travel_distance``, ``dis_diff``
   * - ``performance_attributes``
     - the simple attributes plus ``min_dis_to_junction``, ``control_delay``, ``service_level``,
       ``stop_delay``, ``stop_nums``, ``free_arrival_time``, ``arrival_time``, ``free_v``,
       ``queue_dis``, ``stop_details``, ``split_failure``
   * - ``complete_trajs_attributes``
     - ``comments`` plus the performance attributes plus ``distance_list``
   * - ``basic_trajs_table_attributes``
     - the basic attributes plus the point lists ``timestamp_list``, ``latitude_list``,
       ``longitude_list``, ``speed_list``, ``error_list``
   * - ``complete_points_attributes``
     - the point lists plus ``distance_list``

The definition of each performance attribute is given in :ref:`pipeline_process_trajs`.

``TrajectoryDict``
~~~~~~~~~~~~~~~~~~

A ``dict`` of ``traj_id`` to ``Trajectory`` with table helpers: ``get_trajs_df(attributes)`` (one
row per trajectory), ``get_points_df(attributes)`` (all points with ``trip_id`` and ``traj_id``
columns), ``groupby(attribute)`` (a dictionary of ``TrajectoryDict`` per value, e.g. per
``movement_id``), ``sort_by(key)``, ``add_trajectory`` and ``del_trajectory``. It is what
``AppRegion.load_trajs_dict`` returns.

``Trip``
~~~~~~~~

``Trip(trip_id)`` groups the trajectories of one journey in ``trajs_dict`` and can rebuild the
route as a ``NetworkLinkPath`` with ``get_trip_path(network)`` (stored in ``link_path``).
``get_trajs_attrs(*names)`` and ``get_points_attrs(*names)`` concatenate attributes over the
trajectories, optionally dropping the first and last trajectory (``head`` / ``tail``), and
``get_trajs_total(*names)`` sums them. ``trip_geometry``, ``origin_coord`` and
``destination_coord`` describe the whole journey.


SPaT
----

Signal Phase and Timing data is stored as the complete **history** of what every controller was
programmed to do, and resolved on demand into a **period** snapshot for a list of dates and a time
of day window. None of the classes inherit from each other; they nest as shown below.

.. code-block:: text

   HistoryRegionSPaT                                  all controllers of a region, all versions
   └── controller_spat_history_dict {controller_id: HistoryControllerSPaT}
         │  get_controller_spat_period(date_list, time_range)
         ▼
       PeriodControllerSPaT                           one controller, one resolved timing plan
       ├── movement_spat_period_dict {node_id: {movement_index: PeriodMovementSPaT}}
       └── phase_spat_period_dict    {node_id: {phase: PeriodPhaseSPaT}}

   HistoryRegionSPaT.get_corridor_spat_period(...) -> PeriodCorridorSPaT
   └── controller_spat_period_dict {controller_id: PeriodControllerSPaT}

``HistoryRegionSPaT``
~~~~~~~~~~~~~~~~~~~~~

Built by ``parse_spat`` from the seven CSV files of ``raw_spat`` (their content is described in
:ref:`spat_prepare`) and stored as ``spat/spat.pickle``. It keeps the seven tables as
``DataFrame`` attributes, the mapping ``node_controller_dict`` from network node ID to controller
ID (a controller can run several intersections), and one ``HistoryControllerSPaT`` per controller
in ``controller_spat_history_dict``.

Query methods:

* ``get_controller_spat_history(node_id)``;
* ``get_controller_spat_period(node_id, date_list, time_range)`` returns a
  ``PeriodControllerSPaT``;
* ``get_corridor_spat_period(corridor_id, node_id_ls, date_list, time_range)`` returns a
  ``PeriodCorridorSPaT``;
* ``get_tod_event_time_splits(date_ls, node_id_ls=None)`` returns the ``[start, end]`` windows
  (hours) delimited by the time-of-day events of the given controllers, i.e. the natural analysis
  periods to pass as ``time_range``.

``HistoryControllerSPaT``
~~~~~~~~~~~~~~~~~~~~~~~~~

One controller with its ``node_id_ls``, the mappings from phases to movement indices
(``phase_mappings``, inverse ``movement_mappings``), permissive movements
(``perm_movements_mappings``), lane counts (``no_lanes_mappings``) and flashing modes
(``flashing_mappings``), and the versioned programs: ``version_history`` (start dates of the
dial-split, ring-structure and TOD-event versions), ``tod_event_versions`` (version → program day →
list of ``TODEvent``) and ``dial_split_versions`` (version → ``"dial:split"`` → ``DialSplit``).

``get_controller_spat_period(date_ls, time_range)`` resolves the plan in effect:

#. pick, for every date, the latest version that started before the date. The ring-structure
   version must be the same for all dates; differing dial-split or TOD versions only warn;
#. map each date to its program week (``program_week_data.csv``) and program day
   (``program_day_data.csv``, exact date first, otherwise day of week plus ten times the week);
#. select the TOD events of each program day that overlap ``time_range`` and read their
   ``dial:split``;
#. look up the ``DialSplit`` (phase splits, phase modes, cycle, offset, sequence). All dates and
   the whole time range must resolve to a single set of parameters, otherwise an exception is
   raised. The reserved dial split ``5:5`` stands for flashing operation (cycle 0).

``PeriodControllerSPaT``
~~~~~~~~~~~~~~~~~~~~~~~~

The resolved plan of one controller: ``cycle``, ``offset``, ``phase_splits``, ``phase_modes`` and
the ring structure (``ring_phases``, ``next_phases``, ``concurrent_phases``, ``lock_starts``,
``lock_ends``, ``min_splits``, ``ped_min``, ``yellow_intervals``, ``clearance_intervals``,
``channels``, ``ped_channels``). From these it computes ``phase_starts`` (green start of each phase
within the cycle), ``fixed_starts`` / ``fixed_ends`` (whether a phase boundary is fixed or
actuated, from the phase mode) and ``early_phase_starts`` (earliest green start under actuation,
with a variant assuming a minor-street call). ``get_movement_spat_period(node_id,
movement_index)`` returns the leaf object below; ``phase_spat_period_dict[node_id][phase]`` gives
the phase view. The provenance of the plan is kept in ``version_history``, ``program_weeks``,
``program_days``, ``tod_events`` and ``dial_splits``.

``PeriodMovementSPaT`` and ``PeriodPhaseSPaT``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``PeriodMovementSPaT`` is the object most analyses consume: for one ``node_id`` and
``movement_index`` over ``date_ls`` and ``time_range`` it gives ``cycle``, ``offset``,
``green_start``, ``green_duration`` (including yellow and all-red), ``yellow``, ``clearance``,
``early_green_start``, ``early_green_start_minor_street_call``, the controlling ``phase_ls`` and
the flags ``fx_start`` / ``fx_end``. Movements that are also served permissively carry
``permissive_start``, ``permissive_duration``, ``permissive_yellow`` and ``permissive_clearance``.

``PeriodPhaseSPaT`` is the same information keyed by ``phase``, with ``movement_index_ls``,
``permissive_movements`` and ``no_lanes``.

``PeriodCorridorSPaT``
~~~~~~~~~~~~~~~~~~~~~~

Coordination view of several controllers along a corridor: ``controller_spat_period_dict``,
``controller_cycle_lengths``, ``same_cycle_lengths`` (``False`` when the non-flashing controllers
disagree), the common ``cycle_length`` and ``node_offsets``. ``get_movement_spat_period(node_id,
movement_index)`` delegates to the right controller.

Typical use
~~~~~~~~~~~

.. code-block:: python

   from mtldp.utils.config import AppRegion

   region = AppRegion('configs/oakland_co_subs/0_1.json')          # loads network and SPaT
   spat = region.spat

   windows = spat.get_tod_event_time_splits(['2025-03-03', '2025-03-04'], ['62590214'])
   period = spat.get_controller_spat_period('62590214', ['2025-03-03', '2025-03-04'], windows[1])
   nb_through = period.get_movement_spat_period('62590214', 4)
   print(nb_through.cycle, nb_through.offset, nb_through.green_start, nb_through.green_duration)