Goals and architecture

Goals

• Share processed data in a well-defined, stable data structure so that every application starts from the same network, trajectories and signal timing;

• provide the common utilities (loading, filtering, aggregation, plotting) that research applications need, so that they are written once;

• avoid duplicated development on raw data pre-processing, which is expensive and error-prone;

• make quality control and long-term maintenance possible through a documented pipeline whose intermediate results can be inspected.

Architecture

The architecture of MTLDP.

MTLDP is layered. mtldp.meta defines the data structures, mtldp.utils provides the functions that work on them, and mtldp.preproc produces them from raw data. Applications and the data production pipelines are built on top of these three packages and never touch raw data directly.

Repository	Content
mtldp-meta-utils	mtldp.meta (networks, trajectories, SPaT) and mtldp.utils (regions, I/O, aggregation, time and geo utilities, filters, visualisation, database, command line tools).
mtldp-preproc	mtldp.preproc: OSM parsing and network building, map matching, trajectory index calculation, and the pipeline commands.
mtldp-review	Web tool to review the network built for a region and record corrections.
mtldp-doc	This documentation and the example notebooks.
Research applications	Signal diagnosis and optimisation, traffic models (for example the OSaaS paper model), network analysis; each in its own repository.
Data production	Offline and real-time pipelines (osaas-pipeline, osaas-real-time) that run the pre-processing at scale.

Two workflows

Data production: raw map, trajectory and SPaT data are processed once into the standard data structures.

Research applications: processed data is loaded through AppRegion and analysed with the utilities, without touching raw data.

The unit of work in both workflows is the region, a bounding box with one configuration file and two data directories. Large regions are tiled into sub-regions that are processed independently; Regions and sub-regions describes this design.