# Input Data

Data files provide structured inputs, which are necessary to describe e.g., networks, zones, customer demand, vehicles, infrastructure, etc. Data files are loaded along with the modules in the initialization of the simulation (e.g., FleetSimulationBase, the parent class for the core simulation).

The files to be loaded for a simulation can be selected in the scenario config files (see [here](1_scenario_setup.md)).

Input data files are stored in the `data`-folder. Example files can be found there. Generally, data-files are included in git-ignore.
In the following the to the corresponding path specifications for the input files are given while under 'Data Specification' the data format of the files is described.

## Network Data

* {network_name} denotes the title of a network
* there are various routing-modules which are based on different preprocessing scripts; the preprocessed data are also saved in the respective network directory
* the specification of all network csv and geojson files is given in this wiki under `3_input_data/Data Specification/`
* each network has to have following mandatory directory and file structure:
<br/>`data/networks/`
<br/>`data/networks/{network_name}/`
<br/>`data/networks/{network_name}/base/`
<br/>`data/networks/{network_name}/base/nodes.csv`
<br/>`data/networks/{network_name}/base/edges.csv`
<br/>`data/networks/{network_name}/base/nodes_all_infos.geojson`
<br/>`data/networks/{network_name}/base/edges_all_infos.geojson`
* if the coordinate frame is not WGS84, an additional file states the used reference system
<br/>`data/networks/{network_name}/base/crs.info`
* in case network travel times are deterministic, but vary over time, the edge travel times are saved in following structure:
<br/>`data/networks/{network_name}/{scenario_time}/`
<br/>`data/networks/{network_name}/{scenario_time}/edges_td_att.csv`
* additionally, the NetworkTable routing module requires fastest node-to-node travel time and distance tables for each travel time directory
<br/>`data/networks/{network_name}/ff/`
<br/>`data/networks/{network_name}/ff/tables/`
<br/>`data/networks/{network_name}/ff/tables/nn_fastest_distance.npy`
<br/>`data/networks/{network_name}/ff/tables/nn_fastest_travel_time.npy`
<br/>`data/networks/{network_name}/{scenario_time}/tables/`
<br/>`data/networks/{network_name}/{scenario_time}/tables/nn_fastest_distance.npy`
<br/>`data/networks/{network_name}/{scenario_time}/tables/nn_fastest_travel_time.npy`
* network dynamics: this file defines loading of time dependent travel time files or(!) travel time factors (optional)

## Zone Systems

[optional input data]

* spatial aggregation into zones is necessary for several use cases, e.g. vehicle repositioning, pricing, tolling, NFD clustering
* {zone_system_name} denotes the name of a GIS zone division
* in the {network_name} subdirectory, the GIS data are matched to an existing network
* definition of respective file formats in this wiki under `3_input_data/Data Specification/`
* data structure:
<br/> `data/zones/`
<br/> `data/zones/{zone_system_name}/`
<br/> `data/zones/{zone_system_name}/general_information.csv`
<br/> `data/zones/{zone_system_name}/polygon_definition.geojson`
<br/> `data/zones/{zone_system_name}/crs.info`
<br/> `data/zones/{zone_system_name}/{network_name}/`
<br/> `data/zones/{zone_system_name}/{network_name}/node_zone_info.csv`
<br/> `data/zones/{zone_system_name}/{network_name}/edge_zone_info.csv`

## Demand Data

* {data_title} should be a name reflecting the data source
* raw data and scripts that reduce them to an unmatched trip format (see specification for unmatched trip data) should also remain on the server for clarity
<br/> `data/`
<br/> `data/demand/`
<br/> `data/demand/{data_title}/`
<br/> `data/demand/{data_title}/raw`

### Trip/Request/User Data

* the script matching trip data to a given network {network_name} can be found in src/demand/pp/
* see in this wiki under `3_input_data/Data Specification/trips` for a format specification of trips_X.csv (where "X" can be replaced with any title given to the trip file)
* data structure:
<br/> `data/demand/{data_title}/matched/`
<br/> `data/demand/{data_title}/matched/{network_name}/`
<br/> `data/demand/{data_title}/matched/{network_name}/trips_X.csv`

### Demand Forecast Data

* {zone_system_name} refers to the name of a zone-system definition
* {temporal_resolution} refers to the time aggregation given in "hh_mm"
* different forecasts methods are saved as different columns; "trips" refers to a perfect forecast for the given spatio-temporal resolution
* see in this wiki under `3_input_data/Data Specification/agg_X.csv` for a format specification of agg_X.csv (where "X" can be replaced with any title given to the forecast file)
* data structure:
<br/> `data/demand/{data_title}/aggregated/`
<br/> `data/demand/{data_title}/aggregated/{zone_system_name}/`
<br/> `data/demand/{data_title}/aggregated/{zone_system_name}/{temporal_resolution}`
<br/> `data/demand/{data_title}/aggregated/{zone_system_name}/{temporal_resolution}/agg_{X}.csv`
<br/> `data/demand/{data_title}/aggregated/{zone_system_name}/{temporal_resolution}/agg_od_{X}.csv`

## Vehicle Data

* saving vehicle data on the server reduces the time to research for new studies
* specification in this wiki under `3_input_data/Data Specification/vehicle_type.csv`
<br/> `data/vehicles/`
<br/> `data/vehicles/EV_type1_20200411.csv`
<br/> `data/vehicles/EV_type2_20200411.csv`

## Infrastructure Data

* infrastructure data can be used to add additional information to certain network nodes, e.g. access points for customers, boarding points, charging stations, depots, parking spaces
* the format of these data files are specified in the next Section `Data Specification`
* {gis_name} reflects the spatial area of the data and in this directory, all data are referenced by coordinates
* if the data are stored in another reference system then WGS84, a crs.info is saved as well
<br/> `data/infra/`
<br/> `data/infra/{gis_name}`
<br/> `data/infra/{gis_name}/crs.info`
<br/> `data/infra/{gis_name}/access_points.geojson`
<br/> `data/infra/{gis_name}/boarding_points.geojson`
<br/> `data/infra/{gis_name}/public_charging_stations.geojson`
<br/> `data/infra/{gis_name}/depots.geojson`
* After matching to a network, the respective files are saved in csv file in the {network_name} directory subdirectory
<br/> `data/infra/{gis_name}/{network_name}`
<br/> `data/infra/{gis_name}/{network_name}/access_points.csv`
<br/> `data/infra/{gis_name}/{network_name}/boarding_points.csv`
<br/> `data/infra/{gis_name}/{network_name}/public_charging_stations.csv`
<br/> `data/infra/{gis_name}/{network_name}/depots.csv`

## Public Transportation Data

tbc

## Fleet-Control Data

### Active Fleet Size Data

* can be used for simulations with flexible fleet size, where fleet size is time controlled
* specification in this wiki under `3_input_data/Data Specification/active_vehicles.csv`
<br/> `data/fleetctrl/elastic_fleet_size/`
<br/> `data/fleetctrl/elastic_fleet_size/active_vehicle_sample.csv`

### Initial Vehicle Distribution

* can be used to specify the vehicle distribution of the initially created vehicles
* specification in this wiki under `3_input_data/Data Specification/init_veh_dist.csv`
* {network_name} corresponds to the network the nodes of the init distribution are matched onto
<br/> `data/fleetctrl/initial_vehicle_distribution/`
<br/> `data/fleetctrl/initial_vehicle_distribution/{network_name}/init_veh_dist.csv`

### Pricing Data

* can be used to define time dependent elastic pricing or utilization dependent pricing
* {pricing_file} corresponds to the name of the applied pricing_file [possible scenario input]
<br/> `data/fleetctrl/elastic_pricing/{pricing_file}.csv`
* TO-DO

## Data Collection

tbc

# Data specification

## access points

### access_points.csv

Mandatory attributes with explanation of data types.

Column Name | Data Type | Description
-- | -- | --
node_index | int | index of node in network which is a access point (BEWARE: should be unique!)

Definition:
access points are all network nodes where customers are allowed to enter/leave the simulation.
in most cases, these location are not necessary for the simulation itself, but might be helpfull to create
demand files.

## active vehicles
### active_vehicles.csv

Mandatory attributes with explanation of data types.

Column Name | Data Type | Description
-- | -- | --
time | int | simulation time in seconds
share_active_fleet_size | float | share of fleet that should be active at this time.

## agg forecast

### agg_X.csv

Mandatory attributes with the explanation of data types.
{fc method} has to be specified in the scenario inputs with the variable G_FC_TYPE

Column Name | Data Type | Description
-- | -- | --
time | int | simulation time in seconds
zone_id | int | index of zone in zone system (BEWARE: should be unique!)
out {fc method} | float | number of outgoing trips for forecast method {fc method}
in {fc method} | float | number of incoming trips for forecast method {fc method}

The following forecasts methods are defined until now:
fc method | Description
-- | -- 
perfect_trips | these "forecasts" are generated from actual aggregation of trip files, thereby making them perfect forecast for the number of trips with respect to the chosen resolution
perfect_pax | these "forecasts" are generated from actual aggregation of trip files; instead of aggregating the number of trips, the number of passengers are aggregated, though


### agg_od_X.csv

Column Name | Data Type | Description
-- | -- | --
time | int | simulation time in seconds
out_zone_id | int | index of zone in zone system (BEWARE: should be unique!)
in_zone_id | int | index of zone in zone system (BEWARE: should be unique!)
{fc_method} | float | number of trips from out_zone_id to in_zone_id for forecast method {fc method}

The following forecasts methods are defined until now:
fc method | Description
-- | -- 
perfect_trips | these "forecasts" are generated from actual aggregation of trip files, thereby making them perfect forecast for the number of trips with respect to the chosen resolution
perfect_pax | these "forecasts" are generated from actual aggregation of trip files; instead of aggregating the number of trips, the number of passengers are aggregated, though

## boarding points

### boarding_points.csv

Mandatory attributes with the explanation of data types.

Column Name | Data Type | Description
-- | -- | --
node_index | int | index of node in network which is a boarding point (BEWARE: should be unique!)

Definition:
boarding points are all network nodes where operators can perform boarding processes.
if no boarding points are given/treated directly, mostly all network nodes are considered boarding points.

## charging events

tbc

## depots

### depots.csv

Mandatory attributes with explanation of data types.

Column Name | Data Type | Description
-- | -- | --
charging_station_id | int | unique identifier for each depot
node_index | int | index of node in network (BEWARE: should be unique!)
charging_units | dict_str | power1:number1;power2:number2
max_nr_parking | int | maximum number of vehicles that can park

## initial state

### init_state.csv

* vehicle locations and utilization at the end of a simulation period (and possible the start of the next is recorded)
* the vehicles are already positioned to the final vehicle location; if the vehicle remains on the middle of a link, it is positioned at the start node of this link
* the vehicles are simply counted as blocked
* attribute fields:

column_name | data_type | comment
-- | -- | --
operator_id | int |
vehicle_id | int |
final_node_index | int | 
final_time | int | in seconds; remember to calculate modulo 24*3600 to not block the vehicle for the full next day
final_soc | float | 

## initial vehicle distribution

### init_veh_dist.csv

* initial random distribution of vehicle locations after simulation init
* specifies node indices and their corresponding random probability when initializing mod fleet locations
* attribute fields:

column_name | data_type | comment
-- | -- | --
node_index | int |
probability | float | probability of choosing this node for a vehicle's inititial location

## network data

A routable network consists of nodes and edges. Vehicles travel along edges, which contain the travel information
and nodes are the connections between these edges and represent the positions in the network, where different
edges can be chosen as next part of the route. Hence, they usually represent junctions of a street network.

```diff
- IMPORTANT: the network definition assumes that node indices are numbered from 0..|N-1|!
```

In the following, the columns of the network data files are described. Please refer to documentation/Data_Directory_Structure.md for the correct placement of the respective files.

### nodes.csv

#### Necessary Attributes

Column Name | Data Type | Description
-- | -- | --
node_index | int | ID of node
is_stop_only | bool | False: normal node; True: node can only be used as first or last part of a route leg
pos_x | float | x-position in projected coordinate system > unit: meters
pos_y | float | y-position in projected coordinate system > unit: meters

#### Optional Attributes

Column Name | Data Type | Description
-- | -- | --
node_order | int | only required for contraction hierarchy

### edges.csv

#### Necessary Attributes

Column Name | Data Type | Description
-- | -- | --
from_node | int | ID of origin node of a street edge
to_node | int | ID of destination node of a street edge
distance | float | length of street edge in meters
travel_time | float | travel duration on street edge in seconds

#### Optional Attributes

Column Name | Data Type | Description
-- | -- | --
shortcut_def | str | only for contraction hierarchy; use “;” as separator between IDs
source_edge_id | str | OSM-Edge ID, Aimsun Section ID; can be "-" separated elements as well

### crs.info

epsg:code

* This file only contains one line and contains the epsg-code 'code', which is valid for the pos_x, pos_y in the nodes.csv.

### edges_td_att.csv

This file specifies edge travel times at specific simulation times. 

Column Name | Data Type | Description
-- | -- | --
from_node | int | ID of origin node of a street edge
to_node | int | ID of destination node of a street edge
edge_tt | float | travel duration on street edge in seconds

### NN_FASTEST_TT.NPY / NN_FASTEST_DISTANCE.NPY

Fully preprocessed (according to the fastest route) node-to-node travel time or distance tables are saved as 2D-Numpy arrays.
The first index (row index) represents the origin node, the second index represents the destination node.
The data entries (travel time/distance) are of type float.
These files are saved under scenario_dir/tables/x.npy, where scenario_dir=ff for free-flow conditions.

### Network Dynamics Files

These files can be used

* to define loading of corresponding travel time files at given simulation time (column "travel_time_folder")

or(!)

* to scale all network travel times with certain factors according to the simulation time. This input is used by the 'NetworkTTMatrix' module. (column "travel_time_factor")

Column Name | Data Type | Description
-- | -- | --
simulation_time | int | simulation time in seconds
travel_time_folder | str | corresponding folder name of travel time directory to be used from this simulation time on
travel_time_factor | float | general travel time factor that is used for complete network

```diff
- IMPORTANT: only one of the columns travel_time_folder/travel_time_factor is allowed to be given!
```

### Partially preprocessed data

Partially preprocessed (according to the fastest route) node-to-node travel time or distance tables are saved as 2D-Numpy arrays.

The first index (row index) represents the origin node, the second index represents the destination node.
The data entries (travel time/distance) are of type float.

The travel time matrix is called tt_matrix.npy, the distance matrix is called dis_matrix.npy

These files are saved in the corresponding travel time folders; free-flow condition is stored in the base-folder.
Note that only the travel times/distances between the first x nodes are stored. x is defined by the shape of the matrix.

These matrices are used by NetworkPartiallyPreprocessed.py and NetworkPartiallyPreprocessedCpp.py.

## pricing

### pricing_file.csv

* this file specifies the functionality used for dynamic pricing
* the name of the file can be adopted
* depending on the application there can be two versions of the pricing file
* either a time dependent pricing_file (given with the global "op_elastic_price_file")
* or a utilization dependent pricing_file (given with the global "op_util_surge_price_file")
* attribute fields of time dependent pricing file:

column_name | data_type | comment
-- | -- | --
time | int | start time of this pricing regime
base_fare_factor | float | factor of the base_fare in this pricing regime
distance_fare_factor | float | factor of the distance_fare in this pricing regime
general_factor | float | global price factor in this pricing regime

* attribute fields of utilization dependent pricing file:

column_name | data_type | comment
-- | -- | --
utilization | float | start utilization [0, 1] for this pricing regime
base_fare_factor | float | factor of the base_fare in this pricing regime
distance_fare_factor | float | factor of the distance_fare in this pricing regime
general_factor | float | global price factor in this pricing regime

## public charging stations

### public_charging_stations.csv
Mandatory attributes with explanation of data types.
Additional attributes can be added as long column-names of mandatory attributes are maintained.

Column Name | Data Type | Description
-- | -- | --
charging_station_id | int | unique identifier for each depot
node_index | int | index of node in network (BEWARE: should be unique!)
charging_units | dict_str | power1:number1;power2:number2
public_util | dict_str | hour1:util1;hour2:util2

unit of power : kW

## trips / requests

### trips_X.csv

Mandatory attributes with explanation data types.
Additional attributes can be added as long column-names of mandatory attributes are maintained.

Column Name | Data Type | Description
-- | -- | --
request_id | int | unique identifier for each request
rq_time | int | time (s) a requests gets active/visible for the system
start | int | index of origin node in network
end | int | index of destination node in network

Additional optional attributes, which can be used to model heterogeneous demand.

Column Name | Data Type | Description
-- | -- | --
latest_arrival_time | int | time a rq must/wants to reach destination; not yet implemented!
rq_type | str/int (TODO) | specifies e.g. the type of service a request wants to use
earliest_pickup_time | int | for reservation
latest_pickup_time | int | latest time for pick-up
latest_decision_time | int | latest time for request to decide for an operator before leaving system
max_rel_detour | int | maximum relative detour in percent
max_fare | int | maximum fare a request is willing to pay (in cent)
global trip destination | int (TODO) | for intermodal trips (mod-destination vs trip-destination)
rq_preferences | (TODO) | other parameters for mode choice
number_passenger | int | passengers within one request

## vehicles

### vehicle_type.md

* fleet costs depend on daily and per km costs of vehicles
* range of vehicles depend on battery size and consumption

parameter_name | data_type | comment
-- | -- | --
vtype_name_full | str | 
daily_fix_cost [cent] | int | Value in Cent to work with integers
per_km_cost [cent] | int | Value in Cent to work with integers
battery_size [kWh] | float | 
range [km] | float | 
source | str | Url and Url-Date for documentation in paper