network_wrangler

Network Wrangler is a Python library for managing travel model network scenarios.

System Requirements

Network Wrangler should be operating system agonistic and has been tested on Ubuntu and Mac OS.

In order to assist in installation, its helpful to have either miniconda, anaconda or Docker CE installed. If you don't have any of these already, we reommend starting with Miniconda for Python 3.7 as it has the smallest footprint. conda is the environment manager that is contained within both the Anaconda and mini-conda applications. All commands described below should be entered into the Ananconda Prompt command window.

Network Wrangler does require Python 3.7+. If you have a different version of Python installed, conda will take care of installing it for you in the installation instructions below.

Installation

Network Wrangler uses Python 3.6 and above. Requirements are stored in requirements.txt but are automatically installed when using pip.

If you are managing multiple python versions, we suggest using virtualenv or conda virtual environments. conda is the environment manager that is contained within both the Anaconda and mini-conda applications. Do not add Anaconda to the system path during installation. This may cause problems with other programs that require python 2.7 to be placed in the system path.

Example installing and running tests using conda in the command line:

conda config --add channels conda-forge
conda create python=3.7 rtree geopandas osmnx -n <my_wrangler_environment>
conda activate <my_wrangler_environment>
pip install network-wrangler
pytest -s -m travis

Network wrangler can be installed in several ways depending on the user's needs. Installing from github is the simplest method and is appropriate when the user does not anticipate needing to update network wrangler. An update will require rebuilding the network wrangler environment. Installing from clone is slightly more involved and requires the user to have a git manager on their machine, but permits the user to install network wrangler with the -e, edit, option so that their network wrangler installation can be updated through pulling new commits from the network wrangler repo without a full reinstallation.

From GitHub

Use the package manager pip to install Network Wrangler from the source on GitHub.

conda config --add channels conda-forge
conda create python=3.7 rtree geopandas osmnx -n <my_wrangler_environment>
conda activate <my_wrangler_environment>
pip install git+https://github.com/wsp-sag/network_wrangler.git@master#egg=network_wrangler

Note: if you wanted to install from a specific tag/version number or branch, replace @master with @<branchname> or @tag

From Clone

If you are going to be working on Network Wrangler locally, you might want to clone it to your local machine and install it from the clone. The -e will install it in editable mode.

This is also useful if you want to continue to update your Network Wrangler as it is developed on GitHub.

1. Open a terminal to get a command prompt.

2. Consider using a virtual environment manager like conda.

Create a new environment by typing the following commands into the command prompt (it might take a few minutes).

conda create python=3.7 -n wrangler_env
conda activate wrangler_env

I chose wrangler_env as the name of my environment, but you could choose something else...just remember it so that you can access it later.

NOTE in order to get back to "this" conda environment (i.e. after you close this command prompt), you will need to access it from the command line by using the following command:

conda activate wrangler_env

3. Install finicky Requirements

Some requirements are best installed using conda rather than "pip"

conda config --add channels conda-forge
conda install rtree geopandas osmnx

4. "Clone" (aka download) network wrangler from Github on to your machine

If you have GitHub desktop installed, you can either do this by using the GitHub user interface by clicking on the green button "clone or download" in the main network wrangler repository page.

Otherwise, you can use the command prompt to navigate to the directory that you would like to store your network wrangler clone and then using a git command to clone it.

cd path to where you want to put wrangler
git clone https://github.com/wsp-sag/network_wrangler

Expected output:

cloning into network_wrangler...
remote: Enumerating objects: 53, done.
remote: Counting objects: 100% (53/53), done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 307 (delta 28), reused 29 (delta 19), pack-reused 254
Receiving objects: 100% (307/307), 15.94 MiB | 10.49 MiB/s, done.
Resolving deltas: 100% (140/140), done.

5. Install Network Wrangler in "develop" mode.

Navigate your command prompt into the network wrangler folder and then install network wrangler in editable mode. This will take a few minutes because it is also installing all the prerequisites.

cd network_wrangler
pip install -e .

There will be a lot of messy output, but it should end with something like:

Running setup.py develop for network-wrangler
Successfully installed Rtree-0.8.3 attrs-19.1.0 cchardet-2.1.4 chardet-3.0.4 click-7.0 click-plugins-1.1.1 cligj-0.5.0 cycler-0.10.0 decorator-4.4.0 descartes-1.1.0 fiona-1.8.6 geojson-2.4.1 geopandas-0.5.1 idna-2.8 isoweek-1.3.3 jsonschema-3.0.2 kiwisolver-1.1.0 matplotlib-3.1.1 munch-2.3.2 network-wrangler networkx-2.3 numpy-1.17.0 osmnx-0.10 pandas-0.25.0 partridge-1.1.0 pyparsing-2.4.2 pyproj-2.2.1 pyrsistent-0.15.4 python-dateutil-2.8.0 pytz-2019.2 pyyaml-5.1.2 requests-2.22.0 shapely-1.6.4.post2 six-1.12.0 urllib3-1.25.3

6. Test the Installation

You can test that network wrangler was properly installed by running the tests as follows:

pytest -s  -m  travis

Using the -s flag will run all the tests in "noisy" mode. The -m travis flag runs only tests that are marked as for "travis" continuous integration

Note: if you are not part of the project team and want to contribute code back to the project, please fork before you clone and then add the original repository to your upstream origin list per these directions on github.

Using Docker

1. Install Docker
2. Clone git repository (see instructions above) NOTE: this is easiest way right now since repo is private. When it is public we can clone right from github without having to muck around with logins or keys
3. From the cloned repository, open a terminal from the /docker folder and build and run the docker container corresponding to what you want to do by running docker-compose run <container name> <entry point (optional)> --build
4. Command to exit container: exit

Containers:

• wrangler-jupyter started by running docker-compose run wrangler-jupyter --build is appropriate for running and testing wrangler.
  • Default action is to start jupyter notebook which can be found at http://127.0.0.1:8888
  • Safe: It creates an empty folder to store jupyter notebooks within the container but wont overwrite the source files on your actual machine.
  • Starting Bash: You can also start the container with a command line using docker-compose run wrangler-jupyter /bin/bash --build.
  • Doesn't install development dependencies (although they can be installed from within the container)
• wrangler-ci is a small image without extras meant for running tests and deploying to continuous integration server.
  • default command is to run pytest.
  • contains development dependencies so that it can run tests and build docs.
• wrangler-dev is the most powerful but dangerous container docker-compose run wrangler-dev /bin/bash --build
  • Warning: It will synchronize code edited from the container to your wrangler clone. This is great for developing within an IDE, but please take this into account.

Common Installation Issues

Issue: clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated] If you are using MacOS, you might need to update your xcode command line tools and headers

Issue: OSError: Could not find libspatialindex_c library file* Try installing rtree on its own from the Anaconda cloud

conda install rtree

Issue: Shapely, a pre-requisite, doesn't install propertly because it is missing GEOS module Try installing shapely on its own from the Anaconda cloud

conda install shapely

Issue: Conda is unable to install a library or to update to a specific library version Try installing libraries from conda-forge

conda install -c conda-forge *library*

Issue: User does not have permission to install in directories Try running Anaconda Prompt as an administrator.

Quickstart

To get a feel for the API and using project cards, please refer to the "Wrangler Quickstart" jupyter notebook.

To start the notebook, open a command line in the network_wrangler top-level directory and type:

jupyter notebook

Documentation

Documentation can be built from the /docs folder using the command: make html

Usage

import network_wrangler

##todo this is just an example for now

## Network Manipulation
my_network = network_wrangler.read_roadway_network(...) # returns
my_network.apply_project_card(...) # returns
my_network.write_roadway_network(...) # returns

## Scenario Building
my_scenario = scenario_from_network(roadway_network, transit_network)
my_scenario.add_projects(directory, keyword)
my_scenario.write_networks(directory, format)

Attribution

This project is built upon the ideas and concepts implemented in the network wrangler project by the San Francisco County Transportation Authority and expanded upon by the Metropolitan Transportation Commission.

While Network Wrangler as written here is based on these concepts, the code is distinct and builds upon other packages such as geopandas and partridge which hadn't been implemented when networkwrangler 1.0 was developed.

Contributing

Pull requests are welcome. Please open an issue first to discuss what you would like to change. Please make sure to update tests as appropriate.

License

Apache-2.0