ACM (Anatomically-constrained model)

A framework for videography based pose tracking of rodents. By Arne Monsees. [In review]

Installation

1. Install Anaconda
2. Clone https://github.com/bbo-lab/ACM.git
3. Start Anaconda Prompt (Windows) / terminal (linux) and navigate into repository directory
4. Create conda environment conda env create -f https://raw.githubusercontent.com/bbo-lab/ACM/main/environment.yml once repo is public)
5. Install using pip install .

Testing

1. Download example dataset.
2. Start Anaconda Prompt (Windows) / terminal (linux)
3. Activate conda environment conda activate bbo_acm.
4. View 2D labels by running python -m ACM --viewer [Path of "table_1_20210511" folder from step 1.]
5. Run python -m ACM [Path of "table_1_20210511" folder from step 1.] (expected to take around 20 minutes on modern 8-core CPU, tested with an AMD Vega iGPU and a Geforce GTX 1080 Ti)
6. View 3D tracked pose by running python -m ACM --viewer [Path of "table_1_20210511" folder from step 1.]

Setting up your own dataset config

Overview

Input

A dataset input config consists of a folder with the following files:

• configuration.py: Configuration file that determines settings and input and out paths.
• model.npy: Defines the underlying skeleton graph of the animal.
• multicalibration.npy: Intrinsic and extrinsic camera parameters obtained from calibrating the camera setup.
• origin_coord.npy: Coordinate system of the final result, relative to the coordinate system of the camera calibration.
• labels_dlc_n_m.npy: Automatically detected labels on the videography data.
• labels_manual.npz: Manual labels on the videography data. The labels are required for training a neural network for automated detection of surface markers as well as for learning the animal's antaomy via calibration.py.

Output

A dataset output consists of a folder with the following files:

• pose.npy: A dictionary containing the inferred 3D joint and marker locations as well as the resulting projected marker locations in the 2D images.
• save_dict.npy: A dictionary containing the model parameters learned with the EM algorithm (i.e. the initial state and covariance matrix as well as the transition and measurement covariance matrix). Additionally, it also contains the smoothed latent variables and covariance matrices for every time point of the reconstructed sequence, which were generated using the learned model parameters.

Further, the following intermediate files are generated by calibration and initialization:

• x_calib.npy: A flattened vector containing the latent variables encoding the learned bone lengths, surface marker locations and poses for all time points of the calibration data set, used to learn the animal's anatomy. For the subsequent pipeline only the first few entries of the flattened vector, corresponding to the bone lengths and surface marker locations, are relevant.
• x_ini.npy: A flattened vector containing the latent variables encoding the learned bone lengths, surface marker locations and poses for the first time point of the sequence, which should be reconstructed.

By default, results are saved into the results subfolder of the datset configuration.

File structure of input and output files

The structure of the inputs is documented in INPUTS.md. The structure of the outputs is documented in OUTPUTS.md.

FAQ