Datatracker

Datatracker is a basic logging Python package that keeps track of files and code within a Project. Each script is logged as an entry and input and output datafiles are recorded in order. Datatracker is able to manage versioning of both files and scripts, and is able to identify the most up-to-date version.

At the moment, this Python package is still in alpha, and I may include changes to both UI and file format that may be breaking.

Installation

To install, run the following command:

pip install git+ssh://git@github.com/TarjinderSingh/datatracker

Usage

New entries

For an entry,

1. tag is a unique identifier to the script in question and should be clear what the general purpose and output of the script is. (ie Merge is not what we want to see here)
2. description needs to be one or two sentences equivalent of the Git commit message that thoroughly describes the general purpose and output of the script.
3. category indicates the general step of analysis the script belongs to.
4. module is the sub-category for which the script belongs to. Type category_template in interactive Python for an idea of the appropriate categories and modules are.

For a InputFile or OutputFile,

1. tag is a unique identifier to the File in question and should be clear what the general purpose and output of the script is. (ie Merge is not what we want to see here).
2. description for a file is a one or two sentences equivalent of the Git commit message that thoroughly describe the general purposes of the File at hand.

from datatracker import *
tr = Tracker()

os.environ['VERSION'] = '0.1.0'

entry = Entry(tag='filter-common-variants',
              description='Filtering common variants in new GWAS data set.',
              category='Processing',
              module='Variant QC')

infile = entry.add(
    InputFile(tag='raw-plink-file',
              path='gs://bucket/raw-plink-file.bed',
              description='Raw PLINK file.'))


outfile = entry.add(
    OutputFile(tag='filt-plink-file',
               path='gs://bucket/raw-plink-file.bed',
               description='Filtered PLINK file.'))

tr.save(entry)

View existing entries

from datatracker import *
tr = Tracker()

tr.table

Use existing entries for pipeline

infile = entry.add(InputFile(entry_tag='filter-common-variants', tag='raw-plink-file', database=tr))

Filter and remove

# filter to entry
tr.filter(tr.entry.tag_version == 'import-array_0.1.6')

# remove entry
tr.remove(tr.entry.tag_version == 'import-array_0.1.6')

Pandas and Excel

df = tr.explode()
df = tr.explode('filt-plink-file')

df = tr.to_pandas()
df = tr.table

df.to_excel('spreadsheet.xlsx')

Data artifacts

infile = entry.add(InputFile(path='gs://checkpoint-cache/tmp/1.bed'))

License

MIT License (see repository)

Maintainer

TJ Singh @ tsingh@broadinstitute.org