PyPastry - the opinionated machine learning experimentation framework

PyPastry is a framework for developers and data scientists to run machine learning experiments. We enable you to:

Iterate quickly. The more experiments you do, the more likely you are to find something that works well.
Experiment correctly and consistently. Anything else is not really an experiment, is it?
Make experiments reproducible. That means keeping track of your code state and results.
Experiment locally. None of that Spark rubbish.
Use standard tools. Everything is based on Scikit-learn, Pandas and Git.

Quick start

PyPastry requires python 3.5 or greater.

> pip install pypastry==0.2.0
> pastry init pastry-test
> cd pastry-test
> pastry run -m "First experiment"
Got dataset with 10 rows
   Git hash Dataset hash            Run start                   Model          Score Duration (s)
0  aa87ce62     71e8f4fd  2019-08-28 06:39:07  DecisionTreeClassifier  0.933 ± 0.067         0.03

The command pastry init creates a file called pie.py in the pastry-test directory. If you open that up, you should see some code. The important bit is:

def get_experiment():
    dataset = pd.DataFrame({
        'feature': [1, 0, 1, 1, 0, 0, 1, 1, 0, 1],
        'class': [True, False, True, True, False, False, True, True, False, False],
    })
    predictor = DecisionTreeClassifier()
    cross_validator = StratifiedKFold(n_splits=5)
    scorer = make_scorer(f1_score)
    label_column = 'class'
    return Experiment(dataset, label_column, predictor, cross_validator, scorer)

This returns an Experiment instance that specifies how the experiment should be run. An experiment consists of:

dataset: a Pandas DataFrame where each row is an instance to be used in the experiment.
label_column: the name of the column in dataset that contains the label we wish to predict.
predictor: a Scikit-learn predictor, e.g. a classifier, regressor or Pipeline object.
cross_validator: a Scikit-learn cross validator that specifies how the data should be split up when running the experiment.
scorer a Scikit-learn scorer that will be used as an indication of how well the classifier has learnt to generate predictions.

When you type pastry run, PyPastry does this:

Splits dataset into one or more train and test sets.
For each train and test set, it trains the predictor on the train set and generate predictions on the test set, and computes the score on the test set using the scorer.
Generates a results file in JSON format and stores it in a folder called results
Outputs the results of the experiment.
Your repo has to be clean (no (un)staged changes) for experiment to run. If you want to use dirty repo, you can with calling pypsatry with force flag -f. However, results will not be possible to bond with exact code state.

The results includes:

Git hash: the commit identifier of the code used to run the experiment. There might be "dirty_" prefix indicating that unclean repo was used with this experiment. The hash belongs to the latest commit, however, the information about (un)staged changes is lost.
Git summary: A summary note
Dataset hash: a hash generated from the dataset that will change if the dataset changes.
Run start: the time that the experiment run started
Model: the name of the predictor class used
Score: the mean ± the standard error in the mean, computed over the different folds generated by the cross_validator.
Duration: how long the experiment took to run, in seconds.