Machine Learning Lab
A lightweight command line interface for the management of arbitrary machine learning tasks.
Documentation is available at: https://bering-ml-lab.readthedocs.io/en/latest/
NOTE: Lab is in active development - expect a bumpy ride!
Installation
The latest stable version can be installed directly from PyPi:
pip install lab-mlDevelopment version can be installed from github.
git clone https://github.com/beringresearch/lab
cd lab
pip install --editable .Concepts
Lab employs three concepts: reproducible environment, logging, and model persistence. A typical machine learning workflow can be turned into a Lab Experiment by adding a single decorator.
Creating a new Lab Project
lab init --name [NAME]Lab will look for a requirements.txt file in the working directory to generate a portable virtual environment for ML experiments.
Setting up a Lab Experiment
Here's a simple script that trains an SVM classifier on the iris data set:
from sklearn import svm, datasets
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, precision_score
C = 1.0
gamma = 0.7
iris = datasets.load_iris()
X = iris.data
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.24, random_state=42)
clf = svm.SVC(C, 'rbf', gamma=gamma, probability=True)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred, average = 'macro')It's trivial to create a Lab Experiment using a simple decorator:
from sklearn import svm, datasets
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, precision_score
from lab.experiment import Experiment ## New Line
e = Experiment() ## New Line
@e.start_run ## New Line
def train():
C = 1.0
gamma = 0.7
iris = datasets.load_iris()
X = iris.data
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.24, random_state=42)
clf = svm.SVC(C, 'rbf', gamma=gamma, probability=True)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred, average = 'macro')
e.log_metric('accuracy_score', accuracy) ## New Line
e.log_metric('precision_score', precision) ## New Line
e.log_parameter('C', C) ## New Line
e.log_parameter('gamma', gamma) ## New Line
e.log_model('svm', clf) ## New LineRunning an Experiment
Lab Experiments can be run as:
lab run <PATH/TO/TRAIN.py>Comparing models
Lab assumes that all Experiments associated with a Project log consistent performance metrics. We can quickly assess performance of each experiment by running:
lab ls
Experiment Source Date accuracy_score precision_score
------------ ------------------ ---------- ---------------- -----------------
49ffb76e train_mnist_mlp.py 2019-01-15 0.97: ██████████ 0.97: ██████████
261a34e4 train_mnist_cnn.py 2019-01-15 0.98: ██████████ 0.98: ██████████Pushing models to a centralised repository
Lab experiments can be pushed to a centralised filesystem through integration with minio. Lab assumes that you have setup minio on a private cloud.
Lab can be configured once to interface with a remote minio instance:
lab config minio --tag my-minio --endpoint [URL:PORT] --accesskey [STRING] --secretkey [STRING]To push a local lab experiment to minio:
lab push --tag my-minio --bucket [BUCKETNAME] .Copyright 2020, Bering Limited
