Python library to run ML/data pipelines on stateless compute infrastructure (that may be ephemeral or serverless). Please see the documentation site with more details and demo:


Keywords
dag, data-science, datascience, jupyter-notebooks, jupytext, mlops, notebooks, papermill, pipeline, pipelines, python, python3, ray
License
MIT
Install
pip install pynb-dag-runner-snapshot==0.0.6.dev1661694514

Documentation

ci/cd: publish snapshot to PyPI PyPI version license=mit Ideas and feedback=welcome

Composable Logs

Composable Logs is a Python library to run ML/data workflows on stateless compute infrastructure (that may be ephemeral or serverless).

In particular, Composable Logs supports ML experiment tracking without a dedicated tracking server (and database) to record ML metrics, models or artifacts. Instead, these are emitted using the OpenTelemetry standard for logging. This is an open standard in software engineering with growing support.

For example, log events emitted from Composable Logs can be directed to a JSON-file, or sent to any log storage supporting OpenTelemetry (span) events. In either case, this means that one does not need a separate tracking service only for ML experiments.

Composable Logs uses the Ray framework for parallel task execution.

For more details:

Documentation and architecture

Live demo

  • Using Composable Logs one can run a ML training pipeline using only a free Github account. This uses:

    • Github actions: trigger the ML pipeline daily and for each PR.
    • Build artifacts: to store OpenTelemetry logs of past runs.
    • Github Pages: to host static website for reporting on past runs.

    The static website is rebuilt after each pipeline run (by extracting relevant data from past OpenTelemetry logs). This uses a fork of MLFlow that can be deployed as a static website, https://github.com/composable-logs/mlflow.

    Screenshot

  • Codes for pipeline (MIT): https://github.com/composable-logs/mnist-digits-demo-pipeline

Public roadmap and planning

Install via PyPI

Latest release
Snapshot of latest commit to main branch

Any feedback/ideas welcome!

License

(c) Matias Dahl 2021-2022, MIT, see LICENSE.md.

(Note: As of 1/2023 this project was renamed from pynb-dag-runner to composable-logs.)