cacheflow

Release 0.3

Caching Workflow Engine

Homepage PyPI

Keywords
workflow, execution, cache, caching, notebook
License
BSD-3-Clause
Install
pip install cacheflow==0.3

Documentation

CacheFlow

CacheFlow is a caching workflow engine, capable of executing dataflows while reusing previous results where appropriate, for efficiency. It is very extensible and can be used in many projects.

Goals

  • ☑ Python 3 workflow system
  • ☑ Executes dataflows from JSON files
  • ☐ Can also load from SQL database
  • ☐ Parallel execution
  • ☐ Streaming
  • ☑ Extensible: can add new modules, new storage formats, new caching mechanism, new executors
  • ☐ Pluggable: extensions can be installed from PyPI without forking
  • ☑ Re-usable: can execute workflows by itself, but can also be embedded into applications. Some I plan on developing myself:
    • Literate programming app: snippets or modules embedded into a markdown file, which are executed on render (similar to Rmarkdown). Results would be cached, making later rendering fast
    • Integrate in some of my NYU research projects (VisTrails Vizier, D3M)

Other ideas:

  • ☐ Use Jupyter kernels as backends to execute code (giving me quick access to all the languages they support)
  • ☐ Isolate script execution (to run untrusted Python/... code, for example with Docker)

Non-goals

  • Make a super-scalable and fast workflow execution engine: I'd rather make executors based on Spark, Dask, Ray than re-implement those

Status

Basic structures are here, extracted from D3M. Execution works.

Stats

Dependencies
5
Dependent packages
0
Dependent repositories
0
Total releases
4
Latest release
First release
Stars
1
Forks
0
Contributors
0
SourceRank
5

Development practices

Source repo 2FA enabled
TEXT!
Package manager 2FA enabled
TEXT!
Is security responsive
TEXT!
Dependencies are managed
TEXT!
Issue-free release available
TEXT!
Succession plan available
TEXT!
Package manager 2FA enabled
TEXT!

The Tidelift Subscription provides access to a continuously curated stream of human-researched and maintainer-verified data on open source packages and their licenses, releases, vulnerabilities, and development practices.

Learn more →

Releases

0.3
0.2
0.1
0.0

Something wrong with this page? Make a suggestion

Export .ABOUT file for this package

Last synced: 2023-11-30 16:31:01 UTC

Login to resync this project