Scheduling
Python task scheduler lets you create tasks, assign dependencies between them granting the correct execution order.
Examples
g = scheduling.Task('Global')
pipeline1 = scheduling.Task('My first pipeline')
pipeline2 = scheduling.Task('My second pipeline')
dump = scheduling.Task('Generate dumps')
norm = scheduling.Task('Normalize')
stats = scheduling.Task('Generate statistics')
notify = scheduling.Task('Notify services')
export = scheduling.Task('Export dumps')
index = scheduling.Task('Index data')
dump.add(
norm.then(
stats,
notify,
export
)
)
pipeline1.add(dump)
# pipeline2 starts normally and index task waits
# until normalize finishes.
pipeline2.add(
index.depends(norm)
)
g.add(pipeline1, pipeline2)
# if you do not want pipeline2 to start until pipeline1 finishes
# you would set:
# pipeline2.depends(pipeline1)
s = scheduling.TaskScheduler(g)
s.run()
Running the schedule guarantees an execution respecting dependencies.