Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
-
Updated
Aug 17, 2020 - Python
Is your feature request related to a problem? Please describe.
The current default back pressure configuration uses the Vegas algorithm with an initial limit of 20. What this means is it will poorly handle initial bursts, and may result in a poor first user experience - it's also generally quite pessimistic and more difficult to configure/explain to users.
See: