Description
Capillaries is an open-source distributed data processing platform for producing enriched, production-quality data within SLA time limits. It is aimed at data engineers running heavy, compute-intensive jobs on a schedule, daily, weekly or quarterly, on VMs or containers that can be provisioned for the run and thrown away afterwards. Pipelines are defined declaratively: the processing DAG and relational operations live in a JSON script, field transformations are one-line Go expressions, and row-level logic can use Python formulas covered by integration tests. It runs DAG nodes in parallel and splits large datasets into batches, with fault tolerance and incremental computing. Operators can step in at chosen points for manual go or no-go checks. Performance is meant to be predictable enough to estimate run times.…
Highlights
- Declarative JSON pipeline definitions
- Go expressions for fields, Python formulas for rows
- Parallel DAG execution with batching
- Fault tolerance and incremental computing
- Operator go/no-go checkpoints in pipelines
- Runs on short-lived cloud or private infrastructure
Ratings & Reviews
No ratings yet
Be the first to rate Capillaries.
People also visit
Capillaries FAQ
What is Capillaries?
A distributed data processing platform focused on delivering customer-ready data within SLA time limits, with source code on GitHub.
How are pipelines defined?
Declaratively, as a JSON script describing the DAG and relational operations, with one-liner Go expressions for fields and optional Python formulas for row-level logic.
What workloads is it best for?
Data-heavy, compute-intensive jobs that run periodically, on infrastructure that is provisioned quickly and disposed of once processing is done.
Can people approve steps in a pipeline?
Yes. Operators can interact at designated steps to validate data and make go/no-go decisions.
What are the best Capillaries alternatives?
Popular alternatives to Capillaries include cptn.io, Metaflow and Bacalhau. See all Capillaries alternatives






