DBND

A framework for data pipeline operations.

DBND is a library and SDK for better DataOps. DBND makes it easy to track pipeline metadata, debug workflows, and add dynamic automation to data processes. You can use DBND directly in your workflow code, or as an extension of Apache Airflow.

Documentation

Github

Power Up Your Airflow

Make your Airflow DAGs data-aware and easier to iterate

DBND makes it easier to construct your Airflow pipelines, centrally manage parameters and configurations, and expand your logging capabilities.

Functional Defintion

Easily construct DAGs through intuitive wiring of Python functions.

Data Versioning

Version data outputs from your tasks based on parameters and input objects.

Data Portability

Run pipelines dynamically with new data sets, both local and cloud storage.

Data Caching

Cache and reuse data from unchanged tasks to cut down on unnecessary compute time.

Metadata Tracking

Track execution logs, function input/output, performance metrics, system resources, and data profiles.

Central Configuration

Easily update pipeline parameters and environment variables.

Optimize Your Executions

Gain Faster and more Dynamic Execution for your Pipelines

Execute pipelines through DBND’s runtime for additional features and support for more complex task graphs.

Dynamic Pipelines

Run new DAG versions based on dynamic input parameters

Sub-Pipelines

Easily build and run "pipelines of pipelines" and complex hierarchies

Compute Portability

Update compute locations to easily migrate pipelines across environments