DBND

A framework for data pipeline operations.

DBND is a library for building better DataOps. Use DBND to track pipeline metadata, debug workflows, and add dynamic automation to data processes.

Documentation

Github

Tracking Pipelines

Store metadata to always be on top of your pipeline and data health

Custom Metrics

Define and report any custom metric about your data every time your pipeline runs.

Input/Output

Track workflow inputs and outputs and lineage of data across tasks and broader pipelines.

Data Profiling

Automatically generate data profiling and statistics on data files and tables.

Extending Airflow

Make your Airflow DAGs data-aware and easier to iterate

DBND makes it easier to construct your Airflow pipelines, centrally manage parameters and configurations, and expand your logging capabilities.

Functional Defintion

Easily construct DAGs through intuitive wiring of Python functions.

Data Versioning

Version data outputs from your tasks based on parameters and input objects.

Data Portability

Run pipelines dynamically with new data sets, both local and cloud storage.

Data Caching

Cache and reuse data from unchanged tasks to cut down on unnecessary compute time.

Metadata Tracking

Track execution logs, function input/output, performance metrics, system resources, and data profiles.

Central Configuration

Easily update pipeline parameters and environment variables.

 

 

Leveraging DBND instrumentation in Airflow DAGs to gain additional capabilities for easy pipeline definition, data management, and central configuration.

Optimizing Orchestration

Gain Faster and more Dynamic Execution for your Pipelines

Execute pipelines through DBND’s runtime for additional features and support for more complex task graphs.

Dynamic Pipelines

Run new DAG versions based on dynamic input parameters

Sub-Pipelines

Easily build and run "pipelines of pipelines" and complex hierarchies

Compute Portability

Update compute locations to easily migrate pipelines across environments

 

 

Leveraging DBND runtime for running more dynamic processes