Webinar: Learn how to Strengthen DataOps with Continuous Data Observability. Watch now

Apache Airflow

Marco Alcaria
2022-12-09 08:02:51

Apache Airflow Observability with Databand

Simplify and centralize your Apache Airflow observability.

Continuous Apache Airflow observability and monitoring

Your data’s health is more complicated than a task or run failure. You need to know your Airflow pipelines are going to deliver complete and accurate data on time. Even more important, you need alerts on data quality issues before they are used by downstream consumers.

When you integrate Databand to your Airflow environments, you get continuous Airflow observability. This allows you to centralize the pipeline metadata, logs, and statuses to get the insights you need to consistently deliver high quality data.

What’s in it for you?

Integrate easily

Integrate with Databand’s Airflow connector to begin tracking Airflow metadata

Centralize pipeline metadata to track your data health

Track pipeline statuses, run durations, data volumes, and data quality metrics

Preventative alerts on issues in pipelines and data sets

Analyze and alert on metadata anomalies or missing data, trace the root cause of pipeline failures, data quality problems, and the impact issues on your data deliveries

Extend with connections to additional Airflow services

Integrate with tools built on Airflow and Airflow 2.0+ like Google Cloud Composer and Amazon MWAA to centralize additional logging and data quality information

Know what’s happening to your data while it’s in motion

Databand.ai makes it easy to integrate powerful preventative alerting into your Airflow environments and your organization. Get alerts on Airflow pipelines that are at risk for late deliveries due to long task duration, discover anomalies in your data volume,  gain visibility into data quality issues like breaking changes to in dataset structure by your data sources that normally fly under the radar.

Best of all, Databand.ai makes it easy to push these notifications to Slack, OpsGenie, Pagerduty, or whatever other notification systems your data team uses.

Airflow Observability - Alerting
Airflow Observability - Root Cause

Jump right into the root cause of data health issues

Drill beneath the surface of your Airflow environments and cut down on engineering’s Time-to-Remediation.

When Databand.ai sends you an alert, clicking brings you right where the incident occurs so you can begin to get context on the root cause. Databand.ai makes it easy to see all the relevant information you’ll need to resolve the issue. View your pipeline inputs and outputs, error traces, logs, data source, parameters, xcoms, and user metrics in one easy-to-use Dashboard.

Get a bird’s eye view of all your Airflow instances

With Databand.ai, all of your Airflow observability activities can live in one place.

The Databand Dashboard makes it easy to highlight all the important metrics for all of your high-stakes Airflow DAGs. Visualizations and charts of your critical data assets allow you to see whether pipeline metrics are in the right ranges and Airflow throughput is on schedule for delivery.

Airflow Observability Dashboard

Fix data incidents fast

See how Databand can transform data observability at your organization today.

dbt

Ryan Yackel
2022-12-05 12:47:33

Continuous dbt observability with Databand

Databand provides dbt observability across your jobs, tests, and models you can know when a dbt process breaks, and how to fix it fast.

dbt observability with databand

Why dbt observability with Databand?

Teams use dbt Core and dbt Cloud to quickly deploy analytics code. But what happens when your dbt commands don’t work as expected? That’s where Databand’s dbt observability helps. 

Icon Icon

Alert earlier

Get proactive dbt alerts around execution times, test failures, model anomalies, and more. 

Icon Icon

Debug faster

Save engineering time by centralizing metadata and root cause analysis from all your dbt commands under one roof. 

Icon Icon

Analyze impacts

Leverage Databand’s lineage capabilities to see which tables are impacted from dbt population. 

Watch dbt + Databand in action.

Watch how easy it is to define alerts on dbt tests, models, and jobs to receive alerts when your dbt processes fail.

dbt Alerts

INCIDENT MANAGEMENT

Receive proactive alerts for all dbt incidents.

Leverage the power of Databand’s alerting capabilities to notify your teams of critical issues as soon as they happen.  

 Generate alerts for incidents like: 

  • Failure of dbt commands, individual models, or individual tests. 
  • Duration anomalies for commands, models, and tests. 
  • Anomalous record counts for the tables in your models. 
  • Number of failures for any given test. 

SQL ACCESS & DISCOVERY

Get instant access and debug your dbt SQL.

Simplify how analytics engineers and analysts access SQL for their models and tests.  

Databand removes the manual discovery of .sql and .yaml files from your dbt projects. 

  • Identify key information like the materialization type of your tables and schemas. 
  • Investigate table logic to better understand how certain calculations are derived. 
dbt - SQL Access and Discovery

ON DEMAND WEBINAR

How to Strengthen DataOps with Continuous Data Observability

dbt - Central logging 2

CENTRAL LOGGING

Stop wasting time. View all dbt commands in one place.

Quickly view information about the state and duration of each dbt command from the Databand console. 

  • See the state and duration of each individual dbt models and tests. 
  • Pinpoint the root causes of a dbt failures and resolve them fast.  

LINEAGE & IMPACT ANALYSIS

Get the whole picture with impact analysis and lineage.

Databand’s dbt observability automatically displays dbt commands in the context of your other data processes.  

For example, if your using Apache Airflow to kick off a dbt command, Databand will show the entire flow. 

  • Gain insight into the affected processes if a dbt model fails. 
  • Proactively determine which downstream processes are at risk due to upstream issues. 
dbt - Lineage
Testimonial Image

Databand automatically monitors the health of our Airflow pipelines so we spend less time debugging, and more time building ML models. Before Databand, 60% of our pipelines had at least one data incident. Now less than 1% of pipelines have incidents. This resulted in a 3X increase in our customers since we can now manage our ML deep learning models at scale.

Tzoof Hemed
AI-Engineering Team Leader
Testimonial Image

Databand helps us detect data quality issues faster so we can meet our data SLAs. Without Databand, we didn’t know we had problems until two or three days later – forcing us to backfill the data. 

Fithrah Fauzan
Data Engineering Lead
Testimonial Image

Databand is helping us achieve better pipeline reliability and higher velocity releases for our data products. The platform saves our team a lot of troubleshooting time by providing one holistic view for job monitoring and dependencies so that everyone can see what’s happening in our pipelines.

Amir Ara
Senior Engineering

Keep up with the Databand community

Fix data incidents fast

See how Databand can transform data observability at your organization today.

Google Composer

Marco Alcaria
2022-10-07 10:02:23

Google Cloud Composer Observability with Databand

Databand.ai’s instant integration to Google Cloud Composer provides comprehensive monitoring and observability into user pipelines.

What’s in it for you?

Integrate easily

Integrate with Databand’s Google Cloud Composer connection to begin tracking Composer metadata

Centralize pipeline metadata to track your data health

Track pipeline statuses, run durations, data volumes, and data quality metrics

Automatically alert on issues in pipelines and data sets

Analyze and alert on metadata anomalies or missing data, trace the root cause of pipeline failures, data quality problems, and the impact issues on your data deliveries

Extend with connections to additional GCP services

Integrate with more GCP tools like Dataproc, BigQuery, and Google Storage, to centralize additional logging and data quality information

FAQs

No, this integration is built into the platform at no extra cost.

No, you can integrate Databand.ai to your Google Cloud Composer environment in one click. If you would like to learn more about how the integration works, please read our documentation on this integration.

Getting started is easy. First, you start your free trial or schedule a product demo. From there, the Solutions Architect team will advise you how to customize the platform to best suit your needs.

Fix data incidents fast

See how Databand can transform data observability at your organization today.