Plain and simple, most organizations believe their data is unreliable:

• 82% say data quality concerns are a barrier to data integration projects¹
• 80% of executives do not trust their data²

The impact of this bad data can’t be underestimated. In May 2022, Unity Software discovered it had been ingesting bad data from a large customer, which led to a 30% plunge in the company’s stock³ and ultimately cost the business USD 110 million in lost revenue⁴.

Traditionally, it’s been difficult to identify bad data until it’s too late. Unlike when an application goes down and it affects thousands of users immediately, businesses can operate on bad data unknowingly for quite some time. For example, a sales team would know right away if a Salesforce dashboard wasn’t loading, but there’s no telling how long it would take them to figure out that a dashboard was showing incorrect data.

Data observability is the best defense against bad data getting through. It monitors data pipelines to ensure complete, accurate and timely delivery of data so that data teams can prevent data downtime, meet data SLAs and maintain the business’s trust in the data it sees.

Modern data systems provide a wide variety of functionality, allowing users to store and query their data in many different ways. Of course the more functionality you add, the more complicated it becomes to ensure that your system works correctly. This complication includes:

More external data sources

In the past, data infrastructure was built to handle small amounts of data—usually operational data from a few internal data sources—and the data was not expected to change very much. Now, many data products rely on data from internal and external sources, and the sheer volume and velocity in which this data is collected can cause unexpected drift, schema changes, transformations and delays.

More complicated transformations

More data ingested from external data sources means you need to transform, structure and aggregate all that data in all other formats to make it all usable. Even worse, if those formats change at all, it causes a domino effect of failures downstream as the strictly coded logic fails to adapt to the new schema.

Too much focus on analytics engineering

Complex ingestion pipelines have created a market for tools to simplify this end-to-end process, mostly automating the ingestion and extract, transform, load (ETL)/extract, load, transform (ELT) processes. Combining them together, you get a data platform the analytics industry has dubbed the “modern data stack,” or MDS. The goal of the MDS is to reduce the amount of time it takes for data to be made usable for end-users (typically analysts) so they can start leveraging that data faster. However, the more automation you have, the less control you have over how data is delivered. These organizations need to build out custom data pipelines so they can better guarantee data is delivered as expected.

Data operations (DataOps) is a workflow that enables an agile delivery pipeline and feedback loop so that businesses can create and maintain their products more efficiently. DataOps allows companies to use the same tools and strategies throughout all phases of their analytics projects, from prototyping to product deployment.

The DataOps cycle outlines the fundamental activities needed to improve data management within the DataOps workflow. This cycle consists of three distinct stages: detection, awareness and iteration.

Detection

It’s important that this cycle starts with detection because the bedrock of the DataOps movement is founded on a data quality initiative. 

This first stage of the DataOps cycle is validation-focused. These include the same data quality checks that have been used since the inception of the data warehouse. They were looking at column schema and row-level validations. Essentially, you are ensuring all datasets adhere to the business rules in your data system.

This data quality framework that lives in the detection stage is important but reactionary by its very nature. It’s giving you the ability to know whether the data that’s already stored in your data lake or data warehouse (and likely already being utilized) is in the form you expect.

It’s also important to note that you are validating datasets and following business rules you know. If you don’t know the causes of issues, you cannot establish new business rules for your engineers to follow. This realization fuels the demand for a continuous data observability approach that ties directly into all stages of your data lifecycle, starting with your source data.

Awareness

Awareness is a visibility-focused stage of the DataOps phase. This is where the conversation around data governance comes into the picture and a metadata-first approach is introduced. Centralizing and standardizing pipeline and dataset metadata across your data ecosystem gives teams visibility into issues within the entire organization.

The centralization of metadata is crucial to giving the organization awareness into the end-to-end health of its data. Doing this allows you to move toward a more proactive approach to solving data issues. If there is bad data that is entering your “domain,” you can trace the error to a certain point upstream in your data system. For example, Data Engineering Team A can now go on to look at Data Engineering Team B’s pipelines and be able to understand what’s going on and collaborate with them to fix the issue.

The vice-versa also applies. Data Engineering Team B can detect an issue and trace what impact it will have on downstream dependencies. This means Data Engineering Team A will know that an issue will happen and can take whatever measures are necessary to contain it.

Iteration

Here, teams focus on data-as-code. This stage of the cycle is process-focused. Teams are ensuring that they have repeatable and sustainable standards that will be applied to all data development to ensure that they get the same trustworthy data at the end of those pipelines.

The gradual improvement of the data platform’s overall health is now made possible by the detection of issues, awareness of the upstream root causes and efficient processes for iteration.

Together, the five pillars of data observability provide valuable insight into the quality and reliability of your data.

1. Freshness

Freshness describes how up to date your data is and how frequently it is updated. Data staleness occurs when there are important gaps in time when the data has not been updated. Often, when data pipelines break it is due to a freshness issue.   

2. Distribution

An indicator of your data’s field-level health, distribution refers to whether or not the data falls within an accepted range. Deviations from the expected distribution might indicate data quality issues, errors or changes in the underlying data sources.   

3. Volume

Volume refers to the amount of data being generated, ingested, transformed and moved through various processes and pipelines. It also refers to the completeness of your data tables. Volume is a key indicator as to whether or not your data intake is meeting expected thresholds.

4. Schema

Schema describes the organization of your data. Schema changes often result in broken data. Data observability helps ensure that your data is organized consistently, is compatible across different systems and maintains its integrity throughout its lifecycle.  

5. Lineage

Lineage’s purpose is to answer the question, “Where?” when data breaks. It looks at the data from its source to its end location and notes any changes, including what changed, why it changed and how it changed along the way. Lineage is most often represented visually.

Data observability supports data quality, but the two are different aspects of managing data.

While data observability practices can point out quality problems in data sets, they can’t on their own guarantee good data quality. That requires efforts to fix data issues and to prevent them from occurring in the first place. On the other hand, an organization can have strong data quality even if it doesn’t implement a data observability initiative.

Data quality monitoring measures whether the condition of data sets is good enough for their intended uses in operational and analytics applications. To make that determination, data is examined based on various dimensions of quality, such as accuracy, completeness, consistency, validity, reliability, and timeliness.

Data observability and data governance are complementary processes that support each other.

Data governance aims to ensure that an organization’s data is available, usable, consistent and secure, and that it’s used in compliance with internal standards and policies. Governance programs often incorporate or are closely tied to data quality improvement efforts. 

A strong data governance program helps eliminate the data silos, data integration problems and poor data quality that can limit the value of data observability practices. 

Data observability can aid the governance program by monitoring changes in data quality, availability and lineage.

All data observability isn’t created equal. The level of context you can achieve depends on what metadata you can collect and provide visibility into. This is known as the hierarchy of data observability. Each level is a foundation for the next and allows you to attain increasingly finer grains of observability.

Below are the main steps typically involved in building a successful observability pipeline. The process involves the integration of various tools and technologies, as well as the collaboration of different teams within an organization.

• Define key metrics: Start by identifying the critical metrics that you need to track. This could include data quality metrics, data volumes, latency, error rates and resource utilization. The choice of metrics will depend on your specific business needs and the nature of your data pipeline.
  
  
• Choose appropriate tools: Next, choose the tools you’ll need for data collection, storage, analysis and alerting. Ensure that the tools you select, including open-source, are compatible with your existing infrastructure and can handle the scale of your operations.
  
  
• Standardize libraries: Put infrastructure in place that allows teams to speak the same language and openly communicate about issues. This includes standardized libraries for API and data management (i.e., querying data warehouse, read/write from the data lake, pulling data from APIs, etc.) and data quality.
  
  
• Instrument your data pipeline: Instrumentation involves integrating data collection libraries or agents into your data pipeline. This allows you to collect the defined metrics from various stages of your pipeline. The goal is to achieve comprehensive visibility, so it’s key to ensure every crucial stage is instrumented.
  
  
• Set up a data storage solution: The collected metrics need to be stored in a database or a time-series platform that can scale as your data grows. Ensure the storage solution you choose can handle the volume and velocity of your data.
  
  
• Implement data analysis tools: These tools help derive insights from the stored metrics. For more in-depth analysis, consider using tools that provide intuitive visualizations and support complex queries.
  
  
• Configure alerts and notifications: Establish a system for sending automated alerts when predefined thresholds are crossed or anomaly detection occurs. This will help your team respond promptly to issues, minimizing any potential downtime.
  
  
• Integrate with incident management platforms: In addition to detecting issues, observability also involves the effective management of issues. Integrating your observability pipeline with an incident management system can help streamline your response workflows.
  
  
• Regularly review and update your observability pipeline: As your business evolves, so will your data and requirements. Regularly reviewing and updating your observability pipeline ensures it will continue to provide the necessary insights and performance.

Building an observability pipeline is a continuous process of learning and refinement. It’s crucial to start small, learn from the experience and incrementally expand your observability capabilities.