Are you curious about the evolution of Apache Airflow? In this episode, we discuss the open-source platform used by data engineers worldwide to program and monitor workflows. Joined together with Marc Lamberti’s expert insights and analysis of his experiences with Airflow, he covers the history, key takeaways, and tips for getting started with the popular tool. Whether you’re a data engineer or just interested in learning more about workflow management tools, this episode is designed to keep listeners engaged with a practical and resourceful discussion.

Following the journey of an analytics engineer as they navigate through the challenges of data quality is no easy task. Fortunately, we were happy to be joined by Madison Schott, an Analytics Engineer over at Winc. With her expertise, she helps describe the importance of maintaining clean and accurate data in the analytics process. She also goes into greater detail on common data quality issues and their potential causes from her own point of view.

As our first guest from IBM, we are thrilled to have Jessie Snyder, Program Director for IBM Data Integration, stop by MAD Data Studio to talk about the past, future, and next big thing for the Data Fabric. We also dive deeper into the topic of Data Integration and how it’s quickly gaining popularity within modern data teams.

Identifying the key factors of data system complexities and how to reduce them is critical for any company. Joseph Machado, Senior Data Engineer at LinkedIn, shares his insights on ways to keep data systems from becoming unwieldy. In addition, Joseph offers tips for data teams to manage their data warehouses and keep data pipelines running reliably.

Matthew Blasa, a Data Scientist at The Home Depot, stopped by the MAD Data Studio to talk about how incident management as a part of observability and governance can affect various data areas. Learn how Matthew and his team tackle data observability, as well as different ways of approaching data governance on a small scale versus an enterprise scale.

Check out Matt’s Medium @ blaza-matt and his YouTube channel @ DataLife360

Neelesh Salian is a Staff Software Engineer at dbt Labs, creators, and maintainers of dbt. Neelesh stopped by the MAD Data Studio to talk about how engineering can deliver better value for data science. Get in touch with him on LinkedIn or check out his Substack – hysterical.substack.com

Sid Anand, Chief Architect & Head of Engineering at Datazoom, dropped by to give his lessons learned in building world-class data engineering teams at Ebay, Netflix, LinkedIn, and PayPal. For anyone who’s leading a team now or in the future, you don’t want to miss Sid’s insights in working at some of the world’s biggest brands.

We love Maxime so much that we had him back for a second podcast! In this episode, Max makes the case for dataset centric visualization. He walks through how different BI tools offer different approaches to building dashboards. Then he discusses how dataset centric modeling is a powerful approach that combines the best of the query-centric and semantic-centric visualization approaches.

Check out Maxime’s blog post here – The Case for Dataset-Centric Visualization

In our first two-part guest special we’re happy to introduce, Maxime Beauchemin, CEO and Founder of Preset. Eager to learn more about the recent IBM acquisition of Databand, Maxime asks us to describe it in more detail and explains what it means for the future of the data observability space.

Tristan Handy is the Founder & CEO of dbt Labs, creators, and maintainers of dbt. dbt Labs is building the modern analytics workflow used by tens of thousands of data analysts. Tristan stopped by to talk to our CEO, and fellow Philadelphian, Josh Benamram about how he founded and built dbt Labs, and how they are building an analytics community.

Jimmy Shah is a Senior Data Engineer at Warby Parker, online retailer of high-quality eyeglasses, sunglasses, contacts, and eye exams at an affordable price. Jimmy talked to us about how he builds repeatable data engineering playbooks to cut engineering time by 50-60%.

Jessica Larson is a Data Engineer at Pinterest and the author of Snowflake Access Control: Mastering the Features for Data Privacy and Regulatory Compliance. Jessica stopped by to talk about how Pinterest is migrating to the modern data cloud with Snowflake and the lessons learned in her new book.

Chad Sanderson Head of Product and Data Platform at Convoy, a digital freight network that’s solving problems in the $800B trucking industry. Chad has a lot on his plate, including being responsible for Convoy’s Data Engineering team, Data Warehouse, Data Pipeline tooling, Experimentation Platform, Machine Learning Platform, Analytics Platform, and Streaming. Chad swung by to talk about how his team is building one of the most advanced experimentations and machine learning platforms in the world from the ground up.

Colleen Tartow is the Director of Engineering for Starburst Data, a product suite that gives users the ability to self-manage their data infrastructure. We had a great time in the MAD Data studio as she shared insights on how data-driven companies should treat their data as a product and dove into the category of data mesh. Colleen also holds a Ph.D. in astrophysics which makes her truly out of this world!

George Firican is an award-winning data governance leader, founder of LightsOnData, and podcast host of the Lights On Data Show. George swung by the MAD Data studio to talk about what actually makes a company data-driven and why companies need to be more aware of their data quality issues.

Scott Taylor, The Data Whisperer, and Kate Strachnyi, Founder of DATAcated, discuss the relevance and irrelevance of data quality. In order to make data quality achievable, our guests explain that it must begin with a big picture business context.

What new data reality are we anticipating? Sr. Director of Product Management at Google Cloud, Sudhir Hasbe, shares insights from Google customers’ evolving needs. The modern data stack will take on greater variety and tooling must adapt to real-time demands.

As companies scale out their data operations and data demands continue to mount, communications and knowledge alignment form every team’s operational backbone. Castor CTO, Amaury Dumoulin, explains the foundational importance of data catalogs as teams expand in size and knowledge.

Data quality is universally important to data teams but where data quality checks should sit along the data process depends on your business use case. Netflix Sr. Software Engineer (Data Pipelines), Vivek Kaushal, shares his experience on the differences in data quality criteria from his time at Apple, Amazon, and now at Netflix.

How do you ensure data quality when your business relies on data from a high variety of external data sources? Johannes Leppä, Sr. Data Engineer at Komodo Health, shares his insights on how a data-intensive operation can design its data infrastructure to prevent common errors and high-impact issues. Johannes offers his tips for data teams to get ahead of data errors and proactively manage data quality.

Following up on our last episode on data quality, this episode takes aim at a common culture of reactivity in data teams. Gary Cheung, Staff Analytics Engineer at Eventribite, highlights the importance of taking a proactive approach to data quality and why prevention is the cure that every data team needs.

Data quality is the data industry’s holy grail – desired by all but mysteriously elusive. Sam Bail, who speaks frequently on the topic of data quality, joins Sarah Krasnik, Data Engineer at Perpay, to discuss the building blocks, organizational mindset, and strategic planning that are needed to take data quality from theory to practice.

Taylor Murphy, Head of Product and Data at Meltano, introduces the history and vision behind the open source brainchild that spun out of GitLab. In a world where businesses rely on a high variety of external data sources, many of which may be obscure and unique to an industry, the ability to develop custom connectors is indispensable. The growth of the Meltano community signals a larger trend where data teams are choosing tech stacks that adapt to their business contexts rather than the other way around. While data observability has become embraced as a data quality must-have, the industry recognizes that growing complexity calls for a distinctly proactive approach to data observability.

Matt Turck, creator of the term “MAD landscape” and Partner at FirstMark Capital, joins Joe Reis, CEO of Ternary Data and Monday Morning Data Chat podcast host, and Databand.ai Co-Founder & CEO, Josh Benamram, to discuss the explosion of data tooling, the creation of data observability as a new category, and what’s next in innovation and data engineering. What old problems have we solved? What new problems are emerging? What’s ahead in the next 5 years?

Nick Schrock, Founder & CEO of Elementl, and Scott Breitenother, Founder of Brooklyn Data Co., discuss the evolution of data from Big Data to Big Complexity – what’s next now that the data industry has solved the problem of data storage? While the modern data stack has become embraced as every data team’s “must-have” to address ‘modern data problems,’ Nick and Scott muse on the struggles that continue to plague data teams and the next wave of potential in data infrastructure innovation. With one problem solved, a new era of possibility and complexity is now unleashed.

Our modern data reality is highly complex. Data teams are recognizing that previous data processes made for a bygone era are tough to scale. Airbyte Co-Founder & CEO, Michel Tricot, and Databand.ai Co-Founder & CEO, Josh Benamram, offer insights and advice on what to do when the sheer number and variety of external data sources that a business counts on multiply at a dizzying rate – and show no sign of stopping. Our guests speak to the necessity of paying special attention to the left of the warehouse to ensure that all your data gets in and all your data is correct. In order to achieve data quality at scale and future-proof your data operations, data observability and data integration must work together hand-in-hand.

Josh Laurito, Director of Data at Squarespace and editor of NYC Data, shares his observations and insights on how data engineering has matured from a previously nebulous identity to a clearer purpose today. Josh, Harper, and Honor take a quick stroll down the data memory lane to take stock of all the changes and learnings that are shaping data engineering into a distinct discipline. The explosion in data tooling shows no sign of stopping – what does this mean for data engineers, data scientists, and data analysts who are growing in their understanding of what they love to do and the part they want to play?

Databand.ai Data Solution Architect, Michael Harper, joins as MAD Data co-host to share insights on how the treatment of data is rapidly evolving from a principle of least privileged to a principle of data access and data observability. Harper and Honor discuss what needs to occur in order for data to become available without security risk and the kind of education needed at all levels to advance data literacy. As data becomes an indispensable asset that directly contributes to the bottom line, one has to ask, Does it make sense to only empower a select few with essential business intelligence to make smart decisions?

Databricks Sr. Staff Developer Advocate, Denny Lee, Citadel Head of Business Engineering, Vinoo Ganesh, and Databand.ai Co-Founder & CEO, Josh Benamram, discuss the complexities and business necessity of setting clear data service-level agreements (SLAs). They share their experiences around the importance of contractual expectations and why data delivery success criteria are prone to disguise failures as success in spite of our best intentions. Denny, Vinoo, and Josh challenge businesses of all industries to see themselves as data companies by driving home a costly reality – what do businesses have to lose when their data is wrong? A lot more than they’d like to believe.

Databand.ai Director of Product, Shani Keynan, provides a fresh perspective on how to define data quality and how to control data quality when your data is in motion. Data observability is well understood to be a means to quality data. However, what’s often overlooked is the sheer distance that data must travel from the moment it’s collected all the way to data consumers. This means that data observability must be performed truly from end-to-end by starting right from the beginning (at the data ingestion layer) in order for data observability to be effective at all. Shani offers examples that illustrate how to make data quality an achievable goal and how to apply real-world logic and context create business impact.