Data Quality At Scale

Week 1

Detecting Problems in the Data Pipeline

Understanding where in the data pipeline problems occur, and what peoples’ roles are is a crucial first step to detecting and preventing data quality issues. In this session, we will:

• Map out key stakeholders’ roles in a data pipeline
• Discover where in the pipeline data quality issues can occur and why
• Learn from case studies such as Convoy on how to detect data quality issues

By the end of the session, learners will be able to:

• Map out key stakeholders’ roles in your data pipeline
• Describe where in the pipeline data quality issues can occur and why

Week 2

Creating Requirements for your Data Quality System

Garbage in - garbage out: The quality of your data model is determined by the quality of its data inputs. In this session, we will:

• Discuss how you can identify the type of data you need for your model to run effectively
• Determine the requirements for a data quality system
• Get to know specific tools & frameworks, such as medallion data infrastructure, data warehouses, data marts, data modeling and data mesh

By the end of the session, learners will be able to:

• Identify what data you need and where in the pipeline it is processed (constraints)
• Define the requirements for your data quality system

Week 3

Architecting a Data Observability System

With your requirements defined, we can now move onto implementing the technical solution for your data observability system. In this session, we will:

• Contrast data observability with data contracts
• Review the technical components of observability, including alerting systems, ownership and process, including 3 pillars: logs, metrics and traces

By the end of the session, learners will be able to:

• Decide whether you need an observability system or data contract
• Architect an observability system

Week 4

Developing Data Contracts

Data Contracts are API-like agreements between software engineers who produce operational data and data consumers who use them for business-critical analytics. In this session, we will:

• Review a CDC-based implementation of entity-based data contracts, covering contract definition, schema enforcement, fulfillment and monitoring
• Discuss the elements of an effective data contract definition as code, including using the Transactional Outbox pattern
• Learn from guest lecturer Adrian Krueziger

By the end of the session, learners will be able to:

• Define a data contract (as code) between data producers and data consumers

Week 5

Implementing Data Contracts

With a well-defined data contract in place, you need to enforce it in production and monitor it for any bugs slipping through. In this session, we will:

• Learn how to enforce contracts in our CI/CD pipeline as part of the normal deployment process
• Review an implementation for the enforcement of semantic contracts and the need for good monitoring
• Discuss open-source solutions for data contracts such as flink, airflow, DBT, great expectations
• Bring it all together - review how to implement an end-to-end data quality system
• Learn from guest lecturer Daniel Dicker

By the end of the session, learners will be able to:

• Enforce, fulfill, and monitor data contracts for a continuous data quality system