Modern Data Stack; Data Quality and Governance

Modern Data Stack

What is the Modern Data Stack?

A set of cloud-native tools for data engineering:

Ingestion:      Fivetran, Airbyte  → extract from sources
Storage:        Snowflake, BigQuery, Redshift → cloud DWH
Transformation: dbt (data build tool) → SQL transformations
Orchestration:  Apache Airflow → schedule and monitor pipelines
BI:             Looker, Tableau, Metabase → visualization

DBT (Data Build Tool)

dbt allows data transformation using SQL models:

-- models/marts/sales_summary.sql
SELECT
    date_trunc('month', order_date) as month,
    product_category,
    SUM(revenue) as total_revenue,
    COUNT(*) as order_count,
    AVG(revenue) as avg_order_value
FROM {{ ref('stg_orders') }}   -- reference another dbt model
WHERE order_status = 'completed'
GROUP BY 1, 2
ORDER BY 1, 3 DESC

Apache Airflow

Airflow orchestrates complex workflows as DAGs (Directed Acyclic Graphs):

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def extract(): print("Extracting data...")
def transform(): print("Transforming data...")
def load(): print("Loading to DWH...")

with DAG("etl_pipeline", schedule_interval="@daily",
         start_date=datetime(2024, 1, 1)) as dag:

    t1 = PythonOperator(task_id="extract", python_callable=extract)
    t2 = PythonOperator(task_id="transform", python_callable=transform)
    t3 = PythonOperator(task_id="load", python_callable=load)

    t1 >> t2 >> t3   # define dependencies

Data Quality and Governance

Data Quality Dimensions

Data Governance

Data Governance is the framework for managing data availability, usability, integrity and security.

Key Components:

• Data Catalog — inventory of all data assets (metadata)
• Data Lineage — track where data comes from and how it transforms
• Access Control — who can access what data
• Data Quality Rules — define and monitor quality checks
• Privacy Compliance — GDPR, CCPA, HIPAA regulations

Layers of a Modern Data Platform

A modern data platform separates ingestion, storage, processing, serving and governance because each layer has a different responsibility. Batch ingestion copies bounded files or database snapshots. Streaming ingestion carries a continuing sequence of events. Object storage commonly holds raw and curated data because it scales independently of compute. Processing engines clean, join and aggregate records. Warehouses, lakehouses, search systems and feature stores then serve different workloads. Orchestrators record dependencies and start each task only after its inputs are ready.

Every boundary needs a contract that states schema, ownership, freshness and failure behavior. A pipeline that writes a table successfully can still be wrong if it silently drops rows or changes the meaning of a field. Raw data is usually preserved so that corrected processing can reproduce later layers.

Data Quality, Metadata and Governance

Accuracy asks whether a value represents reality. Completeness measures missing records or attributes. Validity checks values against allowed types, ranges and formats. Consistency compares representations of the same fact. Uniqueness detects unintended duplicates. Timeliness asks whether data arrives soon enough. Referential integrity checks relationships such as an order referring to an existing customer.

Useful controls include schema validation, null thresholds, accepted-value lists, range checks, reconciliation totals and duplicate-key tests. Technical metadata describes schemas, partitions, formats and locations. Business metadata defines terms such as active customer or net revenue. Operational metadata records runs, row counts and failures. Lineage connects an output to its source fields and transformations.

Governance assigns accountability. Owners approve meaning and acceptable use. Stewards maintain definitions and quality rules. Platform teams implement storage, access and monitoring. Least-privilege access may use roles, row filters, column masking and encryption. Retention rules define how long records remain available. Audit logs record access to protected information.