Parquet files accept new columns without complaint. Your Spark job does not crash when a source adds a field — it just drops that column on read and keeps going. No error, no alert, no incident. Your dashboard shows the same numbers, your dbt models pass their tests, and your downstream consumers assume the data is complete. It is not.

Schema drift is the most common silent data corruption pattern we see in production data lakes. The file format tolerates it, the engine tolerates it, and the pipeline tolerates it — right up until the CFO asks why the Q2 numbers shifted by 12% and nobody can trace it.

How Parquet Hides Schema Drift

Parquet stores data in a columnar format with embedded schema metadata per file. When you write a Parquet file with columns [A, B, C] and then write a second file with columns [A, B, C, D], both files coexist peacefully in the same directory. No error is raised.

What happens at read time depends on the reader configuration:

import pyarrow.parquet as pq

# Default behavior: silently drops unknown columns
table = pq.read_table("s3://lake/events/")
# Column 'D' is present, but if you select schema from the first file only, it is gone

When you use a schema-on-read engine like Spark (default mode) or DuckDB, the engine merges schemas across files using a union strategy — columns present in some files but not others become NULL. If your deduplication logic runs before the merge, or if your ELT tool uses a fixed-schema SELECT *, columns are lost without a trace.

The real damage occurs downstream. Aggregations produce incorrect results, surrogate keys break because the source changed partition ordering, and the daily metrics pipeline shows a “dip” that three engineers spend a week debugging.

Pattern 1: Silent Column Addition

A production API adds a refund_reason field to its event payload. The Parquet writer adds it as column eight. Old files have seven columns, new files have eight. Your Spark reader loads a batch spanning both old and new partitions:

# spark.conf.set("spark.sql.schema.merge", "true")  # default behavior
df = spark.read.parquet("s3://lake/api_events/")
df.filter(col("refund_reason").isNotNull()).count()
# Returns a number, but you never noticed this column existed

With merge disabled, only the first file’s schema is used and refund_reason is silently dropped. The reader does not warn you. Every downstream pipeline referencing that column gets NULLs for historical partitions.

Pattern 2: Type Widening Without Warning

A source team changes a device_id field from INT to STRING. Parquet allows this per file. Any downstream model that performs a join on device_id using an INT key now misses every new record. The join cardinality is wrong, but the job succeeds.

-- dbt model, runs without error
select a.*, b.session_count
from source_data a
left join device_stats b on a.device_id = b.device_id
-- After type change, all new device_ids fail the join

No error is thrown. Joins that fail type coercion silently yield NULLs.

Pattern 3: Column Removal (Orphaned Schema Expectations)

A source removes legacy_field from its output. The Parquet files stop including it. Downstream models that still SELECT legacy_field receive NULLs. If that column was used in a WHERE clause or partition filter, the query returns zero rows — and the dashboard goes blank.

Detection: Schema Drift Is an Observability Problem, Not a Validation Problem

You cannot catch drift with dbt tests alone. A not_null test passes as long as the column exists. A unique test passes as long as the data is valid. Neither catches a new column or a missing column.

The fix is to compare schemas across partitions, batches, or time windows — and alert when they diverge.

PyArrow Schema Diff Checker

import pyarrow.parquet as pq
from pyarrow import Schema
from datetime import datetime, timedelta
import boto3

def get_schema_for_prefix(bucket: str, prefix: str) -> Schema:
    s3 = boto3.client("s3")
    response = s3.list_objects_v2(Bucket=bucket, Prefix=prefix, MaxKeys=1)
    if "Contents" not in response or len(response["Contents"]) == 0:
        return None
    key = response["Contents"][0]["Key"]
    table = pq.read_table(f"s3://{bucket}/{key}")
    return table.schema

drift_detected = []

# Compare today's schema against yesterday's
today_prefix = f"events/dt={datetime.now().strftime('%Y-%m-%d')}"
yesterday_prefix = f"events/dt={(datetime.now() - timedelta(days=1)).strftime('%Y-%m-%d')}"

today_schema = get_schema_for_prefix("my-lake", today_prefix)
yesterday_schema = get_schema_for_prefix("my-lake", yesterday_prefix)

if today_schema and yesterday_schema:
    if today_schema != yesterday_schema:
        drift_detected.append({
            "table": "events",
            "yesterday": str(yesterday_schema),
            "today": str(today_schema),
            "added": today_schema.remove(yesterday_schema),
            "removed": yesterday_schema.remove(today_schema),
        })

This produces the exact list of columns added, removed, or type-changed. Route it to PagerDuty or Slack — do not let it sit in a log file.

Great Expectations: Expectation Suite for Schema Lock

Great Expectations provides the right abstraction: an expectation suite that describes what a batch’s schema must look like.

import great_expectations as gx

context = gx.get_context()
datasource = context.sources.add_pandas_s3("events", bucket="my-lake")
batch_request = datasource.build_batch_request(
    data_asset_name="events",
    options={"dt": "2026-05-15"}
)

validator = context.get_validator(batch_request=batch_request)

# Lock column names
validator.expect_table_columns_to_match_set(
    expected_column_set=[
        "event_id", "user_id", "device_id", "event_type",
        "timestamp", "revenue", "refund_reason",
    ]
)

# Lock column types
validator.expect_column_type("device_id", "string")
validator.expect_column_type("revenue", "float64")

results = validator.validate()

Run this expectation suite as a checkpoint after every ingestion job. If schema drift is detected, the checkpoint fails before data reaches the warehouse.

Iceberg/Deltalake Schema Enforcement

If you use Apache Iceberg or Delta Lake, schema enforcement happens at the table level rather than the file level. Both require explicit schema evolution commands.

from delta import DeltaTable

delta_table = DeltaTable.forPath(spark, "s3://lake/events")

# Attempting to write a column that doesn't exist
# raises an AnalysisException unless schema evolution is explicitly enabled
spark.conf.set("spark.databricks.delta.schema.autoUpdate.enabled", "false")

df.write \
  .format("delta") \
  .mode("append") \
  .option("mergeSchema", "false") \
  .save("s3://lake/events")
  # Fails with AnalysisException if schema doesn't match

With mergeSchema = false, writes block on schema mismatch. You must review and approve every change before it lands in the table. This is the correct default for production — require explicit evolution.

Schema Registry for Streaming Sources

If your data arrives via Kafka or Kinesis, the fix belongs upstream. Enforce Avro or Protobuf with a schema registry. Confluent Schema Registry rejects any message whose schema is incompatible with the registered subject.

# Register schema
curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"schema": "{\"type\":\"record\",\"name\":\"Event\",\"fields\":[...]}"}' \
  https://registry.example.com/subjects/events-value/versions

# Any producer sending a non-matching schema gets a 409 Conflict

This catches drift at the producer, not at the lake — which is where it belongs.

The Observability Pipeline

A complete drift detection pipeline requires three layers:

Layer 1 — Registry enforcement (upstream): Schema registry rejects incompatible writes at the producer. This is the cheapest fix.

Layer 2 — File-level checksum (ingestion): After every batch load, run a schema comparison against the previous partition. Any diff routes to an alert channel.

Layer 3 — Table-level evolution (storage): Use Iceberg or Delta Lake with explicit schema evolution. Require a code review and migration script for every schema change.

def drift_check_pipeline(bucket: str, table: str, partitions: list[str]):
    schemas = {p: get_schema_for_prefix(bucket, p) for p in partitions}
    reference = schemas.get(partitions[0])
    for partition, schema in schemas.items():
        if schema and reference and schema != reference:
            handle_drift(table, partition, reference, schema)

def handle_drift(table: str, partition: str, expected, actual):
    # Send to Slack, file a ticket, block the pipeline
    print(f"DRIFT: {table}/{partition}")
    print(f"  expected: {expected.names}")
    print(f"  actual:   {actual.names}")

Trade-offs

Schema registries and Iceberg both add operational complexity. You now have a registry to manage, schema evolution workflows to document, and migration scripts to write for every change. Teams that treat schemas as an afterthought will chafe against these constraints.

Iceberg/Delta Lake write overhead is roughly 5-15% higher than bare Parquet due to manifest files, metadata logs, and compaction. For tables under 100 GB, bare Parquet with a drift detection script is simpler and cheaper. For tables above 1 TB, the transactional guarantees of a table format are worth the overhead.

Schema enforcement prevents rapid iteration. If your data sources change weekly and you have 50 of them, a schema registry becomes a bottleneck. Apply enforcement selectively — lock schemas for production-facing tables, allow evolution for exploratory zones.

Great Expectations can become a maintenance burden. Suites that check for exact column sets break on every legitimate schema evolution. Use a “known columns” pattern: track the baseline and alert only on unexpected additions, not all changes.

Decision Framework

Dataset Type	Drift Risk	Recommended Approach
Production BI tables (finance, revenue)	Critical	Iceberg + schema registry + GX checkpoint
Operational dashboards (product analytics)	High	Delta Lake with mergeSchema = false + partition diff
Exploratory data marts for data science	Medium	Bare Parquet + weekly drift report
Logs and raw event dumps	Low	Accept drift, document known columns

Schema drift is not a technical failure of Parquet — it is an engineering gap between what the format allows and what your pipeline assumes. Close that gap with one of the three layers above, or expect a call from the CFO who noticed the numbers do not add up.