Schema Evolution for Events and Backward-Compatible Change

Why schema evolution is different for events

Event data is durable by design: once an event is recorded, it should remain a faithful representation of what was known at that time. That durability makes schema evolution more constrained than evolving a typical OLTP table. With OLTP, you can often migrate rows in place and move on. With events, you usually have years of historical records, multiple consumers, and replay/rebuild workflows that depend on old events remaining readable.

Schema evolution for events is the practice of changing event definitions (names, fields, types, semantics) while keeping existing stored events usable and keeping existing consumers functioning. “Backward-compatible change” means older consumers can continue to process newer events without breaking. In practice you also care about “forward compatibility” (new consumers can process old events) and “replay compatibility” (rebuilding projections from the full history still works after changes).

This chapter focuses on concrete patterns for evolving event schemas in PostgreSQL when events are stored in an append-only event table and consumed by multiple services or jobs. The key idea is to treat event schema as a contract and evolve it with explicit versioning, compatibility rules, and controlled rollout steps.

Compatibility goals and what can break

Backward compatibility (old consumers read new events)

Backward compatibility is usually the primary operational goal during rollout. If you deploy a producer that emits a new field or changes a value format, older consumers should not crash or misinterpret the event. Backward compatibility is easiest when changes are additive and optional.

Forward compatibility (new consumers read old events)

Forward compatibility matters for replay and for consumers that start after a change but must process historical events. A new consumer should be able to interpret old events, often by applying defaults or by supporting multiple versions.

Continue in our app.

• Listen to the audio with the screen off.
• Earn a certificate upon completion.
• Over 5000 courses for you to explore!

Or continue reading below...

Download the app

Replay compatibility (rebuilds remain deterministic)

When you rebuild projections from the entire event history, you need deterministic interpretation across versions. If semantics changed, you must encode enough information in events (or in version metadata) to interpret them correctly later.

Common breaking changes

• Renaming a field without keeping an alias (consumer looks for old name and fails).
• Changing a field type (string to integer) without dual-writing or conversion logic.
• Changing semantics (e.g., amount changes from “gross” to “net”) without changing event type/version.
• Removing a field that consumers still rely on.
• Changing enum values (e.g., “CANCELLED” becomes “CANCELED”) without mapping.
• Changing meaning of timestamps/time zones without explicit metadata.

Versioning strategies for event schemas

1) Version in the event type name

Example: order.created.v1, order.created.v2. This is explicit and makes semantic changes obvious. It also allows consumers to subscribe to specific versions. The trade-off is proliferation of event types and the need to support multiple types in consumers.

2) Version in metadata

Store a version field such as schema_version in event metadata (or in the payload). Example: {"schema_version":2}. This keeps a stable event type name but requires consumers to branch on version. It works well when you want a single logical event type with multiple encodings.

3) No explicit version, only additive evolution

This can work if you enforce strict rules: only add optional fields, never change meaning, never remove. In real systems, semantic changes eventually happen, so explicit versioning is usually safer.

Recommended approach

Use explicit versioning for semantic changes (new meaning, new required fields, incompatible type changes). For purely additive changes, you can keep the same version and treat new fields as optional. Many teams combine both: a stable event type plus a schema_version for encoding changes, and a new event type/version for semantic changes.

Compatibility rules you can enforce

Additive changes (usually safe)

• Add a new optional field with a default behavior when absent.
• Add new enum values if consumers treat unknown values safely (e.g., map to “OTHER”).
• Add new nested objects in JSON payloads.

Changes that require a multi-step rollout

• Make an optional field required (must backfill or dual-write first).
• Change a field type (must dual-write old and new representations).
• Rename a field (must write both names for a period).

Changes that should create a new event version/type

• Semantic changes where the same field name would mean something different.
• Splitting one event into multiple events, or merging multiple into one, when consumers cannot infer equivalence.
• Any change that would make old consumers produce incorrect side effects even if they don’t crash.

PostgreSQL storage patterns that help evolution

Store schema version and producer info in columns

Even if your payload is JSONB, keep a few stable columns for evolution control and observability, such as event_type, schema_version, producer, producer_version, and occurred_at. This makes it easier to route, validate, and analyze compatibility.

-- Example event table additions for evolution control (illustrative only)

ALTER TABLE events ADD COLUMN IF NOT EXISTS schema_version int NOT NULL DEFAULT 1;

ALTER TABLE events ADD COLUMN IF NOT EXISTS producer text;

ALTER TABLE events ADD COLUMN IF NOT EXISTS producer_version text;

Keeping schema_version as a column (not only inside JSON) enables indexing and fast filtering during replays or migrations.

Use JSONB for flexible payloads, but validate at the boundary

JSONB makes additive evolution easy, but it also makes it easy to accidentally emit malformed events. The key is to validate at write time (producer-side) and optionally at insert time (database-side) for critical invariants.

Database-side validation can be done with CHECK constraints for simple rules. For example, if version 2 requires a field currency:

ALTER TABLE events ADD CONSTRAINT events_v2_currency_required CHECK (schema_version < 2 OR (payload ? 'currency'));

Keep constraints lightweight; heavy validation can harm ingest throughput. Use constraints for invariants that prevent irreparable data corruption.

Generated columns for evolving fields

When you introduce a new canonical field but must support old events, a generated column can provide a unified view. Suppose older events stored customerId and newer events store customer_id. You can expose a generated column that coalesces both.

ALTER TABLE events ADD COLUMN customer_id_text text GENERATED ALWAYS AS (COALESCE(payload->>'customer_id', payload->>'customerId')) STORED;

This helps consumers and projections query a stable column while you transition. It also makes indexing easier than indexing multiple JSON paths.

Views as compatibility layers

Create a view that normalizes payload differences across versions. Consumers that read from PostgreSQL directly (analytics jobs, projection builders) can use the view instead of raw events.

CREATE OR REPLACE VIEW events_normalized AS SELECT id, event_type, schema_version, occurred_at, CASE WHEN event_type = 'order.created' AND schema_version = 1 THEN jsonb_set(payload, '{customer_id}', to_jsonb(payload->>'customerId'), true) WHEN event_type = 'order.created' AND schema_version >= 2 THEN payload ELSE payload END AS payload FROM events;

A view does not change stored history; it provides a stable read contract while producers and consumers evolve.

Step-by-step rollout patterns for backward-compatible change

Pattern A: Add a new optional field

Scenario: You want to add currency to payment.authorized events. Old consumers ignore it; new consumers use it when present.

• Step 1: Update schema contract (documentation and validation rules). Mark currency as optional with a default behavior when absent (e.g., assume “USD” only if historically true, otherwise treat as unknown and route to remediation).

• Step 2: Update consumers first (if possible). Deploy consumers that can handle both cases: field present or missing.

• Step 3: Update producer to emit the field. Start writing currency for new events.

• Step 4: Monitor. Track the percentage of events with currency present and any consumer errors.

In PostgreSQL, you might add a lightweight check only after rollout if you want to enforce that all new events include currency from a certain date onward, but be careful: enforcing “required” retroactively can block ingestion if any producer lags.

Pattern B: Rename a field without breaking older consumers

Scenario: You want to rename customerId to customer_id.

• Step 1: Consumer tolerance. Update consumers to read customer_id if present, else fall back to customerId.

• Step 2: Dual-write. Update producer to emit both fields for a transition period.

• Step 3: Backfill (optional). If you need uniformity for analytics, you can backfill old events into a normalized view or derived column rather than rewriting the event table.

• Step 4: Stop writing the old field after you confirm all consumers have been updated and historical replays are safe.

Dual-writing keeps backward compatibility because old consumers still find customerId. New consumers can move to customer_id immediately.

Pattern C: Change a field type with dual representation

Scenario: You stored amount as a string (e.g., “12.34”) and want to move to an integer minor-unit representation (e.g., 1234) to avoid floating issues.

• Step 1: Introduce a new field such as amount_minor while keeping amount unchanged.

• Step 2: Update consumers to prefer amount_minor when present, else parse amount.

• Step 3: Producer dual-write for a period: write both amount and amount_minor.

• Step 4: Optional enforcement. Add a constraint for new versions: schema_version >= 2 requires amount_minor.

• Step 5: Deprecate old field for new events, but keep reading it for historical events indefinitely.

In PostgreSQL, you can add a generated column to unify reads:

ALTER TABLE events ADD COLUMN amount_minor bigint GENERATED ALWAYS AS (CASE WHEN payload ? 'amount_minor' THEN (payload->>'amount_minor')::bigint WHEN payload ? 'amount' THEN round(((payload->>'amount')::numeric) * 100)::bigint ELSE NULL END) STORED;

This lets downstream queries use amount_minor consistently, regardless of event version.

Pattern D: Semantic change (create a new event version/type)

Scenario: discount.applied used to mean “discount applied to the cart total” but now it means “discount applied per line item,” which changes how projections compute totals. This is not just a new field; it changes meaning.

• Step 1: Introduce a new version/type, e.g., discount.applied.v2 or discount.applied with schema_version=2, but only if consumers can reliably branch.

• Step 2: Update consumers to support both. For projections, implement handlers for v1 and v2 separately.

• Step 3: Producer emits v2 for new behavior. Keep v1 for legacy flows only if needed.

• Step 4: Replay testing. Rebuild projections from scratch in a staging environment to ensure deterministic results across mixed histories.

When semantics change, trying to “normalize” by rewriting old events is risky because it changes history. Prefer new versions and explicit interpretation logic.

Managing deprecations and consumer coordination

Deprecation windows and “compatibility budgets”

Define a deprecation policy: how long you will dual-write fields or support old versions. Without a policy, old versions linger indefinitely and complexity grows. A practical approach is to set a time-based window (e.g., 90 days) plus a replay requirement (e.g., “we support all versions that exist in the last 2 years of events” or “we support all versions ever emitted for core domains”).

Consumer-driven contracts

Backward compatibility is easiest when producers know what consumers require. Maintain a contract per event type/version: required fields, optional fields, allowed values, and semantic notes. Even if you don’t use a full schema registry, you can store JSON Schema documents in a repository and reference them by version.

Detecting lagging consumers

To safely stop dual-writing or stop emitting a version, you need evidence that no consumer depends on it. Techniques include:

• Consumer telemetry that reports supported versions/types.
• Database queries that show whether old versions are still being produced.
• Dead-letter queues or error logs that indicate parsing failures after changes.

Testing schema evolution with replay-focused checks

Golden replay tests

Maintain a small curated set of historical events (fixtures) for each event type/version. When you change consumer code, run a replay over these fixtures and assert the resulting projection state. This catches subtle semantic drift.

Property-based compatibility tests

For additive changes, test that consumers ignore unknown fields and tolerate missing optional fields. For example, generate payloads with extra keys and ensure parsing still succeeds.

Database-level validation tests

If you use CHECK constraints or triggers for validation, include migration tests that insert representative events for each supported version to ensure constraints don’t block valid historical shapes.

Operational techniques in PostgreSQL for evolving event schemas

Using partial indexes by version

When a new version introduces a field heavily used in queries, you can add a partial index that targets only events where that field exists or where schema_version is high enough. This avoids indexing sparse historical data.

CREATE INDEX IF NOT EXISTS events_order_created_v2_customer_id_idx ON events ((payload->>'customer_id')) WHERE event_type = 'order.created' AND schema_version >= 2;

This supports new consumers efficiently while leaving old data untouched.

Online migrations for compatibility columns

If you add generated columns or new metadata columns, do it in a way that avoids long locks. In PostgreSQL, adding a column with a constant default is optimized in modern versions, but adding a stored generated column may require computation. Consider adding the column nullable first, then backfilling in batches, then adding constraints.

ALTER TABLE events ADD COLUMN IF NOT EXISTS customer_id_text text;

-- Backfill in batches (example pattern; run repeatedly with a WHERE clause)

UPDATE events SET customer_id_text = COALESCE(payload->>'customer_id', payload->>'customerId') WHERE customer_id_text IS NULL AND event_type = 'order.created';

ALTER TABLE events ALTER COLUMN customer_id_text SET NOT NULL;

Whether you can set NOT NULL depends on your true invariants; for many event types, leaving it nullable is acceptable.

Controlled “upcasting” at read time

Upcasting means transforming older event versions into the newest in-memory representation when reading, without rewriting stored events. In PostgreSQL-centric pipelines, you can implement upcasting in:

• Consumer code (preferred for complex transformations).
• A normalization view (good for simple renames/defaults).
• A SQL function that returns a normalized JSONB payload.

Example SQL function for simple upcasting:

CREATE OR REPLACE FUNCTION upcast_order_created(p_payload jsonb, p_version int) RETURNS jsonb LANGUAGE sql IMMUTABLE AS $$ SELECT CASE WHEN p_version = 1 THEN jsonb_set(p_payload, '{customer_id}', to_jsonb(p_payload->>'customerId'), true) - 'customerId' WHEN p_version >= 2 THEN p_payload ELSE p_payload END $$;

Use IMMUTABLE only if the function truly is deterministic and does not depend on database state.

Handling event splitting, merging, and de-normalization changes

Splitting one event into multiple events

Scenario: A single order.updated event used to include many optional fields; you want to split into more specific events like order.shipping_address_changed and order.status_changed.

This is typically a semantic change. A backward-compatible rollout often looks like:

• Emit the new specific events in addition to the old generic event (dual stream) for a period.
• Update consumers to prefer specific events when present, but still handle the generic event for older history.
• Eventually stop emitting the generic event for new changes, but keep the handler for replay.

In PostgreSQL, you can help consumers by providing a view that expands generic events into a normalized set, but be cautious: expanding one row into multiple rows in a view can complicate ordering assumptions and replay logic. Prefer doing this expansion in consumer code where you can control ordering and idempotency.

Merging multiple events into one

Merging is harder to do compatibly because older consumers may expect intermediate events. If you must merge, consider emitting both: keep emitting the old sequence for compatibility and add the new combined event for new consumers. If you stop emitting the old sequence, you likely need a new event version/type and a clear cutover date, plus replay logic that can interpret both histories.

Practical checklist for safe event schema evolution

• Decide whether the change is additive, representational (rename/type), or semantic. Use new versions/types for semantic changes.
• Ensure old consumers won’t crash on new events: ignore unknown fields, tolerate missing optional fields, handle unknown enum values.
• Prefer consumer-first deployments: update consumers to accept both shapes before producers start emitting the new shape.
• Use dual-write for renames and type changes, with a defined deprecation window.
• Store schema_version and producer metadata in stable columns for observability and filtering.
• Use views/generated columns/functions as compatibility layers for SQL-based readers.
• Test with replay fixtures across versions and verify deterministic projection results.
• Monitor production: percentage of events by version, parsing failures, and lagging producers/consumers.