The most visible field-change failure is a program error. The harder failure is a program that keeps running while numbers acquire a different meaning. A safe release must establish that required consumers can read old and new data, fields still express the agreed business facts, and cutover or recovery does not lose or mix state. This article addresses pipelines serving reports, data marts and AI analytics. Examples are constructed; no system experiment was performed.
Map the impact before comparing column lists
Treat the source database, messages, transformation models and published data product as four separate interfaces. A database permitting an extra nullable column does not establish that a strict consumer accepts extra fields. A decodable message does not establish that its values fit warehouse columns. Check the actual format, serializers, converters, adapters and consumer configuration instead of calling the entire chain compatible.
Separate structural, semantic and operational changes. Structure includes names, types, nullability and precision. Semantics includes units, status meanings, statistical grain and time definitions. Operations includes mixed producer versions, backlogs, replay and downstream cutover. One change can span all three and needs separate evidence for each.
Consider a constructed V1 order interface: amount is a tax-inclusive RMB value expressed in yuan with two decimal places. V2 should store the tax-inclusive amount as an integer number of fen and also provide a tax-exclusive amount. If the old amount is merely multiplied by 100 while its name and description remain unchanged, downstream systems may read 12345 as 12,345 yuan when the actual tax-inclusive amount is 123.45 yuan.
This need not cause a type error: a sufficiently wide numeric type can store both values. Field names, units and version boundaries are conditions for correct interpretation, not incidental documentation. An AI analytics system that infers meaning from names cannot independently resolve the ambiguity.
Understand compatibility from the reader's direction
In Confluent Schema Registry, BACKWARD lets a new-schema consumer read data written with the previous schema; FORWARD addresses whether the previous consumer can read new data; FULL covers both.[1] Success in one direction does not prove the other. Test the deployed reader/writer combinations.
Non-transitive checks compare with the latest schema; TRANSITIVE checks extend across registered history.[1] A three-month replay needs coverage of the schemas actually used in that period. Align registry history checks with message retention and the required replay window.
Rules differ across Avro, Protobuf and JSON Schema.[1] Do not transplant an 'optional fields are safe' rule between formats. Pin the format and component versions, then test actual serialization and deserialization rather than JSON syntax alone.
Even compatible decoding and types can misrepresent yuan as fen or tax-inclusive values as tax-exclusive. Structural checks need separate rules and expected outcomes for units, enumerations and metric definitions.
Make contracts executable and identify their limits
An order product should specify keys and grain, currency and units, tax relationships, numeric precision, null semantics, business time and ownership. Each important rule needs a validation location, failure behavior and exception process. Describing a column only as 'amount' cannot support acceptance. A configured rule that is never enforced provides no demonstrated protection.
dbt model contracts can check column names and types before a model is built. Whether particular constraints are enforced by the data platform depends on the adapter and materialization.[2] Verify actual enforcement rather than a metadata declaration. Source-message compatibility, transformation-model contracts and data-quality tests are different controls.
A monetary contract might require tax-inclusive fen to equal tax-exclusive fen plus tax in fen, but only where applicable tax and rounding rules have been specified. Discounts, cross-line rounding and transactions without an applicable tax amount need their own treatment. Do not alter source facts to satisfy a generic equation; retain unexplained differences in exception details.
If historical V1 records contain only tax-inclusive amounts without reliable tax evidence, do not default the new tax-exclusive field to zero or use today's rate as a substitute for historical fact. V2 can explicitly mark it unknown with provenance status, or limit relevant analysis to evidenced records. A mandatory non-null requirement means collecting evidence first or narrowing the release scope.
An unknown status value must not silently mean completed either. Missing optional notes may be tolerable, missing keys can break joins and idempotency, and unknown units undermine financial aggregation. Design blocking at the affected object or partition level and expose completeness status. Avoid both silent drops and stopping the entire platform over an unrelated optional note.
Expand, validate in parallel and cut over instead of overwriting
This article recommends retaining V1 while adding clearly named V2 fields or a separate versioned product. Inventory consumers, including reports, subscriptions, caches, exports and AI semantic mappings, with their dependencies and owners. Direct SQL queries may escape a dependency graph, so supplement it with access records and owner confirmation.
Prefer generating V2 from the same authoritative input, preserving stable business keys, input versions and transformation-rule versions to reduce divergence from independent dual writes. Replay history and catch up with live inputs for new consumers, then reconcile at a consistent cutoff. Comparing two independently latest outputs mistakes timing differences for migration errors.
For the constructed 123.45-yuan amount, V2 should contain 12345 tax-inclusive fen. Define expected outputs for zero, negative refunds, large values and decimal boundaries. Apply agreed decimal precision and rounding, checking overflow and truncation. Converting strings casually to floating point and displaying two decimal places does not demonstrate lossless conversion.
Move consumers after acceptance. dbt model versions support retained versions and explicit version references; unpinned references follow the version declared latest by the maintainer.[3] Pin critical consumers during migration and include cache keys, semantic mappings, exports and report definitions in the cutover inventory.
Parallel operation adds storage, compute and maintenance costs and should not continue indefinitely. Retire V1 only after consumer migration is confirmed, agreed replay and recovery windows have closed, old-version usage is checked and operations has accepted responsibility. A deprecation announcement is not evidence that no old consumers remain.
Rolling back code does not restore overwritten data
Renaming a field while retaining its original value often allows a compatibility mapping. Truncated precision, deleted fields or overwritten meaning may be irrecoverable by switching programs back. Before migration, identify reversible transformations, those needing original copies and those that cannot be recovered. Retained data remains subject to the organization's access and retention policies.
If a new consumer fails, freeze the affected serving cutover and isolate incorrect output. Establish whether the old path can still process messages received after the change. Returning to an old consumer that was never tested against new messages can compound the incident. If necessary, rebuild the old product from retained authoritative input and state the recovery cutoff and remaining gaps.
Recovery replays records, so field migration needs the replay and reconciliation design for CDC data products. Version conversion must not assign untraceable new identities to the same business event. Repeated processing should converge to the declared state, while additive downstream metrics must avoid duplicate contributions. Schema correctness does not provide these guarantees automatically.
For published reports or results already acted upon, produce an affected-record and revision list. Restoring correct data does not reverse business actions already taken. Decide whether reports need replacement, consumers need notification or records need manual reconciliation according to actual impact; a green pipeline is not sufficient closure.
Cross-check version combinations with counterexamples
First test the actual deployment combinations: old producer and old consumer, new consumer reading old data, new data reaching old consumers still in service, and the new/new combination. Not every combination must be supported. Explicitly identify permitted coexistence and prohibited combinations, then prevent unsupported combinations from reaching production.
Second test structural boundaries: added, missing and extra fields; nulls; unknown enums; excess precision; and case or alias changes. Exercise the actual component chain from source output through transformation to the published product, not just schema text in a registry.
Third test meaning. The same order's tax-inclusive amount should reconcile between versions under the declared unit conversion. Tax-exclusive amounts without evidence remain unknown. Refund signs, currencies and grain should remain consistent. Match key sets, check fields individually and reconcile by date, currency and status as well as grand totals, preventing positive and negative differences from cancelling out.
Fourth test operational failures: old-message backlog, interleaved producer versions, interrupted backfills, consumer restarts after cutover and new-version messages arriving during rollback. Check completeness, duplicate contributions and consistency between contract versions and displayed explanations. Freeze test cases and expected outcomes before release.
The acceptance record should include results for required reader/writer combinations, schema-failure counts, amounts affected by semantic differences, quarantined records and business impact, unmigrated consumers, and replay and recovery results. Mark untested historical versions as unverified. Delivery means every approved consumer still receives data with explicit, reconcilable meaning—not merely that the columns have changed.