Why Data Consumers Read Incomplete Batches: Immutable Manifests, Conditional Commits and Publication Tests

Successful uploads do not give consumers an atomic batch. An orders-and-returns example develops immutable files, a fixed manifest and a conditional publication pointer, compares a lightweight protocol with table formats, and tests concurrency, ambiguous commits, mixed reads and cleanup races.

Preventing incomplete reads requires more than letting consumers scan a directory that is still receiving files and guess when uploading has ended. Prepare and validate all immutable files first, publish their manifest through one conditional operation, and make each consumer retain the chosen batch until processing finishes. Atomicity comes from an explicit commit point and reader–writer rules, not from every upload returning success.

Define the delivery unit: two correct files can form an incorrect report

This design covers one logical dataset delivered through multiple files, with consumers able to follow a batch-reading protocol. The object-store example uses the documented semantics of ordinary Amazon S3 object PUT, GET and conditional writes as read on the access date. Not every S3-compatible service provides identical capabilities. This is a proposed publication protocol, not a claim that a database, data platform or product already implements it.

Construct two batches of orders and returns. In old batch B100, orders O1 and O2 are CNY 100 and 200, and return R1 is CNY 20, yielding a net CNY 280. New batch B101 corrects O2 to CNY 180 and adds O3 for CNY 120; returns are R1 for CNY 20 and R2 for CNY 30. Its net is 100 + 180 + 120 − 20 − 30 = CNY 350. All amounts use the same currency and definition.

If orders are overwritten before returns, an intervening reader can calculate new orders of CNY 400 minus old returns of CNY 20, yielding CNY 380. Mixing in the opposite direction gives old orders of CNY 300 minus new returns of CNY 50, yielding CNY 250. Each file can pass its own byte-integrity check while the combination belongs to neither approved batch. That is the failure this protocol must prevent.

First agree on the files' business cutoff, correction rules, units and keys. Publication can guarantee that consumers receive the declared combination; it cannot establish that upstream orders and returns already represent a consistent business scope. If reconciliation across the source systems is incomplete, preserve that status rather than hide a source-data gap behind an atomic switch.

Why single-key strong consistency does not prevent partial batches

Amazon S3 documents atomic updates to an individual object key and strongly consistent reads after successful writes. It also states that updates across different keys are not automatically atomic.[1] Seeing the newly uploaded orders does not establish that returns are ready. Eventual consistency is unnecessary to explain this failure: the two writes simply complete at different times.

A directory prefix filters names; it is not a transaction. Even an accurate listing of every currently existing object cannot identify how many files this batch was supposed to contain, whether a fragment failed or which older files must be excluded. Choosing the newest objects by modification time likewise cannot establish one consistent cross-file combination.

A completion marker also requires a contract. An empty marker followed by permitted data overwrites still exposes changing contents. At minimum, the marker must bind an immutable manifest and its identity, and be created only after data validation. It then serves as a commit record, rather than a disposable hint that may be overwritten at will.

Separate preparing, validating, ready to publish, committed and consumed. Uploading is part of preparation; committed means a particular manifest has been made authoritative; consumed means a specific downstream system has completed its own processing. One success flag cannot represent all three, and a duplicate notification must not cause the same sales batch to be accumulated again.

Record file identities and business totals in an immutable manifest

Give B101 separate object names, such as B101/orders.csv and B101/returns.csv, and prohibit content overwrites after completion. Where version-addressed reads are available, bind the exact object version to the manifest as well. A mutable path alone is insufficient. Separate staging names, batch names and the publication pointer, and restrict consumers from bypassing the entry point by reading arbitrary staging locations.

The manifest needs dataset and batch identifiers, parent batch, format version, business cutoff and generation-rule version. For every file, record its logical role, object key or version, length, whole-byte digest, row count and total. Consumers also need rules defining required roles and legitimate empty roles. A missing returns file is different from an explicitly verified file containing zero returns.

This example requires exactly one orders file and one returns file. B101 records three orders totaling CNY 400, two returns totaling CNY 50, and a net of CNY 350, together with uniqueness rules for O1–O3 and R1–R2. If orders later span multiple fragments, the manifest must enumerate all of them and update the role rule. The old requirement of one orders file must not silently remain in effect.

Specify the digest algorithm and byte scope, such as SHA-256 over the original delivered bytes. Amazon S3 distinguishes checksum types and explains why a multipart-upload ETag is not a whole-file MD5.[2] The pointer's ETag supports conditional comparison; file digests support content checks. Reuse the existing method for single-file resume and integrity verification instead of reinventing that transfer layer here.

Before generating the manifest, inspect evidence from the completed objects and recalculate the agreed totals. Keep the resulting manifest immutable and record its own digest. Failed content checks quarantine the batch with a reason; do not borrow the previous batch's returns merely to fill a required role. Matching digests establish byte agreement, while trustworthy origin, authorization and business totals require separate checks.

Make one conditional update the sole activation point

Maintain a small publication pointer for each logical dataset. Here, generation 40 points to B100. The proposed generation 41 contains B101, the manifest object name, its digest and a unique commit identifier. Read the current pointer and its original ETag, prepare the complete candidate, and replace the pointer only while that ETag still matches.

RFC 9110 requires strong validator comparison for If-Match, helping prevent stale writes from overwriting concurrent changes.[3] Supported S3 conditional writes can compare an existing object through If-Match or create an absent object with If-None-Match and an asterisk.[4] Initial pointer creation and subsequent updates are different branches. A condition meaning only that an object exists is not an exact-version comparison.

Prevent overwrites of immutable data and manifests too. On rerun, check the identity and complete digest of an existing object; stop on disagreement instead of writing new bytes under an old batch name. Restrict unconditional writes that bypass the pointer protocol. One well-behaved publisher cannot guarantee correctness while another process can overwrite the pointer freely.

The numbers 40 and 41 are application commit generations, not numerical ETags. Treat the service's complete ETag as an opaque value. Every commit contains a new unique identity, and a rollback also creates a new generation. Do not restore an old pointer byte for byte and create ambiguity in which a value changed and later appears unchanged.

A successful conditional update is the activation point for this dataset. Before it, candidate files may exist but compliant consumers still process B100. Afterwards, newly starting consumers may pin B101. This does not simultaneously update every consumer's memory or establish a transaction across independent datasets. Reports requiring several datasets together need an explicit higher-level publication unit.

Pin one manifest before deciding whether results can be delivered

At startup, read the publication pointer and retain its commit ID, generation, batch, manifest location and digest. Fetch and validate that manifest, then read only the object versions it lists. Do not scan directories for additional files or fetch the latest pointer again when processing the second role. Batch identity stays fixed throughout one task.

During the B100-to-B101 switch, an earlier task may correctly produce CNY 280 and a later task CNY 350. Both are consistent with their pinned batches. CNY 380 or CNY 250 signals mixed reads. Display the business cutoff and batch alongside results so a complete old snapshot is not described as the latest data.

Check each object's identity, length, digest, format, rows and fields, followed by batch-wide roles, keys and totals. A missing object or permission denial leaves the task failed or awaiting investigation; it must not trigger fallback to an old path. Inability to read an object does not prove absence and must not cause a producer to manufacture a same-name replacement.

A downstream database can first stage records by batch. After complete validation, it can record consumption and switch the results entry point within its own supported transaction. If the destination cannot perform that atomic operation, define a recoverable state machine. Do not let the first half affect business records and then treat a failure in the second half as a simple import retry.

Notifications only announce that a batch may be available; consumers still consult trusted publication evidence. Duplicate notifications must not duplicate accumulation, and delayed notices must not make an older batch current. If the business requires processing every batch instead of reading the latest snapshot, one current pointer is insufficient: add an ordered, recoverable commit log and consumption position. That is a different delivery contract.

Resume safely after conflicts and ambiguous outcomes

Suppose publishers A and B both read generation 40, prepare B101 and B102 respectively, and attempt commits with the same old ETag. A correct implementation allows one to change the pointer first; the other must reconsider once the condition changes. S3 documents 412 for an unsatisfied condition and possible 409 responses in concurrent cases.[4] Do not classify all such responses as network errors and retry indefinitely.

If A succeeds and B observes B101 as current, B must determine whether B102 was computed from stale business inputs. Only after reconfirming scope and parent version may it prepare another valid commit. Simply obtaining the new ETag and pushing the old result is insufficient. Conditional updates prevent overwriting; they do not decide whether an older computation remains usable.

A timed-out pointer write is more difficult: the server may have accepted it while the response was lost. Start with a read-only check. If commit ID, batch and manifest digest match the original request, confirm that this content became active and repair the missing receipt. A missing local success record is not a reason to generate another batch or rewrite data.

If the pointer now contains a higher generation from another commit, that does not prove the original write never succeeded. It may have become active briefly before being superseded. Resolve this through authorized object-version history, reliable commit auditing or other durable evidence. Without such evidence, preserve an unknown state and prohibit overwriting the current result. Design recovery evidence before the incident rather than guessing its timeline afterwards.

Even if a read returns the old pointer, consider whether the original request remains in flight. Continuation must preserve the same candidate and original condition, under verified client retry semantics; never switch to unconditional PUT. If the original condition subsequently fails, inspect again rather than replace the identifier to conceal ambiguity. Stop automatic commits and hand investigation to an operator when the outcome cannot be established safely.

If B101 is later found to contain incorrect business calculations, an approved rollback can publish a new generation referencing a verified old or corrected snapshot and recording the reason. Reversing downstream business actions is a separate workflow. Switching a file pointer cannot undo orders, accounting operations or external notifications already produced.

Choose between a lightweight manifest and a table-format platform

A lightweight manifest suits deliveries with few roles, explicit batch contents and a manageable consumer population, such as a daily orders-and-returns reconciliation package. It directly represents the files as one business unit, but the team owns conditional commits, pinned reads, conflict recovery, compatibility and cleanup. A small file count does not remove those responsibilities.

If the data already form large analytic tables with continuing appends or corrections, partition pruning, concurrent writers and historical snapshots, evaluate an established table format and its catalog. Apache Iceberg represents snapshots using immutable files and metadata, with atomic metadata commits; readers use the snapshot they loaded.[5] This invokes its commit principle, not a complete implementation recipe. Confirm compatible catalog and engine versions in the actual deployment.

An ordinary JSON manifest is not a complete Iceberg implementation, and a single-table commit is not automatic consistency across business tables. If orders and returns become separate tables, independent commits may still be mixed by a reader. Bind the two snapshots at the report level or use coordination that genuinely supports the required combination semantics.

Compare alternatives against the same consumer requirements: whether old snapshots are acceptable, every batch must be consumed, only filtered subsets need reading, and historical replay is required. For a small team delivering fixed packages, platform operations may cost more than they save. Where a mature analytic platform already exists and file counts keep growing, extending a custom manifest can duplicate catalog, indexing and concurrency work.

Assign failure ownership explicitly. Producers prove completeness of objects named in the manifest; the publication service proves the conditional commit; consumers prove pinned reads and correct results; platform maintainers prove retention and cleanup boundaries. A component's success receipt covers its own responsibility. Procurement should not collapse these into an unexplained claim of data consistency.

Accept fault behavior, not just a normal upload

Test one: upload B101 orders successfully, then interrupt returns. Expect the pointer to remain at B100 and consumers to produce CNY 280. Inspect pointer and manifest evidence rather than treating an orders object in a directory as success. Repair transfer, validate all of B101 and commit it. Recheck that new tasks produce CNY 350 while old tasks continue using only B100.

Test two: both files exist, but returns contain incorrect bytes of the same length. Expect a whole-file digest or business-total failure and no commit. Retain expected and actual digests, object version and validation phase. Repair through candidate objects and manifests that respect immutable identity, not by changing an expected digest until corrupted data pass.

Test three: two producers publish concurrently from generation 40. Expect one successful conditional change and a conflict record from the other, followed by business-parent revalidation. Inspect the authoritative pointer and request receipts, not just process logs, and establish that unconditional retries cannot overwrite the winner. Replay the concurrent trigger after repair.

Test four: discard the client response after commit and invoke recovery. Expect inspection of the original commit identity without creation of another batch. Matching original manifest evidence repairs the success record; unresolvable history remains unknown. Then advance the pointer with another commit and test that recovery neither declares the original request failed without evidence nor rolls the pointer backwards.

Test five: pause a consumer after it loads B100's manifest, commit B101, and let the old consumer continue reading returns. Expect CNY 280 from the old task and CNY 350 from a new one, never either mixed net amount. Inspect the manifest and version bound to each object request. Repair any code that refreshes latest addresses mid-task, then repeat the timeline.

Test six: deliver the same batch notification twice, or attempt cleanup while an old snapshot still has readers. Expect one committed business result in the consumption ledger and continued access to the active snapshot. Inspect consumption transactions and retention records. Do not delete an object whose lack of references cannot be established. Include long-running readers, not only tiny tasks that finish immediately.

Deliver scaling limits and maintenance rules together

As file counts grow, manifest parsing, per-object requests and full-digest reads add latency. Record file count, manifest size, bytes validated, slowest read and commit waiting time to locate the bottleneck in preparation, checking or consumption. Hierarchical manifests and partitioned reads must still pin the root and every child identity. Do not reintroduce latest-file selection farther down the tree.

Separate cleanup candidates into uncommitted preparations, superseded snapshots and currently referenced objects. Delete only after the agreed recovery window and after establishing that current pointers, retained history and active tasks do not reference them. Reader leases require handling the case where a lease expires while its task continues running. Age-only deletion can break a previously correct pinned snapshot halfway through reading.

Format upgrades need compatibility checks. Unknown manifest versions, duplicate roles, unrecognized units or missing required digests should cause explicit rejection rather than guessed defaults. Preserve a usable old format during staggered producer and consumer upgrades. Revalidate new conditional-write behavior, version addressing and proxy-cache configuration in the target environment.

The final receipt should identify batch and commit, business cutoff, complete object set, individual validation outcomes, conditional-commit status, actual consumer result and unresolved exceptions. The expected example output can be recalculated from five business records, but network and concurrency fault tests still require execution and evidence in the deployment environment. Complete, current and consumed are separate properties; preserving each makes publication diagnosable and recoverable.

References

  1. [1] Amazon S3 User Guide — Data consistency model (accessed 17 November 2026)
  2. [2] Amazon S3 User Guide — Checking object integrity for data uploads (accessed 17 November 2026)
  3. [3] RFC 9110 — HTTP Semantics, sections 8.8.3 and 13.1.1, June 2022
  4. [4] Amazon S3 User Guide — How to prevent object overwrites with conditional writes (accessed 17 November 2026)
  5. [5] Apache Iceberg Table Spec — Overview, Optimistic Concurrency and Table Metadata (accessed 17 November 2026)
Back to insights
鲁ICP备2024109755号-2
Drag to move. Right-click, touch and hold, or press Shift+F10 to choose a corner.