Reports, training datasets and industrial data files commonly arrive through HTTP or object storage. After an interrupted transfer, the dangerous outcome is often a plausible-length file assembled from different versions. This article addresses transmission of an already generated file. Examples are constructed reasoning, not performance measurements or executed fault experiments.
Freeze file identity and separate bytes from business completeness
A recoverable download record should include the export job, object version or strong validator, expected length, representation encoding, checksum algorithm, scope and expected value. A URL is an access route: refreshing a signed URL need not change the object, while an unchanged URL can serve changed content. Bind recovery to stable file identity rather than its filename alone.
Separate three questions: are the bytes from one version, are all of that version's bytes correct, and does the file represent the intended business scope with usable content? Length detects some truncation but not equal-length replacement. A digest compares bytes; an expected digest taken from the wrong export cannot detect incorrect export filters. Check authorization scope, export parameters and data boundaries separately.
If generating the source requires scanning changing data, use the snapshot acceptance method for paginated collection. Our concern begins after generation. A correct snapshot can still be damaged by resume logic, encoding or storage failures; successful transmission cannot establish the source snapshot's business completeness.
Do not equate ETags, composite checksums and whole-file hashes
An HTTP ETag is a representation validator, not a universal file-digest format.[1] Preserve a strong ETag for conditional requests rather than assuming its unquoted value is the local file's MD5. A weak validator prefixed with W/ is unsuitable for the byte-exact resume condition used here.
Storage services add specific boundaries. Amazon S3 documents that multipart-upload ETags are not whole-object MD5 values, and certain encryption modes also invalidate that assumption. Full-object and composite checksums have different scopes and calculation rules.[3] Removing a suffix from a checksum does not turn it into a whole-file hash.
Specify the triple of algorithm, checksum type and byte scope. One option is a trusted export manifest containing SHA-256 for the complete delivered file; another is a documented storage-service mechanism correctly implemented by the client. The manifest option does not imply that a particular S3 multipart checksum algorithm supports full-object validation. Check the relevant service operation.
Bind expected digests to object versions and obtain them through a trusted channel. A digest alone is not authentication. If an attacker can replace both file and expected digest, equality does not establish trustworthy provenance. Transmission-corruption detection and supply-chain authenticity are different requirements; adding a hash is not a complete security design.
Define the checked bytes before handling compression and ranges
RFC 9530 separates Content-Digest, covering HTTP message content, from Repr-Digest, covering selected representation data.[2] In a range response, the transferred fragment and full representation are different scopes. Passing a fragment checksum does not establish completion of the whole file.
Content-Encoding also affects which bytes are checked. A client that transparently decompresses gzip and hashes decoded text may be comparing a different byte sequence with an HTTP digest. Text decoding, newline conversion, BOM handling and reserialization likewise change bytes. Perform such transformations after accepting the original deliverable.
A simpler implementation contract requests identity encoding, verifies that the actual response follows it, and stores and resumes raw file bytes. A preference header does not guarantee server compliance; unexpected encoding must stop resume and follow an explicit policy. If an encoded representation is required, fix it, preserve its byte coordinates and distinguish the compressed artifact's digest from that of decoded output. Never reuse offsets across those coordinate systems.
Digest values may arrive in trailers. A client that neither reads nor validates them has not performed that check; downstream consumption before the expected value arrives also bypasses the integrity gate.[2] Choose a client exposing the required byte stream and fields, and record its operating mode rather than merely an HTTP status.
Branch on resume responses instead of appending blindly
Construct a ten-byte ASCII file: V1 is ABCDEFGHIJ. The client durably saves ABCD, then the source becomes equal-length V2, abcd123456. Resuming at offset four without version binding produces ABCD123456. It is ten bytes long but is neither version. This is a byte-level illustration, not a network experiment.
For a mutable URL, send Range with If-Range containing the original strong ETag. RFC 9110 specifies ignoring the range and returning the full representation when that condition does not match.[1] If immutable object-version addressing is available, bind that version to the manifest too. Without a reliable version mechanism, retrieve and validate a complete file rather than describe cross-version assembly as safe resume.
For 206, check that Content-Range starts at the requested checkpoint, that its end and total length are reasonable, and that the actual fragment length matches the declared range. The six remaining bytes in this example are bytes 4-9/10, with inclusive endpoints. Also verify agreement with the task's identity and representation. Stop merging on version or encoding changes. This workflow accepts single ranges only, not multipart/byteranges.
For 200, do not append a full response to the existing prefix. Isolate old staging data and handle a complete download. If the contract requires the original version, establish whether it remains retrievable rather than silently switching to the latest one. A 416 response says the requested range cannot be satisfied; it does not automatically certify local completion. Identity, total length and the whole-file digest still require validation.
Checkpoints represent durably committed data, not bytes merely received by the network layer. Retry a fragment at its defined offset or recognize its committed identity; do not append it twice. Parallel downloads require each fragment's start, end, length, version and completion state. Confirm complete interval coverage with no gaps or unintended overlap. Preallocating a file to its expected size does not prove that its bytes were written.
Order complete validation before visible delivery
Use distinct states: version located, downloading, bytes verified, format verified and delivered. Write fragments into an isolated staging object. Once assembled, recompute the agreed whole-file digest from the actual stored file and check length. Hashing the incoming network stream alone does not validate the subsequent storage write.
After byte acceptance, parse the format and check encoding, required fields, record counts and business totals supplied by the source manifest. If business validation fails, retain the successful byte check but withhold delivery. This distinguishes transmission faults from source-data problems rather than treating both as reasons for network retries.
Consumers should read explicitly completed versions. Where the same filesystem supports the necessary semantics, prepare a temporary file and switch visibility with atomic rename; crash durability still requires the filesystem's synchronization rules. Object storage must not be assumed to provide filesystem rename semantics. Upload a separate completed object and update a publication manifest or pointer conditionally according to service capabilities.
If a database import follows, bind its batch identifier to file identity so repeated notifications do not duplicate data. Record file acceptance, business import and consumer notification separately. Without a shared transaction across stores, use recoverable commit records rather than hiding intermediate stages behind one success boolean.
Accept recovery with a fault matrix, not one successful transfer
Test one: disconnect during a response. Expect no delivery and resume only from committed fragments. Test two: replace the source with an equal-length version. Expect rejection of mixed assembly, followed by restart or failure according to the original-version contract. Test three: ignore Range and return 200. Expect the full-download branch, never an old prefix followed by a full file.
Test four: inject incorrect Content-Range values, duplicate fragments or missing fragments. Expect a located range error and withheld delivery. Test five: enable unexpected decompression or return unexpected encoding. Expect detection of byte-scope mismatch rather than changing algorithms to hide it. Test six: alter one byte without changing length. Expect whole-file digest failure.
Also cover empty files, unread digest trailers, 416 responses, crashes after storage writes but before checkpoint updates, and reruns after delivery. Declare expected states and permitted recovery actions in advance. This article proposes an acceptance design; it has not executed these injections and supplies no pass rate, speedup or universal chunk size.
A completion record should contain file identity, representation encoding, expected and actual byte counts, algorithm and scope, digest comparison, format and business checks, delivered version and unresolved exceptions. Full rereads cost disk I/O; parallel fragments add state-management work. Choose according to file size and recovery requirements. Those costs can be measured and traded, but HTTP success cannot replace explicit evidence of an intact deliverable.