Safely Importing ZIP Data: Path Boundaries, Name Collisions and Decompression Budgets

Treat an archive as a batch awaiting acceptance

An ingestion workflow needs at least five states: original received, manifest accepted, members completely read, content checks passed, and batch available downstream. Successful HTTP upload establishes only the first. Triggering indexing after extracting the first file can expose half a batch when a later member fails, leaving difficult-to-trace gaps in a knowledge base or training dataset.

Assign a stable batch identifier and record source, receipt time, original-archive digest, processing policy and runtime version. Retain the original under access controls for investigation and give staging storage a lifecycle. A digest can identify an input, but without a trusted comparison value it does not establish authenticity or trustworthiness.

Specify permitted formats and business needs first. A workflow needing a few CSV files need not accept arbitrary executables, encrypted archives or unlimited nesting. Unsupported algorithms, unavailable decryption and malformed input require explicit failure states. An unread member must not silently become an empty file.

Keep archive names out of physical storage paths where possible

Path traversal occurs when an external name resolves outside an intended directory; CWE-22 covers both relative and absolute paths.[2] Removing one dangerous substring is not an adequate boundary. If the business does not need the original hierarchy, assign server-generated storage identifiers and retain original names only in a manifest for presentation and traceability.

For example, reports/nov.csv can map to member 0007 within a batch, while the manifest records its original name, entry ordinal, actual byte count and content digest. Downstream parsers access files through the manifest rather than reconstructing paths from original names. Names still need context-appropriate escaping when displayed; not using them for storage does not make them trusted text.

If preserving directories is necessary, define an allowed relative-path grammar, maximum depth and component length. Validate absolute paths, parent components, drive or network paths, separators and reserved names against the destination platform. Determine containment using path components, not a string prefix. A name beginning with a directory's text is not necessarily inside it.

Validation must be followed by writes that resist symbolic-link redirection and concurrent replacement. Use a fresh server-created directory writable only by the worker, and do not restore links, devices or other non-regular members. Where hierarchy fidelity is required, use platform-appropriate safe file-creation mechanisms and test races. Static path checks do not replace isolation at write time.

Python distinguishes filename handling in ZipFile.extract/extractall from zipfile.Path, whose member names require caller validation.[1] Protection in one extraction interface does not automatically transfer to an application that uses Path or reads member streams and writes files manually.

Duplicate names must not become last-write-wins

An archive may contain repeated names or names that compare equal in the destination environment. Do not first build a name-keyed dictionary and only then search for duplicates: entries may already be lost. Preserve directory-entry order and distinct identities, and define whether ambiguity rejects the entire batch or requires review. The last entry must not silently overwrite an earlier one.

Unicode normalization and identifier collisions explains how transformations collapse distinctions. File ingestion must also examine case, trailing characters, path transformations and destination comparison rules. Equality differs across platforms, so demonstrating distinct names on a developer's computer is insufficient.

Generated storage identifiers prevent physical overwrite but leave business ambiguity. Two identically named sales tables might be revisions, different stores or repeated uploads. If the manifest cannot distinguish them, do not arbitrarily choose the official data. Separate safe storage from business version selection: retaining both files does not prove correct ingestion.

Budget bytes, entry counts and execution time together

CWE-409 describes resource amplification from highly compressed input.[3] OWASP's upload guidance likewise calls for considering decompressed size, not just upload size.[4] Neither establishes one compression-ratio threshold suitable for every enterprise. Repetitive text naturally compresses well; a large, poorly compressed file can also exhaust capacity.

Manifest inspection can reject obvious excesses using declared sizes, entry counts and compression methods, but those are input metadata rather than runtime enforcement. Read bounded chunks and meter actual output. Separately limit bytes per member, bytes per batch, entry count, directory depth and nesting depth. Check that the remaining budget accommodates the next chunk before writing it.

Outside the worker, enforce memory, CPU, wall-clock and disk limits. Parsing or decompression can consume resources without producing many bytes, and checks inside a read loop do not cover directory parsing or a single library call. Enumerate the manifest inside the constrained worker too. Bound queues and concurrency so individually acceptable batches cannot collectively exhaust storage.

Consider a constructed pilot policy: at most 100 members, 20 MiB per file, 200 MiB total output, and no automatic extraction of inner archives. These are illustrative business choices, not an industry safety standard. After nine 20 MiB files, if the tenth exceeds the remaining 20 MiB, stop, fail the batch and remove unpublished output. Do not truncate it into an apparently valid file.

If nested extraction is necessary, all levels share the root batch budget, with additional depth and cumulative-processing limits. Each level must not receive a fresh full allowance. Downstream PDF, image and spreadsheet parsing also needs separate resource controls; successful decompression does not establish cheap content processing.

Stream into isolation, check content, then publish the complete batch

A recommended sequence is to retain and register the archive; enumerate members in a constrained worker; validate format, count, mappings and conflicts; read each member in bounded chunks into fresh staging files; confirm complete reads and library checks; perform approved content checks; write the final manifest; and only then mark the batch available. Failure at any stage must prevent partial publication.

Python's documentation identifies format, password, CRC, compression-method and resource problems as possible failure causes.[1] Partial bytes are not successful extraction when checks remain unfinished. Integrity checks such as CRC do not establish that content is harmless. Type identification, malicious-content inspection and business-field validation have distinct responsibilities.

OWASP recommends separating upload storage and limiting permissions.[4] In this ingestion design, staging is not a web-served static directory, files receive no execute permissions, and the worker has only the access needed for its batch. A timed-out or unavailable content scanner leaves a check incomplete; it must not silently become a pass.

Whole-batch publication can use versioned directories and a controlled manifest pointer. Downstream readers admit only accepted batches with complete manifests. Atomic rename depends on the storage and filesystem; behavior on one local filesystem does not generalize to object storage or cross-volume moves. Object storage can use a separate completion record as the visibility gate, with verification of every referenced object.

Prove rejection, cleanup and recovery with counterexamples

Acceptance cases should include an ordinary small archive, an empty archive, exact-budget input, output exceeding a budget by one read chunk, duplicate names, destination name collisions, escaping paths, unsupported algorithms, damaged members and forbidden nested content. Define the expected state beforehand. Business policy decides whether an empty archive is permitted, but it must not be reported as containing valid imported data.

Check side effects as well: rejection creates or changes nothing outside the permitted directory; existing published batches remain intact; staging space is released; downstream consumers receive no partial batch; and diagnostics identify the original input and failure stage. Run these assertions in an isolated test environment rather than using important real files as test targets.

Simulate worker termination, exhausted disk budget, content-check timeout and failure to write the completion marker. Recover from the batch record; unfinished output must not be promoted directly. Reprocessing the retained archive in a new isolated staging area can be appropriate for the same batch, but outward publication must remain idempotent so two completion events cannot cause duplicate ingestion.

The handoff should include permitted formats, member-identity policy, budgets, rejection rules, cleanup deadlines, counterexample results and downstream visibility rules. Safe import means every member reaches only an authorized location, resource use remains bounded, and failure cannot become half a published dataset. It requires more than a function that reports decompression success.