Why JSON Interfaces Silently Change Data: Duplicate Keys, Large Integers and Parser-Boundary Tests

Successful parsing and schema validation can conceal overwritten fields and rounded identifiers. This article develops representation contracts, early duplicate detection, explicit numeric conversion, cross-runtime tests and recoverable quarantine before data reaches business tables.

One system sends order identifier 9007199254740993; another receives 9007199254740992. Both may report successful JSON parsing. This failure can occur after model generation but before business validation, or in ordinary APIs, queues and browsers. Constructed messages show how to move fidelity checks ahead of irreversible conversion.

Give the parser boundary its own interface contract

Validation before database entry for structured outputs covers complete responses, field constraints, business consistency and evidence. Here we examine an earlier boundary: what disappears when raw JSON becomes a dictionary or object? A validator normally sees the parsed value. Once members merge or a number is rounded, stricter range checks cannot reconstruct the source.

Map the real route: producer object, serialized text, gateway or broker, consumer parser, business types, database and re-export. Record language, library version and configuration at each point. Testing only the final service parser is insufficient when an SDK, gateway plugin or frontend has already performed a lossy conversion.

Our proposed exchange contract requires unique names within each object, string identifiers, an agreed decimal representation for exact amounts, and explicit rejection of numbers outside business limits. This is an engineering choice for interoperable exchange, not a claim that JSON requires every number to be a string.

Use two layers: check framing, encoding, duplicate names and numeric lexical form at the raw-message boundary; check required fields, units, ranges and relationships on typed results. Retain rule versions for both. A receipt for the latter alone cannot establish end-to-end fidelity.

Reject duplicate names before constructing ordinary objects

RFC 8259 uses SHOULD for unique object names and describes differing receiver behavior for duplicates: keeping the last value, raising an error or retaining all members.[1] Duplicates are therefore not universally rejected as syntax errors, and one parser accepting them does not establish shared interpretation.

The constructed message {"amount":100,"amount":900} contains two members with the same name. If parsing leaves only amount=900, subsequent checks for an integer greater than zero can pass. Under our policy, even identical repeated values are rejected so producer defects are not concealed. This is an input policy, not a judgment about the correct amount.

Obtain member sequences for each object, compare decoded names for duplicates, then construct the mapping. A regular-expression search across the document is unsuitable: separate nested objects may legitimately share a name, and strings can contain field-like text. Decode JSON escapes first, so literal a and escaped \u0061 count as the same name.

Python json keeps the last repeated member by default; object_pairs_hook allows construction to be intercepted while member pairs remain visible.[2] A per-object seen set can reject duplicates before returning a dictionary. object_hook receives an already constructed dictionary and cannot provide that check. Do not silently trim, case-fold or Unicode-normalize field names as part of duplicate detection.

A JSON integer is not necessarily exact in every consumer

RFC 8259 permits implementation limits on number range and precision and identifies integers within plus or minus 2 to the power 53 minus one as exactly interoperable under common binary64 capabilities.[1] Outside that interval, not every integer is wrong; the guarantee that consecutive integers remain distinguishable is lost.

Constructed order A has identifier 9007199254740992 and B has 9007199254740993. Their decimal texts differ, but converting the latter to a conventional binary64 value rounds it to the former. A consumer using that value as a deduplication key can merge the orders. Turning the rounded value back into text does not recover B.

For identifiers that are not quantities, use JSON strings at the producer and specify character set, length, leading zeros and case rules. Avoid mixed number/string representations followed by permissive consumer conversion. A legacy numeric interface needs a declared range checked before lossy conversion, against the original number representation.

Making numeric fields strings does not finish the design. Monetary strings still need currency, scale, range and rounding rules before direct decimal conversion. Integer quantities must fit every consumer. Decide explicitly whether exponent notation is allowed: numerical equality does not establish compliance with a lexical contract.

Combine strict parsing with explicit business conversion

The first gate receives a complete, size-limited message, decodes it under the contract, and rejects duplicate names, extra trailing material and unapproved nonstandard numbers. Limit large arrays, nesting and numeric length as well. Parser capacity is not a business acceptance policy. Retain raw evidence only in controlled storage, with ordinary logs carrying references.

Python can combine object_pairs_hook, parse_float and parse_constant: the first checks members, the second can construct Decimal directly from the numeric text, and the third can reject extensions such as NaN and Infinity.[2] Converting through float before Decimal cannot preserve original decimal precision. Apply integer range rules explicitly; Python's larger integer capacity says nothing about browser or other-language safety.

The second gate performs field-specific business conversion. String identifiers remain strings; amounts need scale and currency checks; quantities need integral values and bounds; booleans must not accept numeric 0 and 1 as substitutes. Treat missing, null, empty text and zero separately. Verify implicit coercion through the actual request path.

A setting named strict is insufficient. Python json's strict option primarily concerns control characters in strings, rather than automatically rejecting all duplicate names or special numbers.[2] Assign every policy an enforcement point and prove with counterexamples that it runs before the first lossy parse. An interface that discarded lexical form cannot promise to identify the original spelling afterward.

Keep business values and necessary source evidence separately

PostgreSQL 18 distinguishes json from jsonb: json retains input text and repeated members, while processing functions use the last value; jsonb retains only the last duplicate value.[3] Inserting raw messages into jsonb and then looking for repeated names is too late. A database type does not replace input policy.

Where investigation requires it, retain original bytes or text and a digest in a controlled evidence area, with verified business fields in authoritative tables. Text preservation by json neither rejects duplicates for the business nor proves upstream bytes never changed. Original evidence and queryable typed results serve distinct purposes.

An exact database numeric type cannot repair rounding that already happened in a client. Read values back through the actual driver, API and browser during acceptance, confirming identifiers remain intact. Successful insertion into numeric or inspection of a formatted table does not cover the return path.

If raw messages carry signatures or agreed digests, verify the relevant bytes under the established protocol. Do not silently parse and reorder fields before calculating and call that original-message verification. Semantic equivalence and byte identity are different requirements. This article defines no signing standard; canonicalized signing needs its own explicit protocol and implementation.

Pass a small counterexample set through every real runtime

Start with duplicate names in one object, repeated identical values, duplicates inside nested objects, names equal after escape decoding, and legitimate equal names in separate objects. Reject the first four under this contract and accept the last. Record the rejection layer, whether side effects occurred, and a locatable path or parsing error without copying the entire sensitive message into logs.

Test 9007199254740991, 9007199254740992 and 9007199254740993 as both numbers and strings. Identifier strings must survive character for character; numeric forms follow the declared range. Add 12.30, exponent notation, out-of-range amounts, NaN, Infinity, nulls and absent fields. Specify expected results individually instead of merely requiring no exceptions.

Compare two routes: identical raw text sent directly to the service, and the same text passed through the real gateway, SDK, queue and frontend. Compare types, values, accept/reject decisions and final records at each stage. If only the real route fails, locate the first conversion instead of hiding the loss with a tolerance at the final layer.

Recovery tests should cover interruption before commit, a successful database commit with a lost receipt, and repeated arrival of the same business request. Changed parsing policies require revalidation of source input with a recorded processing version. Revalidation does not authorize another payment or shipment; original business idempotency and authorization still govern side effects.

Stop failures where recovery remains possible

Reject or quarantine duplicate-name messages for the producer to resolve. Do not choose the larger or last value, or ask a model to guess. For rounded identifiers, recover the authoritative value and investigate incorrect associations already created. Converting the current value to a string prevents a future conversion but does not reverse earlier loss.

A concrete delivery consists of a representation contract, a checker at the first parsing boundary, cross-route counterexamples and quarantine records linked to originals and rule versions. Distinguish receipt, parsing-fidelity acceptance, business validation and authoritative commit. One HTTP 200 cannot represent all four.

Sources are RFC 8259, Python 3.12 and PostgreSQL 18 documentation, checked on October 26, 2026. Identifiers and messages are constructed examples; the acceptance workflow is for implementation in the actual chain, not evidence of completed enterprise production tests. Earlier checks preserve information needed for decisions; no single parser option resolves every cross-language difference.

References

  1. [1] IETF: RFC 8259, The JavaScript Object Notation (JSON) Data Interchange Format, December 2017
  2. [2] Python 3.12: json — JSON encoder and decoder (accessed 2026-10-26)
  3. [3] PostgreSQL 18: JSON Types (accessed 2026-10-26)
Back to insights
鲁ICP备2024109755号-2
Drag to move. Right-click, touch and hold, or press Shift+F10 to choose a corner.