Two part numbers look almost identical but fail to join. A character-cleaning step improves the match rate, yet inventory is assigned to the wrong object. This need not be a model error or damaged encoding. This article addresses string normalization: preserve traceable originals, define comparison semantics and block unapproved identity collisions. All part numbers and object relationships are constructed examples.
Define the comparison before transforming characters
String equality, search similarity and shared business identity are different judgments. Customer names may benefit from permissive retrieval, whereas supplier part numbers may distinguish case and hyphens. Enterprise customer entity resolution asks whether records refer to one object. This article examines the preceding step: what character transformations preserve and what they discard.
Distinguish raw_value, display_value, search_key and identifier_key. Raw text supports provenance, display values support presentation, search keys broaden candidate retrieval, and identity keys must follow the issuing authority's rules. Not every field needs four physical copies, but one cleaned field must not silently conflate those purposes.
Register the namespace, allowed characters, case sensitivity, whitespace and punctuation policies, and who can approve aliases. Code 001 in source A is not automatically the same as 001 in source B. Stable record identity should retain source and original record identifier rather than depend solely on a transformed name.
Establish the cost of errors. Missing a search candidate differs from merging inventory for two components. A search key can be many-to-one while automatic identity mapping remains prohibited. Without an approved code policy, retain the original identifier and flag uncertainty instead of inventing a global cleaning rule.
NFC and NFKC address different forms of equivalence
Unicode UAX #15 distinguishes canonical and compatibility equivalence. NFC performs canonical decomposition followed by composition; NFKC additionally addresses compatibility distinctions.[1] For example, é can be represented by U+00E9 or by e followed by combining accent U+0301. NFC unifies those representations. It is not a general accent-removal or ASCII-conversion operation.
Compatibility normalization is broader. Fullwidth Latin letters and ordinary Latin letters, or circled and ordinary digits, can yield identical NFKC results. The standard cautions against blindly applying compatibility normalization to arbitrary text because erased distinctions may carry meaning.[1] A tidier appearance does not authorize overwriting originals.
Consider a constructed name search: two records display Café, one with precomposed é and another with the combining sequence. An NFC-normalized search index can reduce representation-related retrieval misses. It does not establish that both records identify one company. Character equivalence resolves a representation difference; entity identity still needs further evidence.
For serial numbers, contract references, external-system keys and text awaiting signatures, establish whether the source protocol permits any transformation. Even when NFC is approved, derive a new value from the retained original rather than claim that original bytes are unchanged. Character transformations can alter bytes; byte-level verification and business text comparison require separate treatment.
Four part numbers expose a collision that cannot authorize merging
Construct four independent records R1, R2, R3 and R4 in one source, with codes AB-01, ＡＢ-01, AB01 and АB-01. R2 uses fullwidth A and B. R4 begins with Cyrillic А, U+0410, whereas R1 begins with Latin A, U+0041. There is no business evidence that any pair identifies the same component.
Applying only NFKC gives R1 and R2 the search key AB-01. R3 remains AB01, and R4 retains its distinct initial character. This creates one candidate collision group containing two source records, not one confirmed duplicate. Keeping only the first row per search key would discard R1 or R2 without identity evidence.
Do not remove hyphens merely to improve match rate. Adding that transformation brings R3 into the first two records' group; this is an additional policy, not an effect of NFKC itself. Record each transformation's input, output and rule name so a new association can be attributed to width conversion or punctuation removal.
Handling depends on the code system. If only ASCII is permitted, report R2 and R4 as contract violations for source correction or approved aliases. If wider character sets are permitted, preserve them and create reviewable candidate links. A policy must not allow distinct original codes while silently collapsing their identities through a unique index.
Case folding and visual confusability are separate operations
Caseless comparison is an independent decision. Python documents that casefold is stronger than lower: German ß becomes ss.[2] It can therefore change length rather than substitute characters one-for-one. Search may permit that equivalence; an external identifier may not.
Do not treat NFKC, casefold, whitespace removal, accent removal and punctuation deletion as a universally interchangeable recipe. Version the approved order and check length, character repertoire and empty output afterward. A custom function composition must not be labeled as a particular Unicode-standard algorithm without verifying that claim.
Visual confusability is another problem. Unicode UTS #39 supplies detection mechanisms and states that a skeleton is for internal confusability testing, not identifier normalization or direct user display.[3] A similar character such as R4's initial may justify review, but possible visual confusion does not establish identical codes.
Show reviewers the original text, code points, scripts and source record, highlighting invisible characters when needed. Do not delete every zero-width or joining-control character universally; some writing systems use them meaningfully. A restricted code field can reject them under its contract, while personal names and natural language require appropriate language rules.
Identical cleaning code does not guarantee identical database equality
Strings continue through databases, search indexes, caches and exports. PostgreSQL 17 distinguishes deterministic and nondeterministic collations; the latter can, depending on configuration, consider differently encoded strings equal, including case-insensitive or other insensitive comparisons.[4] Inequality in Python alone does not establish that a destination unique index permits both values.
For derived keys requiring exact identity comparison, specify storage types and comparison behavior, then test equality, joins, grouping and uniqueness in the actual database. Natural-language ordering and code identity need not use one universal collation. The requirement is to know where cross-layer semantics differ.
Group potential collisions by namespace, rule version and comparison key, counting distinct source records, originals and established entity identities. Separate existing duplicates, new rule-induced equivalences and approved aliases. Do not count distinct originals under a permissive database comparison that may already erase their differences; use a controlled byte representation where necessary.
Quarantine candidate collision groups with their members and resolution reasons while fact tables continue to reference stable record identities. For inventory and amounts, reconcile fact rows and totals before and after transformation. A text mapping can change grouping but must not silently delete orders or stock facts. Metric changes need a traceable mapping version.
Compute in shadow before switching rule versions
Preserve original input and source metadata, and decode using the declared encoding. Send decoding failures to an exception path instead of ignoring invalid bytes to obtain plausible text. Derive old and new comparison keys independently from the original, recording rules, runtime and Unicode data versions. Python exposes its character-database version through unicodedata.unidata_version.[5]
Referencing Unicode 18.0.0 documentation here does not establish that a deployed runtime supports that version. Version records explain actual behavior and character coverage; they do not imply every old character must change after an upgrade. Compare custom rules, case handling, collation and newly supported characters using the actual implementation.
In shadow operation, record new collision groups, affected records, invalid inputs, candidate-retrieval changes and review workload. Resolve identity conflicts before enabling the new rule in production queries. Switch field by field if scope differs. Names, passwords, serial numbers and signature inputs should not share one cleaner merely to simplify code.
Rollback should restore the old policy and corresponding index, not try to reconstruct original text from a lossy search key. Retain originals, mapping history and downstream dependencies for replay. Storage and index maintenance have costs, but overwritten distinctions often cannot be recovered through an inverse function; decide that tradeoff at ingestion.
Test values that should match and values that must remain distinct
Build fixtures with explicit expectations: precomposed and combining-accent forms match under NFC; fullwidth AB and ASCII AB match under NFKC; Latin A and Cyrillic А remain distinct under either form; and AB-01 differs from AB01 when punctuation is retained. State whether each assertion concerns a string transformation or an approved business identity.
Include nulls, empty strings, whitespace-only values, length boundaries, case-sensitive codes, circled digits and control characters. Null must not accidentally become the literal string 'None' or share an identity with empty text. Compare code points or bytes, not screenshots alone, because fonts and rendering can obscure differences.
Check idempotence and consistency throughout the path. Repeated processing of the same original under one rule should produce the same key; reprocessing normalized input should not introduce undesigned changes. Application code, storage, queries, caches and export/reimport must satisfy their respective contracts. Idempotence is necessary but insufficient: a rule that consistently merges the wrong objects remains wrong.
The handoff should include field contracts, original-retention policy, versioned rules, counterexamples, collision resolutions and switch-and-rollback evidence. Better search recall can be valuable, but it cannot replace identity-correctness acceptance. Reliable normalization makes approved differences comparable while preserving differences that still matter.
References
- [1] Unicode Standard Annex #15 — Unicode Normalization Forms, revision 58, Unicode 18.0.0, 2026-08-12
- [2] Python — str.casefold (accessed 2026-10-24)
- [3] Unicode Technical Standard #39 — Unicode Security Mechanisms, revision 34, 2026-08-27
- [4] PostgreSQL 17 — Collation Support
- [5] Python — unicodedata: normalize and unidata_version (accessed 2026-10-24)