Unicode Normalization Explained: Why Identical Strings Break
Two strings look identical on your screen. Compare them with === and you get false. Before debugging your code, check the strings themselves: one of them probably encodes "é" in a different byte sequence than the other. This is the single most common Unicode bug, and it hits login forms, search, deduplication, and slugs alike.
The problem: one character, several encodings
Unicode has more than one way to represent accented characters. "é" (U+00E9) is a precomposed code point — one character, one unit. The same visible letter can also be written as "e" (U+0065) followed by a combining acute accent (U+0301). Two code points instead of one. Both render identically in every browser, but they are different byte sequences:
0x65 0xCC 0x81(decomposed, NFD-style) — 3 bytes in UTF-80xC3 0xA9(composed, NFC) — 2 bytes in UTF-8
A database column, a hash, or a strict string comparison sees them as different values. A user sees nothing unusual. That is the whole bug.
Where the two forms come from in practice
Neither form is a glitch. Different platforms just default differently:
- macOS (HFS+/APFS) stores filenames in NFD. A filename containing "é" created on a Mac arrives decomposed; the same name typed on Linux or Windows defaults to NFC. Sync these directories and you can end up with two entries that look like duplicates but aren't.
- JavaScript and web forms pass through whatever the OS keyboard produced, so the same user can submit NFC today and NFD tomorrow.
- Copy-paste from PDFs, Word documents, and older CMS exports is the classic source of decomposed text and stray combining marks in database fields.
- Languages like Vietnamese stack two diacritics on one base letter, producing even more decomposed combinations that render fine but compare false.
NFC and NFD: the two forms that matter
The Unicode standard defines four normalization forms, but two cover almost every real case:
- NFC (Canonical Composition, "normalize to composed") — decomposed sequences are merged into precomposed code points where possible.
e + U+0301becomesU+00E9. Shortest storage, matches what most users type. - NFD (Canonical Decomposition, "normalize to decomposed") — precomposed characters are split into base plus combining marks. Best for text processing that wants access to individual marks.
NFKC and NFKD additionally replace compatibility characters: ligatures like fi break into "fi", superscript ² becomes the digit 2, and full-width Latin letters collapse to ASCII. That is a different tool — the visual identity is destroyed on purpose. Never apply NFK* to user-entered text you want to display back; use it only when comparing or extracting, and only after understanding what it will flatten.
One hard rule: choose one form for your whole system and normalize every string at every boundary where text enters it — form submissions, API payloads, file uploads, CSV imports. NFC is the sensible default for web apps. Mixed-form data is what turns into unfindable duplicate rows years later.
Real bugs this causes
- Login and identity. A user registers with an NFC-encoded "Müller", then their Mac keyboard or password manager autofills NFD. Hash the string and the two never match — password reset loop, with no visible difference for the support team.
- Slugs and URLs. Slug generators that strip accents by removing non-ASCII code points can drop a decomposed letter entirely or emit both the base letter and the mark, producing different slugs for the same word on different runs. Slugify after normalizing.
- Search and deduplication. Even exact matches fail when the two copies of one word differ in encoding. Hash- or index-based dedup silently misses them.
- Limits and counters. A MySQL
VARCHAR(255)counts bytes under utf8mb4; a 3-byte decomposed representation can overflow where the composed one fits. The same text passes validation on one path and fails on another. - Regex matching. A pattern like
^\w+$treats combining marks as separate tokens, so a visually simple name matches differently depending on which form arrived.
How to fix it at every layer
Normalization is well-supported everywhere; the fix is usually one call at the right boundary:
- JavaScript:
str.normalize('NFC')— built intoString.prototype, cheap enough to run on every form field. - PHP:
Normalizer::normalize($s, Normalizer::FORM_C)from ext-intl. Plain PHP has no fallback, so verify intl is enabled on the server before relying on it. - Python:
unicodedata.normalize('NFC', s). - Database: MySQL does not normalize for you in most versions — store normalized data, don't expect the storage layer to fix it. Postgres 13+ has the
normalize()function and theis normalizedcheck. - Filesystem: re-normalize filenames on sync/copy jobs, or a Mac-produced NFD name and a Linux NFC clone become two entries.
Also normalize before hashing, before inserting, and before writing anything into a unique-keyed column. A hash of unnormalized text is just a shorter version of the bug.
How to check your own data
Dump suspect strings to code points and look at the sequence. "é".codePointAt(0).toString(16) in a console returns e9 for NFC; iterating the full string under NFD returns two values. In practice you don't inspect every string by hand — run bulk normalization over a copy of your tables, then diff before/after to see which fields actually change. A quick pass with a Unicode normalizer tool surfaces exactly those fields.
Combining-mark noise rides along with encoding issues. Stray zero-width spaces, soft hyphens, and CRLF-only line endings cause their own invisible mismatches — a cleanup with a line-ending normalizer or a comparison in a text diff checker exposes pairs of strings that look equal but are not, and shows exactly which bytes differ.
The one-minute checklist
- Pick NFC. Write it into the coding standard as the only accepted form.
- Normalize every text input at the boundary — forms, APIs, file uploads, CSV rows.
- Never hash or unique-key unnormalized text.
- Slugify after normalization, so accent stripping works on composed characters.
- Pick NFKC/NFKD only for comparison or extraction — never for data you display back.
- Test with a decomposed string in your fixtures: macOS filename, copy-paste artifacts, Vietnamese diacritics.
The bug is invisible in every review — the strings render identically. The fix is one deterministic call in the right place. Add the boundary normalization once and the whole class of "identical" strings stops failing comparisons.