Skip to main content
Development

Unicode Normalization Explained: Why Identical Strings Break

Two strings look identical on your screen. Compare them with === and you get false. Before debugging your code, check the strings themselves: one of them probably encodes "é" in a different byte sequence than the other. This is the single most common Unicode bug, and it hits login forms, search, deduplication, and slugs alike.

The problem: one character, several encodings

Unicode has more than one way to represent accented characters. "é" (U+00E9) is a precomposed code point — one character, one unit. The same visible letter can also be written as "e" (U+0065) followed by a combining acute accent (U+0301). Two code points instead of one. Both render identically in every browser, but they are different byte sequences:

  • 0x65 0xCC 0x81 (decomposed, NFD-style) — 3 bytes in UTF-8
  • 0xC3 0xA9 (composed, NFC) — 2 bytes in UTF-8

A database column, a hash, or a strict string comparison sees them as different values. A user sees nothing unusual. That is the whole bug.

Where the two forms come from in practice

Neither form is a glitch. Different platforms just default differently:

  • macOS (HFS+/APFS) stores filenames in NFD. A filename containing "é" created on a Mac arrives decomposed; the same name typed on Linux or Windows defaults to NFC. Sync these directories and you can end up with two entries that look like duplicates but aren't.
  • JavaScript and web forms pass through whatever the OS keyboard produced, so the same user can submit NFC today and NFD tomorrow.
  • Copy-paste from PDFs, Word documents, and older CMS exports is the classic source of decomposed text and stray combining marks in database fields.
  • Languages like Vietnamese stack two diacritics on one base letter, producing even more decomposed combinations that render fine but compare false.

NFC and NFD: the two forms that matter

The Unicode standard defines four normalization forms, but two cover almost every real case:

  • NFC (Canonical Composition, "normalize to composed") — decomposed sequences are merged into precomposed code points where possible. e + U+0301 becomes U+00E9. Shortest storage, matches what most users type.
  • NFD (Canonical Decomposition, "normalize to decomposed") — precomposed characters are split into base plus combining marks. Best for text processing that wants access to individual marks.

NFKC and NFKD additionally replace compatibility characters: ligatures like fi break into "fi", superscript ² becomes the digit 2, and full-width Latin letters collapse to ASCII. That is a different tool — the visual identity is destroyed on purpose. Never apply NFK* to user-entered text you want to display back; use it only when comparing or extracting, and only after understanding what it will flatten.

One hard rule: choose one form for your whole system and normalize every string at every boundary where text enters it — form submissions, API payloads, file uploads, CSV imports. NFC is the sensible default for web apps. Mixed-form data is what turns into unfindable duplicate rows years later.

Real bugs this causes

  • Login and identity. A user registers with an NFC-encoded "Müller", then their Mac keyboard or password manager autofills NFD. Hash the string and the two never match — password reset loop, with no visible difference for the support team.
  • Slugs and URLs. Slug generators that strip accents by removing non-ASCII code points can drop a decomposed letter entirely or emit both the base letter and the mark, producing different slugs for the same word on different runs. Slugify after normalizing.
  • Search and deduplication. Even exact matches fail when the two copies of one word differ in encoding. Hash- or index-based dedup silently misses them.
  • Limits and counters. A MySQL VARCHAR(255) counts bytes under utf8mb4; a 3-byte decomposed representation can overflow where the composed one fits. The same text passes validation on one path and fails on another.
  • Regex matching. A pattern like ^\w+$ treats combining marks as separate tokens, so a visually simple name matches differently depending on which form arrived.

How to fix it at every layer

Normalization is well-supported everywhere; the fix is usually one call at the right boundary:

  • JavaScript: str.normalize('NFC') — built into String.prototype, cheap enough to run on every form field.
  • PHP: Normalizer::normalize($s, Normalizer::FORM_C) from ext-intl. Plain PHP has no fallback, so verify intl is enabled on the server before relying on it.
  • Python: unicodedata.normalize('NFC', s).
  • Database: MySQL does not normalize for you in most versions — store normalized data, don't expect the storage layer to fix it. Postgres 13+ has the normalize() function and the is normalized check.
  • Filesystem: re-normalize filenames on sync/copy jobs, or a Mac-produced NFD name and a Linux NFC clone become two entries.

Also normalize before hashing, before inserting, and before writing anything into a unique-keyed column. A hash of unnormalized text is just a shorter version of the bug.

How to check your own data

Dump suspect strings to code points and look at the sequence. "é".codePointAt(0).toString(16) in a console returns e9 for NFC; iterating the full string under NFD returns two values. In practice you don't inspect every string by hand — run bulk normalization over a copy of your tables, then diff before/after to see which fields actually change. A quick pass with a Unicode normalizer tool surfaces exactly those fields.

Combining-mark noise rides along with encoding issues. Stray zero-width spaces, soft hyphens, and CRLF-only line endings cause their own invisible mismatches — a cleanup with a line-ending normalizer or a comparison in a text diff checker exposes pairs of strings that look equal but are not, and shows exactly which bytes differ.

The one-minute checklist

  • Pick NFC. Write it into the coding standard as the only accepted form.
  • Normalize every text input at the boundary — forms, APIs, file uploads, CSV rows.
  • Never hash or unique-key unnormalized text.
  • Slugify after normalization, so accent stripping works on composed characters.
  • Pick NFKC/NFKD only for comparison or extraction — never for data you display back.
  • Test with a decomposed string in your fixtures: macOS filename, copy-paste artifacts, Vietnamese diacritics.

The bug is invisible in every review — the strings render identically. The fix is one deterministic call in the right place. Add the boundary normalization once and the whole class of "identical" strings stops failing comparisons.