Skip to main content
Text Processing

Sorting Text the Wrong Way: ASCII Order and Its Traps

Sorting looks like a solved problem until it produces output like: file10 before file2, Zebra before apple, and a lowercase list interleaved with uppercase nonsense. That is not a bug in your editor. That is lexicographic sorting doing exactly what the spec says: comparing character code points one by one.

Once you understand what the comparator actually sees, every "wrong sort" becomes predictable. And most of them become fixable with the same three moves: normalize first, key on a transformation, or switch to natural sort.

What lexicographic sort really compares

A basic sort in any language compares strings character by character, and the comparison uses the characters' code points. In ASCII, and in the first 127 positions of Unicode, the order is:

  • Digits 0–9 come first (0x30–0x39)
  • Then uppercase A–Z (0x41–0x5A)
  • Then a few punctuation characters
  • Then lowercase a–z (0x61–0x7A)

So "Zebra" (Z = 0x5A) sorts before "apple" (a = 0x61). Case is not ignored — it is a primary sort key. The digits inside file2 and file10 get compared letter by letter: '1' < '2', so file10 wins before the digit 2 is ever considered. Neither result is illogical; both are just code-point order.

This is why the same data sorts "correctly" in a language-aware tool and "randomly" in a script. Tools that claim to sort alphabetically often mean "collated per locale," which is a completely different algorithm.

Trap 1: mixed case breaks grouping

A common real-world failure: you paste a list of company names from a CRM export, sort it, and all UPPERCASE names jump to the top, away from their lowercase variants. Nobody reading the output asked for that.

Fixes, in order of preference:

  • Casefold the key, not the value. Use sorted(items, key=str.lower) in Python, -f in GNU sort, or toLowerCase() in JavaScript. The output keeps original casing but the comparison is case-insensitive.
  • Normalize before sorting. If the list has stray whitespace, inconsistent punctuation, or duplicate lines, sort after cleaning. A trailing space at 0x20 sorts before everything, and it is invisible: a value with a trailing space will silently land before the same value without one. That is exactly the class of bug a quick pass through the whitespace remover and a dedupe step will catch before you sort.
  • Watch for Unicode look-alikes and composed forms. é written as a single code point (U+00E9) and as e + combining accent (U+0065 U+0301) are the same human letter but different byte sequences. A code-point sort scatters them apart. Normalize to NFC first — the unicode normalizer tool does this in one pass.

Trap 2: numbers inside strings

The classic: log files, invoice IDs, or asset names like img2.png, img10.png. Code-point order gives img1, img10, img100, img2. Every developer has shipped something with this ordering visible to a user.

The pattern that fixes it is called natural sort: instead of comparing raw characters, split each string into digit runs and non-digit runs, compare digit runs as integers. img2 then sorts before img10.

Rolling your own seems trivial until versions come in: v1.0.10 vs v1.0.9, zero-padded IDs, negative numbers, or values like 2abc vs 2. Libraries exist in every ecosystem precisely because these edge cases keep appearing. But for a one-off cleanup of a list — sorting filenames for a manifest, ordering test cases for a report — you do not need a library. The text sort and dedupe tool has a natural sort mode that handles digit runs correctly, and dedupes in the same pass.

Trap 3: locales, accents, and collation

Swedish, German, and Estonian all sort differently from ASCII, and from each other:

  • Swedish: Ö comes after Z, as its own letter.
  • German: Ä sorts near A (telephone-book collation expands it to ae).
  • Lithuanian: Y sits between I and J.

If your product ever shows sorted lists to European users, code-point order is wrong for them. The fix is locale-aware collation — Intl.Collator in JavaScript or equivalent APIs elsewhere. A middle ground many tools use: decompose accented letters down to their base letter (NFD, strip combining marks), sort, then restore. That gives a stable, predictable order that is correct for most Western European text without pulling in full locale tables.

Test case worth remembering: sort Arkiv, Ärnö, Örebro, Zebra under two collations. If the list contains customer addresses, the difference is not academic.

Trap 4: invisible characters that reorder

Some characters in real text have code points that sort in surprising places:

  • Non-breaking space (U+00A0) looks identical to a space but sits in the Latin-1 range, far from ASCII space.
  • Zero-width characters are invisible but still participate in the comparison, shifting everything around them.
  • Directional override characters copied from certain websites silently corrupt the surrounding comparison.

Symptom to watch for: two values that look identical are not adjacent after sorting. When that happens, do not eyeball — diff the actual bytes. A character-level diff tool will show which positions differ even when the strings look the same in your terminal.

Practical sorting checklist

Before sorting anything user-facing, run through this list:

  1. Empty lines and whitespace: remove or handle explicitly. They sort to the top of any code-point sort.
  2. Duplicates: decide if you want them deduplicated before or after sorting; sorting with dupes makes near-duplicates harder to spot.
  3. Case: casefold the sort key or every mixed-case list will look shuffled.
  4. Numeric fields: natural sort, or pad to fixed width with zeros before comparing as text.
  5. Unicode: normalize to NFC first, ideally strip diacritics for a Western audience.
  6. Locale: if users outside English will see sorted output, use locale collation, not code-point order.

The one rule that prevents most of this

Sort with the same key convention your pipeline uses everywhere else. If filenames are zero-padded in your build, no natural sort is needed at all — img002 through img999 sorts correctly under plain lexicographic order. Pad first where you control the data; use natural sort where you don't. Normalizing the input once beats writing a clever comparator to compensate later.

Sorting is a contract with whoever reads the output next. When the contract is "human alphabetical order," raw code-point comparison almost never honors it. Know which mode your tool is in, clean the input first, and most wrong-sort bugs disappear before they exist.