Text Diffing Algorithms: How Your Diff Tool Actually Works
Every merge conflict, code review comment, and deployment rollback trace ends with a diff. Most developers read diffs daily and trust them blindly. The trust is mostly earned, but the algorithm underneath has quirks that explain every "why is half the file marked as changed?" moment. Knowing how diffing actually works takes fifteen minutes and saves real debugging time.
The core idea: longest common subsequence
Nearly every diff tool, including git diff, builds on the Myers algorithm (1986) or a variant of it. The problem it solves: find the longest common subsequence (LCS) of two texts — the largest set of lines that appear in both, in the same order. Everything before or after those shared lines becomes an insertion or a deletion.
Example. Old file lists alpha, beta, gamma, delta. New file lists alpha, echo, beta, delta, gamma. The LCS is alpha, beta, delta. So the tool reports: delete gamma, insert echo after alpha, insert gamma after delta. A human might just say "gamma moved down" — but a pure LCS diff does not do move detection. It has no concept of "moved", only delete plus insert. Git's --color-moved is a post-processing layer on top, not part of the core algorithm.
Myers is O(ND) where N is the sum of both file lengths and D is the number of edits. For similar files D is small, so it is fast. For completely different files it degrades, which is why every diff implementation has a fallback: if files share so little that LCS is expensive, mark the whole file as rewritten instead of computing an exact answer.
Line-based, word-based, character-based
Diff granularity decides whether output is readable. Line-based diff is the default everywhere because it keeps output small and stable. But it has a weakness: change one word inside a 40-line function and a naive diff shows one huge hunk instead of the actual edit.
That is why production tools use the Histogram or Patience algorithms: they first find lines that are unique and common in both files — anchors — and only diff between anchors. If a changed function is surrounded by unique lines, the diff shrinks to the real edit. git diff --patience and --histogram are opt-in on the command line; GitHub's diffing has used Histogram-style anchoring for years.
Word-level diff exists separately: editors show "inline" views where a changed line is displayed with the exact word highlighted. That is a character-level LCS run on top of the line diff that matched the pair. Useful when reviewing copy, JSON config, or single-line template changes.
The three false positives that eat your time
Most "noisy diff" complaints come from exactly three causes, and all three are data problems, not algorithm problems.
- Line endings. A Windows checkout diffed against a Unix-original file shows every line changed. CRLF vs LF is invisible in the content but changes every byte pattern the algorithm compares. Fix the data, not the review: normalize before comparing — use
.gitattributeswith* text=auto, or run a dedicated normalizer like Line Ending Normalizer Pro before pasting text into any comparison. - Trailing whitespace. Editors that strip trailing spaces on save silently rewrite hundreds of lines. Same symptom as line endings: entire file "changed", zero semantic edits. Configure one whitespace policy across the team, or the diff becomes the only thing enforcing it.
- Reordered blocks. Sort a JSON list of 500 keys and the diff shows 500 deletions plus 500 insertions, even though content is identical. LCS cannot see order changes as moves. Diff sorted copies of the data to confirm "same content, different order" — a quick pass through Text Sort and Dedupe Pro settles the question before you read 1,000 changed lines.
Where diffing bites people in practice
Environment files. Comparing an example env template against a production env file by diff works until someone reorders entries, and then you are staring at a wall of changed lines containing identical values. Sort both sides first — the semantic question is "which keys exist in one and not the other", not line order.
Generated code. Diffing generated migrations or lock files line-by-line is meaningless when regenerating reorders output. The review question is not "what changed character by character" but "did the semantics move" — which a sorted or canonicalized comparison answers in seconds.
Large refactor scripts. When you run a bulk replacement across 80 files, you do not want to open each one. A scripted pass with something like Bulk Text Replacer Pro plus a per-file diff summary gives you an auditable list: file, before, after. Reviewing that list is faster and safer than reviewing raw hunks.
Copy review. Comparing two versions of marketing text line-by-line is useless because everything is "one line". Character-level diff solves this — that is the mode to switch to when the text has fewer than a few dozen lines total.
Practical workflow rules
- Normalize first, diff second. Line endings and trailing whitespace stripped, then compare. One normalization step kills the majority of noise diffs.
- For structured data, diff a canonical form (sorted keys, consistent indentation), not the raw file.
- Switch to word-level diff when the line diff is longer than the change could possibly be.
- Treat "1,000 lines changed" not as a review problem but as a data-canonicalization check first: run both files through a normalizer before reading a single hunk.
When you need a quick, no-setup comparison — a client sends an updated copy of a file, a teammate pastes a config from a server — a browser tool like Text Diff Checker Pro covers the exact loop: paste both sides, see line and character differences, spot the CRLF or whitespace culprit immediately. For the 800-line merge, keep using git's own algorithm — but now you know why it drew that odd hunk around line 214.