Skip to main content
Dev Tools

CRLF vs LF: The Invisible Bug That Breaks CI

A commit passes tests on your machine. The same commit fails in CI ten seconds later. Nobody changed the code between the two runs — but the diff is full of lines that "look identical" and don't match. In a large fraction of those cases, the culprit is invisible: carriage return characters, `\r`, that your editor added or removed without telling you.

Why two line ending conventions exist

Line endings are one byte in ASCII for the newline character (LF, `0x0A`, `\n`). Windows tools historically end lines with two bytes: CR followed by LF (`0x0D 0x0A`, `\r\n`). macOS before OS X used a bare CR. Unix, Linux, and modern macOS use LF only.

Neither convention is wrong on its own. The problem is mixing them in one repository. Git, editors, shells, and CI runners all have opinions about which one is correct, and those opinions silently disagree.

Where the mismatch sneaks in

  • A Windows contributor has git's `core.autocrlf=true`, which converts LF to CRLF on checkout and back to LF on commit. Teammates on Linux checkout CRLF files raw if the repo already contains them.
  • A config file generated on Windows — a `.env`, a CSV fixture, a YAML template — is committed with CRLF. A Linux CI runner then parses it. Shell scripts are the worst case: a `#!/bin/bash` script with CRLF endings fails with "/usr/bin/env^M: bad interpreter".
  • Copy-paste from a tutorial or a PDF pastes CRLF into your source. Many editors hide this completely.
  • Tools that "fix" formatting re-save files with the opposite convention. The diff shows every line changed, the PR looks like a rewrite, and review time triples.

The classic CI failures

These show up repeatedly in real build logs:

  1. Shell script won't run: `/bin/bash^M: bad interpreter` or `$'\r': command not found`. The YAML runner executes a script with CRLF endings; bash reads `\r` as part of every token.
  2. Diff tests fail: a golden-file test compares output against a fixture committed by a Windows user. Byte-for-byte comparison fails on 100% of lines.
  3. YAML parse errors: some parsers tolerate `\r`, others don't. The same file parses locally and explodes in CI, depending on library versions.
  4. Huge diffs: a formatting PR that changed nothing logically shows thousands of modified lines because the endings flipped.
  5. Hashes don't match: checksums of "the same" config file differ between environments because the bytes differ.

Fix it once at the repo level

The durable fix is a single `.gitattributes` file at the repo root:

* text=auto
*.sh text eol=lf
*.yml text eol=lf
*.bat text eol=crlf
*.png -text

This tells git to normalize text files to LF in the repository, force LF for shell scripts and YAML regardless of platform, and keep CRLF for Windows batch files where it's genuinely required. Binary files stay untouched.

After adding it, renormalize once: `git add --renormalize .`, commit, and have every developer re-clone or rebase. Any CRLF already lurking in blobs disappears from history going forward.

Fix it when a file is already broken

When a single file or a whole directory arrives with wrong endings, don't hand-edit or retype. Convert them mechanically and verify the result. A line ending normalizer converts CRLF/CR/LF in one pass and reports the counts, so a 40,000-line CSV can be cleaned in seconds without touching its content.

After converting, always verify nothing else changed. A text diff between the before and after versions should show zero content differences — if any line shows a "change" beyond line endings, stop and investigate before committing.

A related trap: pastes from chat apps and email often carry trailing whitespace and stray blank lines along with CRLF. Cleaning the paste through remove line breaks before saving the file prevents the formatting PR problem entirely.

Prevent it in CI itself

Trust but verify. Add a one-line check to your pipeline so endings can never regress:

test -z "$(git grep -Il $'\r' -- ':!*.bat')"

This greps tracked files for carriage returns and fails the build if any appear (excluding Windows batch files). It runs in milliseconds. Teams that add this check never debug the "bad interpreter" error on a Friday afternoon again.

Also configure your linters with explicit expectations: EditorConfig (`end_of_line = lf`), Prettier (`endOfLine: "lf"`), and ESLint's `linebreak-style` rule all cover this if you pin the values instead of leaving defaults. EditorConfig alone prevents most editor-side reintroductions.

Practical rules worth remembering

  • Shipped shell scripts, YAML, and Dockerfiles must be LF. No exceptions.
  • `.bat` and `.cmd` files keep CRLF — cmd.exe misparses LF-only labels.
  • Normalize in git (`.gitattributes`), not in people's editors. Convention in the repo beats convention in a handbook nobody reads.
  • Never commit a formatting PR that flips every line ending mixed with real changes — review it separately or reviewers will miss the actual bug.
  • If a CI log shows `^M` anywhere, you already know the diagnosis. Convert the culprit file, diff it, push.

Line endings are a solved problem — the tools to detect, convert, and prevent them are trivial. What makes them expensive is that the failure is invisible in every editor and only appears in a context you don't control. One `.gitattributes` file plus one grep in CI closes it permanently.