10.4 — Character Encodings & The 1024‑Byte Buffer Rule

The entire parsing pipeline depends on one critical assumption: the browser knows the character encoding of the incoming bytes. A wrong guess can corrupt tokenization, destroy layouts, and produce the garbled text known as Mojibake. The browser uses a strict multi‑layer evaluation path to determine the encoding, governed by the hard 1024‑byte rule.

Diagram showing the three-layer encoding decision: Byte Order Mark overrides HTTP headers, HTTP headers override meta charset — **Figure 10.15:** The encoding determination hierarchy – BOM beats HTTP, HTTP beats meta.

The Three‑Layer Codec Evaluation Path

When the first bytes arrive, the browser determines the encoding by checking three sources in strict order of authority:

Byte Order Mark (BOM) – If the stream starts with the bytes EF BB BF (UTF‑8 BOM), the encoding is locked instantly. The BOM overrides everything else.
HTTP Content‑Type Header – If no BOM, the engine looks at the charset parameter from the server’s response header (e.g., Content-Type: text/html; charset=utf-8).
Inline <meta charset> – If both BOM and HTTP header are absent, the parser must read the HTML itself to find <meta charset="UTF-8">.

The absence of the first two layers forces the browser into speculative parsing, which is where the 1024‑byte rule becomes critical.

Diagram showing the first 1024 bytes of a document with a meta charset tag inside; beyond 1024, the encoding is locked — **Figure 10.16:** The browser only looks for charset declarations within the first 1024 bytes.

The 1024‑Byte Rule & Speculative Restart

Without a definite encoding from the BOM or HTTP header, the browser must guess. It chooses a default (often Windows‑1252 or ISO‑8859‑1) and begins tokenizing and building a speculative DOM tree.

The engine continuously watches for an inline <meta charset>. The specification enforces a strict spatial boundary: this search only happens within the first 1024 bytes of the HTML stream.

Found within 1024 bytes and matches guess: parsing continues seamlessly.
Found within 1024 bytes and differs from guess: the browser triggers a speculative restart. It destroys the provisional DOM, resets the stream, switches to the declared encoding, and starts over.
Not found within 1024 bytes: the guess is locked permanently. If the guess was wrong, all subsequent multi‑byte characters become garbled – Mojibake.

Side-by-side: correct UTF-8 text vs. garbled Mojibake output when interpreted as Windows-1252 — **Figure 10.17:** Mojibake – the same bytes decoded as UTF‑8 (left) and as Windows‑1252 (right).

Preventing the Catastrophe

To eliminate speculative restarts and avoid Mojibake, the <meta charset> tag (or better, the HTTP header) must appear as early as possible – ideally at byte zero or within the first few characters. Place it immediately after the opening <head> tag, before any other elements, scripts, or comment blocks.

<head>
  <meta charset="UTF-8">
  <title>…</title>
  …
</head>

This simple discipline guarantees that the browser never guesses, never restarts, and always decodes your content perfectly.

TAKEAWAY

Character encoding is the foundation of parsing. The browser follows a strict hierarchy: BOM, then HTTP header, then inline <meta charset>. The 1024‑byte rule dictates that the charset must be declared early, or the browser will lock in a potentially wrong guess, leading to catastrophic layout drops and garbled text. Always declare your encoding as close to byte zero as possible – ideally in the HTTP header or as the first child of <head>.