Understanding Character Encoding: ASCII, Unicode & UTF-8 Explained

Every character you see on a screen — every letter, digit, emoji, and symbol — is stored in memory as a sequence of numbers. The rules that map those numbers to human-readable characters are called character encodings. Misunderstanding or misconfiguring character encoding is one of the most common sources of data corruption, garbled text (mojibake), and subtle security vulnerabilities in web applications. This guide walks you through the history, mechanics, and practical implications of the encodings that power the modern web.

ASCII: Where It All Started

The American Standard Code for Information Interchange (ASCII) was published in 1963. It assigns numeric values to 128 characters: the English alphabet (upper and lower case), digits 0–9, punctuation, and a handful of control characters like newline (0x0A) and tab (0x09).

Character   Decimal   Hex    Binary
A           65        0x41   01000001
Z           90        0x5A   01011010
a           97        0x61   01100001
0           48        0x30   00110000
Space       32        0x20   00100000

Because ASCII uses only 7 bits, it can represent at most 128 characters. That was plenty for mid-20th-century American computing, but it left no room for accented letters (é, ñ, ü), non-Latin scripts (Chinese, Arabic, Cyrillic), or the thousands of symbols used around the world.

The Code Page Era & Its Problems

To work around ASCII's limitations, manufacturers created code pages — extended 8-bit encodings that used the upper 128 values (128–255) for region-specific characters. For example:

• ISO 8859-1 (Latin-1) — Western European languages
• ISO 8859-5 — Cyrillic alphabets
• Windows-1252 — Microsoft's superset of Latin-1
• Shift_JIS — Japanese text

The fundamental problem was that byte value 0xE9 meant é in Latin-1 but щ in ISO 8859-5. If a document encoded in one code page was opened with another, the result was mojibake — garbled, unreadable text. There was no universal scheme that could handle every script simultaneously.

Unicode: One Character Set to Rule Them All

The Unicode Consortium set out to solve this by assigning a unique code point to every character in every writing system. A code point is written in the form U+XXXX, where XXXX is a hexadecimal number. As of Unicode 16.0, there are over 154,000 characters spanning 168 scripts.

Character   Code Point   Description
A           U+0041       Latin Capital Letter A
é           U+00E9       Latin Small Letter E with Acute
中          U+4E2D       CJK Unified Ideograph (Middle)
😀          U+1F600      Grinning Face Emoji
∞           U+221E       Infinity Symbol

It is critical to understand that Unicode is not an encoding — it is a character set. It defines which number corresponds to which character, but it says nothing about how those numbers are stored as bytes. That job falls to encoding forms like UTF-8, UTF-16, and UTF-32.

UTF-8: The Web's Dominant Encoding

UTF-8 (Unicode Transformation Format — 8-bit) is a variable-width encoding that uses one to four bytes per character. It was designed by Ken Thompson and Rob Pike in 1992 and has become the de facto encoding of the internet — over 98% of web pages use UTF-8.

How UTF-8 Byte Sequences Work

UTF-8 encodes code points into byte sequences according to these rules:

Code Point Range        Bytes   Byte Pattern
U+0000  – U+007F       1       0xxxxxxx
U+0080  – U+07FF       2       110xxxxx 10xxxxxx
U+0800  – U+FFFF       3       1110xxxx 10xxxxxx 10xxxxxx
U+10000 – U+10FFFF     4       11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

The leading bits of the first byte indicate how many bytes are in the sequence. Continuation bytes always start with 10, making UTF-8 self-synchronizing — you can jump into the middle of a stream and find the next valid character boundary.

Key Advantages of UTF-8

• Backward compatible with ASCII — any valid ASCII document is also valid UTF-8
• No byte-order issues — unlike UTF-16/32, byte order is unambiguous
• Space efficient for Latin-heavy text (1 byte per ASCII character)
• Self-synchronizing — easy to find character boundaries in a byte stream

UTF-16 & the BOM

UTF-16 uses two or four bytes per character. Characters in the Basic Multilingual Plane (BMP, U+0000 to U+FFFF) use two bytes, while supplementary characters use a surrogate pair — two 16-bit code units.

Because UTF-16 stores data in 16-bit units, byte order matters. A system using big-endian byte order stores the most significant byte first; little-endian stores it last. To signal which order is used, files may begin with a Byte Order Mark (BOM):

Encoding         BOM Bytes
UTF-8            EF BB BF  (optional, often discouraged)
UTF-16 BE        FE FF
UTF-16 LE        FF FE
UTF-32 BE        00 00 FE FF
UTF-32 LE        FF FE 00 00

In UTF-8, the BOM is technically valid but discouraged. It can cause invisible bugs — for instance, a PHP file beginning with a UTF-8 BOM will output three garbage bytes before any intended content, potentially breaking HTTP headers or JSON responses.

Mojibake: When Encodings Collide

Mojibake (文字化け, literally "character transformation") is the garbled output that appears when text encoded in one scheme is decoded with another. Common examples include:

• é (U+00E9) encoded in UTF-8 becomes bytes C3 A9. If decoded as Latin-1, you see é instead.
• Japanese text encoded in Shift_JIS but decoded as UTF-8 produces sequences of replacement characters (�).
• A database storing UTF-8 data in a Latin-1 column silently corrupts multi-byte characters.

How to Prevent Mojibake

1. Declare encoding explicitly — always include <meta charset="UTF-8"> in your HTML <head>.
2. Set HTTP headers — send Content-Type: text/html; charset=utf-8.
3. Configure your database — use utf8mb4 in MySQL (not the legacy utf8, which only supports 3-byte sequences and breaks emoji).
4. Save source files as UTF-8 without BOM — configure your editor accordingly.
5. URL-encode non-ASCII characters in URLs — the percent-encoding scheme converts bytes like C3 A9 into %C3%A9.

Character Encoding in Web Development

Encoding issues can surface at every layer of a web application. Here is where to pay attention:

HTML & the Viewport

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>My Page</title>
</head>

The <meta charset> tag must appear within the first 1024 bytes of the document. Browsers use it to determine how to decode the rest of the page. If it is missing or incorrect, the browser falls back to heuristics that frequently guess wrong.

JavaScript String Handling

JavaScript strings are internally encoded as UTF-16. This means that characters outside the BMP (like emoji) are represented as surrogate pairs and have a .length of 2, not 1:

const emoji = '😀';
console.log(emoji.length);       // 2 (surrogate pair)
console.log([...emoji].length);  // 1 (iterator is code-point aware)

// Safe way to count characters:
const charCount = [...str].length;

URLs & Percent Encoding

URLs can only contain a limited set of ASCII characters. Any character outside that set — spaces, accented letters, CJK characters — must be percent-encoded. The process converts each byte of the UTF-8 representation into %HH format:

Original:  café
UTF-8 bytes of 'é': C3 A9
Encoded:   caf%C3%A9

Original:  hello world
Encoded:   hello%20world

Encoding URLs correctly is essential for avoiding broken links, injection attacks, and interoperability issues. Use built-in functions like JavaScript's encodeURIComponent() or Python's urllib.parse.quote() rather than hand-rolling encoding logic.

Databases

Always ensure your database connection, table, and column character sets match. In MySQL, the recommended configuration for full Unicode support is:

CREATE DATABASE myapp
  CHARACTER SET utf8mb4
  COLLATE utf8mb4_unicode_ci;

-- Connection string should also specify charset:
-- mysql://user:pass@host/myapp?charset=utf8mb4

Encoding & Security

Character encoding is also a security concern. Attackers have exploited encoding ambiguities to bypass input validation:

• Double encoding — encoding an already-encoded string (%253C decodes to %3C, which decodes to <) can bypass WAF rules.
• Overlong UTF-8 sequences — representing a character with more bytes than necessary (e.g., encoding / as C0 AF) was used to bypass path traversal filters in older systems.
• Homoglyph attacks — visually identical characters from different scripts (e.g., Latin "a" vs. Cyrillic "а") can spoof domain names or usernames.

Modern UTF-8 decoders reject overlong sequences, but you should still validate and normalize input using functions like Normalizer::normalize() in PHP or str.normalize() in Python.

Quick Reference Cheat Sheet

Feature          ASCII    UTF-8         UTF-16        UTF-32
Max Characters   128      1,112,064     1,112,064     1,112,064
Bytes/Char       1        1–4           2 or 4        4
ASCII Compatible Yes      Yes           No            No
Byte Order Issue No       No            Yes (BOM)     Yes (BOM)
Web Usage        Legacy   ~98%          Rare          Very Rare

Conclusion

Character encoding is one of those foundational topics that every developer encounters but few take the time to fully understand. The key takeaways are simple: use UTF-8 everywhere, declare your encoding explicitly at every layer (HTML, HTTP, database), and never assume that one byte equals one character. When you work with URLs, always percent-encode non-ASCII characters properly to avoid broken links and security vulnerabilities.