Character encoding

Lore & Background

Early character encodings originated with optical or electrical telegraphy and early computers, representing only a subset of characters, often restricted to upper case letters, numerals, and limited punctuation. These systems used numeric values, known as code points, to represent each character of a writing script, including natural language symbols, control characters, and whitespace. The numerical values collectively form a code space or code page. Early examples include Morse code, which employed variable-length sequences of short and long signals, and the Baudot code, a five-bit encoding created by Émile Baudot in 1870, later modified and standardized as International Telegraph Alphabet No. 2. Herman Hollerith’s punch card data encoding, invented in the late 19th century, initially used hole positions to represent data elements, later encoding numeric information by row numbers and alphabetic data with multiple punches per column. IBM’s early electronic computers used six-bit binary-coded decimal (BCD) schemes tied to punch card codes, limited to digits, uppercase letters, and a few special characters. These evolved into the eight-bit Extended Binary-Coded Decimal Interchange Code (EBCDIC) in 1963 for the IBM System/360, which added lowercase letters. The U.S. military’s Fieldata code, a six- or seven-bit code introduced in 1959, addressed machine collation but was short-lived. ASCII, first released in 1963, used a simpler seven-bit code and was widely adopted, with a 1967 revision adding lowercase letters. Over time, more comprehensive encodings emerged, such as ASCII, ISO/IEC 8859, and Unicode encodings like UTF-8 and UTF-16. UTF-8 is the most popular on the World Wide Web, used in the vast majority of surveyed websites, while both UTF-8 and UTF-16 are common in applications and operating systems.

Reader's Guide

Character encoding governs how computers store, transmit, and transform textual data by assigning a unique numeric value, or code point, to each character in a writing system. These code points collectively form a code space or code page. Encodings can represent not only natural language symbols but also control characters and whitespace, and they have been defined for constructed languages as well. Early systems, such as Morse code from the 1840s, used variable-length sequences of four symbols (short and long signals, short and long spaces) and were often manually generated and deciphered by ear, persisting in amateur radio and aeronautics. Later, the Baudot code emerged as a five-bit encoding, patented in 1874, later modified and standardized as International Telegraph Alphabet No. 2, though it suffered from compatibility issues due to manufacturer-specific improvements. Herman Hollerith’s late-19th-century punch card encoding initially used hole positions to represent data elements, later encoding numbers by row position and alphabetic data through multiple punches per column. With electronic computing, IBM developed six-bit binary-coded decimal (BCD) schemes tied to punch card codes, limited to digits, uppercase letters, and a few special characters. These evolved into the eight-bit Extended Binary-Coded Decimal Interchange Code (EBCDIC) for the IBM System/360, which added lowercase letters. The U.S. military’s Fieldata code, a six- or seven-bit code introduced in 1959, aimed to improve machine collation but was short-lived. ASCII, released in 1963 and revised in 1967 to include lowercase letters, became widely adopted, though its American-centric nature was later addressed by Unicode, which broke the direct mapping of characters to bit sequences by using abstract code points and variable-length encodings like UTF-8 and UTF-16. Today, UTF-8 is the dominant encoding on the World Wide Web, used in the vast majority of surveyed websites.

Did You Know?

• The earliest well-known electrically transmitted character code was Morse code, introduced in the 1840s.
• Herman Hollerith invented punch card data encoding in the late 19th century to analyze census data.

The Architecture of Symbolic Representation

Character encoding is fundamentally a mapping convention: it assigns a specific numeric value to every symbol in a writing system so that a machine can recognize, store, transmit, and transform that symbol. The scope of what gets encoded extends well beyond the letters and digits of natural languages. A well-designed code space also reserves positions for control characters, whitespace markers, and other functional symbols that carry meaning outside of ordinary prose. Encodings have even been formalized for constructed languages, demonstrating that the principle is not limited to any single linguistic tradition. The individual numeric values that make up an encoding are called code points, and the full collection of those values is referred to as either a code space or a code page. This terminology matters because it frames encoding not as a single fixed table but as a structured, addressable region of numbers. Once a character has been reduced to its code point, it becomes a unit of data that any computer system can manipulate—copied across a network, written to disk, or converted into a different encoding altogether. In this sense, character encoding is the foundational layer that makes all higher-level text processing possible.

From Telegraph Keys to Punch Cards

The story of machine-readable character codes begins long before electronic computers. When electrical telegraphy arrived, those ideas were adapted to the new medium. Morse code, introduced in the 1840s, used just four building blocks (short signal, long signal, short space, long space) to build variable-length sequences. Though later transmitted by machinery, it was often keyed by hand and read by ear, a practice that endures in amateur radio and aviation. Donald Murray modified it in 1901, and the CCITT standardized it as International Telegraph Alphabet No. 2 in 1930. Meanwhile, Herman Hollerith's punch-card system, invented in the late nineteenth century to tabulate census data, encoded information through the presence and position of holes.

The ASCII Decade and the Fragmentation Problem

By the late 1950s, the need for a common seven-bit character standard was pressing. The U.S. Army Signal Corps had defined Fieldata in 1959, a six-or-seven-bit code that addressed modern concerns like machine collation of letters and digits, but it proved short-lived and fell short of its ambitions. Notably, at least one Fieldata committee member, W. F. Leubbert, also sat on the ASCII committee, ensuring continuity of thinking. The 1967 revision added lower-case letters and resolved lingering control-code ambiguities, and ASCII67 was adopted broadly across industry. Yet the landscape remained fragmented. Other vendors like UNIVAC I ran their own six-bit codes.

The Convergence on Unicode and the Web's Default

The 1980s presented researchers with a genuine dilemma. On one side, accommodating the full breadth of world scripts demanded more bits per character. On the other, the majority of computer users at the time worked with the relatively compact Latin alphabet, and those extra bits represented a significant waste of scarce and expensive computing resources. This tension drove the search for a universally interchangeable encoding that could scale without penalizing smaller character sets. Unicode emerged as the answer: a well-defined, extensible system that ultimately replaced most earlier character encodings. Its variable-length implementations, particularly UTF-8 and UTF-16, allowed the encoding to be efficient for Latin text while still covering every script in the code space. In application programs and operating-system tasks, both UTF-8 and UTF-16 remain popular choices, reflecting the fact that different contexts favor different trade-offs between byte efficiency and internal processing convenience.