You open a spreadsheet a colleague sent you and, instead of “café”, you see “café”. Or a name with an accent turns into a row of question marks. These glitches are not random — they are the visible result of a mismatch in text encoding, one of the most fundamental and least understood concepts in computing. This guide explains how computers actually store letters, why so many encodings exist, and how to avoid the problem in your own files.
Computers only store numbers
A computer’s memory holds nothing but numbers, ultimately expressed as sequences of bits — zeros and ones. There is no physical “letter A” anywhere in a machine. What exists is an agreement: a table that says the number 65 shall be displayed as the character A.
That agreement is called a character encoding. Writing a text file means converting characters into numbers; reading it means converting numbers back into characters. If the program that writes and the program that reads use different tables, the text comes out wrong. That’s the whole story behind almost every garbled-text bug you have ever seen.
ASCII: the original agreement
ASCII, standardized in the 1960s, assigned numbers 0 to 127 to a small set of characters: uppercase and lowercase English letters, digits, punctuation, the space, and a handful of control codes such as tab and newline. Seven bits were enough to cover all of them.
ASCII worked beautifully — as long as you wrote in English. It has no é, no ñ, no ç, no Greek letters, no Cyrillic, no Chinese characters. For most of the world’s languages, it simply had nothing to offer.
The messy middle years: code pages
Since computers of the era used 8-bit bytes, there were 128 unused numbers left over (128 to 255). Different regions filled that space with the characters they needed, producing dozens of incompatible “code pages”: one for Western European languages, another for Cyrillic, another for Greek, and so on.
The consequence was predictable. The same byte meant different things depending on which table you assumed. A document written on a Western European system and opened on a Cyrillic one would display perfectly readable — but completely wrong — letters. And no single code page could hold a document mixing Russian, Greek and Japanese.
Unicode: one number for every character
Unicode was created to end that fragmentation. Its goal is simple to state: give every character in every writing system its own unique number, called a code point. Code points are written in a conventional format, like U+0041 for A or U+00E9 for é.
Unicode covers modern alphabets, historical scripts, mathematical symbols, punctuation and emoji. Crucially, it kept the first 128 code points identical to ASCII, so existing English text remained valid.
Here is the point that trips up most beginners: Unicode is not an encoding. It is a catalogue of characters and their numbers. It does not say how those numbers should be stored as bytes on disk. That job belongs to encodings such as UTF-8, UTF-16 and UTF-32.
UTF-8 and its relatives
UTF-8 is a variable-length encoding: it uses one byte for the original ASCII characters, and two, three or four bytes for everything else. This design gives it two big advantages. First, any plain English text encoded in UTF-8 is byte-for-byte identical to the same text in ASCII, so decades of old files and systems keep working. Second, common Latin text stays compact.
| Encoding | Bytes per character | ASCII compatible? | Typical use |
|---|---|---|---|
| ASCII | 1 | Yes (it is the origin) | Legacy systems, protocol keywords |
| UTF-8 | 1 to 4 (variable) | Yes | The web, most modern files and APIs |
| UTF-16 | 2 or 4 | No | Internal strings in Java, .NET, Windows |
| UTF-32 | 4 (fixed) | No | Rare; simple indexing, high memory cost |
UTF-8 has become the dominant encoding on the internet by a wide margin, which is why “always use UTF-8” is standard advice for new projects.
Why “café” becomes “café”
Now the classic bug makes sense. In UTF-8, the character é is stored as two bytes. If a program reads that file assuming a single-byte Western European code page, it interprets each of those two bytes as a separate character — and prints two odd symbols instead of one accented letter.
The opposite mistake produces different symptoms. If a file was saved in a single-byte code page and is read as UTF-8, the decoder finds byte sequences that are not valid UTF-8 and replaces them with the replacement character � or with question marks. Once that substitution is written back to disk, the original information is gone — which is why the damage is often permanent.
How to avoid encoding problems
- Default to UTF-8 everywhere. Editors, databases, source files, exports. Consistency matters more than the specific choice.
- Declare the encoding explicitly. In HTML, include the meta charset tag in the head. In HTTP responses, set the charset in the Content-Type header. In databases, define the collation and charset when creating tables.
- Be specific when exporting CSVs. Spreadsheet software often offers “CSV” and “CSV UTF-8” as separate options. Choose the UTF-8 variant when data contains accents or non-Latin scripts.
- Don’t guess in code. When opening files programmatically, state the encoding rather than relying on the operating system default, which varies between machines.
- Keep an eye on the BOM. The byte order mark is an optional marker at the start of a file. It helps some Windows tools detect encoding, but can break scripts and configuration files that expect the first character to be meaningful.
Two extra concepts worth knowing
A character is not always one code point. Some accented letters can be represented either as a single code point or as a base letter followed by a combining accent mark. Both look identical on screen but compare as different strings. Unicode normalization converts text to a canonical form so that comparisons behave as users expect.
Length can be ambiguous. Depending on the programming language, asking for the “length” of a string may return the number of bytes, the number of code points, or the number of visible characters. Emoji make this obvious: a single emoji with a skin-tone modifier may count as several code points. When validating input length, decide which definition you actually mean.
Conclusion
Text encoding is invisible until it breaks, and then it is remarkably confusing. Remembering three ideas keeps you out of trouble: computers store numbers, an encoding is the agreed table linking numbers to characters, and UTF-8 is the modern default that keeps almost everything compatible. Once you internalize that, garbled characters stop being a mystery and become a diagnosis.
If you want to build a stronger foundation in how computers really work, Cursa offers free online courses in computer basics, programming fundamentals and IT support, with video lessons and a certificate of completion — a practical way to turn concepts like this into everyday skills.
