Unicode and UTF-8 Explained: Why String Length Lies

Ask JavaScript how long the string "🚀" is and it says 2. Ask Python and it says 1. Ask the database how many bytes it takes and the answer is 4. Ask a user and they say "one rocket". All four answers are correct, because "length" means different things at different layers of text.

Most string bugs in internationalised software come from mixing those layers: truncating in the middle of an emoji, comparing two strings that look identical but aren't, or reading UTF-8 bytes as the wrong encoding. This guide explains the layers with real examples, so you can choose the right one each time.

Layer 1: code points

Unicode assigns a number, called a code point, to every character in every writing system, plus symbols and emoji. Code points are written U+ followed by hex:

A   U+0041
é   U+00E9
€   U+20AC
🚀  U+1F680

There are over a million possible code points, and about 160,000 are assigned to characters. Unicode itself doesn't say how to store them as bytes. That's the job of an encoding.

Layer 2: UTF-8 bytes

UTF-8 is the encoding of the web, used by over 98% of websites. It stores each code point in one to four bytes:

Its clever property is that ASCII text is unchanged: an English README is byte-for-byte the same in ASCII and UTF-8. The bytes in the table are also exactly what you see in percent-encoded URLs (%C3%A9), as explained in our guide to URL encoding.

Byte length matters whenever something has a byte limit: database columns, HTTP headers, SMS messages, file names, or a 255-byte field in an API.

Layer 3: UTF-16 code units (JavaScript, Java, C#)

JavaScript strings, like Java and C# strings, are sequences of 16-bit code units in UTF-16. Code points up to U+FFFF fit in one unit. Anything above, including most emoji, needs two units called a surrogate pair:

'🚀'.length;               // 2
'🚀'.charCodeAt(0).toString(16);  // "d83d" — half of a surrogate pair
'🚀'.codePointAt(0).toString(16); // "1f680"
'🚀' === '🚀';  // true

.length, .slice(), .substring() and indexing all count code units. That's how you get a broken character:

'🚀'.slice(0, 1);                       // "\ud83d" — a lone surrogate, shown as �
'hello🚀'.split('').reverse().join(''); // "��olleh"

Iterating a string with for…of or the spread operator works on code points instead: [...'🚀'].length is 1. Python 3 strings are sequences of code points, so len('🚀') is 1 there.

Layer 4: graphemes — what users call characters

Even code points aren't what people see. A grapheme cluster is one user-perceived character, and it can contain several code points:

The flag is two "regional indicator" letters (I and N). The family emoji is three person emoji joined by two invisible zero-width joiners. To count graphemes in JavaScript, use Intl.Segmenter:

const graphemes = (text) => [...new Intl.Segmenter('en', { granularity: 'grapheme' }).segment(text)].length;

graphemes('👨‍👩‍👧'); // 1
'👨‍👩‍👧'.length;     // 8

Use graphemes for anything shown to users: character counters, "max 20 characters" limits on names, truncation with an ellipsis, cursor movement. The Word & Character Counter is a quick way to see how different counts compare for a piece of text.

Same text, different code points: normalisation

The two café rows above look identical, but they are different strings:

const precomposed = 'café';   // é as one code point
const decomposed = 'café';   // e + COMBINING ACUTE ACCENT

precomposed === decomposed;                    // false
precomposed === decomposed.normalize('NFC');   // true

Text typed on one system, pasted from a PDF, or coming from macOS file names (which historically used the decomposed form) may use either. Without normalisation, a search for "café" misses half the matches, a uniqueness check lets "duplicate" usernames through, and the same file name appears twice.

Normalise to NFC when you store or compare text. For usernames and identifiers, many systems also apply case folding and the stricter NFKC form, which maps look-alike compatibility characters (like the fi ligature) to their plain equivalents.

Mojibake: decoding with the wrong encoding

When UTF-8 bytes are read as a single-byte encoding such as Windows-1252 or Latin-1, every multi-byte character turns into two or three wrong ones. This garbage has a name: mojibake.

café   saved as UTF-8 bytes   63 61 66 C3 A9
read as Windows-1252          café

C3 is Ã and A9 is © in Windows-1252. If you see é, ’ (a curly apostrophe) or € (a euro sign), you're looking at UTF-8 decoded as Windows-1252. Common causes:

• A database connection that doesn't specify UTF-8 (in MySQL, set the connection charset to utf8mb4).
• An HTTP response without charset=utf-8 in its Content-Type.
• A CSV opened in Excel, which may guess the wrong encoding. Our JSON to CSV guide covers that case.

The fix is to declare UTF-8 everywhere: files, HTML (<meta charset="utf-8">), HTTP headers, database columns and connections. Once text has been decoded wrongly and saved, recovering it means re-encoding the garbage with the wrong encoding and decoding it correctly, which only works if no information was lost.

MySQL warning: MySQL's old utf8 charset stores at most 3 bytes per character, so it can't store emoji at all. Always use utf8mb4.

Escape sequences: writing any character in ASCII

When a file or protocol must stay ASCII, characters are written as escapes. JSON and JavaScript use \u followed by four hex digits of a UTF-16 unit, so emoji need two escapes or the newer brace syntax:

'é'        // "é"
'🚀'  // "🚀" (surrogate pair, valid in JSON too)
'\u{1F680}'     // "🚀" (JavaScript only, not JSON)

The Unicode Escape tool converts text to escapes and back, including emoji. It's also useful for seeing invisible characters: zero-width spaces, non-breaking spaces and byte order marks become visible ​,   and  escapes. Those invisible characters are a classic cause of baffling errors, as our list of JSON syntax errors shows.

Practical rules

1. Use UTF-8 everywhere, and say so explicitly in headers, files and database connections.
2. Decide which length you need: bytes for storage limits, graphemes for anything a user sees, code units only when an API demands it.
3. Never cut strings with .slice() for display. Segment into graphemes first, or you'll split emoji and accented letters.
4. Normalise to NFC before storing, comparing or hashing text.
5. Recognise mojibake (é, ’) as an encoding mismatch, not corrupted data.

Summary

Text has four layers: graphemes (what users see), code points (Unicode's numbers), code units (how JavaScript, Java and C# store strings) and bytes (how UTF-8 stores text). "🚀".length is 2 because JavaScript counts UTF-16 units. Pick the layer that matches the job, normalise before comparing, and declare UTF-8 at every boundary to avoid mojibake.