Devanagari Text: Fonts, Search and Sorting

Latin text lets you be sloppy. One character is one code point, sorting is alphabetical, and a string looks the way it is stored. Devanagari holds none of those assumptions, and code written on the first set breaks quietly on the second.

Normalisation, or why search misses

Unicode allows the same visible text to be encoded in more than one sequence of code points. A character with a diacritic can be a single composed code point or a base plus a combining mark. Both render identically. Neither equals the other under a byte comparison.

Nepali content typed on different keyboards, or pasted from different sources, ends up in a database in mixed forms. Search then fails for a word the user can see on the screen, which is a bug report that sounds impossible until you look at the bytes.

The fix is to normalise on the way in and on the way to a query, both to the same form. NFC is the usual choice. Do it in one place, at the boundary, rather than scattering normalisation calls through the code.

Length is not what you think

A string's length in code points is not the number of characters a reader sees. Devanagari combines consonants and vowel signs into clusters that display as a unit, so a five-cluster word can be eight or nine code points.

This matters wherever you count or cut. A field validated at twenty characters rejects a Nepali name that looks shorter than an English one that passes. An excerpt truncated at a fixed length can slice a cluster in half and render a broken glyph.

Where a limit is about what fits on screen, measure rendered width rather than length. Where it is about storage, be generous, because the cost of a larger column is nothing next to a form users cannot complete.

Sorting

Devanagari has its own order, and a byte sort does not produce it. A list sorted with a binary collation looks arbitrary to a Nepali reader.

Use a Unicode collation in the database rather than sorting in application code. In MySQL that means utf8mb4 with a Unicode collation, set at the database, table, and connection. A correct table with a mismatched connection charset still produces wrong results, and it is a maddening bug to chase.

Check the connection specifically. It is the level most often left at a default that nobody set deliberately.

Fonts and the blank screen

A Devanagari webfont is large, because it carries far more glyphs and the conjunct forms the script requires. Loaded carelessly, it leaves a visitor on a slow connection looking at a blank area while several hundred kilobytes download.

Subset to the ranges you actually use, self-host, and set font-display to swap so text appears immediately in a fallback. Then check the fallback: many system stacks render Devanagari poorly or not at all, so name a specific fallback rather than relying on sans-serif.

Test on a real Android device, not only in a desktop browser. Glyph rendering differs, and conjunct forms are where the differences show.

The tests worth running

Before launch: enter a Nepali name in every form and search for it. Sort a list of Nepali names and have a Nepali speaker confirm the order. Load the site on an Android phone on a throttled connection and watch what happens in the first two seconds. Paste text copied from a Word document, which is the most common source of mixed normalisation forms.

Four tests, half an hour, and they catch nearly every bug in this class before a user does.