Wednesday, 12 January 2022

Unicode Trivial U+0256

Codepoint: U+0256 "LATIN SMALL LETTER D WITH TAIL"
Block: U+0250..02AF "IPA Extensions"

The "Ð" (U+00D0 "LATIN CAPITAL LETTER ETH") that we first met in the Icelandic alphabet is easily confused with other Unicode codepoints:

  • "Đ" (U+0110 "LATIN CAPITAL LETTER D WITH STROKE")
  • "Ɖ" (U+0189 "LATIN CAPITAL LETTER AFRICAN D")

These three codepoints are considered distinct according to the Unicode standard but are typically rendered (almost) identically.

The lowercase mapping of the African D (which sits in the "Latin Extended-B" block) is "ɖ" (U+0256 "LATIN SMALL LETTER D WITH TAIL") which sits in the "IPA Extensions" block. Note that the lowercase is not called "LATIN SMALL LETTER AFRICAN D" as you might expect. However, the uppercase mapping of "LATIN SMALL LETTER D WITH TAIL" is indeed "LATIN CAPITAL LETTER AFRICAN D", so they're obviously a pair.

At first, I thought the African D naming inconsistency was because the two blocks ("Latin Extended-B" and "IPA Extensions") were added to the Unicode standard at different times. But the UCD tells us the codepoint were both in the original version 1.0, though under differently-named blocks.

An alternative reason for the codepoint naming inconsistency may be due to history of the African D itself.

The African D is the sixth letter of the International African Alphabet developed in 1928 by Diedrich Hermann Westermann and others. This alphabet was a precursor to the African Reference Alphabet (1978) and the World Orthography alphabet (1948).

International African Alphabet

The International African Alphabet was itself developed from the International Phonetic Alphabet which has been evolving since 1888. However, the IPA famously does not have uppercase versions of its letters. Fortunately, the International Institute of African Languages and Cultures had already prepared such a mapping in "The Practical Orthography of African Languages" (1928) [transcript]:

[source]

The International Phonetic Alphabet does not have a dedicated block of its own in Unicode as its characters are mainly "borrowed" from other sources (e.g. Latin, Greek and Cyrillic blocks). During the original compilation of Unicode, any phonetic character that wasn't extant elsewhere was added to the new "Standard Phonetic" (later renamed "IPA Extensions") block. The original code charts show U+0189 (p.185 "Extended Latin") and U+0256 (p.189 "Standard Phonetic") with their current names and the expected case mapping between them.

In a Unicode meeting (ISO/IEC JTC 1/SC 2/WG 2 on 1994-06-01 item N 989), a suggestion was made to rename U+0189 to "LATIN CAPITAL LETTER OF LETTER D WITH TAIL", but this was withdrawn for unknown reasons. The alternatives are to rename U+0189 to "LATIN CAPITAL LETTER D WITH TAIL" which is obviously visually incorrect, or to rename U+0256 to "LATIN SMALL LETTER AFRICAN D". The last suggestion is problematic because I'm sure codepoint U+0256 is primarily used in the context of phonetics.

I believe the two codepoints were added to Unicode 1.0 with their current names knowingly inconsistent as a compromise. And compromises are what standards committees are all about.

Tuesday, 11 January 2022

Unicode Trivia U+01BF

Codepoint: U+01BF "LATIN LETTER WYNN"
Block: U+0180..024F "Latin Extended-B"

The Old English (Anglo-Saxon) alphabet (circa 8th to 12th centuries) had 24 letters:

Aa Ææ Bb Cc Dd Ðð Ee Ff Ᵹᵹ Hh Ii Ll Mm Nn Oo Pp Rr Ss Tt Uu Ƿƿ Xx Yy Þþ

The twenty-first letter (uppercase "Ƿ", lowercase "ƿ") is named "wynn". This alphabet was derived from Latin, which lacked a /w/ sound, so the digraph "uu" was used instead. Hence the name "double-u".

Later, "uu" was replaced by the runic symbol "ᚹ" (U+16B9 "RUNIC LETTER WUNJO WYNN W") that morphed into the Latin wynn. After the Norman Conquest, French scribes abandoned the wynn, possibly because it was easily confused with "Pp" and/or "Þþ" (thorn), and reverted to "double-u".

There are many infographics about the evolution of the English alphabet, including one by Useful Charts. I've tried to construct one from the perspective of codepoints in Unicode 14.0. I've also ignored minuscules which means that topics such as Carolingian are omitted.

There's an interactive rendition of this table on the "Unicode Tour" web page that accompanies these blog posts.

Unicode Trivia U+0132

Codepoint: U+0132 "LATIN CAPITAL LIGATURE IJ"
Block: U+0100..017F "Latin Extended-A"

[If you haven't already noticed, I'm trying to come up with an mildly interesting fact about one codepoint in every Unicode block. As of Version 14.0, that's 320 blocks]

The Latin script is the basis of many alphabets. Of the languages whose Latin-script alphabets can (currently) be expressed as single codepoints, here are a selection [source]:

Danish
Aa Bb Cc Dd Ee Ff Gg Hh Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Vv Ww Xx Yy Zz Ææ Øø Åå

Dutch
Aa Bb Cc Dd Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Vv Ww Xx Yy IJij Zz

English
Aa Bb Cc Dd Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Vv Ww Xx Yy Zz

Estonian
Aa Bb Cc Dd Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Šš Zz Žž Tt Uu Vv Ww Õõ Ää Öö Üü Xx Yy

Faroese
Aa Áá Bb Dd Ðð Ee Ff Gg Hh Ii Íí Jj Kk Ll Mm Nn Oo Óó Pp Rr Ss Tt Uu Úú Vv Yy Ýý Ææ Øø 

Finnish
Aa Bb Cc Dd Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Vv Xx Yy Zz Ää Öö

French
Aa Bb Cc Dd Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Vv Ww Xx Yy Zz

German
Aa Bb Cc Dd Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Vv Ww Xx Yy Zz Ää Öö Üü
ẞß

Icelandic
Aa Áá Bb Dd Ðð Ee Éé Ff Gg Hh Ii Íí Jj Kk Ll Mm Nn Oo Óó Pp Rr Ss Tt Uu Úú Vv Xx Yy Ýý Þþ Ææ Öö

Irish
Aa Bb Cc Dd Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Vv Ww Xx Yy Zz

Italian
Aa Bb Cc Dd Ee Ff Gg Hh Ii Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Vv Zz

Latvian
Aa Āā Bb Cc Čč Dd Ee Ēē Ff Gg Ģģ Hh Ii Īī Jj Ķķ Ll Ļļ Mm Nn Ņņ Oo Pp Rr Ss Šš Tt Uu Ūū Vv Zz Žž

Lithuanian
Aa Ąą Bb Cc Čč Ee Ęę Ėė Ff Gg Hh Ii Įį Yy Jj Kk Ll Mm Nn Oo Pp Rr Ss Šš Tt Uu Ųų Ūū Vv Zz Žž

Norwegian
Aa Bb Cc Dd Ee Ff Gg Hh Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Vv Ww Xx Yy Zz Ææ Øø Åå

Polish
Aa Ąą Bb Cc Ćć Dd Ee Ęę Ff Gg Hh Ii Jj Kk Ll Łł Mm Nn Ńń Oo Óó Pp Rr Ss Śś Tt Uu Ww Yy Zz Źź Żż

Portuguese
Aa Bb Cc Dd Ee Ff Gg Hh Ii Jj Ll Mm Nn Oo Pp Qq Ss Tt Uu Vv Xx Zz

Romanian
Aa Ăă Ââ Bb Cc Dd Ee Ff Gg Hh Ii Îî Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Şş Tt Ţţ Uu Vv Ww Xx Yy Zz

Sami
Aa Áá Bb Cc Čč Dd Đđ Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Ŋŋ Oo Pp Rr Ss Šš Ŧŧ Uu Vv Zz Žž

Swedish
Aa Bb Cc Dd Ee Ff Gg Hh Ii Jj Kk Ll Mm Nn Oo Pp Qq Rr Ss Tt Uu Vv Ww Xx Yy Zz Åå Ää Öö

Turkish
Aa Bb Cc Çç Dd Ee Ff Gg Ğğ Hh Iı Ii Jj Kk Ll Mm Nn Oo Öö Pp Rr Ss Şş Tt Uu Üü Vv Vv Yy Zz

Some languages use digraphs (or similar) in their alphabets which are not single codepoints in Unicode:

Albanian
Aa Bb Cc Çç Dd DH/dh Ee Ëë Ff Gg GJ/gj Hh Ii Jj Kk Ll LL/ll Mm Nn NJ/nj Oo Pp Qq Rr RR/rr Ss SH/sh Tt TH/th Uu Vv Xx XH/xh Yy Zz ZH/zh

Croatian
Aa Bb Cc Čč Ćć Dd DŽ/dž Đđ Ee Ff Gg Hh Ii Jj Kk Ll LJlj Mm Nn NJ/nj Oo Pp Rr Ss Šš Uu Vv Zz Žž

Czech
Aa Bb Cc Čč Dd Ee Ff Gg Hh CH/ch Ii Jj Kk Ll Mm Nn Oo Pp Rr Řř Ss Šš Tt Uu Vv Ww Xx Yy Zz Žž

Hungarian (standard)
Aa Áá Bb Cc CS/cs Dd DZ/dz DZS/dzs Ee Éé Ff Gg GY/gy Hh Ii Íí Jj Kk Ll LY/ly Mm Nn NY/ny Oo Óó Öö Őő Pp Qq Rr Ss SZ/sz Tt TY/ty Uu Úú Üü Űű Vv ZZS/zzs

Spanish
Aa Bb Cc CH/ch Dd Ee Ff Gg Hh Ii Jj Kk Ll LL/ll Mm Nn Ññ Oo Pp Qq Rr RR/rr Ss Tt Uu Vv Ww Xx Yy Zz

Welsh
Aa Bb Cc CH/ch Dd DD/dd Ee Ff FF/ff Gg NG/ng Hh Ii Jj Ll LL/ll Mm Nn Oo Pp PH/ph Rr RH/rh Ss Tt TH/th Uu Ww Yy

According to Wikipedia, the largest Latin (and European) true alphabet is Slovak with 46 letters:

Slovak
Aa Áá Ää Bb Cc Čč Dd Ďď DZ/dz DŽ/dž Ee Éé Ff Gg Hh CH/ch Ii Íí Jj Kk Ll Ĺĺ Ľľ Mm Nn Ňň Oo Óó Ôô Pp Qq Rr Ŕŕ Ss Šš Tt Ťť Uu Úú Vv Ww Xx Yy Ýý Zz Žž

Italian has just 21.

A number of digraphs (and similar) are encoded in Unicode:

  • DZ, Dz, dz (U+01F1, U+01F2, U+01F3)
  • DŽ, Dž, dž (U+01C4, U+01C5, U+01C6)
  • IJ, ij (U+0132, U+0133)
  • LJ, Lj, lj (U+01C7, U+01C8, U+01C9)
  • NJ, Nj, nj (U+01CA, U+01CB, U+01CC)

Of these, the Dutch "IJ" (U+0132 "LATIN CAPITAL LIGATURE IJ") is unusual in not having a titlecase mapping distinct from its uppercase mapping. Consider the Dutch word "ijsje" meaning ice cream; in uppercase it is "IJSJE" and in titlecase it is "IJsje" not "Ijsje".

This quirk is considered by many as evidence that "IJ" should be considered a true letter in its own right. That possibly includes this purveyor of alcohol:

"Slijterij" means "Off-Licence"

However, the replacement of the "IJ" tile with "Y" in Dutch Scrabble may be another nail in the coffin.

Monday, 10 January 2022

Unicode Trivia U+00F0

Codepoint: U+00F0 "LATIN SMALL LETTER ETH"
Block: U+0080..00FF "Latin-1 Supplement"

The Basic Latin Unicode block (U+0000..007F) is fine if you're writing English, but it quickly runs out of steam for other languages. For example, the Icelandic alphabet ("stafrófið") has 32 letters:

Aa Áá Bb Dd Ðð Ee Éé Ff Gg Hh Ii Íí Jj Kk Ll Mm Nn Oo Óó Pp Rr Ss Tt Uu Úú Vv Xx Yy Ýý Þþ Ææ Öö

The fifth letter "ð" is "eth", seen here in a sign in Landmannalaugar:

One of Unicode's founding principles is "universal repertoire" and, indeed, "LATIN SMALL LETTER ETH" has been assigned the unique codepoint U+00F0. Before Unicode, region-specific characters had a tendency to "move around" in codepoint space. For instance, in IBM DOS Code Page 861, small eth was at position 0x8C.

For every codepoint, the Unicode Character Database maintains a plethora of information. For U+00F0, we can view that data using an online utility:

https://util.unicode.org/UnicodeJsps/character.jsp?a=00F0

Amongst other things, we see that:

  • The official name of that codepoint is "LATIN SMALL LETTER ETH"
  • It belongs to the "Latin-1 Supplement" block (U+0080..00FF)
  • It primarily belongs to the "Latin" script
  • It was introduced in Unicode 1.1
  • Its General Category is "Lowercase Letter"
  • Its uppercase mapping is "Ð" U+00D0
  • Its titlecase mapping is also "Ð" U+00D0
  • etc.

The titlecase mapping is somewhat moot as there are no words in Icelandic that begin with "eth". This makes children's "A is for Apple"-style alphabet posters somewhat difficult to produce:

It also means that the uppercase letter "Ð" (U+00D0) only occurs in text where the whole word is uppercase, such as … erm … "STAFRÓFIÐ".

Saturday, 8 January 2022

Unicode Trivia U+0000

[This blog post is part of the Universe series of investigations]

Codepoint: U+0000 "CONTROL CODE <NUL>"
Block: U+0000..007F "Basic Latin"

Unicode is built on top of any number of standards. The history of the first Unicode codepoint (U+0000, a.k.a. control code <NUL>) is an object lesson in the workings of standards committees.

  • The Unicode 1.0 (1991) standard, along with ISO/IEC 10646, bases its first 128 codepoints on ISO/IEC 646.
  • ISO/IEC 646 (1972) was itself based on ASCII.
  • The first major release of the ASCII standard was ASA X3.4-1963.

In the late 1950s, American Telephone and Telegraph Company (AT&T) had stated to the predecessors of the ASA X3 committee a functional requirement for any new standard to have an all-zeroes character named NULL (or IDLE). See Chapter 13 of Coded Character Sets, History and Development by Charles E. Mackenzie, 1980.

This "all-zeroes" requirement was almost certainly due to the practice of leaving gaps in punched tape or cards that could be "overwritten" with other characters later on without having to reissue the whole tape or card deck. Similarly, AT&T also requested an all-ones bit pattern to delete a character by "punching out" all the holes in the row. As ASA X3.4-1963 was a 7-bit character set, this led to the <DEL> control code eventually ending up at U+007F.

Further back in time, the "all-zeroes" control code was part of the International Telegraph Alphabet No. 2 (ITA2) code of 1924. This, inexorably, was a development of Baudot (ITA1) code developed in the 1870s and patented in the United States in 1888:

The final three rows of the table above are:

  1. Figure switch
  2. Letter switch
  3. Instrument at rest

where "Instrument at rest" is the idle state for teleprinters, a.k.a. NUL.

Sunday, 19 December 2021

Universe 5: Font Detection

Any Unicode codepoint browser ends up as an exercise in font wrangling. Universe is no exception.

The threat of fingerprinting means a browser script cannot typically enumerate the locally installed fonts. There is also currently no native way of determining if a font supports a particular codepoint. One would think that determining font support for codepoints would therefore be problematic, but it turns out to be relatively easy for our use-case with Chrome.

Consider the following JavaScript snippet:

var canvas = document.createElement("canvas");
var context = canvas.getContext("2d");
context.font = "10px font-a,font-b,font-c";
var measure = context.measureText(text);

If a character in the "text" string is not supported by "font-a", Chrome text rendering falls back to using "font-b". If "font-b" also doesn't support the character, "font-c" is used. If the character is not supported by "font-c" either, a system default is used.

We can take advantage of this fall back mechanism by using a "blank" font that guarantees to render any glyph as zero-width/zero-mark. Fortunately, there's just such a font already out there: Adobe Blank:

@font-face {
  font-family: "blank";
  src: url("https://raw.githubusercontent.com/adobe-fonts/
         adobe-blank/master/AdobeBlank.otf.woff") format("woff");
}

Now, we can write a function to test a font for supported characters:

function IsBlank(font, text) {
  var canvas = document.createElement("canvas");
  var context = canvas.getContext("2d");
  context.font = `10px "${font}",blank`;
  var measure = context.measureText(text);
  return (measure.width <= 0) &&
    (measure.actualBoundingBoxRight <= -measure.actualBoundingBoxLeft);
}

The actual code in universe.js has some optimisations and additional features; see "MeasureTextContext()" and "TextIsBlank()".

Using this technique, we can iterate around some "well-known" fonts and render them where appropriate. For our example of codepoint U+0040:

The origins of each of the glyphs above are:

  1. "notoverse" is the 32-by-32 pixel bitmap glyph described previously.
  2. "(default)" is the default font used for this codepoint by Chrome. In my case, it's "Times New Roman".
  3. "(sans-serif)" is the default sans serif font: "Arial".
  4. "(serif)" is the default serif font: "Times New Roman".
  5. "(monospace)" is the default monospace font: "Consolas".
  6. "r12a.io" is the PNG from Richard Ishida's UniView online app.
  7. "glyphwiki.org" is the SVG from GlyphWiki. It looks a bit squashed because it's a half-width glyph; GlyphWiki is primarily concerned with CJK glyphs.
  8. "unifont" is the GNU Unifont font. I couldn't find a webfont-friendly source for version 14, so I had to piggyback version 12 from Terence Eden.
  9. "noto" is a composite font of several dozen Google Noto fonts. See ".noto" in universe.css.
  10. Subsequent "Noto ..." glyphs are from the corresponding, individual Noto font, in priority order.

Saturday, 18 December 2021

Universe 4: User Interface

The Universe character browser has a very basic user interface. In addition to the HTML5 presentation, there is a URL scheme that allows you to jump to various subpages.

Landing Page

universe.html#

The landing page gives a brief introduction and a collapsible table of statistics. As with all pages, a header contains breadcrumbs and a footer lists some useful jumping-off points.

Codepoints

To examine a specific codepoint, expressed in hexadecimal between "0000" and "10FFFF" inclusive, use:

universe.html#U+<hex>

In this case, "#U+0040" examines "@". The subpage lists the block, column and codepoint details, as well as font support, UCD data fields and external links, where appropriate.

Planes

To list all 17 Unicode planes, use:

universe.html#P

To examine details of the plane containing codepoint "U+<hex>", use:

universe.html#P+<hex>

Sheets

In this context, a "sheet" is a 32-by-32 grid of 1024 contiguous codepoints; it is how the Notoverse glyphs are organised.

To list all 161 sheets which contain allocated codepoints, use:

universe.html#S

To examine the sheet containing codepoint "U+<hex>", use:

universe.html#S+<hex>

Blocks

To list all 320 named Unicode blocks, use:

universe.html#B

To examine the block containing codepoint "U+<hex>", use:

universe.html#B+<hex>

This will list constituent codepoints, organised by column.

Columns

To examine just the column containing codepoint "U+<hex>", use:

universe.html#C+<hex>

This will list constituent codepoints in more detail.

Queries

To query the UCD for matching codepoints, use:

universe.html#<key>=<value>

The "<key>" can be:

  • One of the abbreviated UCD field names, such as "gc" or "na".
  • "id" to search the codepoint id (e.g. "U+0040").
  • "basic" to search the computed codepoint basic category.
  • "kind" to search the computed codepoint kind.
  • "script" to search the codepoint script list (see below).
  • "extra" to search NamesList.txt annotations.
  • "text" to search all the above.

The "<value>" can be simple text or a JavaScript regular expression in the form "/<expr>/<flags>". For example:

universe.html#na=/\bANT\b/i

This searches for codepoints whose name field contains the whole word "ANT" case-insensitively.

Scripts

Queries with the special key "script" search the fields "sc" and "scx". To query for the 29 codepoints of the Ogham script, use:

universe.html#script=Ogam

List all the 210 scripts of the ISO-15924 standard, use:

universe.html#script

Search

Queries with the special key "search" perform full-text searches. To bring up a search dialog, use:

universe.html#search

Gears

As mentioned earlier, loading the full UCD database and glyph sheets for the first time can take quite a few minutes. Searches can also take a few seconds. For long-running JavaScript functions, we display animated gears:

To keep the page responsive, we wrap the long-running functionality inside a call to the "Gears()" function in universe.js:

function Gears(parent, asynchronous, complete) {
  var gears = document.createElement("img");
  gears.className = "gears";
  gears.title = "Please wait...";
  gears.src = "gears.svg";
  parent.appendChild(gears);
  gears.onload = async () => {
    var result = await asynchronous();
    if (gears.parentNode === parent) {
      parent.removeChild(gears);
    }
    if (complete) {
      complete(result);
    }
  };
}

Inside the asynchronous, long-running function we have to make sure we periodically call "YieldAsync()":

function YieldAsync(milliseconds) {
  // Yield every few milliseconds 
  // (or every time this function is called if argument is missing)
  var now = performance.now();
  if (!YieldAsync.already) {
    // First call
    YieldAsync.already = Promise.resolve(undefined);
    YieldAsync.channel = new MessageChannel();
  } else if (milliseconds && ((now - YieldAsync.previous) < milliseconds)) {
    // Resolve immediately
    return YieldAsync.already;
  }
  YieldAsync.previous = now;
  return new Promise(resolve => {
    YieldAsync.channel.port1.addEventListener("message",
      () => resolve(), { once: true });
    YieldAsync.channel.port1.start();
    YieldAsync.channel.port2.postMessage(null);
  });
}

This was inspired by a much-underrated StackOverflow answer.