Friday, 4 February 2022

Unicode Trivia U+0BD0

Codepoint: U+0BD0 "TAMIL OM"
Block: U+0B80..0BFF "Tamil"

Looking for mildly interesting facts within a Unicode block means a bit of research. Take block U+0B80..0BFF "Tamil"; here are my go-to places for information:

  1. Wikipedia (language, script and block)
  2. Unicode code chart
  3. Unicode core specification (Section 12.6)
  4. ScriptSource
  5. Omniglot
  6. Richard Ishida's excellent r12a

Obviously, many of these sites link to other resources. And therein lies the "fun".

Looking at the official code chart I found U+0BD0 "TAMIL OM". Of this codepoint, r12a says:

OM is a religious concept found in all three major religions born in India viz. Hinduism, Jainism and Buddhism. ௐ [U+0BD0 TAMIL OM] is widely used in Hindu religious texts, temple publications, and as neon lamps of sign boards in shops etc.

Hmm. That's a bit jarring, isn't it? How did the reference to "neon lamps of sign boards in shops" make it into a list of sacred uses? A quick google of that exact phrase only turns up references to r12a. But I cannot imagine Richard Ishida conjuring up that phrase from thin air.

U+0BD0 "TAMIL OM" was added in Unicode 5.1 (April 2008); recently enough for there to be quite a good paper trail. Indeed, the proposal (N3119) to add it was submitted in April 2006 by the International Forum for Information Technology in Tamil (INFITT) Working Group 2 (WG02). Section 2.1 of the proposal says:

Devanagari and Gujarati scripts have a sign named OM in their Unicode ranges. However in Tamil the corresponding slot is left vacant. Gurmukhi script also has an OM sign. Tamil OM sign is widely used in Hindu religious texts, temple publications, and as neon lamps of sign boards in shops etc. OM is a religious concept found in all three major religions born in India viz. Hinduism, Jainism and Buddhism. This document proposes to add the character TAMIL OM in Unicode Tamil range at U+0BD0.

Surely this must be the source of the "neon lamps" narrative? Somehow Google haven't (yet) indexed it.

Written proposals to the Unicode committee usually have examples (known as "attestations") attached to their end. Alas, proposal N3119 does not provide a photograph of a neon shop sign.

It is also surprisingly difficult to find shop signage featuring the Tamil Om on the internet; though other Oms are available. The only good match I found was this:

[source]

This is from the IndiaMART page of Sudha Neon Lights of Chennai, Tamil Nadu. The Tamil Om is in green; the red "spear" is Vel , the divine javelin of Murugan, the Hindu God of war. The blue text underneath is, I believe, "முருகா" or "Muruga", an alternative spelling of Murugan.

Given the paucity of images of neon lamp signage in shops incorporating Tamil Om, I wonder just how common it is in Southern India and where the suggestion in N3119 actually comes from. Alas, INFITT/WG02 was dissolved some time before May 2020, so we may never know.

Thursday, 3 February 2022

Unicode Trivia U+0B77

Codepoint: U+0B77 "ORIYA FRACTION THREE SIXTEENTHS"
Block: U+0B00..0B7F "Oriya"

The Odia language (formerly named Oriya) is spoken in Odisha (formerly Orissa): 

[source]

Unlike many Brahmic scripts, the head bar of each glyph is not a contiguous, straight line. As Omniglot says:

The Odia script developed from the Kalinga script, one of the many descendants of the Brahmi script of ancient India. The earliest known inscription in the Odia language, in the Kalinga script, dates from 1051.

The curved appearance of the Odia script is a result of the practice of writing on palm leaves, which have a tendency to tear if you use too many straight lines.

There are six "fractions signs" added to the "Oriya" block in Unicode 6.0 (October 2010):

  • U+0B72 "୲" ORIYA FRACTION ONE QUARTER
  • U+0B73 "୳" ORIYA FRACTION ONE HALF
  • U+0B74 "୴" ORIYA FRACTION THREE QUARTERS
  • U+0B75 "୵" ORIYA FRACTION ONE SIXTEENTH
  • U+0B76 "୶" ORIYA FRACTION ONE EIGHTH
  • U+0B77 "୷" ORIYA FRACTION THREE SIXTEENTHS

The original proposal by Anshuman Pandey explains that they were primarily used to subdivide one rupee into sixteen annas. See also Section 9.5 of South Asian Scripts-I (6.0).

Why does it stop at U+0B77 "ORIYA FRACTION THREE SIXTEENTHS"? It first glance, it looks like there must be some codepoints missing, but Anshuman Pandey explains that this is an additive base-4 system, where you can express "N/16" for N=1..15 with at most two of the above codepoints:

  • 1/16 = "୵" = 1/16
  • 2/16 = "୶" = 1/8
  • 3/16 = "୷" = 3/16
  • 4/16 = "୲" = 1/4
  • 5/16 = "୲୵" = 1/4 + 1/16
  • 6/16 = "୲୶" = 1/4 + 1/8
  • 7/16 = "୲୷" = 1/4 + 3/16
  • 8/16 = "୳" = 1/2
  • 9/16 = "୳୵" = 1/2 + 1/16
  • 10/16 = "୳୶" = 1/2 + 1/8
  • 11/16 = "୳୷" = 1/2 + 3/16
  • 12/16 = "୴" = 3/4
  • 13/16 = "୴୵" = 3/4 + 1/16
  • 14/16 = "୴୶" = 3/4 + 1/8
  • 15/16 = "୴୷" = 3/4 + 3/16

As supporting evidence, he also includes a passage from "First Lessons in Oriya" by A. H. Young (1953, revised by B. Das. Cuttack: Orissa Mission Press):

The leading principle of Oriya arithmetic, to divide by four rather than any other number, pervades also the system of fractions.

This suggests base-4 was used elsewhere in the region's number system. I haven't been able to find any other concrete examples for Odia, but Kharosthi numbers have a base-4 component and most other Brahmic scripts have fractions built upon quarters or sixteenths, such as Bengali that we saw earlier.

Wednesday, 2 February 2022

Unicode Trivia U+0AF1

Codepoint: U+0AF1 "GUJARATI RUPEE SIGN"
Block: U+0A80..0AFF "Gujarati"

Sometimes a codepoint loses its lustre. Take U+0AF1 "GUJARATI RUPEE SIGN" as an example.

  • October 1991 — The "Gujarati" block is imported from ISCII into Unicode 1.0 without a specific rupee symbol

The rise...

  • July 2001 — The Indian Ministry of Information Technology suggests the addition of a Gujarati rupee symbol
  • November 2001 — The Unicode Technical Committee agrees to "add this rupee sign for Gujarati to the list of proposed additions, since the symbol is not made from pieces that are already encoded Gujarati characters. The form of this character is very Gujarati-like, and it will be proposed for encoding at this location, rather than in the Currency Symbols block."
  • April 2003 — U+0AF1 "GUJARATI RUPEE SIGN" is formally added to Unicode 4.0

U+0AF1

And fall...

  • October 2009 — Anshuman Pandey proposes the addition of a Gujarati abbreviation sign
  • October 2009 — Anshuman Pandey also proposes that U+0AF1 be deprecated as, with the addition of the abbreviation sign, the Gujarati rupee can be rendered using the codepoint sequence:

    • U+0AB0 "GUJARATI LETTER RA"
    • U+0AC2 "GUJARATI VOWEL SIGN UU"
    • U+0AF0 "GUJARATI ABBREVIATION SIGN"

  • January 2012 — U+0AF0 "GUJARATI ABBREVIATION SIGN" is formally added to Unicode 6.1

Of course, you cannot just remove an existing codepoint from the Unicode standard. What would you do with all the documents that had already embedded U+0AF1 as the rupee symbol? Instead, an annotation was added to U+0AF1 saying

preferred spelling is 0AB0 0AC2 0AF0

Job done? Not quite...

  • September 2018 — Charlotte Buff points out an inconsistency. She "identified the following 18 characters [including U+0AF1] that are strongly implied to be deprecated in the code charts, but actually aren’t in the UCD". She also raises the point that "U+0AF1 does not decompose into its preferred representation"

Should U+0AF1 be formally deprecated? Or should its usage be "discouraged"? Should codepoints in general be decomposed into their preferred spellings?

Personally, I think this is a case that's getting beyond the purview of the core Unicode Standard. Let's face it, U+0AF1 is already out there. Of course, it's difficult to know how prevalent it is; but even one occurrence makes it irrevocable.

And how exactly do you discourage the use of a codepoint, let alone deprecate it? Do you raid people's homes in the middle of the night and confiscate all the Gujarati Rupee codepoints?

The keen-eyed reader will have noticed I haven't actually used codepoint U+0AF1 in this post. I don't want to be woken up at 2am, thank you very much!

Tuesday, 1 February 2022

Unicode Trivia U+0A70

Codepoint: U+0A70 "GURMUKHI TIPPI"
Block: U+0A00..0A7F "Gurmukhi"

Sometimes the information in the Unicode Character Database (UCD) is either insufficient for some purpose or requires clarification. This is the role of Unicode Technical Reports (UTR) and Unicode Technical Notes (UTN).

Like other Brahmic scripts, the Gurmukhi script was imported into Unicode 1.0 as part of ISCII, where it was known as Punjabi.

U+0A02 and U+0A70 highlighted in yellow [source]

However, Gurmukhi differs in having two diacritics for nasalisation, the Bindi and Tippi:

U+0A02 "GURMUKHI SIGN BINDI"
U+0A70 "GURMUKHI TIPPI"

The ISCII codepage for Punjabi uses character 0xA2 for both diacritics with the expectation that the correct combining glyph will be rendered according to context. This logic is clarified in UTN #30 from Sukhjinder Sidhu (2006):

Bindi and Tippi are encoded using a single code point in ISCII (0xA2) and the underlying rendering engine selects the correct glyph. However, in Unicode they are given two separate code points.

Thus, 0xA2 should be converted to U+0A70 (Tippi) when:

  • The preceding letter is a consonant (ignoring any Nuktas)
  • The preceding letter is Vowel Sign I (U+0A3F), Vowel Sign U (U+0A41), Vowel Sign UU (U+0A42)
  • The preceding letter is Letter A (U+0A05), Letter I (U+0A07)

In all other cases, the sign should remain a Bindi (U+0A02).

When converting from Unicode to ISCII, both Bindi and Tippi should be converted to Bindi (0xA2).

This special case logic isn't part of the core Unicode Standard; it is advisory only. But Sukhjinder Sidhu points out

If the advice in this document is not heeded, any resulting conversion will not be legible to readers of the Gurmukhi script,

One would hope that software vendors would take heed, but a casual read of Microsoft's .NET core library source reveals no implementation (or even mention) of UTN #30 in ISCIIEncoding,cs. The code maps 0xA2 to and from U+0A02 (Bindi) but provides no transformations for Tippi. At the top of the C# source file is a comment:

Ported from windows c_iscii. If you find bugs here, there're likely similar bugs in the windows version

I decided not to look any further.

Monday, 31 January 2022

Unicode Trivia U+09F8

Codepoint: U+09F8 "BENGALI CURRENCY NUMERATOR ONE LESS THAN THE DENOMINATOR"
Block: U+0980..09FF "Bengali"

Decimal Day. 15 February 1971. A Monday. The day the United Kingdom and the Republic of Ireland converted to decimal currency. Before that, each pound was divided into twenty shillings and each shilling into twelve pence. We'll ignore farthings.

So, if I bought something worth one penny with an old, pre-decimalisation five pound note, I'd get a dirty look and the following change:

£4 19/11 = £4. 19s. 11d = 4 pounds, 19 shillings, 11 pence

Such mixed-radix currencies were not uncommon. In British India, the rupee had been divided into sixteen annas, each anna into four pice (paisa), and each pice into three pies. The change from five rupees for a one pie item would be:

Rs. 4/15/3/2 = 4 rupees, 15 annas, 3 pice, 2 pies

In pre-decimal Bengal, the taka (rupee) had been divided into sixteen ana, and each ana into twenty ganda. The change from five taka for a one ganda item would be:

Tk. 4/15/19 = 4 taka, 15 ana, 19 ganda

Of course, that's in English using the Latin script and Western Arabic numerals. In Bengali, one could have written:

৪৲৸৶৹৻১৯

U+09EA U+09F2 U+09F8 U+09F6 U+09F9 U+09FB U+09E7 U+09EF

As Anshuman Pandey, points out, only one currency mark was actually used when multiple units were written. We'll return to this in due course, but in the meantime I've left that refinement out of the example above.

Bengali is a Brahmic script written left-to-right, so in Unicode this example is:

  1. U+09EA "BENGALI DIGIT FOUR"
  2. U+09F2 "BENGALI RUPEE MARK"
  3. U+09F8 "BENGALI CURRENCY NUMERATOR ONE LESS THAN THE DENOMINATOR"
  4. U+09F6 "BENGALI CURRENCY NUMERATOR THREE"
  5. U+09F9 "BENGALI CURRENCY DENOMINATOR SIXTEEN"
  6. U+09FB "BENGALI GANDA MARK"
  7. U+09E7 "BENGALI DIGIT ONE"
  8. U+09EF "BENGALI DIGIT NINE"

The first two glyphs ("৪৲") represent "4 taka" in decimal; the Bengali digit four just happens to look like a Western Arabic digit eight. The next three glyphs ("৸৶৹") represent "15 ana". This is complicated by the fact that, traditionally, ana were written as fractions of a taka. Finally, the last three glyphs ("৻১৯") represent "19 ganda" in decimal where, just to confuse us further, the ganda mark comes before the digits, not after them as with taka and ana.

The ana component is the most perplexing. The Unicode codepoint name for U+09F8, "BENGALI CURRENCY NUMERATOR ONE LESS THAN THE DENOMINATOR", doesn't really help. Fortunately, there's an explanation within the much later proposal to add the ganda mark in 2007.

The fifteen possible quantities of ana are:

  • ৴৹ = 1 ana (Numerator 1)
  • ৵৹ = 2 ana (Numerator 2)
  • ৶৹ = 3 ana (Numerator 3)
  • ৷৹ = 4 ana (Numerator 4)
  • ৷৴৹ = 5 ana
  • ৷৵৹ = 6 ana
  • ৷৶৹ = 7 ana
  • ৷৷৹ = 8 ana
  • ৷৷৴৹ = 9 ana
  • ৷৷৵৹ = 10 ana
  • ৷৷৶৹ = 11 ana
  • ৸৹ = 12 ana (Numerator One Less Than the Denominator)
  • ৸৴৹ = 13 ana
  • ৸৵৹ = 14 ana
  • ৸৶৹ = 15 ana

This looks like a modified base-4 tally mark system. But, thinking back to what Anshuman Pandey said about elided currency marks, I wonder if this scheme didn't originate in a finer-grained positional system.

Imagine that instead of the taka being divided directly into sixteen ana, it was divided into four virtual "beta", which were themselves divided into four virtual "alpha". Obviously:

  • ana = alpha + beta * 4

But now we have the following encoding:

  • ৴ = 1 alpha (Numerator 1)
  • ৵ = 2 alpha (Numerator 2)
  • ৶ = 3 alpha (Numerator 3)
  • ৷ = 1 beta
  • ৷৷ = 2 beta
  • ৸ = 3 beta (Numerator One Less Than the Denominator)

For beta, the "denominator" is indeed four, to the mysterious U+09F8 "BENGALI CURRENCY NUMERATOR ONE LESS THAN THE DENOMINATOR" suddenly makes sense.

We can now come up with an algorithm for writing out a currency amount according to the scheme described by Anshuman Pandey:

  • Let T, A, G be the number of taka (0 or more), ana (0 to 15), ganda (0 to 19) respectively
  • If T is not zero then
    • Write out the Bengali decimal representation of T
    • If both A and G are zero
      • Write out U+09F2 "BENGALI RUPEE MARK"
      • We're finished
  • Let α be A modulo 4 (0 to 3)
  • Let β be A divided by 4, rounded down (0 to 3)
  • If β is 1, write out U+09F7 "BENGALI CURRENCY NUMERATOR FOUR"
  • If β is 2, write out U+09F7 "BENGALI CURRENCY NUMERATOR FOUR" twice
  • If β is 3, write out U+09F8 "BENGALI CURRENCY NUMERATOR ONE LESS THAN THE DENOMINATOR"
  • If α is 1, write out U+09F4 "BENGALI CURRENCY NUMERATOR ONE"
  • If α is 2, write out U+09F5 "BENGALI CURRENCY NUMERATOR TWO"
  • If α is 3, write out U+09F6 "BENGALI CURRENCY NUMERATOR THREE"
  • If G is zero then
    • Write out U+09F9 "BENGALI CURRENCY DENOMINATOR SIXTEEN"
    • We're finished
  • Write out U+09FB "BENGALI GANDA MARK"
  • Write out the Bengali decimal representation of G
  • We're finished

For our example, T=4, A=15, G=19, α=3, β=3 and the output is:

৪৸৶৻১৯


U+09EA U+09F8 U+09F6 U+09FB U+09E7 U+09EF

This representation is surprisingly concise and totally unambiguous.

Sunday, 30 January 2022

Unicode Trivial U+0950

Codepoint: U+0950 "DEVANAGARI OM"
Block: U+0900..097F "Devanagari"

Next, we come to our first Brahmic script: Devanagari. Devanagari is the most widely-used Brahmic script. Almost 50% of the Indian population use it to write their native language. It is a left-to-right abugida used to write dozens of languages.

It is sometimes glibly called a "washing line" script because, unlike Latin/Greek/Cyrillic scripts that "sit" on top of their baselines, Devanagari also "hangs" from a head line:

Devanagari script
देवनागरी लिपि

Devanagari typography is non-trivial, even when the letterforms are isolated:

Devanagari Type Anatomy [source]

Of course, anthropomorphism in typography isn't limited to Brahmic scripts, but one can take "type anatomy" quite literally:

Design Parameters of Devanagari (Gokhale, 1983) [source]

The flipside of grammatology is phonology.

"Om" is the sound of a sacred spiritual symbol in Indic religions. In Devanagari, in its simplest form, it is written:

Om
ओम
U+0913 U+092E

However, there is also a ligatured codepoint in the Devanagari block named "DEVANAGARI OM":

Om (sign)
U+0950

There are currently many Oms encoded into Unicode. The Devanagari Om was added right at the onset in Unicode 1.0 (1991). This incorporated a wholesale import of the ISCII (Indian Script Code for Information Interchange, 1988) character sets which encode Om as a multi-byte sequence (0xA1 0xE9).  The Unicode Consortium allocated U+0950 "DEVANAGARI OM" to allow one-to-one mapping of ISCII on this basis. Of course, once you allow one Om through...

Saturday, 29 January 2022

Unicode Trivia U+08C8

Codepoint: U+08C8 "ARABIC LETTER GRAF"
Block: U+08A0..08FF "Arabic Extended-A"

Unicode blocks are not allocated sequentially. Consequently, "Arabic Extended-A" (U+08A0..08FF, originating in Unicode 6.1, January 2012) comes numerically after "Arabic Extended-B" (U+0870..089F, originating in Unicode 14.0, September 2021).

Even more confusing, codepoints within a block can be allocated at different times. For example, U+08C8 "ARABIC LETTER GRAF" was assigned in Unicode 14.0 (September 2021); but its neighbour, U+08C7 "ARABIC LETTER LAM WITH SMALL ARABIC LETTER TAH ABOVE" was assigned in Unicode 13.0 (March 2020).

U+08C8 "ARABIC LETTER GRAF" is an addition to the Arabic script for writing the Balti language:

Isolated form of U+08C8 [source]

For example, the Balti word for knife (U+08C8 U+06CC):

[source]

U+08C8 is the only specific addition required for the Arabic script to be able to write Balti, although two other codepoints (U+0F6B "TIBETAN LETTER KKA" and U+0F6C "TIBETAN LETTER RRA", added in Unicode 5.1, April 2008) are needed to write Balti using the Tibetan script.

Balti is spoken in Baltistan, or Little Tibet, a mountainous region in the Gilgit-Baltistan part of Pakistan-administered Kashmir. This may be the origins of the balti curry dish popular in the UK since the nineteen-seventies.