LocaleNewApproach [FreeBSD WIKI]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== New approach to the FreeBSD locale database ======

===== Background =====

Over the years the %%FreeBSD%% locale database (share/colldef,
share/monetdef, share/msgdef, share/numericdef, share/timedef) has
accumulated a total of 165 definitions (language - country-code -
character-set triplets). The contents of the files is for Western
European languages often low-ASCII but for Eastern European and
Asian languages partly or fully high-ASCII. Without knowing how to
display or interpret the character-sets, it is difficult to make
sure by the general audience that the local languages (language -
country-code) definitions is displayed properly in various
character-sets.

===== Solution =====

With a per definition (language - country-code) low-ASCII file with
the definitions of the characters for the fields, it would be
possible to generate the various character-sets for that language.

===== What do we need =====

  * A database with all character encoding definitions. The Unicode Project defines these.
  * An intermittent format which can be used to convert these encodings into unique characters. The UTF-8 character-set supports this.
  * A tool to convert from the intermittent format into the various character-sets. Libiconv (GPL) and bsdiconv (BSDL) can do this.
  * A Makefile which glues everything together.

===== Gotchas =====

  * Some countries do not only have multiple languages (nl_BE and fr_BE for example), but some of them have also different font families: sr_Cyrl_RS and sr_Latn_RS.
  * Duplicate detection has always been a manual thing and is tricky to do initially. Right now this keeps being the job of the maintainers of the locale data in the SCM repository.

===== Examples =====

The word for the last day of the week in the en_US language - country
code would be in Unicode format:

<code>
<LATIN CAPITAL LETTER S><LATIN SMALL LETTER A><LATING SMALL LETTER T><LATIN SMALL LETTER U><LATIN SMALL LETTER R><LATIN SMALL LETTER D><LATIN SMALL LETTER A><LATIN SMALL LETTER Y>
</code>

Converted into UTF-8 this will be:

<code>
Saturday
</code>

Converted into ISO-8859 this will be:

<code>
Saturday
</code>

The word for the last day of the week in the ru_RU language -
country code would be in Unicode format:

<code>
<CYRILLIC SMALL LETTER ES><CYRILLIC SMALL LETTER U><CYRILLIC SMALL LETTER BE><CYRILLIC SMALL LETTER BE><CYRILLIC SMALL LETTER O><CYRILLIC SMALL LETTER TE><CYRILLIC SMALL LETTER A>
</code>

Converted into UTF-8 this will be:

<code>
<D1><81><D1><83><D0><B1><D0><B1><D0><BE><D1><82><D0><B0>
</code>

Converted into KOI8-R this will be:

<code>
<D3><D5><C2><C2><CF><D4><C1>
</code>

===== Careful! =====

  * In the timedef definitions, do not convert the %A into Unicode format because the %A is a low-ASCII input for strftime(). Also don't put the md_order in Unicode format because that is a low-ASCII definition.
  * libiconv doesn't understand ISCII-DEV, bsdiconv calls it macdevanaga.
  * Backwards compatibility: There are a bunch of old or obsolete names in the %%FreeBSD%% locale definitions (sr_YU -> sr_Cyrl_RS and sr_Latn_RS, zh_HK -> zh_Hant_HK, zh_CN -> zh_Hans_CN) which still might be needed.

===== Current status =====

Finished:
  * Conversion of the current locale data into the Unicode format for share/monetdef, share/msgdef, share/numericdef, share/timedef.
  * Conversion of the current Makefiles to support the new approach. It also adds the file src/share/Makefile.def.inc which does do the magic between the definitions in the Makefile and the %%FreeBSD%% bsd.*.mk. Done for share/colldef, share/monetdef, share/msgdef, share/numericdef, share/timedef.
  * Regression check.
  * Conversion of the Unicode definitions to the UTF-8 character-set. It is residing in usr.bin/unicode2utf8 and requires the file posix/UTF-8.cm from the CLDR distribution.

Pending:
  * Checking of the data with the CLDR (Common Locale Data Repository) for completeness of the current data.
  * Conversion of Makefiles for share/mklocale.
  * Import of the file UTF-8.cm (from the CLDR project) into the base operating system. These files for now live in src/tools/tools/locale/

Pending third parties:
  * bsdiconv in the base operating system.

===== SCM =====

(Currently the SCM contains all the definitions (language - country-code
- character-set) in low and high-ASCII. To keep the SCM history, we
will once move them to their .unicode extension and then overwrite
them with the Unicode encoding definitions)

The .unicode files are stored in SCM and will be, in the long term,
be the only source in SCM. Right now due to lack of bsdiconv in the
base operating system we will have to store also the character-map
sources (.src) files into the SCM. Once bsdiconv is in the base
system these files can be removed and the whole database can be
made self-hosted.

===== Testing (before move to src/tools/tools/locale) =====

To test the current system, you need the following data:

  * A copy of the CLDR, available from http://cldr.unicode.org/. Currently version 1.7.1 is used. We only use the file posix/UTF-8.cm from it.
  * A copy of svn:%%//%%svn.freebsd.org/base/user/edwin/locale/.
  * A copy of bsdiconv from p4:%%//%%/depot/gabor/something.

Local configuration:

  * Add to /etc/make.conf (make sure they match your directory layout)

<code>
CLDRDIR=        /home/edwin/unicode/cldr/1.7.1
LOCALE_DESTDIR= /home/edwin/locale/new
LOCALE_SHAREOWN=edwin
LOCALE_SHAREGRP=edwin
</code>

Test it out:

  * Go to the SVN directory /user/edwin/locale/share. The Makefile there only includes the locale directories, so there is no need to be worried about the other .
  * Run "FULL=1 make clean" to get rid of all generated files, even the ones in the SCM. You should only have the *.unicode and the Makefiles now.
  * Run "FULL=1 make" to recreate everything. 
  * Run "make clean" to get rid of all data not in the SCM.
  * Run "make" to recreate the data not in the SCM.

<code>
#
# All targets for TARGET_CHARACTERMAP
#
# .unicode -> .utf-8.src -> .utf-8.out
#                 \__ .iso8859-1.src -> .iso8859-1.out
# <----1---><--2---><------3--------><----4----->
#
# 1. The files .unicode are stored in the SCM and are the source
#    for the whole further system
# 2. The Perl script converts the .unicode files and the Unicode
#    CLDR database into UTF-8 code
# 3. The UTF-8 gets converted by libiconv or bsdiconv in the specific
#    character-map.
# 4. Get rid of the comments.
#
# As long as there is no bsdiconv, the files with the extension
# .unicode and .src must be stored in the SCM and will not be
# generated as part of the build process.
#
</code>