SysconsUnicodeProject

For any reader not familiar with Unicode, reading SysconsUnicodeProject/UnicodeBasics is suggested.

This project aims to enhance Unicode support on FreeBSD operating system by implementing support for UTF-8 in the Syscons console driver. This is not about full support for Unicode-based applications (because has no impact on correctness of sorting text strings, for example). Currently system console driver deals with 8-bit character sets only.

23/08/2008: Project initiated by Alexander Churanov. See this message to the lists.
01/01/2009: Syscons terminal emulator replaced with new implementation by Ed Schouten, which has optional xterm and UTF-8 support.
29/01/2009: The implementation of unicode maps, that allow using existing fonts in UTF-8 mode was created by Alexander Churanov.
04/08/2009: Unicode support for the keyboard code is present in Perforce and will be committed to HEAD after it has been unfrozen.

Description	Assigned to	Status
Make terminal emulator UTF-8 aware	Ed	Committed (01/01/2009)
Implement unicode-to-8-bit mapper for existing fonts	Alexander	Implemented (29/01/2009)
UTF-8 aware input layer, keymaps.	Ed	Semi-done
UTF-8 aware font renderer	Ed	TBD
Unicode fonts, not limited to 256 chars	Alexander	TBD, not scheduled

The old terminal renderer used by Syscons is not suitable to use with UTF-8. Right now we use a terminal type called cons25. The disadvantages of cons25:

At least on FreeBSD, we use non-ASCII characters by default. The box drawing is done using characters above 0x7f. This means it is impossible to use UTF-8, because if someone runs a non-UTF-8 application or sets LC_CTYPE to, say, C, applications like dialog(1) would generate illegal output.
cons25 and UTF-8 is not something that exists right now and is proven to work. xterm and UTF-8 is known to Just Work ™.
cons25 also has some other problems, not related to UTF-8. It is not supported by many operating systems and devices and lacks a lot of optimizations (scrolling regions).

This is why we should migrate to xterm at the same time. This adds a little more complexity to the terminal emulator (old VT100 character sets, keypad application mode, etc), but has its advantages.

As of 01/01/2009, the old Syscons emulator has been replaced by libteken. libteken is still configured to be a cons25 emulator, but already supports a lot more escape sequences. This means you should already be able to SSH to your Cisco/HP switch without problems.

If you want to try the new xterm + UTF-8 support, recompile your kernel with TEKEN_XTERM and TEKEN_UTF8 in your kernel configuration file. All non-ASCII characters will still be displayed as a yellow/red question mark. Be sure to replace cons25 with xterm in /etc/ttys!

Right now the input layer only has 8-bit keymaps. This should be changed to Unicode. XXX: Extend.

Because Syscons (and the hardware!) is restricted to 8-bit fontmaps, we should add a Unicode-to-8-bit fontmap conversion. An advantage is that we should be able to display exactly 256 different characters now, instead of 240-250 (not sure). We could even map multiple Unicode characters to the same graphical character (German Eszett and Beta?).

We could consider writing only a very small amount of fonts and some kind of file format, describing which characters should be included in the 8-bit map. This means we can generate a lot of fontmaps. I guess it's a little more tricky to add support for full width characters, because these should consume two graphical characters. Maybe we can save some space by just rendering full-width ASCII characters as regular characters, followed by a space?

This is still very premature. WIP. TBD.

In order to make integration into the operating system as easy as possible, we should at least make UTF-8 support a compile-time switch. Users/developers may still want to migrate their key/fontmaps to the new format; POLA.

To verify that the project has reached its goals and met all requirements performing the following is necessary: