SysconsUnicodeProject [FreeBSD WIKI]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== SysconsUnicodeProject ======

For any reader not familiar with Unicode, reading [[SysconsUnicodeProject/UnicodeBasics|SysconsUnicodeProject/UnicodeBasics]] is suggested.

This project aims to enhance Unicode support on %%FreeBSD%% operating system by implementing support for UTF-8 in the Syscons console driver. This is not about full support for Unicode-based applications (because has no impact on correctness of sorting text strings, for example). Currently system console driver deals with 8-bit character sets only.

===== News =====

  * 23/08/2008: Project initiated by Alexander Churanov. See [[http://lists.freebsd.org/pipermail/freebsd-current/2008-August/087876.html|this]] message to the lists.
  * 01/01/2009: Syscons terminal emulator replaced with new implementation by Ed Schouten, which has optional xterm and UTF-8 support.
  * 29/01/2009: The implementation of unicode maps, that allow using existing fonts in UTF-8 mode was created by Alexander Churanov.
  * 04/08/2009: Unicode support for the keyboard code is present in Perforce and will be committed to HEAD after it has been unfrozen.

===== TODO =====

^ Description ^ Assigned to ^ Status ^
| Make terminal emulator UTF-8 aware | Ed | Committed (01/01/2009) |
| Implement unicode-to-8-bit mapper for existing fonts | Alexander | Implemented (29/01/2009) |
| UTF-8 aware input layer, keymaps. | Ed | Semi-done |
| UTF-8 aware font renderer | Ed | TBD |
| Unicode fonts, not limited to 256 chars | Alexander | TBD, not scheduled |

===== Terminal emulator =====
The old terminal renderer used by Syscons is not suitable to use with UTF-8. Right now we use a terminal type called ''%%cons25%%''. The disadvantages of ''%%cons25%%'':

  * At least on %%FreeBSD%%, we use non-ASCII characters by default. The box drawing is done using characters above ''%%0x7f%%''. This means it is impossible to use UTF-8, because if someone runs a non-UTF-8 application or sets ''%%LC_CTYPE%%'' to, say, ''%%C%%'', applications like ''%%dialog(1)%%'' would generate illegal output.
  * ''%%cons25%%'' and UTF-8 is not something that exists right now and is proven to work. ''%%xterm%%'' and UTF-8 is known to Just Work (tm).
  * ''%%cons25%%'' also has some other problems, not related to UTF-8. It is not supported by many operating systems and devices and lacks a lot of optimizations (scrolling regions).

This is why we should migrate to ''%%xterm%%'' at the same time. This adds a little more complexity to the terminal emulator (old VT100 character sets, keypad application mode, etc), but has its advantages.

As of 01/01/2009, the old Syscons emulator has been replaced by libteken. libteken is still configured to be a ''%%cons25%%'' emulator, but already supports a lot more escape sequences. This means you should already be able to SSH to your Cisco/HP switch without problems.

If you want to try the new ''%%xterm%%'' + UTF-8 support, recompile your kernel with ''%%TEKEN_XTERM%%'' and ''%%TEKEN_UTF8%%'' in your kernel configuration file. All non-ASCII characters will still be displayed as a yellow/red question mark. Be sure to replace ''%%cons25%%'' with ''%%xterm%%'' in ''%%/etc/ttys%%''!

===== Input layer =====

Right now the input layer only has 8-bit keymaps. This should be changed to Unicode. XXX: Extend.

===== Font rendering =====

Because Syscons (and the hardware!) is restricted to 8-bit fontmaps, we should add a Unicode-to-8-bit fontmap conversion. An advantage is that we should be able to display exactly 256 different characters now, instead of 240-250 (not sure). We could even map multiple Unicode characters to the same graphical character (German Eszett and Beta?).

We could consider writing only a very small amount of fonts and some kind of file format, describing which characters should be included in the 8-bit map. This means we can generate a lot of fontmaps. I guess it's a little more tricky to add support for full width characters, because these should consume two graphical characters. Maybe we can save some space by just rendering full-width ASCII characters as regular characters, followed by a space?

This is still very premature. WIP. TBD.

===== Integration =====

In order to make integration into the operating system as easy as possible, we should at least make UTF-8 support a compile-time switch. Users/developers may still want to migrate their key/fontmaps to the new format; POLA.

===== Testing =====

To verify that the project has reached its goals and met all requirements performing the following is necessary:

  * performing all manual tests in /usr/src/tools/test/syscons
  * verifying font correctness (for each added/modified font)
  * verifying keyboard input map correctness (for each added/modified map)
  * verifying font catalog completness
  * verifying keyboard input map catalog completness

----

{{tag>InactiveProject}}