SysconsUnicodeProject

This is an old revision of the document!

For any reader not familiar with unicode, reading SysconsUnicodeProject/UnicodeBasics is suggested.

This project aims to enhance unicode support on FreeBSD operating system by implementing support for UTF-8 in console driver. This is not about full support for unicode-based applications (because has no impact on correctness of sorting text strings, for example), but just the necessary part. Currently system console driver deals with 8-bit characters only.

The project started on August 23, 2008 and is not finished yet.

Many project decisions originate from this discussion in mailing list. Any of the mailing list threads with the topic “Unicode-based FreeBSD” is appropriate for discussing the project.

Provide UTF-8 input-output mode for syscons driver.

UTF-8 i/o mode should be optional.
Full range of unicode characters according to Unicode 5.0 must be supported.
Driver should respect character width.
Console driver should be correct. New code should be robust and behave consistently.

Details:

Ability to turn UTF-8 i/o mode on and off is necessary, because at the time of this writing other components of operation system are not unicode-compliant in full. And it's not clear whether syscons is a bottleneck in moving to full UTF-8 support or not. Compile-time switch like kernel build option is appropriate.

Support for the full range of unicode code points is necessary, because Unicode 5.0 specification was published 3 years ago and not complying with it means being too outdated. It has minor impact on implementation cost because correct unicode implementation is not harder than incorrect.

Console Driver should respect at least normal and double character width. It's allowed to insert spaces for making character of normal width look like a double-width char.

Transcode application output to selected 8-bit charset when producing output to screen.
Create unit-tests for new code. Create manual integration tests for cases where writing unit-tests is currently impossible.

Rationale:

Transcoding to 8-bit will utilize most of current screenmaps, keymaps and fonts. Creating new maps and fonts would dramatically increase implementation cost. Modifying the renderer to support more than 256 characters at time can be handled as a separate task.

Completed items

Implement manual integration test application for detecting whether syscons supports UTF-8.
Implement UTF-8 decoder.
Implement UTF-8 encoder.
Enhance kernel data structures and code logic for turning UTF-8 mode on and off on demand.
Connect UTF-8 decoder with syscons (process application output to screen).
Implement unicode-to-8-bit conversion map support.

Items in-progress

Provide interface enabling user-mode applications to turn UTF-8 on or off.
Connect UTF-8 encoder with syscons (process application input - keystrokes).
Implement keyboard-to-unicode conversion map support.
Implement interface enabling user-mode applications to load and select keyboard-to-unicode conversion map.
Implement interface enabling user-mode applications to load and select unicode-to-8-bit conversion map.
Enhance user-mode applications to utilize provided APIs for turning UTF-8 on and off and loading/selecting IO maps.