Differences

This shows you the differences between two versions of the page.

--- sysconsunicodeproject [2009/01/20 15:41] – AlexanderChuranov
+++ sysconsunicodeproject [2022/09/19 01:34] (current) – Remove ACL: not devs only for a long time, all:read is already default KubilayKocak
@@ Line 1: / Line 1: @@
 ====== SysconsUnicodeProject ======
-For any reader not familiar with unicode, reading [[SysconsUnicodeProject/UnicodeBasics|SysconsUnicodeProject/UnicodeBasics]] is suggested.
+For any reader not familiar with Unicode, reading [[SysconsUnicodeProject/UnicodeBasics|SysconsUnicodeProject/UnicodeBasics]] is suggested.
-This project aims to enhance unicode support on %%FreeBSD%% operating system by implementing support for UTF-8 in console driver. This is not about full support for unicode-based applications (because has no impact on correctness of sorting text strings, for example), but just the necessary part. Currently system console driver deals with 8-bit characters only.
+This project aims to enhance Unicode support on %%FreeBSD%% operating system by implementing support for UTF-8 in the Syscons console driver. This is not about full support for Unicode-based applications (because has no impact on correctness of sorting text strings, for example). Currently system console driver deals with 8-bit character sets only.
-The project started on August 23, 2008 and is not finished yet.
+===== News =====
-Many project decisions originate from [[http://lists.freebsd.org/pipermail/freebsd-current/2008-August/087876.html|this]] discussion in mailing list. Any of the mailing list threads with the topic "Unicode-based %%FreeBSD%%" is appropriate for discussing the project.
+  * 23/08/2008: Project initiated by Alexander Churanov. See [[http://lists.freebsd.org/pipermail/freebsd-current/2008-August/087876.html|this]] message to the lists.
+  * 01/01/2009: Syscons terminal emulator replaced with new implementation by Ed Schouten, which has optional xterm and UTF-8 support.
+  * 29/01/2009: The implementation of unicode maps, that allow using existing fonts in UTF-8 mode was created by Alexander Churanov.
+  * 04/08/2009: Unicode support for the keyboard code is present in Perforce and will be committed to HEAD after it has been unfrozen.
-\\
+===== TODO =====
-===== Goals =====
-  * Provide UTF-8 input-output mode for syscons driver.
+^ Description ^ Assigned to ^ Status ^
+| Make terminal emulator UTF-8 aware | Ed | Committed (01/01/2009) |
+| Implement unicode-to-8-bit mapper for existing fonts | Alexander | Implemented (29/01/2009) |
+| UTF-8 aware input layer, keymaps. | Ed | Semi-done |
+| UTF-8 aware font renderer | Ed | TBD |
+| Unicode fonts, not limited to 256 chars | Alexander | TBD, not scheduled |
-\\
+===== Terminal emulator =====
-===== Requirements =====
+The old terminal renderer used by Syscons is not suitable to use with UTF-8. Right now we use a terminal type called ''%%cons25%%''. The disadvantages of ''%%cons25%%'':
-  * UTF-8 i/o mode should be optional.
+  * At least on %%FreeBSD%%, we use non-ASCII characters by default. The box drawing is done using characters above ''%%0x7f%%''. This means it is impossible to use UTF-8, because if someone runs a non-UTF-8 application or sets ''%%LC_CTYPE%%'' to, say, ''%%C%%'', applications like ''%%dialog(1)%%'' would generate illegal output.
-  * Full range of unicode characters according to Unicode 5.0 must be supported.
+  * ''%%cons25%%'' and UTF-8 is not something that exists right now and is proven to work. ''%%xterm%%'' and UTF-8 is known to Just Work (tm).
-  * Driver should respect character width.
+  * ''%%cons25%%'' also has some other problems, not related to UTF-8. It is not supported by many operating systems and devices and lacks a lot of optimizations (scrolling regions).
-  * Console driver should be correct. New code should be robust and behave consistently.
-Details:
+This is why we should migrate to ''%%xterm%%'' at the same time. This adds a little more complexity to the terminal emulator (old VT100 character sets, keypad application mode, etc), but has its advantages.
-Ability to turn UTF-8 i/o mode on and off is necessary, because at the time of this writing other  components of operation system are not unicode-compliant in full. And it's not clear whether syscons is a bottleneck in moving to full UTF-8 support or not. Compile-time switch like kernel build option is appropriate.
+As of 01/01/2009, the old Syscons emulator has been replaced by libteken. libteken is still configured to be a ''%%cons25%%'' emulator, but already supports a lot more escape sequences. This means you should already be able to SSH to your Cisco/HP switch without problems.
-Support for the full range of unicode code points is necessary, because Unicode 5.0 specification was published 3 years ago and not complying with it means being too outdated. It has minor impact on implementation cost because correct unicode implementation is not harder than incorrect.
+If you want to try the new ''%%xterm%%'' + UTF-8 support, recompile your kernel with ''%%TEKEN_XTERM%%'' and ''%%TEKEN_UTF8%%'' in your kernel configuration file. All non-ASCII characters will still be displayed as a yellow/red question mark. Be sure to replace ''%%cons25%%'' with ''%%xterm%%'' in ''%%/etc/ttys%%''!
-Console Driver should respect at least normal and double character width. It's allowed to insert spaces for making character of normal width look like a double-width char.
+===== Input layer =====
-\\
+Right now the input layer only has 8-bit keymaps. This should be changed to Unicode. XXX: Extend.
-===== Implementation Decisions =====
-  * Copy UTF-8 encoder and decoder from IBM ICU libray.
+===== Font rendering =====
-  * Transcode application output to selected 8-bit charset when producing output to screen.
-  * Create unit-tests for new code. Create manual integration tests for cases where writing unit-tests is currently impossible.
-Rationale:
+Because Syscons (and the hardware!) is restricted to 8-bit fontmaps, we should add a Unicode-to-8-bit fontmap conversion. An advantage is that we should be able to display exactly 256 different characters now, instead of 240-250 (not sure). We could even map multiple Unicode characters to the same graphical character (German Eszett and Beta?).
-Using IBM ICU is permitted by license. The code is already tested for correctness. It's expected that code is highly compliant with Standard version 5.0. This will reduce implementation cost.
+We could consider writing only a very small amount of fonts and some kind of file format, describing which characters should be included in the 8-bit map. This means we can generate a lot of fontmaps. I guess it's a little more tricky to add support for full width characters, because these should consume two graphical characters. Maybe we can save some space by just rendering full-width ASCII characters as regular characters, followed by a space?
-Transcoding to 8-bit will utilize most of current screenmaps, keymaps and fonts. Creating new maps and fonts would dramatically increase implementation cost. Modifying the renderer to support more than 256 characters at time can be handled as a separate task.
+This is still very premature. WIP. TBD.
-\\
+===== Integration =====
-===== Implementation Plan =====
-=== Completed items ===
+In order to make integration into the operating system as easy as possible, we should at least make UTF-8 support a compile-time switch. Users/developers may still want to migrate their key/fontmaps to the new format; POLA.
-  * Implement manual integration test application for detecting whether syscons supports UTF-8.
-  * Implement UTF-8 decoder.
-  * Implement UTF-8 encoder.
-  * Enhance kernel data structures and code logic for turning UTF-8 mode on and off on demand.
-  * Connect UTF-8 decoder with syscons (process application output to screen).
-  * Implement unicode-to-8-bit conversion map support.
-=== Items in-progress ===
+===== Testing =====
-  * Provide interface enabling user-mode applications to turn UTF-8 on or off.
+To verify that the project has reached its goals and met all requirements performing the following is necessary:
-  * Connect UTF-8 encoder with syscons (process application input - keystrokes).
-  * Implement keyboard-to-unicode conversion map support.
-  * Implement interface enabling user-mode applications to load and select keyboard-to-unicode conversion map.
-  * Implement interface enabling user-mode applications to load and select unicode-to-8-bit conversion map.
-  * Enhance user-mode applications to utilize provided APIs for turning UTF-8 on and off and loading/selecting IO maps.
-\\
+  * performing all manual tests in /usr/src/tools/test/syscons
-===== Project Contributors =====
+  * verifying font correctness (for each added/modified font)
+  * verifying keyboard input map correctness (for each added/modified map)
+  * verifying font catalog completness
+  * verifying keyboard input map catalog completness
-[[AlexanderChuranov|Alexander Churanov]]
+----
+{{tag>InactiveProject}}