Differences

This shows you the differences between two versions of the page.

--- sysconsunicodeproject [2009/01/20 18:00] – AlexanderChuranov
+++ sysconsunicodeproject [2022/09/19 01:34] (current) – Remove ACL: not devs only for a long time, all:read is already default KubilayKocak
@@ Line 1: / Line 1: @@
 ====== SysconsUnicodeProject ======
-For any reader not familiar with unicode, reading [[SysconsUnicodeProject/UnicodeBasics|SysconsUnicodeProject/UnicodeBasics]] is suggested.
+For any reader not familiar with Unicode, reading [[SysconsUnicodeProject/UnicodeBasics|SysconsUnicodeProject/UnicodeBasics]] is suggested.
-This project aims to enhance unicode support on %%FreeBSD%% operating system by implementing support for UTF-8 in console driver. This is not about full support for unicode-based applications (because has no impact on correctness of sorting text strings, for example), but just the necessary part. Currently system console driver deals with 8-bit characters only.
+This project aims to enhance Unicode support on %%FreeBSD%% operating system by implementing support for UTF-8 in the Syscons console driver. This is not about full support for Unicode-based applications (because has no impact on correctness of sorting text strings, for example). Currently system console driver deals with 8-bit character sets only.
-The project started on August 23, 2008 and is not finished yet.
+===== News =====
-Many project decisions originate from [[http://lists.freebsd.org/pipermail/freebsd-current/2008-August/087876.html|this]] discussion in mailing list. Any of the mailing list threads with the topic "Unicode-based %%FreeBSD%%" is appropriate for discussing the project.
+  * 23/08/2008: Project initiated by Alexander Churanov. See [[http://lists.freebsd.org/pipermail/freebsd-current/2008-August/087876.html|this]] message to the lists.
+  * 01/01/2009: Syscons terminal emulator replaced with new implementation by Ed Schouten, which has optional xterm and UTF-8 support.
+  * 29/01/2009: The implementation of unicode maps, that allow using existing fonts in UTF-8 mode was created by Alexander Churanov.
+  * 04/08/2009: Unicode support for the keyboard code is present in Perforce and will be committed to HEAD after it has been unfrozen.
-\\
+===== TODO =====
-===== Goals =====
-  * Provide UTF-8 input-output mode for syscons driver.
+^ Description ^ Assigned to ^ Status ^
+| Make terminal emulator UTF-8 aware | Ed | Committed (01/01/2009) |
+| Implement unicode-to-8-bit mapper for existing fonts | Alexander | Implemented (29/01/2009) |
+| UTF-8 aware input layer, keymaps. | Ed | Semi-done |
+| UTF-8 aware font renderer | Ed | TBD |
+| Unicode fonts, not limited to 256 chars | Alexander | TBD, not scheduled |
-\\
+===== Terminal emulator =====
-===== Requirements =====
+The old terminal renderer used by Syscons is not suitable to use with UTF-8. Right now we use a terminal type called ''%%cons25%%''. The disadvantages of ''%%cons25%%'':
-  * UTF-8 i/o mode should be optional.
+  * At least on %%FreeBSD%%, we use non-ASCII characters by default. The box drawing is done using characters above ''%%0x7f%%''. This means it is impossible to use UTF-8, because if someone runs a non-UTF-8 application or sets ''%%LC_CTYPE%%'' to, say, ''%%C%%'', applications like ''%%dialog(1)%%'' would generate illegal output.
-  * Full range of unicode characters according to Unicode 5.0 must be supported.
+  * ''%%cons25%%'' and UTF-8 is not something that exists right now and is proven to work. ''%%xterm%%'' and UTF-8 is known to Just Work (tm).
-  * Driver should respect character width.
+  * ''%%cons25%%'' also has some other problems, not related to UTF-8. It is not supported by many operating systems and devices and lacks a lot of optimizations (scrolling regions).
-  * Console driver should be correct. New code should be robust and behave consistently.
-Details:
+This is why we should migrate to ''%%xterm%%'' at the same time. This adds a little more complexity to the terminal emulator (old VT100 character sets, keypad application mode, etc), but has its advantages.
-Ability to turn UTF-8 i/o mode on and off is necessary, because at the time of this writing other  components of operation system are not unicode-compliant in full. And it's not clear whether syscons is a bottleneck in moving to full UTF-8 support or not. Compile-time switch like kernel build option is appropriate.
+As of 01/01/2009, the old Syscons emulator has been replaced by libteken. libteken is still configured to be a ''%%cons25%%'' emulator, but already supports a lot more escape sequences. This means you should already be able to SSH to your Cisco/HP switch without problems.
-Support for the full range of unicode code points is necessary, because Unicode 5.0 specification was published 3 years ago and not complying with it means being too outdated. It has minor impact on implementation cost because correct unicode implementation is not harder than incorrect.
+If you want to try the new ''%%xterm%%'' + UTF-8 support, recompile your kernel with ''%%TEKEN_XTERM%%'' and ''%%TEKEN_UTF8%%'' in your kernel configuration file. All non-ASCII characters will still be displayed as a yellow/red question mark. Be sure to replace ''%%cons25%%'' with ''%%xterm%%'' in ''%%/etc/ttys%%''!
-Console Driver should respect at least normal and double character width. It's allowed to insert spaces for making character of normal width look like a double-width char.
+===== Input layer =====
-\\
+Right now the input layer only has 8-bit keymaps. This should be changed to Unicode. XXX: Extend.
-===== Implementation Decisions =====
-  * Map decoded unicode code point numbers to 8-bit indices of glyphs in font.
+===== Font rendering =====
-  * Create unit-tests for new code. Create manual integration tests for cases where writing unit-tests is currently impossible.
-Rationale:
+Because Syscons (and the hardware!) is restricted to 8-bit fontmaps, we should add a Unicode-to-8-bit fontmap conversion. An advantage is that we should be able to display exactly 256 different characters now, instead of 240-250 (not sure). We could even map multiple Unicode characters to the same graphical character (German Eszett and Beta?).
-Mapping to 8-bit indices will utilize most of current fonts. Creating fonts would dramatically increase implementation cost. Modifying the renderer to support more than 256 characters at time can be handled as a separate task. The suggested approach is to enhance font file format to contain a separate "unicode code point number to 8-bit glyph index" table inside the font file. Adding new information to existing font files is a doable task.
+We could consider writing only a very small amount of fonts and some kind of file format, describing which characters should be included in the 8-bit map. This means we can generate a lot of fontmaps. I guess it's a little more tricky to add support for full width characters, because these should consume two graphical characters. Maybe we can save some space by just rendering full-width ASCII characters as regular characters, followed by a space?
-\\
+This is still very premature. WIP. TBD.
-===== Implementation Plan =====
-=== Completed items ===
+===== Integration =====
-  * Create or find text files in UTF-8 for testing.
-  * Implement UTF-8 decoder.
-  * Implement UTF-8 encoder.
-  * Provide compile-time switch for enabling and disabling UTF-8 mode.
-  * Connect UTF-8 decoder with syscons (process application output to screen).
-=== Items in-progress ===
+In order to make integration into the operating system as easy as possible, we should at least make UTF-8 support a compile-time switch. Users/developers may still want to migrate their key/fontmaps to the new format; POLA.
-  * Design new font file format.
-  * Implement unicode-to-8-bit-glyph-index conversion in syscons.
-  * Review existing font files, find files that contains the same glyphs.
-  * Create new fonts/glyphs lists.
-  * Merge/modify font files if necessary.
-  * Add conversion table to all font files.
-  * Connect UTF-8 encoder with syscons (process application input - keystrokes).
-  * Implement keyboard-to-unicode conversion map support.
-  * Implement interface enabling user-mode applications to load and select keyboard-to-unicode conversion map.
-  * Enhance user-mode applications to utilize provided API for loading/selecting keyboard maps.
-\\
 ===== Testing =====
-To verify that the project has reached its goals and met all requirements performing the following tests is necessary:
+To verify that the project has reached its goals and met all requirements performing the following is necessary:
-  * [[SysconsUnicodeProject/DecodingUTF8SequencesOfVariousLengthTest|decoding UTF-8 sequences of various length]]
+  * performing all manual tests in /usr/src/tools/test/syscons
-  * decoding incorrect sequences
-  * rendering characters of various width
-  * [[SysconsUnicodeProject/ReplacingCharactersThatAreMissingInCurrentFontTest|replacing characters that are missing in current font]]
-  * [[SysconsUnicodeProject/SwitchingAFontTest|switching a font]]
-  * switching a keyboard input map
   * verifying font correctness (for each added/modified font)
   * verifying keyboard input map correctness (for each added/modified map)
@@ Line 79: / Line 60: @@
   * verifying keyboard input map catalog completness
-\\
+----
-===== Project Contributors =====
-[[AlexanderChuranov|Alexander Churanov]]
+{{tag>InactiveProject}}