Feature request: UTF-8 on the serial console
============================================

Hi Peter,

a suggestion for MMBasic. My wish is this for the CMM2 with
firmware 8, but it would be a feature for MMBasic in general: the CMM2
and the PicoMite family (PicoMite, PicoCalc, ...) use the same Font 1
(font1.h). So far I have tested everything on the PicoCalc only.

The problem
-----------
Font 1 draws its own characters for bytes 128-255: box drawing,
shading, Greek, maths and symbols. A terminal on the PC cannot show
them the same way:

- Modern terminals (Tera Term 5, PuTTY, Linux) work in Unicode. In
  UTF-8 mode a single byte 128-255 is not valid and is dropped.
- In ISO8859-1 mode bytes 128-159 are C1 control codes. Tera Term 5
  drops them even with Accept8BitCtrl=off, so those two rows stay
  dark whatever font is used.
- Text from outside (for example SQLite data, which is always UTF-8)
  cannot be shown in the terminal correctly next to the box drawing
  of the program.

Suggestion
----------
OPTION SERIAL UTF8 ON | RAW | OFF   (the name is only a suggestion)

It changes the serial console output only. The screen (VGA, HDMI,
LCD) stays exactly as it is.

ON  - each byte the console sends to the serial port is sent as the
      UTF-8 of the character that Font 1 draws for it. One table with
      130 entries (bytes &H60, &H7F, &H80-&HFF; see the attached
      table). No program has to change: the CLI, the editor and all
      existing programs look the same in a UTF-8 terminal as on the
      screen, including bytes 128-159.

RAW - for programs that write UTF-8 themselves. The serial port gets
      the bytes unchanged. On the screen each UTF-8 character takes
      one cell: the Font 1 character if the table has it (so a box
      drawn with Unicode box characters is still a box on the screen),
      otherwise a dot CHR$(250). Screen and terminal keep the same
      columns.

OFF - as today (default).

Input (optional): with ON or RAW a UTF-8 sequence typed in the
terminal could become the Font 1 byte from the table, or "?".

Why two modes: some pairs of Font 1 bytes happen to be valid UTF-8
(for example &HC8 &HBC, two box characters), so the firmware cannot
tell by itself whether a program sends Font 1 bytes or UTF-8.

The table
---------
Rule: each byte maps to the character Font 1 draws. Where Unicode has
no such character (the icons), it uses U+E000 + byte from the Private
Use Area. Examples:

  &HC9 -> U+2554 (box drawing double down and right)
  &HE0 -> U+03B1 (alpha)
  &H60 -> U+00B0 (Font 1 draws a degree sign there)
  &H97 -> U+E097 (light bulb icon, no Unicode equivalent)

Open question
-------------
The table follows one rule: the character Font 1 draws. So &HE1 is
beta (that is what Font 1 shows), and a sharp s from UTF-8 becomes a
dot on the screen in RAW mode. Whether RAW should use look-alikes
(sharp s -> &HE1) is up to you.

Demo
----
utf8demo.bas shows both modes, written in BASIC with OPTION CONSOLE
switching. With Tera Term in UTF-8 mode and the font "MMBasic Font1
Unicode" (a TrueType copy of font1.h with the glyphs on their Unicode
code points) the terminal shows the same as the screen, plus real
UTF-8 text such as "Mueller" with a u-umlaut. Tested on the PicoCalc
only; the CMM2 is not tested yet.

In BASIC this needs two outputs per line and OPTION CONSOLE switching,
which also loses characters when input arrives at the same time. In
the firmware it would be one table lookup in the serial output
routine.

Everything is in the attached utf8-terminal.zip (see README.txt):
  feature-request.txt, utf8demo.bas, font1_unicode_table.txt,
  MMBasic_Font1.ttf, MMBasic_Font1_Unicode.ttf, three screenshots
  (two of Tera Term, one of the PicoCalc LCD).

Matthias
