Bug 50171

This bug is related to TUI user cases, for people who switch from traditional VGA to new KMS frame buffer consoles. It is really annoying for them. Please, read carefully.


PREAMBLE.

There are so called "margins" of framebuffer consoles. It is an unused areas on bottom, right or both margins of the screen.

On VGA console, people normally use 8x16 fonts (9x16), and with console size of 640x400 (720x400) all the 25 rows and 80 columns fit into the console dimensions completely. So, there are no any margins here.

After the KMS modeset had appeared, people start to use frame buffer consoles. Normally, it utilizes the maximum dimensions provided by hardware (ie. 1024x768, 1920x1080 etc.). Since default 8x16 font looks too small on such consoles, people choose bigger fonts (for example -- from terminus-console-fonts project). And often enough "cols * font_x" and "rows * font_y" do not fit the hardware sizes.

Consider 1024x768 console with 12x24 font -- it is 32x85 char console with 4 bits "margin" at the right (1024bit - (12bit * 85col))
Consider 1920x1080 console with 16x32 font -- it is 33x120 char console with 24 bits "margin" at the bottom (1080bit - (32bit * 33row))


ISSUE.

When consoles are switched (by Alt-Fx), the margins (if any) of the new active console are cleared. To clear margins, fbcon driver uses color from "vc->vc_video_erase_char" character, whereas the correct way seems to use "vc->vc_def_color" .

The "vc->vc_video_erase_char" might be changed during console session (by an application, see below). OTOH the margin colors must be the same during the session.


TEST CASE.

1.
A "normal" machine, with any modern Intel/Nvidia/Radeon, booted with modeset enabled, and having 6 usual consoles (at least 2 for test case).

Choose font/screen combination which provides console margins. Or just decrease the number of rows and/or columns by stty command (see the curent sizes by "stty size", then decrease either "stty cols N" or "stty rows N" or both).

Ncurses package with "tput" binary should be installed.

2.
Let's choose the first console (Alt-F1). Optionally clear it by "tput clear".

Run "tput setab 4", to switch to blue background of characters.

Switch to another console (Alf-F2), then switch back (Alt-F1).

You see the "blue" margins. There is no way to clear it staying on the same console (neither "tput clear", nor "reset").

The only way to clear this "blue" margins is "tput setab 0" (or "reset") and then "switch to another console and come back".

3.
The problem is when you switch to a console, "vt" driver calls "con_switch" method:
http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=blob;f=drivers/tty/vt/vt.c;h=f87d7e8964bf0849c1ab3e3da327e3a14c9b35a2;hb=HEAD#l689

which in a case of framebuffer console is drivers/video/console/fbcon.c:fbcon_switch()
http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=blob;f=drivers/video/console/fbcon.c;h=fdefa8fd72c4da070787d95260b2744d6a9a7dab;hb=HEAD#l2145

which calls fbcon_clear_margins(), which actually calls drivers/video/console/bitblit.c:bit_clear_margins()
http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=blob;f=drivers/video/console/bitblit.c;h=61b182bf32a22962492cc79a3b1e60f3f49ef388;hb=HEAD#l205

which uses attr_bgcol_ec() macro for color, which according to fbcon.h uses vc->vc_video_erase_char to obtain such a color:
http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=blob;f=drivers/video/console/fbcon.h;h=6bd2e0c7f209d402cbbf75bb3b088a7c58f3ed24;hb=HEAD#l150

But after the "tput setab 4", we have "blue" vc->vc_video_erase_char ...

The only way to clear margins properly is to clear background color from vc_video_erase_char ("tput setab 0", or just "reset"), and then perform switch to another console and back (this switch causes call to fbcon_clear_margins() again).


ACTUAL USER CASES.

There are two wide-spread libraries for terminal user interface (TUI) on Linux -- "ncurses" and "slang".

A lot of TUI applications linked with "slang" library (ntsysv, mc, whiptail etc.). Slang library has a feature to minimize terminal output. This way it leaves background color attributes unchanged when waiting for user input (because it is probably enough that the same attributes might be needed further). If, instead of an application input, the user switches consoles, it catches the case.

Unlike "slang", it seems that "ncurses" library always drop color attributes when waiting for user input. I still cannot reproduce the issue with any ncurses-linked application.

You can repeat the test case above, running
"whiptail --yesno test 0 0" (whiptail linked with slang)
and
"dialog -- yesno test 0 0" (dialog linked with ncurses).
After invoking, move between "yes and no" some times (this causes slang to change color and leave it for a while), then perform console switches. The similar results as for 'tput setab" is for "whiptail" case, and no issues with "dialog" case.

Please note, that it is not a slang issue. Besides the slang, there maybe another TUI applications who behave similar.


SOLUTION.

An easiest kludge is to wraparound vc->vc_sw->con_switch() call by something like:

orig_erase_char = vc->vc_video_erase_char;
vc->vc_video_erase_char = 0;   // useful for black background consoles only
update = vc->vc_sw->con_switch(vc);
vc->vc_video_erase_char = orig_erase_char;

but more intelligent way seems to utilize vc->vc_def_color in clear_margins methods.


PS.

Please, fix an issue. There are still enough users of TUI applications. It is real annoying them.