Why is this a kernel problem?  Is it not simply a case of userspace
setting the display to some mode which it shouldn't be setting?

Seems to me that because it's so common that framebuffer is the only place the
root of the problem could lie. What in userspace could be responsible?

When a 132 column non-25 line mode is selected from the list following vga=ask
on the kernel line the same behavior results as selecting a non-25 line mode
directly.

This is still in text mode correct, because i810 and i815 does not support VESA
framebuffer.

I received similar reports before, and the cause is typically kbd.

Try doing stty -a and if rows and cols agree with your display, then it is a
userspace problem.

I just tried using 2.6.21rc4 (SUSE factory installation kernel) with ask/scan on
mga (G400), and am offered only one non-80 column mode, 030A, 132x43. 030A
initializes correctly, but before init finishes the tty shifts to about 25 rows,
which is the same behavior as 2.6.18-34 (SUSE 10.2 release kernel). On 2.6.18-34
stty -a reports 132 columns, which is correct; and 43 rows, which is what it did
early in init, but not what it is doing post-init, which is 21 rows.

I tried ask/scan also on 2.6.18-3 (Etch) on i815, selecting 0121 (132x43). The
behavior is partially the same, in that early in init there are 43 rows which
later become 21 rows. The difference is that all 21 lines are using only the top
half of the screen, and only the top half of each line displays. Even so, I can
see stty -a reports the row count at 21.

I tried ask/scan also on 2.6.18-1.2798 (FC6) on ET6100, selecting 0324 (132x28).
The behavior is essentially the same as mga, in that early in init there are 28
rows which later become 24 rows. stty -a reports 24 rows, unlike the 28 rows it
initialized at and should be continuing. With ET6100 I also tried selecting 0322
(132x44). It initialized at 44 rows and switched to 24 rows. stty -a reports 24
rows.

On Ubuntu 2.6.15-27 with nv (MX400) I selected 0154 (132x43). It initialized 43
rows on a screen of approximately 80% height, then switched to 24 rows. stty -a
reports 24 rows.

So only on mga is stty -a out of sync with the actual display, but in every case
here the display fails to remain in the selected, initialized, and
hardware-supported non-25 line mode.

I found an ATI 3D Rage PCI to try on 2.6.13-15 (SUSE 10.0), selecting 0133
(132x44). It initialized to 44 rows on an ~80% height screen, then switched to
22 visible rows on ~80% height. stty -a reports 44 rows.

Try disabling font setting (ie, in Suse, you can chmod -x /etc/rc.d/kbd). I
don't know about other distributions, but you can grep for setfont in /etc/rc.d.
Let me know how it goes. 

On mga at least, chmod -x /etc/rc.d/kbd solves the problem. I'll check other
hardware and distros other than SUSE as time permits.

> On mga at least, chmod -x /etc/rc.d/kbd solves the problem. I'll check other
> hardware and distros other than SUSE as time permits.

Ok, thanks for testing. It looks to be related to setting a new font with
dimensions different from the original. I'll see what I can do with this
problem, though I don't have graphics hardware that supports non-80 column  modes.

Created attachment 11011 [details]
Inform vgacon whether resize request comes from kernel or userspace

Can you try this patch, and let me know if it works.  It is against
2.6.21-rc5-mm3, but it should apply on a reasonably recent kernel.

Etch, Mandriva Cooker and FC6 have no /etc/rc.d/kbd or /etc/init.d/kbd and I
couldn't manage to find their equivalent to test the workaround. chmod -x on
/etc/init.d/S48console-screen failed on Etch. Nothing on FC6 or Cooker in
/etc/init.d or /etc/rc.d contains setfont.

I just checked the old distros I have that still boot. Corel 1.2, Mandrake 7.1 &
RedHat 6.2 with 2.2 kernels do not have the problem. My oldest SUSE 8.1 has 2.4
and it has this problem.

Hopefully tomorrow I can try the patch in SUSE Factory's 2.6.21rcwhatever kernel.

> Etch, Mandriva Cooker and FC6 have no /etc/rc.d/kbd or /etc/init.d/kbd and I
> couldn't manage to find their equivalent to test the workaround. chmod -x on
> /etc/init.d/S48console-screen failed on Etch. Nothing on FC6 or Cooker in
> /etc/init.d or /etc/rc.d contains setfont.

Try 

cd /etc
grep -r setfont

Then just try to comment out any lines with setfont in it.


> I just checked the old distros I have that still boot. Corel 1.2, Mandrake 7.1
> & RedHat 6.2 with 2.2 kernels do not have the problem. My oldest SUSE 8.1 has
> 2.4 and it has this problem.

The older distributions are probably not resetting the console fonts (Even the
latest version of slackware leaves the console fonts alone).

> Hopefully tomorrow I can try the patch in SUSE Factory's 2.6.21rcwhatever
> kernel.

Yes, please, and let me know of the results.

I succeeded in building the 2.6.21-rc5-git6-2 Factory kernel with the comment 10
patch applied on the box using mga. It didn't change the behavior.

As an aside, su -s on the installed kernel modules directory is 60804; on the
dir for the kernel I built, 413432, nearly 7 times as much. The kernel I built
is about 7% smaller, but the initrd is more than double in size. When I first
tried to run make modules_install the root partition filled up. I had to make a
new partition to mount on /usr/src and rebuild to solve the space problem. As a
consequence of the space problem, before rebuilding I reran make menuconfig and
disabled a lot of modules for experimental and laptops and filesystems I don't
use in attempt to reduce space consumption. After adding the new partition for
/usr/src, / had 50% free before make modules_install, only 41% free after. How
could all that disk space consumption have happened?

Sorry for not coming back to you sooner.

If the patch does not work for you, I don't think this problem can be resolved
easily.  If you are going to use a non-80x25 VGA mode, then userspace just have
to be careful not set a new font.

You can do this by searching for 'setfont' in your init scripts and then
commenting them all out.  Sorry, that's the best I can do.

Will close this bug, but you can reopen it if you think this really needs to be
resolved.

How easy would be a fix is irrelevant to whether it should be fixed. It used to
work, so it still should work. The only valid reason to reject it as invalid is
if it is in fact invalid as a pure userspace problem. I'm not yet convinced of
where the problem actually lies, and not competent without help to make that
determination. Since this applies across multiple distros and graphics adapters,
it is clearly a problem which should be worth a fix, and worth the effort to
establish either a kernel fix or clear evidence that the kernel is not the best
or even an appropriate place for a fix. 

This bug is a consequence of the VGA console getting the capability to adjust
the window size. Before this capability was added, userspace is assuming a lot
of things that are technically incorrect (primarily, the utility svgatextmode).
When this capability was added, this assumptions were exposed and exhibited as
new bugs.

Anyway, here's one more thing you can try.  Open

drivers/video/console/vgacon.c and look for this function:

static int vgacon_resize(struct vc_data *c, unsigned int width,
				unsigned int height)
{
	if (width % 2 || width > ORIG_VIDEO_COLS ||
	    height > (ORIG_VIDEO_LINES * vga_default_font_height)/
	    c->vc_font.height)
		/* let svgatextmode tinker with video timings */
		return 0;

	if (CON_IS_VISIBLE(c) && !vga_is_gfx) /* who knows */
		vgacon_doresize(c, width, height);
	return 0;
}

Then change the first 'return 0' statement to -EINVAL;

I tried the comment 16 suggestion on mga and from vga=ask selected 30A. When
init ran setfont the video went nuts, constantly redrawing at several of various
odd horizontal positions instead of from the left edge of screen.

Samuel,

Can you help us with this problem?  It looks to be related to resizing of vgacon.

Tony

I can't reproduce the problem with my video board that only has 80x modes.  I'll see if I can find a board that provides other modes. 

I can't reproduce the problem with the debian 2.6.20 kernel and a 132x44 video mode on an ATI video board. What font are you loading? (I've tried lat9-08/10/12/14/16 as well as 512-chars lat9wbrl-08/10/12/14/16 without any failure) 

> What font are you loading?

I don't know how to find the answer to this.

It's in some distribution-dependant file in /etc. If your distribution uses console-tools, maybe it is in /etc/console-tools/config. If your distribution uses kbd, maybe it is in /etc/kbd/config. As a last resort, maybe you can just  grep -r CONSOLE_FONT /etc 

I have a similar problem with OpenSuse-10.2. These look to be the steps, and
opensuse's init scripts are partially to blame:

1. Initial mode is 80x50
2. Load an 8x16 font /* screen is not 80x25 */
3. Init scripts does an stty cols 80 rows 50
4. This silently succeeds /* vgacon always return success if resulting size is
physically larger - to accomodate svgatextmode */
5. Now, I have an 80x50 screen, but only the upper 25 lines are visible.

In this particular case, the attached patch (Inform vgacon whether resize
request comes from kernel or userspace) fixed the problem.  However, Felix's
case seems to be different, and I can't duplicate that also.

> 2. Load an 8x16 font /* screen is not 80x25 */

This should read:

2. Load an 8x16 font /* screen is _now_ 80x25 */
                                   

> 3. Init scripts does an stty cols 80 rows 50

Mmm, why is it doing that?

>> 3. Init scripts does an stty cols 80 rows 50
>
> Mmm, why is it doing that?

I don't know, I haven't fully examined opensuse's init scripts. I discovered it
when I instrumented the VT_RESIZE, VT_RESIZEX, and TIOCSWINSZ ioctls. It
probably wanted to restore the original screen size after setting the font,
which is completely unnecessary.

> >> 3. Init scripts does an stty cols 80 rows 50
> >
> > Mmm, why is it doing that?
> 
> I don't know, I haven't fully examined opensuse's init scripts.

Ok. I think init scripts are culprit.

If I do the following on my box:

- Initial mode 80x50
- Load 8x16 font
- run stty cols 80 rows 50

I indeed get a 25-line display but applications use 50 lines so that half of
them are hidden.

But that's on purpose: programs (e.g. svgatextmode) that run stty cols/rows are
assumed by the kernel to know what they are doing.  If a suse init script thinks
that it is possible to get 80x50 with a 8x16 font, then it doesn't know what it
is doing :)

> If I do the following on my box:
> 
> - Initial mode 80x50
> - Load 8x16 font
> - run stty cols 80 rows 50
> 
> I indeed get a 25-line display but applications use 50 lines so that half of
> them are hidden.

Yes.  And the patch I posted here at least kinda solves this problem.  What the
patch does is that if the request came from a VT_RESIZE/VT_RESIZEX ioctl, then
vgacon_resize() will return success (svgatextmode uses these ioctls). If the
request comes from a TIOCSWINSZ ioctl (which is what stty uses) or an internal
kernel call, then vgacon_resize() does the right thing, which is to return
-EINVAL if the new physical dimensions exceed the original one.

But according to Felix, the patch does not work for him, which leaves me at a
loss as to what is the cause of his problem.

Ah, ok, I didn't remember that svgatextmode and stty were using different
ioctls.  Well, to be frank your patch is a bit ugly :) Maybe I'd rather add a
flag parameter to the resize method, for letting drivers know whether it's a
VT_RESIZE or TIOCSWINSZ call.

Now, why it does not work for Felix, I don't know.

> Ah, ok, I didn't remember that svgatextmode and stty were using different
> ioctls.

I have to look at the source code of svgatextmode for that.

> Well, to be frank your patch is a bit ugly :)

Ouch, you hurt me :-) No, it was a test patch and was never meant for inclusion.

> Maybe I'd rather add a
> flag parameter to the resize method, for letting drivers know whether it's a
> VT_RESIZE or TIOCSWINSZ call.

That can be done, but it will require a lot more work as this is an interface
changed.

> 
> Now, why it does not work for Felix, I don't know.

I'm stumped too.

As soon as I can I'll try other video cards with that patched kernel. Maybe
Matrox has another problem too.

The computer I've been compiling on to test this with is broken, so it may be
some time longer before I get back to this.

Got system back up last week, but forgot about this before updating to current factory. NAICT, the custom kernel I built according to comment 16 is still working, though the timestamp on it and the initrd seem to be a few hours newer than comment 17. My comment 17 no longer holds with that kernel, or with the unmodified Factory 2.6.22-rc4-git6-2 kernel. Now with mga mode 30A, nv (modes tested 165, 30B) and tseng (with all 5 native 100 & 132 column modes), just work - until I start mc, but that's probably a separate problem.

Does testing in #33 sufficient to consider the problem to be resolved, and the bug can be closed?
Thanks.

I should see what happens in F7 or Rawhide, Cooker, and maybe others first. So far I've been unable to get F7 to install anywhere I've tried.

Anything new? Should I create a definitive patch for this problem?

I'd say yes. 

Okay. I will submit a patch for the -mm tree soon. I will CC both of you.

Can this bug be closed now, was the patch submitted?

To my knowledge, the patch was applied (or a least I can see it in 2.6.24)

What kernel version did the fix go into? Without disabling mode setting in the init scripts, 0x121 is not improved with 2.6.24.3-desktop-4mnb in Mandriva Cooker, nor 0x122 or 0x123 with 2.6.24.1-6-default in openSUSE Factory 11.0 on a Dell GX110 (i810e), nor 0x30a with 2.6.25-rc7-git2-11-default in Factory using mga. chmod -x /etc/rc.d/kbd gets the right rows and columns in Factory with mga & 0x30a, but only uses about 3/4 of the screen height. With chmod -x /etc/rc.d/kbd in Factory 0x122-0x123 with i810e size OK, but with 0x121 less than 90% of the screen height is used. Is distro/userspace follow-up required for it to work right by default in all native modes?

As I said, the fix is there at least since 2.6.24: I tried  - initialize with 80x50 - load 8x16 font - stty rows 50  It now correctly fails, so the fix really went in.  My guess is hence that it's really a distro problem. 

Just to make sure: you are using an 8x8 font, right? 

Mmm, are 50 and 60 lines modes really broken? I indeed have oddities with e.g. 43 lines, but not 50/60.

Could you also try to disable parallel boot? The oddities go away when I disable it (see /etc/sysconfig/boot)

What may be at stake here is that for instance in OpenSUSE's /etc/init.d/boot
script, you can see

otty=$(stty -g)

and then later on

stty $otty

If the font size is changed in the meanwhile (because kbd's script is run in
parallel), it is little wonder that things go wrong.  With 2.6.24 however there
should never be non-displayed text, i.e. text that goes off the screen, only
partial use of the screen.

Using 2.6.25.4-8-pae in SUSE 11.0b3+ the behavior with 0x30A on mga is no different than it was when this bug was filed, still 3/4 screen height, and drop from 43 lines back to 21 midway through init unless /etc/init.d/kbd execute is disabled. I don't know how to tell what font is in use, but I'm guessing the initial font with 0x30A is 8x8, and switches to 8x14 when kbd gets run.

As I said, it looks like a distro problem, you should raise the problem there, quoting my comment #45. 

(In reply to comment #47)
> As I said, it looks like a distro problem, you should raise the problem
> there,
> quoting my comment #45. 

I already raised it long ago: https://bugzilla.novell.com/show_bug.cgi?id=259577#c1

Here's the only way that seems appropriate to track such attempts. I still have to figure out how to try some others besides SUSE.

I tried to register to their bugzilla to add comments, but I'm still
waiting for the confirmation code... 

I think you should copy/paste my comment #45 there.

If admission doesn't come through in short order, try again from the beginning. If you don't succeed, try emailing novell.com Andreas Jaeger = aj@ for assistance. Better ref comments come from a developer who actually understands the issues than a mere mortal user like me.

Ok, so I think we fixed the userland part, but you're still having only 3/4 of the screen used _with proprietary modes_. I guess it is so right from the beginning (even before init gets started)?  Could you put return 0; at the beginning of linux/drivers/video/console/vgacon.c to make sure that the kernel doesn't tinker with CRTC_MAX_SCAN, and tell us whether that makes a difference before init boots, and after during the boot process? 

"Back in the 2.2.x kernel, ... your distribution was most
probably just not loading an 8x16 font by default."

I think I still have a functional old box with 2.2.x on RedHat 6.somethingorother where native modes work in case we want to check anything. Video card in it is Tseng, with native 132x25, 132x30, 132x43 & 132x50 or 60.

Where am I supposed to find linux/drivers/video/console/vgacon.c?

In the linux source code (we're back in the kernel bugzilla ;) )

Any preference whether I use Factory or 11.0? Either way, this is going to take a while. I've built no kernels since last you asked me to, and I don't remember much about doing it.

Either should be fine. 

(In reply to comment #51)
Could you put return 0; at the
> beginning of linux/drivers/video/console/vgacon.c to make sure that the
> kernel
> doesn't tinker with CRTC_MAX_SCAN, and tell us whether that makes a
> difference
> before init boots, and after during the boot process? 
 
I tried with 2.6.25.9-0.2-pae on OpenSUSE 11.0, but make halted:
...
CC drivers/video/console/dummycon.o
CC drivers/video/console/vgacon.o
drivers/video/console/vgacon.c:1: error: expected identifier or ?(? before ?return?
make[3]: *** [drivers/video/console/vgacon.o] Error 1
make[2]: *** [drivers/video/console] Error 2
make[1]: *** [drivers/video] Error 2
make: *** [drivers/video] Error 2

Sorry, I forgot to give you the function name: in vgacon_doresize, put
return 0; just after the opening brace and the 3 lines of variable
declaration.

Is this what you mean?

static int vgacon_doresize(struct vc_data *c,
		unsigned int width, unsigned int height)
{
	unsigned long flags;
	unsigned int scanlines = height * c->vc_font.height;
	u8 scanlines_lo = 0, r7 = 0, vsync_end = 0, mode, max_scan;
return 0;

	spin_lock_irqsave(&vga_lock, flags);

Yes.

I built 2.6.25.9-0.2-pae on OpenSUSE 11.0 on i845G:
0x122 on i810e - no change (OK only until late in init)
0x30A on mga - no change (OK only until late in init)
0x122 on i845G - apparently unsupported (puts display into sleep mode)

In the dmesg, you should have a line looking like

Console: colour VGA+ 80x50

what is that line saying in your dmesg?

BTW, how many rows & cols 0x122 is supposed to be? (It's not written
anywhere in the bugzillas). Just as a reminder, 0x30A is supposed to be
132x43.

Created attachment 16817 [details]
verbose BIOS-announced height

Could you try to boot with the attached patch and see what it tells in
dmesg?  I'm afraid the bios may not correctly give the font height used
by the extended modes :/

(In reply to comment #62)
> In the dmesg, you should have a line looking like
> Console: colour VGA+ 80x50
> what is that line saying in your dmesg?
(In reply to comment #63)
> BTW, how many rows & cols 0x122 is supposed to be?

using 0x30A on mga : Console: colour VGA+ 132x43
using 0x122 on i810: Console: colour VGA+ 132x50

IIRC, on i810e: 120=132x25, 121=132x43, 122=132x50, 123=132x60

Do you also happen to know the resolution that these modes produce (like
800x600)? e.g. maybe the monitor tells you that?

I don't know how a monitor might tell, but I don't think there are any SVGA text (hardware native proprietary) modes that are not 800x600. I'm pretty sure I had that figured out many years ago when I created the attachment to http://groups.google.com/group/comp.os.msdos.apps/msg/fe17541c60aad276?hl=en

Created attachment 16819 [details]
dmesg output booted using vga=0x122 on i810e & kernel built with comment 64 patch

I just tried an old Trident 9680 PCI, and noticed starting with vga=ask:

1-of 19 total modes, only 1 is VESA, 11 are BIOS, the rest VGA. I can't recall anything other than VGA & VESA with other gfxcards
2-if I select either of the 132x30 BIOS modes, init does not change the rows & columns, while if I select either 132x43 mode, it behaves the same as ATI, Intel, MGA & Nvidia.

FWIW in case you hadn't thought about it: 800x600 SVGA text modes are all 4bpp/16 color.

Err... 132 columns with a 8pixel wide font produces 1056 pixels. Or are these modes using non-8pixel fonts?  After init loads a font, you still have 132 columns, don't you?  I guess it would be difficult to actually count the number of pixels shown on the screen?  About the dmesg message, 50 * 8 = 400, that's what I was afraid of.  I guess your video rom is loading a font with more than 8 pixels height, but doesn't record that into the bios parameters where we fetch the font height.  I'm afraid we might have to read it directly from the card.  It's a shame I don't have my video boards here, I won't have them until September. 

OK, some memory came back, and I rediscovered a VGA programming book I bought in 1991. At least some SVGA text modes are not 800x600, and certainly all 132 column modes:
Mode Col Row Hori Ver Font
ATI 18800
309h 132 25  1056 350 8x14
(33) 132 44  1056 352 8x8
C&T 453
(60) 132 25  1056 400
(61) 132 50  1056 400 8x8
Trident 8900
153h 132 25  1056 350 8x14 
154h 132 30  1056 480 8x16
155h 132 43  1056 473 8x11
156h 132 60  1056 480 8x8
157h 132 25  1188 350 9x14 
158h 132 30  1188 480 9x16
159h 132 43  1188 473 9x11
15Ah 132 60  1188 480 9x8
Tseng 3000
309h 132 25  1056 350 8x14
30Ch 132 60
322h 132 44  1056 352 8x8
324h 132 28  1056 364 8x13
32Ah 100 40  800  600 8x15
Video7/Headland V7VGA
(41) 132 25  1056 350 8x14
(42) 132 43  1056 344 8x8
(45) 132 28  1056 392 8x14

The book has others, but they are all chips from companies that no longer exist. NVidia might be a name change from one of those (Genoa & Paradise), but if it is, I wouldn't know which. The modes in parenthesis are the DOS modes from the book that I couldn't translate into INT10 4F modes.

In every case, what starts as 132 col stays at 132 col after the init switch.

Ok, what I guess might be problematic are modes like

155h 132 43  1056 473 8x11

Maybe in that case the bios still reports a 8 pixel font height.

Could you install (but not enable!) the svgatextmode in order to get
the getVGAreg program, and then report what the following says when you
properly have the 43 or 50 lines (i.e. in emergency mode or by disabling
the start of the font loading script):

getVGAreg CRTC 0x6
getVGAreg CRTC 0x7
getVGAreg CRTC 0x9
getVGAreg CRTC 0x10
getVGAreg CRTC 0x11
getVGAreg CRTC 0x12
getVGAreg CRTC 0x15
getVGAreg CRTC 0x16
getVGAreg CRTC 0x17

I'm having no luck finding getVGAreg or svgatextmode with webpin, YaST2 and
http://software.opensuse.org/search for OpenSUSE 11.0. :-(

Created attachment 16822 [details]
getVGAreg

Ah, well then here is the debian version.

Created attachment 16823 [details]
comment 72 response: getVGAreg output

3 BIOS modes on i810E, 1 VESA mode each on MGA & i810E

Let's concentrate on mode 0x122, since that's the one you provided most information about.  Mmm, it looks like it's indeed a 8x8 font that is loaded by your video ROM, and the number of scanlines is then properly detected.  How many lines do you get after init loads a font?  (run stty -a to know) Could you also run the getVGAreg series after init loads a font, for comparison? 

Created attachment 16828 [details]
output of comment 76 request

Err, have you really configured your distribution to load an 8x8 font?
Please make sure that CONSOLE_FONT in /etc/sysconfig/console is a -08
variant, not a -16 variant (else there is little wonder you have
troubles).

Created attachment 16830 [details]
output of comment 76 request on same system booted to Cooker instead of SUSE

From Cooker: -rw-r--r-- 1 root root 1977 2008-01-19 17:31 /etc/sysconfig/console/consolefonts/lat0-16.psfu.gz

I tried replacing that file, the only thing that looked relevant in the whole /etc/sysconfig/console tree, with lat0-08.psfu.gz, but Cooker behavior was unchanged, and I noted during init that lat0-16 font was being loaded.

OTOH, replacing the -16 CONSOLE_FONT in SUSE's /etc/sysconfig/console caused all of init to continue with apparently the exact same appropriate font throughout whether cmdline vga= had either 0x121 or 0x122. That file had never previously been touched by me. Whatever is in it was put there by SUSE.

It's nice most gfxcards support the more usual framebuffer modes. Having to know in advance that to successfully continue to use a native/BIOS console mode that the kernel had no problem managing requires preconfiguring some config file that varies according to distro is rather antithetical to the precept that puters are supposed to make things easier, do things for us rather than the other way around.

Thanks for putting grumbling comments aside, they are not welcome:
no, programmers don't do it on purpose to break your particular
way of using a computer.

That being said, if loading an 8pixel font fixes the issue then that's
again a distribution bug actually: it should load a font whose size is
the same as what is currently loaded.  They can do that e.g. by using
the GIO_FONTX ioctl.  You can file a bug in that direction.

I'm quite confused by the Cooker behavior. I guess it is loading the
font from somewhere else.  In any case, that's a distribution bug.

Comment 79's "grumbling" was really just an expression of disappointment that all the time we just spent on this, which seemed to be leading to a cross-distro fix via the kernel, turned out to be nothing more than proof the distros are where the problem lies. Thanks for all your focused time pinning the blame where it belongs.

Well, ok, but still :)