Bug 119391

Summary: ASUS Q500A: corrupted kaycodes on dualboot system
Product: Platform Specific/Hardware Reporter: Oleksij Rempel (linux)
Component: x86-64Assignee: platform_x86_64 (platform_x86_64)
Status: ASSIGNED ---    
Severity: normal CC: alexhenrie24, corentin.chary, dvhart, lv.zheng, peter, rui.zhang
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 4.7.0-rc1-00023-g367d3fd Subsystem:
Regression: No Bisected commit-id:
Bug Depends on: 150811    
Bug Blocks:    
Attachments: acpi_dump
acpi_dump - no bug
quirk.patch
dmesg with ec patch

Description Oleksij Rempel 2016-05-31 20:56:53 UTC
Created attachment 218471 [details]
acpi_dump

This problem is 100% reproducible on ASUS Q500A.

Description:
By using any of hotkays (volume, brightness, etc) result can be unpredictable. some times wrong event is triggered, some times it will generate asci code instead of ACPI event.

This issue is only reproducible if windows was ever started on this notebook.

After complete reset (unplug power and battery), linux (i mean all kay) are working properly.

Windows seems to work without any issue even on dual boot system, so only linux is affected.

Till now i tried:
- disable BIOS fast boot function
- disable UEFI
- disable windows fastboot function.

Tested software:
- Windows 10
- Ubuntu 16.04 + latest kernel.git
Comment 1 Zhang Rui 2016-06-01 02:02:03 UTC
so, it seems that windows is doing something different, which can not be reset unless ACPI G3(mechanical off), and windows is also doing something different during boot, which can handle the first difference after an ACPI S5(soft off).

TBH, I don't know how to handle this unless we know what is done in windows.
Anyway, please check if any of the following tricks helps or not
1. boot Linux with kernel parameter: acpi_osi=! acpi_osi="Windows 2015"
or
2. boot Linux with kernel parameter: acpi=off
or
3. try a different windows version like win7/win8
Comment 2 Oleksij Rempel 2016-06-01 04:05:14 UTC
Results:
- acpi_osi=!
Hotkays seems to work properly, but keyboard is screwed completely now. No other kay is working expect of hotkays.

- acpi_osi="Windows 2015"
Same result as with default boot, keyboard is working but hotkays are corrupt.
If is see it correctly, this DSDT supports max Windows 2012.

- acpi=off
No hotkays are working.

- win7/win8
Will take some time untill i get them
Comment 3 Zhang Rui 2016-06-01 05:04:35 UTC
(In reply to Oleksij Rempel from comment #2)
> Results:
> - acpi_osi=!
> Hotkays seems to work properly, but keyboard is screwed completely now. No
> other kay is working expect of hotkays.
> 
when you say hotkeys are working, do you mean they are always working, even if a soft shutdown from windows?

> - acpi_osi="Windows 2015"
> Same result as with default boot, keyboard is working but hotkays are
> corrupt.
> If is see it correctly, this DSDT supports max Windows 2012.

yes, my intention was to use this parameter together with acpi_osi=!.
Comment 4 Oleksij Rempel 2016-06-01 07:44:16 UTC
(In reply to Zhang Rui from comment #3)
> (In reply to Oleksij Rempel from comment #2)
> > Results:
> > - acpi_osi=!
> > Hotkays seems to work properly, but keyboard is screwed completely now. No
> > other kay is working expect of hotkays.
> > 
> when you say hotkeys are working, do you mean they are always working, even
> if a soft shutdown from windows?

Yes.

> > - acpi_osi="Windows 2015"
> > Same result as with default boot, keyboard is working but hotkays are
> > corrupt.
> > If is see it correctly, this DSDT supports max Windows 2012.
> 
> yes, my intention was to use this parameter together with acpi_osi=!.

ah... i'll retest it this evening.
Comment 5 Oleksij Rempel 2016-06-01 17:54:12 UTC
- acpi_osi=! acpi_osi="Windows 2015"
Hmm... looks like this options really solve this issue. What is happening here?
Comment 6 Oleksij Rempel 2016-06-01 18:03:21 UTC
But after some suspend/resume rounds keyboard started to be useless.
Comment 7 Zhang Rui 2016-06-02 00:42:21 UTC
(In reply to Oleksij Rempel from comment #5)
> - acpi_osi=! acpi_osi="Windows 2015"
> Hmm... looks like this options really solve this issue. What is happening
> here?

TBH, I don't know, I'm just trying to make Linux pretend to be win10 in this case.
Comment 8 Zhang Rui 2016-06-02 00:46:16 UTC
(In reply to Oleksij Rempel from comment #6)
> But after some suspend/resume rounds keyboard started to be useless.

please try some other combinations,
1. acpi_osi=! acpi_osi="Windows 2012"
2. acpi_osi=! acpi_osi="Windows 2009"
Comment 9 Oleksij Rempel 2016-06-02 06:12:18 UTC
(In reply to Zhang Rui from comment #8)
> (In reply to Oleksij Rempel from comment #6)
> > But after some suspend/resume rounds keyboard started to be useless.
> 
> please try some other combinations,
> 1. acpi_osi=! acpi_osi="Windows 2012"
> 2. acpi_osi=! acpi_osi="Windows 2009"

Same results. as with "acpi_osi=! acpi_osi="Windows 2015"

I would describe it a two level of fail:
1. ally keys seems to work properly but kernel warns about unknown keycode <e061>
2. hotkeys are kind of working, but normal keys are completely screwed.

After dirty start (windows was started on this machine and not hard reset was made), 1. level is 100% reproducible with all suggested kernel options. 2. level is usually reproducible after some cycles of syspend/resume or probably more intensive testing.

I would assume, that most users will notice only 2. level which is reproducible only after dirty start and after hot keys was used.
Comment 10 Oleksij Rempel 2016-06-02 18:12:56 UTC
before i start reinstalling the system, probably it would be good to gather some debug information from the current installation?
Comment 11 Oleksij Rempel 2016-06-04 18:40:38 UTC
I two days on reinstalling different Windows versions...
Result is fallowing:
- Windows version makes no difference.
- ASUS ATK Software (ACPI WMI Driver?), do make a difference. If it is not installed on Windows, Linux has no problem.
Comment 12 Zhang Rui 2016-06-06 05:56:27 UTC
(In reply to Oleksij Rempel from comment #11)
> I two days on reinstalling different Windows versions...
> Result is fallowing:
> - Windows version makes no difference.
> - ASUS ATK Software (ACPI WMI Driver?), do make a difference. If it is not
> installed on Windows, Linux has no problem.

cool.
To me, the problem is that the ASUS ATK software does not do something cleanly upon shutdown. But it MAY BE possible to enhance Linux to be able to handle this, say, force a cleanup/reset of the device during boot, etc. But I don't know how to do this w/o knowing what is done in Asus ATK software.

Anyway, this is not a Linux kernel bug to me. Reassign to the Asus expert to see if we can handle this in Linux, before just closing the bug report.
Comment 13 Oleksij Rempel 2016-06-06 06:14:57 UTC
I got one more ASUS Q500A with preinstalled BIOS version 208, this bug is not reproducible with this version.
The buggy version is 207.
The problem is that v 208 is not available for public download. I send i request to ASUS, lets see what will happen.
Comment 14 Oleksij Rempel 2016-06-06 06:26:13 UTC
Created attachment 219171 [details]
acpi_dump - no bug

acpu dump from BIOS v208.

Looks like there is some additional sanity checks in DSDT.
Comment 15 Oleksij Rempel 2016-06-12 12:10:36 UTC
Status update...
- ASUS Support from Germany answered that: 1. ASUS Q500A-BHI5N01 BIOS version 207; 2. ASUS Q500A-BSI5N04 BIOS version 208; have different main boards revisions, so this should be reason why they have different BIOS version. They also suggesied to contact ASUS USA, since this model are not sold in Gemrany.

- In my investigation i found that file named "atkwmiacpi64.sys" is triggering this issue. Ill try to get some traces on windows...

- with "showkey -s" (linux) it is easier to test this issue.
I made fallowing test on "ASUS Q500A-BHI5N01 BIOS version 207":
* press 4x Fn+space (power4gear)
* press 4x Fn+c (Splendid)
* press 4x Fn+v (WebCam)
* press 4x Fn+F11 (volume down)
* press 4x Fn+F12 (volume up)

results after windows:
0x5f 0xdf 
0x5f 0xdf 0x23 
0x5f 0xdf 
0x5f 0xdf 0x23 
0xe0 0x1f 0xe0 0x9f 
0xe0 0x1f 0xe0 0x9f 0x23 
0xe0 0x1f 0xe0 0x9f 
0xe0 0x1f 0xe0 0x9f 0x23 
0xe0 0x3b 0xe0 0xbb 
0xe0 0x3b 0xe0 0xbb 0x23 
0xe0 0x3b 0xe0 0xbb 
0xe0 0x3b 0xe0 0xbb 0x23 
0x30 
0x30 
0x30 
0x30 
0x2e 
0x2e 
0x2e 
0x2e 


results after complete reset:
0x5f 0xdf 
0x5f 0xdf 
0x5f 0xdf 
0x5f 0xdf 
0xe0 0x1f 0xe0 0x9f 
0xe0 0x1f 0xe0 0x9f 
0xe0 0x1f 0xe0 0x9f 
0xe0 0x1f 0xe0 0x9f 
0xe0 0x3b 0xe0 0xbb 
0xe0 0x3b 0xe0 0xbb 
0xe0 0x3b 0xe0 0xbb 
0xe0 0x3b 0xe0 0xbb 
0xe0 0x30 0xe0 0xb0 
0xe0 0x30 0xe0 0xb0 
0xe0 0x30 0xe0 0xb0 
0xe0 0x30 0xe0 0xb0 
0xe0 0x2e 0xe0 0xae 
0xe0 0x2e 0xe0 0xae 
0xe0 0x2e 0xe0 0xae 
0xe0 0x2e 0xe0 0xae 


On "ASUS Q500A-BSI5N04 BIOS version 208" both results are identical.

For now looks like "ASUS Q500A-BHI5N01" has missing KeyRelease events.
Interesting how it is handled on Windows? EC is reseted/reconfigured to provide proper key events, or they "fix" it by keyboard filter like we do it for some buggy HW on linux side?
Comment 16 Oleksij Rempel 2016-07-28 07:19:52 UTC
I still can't find how exactly atkwmiacpi64.sys reconfigures EC. So right now it looks like easiest way is to make use of keyboard force-release quirk:

echo 32,35,46,48 > /sys/devices/platform/i8042/serio0/force_release

It seems to work good in both clean and brocken cases.

Affected keys are:
volume up 0x30, down 0x2e, mute 0x20
display off 0x23

Previously looks like udev needed to be updated to add new quirks:
https://github.com/lu-zero/udev/blob/master/src/keymap/95-keyboard-force-release.rules

But looks like it is not the case now. Which userspace application should be updated/extended?
Comment 17 Alex Henrie 2016-07-28 16:21:38 UTC
Maybe /lib/udev/hwdb.d/60-keyboard.hwdb?
Comment 19 Oleksij Rempel 2016-07-29 19:50:03 UTC
Suddenly i was not able to reliable fix it from user space. Reported keycodes was not consistent. The reason is that actual atkbd.c was getting confusing prefix to actual codes.
For example, this result i got directly in atkbd.c for volume buttons after clean start:
[  109.353442] t:1;e:0;c:e0
[  109.355981] t:1;e:1;c:30
[  109.358052] t:1;e:0;c:e0
[  109.363052] t:1;e:1;c:b0

[  109.818031] t:1;e:0;c:e0
[  109.820548] t:1;e:1;c:2e
[  109.822677] t:1;e:0;c:e0
[  109.827697] t:1;e:1;c:ae

[  110.189919] t:1;e:0;c:e0
[  110.191935] t:1;e:1;c:20
[  110.194612] t:1;e:0;c:e0
[  110.199632] t:1;e:1;c:a0

this i got after windows:
[  953.911329] t:1;e:0;c:e1
[  953.918923] t:1;e:2;c:23
[  953.922925] t:1;e:1;c:e0
[  953.925653] t:1;e:0;c:30
[  953.927707] t:1;e:0;c:e0
[  953.932691] t:1;e:1;c:b0

[  954.470419] t:1;e:0;c:e1
[  954.477625] t:1;e:2;c:23
[  954.488170] t:1;e:1;c:e0
[  954.490639] t:1;e:0;c:2e
[  954.492798] t:1;e:0;c:e0
[  954.498079] t:1;e:1;c:ae

[  955.131170] t:1;e:0;c:e1
[  955.139022] t:1;e:2;c:23
[  955.144700] t:1;e:1;c:e0
[  955.146725] t:1;e:0;c:20
[  955.149385] t:1;e:0;c:e0
[  955.153537] t:1;e:1;c:a0

If i understand it correctly. 0xe1 means that next two codes should be interpreted as one scancode, but in reality only one: 0x23 is available.

I added a quirk to filter this codes out, so far no other changes was needed.
Comment 20 Oleksij Rempel 2016-07-29 19:50:51 UTC
Created attachment 226891 [details]
quirk.patch
Comment 21 Peter Wu 2016-08-27 12:47:12 UTC
Can confirm the spurious keys behavior on an Asus X550ZE after rebooting. Keys are coming straight from the EC, I did not check whether it also occurs after rebooting without installing the ATK package though.

Press Fn+F12 four times and this is visible (showkey -s over SSH):
0x30 0xe0 0xb0 
0xe0 0x30 0xe0 0xb0 
0xe0 0x30 0xe0 0xb0 
0xe0 0x30 0xe0 0xb0 

In the text console (zsh), I also see one '<FFFFFFFF>' during this. I have uploaded some logs to https://lekensteyn.nl/files/Asus-X550ZE/

- dmesg-4.8.0-rc3-testing-00183-g5e608a0.txt
  Various debug options enabled, output of showkey -s written to /dev/kmsg
- windbg-kd-acpi-trace-atk-installed.txt
  Verbose ACPI trace of boot where ATK v1.0.0041 was installed.
- windbg-kd-acpi-trace-atk-installed-rebooted.txt
  Verbose ACPI trace after rebooting (search for "kd>" to see moments where I added an annotation like "Press Fn+f10 later", search for "ATKD." for the ATKD device).

The WinDbg ACPI traces also show how ATKD methods are invoked, this might be useful for the asus-nb-wmi developer (cc'ing Corentin Chary).

(If you are curious, the DSDT override in the dmesg is due to bug 115021.)
Comment 22 Lv Zheng 2016-08-30 23:22:58 UTC
Hi,

Please give this patch a try:
attachment 231401 [details]

Thanks
Comment 23 Oleksij Rempel 2016-08-31 16:35:04 UTC
Created attachment 231611 [details]
dmesg with ec patch

Suddenly not helping in my case. Dmesg is attached.
Comment 24 Lv Zheng 2016-09-01 01:32:54 UTC
(In reply to Oleksij Rempel from comment #23)
> Created attachment 231611 [details]
> dmesg with ec patch
> 
> Suddenly not helping in my case. Dmesg is attached.

OK, resetting this to platform driver.
It looks the ECDT case is not a common case for hotkey bugs.
Comment 25 Oleksij Rempel 2016-09-04 06:26:43 UTC
For the log. I tried to upstrem my patch. Here is the response from maintainer Dmitry Torokhov:
I'd prefer it was done as a i8042_filter() installed in a platform
driver (asus-laptop.c?). I a filter you'd need to note the fact that you
received e1 and when you receive next character you either swallo both
or re-inject e1 and the new character. See filter implementation in
./drivers/input/misc/ideapad_slidebar.c.

The scancode fixup method is quite old and is better suited when you
actually want to use the scancode, just a different one. Here you are
forced to user ATKBD_RET_ERR which causes debug messages and increment
error counter.

Will continue on it ASAP.
Comment 26 Alex Henrie 2016-10-12 05:05:46 UTC
For what it's worth, I can confirm that attachment 226891 [details] fixes the problem, although there is already a better patch in linux-next. Thanks Oleksij for all the hard work!