Bug 219108 - [REGRESSION] Inbuilt keyboard on the 2022 Asus Vivobook S15 OLED has bad key registration after charging in kernel versions 6.10 and beyond
Summary: [REGRESSION] Inbuilt keyboard on the 2022 Asus Vivobook S15 OLED has bad key ...
Status: RESOLVED CODE_FIX
Alias: None
Product: Drivers
Classification: Unclassified
Component: Input Devices (show other bugs)
Hardware: All Linux
: P3 normal
Assignee: drivers_input-devices
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-07-30 13:04 UTC by icaliberdev
Modified: 2024-10-02 11:32 UTC (History)
3 users (show)

See Also:
Kernel Version: 6.10
Subsystem:
Regression: Yes
Bisected commit-id: de52aca4d9d56c3b2f00b638d457075914b1a227


Attachments
dmesg log (82.02 KB, text/plain)
2024-07-30 13:17 UTC, icaliberdev
Details
dmesg error log (for convinience) (704 bytes, text/plain)
2024-07-30 13:18 UTC, icaliberdev
Details

Description icaliberdev 2024-07-30 13:04:46 UTC
As the summary says, on the 2022 Asus Vivobook S15 OLED (model K3502ZA.306), the inbuilt keyboard has bad key registration after plugging the laptop in for charging. Sometimes, the keys pressed do not register at all, and sometimes they register very late. The keyboard works properly if I don't plug the laptop in to charge, and this issue doesn't manifest ever on kernels older than 6.6.37-1.

I have tested every lts kernel after 6.6.37-1 and all of them have the same problem. I have also tested the latest mainline and zen kernels (versions 6.10.1 for both) and they too have the the same problem.
Once the laptop is plugged into it's charging cable, the keyboard fails and unplugging the laptop doesn't fix the situation. However, if I don't plug the laptop into it's charging cable, the problem never manifests itself. The charging cable itself is a usb type c cable and the laptop charges over thunderbolt.
Comment 1 icaliberdev 2024-07-30 13:16:35 UTC
I have tested usb keyboards, and they do not have this problem at all.

Some system details are as follows:
      Processor: Intel i5-12500H
  
      Current Kernel: 6.6.36-1
   
      Model Number: K3502ZA
        
      BIOS verison: 306

I cannot think of any other system specification that would be important.
I am willing to assist in any way possible, including compiling kernels with any patches necessary.

Steps to reproduce:
   
      Own a 2022 Asus Vivobook S15 OLED (model number K3502ZA)
    
      Boot into kernel version 6.6.37-1 or higher
       
      Plug the laptop to charge it
       
      The keyboard has bad registration.

There is another bug there might be similar here: https://bugzilla.kernel.org/show_bug.cgi?id=218929

However, it seems to be different, and also might be resolved pretty easily in the future.

I have attached the dmesg logs from Arch's linux-lts-6.6.42-1 package.

None of these errors are new to the 6.6.36 kernel where the issue isn't present, save for the `USBC000:00` errors. But the keyboard issue is still there in linux-lts 6.6.37-1 and that kernel doesn't have those particular errors.

Once again, I am willing to help in any way possible, like testing patches, recording video, etc. Thank you in advance.
Comment 2 icaliberdev 2024-07-30 13:17:36 UTC
Created attachment 306634 [details]
dmesg log
Comment 3 icaliberdev 2024-07-30 13:18:12 UTC
Created attachment 306635 [details]
dmesg error log (for convinience)
Comment 4 icaliberdev 2024-07-30 13:19:21 UTC
I have also filed a bug in the Arch Linux bug tracker.
https://gitlab.archlinux.org/archlinux/packaging/packages/linux-lts/-/issues/9
Comment 5 Artem S. Tashkinov 2024-07-30 17:42:36 UTC
First test 6.10.2.

If it's still broken, please perform regression testing using:

https://docs.kernel.org/admin-guide/bug-bisect.html
Comment 6 icaliberdev 2024-08-01 05:43:49 UTC
(In reply to Artem S. Tashkinov from comment #5)
> First test 6.10.2.

I checked the latest LTS release (6.6.43), so I don't see why it should matter if I checked the latest mainline kernel.

Never the less, I have checked 6.10.2, and can confirm that it has the same issue.

> If it's still broken, please perform regression testing using:
> 
> https://docs.kernel.org/admin-guide/bug-bisect.html

I will do this as soon as I get enough time to do so. Thanks for the guide ! I have never bisected something before, so I might take some time....
Comment 7 The Linux kernel's regression tracker (Thorsten Leemhuis) 2024-08-01 05:54:39 UTC
(In reply to icaliberdev from comment #6)
> (In reply to Artem S. Tashkinov from comment #5)
>
> I checked the latest LTS release (6.6.43), so I don't see why it should
> matter if I checked the latest mainline kernel.

In short: because that series is maintained by different people -- and those won't fix the problem. More on this can be found on https://linux-regtracking.leemhuis.info/post/frequent-reasons-why-linux-kernel-bug-reports-are-ignored/
 
> > https://docs.kernel.org/admin-guide/bug-bisect.html

FWIW, a more detailed guide can be found here: https://docs.kernel.org/admin-guide/verify-bugs-and-bisect-regressions.html
Comment 8 icaliberdev 2024-08-03 16:29:42 UTC
Christian Heusel over at the Arch Linux forum bisected the kernel for me !

Here is the defective commit and it's message : de52aca4d9d56c3b2f00b638d457075914b1a227

```
Author: Christian A. Ehrhardt <lk@c--e.de>
Date:   Wed Mar 27 23:45:53 2024 +0100

    usb: typec: ucsi: Never send a lone connector change ack

    Some PPM implementation do not like UCSI_ACK_CONNECTOR_CHANGE
    without UCSI_ACK_COMMAND_COMPLETE. Moreover, doing this is racy
    as it requires sending two UCSI_ACK_CC_CI commands in a row and
    the second one will be started with UCSI_CCI_ACK_COMPLETE already
    set in CCI.

    Bundle the UCSI_ACK_CONNECTOR_CHANGE with the UCSI_ACK_COMMAND_COMPLETE
    for the UCSI_GET_CONNECTOR_STATUS command that is sent while
    handling a connector change event.

    Signed-off-by: Christian A. Ehrhardt <lk@c--e.de>
    Reviewed-by: Heikki Krogerus <heikki.krogerus@linux.intel.com>
    Tested-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
    Link: https://lore.kernel.org/r/20240327224554.1772525-3-lk@c--e.de
    Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

```
Bisected-by: Christian Heusel <christian@heusel.e>

Seems like it was something related to USB C, which is the charging cable type for my laptop.
Comment 9 Christian Heusel 2024-08-03 17:06:34 UTC
I would have also liked to test a mainline commit with the questionable commit reverted but I get a build error with the commit reverted on 6.10 and a conflict on 6.11-rc1 :(

```
  CC [M]  drivers/input/rmi4/rmi_f34v7.o
drivers/usb/typec/ucsi/ucsi.c: In function ‘ucsi_exec_command’:
drivers/usb/typec/ucsi/ucsi.c:163:31: error: implicit declaration of function ‘ucsi_acknowledge’; did you mean ‘ucsi_acknowledge_command’? [-Wimplicit-function-declaration]
  163 |                         ret = ucsi_acknowledge(ucsi, false);
      |                               ^~~~~~~~~~~~~~~~
      |                               ucsi_acknowledge_command
  CC [M]  fs/dlm/rcom.o
  CC [M]  drivers/net/ethernet/intel/i40e/i40e_adminq.o
  CC [M]  drivers/net/wireless/ath/ath11k/mhi.o
  CC [M]  drivers/net/ethernet/intel/i40e/i40e_common.o
  LD [M]  drivers/staging/fieldbus/fieldbus_dev.o
  LD [M]  drivers/staging/vt6655/vt6655_stage.o
make[6]: *** [scripts/Makefile.build:244: drivers/usb/typec/ucsi/ucsi.o] Error 1
make[5]: *** [scripts/Makefile.build:485: drivers/usb/typec/ucsi] Error 2
```
Comment 10 The Linux kernel's regression tracker (Thorsten Leemhuis) 2024-08-04 07:39:48 UTC
Forwarded by mail: https://lore.kernel.org/all/8a541683-36da-4cea-a8a9-db6ee875c86d@leemhuis.info/
Comment 11 Christian A. Ehrhardt 2024-08-04 18:15:51 UTC
Hm, I don't see how this change could trigger the symptoms described. But apparently, it does.

I take it the ucsi related error messages are not present on a working kernel?

For starters and given that this is also an ASUS, can you try to enable the ASUS zenbook quirk in drivers/usb/typec/ucsi/ucsi_acpi.c?

If that doesn't help we will have to dig deeper.
Comment 12 icaliberdev 2024-08-18 16:54:51 UTC
Hey everyone, sorry about the long silence, I was busy with college.

(In reply to Christian A. Ehrhardt from comment #11)
> I take it the ucsi related error messages are not present on a working
> kernel?

No, currently the 6.6.36-1 kernel also has this error:
```
kernel: ucsi_acpi USBC000:00: ucsi_handle_connector_change: GET_CONNECTOR_STATUS failed (-110)
```

> For starters and given that this is also an ASUS, can you try to enable the
> ASUS zenbook quirk in drivers/usb/typec/ucsi/ucsi_acpi.c?
I have no idea what that is, I couldn't find it on google. Can you direct me to it ? Also some resources on how to apply patches because I don't know how to do that :D.

I have some good news though. I got rid of this error:
(In reply to Christian Heusel from comment #9)
> I would have also liked to test a mainline commit with the questionable
> commit reverted but I get a build error with the commit reverted on 6.10 and
> a conflict on 6.11-rc1 :(

I just replaced the ucsi_acknowledge with ucsi_acknowledge_command after reverting the commit. It compiled (version 6.10.5), but I'm scared to boot into it to test it. If someone can reassure me that my laptop won't explode if I boot into it, I'll test it out.

As mentioned though all 6.11 versions had merge conflicts, which I didn't want to attempt to fix.
Comment 13 icaliberdev 2024-08-18 16:57:28 UTC
The change:

```
diff --git a/drivers/usb/typec/ucsi/ucsi.c b/drivers/usb/typec/ucsi/ucsi.c
index d42fc7578e2d..719065b2da83 100644
--- a/drivers/usb/typec/ucsi/ucsi.c
+++ b/drivers/usb/typec/ucsi/ucsi.c
@@ -160,7 +160,7 @@ static int ucsi_exec_command(struct ucsi *ucsi, u64 cmd)

        if (cci & UCSI_CCI_ERROR) {
                if (cmd == UCSI_GET_ERROR_STATUS) {
-                       ret = ucsi_acknowledge_command(*ucsi);
+                       ret = ucsi_acknowledge_command(ucsi);
                        if (ret)
                                return ret;
```
Comment 14 Christian A. Ehrhardt 2024-08-25 17:19:48 UTC
I'm looking into this...
Comment 15 Christian A. Ehrhardt 2024-08-27 09:52:04 UTC
Debugging showed that there are several issues here:

First the quirk for some ASUS zenbooks is required for the VivoBook, too. Otherwise we will read bogus values in CCI.

Doing this reveals that UCSI on the affected laptop does two things that are in violation of the spec:
- It sends busy notifications that contain additional bogus data (e.g. a
  bogus connector change notification or stale command completion bits).
- In response to a GET_CONNECTOR_STATUS command the CCI value always contains
  the connector number (which is interpreted as a new connector change
  event).

The bogus busy notifications result in a command timeout. In response to
this command timeout the EVENT_PENDING bit is cleared in an attempt to recover from the timeout. Most of the time this is ok but in this case the timeout happens on the ACK command and this causes the EVENT_PENDING bit to go out of sync.

As a result ucsi_handle_connector_change() is entered with the EVENT_PENDING bit clear. This again results in an infinite loop:
- ucsi_handle_connector_change is entered with EVENT_PENDING clear.
- ucsi_handle_connector_change() sends a GET_CONNECTOR_STATUS command.
- The response to this command includes the connector number in the lower bits.
- This is interpreted as a new connector change event. Normally, this event
  would be ignored as EVENT_PENDING is still set. However, in this case
  EVENT_PENDING is set and the connector work is scheduled again.
- When we send the ACK command for the GET_CONNECTOR_STATUS command,
  we immediately clear EVENT_PENDING on the grounds that we just handled
  and cleared the connector change event.
- However, the connector change work is still scheduled and everything starts
  all over again.

Apparently, this infinite loop keeps some resource (the embedded controller?) so busy that it fails to handle keystrokes on the builtin keyboard...

I will send a patch that:
- Enables the zenbook quirks for the affected laptops
- ignores the contents of CCI if the busy indicator is set.
- makes sure that EVENT_PENDING is set on entry to
  ucsi_handle_connector_change().
Comment 16 Christian A. Ehrhardt 2024-09-29 16:12:31 UTC
Fix is merged:
| commit 7fa6b25dfb43dafc0e16510e2fcfd63634fc95c2
| Author: Christian A. Ehrhardt <lk@c--e.de>
| Date:   Thu Sep 12 09:41:32 2024 +0200
|
|     usb: typec: ucsi: Fix busy loop on ASUS VivoBooks

Note You need to log in before you can comment on or make changes to this bug.