Bug 217336

Summary: keyboard not working Asus TUF FA617NS
Product: Drivers Reporter: seb.j
Component: Input DevicesAssignee: Mario Limonciello (AMD) (mario.limonciello)
Status: RESOLVED CODE_FIX    
Severity: normal CC: clay.white, dridri85, friedrich.vock, hui.wang, julien.colinet, jwrdegoede, mario.limonciello, marslogout, nmschulte, npliashechnikov, pearfactory, pmenzel+bugzilla.kernel.org, raphael.alampay, redondogomezgoicoechea, starquake, tamim, teslan223
Priority: P3    
Hardware: AMD   
OS: Linux   
Kernel Version: Subsystem:
Regression: No Bisected commit-id:
Attachments: dmesg output
acpi.log
dmesg 5.19 output
dmesg grep acpi
acpi.log kernel 5.19
Reenable IRQ override on Asus TUF FA617NS
git error
patched dmesg output
dmesg 6.3 patch removed
kernel 6.2.11 no patch
added debug to grub
Test patch to force IRQ 1 to level high
Test patch to force IRQ 1 to level low
kernel_high_patch_dmesg_output
dmesg 6.3 patch lo
bios 308 dmesg asus patch applied
dmesg with dumbkbd
dmesg noaux option
dmesg nomux option
ASUS Live Chat support
ASUS Live Chat support
Debian 6.3.0-0 amd64 i8042 option variations logs
/sys/kernel/debug/gpio
FA617XT.309 gpio, interrupts, RWEverything-mmio
PDF of GPIO register configuration; Win 11 vs Linux
potential patch (v1)
potential patch (v2)
ACPI dump from FA617NS
FA617XT.309 gpio, acpidump -- w/ both patch sets (on v6.4-rc4)
dmesg - Asus A15 2023 - generic 6.4.6

Description seb.j 2023-04-14 16:27:15 UTC
Hi Guys, my problem, my keyboard is not working on my laptop asus TUF FA617NS. i was trying ubuntu 23.04 lts kernel 6.2  but it's sill failing.

lsmod:

Module                  Size  Used by
ccm                    20480  6
rfcomm                 98304  4
snd_seq_dummy          16384  0
snd_hrtimer            16384  1
hid_logitech_hidpp     61440  0
hid_logitech_dj        32768  0
input_leds             16384  0
cmac                   16384  3
algif_hash             20480  1
algif_skcipher         16384  1
af_alg                 32768  6 algif_hash,algif_skcipher
bnep                   32768  2
binfmt_misc            24576  1
uvcvideo              139264  0
videobuf2_vmalloc      20480  1 uvcvideo
videobuf2_memops       20480  1 videobuf2_vmalloc
videobuf2_v4l2         36864  1 uvcvideo
videodev              319488  2 videobuf2_v4l2,uvcvideo
videobuf2_common       86016  4 videobuf2_vmalloc,videobuf2_v4l2,uvcvideo,videobuf2_memops
nls_iso8859_1          16384  1
mc                     81920  4 videodev,videobuf2_v4l2,uvcvideo,videobuf2_common
usbhid                 69632  2 hid_logitech_dj,hid_logitech_hidpp
amdgpu              14499840  18
snd_soc_dmic           16384  0
snd_soc_acp6x_mach     24576  0
snd_acp6x_pdm_dma      16384  0
snd_sof_amd_rembrandt    16384  0
intel_rapl_msr         20480  0
snd_sof_amd_renoir     16384  0
intel_rapl_common      40960  1 intel_rapl_msr
snd_sof_amd_acp        45056  2 snd_sof_amd_rembrandt,snd_sof_amd_renoir
edac_mce_amd           40960  0
snd_sof_pci            24576  2 snd_sof_amd_rembrandt,snd_sof_amd_renoir
snd_sof_xtensa_dsp     16384  1 snd_sof_amd_acp
kvm_amd               204800  0
snd_sof               311296  2 snd_sof_amd_acp,snd_sof_pci
mt7921e                28672  0
snd_hda_codec_realtek   192512  1
joydev                 32768  0
mt7921_common         114688  1 mt7921e
snd_sof_utils          20480  1 snd_sof
snd_hda_codec_generic   118784  1 snd_hda_codec_realtek
snd_hda_codec_hdmi     94208  2
kvm                  1347584  1 kvm_amd
mt76_connac_lib        90112  2 mt7921e,mt7921_common
snd_soc_core          417792  4 snd_soc_acp6x_mach,snd_sof,snd_acp6x_pdm_dma,snd_soc_dmic
irqbypass              16384  1 kvm
crct10dif_pclmul       16384  1
btusb                  69632  0
snd_compress           32768  1 snd_soc_core
iommu_v2               24576  1 amdgpu
polyval_clmulni        16384  0
mt76                  122880  3 mt7921e,mt7921_common,mt76_connac_lib
snd_hda_intel          61440  3
polyval_generic        16384  1 polyval_clmulni
snd_seq_midi           20480  0
ac97_bus               16384  1 snd_soc_core
btrtl                  28672  1 btusb
drm_buddy              20480  1 amdgpu
ghash_clmulni_intel    16384  0
snd_seq_midi_event     16384  1 snd_seq_midi
snd_pcm_dmaengine      20480  1 snd_soc_core
snd_intel_dspcfg       36864  2 snd_hda_intel,snd_sof
btbcm                  28672  1 btusb
gpu_sched              61440  1 amdgpu
btintel                53248  1 btusb
snd_pci_ps             20480  0
sha512_ssse3           53248  0
snd_intel_sdw_acpi     20480  1 snd_intel_dspcfg
drm_ttm_helper         16384  1 amdgpu
btmtk                  16384  1 btusb
ttm                   110592  2 amdgpu,drm_ttm_helper
mac80211             1617920  3 mt76,mt7921_common,mt76_connac_lib
snd_rawmidi            53248  1 snd_seq_midi
snd_rpl_pci_acp6x      20480  0
aesni_intel           397312  8
snd_hda_codec         204800  4 snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec_realtek
snd_acp_pci            16384  0
bluetooth            1036288  36 btrtl,btmtk,btintel,btbcm,bnep,btusb,rfcomm
drm_display_helper    212992  1 amdgpu
snd_seq                94208  9 snd_seq_midi,snd_seq_midi_event,snd_seq_dummy
snd_hda_core          139264  5 snd_hda_codec_generic,snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec,snd_hda_codec_realtek
crypto_simd            20480  1 aesni_intel
cryptd                 28672  3 crypto_simd,ghash_clmulni_intel
snd_pci_acp6x          20480  0
snd_hwdep              20480  1 snd_hda_codec
cec                    94208  1 drm_display_helper
ecdh_generic           16384  2 bluetooth
ecc                    45056  1 ecdh_generic
rc_core                77824  1 cec
snd_seq_device         16384  3 snd_seq,snd_seq_midi,snd_rawmidi
serio_raw              20480  0
snd_pcm               192512  13 snd_sof_amd_acp,snd_hda_codec_hdmi,snd_pci_acp6x,snd_hda_intel,snd_hda_codec,snd_sof,snd_acp6x_pdm_dma,snd_compress,snd_soc_core,snd_sof_utils,snd_hda_core,snd_pci_ps,snd_pcm_dmaengine
rapl                   20480  0
cfg80211             1241088  4 mt76,mac80211,mt7921_common,mt76_connac_lib
snd_timer              49152  3 snd_seq,snd_hrtimer,snd_pcm
asus_nb_wmi            28672  0
wmi_bmof               16384  0
drm_kms_helper        249856  5 drm_display_helper,amdgpu
snd_pci_acp5x          20480  0
snd_rn_pci_acp3x       20480  0
snd                   135168  22 snd_hda_codec_generic,snd_seq,snd_seq_device,snd_hda_codec_hdmi,snd_hwdep,snd_hda_intel,snd_hda_codec,snd_hda_codec_realtek,snd_sof,snd_timer,snd_compress,snd_soc_core,snd_pcm,snd_rawmidi
i2c_algo_bit           16384  1 amdgpu
syscopyarea            16384  1 drm_kms_helper
snd_acp_config         16384  4 snd_rn_pci_acp3x,snd_sof_amd_rembrandt,snd_acp_pci,snd_sof_amd_renoir
ucsi_acpi              16384  0
libarc4                16384  1 mac80211
sysfillrect            20480  1 drm_kms_helper
ccp                   131072  1 kvm_amd
snd_soc_acpi           16384  2 snd_sof_amd_acp,snd_acp_config
typec_ucsi             53248  1 ucsi_acpi
sysimgblt              16384  1 drm_kms_helper
soundcore              16384  1 snd
snd_pci_acp3x          20480  0
k10temp                16384  0
typec                 106496  1 typec_ucsi
amd_pmc                32768  0
hid_multitouch         36864  0
mac_hid                16384  0
msr                    16384  0
parport_pc             53248  0
ppdev                  24576  0
lp                     28672  0
drm                   692224  16 gpu_sched,drm_kms_helper,drm_display_helper,drm_buddy,amdgpu,drm_ttm_helper,ttm
parport                73728  3 parport_pc,lp,ppdev
efi_pstore             16384  0
dmi_sysfs              24576  0
ip_tables              36864  0
x_tables               65536  1 ip_tables
autofs4                57344  2
nvme                   61440  2
mfd_aaeon              16384  0
asus_wmi               73728  2 asus_nb_wmi,mfd_aaeon
nvme_core             204800  3 nvme
r8169                 110592  0
ledtrig_audio          16384  2 snd_hda_codec_generic,asus_wmi
sparse_keymap          16384  1 asus_wmi
hid_generic            16384  0
platform_profile       16384  1 asus_wmi
crc32_pclmul           16384  0
xhci_pci               24576  0
i2c_piix4              28672  0
xhci_pci_renesas       20480  1 xhci_pci
realtek                36864  1
nvme_common            28672  1 nvme_core
i2c_hid_acpi           16384  0
video                  69632  2 asus_wmi,amdgpu
i2c_hid                40960  1 i2c_hid_acpi
wmi                    40960  4 video,asus_wmi,wmi_bmof,mfd_aaeon
hid                   176128  6 i2c_hid,usbhid,hid_multitouch,hid_generic,hid_logitech_dj,hid_logitech_hidpp
Comment 1 seb.j 2023-04-14 16:30:31 UTC
Created attachment 304135 [details]
dmesg output
Comment 2 seb.j 2023-04-14 16:34:32 UTC
grep -A 40 PS2K dsdt.dsl | grep IRQ -A 1

gives me the output:

                IRQNoFlags ()
                    {1}
Comment 3 seb.j 2023-04-14 16:44:10 UTC
Created attachment 304136 [details]
acpi.log
Comment 4 Paul Menzel 2023-04-15 11:33:43 UTC
The ASL for the PS2K device:

```
    Scope (_SB.PCI0)
    {
        Device (PS2K)
        {
            Name (_HID, "MSFT0003")  // _HID: Hardware ID
            Name (_CID, EisaId ("PNP0303") /* IBM Enhanced Keyboard (101/1
02-key, PS/2 Mouse) */)  // _CID: Compatible ID
            Name (_UID, Zero)  // _UID: Unique ID
            Method (_STA, 0, NotSerialized)  // _STA: Status
            {
                Return (0x0F)
            }

            Name (_CRS, ResourceTemplate ()  // _CRS: Current Resource Set
tings
            {
                IO (Decode16,
                    0x0060,             // Range Minimum
                    0x0060,             // Range Maximum
                    0x00,               // Alignment
                    0x01,               // Length
                    )
                IO (Decode16,
                    0x0064,             // Range Minimum
                    0x0064,             // Range Maximum
                    0x00,               // Alignment
                    0x01,               // Length
                    )
                IRQNoFlags ()
                    {1}
            })
            Name (_PRS, ResourceTemplate ()  // _PRS: Possible Resource Settings
            {
                StartDependentFn (0x00, 0x00)
                {
                    IO (Decode16,
                        0x0060,             // Range Minimum
                        0x0060,             // Range Maximum
                        0x00,               // Alignment
                        0x01,               // Length
                        )
                    IO (Decode16,
                        0x0064,             // Range Minimum
                        0x0064,             // Range Maximum
                        0x00,               // Alignment
                        0x01,               // Length
                        )
                    IRQNoFlags ()
                        {1}
                }
                EndDependentFn ()
            })
            Method (_PSW, 1, NotSerialized)  // _PSW: Power State Wake
            {
                KBFG = Arg0
            }
        }

        Scope (\)
        {
            Name (KBFG, One)
        }
    }
```
Comment 5 Paul Menzel 2023-04-15 11:53:19 UTC
Tamin commented in https://bugzilla.kernel.org/show_bug.cgi?id=216158#c83 that it might be a regression by commit 9946e39fe8d0 (ACPI: resource: skip IRQ override on AMD Zen platforms) [1], which was added in Linux 6.0-rc1.

Could you please test Linux 5.20 [2]?

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9946e39fe8d0a5da9eb947d8e40a7ef204ba016e
[2]: https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.19/
Comment 6 seb.j 2023-04-15 12:26:47 UTC
@Tamin,

actually have the notebook now for a month and tried to get this running
initially with ubuntu 22.04 lts and 18.04 lts.
earliest kernel i tried was 5.18, but it neither worked under this one.
That was why i switched back and hoped for kernel 6.2 to fix my issue.
Comment 7 Paul Menzel 2023-04-15 12:33:20 UTC
Just for the record, Linux 6.2.6 logs:

```
[    0.000000] Linux version 6.2.0-20-generic (buildd@lcy02-amd64-035) (x86_64-linux-gnu-gcc-12 (Ubuntu 12.2.0-17ubuntu1) 12.2.0, GNU ld (GNU Binutils for Ubuntu) 2.40) #20-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr  6 07:48:48 UTC 2023 (Ubuntu 6.2.0-20.20-generic 6.2.6)
[…]
[    0.000000] DMI: ASUSTeK COMPUTER INC. ASUS TUF Gaming A16 FA617NS_FA617NS/FA617NS, BIOS FA617NS.306 03/07/2023
[…]
[    0.437446] i8042: PNP: PS/2 Controller [PNP0303:PS2K] at 0x60,0x64 irq 1
[    0.437448] i8042: PNP: PS/2 appears to have AUX port disabled, if this is incorrect please boot with i8042.nopnp
[    0.438665] serio: i8042 KBD port at 0x60,0x64 irq 1
[    0.438721] mousedev: PS/2 mouse device common for all mice
```
Comment 8 seb.j 2023-04-15 13:00:59 UTC
The touchpad, mute microfon button and display brightness controls actually work.
But all other keys don't. 
Should i run kernel 5.19 again and give the outputs again?
Comment 9 seb.j 2023-04-15 13:03:04 UTC
i8042.nopnp
i actually modified this earlier, i think under 22.04, but without any effect.
Comment 10 seb.j 2023-04-15 13:06:38 UTC
Sorry i mean i modified grub to exclude from boot. sorry english is not my first language
Comment 11 seb.j 2023-04-15 14:15:08 UTC
Created attachment 304142 [details]
dmesg 5.19 output
Comment 12 seb.j 2023-04-15 14:16:50 UTC
@Paul
just reinstalled 22.04 and back to kernel 5.19, demsg output is above. Behavior is the same.
Comment 13 seb.j 2023-04-15 14:25:38 UTC
grep -A 40 PS2K dsdt.dsl | grep IRQ -A 1
                IRQNoFlags ()
                    {1}
Comment 14 seb.j 2023-04-15 14:26:04 UTC
Created attachment 304143 [details]
dmesg grep acpi
Comment 15 seb.j 2023-04-15 14:29:48 UTC
Created attachment 304144 [details]
acpi.log kernel 5.19
Comment 16 seb.j 2023-04-15 14:52:40 UTC
@Paul just insalled the kernel from your link
now there is some response on the keyboard, but its uncontrollable (just repeating some letters as h continously and can't type anything anymore)
Comment 17 Tamim Khan 2023-04-15 15:13:25 UTC
Created attachment 304145 [details]
Reenable IRQ override on Asus TUF FA617NS

@sebj: Can you try applying the following patch to 6.2.11+ kernel? It basically re-enables he override that https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9946e39fe8d0a5da9eb947d8e40a7ef204ba016e disabled. 

I took some educated guesses as to the dmi_system_id and irq_override_cmp structure based on your dmesg posts. However, if this doesn't work we can confirm the values to make sure the reason its not working is not just a typo in my patch.

Tamim
Comment 18 seb.j 2023-04-15 15:47:41 UTC
Thanks Tamim,

Will check asap. Just went oustide, but will provide as soon as I'm nack.

Thanks so much for your all support
Comment 19 Tamim Khan 2023-04-15 16:28:18 UTC
No problem. 

Also just so everyone knows, the disable IRQ override patch was unfortunately back ported to both 5.19 [1] and 5.15 [2]. Also based on the package source [3] for the kernel seb.j tried I think that 5.19 kernel might have had it too. As a result, if this is indeed the problem we are going to need to apply a similar patch to those kernels or use an older version of them that doesn't have the patch applied

Tamim 


[1] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/drivers/acpi/resource.c?h=linux-5.19.y&id=37c81d9f1d1b1458894454efcb857f6a769b6bc4
[2] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/drivers/acpi/resource.c?h=v5.15.107&id=3bb12efc5e4d2b237230a4c6919b6fcf81d61190
[3] https://bugs.launchpad.net/ubuntu/+source/linux/5.19.0-38.39
Comment 20 Paul Menzel 2023-04-15 17:10:38 UTC
(In reply to seb.j from comment #12)
> @Paul
> just reinstalled 22.04 and back to kernel 5.19, demsg output is above.
> Behavior is the same.

If you want to avoid the rebuild to try Tamim’s patch, please use 5.19 (.0) proper. The test you did with 22.04, used Linux 5.19.17, which had the problematic patch backported. (Big thanks to Tamim for spotting that.)

    [    0.000000] Linux version 5.19.0-38-generic (buildd@lcy02-amd64-001) (x86_64-linux-gnu-gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #39~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 17 21:16:15 UTC 2 (Ubuntu 5.19.0-38.39~22.04.1-generic 5.19.17)

But, if you know how to build a Linux kernel [1], trying Tamim’s patch is easiest.


[1]: https://bugzilla.kernel.org/show_bug.cgi?id=217323#c5
Comment 21 seb.j 2023-04-16 03:29:40 UTC
@paul 
tried the patch again, but rn into some issue, that i couldn't identify
Comment 22 seb.j 2023-04-16 03:30:08 UTC
Created attachment 304146 [details]
git error
Comment 23 seb.j 2023-04-16 05:12:49 UTC
I now try to patch the kernal with tamim's patch and it's running now (after a few tries without the right environment).
It's the first time, that i compile a kernel... I'm normally just a normal user, so please forgive me in case i do some stupid error here, I will follow your explanations as good as i can
Comment 24 Paul Menzel 2023-04-16 06:37:00 UTC
(In reply to seb.j from comment #21)
> @paul 
> tried the patch again, but run into some issue, that i couldn't identify

Ah, sorry for not explaining further:

The `wget` line downloads the patch for the problematic device in the bug 217323. Instead download and save Tamim’s patch with the browser:

    $ git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
    $ cd linux
    $ # download and save Tamim’s patch
    $ patch -p1 < /path/to/the/patch/from/tamim/asus-tuf-6.2.patch
    $ git commit -a -m "Re-enable IRQ for Asus Tuf,
    $ make localmodconfig
    $ make bindeb-pkg # install missing program X with `sudo apt install X`
    $ sudo dpkg -i ../linux-image-*.deb

Hopefully that works.
Comment 25 seb.j 2023-04-16 07:01:22 UTC
Hey Paul,

Thanks so much for the additional help and explanation.
Will try that asap (currenlty outside again).
I don't want to trash to much here in the thread since i'm total new to all of this.
Comment 26 seb.j 2023-04-16 11:03:50 UTC
Actually, when i build the kernel with patch, the config making worked for me, but when i apply it with Paul's method, I get questions for the config

  64bit/32bit division and modulo test (TEST_DIV64) [N/m/y/?] n

sorry to ask, but where do i get the infos for this?
Comment 27 Tamim Khan 2023-04-16 13:57:49 UTC
I think the default value here is fine. That would be the option with the capital letter (ie: `N` in this case) or just pressing enter will select it too. The reason this is happening is that `make localmodconfig` tries to your current kernel compilation to fill in the values for as much of these as possible but if a newer kernel introduces a new config you will need to still specify a value for it.

For any of the kernels you have compiled with the patch, did a reboot into them fix your keyboard?

Thanks,
Tamim
Comment 28 seb.j 2023-04-16 14:03:19 UTC
Hey Tamim,

Somehow,
I installed accroding to Paul's instructions, while compiling of the kernel by myself actually took very long and wasn't finished, when i came back.

My keyboard responds now, but in a very strange way.
Gives wrong key outputs, freezes sometimes and if i press keys, i will stuck on some letter and repeat it almost endlessly (until pressing oter keys, han i will repeat other letters, but all are the wrong ones, (n is like r etc).
i have the normal std. us layout in settings.
should i give another dmesg output?
Comment 29 seb.j 2023-04-16 14:06:05 UTC
kernel is now
6.3.0-rc6+
Comment 30 seb.j 2023-04-16 14:09:12 UTC
Created attachment 304147 [details]
patched dmesg output
Comment 31 seb.j 2023-04-16 14:10:51 UTC
In the past, I had actually already tried your older patch for the expert book and the behaviour was very similar.
Comment 32 Tamim Khan 2023-04-16 15:32:43 UTC
@seb.j: I looked into your dmesg and it does look like its now overriding the IRQ now, so there doesn't seem to be a typo in the patch:

[    0.315700] ACPI: IRQ 1 override to edge, low(!)

However, your keyboard is not working so I think there is another problem. Can you try reverting the patch and running the stock 6.3.0-rc6 kernel and let us know how the keyboard behaves? You can do that by:

    $ cd linux # go to directory where you compiled the kernel previously
    $ patch -R -p1 < /path/to/the/patch/from/tamim/asus-tuf-6.2.patch
    $ git commit -a -m "Revert IRQ override for Asus TUF"
    $ make localmodconfig
    $ make bindeb-pkg # install missing program X with `sudo apt install X`
    $ sudo dpkg -i ../linux-image-*.deb

I am just trying to see if our patch made the keyboard at least register input and we need to change other things or if the patch didn't help at all.

Thanks,
Tamim
Comment 33 Tamim Khan 2023-04-16 15:34:44 UTC
Also for the expertbook patch you tried, which kernel did you do that on? The same we are testing with now (ie: 6.3.0-rc6+)?

Tamim
Comment 34 seb.j 2023-04-16 16:32:27 UTC
Hey Tamim,

Just running the unpatching now.
I think the expert patch i've tried on the kernel delivered with 22.04 (5.19).

BTW, thanks for all the help here
Comment 35 seb.j 2023-04-16 16:40:37 UTC
With the patch applied, there seems to be feedback on he keyboard, but also a lot of ghost typing.
Had to go back to 6.2 to be able to work on the laptop again.
Comment 36 Tamim Khan 2023-04-16 16:54:27 UTC
So basically the same with the patch reverted? If possible could you post the dmesg of the stock kernel too for comparison purposes?

Tamim
Comment 37 seb.j 2023-04-16 17:16:07 UTC
Created attachment 304148 [details]
dmesg 6.3 patch removed
Comment 38 seb.j 2023-04-16 17:17:00 UTC
You mean this one right?
Comment 39 seb.j 2023-04-16 17:20:12 UTC
Keyboard is now again fully unresponsive
Comment 40 seb.j 2023-04-16 17:35:00 UTC
Created attachment 304149 [details]
kernel 6.2.11 no patch
Comment 41 Tamim Khan 2023-04-16 19:00:21 UTC
I was reading the original thread [1] where the Ryzen patch was submitted and it seems like with the patch I proposed above we are basically back to the old behavior (ie: ghost typing, keyboard typing delay etc). So unfortunately even though our patch is making the keyboard do "something" I don't think its helping.

Can you try adding the kernel command line flag i8042.debug to the kernel command line on the unpatched 6.3 kernel and attach the dmesg? Instructions on doing that in ubuntu can be found on https://wiki.ubuntu.com/Kernel/KernelBootParameters. Feel free to add either temporarily or permanently. Hopefully with that we will have clues as to what to look at next since it looks like my original idea about the regression is incorrect.

Tamim 

[1] https://bbs.archlinux.org/viewtopic.php?id=277260
Comment 42 seb.j 2023-04-17 00:59:24 UTC
Created attachment 304150 [details]
added debug to grub
Comment 43 seb.j 2023-04-17 01:01:50 UTC
Hey Tamim,

Added the debug and dmesg is above
Comment 44 Tamim Khan 2023-04-17 02:23:05 UTC
Created attachment 304151 [details]
Test patch to force IRQ 1 to level high

@seb.j: Comparing your i8042 debug output to https://bbs.archlinux.org/viewtopic.php?pid=2041563#p2041563 it seems like your keyboard is behaving pretty much identically to other Ryzen laptops before the IRQ override disabling patch was added even though this patch is in the 6.3 kernel you are running.

Due to that, I suspect the problem is still IRQ setup related. As a result, taking inspiration from https://bugzilla.kernel.org/show_bug.cgi?id=213031#c13, I have attached two patches, one that sets the IRQ up as level_high and another that sets it up as level_low to see if either of these options gets your keyboard to work. Unfortunately, we wont be able to merge these into the kernel but if either works we can hopefully adapt them to something more generic that can be merged. 

In order to apply them, you should be able to use the instructions mentioned previously [1] expect without the `git clone`. If the first patch doesn't work though make sure to revert it though with `patch -R -p1 < /path/to/first/patch` before you do the `patch -p1 < /patch/to/second/patch`.

Let me know if you have any questions.

Tamim

[1] https://bugzilla.kernel.org/show_bug.cgi?id=217336#c24
Comment 45 Tamim Khan 2023-04-17 02:25:22 UTC
Created attachment 304152 [details]
Test patch to force IRQ 1 to level low
Comment 46 seb.j 2023-04-17 05:24:16 UTC
Hey Tamim,

Thanks for this.
Running the patching now, but will be out for work again soon. Will let you know tonight, if one of those did it.
Comment 47 seb.j 2023-04-17 05:40:01 UTC
Created attachment 304153 [details]
kernel_high_patch_dmesg_output
Comment 48 seb.j 2023-04-17 05:40:45 UTC
high patch output, keyboard isn't responisve
will test the low patch now
Comment 49 seb.j 2023-04-17 08:25:09 UTC
Created attachment 304154 [details]
dmesg 6.3 patch lo
Comment 50 seb.j 2023-04-17 08:25:53 UTC
ddddddddddddddddddddddddddddddddddddddddddddddddeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeesssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaahhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhgggggggggggggggggggggggggggggggggggggggggggwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww
Comment 51 seb.j 2023-04-17 08:39:25 UTC
sorry for above, but this is what i got in ghost inputs, after opening libre office. interestingly it also disabled some keys on my usb connected keyboard (t and w, i saw), wiht ht elow patch applied. overall it actually behaves very similar to the ryzen patch
Comment 52 Paul Menzel 2023-04-17 09:05:13 UTC
Adding Hui Wang to Cc:. Hui, this problem is happening also with Ubuntu GNU/Linux.
Comment 53 Tamim Khan 2023-04-17 11:17:33 UTC
Unfortunately, at this point I don't really have any good ideas. We have tried all 4 setup options and none of them gotten us anything that works:

* EDGE_HIGH: Default unpatched kernel. Keyboard was unresponsive
* EDGE_LOW: First patch to re-enable override. Keyboard producing ghost typing
* LEVEL_HIGH: test-asus-tuf-level-high.patch. Keyboard was unresponsive
* LEVEL_LOW: test-asus-tuf-level-low.patch. Keyboard producing ghost typing

Interestingly, based on the i8042.debug dmesg output it looks like the LOW modes have a more complete i8042 setup but a lot of ghost typing when compared to the HIGH modes where it seems like it just hangs. We could try the low mode (ie: the originally proposed patch) and some i8042 options like `i8042.dumbkbd`, `i8042.noaux` and `i8042.nomux` independently and see if that does anything. Not sure if that will help anything though.

However, I am hoping @Hui Wang has some better ideas as to what to try or look into since he was the originally the one that patched this issue on the original mideon laptop that was impacted.

Tamim
Comment 54 seb.j 2023-04-17 11:33:20 UTC
Thanks for all the help Tamim and Paul,

At least we got something. I will get v
back to request asus one more time, to support on this. Actually they had advertised linux compatibility  in my region for this model (that was actually the selling point to me).
Maybe they can give at least some hint, what we have to improve here.
Comment 55 Paul Menzel 2023-04-18 10:57:54 UTC
One more thing, does the keyboard work correctly with a non-GNU/Linux like *BSD or Microsoft Windows?
Comment 56 seb.j 2023-04-19 00:11:10 UTC
Dear Paul,

Yes, the laptop actually arrived with Windows 11 and everything worked fine.
The problem only occurs ubder Ubuntu.
Other distro's of ubuntu, actually never had time to reallyy test this.
Comment 57 seb.j 2023-04-20 06:54:42 UTC
Is here eventual a specific question, I should ask Asus, to helpt here forward?
Eventually we will need details on IRQ settings for the keyboard, am I correct here?
Comment 58 Paul Menzel 2023-04-20 06:59:47 UTC
Sorry, no idea. Did ASUS already tell you what system firmware version and GNU/Linux distribution they tested with?

It might also be a good idea to create an issue in Canonical’s/Ubuntu’s issue/bug tracker Launchpad, and set the metadata there to this upstream report.
Comment 59 seb.j 2023-04-20 09:06:30 UTC
Actually didn't ask this initially.
Thanks for the hints and will let you know if there is any update
Comment 60 seb.j 2023-05-01 06:13:12 UTC
Hey guys,

Just got an update by asus and amd, that they will not support on the issue....

But on the other hand, asus released a new bios update (308) and after i updated and tried the asus patch on kernel 6.3 again and see, there is some movement.

Initially, go he ghost typing issue again, i thn activated "slow keys" "sticky keys" and "bouncy keys" from the accessibility menu (i always did his earleier too, but without any good results)

but this time, i was able to type my password and login wiht enter, even got number response form the numpad (tried in libre office after start).

the input will be hardly delayed and it hangs after a while, bt i'm amazd it is working somehow.

does this give you more info? will attach a new dmesg, when back form work today
Comment 61 seb.j 2023-05-01 08:16:57 UTC
Created attachment 304198 [details]
bios 308 dmesg asus patch applied
Comment 62 seb.j 2023-05-04 17:00:06 UTC
could it eventually be, that the acpi usb hub config error is connected to this?
i assum the keyboard is connected on that bus.
Comment 63 Tamim Khan 2023-05-07 23:47:07 UTC
Wow, yah that bios must have changed something with their setup of the keyboard. The kernels communication with the keyboard as outputted by `i8042.debug` actually has the keyboard responding back rather then just ignoring the kernel (you can see it too by downloading your dmesg attachment and then running `grep i8042 path_to_dmesg_file.txt`). At this point, since you are much closer to something working it would be good to try the kernel options `i8042.dumbkbd`, `i8042.noaux` and `i8042.nomux` individually in various combinations to see if we can get it to work reliably. Hopefully, that gives as idea about what is really broken but if not at least you have an option to get your keyboard working for now.

> could it eventually be, that the acpi usb hub config error is connected to
> this?
> i assum the keyboard is connected on that bus.

Unfortunately, I think if the i8042 driver is detecting it then it should be using the PS/2 bus and not USB. However, if the keyboard is meant to use USB but is being picked up by i8042 then that would definitely explain why its behaving so badly. 

One other thing that would be good to look at as well just in case the bios changed it is the the value of PS2K in the DSDT table. You can obtain that doing: 

cat /sys/firmware/acpi/tables/DSDT > dsdt.dat
iasl -d dsdt.dat
grep -A 40 PS2K dsdt.dsl

Thanks,
Tamim
Comment 64 seb.j 2023-05-08 19:44:51 UTC
Hey Tamim, 

Thanks for the help again.

output of the commands as below:



sudo grep -A 40 PS2K dsdt.dsl
        Device (PS2K)
        {
            Name (_HID, "MSFT0003")  // _HID: Hardware ID
            Name (_CID, EisaId ("PNP0303") /* IBM Enhanced Keyboard (101/102-key, PS/2 Mouse) */)  // _CID: Compatible ID
            Name (_UID, Zero)  // _UID: Unique ID
            Method (_STA, 0, NotSerialized)  // _STA: Status
            {
                Return (0x0F)
            }

            Name (_CRS, ResourceTemplate ()  // _CRS: Current Resource Settings
            {
                IO (Decode16,
                    0x0060,             // Range Minimum
                    0x0060,             // Range Maximum
                    0x00,               // Alignment
                    0x01,               // Length
                    )
                IO (Decode16,
                    0x0064,             // Range Minimum
                    0x0064,             // Range Maximum
                    0x00,               // Alignment
                    0x01,               // Length
                    )
                IRQNoFlags ()
                    {1}
            })
            Name (_PRS, ResourceTemplate ()  // _PRS: Possible Resource Settings
            {
                StartDependentFn (0x00, 0x00)
                {
                    IO (Decode16,
                        0x0060,             // Range Minimum
                        0x0060,             // Range Maximum
                        0x00,               // Alignment
                        0x01,               // Length
                        )
                    IO (Decode16,
                        0x0064,             // Range Minimum
                        0x0064,             // Range Maximum
                        0x00,               // Alignment
Comment 65 seb.j 2023-05-08 19:46:44 UTC
Actually I had updated the bios to now version 403 (last one with working keyboard was 308)

and now funnily it's totally unresponsive again
Comment 66 Paul Menzel 2023-05-08 20:04:21 UTC
> Actually I had updated the bios to now version 403 (last one with working
> keyboard was 308)

Do you mean with *partially* working keyboard? (Or did I miss something, that it worked fine?)
Comment 67 seb.j 2023-05-08 20:27:09 UTC
No you got that right. was partially working (initial bootup to some time after that)
under grub, it actually works fine too
Comment 68 seb.j 2023-05-08 20:43:29 UTC
Hey Paul,

Sorry that with not working again, seemed to be my fault.
Had run update and updated the kernel.
Just swapped back to the asus patch kernel and it works again.
Just logged in with the keyboard on the laptop, but it hangs after some time again and repeats letters when typing.

Will try the options given by Tamim and feedback again with demsg
Comment 69 Tamim Khan 2023-05-09 13:38:26 UTC
Would it also be possible for you attach the dmesg or `dmesg | grep i8042` right after the keyboard dies after its been working for a while? We can probably use the i8042 logs to figure out how the communication is breaking down and try to patch that.

Tamim
Comment 70 seb.j 2023-05-09 15:57:21 UTC
Created attachment 304237 [details]
dmesg with dumbkbd
Comment 71 seb.j 2023-05-09 16:01:42 UTC
Hey Tamim,

Actually will not be possible, already tried, but also my external keyboard stops working somehow, after the internal trips.
I tried by now each i80442 option seprately and get the dmesg directly after boot.
All keys are now the right ones when i can type with the nternal one, before they were randomly worng.

but the repea error keeps persistent for me, after shortly orking, the internal one will let me write like this:
example word enter looks when typing like this:

eeeeeeeeeeeeennnnnnnnnnnnnnnnnnnnnttttttttttttttttttttttttttteeeeeeeeeeeeeeeeerrrrrrrrrr

Will continue with the diferent possible combinations and upload further.
I alos loaded openSUSE and will try ton run it asap.
Comment 72 seb.j 2023-05-09 16:02:07 UTC
Created attachment 304238 [details]
dmesg noaux option
Comment 73 seb.j 2023-05-09 16:02:35 UTC
Created attachment 304239 [details]
dmesg nomux option
Comment 74 seb.j 2023-05-09 16:05:02 UTC
The error above happens also after i deactivated repeat keys in accessibility menu
Comment 75 seb.j 2023-05-10 05:48:39 UTC
I tired opensuse yesterday evening, but the behavior is similar.
the keyboard works, for selecting installation options (i think it's under yast), but as soon as i come to the graphical installation menu, the keyboard isn't working anymore.
Comment 76 seb.j 2023-05-10 07:17:52 UTC
for suse also found a similar thread
https://lists.opensuse.org/archives/list/bugs@lists.opensuse.org/thread/O7JCC6R7ZQDTOOE6D7K42IFKAIN3J257/

Seems the combination of asus and ryzen is very bad for the linux world.

i installed windows again, just to try if something else broke eventually during all the installations etc.

but in windows all keys are working fine without any issues.
Comment 77 seb.j 2023-05-10 16:49:19 UTC
Ok guys,

Thanks Tamim and Paul for all the work here.
I will exchange the laptop with one of my workers, since he has almost the same model, but with intel chipset and geforce graphics.
I actually need the unit for work so might be better that way.

Let me know, if I can donate you both something?
Comment 78 Nathan Schulte 2023-05-12 16:17:15 UTC
I am experiencing what seems to be this same issue on an ASUS FA617XT: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1036004

Ubuntu 23.04LTS (kernel 6.2), Debian Bookworm RC2 (kernel 6.1 and experimental kernel 6.3), seem to behave as though the keyboard is entirely inoperable, though it seemed just extremely broken and sometimes inputs would occur and get stuck: generally though, it simply does not work, no input.

With older kernels, like Ubuntu 22.04LTS (kernel 5.19?) or Debian Stable (kernel 5.18?), the behavior is also bad: keys _might_ input and then get "stuck": essentially unusable.

The machine shipped with BIOS version FA617XT.303, and updating to the FA617XT.307 (mid April), does not resolve this issue.

I hope to try with builds of kernel 6.3.2, though I expect this won't resolve anything given the comments here.

Please let me know what I can do to help diagnose this issue: I don't believe ASUS nor AMD would be happy to simply have these machines non-functional with Linux for such a seemingly simple protocol/signalling issue.

As well, for me, kernels less than v6.3 suffer from kernel warning traces/taints from amdgpu; nomodeset option was the only way in Debian to achieve a booting Linux; Ubuntu seems to behave better and allow booting the system when it failed to init the GPUs (Ryzen 780M and Radeon RX 7700S), but Debian would simply reset the system during boot (or sometimes hang the system at a black screen, as though the reset failed): https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1035971
Comment 79 Julien 2023-05-13 21:04:56 UTC
Hello, I am having the same issue and symptoms here on ASUS TUF FA507NU (AMD 7735HS/RTX4050): https://bugzilla.suse.com/show_bug.cgi?id=1211380#c0.

Kernel 6.3.1 and latest bios doesn't change anything for me on Tumbleweed. I will also be more than happy to do any tests to help fix this issue.
Comment 80 Paul Menzel 2023-05-14 07:08:26 UTC
Nathan, Julien, please also contact ASUS about this issue. Maybe Seb can share a ticket number, you can reference.
Comment 81 Julien 2023-05-14 07:46:21 UTC
I contacted them and link to this bug. I am waiting for their reply.
Comment 82 Nathan Schulte 2023-05-15 20:21:48 UTC
Created attachment 304271 [details]
ASUS Live Chat support

part 1/2 from ASUS Live Chat support about the issue -- expectedly unfruitful

> ...
> Kevin_L.: I do apologize as we do not assist with the Linux operating system.
> Me: I understand, but that's a disingenuous position: this issue is about
> supporting the hardware.
> Me: There's no one else you can refer me to for assistance?
> Kevin_L.: No I do apologize, we do not assist with the Linux operating
> system.
> ...
> Kevin_L.: Your case number for future reference is: N2305063195-0001 We do
> appreciate your understanding and we do value your business, thank you for
> being a part of the ASUS family.
Comment 83 Nathan Schulte 2023-05-15 20:22:26 UTC
Created attachment 304272 [details]
ASUS Live Chat support

part 2/2 from ASUS Live Chat support about the issue -- expectedly unfruitful
Comment 84 Nathan Schulte 2023-05-17 23:15:34 UTC
Created attachment 304286 [details]
Debian 6.3.0-0 amd64 i8042 option variations logs

Attached is a tarball of kernel logs and DSDT output, from Debian's experimental 6.3.0-0-amd64 kernel, with i8042.debug + i8042.dumbkbd/i8042.noaux/i8042.nomux permutations on an FA617XT w/ .307 BIOS version.  The kernel is booted with nomodeset because amdgpu also outputs IRQ warning traces when the GPUs are initialized; perhaps this is interesting.

I wish I had checked Windows 11 Home to see how the keyboard works; if it does and how it is attached (USB vs PS/2).
Comment 85 seb.j 2023-05-18 15:03:03 UTC
@Nathan
Yeah Asus feedback is indeed very nice. Had the same and I actually had gotten the account from my dealer (sells mainly Asus products).
I thought from his account it might have more weight, but I've got the same feedback.
I would try Win 11 just to check everything's works fine over there.
Actually reinstalled it for the guy I made the exchange with and all was good. Never had any issue with Keyboard there.
Does your's work in grub?
There is also a keyboard test tool in the bios actually.
Comment 86 Paul Menzel 2023-05-19 06:38:04 UTC
Nathan, Julien, despite this probably being a Linux kernel shortcoming, I still recommend to return the device to show Asus, that they can’t sell these, when they do not verify that the Linux kernel works. (Especially when they lie about working GNU/Linux support.)
Comment 87 Nathan Schulte 2023-05-19 16:38:06 UTC
(In reply to seb.j from comment #85)
> Does your's work in grub?
> There is also a keyboard test tool in the bios actually.

Yes, the keyboard works well in GRUB, and presumably in the BIOS hardware test tool.

I wonder how it behaves on e.g. Windows 8, 10, vs the 11 it shipped with.

(In reply to Paul Menzel from comment #86)
> Nathan, Julien, despite this probably being a Linux kernel shortcoming, I
> still recommend to return the device to show Asus, that they can’t sell
> these, when they do not verify that the Linux kernel works. (Especially when
> they lie about working GNU/Linux support.)

Where did ASUS claim working GNU/Linux support?  I didn't see this, but instead simply that some models could be purchased without an operating system installed: note this was not an option for the FA617XT model in the US, just one configuration is/was available.
Comment 88 seb.j 2023-05-20 03:18:43 UTC
@Nathan,

I think Asus not really advertises this, but when I asked my dealer, he actually gave me the laptop as option (FA617NS) and told me it would be compatible.
I think that was just a failure by the dealer, while when I checked later, Asus also refused to support.
My dealer would have exchanged the unit, but since I bought a similar unit for one of my workers, that wasn't necesssary anymore (since he sticks to windows, i just exchanged the units and prepared the old one for him).

Long story short, my dealer advertised it as Linux compatible (i live in Thailand / not sure if that makes a difference)
Comment 89 Paul Menzel 2023-05-20 07:30:42 UTC
On page 100 of the English manual edition it says:

> NOTE: Energy Star is NOT supported on FreeDOS and Linux-based operating
> systems

But otherwise it’s indeed not mentioned for the FA617XT. One reason more to not support such a vendor and return the device. (Sorry for going off-topic.)

If you still want to keep that device my last suggestion for this thread is to contact linux-input@vger.kernel.org and the maintainer (see `MAINTAINERS`) and make them aware of the problem. Please also carbon-copy Greg Kroah-Hartman, LKML, and the ACPI mailing list. I think, the INPUT subsystem does not use the Linux kernel Bugzilla bug tracker.
Comment 90 drich 2023-05-27 12:22:18 UTC
Hello, this kernel patch fixes the issue : https://lore.kernel.org/linux-acpi/20230518183920.93472-1-mario.limonciello@amd.com/
Comment 91 Nathan Schulte 2023-05-28 09:00:19 UTC
(In reply to drich from comment #90)
> Hello, this kernel patch fixes the issue :
> https://lore.kernel.org/linux-acpi/20230518183920.93472-1-mario.
> limonciello@amd.com/

I was able to test this patch against v6.3 in Debian.  It does not resolve the issue entirely for me; it does make the keyboard _slightly_ more functional: I can type my username at the prompt and perhaps 60-120 seconds later the characters will appear to register.  This seems like the same behavior w/ the 5.x kernels before the set of patches noted above were applied.
Comment 92 drich 2023-05-28 09:18:12 UTC
Happened to me also, didn't know if it was down to my setup or not.

I found a workaround :
- open root shell
- go to /sys/bus/platform/devices and search for ITE5570:00 in the AMDI00**/i2c-* folders (mine was in AMDI0010:01)
- run the following command : echo 'AMDI0010:01' > /sys/bus/platform/drivers/i2c_designware/unbind (change AMDI0010:01 for the one you found)

This should just fix the issue without side effect (still don't know what this ITE5570:00 is, its recognized as a generic-HID by kernel, and as a keyboard by libinput).

I created a oneshot systemd service to run it on boot as WantedBy multi-user.target
Comment 93 Paul Menzel 2023-05-28 10:31:48 UTC
As this issue has a lot of comments, and is not only about the ** anymore, please always list your device.

1.  Nathan has a ASUS FA617XT.
2.  drich, what device do you have?

drich, thank you very much for your workaround. What distribution and Linux version did you test it with?
Comment 94 drich 2023-05-28 10:41:55 UTC
Oh yeah sorry, here are my system infos :

System:
  Host: laptop
  Kernel: 6.4.0-rc3-amd64+ (custom build)
  Cmdline: BOOT_IMAGE=/boot/vmlinuz-6.4.0-rc3-amd64+ root=UUID=9012f2a0-b3f6-49eb-ae42-a08feded24dc ro (default one)
  Distro: Debian GNU/Linux 12 (bookworm)
Machine:
  Type: Laptop
  System: ASUSTeK
  Product: ASUS TUF Gaming A16 FA617XS_TUF617XS v: 1.0
  Mobo: ASUSTeK FA617XS v: 1.0 serial: C5W33M100AY
  UEFI: American Megatrends LLC. v: FA617XS.303 date: 03/23/2023
Comment 95 Nathan Schulte 2023-05-28 17:17:53 UTC
(In reply to drich from comment #92)
> Happened to me also, didn't know if it was down to my setup or not.
> 
> I found a workaround :
> - open root shell
> - go to /sys/bus/platform/devices and search for ITE5570:00 in the
> AMDI00**/i2c-* folders (mine was in AMDI0010:01)
> - run the following command : echo 'AMDI0010:01' >
> /sys/bus/platform/drivers/i2c_designware/unbind (change AMDI0010:01 for the
> one you found)
> 
> This should just fix the issue without side effect (still don't know what
> this ITE5570:00 is, its recognized as a generic-HID by kernel, and as a
> keyboard by libinput).

This, along with the patch you shared, makes the keyboard work on my ASUS FA617XT.  The device is the same: AMDI0010:01.

Thanks.  How did you uncover this?
Comment 96 drich 2023-05-28 17:35:28 UTC
> How did you uncover this?

I had another problem : the keyboard was perfectly working with the patch but the touchpad was not detected at all on my setup, it appeared that I just had the I2C-HID modules disabled in my kernel config.

Once enabled, the keyboard started behaving the same way described in this thread, so at this point I knew it was caused by something on the I2C stack. I then played around with all the I2C peripherals and settings to finally find that AMDI0010:01 / ITE5570:00 was faulty.
Comment 97 Mario Limonciello (AMD) 2023-06-01 22:28:32 UTC
FYI I've posted a v3 of the patch to revert the heuristic.  I don't expect this to help the issue with the i2c-hid version of the keyboard that Nathan and Drich had, but it should help PS/2 keyboards.

https://lore.kernel.org/linux-acpi/20230601221151.670-1-mario.limonciello@amd.com/T/#u
Comment 98 Clay 2023-06-02 02:29:18 UTC
I have an Asus Fa617XT as well and experiencing this same issue and found this that might be helpful regarding this issue: https://lore.kernel.org/linux-input/20230530154058.17594-1-friedrich.vock@gmx.de/T/
Comment 99 Mario Limonciello (AMD) 2023-06-02 04:09:04 UTC
*** Bug 217493 has been marked as a duplicate of this bug. ***
Comment 100 Nathan Schulte 2023-06-03 09:15:24 UTC
(In reply to Mario Limonciello (AMD) from comment #97)
> FYI I've posted a v3 of the patch to revert the heuristic.  I don't expect
> this to help the issue with the i2c-hid version of the keyboard that Nathan
> and Drich had, but it should help PS/2 keyboards.
> 
> https://lore.kernel.org/linux-acpi/20230601221151.670-1-mario.
> limonciello@amd.com/T/#u

v3, w/ the addition to disable the "GPIO 7" / ITE5570 device, continues to work for me on 6.3.5.
Comment 101 marslogout 2023-06-03 16:52:44 UTC
Hello, I have an Asus FA507NV laptop. I installed Linux Mint 21.1 with kernel version 5.15.0-56, and the keyboard works fine. However, after updating the kernel to version 5.15.0-73, it stops working.
Comment 102 Nathan Schulte 2023-06-04 04:13:50 UTC
Update:

My ASUS FA617XT is completely dead, no signs of life: will not power on, does not seem to charge from AC adapter or USB Type-C port; dis/reconnecting the battery has no effect, nor removing the RAM DIMMs or NVMe cards (SSD, WiFi).

I was running v6.4-rc4 w/ v3 patch, and with the ITE 5570 device unbound.  After using the machine (GPU intensive task) and letting it idle for many hours, it suddenly powered off and now appears lifeless, a semi-expensive fancy-looking brick.

I'll be contacting ASUS for warranty support.  I suspect this is simply a hardware issue with my specific device, but wanted to share in case others are running v6.4 and/or the above patches.
Comment 103 seb.j 2023-06-04 06:20:12 UTC
@Nathan

eventually check for thermal management issues, i actually had my laptop ran very hot in the beginning and found that the thermal management wasn't working correctly, but ofr me it was fixed with kernel 6.3.
sorrry didn't keep track of the specific error.
Comment 104 seb.j 2023-06-04 06:22:00 UTC
I think it was related to the amd-gpu error, i had at startup, when now remembering.
eventually your gc burned itself?
Comment 105 drich 2023-06-04 10:15:35 UTC
Using 6.4.y with the patches too 👀 , will avoid GPU intensive tasks for now.

Just installed a second SSD with Windows 11 on it, keyboard works perfectly well on both installer and the installed system.

The keyboard is detected on the same PS/2 interface as Linux, and ITE5570 as a generic "I2C HID Device" (don't know how to gather more precise informations)
Comment 106 Nathan Schulte 2023-06-04 18:07:40 UTC
FWIW, I don't think the issue is GPU related, though it could be.  I may pop out the SSD and try to check the logs; I think the amdgpu_irq_put kernel taint warnings were still occurring but unsure: OpenCL via ROCm was working great, just ran a task for maybe 15m and then left the laptop on afterward.  The fans would throttle up and down with the load.

Thanks for checking in Windows; I was tempted to install a second SSD as well.  I wiped the SSD that came with it; I wonder if it's possible to recover my license or how that works.

I'll be contacting ASUS on Monday about this all.
Comment 107 Mario Limonciello (AMD) 2023-06-05 14:42:34 UTC
> The keyboard is detected on the same PS/2 interface as Linux, and ITE5570 as
> a generic "I2C HID Device" (don't know how to gather more precise
> informations)

Could you 
> I think the amdgpu_irq_put kernel taint warnings were still occurring

The last of these for Phoenix should be fixed in 6.4-rc5.

> I wiped the SSD that came with it; I wonder if it's possible to recover my
> license or how that works.

It should be stored in BIOS and found automatically by the Windows installer.

> The keyboard is detected on the same PS/2 interface as Linux, and ITE5570 as
> a generic "I2C HID Device" (don't know how to gather more precise
> informations)

On 
https://lore.kernel.org/linux-input/20230530154058.17594-1-friedrich.vock@gmx.de/T/ I was looking for a few things.  Most notably:

1) The output from /sys/kernel/debug/gpio
2) Sleuthing to figure out which interrupt is causing the storm in /proc/interrupts, the master interrupt or the one tagged to that GPIO
3) Use R/W everything in Windows to check the GPIO register configuration against Linux.  For that individual register the address will be in /sys/kernel/debug/gpio in Linux and you can use the MMIO function in R/W everything to access it in Windows.
Comment 108 Mario Limonciello (AMD) 2023-06-05 14:43:40 UTC
> Could you 

My apologies, I forgot to delete this block, I completed my thought at the end of the comment.
Comment 109 raphael.alampay 2023-06-05 16:37:53 UTC
I'm having the same problem with marslogout@gmail.com with the same model. There's a post on opensuse.org forum (https://forums.opensuse.org/t/keyboard-not-working-on-laptop-amd-asus-tuf/165741/14) which seems to make it work for them by commenting out some lines in drivers/acpi/resource.c. Maybe this can help?
Comment 110 Mario Limonciello (AMD) 2023-06-05 16:39:22 UTC
> I'm having the same problem with marslogout@gmail.com with the same model.
> There's a post on opensuse.org forum
> (https://forums.opensuse.org/t/keyboard-not-working-on-laptop-amd-asus-tuf/165741/14)
> which seems to make it work for them by commenting out some lines in
> drivers/acpi/resource.c. Maybe this can help?

This patch which is under testing will fix it.

https://lore.kernel.org/linux-acpi/20230601221151.670-1-mario.limonciello@amd.com/T/#u
Comment 111 raphael.alampay 2023-06-05 16:55:27 UTC
Not too familiar with patching up the kernel myself. Will try to test it over the week. But will this be expected to be in 6.4?
Comment 112 Mario Limonciello (AMD) 2023-06-05 16:56:48 UTC
> Not too familiar with patching up the kernel myself. Will try to test it over
> the week. But will this be expected to be in 6.4?

It depends on how much testing it gets; but I think unlikely.
Comment 113 marslogout 2023-06-05 18:22:46 UTC
(In reply to Mario Limonciello (AMD) from comment #112)
> > Not too familiar with patching up the kernel myself. Will try to test it
> over
> > the week. But will this be expected to be in 6.4?
> 
> It depends on how much testing it gets; but I think unlikely.

So, you mean to say that the more people test this patch, the higher the likelihood that it will be included in the upcoming kernel releases?

Does this patch work for any kernel version or only for specific ones? Can you provide a link to an article that describes the patch installation process step by step? It would be quite difficult for an inexperienced user to do it without proper instructions.
Comment 114 Mario Limonciello (AMD) 2023-06-05 18:25:06 UTC
> So, you mean to say that the more people test this patch, the higher the
> likelihood that it will be included in the upcoming kernel releases?

I explicitly said in the patch it needs testing from a majority of people before it should go in.  It's risky otherwise.

> Does this patch work for any kernel version or only for specific ones? Can
> you provide a link to an article that describes the patch installation
> process step by step? It would be quite difficult for an inexperienced user
> to do it without proper instructions.

Contextually it might change from one version to another depending upon what other commits are there already.

In terms of building a kernel, you should reference your distro for the best directions.  For example Fedora has this:
https://docs.fedoraproject.org/en-US/quick-docs/kernel/build-custom-kernel/
Comment 115 Nathan Schulte 2023-06-05 18:52:55 UTC
I've built with versions of the patch against 6.3, 6.4, and 6.1 IIUC.

On Debian, there is a utility to build a kernel with a set of patches with a single command; this is the guide:

https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#id-1.6.6.4

I had to use `--fuzz 3` due to some slight differences, but it applied cleanly.  The `b4` command (and an existing Git repository) will allow to create a .patch if copy/paste from the mail doesn't work.
Comment 116 raphael.alampay 2023-06-14 14:21:27 UTC
Is this line in drivers/acpi/resource.c really needed? Seems to be present in the stable releases but already removed in linux-next.

#ifdef CONFIG_X86
	/*
	 * IRQ override isn't needed on modern AMD Zen systems and
	 * this override breaks active low IRQs on AMD Ryzen 6000 and
	 * newer systems. Skip it.
	 */
	if (boot_cpu_has(X86_FEATURE_ZEN))
		return false;
#endif
Comment 117 Nik_P 2023-06-18 20:17:55 UTC
(In reply to Mario Limonciello (AMD) from comment #114)
> > So, you mean to say that the more people test this patch, the higher the
> > likelihood that it will be included in the upcoming kernel releases?
> 
> I explicitly said in the patch it needs testing from a majority of people
> before it should go in.  It's risky otherwise.

Hello,
just wanted to confirm the latest version of your patch working with the Asus FA617XS variant, together with the I2C unbind fix on the Ubuntu 23.04&kernel 6.4-rc6.
Touchpad became unresponsive after moving to this kernel, but the unbind/bind of AMDI0010:00 did the trick.

What I have noticed, when I dropped out to the initramfs shell with the patched kernel, the keyboard was working perfectly even without the I2C unbind. But after I got to the graphics, it had effectively stopped working and needed to be unbound. Not sure if it is relevant, just sharing an observation.
Thank you and everyone else for getting my laptop running!
Comment 118 Mario Limonciello (AMD) 2023-06-19 04:04:27 UTC
(In reply to Nik_P from comment #117)
> (In reply to Mario Limonciello (AMD) from comment #114)
> > > So, you mean to say that the more people test this patch, the higher the
> > > likelihood that it will be included in the upcoming kernel releases?
> > 
> > I explicitly said in the patch it needs testing from a majority of people
> > before it should go in.  It's risky otherwise.
> 
> Hello,
> just wanted to confirm the latest version of your patch working with the
> Asus FA617XS variant, together with the I2C unbind fix on the Ubuntu
> 23.04&kernel 6.4-rc6.
> Touchpad became unresponsive after moving to this kernel, but the
> unbind/bind of AMDI0010:00 did the trick.

Thanks!  It's queued up for 6.5.

> 
> What I have noticed, when I dropped out to the initramfs shell with the
> patched kernel, the keyboard was working perfectly even without the I2C
> unbind. But after I got to the graphics, it had effectively stopped working
> and needed to be unbound. Not sure if it is relevant, just sharing an
> observation.
> Thank you and everyone else for getting my laptop running!

Do you have i2c-hid in your initramfs?  I would think it's not tied to graphics specifically, but rather that i2c-hid is in your rootfs instead of initramfs so issue didn't trigger until i2c-hid loaded.

If possible; do you or anyone else have a dual boot with Windows?  I would really like if you can compare the configuration for the problematic GPIO in Windows (using R/W everything) and Linux (using /sys/kernel/debug/gpio).
Comment 119 marslogout 2023-06-19 06:34:21 UTC
(In reply to Nik_P from comment #117)
> (In reply to Mario Limonciello (AMD) from comment #114)
> > > So, you mean to say that the more people test this patch, the higher the
> > > likelihood that it will be included in the upcoming kernel releases?
> > 
> > I explicitly said in the patch it needs testing from a majority of people
> > before it should go in.  It's risky otherwise.
> 
> Hello,
> just wanted to confirm the latest version of your patch working with the
> Asus FA617XS variant, together with the I2C unbind fix on the Ubuntu
> 23.04&kernel 6.4-rc6.
> Touchpad became unresponsive after moving to this kernel, but the
> unbind/bind of AMDI0010:00 did the trick.
> 
> What I have noticed, when I dropped out to the initramfs shell with the
> patched kernel, the keyboard was working perfectly even without the I2C
> unbind. But after I got to the graphics, it had effectively stopped working
> and needed to be unbound. Not sure if it is relevant, just sharing an
> observation.
> Thank you and everyone else for getting my laptop running!

Hello, I also encountered a similar issue when installing the Nvidia proprietary driver with open-source code. There were no problems when installing the driver with closed-source code, so you can check that. Keep in mind that simply reinstalling the driver doesn't help. I had to reinstall the system to verify this.
Comment 120 Marek Šanta 2023-06-23 08:08:37 UTC
I confirm the issue on FA507NU laptop.
The funny thing is that after installation of current Linux Mint 21.1 with 5.15 Kernel, everything worked fine. But after switching to Kernel 5.19, the keyboard stopped working. 

I have two pieces of the same model - it is the same for both of them.

The discussion is quite confusing for me. What is the current status? What mainline kernel version will have the fix?
Comment 121 Mario Limonciello (AMD) 2023-06-23 19:11:51 UTC
The patch will be landing in kernel 6.5-rc1.  If no other problems are reported we'll intend to bring it back to stable later on as well.
Comment 122 Mario Limonciello (AMD) 2023-06-23 19:12:46 UTC
> The discussion is quite confusing for me. What is the current status? What
> mainline kernel version will have the fix?

There is a separate issue going on where an I2C device is causing an interrupt storm which makes the keyboard seem like it's not working.  Still looking for debugging information on that to come up with a proper solution.
Comment 123 Marek Šanta 2023-06-26 12:28:23 UTC
Thanks, Mario.
I thereby confirm that your patch applied on latest v6.4 tag works on FA507NU.

Apart from a few ACPI bugs in dmesg (I have never ever seen a system without some) I had no problems so far. Is there anything I should focus on during my testing?
Comment 124 Mario Limonciello (AMD) 2023-06-26 13:26:01 UTC
> Is there anything I should focus on during my testing?

If you are affected by the interrupt storm, I would like to see a comparison of registers from R/W everything.
Comment 125 drich 2023-06-26 13:43:38 UTC
Created attachment 304477 [details]
/sys/kernel/debug/gpio

Here is the content of my /sys/kernel/debug/gpio "file", but I don't understand how R/W Everything works. How should I set it up to get the GPIO output from Windows side ?
Comment 126 Mario Limonciello (AMD) 2023-06-26 14:18:37 UTC
For using R/W everything in Windows you will use the "MMIO" function.  
1. Press the MMIO button and then hit the "+" button.
2. Add a mapping for the base address (FED81500) and save it.
3. Add line items for all the relevant GPIO registers and their offsets from that base address.

For example if you wanted to look at pin 7, it should be 1C.
The master register is FC (See https://github.com/torvalds/linux/blob/master/drivers/pinctrl/pinctrl-amd.h#L19).

At a minimum we want to match the values in R/W everything against those in /sys/kernel/debug/gpio for:
1. the GPIO that causes the storm
2. the power button interrupt (offset 0)
3. the master controller register (offset (FC)

The names aren't exactly the same platform but as it's the most open document I know of and the offsets don't change you can reference https://www.amd.com/system/files/TechDocs/50742_15h_Models_60h-6Fh_BKDG.pdf to try to find the offsets if you have a hard time for your problematic GPIO.
Comment 127 Nathan Schulte 2023-06-27 09:39:22 UTC
Created attachment 304483 [details]
FA617XT.309 gpio, interrupts, RWEverything-mmio

Attached are `/proc/interrupts`, `/sys/kernel/debug/gpio`, and the output from RW Everything utility, from my repaired ASUS FA617XT, which "conveniently" came back with Windows 11; the captures are from Ubuntu 23.04 install/live CD.

I'm not sure which interrupt is the noisy one, so I captured as much as I could: please let me know if this captures what's necessary, or what else I should look at.  Without your (Mario) notes about the GPIO base address, I wouldn't have been able to get this.  I captured banks 0, 1, and 2, though `.../gpio` lists a third bank not listed in "BKDG for AMD Family 15h Models 60h-6Fh Processors."

----

For those interested, the repair sheet said they replaced the MAIN BOARD ("no power on (LED indicators did not light up)") and the KEYBOARD MODULE ("certain key pressed and triggered multiple response").  I don't think I took good enough notes to tell if anything else was replaced or if any of this is original (beside the S/N / sticker).

Also, I was able to capture the kernel log from when the system turned into a brick, and there's some interesting output from amdgpu; as well it almost seems like systemd restarted or the system tried to reinitialize without power cycling somehow.  I'll share them later if anyone is interested, or can help assure me it won't happen again due to some misconfiguration or use ...
Comment 128 Nathan Schulte 2023-06-27 13:30:51 UTC
Created attachment 304485 [details]
PDF of GPIO register configuration; Win 11 vs Linux

I did some analysis on the captured values (FA617XT.309); see the last page for comparison: non-zero values are differences (simple subtraction: red lettering indicating the Win 11 value is "larger" -- I'm aware decimal comparison isn't ideal ...).

Hopefully this helps; there are a number of registers that are configured differently.
Comment 129 Friedrich Vock 2023-06-27 14:34:27 UTC
Hi,

FA617NS here, and some registers are also configured differently here. I *think* the GPIO pin that causes the storm might be pin 7, so I captured that pin, pin 0 and the master controller register as well as some random others in R-W everything and on Linux. It looks something like this:

pin 0: 805F800 on Windows, 815F8E3 on Linux. Windows clears bit 20 and the whole first byte, Linux sets bit 20 and some of the first byte.

pin 1: 150000 on both Windows and Linux

pin 2: 15EB00 on both Windows and Linux

pin 3: 150000 on both Windows and Linux

pin 4: 41940 on Windows, 247C00 on Linux. Different bits are bits 21, 14, 13, 8 and 6.
Windows clears bits 21, 14 and 13 while Linux sets them, and Linux clears bits 8 and 6 while Windows sets them.

pin 5: 240000 on both Windows and Linux

pin 6: 240000 on both Windows and Linux

pin 7: 151B00 on Windows, 10040B00 on Linux. Different bits are bits 28, 20, 16 and 12.
Windows clears bit 28 while Linux sets it, and Linux clears bits 20, 16 and 12 while Windows sets them. Bit 28 is interesting, according to the docs, if it is cleared, it doesn't generate interrupts. Sounds like the storm might go away if that bit is cleared?

Master control register: FF00000000 on both Windows and Linux.
Comment 130 Mario Limonciello (AMD) 2023-06-27 17:44:52 UTC
Created attachment 304487 [details]
potential patch (v1)

It looks like pin 7 has the same results for both of you guys; let's focus on that.

> pin 0: 805F800 on Windows, 815F8E3 on Linux. Windows clears bit 20 and the
> whole first byte, Linux sets bit 20 and some of the first byte.

OK I expect pin 0 is closer to the same after the series going to 6.5-rc1 hopefully.  This shouldn't affect this issue.

> pin 7: 151B00 on Windows, 10040B00 on Linux. Different bits are bits 28, 20,
> 16 and 12.
> Windows clears bit 28 while Linux sets it, and Linux clears bits 20, 16 and
> 12 while Windows sets them. Bit 28 is interesting, according to the docs, if
> it is cleared, it doesn't generate interrupts. Sounds like the storm might go
> away if that bit is cleared?

bit 28: Linux does write the GPIO again to clear it.
https://github.com/torvalds/linux/blob/v6.4/drivers/pinctrl/pinctrl-amd.c#L668C18-L668C18

bit 20: This is most concerning to me.  Windows configured the pin as pull up, but Linux didn't.

bit 16: this is read only; just a status bit based on current status.

bit 12: this controls the interrupt delivery.  It appears that interrupt is enabled on Windows but not on Linux.

So I think we should be focusing on bit 20 to figure out why that got programmed differently.

I don't see in the spec anything about bit 19 being used for pull up select.  I wonder if we've been programming bit 19/20 incorrectly as a result.

Please try this patch.
Comment 131 Friedrich Vock 2023-06-27 18:32:46 UTC
This patch doesn't seem to change anything. The issue is still there, and the GPIO register value seems the same as well.
Comment 132 Mario Limonciello (AMD) 2023-06-27 19:04:15 UTC
Can you please share an acpidump from your system?
Comment 133 Mario Limonciello (AMD) 2023-06-27 19:24:06 UTC
Actually assuming it's the same as https://bugzilla.kernel.org/show_bug.cgi?id=217493 then it should have been configured as a PullUp.

                    GpioInt (Level, ActiveLow, Exclusive, PullUp, 0x0000,
                        "\\_SB.GPIO", 0x00, ResourceConsumer, ,
                        )
                        {   // Pin list
                            0x0007
                        }

I'll keep looking around for where the bug is.
Comment 134 Mario Limonciello (AMD) 2023-06-27 20:05:45 UTC
I wonder if this is a situation that the interrupt for the GPIO is masked because no driver binds to the ACPI device.  
To rule out if we still have a problem in pinctrl-amd code, can you please share 
1. the output /sys/kernel/debug/gpio
and
2. of this with my bit 19/20 patch applied:

diff --git a/drivers/pinctrl/pinctrl-amd.c b/drivers/pinctrl/pinctrl-amd.c
index 9f7fabaffef59..b808af7752dcf 100644
--- a/drivers/pinctrl/pinctrl-amd.c
+++ b/drivers/pinctrl/pinctrl-amd.c
@@ -783,6 +783,8 @@ static int amd_pinconf_set(struct pinctrl_dev *pctldev, unsigned int pin,
                arg = pinconf_to_config_argument(configs[i]);
                pin_reg = readl(gpio_dev->base + pin*4);

+               pr_info("pinconf_set: pin %d, param %d, arg %d\n",
+                       pin, param, arg);
                switch (param) {
                case PIN_CONFIG_INPUT_DEBOUNCE:
                        pin_reg &= ~DB_TMR_OUT_MASK;
Comment 135 Mario Limonciello (AMD) 2023-06-29 22:09:58 UTC
Created attachment 304508 [details]
potential patch (v2)

I've found the reason that potential patch v1 wasn't calling that function.  I've adjusted some of the calling structure.

Can you guys please test "potential patch (v2)"?

Note: if this works on 6.4/6.3.y, I'll be splitting it into multiple patches and rebase it on pinctrl.git.
Comment 136 Nathan Schulte 2023-06-30 05:37:12 UTC
(In reply to Mario Limonciello (AMD) from comment #135)
> Can you guys please test "potential patch (v2)"?

Mario, with this patch applied atop the first (https://lore.kernel.org/linux-acpi/20230601221151.670-1-mario.limonciello@amd.com/T/#u), my keyboard is now working without having to perform any extra dancing.  Linux v6.4-rc4 + patches (ASUS FA617XT.309).
Comment 137 Nathan Schulte 2023-06-30 05:50:44 UTC
What I find interesting is that these two interrupts seem to share the same GPIO or such; with this patch, #61 (amd_gpio 7) has many fewer entries than without (in fact it seems stable at 2 in my limited testing):

            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12      CPU13      CPU14      CPU15······
   7:          0          0          0          0          0          0       8031          0          0          0          0          0          0          0          0          0  IR-IO-APIC    7-fasteoi   pinctrl_amd
  61:          0          0          0          0          0          0          2          0          0          0          0          0          0          0          0          0  amd_gpio    7  ITE5570:00

versus before:

            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12      CPU13      CPU14      CPU15······
   7:          0          0          0          0          0          0     262745          0          0          0          0          0          0          0          0          0  IR-IO-APIC    7-fasteoi   pinctrl_amd
  44:          0          0          0          0          0          0     262413          0          0          0          0          0          0          0          0          0  amd_gpio    7
Comment 138 Mario Limonciello (AMD) 2023-06-30 11:41:33 UTC
Can I please see updated /sys/kernel/debug/gpio output and an acpidump from your specific system?  I want to make sure this still lines up with what the firmware had intended.
Comment 139 Friedrich Vock 2023-06-30 15:08:58 UTC
Created attachment 304511 [details]
ACPI dump from FA617NS

Hi,

I can confirm the v2 patch also fixes the issue for me.

I attached an ACPI dump from my system. I also looked at the /sys/kernel/debug/gpio values again. Compared to last time I checked, some values changed, but most of them were due to other changes unrelated to this patch. This patch only affects the value of pin 7, changing it from the erroneous value from before to 151B00 (i.e. the same as Windows).

Thanks a lot for doing the proper fix for this!
Comment 140 Nathan Schulte 2023-06-30 18:03:37 UTC
Created attachment 304512 [details]
FA617XT.309 gpio, acpidump -- w/ both patch sets (on v6.4-rc4)

Mario, attached is the acpidump and /sys/kernel/debug/gpio from my v6.4-rc4 kernel on FA617XT.309 w/ the two patches.
Comment 141 Mario Limonciello (AMD) 2023-06-30 19:49:13 UTC
Thanks guys!

I've split up the series and rebased on pinctrl.git and sent it up for review:
https://lore.kernel.org/linux-gpio/20230630194716.6497-1-mario.limonciello@amd.com/T/#t
Comment 142 raphael.alampay 2023-07-03 01:44:02 UTC
Did this patch make it to 6.4.1? Sorry not too familiar with trying out patches. I tried it yesterday with 6.4 on Ubuntu 23.04 using mainline kernel but keyboard is still not working. I plan to try 6.4.1. Any logs that I can post to help?
Comment 143 Mario Limonciello (AMD) 2023-07-03 02:55:20 UTC
It's in Linus' tree, and it has been submitted for a backport request to 6.4.2.  You can pick it now on top of 6.4.1 if you would like.
Comment 144 Mario Limonciello (AMD) 2023-07-03 02:55:52 UTC
But please keep in mind that only fixes the keyboard, if you want to fix the interrupt storm you'll need the other series or other workaround.
Comment 145 raphael.alampay 2023-07-03 03:16:53 UTC
Got it. Thanks again for all the hard work. Will try to report what I find out similar to the ones posted here.
Comment 146 Nathan Schulte 2023-07-04 02:38:43 UTC
Curious ... what's it take to get the vendors (ASUS in this case) to ship a correct DSDT with their products?
Comment 147 Jan Visser 2023-07-08 21:44:17 UTC
(In reply to Mario Limonciello (AMD) from comment #143)
> It's in Linus' tree, and it has been submitted for a backport request to
> 6.4.2.  You can pick it now on top of 6.4.1 if you would like.

I have an ASUS TUF Gaming A16 FA617NS. If I build the kernel from the master branch of linux-torvalds. Should that include the fix?

I tried booting it but the keyboard still didn't work.
Comment 148 Mario Limonciello (AMD) 2023-07-09 04:22:16 UTC
The fix for the keyboard is landed in Linus' tree, but the interrupt storm solution isn't.  It's possible that you're hitting the interrupt storm making it seem the keyboard isn't working.

If you check out the fixes branch of https://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl.git/ it should get you both solutions.
Comment 149 Jan Visser 2023-07-10 16:45:26 UTC
Thanks! The keyboard now works!
Comment 150 Mario Limonciello (AMD) 2023-07-10 16:58:11 UTC
If possible, can you reply to this thread https://lore.kernel.org/linux-gpio/994042dd-9532-0dfb-8f86-98c897db75fd@amd.com/

with a tag like this:

Tested-by: Jan Visser <public_email_you_want_to_use>
Comment 151 Mario Limonciello (AMD) 2023-07-17 00:40:31 UTC
All of these fixes are landed into 6.5-rc2, closing this issue.
Comment 152 Nathan Schulte 2023-07-17 02:23:24 UTC
Thank you so much Mario, all.  I'll give it a build soon; glad this is resolved.

Still curious what it takes to get ASUS et al to play ball, but ... so it goes.
Comment 153 raphael.alampay 2023-07-17 03:24:21 UTC
Thank you Mario and all who contributed! Might finally get my laptop to work with this. Just curious, will this be backported to 6.4.3?

Thank you again.
Comment 154 Mario Limonciello (AMD) 2023-07-17 20:20:45 UTC
> Just curious, will this be backported to 6.4.3?

Sure, I don't see a problem with doing this.  I just tested it and found a new error message from this series so I sent this up:

https://lore.kernel.org/linux-gpio/20230717201652.17168-1-mario.limonciello@amd.com/T/#u

After testing 6.1.y and 6.4.y I've submitted the following to request a backport:
https://lore.kernel.org/stable/10fef988-8460-f96a-2d3d-cc8c6a37f3eb@amd.com/T/#u
Comment 155 Jan Visser 2023-07-17 20:33:14 UTC
That's great! If you need more testing just let me know!
Comment 156 Mario Limonciello (AMD) 2023-07-17 22:20:19 UTC
Any more testing is definitely welcome.  You can backport the patches to 6.1.y or 6.4.y as I described there.
Comment 157 Marina 2023-07-25 10:04:44 UTC
(In reply to Mario Limonciello (AMD) from comment #156)
> Any more testing is definitely welcome.  You can backport the patches to
> 6.1.y or 6.4.y as I described there.

I have tryed and when i do make -j 16 it gives me **make [Makefile:2032: .] Error2 and I can´t find how to solve it. Also I don't know what's the beggining and the final of the patch. 
thank you
Comment 158 Mario Limonciello (AMD) 2023-07-25 13:33:23 UTC
It's landed in stable 6.1 and 6.4 kernels now, if you upgrade to latest stable you'll have it.  It's also on it's way to 5.15.y and 5.10.y as well.
Comment 159 Marek Šanta 2023-07-25 14:54:58 UTC
Created attachment 304693 [details]
dmesg - Asus A15 2023 - generic 6.4.6

I am still running 6.4.2+ with your first patch:
https://lore.kernel.org/linux-acpi/20230601221151.670-1-mario.limonciello@amd.com/T/#u

It does not seem to be in current stable yet. However, I am not able to start 6.4.6 anyway as it just freezes during the boot.

Attached dmesg shows the freeze for approx 3 minutes until I hard shut the PC (and ACPI power button press was detected)

During the boot the cursor stopped blinking, I could not switch to TTY and it did not even react to CTRL+ALT+SysRq commands...

Another difference to working 6.4.2 is that "amd_gpio AMDI0030:00: Invalid config param 0014" now show up during the boot.

Should I maybe open up another bug here?
Comment 160 Mario Limonciello (AMD) 2023-07-25 15:02:44 UTC
> It does not seem to be in current stable yet.

Everything for this issue should be in 6.4.y now.

> Another difference to working 6.4.2 is that "amd_gpio AMDI0030:00: Invalid
> config param 0014" now show up during the boot.

https://bugzilla.kernel.org/show_bug.cgi?id=217698

> Attached dmesg shows the freeze for approx 3 minutes until I hard shut the PC
> (and ACPI power button press was detected)
> Should I maybe open up another bug here?

Please bisect between 6.4.2 and 6.4.6 and open a new bug with the result.  If it turns out to be caused by anything in drm or amdgpu please open the bug at https://gitlab.freedesktop.org/drm/amd
Comment 161 Marek Šanta 2023-07-25 15:14:58 UTC
Thanks. I will try to find the commit that causes the issue.

However, your IRQ override patch is definitely not in stable 6.4.y right now:

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/acpi/resource.c?h=linux-6.4.y
Comment 163 Nathan Schulte 2023-07-25 16:22:57 UTC
(In reply to Marek Šanta from comment #159)
> Another difference to working 6.4.2 is that "amd_gpio AMDI0030:00: Invalid
> config param 0014" now show up during the boot.
> 
> Should I maybe open up another bug here?

I started noticing this as well about the same time (haven't tried 6.4.6 yet; Debian has only 6.4.4 and 6.4.5 has the fixes here) but unsure what it means.  If you do open a bug report for this, please link it.
Comment 164 Mario Limonciello (AMD) 2023-07-25 16:49:19 UTC
BTW It's possible it's not kernel but instead GPU F/W. You can try to revert https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/commit/?id=cd524606ce5b43745c16554de8a5a5476de0f557 to see if that helps.
Comment 165 Marina 2023-07-25 18:15:54 UTC
I did it with kernel 6.4 and the lasts 3 sentences are:
  LD [M]  drivers/comedi/drivers/ni_routing.o
  LD [M]  drivers/infiniband/hw/hfi1/hfi1.o
make: *** [Makefile:2026: .] Error 2

Now its not 2032, but its another line, i don't know what to do... i'm a bit exhausted and frustrated...
Comment 166 raphael.alampay 2023-07-26 07:53:44 UTC
Can confirm that it's NOT working on 6.4.6 but working on 6.5rc3 (Keyboard's working). Probably worth noting as well that 6.5rc3 still breaks nvidia drivers 4xx - 5xx on Ubuntu 23.04.
Comment 167 Marina 2023-07-26 10:58:51 UTC
I tryed all kind of things and the best one is downloading this https://mega.nz/file/6qYkiIZQ#Y8w79Ilh0n61B-A_8O7v5fg6lMKYLeDXg1LCOXg8kZU and installing all three .dev 
and before reboot, putting this command udo apt-get install build-essential libncurses-dev bison flex libssl-dev libelf-dev

the only thing i have now is that i need to know how to enter whithout recovering mode, anyone knows?
Comment 168 Hans de Goede 2023-08-09 19:31:40 UTC
Unfortunately the dropping of:

```
#ifdef CONFIG_X86
	/*
	 * IRQ override isn't needed on modern AMD Zen systems and
	 * this override breaks active low IRQs on AMD Ryzen 6000 and
	 * newer systems. Skip it.
	 */
	if (boot_cpu_has(X86_FEATURE_ZEN))
		return false;
#endif
```

As done in commit a9c4a912b7dc ("ACPI: resource: Remove "Zen" specific match and
quirks") is causing keyboard problems for quite a lot of AMD based laptop users using other models, see e.g. :

https://bugzilla.redhat.com/show_bug.cgi?id=2228891
https://bugzilla.redhat.com/show_bug.cgi?id=2229165
https://bugzilla.redhat.com/show_bug.cgi?id=2229317
https://bugzilla.kernel.org/show_bug.cgi?id=217718
https://bugzilla.kernel.org/show_bug.cgi?id=217726
https://bugzilla.kernel.org/show_bug.cgi?id=217731

So I have submitted a revert upstream:
https://lore.kernel.org/linux-acpi/20230808103335.95339-1-hdegoede@redhat.com/

I did add an extra patch on top:
https://lore.kernel.org/linux-acpi/20230808103335.95339-4-hdegoede@redhat.com/

which should keep the behavior of a9c4a912b7dc for any laptop with a line like this in dmesg:

[    0.021867] ACPI: INT_SRC_OVR (bus 0 bus_irq 1 global_irq 1 low edge)

Note the "bus_irq 1" is important here. There will also be other INT_SRC_OVR messages with a different bus_irq!

If possible please give this patch-series a try:
https://lore.kernel.org/linux-acpi/20230808103335.95339-1-hdegoede@redhat.com/

With this series there should still be a line like:

[    0.410670] ACPI: IRQ 1 override to edge, low(!)

in dmesg for laptops which need the override (based on the INT_SRC_OVR bus_irq 1),
so this patch-series should not cause a regression on the laptop from the original reporter of this bug.
Comment 169 Marek Šanta 2023-08-10 06:57:45 UTC
I confirm that after applying your set of three patches on top of current 6.4.9 stable, the keyboard on ASUS FA507NU still works well! :)

the line

ACPI: IRQ 1 override to edge, low(!)

is present in my dmesg
Comment 170 Alberto Baglietto 2023-10-03 08:19:55 UTC
I replicate my other post on a very similar thread just to point out what follows: I solved the problem pretty easily after racking my brains for several weeks. My B150 series Asus ExpertBook keyboard, otherwise  perfectly working, stopped responding once Linux had booted. I took the following steps:

I added this repository:

sudo add-apt-repository ppa:damentz/liquorix && sudo apt-get update

I downloaded the kernel 

sudo apt-get install linux-image-liquorix-amd64 linux-headers-liquorix-amd64

No need to add patches or anything, keyboard was right away alive and kicking!