Bug 212371 - iwlwifi: "Timeout waiting for PNVM load!" with AX210 on 5.11.3+ and 5.12.0-rc3
Summary: iwlwifi: "Timeout waiting for PNVM load!" with AX210 on 5.11.3+ and 5.12.0-rc3
Status: ASSIGNED
Alias: None
Product: Drivers
Classification: Unclassified
Component: network-wireless-intel (show other bugs)
Hardware: Intel Linux
: P1 normal
Assignee: Default virtual assignee for network-wireless-intel
URL:
Keywords:
: 213683 (view as bug list)
Depends on:
Blocks:
 
Reported: 2021-03-21 02:57 UTC by David Ross
Modified: 2021-07-08 21:16 UTC (History)
14 users (show)

See Also:
Kernel Version: 5.11.7-200.fc33
Tree: Mainline
Regression: No


Attachments
BROKEN dmesg log for 5.11.7 (125.22 KB, text/plain)
2021-03-21 02:57 UTC, David Ross
Details
BROKEN dmesg log for 5.12.0-0.rc3 (128.18 KB, text/plain)
2021-03-21 02:58 UTC, David Ross
Details
WORKING dmesg log for 5.11.2 (82.91 KB, text/plain)
2021-03-21 02:58 UTC, David Ross
Details
BROKEN dmesg log for 5.11.3 (80.71 KB, text/plain)
2021-03-21 03:13 UTC, David Ross
Details
lspci -vvnn (10.40 KB, text/plain)
2021-03-21 03:26 UTC, David Ross
Details

Description David Ross 2021-03-21 02:57:40 UTC
Created attachment 295967 [details]
BROKEN dmesg log for 5.11.7

kernel 5.11.7-200.fc33.x86_64+debug

Likely related to my running an Intel AX210 chip.

iwlwifi detects the AX210 chip, but it errors out while loading, and NetworkManager does not find the device.

This is broken on at least 5.11.3-300.fc34.x86_64, 5.11.7-200.fc33.x86_64+debug and 5.12.0-0.rc3.20210319git8b12a62a4e3e.172.fc35.x86_64.

This works (the bug does NOT occur) on 5.11.2-300.fc34.x86_64.

Related arch linux bug at https://bugs.archlinux.org/task/70071. I'm submitting this here because I'm on Fedora, and thus I believe this is a cross-system kernel issue.

[   11.466042] iwlwifi 0000:04:00.0: enabling device (0000 -> 0002)
[   11.513767] iwlwifi 0000:04:00.0: api flags index 2 larger than supported by driver
[   11.513897] iwlwifi 0000:04:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 93.8.63.28
[   11.515808] iwlwifi 0000:04:00.0: loaded firmware version 59.601f3a66.0 ty-a0-gf-a0-59.ucode op_mode iwlmvm
[   11.724017] iwlwifi 0000:04:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[   11.912643] iwlwifi 0000:04:00.0: loaded PNVM version 0x324cd670
[   12.014922] iwlwifi 0000:04:00.0: Timeout waiting for PNVM load!
[   12.014931] iwlwifi 0000:04:00.0: Failed to start RT ucode: -110
[   12.014941] iwlwifi 0000:04:00.0: iwl_trans_send_cmd bad state = 1
[   12.222898] iwlwifi 0000:04:00.0: firmware didn't ACK the reset - continue anyway
[   12.236033] iwlwifi 0000:04:00.0: Failed to run INIT ucode: -110


Full dmesg logs for 5.12.0, 5.11.3 (BROKEN) and 5.11.2 (WORKING) attached.
Comment 1 David Ross 2021-03-21 02:58:20 UTC
Created attachment 295969 [details]
BROKEN dmesg log for 5.12.0-0.rc3
Comment 2 David Ross 2021-03-21 02:58:54 UTC
Created attachment 295971 [details]
WORKING dmesg log for 5.11.2
Comment 3 David Ross 2021-03-21 03:13:58 UTC
Created attachment 295973 [details]
BROKEN dmesg log for 5.11.3
Comment 4 David Ross 2021-03-21 03:26:49 UTC
Created attachment 295975 [details]
lspci -vvnn
Comment 5 David Ross 2021-03-21 03:29:02 UTC
May be related to https://bugzilla.kernel.org/show_bug.cgi?id=211165. Same error message, but that bug has only 2/100 reproducibility, while this bug is 100% reproducible on my system.
Comment 6 Laurent Carlier 2021-03-22 15:17:16 UTC
Archlinux bug report: https://bugs.archlinux.org/task/70071
Comment 7 Peter Robinson 2021-03-22 15:23:21 UTC
Fedora bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1940263
Comment 8 Dannel Albert 2021-03-22 19:11:13 UTC
I have the same issue on a brand new system that came with the AX210 and haven't been able to get it to work at all. I get pretty much the same error messages:

➜  ~ sudo dmesg | grep iwlwifi
[    5.300620] iwlwifi 0000:03:00.0: enabling device (0000 -> 0002)
[    5.315109] iwlwifi 0000:03:00.0: api flags index 2 larger than supported by driver
[    5.315125] iwlwifi 0000:03:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 93.8.63.28
[    5.315357] iwlwifi 0000:03:00.0: loaded firmware version 59.601f3a66.0 ty-a0-gf-a0-59.ucode op_mode iwlmvm
[    5.649887] iwlwifi 0000:03:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[    5.819781] iwlwifi 0000:03:00.0: loaded PNVM version 0x324cd670
[    5.922953] iwlwifi 0000:03:00.0: Timeout waiting for PNVM load!
[    5.922956] iwlwifi 0000:03:00.0: Failed to start RT ucode: -110
[    5.922959] iwlwifi 0000:03:00.0: iwl_trans_send_cmd bad state = 1
[    6.126286] iwlwifi 0000:03:00.0: firmware didn't ACK the reset - continue anyway
[    6.138669] iwlwifi 0000:03:00.0: Failed to run INIT ucode: -110
Comment 9 Dannel Albert 2021-03-22 19:12:44 UTC
Oh, I almost forgot, here's the output of my uname -a:

➜  ~ uname -a
Linux gladiator 5.11.7-arch1-1 #1 SMP PREEMPT Wed, 17 Mar 2021 16:59:58 +0000 x86_64 GNU/Linux
Comment 10 David Ross 2021-03-23 05:38:32 UTC
Dannel Albert, if you're able to find a way to downgrade your kernel version to something that's at least 5.11.0 (the version which added support for the AX210), but older than 5.11.3 (the version which broke it), then I would recommend trying that.

Since it seems like you're on arch, maybe you could download linux-5.11.2 from https://archive.archlinux.org/packages/l/linux/ and follow https://wiki.archlinux.org/index.php/Arch_Linux_Archive#How_to_downgrade_one_package to downgrade?

Or maybe it'd be better to follow what the person making the arch report (linked above) said, and downgrade linux-firmware to 20210208.b79d239-1 from https://archive.archlinux.org/packages/l/linux-firmware/.

I'm not an arch linux user myself, but downgrading the kernel in fedora by manually downloading & installing older kernel packages was a good temporary fix for me.

I'm not sure what the etiquette for providing help/workarounds on kernel bug reports is, so I apologize if this post is out of line and/or spamming everyone :)
Comment 11 tornado99 2021-03-23 09:43:27 UTC
Same issue with AX210 and 5.11.6 for me. Workaround by downgrading firmware to previous also worked.
Comment 12 Dannel Albert 2021-03-23 11:40:32 UTC
Created attachment 296011 [details]
attachment-4622-0.html

Thanks for the suggestion. Those are good ideas, but I actually anticipated having issues when I bought this laptop and got an AX200 just in case, so when it didn't work, I just swapped that in.

I'll probably keep that in until I hear reports that this is working and/or there is a fix that needs to be tested and I can help contribute.
On Mar 23, 2021, 1:38 AM -0400, bugzilla-daemon@bugzilla.kernel.org, wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=212371
>
> --- Comment #10 from David Ross (daboross@daboross.net) ---
> Dannel Albert, if you're able to find a way to downgrade your kernel version
> to
> something that's at least 5.11.0 (the version which added support for the
> AX210), but older than 5.11.3 (the version which broke it), then I would
> recommend trying that.
>
> Since it seems like you're on arch, maybe you could download linux-5.11.2
> from
> https://archive.archlinux.org/packages/l/linux/ and follow
>
> https://wiki.archlinux.org/index.php/Arch_Linux_Archive#How_to_downgrade_one_package
> to downgrade?
>
> Or maybe it'd be better to follow what the person making the arch report
> (linked above) said, and downgrade linux-firmware to 20210208.b79d239-1 from
> https://archive.archlinux.org/packages/l/linux-firmware/.
>
> I'm not an arch linux user myself, but downgrading the kernel in fedora by
> manually downloading & installing older kernel packages was a good temporary
> fix for me.
>
> I'm not sure what the etiquette for providing help/workarounds on kernel bug
> reports is, so I apologize if this post is out of line and/or spamming
> everyone
> :)
>
> --
> You may reply to this email to add a comment.
>
> You are receiving this mail because:
> You are on the CC list for the bug.
Comment 13 Paulius Zaleckas 2021-03-24 16:40:12 UTC
Another simple workaround is to remove/rename /lib/firmware/iwlwifi-ty-a0-gf-a0.pnvm file.
Apparently currently used firmware is not compatible with this PNVM:
https://bugs.gentoo.org/777324#c6
Comment 14 David Ross 2021-03-25 05:54:11 UTC
(In reply to Paulius Zaleckas from comment #13)
> Another simple workaround is to remove/rename
> /lib/firmware/iwlwifi-ty-a0-gf-a0.pnvm file.
> Apparently currently used firmware is not compatible with this PNVM:
> https://bugs.gentoo.org/777324#c6

Thank you! This workaround works for me.

I've confirmed that renaming that file and rebooting with 5.11.8 gets me working wifi. Additionally, renaming it back to the original name and rebooting breaks wifi again (and with the same kernel error messages, if I'm reading them correctly).
Comment 15 Dannel Albert 2021-03-26 04:08:31 UTC
I swapped my AX210 back into my system and removing the pnvm file mentioned in the workaround works for me too!

Thanks!
Comment 16 Luca Coelho 2021-03-26 06:49:23 UTC
Unfortunately we can't just backport the FW version bump.  There are
actual API changes that need to be supported, so that would require an
unfeasible amount of changes for a stable tree.

We have two options:

1. Try to take a fix to the -59.ucode firmware (I'll take this
internally)

2. Prevent the driver from loading the PNVM file when older firmwares
are used (I'll check if there is a proper way to do this without
relying on the actual version number).

I'll take this forward.
Comment 17 Dannel Albert 2021-03-26 12:17:53 UTC
Created attachment 296073 [details]
attachment-13323-0.html

Thank you for handling this Luca!
On Mar 26, 2021, 2:49 AM -0400, bugzilla-daemon@bugzilla.kernel.org, wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=212371
>
> Luca Coelho (luca@coelho.fi) changed:
>
> What |Removed |Added
> ----------------------------------------------------------------------------
> Status|NEW |ASSIGNED
> CC| |luca@coelho.fi
>
> --- Comment #16 from Luca Coelho (luca@coelho.fi) ---
> Unfortunately we can't just backport the FW version bump. There are
> actual API changes that need to be supported, so that would require an
> unfeasible amount of changes for a stable tree.
>
> We have two options:
>
> 1. Try to take a fix to the -59.ucode firmware (I'll take this
> internally)
>
> 2. Prevent the driver from loading the PNVM file when older firmwares
> are used (I'll check if there is a proper way to do this without
> relying on the actual version number).
>
> I'll take this forward.
>
> --
> You may reply to this email to add a comment.
>
> You are receiving this mail because:
> You are on the CC list for the bug.
Comment 18 wil 2021-07-08 21:14:59 UTC
*** Bug 213683 has been marked as a duplicate of this bug. ***
Comment 19 wil 2021-07-08 21:16:28 UTC
Hey is there any updates?

This is still an issue for me, I cannot update my linux kernel.

Note You need to log in before you can comment on or make changes to this bug.