Bug 212371

Summary: iwlwifi: "Timeout waiting for PNVM load!" with AX210 on 5.11.3+ and 5.12.0-rc3
Product: Drivers Reporter: David Ross (daboross)
Component: network-wireless-intelAssignee: Default virtual assignee for network-wireless-intel (drivers_network-wireless-intel)
Status: ASSIGNED ---    
Severity: normal CC: artur, bugzilla, ccdrecc, CoelacanthusHex, cory, cuebix, doug.coleman, duarte.rocha, eddik98, inhui.hwang, janpieter.sollie, kernel-org.myq, kernel.org, kiran.telukunta, lenchan139, lordheavym, luca, niclas, paulius.zaleckas, pbrobinson, philipp+Kernel, raymond, richrocksmyworld, samwinchester19999, seth, tais.hansen, tsangares, UshakovVasilii
Priority: P1    
Hardware: Intel   
OS: Linux   
Kernel Version: 5.11.7-200.fc33 Subsystem:
Regression: No Bisected commit-id:
Attachments: BROKEN dmesg log for 5.11.7
BROKEN dmesg log for 5.12.0-0.rc3
WORKING dmesg log for 5.11.2
BROKEN dmesg log for 5.11.3
lspci -vvnn
app

Description David Ross 2021-03-21 02:57:40 UTC
Created attachment 295967 [details]
BROKEN dmesg log for 5.11.7

kernel 5.11.7-200.fc33.x86_64+debug

Likely related to my running an Intel AX210 chip.

iwlwifi detects the AX210 chip, but it errors out while loading, and NetworkManager does not find the device.

This is broken on at least 5.11.3-300.fc34.x86_64, 5.11.7-200.fc33.x86_64+debug and 5.12.0-0.rc3.20210319git8b12a62a4e3e.172.fc35.x86_64.

This works (the bug does NOT occur) on 5.11.2-300.fc34.x86_64.

Related arch linux bug at https://bugs.archlinux.org/task/70071. I'm submitting this here because I'm on Fedora, and thus I believe this is a cross-system kernel issue.

[   11.466042] iwlwifi 0000:04:00.0: enabling device (0000 -> 0002)
[   11.513767] iwlwifi 0000:04:00.0: api flags index 2 larger than supported by driver
[   11.513897] iwlwifi 0000:04:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 93.8.63.28
[   11.515808] iwlwifi 0000:04:00.0: loaded firmware version 59.601f3a66.0 ty-a0-gf-a0-59.ucode op_mode iwlmvm
[   11.724017] iwlwifi 0000:04:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[   11.912643] iwlwifi 0000:04:00.0: loaded PNVM version 0x324cd670
[   12.014922] iwlwifi 0000:04:00.0: Timeout waiting for PNVM load!
[   12.014931] iwlwifi 0000:04:00.0: Failed to start RT ucode: -110
[   12.014941] iwlwifi 0000:04:00.0: iwl_trans_send_cmd bad state = 1
[   12.222898] iwlwifi 0000:04:00.0: firmware didn't ACK the reset - continue anyway
[   12.236033] iwlwifi 0000:04:00.0: Failed to run INIT ucode: -110


Full dmesg logs for 5.12.0, 5.11.3 (BROKEN) and 5.11.2 (WORKING) attached.
Comment 1 David Ross 2021-03-21 02:58:20 UTC
Created attachment 295969 [details]
BROKEN dmesg log for 5.12.0-0.rc3
Comment 2 David Ross 2021-03-21 02:58:54 UTC
Created attachment 295971 [details]
WORKING dmesg log for 5.11.2
Comment 3 David Ross 2021-03-21 03:13:58 UTC
Created attachment 295973 [details]
BROKEN dmesg log for 5.11.3
Comment 4 David Ross 2021-03-21 03:26:49 UTC
Created attachment 295975 [details]
lspci -vvnn
Comment 5 David Ross 2021-03-21 03:29:02 UTC
May be related to https://bugzilla.kernel.org/show_bug.cgi?id=211165. Same error message, but that bug has only 2/100 reproducibility, while this bug is 100% reproducible on my system.
Comment 6 Laurent Carlier 2021-03-22 15:17:16 UTC
Archlinux bug report: https://bugs.archlinux.org/task/70071
Comment 7 Peter Robinson 2021-03-22 15:23:21 UTC
Fedora bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1940263
Comment 8 Dannel Albert 2021-03-22 19:11:13 UTC
I have the same issue on a brand new system that came with the AX210 and haven't been able to get it to work at all. I get pretty much the same error messages:

➜  ~ sudo dmesg | grep iwlwifi
[    5.300620] iwlwifi 0000:03:00.0: enabling device (0000 -> 0002)
[    5.315109] iwlwifi 0000:03:00.0: api flags index 2 larger than supported by driver
[    5.315125] iwlwifi 0000:03:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 93.8.63.28
[    5.315357] iwlwifi 0000:03:00.0: loaded firmware version 59.601f3a66.0 ty-a0-gf-a0-59.ucode op_mode iwlmvm
[    5.649887] iwlwifi 0000:03:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[    5.819781] iwlwifi 0000:03:00.0: loaded PNVM version 0x324cd670
[    5.922953] iwlwifi 0000:03:00.0: Timeout waiting for PNVM load!
[    5.922956] iwlwifi 0000:03:00.0: Failed to start RT ucode: -110
[    5.922959] iwlwifi 0000:03:00.0: iwl_trans_send_cmd bad state = 1
[    6.126286] iwlwifi 0000:03:00.0: firmware didn't ACK the reset - continue anyway
[    6.138669] iwlwifi 0000:03:00.0: Failed to run INIT ucode: -110
Comment 9 Dannel Albert 2021-03-22 19:12:44 UTC
Oh, I almost forgot, here's the output of my uname -a:

➜  ~ uname -a
Linux gladiator 5.11.7-arch1-1 #1 SMP PREEMPT Wed, 17 Mar 2021 16:59:58 +0000 x86_64 GNU/Linux
Comment 10 David Ross 2021-03-23 05:38:32 UTC
Dannel Albert, if you're able to find a way to downgrade your kernel version to something that's at least 5.11.0 (the version which added support for the AX210), but older than 5.11.3 (the version which broke it), then I would recommend trying that.

Since it seems like you're on arch, maybe you could download linux-5.11.2 from https://archive.archlinux.org/packages/l/linux/ and follow https://wiki.archlinux.org/index.php/Arch_Linux_Archive#How_to_downgrade_one_package to downgrade?

Or maybe it'd be better to follow what the person making the arch report (linked above) said, and downgrade linux-firmware to 20210208.b79d239-1 from https://archive.archlinux.org/packages/l/linux-firmware/.

I'm not an arch linux user myself, but downgrading the kernel in fedora by manually downloading & installing older kernel packages was a good temporary fix for me.

I'm not sure what the etiquette for providing help/workarounds on kernel bug reports is, so I apologize if this post is out of line and/or spamming everyone :)
Comment 11 tornado99 2021-03-23 09:43:27 UTC
Same issue with AX210 and 5.11.6 for me. Workaround by downgrading firmware to previous also worked.
Comment 12 Dannel Albert 2021-03-23 11:40:32 UTC
Created attachment 296011 [details]
attachment-4622-0.html

Thanks for the suggestion. Those are good ideas, but I actually anticipated having issues when I bought this laptop and got an AX200 just in case, so when it didn't work, I just swapped that in.

I'll probably keep that in until I hear reports that this is working and/or there is a fix that needs to be tested and I can help contribute.
On Mar 23, 2021, 1:38 AM -0400, bugzilla-daemon@bugzilla.kernel.org, wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=212371
>
> --- Comment #10 from David Ross (daboross@daboross.net) ---
> Dannel Albert, if you're able to find a way to downgrade your kernel version
> to
> something that's at least 5.11.0 (the version which added support for the
> AX210), but older than 5.11.3 (the version which broke it), then I would
> recommend trying that.
>
> Since it seems like you're on arch, maybe you could download linux-5.11.2
> from
> https://archive.archlinux.org/packages/l/linux/ and follow
>
> https://wiki.archlinux.org/index.php/Arch_Linux_Archive#How_to_downgrade_one_package
> to downgrade?
>
> Or maybe it'd be better to follow what the person making the arch report
> (linked above) said, and downgrade linux-firmware to 20210208.b79d239-1 from
> https://archive.archlinux.org/packages/l/linux-firmware/.
>
> I'm not an arch linux user myself, but downgrading the kernel in fedora by
> manually downloading & installing older kernel packages was a good temporary
> fix for me.
>
> I'm not sure what the etiquette for providing help/workarounds on kernel bug
> reports is, so I apologize if this post is out of line and/or spamming
> everyone
> :)
>
> --
> You may reply to this email to add a comment.
>
> You are receiving this mail because:
> You are on the CC list for the bug.
Comment 13 Paulius Zaleckas 2021-03-24 16:40:12 UTC
Another simple workaround is to remove/rename /lib/firmware/iwlwifi-ty-a0-gf-a0.pnvm file.
Apparently currently used firmware is not compatible with this PNVM:
https://bugs.gentoo.org/777324#c6
Comment 14 David Ross 2021-03-25 05:54:11 UTC
(In reply to Paulius Zaleckas from comment #13)
> Another simple workaround is to remove/rename
> /lib/firmware/iwlwifi-ty-a0-gf-a0.pnvm file.
> Apparently currently used firmware is not compatible with this PNVM:
> https://bugs.gentoo.org/777324#c6

Thank you! This workaround works for me.

I've confirmed that renaming that file and rebooting with 5.11.8 gets me working wifi. Additionally, renaming it back to the original name and rebooting breaks wifi again (and with the same kernel error messages, if I'm reading them correctly).
Comment 15 Dannel Albert 2021-03-26 04:08:31 UTC
I swapped my AX210 back into my system and removing the pnvm file mentioned in the workaround works for me too!

Thanks!
Comment 16 Luca Coelho 2021-03-26 06:49:23 UTC
Unfortunately we can't just backport the FW version bump.  There are
actual API changes that need to be supported, so that would require an
unfeasible amount of changes for a stable tree.

We have two options:

1. Try to take a fix to the -59.ucode firmware (I'll take this
internally)

2. Prevent the driver from loading the PNVM file when older firmwares
are used (I'll check if there is a proper way to do this without
relying on the actual version number).

I'll take this forward.
Comment 17 Dannel Albert 2021-03-26 12:17:53 UTC
Created attachment 296073 [details]
attachment-13323-0.html

Thank you for handling this Luca!
On Mar 26, 2021, 2:49 AM -0400, bugzilla-daemon@bugzilla.kernel.org, wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=212371
>
> Luca Coelho (luca@coelho.fi) changed:
>
> What |Removed |Added
> ----------------------------------------------------------------------------
> Status|NEW |ASSIGNED
> CC| |luca@coelho.fi
>
> --- Comment #16 from Luca Coelho (luca@coelho.fi) ---
> Unfortunately we can't just backport the FW version bump. There are
> actual API changes that need to be supported, so that would require an
> unfeasible amount of changes for a stable tree.
>
> We have two options:
>
> 1. Try to take a fix to the -59.ucode firmware (I'll take this
> internally)
>
> 2. Prevent the driver from loading the PNVM file when older firmwares
> are used (I'll check if there is a proper way to do this without
> relying on the actual version number).
>
> I'll take this forward.
>
> --
> You may reply to this email to add a comment.
>
> You are receiving this mail because:
> You are on the CC list for the bug.
Comment 18 wil 2021-07-08 21:14:59 UTC
*** Bug 213683 has been marked as a duplicate of this bug. ***
Comment 19 wil 2021-07-08 21:16:28 UTC
Hey is there any updates?

This is still an issue for me, I cannot update my linux kernel.
Comment 20 Doug Coleman 2021-09-01 17:40:10 UTC
A solution to the AX210 pnvm error that worked for me on Ubuntu 20.04.3 kernel 5.11.0-27-generic with an AX210 is to delete /lib/firmware/iwlwifi-ty-a0-gf-a0.pnvm

I found this at https://www.reddit.com/r/Fedora/comments/m9p2h2/ax210_drivers/
Comment 21 kernel.org 2021-10-11 20:58:11 UTC
I also had to delete the /lib/firmware/iwlwifi-so-a0-gf-a0.pnvm file even though I have a `ty` aka AX210 otherwise it was causing the card to loop and not actually connect on 5.10.
Comment 22 Niclas 2021-11-02 18:37:55 UTC
GUys,

I have a Samsung laptop with the AX210, so also the ty.

I've been running Debian on the laptop on stable, but decided to do a complete new fresh install with the latest testing variant instead.

Everything works except iwlwifi has this .pvnm file and if..

.. I remove it, wifi stops working and bluetooth starts fine
.. I put it back, bluetooth stops working and wifi starts fine

Well. That means I cant have both working at the same time. Any suggestions as to where to begin? Can I somehow make either bluetooth accept pvnm or have wifi ignore it?

Regards,

Niclas
Comment 23 wil 2021-12-02 04:23:35 UTC
I have been remaining in a 5.11.10 this whole time to make sure my wifi works. Now I need some features in later kernal versions although this still seems to be a persistent issue.

Is there any progress being made on this iwlwifi module loading issue?
Comment 24 Niclas 2021-12-02 06:39:50 UTC
For me, I'm on Debian testing (bookworm) with the latest debian-released kernel 5.15.0, I have no problems anymore. I did'nt have problem with 5.13 either actually.

And the mentioned problem from me above, about not having both bluetooth and the wifi side of the chip working at the same time, has also gone away.

So which kernel are you considering? The fix of removing the pnvm-file in /lib/firmware works without problem if you really need to move up to a higher revision of the kernel I guess.

//Niclas
Comment 25 Janpieter Sollie 2021-12-11 07:44:58 UTC
I have the same issue on a mips24kc machine with AX210 card.  I'm using openwrt snapshots, which currently uses 5.15-rc6 backport drivers.  I generated an additonal patch to update drivers/.../iwlwifi to 5.15.5, but it didn't help.
Tried with all currently released firmwares in linux-firmware tree, 59-66, and a few pnvm files visible though the commit log, none of them works.
Comment 26 Raymond 2022-01-17 11:28:47 UTC
I had similar problem on a gaming PC, I'm on Ubuntu 5.11.0-46-generic kernel. I renamed pnvm file, rebooted and it worked for me. Bluetooth also seems to be working for me. Any idea when will it be fixed?
Comment 27 Doug Coleman 2022-01-17 18:05:00 UTC
I updated Ubuntu and got a broken wifi driver again. I got fed up with old kernels (5.11, 5.13) and installed 5.16 and it fixed the wifi and a screen-tearing issue that was driving me mad!

sudo add-apt-repository ppa:tuxinvader/lts-mainline -y
sudo apt update
sudo apt-get install linux-generic-5.16 -y
Comment 28 Raymond 2022-01-17 18:48:55 UTC
(In reply to Doug Coleman from comment #27)
> I updated Ubuntu and got a broken wifi driver again. I got fed up with old
> kernels (5.11, 5.13) and installed 5.16 and it fixed the wifi and a
> screen-tearing issue that was driving me mad!
> 
> sudo add-apt-repository ppa:tuxinvader/lts-mainline -y
> sudo apt update
> sudo apt-get install linux-generic-5.16 -y

Yes, it fixed for me too when I upgraded to 5.16 but but..
5.16 isn't official kernel?
Comment 29 pepito 2022-02-21 15:40:48 UTC
I'm getting the same error but with the recent kernel version 5.16.9, and the workaround of renaming the .pnvm file does not work. I can see from the dmesg log that it tries loading the -67.ucode firmware (which gives the same timeout error reported by others; see below), and I've tried renaming every version after -59 as well as the .pnvm file to force it to use the -59 firmware (which is the version Intel provides on their website), but I still get the timeout error in dmesg:

> Timeout waiting for PNVM load!
> Failed to start RT ucode: -110

Note: I *did* get this working by downgrading the kernel to 5.11 and renaming the .pnvm file, but I don't want to be stuck on such an old kernel. I was expecting everything to "just work" after I upgraded to the 5.16 kernel.
Comment 30 Paulius Zaleckas 2022-02-21 15:53:18 UTC
(In reply to pepito from comment #29)
> Note: I *did* get this working by downgrading the kernel to 5.11 and
> renaming the .pnvm file, but I don't want to be stuck on such an old kernel.
> I was expecting everything to "just work" after I upgraded to the 5.16
> kernel.

Currently I am using 5.14.18 and it does not have this problem. 5.11 had it.
Later if I upgrade to 5.16, I will let you know if it happens again.
Comment 31 Raymond 2022-04-01 14:47:35 UTC
Renaming/deleting .pvnm file solution stopped working for me after updating to 20.04.

Does anyone know how to fix it now?
Comment 32 Raymond 2022-04-01 14:48:20 UTC
(In reply to Raymond from comment #31)
> Renaming/deleting .pvnm file solution stopped working for me after updating
> to 20.04.
> 
> Does anyone know how to fix it now?

Kernel 5.13.0-39-generic
Comment 33 David Ross 2022-04-03 00:38:21 UTC
This no longer occurs for me on Fedora 35 with Kernel version 5.16.5-200.fc35.x86_64. I'm going to unsubscribe but leave this open, as I don't know where the fix was, and don't know the proper procedure for figuring out if this is both fixed & if the kernel has a test.
Comment 34 Cedric 2022-10-30 02:10:39 UTC
I had this problem, even after upgrading to 5.19.0-23-generic. 

The answer turned out to be that I had to remove "pci=nomsi,noaer" which was in my /etc/default/grub file.
Comment 35 samwinchester19999 2023-03-09 18:47:07 UTC
Created attachment 303914 [details]
app
Comment 36 Eduard Sydorenko 2023-04-10 12:40:35 UTC
Hello, I have the same problem, with a lot of linux-firmware versions.
I have found this issue also with linux-firmware-20220329.git681281e4-0ubuntu3.12.

After upgrading linux-firmware lib, I always have to use this command `sudo mv /usr/lib/firmware/iwlwifi-ty-a0-gf-a0.pnvm /usr/lib/firmware/iwlwifi-ty-a0-gf-a0.bak` (https://askubuntu.com/questions/1360175/intel-wifi-6-ax210-wifi-not-working-after-update), and after this command, Wi-Fi module works.

I have the same logs as other users have above.

- linux-firmware-20220329.git681281e4-0ubuntu3.12
- Linux version 5.10.110-rockchip-rk3588 (aarch64-linux-gnu-gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1 SMP Wed Mar 29 08:28:12 BST 2023

00:34:55 kernel: iwlwifi 0004:41:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
00:34:55 kernel: iwlwifi 0004:41:00.0: Failed to run INIT ucode: -110
00:34:55 kernel: iwlwifi 0004:41:00.0: iwl_trans_send_cmd bad state = 0
00:34:54 kernel: iwlwifi 0004:41:00.0: loaded PNVM version 0x881c99e1
Comment 37 Eduard Sydorenko 2023-04-10 18:25:20 UTC
Additional logs:

- Before executing above mv command:
[   18.502463] Intel(R) Wireless WiFi driver for Linux
[   18.502891] iwlwifi 0004:41:00.0: enabling device (0000 -> 0002)
[   18.543590] iwlwifi 0004:41:00.0: api flags index 2 larger than supported by driver
[   18.543628] iwlwifi 0004:41:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 93.8.63.28
[   18.544154] iwlwifi 0004:41:00.0: loaded firmware version 59.601f3a66.0 ty-a0-gf-a0-59.ucode op_mode iwlmvm
[   18.552252] iwlwifi 0004:41:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[   18.631147] sd 0:0:0:0: Attached scsi generic sg0 type 0
[   18.728168] iwlwifi 0004:41:00.0: loaded PNVM version 0x881c99e1
[   19.001708] iwlwifi 0004:41:00.0: Timeout waiting for PNVM load!
[   19.001730] iwlwifi 0004:41:00.0: Failed to start RT ucode: -110
[   19.001740] iwlwifi 0004:41:00.0: iwl_trans_send_cmd bad state = 0
[   19.014020] iwlwifi 0004:41:00.0: Failed to run INIT ucode: -110
[   19.026420] iwlwifi 0004:41:00.0: retry init count 0
[   19.028901] iwlwifi 0004:41:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[   19.081044] Process accounting resumed
[   19.479068] iwlwifi 0004:41:00.0: Timeout waiting for PNVM load!
[   19.479114] iwlwifi 0004:41:00.0: Failed to start RT ucode: -110
[   19.479137] iwlwifi 0004:41:00.0: iwl_trans_send_cmd bad state = 0
[   19.491411] iwlwifi 0004:41:00.0: Failed to run INIT ucode: -110
[   19.504714] iwlwifi 0004:41:00.0: retry init count 1
[   19.958375] iwlwifi 0004:41:00.0: Timeout waiting for PNVM load!
[   19.958420] iwlwifi 0004:41:00.0: Failed to start RT ucode: -110
[   19.958445] iwlwifi 0004:41:00.0: iwl_trans_send_cmd bad state = 0
[   19.971024] iwlwifi 0004:41:00.0: Failed to run INIT ucode: -110
[   19.983452] iwlwifi 0004:41:00.0: retry init count 2

- After executing above mv command:
[   18.750383] iwlwifi 0004:41:00.0: enabling device (0000 -> 0002)
[   18.765745] iwlwifi 0004:41:00.0: api flags index 2 larger than supported by driver
[   18.765774] iwlwifi 0004:41:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 93.8.63.28
[   18.766197] iwlwifi 0004:41:00.0: loaded firmware version 59.601f3a66.0 ty-a0-gf-a0-59.ucode op_mode iwlmvm
[   18.777427] iwlwifi 0004:41:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[   18.896842] sd 0:0:0:0: Attached scsi generic sg0 type 0
[   19.033116] iwlwifi 0004:41:00.0: base HW address: 04:cf:4b:21:24:e2
[   19.047555] thermal thermal_zone7: power_allocator: sustainable_power will be estimated
[   19.047621] thermal thermal_zone7: failed to read out thermal zone (-61)
[   19.055209] iwlwifi 0004:41:00.0 wlP4p65s0: renamed from wlan0