Bug 212371

Created attachment 295969 [details]
BROKEN dmesg log for 5.12.0-0.rc3

Created attachment 295971 [details]
WORKING dmesg log for 5.11.2

Created attachment 295973 [details]
BROKEN dmesg log for 5.11.3

Created attachment 295975 [details]
lspci -vvnn

May be related to https://bugzilla.kernel.org/show_bug.cgi?id=211165. Same error message, but that bug has only 2/100 reproducibility, while this bug is 100% reproducible on my system.

Archlinux bug report: https://bugs.archlinux.org/task/70071

Fedora bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1940263

I have the same issue on a brand new system that came with the AX210 and haven't been able to get it to work at all. I get pretty much the same error messages:

➜  ~ sudo dmesg | grep iwlwifi
[    5.300620] iwlwifi 0000:03:00.0: enabling device (0000 -> 0002)
[    5.315109] iwlwifi 0000:03:00.0: api flags index 2 larger than supported by driver
[    5.315125] iwlwifi 0000:03:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 93.8.63.28
[    5.315357] iwlwifi 0000:03:00.0: loaded firmware version 59.601f3a66.0 ty-a0-gf-a0-59.ucode op_mode iwlmvm
[    5.649887] iwlwifi 0000:03:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[    5.819781] iwlwifi 0000:03:00.0: loaded PNVM version 0x324cd670
[    5.922953] iwlwifi 0000:03:00.0: Timeout waiting for PNVM load!
[    5.922956] iwlwifi 0000:03:00.0: Failed to start RT ucode: -110
[    5.922959] iwlwifi 0000:03:00.0: iwl_trans_send_cmd bad state = 1
[    6.126286] iwlwifi 0000:03:00.0: firmware didn't ACK the reset - continue anyway
[    6.138669] iwlwifi 0000:03:00.0: Failed to run INIT ucode: -110

Oh, I almost forgot, here's the output of my uname -a:

➜  ~ uname -a
Linux gladiator 5.11.7-arch1-1 #1 SMP PREEMPT Wed, 17 Mar 2021 16:59:58 +0000 x86_64 GNU/Linux

Dannel Albert, if you're able to find a way to downgrade your kernel version to something that's at least 5.11.0 (the version which added support for the AX210), but older than 5.11.3 (the version which broke it), then I would recommend trying that.

Since it seems like you're on arch, maybe you could download linux-5.11.2 from https://archive.archlinux.org/packages/l/linux/ and follow https://wiki.archlinux.org/index.php/Arch_Linux_Archive#How_to_downgrade_one_package to downgrade?

Or maybe it'd be better to follow what the person making the arch report (linked above) said, and downgrade linux-firmware to 20210208.b79d239-1 from https://archive.archlinux.org/packages/l/linux-firmware/.

I'm not an arch linux user myself, but downgrading the kernel in fedora by manually downloading & installing older kernel packages was a good temporary fix for me.

I'm not sure what the etiquette for providing help/workarounds on kernel bug reports is, so I apologize if this post is out of line and/or spamming everyone :)

Same issue with AX210 and 5.11.6 for me. Workaround by downgrading firmware to previous also worked.

Created attachment 296011 [details]
attachment-4622-0.html

Thanks for the suggestion. Those are good ideas, but I actually anticipated having issues when I bought this laptop and got an AX200 just in case, so when it didn't work, I just swapped that in.

I'll probably keep that in until I hear reports that this is working and/or there is a fix that needs to be tested and I can help contribute.
On Mar 23, 2021, 1:38 AM -0400, bugzilla-daemon@bugzilla.kernel.org, wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=212371
>
> --- Comment #10 from David Ross (daboross@daboross.net) ---
> Dannel Albert, if you're able to find a way to downgrade your kernel version
> to
> something that's at least 5.11.0 (the version which added support for the
> AX210), but older than 5.11.3 (the version which broke it), then I would
> recommend trying that.
>
> Since it seems like you're on arch, maybe you could download linux-5.11.2
> from
> https://archive.archlinux.org/packages/l/linux/ and follow
>
> https://wiki.archlinux.org/index.php/Arch_Linux_Archive#How_to_downgrade_one_package
> to downgrade?
>
> Or maybe it'd be better to follow what the person making the arch report
> (linked above) said, and downgrade linux-firmware to 20210208.b79d239-1 from
> https://archive.archlinux.org/packages/l/linux-firmware/.
>
> I'm not an arch linux user myself, but downgrading the kernel in fedora by
> manually downloading & installing older kernel packages was a good temporary
> fix for me.
>
> I'm not sure what the etiquette for providing help/workarounds on kernel bug
> reports is, so I apologize if this post is out of line and/or spamming
> everyone
> :)
>
> --
> You may reply to this email to add a comment.
>
> You are receiving this mail because:
> You are on the CC list for the bug.

Another simple workaround is to remove/rename /lib/firmware/iwlwifi-ty-a0-gf-a0.pnvm file.
Apparently currently used firmware is not compatible with this PNVM:
https://bugs.gentoo.org/777324#c6

(In reply to Paulius Zaleckas from comment #13)
> Another simple workaround is to remove/rename
> /lib/firmware/iwlwifi-ty-a0-gf-a0.pnvm file.
> Apparently currently used firmware is not compatible with this PNVM:
> https://bugs.gentoo.org/777324#c6

Thank you! This workaround works for me.

I've confirmed that renaming that file and rebooting with 5.11.8 gets me working wifi. Additionally, renaming it back to the original name and rebooting breaks wifi again (and with the same kernel error messages, if I'm reading them correctly).

I swapped my AX210 back into my system and removing the pnvm file mentioned in the workaround works for me too!

Thanks!

Unfortunately we can't just backport the FW version bump.  There are
actual API changes that need to be supported, so that would require an
unfeasible amount of changes for a stable tree.

We have two options:

1. Try to take a fix to the -59.ucode firmware (I'll take this
internally)

2. Prevent the driver from loading the PNVM file when older firmwares
are used (I'll check if there is a proper way to do this without
relying on the actual version number).

I'll take this forward.

Created attachment 296073 [details]
attachment-13323-0.html

Thank you for handling this Luca!
On Mar 26, 2021, 2:49 AM -0400, bugzilla-daemon@bugzilla.kernel.org, wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=212371
>
> Luca Coelho (luca@coelho.fi) changed:
>
> What |Removed |Added
> ----------------------------------------------------------------------------
> Status|NEW |ASSIGNED
> CC| |luca@coelho.fi
>
> --- Comment #16 from Luca Coelho (luca@coelho.fi) ---
> Unfortunately we can't just backport the FW version bump. There are
> actual API changes that need to be supported, so that would require an
> unfeasible amount of changes for a stable tree.
>
> We have two options:
>
> 1. Try to take a fix to the -59.ucode firmware (I'll take this
> internally)
>
> 2. Prevent the driver from loading the PNVM file when older firmwares
> are used (I'll check if there is a proper way to do this without
> relying on the actual version number).
>
> I'll take this forward.
>
> --
> You may reply to this email to add a comment.
>
> You are receiving this mail because:
> You are on the CC list for the bug.

*** Bug 213683 has been marked as a duplicate of this bug. ***

Hey is there any updates?

This is still an issue for me, I cannot update my linux kernel.

A solution to the AX210 pnvm error that worked for me on Ubuntu 20.04.3 kernel 5.11.0-27-generic with an AX210 is to delete /lib/firmware/iwlwifi-ty-a0-gf-a0.pnvm

I found this at https://www.reddit.com/r/Fedora/comments/m9p2h2/ax210_drivers/

I also had to delete the /lib/firmware/iwlwifi-so-a0-gf-a0.pnvm file even though I have a `ty` aka AX210 otherwise it was causing the card to loop and not actually connect on 5.10.

GUys,

I have a Samsung laptop with the AX210, so also the ty.

I've been running Debian on the laptop on stable, but decided to do a complete new fresh install with the latest testing variant instead.

Everything works except iwlwifi has this .pvnm file and if..

.. I remove it, wifi stops working and bluetooth starts fine
.. I put it back, bluetooth stops working and wifi starts fine

Well. That means I cant have both working at the same time. Any suggestions as to where to begin? Can I somehow make either bluetooth accept pvnm or have wifi ignore it?

Regards,

Niclas

I have been remaining in a 5.11.10 this whole time to make sure my wifi works. Now I need some features in later kernal versions although this still seems to be a persistent issue.

Is there any progress being made on this iwlwifi module loading issue?

For me, I'm on Debian testing (bookworm) with the latest debian-released kernel 5.15.0, I have no problems anymore. I did'nt have problem with 5.13 either actually.

And the mentioned problem from me above, about not having both bluetooth and the wifi side of the chip working at the same time, has also gone away.

So which kernel are you considering? The fix of removing the pnvm-file in /lib/firmware works without problem if you really need to move up to a higher revision of the kernel I guess.

//Niclas

I have the same issue on a mips24kc machine with AX210 card.  I'm using openwrt snapshots, which currently uses 5.15-rc6 backport drivers.  I generated an additonal patch to update drivers/.../iwlwifi to 5.15.5, but it didn't help.
Tried with all currently released firmwares in linux-firmware tree, 59-66, and a few pnvm files visible though the commit log, none of them works.

I had similar problem on a gaming PC, I'm on Ubuntu 5.11.0-46-generic kernel. I renamed pnvm file, rebooted and it worked for me. Bluetooth also seems to be working for me. Any idea when will it be fixed?

I updated Ubuntu and got a broken wifi driver again. I got fed up with old kernels (5.11, 5.13) and installed 5.16 and it fixed the wifi and a screen-tearing issue that was driving me mad!

sudo add-apt-repository ppa:tuxinvader/lts-mainline -y
sudo apt update
sudo apt-get install linux-generic-5.16 -y

(In reply to Doug Coleman from comment #27)
> I updated Ubuntu and got a broken wifi driver again. I got fed up with old
> kernels (5.11, 5.13) and installed 5.16 and it fixed the wifi and a
> screen-tearing issue that was driving me mad!
> 
> sudo add-apt-repository ppa:tuxinvader/lts-mainline -y
> sudo apt update
> sudo apt-get install linux-generic-5.16 -y

Yes, it fixed for me too when I upgraded to 5.16 but but..
5.16 isn't official kernel?

I'm getting the same error but with the recent kernel version 5.16.9, and the workaround of renaming the .pnvm file does not work. I can see from the dmesg log that it tries loading the -67.ucode firmware (which gives the same timeout error reported by others; see below), and I've tried renaming every version after -59 as well as the .pnvm file to force it to use the -59 firmware (which is the version Intel provides on their website), but I still get the timeout error in dmesg:

> Timeout waiting for PNVM load!
> Failed to start RT ucode: -110

Note: I *did* get this working by downgrading the kernel to 5.11 and renaming the .pnvm file, but I don't want to be stuck on such an old kernel. I was expecting everything to "just work" after I upgraded to the 5.16 kernel.

(In reply to pepito from comment #29)
> Note: I *did* get this working by downgrading the kernel to 5.11 and
> renaming the .pnvm file, but I don't want to be stuck on such an old kernel.
> I was expecting everything to "just work" after I upgraded to the 5.16
> kernel.

Currently I am using 5.14.18 and it does not have this problem. 5.11 had it.
Later if I upgrade to 5.16, I will let you know if it happens again.

Renaming/deleting .pvnm file solution stopped working for me after updating to 20.04.

Does anyone know how to fix it now?

(In reply to Raymond from comment #31)
> Renaming/deleting .pvnm file solution stopped working for me after updating
> to 20.04.
> 
> Does anyone know how to fix it now?

Kernel 5.13.0-39-generic

This no longer occurs for me on Fedora 35 with Kernel version 5.16.5-200.fc35.x86_64. I'm going to unsubscribe but leave this open, as I don't know where the fix was, and don't know the proper procedure for figuring out if this is both fixed & if the kernel has a test.

I had this problem, even after upgrading to 5.19.0-23-generic. 

The answer turned out to be that I had to remove "pci=nomsi,noaer" which was in my /etc/default/grub file.

Created attachment 303914 [details]
app

Hello, I have the same problem, with a lot of linux-firmware versions.
I have found this issue also with linux-firmware-20220329.git681281e4-0ubuntu3.12.

After upgrading linux-firmware lib, I always have to use this command `sudo mv /usr/lib/firmware/iwlwifi-ty-a0-gf-a0.pnvm /usr/lib/firmware/iwlwifi-ty-a0-gf-a0.bak` (https://askubuntu.com/questions/1360175/intel-wifi-6-ax210-wifi-not-working-after-update), and after this command, Wi-Fi module works.

I have the same logs as other users have above.

- linux-firmware-20220329.git681281e4-0ubuntu3.12
- Linux version 5.10.110-rockchip-rk3588 (aarch64-linux-gnu-gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1 SMP Wed Mar 29 08:28:12 BST 2023

00:34:55 kernel: iwlwifi 0004:41:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
00:34:55 kernel: iwlwifi 0004:41:00.0: Failed to run INIT ucode: -110
00:34:55 kernel: iwlwifi 0004:41:00.0: iwl_trans_send_cmd bad state = 0
00:34:54 kernel: iwlwifi 0004:41:00.0: loaded PNVM version 0x881c99e1

Additional logs:

- Before executing above mv command:
[   18.502463] Intel(R) Wireless WiFi driver for Linux
[   18.502891] iwlwifi 0004:41:00.0: enabling device (0000 -> 0002)
[   18.543590] iwlwifi 0004:41:00.0: api flags index 2 larger than supported by driver
[   18.543628] iwlwifi 0004:41:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 93.8.63.28
[   18.544154] iwlwifi 0004:41:00.0: loaded firmware version 59.601f3a66.0 ty-a0-gf-a0-59.ucode op_mode iwlmvm
[   18.552252] iwlwifi 0004:41:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[   18.631147] sd 0:0:0:0: Attached scsi generic sg0 type 0
[   18.728168] iwlwifi 0004:41:00.0: loaded PNVM version 0x881c99e1
[   19.001708] iwlwifi 0004:41:00.0: Timeout waiting for PNVM load!
[   19.001730] iwlwifi 0004:41:00.0: Failed to start RT ucode: -110
[   19.001740] iwlwifi 0004:41:00.0: iwl_trans_send_cmd bad state = 0
[   19.014020] iwlwifi 0004:41:00.0: Failed to run INIT ucode: -110
[   19.026420] iwlwifi 0004:41:00.0: retry init count 0
[   19.028901] iwlwifi 0004:41:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[   19.081044] Process accounting resumed
[   19.479068] iwlwifi 0004:41:00.0: Timeout waiting for PNVM load!
[   19.479114] iwlwifi 0004:41:00.0: Failed to start RT ucode: -110
[   19.479137] iwlwifi 0004:41:00.0: iwl_trans_send_cmd bad state = 0
[   19.491411] iwlwifi 0004:41:00.0: Failed to run INIT ucode: -110
[   19.504714] iwlwifi 0004:41:00.0: retry init count 1
[   19.958375] iwlwifi 0004:41:00.0: Timeout waiting for PNVM load!
[   19.958420] iwlwifi 0004:41:00.0: Failed to start RT ucode: -110
[   19.958445] iwlwifi 0004:41:00.0: iwl_trans_send_cmd bad state = 0
[   19.971024] iwlwifi 0004:41:00.0: Failed to run INIT ucode: -110
[   19.983452] iwlwifi 0004:41:00.0: retry init count 2

- After executing above mv command:
[   18.750383] iwlwifi 0004:41:00.0: enabling device (0000 -> 0002)
[   18.765745] iwlwifi 0004:41:00.0: api flags index 2 larger than supported by driver
[   18.765774] iwlwifi 0004:41:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 93.8.63.28
[   18.766197] iwlwifi 0004:41:00.0: loaded firmware version 59.601f3a66.0 ty-a0-gf-a0-59.ucode op_mode iwlmvm
[   18.777427] iwlwifi 0004:41:00.0: Detected Intel(R) Wi-Fi 6 AX210 160MHz, REV=0x420
[   18.896842] sd 0:0:0:0: Attached scsi generic sg0 type 0
[   19.033116] iwlwifi 0004:41:00.0: base HW address: 04:cf:4b:21:24:e2
[   19.047555] thermal thermal_zone7: power_allocator: sustainable_power will be estimated
[   19.047621] thermal thermal_zone7: failed to read out thermal zone (-61)
[   19.055209] iwlwifi 0004:41:00.0 wlP4p65s0: renamed from wlan0