Bug 217207

Summary: firmware rtw89/rtw8852b_fw.bin does not load/not recognized in linux-firmware version 2023_03_10
Product: Networking Reporter: rab0171610 (rb0171610)
Component: WirelessAssignee: networking_wireless (networking_wireless)
Status: NEW ---    
Severity: high CC: jwboyer, Larry.Finger, pkshih, rb0171610, regressions
Priority: P1    
Hardware: AMD   
OS: Linux   
Kernel Version: 6.2.5 Subsystem:
Regression: No Bisected commit-id:

Description rab0171610 2023-03-15 20:45:28 UTC
First, this is issue is independent of which kernel version I use. I use rtw89 wifi drivers for rtw_8852be from 6.2 kernel. Second, the affected firmware is not an issue on linux-firmware 2023_02_10 and below (wifi works):

 
[    8.786565] Loading firmware: rtw89/rtw8852b_fw.bin

After upgrade to linux-firmware-2023_03_10 (wifi does not work/no devices found):
[] loading firmare: rwt89/rtw8852b_fw.bin
[] rtw89_8852be 0000:03:00.0: no suitable firmware found
[] rtw89_8852be 0000:03:00.0: failed to recognize firmware


Reproducible: Always

Steps to Reproduce:
1.Upgrade to linux-firmware 2023_03_10 and above (specifically rtw89/rtw8852b_fw.bin)
2. Reboot
3.Wifi does not work, devices not found, dmesg reports firmware not loaded/not recognized (rwt89/rtw8852b_fw.bin)
4. Downgrade to previous version of firmware binary (rtw89/rtw8852b_fw.bin) from linux-firmware 2023_02_10
5. Reboot
6. Wifi working, dmesg reports firmware loaded, no errors (rwt89/rtw8852b_fw.bin)

Actual Results:  
Wifi did not work after upgrade, worked before upgrade and worked after downgrading.

Expected Results:  
Wifi would continue to work after upgrade, firmware would load normally after upgrade, firmware would be recognized after upgrade

Harware Details:

Network controller [0280]: Realtek Semiconductor Co., Ltd. Device [10ec:b852]

Lenovo Ideapad 1i

Additional Details:

There were two commits made upstream on 2-24-23 to the rtw8852b firmware binary: 

 "rtw89: 8852b: update fw to v0.29.26.0
 New fimrware type 5 to support hardware scan, P2P-PS and etc."

 "rtw89: 8852b: update fw to v0.29.29.0
 Correct channel report of hardware scan."

 "--- a/rtw89/rtw8852b_fw.bin
 +++ b/rtw89/rtw8852b_fw.bin
 . . .Binary files differ"

This newest of this specific binary blob version does not load on my machine (no suitable firmware found/firmware not recognized.)
Comment 1 Ping-Ke Shih 2023-03-17 00:36:37 UTC
The firmware format is changed, and wireless-next tree or upcoming 6.3 will
support it.
Comment 2 Larry Finger 2023-03-17 05:52:58 UTC
You can also run the following commands:

wget lwfinger.com/download/rtw8852b_fw.bin
sudo cp rtw8852b_fw.bin /lib/firmware/rtw89/.

That will download and install version 0.27.x of the firmware.
Comment 3 The Linux kernel's regression tracker (Thorsten Leemhuis) 2023-03-17 13:30:41 UTC
(In reply to Ping-Ke Shih from comment #1)
> The firmware format is changed, and wireless-next tree or upcoming 6.3 will
> support it.

So people have to update those two in sync? 

Why wasn't some tag like "v2" added to the new firmware (e.g. rtw89/rtw8852b_fw_v2.bin), so that people that update kernel-firmware now are not affected by it?
Comment 4 The Linux kernel's regression tracker (Thorsten Leemhuis) 2023-03-17 13:40:40 UTC
Just to understand this fully: 6.3-rc/wireless-next work fine with the old firmware? From mainline commit 18ddf102d4b8 it sounds like it, but I wonder of there were other commits that are not backward compatible.
Comment 5 The Linux kernel's regression tracker (Thorsten Leemhuis) 2023-03-17 13:48:48 UTC
To me everything looks like the firmware in kernel-firmware was updated to a version that only works on 6.3-rc and later, but doesn't work on 6.2 and earlier kernels.

Is that correct? If it's the case I'd argue that the commits with the updated firmware should be reverted, as such a update is not allowed according to Documentation/driver-api/firmware/firmware-usage-guidelines.rst; to quote: 
```
Users switching to a newer kernel should *not* have to install newer
firmware files to keep their hardware working. At the same time updated
firmware files must not cause any regressions for users of older kernel
releases.
```
Comment 6 The Linux kernel's regression tracker (Thorsten Leemhuis) 2023-03-17 14:06:24 UTC
Ahh, I only now see that a similar discussion happened on linux-wireless and there a few questions of mine where answered 
https://lore.kernel.org/all/df1ce994-3368-a57e-7078-8bdcccf4a1fd@gmail.com/

So it seems this really is something that shouldn't happen and that the updated firmware should be reverted.
Comment 7 Larry Finger 2023-03-17 15:35:36 UTC
The problem only affects kernel 6.2, and only the rtw8852be driver. There are distros that have backported this driver to 6.1, but that is not common.

Yes, the newest driver works with the older firmware, and there certainly should have been a new name used for the newer firmware.
Comment 8 rab0171610 2023-03-17 19:14:52 UTC
I was eager to adopt the 6.2 stable kernel as the rtw89 drivers are now in the kernel tree.  Before I was using 6.0, 6.1 series and using Larry's (lwfinger) git rtw89 driver.  I was eager to switch to 6.2, so I would not have to recompile the wifi driver with each kernel update. 
The latest release of linux-firmware-2023_03_10 is not backward compatible for anyone that requires said firmware ( rtw8852b_fw.bin), unless they are testing the mainline kernel.  Distros are not aware of this and may inadvertently version bump to the latest linux-firmware release, not realizing that some firmware no longer works with any current stable or long-term kernel. This is the case with Gentoo, which has stabilized linux-firmware-2023_03_10 -- it is now offered as an update to users. Regardless, a user could still download the latest linux-firmware from kernel.org and unzip it to their system, unaware that it may not work with the currently stable kernel as the rtw89 firmware only works with a kernel that isn't actually finalized and released yet (6.3).

Considering the recent changes to the firmware version of rtw89/rtw8852b_fw.bin:
        A) At a minimum package maintainers at the distro level and users should be made aware of the 6.3+ kernel dependency with the latest release of linux-firmware(2023_03_10) 
                            OR
        B)The rtw8852b firmware version changes need to be temporarily reverted, in which case the last release of linux-firmware (2023_03_10) should be dropped, at least until there is a working resolution.

With regard to the kernel compatibility issues with rtw89 firmware, I don't think linux-firmware-2023_03_10 should be available for download at the present time.
Comment 9 rab0171610 2023-03-17 19:27:58 UTC
To be clear, from my tests:

The rtw89/rtw8852b_fw.bin firmware from the Februrary release of linux-firmware_2023_02_10 is recognized and loads on both the stable 6.2 and mainline 6.3 kernels. 

The rtw89/rtw8852b_fw.bin, which has a new firmware format, in the March release of  linux-firmware_2023_03_10 is only recognized and loads on the mainline 6.3 kernel. It does not load and is not recognized on the 6.2 kernel:
[] loading firmare: rwt89/rtw8852b_fw.bin
[] rtw89_8852be 0000:03:00.0: no suitable firmware found
[] rtw89_8852be 0000:03:00.0: failed to recognize firmware

What the results would be with 6.1 and earlier kernels using Larrys git rtw89 out of tree kernel rtw89 driver I do not know.
Comment 10 Larry Finger 2023-03-17 19:38:13 UTC
My out of the tree drivers match the code of kernel 6.3. They will work with both firmware formats.

In comment #2, I also provided the commands to load the old-format firmware that works with kernel 6.2!
Comment 11 rab0171610 2023-03-17 19:48:44 UTC
Thank you Larry. In addition to your firmware, users can also extract the firmware from the rtw89 directory in the February release linux-firmware-2023_02_10 from kernel.org as a workaround in the 6.2 kernel.
Comment 12 Ping-Ke Shih 2023-03-18 14:03:26 UTC
I have send a pull-request to revert the firmware

https://lore.kernel.org/linux-firmware/889f3db6ca4008047595f4392388acc1f165e03c.camel@realtek.com/T/#u

Sorry for the inconvenience.