Bug 205829 - intel ax200 not properly reinitialized on reboot
Summary: intel ax200 not properly reinitialized on reboot
Status: ASSIGNED
Alias: None
Product: Drivers
Classification: Unclassified
Component: network-wireless-intel (show other bugs)
Hardware: Intel Linux
: P1 high
Assignee: DO NOT USE - assign "network-wireless-intel" component instead
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-12-10 20:37 UTC by Jason A. Donenfeld
Modified: 2020-07-17 10:35 UTC (History)
3 users (show)

See Also:
Kernel Version: 5.4, 5.5
Subsystem:
Regression: No
Bisected commit-id:


Attachments

Description Jason A. Donenfeld 2019-12-10 20:37:40 UTC
I have a Thinkpad P1 gen 2 (same as X1 extreme gen 2), which has an
AX200 -- 52:00.0 0280: 8086:2723 (rev 1a). Things mostly work, but
I've noticed that after a reboot, the card stops working.
Specifically, wpa_supplicant can't talk to it anymore. Or, sometimes
it appears to drop off the bus completely. After doing a full
poweroff+poweron (not a reboot), it works again.

Here's what one of the failure modes looks like:

zx2c4@thinkpad ~ $ journalctl -b -1|egrep '(iwl|wpa)'
Dec 10 11:57:55 thinkpad systemd[1]: Created slice system-wpa_supplicant.slice.
Dec 10 11:57:55 thinkpad kernel: iwlwifi 0000:52:00.0: enabling device
(0000 -> 0002)
Dec 10 11:57:55 thinkpad kernel: iwlwifi 0000:52:00.0: Direct firmware
load for iwlwifi-cc-a0-52.ucode failed with error -2
Dec 10 11:57:55 thinkpad kernel: iwlwifi 0000:52:00.0: Direct firmware
load for iwlwifi-cc-a0-51.ucode failed with error -2
Dec 10 11:57:55 thinkpad kernel: iwlwifi 0000:52:00.0: Direct firmware
load for iwlwifi-cc-a0-50.ucode failed with error -2
Dec 10 11:57:55 thinkpad kernel: iwlwifi 0000:52:00.0: Direct firmware
load for iwlwifi-cc-a0-49.ucode failed with error -2
Dec 10 11:57:55 thinkpad kernel: iwlwifi 0000:52:00.0:
TLV_FW_FSEQ_VERSION: FSEQ Version: 43.2.23.17
Dec 10 11:57:55 thinkpad kernel: iwlwifi 0000:52:00.0: Found debug
destination: EXTERNAL_DRAM
Dec 10 11:57:55 thinkpad kernel: iwlwifi 0000:52:00.0: Found debug
configuration: 0
Dec 10 11:57:55 thinkpad kernel: iwlwifi 0000:52:00.0: loaded firmware
version 48.4fa0041f.0 op_mode iwlmvm
Dec 10 11:57:55 thinkpad kernel: iwlwifi 0000:52:00.0: Detected
Intel(R) Wi-Fi 6 AX200 160MHz, REV=0x340
Dec 10 11:57:55 thinkpad kernel: iwlwifi 0000:52:00.0: Applying debug
destination EXTERNAL_DRAM
Dec 10 11:57:55 thinkpad kernel: iwlwifi 0000:52:00.0: Allocated
0x00400000 bytes for firmware monitor.
Dec 10 11:57:55 thinkpad kernel: iwlwifi 0000:52:00.0: base HW
address: sanitized
Dec 10 11:57:56 thinkpad wpa_supplicant[747]: Successfully initialized
wpa_supplicant
Dec 10 11:57:56 thinkpad wpa_supplicant[747]: Could not set interface
wlan0 flags (UP): Connection timed out
Dec 10 11:57:56 thinkpad wpa_supplicant[747]: nl80211: Could not set
interface 'wlan0' UP
Dec 10 11:57:56 thinkpad wpa_supplicant[747]: nl80211: deinit
ifname=wlan0 disabled_11b_rates=0
Dec 10 11:57:56 thinkpad wpa_supplicant[747]: wlan0: Failed to
initialize driver interface
Dec 10 11:57:56 thinkpad systemd[1]: wpa_supplicant@wlan0.service:
Main process exited, code=exited, status=255/EXCEPTION
Dec 10 11:57:56 thinkpad systemd[1]: wpa_supplicant@wlan0.service:
Failed with result 'exit-code'.
Dec 10 11:58:10 thinkpad systemd[1]: Removed slice system-wpa_supplicant.slice.

I assume this is something to do with the bios not powercycling the
pcie bus during reboot, leaving the card in some kind of inconsistent
state. Perhaps it's possible to add a quirk to always powercycle it
before initializing and loading firmware?

This happens 100% of the time.
Comment 1 Blaž Hrastnik 2019-12-11 07:11:02 UTC
You also seem to be missing some of the firmware that's still missing in `linux-firmware`. I addressed that by following the workaround here: https://gitlab.com/emrose/xps13-7390_debian/issues/5#note_240886447


Regardless, I get a similar error on my Surface Laptop 3 with a bit different dmesg output (I'll provide the log when I get back home). Rebooting seems to work, but a dual boot into Windows 10, then reboot into Linux breaks the card. It's broken enough that after the Linux boot that it stops working on both Windows and Linux.
Comment 2 Johannes Berg 2019-12-11 21:28:08 UTC
(In reply to Blaž Hrastnik from comment #1)
> You also seem to be missing some of the firmware that's still missing in
> `linux-firmware`. I addressed that by following the workaround here:
> https://gitlab.com/emrose/xps13-7390_debian/issues/5#note_240886447

So that means you actually have a Qu device, not a "cc" like Jason?

> Regardless, I get a similar error on my Surface Laptop 3 with a bit
> different dmesg output (I'll provide the log when I get back home).
> Rebooting seems to work, but a dual boot into Windows 10, then reboot into
> Linux breaks the card. It's broken enough that after the Linux boot that it
> stops working on both Windows and Linux.

Must be some initialization then ... Hmm.

We had a similar report on Qu where it failed only a few percent of the time, but sounds like that's completely different.
Comment 3 Johannes Berg 2019-12-11 21:28:53 UTC
(In reply to Johannes Berg from comment #2)

> We had a similar report on Qu where it failed only a few percent of the
> time, but sounds like that's completely different.

Oh and I forgot to say - the underlying reason for that Qu failure was something that cannot happen on cc :)
Comment 4 Blaž Hrastnik 2019-12-12 10:17:03 UTC
(In reply to Johannes Berg from comment #2)
> 
> So that means you actually have a Qu device, not a "cc" like Jason?
> 


Ah right, I think so. I only got the machine recently so I wasn't sure what the exact chip was. If you want, I can open a separate issue?

Here's how the failure looks like (before doing the firmware update, it's the same after):


[ 2.269893] iwlwifi 0000:00:14.3: enabling device (0000 -> 0002)
[ 2.274564] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-Qu-c0-hr-b0-50.ucode failed with error -2
[ 2.274644] iwlwifi 0000:00:14.3: Direct firmware load for iwlwifi-Qu-c0-hr-b0-49.ucode failed with error -2
[ 2.276989] iwlwifi 0000:00:14.3: TLV_FW_FSEQ_VERSION: FSEQ Version: 43.2.23.17
[ 2.276991] iwlwifi 0000:00:14.3: Found debug destination: EXTERNAL_DRAM
[ 2.276992] iwlwifi 0000:00:14.3: Found debug configuration: 0
[ 2.277130] iwlwifi 0000:00:14.3: loaded firmware version 48.4fa0041f.0 op_mode iwlmvm

...

[ 3.469801] iwlwifi 0000:00:14.3: Collecting data: trigger 15 fired.
[ 3.470906] iwlwifi 0000:00:14.3: Loaded firmware version: 48.4fa0041f.0
[ 3.470908] iwlwifi 0000:00:14.3: 0x00000000 | ADVANCED_SYSASSERT
[ 3.470910] iwlwifi 0000:00:14.3: 0x00000000 | trm_hw_status0
[ 3.470910] iwlwifi 0000:00:14.3: 0x00000000 | trm_hw_status1
[ 3.470911] iwlwifi 0000:00:14.3: 0x00000000 | branchlink2
[ 3.470912] iwlwifi 0000:00:14.3: 0x00000000 | interruptlink1
[ 3.470913] iwlwifi 0000:00:14.3: 0x00000000 | interruptlink2
[ 3.470914] iwlwifi 0000:00:14.3: 0x00000000 | data1
[ 3.470915] iwlwifi 0000:00:14.3: 0x00000000 | data2
[ 3.470915] iwlwifi 0000:00:14.3: 0x00000000 | data3
[ 3.470916] iwlwifi 0000:00:14.3: 0x00000000 | beacon time
[ 3.470917] iwlwifi 0000:00:14.3: 0x00000000 | tsf low
[ 3.470918] iwlwifi 0000:00:14.3: 0x00000000 | tsf hi
[ 3.470919] iwlwifi 0000:00:14.3: 0x00000000 | time gp1
[ 3.470920] iwlwifi 0000:00:14.3: 0x00000000 | time gp2
[ 3.470920] iwlwifi 0000:00:14.3: 0x00000000 | uCode revision type
[ 3.470921] iwlwifi 0000:00:14.3: 0x00000000 | uCode version major
[ 3.470921] iwlwifi 0000:00:14.3: 0x00000000 | uCode version minor
[ 3.470922] iwlwifi 0000:00:14.3: 0x00000000 | hw version
[ 3.470922] iwlwifi 0000:00:14.3: 0x00000000 | board version
[ 3.470923] iwlwifi 0000:00:14.3: 0x00000000 | hcmd
[ 3.470923] iwlwifi 0000:00:14.3: 0x00000000 | isr0
[ 3.470924] iwlwifi 0000:00:14.3: 0x00000000 | isr1
[ 3.470924] iwlwifi 0000:00:14.3: 0x00000000 | isr2
[ 3.470925] iwlwifi 0000:00:14.3: 0x00000000 | isr3
[ 3.470925] iwlwifi 0000:00:14.3: 0x00000000 | isr4
[ 3.470926] iwlwifi 0000:00:14.3: 0x00000000 | last cmd Id
[ 3.470927] iwlwifi 0000:00:14.3: 0x00000000 | wait_event
[ 3.470927] iwlwifi 0000:00:14.3: 0x00000000 | l2p_control
[ 3.470927] iwlwifi 0000:00:14.3: 0x00000000 | l2p_duration
[ 3.470928] iwlwifi 0000:00:14.3: 0x00000000 | l2p_mhvalid
[ 3.470929] iwlwifi 0000:00:14.3: 0x00000000 | l2p_addr_match
[ 3.470929] iwlwifi 0000:00:14.3: 0x00000000 | lmpm_pmg_sel
[ 3.470930] iwlwifi 0000:00:14.3: 0x00000000 | timestamp
[ 3.470930] iwlwifi 0000:00:14.3: 0x00000000 | flow_handler
[ 3.470973] iwlwifi 0000:00:14.3: Start IWL Error Log Dump:
[ 3.470974] iwlwifi 0000:00:14.3: Status: 0x00000000, count: 7
[ 3.470975] iwlwifi 0000:00:14.3: 0x20000066 | NMI_INTERRUPT_HOST
[ 3.470975] iwlwifi 0000:00:14.3: 0x00000000 | umac branchlink1
[ 3.470976] iwlwifi 0000:00:14.3: 0xC008D49C | umac branchlink2
[ 3.470976] iwlwifi 0000:00:14.3: 0x8048DBD6 | umac interruptlink1
[ 3.470977] iwlwifi 0000:00:14.3: 0x8048DBD6 | umac interruptlink2
[ 3.470977] iwlwifi 0000:00:14.3: 0x01000000 | umac data1
[ 3.470978] iwlwifi 0000:00:14.3: 0x8048DBD6 | umac data2
[ 3.470978] iwlwifi 0000:00:14.3: 0x00000000 | umac data3
[ 3.470979] iwlwifi 0000:00:14.3: 0x00000030 | umac major
[ 3.470979] iwlwifi 0000:00:14.3: 0x4FA0041F | umac minor
[ 3.470980] iwlwifi 0000:00:14.3: 0x000F526A | frame pointer
[ 3.470980] iwlwifi 0000:00:14.3: 0xC0886284 | stack pointer
[ 3.470981] iwlwifi 0000:00:14.3: 0x0000FFFF | last host cmd
[ 3.470982] iwlwifi 0000:00:14.3: 0x00000000 | isr status reg
[ 3.471004] iwlwifi 0000:00:14.3: Fseq Registers:
[ 3.471007] iwlwifi 0000:00:14.3: 0x20000000 | FSEQ_ERROR_CODE
[ 3.471010] iwlwifi 0000:00:14.3: 0x80290030 | FSEQ_TOP_INIT_VERSION
[ 3.471013] iwlwifi 0000:00:14.3: 0x80070043 | FSEQ_CNVIO_INIT_VERSION
[ 3.471016] iwlwifi 0000:00:14.3: 0x0000A481 | FSEQ_OTP_VERSION
[ 3.471019] iwlwifi 0000:00:14.3: 0x00000002 | FSEQ_TOP_CONTENT_VERSION
[ 3.471022] iwlwifi 0000:00:14.3: 0x4552414E | FSEQ_ALIVE_TOKEN
[ 3.471024] iwlwifi 0000:00:14.3: 0x02000300 | FSEQ_CNVI_ID
[ 3.471027] iwlwifi 0000:00:14.3: 0x01300504 | FSEQ_CNVR_ID
[ 3.471030] iwlwifi 0000:00:14.3: 0x02000300 | CNVI_AUX_MISC_CHIP
[ 3.471035] iwlwifi 0000:00:14.3: 0x01300504 | CNVR_AUX_MISC_CHIP
[ 3.471040] iwlwifi 0000:00:14.3: 0x05B0905B | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
[ 3.471044] iwlwifi 0000:00:14.3: 0x0000025B | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR
[ 3.471082] iwlwifi 0000:00:14.3: SecBoot CPU1 Status: 0x5fb9, CPU2 Status: 0x3
[ 3.471083] iwlwifi 0000:00:14.3: Failed to start RT ucode: -110
[ 3.471085] iwlwifi 0000:00:14.3: Firmware not running - cannot dump error
[ 3.482587] iwlwifi 0000:00:14.3: Failed to run INIT ucode: -110
Comment 5 daniel.a.jenson 2020-02-04 17:18:30 UTC
I have a very similar issue, tracked here: https://bugzilla.redhat.com/show_bug.cgi?id=1790231. Brand new lenovo P1 Gen 2 AX200. Looks like it's loading the wrong firmware:

Jan 27 08:03:09 m2 kernel: iwlwifi 0000:52:00.0: BIOS contains WGDS but no WRDS
Jan 27 08:03:09 m2 kernel: iwlwifi 0000:52:00.0: FW already configured (0) - re-configuring
Jan 27 08:03:09 m2 kernel: iwlwifi 0000:52:00.0: Applying debug destination EXTERNAL_DRAM
Jan 27 08:03:05 m2 kernel: iwlwifi 0000:52:00.0: BIOS contains WGDS but no WRDS
Jan 27 08:03:05 m2 kernel: iwlwifi 0000:52:00.0: FW already configured (0) - re-configuring
Jan 27 08:03:05 m2 kernel: iwlwifi 0000:52:00.0: Applying debug destination EXTERNAL_DRAM
Jan 27 08:03:05 m2 kernel: iwlwifi 0000:52:00.0: BIOS contains WGDS but no WRDS
Jan 27 08:03:05 m2 kernel: iwlwifi 0000:52:00.0: FW already configured (0) - re-configuring
Jan 27 08:03:05 m2 kernel: iwlwifi 0000:52:00.0: Applying debug destination EXTERNAL_DRAM
Jan 27 08:03:05 m2 NetworkManager[1457]: <info>  [1580140985.4611] rfkill1: found Wi-Fi radio killswitch (at /sys/devices/pci0000:00/0000:00:1d.6/0000:52:00.0/ieee80211/phy0/rfkill1) (driver iwlwifi)
Jan 27 08:03:04 m2 kernel: iwlwifi 0000:52:00.0 wlp82s0: renamed from wlan0
Jan 27 08:03:04 m2 kernel: iwlwifi 0000:52:00.0: base HW address: dc:71:96:d9:e5:95
Jan 27 08:03:04 m2 kernel: iwlwifi 0000:52:00.0: Allocated 0x00400000 bytes for firmware monitor.
Jan 27 08:03:04 m2 kernel: iwlwifi 0000:52:00.0: Applying debug destination EXTERNAL_DRAM
Jan 27 08:03:04 m2 kernel: iwlwifi 0000:52:00.0: Detected Intel(R) Wi-Fi 6 AX200 160MHz, REV=0x340
Jan 27 08:03:04 m2 kernel: iwlwifi 0000:52:00.0: loaded firmware version 48.4fa0041f.0 op_mode iwlmvm
Jan 27 08:03:04 m2 kernel: iwlwifi 0000:52:00.0: Found debug configuration: 0
Jan 27 08:03:04 m2 kernel: iwlwifi 0000:52:00.0: Found debug destination: EXTERNAL_DRAM
Jan 27 08:03:04 m2 kernel: iwlwifi 0000:52:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 43.2.23.17
Jan 27 08:03:04 m2 kernel: iwlwifi 0000:52:00.0: Direct firmware load for iwlwifi-cc-a0-49.ucode failed with error -2
Jan 27 08:03:04 m2 kernel: iwlwifi 0000:52:00.0: Direct firmware load for iwlwifi-cc-a0-50.ucode failed with error -2
Jan 27 08:03:04 m2 kernel: iwlwifi 0000:52:00.0: enabling device (0000 -> 0002)

Note You need to log in before you can comment on or make changes to this bug.