Bug 206067

Summary: ax200: Crash with status 0x00000080 and rx duplicated frames.
Product: Drivers Reporter: Ben Greear (greearb)
Component: network-wireless-intelAssignee: Default virtual assignee for network-wireless-intel (drivers_network-wireless-intel)
Status: NEW ---    
Severity: normal CC: johannes.gueldenring
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 5.2.21+ Tree: Mainline
Regression: No

Description Ben Greear 2020-01-03 21:37:16 UTC
Our test team reported this problem.  The hardware is Intel x86-64 with pcie to m.2 adapter boards and 8 total AX200 radios in one system.

Notes on the AX crashes:

- When the crash happens, the station does not seem to make a full recovery. There are many duplicate rx packets on the UDP download side. The upload direction is normal with no duplicate packets.

- Traffic is fixed after resetting the station.

- Sometimes traffic just stops completely for both sides and cannot resume until resetting the station.

- I've found a way to reproduce a crash fairly reliably. Just in case it's a different crash, I'm uploading a separate crash file. It'll have wiphy0 in the file name.

- To reproduce the crash, I ran UDP upload traffic at 100 Mbps on one radio (wiphy1). Then I brought up a second station on another radio (wiphy0). The station on wiphy0 will then crash as it comes up. If there is no traffic flowing on the wiphy1 radio, or if it is only doing 30Mbps upload, the station doesn't crash.

- Traffic is over the air and not encrypted. I'm using our LANforge virtual AP on channel 36.

- I was also able to get a crash with an Asus AX6000 AP. My earlier method of reproducing the crash did not work. Basically I got it to crash running 160 Mbps UDP upload on one AX station then resetting 7 other stations (each on their own AX radio).

Firmware crash log:

Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: Microcode SW error detected. Restarting 0x0.
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: Start IWL Error Log Dump:
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: Status: 0x00000080, count: 6
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: Loaded firmware version: 48.4fa0041f.0
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000942 | ADVANCED_SYSASSERT          
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00A0A2F0 | trm_hw_status0
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000000 | trm_hw_status1
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x004F8E3C | branchlink2
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000E26 | interruptlink1
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000E26 | interruptlink2
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0xFFEFFFFC | data1
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x26010101 | data2
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x040C0605 | data3
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0xBB01867B | beacon time
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x92BDF98E | tsf low
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x0000000F | tsf hi
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000000 | time gp1
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x0AF4D781 | time gp2
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000001 | uCode revision type
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000030 | uCode version major
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x4FA0041F | uCode version minor
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000340 | hw version
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00C89000 | board version
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x807BFD06 | hcmd
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0xE7821000 | isr0
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00040000 | isr1
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x08F00002 | isr2
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00C1871D | isr3
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000000 | isr4
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x0016019C | last cmd Id
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x004EB8B4 | wait_event
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000000 | l2p_control
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000000 | l2p_duration
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x000000BF | l2p_mhvalid
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x0000000E | l2p_addr_match
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000009 | lmpm_pmg_sel
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000000 | timestamp
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x0000B0D0 | flow_handler
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: Start IWL Error Log Dump:
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: Status: 0x00000080, count: 7
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x20000070 | NMI_INTERRUPT_LMAC_FATAL
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000000 | umac branchlink1
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0xC008CC3C | umac branchlink2
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x8048D0E6 | umac interruptlink1
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x8048D0E6 | umac interruptlink2
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000400 | umac data1
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x8048D0E6 | umac data2
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000000 | umac data3
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000030 | umac major
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x4FA0041F | umac minor
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x0AF4D791 | frame pointer
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0xC0886284 | stack pointer
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x0017010C | last host cmd
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000000 | isr status reg
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: Fseq Registers:
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0xE0000000 | FSEQ_ERROR_CODE
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x80290001 | FSEQ_TOP_INIT_VERSION
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x80050008 | FSEQ_CNVIO_INIT_VERSION
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x0000A503 | FSEQ_OTP_VERSION
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x80000003 | FSEQ_TOP_CONTENT_VERSION
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x4552414E | FSEQ_ALIVE_TOKEN
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00100530 | FSEQ_CNVI_ID
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000532 | FSEQ_CNVR_ID
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00100530 | CNVI_AUX_MISC_CHIP
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x00000532 | CNVR_AUX_MISC_CHIP
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x05B0905B | CNVR_SCU_SD_REGS_SD_REG_DIG_DCDC_VTRIM
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: 0x0000025B | CNVR_SCU_SD_REGS_SD_REG_ACTIVE_VDIG_MIRROR
Jan  3 05:38:55 localhost kernel: iwlwifi 0000:0e:00.0: Collecting data: trigger 2 fired.
Jan  3 05:38:55 localhost kernel: ieee80211 wiphy5: Hardware restart was requested

We also have binary crash files available, but they are too big to attach to bugzilla.  Let me know if there is someplace helpful that I can send them.
Comment 1 Johannes 2020-04-09 10:46:07 UTC
We are experiencing the same bug. Is there an update yet?