Bug 203277

Summary: iwlwifi 8265 Microcode SW error detected (queue stuck)
Product: Drivers Reporter: Alexander Mikhalitsyn (alexander)
Component: network-wirelessAssignee: DO NOT USE - assign "network-wireless-intel" component instead (linuxwifi)
Status: CLOSED WILL_NOT_FIX    
Severity: normal CC: alexander
Priority: P1    
Hardware: Intel   
OS: Linux   
See Also: https://bugzilla.kernel.org/show_bug.cgi?id=202513
Kernel Version: 4.20 Subsystem:
Regression: No Bisected commit-id:
Attachments: Dumps and full dmesg output ~12:00 12.04.2019

Description Alexander Mikhalitsyn 2019-04-11 18:49:04 UTC
Hello,

For a long time I meet with the following problem with WiFi (part of dmesg output):
[28028.577327] iwlwifi 0000:04:00.0: Queue 10 is active on fifo 1 and stuck for 10000 ms. SW [193, 148] HW [193, 148] FH TRB=0x0c010a0d0
[28028.577474] iwlwifi 0000:04:00.0: Microcode SW error detected.  Restarting 0x2000000.

After that I lose the connection or see serious network performance problem.

1. Interesting notice that in Windows 10 with standard drivers I have not that problem.
2. It is very likely that the problem is due to interference of electromagnetic waves from wifi adapter...
Some links:
https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/1804841/comments/17
https://bugs.launchpad.net/ubuntu/+source/linux-firmware/+bug/1804841/comments/24
possibly problem is with transmitter power settings or something similar.

Kernel: I've tried several kernel versions and that have no any affect to a problem.
Firmware: 36.e91976c0.0 (I've also tried several versions but... no affect)
Hardware: Intel Corporation Wireless 8265 / 8275 (VivoBook_ASUSLaptop X580GD_N580GD)

I would like to notice that very similar situation described in bug
https://bugzilla.kernel.org/show_bug.cgi?id=202513
but it's already closed.

A few examples from dmesg:
1)
[28028.577327] iwlwifi 0000:04:00.0: Queue 10 is active on fifo 1 and stuck for 10000 ms. SW [193, 148] HW [193, 148] FH TRB=0x0c010a0d0
[28028.577474] iwlwifi 0000:04:00.0: Microcode SW error detected.  Restarting 0x2000000.
[28028.577692] iwlwifi 0000:04:00.0: Start IWL Error Log Dump:
[28028.577699] iwlwifi 0000:04:00.0: Status: 0x00000100, count: 6
[28028.577705] iwlwifi 0000:04:00.0: Loaded firmware version: 36.e91976c0.0
[28028.577712] iwlwifi 0000:04:00.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN       
[28028.577718] iwlwifi 0000:04:00.0: 0x008006F4 | trm_hw_status0
[28028.577724] iwlwifi 0000:04:00.0: 0x00000000 | trm_hw_status1
[28028.577729] iwlwifi 0000:04:00.0: 0x000248DC | branchlink2
[28028.577735] iwlwifi 0000:04:00.0: 0x0003A7DA | interruptlink1
[28028.577740] iwlwifi 0000:04:00.0: 0x00018918 | interruptlink2
[28028.577745] iwlwifi 0000:04:00.0: 0x00000000 | data1
[28028.577751] iwlwifi 0000:04:00.0: 0x00000080 | data2
[28028.577756] iwlwifi 0000:04:00.0: 0x07830000 | data3
[28028.577761] iwlwifi 0000:04:00.0: 0x28C0B72C | beacon time
[28028.577767] iwlwifi 0000:04:00.0: 0x548738D5 | tsf low
[28028.577772] iwlwifi 0000:04:00.0: 0x0000003D | tsf hi
[28028.577778] iwlwifi 0000:04:00.0: 0x00000000 | time gp1
[28028.577783] iwlwifi 0000:04:00.0: 0x01858D64 | time gp2
[28028.577789] iwlwifi 0000:04:00.0: 0x00000001 | uCode revision type
[28028.577794] iwlwifi 0000:04:00.0: 0x00000024 | uCode version major
[28028.577799] iwlwifi 0000:04:00.0: 0xE91976C0 | uCode version minor
[28028.577805] iwlwifi 0000:04:00.0: 0x00000230 | hw version
[28028.577810] iwlwifi 0000:04:00.0: 0x00489000 | board version
[28028.577815] iwlwifi 0000:04:00.0: 0x0AC1001C | hcmd
[28028.577821] iwlwifi 0000:04:00.0: 0x80022003 | isr0
[28028.577826] iwlwifi 0000:04:00.0: 0x00800000 | isr1
[28028.577831] iwlwifi 0000:04:00.0: 0x0800180A | isr2
[28028.577837] iwlwifi 0000:04:00.0: 0x00417CC5 | isr3
[28028.577842] iwlwifi 0000:04:00.0: 0x00000000 | isr4
[28028.577847] iwlwifi 0000:04:00.0: 0x00D5019C | last cmd Id
[28028.577852] iwlwifi 0000:04:00.0: 0x00000000 | wait_event
[28028.577858] iwlwifi 0000:04:00.0: 0x00004305 | l2p_control
[28028.577863] iwlwifi 0000:04:00.0: 0x00002020 | l2p_duration
[28028.577869] iwlwifi 0000:04:00.0: 0x00000003 | l2p_mhvalid
[28028.577874] iwlwifi 0000:04:00.0: 0x000000FE | l2p_addr_match
[28028.577879] iwlwifi 0000:04:00.0: 0x0000000D | lmpm_pmg_sel
[28028.577885] iwlwifi 0000:04:00.0: 0x28031619 | timestamp
[28028.577890] iwlwifi 0000:04:00.0: 0x00342030 | flow_handler
[28028.577984] iwlwifi 0000:04:00.0: 0x00000000 | ADVANCED_SYSASSERT
[28028.577990] iwlwifi 0000:04:00.0: 0x00000000 | umac branchlink1
[28028.577995] iwlwifi 0000:04:00.0: 0x00000000 | umac branchlink2
[28028.578000] iwlwifi 0000:04:00.0: 0x00000000 | umac interruptlink1
[28028.578005] iwlwifi 0000:04:00.0: 0x00000000 | umac interruptlink2
[28028.578011] iwlwifi 0000:04:00.0: 0x00000000 | umac data1
[28028.578016] iwlwifi 0000:04:00.0: 0x00000000 | umac data2
[28028.578021] iwlwifi 0000:04:00.0: 0x00000000 | umac data3
[28028.578026] iwlwifi 0000:04:00.0: 0x00000000 | umac major
[28028.578031] iwlwifi 0000:04:00.0: 0x00000000 | umac minor
[28028.578037] iwlwifi 0000:04:00.0: 0x00000000 | frame pointer
[28028.578042] iwlwifi 0000:04:00.0: 0x00000000 | stack pointer
[28028.578047] iwlwifi 0000:04:00.0: 0x00000000 | last host cmd
[28028.578052] iwlwifi 0000:04:00.0: 0x00000000 | isr status reg
[28028.578061] ieee80211 phy0: Hardware restart was requested


2)
[32770.273182] iwlwifi 0000:04:00.0: Queue 10 is active on fifo 1 and stuck for 10000 ms. SW [163, 237] HW [163, 237] FH TRB=0x0c010a0b2
[32770.273343] iwlwifi 0000:04:00.0: Microcode SW error detected.  Restarting 0x2000000.
[32770.273558] iwlwifi 0000:04:00.0: Start IWL Error Log Dump:
[32770.273566] iwlwifi 0000:04:00.0: Status: 0x00000100, count: 6
[32770.273572] iwlwifi 0000:04:00.0: Loaded firmware version: 36.e91976c0.0
[32770.273579] iwlwifi 0000:04:00.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN       
[32770.273584] iwlwifi 0000:04:00.0: 0x008006F4 | trm_hw_status0
[32770.273590] iwlwifi 0000:04:00.0: 0x00000000 | trm_hw_status1
[32770.273595] iwlwifi 0000:04:00.0: 0x000248DC | branchlink2
[32770.273601] iwlwifi 0000:04:00.0: 0x0003A7DA | interruptlink1
[32770.273606] iwlwifi 0000:04:00.0: 0x00004230 | interruptlink2
[32770.273611] iwlwifi 0000:04:00.0: 0x00000000 | data1
[32770.273617] iwlwifi 0000:04:00.0: 0x00000080 | data2
[32770.273622] iwlwifi 0000:04:00.0: 0x07830000 | data3
[32770.273627] iwlwifi 0000:04:00.0: 0x0DC00CDD | beacon time
[32770.273633] iwlwifi 0000:04:00.0: 0x6F277323 | tsf low
[32770.273638] iwlwifi 0000:04:00.0: 0x0000003E | tsf hi
[32770.273644] iwlwifi 0000:04:00.0: 0x00000000 | time gp1
[32770.273649] iwlwifi 0000:04:00.0: 0x00BA7DAA | time gp2
[32770.273654] iwlwifi 0000:04:00.0: 0x00000001 | uCode revision type
[32770.273660] iwlwifi 0000:04:00.0: 0x00000024 | uCode version major
[32770.273665] iwlwifi 0000:04:00.0: 0xE91976C0 | uCode version minor
[32770.273670] iwlwifi 0000:04:00.0: 0x00000230 | hw version
[32770.273676] iwlwifi 0000:04:00.0: 0x00489000 | board version
[32770.273681] iwlwifi 0000:04:00.0: 0x0AA3001C | hcmd
[32770.273687] iwlwifi 0000:04:00.0: 0x80022002 | isr0
[32770.273692] iwlwifi 0000:04:00.0: 0x00800000 | isr1
[32770.273697] iwlwifi 0000:04:00.0: 0x0800180A | isr2
[32770.273702] iwlwifi 0000:04:00.0: 0x004168C5 | isr3
[32770.273708] iwlwifi 0000:04:00.0: 0x00000000 | isr4
[32770.273713] iwlwifi 0000:04:00.0: 0x005F019C | last cmd Id
[32770.273718] iwlwifi 0000:04:00.0: 0x00000000 | wait_event
[32770.273724] iwlwifi 0000:04:00.0: 0x0000B33B | l2p_control
[32770.273729] iwlwifi 0000:04:00.0: 0x00002020 | l2p_duration
[32770.273734] iwlwifi 0000:04:00.0: 0x00000003 | l2p_mhvalid
[32770.273739] iwlwifi 0000:04:00.0: 0x000000FE | l2p_addr_match
[32770.273745] iwlwifi 0000:04:00.0: 0x0000000D | lmpm_pmg_sel
[32770.273750] iwlwifi 0000:04:00.0: 0x28031619 | timestamp
[32770.273755] iwlwifi 0000:04:00.0: 0x00341018 | flow_handler
[32770.273850] iwlwifi 0000:04:00.0: 0x00000000 | ADVANCED_SYSASSERT
[32770.273856] iwlwifi 0000:04:00.0: 0x00000000 | umac branchlink1
[32770.273861] iwlwifi 0000:04:00.0: 0x00000000 | umac branchlink2
[32770.273866] iwlwifi 0000:04:00.0: 0x00000000 | umac interruptlink1
[32770.273872] iwlwifi 0000:04:00.0: 0x00000000 | umac interruptlink2
[32770.273877] iwlwifi 0000:04:00.0: 0x00000800 | umac data1
[32770.273882] iwlwifi 0000:04:00.0: 0xC0083A94 | umac data2
[32770.273888] iwlwifi 0000:04:00.0: 0xDEADBEEF | umac data3
[32770.273893] iwlwifi 0000:04:00.0: 0x00000024 | umac major
[32770.273898] iwlwifi 0000:04:00.0: 0xE91976C0 | umac minor
[32770.273904] iwlwifi 0000:04:00.0: 0xC088628C | frame pointer
[32770.273909] iwlwifi 0000:04:00.0: 0xC088628C | stack pointer
[32770.273914] iwlwifi 0000:04:00.0: 0x005F019C | last host cmd
[32770.273919] iwlwifi 0000:04:00.0: 0x00000000 | isr status reg
[32770.273929] ieee80211 phy0: Hardware restart was requested

Thanks.
Comment 1 Emmanuel Grumbach 2019-04-12 07:14:01 UTC
Hello,

Please refer to https://wireless.wiki.kernel.org/en/users/drivers/iwlwifi/debugging#firmware_debugging

You'll need to install the debug version of the firmware and collect the debug data as explained. Please take the time to read the privacy notice.
Comment 2 Alexander Mikhalitsyn 2019-04-12 08:59:20 UTC
Hello,

I've tried to collect dump but.. nothing happened after firmware crashes. And there are no /sys/devices/virtual/devcoredump/... devices at all.

Of course I've removed all firmwares and add debug firmware and executed
$ echo 1 > /sys/kernel/debug/iwlwifi/0000\:04\:00.0/iwlmvm/fw_dbg_collect
as needed.

1) Kernel configuration:
$ cat "/boot/config-`uname -r`" | grep COREDUMP
CONFIG_COREDUMP=y
CONFIG_WANT_DEV_COREDUMP=y
CONFIG_ALLOW_DEV_COREDUMP=y
CONFIG_DEV_COREDUMP=y

2)
$ ls /sys/devices/virtual/
bdi  block  dmi  graphics  input  mem  misc  msr  net  powercap  ppp  sound  thermal  tty  vc  vtconsole  workqueue

3) dmesg:

[ 1414.356465] iwlwifi 0000:04:00.0: Collecting data: trigger 1 fired.
[ 1454.603978] iwlwifi 0000:04:00.0: Queue 10 is active on fifo 1 and stuck for 10000 ms. SW [51, 91] HW [51, 90] FH TRB=0x0c010a042
[ 1454.604180] iwlwifi 0000:04:00.0: Microcode SW error detected.  Restarting 0x2000000.
[ 1454.604482] iwlwifi 0000:04:00.0: Start IWL Error Log Dump:
[ 1454.604490] iwlwifi 0000:04:00.0: Status: 0x00000100, count: 6
[ 1454.604495] iwlwifi 0000:04:00.0: Loaded firmware version: 36.9f0a2d68.0
[ 1454.604502] iwlwifi 0000:04:00.0: 0x00000084 | NMI_INTERRUPT_UNKNOWN       
[ 1454.604508] iwlwifi 0000:04:00.0: 0x008006F4 | trm_hw_status0
[ 1454.604514] iwlwifi 0000:04:00.0: 0x00000000 | trm_hw_status1
[ 1454.604520] iwlwifi 0000:04:00.0: 0x0002485C | branchlink2
[ 1454.604525] iwlwifi 0000:04:00.0: 0x0003A7CA | interruptlink1
[ 1454.604530] iwlwifi 0000:04:00.0: 0x00006E00 | interruptlink2
[ 1454.604536] iwlwifi 0000:04:00.0: 0x00000000 | data1
[ 1454.604541] iwlwifi 0000:04:00.0: 0x00000080 | data2
[ 1454.604546] iwlwifi 0000:04:00.0: 0x07830000 | data3
[ 1454.604552] iwlwifi 0000:04:00.0: 0x28C0F21E | beacon time
[ 1454.604557] iwlwifi 0000:04:00.0: 0xF0EEADE1 | tsf low
[ 1454.604563] iwlwifi 0000:04:00.0: 0x0000004C | tsf hi
[ 1454.604568] iwlwifi 0000:04:00.0: 0x00000000 | time gp1
[ 1454.604574] iwlwifi 0000:04:00.0: 0x07767394 | time gp2
[ 1454.604579] iwlwifi 0000:04:00.0: 0x00000001 | uCode revision type
[ 1454.604585] iwlwifi 0000:04:00.0: 0x00000024 | uCode version major
[ 1454.604590] iwlwifi 0000:04:00.0: 0x9F0A2D68 | uCode version minor
[ 1454.604596] iwlwifi 0000:04:00.0: 0x00000230 | hw version
[ 1454.604601] iwlwifi 0000:04:00.0: 0x18489000 | board version
[ 1454.604606] iwlwifi 0000:04:00.0: 0x0A33001C | hcmd
[ 1454.604612] iwlwifi 0000:04:00.0: 0x80022002 | isr0
[ 1454.604617] iwlwifi 0000:04:00.0: 0x00800000 | isr1
[ 1454.604622] iwlwifi 0000:04:00.0: 0x0800180A | isr2
[ 1454.604627] iwlwifi 0000:04:00.0: 0x00417CC5 | isr3
[ 1454.604632] iwlwifi 0000:04:00.0: 0x00000000 | isr4
[ 1454.604638] iwlwifi 0000:04:00.0: 0x8D3D009D | last cmd Id
[ 1454.604643] iwlwifi 0000:04:00.0: 0x00000000 | wait_event
[ 1454.604648] iwlwifi 0000:04:00.0: 0x000087B6 | l2p_control
[ 1454.604654] iwlwifi 0000:04:00.0: 0x00002020 | l2p_duration
[ 1454.604659] iwlwifi 0000:04:00.0: 0x00000003 | l2p_mhvalid
[ 1454.604664] iwlwifi 0000:04:00.0: 0x00000030 | l2p_addr_match
[ 1454.604670] iwlwifi 0000:04:00.0: 0x0000000D | lmpm_pmg_sel
[ 1454.604675] iwlwifi 0000:04:00.0: 0x04120134 | timestamp
[ 1454.604681] iwlwifi 0000:04:00.0: 0x00340010 | flow_handler
[ 1454.604791] iwlwifi 0000:04:00.0: Start IWL Error Log Dump:
[ 1454.604797] iwlwifi 0000:04:00.0: Status: 0x00000100, count: 7
[ 1454.604803] iwlwifi 0000:04:00.0: 0x00000070 | ADVANCED_SYSASSERT
[ 1454.604809] iwlwifi 0000:04:00.0: 0x00000000 | umac branchlink1
[ 1454.604814] iwlwifi 0000:04:00.0: 0xC00868A4 | umac branchlink2
[ 1454.604819] iwlwifi 0000:04:00.0: 0xC0083A94 | umac interruptlink1
[ 1454.604825] iwlwifi 0000:04:00.0: 0xC0083A94 | umac interruptlink2
[ 1454.604830] iwlwifi 0000:04:00.0: 0x00000800 | umac data1
[ 1454.604835] iwlwifi 0000:04:00.0: 0xC0083A94 | umac data2
[ 1454.604840] iwlwifi 0000:04:00.0: 0xDEADBEEF | umac data3
[ 1454.604846] iwlwifi 0000:04:00.0: 0x00000024 | umac major
[ 1454.604851] iwlwifi 0000:04:00.0: 0x9F0A2D68 | umac minor
[ 1454.604856] iwlwifi 0000:04:00.0: 0xC088628C | frame pointer
[ 1454.604862] iwlwifi 0000:04:00.0: 0xC088628C | stack pointer
[ 1454.604867] iwlwifi 0000:04:00.0: 0x00C3019C | last host cmd
[ 1454.604873] iwlwifi 0000:04:00.0: 0x00000000 | isr status reg
[ 1454.604882] ieee80211 phy1: Hardware restart was requested
[ 1455.273529] iwlwifi 0000:04:00.0: Applying debug destination EXTERNAL_DRAM
[ 1455.408127] iwlwifi 0000:04:00.0: Applying debug destination EXTERNAL_DRAM
[ 1455.479145] iwlwifi 0000:04:00.0: FW already configured (0) - re-configuring
Comment 3 Alexander Mikhalitsyn 2019-04-12 09:11:48 UTC
Oh... my bad. Problem with udevd daemon... :(
Comment 4 Alexander Mikhalitsyn 2019-04-12 09:13:21 UTC
Created attachment 282317 [details]
Dumps and full dmesg output ~12:00 12.04.2019
Comment 5 Emmanuel Grumbach 2019-04-14 17:32:24 UTC
I have looked at the data.

What I see is that we are trying to send data over the air, but we constantly hear noise on the channel. Note that it seems that we hear noise on the extension channel (because we seem to be on 40MHz configuration). You may try to reduce the bandwidth to 20MHz, but you intuition is perfectly right.
This is a radio problem that we can't fix from the driver.
I'll close this bug since we can't do anything about it besides maybe reducing the bandwidth and the other work arounds that you seem to have already found.
Comment 6 Emmanuel Grumbach 2019-04-14 17:32:55 UTC
Closing, please re-open if you feel the need.
We will be copied to anything you'll add in the bug regardless of its state.
Comment 7 Alexander Mikhalitsyn 2019-04-15 07:10:33 UTC
Thank you very much. I will play with options and post workaround. May be it will be helpful for someone..