Bug 93461

Summary: iwlmvm: 7260: IBSS causes panic - MWG100229852
Product: Drivers Reporter: Richard Taylor (rjt-kernel)
Component: network-wirelessAssignee: drivers_network-wireless (drivers_network-wireless)
Status: CLOSED CODE_FIX    
Severity: high CC: haim.dreyfuss, ilw, linville
Priority: P1    
Hardware: Intel   
OS: Linux   
Kernel Version: 3.19 Subsystem:
Regression: No Bisected commit-id:
Attachments: dmesg
iw list
Full console log
syslog output
first_with_debug.syslog
first_with_debug.trace.tgz
first_with_debug.out
patch_with_debug.syslog
patch_with_debug.out
new_patch.trace.dat
new_patch.syslog

Description Richard Taylor 2015-02-18 16:35:37 UTC
Created attachment 167491 [details]
dmesg

Kernel panic when receiving an ICMP packet.

I am happy to help conduct further tests to narrow this down.

To reproduce:

1. Setup ibss net:

    ip link set dev wlp2s0 down
    iw wlp2s0 set type ibss
    ip link set dev wlp2s0 up
    iw wlp2s0 ibss join mynet 2412
    ifconfig wlp2s0 10.2.1.201 netmask 255.255.255.0

   --- log messages ---

   [  149.106858] wlp2s0: Trigger new scan to find an IBSS to join
   [  149.384256] wlp2s0: Selected IBSS BSSID 4a:25:92:b8:f3:9d based on configured SSID
   [  149.387053] IPv6: ADDRCONF(NETDEV_CHANGE): wlp2s0: link becomes ready

2. ping known host on net:

    ping -c 1 10.2.1.11   # This works

3. leave net:

    iw wlp2s0 ibss leave

4. Rejoin net:

    ip link set dev wlp2s0 down
    iw wlp2s0 set type ibss
    ip link set dev wlp2s0 up
    iw wlp2s0 ibss join mynet 2412
    ifconfig wlp2s0 10.2.1.201 netmask 255.255.255.0

5. This produced a traceback (but not a panic):

    [  149.726300] ------------[ cut here ]------------
[  149.727341] WARNING: CPU: 1 PID: 100 at drivers/net/wireless/iwlwifi/mvm/sta.c:273 iwl_mvm_add_sta+0x1b0/0x340 [iwlmvm]()
[  149.728136] Modules linked in: netconsole bridge stp llc arc4 iwlmvm bnep mac80211 intel_rapl iosf_mbi x86_pkg_temp_thermal[  149.728314] ------------[ cut here ]------------
[  149.728317] WARNING: CPU: 1 PID: 100 at net/core/netpoll.c:362 netpoll_send_skb_on_dev+0x25e/0x270()
[  149.728332] netpoll_send_skb_on_dev(): eno1 enabled interrupts in poll (e1000_xmit_frame+0x0/0xf70 [e1000e])
[  149.728333] Modules linked in: netconsole bridge stp llc arc4 iwlmvm bnep mac80211 intel_rapl iosf_mbi x86_pkg_temp_thermal coretemp kvm_intel kvm iwlwifi snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic iTCO_wdt iTCO_vendor_support dell_laptop snd_hda_intel dcdbas crct10dif_pclmul snd_hda_controller dell_wmi crc32_pclmul snd_hda_codec sparse_keymap cfg80211 crc32c_intel i8k snd_hwdep snd_seq ghash_clmulni_intel pcspkr serio_raw snd_seq_device btusb uvcvideo snd_pcm bluetooth videobuf2_vmalloc videobuf2_core videobuf2_memops v4l2_common videodev i2c_i801 joydev lpc_ich snd_timer media mfd_core usbtouchscreen mei_me snd i2c_hid mei sdhci_pci rfkill soundcore dw_dmac shpchp i2c_designware_platform dell_smo8800 i2c_designware_core dw_dmac_core nfsd auth_rpcgss nfs_acl lockd grace sunrpc nouveau i915 ttm i2c_algo_bit drm_kms_helper drm e1000e mxm_wmi sdhci_acpi ptp sdhci mmc_core pps_core wmi video r8152 mii [last unloaded: iptable_raw]
[  149.728355] CPU: 1 PID: 100 Comm: kworker/u8:3 Not tainted 3.19.0 #2
[  149.728355] Hardware name: Dell Inc. Latitude 14 Rugged (5404)/07RKHG, BIOS A01 10/06/2014
[  149.728365] Workqueue: phy0 ieee80211_iface_work [mac80211]
[  149.728365]  0000000000000000 000000001eb6534b ffff88021df9b308 ffffffff81763ee4
[  149.728367]  0000000000000000 ffff88021df9b360 ffff88021df9b348 ffffffff8109adfa
[  149.728368]  0000000600000000 ffff8800c2abbc00 ffff8800ca391090 ffff880033c9f200
[  149.728369] Call Trace:
[  149.728370]  [<ffffffff81763ee4>] dump_stack+0x45/0x57
[  149.728372]  [<ffffffff8109adfa>] warn_slowpath_common+0x8a/0xc0
[  149.728375]  [<ffffffff8109ae85>] warn_slowpath_fmt+0x55/0x70
[  149.728376]  [<ffffffff8139c49b>] ? delay_tsc+0x3b/0x80
[  149.728379]  [<ffffffffa00a79d0>] ? e1000_clean_tx_ring+0x100/0x100 [e1000e]
[  149.728384]  [<ffffffff8166d10e>] netpoll_send_skb_on_dev+0x25e/0x270
[  149.728385]  [<ffffffff8166d3e2>] netpoll_send_udp+0x2c2/0x410
[  149.728387]  [<ffffffffa063f92f>] write_msg+0xcf/0x140 [netconsole]
[  149.728388]  [<ffffffff810ed6ea>] call_console_drivers.constprop.22+0xaa/0xf0
[  149.728390]  [<ffffffff810eec2d>] console_unlock+0x1dd/0x440
[  149.728391]  [<ffffffff810e07be>] ? down_trylock+0x2e/0x40
[  149.728393]  [<ffffffff810ef240>] vprintk_emit+0x3b0/0x550
[  149.728394]  [<ffffffffa08a5ae0>] ? iwl_mvm_add_sta+0x1b0/0x340 [iwlmvm]
[  149.728399]  [<ffffffff810ef569>] vprintk_default+0x29/0x40
[  149.728400]  [<ffffffff81762dc2>] printk+0x55/0x6b
[  149.728401]  [<ffffffff8111d503>] print_modules+0xa3/0xd0
[  149.728403]  [<ffffffff8109adf5>] warn_slowpath_common+0x85/0xc0
[  149.728404]  [<ffffffff8109af2a>] warn_slowpath_null+0x1a/0x20
[  149.728406]  [<ffffffffa08a5ae0>] iwl_mvm_add_sta+0x1b0/0x340 [iwlmvm]
[  149.728409]  [<ffffffffa08982ff>] iwl_mvm_mac_sta_state+0x25f/0x420 [iwlmvm]
[  149.728412]  [<ffffffffa092ebfe>] sta_info_insert_finish+0x15e/0x800 [mac80211]
[  149.728416]  [<ffffffffa0799987>] ? __iwl_dbg+0xf7/0x130 [iwlwifi]
[  149.728420]  [<ffffffffa092f310>] sta_info_insert_rcu+0x70/0xd0 [mac80211]
[  149.728424]  [<ffffffffa093ad65>] ieee80211_ibss_finish_sta+0x115/0x2c0 [mac80211]
[  149.728429]  [<ffffffffa093afe8>] ieee80211_ibss_add_sta+0xd8/0x170 [mac80211]
[  149.728434]  [<ffffffffa093c27a>] ieee80211_rx_mgmt_probe_beacon+0x5fa/0x8d0 [mac80211]
[  149.728439]  [<ffffffffa0951690>] ? __ieee80211_tx+0x2d0/0x3d0 [mac80211]
[  149.728446]  [<ffffffffa093cca0>] ieee80211_ibss_rx_queued_mgmt+0x3c0/0x470 [mac80211]
[  149.728450]  [<ffffffff811f5aad>] ? __slab_free+0xbd/0x300
[  149.728452]  [<ffffffff811f5aad>] ? __slab_free+0xbd/0x300
[  149.728453]  [<ffffffff811f60be>] ? kfree+0x14e/0x160
[  149.728454]  [<ffffffffa093e9f8>] ? ieee80211_iface_work+0x108/0x460 [mac80211]
[  149.728458]  [<ffffffff811f5eb6>] ? kmem_cache_free+0x1c6/0x210
[  149.728459]  [<ffffffffa093e9f8>] ? ieee80211_iface_work+0x108/0x460 [mac80211]
[  149.728464]  [<ffffffffa093ec2b>] ieee80211_iface_work+0x33b/0x460 [mac80211]
[  149.728468]  [<ffffffff811617e1>] ? tracing_record_cmdline+0x21/0x120
[  149.728470]  [<ffffffff810b357b>] process_one_work+0x14b/0x3f0
[  149.728472]  [<ffffffff810b3f43>] worker_thread+0x53/0x470
[  149.728474]  [<ffffffff810b3ef0>] ? rescuer_thread+0x300/0x300
[  149.728475]  [<ffffffff810b8e08>] kthread+0xd8/0xf0
[  149.728477]  [<ffffffff810b8d30>] ? kthread_create_on_node+0x1b0/0x1b0
[  149.728478]  [<ffffffff8176a5bc>] ret_from_fork+0x7c/0xb0
[  149.728479]  [<ffffffff810b8d30>] ? kthread_create_on_node+0x1b0/0x1b0
[  149.728481] ---[ end trace db9b5455a6d7770c ]---
    
6. ping known host (may take a few pings and sometimes requires a incoming ping to trigger panic):

[  149.775009]  coretemp kvm_intel kvm iwlwifi snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic iTCO_wdt iTCO_vendor_support dell_laptop snd_hda_intel dcdbas crct10dif_pclmul snd_hda_controller dell_wmi crc32_pclmul snd_hda_codec sparse_keymap cfg80211 crc32c_intel i8k snd_hwdep snd_seq ghash_clmulni_intel pcspkr serio_raw snd_seq_device btusb uvcvideo snd_pcm bluetooth videobuf2_vmalloc videobuf2_core videobuf2_memops v4l2_common videodev i2c_i801 joydev lpc_ich snd_timer media mfd_core usbtouchscreen mei_me snd i2c_hid mei sdhci_pci rfkill soundcore dw_dmac shpchp i2c_designware_platform dell_smo8800 i2c_designware_core dw_dmac_core nfsd auth_rpcgss nfs_acl lockd grace sunrpc nouveau i915 ttm i2c_algo_bit drm_kms_helper drm e1000e mxm_wmi sdhci_acpi ptp sdhci mmc_core pps_core wmi video r8152 mii [last unloaded: iptable_raw]
[  149.779028] CPU: 1 PID: 100 Comm: kworker/u8:3 Tainted: G        W      3.19.0 #2
[  149.779682] Hardware name: Dell Inc. Latitude 14 Rugged (5404)/07RKHG, BIOS A01 10/06/2014
[  149.780329] Workqueue: phy0 ieee80211_iface_work [mac80211]
[  149.780948]  0000000000000000 000000001eb6534b ffff88021df9b678 ffffffff81763ee4
[  149.781573]  0000000000000000 0000000000000000 ffff88021df9b6b8 ffffffff8109adfa
[  149.782219]  0000000000000000 ffff8802217ec680 0000000000000010 0000000000000001
[  149.782860] Call Trace:
[  149.783483]  [<ffffffff81763ee4>] dump_stack+0x45/0x57
[  149.784118]  [<ffffffff8109adfa>] warn_slowpath_common+0x8a/0xc0
[  149.784738]  [<ffffffff8109af2a>] warn_slowpath_null+0x1a/0x20
[  149.785430]  [<ffffffffa08a5ae0>] iwl_mvm_add_sta+0x1b0/0x340 [iwlmvm]
[  149.786065]  [<ffffffffa08982ff>] iwl_mvm_mac_sta_state+0x25f/0x420 [iwlmvm]
[  149.786709]  [<ffffffffa092ebfe>] sta_info_insert_finish+0x15e/0x800 [mac80211]
[  149.787400]  [<ffffffffa0799987>] ? __iwl_dbg+0xf7/0x130 [iwlwifi]
[  149.788045]  [<ffffffffa092f310>] sta_info_insert_rcu+0x70/0xd0 [mac80211]
[  149.788693]  [<ffffffffa093ad65>] ieee80211_ibss_finish_sta+0x115/0x2c0 [mac80211]
[  149.789391]  [<ffffffffa093afe8>] ieee80211_ibss_add_sta+0xd8/0x170 [mac80211]
[  149.790041]  [<ffffffffa093c27a>] ieee80211_rx_mgmt_probe_beacon+0x5fa/0x8d0 [mac80211]
[  149.790694]  [<ffffffffa0951690>] ? __ieee80211_tx+0x2d0/0x3d0 [mac80211]
[  149.791361]  [<ffffffffa093cca0>] ieee80211_ibss_rx_queued_mgmt+0x3c0/0x470 [mac80211]
[  149.792012]  [<ffffffff811f5aad>] ? __slab_free+0xbd/0x300
[  149.792685]  [<ffffffff811f5aad>] ? __slab_free+0xbd/0x300
[  149.793341]  [<ffffffff811f60be>] ? kfree+0x14e/0x160
[  149.793989]  [<ffffffffa093e9f8>] ? ieee80211_iface_work+0x108/0x460 [mac80211]
[  149.794647]  [<ffffffff811f5eb6>] ? kmem_cache_free+0x1c6/0x210
[  149.795315]  [<ffffffffa093e9f8>] ? ieee80211_iface_work+0x108/0x460 [mac80211]
[  149.795967]  [<ffffffffa093ec2b>] ieee80211_iface_work+0x33b/0x460 [mac80211]
[  149.796611]  [<ffffffff811617e1>] ? tracing_record_cmdline+0x21/0x120
[  149.797290]  [<ffffffff810b357b>] process_one_work+0x14b/0x3f0
[  149.797939]  [<ffffffff810b3f43>] worker_thread+0x53/0x470
[  149.798571]  [<ffffffff810b3ef0>] ? rescuer_thread+0x300/0x300
[  149.799260]  [<ffffffff810b8e08>] kthread+0xd8/0xf0
[  149.799908]  [<ffffffff810b8d30>] ? kthread_create_on_node+0x1b0/0x1b0
[  149.800538]  [<ffffffff8176a5bc>] ret_from_fork+0x7c/0xb0
[  149.801194]  [<ffffffff810b8d30>] ? kthread_create_on_node+0x1b0/0x1b0
[  149.801866] ---[ end trace db9b5455a6d7770d ]---
[  149.802522] wlp2s0: failed to move IBSS STA 00:26:c6:49:15:1e to state 1 (-28) - keeping it anyway
[  303.228612] wlp2s0: failed to move IBSS STA 28:b2:bd:82:5c:99 to state 1 (-28) - keeping it anyway
[  320.924196] iwlwifi 0000:02:00.0: L1 Disabled - LTR Enabled
[  320.925083] iwlwifi 0000:02:00.0: L1 Disabled - LTR Enabled
[  320.938578] IPv6: ADDRCONF(NETDEV_UP): wlp2s0: link is not ready
[  323.218639] wlp2s0: Trigger new scan to find an IBSS to join
[  323.497962] wlp2s0: Selected IBSS BSSID 4a:25:92:b8:f3:9d based on configured SSID
[  323.502403] IPv6: ADDRCONF(NETDEV_CHANGE): wlp2s0: link becomes ready
[  323.702991] wlp2s0: failed to move IBSS STA 00:1e:65:48:5d:68 to state 1 (-28) - keeping it anyway
[  339.966162] wlp2s0: failed to move IBSS STA 28:b2:bd:82:5c:99 to state 1 (-28) - keeping it anyway
[  340.971343] BUG: unable to handle kernel NULL pointer dereference at           (null)
[  340.972207] IP: [<ffffffffa08ae073>] rs_fill_lq_cmd+0x83/0x3e0 [iwlmvm]
[  340.973018] PGD 0 
[  340.973601] Oops: 0000 [#1] SMP 
[  340.974173] Modules linked in: netconsole bridge stp llc arc4 iwlmvm bnep mac80211 intel_rapl iosf_mbi x86_pkg_temp_thermal coretemp kvm_intel kvm iwlwifi snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic iTCO_wdt iTCO_vendor_support dell_laptop snd_hda_intel dcdbas crct10dif_pclmul snd_hda_controller dell_wmi crc32_pclmul snd_hda_codec sparse_keymap cfg80211 crc32c_intel i8k snd_hwdep snd_seq ghash_clmulni_intel pcspkr serio_raw snd_seq_device btusb uvcvideo snd_pcm bluetooth videobuf2_vmalloc videobuf2_core videobuf2_memops v4l2_common videodev i2c_i801 joydev lpc_ich snd_timer media mfd_core usbtouchscreen mei_me snd i2c_hid mei sdhci_pci rfkill soundcore dw_dmac shpchp i2c_designware_platform dell_smo8800 i2c_designware_core dw_dmac_core nfsd auth_rpcgss nfs_acl lockd grace sunrpc nouveau i915 ttm i2c_algo_bit drm_kms_helper drm e1000e mxm_wmi sdhci_acpi ptp sdhci mmc_core pps_core wmi video r8152 mii [last unloaded: iptable_raw]
[  340.978354] CPU: 1 PID: 703 Comm: irq/50-iwlwifi Tainted: G        W      3.19.0 #2
[  340.979060] Hardware name: Dell Inc. Latitude 14 Rugged (5404)/07RKHG, BIOS A01 10/06/2014
[  340.979774] task: ffff88021c509bc0 ti: ffff8802249cc000 task.ti: ffff8802249cc000
[  340.980494] RIP: 0010:[<ffffffffa08ae073>]  [<ffffffffa08ae073>] rs_fill_lq_cmd+0x83/0x3e0 [iwlmvm]
[  340.981235] RSP: 0018:ffff8802249cf908  EFLAGS: 00010246
[  340.981969] RAX: 0000000000000000 RBX: ffff8802217ee7b0 RCX: ffff8802217ee868
[  340.982710] RDX: 0000000000000000 RSI: ffff8802217ee680 RDI: 0000000000000000
[  340.983447] RBP: ffff8802249cf988 R08: 00000000000005dc R09: ffff880223b0b5f4
[  340.984190] R10: 0000000000000000 R11: 00000000000005dc R12: ffff8802217ee680
[  340.984923] R13: ffff8802237bdac8 R14: ffff8802217ee680 R15: ffff8802217ee868
[  340.985653] FS:  0000000000000000(0000) GS:ffff88022ea80000(0000) knlGS:0000000000000000
[  340.986385] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  340.987115] CR2: 0000000000000000 CR3: 0000000001c11000 CR4: 00000000001407e0
[  340.987852] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  340.988589] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  340.989317] Stack:
[  340.990037]  ffffffffa08cd24b ffff8802249cf918 ffff880200000030 ffff8802249cf9a0
[  340.990766]  ffff8802249cf938 00000000c7a977f4 0000000000000000 0000000100000001
[  340.991485]  0000000000000000 00000000c7a977f4 0000000000000001 ffff8802237bdac8
[  340.992206] Call Trace:
[  340.992918]  [<ffffffffa08aec65>] iwl_mvm_rs_rate_init+0x5b5/0x7e0 [iwlmvm]
[  340.993649]  [<ffffffffa08af5b8>] iwl_mvm_rs_tx_status+0x728/0x21e0 [iwlmvm]
[  340.994378]  [<ffffffffa07ab3ec>] ? ftrace_raw_event_iwlwifi_dbg+0xec/0x1b0 [iwlwifi]
[  340.995106]  [<ffffffffa08b10c2>] rs_mac80211_tx_status+0x52/0x80 [iwlmvm]
[  340.995830]  [<ffffffffa092ca44>] ieee80211_tx_status+0x244/0xd90 [mac80211]
[  340.996545]  [<ffffffff81769f2a>] ? _raw_spin_unlock_bh+0x1a/0x20
[  340.997252]  [<ffffffffa07a23e1>] ? iwl_trans_pcie_reclaim+0x211/0x3b0 [iwlwifi]
[  340.997953]  [<ffffffffa08a3795>] iwl_mvm_rx_tx_cmd+0x405/0x7c0 [iwlmvm]
[  340.998657]  [<ffffffff81172b42>] ? ftrace_event_buffer_commit+0xa2/0x1b0
[  340.999364]  [<ffffffffa089d765>] iwl_mvm_rx_dispatch+0x165/0x1b0 [iwlmvm]
[  341.000070]  [<ffffffffa079ec38>] iwl_pcie_irq_handler+0xac8/0x1520 [iwlwifi]
[  341.000791]  [<ffffffff810d572b>] ? pick_next_task_fair+0x1bb/0x8b0
[  341.001518]  [<ffffffff810f1930>] ? irq_finalize_oneshot.part.30+0xf0/0xf0
[  341.002243]  [<ffffffff810f1950>] irq_thread_fn+0x20/0x50
[  341.002966]  [<ffffffff810f1c6f>] irq_thread+0x13f/0x170
[  341.003679]  [<ffffffff810f19f0>] ? irq_forced_thread_fn+0x70/0x70
[  341.004391]  [<ffffffff810f1b30>] ? irq_thread_check_affinity+0x90/0x90
[  341.005090]  [<ffffffff810b8e08>] kthread+0xd8/0xf0
[  341.005769]  [<ffffffff810b8d30>] ? kthread_create_on_node+0x1b0/0x1b0
[  341.006449]  [<ffffffff8176a5bc>] ret_from_fork+0x7c/0xb0
[  341.007124]  [<ffffffff810b8d30>] ? kthread_create_on_node+0x1b0/0x1b0
[  341.007804] Code: 08 c2 0f 85 20 03 00 00 48 8b 01 48 8b 96 a8 0f 00 00 c7 45 b0 00 00 00 00 48 89 45 b4 48 8b 41 08 48 89 45 bc 8b 41 10 89 45 c4 <83> 3a 02 49 8b 45 10 44 0f b6 b0 a0 03 00 00 0f b6 43 3a 0f 84 
[  341.009286] RIP  [<ffffffffa08ae073>] rs_fill_lq_cmd+0x83/0x3e0 [iwlmvm]
[  341.009952]  RSP <ffff8802249cf908>
[  341.010608] CR2: 0000000000000000
[  341.011214] ---[ end trace db9b5455a6d7770e ]---
[  341.011797] Kernel panic - not syncing: Fatal exception in interrupt
[  341.012399] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff)
[  341.013047] drm_kms_helper: panic occurred, switching back to text console
[  341.013706] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
[  341.014353] ------------[ cut here ]------------
[  341.014986] WARNING: CPU: 1 PID: 703 at arch/x86/kernel/smp.c:124 native_smp_send_reschedule+0x61/0x70()
[  341.015634] Modules linked in: netconsole bridge stp llc arc4 iwlmvm bnep mac80211 intel_rapl iosf_mbi x86_pkg_temp_thermal coretemp kvm_intel kvm iwlwifi snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic iTCO_wdt iTCO_vendor_support dell_laptop snd_hda_intel dcdbas crct10dif_pclmul snd_hda_controller dell_wmi crc32_pclmul snd_hda_codec sparse_keymap cfg80211 crc32c_intel i8k snd_hwdep snd_seq ghash_clmulni_intel pcspkr serio_raw snd_seq_device btusb uvcvideo snd_pcm bluetooth videobuf2_vmalloc videobuf2_core videobuf2_memops v4l2_common videodev i2c_i801 joydev lpc_ich snd_timer media mfd_core usbtouchscreen mei_me snd i2c_hid mei sdhci_pci rfkill soundcore dw_dmac shpchp i2c_designware_platform dell_smo8800 i2c_designware_core dw_dmac_core nfsd auth_rpcgss nfs_acl lockd grace sunrpc nouveau i915 ttm i2c_algo_bit drm_kms_helper drm e1000e mxm_wmi sdhci_acpi ptp sdhci mmc_core pps_core wmi video r8152 mii [last unloaded: iptable_raw]
[  341.020412] CPU: 1 PID: 703 Comm: irq/50-iwlwifi Tainted: G      D W      3.19.0 #2
[  341.021211] Hardware name: Dell Inc. Latitude 14 Rugged (5404)/07RKHG, BIOS A01 10/06/2014
[  341.022023]  0000000000000000 00000000c7a977f4 ffff88022ea83d58 ffffffff81763ee4
[  341.022854]  0000000000000000 0000000000000000 ffff88022ea83d98 ffffffff8109adfa
[  341.023675]  ffff88021c509cd0 0000000000000000 ffff88022ea14540 0000000000000001
[  341.024489] Call Trace:
[  341.025286]  <IRQ>  [<ffffffff81763ee4>] dump_stack+0x45/0x57
[  341.026167]  [<ffffffff8109adfa>] warn_slowpath_common+0x8a/0xc0
[  341.026974]  [<ffffffff8109af2a>] warn_slowpath_null+0x1a/0x20
[  341.027771]  [<ffffffff81049e71>] native_smp_send_reschedule+0x61/0x70
[  341.028571]  [<ffffffff810d648d>] trigger_load_balance+0x13d/0x1f0
[  341.029369]  [<ffffffff810c4d49>] scheduler_tick+0x99/0xe0
[  341.030159]  [<ffffffff81100a81>] update_process_times+0x51/0x60
[  341.030949]  [<ffffffff811103c5>] tick_sched_handle.isra.19+0x25/0x60
[  341.031737]  [<ffffffff81110444>] tick_sched_timer+0x44/0x80
[  341.032517]  [<ffffffff81101753>] __run_hrtimer+0x73/0x1d0
[  341.033293]  [<ffffffff81110400>] ? tick_sched_handle.isra.19+0x60/0x60
[  341.034076]  [<ffffffff81101b67>] hrtimer_interrupt+0x107/0x250
[  341.034854]  [<ffffffff8104c93c>] local_apic_timer_interrupt+0x3c/0x70
[  341.035633]  [<ffffffff8176d4e1>] smp_apic_timer_interrupt+0x41/0x60
[  341.036407]  [<ffffffff8176b57d>] apic_timer_interrupt+0x6d/0x80
[  341.037177]  <EOI>  [<ffffffff81762be0>] ? panic+0x1c3/0x204
[  341.037953]  [<ffffffff81762bd9>] ? panic+0x1bc/0x204
[  341.038730]  [<ffffffff810187d7>] oops_end+0xd7/0xe0
[  341.039504]  [<ffffffff8106326f>] no_context+0x13f/0x3a0
[  341.040278]  [<ffffffff8106354d>] __bad_area_nosemaphore+0x7d/0x210
[  341.041054]  [<ffffffff8101e7ea>] ? native_sched_clock+0x2a/0x90
[  341.041828]  [<ffffffff810636f3>] bad_area_nosemaphore+0x13/0x20
[  341.042600]  [<ffffffff810639fe>] __do_page_fault+0xce/0x5b0
[  341.043369]  [<ffffffff8139901f>] ? number.isra.2+0x33f/0x370
[  341.044114]  [<ffffffff8139992f>] ? string.isra.7+0x3f/0xf0
[  341.044831]  [<ffffffff81162d90>] ? trace_buffer_unlock_commit+0x50/0x60
[  341.045545]  [<ffffffff81063f11>] do_page_fault+0x31/0x70
[  341.046252]  [<ffffffff8176c628>] page_fault+0x28/0x30
[  341.046964]  [<ffffffffa08ae073>] ? rs_fill_lq_cmd+0x83/0x3e0 [iwlmvm]
[  341.047680]  [<ffffffffa08aec65>] iwl_mvm_rs_rate_init+0x5b5/0x7e0 [iwlmvm]
[  341.048395]  [<ffffffffa08af5b8>] iwl_mvm_rs_tx_status+0x728/0x21e0 [iwlmvm]
[  341.049098]  [<ffffffffa07ab3ec>] ? ftrace_raw_event_iwlwifi_dbg+0xec/0x1b0 [iwlwifi]
[  341.049794]  [<ffffffffa08b10c2>] rs_mac80211_tx_status+0x52/0x80 [iwlmvm]
[  341.050493]  [<ffffffffa092ca44>] ieee80211_tx_status+0x244/0xd90 [mac80211]
[  341.051183]  [<ffffffff81769f2a>] ? _raw_spin_unlock_bh+0x1a/0x20
[  341.051864]  [<ffffffffa07a23e1>] ? iwl_trans_pcie_reclaim+0x211/0x3b0 [iwlwifi]
[  341.052526]  [<ffffffffa08a3795>] iwl_mvm_rx_tx_cmd+0x405/0x7c0 [iwlmvm]
[  341.053160]  [<ffffffff81172b42>] ? ftrace_event_buffer_commit+0xa2/0x1b0
[  341.053770]  [<ffffffffa089d765>] iwl_mvm_rx_dispatch+0x165/0x1b0 [iwlmvm]
[  341.054356]  [<ffffffffa079ec38>] iwl_pcie_irq_handler+0xac8/0x1520 [iwlwifi]
[  341.054916]  [<ffffffff810d572b>] ? pick_next_task_fair+0x1bb/0x8b0
[  341.055450]  [<ffffffff810f1930>] ? irq_finalize_oneshot.part.30+0xf0/0xf0
[  341.055972]  [<ffffffff810f1950>] irq_thread_fn+0x20/0x50
[  341.056482]  [<ffffffff810f1c6f>] irq_thread+0x13f/0x170
[  341.056978]  [<ffffffff810f19f0>] ? irq_forced_thread_fn+0x70/0x70
[  341.057481]  [<ffffffff810f1b30>] ? irq_thread_check_affinity+0x90/0x90
[  341.057983]  [<ffffffff810b8e08>] kthread+0xd8/0xf0
[  341.058475]  [<ffffffff810b8d30>] ? kthread_create_on_node+0x1b0/0x1b0
[  341.058970]  [<ffffffff8176a5bc>] ret_from_fork+0x7c/0xb0
[  341.059459]  [<ffffffff810b8d30>] ? kthread_create_on_node+0x1b0/0x1b0
[  341.059956] ---[ end trace db9b5455a6d7770f ]---

Output from dmesg attached. 

Output from trace-cmd send directly to: ilw <at> linux.intel.com


Regards

Richard
Comment 1 Richard Taylor 2015-02-18 16:36:22 UTC
Created attachment 167501 [details]
iw list
Comment 2 Emmanuel Grumbach 2015-02-19 17:07:47 UTC
triaged and mirrored to our internal bug DB
Comment 3 Emmanuel Grumbach 2015-02-19 17:12:32 UTC
can we get the full log and not only snippet?

thanks
Comment 4 Richard Taylor 2015-02-19 19:12:07 UTC
Created attachment 167581 [details]
Full console log

This is the full log as captured via netconsole.
Comment 5 Johannes Berg 2015-03-02 14:27:22 UTC
The first warning is certainly an e1000 bug, not iwlwifi.

The second problem seems related to the integration of iwlwifi rs as a mac80211-based algorithm, which has always been wrong.
Comment 6 haim.dreyfuss 2015-03-09 08:13:28 UTC
Can we get trace-cmd output.
Comment 7 Richard Taylor 2015-03-10 07:37:36 UTC
(In reply to haim.dreyfuss from comment #6)
> Can we get trace-cmd output.

The trace-cmd output was sent to: ilw <at> linux.intel.com as suggested at: https://wireless.wiki.kernel.org/en/users/drivers/iwlwifi 

I have copied them directly to you as well.

Regards

Richard
Comment 8 Emmanuel Grumbach 2015-03-10 07:42:26 UTC
(In reply to Richard Taylor from comment #7)
> (In reply to haim.dreyfuss from comment #6)
> > Can we get trace-cmd output.
> 
> The trace-cmd output was sent to: ilw <at> linux.intel.com as suggested at:
> https://wireless.wiki.kernel.org/en/users/drivers/iwlwifi 
> 
> I have copied them directly to you as well.
> 

Somehow this mail never made it (I get all the mails sent to ilw@linux.intel.com).

(I got the mail form the bugzilla which is sent to ilw@linux.intel.com as well).
Comment 9 Richard Taylor 2015-03-10 08:04:17 UTC
(In reply to Emmanuel Grumbach from comment #8)
> (In reply to Richard Taylor from comment #7)
> > (In reply to haim.dreyfuss from comment #6)
> > > Can we get trace-cmd output.
> > 
> > The trace-cmd output was sent to: ilw <at> linux.intel.com as suggested at:
> > https://wireless.wiki.kernel.org/en/users/drivers/iwlwifi 
> > 
> > I have copied them directly to you as well.
> > 
> 
> Somehow this mail never made it (I get all the mails sent to
> ilw@linux.intel.com).
> 
> (I got the mail form the bugzilla which is sent to ilw@linux.intel.com as
> well).

Do you have it now? I sent it at 07:35 GMT this morning.
Comment 10 Emmanuel Grumbach 2015-03-10 08:06:56 UTC
no
Comment 11 Richard Taylor 2015-03-10 08:42:16 UTC
OK, grab it from here:

https://drive.google.com/folderview?id=0B_KhWcHoP-j5cFVUZ3dJOHZ2ZmM&usp=sharing
Comment 12 Emmanuel Grumbach 2015-03-10 17:21:45 UTC
Hi,


this tracing output is invalid. We should have one single file: trace.dat, although if the machine crashes, it may explain why we don't see the unified file...
Comment 13 Richard Taylor 2015-03-10 17:26:55 UTC
(In reply to Emmanuel Grumbach from comment #12)
> Hi,
> 
> 
> this tracing output is invalid. We should have one single file: trace.dat,
> although if the machine crashes, it may explain why we don't see the unified
> file...

I ran the following:

 trace-cmd record -e iwlwifi -e mac80211 -e cfg80211 -e iwlwifi_msg

and that was the output that I recovered after rebooting the machine.

Is there anything else I can try that would help to provide the information that you require?
Comment 14 Emmanuel Grumbach 2015-03-10 18:02:02 UTC
please compile with MAC80211_STA_DEBUG and MAC80211_MLME_DEBUG and run the scenario again.
Haim tried to reproduce but it worked for him.

Also, please start netconsole, unload iwlwifi and load iwlwifi again.
That will allow us to see the story from the beginning of the driver's life.

What we see is that the driver claims it doesn't have any room for any additional station.
Do you have 15 stations in your IBSS?
Comment 15 Johannes Berg 2015-03-10 20:30:54 UTC
Can you try this? http://p.sipsolutions.net/a501a695da371a24.txt
Comment 16 Richard Taylor 2015-03-11 11:17:08 UTC
(In reply to Emmanuel Grumbach from comment #14)
> please compile with MAC80211_STA_DEBUG and MAC80211_MLME_DEBUG and run the
> scenario again.
> Haim tried to reproduce but it worked for him.
> 
> Also, please start netconsole, unload iwlwifi and load iwlwifi again.
> That will allow us to see the story from the beginning of the driver's life.
> 
> What we see is that the driver claims it doesn't have any room for any
> additional station.
> Do you have 15 stations in your IBSS?

I have added the debug flags as requested.

Log out attached:
 netconsole: first_with_debug.out
 syslog: first_with_debug.syslog
 trace-cmd: first_with_debug.tgz

The IBSS has 25 stations.
Comment 17 Richard Taylor 2015-03-11 11:18:02 UTC
Created attachment 170411 [details]
syslog output
Comment 18 Richard Taylor 2015-03-11 11:18:58 UTC
Created attachment 170421 [details]
first_with_debug.syslog
Comment 19 Richard Taylor 2015-03-11 11:19:21 UTC
Created attachment 170431 [details]
first_with_debug.trace.tgz
Comment 20 Richard Taylor 2015-03-11 11:19:49 UTC
Created attachment 170441 [details]
first_with_debug.out
Comment 21 Richard Taylor 2015-03-11 11:23:13 UTC
(In reply to Johannes Berg from comment #15)
> Can you try this? http://p.sipsolutions.net/a501a695da371a24.txt

This did not apply cleanly on a stock 3.19 kernel. I had to hand apply the changes to "rs_add_debugfs".

Running this resulted in a kernel panic when joining the ibss net.

Logs attached:
   netconsole: patch_with_debug.out
   syslog: patch_with_debug.syslog

trace-cmd did not capture anything.

Do you want me to try with the latest kernel from https://github.com/torvalds/linux ?
Comment 22 Richard Taylor 2015-03-11 11:23:36 UTC
Created attachment 170451 [details]
patch_with_debug.syslog
Comment 23 Richard Taylor 2015-03-11 11:23:53 UTC
Created attachment 170461 [details]
patch_with_debug.out
Comment 24 Johannes Berg 2015-03-11 11:56:36 UTC
Yes, sorry about that, that was a stupid patch...

Try this one instead http://p.sipsolutions.net/a3d4ab8307aff36c.txt

Note that I suspect there are other going to be other issues in this area, like trying to start aggregation sessions with stations that don't exist as far as the driver is concerned.
Comment 25 Richard Taylor 2015-03-11 14:39:46 UTC
Created attachment 170481 [details]
new_patch.trace.dat
Comment 26 Richard Taylor 2015-03-11 14:40:14 UTC
Created attachment 170491 [details]
new_patch.syslog
Comment 27 Richard Taylor 2015-03-11 14:41:59 UTC
(In reply to Johannes Berg from comment #24)
> Yes, sorry about that, that was a stupid patch...
> 
> Try this one instead http://p.sipsolutions.net/a3d4ab8307aff36c.txt
> 
> Note that I suspect there are other going to be other issues in this area,
> like trying to start aggregation sessions with stations that don't exist as
> far as the driver is concerned.

That does appear to fix the kernel panic.

There is still a warning traceback when first joining the ibss.

Logs attached:

    syslog: new_patch.syslog
    trace-cmd: new_patch.trace.dat


Thanks
Comment 28 Emmanuel Grumbach 2015-03-11 14:43:31 UTC
tons of stations on the same IBSS here... :)
Comment 29 Richard Taylor 2015-03-11 14:45:33 UTC
(In reply to Emmanuel Grumbach from comment #28)
> tons of stations on the same IBSS here... :)

We are expecting to get up to 60+ stations!
Comment 30 Johannes Berg 2015-03-11 16:13:48 UTC
Right - the warning was there before, you can ignore it. I'll simply remove it, I think, perhaps leaving a debug statement.

Note that you can only get good performance with the first 14 (or so) stations as the iwlwifi firmware isn't able to deal with more, and thus rate scaling etc. will not work and basically all frames to the other stations have to be transmitted at very low rates.
Comment 31 Emmanuel Grumbach 2015-03-11 16:29:42 UTC
if you are using security, you won't be able to have more than 14 peers either.
Comment 32 Richard Taylor 2015-03-11 18:01:39 UTC
(In reply to Emmanuel Grumbach from comment #31)
> if you are using security, you won't be able to have more than 14 peers
> either.

Are the features of the firmware documented somewhere?
Comment 33 Johannes Berg 2015-03-11 18:10:33 UTC
You'll get software security, and since it's slow anyway that won't matter, so you will have security afaict?

Sadly, these "features" aren't documented...
Comment 34 Emmanuel Grumbach 2015-03-11 18:29:09 UTC
Yeah. Sorry, I always forget about software crypto
Comment 35 Emmanuel Grumbach 2015-03-12 13:32:53 UTC
fix will be sent upstream.