Bug 201141

Summary:	ath10k failed to delete peer
Product:	Networking	Reporter:	Matteo Croce (rootkit85)
Component:	Wireless	Assignee:	networking_wireless (networking_wireless)
Status:	RESOLVED OBSOLETE
Severity:	normal	CC:	leonardo, raghujindia
Priority:	P1
Hardware:	x86-64
OS:	Linux
Kernel Version:	4.19.1	Subsystem:
Regression:	No	Bisected commit-id:
Attachments:	syslog

Description Matteo Croce 2018-09-15 20:21:32 UTC

Created attachment 278561 [details]
syslog

ath10k prints some errors and the whole system become unusable.

software versions:

root@apu:~# uname -a
Linux apu 4.18.8-apu #54 SMP Sat Sep 15 21:51:37 CEST 2018 x86_64 GNU/Linux

root@apu:~# hostapd -v
hostapd v2.7-devel-hostap_2_6-2095-g3d019578b
User space daemon for IEEE 802.11 AP management,
IEEE 802.1X/WPA/WPA2/EAP/RADIUS Authenticator
Copyright (c) 2002-2017, Jouni Malinen <j@w1.fi> and contributors

root@apu:~# dmesg |grep 'ath10k.* firmware ver'
[   53.240509] ath10k_pci 0000:04:00.0: firmware ver 10.2.4-1.0-00041 api 5 features no-p2p,raw-mode,mfp,allows-mesh-bcast crc32 f43fa422

syslog content attached

Comment 1 Matteo Croce 2018-09-16 16:37:26 UTC

ath10k_pci 0000:04:00.0: failed to delete peer 34:80:b3:f8:ca:14 for vdev 0: -110
ath10k_pci 0000:04:00.0: found sta peer 34:80:b3:f8:ca:14 (ptr 000000001c8be7d5 id 189) entry on vdev 0 after it was supposedly removed
WARNING: CPU: 1 PID: 1256 at net/mac80211/sta_info.c:1004 __sta_info_destroy_part2+0x138/0x140 [mac80211]
Modules linked in: ecb algif_skcipher cmac ctr ccm sha256_generic sha1_ssse3 sha1_generic hmac algif_hash af_alg tun pppoe pppox ppp_generic slhc 8021q bridge stp llc arc4 usb_storage ath10k_pci ath9k ath9k_common ath9k_hw ath10k_core mac80211 ath ohci_pci k10temp hwmon nft_objref cfg80211 ohci_hcd ehci_pci ehci_hcd sp5100_tco sg r8169 i2c_piix4 mii usbcore i2c_core usb_common nft_reject_ipv4 nf_reject_ipv4 nft_reject acpi_cpufreq nft_nat nft_masq_ipv4 nft_masq nf_tables_set nft_ct nft_chain_nat_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_tables nfnetlink nf_nat_sip nf_conntrack_sip nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack ipv6 autofs4
CPU: 1 PID: 1256 Comm: hostapd Not tainted 4.18.8-apu #54
Hardware name: PC Engines APU, BIOS SageBios_PCEngines_APU-45 04/05/2014
RIP: 0010:__sta_info_destroy_part2+0x138/0x140 [mac80211]
Code: bb 04 01 00 00 00 0f 84 4d ff ff ff 45 31 c0 b9 01 00 00 00 48 89 da 4c 89 e6 48 89 ef e8 60 ae ff ff 85 c0 0f 84 2f ff ff ff <0f> 0b e9 28 ff ff ff 90 53 48 89 fb 48 83 ec 08 e8 23 d6 ff ff 85 
RSP: 0018:ffffc90000ab7a88 EFLAGS: 00010282
RAX: 00000000ffffff92 RBX: ffff880111eb2000 RCX: 0000000000000000
RDX: ffff880117a2cd00 RSI: 00000000fffffe01 RDI: 0000000000000000
RBP: ffff880115d00720 R08: ffff88010ac3c249 R09: 0000000000000040
R10: ffff88010ac3c248 R11: 0000000000000000 R12: ffff880115c7c780
R13: ffff880115d00000 R14: ffffc90000ab7ad8 R15: ffff8801159d6000
FS:  00007f87e5fa50c0(0000) GS:ffff88011ed00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fce10db1130 CR3: 00000001115e8000 CR4: 00000000000006e0
Call Trace:
 __sta_info_destroy+0x22/0x30 [mac80211]
 sta_info_destroy_addr_bss+0x33/0x60 [mac80211]
 nl80211_del_station+0xe1/0x1e0 [cfg80211]
 genl_family_rcv_msg+0x1c4/0x3c0
 ? check_preempt_curr+0x76/0x90
 genl_rcv_msg+0x50/0x9a
 ? genl_family_rcv_msg+0x3c0/0x3c0
 netlink_rcv_skb+0x44/0x110
 genl_rcv+0x1f/0x30
 netlink_unicast+0x1b0/0x250
 netlink_sendmsg+0x1d1/0x3f0
 ? netlink_unicast+0x250/0x250
 ___sys_sendmsg+0x294/0x2e0
 ? ep_item_poll.isra.22+0x46/0xc0
 ? ep_send_events_proc+0x7d/0x140
 ? ep_poll+0x1fc/0x3c0
 __sys_sendmsg+0x5d/0xb0
 do_syscall_64+0x4a/0xf0
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7f87e60c0354
Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b5 0f 1f 80 00 00 00 00 48 8d 05 91 36 0c 00 8b 00 85 c0 75 13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 f3 c3 66 90 41 54 55 41 89 d4 53 48 89 f5 
RSP: 002b:00007ffcfe92b6e8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 0000565418d04710 RCX: 00007f87e60c0354
RDX: 0000000000000000 RSI: 00007ffcfe92b720 RDI: 0000000000000005
RBP: 0000565418d0b110 R08: 0000000000000000 R09: 000000000000000d
R10: 0000565418d01010 R11: 0000000000000246 R12: 0000565418d04620
R13: 00007ffcfe92b720 R14: 0000000000000000 R15: 00000000ffffffff
---[ end trace 0cd4a1c22c80dae2 ]---
ath10k_pci 0000:04:00.0: failed to wait for created wmi peer 34:80:b3:f8:ca:14 on vdev 0: -110
ath10k_pci 0000:04:00.0: failed to add peer 34:80:b3:f8:ca:14 for vdev 0 when adding a new sta: -110
ath10k_pci 0000:04:00.0: failed to set beacon mode for vdev 0: -11
ath10k_pci 0000:04:00.0: failed to set dtim period for vdev 0: -11
ath10k_pci 0000:04:00.0: failed to recalculate rts/cts prot for vdev 0: -11
ath10k_pci 0000:04:00.0: failed to set cts protection for vdev 0: -11
ath10k_pci 0000:04:00.0: failed to set preamble for vdev 0: -11

Comment 2 Matteo Croce 2018-11-06 21:11:04 UTC

Still present in 4.19.1 and firmware 10.2.4-1.0-00042:

[108053.393017] ath10k_pci 0000:04:00.0: failed to delete peer 34:80:b3:f8:ca:14 for vdev 0: -110
[108053.393033] ath10k_pci 0000:04:00.0: found sta peer 34:80:b3:f8:ca:14 (ptr 00000000c1a79fe6 id 189) entry on vdev 0 after it was supposedly removed
[108053.393187] WARNING: CPU: 1 PID: 1234 at net/mac80211/sta_info.c:1004 __sta_info_destroy_part2+0x138/0x140 [mac80211]
[108053.393190] Modules linked in: ecb algif_skcipher tun cmac ctr ccm sha256_generic sha1_ssse3 sha1_generic hmac algif_hash af_alg pppoe pppox ppp_generic slhc 8021q bridge stp llc arc4 usb_storage ath10k_pci nft_objref ath9k ath9k_common ath10k_core ath9k_hw sp5100_tco nft_reject_ipv4 k10temp ohci_pci nf_reject_ipv4 hwmon nft_reject i2c_piix4 mac80211 ehci_pci ohci_hcd ehci_hcd ath usbcore i2c_core usb_common realtek nft_nat cfg80211 r8169 sg libphy acpi_cpufreq nft_masq_ipv4 nft_masq nf_tables_set nft_ct nft_chain_nat_ipv4 nf_nat_ipv4 nf_tables nfnetlink edac_mce_amd nf_nat_sip nf_conntrack_sip nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipv6 autofs4
[108053.393324] CPU: 1 PID: 1234 Comm: hostapd Not tainted 4.19.1-apu #57
[108053.393327] Hardware name: PC Engines APU, BIOS SageBios_PCEngines_APU-45 04/05/2014
[108053.393364] RIP: 0010:__sta_info_destroy_part2+0x138/0x140 [mac80211]
[108053.393370] Code: bb 04 01 00 00 00 0f 84 4d ff ff ff 45 31 c0 b9 01 00 00 00 48 89 da 4c 89 e6 48 89 ef e8 20 ae ff ff 85 c0 0f 84 2f ff ff ff <0f> 0b e9 28 ff ff ff 90 53 48 89 fb 48 83 ec 08 e8 03 d6 ff ff 85
[108053.393374] RSP: 0018:ffffc900007c3a88 EFLAGS: 00010282
[108053.393380] RAX: 00000000ffffff92 RBX: ffff88010d382000 RCX: 0000000000000000
[108053.393384] RDX: ffff8801149a2c00 RSI: 00000000fffffe01 RDI: 0000000000000000
[108053.393387] RBP: ffff880114a10720 R08: ffff880102ef86d9 R09: 0000000000000040
[108053.393391] R10: ffff880102ef86d8 R11: 0000000000000000 R12: ffff880117856780
[108053.393394] R13: ffff880114a10000 R14: ffffc900007c3ad8 R15: ffff88011799a000
[108053.393400] FS:  00007f0f75ace0c0(0000) GS:ffff88011ab00000(0000) knlGS:0000000000000000
[108053.393403] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[108053.393407] CR2: 00007f66ddc37080 CR3: 000000010fd18000 CR4: 00000000000006e0
[108053.393410] Call Trace:
[108053.393454]  __sta_info_destroy+0x22/0x30 [mac80211]
[108053.393491]  sta_info_destroy_addr_bss+0x33/0x60 [mac80211]
[108053.393532]  nl80211_del_station+0xe1/0x1e0 [cfg80211]
[108053.393546]  genl_family_rcv_msg+0x1c4/0x3c0
[108053.393557]  ? tracing_record_taskinfo_skip+0x45/0x50
[108053.393562]  ? tracing_record_taskinfo+0x19/0xc0
[108053.393569]  genl_rcv_msg+0x50/0x9a
[108053.393576]  ? genl_family_rcv_msg+0x3c0/0x3c0
[108053.393582]  netlink_rcv_skb+0x44/0x110
[108053.393589]  genl_rcv+0x1f/0x30
[108053.393595]  netlink_unicast+0x1b0/0x250
[108053.393602]  netlink_sendmsg+0x1d1/0x3f0
[108053.393608]  ? netlink_unicast+0x250/0x250
[108053.393614]  ___sys_sendmsg+0x294/0x2e0
[108053.393622]  ? ep_item_poll.isra.22+0x46/0xc0
[108053.393627]  ? ep_send_events_proc+0x7d/0x140
[108053.393632]  ? ep_modify+0x140/0x140
[108053.393637]  ? ep_poll+0x1e4/0x390
[108053.393643]  __sys_sendmsg+0x5d/0xb0
[108053.393652]  do_syscall_64+0x4a/0xf0
[108053.393660]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[108053.393666] RIP: 0033:0x7f0f75be9354
[108053.393672] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b5 0f 1f 80 00 00 00 00 48 8d 05 91 36 0c 00 8b 00 85 c0 75 13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 f3 c3 66 90 41 54 55 41 89 d4 53 48 89 f5
[108053.393675] RSP: 002b:00007ffe8ba11458 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
[108053.393681] RAX: ffffffffffffffda RBX: 000055d34e4cc710 RCX: 00007f0f75be9354
[108053.393684] RDX: 0000000000000000 RSI: 00007ffe8ba11490 RDI: 0000000000000005
[108053.393687] RBP: 000055d34e4d3110 R08: 0000000000000004 R09: 00007f0f75c2b080
[108053.393691] R10: 00007ffe8ba1156c R11: 0000000000000246 R12: 000055d34e4cc620
[108053.393694] R13: 00007ffe8ba11490 R14: 0000000000000000 R15: 00000000ffffffff
[108053.393699] ---[ end trace db77cb3f6f00f9f3 ]---
[108056.432467] ath10k_pci 0000:04:00.0: failed to wait for created wmi peer 34:80:b3:f8:ca:14 on vdev 0: -110
[108056.432480] ath10k_pci 0000:04:00.0: failed to add peer 34:80:b3:f8:ca:14 for vdev 0 when adding a new sta: -110
[108198.734201] ath10k_pci 0000:04:00.0: failed to install key for vdev 0 peer 18:01:f1:1e:ae:a0: -11
[108198.734212] wlp4s0: failed to remove key (0, 18:01:f1:1e:ae:a0) from hardware (-11)
[108198.742815] ath10k_pci 0000:04:00.0: cipher 0 is not supported
[108198.742823] ath10k_pci 0000:04:00.0: failed to remove peer wep key 0: -95
[108198.742830] ath10k_pci 0000:04:00.0: failed to clear all peer wep keys for vdev 0: -95
[108198.742837] ath10k_pci 0000:04:00.0: failed to disassociate station: 18:01:f1:1e:ae:a0 vdev 0: -95
[108198.742962] WARNING: CPU: 1 PID: 1234 at net/mac80211/sta_info.c:996 __sta_info_destroy_part2+0x10b/0x140 [mac80211]
[108198.742964] Modules linked in: ecb algif_skcipher tun cmac ctr ccm sha256_generic sha1_ssse3 sha1_generic hmac algif_hash af_alg pppoe pppox ppp_generic slhc 8021q bridge stp llc arc4 usb_storage ath10k_pci nft_objref ath9k ath9k_common ath10k_core ath9k_hw sp5100_tco nft_reject_ipv4 k10temp ohci_pci nf_reject_ipv4 hwmon nft_reject i2c_piix4 mac80211 ehci_pci ohci_hcd ehci_hcd ath usbcore i2c_core usb_common realtek nft_nat cfg80211 r8169 sg libphy acpi_cpufreq nft_masq_ipv4 nft_masq nf_tables_set nft_ct nft_chain_nat_ipv4 nf_nat_ipv4 nf_tables nfnetlink edac_mce_amd nf_nat_sip nf_conntrack_sip nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipv6 autofs4
[108198.743099] CPU: 1 PID: 1234 Comm: hostapd Tainted: G        W         4.19.1-apu #57
[108198.743102] Hardware name: PC Engines APU, BIOS SageBios_PCEngines_APU-45 04/05/2014
[108198.743140] RIP: 0010:__sta_info_destroy_part2+0x10b/0x140 [mac80211]
[108198.743146] Code: 48 89 df e8 07 42 05 00 48 8b 43 50 48 89 df 48 8b a8 10 04 00 00 e8 64 d1 ff ff 48 89 de 5b 48 89 ef 5d 41 5c e9 75 e0 ff ff <0f> 0b 80 bb 04 01 00 00 00 0f 84 4d ff ff ff 45 31 c0 b9 01 00 00
[108198.743150] RSP: 0018:ffffc900007c3a88 EFLAGS: 00010282
[108198.743156] RAX: 00000000ffffffa1 RBX: ffff88010d384000 RCX: 0000000000000000
[108198.743159] RDX: ffff8801149a2c00 RSI: 0000000000000092 RDI: 0000000000000000
[108198.743163] RBP: ffff880114a10720 R08: 000000000000008c R09: 0000000000000004
[108198.743166] R10: 0000000000000000 R11: 0000000000000001 R12: ffff880117856780
[108198.743170] R13: ffff880114a10000 R14: ffffc900007c3ad8 R15: ffff88011799a000
[108198.743175] FS:  00007f0f75ace0c0(0000) GS:ffff88011ab00000(0000) knlGS:0000000000000000
[108198.743179] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[108198.743183] CR2: 00007ffe95002000 CR3: 000000010fd18000 CR4: 00000000000006e0
[108198.743186] Call Trace:
[108198.743231]  __sta_info_destroy+0x22/0x30 [mac80211]
[108198.743267]  sta_info_destroy_addr_bss+0x33/0x60 [mac80211]
[108198.743312]  nl80211_del_station+0xe1/0x1e0 [cfg80211]
[108198.743326]  genl_family_rcv_msg+0x1c4/0x3c0
[108198.743334]  ? skb_queue_tail+0x24/0x60
[108198.743340]  ? __netlink_sendskb+0x57/0x80
[108198.743347]  genl_rcv_msg+0x50/0x9a
[108198.743354]  ? genl_family_rcv_msg+0x3c0/0x3c0
[108198.743360]  netlink_rcv_skb+0x44/0x110
[108198.743367]  genl_rcv+0x1f/0x30
[108198.743374]  netlink_unicast+0x1b0/0x250
[108198.743380]  netlink_sendmsg+0x1d1/0x3f0
[108198.743387]  ? netlink_unicast+0x250/0x250
[108198.743393]  ___sys_sendmsg+0x294/0x2e0
[108198.743404]  ? __inode_wait_for_writeback+0x7a/0xe0
[108198.743413]  ? init_wait_var_entry+0x40/0x40
[108198.743422]  ? __dentry_kill+0xfd/0x150
[108198.743430]  ? _cond_resched+0x10/0x20
[108198.743436]  __sys_sendmsg+0x5d/0xb0
[108198.743445]  do_syscall_64+0x4a/0xf0
[108198.743453]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[108198.743459] RIP: 0033:0x7f0f75be9354
[108198.743464] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b5 0f 1f 80 00 00 00 00 48 8d 05 91 36 0c 00 8b 00 85 c0 75 13 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 f3 c3 66 90 41 54 55 41 89 d4 53 48 89 f5
[108198.743467] RSP: 002b:00007ffe8ba124f8 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
[108198.743474] RAX: ffffffffffffffda RBX: 000055d34e4cc710 RCX: 00007f0f75be9354
[108198.743477] RDX: 0000000000000000 RSI: 00007ffe8ba12530 RDI: 0000000000000005
[108198.743480] RBP: 000055d34e4dccb0 R08: 0000000000000004 R09: 000000000000000d
[108198.743484] R10: 00007ffe8ba1260c R11: 0000000000000246 R12: 000055d34e4cc620
[108198.743487] R13: 00007ffe8ba12530 R14: 0000000000000000 R15: 00000000ffffffff
[108198.743493] ---[ end trace db77cb3f6f00f9f4 ]---
[108201.773831] ath10k_pci 0000:04:00.0: failed to delete peer 18:01:f1:1e:ae:a0 for vdev 0: -11
[108201.773847] ath10k_pci 0000:04:00.0: found sta peer 18:01:f1:1e:ae:a0 (ptr 000000006b1fb63c id 28) entry on vdev 0 after it was supposedly removed
[108204.813559] ath10k_pci 0000:04:00.0: failed to set beacon mode for vdev 0: -11
[108207.853399] ath10k_pci 0000:04:00.0: failed to set dtim period for vdev 0: -11
[108210.892766] ath10k_pci 0000:04:00.0: failed to recalculate rts/cts prot for vdev 0: -11
[108213.932236] ath10k_pci 0000:04:00.0: failed to set cts protection for vdev 0: -11
[108216.971985] ath10k_pci 0000:04:00.0: failed to set preamble for vdev 0: -11
[108360.073601] ath10k_pci 0000:04:00.0: failed to install key for vdev 0 peer 00:21:6a:9d:7c:68: -11
[108360.073612] wlp4s0: failed to remove key (0, 00:21:6a:9d:7c:68) from hardware (-11)
[108360.082174] ath10k_pci 0000:04:00.0: cipher 0 is not supported
[108360.082182] ath10k_pci 0000:04:00.0: failed to remove peer wep key 0: -95
[108360.082189] ath10k_pci 0000:04:00.0: failed to clear all peer wep keys for vdev 0: -95
[108360.082196] ath10k_pci 0000:04:00.0: failed to disassociate station: 00:21:6a:9d:7c:68 vdev 0: -95
[108363.113448] ath10k_pci 0000:04:00.0: failed to delete peer 00:21:6a:9d:7c:68 for vdev 0: -11
[108363.113465] ath10k_pci 0000:04:00.0: found sta peer 00:21:6a:9d:7c:68 (ptr 00000000633fd492 id 40) entry on vdev 0 after it was supposedly removed

Comment 3 WhoAmEye 2019-04-22 18:28:33 UTC

Do you have reproduction steps for this issue?
Thanks.

Comment 4 WhoAmEye 2019-04-23 18:34:48 UTC

From the log I see "ath10k_pci 0000:04:00.0: failed to add peer 34:80:b3:f8:ca:14 for vdev 0 when adding a new sta: -110" which is a known issue I believe as per 

https://wireless.wiki.kernel.org/en/users/drivers/ath10k#known_bugslimitations

Comment 5 Matteo Croce 2019-04-28 10:51:29 UTC

(In reply to WhoAmEye from comment #3)
> Do you have reproduction steps for this issue?
> Thanks.

Hi,

not really, the issue happens randomly after a few days of usage.
I have some smartphones and notebook connected.

Comment 6 leonardo 2019-12-28 22:02:10 UTC

I am experiencing the exact same issue on a Turris Omnia router (running a fork of OpenWrt).

root@turris:~# uname -a
Linux turris 4.14.158 #0 SMP Fri Dec 20 00:24:37 2019 armv7l GNU/Linux

root@turris:~# hostapd -v
hostapd v2.7-devel
User space daemon for IEEE 802.11 AP management,
IEEE 802.1X/WPA/WPA2/EAP/RADIUS Authenticator
Copyright (c) 2002-2017, Jouni Malinen <j@w1.fi> and contributors

root@turris:~# dmesg | grep ath10k | grep 'firmware ver'
[   13.991887] ath10k_pci 0000:02:00.0: firmware ver 10.2.4-1.0-00037 api 5 features no-p2p,raw-mode,mfp,allows-mesh-bcast crc32 a4a52adb

Unfortunately I lost the kernel log (it's not persisted across reboots), but I can attach it here when it happens again. As it is the case with original submitter, it happens randomly -- it may take a few hours or a few days.