In the mean time I forgot to add that the last version of kernel which didn't have this problem was the one in Ubuntu 11.04, which is 2.6.38
In the following ubuntu releases I've always had this problem

Created attachment 72698 [details]
dmesg

this is a dmesg of an unlucky boot which wifi not working.

0c:00.0 Network controller [0280]: Intel Corporation WiFi Link 5100 [8086:4232]

Same problem with 3.13.0-24-generic (Ubuntu 14.04) and Intel Wireless 7260 (rev 6b). Some problem with Linux 3.11 (Mint) and Intel WiFi Link 5100.

Affects only my home AP. No problems with Windows; Other clients connect and transmit without problems. In fact both my notebooks have this problem when running Linux.

Problem disappears for various amount of time (minutes–hours–days) when changing the APs frequency or WPA2 passphrase. Reconnecting to it without changing its setting does not solve the problem. Restarting the router does.

I already tried to purge resolveconf, tried to re-install it, tried to dpkg-reconfigure it. No change.

Created attachment 133831 [details]
syslog with various successful and unsuccessful connections

Looking at Wireshark I see NBNS, IGMP and ARP messages go through.

I've just seen what looks like the same issue with 3.14.2 and intel 7260.

(IP connectivity dead, I can see ARP packets going through with tcpdump, but no other traffic)

I'm pretty sure this is not the first time I run into it, but I think it was largely shadowed with occurences of the more frequent bug #72601.

I have a trace with iwlwifi debug from *after* the link went dead, I can post it if it helps.

Setting my router to b/g mode only fixed it. At the same time I set my channel to <10 (which I read somewhere, couldn't find it anymore).
I cannot reproduce which change solved the problem.

Created attachment 136791 [details]
tpcdump log

The issue popped up again tonight, here is a tcpdump log.

As you can such a few things such as ARP packets are going through.

Created attachment 136801 [details]
trace file

Trace file captured after the 'blackout' and during the tcpdump session above

I ran it with :
trace-cmd record -e iwlwifi -e iwlwifi_msg -e mac80211 -e cfg80211

@Florian

From your trace I can see that we (driver) don't get anything from the firmware during that period of time. This can mean 2 things:
1) there were no packets in the air - in case it is not our fault
2) there were packets in the air and they have been discarded by the Intel Wifi firmware for some reason.

They best way to know is to have another device set as a sniffer that can hear *everything*. If you have another machine that can do that, this can be nice.

I'd first disable power save (iw wlan0 set power_save off)
Another thing you can is to try several versions of the firmware.
Yet another thing you can is to try disable 11n: 11n_disable=1 just to see what happens.

Hi Emmanuel, I can sniff on the router directly, would that be interesting ? Are tcpdumps enough in that case ?

Otherwise, I think I have a spare WiFi USB Key lying somewhere, I can probably set up a sniffer with dev boards I have lying around.

I'll make more tests tonight, starting with going back to fw .8

Florian

(In reply to Florian Vallee from comment #12)
> Hi Emmanuel, I can sniff on the router directly, would that be interesting ?
> Are tcpdumps enough in that case ?

Not really, I'd need wifi sniffer.

> 
> Otherwise, I think I have a spare WiFi USB Key lying somewhere, I can
> probably set up a sniffer with dev boards I have lying around.
> 

That can work

> I'll make more tests tonight, starting with going back to fw .8
> 
Consider even fw.7

Created attachment 137051 [details]
monitor-mode capture on AP

Hi there,

the blackout issue paid me a visit again tonight ;)

Sadly I don't have much time to look into it today, nor have I worked on the sniffing setup yet.

What I did however, is a capture in monitor mode on the AP, after shutting down most of the wifi-enabled device in the home (during a "blackout")

My intel 7260 is 5c:51:4f:72:f2:81, my AP is d4:ca:6d:25:f3:fc

I'm currently having a quick look at it feel to check it out too ;)

Looking at it - I'll reply privately.

BTW - if you care about privacy, feel free to send this kind of things to me only, and / or to encrypt them with my PGP key.

Unfortunately, I couldn't find anything useful in this sniffer capture.
Lots retransmissions of CTS, but that's not something I can debug through tracing...

Hi Emmanuel,

Regarding privacy, I'll keep that in mind. I'm curious is there anything sensitive in the capture ? I figured there was pretty much nothing in it someone passing by couldn't have obtained by passively listening to radio activity.

Regarding your latest comment is there anything more I can do to help ? Would a capture from a third, sniffer station, still be useful ?

Can you please try this?

diff --git a/drivers/net/wireless/iwlwifi/mvm/mac-ctxt.c b/drivers/net/wireless/iwlwifi/mvm/mac-ctxt.c
index bc57c27..a32dce9 100644
--- a/drivers/net/wireless/iwlwifi/mvm/mac-ctxt.c
+++ b/drivers/net/wireless/iwlwifi/mvm/mac-ctxt.c
@@ -669,7 +669,7 @@ static void iwl_mvm_mac_ctxt_cmd_common(struct iwl_mvm *mvm,
 
        if (vif->bss_conf.use_cts_prot) {
                cmd->protection_flags |= cpu_to_le32(MAC_PROT_FLG_TGG_PROTECT);
-               cmd->protection_flags |= cpu_to_le32(MAC_PROT_FLG_SELF_CTS_EN);
+//             cmd->protection_flags |= cpu_to_le32(MAC_PROT_FLG_SELF_CTS_EN);
        }
        IWL_DEBUG_RATE(mvm, "use_cts_prot %d, ht_operation_mode %d\n",
                       vif->bss_conf.use_cts_prot,

Ok, I'll build this tonight against 3.15

Done, I'll start running this tomorrow.

I've put the package online in case anyone else wants to give this a try (archlinux binaries), it's here :

http://ey3ball.net/arch/linux-3.15-1-b42978-1/

I had to slighthly modify emmanuel's patch as it wouldn't apply directly on top of the 3.15 release, see http://ey3ball.net/arch/linux-3.15-1-b42978-1/0099-dead-wifi.patch

any news?

Sadly not much, I've been running this, but not in my usual radio environment (I was away from home lately), so this is not really helpful.

I'm now back to my usual setup so I'll be able to give you an update in a few days.

Going back to my comment #17 and what you said in comment #16 in case I see anything happening which data would be useful for you to debug ? Do we stick with traces and sniffing on the router ?

Ok.

Regarding other debug options, the only other input I can think about is a sniffer capture on another device.
The problem is that typically, the air is very polluted which make it hard to catch the problem.

Hi there, a quick update :

No sign of the issue so far, I'm sticking to the same kernel for the upcoming week and will report if anything new happens.

Florian

Saw the issue twice tonight.

I was in a bit of a rush so I did not capture logs. I'll work on a "continuous sniffing" setup from a 3rd station tomorrow.

As you wish - but I doubt I'll have time to look at the captures.

I am working at enabling better debug for the firmware which might shed more light here.

Are you in 20Mhz or 40Mhz?

It seems I'm connected in 20Mhz mode

$ iw  wlp3s0 info
Interface wlp3s0
        ifindex 3
        wdev 0x1
        addr -
        type managed
        wiphy 0
        channel 1 (2412 MHz), width: 20 MHz, center1: 2412 MHz

Ok, I'll wait for a new firmware / debug patch, in the meantime, I'm reverting to a vanilla mainline kernel.

Florian

the patch from Comment 18 is on its way upstream.

Can you please try with Core6 FW. It is available here:
https://git.kernel.org/cgit/linux/kernel/git/egrumbach/linux-firmware.git/diff/iwlwifi-7260-9.ucode?h=Core6

Thanks

[   22.076614] iwlwifi 0000:03:00.0: loaded firmware version 25.223.9.0 op_mode iwlmvm

I'll let you know if anything interesting happens

Hi Emmanuel,

I've been running this fw for a week and the issue occured a dozen of times since.

You mentionned the possibility of gathering more debug logs with this firmware, how do I enable this ?

Thank you,

Regards,

Florian

Created attachment 146951 [details]
attachment-3032-0.html

Try adding pcie_aspm=off to your kernel parameters. It made mine much more
stable.
On Aug 17, 2014 3:14 PM, <bugzilla-daemon@bugzilla.kernel.org> wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=42978
>
> --- Comment #32 from Florian Vallee <florian@ey3ball.net> ---
> Hi Emmanuel,
>
> I've been running this fw for a week and the issue occured a dozen of times
> since.
>
> You mentionned the possibility of gathering more debug logs with this
> firmware,
> how do I enable this ?
>
> Thank you,
>
> Regards,
>
> Florian
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.
>

@Florian, dmesg would be a good start... :)

Created attachment 148911 [details]
Script to ping gateway and bounce it on freeze

I've been having this problem too since I got my laptop.

In case this is helpful to someone, I've attached a hacky script which pings your gateway and bounces the wlan link if it doesn't get a response for a few seconds.  On my laptop at least, the network works again after an rfkill cycle.

You'll see this as a disconnect/reconnect so it's hardly seamless, but it's somewhat better than having to do it by hand when your network freezes.

If you also have other network issues (router problems, etc.) this can sometimes go crazy.  I wouldn't leave it running all the time.

Confirmed this still appears to happen on kernel 3.16.1 using the core6 firmware.

Created attachment 149091 [details]
dmesg with core6 fw

Hi there,

Here is a dmesg with the core6 fw once the issue has occured.

Apologies for the delay in getting back to you, I've been away for a while.

Hope it helps,

Florian

hmm... nothing there....


For more debug - you'd need a special firmware that I can prepare for you, but you need 3.17...

Ok, I'll get started on a 3.17 kernel build later today.

I haven't been able to repro the problem with pcie_aspm=off in several days (but of course, it kills my battery life).

Thanks Elladan, this is extremely useful. I'll ask the firmware guys since they have some impact on the way PCI goes to power states.

Hi,

I'm sorry to say I've seen the issue with pcie_aspm=off as well, I've been running this option since Stuart's message (but reverted before I sent my last dmesg because of battery life implications)

Since this seems to contradict Elladan's report, I'll double check this week and will provide the usual debug info if I can reproduce again.

Also, I have a 3.17-rc3 kernel ready to run.

Created attachment 149381 [details]
Core7 FW with DBGM

Ok - thanks Florian.

Can you please run with the FW attached and run the FW monitor?
You can find the instructions here: http://wireless.kernel.org/en/users/Drivers/iwlwifi#Debugging

Note the user privacy section.

When you see bad latency, do *quickly* the following:

echo 1 > /sys/kernel/debug/iwlwifi/<whatever>/iwlmvm/fw_restart
Then,

cat /sys/kernel/debug/iwlwifi/*/iwlmvm/fw_error_dump > iwl.bin

and send me the iwl.bin

Don't forget to run with fw_monitor=1.

Thanks!

One more thing - you'll need https://git.kernel.org/cgit/linux/kernel/git/iwlwifi/iwlwifi-fixes.git/commit/?id=d88c8958dc13b4e4eb7fc57e3f06dc1c4abc7b1f to be able to load the -10 firmware.
Thanks!

Sorry, been a bit busy lately

So, I've just started running the kernel with your patch + debug fw.

[   22.080027] iwlwifi 0000:03:00.0: loaded firmware version 23.10.10.0 op_mode iwlmvm

I have added fw_monitor=1 and pcie_aspm=off to my kernel command line

My archlinux kernel build is available here : http://ey3ball.net/arch/linux-3.17-rc3/

Will report on what happened (or didn't happen) by the end of the week

Florian

I guess you can wait with that...
The firmware folks just told me that this firmware doesn't have all the debug thing they need...
I am working on enabling more and more debug infrastructure for the firmware to be able to understand what happens in cases like yours...

Ok, I'll wait for a new FW.

FYI regarding my comment #42, I've just been able to confirm the issue does occur with pcie_aspm=off, I've seen it happen just now.

$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/omnicron-arch--root rw cryptdevice=/dev/sda3:omnicron "acpi_osi=!Windows 2012" fw_monitor=1 pcie_aspm=off

I'm having issues with 7260 on Lenovo Yoga 2 13'' . Wireless works fine only on close distance (5-10 m) from the router (checked in different places with different router types) otherwise connection drops. On the boundary of this range speed drops to extremely low.
Tried on 3.13.0-24, 3.13.0-35, 3.17.0.rc4
Originally filed a bug here (with all appropriate details):
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1369101?comments=all

please disable ASPM to see what happens.

Created attachment 151611 [details]
Configure the LTR

Can you please test this?

I am not sure it will work on old firmware - I have tested yet.
This is meant to fix the issues caused by ASPM.

Thanks

I forgot to say - not tested at all :)

(In reply to Emmanuel Grumbach from comment #50)
> please disable ASPM to see what happens.

Disabled on 3.17.0.rc4 and it improved a situation. It the spot where I didn't have any connectivity I have it now but speed is extremely low.

Created attachment 152001 [details]
Configure the LTR

This is a good candidate for a fix.
I'd like to have feedback to be able to send it upstream ASAP.
Thanks.

Hi Emmanuel,

I've been testing a 3.17 rc6 kernel with this patch since you posted it last Sunday and have not seen the issue since.

I think we need some more testing to be sure it's gone, but it's probably a good sign, it usually occurs faster ;)

for reference:
[  192.590629] iwlwifi 0000:03:00.0: loaded firmware version 23.10.10.0 op_mode iwlmvm

Linux omnicron 3.17.0-1-mainline #1 SMP PREEMPT Sun Sep 28 09:41:03 UTC 2014 x86_64 GNU/Linux

Once again I've spoken too fast, just saw the issue happening, same symptoms as usual.

(In reply to Florian Vallee from comment #56)
> Once again I've spoken too fast, just saw the issue happening, same symptoms
> as usual.


Thanks for testing.
As I said above, this patch (comment 54) is supposed to help people that had an improvement when disabling ASPM. This was not you case, so I am not very surprised...

Yes I get that, just though I'd give this a try just in case, I had a few minutes to spare ;).

Can you please disable power save?

sudo iw wlan0 set power_save off

I am also suspecting U-APSD...

Ok, added this to my rc.local

saw the issue with power_save off a few minutes ago

Ok - yet another FW released.
In this last release, the FW folks fixed a big problem in scan that can cause much latency.

Please test:

https://git.kernel.org/cgit/linux/kernel/git/egrumbach/linux-firmware.git/plain/iwlwifi-7260-9.ucode

Thanks

ok, grabbed it

[   11.251271] iwlwifi 0000:03:00.0: loaded firmware version 25.228.9.0 op_mode iwlmvm

Tried 25.228.9.0, network seemed unstable to the point of being unusable, reverted.

@elladan

Do you have aspm enable?
Do you have my ltr patch?

@emmanual

ASPM is enabled. I'm using a 3.16.0-18 kernel which should have your LTR patch.

Can you please let me know what you mean by "unusable"?
Latency, throughput?

In 3.17 I have the ability to debug the firmware with better tools. Would you consider moving to 3.17 and apply the LTR patch?

Hi Emmanuel,

I've been in out and of home lately, but I haven't seen the issue in a while. I'm using the firmware from comment #62.

I'll keep my current setup it and report if anything happens by next week.

Ok

FWIW - another user provide FW logs and we saw an issue there that can explain that we don't Tx in his environment.

Hi,

I've got the exact same issue with intel 7260 and 3.16.6.
I have to disconnect/reconnect several time to have an active connection back.

I can help, but I'm not really familiar in wifi problem diagnostics.

Please tell me if I can help by running some diagnostics tool.

Regards

(In reply to Sticky Dev from comment #71)
> Hi,
> 
> I've got the exact same issue with intel 7260 and 3.16.6.
> I have to disconnect/reconnect several time to have an active connection
> back.
> 
> I can help, but I'm not really familiar in wifi problem diagnostics.
> 
> Please tell me if I can help by running some diagnostics tool.
> 
> Regards

Please record tracing while the traffic is stuck:

sudo trace-cmd record -e iwlwifi -e iwlwifi_msg -e mac80211 -e cfg80211

and send trace.dat.

thanks

@Olivier - any news?

Also - what version of the firmware do you use?
dmesg | grep iwlwifi | grep -i version

@Emmanuel

I tried to use trace-cmd but I notice that my opensuse 13.2 kernel is not compiled with IWLWIFI_DEVICE_TRACING. So I will compile the 3.17.4 vanilla kernel this weekend to enable tracing.  

I noticed also that download rate was very slow between the computer and my router, decreasing rapidely to arround 5 ko/s.

my firmware is from memory (I'm not currently at home) : iwlwifi-7260-9

regards

iwlwifi-7290-9 doesn't tell me exactly what version you have.
Please paste your dmesg output when you'll be able to.

thanks.

@Emmanuel 
[    3.308537] iwlwifi 0000:02:00.0: loaded firmware version 23.214.9.0 op_mode iwlmvm

@Olivier


Please take a newer FW from here:
https://git.kernel.org/cgit/linux/kernel/git/egrumbach/linux-firmware.git/plain/iwlwifi-7260-9.ucode?id=871ab78c67c78eefb0b8bc22eac967a3872526a6

copy that file to /lib/firmware and reboot

@Emmanuel
A small update : With vanila kernel 3.17.4 and the firmware you linked, I had no pb during 2 days yet, and the transfer rate are normal.

I will continue to test during this week.

Thanks for your help

based on comment 69 and comment 78, I am now closing this bug.

Note that the new firmware has just been merged to linux-firmware.git.

If you still experience issues with the new firmware (25.228.9.0), please open a new bug.

IMPORTANT NOTE TO ANYBODY TRACKING THIS BUG:
********************************************

The firmware 25.228.9.0 had several re-spin. We updated the firmware without changing the version. Since the firmware hadn't been officially released / merged linux-firmware.git, this is not an issue.
BUT if you took a version from this bug / from my git repository before it was merged to linux-firmware.git, I highly recommend you take it again from here: https://git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git/plain/iwlwifi-7260-9.ucode

$ md5sum iwlwifi-7260-9.ucode
e44b5de2a16cd0009ac464f3cf80c347  iwlwifi-7260-9.ucode