(Shameless repost from my mail to LKML. Seems to have gone quiet so I thought I wouldn't let this bugger get lost.) I have recently assembled a Core 2 Duo system with 4GB RAM and I believe there might be a bug in the r8169 driver in >4GB RAM configurations. Initially I can use one of two active r8169 NICs on the motherboard with this quantity of RAM with other devices, without issue. But after some amount of data (generally about 50MB), no more network packets are sent/received. The "choke" affects other devices on the system too, notably libata, which does not recover gracefully. In my logs, I see a stream of: DMA: Out of SW-IOMMU space for 7222 bytes at device 0000:04:00.0 DMA: Out of SW-IOMMU space for 7222 bytes at device 0000:04:00.0 The device 0000:04:00.0 corresponds to one of the r8169s. The reason I believe r8169 is at fault is that I was doing a rebuild of my RAID5 across 3 SATA drives via libata's ahci driver, and transferring over the network. When the "choke" occurred the RAID sync stopped, libata errors were seen, and I simply did a "ifconfig br0 down" (which contained the r8169) and the messages went away. Bringing the NIC up again would see some initial functionality then very rapidly it would go back to the same error messages. The Intel chipset I am using does not support any kind of hardware IOMMU, so I am forced to use swiotlb in a 4GB RAM configuration. In an attempt to delay the failures, I used the swiotlb option to increase the swiotlb's page allocation with "swiotlb=65536" (which seems to correspond to a 256MB bounce buffer). Assuming both libata and r8169 use the swiotlb, and both systems are impaired when these messages appear, removing r8169 would appear to be key. Indeed, if there is no significant libata activity, the problem still occurs on the NIC within approximately the same amount of transfer. This option delays the failure for some time but it will happen eventually, which makes me suspicious that maybe the driver is somehow pinning an area of the buffer and not releasing it. (I hunted bugzilla for reports similar to this one, but couldn't find anything.) Having tested the r8169 driver on an AMD system I did not experience the same problems with 4GB RAM, so this could be a bug specific to swiotlb. I would have added more people to CC but I have no idea who might be responsible. mem=2000M on the cmdline works around this issue.
No change with 2.6.25-rc5. Disabling the IOMMU still works around the problem (albeit at the loss of half my RAM).
Still a problem with 2.6.26-rc2.
Same here with 2.6.25-r4 ( gentoo-sources) but on the different system -AMD Phenom with 4GB RAM. Interesting things are: If I take out phenom 9850 and plug in X2 6000+, IOMMU error vanishes and system works perfectly normally - on Phenom workaround is to use smaller MTU. I had mtu 7200 which was pro9blematic, so I have lowered it. It seems that anything up to 3600 works fine for me, but I use 2048 just to be sure....
As I tried to explain in my original report, my problem is with the swiotlb support, NOT the AMD hardware IOMMU. My Intel chipset does not support any hardware IOMMU. Unless you're explicitly disabling GART IOMMU on AMD, I can't imagine your problem is the same as mine. Of course, it would be really interesting if it were. Do you see exactly the same messages? Could I see your dmesg? Other posters have mentioned the lower MTU, but I still have problems (packet loss, unreliability) even at 1500, it just won't crash the machine (but who knows, maybe it just takes 5-6 times longer to do so).
Created attachment 17344 [details] pci_unmap unbalance Alistair, can you give the attached patch a try agaisnt 2.6.27-rc ? -- Ueimor
Thanks a LOT. This patch seems to fix the issue completely and I'm running with my swiotlb once again in 2.6.27-rc3. Transferred >4GB data over the link with 7.2k frames, which is the most I've ever been able to do with the r8169 driver + Intel + 4GB RAM, no warnings to dmesg. I think you've plugged this particular leak. Well done! Tested-by: Alistair John Strachan <alistair@devzero.co.uk>
fixed by commit a866bbf6aacf95f849810079442a20be118ce905
how would I use this patch for linux mint 11 x64?
(In reply to comment #8) > how would I use this patch for linux mint 11 x64 ? Your distribution claims to be 2.6.38 based. If so the patch is already included in it. Please contact your distribution vendor. -- Ueimor