Yesterday I have configured bridge(br0) between my wifi interface (wlp3s0) and ethernet interface(enp4s0) using hostapd, and today in dmesg I found 2 call traces, after starting rtorrent on bridge: [175827.356666] WARNING: CPU: 0 PID: 0 at net/core/stream.c:201 sk_stream_kill_queues+0xfe/0x126() [175827.356670] Modules linked in: bridge joydev nvidia(PO) x86_pkg_temp_thermal coretemp pcspkr nouveau i2c_i801 snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic mxm_wmi r8169 mii ttm snd_hda_intel snd_hda_codec snd_hwdep rtc_cmos wmi efivarfs xts gf128mul aes_x86_64 sha512_generic libiscsi scsi_transport_iscsi tg3 fuse nfs lockd sunrpc jfs hid_sunplus hid_sony hid_samsung hid_pl hid_petalynx hid_monterey hid_microsoft hid_logitech hid_gyration hid_ezkey hid_cypress hid_chicony hid_cherry hid_belkin hid_apple hid_a4tech sl811_hcd aic94xx libsas scsi_transport_sas lpfc qla2xxx scsi_transport_fc megaraid_sas megaraid_mbox megaraid_mm megaraid aacraid sx8 DAC960 cciss 3w_9xxx 3w_xxxx atp870u dc395x qla1280 dmx3191d sym53c8xx gdth advansys initio BusLogic arcmsr aic7xxx aic79xx scsi_transport_spi [175827.356772] pdc_adma sata_inic162x sata_qstor sata_vsc sata_uli sata_sis sata_sx4 sata_via sata_svw sata_sil24 sata_sil sata_promise pata_sl82c105 pata_cs5530 pata_cs5520 pata_via pata_jmicron pata_marvell pata_sis pata_netcell pata_sc1200 pata_pdc202xx_old pata_triflex pata_atiixp pata_opti pata_amd pata_ali pata_it8213 pata_pcmcia pcmcia pcmcia_core pata_ns87415 pata_ns87410 pata_serverworks pata_platform pata_artop pata_it821x pata_optidma pata_hpt3x2n pata_hpt3x3 pata_hpt37x pata_hpt366 pata_cmd64x pata_efar pata_rz1000 pata_sil680 pata_radisys pata_pdc2027x pata_mpiix usbhid [175827.356836] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P O 3.14.3-gentoo #1 [175827.356841] Hardware name: ASUSTeK COMPUTER INC. G56JR/G56JR, BIOS G56JR.200 01/13/2014 [175827.356845] 0000000000000009 ffff88042ee036f8 ffffffff818532e4 0000000000000103 [175827.356853] 0000000000000000 ffff88042ee03738 ffffffff8106c79c ffff8803a25607e0 [175827.356859] ffffffff816e0bff ffff8803a25606c0 ffff8803a2560838 ffff880088bdd8e2 [175827.356867] Call Trace: [175827.356871] <IRQ> [<ffffffff818532e4>] dump_stack+0x4f/0x7c [175827.356887] [<ffffffff8106c79c>] warn_slowpath_common+0x77/0x91 [175827.356896] [<ffffffff816e0bff>] ? sk_stream_kill_queues+0xfe/0x126 [175827.356903] [<ffffffff8106c7cb>] warn_slowpath_null+0x15/0x17 [175827.356910] [<ffffffff816e0bff>] sk_stream_kill_queues+0xfe/0x126 [175827.356918] [<ffffffff8171af14>] inet_csk_destroy_sock+0x78/0xfe [175827.356925] [<ffffffff8171cf91>] tcp_done+0x78/0x7d [175827.356931] [<ffffffff8172581f>] tcp_reset+0x55/0x59 [175827.356937] [<ffffffff81726c81>] tcp_rcv_state_process+0xa05/0xab9 [175827.356944] [<ffffffff81114560>] ? kfree+0xc7/0x125 [175827.356952] [<ffffffff81087612>] ? notifier_call_chain+0x32/0x5c [175827.356960] [<ffffffff810ed9d0>] ? put_page+0x12/0x23 [175827.356967] [<ffffffff8172c7b6>] tcp_v4_do_rcv+0x2c7/0x313 [175827.356973] [<ffffffff8172f1d6>] ? tcp_v4_rcv+0x6b4/0x734 [175827.356979] [<ffffffff8172eeb9>] tcp_v4_rcv+0x397/0x734 [175827.356986] [<ffffffff81087612>] ? notifier_call_chain+0x32/0x5c [175827.356996] [<ffffffffa0616009>] ? rtl8169_start_xmit+0x29a/0x716 [r8169] [175827.357003] [<ffffffff810911ad>] ? update_curr+0x89/0xe1 [175827.357010] [<ffffffff81711b82>] ip_local_deliver_finish+0x10c/0x1ed [175827.357016] [<ffffffff81711d9d>] ip_local_deliver+0x47/0x78 [175827.357022] [<ffffffff81711a3f>] ip_rcv_finish+0x29e/0x2d5 [175827.357028] [<ffffffff81712075>] ip_rcv+0x2a7/0x316 [175827.357035] [<ffffffff816e747d>] __netif_receive_skb_core+0x569/0x5cb [175827.357045] [<ffffffffa047d770>] ? br_handle_local_finish+0x41/0x41 [bridge] [175827.357051] [<ffffffff816e7528>] __netif_receive_skb+0x49/0x5b [175827.357058] [<ffffffff816e76e4>] netif_receive_skb_internal+0x56/0x8d [175827.357066] [<ffffffffa047d770>] ? br_handle_local_finish+0x41/0x41 [bridge] [175827.357073] [<ffffffff816e7724>] netif_receive_skb+0x9/0xb [175827.357081] [<ffffffffa047d9e6>] br_handle_frame_finish+0x276/0x2e1 [bridge] [175827.357090] [<ffffffffa04830c7>] br_nf_pre_routing_finish+0x2ad/0x30d [bridge] [175827.357099] [<ffffffffa047d770>] ? br_handle_local_finish+0x41/0x41 [bridge] [175827.357107] [<ffffffffa048364e>] br_nf_pre_routing+0x527/0x59f [bridge] [175827.357115] [<ffffffff8170c74f>] nf_iterate+0x3f/0x7a [175827.357122] [<ffffffff8170c7f8>] nf_hook_slow+0x6e/0x106 [175827.357130] [<ffffffffa047d770>] ? br_handle_local_finish+0x41/0x41 [bridge] [175827.357138] [<ffffffffa047dc35>] br_handle_frame+0x1e4/0x1f9 [bridge] [175827.357145] [<ffffffffa047da51>] ? br_handle_frame_finish+0x2e1/0x2e1 [bridge] [175827.357152] [<ffffffff816e7328>] __netif_receive_skb_core+0x414/0x5cb [175827.357158] [<ffffffff816e7528>] __netif_receive_skb+0x49/0x5b [175827.357165] [<ffffffff816e76e4>] netif_receive_skb_internal+0x56/0x8d [175827.357172] [<ffffffff816e8664>] napi_gro_receive+0x27/0x68 [175827.357179] [<ffffffffa06134bc>] rtl8169_poll+0x337/0x4d0 [r8169] [175827.357186] [<ffffffff81087612>] ? notifier_call_chain+0x32/0x5c [175827.357193] [<ffffffff816e78f0>] net_rx_action+0x6f/0x13e [175827.357198] [<ffffffff8106fd7b>] __do_softirq+0xc0/0x1c0 [175827.357204] [<ffffffff8106ffe9>] irq_exit+0x38/0x7e [175827.357212] [<ffffffff81036dab>] do_IRQ+0xb5/0xd0 [175827.357221] [<ffffffff8185baaa>] common_interrupt+0x6a/0x6a [175827.357224] <EOI> [<ffffffff81086998>] ? __hrtimer_start_range_ns+0x25f/0x27b [175827.357237] [<ffffffff8168bbc5>] ? cpuidle_enter_state+0x49/0xac [175827.357243] [<ffffffff8168bbbe>] ? cpuidle_enter_state+0x42/0xac [175827.357250] [<ffffffff8168bcf0>] cpuidle_idle_call+0xc8/0x111 [175827.357256] [<ffffffff8103d666>] arch_cpu_idle+0x9/0x18 [175827.357262] [<ffffffff810a83fc>] cpu_startup_entry+0xb5/0x11c [175827.357269] [<ffffffff8184c287>] rest_init+0x7b/0x7f [175827.357275] [<ffffffff81c02d04>] start_kernel+0x3b4/0x3c1 [175827.357281] [<ffffffff81c02757>] ? repair_env_string+0x5a/0x5a [175827.357287] [<ffffffff81c0248b>] x86_64_start_reservations+0x2a/0x2c [175827.357293] [<ffffffff81c0255a>] x86_64_start_kernel+0xcd/0xd1 [175827.357297] ---[ end trace e102d0a9c3759f49 ]--- [175827.357336] ------------[ cut here ]------------ [175827.357345] WARNING: CPU: 0 PID: 0 at net/ipv4/af_inet.c:153 inet_sock_destruct+0x161/0x1af() [175827.357348] Modules linked in: bridge joydev nvidia(PO) x86_pkg_temp_thermal coretemp pcspkr nouveau i2c_i801 snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic mxm_wmi r8169 mii ttm snd_hda_intel snd_hda_codec snd_hwdep rtc_cmos wmi efivarfs xts gf128mul aes_x86_64 sha512_generic libiscsi scsi_transport_iscsi tg3 fuse nfs lockd sunrpc jfs hid_sunplus hid_sony hid_samsung hid_pl hid_petalynx hid_monterey hid_microsoft hid_logitech hid_gyration hid_ezkey hid_cypress hid_chicony hid_cherry hid_belkin hid_apple hid_a4tech sl811_hcd aic94xx libsas scsi_transport_sas lpfc qla2xxx scsi_transport_fc megaraid_sas megaraid_mbox megaraid_mm megaraid aacraid sx8 DAC960 cciss 3w_9xxx 3w_xxxx atp870u dc395x qla1280 dmx3191d sym53c8xx gdth advansys initio BusLogic arcmsr aic7xxx aic79xx scsi_transport_spi [175827.357429] pdc_adma sata_inic162x sata_qstor sata_vsc sata_uli sata_sis sata_sx4 sata_via sata_svw sata_sil24 sata_sil sata_promise pata_sl82c105 pata_cs5530 pata_cs5520 pata_via pata_jmicron pata_marvell pata_sis pata_netcell pata_sc1200 pata_pdc202xx_old pata_triflex pata_atiixp pata_opti pata_amd pata_ali pata_it8213 pata_pcmcia pcmcia pcmcia_core pata_ns87415 pata_ns87410 pata_serverworks pata_platform pata_artop pata_it821x pata_optidma pata_hpt3x2n pata_hpt3x3 pata_hpt37x pata_hpt366 pata_cmd64x pata_efar pata_rz1000 pata_sil680 pata_radisys pata_pdc2027x pata_mpiix usbhid [175827.357483] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P W O 3.14.3-gentoo #1 [175827.357486] Hardware name: ASUSTeK COMPUTER INC. G56JR/G56JR, BIOS G56JR.200 01/13/2014 [175827.357490] 0000000000000009 ffff88042ee037f8 ffffffff818532e4 0000000000000102 [175827.357496] 0000000000000000 ffff88042ee03838 ffffffff8106c79c ffff8803a2560730 [175827.357502] ffffffff8173f483 ffff8803a25606c0 ffff8803a2560838 0000000000000000 [175827.357508] Call Trace: [175827.357512] <IRQ> [<ffffffff818532e4>] dump_stack+0x4f/0x7c [175827.357522] [<ffffffff8106c79c>] warn_slowpath_common+0x77/0x91 [175827.357529] [<ffffffff8173f483>] ? inet_sock_destruct+0x161/0x1af [175827.357535] [<ffffffff8106c7cb>] warn_slowpath_null+0x15/0x17 [175827.357541] [<ffffffff8173f483>] inet_sock_destruct+0x161/0x1af [175827.357548] [<ffffffff816d86cb>] __sk_free+0x18/0x10c [175827.357553] [<ffffffff816d881c>] sk_free+0x13/0x15 [175827.357560] [<ffffffff8172f1ff>] tcp_v4_rcv+0x6dd/0x734 [175827.357567] [<ffffffff81087612>] ? notifier_call_chain+0x32/0x5c [175827.357575] [<ffffffffa0616009>] ? rtl8169_start_xmit+0x29a/0x716 [r8169] [175827.357581] [<ffffffff810911ad>] ? update_curr+0x89/0xe1 [175827.357587] [<ffffffff81711b82>] ip_local_deliver_finish+0x10c/0x1ed [175827.357593] [<ffffffff81711d9d>] ip_local_deliver+0x47/0x78 [175827.357599] [<ffffffff81711a3f>] ip_rcv_finish+0x29e/0x2d5 [175827.357605] [<ffffffff81712075>] ip_rcv+0x2a7/0x316 [175827.357612] [<ffffffff816e747d>] __netif_receive_skb_core+0x569/0x5cb [175827.357620] [<ffffffffa047d770>] ? br_handle_local_finish+0x41/0x41 [bridge] [175827.357626] [<ffffffff816e7528>] __netif_receive_skb+0x49/0x5b [175827.357632] [<ffffffff816e76e4>] netif_receive_skb_internal+0x56/0x8d [175827.357640] [<ffffffffa047d770>] ? br_handle_local_finish+0x41/0x41 [bridge] [175827.357646] [<ffffffff816e7724>] netif_receive_skb+0x9/0xb [175827.357653] [<ffffffffa047d9e6>] br_handle_frame_finish+0x276/0x2e1 [bridge] [175827.357661] [<ffffffffa04830c7>] br_nf_pre_routing_finish+0x2ad/0x30d [bridge] [175827.357669] [<ffffffffa047d770>] ? br_handle_local_finish+0x41/0x41 [bridge] [175827.357677] [<ffffffffa048364e>] br_nf_pre_routing+0x527/0x59f [bridge] [175827.357684] [<ffffffff8170c74f>] nf_iterate+0x3f/0x7a [175827.357691] [<ffffffff8170c7f8>] nf_hook_slow+0x6e/0x106 [175827.357699] [<ffffffffa047d770>] ? br_handle_local_finish+0x41/0x41 [bridge] [175827.357707] [<ffffffffa047dc35>] br_handle_frame+0x1e4/0x1f9 [bridge] [175827.357714] [<ffffffffa047da51>] ? br_handle_frame_finish+0x2e1/0x2e1 [bridge] [175827.357720] [<ffffffff816e7328>] __netif_receive_skb_core+0x414/0x5cb [175827.357727] [<ffffffff816e7528>] __netif_receive_skb+0x49/0x5b [175827.357733] [<ffffffff816e76e4>] netif_receive_skb_internal+0x56/0x8d [175827.357740] [<ffffffff816e8664>] napi_gro_receive+0x27/0x68 [175827.357747] [<ffffffffa06134bc>] rtl8169_poll+0x337/0x4d0 [r8169] [175827.357754] [<ffffffff81087612>] ? notifier_call_chain+0x32/0x5c [175827.357761] [<ffffffff816e78f0>] net_rx_action+0x6f/0x13e [175827.357766] [<ffffffff8106fd7b>] __do_softirq+0xc0/0x1c0 [175827.357772] [<ffffffff8106ffe9>] irq_exit+0x38/0x7e [175827.357778] [<ffffffff81036dab>] do_IRQ+0xb5/0xd0 [175827.357785] [<ffffffff8185baaa>] common_interrupt+0x6a/0x6a [175827.357788] <EOI> [<ffffffff81086998>] ? __hrtimer_start_range_ns+0x25f/0x27b [175827.357800] [<ffffffff8168bbc5>] ? cpuidle_enter_state+0x49/0xac [175827.357807] [<ffffffff8168bbbe>] ? cpuidle_enter_state+0x42/0xac [175827.357813] [<ffffffff8168bcf0>] cpuidle_idle_call+0xc8/0x111 [175827.357819] [<ffffffff8103d666>] arch_cpu_idle+0x9/0x18 [175827.357824] [<ffffffff810a83fc>] cpu_startup_entry+0xb5/0x11c [175827.357830] [<ffffffff8184c287>] rest_init+0x7b/0x7f [175827.357836] [<ffffffff81c02d04>] start_kernel+0x3b4/0x3c1 [175827.357842] [<ffffffff81c02757>] ? repair_env_string+0x5a/0x5a [175827.357848] [<ffffffff81c0248b>] x86_64_start_reservations+0x2a/0x2c [175827.357854] [<ffffffff81c0255a>] x86_64_start_kernel+0xcd/0xd1 [175827.357858] ---[ end trace e102d0a9c3759f4a ]--- and out of memory message: [183972.519743] TCP: out of memory -- consider tuning tcp_mem tcp_mem and sockstat: # cat /proc/sys/net/ipv4/tcp_mem ; cat /proc/net/sockstat 376935 502581 753870 sockets: used 467 TCP: inuse 3 orphan 0 tw 0 alloc 5 mem 753741 UDP: inuse 1 mem 1 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 After exiting rtorrent stats above didn't change. # free -m total used free shared buffers cached Mem: 15908 9873 6035 3 50 8890 -/+ buffers/cache: 932 14976 Swap: 8175 1 8174 # sysctl net.ipv4.ip_forward net.ipv4.ip_forward = 1 # uptime 00:40:06 up 2 days, 4:45, 8 users, load average: 0,01, 0,15, 0,33 Hardware: Network controller: Intel Corporation Wireless 7260 (rev 73) Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
I see what looks like the same problem about every 3 days on version 3.15.7-200.fc20.x86_64. The hardware is different, the process running is squid not rtorrent, but the signature is the same with log entries: TCP: out of memory -- consider tuning tcp_mem WARNING: CPU: 2 PID: 2536 at net/core/stream.c:201 sk_stream_kill_queues+0x12c/0x130() WARNING: CPU: 2 PID: 2536 at net/ipv4/af_inet.c:153 inet_sock_destruct+0x1d9/0x1e0() I can supply the full log with traceback if that is helpful. The key point is that this bug is not specific to the originators kernel version, hardware or configuration. This is a regression in so far as it wasn't there in 3.6.10. I appreciate that is a pretty big window.
I'm running into the same bug on Arch Linux kernel 3.17.2. I also have a bridge interface but only 1 NIC (I use the bridge for virtual machines but at the time of the bug I had no VMs loaded.) Also likewise the system had been up for 3-4 days at the time of the bug. Hardware as reported by lspci: $ lspci | grep Ethernet 07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06) And it's using the r8169 kernel module as the driver. Here's the dmesg output: [322438.563394] TCP: out of memory -- consider tuning tcp_mem [322475.994051] TCP: out of memory -- consider tuning tcp_mem [322479.087904] TCP: out of memory -- consider tuning tcp_mem [322486.753438] TCP: out of memory -- consider tuning tcp_mem [322618.973412] ------------[ cut here ]------------ [322618.973426] WARNING: CPU: 0 PID: 0 at net/core/stream.c:201 sk_stream_kill_queues+0x151/0x160() [322618.973427] Modules linked in: fuse bridge stp llc snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec iTCO_wdt iTCO_vendor_support nvidia(PO) snd_hwdep snd_pcm mxm_wmi evdev drm ecb serio_raw snd_timer btusb coretemp hwmon intel_rapl x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm r8169 pcspkr mousedev joydev bluetooth snd mac_hid mii rfkill soundcore thermal shpchp fan mei_me mei i2c_i801 button i2c_core lpc_ich battery wmi tpm_infineon video tpm_tis tpm processor ext4 crc16 mbcache jbd2 algif_skcipher af_alg dm_crypt dm_mod sr_mod cdrom sd_mod crc_t10dif ata_generic pata_acpi hid_generic usbhid hid atkbd libps2 crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper [322618.973464] ablk_helper cryptd ahci ata_piix libahci libata scsi_mod ehci_pci xhci_hcd ehci_hcd usbcore usb_common i8042 serio [322618.973471] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P O 3.17.2-1-ARCH #1 [322618.973473] Hardware name: MSI MS-7681/Z68A-GD65 (G3) (MS-7681), BIOS V25.8 01/11/2013 [322618.973474] 0000000000000000 67bcc9c1036c0b93 ffff88041ec03d40 ffffffff815367d0 [322618.973476] 0000000000000000 ffff88041ec03d78 ffffffff8107054d ffff88030f0dc600 [322618.973478] ffff88030f0dc780 ffff88030f0dc728 0000000000000000 ffff88030f0dc600 [322618.973480] Call Trace: [322618.973482] <IRQ> [<ffffffff815367d0>] dump_stack+0x4d/0x6f [322618.973488] [<ffffffff8107054d>] warn_slowpath_common+0x7d/0xa0 [322618.973490] [<ffffffff8107067a>] warn_slowpath_null+0x1a/0x20 [322618.973493] [<ffffffff81438a31>] sk_stream_kill_queues+0x151/0x160 [322618.973496] [<ffffffff814890a6>] inet_csk_destroy_sock+0x56/0x170 [322618.973498] [<ffffffff8148a9a7>] tcp_done+0x67/0xa0 [322618.973500] [<ffffffff8149cd61>] tcp_write_err+0x31/0x50 [322618.973502] [<ffffffff8149d628>] tcp_retransmit_timer+0x208/0x770 [322618.973504] [<ffffffff8149dd40>] ? tcp_write_timer_handler+0x1b0/0x1b0 [322618.973508] [<ffffffff8149dc20>] tcp_write_timer_handler+0x90/0x1b0 [322618.973510] [<ffffffff8149ddbc>] tcp_write_timer+0x7c/0x80 [322618.973512] [<ffffffff810d5aa6>] call_timer_fn+0x36/0x160 [322618.973514] [<ffffffff8149dd40>] ? tcp_write_timer_handler+0x1b0/0x1b0 [322618.973516] [<ffffffff810d64d4>] run_timer_softirq+0x274/0x320 [322618.973519] [<ffffffff81074402>] __do_softirq+0xf2/0x2e0 [322618.973521] [<ffffffff81074746>] irq_exit+0x86/0xb0 [322618.973524] [<ffffffff8153f334>] smp_apic_timer_interrupt+0x44/0x50 [322618.973526] [<ffffffff8153d5cd>] apic_timer_interrupt+0x6d/0x80 [322618.973527] <EOI> [<ffffffff813f27f2>] ? cpuidle_enter_state+0x62/0x1a0 [322618.973532] [<ffffffff813f2a17>] cpuidle_enter+0x17/0x20 [322618.973535] [<ffffffff810afd1c>] cpu_startup_entry+0x3dc/0x430 [322618.973538] [<ffffffff8152cab4>] rest_init+0x84/0x90 [322618.973540] [<ffffffff818fbff1>] start_kernel+0x471/0x492 [322618.973542] [<ffffffff818fb120>] ? early_idt_handlers+0x120/0x120 [322618.973544] [<ffffffff818fb4d7>] x86_64_start_reservations+0x2a/0x2c [322618.973546] [<ffffffff818fb62b>] x86_64_start_kernel+0x152/0x175 [322618.973547] ---[ end trace 3468a80001ace10a ]--- [322618.973550] ------------[ cut here ]------------ [322618.973553] WARNING: CPU: 0 PID: 0 at net/ipv4/af_inet.c:153 inet_sock_destruct+0x191/0x200() [322618.973554] Modules linked in: fuse bridge stp llc snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec iTCO_wdt iTCO_vendor_support nvidia(PO) snd_hwdep snd_pcm mxm_wmi evdev drm ecb serio_raw snd_timer btusb coretemp hwmon intel_rapl x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm r8169 pcspkr mousedev joydev bluetooth snd mac_hid mii rfkill soundcore thermal shpchp fan mei_me mei i2c_i801 button i2c_core lpc_ich battery wmi tpm_infineon video tpm_tis tpm processor ext4 crc16 mbcache jbd2 algif_skcipher af_alg dm_crypt dm_mod sr_mod cdrom sd_mod crc_t10dif ata_generic pata_acpi hid_generic usbhid hid atkbd libps2 crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper [322618.973577] ablk_helper cryptd ahci ata_piix libahci libata scsi_mod ehci_pci xhci_hcd ehci_hcd usbcore usb_common i8042 serio [322618.973582] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P W O 3.17.2-1-ARCH #1 [322618.973583] Hardware name: MSI MS-7681/Z68A-GD65 (G3) (MS-7681), BIOS V25.8 01/11/2013 [322618.973584] 0000000000000000 67bcc9c1036c0b93 ffff88041ec03db0 ffffffff815367d0 [322618.973586] 0000000000000000 ffff88041ec03de8 ffffffff8107054d ffff88030f0dc600 [322618.973588] ffff88030f0dc780 ffff88030f0dc66c ffffffff8149dd40 ffff88030f0dc600 [322618.973589] Call Trace: [322618.973590] <IRQ> [<ffffffff815367d0>] dump_stack+0x4d/0x6f [322618.973593] [<ffffffff8107054d>] warn_slowpath_common+0x7d/0xa0 [322618.973595] [<ffffffff8149dd40>] ? tcp_write_timer_handler+0x1b0/0x1b0 [322618.973597] [<ffffffff8107067a>] warn_slowpath_null+0x1a/0x20 [322618.973599] [<ffffffff814b5281>] inet_sock_destruct+0x191/0x200 [322618.973601] [<ffffffff8142d70d>] __sk_free+0x1d/0x130 [322618.973603] [<ffffffff8142d839>] sk_free+0x19/0x20 [322618.973605] [<ffffffff8149dda8>] tcp_write_timer+0x68/0x80 [322618.973607] [<ffffffff810d5aa6>] call_timer_fn+0x36/0x160 [322618.973609] [<ffffffff8149dd40>] ? tcp_write_timer_handler+0x1b0/0x1b0 [322618.973611] [<ffffffff810d64d4>] run_timer_softirq+0x274/0x320 [322618.973613] [<ffffffff81074402>] __do_softirq+0xf2/0x2e0 [322618.973615] [<ffffffff81074746>] irq_exit+0x86/0xb0 [322618.973617] [<ffffffff8153f334>] smp_apic_timer_interrupt+0x44/0x50 [322618.973619] [<ffffffff8153d5cd>] apic_timer_interrupt+0x6d/0x80 [322618.973620] <EOI> [<ffffffff813f27f2>] ? cpuidle_enter_state+0x62/0x1a0 [322618.973623] [<ffffffff813f2a17>] cpuidle_enter+0x17/0x20 [322618.973625] [<ffffffff810afd1c>] cpu_startup_entry+0x3dc/0x430 [322618.973627] [<ffffffff8152cab4>] rest_init+0x84/0x90 [322618.973629] [<ffffffff818fbff1>] start_kernel+0x471/0x492 [322618.973632] [<ffffffff818fb120>] ? early_idt_handlers+0x120/0x120 [322618.973634] [<ffffffff818fb4d7>] x86_64_start_reservations+0x2a/0x2c [322618.973635] [<ffffffff818fb62b>] x86_64_start_kernel+0x152/0x175 [322618.973637] ---[ end trace 3468a80001ace10b ]--- [322654.327487] Netfilter messages via NETLINK v0.30. [322655.186174] cfg80211: Calling CRDA to update world regulatory domain
Also forgot to mention the symptoms of this bug. It happened while I was downloading updates with pacman (Arch package manager.) I wasn't doing any kind of Bittorrent at the time, just a simple HTTP download, unlike the previous reporters. The download timed out and at the same time I was unable to load any web pages in Firefox. That's when I checked dmesg and saw the error. I could still ping external websites fine (so I guess ICMP is unaffected) but basically nothing else I tried on the network worked. I loaded up Wireshark and noticed it seemed to be succsesfully completing the SYN, SYN-ACK, ACK handshake and it even did a SSL/TLS handshake but didn't do anything after that. Firefox timed out every website I tried and pacman timed out every time trying to connect to my mirror. After repeated timeouts, I ended up just rebooting the system and that brought my network back to working order. Also despite the "out of memory" error at the time I was only using ~1GB of 16GB available RAM so that wouldn't be the issue... I suppose it was just refering whatever range of memory the kernel allocated for itself, not system-wide memory. Here's the link to the post I made on the Arch forums in case anything develops there: https://bbs.archlinux.org/viewtopic.php?id=189695
I just hit this bug again... this time my uptime was less than 3 hours. I may have been using the networking stack more heavily because I was running Wireshark at the time of the bug (and the kernel bug actually crashed the Wireshark process and made Firefox max out 1 of my cores.) I didn't get the "out of memory" this time but the stack traces look very similar. I'm about to remove my network bridge and see if I don't run into this again because I suspect the problem lies there (I'll have to use NAT or something for my VMs but I don't use them all that much and wasn't using them at the time of the bug) [ 8991.050876] Netfilter messages via NETLINK v0.30. [ 8991.907237] cfg80211: Calling CRDA to update world regulatory domain [ 9886.005800] ------------[ cut here ]------------ [ 9886.005811] WARNING: CPU: 0 PID: 0 at net/core/stream.c:201 sk_stream_kill_queues+0x151/0x160() [ 9886.005813] Modules linked in: cfg80211 nfnetlink_queue nfnetlink_log nfnetlink nls_iso8859_1 nls_cp437 vfat fat loop uas usb_storage fuse bridge stp llc mousedev joydev ecb btusb nvidia(PO) bluetooth iTCO_wdt rfkill iTCO_vendor_support mxm_wmi coretemp snd_hda_codec_hdmi hwmon intel_rapl r8169 drm x86_pkg_temp_thermal mii intel_powerclamp kvm_intel snd_hda_codec_realtek thermal wmi evdev kvm mac_hid snd_hda_codec_generic serio_raw pcspkr i2c_i801 fan snd_hda_intel snd_hda_controller snd_hda_codec i2c_core shpchp snd_hwdep snd_pcm snd_timer snd mei_me mei lpc_ich tpm_infineon tpm_tis tpm battery soundcore button video processor sch_fq_codel ext4 crc16 mbcache jbd2 algif_skcipher af_alg dm_crypt dm_mod sr_mod sd_mod cdrom crc_t10dif ata_generic pata_acpi hid_generic usbhid hid atkbd libps2 crct10dif_pclmul [ 9886.005849] crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ahci ata_piix libahci libata scsi_mod xhci_hcd ehci_pci ehci_hcd usbcore usb_common i8042 serio [ 9886.005859] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P O 3.17.2-1-ARCH #1 [ 9886.005861] Hardware name: MSI MS-7681/Z68A-GD65 (G3) (MS-7681), BIOS V25.8 01/11/2013 [ 9886.005863] 0000000000000000 c27ad83c5cf732d9 ffff88041ec037d8 ffffffff815367d0 [ 9886.005864] 0000000000000000 ffff88041ec03810 ffffffff8107054d ffff880235ee4d00 [ 9886.005866] ffff880235ee4e80 0000000000004650 0000000000000000 0000000000003c83 [ 9886.005868] Call Trace: [ 9886.005870] <IRQ> [<ffffffff815367d0>] dump_stack+0x4d/0x6f [ 9886.005876] [<ffffffff8107054d>] warn_slowpath_common+0x7d/0xa0 [ 9886.005880] [<ffffffff8107067a>] warn_slowpath_null+0x1a/0x20 [ 9886.005882] [<ffffffff81438a31>] sk_stream_kill_queues+0x151/0x160 [ 9886.005885] [<ffffffff814890a6>] inet_csk_destroy_sock+0x56/0x170 [ 9886.005887] [<ffffffff8148a9a7>] tcp_done+0x67/0xa0 [ 9886.005890] [<ffffffff814a27dc>] tcp_time_wait+0x6c/0x230 [ 9886.005892] [<ffffffff81497942>] tcp_rcv_state_process+0xda2/0xdc0 [ 9886.005895] [<ffffffff8109b55f>] ? try_to_wake_up+0x1ff/0x380 [ 9886.005897] [<ffffffff8149ec31>] tcp_v4_do_rcv+0xf1/0x3a0 [ 9886.005899] [<ffffffff8107caa0>] ? complete_signal+0x110/0x270 [ 9886.005901] [<ffffffff814a1613>] tcp_v4_rcv+0x7f3/0x810 [ 9886.005904] [<ffffffff8147510a>] ? nf_iterate+0xaa/0xc0 [ 9886.005906] [<ffffffff8147bd52>] ip_local_deliver_finish+0xb2/0x220 [ 9886.005908] [<ffffffff8147c399>] ip_local_deliver+0x39/0x90 [ 9886.005910] [<ffffffff8147bfb5>] ip_rcv_finish+0xf5/0x350 [ 9886.005911] [<ffffffff8147c69e>] ip_rcv+0x2ae/0x380 [ 9886.005914] [<ffffffff81442412>] __netif_receive_skb_core+0x5b2/0x880 [ 9886.005916] [<ffffffff8147510a>] ? nf_iterate+0xaa/0xc0 [ 9886.005918] [<ffffffff814426f6>] __netif_receive_skb+0x16/0x70 [ 9886.005920] [<ffffffff81442790>] netif_receive_skb_internal+0x40/0xd0 [ 9886.005922] [<ffffffff8144283c>] netif_receive_skb+0x1c/0x80 [ 9886.005926] [<ffffffffa057b0a9>] br_handle_frame_finish+0x1d9/0x430 [bridge] [ 9886.005930] [<ffffffffa0581dc3>] br_nf_pre_routing_finish+0x1e3/0x3e0 [bridge] [ 9886.005933] [<ffffffffa0582b00>] br_nf_pre_routing+0x430/0x730 [bridge] [ 9886.005935] [<ffffffff810aefb5>] ? __wake_up_common+0x55/0x90 [ 9886.005938] [<ffffffffa057aed0>] ? br_port_flags_change+0x20/0x20 [bridge] [ 9886.005940] [<ffffffff8147510a>] nf_iterate+0xaa/0xc0 [ 9886.005943] [<ffffffffa057aed0>] ? br_port_flags_change+0x20/0x20 [bridge] [ 9886.005945] [<ffffffff814751a4>] nf_hook_slow+0x84/0x150 [ 9886.005947] [<ffffffffa057aed0>] ? br_port_flags_change+0x20/0x20 [bridge] [ 9886.005950] [<ffffffffa057b540>] br_handle_frame+0x190/0x260 [bridge] [ 9886.005952] [<ffffffff814420f2>] __netif_receive_skb_core+0x292/0x880 [ 9886.005955] [<ffffffffa057b3b0>] ? br_handle_local_finish+0xb0/0xb0 [bridge] [ 9886.005956] [<ffffffff814426f6>] __netif_receive_skb+0x16/0x70 [ 9886.005958] [<ffffffff81442790>] netif_receive_skb_internal+0x40/0xd0 [ 9886.005960] [<ffffffff81443060>] napi_gro_receive+0xd0/0x110 [ 9886.005963] [<ffffffffa04f6a86>] rtl8169_poll+0x316/0x670 [r8169] [ 9886.005965] [<ffffffff81443cca>] net_rx_action+0x12a/0x250 [ 9886.005968] [<ffffffff81074402>] __do_softirq+0xf2/0x2e0 [ 9886.005970] [<ffffffff81074746>] irq_exit+0x86/0xb0 [ 9886.005972] [<ffffffff8153f266>] do_IRQ+0x56/0xe0 [ 9886.005975] [<ffffffff8153d22d>] common_interrupt+0x6d/0x6d [ 9886.005976] <EOI> [<ffffffff813f27f2>] ? cpuidle_enter_state+0x62/0x1a0 [ 9886.005980] [<ffffffff813f2a17>] cpuidle_enter+0x17/0x20 [ 9886.005982] [<ffffffff810afd1c>] cpu_startup_entry+0x3dc/0x430 [ 9886.005985] [<ffffffff8152cab4>] rest_init+0x84/0x90 [ 9886.005987] [<ffffffff818fbff1>] start_kernel+0x471/0x492 [ 9886.005989] [<ffffffff818fb120>] ? early_idt_handlers+0x120/0x120 [ 9886.005991] [<ffffffff818fb4d7>] x86_64_start_reservations+0x2a/0x2c [ 9886.005992] [<ffffffff818fb62b>] x86_64_start_kernel+0x152/0x175 [ 9886.005994] ---[ end trace 1e3918801e34b500 ]--- [ 9886.005996] ------------[ cut here ]------------ [ 9886.005999] WARNING: CPU: 0 PID: 0 at net/ipv4/af_inet.c:153 inet_sock_destruct+0x191/0x200() [ 9886.006000] Modules linked in: cfg80211 nfnetlink_queue nfnetlink_log nfnetlink nls_iso8859_1 nls_cp437 vfat fat loop uas usb_storage fuse bridge stp llc mousedev joydev ecb btusb nvidia(PO) bluetooth iTCO_wdt rfkill iTCO_vendor_support mxm_wmi coretemp snd_hda_codec_hdmi hwmon intel_rapl r8169 drm x86_pkg_temp_thermal mii intel_powerclamp kvm_intel snd_hda_codec_realtek thermal wmi evdev kvm mac_hid snd_hda_codec_generic serio_raw pcspkr i2c_i801 fan snd_hda_intel snd_hda_controller snd_hda_codec i2c_core shpchp snd_hwdep snd_pcm snd_timer snd mei_me mei lpc_ich tpm_infineon tpm_tis tpm battery soundcore button video processor sch_fq_codel ext4 crc16 mbcache jbd2 algif_skcipher af_alg dm_crypt dm_mod sr_mod sd_mod cdrom crc_t10dif ata_generic pata_acpi hid_generic usbhid hid atkbd libps2 crct10dif_pclmul [ 9886.006024] crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd ahci ata_piix libahci libata scsi_mod xhci_hcd ehci_pci ehci_hcd usbcore usb_common i8042 serio [ 9886.006032] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P W O 3.17.2-1-ARCH #1 [ 9886.006033] Hardware name: MSI MS-7681/Z68A-GD65 (G3) (MS-7681), BIOS V25.8 01/11/2013 [ 9886.006034] 0000000000000000 c27ad83c5cf732d9 ffff88041ec038e0 ffffffff815367d0 [ 9886.006035] 0000000000000000 ffff88041ec03918 ffffffff8107054d ffff880235ee4d00 [ 9886.006037] ffff880235ee4e80 ffff880235ee4d70 ffff8801e7d012e2 ffff880235ee4d00 [ 9886.006039] Call Trace: [ 9886.006040] <IRQ> [<ffffffff815367d0>] dump_stack+0x4d/0x6f [ 9886.006043] [<ffffffff8107054d>] warn_slowpath_common+0x7d/0xa0 [ 9886.006049] [<ffffffff8107067a>] warn_slowpath_null+0x1a/0x20 [ 9886.006051] [<ffffffff814b5281>] inet_sock_destruct+0x191/0x200 [ 9886.006053] [<ffffffff8142d70d>] __sk_free+0x1d/0x130 [ 9886.006054] [<ffffffff8142d839>] sk_free+0x19/0x20 [ 9886.006056] [<ffffffff814a15cd>] tcp_v4_rcv+0x7ad/0x810 [ 9886.006058] [<ffffffff8147510a>] ? nf_iterate+0xaa/0xc0 [ 9886.006060] [<ffffffff8147bd52>] ip_local_deliver_finish+0xb2/0x220 [ 9886.006062] [<ffffffff8147c399>] ip_local_deliver+0x39/0x90 [ 9886.006063] [<ffffffff8147bfb5>] ip_rcv_finish+0xf5/0x350 [ 9886.006065] [<ffffffff8147c69e>] ip_rcv+0x2ae/0x380 [ 9886.006067] [<ffffffff81442412>] __netif_receive_skb_core+0x5b2/0x880 [ 9886.006069] [<ffffffff8147510a>] ? nf_iterate+0xaa/0xc0 [ 9886.006071] [<ffffffff814426f6>] __netif_receive_skb+0x16/0x70 [ 9886.006073] [<ffffffff81442790>] netif_receive_skb_internal+0x40/0xd0 [ 9886.006075] [<ffffffff8144283c>] netif_receive_skb+0x1c/0x80 [ 9886.006077] [<ffffffffa057b0a9>] br_handle_frame_finish+0x1d9/0x430 [bridge] [ 9886.006080] [<ffffffffa0581dc3>] br_nf_pre_routing_finish+0x1e3/0x3e0 [bridge] [ 9886.006083] [<ffffffffa0582b00>] br_nf_pre_routing+0x430/0x730 [bridge] [ 9886.006085] [<ffffffff810aefb5>] ? __wake_up_common+0x55/0x90 [ 9886.006087] [<ffffffffa057aed0>] ? br_port_flags_change+0x20/0x20 [bridge] [ 9886.006089] [<ffffffff8147510a>] nf_iterate+0xaa/0xc0 [ 9886.006092] [<ffffffffa057aed0>] ? br_port_flags_change+0x20/0x20 [bridge] [ 9886.006094] [<ffffffff814751a4>] nf_hook_slow+0x84/0x150 [ 9886.006096] [<ffffffffa057aed0>] ? br_port_flags_change+0x20/0x20 [bridge] [ 9886.006099] [<ffffffffa057b540>] br_handle_frame+0x190/0x260 [bridge] [ 9886.006101] [<ffffffff814420f2>] __netif_receive_skb_core+0x292/0x880 [ 9886.006103] [<ffffffffa057b3b0>] ? br_handle_local_finish+0xb0/0xb0 [bridge] [ 9886.006105] [<ffffffff814426f6>] __netif_receive_skb+0x16/0x70 [ 9886.006107] [<ffffffff81442790>] netif_receive_skb_internal+0x40/0xd0 [ 9886.006109] [<ffffffff81443060>] napi_gro_receive+0xd0/0x110 [ 9886.006112] [<ffffffffa04f6a86>] rtl8169_poll+0x316/0x670 [r8169] [ 9886.006114] [<ffffffff81443cca>] net_rx_action+0x12a/0x250 [ 9886.006116] [<ffffffff81074402>] __do_softirq+0xf2/0x2e0 [ 9886.006118] [<ffffffff81074746>] irq_exit+0x86/0xb0 [ 9886.006120] [<ffffffff8153f266>] do_IRQ+0x56/0xe0 [ 9886.006122] [<ffffffff8153d22d>] common_interrupt+0x6d/0x6d [ 9886.006123] <EOI> [<ffffffff813f27f2>] ? cpuidle_enter_state+0x62/0x1a0 [ 9886.006126] [<ffffffff813f2a17>] cpuidle_enter+0x17/0x20 [ 9886.006128] [<ffffffff810afd1c>] cpu_startup_entry+0x3dc/0x430 [ 9886.006129] [<ffffffff8152cab4>] rest_init+0x84/0x90 [ 9886.006131] [<ffffffff818fbff1>] start_kernel+0x471/0x492 [ 9886.006133] [<ffffffff818fb120>] ? early_idt_handlers+0x120/0x120 [ 9886.006135] [<ffffffff818fb4d7>] x86_64_start_reservations+0x2a/0x2c [ 9886.006137] [<ffffffff818fb62b>] x86_64_start_kernel+0x152/0x175 [ 9886.006138] ---[ end trace 1e3918801e34b501 ]---
Adding a me2, I've run into this a few times recently, pruned dmesg from today: [ 0.000000] Linux version 3.16.7 (ranma@nukunuku) (gcc version 4.9.1 (Debian 4.9.1-16) ) #2 SMP Fri Nov 7 00:05:16 CET 2014 [ 0.000000] Command line: auto BOOT_IMAGE=Linux ro root=906 root=/dev/md6 resume=/dev/md7 radeon.modeset=1 radeon.dpm=1 [...] [ 12.014763] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded [ 12.017028] r8169 0000:03:00.0: enabling Mem-Wr-Inval [ 12.017152] r8169 0000:03:00.0: enabling bus mastering [ 12.017205] r8169 0000:03:00.0: irq 43 for MSI/MSI-X [ 12.017341] r8169 0000:03:00.0 eth0: RTL8168f/8111f at 0xffffc90000066000, 08:60:6e:e8:40:f3, XID 08000800 IRQ 43 [ 12.019674] r8169 0000:03:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko] [...] [103497.160409] ------------[ cut here ]------------ [103497.160428] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:264 dev_watchdog+0x185/0x1eb() [103497.160433] NETDEV WATCHDOG: eth1 (r8169): transmit queue 0 timed out [103497.160436] Modules linked in: cfg80211 bluetooth ftdi_sio uvcvideo snd_usb_audio videobuf2_vmalloc videobuf2_memops videobuf2_core snd_usbmidi_lib snd_rawmidi k10temp [103497.160455] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.16.7 #2 [103497.160459] Hardware name: ASUS F2A85-M/F2A85-M, BIOS 4.0-7017-g9310df8-dirty 11/07/2014 [103497.160463] 0000000000000000 ffff88041ed03db8 ffffffff8f9425d1 ffff88041ed03e00 [103497.160469] ffff88041ed03df0 ffffffff8f08cf28 ffffffff8f83cf7a ffff88040d594000 [103497.160474] ffff88040dd07e00 0000000000000001 0000000000000002 ffff88041ed03e58 [103497.160480] Call Trace: [103497.160483] <IRQ> [<ffffffff8f9425d1>] dump_stack+0x45/0x56 [103497.160498] [<ffffffff8f08cf28>] warn_slowpath_common+0x7a/0x93 [103497.160504] [<ffffffff8f83cf7a>] ? dev_watchdog+0x185/0x1eb [103497.160510] [<ffffffff8f08cf84>] warn_slowpath_fmt+0x43/0x4b [103497.160516] [<ffffffff8f83cf7a>] dev_watchdog+0x185/0x1eb [103497.160521] [<ffffffff8f83cdf5>] ? dev_graft_qdisc+0x65/0x65 [103497.160527] [<ffffffff8f096036>] call_timer_fn+0x33/0xe8 [103497.160533] [<ffffffff8f83cdf5>] ? dev_graft_qdisc+0x65/0x65 [103497.160538] [<ffffffff8f096789>] run_timer_softirq+0x1d7/0x232 [103497.160544] [<ffffffff8f0908f7>] __do_softirq+0xde/0x249 [103497.160550] [<ffffffff8f090bee>] irq_exit+0x40/0x4e [103497.160555] [<ffffffff8f071e0a>] smp_apic_timer_interrupt+0x3f/0x4b [103497.160562] [<ffffffff8f94d05d>] apic_timer_interrupt+0x6d/0x80 [103497.160565] <EOI> [<ffffffff8f79b564>] ? cpuidle_enter_state+0x42/0x9b [103497.160575] [<ffffffff8f79b55d>] ? cpuidle_enter_state+0x3b/0x9b [103497.160580] [<ffffffff8f79b662>] cpuidle_enter+0x12/0x14 [103497.160587] [<ffffffff8f0bd6a7>] cpu_startup_entry+0x247/0x3af [103497.160593] [<ffffffff8f0705a7>] start_secondary+0x207/0x20d [103497.160597] ---[ end trace d426860d4ef06a03 ]--- [103500.415434] [sched_delayed] sched: RT throttling activated [103500.466149] r8169 0000:03:00.0 eth1: link up [103532.701047] TCP: out of memory -- consider tuning tcp_mem [103563.541333] TCP: out of memory -- consider tuning tcp_mem [103563.958036] ------------[ cut here ]------------ [103563.958048] WARNING: CPU: 1 PID: 4344 at net/core/stream.c:201 sk_stream_kill_queues+0x105/0x10a() [103563.958050] Modules linked in: cfg80211 bluetooth ftdi_sio uvcvideo snd_usb_audio videobuf2_vmalloc videobuf2_memops videobuf2_core snd_usbmidi_lib snd_rawmidi k10temp [103563.958060] CPU: 1 PID: 4344 Comm: rtorrent Tainted: G W 3.16.7 #2 [103563.958062] Hardware name: ASUS F2A85-M/F2A85-M, BIOS 4.0-7017-g9310df8-dirty 11/07/2014 [103563.958063] 0000000000000000 ffff88041ec83d70 ffffffff8f9425d1 0000000000000000 [103563.958066] ffff88041ec83da8 ffffffff8f08cf28 ffffffff8f81a708 ffff88037bd29740 [103563.958068] ffff88037bd298a0 ffff88037bd29848 ffff88037bd29ab0 ffff88041ec83db8 [103563.958071] Call Trace: [103563.958073] <IRQ> [<ffffffff8f9425d1>] dump_stack+0x45/0x56 [103563.958080] [<ffffffff8f08cf28>] warn_slowpath_common+0x7a/0x93 [103563.958083] [<ffffffff8f81a708>] ? sk_stream_kill_queues+0x105/0x10a [103563.958086] [<ffffffff8f08cfe5>] warn_slowpath_null+0x15/0x17 [103563.958088] [<ffffffff8f81a708>] sk_stream_kill_queues+0x105/0x10a [103563.958091] [<ffffffff8f8937d6>] inet_csk_destroy_sock+0x77/0xb7 [103563.958093] [<ffffffff8f894ef3>] tcp_done+0x74/0x79 [103563.958096] [<ffffffff8f8a2f83>] tcp_out_of_resources+0x83/0x9e [103563.958098] [<ffffffff8f8a355f>] tcp_retransmit_timer+0x3dc/0x62b [103563.958100] [<ffffffff8f8a3920>] ? tcp_write_timer_handler+0x172/0x172 [103563.958103] [<ffffffff8f8a380f>] tcp_write_timer_handler+0x61/0x172 [103563.958105] [<ffffffff8f8a393f>] tcp_write_timer+0x1f/0x47 [103563.958108] [<ffffffff8f096036>] call_timer_fn+0x33/0xe8 [103563.958110] [<ffffffff8f8a3920>] ? tcp_write_timer_handler+0x172/0x172 [103563.958113] [<ffffffff8f096789>] run_timer_softirq+0x1d7/0x232 [103563.958115] [<ffffffff8f0908f7>] __do_softirq+0xde/0x249 [103563.958118] [<ffffffff8f090bee>] irq_exit+0x40/0x4e [103563.958120] [<ffffffff8f071e0a>] smp_apic_timer_interrupt+0x3f/0x4b [103563.958123] [<ffffffff8f94d05d>] apic_timer_interrupt+0x6d/0x80 [103563.958124] <EOI> [103563.958126] ---[ end trace d426860d4ef06a04 ]--- [103563.958129] ------------[ cut here ]------------ [103563.958133] WARNING: CPU: 1 PID: 4344 at net/ipv4/af_inet.c:153 inet_sock_destruct+0x166/0x18f() [103563.958134] Modules linked in: cfg80211 bluetooth ftdi_sio uvcvideo snd_usb_audio videobuf2_vmalloc videobuf2_memops videobuf2_core snd_usbmidi_lib snd_rawmidi k10temp [103563.958140] CPU: 1 PID: 4344 Comm: rtorrent Tainted: G W 3.16.7 #2 [103563.958141] Hardware name: ASUS F2A85-M/F2A85-M, BIOS 4.0-7017-g9310df8-dirty 11/07/2014 [103563.958142] 0000000000000000 ffff88041ec83de0 ffffffff8f9425d1 0000000000000000 [103563.958145] ffff88041ec83e18 ffffffff8f08cf28 ffffffff8f8b5285 ffff88037bd29740 [103563.958147] ffff88037bd298a0 ffffffff8f8a3920 ffff88037bd29ab0 ffff88041ec83e28 [103563.958149] Call Trace: [103563.958150] <IRQ> [<ffffffff8f9425d1>] dump_stack+0x45/0x56 [103563.958154] [<ffffffff8f08cf28>] warn_slowpath_common+0x7a/0x93 [103563.958156] [<ffffffff8f8b5285>] ? inet_sock_destruct+0x166/0x18f [103563.958159] [<ffffffff8f8a3920>] ? tcp_write_timer_handler+0x172/0x172 [103563.958161] [<ffffffff8f08cfe5>] warn_slowpath_null+0x15/0x17 [103563.958163] [<ffffffff8f8b5285>] inet_sock_destruct+0x166/0x18f [103563.958166] [<ffffffff8f811f6a>] __sk_free+0x18/0xc4 [103563.958167] [<ffffffff8f812029>] sk_free+0x13/0x15 [103563.958170] [<ffffffff8f8a3964>] tcp_write_timer+0x44/0x47 [103563.958172] [<ffffffff8f096036>] call_timer_fn+0x33/0xe8 [103563.958174] [<ffffffff8f8a3920>] ? tcp_write_timer_handler+0x172/0x172 [103563.958176] [<ffffffff8f096789>] run_timer_softirq+0x1d7/0x232 [103563.958178] [<ffffffff8f0908f7>] __do_softirq+0xde/0x249 [103563.958181] [<ffffffff8f090bee>] irq_exit+0x40/0x4e [103563.958183] [<ffffffff8f071e0a>] smp_apic_timer_interrupt+0x3f/0x4b [103563.958185] [<ffffffff8f94d05d>] apic_timer_interrupt+0x6d/0x80 [103563.958186] <EOI> [103563.958187] ---[ end trace d426860d4ef06a05 ]--- [103572.215239] TCP: out of memory -- consider tuning tcp_mem [103748.857703] TCP: out of memory -- consider tuning tcp_mem [104128.915899] TCP: out of memory -- consider tuning tcp_mem [104253.625287] TCP: out of memory -- consider tuning tcp_mem [104404.204370] TCP: out of memory -- consider tuning tcp_mem [104434.216580] TCP: out of memory -- consider tuning tcp_mem [104706.030954] TCP: out of memory -- consider tuning tcp_mem [104774.384330] TCP: out of memory -- consider tuning tcp_mem [105101.914632] TCP: out of memory -- consider tuning tcp_mem [105124.662653] TCP: out of memory -- consider tuning tcp_mem [105747.735740] TCP: out of memory -- consider tuning tcp_mem [106657.904120] TCP: out of memory -- consider tuning tcp_mem [107678.119730] TCP: out of memory -- consider tuning tcp_mem [108128.395220] TCP: out of memory -- consider tuning tcp_mem [108158.482858] TCP: out of memory -- consider tuning tcp_mem [108334.484559] TCP: out of memory -- consider tuning tcp_mem [108368.595898] TCP: out of memory -- consider tuning tcp_mem [108429.131470] TCP: out of memory -- consider tuning tcp_mem [108934.969639] TCP: out of memory -- consider tuning tcp_mem [109469.914718] TCP: out of memory -- consider tuning tcp_mem [109540.152197] TCP: out of memory -- consider tuning tcp_mem [109590.198229] TCP: out of memory -- consider tuning tcp_mem [109606.471833] TCP: out of memory -- consider tuning tcp_mem [109700.863850] ICMPv6 checksum failed [2a02:2528:ff00:12c::1 > 2a02:2528:ff00:12c::2] [110739.321815] TCP: out of memory -- consider tuning tcp_mem [110865.679772] TCP: out of memory -- consider tuning tcp_mem [110890.490677] TCP: out of memory -- consider tuning tcp_mem [111725.922208] TCP: out of memory -- consider tuning tcp_mem [111797.634877] TCP: out of memory -- consider tuning tcp_mem [112006.017462] TCP: out of memory -- consider tuning tcp_mem [112042.880820] TCP: out of memory -- consider tuning tcp_mem [112542.978489] TCP: out of memory -- consider tuning tcp_mem [113172.857965] TCP: out of memory -- consider tuning tcp_mem [114098.840788] TCP: out of memory -- consider tuning tcp_mem [115005.893066] TCP: out of memory -- consider tuning tcp_mem [115011.627606] TCP: out of memory -- consider tuning tcp_mem [115034.439233] TCP: out of memory -- consider tuning tcp_mem [115547.670715] TCP: out of memory -- consider tuning tcp_mem [115575.140892] TCP: out of memory -- consider tuning tcp_mem [115786.958281] TCP: out of memory -- consider tuning tcp_mem [116142.284608] TCP: out of memory -- consider tuning tcp_mem [116149.503416] TCP: out of memory -- consider tuning tcp_mem [116505.233453] TCP: out of memory -- consider tuning tcp_mem [117973.485971] TCP: out of memory -- consider tuning tcp_mem [117977.375754] TCP: out of memory -- consider tuning tcp_mem [118041.670282] TCP: out of memory -- consider tuning tcp_mem [118344.094692] TCP: out of memory -- consider tuning tcp_mem [118367.328346] TCP: out of memory -- consider tuning tcp_mem [118615.557292] TCP: out of memory -- consider tuning tcp_mem [119159.614814] TCP: out of memory -- consider tuning tcp_mem [119178.657741] TCP: out of memory -- consider tuning tcp_mem [119254.248690] TCP: out of memory -- consider tuning tcp_mem [119301.505419] TCP: out of memory -- consider tuning tcp_mem [119387.517680] TCP: out of memory -- consider tuning tcp_mem [119480.350785] TCP: out of memory -- consider tuning tcp_mem [119508.869151] TCP: out of memory -- consider tuning tcp_mem [119744.805817] TCP: out of memory -- consider tuning tcp_mem [150725.030873] TCP: out of memory -- consider tuning tcp_mem [150818.327643] TCP: out of memory -- consider tuning tcp_mem [150977.984235] TCP: out of memory -- consider tuning tcp_mem [150984.310043] TCP: out of memory -- consider tuning tcp_mem [150999.433985] TCP: out of memory -- consider tuning tcp_mem [151025.057510] TCP: out of memory -- consider tuning tcp_mem [151162.106973] TCP: out of memory -- consider tuning tcp_mem [151294.573022] TCP: out of memory -- consider tuning tcp_mem [151347.593586] TCP: out of memory -- consider tuning tcp_mem [151348.554469] TCP: out of memory -- consider tuning tcp_mem [151351.437114] TCP: out of memory -- consider tuning tcp_mem [151352.077702] TCP: out of memory -- consider tuning tcp_mem [151353.038584] TCP: out of memory -- consider tuning tcp_mem [151359.444471] TCP: out of memory -- consider tuning tcp_mem [151369.815706] TCP: out of memory -- consider tuning tcp_mem
I'm pretty confident the bug is in the bridge interface at this point. Since my last post I removed the bridge and am using the ethernet interface directly and haven't hit the bug since. Current uptime is almost 5 days. With the bridge in place, I would hit this bug by day 3-4 if not sooner every time. Also could be specific to the Realtek r8169 driver since it looks like everyone here is using that driver.
I saw this bug a couple of days after upgrading from ubuntu 14.04 to 14.10. It certainly wasn't present in 14.04. kernel version is now 3.16.0-28-generic dmesg output about the ethernet driver [ 0.560374] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded [ 0.560809] r8169 0000:02:00.0: can't disable ASPM; OS doesn't have ASPM control [ 0.561285] ahci 0000:00:1f.2: version 3.0 [ 0.561474] r8169 0000:02:00.0: irq 46 for MSI/MSI-X [ 0.561595] ahci 0000:00:1f.2: irq 47 for MSI/MSI-X [ 0.561624] ahci 0000:00:1f.2: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x3 impl SATA mode [ 0.561785] r8169 0000:02:00.0 eth0: RTL8168evl/8111evl at 0xffffc900018c0000, 74:d4:35:a9:d3:ee, XID 0c900800 IRQ 46 [ 0.561786] r8169 0000:02:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko] [ 0.562976] ahci 0000:00:1f.2: flags: 64bit ncq pm led clo pio slum part ems apst I am also using bridged networking for my VMs. syslog output when it went wrong (I wasn't using the computer at the time) :- Dec 20 08:36:07 deneb kernel: [163193.879796] TCP: out of memory -- consider tuning tcp_mem Dec 20 08:36:07 deneb kernel: [163194.282481] ------------[ cut here ]------------ Dec 20 08:36:07 deneb kernel: [163194.282498] WARNING: CPU: 3 PID: 0 at /build/buildd/linux-3.16.0/net/core/stream.c:201 sk_stream_kill_queues+0x151/0x160() Dec 20 08:36:07 deneb kernel: [163194.282501] Modules linked in: pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) bridge stp llc binfmt_misc snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic uas usb_storage hid_generic ppdev usbhid snd_hda_intel snd_hda_controller snd_hda_codec snd_soc_rt5640 snd_hwdep snd_soc_rl6231 snd_soc_core snd_compress snd_pcm_dmaengine snd_seq_midi snd_pcm snd_seq_midi_event snd_rawmidi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_seq aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd serio_raw snd_seq_device snd_timer i915 parport_pc dw_dmac dw_dmac_core snd_soc_sst_acpi video 8250_dw drm_kms_helper snd i2c_hid hid drm tpm_infineon i2c_algo_bit i2c_designware_platform soundcore i2c_designware_core mei_me acpi_pad spi_pxa2xx_platform mei shpchp mac_hid lp parport nls_iso8859_1 psmouse ahci r8169 libahci mii sdhci_acpi sdhci Dec 20 08:36:07 deneb kernel: [163194.282611] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OE 3.16.0-28-generic #38-Ubuntu Dec 20 08:36:07 deneb kernel: [163194.282614] Hardware name: Gigabyte Technology Co., Ltd. H97-HD3/H97-HD3, BIOS F1 04/02/2014 Dec 20 08:36:07 deneb kernel: [163194.282618] 0000000000000009 ffff88041fb83d38 ffffffff82781eaa 0000000000000000 Dec 20 08:36:07 deneb kernel: [163194.282624] ffff88041fb83d70 ffffffff8206fedd ffff88009922f700 ffff88009922f878 Dec 20 08:36:07 deneb kernel: [163194.282629] ffff88009922f820 0000000000000000 ffff88009922f700 ffff88041fb83d80 Dec 20 08:36:07 deneb kernel: [163194.282635] Call Trace: Dec 20 08:36:07 deneb kernel: [163194.282638] <IRQ> [<ffffffff82781eaa>] dump_stack+0x45/0x56 Dec 20 08:36:07 deneb kernel: [163194.282661] [<ffffffff8206fedd>] warn_slowpath_common+0x7d/0xa0 Dec 20 08:36:07 deneb kernel: [163194.282668] [<ffffffff8206ffba>] warn_slowpath_null+0x1a/0x20 Dec 20 08:36:07 deneb kernel: [163194.282674] [<ffffffff82676c31>] sk_stream_kill_queues+0x151/0x160 Dec 20 08:36:07 deneb kernel: [163194.282682] [<ffffffff826c7c06>] inet_csk_destroy_sock+0x56/0x170 Dec 20 08:36:07 deneb kernel: [163194.282688] [<ffffffff826c8fe7>] tcp_done+0x67/0xa0 Dec 20 08:36:07 deneb kernel: [163194.282697] [<ffffffff826dba41>] tcp_out_of_resources+0x91/0xb0 Dec 20 08:36:07 deneb kernel: [163194.282704] [<ffffffff826dc1d7>] tcp_retransmit_timer+0x4d7/0x760 Dec 20 08:36:07 deneb kernel: [163194.282712] [<ffffffff826dc620>] ? tcp_write_timer_handler+0x1c0/0x1c0 Dec 20 08:36:07 deneb kernel: [163194.282719] [<ffffffff826dc500>] tcp_write_timer_handler+0xa0/0x1c0 Dec 20 08:36:07 deneb kernel: [163194.282726] [<ffffffff826dc69c>] tcp_write_timer+0x7c/0x80 Dec 20 08:36:07 deneb kernel: [163194.282734] [<ffffffff8207cb36>] call_timer_fn+0x36/0x110 Dec 20 08:36:07 deneb kernel: [163194.282742] [<ffffffff826dc620>] ? tcp_write_timer_handler+0x1c0/0x1c0 Dec 20 08:36:07 deneb kernel: [163194.282749] [<ffffffff8207e730>] run_timer_softirq+0x260/0x370 Dec 20 08:36:07 deneb kernel: [163194.282755] [<ffffffff82075744>] __do_softirq+0x124/0x2e0 Dec 20 08:36:07 deneb kernel: [163194.282761] [<ffffffff82075afd>] irq_exit+0xfd/0x110 Dec 20 08:36:07 deneb kernel: [163194.282768] [<ffffffff8278cd94>] smp_apic_timer_interrupt+0x44/0x50 Dec 20 08:36:07 deneb kernel: [163194.282775] [<ffffffff8278ae7d>] apic_timer_interrupt+0x6d/0x80 Dec 20 08:36:07 deneb kernel: [163194.282778] <EOI> [<ffffffff8261af0c>] ? cpuidle_enter_state+0x4c/0xc0 Dec 20 08:36:07 deneb kernel: [163194.282795] [<ffffffff8261b067>] cpuidle_enter+0x17/0x20 Dec 20 08:36:07 deneb kernel: [163194.282805] [<ffffffff820b9cf7>] cpu_startup_entry+0x347/0x480 Dec 20 08:36:07 deneb kernel: [163194.282812] [<ffffffff82045b20>] start_secondary+0x230/0x2c0 Dec 20 08:36:07 deneb kernel: [163194.282816] ---[ end trace 1373316db32d3146 ]--- Dec 20 08:36:07 deneb kernel: [163194.282821] ------------[ cut here ]------------ Dec 20 08:36:07 deneb kernel: [163194.282828] WARNING: CPU: 3 PID: 0 at /build/buildd/linux-3.16.0/net/ipv4/af_inet.c:153 inet_sock_destruct+0x181/0x1f0() Dec 20 08:36:07 deneb kernel: [163194.282830] Modules linked in: pci_stub vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) bridge stp llc binfmt_misc snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic uas usb_storage hid_generic ppdev usbhid snd_hda_intel snd_hda_controller snd_hda_codec snd_soc_rt5640 snd_hwdep snd_soc_rl6231 snd_soc_core snd_compress snd_pcm_dmaengine snd_seq_midi snd_pcm snd_seq_midi_event snd_rawmidi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_seq aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd serio_raw snd_seq_device snd_timer i915 parport_pc dw_dmac dw_dmac_core snd_soc_sst_acpi video 8250_dw drm_kms_helper snd i2c_hid hid drm tpm_infineon i2c_algo_bit i2c_designware_platform soundcore i2c_designware_core mei_me acpi_pad spi_pxa2xx_platform mei shpchp mac_hid lp parport nls_iso8859_1 psmouse ahci r8169 libahci mii sdhci_acpi sdhci Dec 20 08:36:07 deneb kernel: [163194.282913] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G W OE 3.16.0-28-generic #38-Ubuntu Dec 20 08:36:07 deneb kernel: [163194.282916] Hardware name: Gigabyte Technology Co., Ltd. H97-HD3/H97-HD3, BIOS F1 04/02/2014 Dec 20 08:36:07 deneb kernel: [163194.282919] 0000000000000009 ffff88041fb83da8 ffffffff82781eaa 0000000000000000 Dec 20 08:36:07 deneb kernel: [163194.282924] ffff88041fb83de0 ffffffff8206fedd ffff88009922f700 ffff88009922f878 Dec 20 08:36:07 deneb kernel: [163194.282929] ffff88009922f76c ffffffff826dc620 ffff88009922f700 ffff88041fb83df0 Dec 20 08:36:07 deneb kernel: [163194.282934] Call Trace: Dec 20 08:36:07 deneb kernel: [163194.282936] <IRQ> [<ffffffff82781eaa>] dump_stack+0x45/0x56 Dec 20 08:36:07 deneb kernel: [163194.282949] [<ffffffff8206fedd>] warn_slowpath_common+0x7d/0xa0 Dec 20 08:36:07 deneb kernel: [163194.282957] [<ffffffff826dc620>] ? tcp_write_timer_handler+0x1c0/0x1c0 Dec 20 08:36:07 deneb kernel: [163194.282963] [<ffffffff8206ffba>] warn_slowpath_null+0x1a/0x20 Dec 20 08:36:07 deneb kernel: [163194.282969] [<ffffffff826f5b81>] inet_sock_destruct+0x181/0x1f0 Dec 20 08:36:07 deneb kernel: [163194.282976] [<ffffffff8266a8cf>] __sk_free+0x1f/0x150 Dec 20 08:36:07 deneb kernel: [163194.282983] [<ffffffff8266aa19>] sk_free+0x19/0x20 Dec 20 08:36:07 deneb kernel: [163194.282990] [<ffffffff826dc688>] tcp_write_timer+0x68/0x80 Dec 20 08:36:07 deneb kernel: [163194.282997] [<ffffffff8207cb36>] call_timer_fn+0x36/0x110 Dec 20 08:36:07 deneb kernel: [163194.283004] [<ffffffff826dc620>] ? tcp_write_timer_handler+0x1c0/0x1c0 Dec 20 08:36:07 deneb kernel: [163194.283011] [<ffffffff8207e730>] run_timer_softirq+0x260/0x370 Dec 20 08:36:07 deneb kernel: [163194.283016] [<ffffffff82075744>] __do_softirq+0x124/0x2e0 Dec 20 08:36:07 deneb kernel: [163194.283022] [<ffffffff82075afd>] irq_exit+0xfd/0x110 Dec 20 08:36:07 deneb kernel: [163194.283027] [<ffffffff8278cd94>] smp_apic_timer_interrupt+0x44/0x50 Dec 20 08:36:07 deneb kernel: [163194.283034] [<ffffffff8278ae7d>] apic_timer_interrupt+0x6d/0x80 Dec 20 08:36:07 deneb kernel: [163194.283036] <EOI> [<ffffffff8261af0c>] ? cpuidle_enter_state+0x4c/0xc0 Dec 20 08:36:07 deneb kernel: [163194.283050] [<ffffffff8261b067>] cpuidle_enter+0x17/0x20 Dec 20 08:36:07 deneb kernel: [163194.283056] [<ffffffff820b9cf7>] cpu_startup_entry+0x347/0x480 Dec 20 08:36:07 deneb kernel: [163194.283062] [<ffffffff82045b20>] start_secondary+0x230/0x2c0 Dec 20 08:36:07 deneb kernel: [163194.283065] ---[ end trace 1373316db32d3147 ]--- Output of commands :- $ sysctl net.ipv4.tcp_mem net.ipv4.tcp_mem = 380193 506924 760386 $ cat /proc/net/sockstat sockets: used 434 TCP: inuse 56 orphan 4 tw 10 alloc 59 mem 760360 UDP: inuse 5 mem 3 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 I had to reboot to get things back to normal.
I note that this bug is marked as "Regression: No" in the bug info at the top. This is definitely wrong. Further, I note that the original reporter had kernel version 3.14.3, whereas I found that kernel 3.13.0.24 (Ubuntu server 14.04) did not have the bug. So that is a fairly narrow range in which the regression occurred.
FWIW, I also have a card which uses the r8169 driver: 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 02) and I'm also using it via a bridge interface and I'm seeing these "TCP: out of memory -- consider tuning tcp_mem" messages. I'm running a 3.17.4 upstream kernel.
Recently we experienced an interesting production problem. This application was running on multiple AWS EC2 instances behind Elastic Load Balancer. The application was running on GNU/Linux OS, Java 8, Tomcat 8 application server. All of sudden one of the application instances became unresponsive. All other application instances were handling the traffic properly. Whenever the HTTP request was sent to this application instance from the browser, we were getting following response to be printed on the browser. Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /. Reason: Error reading from remote server Let us see how we resolved this issue by assigning values for these properties in the server: net.core.netdev_max_backlog=30000 net.core.rmem_max=134217728 net.core.wmem_max=134217728 net.ipv4.tcp_max_syn_backlog=8192 net.ipv4.tcp_rmem=4096 87380 67108864 net.ipv4.tcp_wmem=4096 87380 67108864 https://blog.gceasy.io/2019/08/28/tcp-out-of-memory-consider-tuning-tcp_mem/