Bug 196729 - System becomes unresponsive when swapping - Regression since 4.10.x
Summary: System becomes unresponsive when swapping - Regression since 4.10.x
Status: NEW
Alias: None
Product: Memory Management
Classification: Unclassified
Component: Page Allocator (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Andrew Morton
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-08-22 11:17 UTC by Steven Haigh
Modified: 2019-03-31 15:03 UTC (History)
13 users (show)

See Also:
Kernel Version: 4.11.x / 4.12.x
Tree: Mainline
Regression: Yes


Attachments
vmstat-4.10.17-10Gb-noswap.log (OK - OOM running) (15.89 KB, text/plain)
2017-08-22 11:19 UTC, Steven Haigh
Details
vmstat-4.10.17-10Gb.log (OK with swapping) (40.96 KB, text/plain)
2017-08-22 11:21 UTC, Steven Haigh
Details
vmstat-20Gb.log (OK - all in RAM) (25.65 KB, text/plain)
2017-08-22 11:24 UTC, Steven Haigh
Details
vmstat-10Gb.log (NOT OK - System Unresponsive) (54.69 KB, text/plain)
2017-08-22 11:25 UTC, Steven Haigh
Details
read_vmstat.c (4.91 KB, text/x-csrc)
2017-08-23 13:54 UTC, Michal Hocko
Details
8Gb-noswap.tar.gz (18.65 KB, application/x-compressed-tar)
2017-08-23 14:41 UTC, Steven Haigh
Details
8Gb-swap-on-file.tar.gz (42.12 KB, application/x-compressed-tar)
2017-08-23 14:41 UTC, Steven Haigh
Details
8Gb-swap-on-ssd.tar.gz (96.20 KB, application/x-compressed-tar)
2017-08-23 14:41 UTC, Steven Haigh
Details
signature.asc (833 bytes, application/pgp-signature)
2017-08-23 14:41 UTC, Steven Haigh
Details
signature.asc (833 bytes, application/pgp-signature)
2017-08-24 14:20 UTC, Steven Haigh
Details

Description Steven Haigh 2017-08-22 11:17:08 UTC
I have 10Gb of RAM in this system and run Fedora 26. If I launch Cities: 
Skylines with no swap space, things run well performance wise until I get an 
OOM - and it all dies - which is expected.

When I turn on swap to /dev/sda2 which resides on an SSD, I get complete 
system freezes while swap is being accessed.

The first swap was after loading a saved game, then launching kmail in the 
background. This caused ~500Mb to be swapped to /dev/sda2 on an SSD. The 
system froze for about 8 minutes - barely being able to move the mouse. The 
HDD LED was on constantly during the entire time.

To hopefully rule out the above glibc issue, I started the game via jemalloc - 
but experienced even more severe freezes while swapping. I gave up waiting 
after 13 minutes of non-responsiveness - not even being able to move the mouse 
properly.

During these hangs, I could typed into a Konsole window, and some of the 
typing took 3+ minutes to display on the screen (yay for buffers?).

I have tested this with both the default vm.swappiness values, as well as the 
following:
vm.swappiness = 1
vm.min_free_kbytes = 32768
vm.vfs_cache_pressure = 60

I noticed that when I do eventually get screen updates, all 8 cpus (4 cores / 
2 threads) show 100% CPU usage - and kswapd is right up there in the process 
list for CPU usage. Sadly I haven't been able to capture this information 
fully yet due to said unresponsiveness.

(more to come in comments & attachments)
Comment 1 Steven Haigh 2017-08-22 11:18:57 UTC
First - using kernel 4.10.17 - which does not show any issues in swapping:

I tried doing: swapoff /dev/sda2

Attached output as vmstat-4.10.17-10Gb-noswap.log

18:27:00 - Launched Cities: Skylines
18:27:30 - Started loading the saved game
18:28:25 - About this time the game started doing its thing. Started scrolling 
around.
18:28:47 - System stopped responding and then the C:S was killed by the OOM 
handler
Comment 2 Steven Haigh 2017-08-22 11:19:41 UTC
Created attachment 258045 [details]
vmstat-4.10.17-10Gb-noswap.log (OK - OOM running)
Comment 3 Steven Haigh 2017-08-22 11:21:32 UTC
Created attachment 258047 [details]
vmstat-4.10.17-10Gb.log (OK with swapping)

Second test, same kernel with swap turned on:

I have attached the vmstat output that goes with the following timestamps for 
system utilisation:

15:32:30 - Launch Skylines
15:33:00 - Load the saved game
15:34:11 - Saved game loaded ok.
15:35:00 - Launch Chrome.
15:35:36 - Chrome launched - System responding ok.
15:36:00 - Browsing a few web sites
15:36:50 - Exit Chrome
15:37:30 - Exit Cities: Skylines.

You'll note that there are very few missing vmstat lines - however I did 
notice the following missing:
        15:35:10
        15:35:12
        15:35:15
        15:35:20
        15:35:26
        15:35:29
        15:35:30

Attachment is vmstat-4.10.17-10Gb.log
Comment 4 Steven Haigh 2017-08-22 11:24:02 UTC
Created attachment 258049 [details]
vmstat-20Gb.log (OK - all in RAM)

Now using kernel 4.11.x (same happens with 4.12.x) - and testing with 20Gb of RAM in the system - meaning no swapping.

Attached as: vmstat-20Gb.log

Timestamps of events:
21:57 - launch the game from within Steam.
21:58:00 - Load the saved game.
21:58:48 - Saved game is loaded and I'm scrolling around in the map.
22:00:00 - Hit the quit to desktop button.
22:00:31 - Am back to desktop with all RAM free again.
Comment 5 Steven Haigh 2017-08-22 11:25:59 UTC
Created attachment 258051 [details]
vmstat-10Gb.log (NOT OK - System Unresponsive)

I now drop back to 10Gb of RAM to test the swapping under 4.11.x kernel.

Log attached as vmstat-10Gb.log

Timestamps:
22:10:00 - Launched of the game from within Steam
22:11:00 - Load the same saved game from the previous log
22:12:01 - Saved game is loaded and I can scroll around. Noted a slight
pause when swapd went to 256 - but otherwise all is well.
22:13:00 - Launched Google Chrome browser to make the system swap.

After this point, the whole system went to hell. You'll note many missing
vmstat entries up until around 22:22 when I managed to exit from the game
back to desktop via the normal means (and not getting annoyed and doing a
pkill from tty2).

As such, the system went nuts for ~9 minutes until I was able to exit the
game to stop things going nuts.

I note that with 20Gb RAM - as the system never touches swap, I can still
play the game, browse the web with the Chrome browser, read / write
email, and even watch a DVB-T broadcast in VLC without having any more
than a minor pause in the game for less than a second.
Comment 6 Steven Haigh 2017-08-22 11:29:50 UTC
So overall, this seems to indicate a regression between kernel 4.10.x (I'm pretty sure I tested all ok with 4.10.15?) and the newer 4.11 and 4.12 builds.

I made contact with Rik van Riel and Ying Huang (which I will attempt to add to this as a CC for comment?) - they don't believe it is a swapping issue - however Rik seems to believe that:

> > There is ZERO swap space in use.
> >
> > In other words, it is not actually swapping,
> > but thrashing through the page cache.

You may want to email the people who worked on page cache
replacement stuff recently, and the linux-mm mailing list
as well.
Comment 7 Andrew Morton 2017-08-22 22:56:22 UTC
(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Tue, 22 Aug 2017 11:17:08 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=196729
> 
>             Bug ID: 196729
>            Summary: System becomes unresponsive when swapping - Regression
>                     since 4.10.x
>            Product: Memory Management
>            Version: 2.5
>     Kernel Version: 4.11.x / 4.12.x
>           Hardware: All
>                 OS: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Page Allocator
>           Assignee: akpm@linux-foundation.org
>           Reporter: netwiz@crc.id.au
>         Regression: No

So it's "Regression: yes".  More info at the bugzilla link.

> I have 10Gb of RAM in this system and run Fedora 26. If I launch Cities: 
> Skylines with no swap space, things run well performance wise until I get an 
> OOM - and it all dies - which is expected.
> 
> When I turn on swap to /dev/sda2 which resides on an SSD, I get complete 
> system freezes while swap is being accessed.
> 
> The first swap was after loading a saved game, then launching kmail in the 
> background. This caused ~500Mb to be swapped to /dev/sda2 on an SSD. The 
> system froze for about 8 minutes - barely being able to move the mouse. The 
> HDD LED was on constantly during the entire time.
> 
> To hopefully rule out the above glibc issue, I started the game via jemalloc
> - 
> but experienced even more severe freezes while swapping. I gave up waiting 
> after 13 minutes of non-responsiveness - not even being able to move the
> mouse 
> properly.
> 
> During these hangs, I could typed into a Konsole window, and some of the 
> typing took 3+ minutes to display on the screen (yay for buffers?).
> 
> I have tested this with both the default vm.swappiness values, as well as the 
> following:
> vm.swappiness = 1
> vm.min_free_kbytes = 32768
> vm.vfs_cache_pressure = 60
> 
> I noticed that when I do eventually get screen updates, all 8 cpus (4 cores / 
> 2 threads) show 100% CPU usage - and kswapd is right up there in the process 
> list for CPU usage. Sadly I haven't been able to capture this information 
> fully yet due to said unresponsiveness.
> 
> (more to come in comments & attachments)
> 
> -- 
> You are receiving this mail because:
> You are the assignee for the bug.
Comment 8 Michal Hocko 2017-08-23 13:54:38 UTC
Created attachment 258067 [details]
read_vmstat.c

On Tue 22-08-17 15:55:30, Andrew Morton wrote:
> 
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> On Tue, 22 Aug 2017 11:17:08 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
[...]
> Sadly I haven't been able to capture this information 
> > fully yet due to said unresponsiveness.

Please try to collect /proc/vmstat in the bacground and provide the
collected data. Something like

while true
do
	cp /proc/vmstat > vmstat.$(date +%s)
	sleep 1s
done

If the system turns out so busy that it won't be able to fork a process
or write the output (which you will see by checking timestamps of files
and looking for holes) then you can try the attached proggy
./read_vmstat output_file timeout output_size

Note you might need to increase the mlock rlimit to lock everything into
memory.
Comment 9 Steven Haigh 2017-08-23 14:41:38 UTC
Created attachment 258069 [details]
8Gb-noswap.tar.gz

On Wednesday, 23 August 2017 11:38:48 PM AEST Michal Hocko wrote:
> On Tue 22-08-17 15:55:30, Andrew Morton wrote:
> > (switched to email.  Please respond via emailed reply-to-all, not via the
> > bugzilla web interface).
> 
> > On Tue, 22 Aug 2017 11:17:08 +0000 bugzilla-daemon@bugzilla.kernel.org 
wrote:
> [...]
> 
> > Sadly I haven't been able to capture this information
> > 
> > > fully yet due to said unresponsiveness.
> 
> Please try to collect /proc/vmstat in the bacground and provide the
> collected data. Something like
> 
> while true
> do
>       cp /proc/vmstat > vmstat.$(date +%s)
>       sleep 1s
> done
> 
> If the system turns out so busy that it won't be able to fork a process
> or write the output (which you will see by checking timestamps of files
> and looking for holes) then you can try the attached proggy
> ./read_vmstat output_file timeout output_size
> 
> Note you might need to increase the mlock rlimit to lock everything into
> memory.

Thanks Michal,

I have upgraded PCs since I initially put together this data - however I was 
able to get strange behaviour by pulling out an 8Gb RAM stick in my new system 
- leaving it with only 8Gb of RAM.

All these tests are performed with Fedora 26 and kernel 4.12.8-300.fc26.x86_64

I have attached 3 files with output.

8Gb-noswap.tar.gz contains the output of /proc/vmstat running on 8Gb of RAM 
with no swap. Under this scenario, I was expecting the OOM reaper to just kill 
the game when memory allocated became too high for the amount of physical RAM. 
Interestingly, you'll notice a massive hang in the output before the game is 
terminated. I didn't see this before.

8Gb-swap-on-file.tar.gz contains the output of /proc/vmstat still with 8Gb of 
RAM - but creating a file with swap on the PCIe SSD /swapfile with size 8Gb 
via:
	# dd if=/dev/zero of=/swapfile bs=1G count=8
	# mkswap /swapfile
	# swapon /swapfile

Some times (all in UTC+10):
23:58:30 - Start loading the saved game
23:59:38 - Load ok, all running fine
00:00:15 - Load Chrome
00:01:00 - Quit the game

The game seemed to run ok with no real issue - and a lot was swapped to the 
swap file. I'm wondering if it was purely the speed of the PCIe SSD that 
caused this appearance - as the creation of the file with dd completed at 
~1.4GB/sec.

8Gb-swap-on-ssd.tar.gz contains adding a 32Gb SATA based SSD to the system and 
using the entire block device as swap via:
	# mkswap -f /dev/sda
	# swapon /dev/sda

There are many pauses and unresponsiveness issues while this was loading - 
however we eventually got there.

Some timings (all in UTC+10 again):
00:06:33 - Load the saved game
00:11:22 - Saved game loaded - somewhat responsive
00:12:00 - Load Chrome
00:13:07 - Quit the game + chrome

For the sake of information, the following is a speed test on the SSD in 
question:
# dd if=/dev/zero of=/dev/sda bs=1M count=8192 conv=fsync
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 44.923 s, 191 MB/s
# dd if=/dev/sda of=/dev/null bs=1M count=8192 conv=fsync
dd: fsync failed for '/dev/null': Invalid argument
8192+0 records in
8192+0 records out
8589934592 bytes (8.6 GB, 8.0 GiB) copied, 30.7414 s, 279 MB/s

Running the game on the exact same system with 16Gb of RAM and no swap works 
perfectly - even with multitasking - as we never end up filling physical RAM.

As there is some data missing though, should I still attempt to compile + run 
the program provided? I'm not quite clear on the mlock rlimit mention - I 
haven't really had to debug anything like this before.
Comment 10 Steven Haigh 2017-08-23 14:41:39 UTC
Created attachment 258071 [details]
8Gb-swap-on-file.tar.gz
Comment 11 Steven Haigh 2017-08-23 14:41:39 UTC
Created attachment 258073 [details]
8Gb-swap-on-ssd.tar.gz
Comment 12 Steven Haigh 2017-08-23 14:41:39 UTC
Created attachment 258075 [details]
signature.asc
Comment 13 Michal Hocko 2017-08-24 12:41:44 UTC
On Thu 24-08-17 00:30:40, Steven Haigh wrote:
> On Wednesday, 23 August 2017 11:38:48 PM AEST Michal Hocko wrote:
> > On Tue 22-08-17 15:55:30, Andrew Morton wrote:
> > > (switched to email.  Please respond via emailed reply-to-all, not via the
> > > bugzilla web interface).
> > 
> > > On Tue, 22 Aug 2017 11:17:08 +0000 bugzilla-daemon@bugzilla.kernel.org 
> wrote:
> > [...]
> > 
> > > Sadly I haven't been able to capture this information
> > > 
> > > > fully yet due to said unresponsiveness.
> > 
> > Please try to collect /proc/vmstat in the bacground and provide the
> > collected data. Something like
> > 
> > while true
> > do
> >     cp /proc/vmstat > vmstat.$(date +%s)
> >     sleep 1s
> > done
> > 
> > If the system turns out so busy that it won't be able to fork a process
> > or write the output (which you will see by checking timestamps of files
> > and looking for holes) then you can try the attached proggy
> > ./read_vmstat output_file timeout output_size
> > 
> > Note you might need to increase the mlock rlimit to lock everything into
> > memory.
> 
> Thanks Michal,
> 
> I have upgraded PCs since I initially put together this data - however I was 
> able to get strange behaviour by pulling out an 8Gb RAM stick in my new
> system 
> - leaving it with only 8Gb of RAM.
> 
> All these tests are performed with Fedora 26 and kernel
> 4.12.8-300.fc26.x86_64
> 
> I have attached 3 files with output.
> 
> 8Gb-noswap.tar.gz contains the output of /proc/vmstat running on 8Gb of RAM 
> with no swap. Under this scenario, I was expecting the OOM reaper to just
> kill 
> the game when memory allocated became too high for the amount of physical
> RAM. 
> Interestingly, you'll notice a massive hang in the output before the game is 
> terminated. I didn't see this before.

I have checked few gaps. E.g. vmstat.1503496391 vmstat.1503496451 which
is one minute. The most notable thing is that there are only very few
pagecache pages
			[base]		[diff]
nr_active_file  	1641    	3345
nr_inactive_file        1630    	4787

So there is not much to reclaim without swap. The more important thing
is that we keep reclaiming and refaulting that memory

workingset_activate     5905591 	1616391
workingset_refault      33412538        10302135
pgactivate      	42279686        13219593
pgdeactivate    	48175757        14833350

pgscan_kswapd   	379431778       126407849
pgsteal_kswapd  	49751559        13322930

so we are effectivelly trashing over the very small amount of
reclaimable memory. This is something that we cannot detect right now.
It is even questionable whether the OOM killer would be an appropriate
action. Your system has recovered and then it is always hard to decide
whether a disruptive action is more appropriate. One minute of
unresponsiveness is certainly annoying though. Your system is obviously
under provisioned to load you want to run obviously.

It is quite interesting to see that we do not really have too many
direct reclaimers during this time period
allocstall_normal       30      	1
allocstall_movable      490     	88
pgscan_direct_throttle  0       	0
pgsteal_direct  	24434   	4069
pgscan_direct   	38678   	5868
 
> 8Gb-swap-on-file.tar.gz contains the output of /proc/vmstat still with 8Gb of 
> RAM - but creating a file with swap on the PCIe SSD /swapfile with size 8Gb 
> via:
>       # dd if=/dev/zero of=/swapfile bs=1G count=8
>       # mkswap /swapfile
>       # swapon /swapfile
> 
> Some times (all in UTC+10):
> 23:58:30 - Start loading the saved game
> 23:59:38 - Load ok, all running fine
> 00:00:15 - Load Chrome
> 00:01:00 - Quit the game
> 
> The game seemed to run ok with no real issue - and a lot was swapped to the 
> swap file. I'm wondering if it was purely the speed of the PCIe SSD that 
> caused this appearance - as the creation of the file with dd completed at 
> ~1.4GB/sec.

Swap IO tends to be really scattered and the IO performance is not really
great even on a fast storage AFAIK.
 
Anyway your original report sounded like a regression. Were you able to
run the _same_ workload on an older kernel without these issues?
Comment 14 Steven Haigh 2017-08-24 14:20:08 UTC
Created attachment 258079 [details]
signature.asc

On Thursday, 24 August 2017 10:41:39 PM AEST Michal Hocko wrote:
> On Thu 24-08-17 00:30:40, Steven Haigh wrote:
> > On Wednesday, 23 August 2017 11:38:48 PM AEST Michal Hocko wrote:
> > > On Tue 22-08-17 15:55:30, Andrew Morton wrote:
> > > > (switched to email.  Please respond via emailed reply-to-all, not via
> > > > the
> > > > bugzilla web interface).
> > > > 
> > > > On Tue, 22 Aug 2017 11:17:08 +0000 bugzilla-daemon@bugzilla.kernel.org
> > 
> > wrote:
> > > [...]
> > > 
> > > > Sadly I haven't been able to capture this information
> > > > 
> > > > > fully yet due to said unresponsiveness.
> > > 
> > > Please try to collect /proc/vmstat in the bacground and provide the
> > > collected data. Something like
> > > 
> > > while true
> > > do
> > > 
> > >   cp /proc/vmstat > vmstat.$(date +%s)
> > >   sleep 1s
> > > 
> > > done
> > > 
> > > If the system turns out so busy that it won't be able to fork a process
> > > or write the output (which you will see by checking timestamps of files
> > > and looking for holes) then you can try the attached proggy
> > > ./read_vmstat output_file timeout output_size
> > > 
> > > Note you might need to increase the mlock rlimit to lock everything into
> > > memory.
> > 
> > Thanks Michal,
> > 
> > I have upgraded PCs since I initially put together this data - however I
> > was able to get strange behaviour by pulling out an 8Gb RAM stick in my
> > new system - leaving it with only 8Gb of RAM.
> > 
> > All these tests are performed with Fedora 26 and kernel
> > 4.12.8-300.fc26.x86_64
> > 
> > I have attached 3 files with output.
> > 
> > 8Gb-noswap.tar.gz contains the output of /proc/vmstat running on 8Gb of
> > RAM
> > with no swap. Under this scenario, I was expecting the OOM reaper to just
> > kill the game when memory allocated became too high for the amount of
> > physical RAM. Interestingly, you'll notice a massive hang in the output
> > before the game is terminated. I didn't see this before.
> 
> I have checked few gaps. E.g. vmstat.1503496391 vmstat.1503496451 which
> is one minute. The most notable thing is that there are only very few
> pagecache pages
>                       [base]          [diff]
> nr_active_file        1641            3345
> nr_inactive_file        1630          4787
> 
> So there is not much to reclaim without swap. The more important thing
> is that we keep reclaiming and refaulting that memory
> 
> workingset_activate     5905591       1616391
> workingset_refault      33412538        10302135
> pgactivate            42279686        13219593
> pgdeactivate          48175757        14833350
> 
> pgscan_kswapd         379431778       126407849
> pgsteal_kswapd        49751559        13322930
> 
> so we are effectivelly trashing over the very small amount of
> reclaimable memory. This is something that we cannot detect right now.
> It is even questionable whether the OOM killer would be an appropriate
> action. Your system has recovered and then it is always hard to decide
> whether a disruptive action is more appropriate. One minute of
> unresponsiveness is certainly annoying though. Your system is obviously
> under provisioned to load you want to run obviously.
> 
> It is quite interesting to see that we do not really have too many
> direct reclaimers during this time period
> allocstall_normal       30            1
> allocstall_movable      490           88
> pgscan_direct_throttle  0             0
> pgsteal_direct        24434           4069
> pgscan_direct         38678           5868

Yes, I understand that the system is really not suitable - however I believe 
the test is useful - even from an informational point of view :)

> > 8Gb-swap-on-file.tar.gz contains the output of /proc/vmstat still with 8Gb
> > of RAM - but creating a file with swap on the PCIe SSD /swapfile with
> > size 8Gb> 
> > via:
> >     # dd if=/dev/zero of=/swapfile bs=1G count=8
> >     # mkswap /swapfile
> >     # swapon /swapfile
> > 
> > Some times (all in UTC+10):
> > 23:58:30 - Start loading the saved game
> > 23:59:38 - Load ok, all running fine
> > 00:00:15 - Load Chrome
> > 00:01:00 - Quit the game
> > 
> > The game seemed to run ok with no real issue - and a lot was swapped to
> > the
> > swap file. I'm wondering if it was purely the speed of the PCIe SSD that
> > caused this appearance - as the creation of the file with dd completed at
> > ~1.4GB/sec.
> 
> Swap IO tends to be really scattered and the IO performance is not really
> great even on a fast storage AFAIK.
> 
> Anyway your original report sounded like a regression. Were you able to
> run the _same_ workload on an older kernel without these issues?

When I try the same tests with swap on an SSD under kernel 4.10.x (I believe 
the latest I tried was 4.10.25?) - then swap using the SSD did not cause any 
issues or periods of system unresponsiveness.

The file attached in the original bug report "vmstat-4.10.17-10Gb.log" was 
taken on my old system with 10Gb of RAM - and there were no significant pauses 
while swapping.

I do find it interesting that the newer '8Gb-swap-on-file.tar.gz' does not 
show any issues. I wonder if it would be helpful to attempt the same using a 
file on the SSD that was a swap disk in the '8Gb-swap-on-ssd.tar.gz' so we 
have a constant device - but with a file on the SSD instead of the entire 
block device. That would at least expose any issues on the same device in file 
vs block mode? Or maybe even if there's a difference just having the file on a 
much (much!) faster drive?
Comment 15 Huang Ying 2017-08-28 07:14:14 UTC
Compared with

a) vmstat-4.10.17-10Gb.log (OK with swapping) and
b) vmstat-10Gb.log (NOT OK - System Unresponsive)

The si/so is low in both files.  And si/so in a) is higher than that of b), so the problem may be we swap less than before?

The bi is kept high in b).  I guess we encountered thrashing for file pages.
Comment 16 Steven Haigh 2017-11-27 02:37:31 UTC
To give this a bit of a nudge, I've been seeing reports of others having similar issues. See:
https://www.reddit.com/r/Fedora/comments/7f0dht/system_freezes_for_45min_in_lowmemory_conditions/

Also lodged on the RH BZ a while ago:
https://bugzilla.redhat.com/show_bug.cgi?id=1472336
Comment 17 Daniel 2018-04-02 01:47:19 UTC
Hi,

I’ve experienced what I believe is the same problem. The problem has gone away completely for me after I bumped vm.min_free_kbytes way up to 393216.

As soon as the system ran out of physical memory, the system would freeze for at least 2 minutes and often up to 45 minutes. GNOME desktop would stop. I could move the mouse cursor, and ping the system from a remote computer; but not connect over SSH or do anything other than wave the mouse about. The system clock on the top of GNOME would stop updating for 45 minutes. (Maaaybe it would move forward 1 minute after 20 minutes and still be 19 minutes out of sync.)

I've been having this issue for years on multiple different computer configurations with 8+ GiB of memory and large SWAP partitions. I never saw more than maybe 5 MiB in use on the SWAP partition. After tuning min_free_kbytes, the SWAP partition is now being used properly and the system only does the occasional (and expected) 1 second stutter when running low on physical memory.

I also run Fedora and have kept up with the latest stable release.

Aside 1: The issue would persist with SWAPOFF, just like Steven Haigh describes.
Aside 2: The problem happen much more frequently when I used BtrFS. After switching to XFS, this happen less frequently (weekly instead of daily).
Comment 18 lou 2018-05-07 00:49:26 UTC
Please refer also to this bug report. It is the same problem and has existed for eleven (11!) years if one can believe that.

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/159356

I personally experience this on two 4GB laptops running live versions (no swap) of Debian 8.6 - 9.4, Fedora 25 - 28, Ubuntu, with a myriad of shells; Gnome, Mate, Cinnamon, KDE, etc. (One laptop I expanded to 8GB now but it doesn't matter- it just take a little longer to freeze the system.)

Various browsers from Firefox52 to current Developers 60, Chrome and Chromium

Certain combos eat up memory faster (Gnome has a memory leak for example in which it consumers memory for every window drawn and NEVER relinquishes that memory, without a restart to gnome-shell), new Firefox or Chrom* vs older ESR versions (54 and under) of FF eat up memory much more quickly.

Under the best combo/circumstances, I can open up 25-30 FF tabs before the system SUDDENLY SEIZES (observe the "USB live stick" light flashing non-stop, as if swapping, even tho no swap on Live versions). If not caught within literal seconds to Ctrl-Alt-F5 to an opened root console where I can kill the FF ps and save this "live" session, the computer is entirely unresponsive and requires power cycling.

In rare instances, some 10's of minutes later or even hours (4,8 12) later, the system *might* finally respond to the request to drop to the console. Keystrokes to issue the kill command can take minutes per key, but if successful, I've seen the load reported after the kill as high as 75.

Truly amazing.


It's difficult to fathom this critical a bug in memory management has gone un-addressed/un-noticed for so long but alas. I can't recall but I've read this behavior ONLY occurs on 64-bit kernels, and is un-reproducible on 32-bit kernels.

Also, on non-"live" installs, with swap configured, one can watch the hard drive light come on and remain solid to the same effect. Power cycle time. I've read from others, that they've determined swap isn't even really being used, so not sure what the "read" thrashing going on is (and it must be read thrashing because on Live versions there's no swap and the USB drive light is steady active also).

I just run Linux to not run Windows. Basic browsing, text document editing, file management, a few cli's and an instant messaging program typically opened simultaneously. Nothing computationally heavy but memory intensive (at least for the web browser) for sure.

STILL- difficult to believe the OS cannot handle this situation with some sort of message, or killing a window/throwing an error about an opened Firefox tab or something-- rather, it simply fills up the memory (I watch on gnome-system-status/Resources tab now) to 99% and then it's too late.

I really don't know technically how the Out Of Memory killer works/is supposed to work, but it sure isn't doing anything here.
Comment 19 SlayerProof32 2018-05-18 01:41:55 UTC
I experience this too. Ive tested using Kernel version Kernel 4.17 rc8, Kernel 4.16.8, Kernel 4.12.8, and Kernel 4.14 across Manjaro Linux, Ubuntu Linux, Opensuse leap 15 and Fedora 28. 

Steps to trigger:
-Open firefox with many tabs, or any other high memory usage program
-Wait a second
-System freezes. Sometimes the only fix is a hard reboot

Other findings:
-I notice really high cpu load averages if the system unfreezes
-If the system is not frozen, it is highly unresponsive on high memory usage when swapping
-Hard drive indicator light stays solidly on when system is frozen (excessive hard disk use)
-The reason the system freezes is because it is swapping 

Tested on a 
Intel i5 520m with 4gb ram/ 4gb swap (Lenovo t410)
Intel E6400 with 3gb ram/ 3gb swap 

This bug is really hard to deal with because it usually requires a hard restart. Please fix ASAP if possible
Comment 20 SlayerProof32 2018-05-18 01:44:50 UTC
My fedora report here : https://bugzilla.redhat.com/show_bug.cgi?id=1577528
Comment 21 SlayerProof32 2018-05-18 01:51:18 UTC
If someone gives me some debugging instructions, ill gladly follow them
Comment 22 SlayerProof32 2018-05-19 01:36:26 UTC
https://askubuntu.com/questions/41778/computer-freezing-on-almost-full-ram-possibly-disk-cache-problem

https://unix.stackexchange.com/questions/28175/system-hanging-when-it-runs-out-of-memory

https://bbs.archlinux.org/viewtopic.php?id=231087

Over 5 (Non bug reporting) websites report this bug, with recent posts up to 2018. This memory issue is FATAL to people who rely on linux for gaming/working. Please don't let this issue go unnoticed for 11 more years. It is critical to system function that memory management is good. The issue actually doesn't only occur when swapping. Swapping however greatly aggravates this bug
Comment 23 lou 2018-05-19 20:32:29 UTC
@SlayerProof32 #19
> -The reason the system freezes is because it is swapping 

I don't think this is the case at least not in the traditional sense. if you read my comment above yours, I'm running Live versions of Linux where no swap is configured.

I suppose it could be swapping portions of code into and out of regular memory, but there's no hard disk writes. On my USB sticks, I note the light comes on steady when the system freezes. It's reading (something).

I also noted the bug doesn't exhibit in 32-bit systems (I'd read previously somewhere).

Eleven years.

Shocking.
Comment 24 SlayerProof32 2018-05-19 21:45:20 UTC
@lou #20

I have confirmed that swapping is not the issue and you are correct. 
On my linux system, I tried swapoff --a and to no surprise, it crashed on high memory usage still, and there was still excessive drive usage. Thanks for the insight
Comment 25 SlayerProof32 2018-05-19 21:45:55 UTC
https://bugzilla.kernel.org/show_bug.cgi?id=199763
I filed another report.
Comment 26 Ivan Kalvachev 2018-06-18 17:01:18 UTC
I'm just a user, haunted by similar issues.

1. If you have zswap compiled and enabled, try to disable it.
ZSwap uses some RAM in order to compress pages and hide them there. e.g. reserves 500MB with the idea it could put 1000MB swap in there.


2. I think that there is something very broken in "Transparent HugePage Support" or THP, especially at kernel-4.15 and later. A bug in 4.16 memory compaction led to very noticeable swap pressure. The issue has been fixed, but disabling THP also was able to workaround it.
If you have major issues, try to build a kernel with that option disabled and see if the problem remains as severe.
Comment 27 SlayerProof32 2018-06-19 16:25:32 UTC
What happens for me at least with basically any system I use, is that Linux runs out of memory first, then SLAMS my swap disk with writes, which I believe overloads my system’s I/O susbsystem, and caused a complete system freeze. I have swapiness set to 60.
Comment 28 lou 2018-09-23 08:21:34 UTC
Update:

I'd commented in detail about this bug (comment 18.23 above). I run the live versions of Linux on a 4GB Core-i5 laptop (and another 4GB pentium laptop also.)

Just wanted to add:

I've added 4Gb of RAM to the Core-i5 laptop for 8Gb total now. This obviously helps a lot. Now some notes for 8GB RAM instances.

With Fedora 28, the system will still cease up with maybe 2 dozen (or less depending on what's happening (video, etc) ) FF tabs opened/active.

I came back here to note that, I'm currently using a Live Debian Stretch (9.5).

There are obviously significant differences in the way these variants of Linux manage memory.


Why?

Because under the same system conditions (Gnome, same s/w programs installed and/or running), I can open WAY more tabs in FF on Debian; open more simultaneous programs, without fear of a sudden system heart-attack.

In fact, it is much harder for me to cause the system freeze in Debian, even with approaching 50 tabs opened in FF developer 63...

I understand there are underlying Fedora vs Debian system differences like: systemd vs init, and Wayland vs Xorg, Gnome versions (3.28.1 vs 3.22.5) and kernel revisions (4.16.3-301.fc28.x86_64 vs 4.9.110-1 (2018-07-05) ), but in all, I find Debian WAYYYY more forgiving, and more manageable, ESPECIALLY in light of this FATAL flaw, AND the known Gnome memory leak bug which can easily be remediated for in Debian by restarting Gnome (via Alt-F2, r) to free back up that memory. (The only way to accomplish this in Fedora is to actually log out of your session because of Wayland limitations.)

Anyway, I just thought it's another data point to add to the mystery.

I still have to keep resource monitor opened even in Stretch, just in case, but I only crashed Stretch once over the past 3 months or so when I was in the 80's (mem % used) and let a video play for 2 hrs without checking up.

Normally anyway, that percentage isn't rising above the 70's in my typical "working" environment.

Finally, I'd like to mention to those asking for logs, etc., for this issue, realize that WHEN this issue occurs, it *is* essentially a heart-attack for the system. There is no recourse, and no way to gather logs. EVERYTHING ceases up- usually never to come back. A hard power-cycle is the only recourse, and NO logs  which would shed light on the issue are written. EVERYTHING stops- including log writing.

This is the reality.

I *do* have a few logs from and old (non-live) Jesse 8.7.1 install-- for a few times when, the system did revive, after hours-- and there's nothing in there that would shed light on the issue. The few entries in the log that I've researched pointed to no other instances/causes of this same issue.

It would be nice after 11 or 12 years of this issue, if someone higher up and more knowledgeable in the development "food chain" would would simply replicate the issue, it's not really that hard to do so at all.

It honestly is a show-stopper.

Ciao.
Comment 29 SlayerProof32 2018-12-21 04:49:16 UTC
These kind of reports are all over the web. Please someone who knows how memory management works, fix this behavior
Comment 30 ValdikSS 2018-12-25 10:01:23 UTC
I can confirm this regression. My system does not freeze with 4.9.140 and is overall totally usable on high memory usage and swap (right now 7.5 out of 8 GB RAM is used and 2 GB is swapped), but it becomes unresponsive with 4.19.10 when memory consumption is close to my RAM limit of 8 GB.

My system is:
Lenovo Thinkpad X220 laptop, Intel Core i7-2640M CPU (Sandy Bridge), Intel HD 3000 GPU, 8GB RAM, 8GB SWAP, SSD, UEFI mode.
Fedora 29, kernels 4.9.140 (self-built), 4.19.10-300.fc29.x86_64 from repository.
I'm using KDE Plasma 5.

You can easily reproduce this issue with the following steps:
1. Add "mem=2G" to the kernel command line. If you use GRUB, press "e" button in the bootloader menu and append "mem=2G" to the "linux" line. Your system would be limited to 2 GB RAM.
2. Boot the system (press F10 in bootloader menu after editing)
3. Run several heavy applications: web browser, word processor, GIMP. Open gmail web interface in browser, it's also heavy.

Expected result:
The system (at least the GUI, including mouse cursor) does not freeze/hang, but properly utilizes swap, what could be seen with 4.9.140 kernel (before regression). System is usable, all applications are still accessible, the music plays without hiccups, you can continue doing your work.

Actual result:
The system (at least the GUI, including mouse cursor) freezes/hangs with 4.19.10 kernel, with a very little chance to recover by itself. Disk activity LED is constantly lid. System is unusable.
Comment 31 SlayerProof32 2018-12-25 17:11:47 UTC
This issue appears to slightly improve in 4.20. Some of the issue is that things like klogin are getting swapped, as well as other parts of the GUI. We need a way to tell the kernel what to swap out first (like Firefox tabs, or open programs). The kernel should also start swapping at say 65% memory usage very slowly, and then increase swapping as memory fills, not slamming swap at 95% usage. We also should make sure the OOM killer is doing its job, instead of waiting for alt SysRq f. As ram frees, swap should be unloaded immediately, slowly like MacOs does.
Comment 32 lou 2019-02-13 21:48:25 UTC
I know @SlayerProof32 posted that this was rectified w kernel >4.17.5 in https://bugzilla.redhat.com/show_bug.cgi?id=1577528 , but the bug still exists. Easily reproducible.

Tested on a 3GB desktop (Core 2 Quad), running Live Ubuntu 18.10 LTS off of a flash drive (pendrivelinux.com). Kernel 4.18.0-10.

FF 63.0. I set the download folder to a dir on the hard drive so as not to deliberately stress free RAM.

With the system showing about 1.5GB free (System Monitor), trying to d/l this 1.25GB ROM image from Mega ( https://mega.nz/#!KUAyRKjJ!3hALO7dkuyFdE41BTWf1OfHaZmdTA-Kzd8q0HYiMbYs ), the d/l gets to 100% but system monitor shows RAM at 97% or 98%, the flash drive lights up and stays lit. System frozen, as I've reported previously with my other laptops.

I've mitigated this on the laptops because they now have 8GB of RAM. BUT-- even then, I can STILL crash those systems using Live Debian (or whatever flavor). It just takes more stressing (more open tabs, bigger d/l's, whatever) to get there, but it does.
Comment 33 ValdikSS 2019-03-10 22:40:29 UTC
How to reproduce (light):

1. Open web browser
2. Run the following command:
stress --vm 1 --vm-hang 0 --vm-bytes "$(awk '/MemAvailable/ {print $2"000"}' /proc/meminfo)"
3. Navigate to https://www.tumblr.com/explore/trending in web browser

Actual result:
The system is very slow, almost unresponsive. HDD LED is constantly lid.




How to reproduce (heavy):

1. Open web browser
2. Run the following command:
stress --vm 1 --vm-hang 0 --vm-bytes "$(awk '/MemAvailable/ {print $2"000"}' /proc/meminfo)"
3. Navigate to https://mail.google.com/ in web browser, while being logged into Gmail.

Actual result:
The system is unresponsive. HDD LED is constantly lid.
Comment 34 frfgr 2019-03-31 15:03:22 UTC

      Please pay special attention to the fact that you may need to know some knowledge and take a proper backup before trying the methods mentioned later.




  I have tested and found that the kernel 4.19.32 may not have this problem,the problem in kernel 5.0.5 is obvious. 
  Under the kernel with this problem, I found a way, if the problem occurs, run the following command immediately  under the appropriate permissions:

        sync   &&    echo  3  >  /proc/sys/vm/drop_caches    &&     sync    &&     echo 3  >  /proc/sys/vm/drop_caches     &&    sync  &&  echo  3  >  /proc/sys/vm/drop_caches
        
        
  This may reset some of the system's operations, which may affect the next few seconds of IO operation, but should alleviate the unresponsive situation.

Note You need to log in before you can comment on or make changes to this bug.