Bug 99471
Summary: | System locks with kswapd0 and kworker taking full IO and mem | ||
---|---|---|---|
Product: | Memory Management | Reporter: | Gonzalo Aguilar (gaguilar) |
Component: | Page Allocator | Assignee: | Andrew Morton (akpm) |
Status: | NEW --- | ||
Severity: | blocking | CC: | 32170745, howaboutsynergy, larsnostdal, meisterplanlos, serianox, sgh, spam, viktorpal, Wilhelm.Buchmueller |
Priority: | P1 | ||
Hardware: | x86-64 | ||
OS: | Linux | ||
Kernel Version: | 4.0.4-303.fc22.x86_64 | Subsystem: | |
Regression: | No | Bisected commit-id: | |
Attachments: |
This is situation while blocked
Kernel log with tasks an memory 1 week historic load |
Description
Gonzalo Aguilar
2015-06-05 17:07:36 UTC
It's very strange that linux completly hangs when on memmory pressure. That seems to happen with java 8 but I don't understand why. I set swappines to 1 and then 0. Nothing changes. I removed swap. Nothing changes. The problem with this bug is that the system takes 100% of system usage. Blocks IO by using hd to full. Even without swap!!! How can this be? Kernel is saying ok, I'm without memory, so I will use hard disk (without looking for memory) intesively until I decide to kill the program is taking too much RAM. And selects Java. Why? The normal behaviour should be: I have no ram. What's best candidate to be killed. Java. Kill. Finish. Normally I have to shutdown the computer because is not responsive. And this cannot be. I have 2G ram that's not much, but it used to work with Ubuntu 14.10 kernels. I attach an screenshot of the computer when completely locked. Without swap. Situation is awful: 4 kworkers. Taking 75% of the system. 1 kswap worker. Taking the rest and the IO wait. More than the 24% it shows. Current kernel: Linux azul-laptop.level2crm.com 4.0.5-300.fc22.x86_64 #1 SMP Mon Jun 8 16:15:26 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux Created attachment 182541 [details]
This is situation while blocked
The system is so unresponsive that I have to shudown hard to recover it. Normally kernel should kill offending program before going unresponsive.
Created attachment 182571 [details]
Kernel log with tasks an memory
I added everything I thought it was relevant
I changed to scheduler since I don't think is related to the filesystem but the scheduler and memory management. I have the same problem, although it has nothing to do with the amount of memory used. I'm willing to do every possible attempt to track this down. I have tried upgrading the kenrel to 4.1.3, but the same thing happens. https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1462919 <quoote> In 15.04 even when having plenty of RAM and swap kswpd uses lots of CPU and might bring the machine down to a crawl making is unable to access it over the network. I say "might", because it's only a very clear feeling that I have. I do not know it for sure. Anyway the fact is that I have two server that once every 2 week ends up totally unaccessible from the network. They can be pinged though. Logging in locally takes a long time. Rebooting them locally takes a long time. They have absolutely no noticeable disk IO when in this stalled state. </quote> I have cpuload plotted on this site. http://sgh.dk/~sgh/graphs.html Notice the system time just climbing after every reboot. It happens with 4.2.0 too. (switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface). On Tue, 01 Sep 2015 12:32:10 +0000 bugzilla-daemon@bugzilla.kernel.org wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=99471 Guys, could you take a look please? The machine went oom when there's heaps of unused swap and most memory is being used on active_anon and inactive_anon. We should have just swapped that stuff out and kept going. On 09/10/15 14:04, Andrew Morton wrote: > (switched to email. Please respond via emailed reply-to-all, not via the > bugzilla web interface). > > On Tue, 01 Sep 2015 12:32:10 +0000 bugzilla-daemon@bugzilla.kernel.org wrote: > >> https://bugzilla.kernel.org/show_bug.cgi?id=99471 > Guys, could you take a look please? > > The machine went oom when there's heaps of unused swap and most memory > is being used on active_anon and inactive_anon. We should have just > swapped that stuff out and kept going. Isn't there already logic in the kernel that disables OOM if there's swap space available? I saw it once, what happened to it? > > > -- > To unsubscribe, send a message with 'unsubscribe linux-mm' in > the body to majordomo@kvack.org. For more info on Linux MM, > see: http://www.linux-mm.org/ . > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> Created attachment 187511 [details]
1 week historic load
It is clear to see the system load "singularity".
The system has 8 physical cores and 12 GB of memory. The sigularity happended on a sunday no activity on the system.
On Thu, Sep 10, 2015 at 02:04:18PM -0700, Andrew Morton wrote:
> (switched to email. Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
>
> On Tue, 01 Sep 2015 12:32:10 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
>
> > https://bugzilla.kernel.org/show_bug.cgi?id=99471
>
> Guys, could you take a look please?
>
> The machine went oom when there's heaps of unused swap and most memory
> is being used on active_anon and inactive_anon. We should have just
> swapped that stuff out and kept going.
I think we need to re-evaluate the way we balance file and anon scan
pressure. It's not just the "not swapping" aspect that bugs me, it's
also the fact that the machine has been thrashing page cache at full
load for *minutes* before signalling the OOM.
SSDs can flush and reload pages quick enough that on memory pressure
there are always reclaimable cache pages and the scanner never goes
after anonymous memory. If anonymous memory does not leave enough room
for page cache to hold the libraries and executables, userspace goes
into a state where it's mostly waiting for cache to become uptodate.
It's a very frustrating problem because it's hard to even detect.
One idea I had to address the LRU balance problem in the past was to
always reclaim the pages in the following order: inactive file, active
file, anon*. As one set becomes empty, go after the next one. If the
workingset code detects cache thrashing, it depends on the refault
distances what to do: if they are smaller than the active file size,
deactivate; if they are bigger than that, but smaller than active file
+ anon, we need to start swapping to alleviate the cache thrashing.
Now, if the refault distances are bigger than active file + anon, no
amount of deactivating and swapping are going to stop the thrashing
and we have to think about triggering OOM. But OOM is drastic and the
refaults might happen at a very slow pace (or, with sparse files, not
require any IO at all) and the system might be completely fine. So in
addition this would require a measure of overall time spent on
thrashing IO, comparable to what Tejun proposed in "[RFD] memory
pressure and sizing problem", where we say if thrashing IO takes up X
percent of all execution time spent, we trigger the OOM killer--not to
free memory, but to reduce the tasks that contribute to the thrashing
and let the remaining tasks make progress, similar to the swap token
or a BSD style memory scheduler.
* we can ignore the difference between inactive and active anon here
as anon is not aged the same way as the file LRU is aged
FYI. It's still a problem in 4.3-rc1 I experience the same on a desktop computer (Ubuntu 15.04) maybe once a week. When this happens the whole machine gets unresponsive immediately and only restarting helps. It only turns out from the atop logs what happened after the machine was restarted. Atop is configured to run every 30 secs. It is visible from the atop logs that kswapd0 is eating up all the CPU and IO resources. Strange thing is that this happens despite there is no swap enabled. Kernel version: Linux 3.19.0-30-generic #34-Ubuntu SMP Fri Oct 2 22:08:41 UTC 2015 x86_64 GNU/Linux If you need any additional info please let me know. I would really appreciate if this problem could be fixed as it makes the OS very unreliable. If there is any way (a workaround) to avoid this from happening please share it. For what it's worth it's not as such a problem in ubuntu 15.10. I do not know if the patched the hell out of that kernel but it's in any case not destructive anymore. There are however a strange thing about my cpu load graph. http://sgh.dk/~sgh/system-cpu-ripples.png As you can see the ripples that where there in period with high cpu load is still there when it's low. The interesting thing is that the ripples are now smaller. (In reply to Søren Holm from comment #15) > For what it's worth it's not as such a problem in ubuntu 15.10. For me this is definitely a problem in 15.10 (4.2.0-17-generic #21-Ubuntu SMP x86_64). The moment the system has some memory pressure (but not much needed; even at 2 GB free of my 4 GB in total) kswapd0 will start using 100% CPU sometimes locking the system down permanently and I have to pull the plug or remove the laptop battery. ..pretty darn lame. Still present in 4.3.0 (Linux thinkpad 4.3.0 #2 SMP PREEMPT Mon Nov 9 00:11:27 CET 2015 x86_64 GNU/Linux) and 4.4.0-rc4. still present On Mon, 5 Oct 2015 22:03:46 +0200 Michal Hocko <mhocko@kernel.org> wrote: > On Tue 15-09-15 10:39:19, Johannes Weiner wrote: > > On Thu, Sep 10, 2015 at 02:04:18PM -0700, Andrew Morton wrote: > > > (switched to email. Please respond via emailed reply-to-all, not via the > > > bugzilla web interface). > > > > > > On Tue, 01 Sep 2015 12:32:10 +0000 bugzilla-daemon@bugzilla.kernel.org > wrote: > > > > > > > https://bugzilla.kernel.org/show_bug.cgi?id=99471 > > > > > > Guys, could you take a look please? So this isn't fixed and a number of new reporters (cc'ed) are chiming in (let's please keep this going via email, not via the bugzilla UI!). We have various theories but I don't think we've nailed it down yet. Are any of the reporters able to come up with a set of instructions which will permit the developers to reproduce this bug locally? Can we think up a way of adding some form of debug/instrumentation to the kernel which will permit us to diagnose and fix this? It could be something which a tester manually adds or it could be something permanent, perhaps controlled via a procfs knob. Hi, I'm just a regular user but i'd like to point out some relevant info on the topic. fix proposed that partialy solved the problem http://lkml.iu.edu//hypermail/linux/kernel/1601.2/03564.html also: https://bugzilla.kernel.org/show_bug.cgi?id=65201 in this github issue you can see a small program made to easily reproduce this issue https://github.com/GalliumOS/galliumos-distro/issues/52#issuecomment-174261443 Happens with no swap, no mount, only reading from SSD: -> https://bugzilla.kernel.org/show_bug.cgi?id=65201#c50 I have set "swapoff -a" to disable the swap function when this issue is happen. |