Bug 8947 - [wish] reduce disk usage at point of memory exhaustion
Summary: [wish] reduce disk usage at point of memory exhaustion
Status: REJECTED WILL_NOT_FIX
Alias: None
Product: Memory Management
Classification: Unclassified
Component: Other (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Andrew Morton
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-08-27 11:19 UTC by Maciej Pilichowski
Modified: 2009-03-23 12:01 UTC (History)
2 users (show)

See Also:
Kernel Version: 2.6.18.2
Subsystem:
Regression: ---
Bisected commit-id:


Attachments
messages.bz2 (75.83 KB, application/x-bzip2)
2008-02-12 06:42 UTC, Maciej Pilichowski
Details
dmesg (87.01 KB, text/plain)
2008-02-25 01:03 UTC, Maciej Pilichowski
Details

Description Maciej Pilichowski 2007-08-27 11:19:30 UTC
Distribution: OpenSUSE 10.2
Software Environment: no swap enabled (for reason), 2GB RAM
Problem Description:

The original report can be found here:
https://bugzilla.novell.com/show_bug.cgi?id=192705
however then I was referring to the disk usage as to swapping.

The problem is when hard-computing some stuff when I need every bit from my memory. At the point of memory exhaustion there is really bad, heavy disk usage, the whole system freezes (it could for long) and then the process is killed.

I don't know if it is the smallest possible program to show the problem, but:
https://bugzilla.novell.com/attachment.cgi?id=93743

Compile, run, wait and suffer :-)

I don't know what exactly is going on when the disk is used but I would like to see either a way to block it or smarten things up. I would like also to understand the reason, because so far we can rule out swapping (I don't have one), so does system writes something on disk (what? and where? and why?) or reads something (again, in such situation nothing should be read, after all free memory is decreasing at that point).
Comment 1 Maciej Pilichowski 2007-11-27 11:22:14 UTC
Additional info:
This behaviour is an old one, I mean, I've seen it for many years now.

Recently I've seen googletechtalks about YouTube and there was mentioned race condition related to memory swapping -- could this be an issue for this case too. Of course not for swap, but in general -- flushing back and forth some memory to keep all programs in system running?

Anyway this problem is nasty -- usually to get my system it takes one-two hours. And I can only
a) shutdown the whole system (the hard way)
b) wait...
c) or tried to suspend to ram the whole thing -- it took (suspend alone) about 10 minutes to make it, but after resume the memory-hungry program was killed and the system was operational (thanks god)
Comment 2 Natalie Protasevich 2008-02-10 23:46:03 UTC
Maciej, can you get alt-sysrq-t trace while the system is wedged?
Comment 3 Maciej Pilichowski 2008-02-11 00:56:53 UTC
Natalie, when the system is struggling with the running programs (the critical point) I am unable to do anything -- GUI is completely frozen. The only thing I can think of is to somehow (?) predict how long will it take to critical point, so in advance run something like this:

sleep 30 ; run_trace
Comment 4 Natalie Protasevich 2008-02-11 01:11:08 UTC
Probably so, in fact you can run "echo t > /proc/sysrq-trigger" which does the same thing, continuously, and the last trace will stay in your /var/log/messages. 
Comment 5 Maciej Pilichowski 2008-02-11 01:39:05 UTC
Natalie, ok, so tell me what I should do. 

Run
echo t > /proc/sysrq-trigger
then run programs so I get freeze, kill computer, and then send here log from
/var/log/messages.
?

Is this correct?
Comment 6 Natalie Protasevich 2008-02-11 01:51:52 UTC
I didn't say it right, sorry. The sysrq-t (same as "echo t > /proc/sysrq-trigger") collects one trace of every process into your /var/log/messages. You can try running it once and take a look at it. It will most likely go to both your console screen and the log file (not to the GUI desktop).

You can make a script like this:
while :
do 
echo t > /proc/sysrq-trigger
sleep 10
done &

- and run it in background while running your program as usual.
Comment 7 Maciej Pilichowski 2008-02-12 06:42:50 UTC
Created attachment 14789 [details]
messages.bz2

It is rather short, this time I was lucky and my program was killed just after two minutes (with HDD activity). "Normally" it is not unusual to get freeze for 1, or 2 hours.
Comment 8 Natalie Protasevich 2008-02-23 13:25:57 UTC
You can try setting /proc/sys/vm/overcommit_memory to 1, and see how far your system can get. It seems like your system is very heavily loaded. Can you also provide something like boot trace or dmesg to give an idea about your system components.
Comment 9 Maciej Pilichowski 2008-02-25 01:02:49 UTC
Natalie,

Can you tell me how I do this:
> try setting /proc/sys/vm/overcommit_memory to 1
echo 1 > setting /proc/sys/vm/overcommit_memory
? and is this permanent

and this:
> provide boot trace 
? You had /var/log/boot.msg in mind?

> It seems like your system is very heavily loaded.

Yes, and no. "Normally" (when system starts up) it is I believe network, firewall, KDE, and that's all. Most of the daemons I didn't needed are shutted down.
But _on heavy load_ when I run my computation stuff the system cannot handle this load -- the most typical case for me currently is when I run my program (computation, it needs lots of memory), then I run Windows+VirtualBox and at some point there is too little memory. Instead of immediate kill of one of the programs, system is "swapping" (I have no swap) something, because HDD led keeps flashing all the time.
Comment 10 Maciej Pilichowski 2008-02-25 01:03:24 UTC
Created attachment 14975 [details]
dmesg
Comment 11 Natalie Protasevich 2008-02-25 09:02:26 UTC
You can run without swap, if your system has large enough memory for the load. Your system has 2 GB of RAM and single processor, it might be not enough to run comfortably. When it tries to get more memory and write back dirty pages, everything is done sequentially, including computational work. The overcommit memory will make your system not to "hold back" any memory reserves including those needed for paging and freeing memory itself. You should gather some stats to see how much memory is in use and maybe just add more memory. And multi-processor systems are good to parallelize tasks...
Comment 12 Maciej Pilichowski 2008-02-25 09:43:26 UTC
Natalie, thank you for explaining.

> When it tries to get more memory and write back dirty pages, everything is
> done sequentially

That is exactly opposite from what I would like to see -- no HDD I/O when free memory is at critical amount.

Natalie, could you please confirm that I correctly wrote commands? See #9.
Comment 13 Natalie Protasevich 2008-02-25 16:38:03 UTC
Yes, the command is correct. It will stay until reboot, and you can make it permanent through sysctl if you will have to.
You can do some memory monitoring with top/htop/slabtop or the like to watch dynamics of this.
Comment 14 Maciej Pilichowski 2008-02-26 01:14:50 UTC
I executed:
echo 1 > setting /proc/sys/vm/overcommit_memory

and then I ran some programs to occupy more memory, then I run my computational program (memory hungry) and over the top my test program (which just allocates more and more memory), to make life simpler.

The result: no difference, at the point of memory exhaustion, everything froze (including mouse cursor), huge HDD activity (no program was reading any data from HDD, that is for sure) and after some time, the test program was killed (killing is ok) and I could use computer again.
Comment 15 Natalie Protasevich 2008-02-26 11:16:55 UTC
Maciej, maybe you should explain your objectives here why you wish to use no swap and push kernel to the limits.
Comment 16 Maciej Pilichowski 2008-02-26 12:08:36 UTC
First of all, swap does not change a thing, it is not magic of course, so with or without you will get the point of memory exhaustion. However with swap it is harder to notice it, because you could think "oh, HDD is working it must be swap".

So using swap won't solve this issue, I hope we both agree. 

I didn't know about this issue, however I was anticipating very similar issue. I run programs that needs a lot of memory, so I bought computer with max. what I could put into it (at that time) -- 2GB. Since I have rather slow HDD, 5400 RPMs, and the computations takes a lot of time (not unusual to leave computer for a night to get something computed), there were two possible configurations and problems:

a) use swap -- if there will be not enough real RAM, HDD would boil and program would crawl 

b) don't use swap -- in the same situation program will be killed (in theory, in practice it is not the case --> see: this very report)

ad.a) since I cannot control swap, especially when I am asleep, I don't want my computer get on fire, (b) solution is much better for me. It is solution of "everything or nothing" approach. If I get nothing, I have to improve algorithm (or free some memory, for example running program in runlevel=3)

It would work as I expected, but there is no immediate kill of any program and HDD works hard anyway. With swap things would be just worse (just a guess, but reasonable I think).
Comment 17 Maciej Pilichowski 2008-03-05 06:24:54 UTC
I just get accidentally some new info about such case on Polish newsgroup (Piotr, thank you).

Since code (executable) can be loaded on-demand, and freed on-demand too, maybe kernel when there is not too much memory left frees memory, to load some code, execute it, free it, load another portion (or code from another process),  so in short it struggles with constant exchanging code to fit in memory and execute it.
This would explain HDD usage. "Freezing" of the system is the effect of HDD activity (besides any UI event handling requires more HDD work).

The solution in kernel would be, that if it is required to free some memory (code part) to load some code to execute it -- then stop and kill such process.
Comment 18 Natalie Protasevich 2008-03-05 07:10:54 UTC
Maybe in this case you want to try setting up memory policy by setting proc/sys/vm parameters, namely overcommit memory to 2, and om_kill_allocating_task to non-zero to make sure that process will be killed attempts to grab too much memory. From the Documentation on oom_kill_allocating_task:

If this is set to zero, the OOM killer will scan through the entire
tasklist and select a task based on heuristics to kill.  This normally
selects a rogue memory-hogging task that frees up a large amount of
memory when killed. If this is set to non-zero, the OOM killer simply kills the task that triggered the out-of-memory condition. 

And the overcommit_ratio:

When overcommit_memory is set to 2, the committed address space is not permitted to exceed swap plus this percentage of physical RAM set by overcommit_ratio.
I guess you can try adjusting this ration. 

See Documentation/sysctl/vm.txt.
Comment 19 Maciej Pilichowski 2008-03-06 10:15:25 UTC
Natalie, thanks for your help -- I will read this vm.txt doc, one question though -- where is oom_kill_allocating_task file located? I cannot find it in /proc directory,  there are bunch of oom_score and oom_adj files.
Comment 20 Maciej Pilichowski 2008-09-24 05:04:16 UTC
Ala, why it is invalid/rejected all of the sudden? 
Comment 21 Maciej Pilichowski 2008-09-27 10:01:17 UTC
Well, I guess this report is closed by mistake so reopening -- still waiting for a solution that would fix this problem.
Comment 22 Alan 2009-03-23 11:33:36 UTC
It was closed because you were directed to the documentation explaining how to configure different overcommit modes
Comment 23 Maciej Pilichowski 2009-03-23 12:01:10 UTC
Should the documentation be up to date and explain what to do with the system, instead describing purpose of non-existent files?

Note You need to log in before you can comment on or make changes to this bug.