Bug 32442 - Higher Priority CPU-bound Processes Cause Filesystem Lockup
Summary: Higher Priority CPU-bound Processes Cause Filesystem Lockup
Status: RESOLVED OBSOLETE
Alias: None
Product: File System
Classification: Unclassified
Component: ext3 (show other bugs)
Hardware: All Linux
: P1 high
Assignee: fs_ext3@kernel-bugs.osdl.org
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-04-01 16:06 UTC by Dan Moldenhauer
Modified: 2012-08-20 15:26 UTC (History)
2 users (show)

See Also:
Kernel Version: 2.6.37.3
Subsystem:
Regression: No
Bisected commit-id:


Attachments
test program (tar gz) (2.44 KB, application/octet-stream)
2011-04-01 16:06 UTC, Dan Moldenhauer
Details

Description Dan Moldenhauer 2011-04-01 16:06:04 UTC
Created attachment 52972 [details]
test program (tar gz)

There seems to be a problem somewhere between scheduling and filesystems where a CPU-bound compute process (no file I/O) with higher-than-normal priority (such as SCHED_RR 1 (-2)) that is running on cpuX will cause a default priority process that is running on cpuY to block indefinitely on file access (such as write(fd,...)).  Not only does this process block on file access, but every process in the system that tries to access any file on the same volume will also be blocked indefinitely until the higher-than-normal-priority process is either killed or is temporarily put to sleep. Access to files on other volumes remains unaffected (works fine while the volume in question is "locked up").

It seems that if the higher-than-normal-priority process is changed to run at default priority, the system does not lock up. The odd part about this all is that this higher-than-normal-priority process doesn't do any file I/O at all, yet changing its priority can cause a lockup of the entire filesystem. My guess is that there is some filesystem-specific work that has to be done in the background, and that it's attempting to do so at normal priority, and never getting a chance to since the user process would preempt it.

We first noticed the behavior with production software, and I have been able to create a test utility that reproduces the lockup. The source for the test program is attached (we've ran it on recent Intel and AMD-based servers with 8 to 48 cores - with similar lockups on all).

The test utility fork()'s into two processes, one process runs at minimum SCHED_FIFO priority and just does "math", while the other process runs at default priority and attempts file write()'s. Each process is bound to a unique CPU.

Problem was first noticed on 2.6.29.6, has happened on 2.6.33.7, and is very reproducible on 2.6.37.3 (vanilla).

(the attached test is nice enough to sleep every 10 seconds (for 2 seconds) to give you a chance to kill it)

I can provide additional information if needed.

Note You need to log in before you can comment on or make changes to this bug.