When multiple RealTime threads are executing on their own, individual CPUs, a call to 'mlock()' from a non-realtime thread on a different CPU will hang. If some of the RealTime threads lower their priority, the call to mlock will complete.

-----------------------
Steps to reproduce:
-----------------------

I have created a test case (mlock_test) which demonstrates the problem - this requires a minimum of 4 CPUs. It also requires running in text mode (ie. runlevel 3) due to the RealTime priority of many threads (CPUs).

1. compile mlock_test using:
    gcc -Wall -Werror -o mlock_test mlock_test.c -lpthread
2. open a console window in text mode (i.e. sudo init 3), then execute mlock_test using:
    sudo ./mlock_test
3. wait for the text "aux_main: starting in ..."; this will count down from 5s
4. wait for the text "about to lock..."

- AT THIS POINT, THE THREAD IS HUNG IN 'mlock()'

after a few more seconds, each of the RealTime threads will decrease their priority, displaying the text "... adjusted scheduling to SCHED_OTHER."
5. Once all the RealTime threads are running at normal priority, the 'mlock()' call will complete, displaying the message "memory locked (0x...)"
6. Several allocation/lock messages will be displayed, after which the <ENTER> key can be pressed to exit the application.

-----------------------
Additional Information:
-----------------------

1. The test case executes properly on kernel version 2.6.18.x (CentOS 5.5, CentOS 5.7 & RHEL 5.4); that is, the call to 'mlock()' does NOT hang.

2. The test case fails identically on:
  - Quad-core Xeon (4 CPUs)
  - Dual Quad-core Xeon (8 CPUs)
  - Dual Quad-core Xeon w/Hyperthreads (16 CPUs)

3. Theory of operation:
  - the 'main()' thread creates an 'aux_main()' thread, affinitizes each to CPU 0, then blocks on a barrier, awaiting the 'aux_main' thread completion
  - the 'aux_main()' thread creates several (1 fewer than the CPU count) RealTime threads and affinitizes them to an individual CPU (starting w/highest CPU # and proceeding down to #1). Each of these threads spins, awaiting an 'exit_flag' to be set (via a memory location)
  - after creating the RealTime threads, the 'aux_main()' thread allocates a large block of memory and attempts to 'mlock()' it (where it hangs). It does this the same # of times as there are RealTime threads.

4. If the code is changed such that the RealTime threads are instead created as normally-scheduled threads, then the call to 'mlock()' DOES NOT HANG. This can be demonstrated by changing line 227 to read:
  "if( 0 )"
This prevents the test threads from being elevated to RealTime threads.

5. The amount of memory allocated/locked is large, but I have seen identical results using a memory size of 4Kb.

6. This problem also manifests itself in kernel v3.4.4 (from kernel.org) as well as on v2.6.32-220 (CentOS 6.2).