Bug 12033

Latest working kernel version: 2.6.28-rc1
Earliest failing kernel version: 2.6.28-rc2
Distribution: Debian
Hardware Environment: 4 1TB disks behind Sil3726 PMP connected to on Sil3132
Software Environment: 64bit kernel + 32bit userspace, random debugging enabled in kernel
Problem Description:

It took some time until conversion of SATA from home grown queing to block level one got stable (as bug 11898 was just fixed), but unfortunately although things are now stable, there is quite big performance drop - writes to disks behind PMP are now only 50-70% of speed before Jens's conversion.

When I revert Jen's fixes, I get writes ~48MBps to each of 4 disks - 192MBps total bandwidth (after sata_sil24 change to PCIe during 2.6.28-rc it was actually 52MBps/disk).  All 4 disks are being written to concurrently, and test completes on all 4 disks almost simultaneously.  Now with default values each disk gets completely different bandwidth, and when I watch LEDs on disks, I see that for most of the time I/O goes to only one of disks, and which disk is being used switches every ~2 seconds.  Only way how to get at least some part of bandwidth is to allow only 8 queued commands on each disk - they they mostly fit to 31 commands on channel, and starvation code is almost never triggered.

Test just starts 4 concurrent 'dd' to ext3 filesystems on each of 4 disks, writting 4GB of data to each one.

Default setting:

gwy:~# ./x.sh
4000+0 records in
4000+0 records out
4194304000 bytes (4.2 GB) copied, 110.417 s, 38.0 MB/s
gwy:~# 4000+0 records in
4000+0 records out
4194304000 bytes (4.2 GB) copied, 155.827 s, 26.9 MB/s
4000+0 records in
4000+0 records out
4194304000 bytes (4.2 GB) copied, 206.971 s, 20.3 MB/s
4000+0 records in
4000+0 records out
4194304000 bytes (4.2 GB) copied, 206.301 s, 20.3 MB/s

Only 8 requests per drive; there are 4 drives sharing one tag map with 31
entries:

gwy:~# for a in /sys/block/*/queue/nr_requests; do echo 8 > $a; done
gwy:~# ./x.sh
4000+0 records in
4000+0 records out
4194304000 bytes (4.2 GB) copied, 103.588 s, 40.5 MB/s
gwy:~# 4000+0 records in
4000+0 records out
4194304000 bytes (4.2 GB) copied, 110.86 s, 37.8 MB/s
4000+0 records in
4000+0 records out
4194304000 bytes (4.2 GB) copied, 105.978 s, 39.6 MB/s
4000+0 records in
4000+0 records out
4194304000 bytes (4.2 GB) copied, 107.94 s, 38.9 MB/s