I am using Gentoo on a HP DL380 G3. Intel Xeon (P4) 3.06ghz processors, with a HP COMPAQ CCISS ARRAY in RAID 0 mode (hardware, linux just sees one single drive) hdparm on 2.6.27 shows the speed at ~ 130mbs After performing an upgrade of the kernel hdparm on 2.6.29 shows the speed at ~ 30mbs (30-50mbs varies) I looked at the DMESG comparisons, and the only difference is that 2.6.29 mentions: "IRQF_DISABLED is not guaranteed on shared IRQs" Now I have looked for hours to see if it was a configuration issue, but to no avail. Yes the 2.6.27 kernel was self built, and it works fine. The 2.6.29 was built using the .config file from the 2.6.27. I compiled this kernel on two servers, both same spec (well, one has 4GB ram other has 2GB ram), same setup - and both responded as above. On the one, I recompiled the kernel to just use the new version (NO config changes). The other I re-compiled using config changes (removing things not relevant for the machine etc) - both end up having this problem. I have done a "diff" compare on the configs, but there is nothing I can find that would explain this. I know that this "IRQF_DISABLED" warning isn't present in 2.6.27 boots, and that the CISS driver works full speed in that one. I thought it was originally that I changed the IO Scheduling, but as i said, on the one machine - all i did was copy the config file and recompile. Do you know of anything that could be causing this? I checked proc/interrupts on both kernel versions, and IRQ 30 is unshared (IO-APIC-FASTEOI cciss or something like that), i am just unsure what is causing the issue.
Assigned to Mike, marked as a regression.
I'm running some comparisons between 2.6.27.9 and 2.6.29. I see about a 30% drop in buffered disk reads but nothing as drastic as reported here. I don't have the same HW but I would expect about the same behavior if the driver is the culprit. 2.6.27.9 - buffered disk reads 150.88MB/sec 2.6.29 --- buffered disk reads 102.80MB/sec The timed cache reads are within 10% of each other. What you see in dmesg has no effect on the performance. I've elimated the IRQF_SHARED flag in my testing and it made no difference. The reason you didn't see the message in 2.6.27.x is that it does not exist in older kernels. We have changed very little in the main IO path for a long time.
I report same problem with kernel 2.6.29, 2.6.30 and 2.6.31 running at DL380-G5 . Sample dmesg log. Read speed is ~ 20MBps. -= test =- dd if=/dev/cciss/c0d0p3 of=/dev/null bs=1024 count=1000000 1000000+0 records in 1000000+0 records out 1024000000 bytes (1.0 GB) copied, 95.0507 s, 10.8 MB/s -= dmesg =- HP CISS Driver (v 3.6.20) cciss 0000:06:00.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, low) -> IRQ 10 cciss 0000:06:00.0: irq 35 for MSI/MSI-X cciss 0000:06:00.0: irq 36 for MSI/MSI-X cciss 0000:06:00.0: irq 37 for MSI/MSI-X cciss 0000:06:00.0: irq 38 for MSI/MSI-X e1000e: Intel(R) PRO/1000 Network Driver - 1.0.2-k2 e1000e: Copyright (c) 1999-2008 Intel Corporation. e1000e 0000:0e:00.0: PCI INT A -> Link[LNKB] -> GSI 7 (level, low) -> IRQ 7 e1000e 0000:0e:00.0: setting latency timer to 64 e1000e 0000:0e:00.0: irq 39 for MSI/MSI-X IRQ 37/cciss0: IRQF_DISABLED is not guaranteed on shared IRQs cciss0: <0x3230> at PCI 0000:06:00.0 IRQ 37 using DAC blocks= 1757614684 block_size= 512 heads=255, sectors=32, cylinders=215394 blocks= 1757614684 block_size= 512 heads=255, sectors=32, cylinders=215394 cciss/c0d0: p1 p2 < p5 > p3
I can confirm this regression. The problem started between kernel versions 2.6.28 and 2.6.29. Results from hdparm -t : 2.6.27.35: 86.41 MB/sec 2.6.28.10: 87.10 MB/sec 2.6.29.6: 41.51 MB/sec 2.6.31: 30.51 MB/sec Server model: ProLiant DL360 G5 Output from dmesg: HP CISS Driver (v 3.6.20) cciss 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 cciss 0000:06:00.0: irq 57 for MSI/MSI-X cciss 0000:06:00.0: irq 58 for MSI/MSI-X cciss 0000:06:00.0: irq 59 for MSI/MSI-X cciss 0000:06:00.0: irq 60 for MSI/MSI-X IRQ 59/cciss0: IRQF_DISABLED is not guaranteed on shared IRQs cciss0: <0x3230> at PCI 0000:06:00.0 IRQ 59 using DAC blocks= 286677120 block_size= 512 heads=255, sectors=32, cylinders=35132 blocks= 286677120 block_size= 512 heads=255, sectors=32, cylinders=35132 cciss/c0d0: p1 p2 blocks= 286677120 block_size= 512 heads=255, sectors=32, cylinders=35132 blocks= 286677120 block_size= 512 heads=255, sectors=32, cylinders=35132 cciss/c0d1: p1
Hi, I see a similar performance drop off. I have seen it on a DL360G5 (with current firmware CD applied) and a DL380G5 (older firmware). The figures below are on a Dl380G5. A fresh install of Ubuntu 8.04 with a apt-get update;apt-get upgrade;apt-get dist-upgrade done for all current patches. Two 146GB SAS 10k RPM drives - each in a RAID-0 single logical drive created using the P400 BIOS. I copied the stock kernel config options and then compiled various kernels using those options (after doing a make menuconfig to ensure new options get defaults etc). 2.6.31.4 & 2.6.32-rc5 I also configured the BNX2 driver into the kernel to include the firmware and also enabled the SYSFS deprecated option to ensure that the LVM2 stuff worked on bootup. The following was run for each kernel: dd if=/dev/cciss/c0d1 of=/dev/null bs=1024k count=1024 dd if=/dev/cciss/c0d1 of=/dev/null bs=1024k count=2048 In summary 2.6.28.10 was the last good kernel series and 2.6.29.6 showed the performance drop. I have not run through all the 2.6.29 series. I will try get that run tomorrow to see if its common to the entire 2.6.29 kernel. Obviously, a drop from around 90MB/s to 34MB/s is significant. many thanks, Mike The raw results are below: -------------------------- Wed Oct 21 16:40:54 BST 2009 Linux uk-ditnas902 2.6.24-24-server #1 SMP Fri Sep 18 16:47:05 UTC 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 11.83 s, 90.8 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 23.6072 s, 91.0 MB/s -------------------------- -------------------------- Wed Oct 21 16:44:06 BST 2009 Linux uk-ditnas902 2.6.24.7-mjm #1 SMP Wed Oct 21 11:24:15 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 11.8286 s, 90.8 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 23.5383 s, 91.2 MB/s -------------------------- -------------------------- Wed Oct 21 16:47:16 BST 2009 Linux uk-ditnas902 2.6.25.20-mjm #1 SMP Wed Oct 21 11:51:14 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 11.7866 s, 91.1 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 23.5078 s, 91.4 MB/s -------------------------- -------------------------- Wed Oct 21 16:50:26 BST 2009 Linux uk-ditnas902 2.6.26.8-mjm #1 SMP Wed Oct 21 12:18:45 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 11.7988 s, 91.0 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 23.5338 s, 91.3 MB/s -------------------------- -------------------------- Wed Oct 21 16:53:35 BST 2009 Linux uk-ditnas902 2.6.27.37-mjm #1 SMP Wed Oct 21 12:46:28 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 11.8022 s, 91.0 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 23.5142 s, 91.3 MB/s -------------------------- -------------------------- Wed Oct 21 16:56:42 BST 2009 Linux uk-ditnas902 2.6.28.10-mjm #1 SMP Wed Oct 21 13:13:52 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 11.8088 s, 90.9 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 23.5781 s, 91.1 MB/s -------------------------- -------------------------- Wed Oct 21 16:59:49 BST 2009 Linux uk-ditnas902 2.6.29.6-mjm #1 SMP Wed Oct 21 13:42:24 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 31.041 s, 34.6 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 62.0189 s, 34.6 MB/s -------------------------- -------------------------- Wed Oct 21 17:03:54 BST 2009 Linux uk-ditnas902 2.6.30.9-mjm #1 SMP Wed Oct 21 14:11:58 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 31.0455 s, 34.6 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 62.0342 s, 34.6 MB/s -------------------------- -------------------------- Wed Oct 21 17:07:58 BST 2009 Linux uk-ditnas902 2.6.31.4-mjm #1 SMP Wed Oct 21 14:42:28 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 31.0351 s, 34.6 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 62.0157 s, 34.6 MB/s -------------------------- -------------------------- Wed Oct 21 17:12:02 BST 2009 Linux uk-ditnas902 2.6.32-rc5-mjm #1 SMP Wed Oct 21 15:11:32 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 31.0223 s, 34.6 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 62.0179 s, 34.6 MB/s --------------------------
Hi, I have compiled the 2.6.29 kernels and they all show the reduced performance. So the change resulting in the reduced performance must be around the time the 2.6.29 fork took place. -------------------------- Thu Oct 22 09:56:51 BST 2009 Linux uk-ditnas902 2.6.28.10-mjm #1 SMP Wed Oct 21 13:13:52 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 11.8292 s, 90.8 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 23.5529 s, 91.2 MB/s -------------------------- -------------------------- Thu Oct 22 10:00:02 BST 2009 Linux uk-ditnas902 2.6.29-mjm #1 SMP Wed Oct 21 20:05:22 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 31.0513 s, 34.6 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 62.0565 s, 34.6 MB/s -------------------------- -------------------------- Thu Oct 22 10:04:10 BST 2009 Linux uk-ditnas902 2.6.29.1-mjm #1 SMP Wed Oct 21 17:47:16 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 31.0485 s, 34.6 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 62.0786 s, 34.6 MB/s -------------------------- -------------------------- Thu Oct 22 10:08:18 BST 2009 Linux uk-ditnas902 2.6.29.2-mjm #1 SMP Wed Oct 21 18:14:59 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 31.0697 s, 34.6 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 62.0291 s, 34.6 MB/s -------------------------- -------------------------- Thu Oct 22 10:12:26 BST 2009 Linux uk-ditnas902 2.6.29.3-mjm #1 SMP Wed Oct 21 18:42:37 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 31.0689 s, 34.6 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 62.081 s, 34.6 MB/s -------------------------- -------------------------- Thu Oct 22 10:16:35 BST 2009 Linux uk-ditnas902 2.6.29.4-mjm #1 SMP Wed Oct 21 19:10:12 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 31.055 s, 34.6 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 62.0636 s, 34.6 MB/s -------------------------- -------------------------- Thu Oct 22 10:20:43 BST 2009 Linux uk-ditnas902 2.6.29.5-mjm #1 SMP Wed Oct 21 19:37:53 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 31.0474 s, 34.6 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 62.0479 s, 34.6 MB/s -------------------------- -------------------------- Thu Oct 22 10:24:50 BST 2009 Linux uk-ditnas902 2.6.29.6-mjm #1 SMP Wed Oct 21 13:42:24 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 31.0676 s, 34.6 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 62.0199 s, 34.6 MB/s --------------------------
It seems 2.6.29-rc1 also contains the performance regression. So some change in the 2.6.29-rc1 kernel appears to be the culprit. -------------------------- Thu Oct 22 16:41:12 BST 2009 Linux uk-ditnas902 2.6.29-rc1-mjm #2 SMP Thu Oct 22 16:15:30 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 31.0283 s, 34.6 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 62.0603 s, 34.6 MB/s -------------------------- I have done the same tests in a some of VMs we have running (so the cciss Linux driver is not in play) and it seems that the stock 2.6.24-24-virtual Ubuntu 8.04 kernel is slightly slower than a custom compiled 2.6.30.3 (using pretty much the standard kernel options) which would be expected. Unfortunately, I do not have any non-HP hardware available at the moment to test 2.6.28 vs 2.6.29 to conclusively point at the cciss driver. But at the moment 2.6.29-rc1 seems to have introduced the performance decrease this bug is relating to.
Hi, Here are three more data points from a HP Proliant DL360G6. Single quad-core CPU with 6GB ram. This again shows the slower/reduced CCISS performance when going from kernel 2.6.28 to 2.6.29/2.6.31. Again, the test was: dd if=/dev/cciss/c0d1 of=/dev/null bs=1024k count=1024 dd if=/dev/cciss/c0d1 of=/dev/null bs=1024k count=2048 root@uk-ditnas902:~# cat *txt -------------------------- Fri Oct 23 12:19:25 BST 2009 Linux uk-ditnas903 2.6.28.10-mjm #1 SMP Wed Oct 21 13:13:52 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 11.7862 s, 91.1 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 23.5063 s, 91.4 MB/s -------------------------- -------------------------- Thu Jun 25 08:56:40 BST 2009 Linux uk-ditnas903 2.6.29-rc1-mjm #2 SMP Thu Oct 22 16:15:30 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 31.049 s, 34.6 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 62.022 s, 34.6 MB/s -------------------------- -------------------------- Fri Oct 23 12:27:12 BST 2009 Linux uk-ditnas903 2.6.31.4-mjm #1 SMP Wed Oct 21 14:42:28 BST 2009 x86_64 GNU/Linux 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 29.981 s, 35.8 MB/s 2048+0 records in 2048+0 records out 2147483648 bytes (2.1 GB) copied, 59.7466 s, 35.9 MB/s --------------------------
Posted issue at https://bugs.launchpad.net/ubuntu/+source/linux/+bug/337419
After contacting Mike Miller @ hp and getting very useful help and information, I reran some tests and managed to get improved performance. According to Mike, there were some read ahead changes made to the cciss driver. I am not sure which kernel these were made to. - setting the cciss driver read ahead to 64kB using: echo 64 > /sys/block/cciss\!c0d1/queue/read_ahead_kb OR blockdev --setra 128 /dev/cciss/c0d1 made the most significant improvement. This brought my test results back to pre 2.6.29 levels. From 2.6.29-rc1 the default is now 128kB. Prior kernels seemed to use a default of 4096kB. For my test cases, the 64kB read_ahead_kb setting is optimal across logical drive stripe sizes and SmartArray cache ratio. The 64kB read ahead was optimal for my test case, of a logical drive consisting of either one or two 146GB SAS 10k RPM drives, and brought performance back to pre 2.6.29 levels. I have done further testing (results I will try to summarise here shortly) on a variety of RAID levels on a 6i and 6404 controller which indicate a performance regression between 2.6.28.10 and 2.6.29.2. I intend to rerun my test on a non-HP server and see if it is a generic regression or cciss related.
My additional tests confirm there is a performance regression when using the cciss driver for various workloads. The regression is present since 2.6.29-rc1. I have run my test on a Dell 2950 with PERC6/i and the performance regression is not present. I have attached a spreadsheet with the results of my dd test for various read ahead block sizes, various logical drive configurations and various Smart Array cache ratios (read/write cache ratio). The graphs in the spreadsheet clearly indicate the sequential read test performance differences observed for 2.6.28 vs 2.6.29. Anyone using a Smart Array controller should, in my opinion, do thorough testing prior to using a 2.6.29 or more recent kernel for production work loads. At a minimum the latest 2.6.28 kernel's performance should be compare with the more recent kernel being considered.
Created attachment 23569 [details] Performance test result highlighting performance regression between 2.6.28 and 2.6.29 and more recent Spreadsheet with results from performance tests highlighting the performance regression between 2.6.28 and 2.6.29 and more recent. Test runs showing include various logical drive array configurations (Raid1/5/6), array cache ratios and read ahead block sizes.
Created attachment 25614 [details] timing harddisks some more data on (hopefully) this issue. seems as if in 2.6.31 the performance of a RAID1 improved compared to 2.6.28. OTOH, the performance of RAID0 single drives on the controller decreased _a lot_.
Some time ago, I had been tracking down a performance problem with hpsa somewhere between 2.6.29 and 2.6.30, and I git bisected it to this commit: 2f5cb7381b737e24c8046fd4aeab571fb71315f5 is first bad commit commit 2f5cb7381b737e24c8046fd4aeab571fb71315f5 Author: Jens Axboe Date: Tue Apr 7 08:51:19 2009 +0200 cfq-iosched: change dispatch logic to deal with single requests at the time The IO scheduler core calls into the IO scheduler dispatch_request hook to move requests from the IO scheduler and into the driver dispatch list. It only does so when the dispatch list is empty. CFQ moves several requests to the dispatch list, which can cause higher latencies if we suddenly have to switch to some important sync IO. Change the logic to move one request at the time instead. This should almost be functionally equivalent to what we did before, except that we now honor 'quantum' as the maximum queue depth at the device side from any single cfqq. If there's just a single active cfqq, we allow up to 4 times the normal quantum. Signed-off-by: Jens Axboe It seems to have since been fixed, or at least made somewhat better somewhere between 2.6.32 and v2.6.33-rc1, though I was not able to git bisect to find the fix, as I ran into kernels that wouldn't boot somewhere in there, and didn't perservere enough to get past it. That was with hpsa though, I didn't try cciss. Might be the same problem though.
Seems to me that this is still a problem in 2.6.34.
You are using the cfq i/o scheduler? (completely fair queuing) Does using another i/o scheduler (e.g. deadline) make a difference? -- steve
Default was cfq, and I tried deadline, but it was the same. I use 64 for /sys/block/cciss\!c0d0/queue/read_ahead_kb (~50% better than 128) cciss0: HP Smart Array E200i Controller
If this is still seen on modern kernels then please re-open/update