Bug 13347 - Large I/O operations result in slow performance and high iowait times
Large I/O operations result in slow performance and high iowait times
Status: CLOSED OBSOLETE
Product: IO/Storage
Classification: Unclassified
Component: Block Layer
All Linux
: P1 blocking
Assigned To: Jens Axboe
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2009-05-19 23:00 UTC by Yuriy Lalym
Modified: 2012-06-07 10:37 UTC (History)
26 users (show)

See Also:
Kernel Version: 2.6.18 - 2.6.30
Tree: Mainline
Regression: Yes


Attachments

Description Yuriy Lalym 2009-05-19 23:00:14 UTC
Continue BUG-12309

kernel 2.6.30-rc6
yura@suse:~> export LANG=en
yura@suse:~> dd if=/dev/zero of=test1 bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 129.928 s, 80.7 MB/s

yura@suse:~> vmstat 1                                                           
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----   
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa   
 3  8      0  44708      0 7628596    0    0   249 12283  362  821  4  3 66 28  
 0  8      0  49180      0 7627532    0    0     0 40517 1061 1581  7  4  0 89  
 0  7      0  47188      0 7627180    0    0     0 59694 1156 1505  5  6  0 89  
 1  6      0  46692      0 7628276    0    0     0 55553 1144 1476  6  5  0 90  
 0  8      0  46428      0 7628160    0    0    20 51573  900 1096  5  4  0 90  
 0  7      0  46568      0 7627860    0    0     0 64024 1127 1480  5  5  0 90  
 0  7      0  45796      0 7629100    0    0    12 44597  889  987  6  4  0 90  
 0  8      0  46904      0 7627808    0    0   332 40500 1100 1485  6  4  0 90  
 0  7      0  47772      0 7626884    0    0   168 45300 1158 1628  6  4  0 90
 0  8      0  47216      0 7624456    0    0    72 67116  958 1151  5  5  0 90
 0  7      0  47032      0 7626480    0    0   280 29244 1177 1667  5  4  0 91
 0  7      0  45936      0 7626640    0    0   248 58872  922 1060  6  5  0 89
 0  9      0  44988      0 7626640    0    0   216 62492  945 1359  2  6  0 92
 0  8      0  47548      0 7625932    0    0   152 47164  926 1425  1  4  0 95
 1  6      0  45276      0 7627256    0    0    36 54721  605 1089  2  4  0 94
 0  7      0  48208      0 7626388    0    0    44 43612  834 1198  1  4  0 95
 0  8      0  47096      0 7625644    0    0   132 53789  655 1156  1  4  0 94
 0  7      0  46344      0 7624828    0    0   468 50292  981 2089  2  4  0 94
 0  8      0  46576      0 7625416    0    0   116 44056 1155 2119  1  3  0 96
 0  8      0  47476      0 7624800    0    0   636 38936  734 1125  2  4  0 94
 0  8      0  47348      0 7626676    0    0    32 58410  885 1613  1  5  0 93
 1  6      0  48508      0 7626280    0    0     0 67256  623  969  1  4  0 94
 0  7      0  47984      0 7625328    0    0     0 64888  694 1335  2  6  0 92
 0  7      0  45800      0 7626692    0    0     0 62496 1002 1698  1  4  0 95
 0  7      0  48220      0 7625052    0    0     0 61952  614 1222  2  5  0 93
 0  7      0  48508      0 7623300    0    0     0 69632  890 1586  1  5  0 94
yura@suse:~> time cp bigfile bigfile.cp

real    5m52.457s
user    0m0.343s
sys     0m21.356s

calc speed =>  10485760000 / 352.457 = 29.75 Mb/s

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- 
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa 
 1  0      0  46688      0 7686820    0    0     0    12  564  862  1  0 98  0
 0  0      0  46688      0 7686820    0    0    20     0  387  730  2  0 96  1
 0  0      0  46688      0 7686840    0    0     0     0  559  879  1  0 98  0
 0  0      0  46688      0 7686840    0    0     0     0  598  937  1  1 97  0
 0  0      0  46688      0 7686840    0    0     0     0  315  517  2  1 98  0
 0  0      0  46704      0 7686840    0    0     0    16  600 1058  2  1 97  0
 0  0      0  46704      0 7686840    0    0     0     0  328  473  2  0 98  0
 0  0      0  46704      0 7686840    0    0     0     0  610 1122  2  0 98  0
 0  0      0  46704      0 7686840    0    0     0     0  582 1013  2  0 98  0
 0  0      0  46876      0 7686840    0    0     0     1  341  475  1  0 98  0
 0  0      0  46876      0 7686840    0    0     0     0  577  988  2  0 98  0
 0  0      0  46876      0 7686840    0    0     0     0  339  543  2  1 97  0
start cp
 3  0      0  46500      0 7686704    0    0 17500     0  857 2379  2  2 91  5
 3  0      0  43840      0 7689132    0    0 90624     0 2119 5710  4 11 61 24
 0  1      0  46532      0 7686180    0    0 83968     0 2008 5246  8 11 57 24
 1  1      0  43884      0 7689020    0    0 81024    46 2159 8097  6 10 59 25
 0  1      0  45020      0 7687772    0    0 81920     1 1759 3732  4 10 60 26
 0  1      0  44948      0 7687472    0    0 91264     0 2154 4449  4 10 60 25
 0  1      0  43924      0 7688888    0    0 88064     0 2040 4500  3 11 60 26
 0  1      0  46180      0 7686288    0    0 89984     0 1919 4107  3 11 63 22
 0  2      0  44692      0 7680932    0    0 86784 39184 2156 4820  4 12 47 38
 0  2      0  44568      0 7681376    0    0 64384 22436 1569 4127  3  7 35 54
 0  2      0  44092      0 7682832    0    0 35584 37396 1331 2886  3  5 37 55
 4  2      0  46920      0 7678572    0    0 42624 43336 1544 3311  3  6 28 63
 0  2      0  45724      0 7679280    0    0 49792 31240 1301 3076  2  6 27 64
 0  2      0  45328      0 7681288    0    0 41856 31648 1473 3322  3  5 27 65
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- 
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa 
 0  2      0  46328      0 7679272    0    0 52480 29232 1425 3345  3  8 33 56
 1  2      0  46276      0 7679748    0    0 48768 24844 1564 3539  3  6 32 60
 1  2      0  47196      0 7678088    0    0 63360 28688 1830 4781  3  9 14 74
 5  3      0  44052      0 7681744    0    0 58112 23612 1493 3905  3  8  5 83
 1  2      0  44988      0 7679956    0    0 18560 53021 1107 2129  2  4  0 94
 0  4      0  46872      0 7677272    0    0 55808 22541 1478 4117  3  7  1 89
 0  4      0  43824      0 7681360    0    0 52608 33800 1627 4628  3  7  0 89
 0  4      0  45536      0 7679600    0    0 41856 31720 1491 4026  3  6  2 89
 0  4      0  45688      0 7680620    0    0 39424 34324 1202 3190  3  6  6 85
 1  4      0  46052      0 7679708    0    0 48104 27148 1901 5505  3  7  2 88
 0  2      0  44120      0 7680964    0    0 53280  6660 1531 5967  3  8  6 83
 2  3      0  45692      0 7679440    0    0 55784     0 1837 6209  4  7  4 84
 0  5      0  44796      0 7678908    0    0 52452 10248 1953 6444  4  8  1 87
 0  2      0  46952      0 7676344    0    0 45264 24619 2085 8506  6  7  0 87
 0  6      0  44820      0 7677336    0    0 34588 41550 2438 5289  2  7  4 86
 0  2      0  47084      0 7675392    0    0 31016 34352 1203 3506  3  5 17 74
 2  3      0  46856      0 7674440    0    0 10252 67612  685 1250  2  3 11 84
 0  5      0  45072      0 7677236    0    0 48004 15368 1575 4859  2  7  5 85
 0  5      0  45504      0 7678588    0    0 29824 23240  952 2688  3  3  8 85
 0  5      0  45020      0 7678452    0    0 62080  8196 1865 6184  3  9  0 88
 0  3      0  44564      0 7679208    0    0  6272 61473  607  926  2  2 24 71
 1  3      0  46444      0 7676624    0    0 14216 64046 1059 2083  3  2 41 54
 0  2      0  44444      0 7680932    0    0 53636 16048 1448 5787  3  8 10 79
 0  2      0  45188      0 7680048    0    0 40320 34024 1439 3785  3  6  1 90
 1  3      0  46208      0 7679612    0    0 63872 10248 1628 4998  3  9  1 87
 3  4      0  45808      0 7680852    0    0 47360 27152 2030 5505  3  6  7 83
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- 
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa 
 0  5      0  44496      0 7681320    0    0 45828 29452 1683 4815  3  6  0 90
 0  5      0  46044      0 7679900    0    0 44160 28704 1273 3575  3  7  3 87
 0  5      0  44676      0 7679868    0    0 17280 62076  840 1855  3  3  1 93
 0  5      0  44716      0 7681108    0    0 53504 11268 1862 6838  3  6  0 91
 0  4      0  44504      0 7681556    0    0 42880 30905 1460 3885  3  7  2 88
 0  3      0  44364      0 7680420    0    0 19712 69876  904 1960  2  4  3 91
 0  4      0  48668      0 7678024    0    0 52096 21788 1904 5313  3  7  1 88
 0  4      0  47032      0 7677752    0    0 28032 54580 1066 2496  3  5 16 76
 0  3      0  46600      0 7678904    0    0 41344 33300 1426 4071  3  6 10 80
 0  4      0  45424      0 7679184    0    0 41856 40276 1240 3204  3  6  0 91
 0  4      0  45692      0 7680156    0    0 46080 18172 1479 4094  3  6  8 83
 0  4      0  45908      0 7679112    0    0 45824 40108 1660 4158  3  7  4 86
 0  4      0  44048      0 7681292    0    0 49408 32776 1345 3872  3  7  8 81
 0  3      0  46548      0 7678436    0    0 50816 22604 1609 4448  3  7 12 77
 0  2      0  46156      0 7678672    0    0 46464 35852 1350 3829  2  7 12 79
 0  3      0  45244      0 7678956    0    0 50304 26420 1664 4697  3  7  4 85
 0  3      0  48256      0 7675968    0    0 33280 31759 1325 3366  3  5 13 79
 0  4      0  44852      0 7679008    0    0 35328 53281 1168 3202  3  5 39 52
 0  4      0  46668      0 7677444    0    0 38784 24628 1390 3743  2  5 14 78
 0  5      0  45028      0 7680464    0    0 49920 20492 1373 4244  3  7  3 87
 0  5      0  45356      0 7681500    0    0 33408 35548 1369 3642  3  5  2 90
 0  5      0  46896      0 7679744    0    0 57216 23808 1526 4868  3  7  3 87
 1  1      0  45884      0 7679008    0    0 34432 50311 1427 3789  3  5  3 89
 2  3      0  44592      0 7679592    0    0 45952 40989 1378 4140  2  7  3 87
 1  2      0  44188      0 7680924    0    0 25856 48793 1023 2593  3  4  1 92
 0  5      0  45092      0 7680536    0    0 46336 26184 1533 4502  3  6  5 86
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- 
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa 
 1  5      0  44880      0 7681796    0    0 46208 23228 1560 4530  3  7  4 85
 0  5      0  46412      0 7679076    0    0 60800 23884 1623 5242  3  8  7 81
 0  5      0  46488      0 7679024    0    0 50048 24908 1637 4677  3  7  8 83
 0  2      0  44564      0 7681684    0    0 36224 39584 1140 3492  3  4 11 81
 0  5      0  45200      0 7681196    0    0 50304 22796 1757 5024  3  7  7 82
 0  3      0  46924      0 7672852    0    0 34560 43484 1175 3359  2  6  0 91
 0  6      0  45632      0 7675104    0    0 37504 36780 1346 3847  3  6  4 87
 0  6      0  45988      0 7673412    0    0 43776 42904 1434 4643  3  6  1 91
 0  5      0  48244      0 7669064    0    0 30848 44548 1094 3317  3  5  3 88
 0  5      0 286084      0 7439272    0    0 35456 33520 1403 3765  3  7  8 82
 2  4      0 181440      0 7543056    0    0 51968 27148 1491 4836  3  6  3 87
 0  4      0 119520      0 7601468    0    0 29184 43984  969 2825  3  4  4 89
 0  6      0  44456      0 7680104    0    0 43008 34584 1377 4120  3  5  2 89
 1  5      0  49396      0 7666304    0    0 29956 56836 1091 3636  2  5  0 92
 0  6      0  47008      0 7669068    0    0 37508 22336 1439 4405  3  6  0 91
 0  6      0  45312      0 7670448    0    0 20864 32748 1173 3639  2  4  0 94
 0  7      0  45792      0 7673916    0    0 29568 14856  996 2870  3  5  1 91
 0  6      0  44136      0 7679304    0    0 16128 63532 1052 2344  2  3  4 90
 0  3      0  44800      0 7678884    0    0 49664 15644 1407 5044  2  7  1 89
 0  7      0  45180      0 7679664    0    0 44416 28660 1534 4548  2  6  2 89
 0  7      0  43892      0 7678188    0    0 30080 35848 1365 4087  3  4  2 91
 0  7      0  45176      0 7672668    0    0 35456 39344 1145 3608  3  5  1 90
 0  7      0  46560      0 7671748    0    0 25472 36872 1239 3494  3  5  0 92
 0  4      0  47096      0 7675136    0    0 30464 24052 1044 3068  2  5  5 88
 0  5      0  43864      0 7683468    0    0 34944 31988 1274 3345  3  5 10 82
 0  3      0  44452      0 7676068    0    0 32640 43300 1758 5028  2  5  1 93
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- 
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa 
 1  5      0  46088      0 7679004    0    0 26368 34104  951 2415  3  4 10 83
 1  7      0  44644      0 7680964    0    0 39296 40568 1457 4161  3  5  4 87
 0  3      0  46904      0 7677248    0    0 17280 52767  847 2121  2  4 26 68
 0  6      0  45004      0 7679484    0    0 33408 36716 1135 3297  3  5  8 83
 0  3      0  45324      0 7680364    0    0 32128 41892 1873 4455  2  5  7 85
 0  3      0  45672      0 7679632    0    0 38144 32788 1146 2985  2  5  4 88
 0  4      0  44792      0 7681208    0    0 31488 34320 1255 2825  3  4  6 87
 0  5      0  44856      0 7678824    0    0 40960 31488 1259 3753  3  6  1 89
 5  4      0  46340      0 7678096    0    0 53632 25620 1703 5203  3  7  4 86
 0  3      0  46204      0 7678884    0    0 34816 37500 1395 3642  2  5  4 88
 0  5      0  47052      0 7671596    0    0 51840 40476 1460 4763  3  6  3 87
 0  3      0  44276      0 7676316    0    0 27520 30743 1245 3234  3  4  2 90
 0  5      0  46692      0 7678184    0    0 22784 48161  909 2197  2  5  3 90
 0  5      0  44396      0 7679168    0    0 46336 30268 1546 4511  3  6  5 85
 0  5      0  44352      0 7680436    0    0 43264 26744 1543 4340  2  6  2 90
 0  5      0  45496      0 7672684    0    0 44672 37352 1397 4869  3  7  0 90
 0  5      0  49372      0 7662316    0    0 33792 49596 1403 3972  3  6  1 90
 0  4      0  45644      0 7677232    0    0 24960 22280  926 2576  2  4  3 90
 0  6      0  44964      0 7680548    0    0 53120 33176 1705 5679  3  7  1 88
 0  7      0  45844      0 7671820    0    0 22456 37240 1288 4498  3  5  0 92
 0  8      0  45220      0 7667760    0    0 27404 24584  979 2885  2  5  0 93
 0  7      0  45812      0 7666412    0    0 21308 24720 1304 4636  2  3  5 89
 0  6      0  44268      0 7664988    0    0 29152 36896 1250 4517  3  5  0 92
 1  7      0  48448      0 7667896    0    0 27376 21508 1298 4418  3  5  0 92
 0  3      0  52464      0 7672108    0    0 27776 36073 1239 4920  3  5  0 92
 0  4      0  46272      0 7677508    0    0 43008 40522 1351 4645  2  6  0 91
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  7      0  45520      0 7672060    0    0 39296 36952 1499 4941  3  6  0 90
 2  6      0  44008      0 7667764    0    0 41088 39800 1286 4794  3  6  0 91
 0  5      0  45480      0 7665824    0    0 32512 39428 1362 4147  3  5  1 90
 5  7      0  45260      0 7674556    0    0 41728 18204 1464 5160  2  6  1 90
 0  6      0  44860      0 7674124    0    0 31360 43012 1069 3452  3  5  4 88
 1  5      0  44912      0 7672628    0    0 31744 43300 1365 4663  3  5  0 92
 0  4      0  50228      0 7674084    0    0 22656 51238  922 2896  2  5  3 90
 1  6      0  44768      0 7680248    0    0 51840 24756 1661 6082  3  7  5 84
 0  6      0  47312      0 7678352    0    0 45952 25656 1554 5644  3  6  3 87
 0  8      0  44808      0 7675632    0    0 39552 47932 1247 4241  3  6  1 90
 0  8      0  45832      0 7664640    0    0 33024 47104 1447 4698  2  5  0 93
 0  9      0  46444      0 7664368    0    0 40192 41608 1299 4904  3  6  0 91
 0  9      0  45212      0 7673000    0    0 39296 16156 1461 5472  3  5  0 92
 5  6      0  44980      0 7673068    0    0 25216 51256 1245 4057  2  5  2 90
 0  6      0  44220      0 7681560    0    0 34816 27188 1088 3746  2  6  6 85
 0  9      0  44824      0 7680132    0    0 47616 19960 1891 6827  3  6  9 81
 0  2      0  45324      0 7678940    0    0  3200 81459  615  855  2  2 42 54
 0  6      0  40888      0 7683144    0    0 24080 43565 1102 3157  3  4 28 65
 0  9      0  46280      0 7678620    0    0 51460  7176 1738 6587  3  6  3 88
 1  1      0  44716      0 7681788    0    0 31360 39488 1113 3891  3  5 14 78
 0  9      0  45096      0 7680228    0    0 52864 21536 1640 5803  3  7  6 84
 0  9      0  45744      0 7680164    0    0 44032 31268 1332 4769  4  6  2 88
 0 10      0  45880      0 7669856    0    0 39336 49300 1433 4746  1  5  0 94
^C
Comment 1 Tejun Heo 2009-05-20 06:55:41 UTC
Can you please elaborate what the problem is?  If you're copying from and to the same disk, your transfer speed of course is gonna be lower.  Is it that you think it's too low?

Thanks.
Comment 2 rafal 2009-05-20 08:55:11 UTC
Maybe this is related to:
http://lkml.org/lkml/2009/5/15/320
(patch in -mm)
Comment 3 Yuriy Lalym 2009-05-20 10:08:07 UTC
(In reply to comment #1)

CPU - Core 2 Duo E8400,
RAM - 8 Gb
HDD - S31000340AS (1000 Gb, 32Mb Cache, 7200 об, S-ATA-II)
FileSystem - XFS

http://bugzilla.kernel.org/show_bug.cgi?id=12309#c234
It is the same computer, only a kernel 2.6.28.5-default x86_64 (now 2.6.30-rc6 x86_64)

Yes I consider that speed of copying became very low (comparison of this fact to operation of other operating systems confirms my supposition). Yes I see that at the moment of copying responsiveness of system zero (the system sleeps, and the processor heats air by 90% iowait)

And plus (kernel 2.6.30-rc x86_64)

time cp -r qt-x11-opensource-src-4.5.1 qt-x11-opensource-src-4.5.1-1
real    5m51.075s
user    0m0.147s
sys     0m2.192s

302.6Mb / 351s = 0.9 Mb/s

It too good speed? To me say that it is perfect other problem. But I do not see here other problem. After copying of each small file system that waits and only then starts to copy the following file.
Comment 4 Tejun Heo 2009-05-20 10:20:53 UTC
Hello,

(In reply to comment #3)
> (In reply to comment #1)
> 
> CPU - Core 2 Duo E8400,
> RAM - 8 Gb
> HDD - S31000340AS (1000 Gb, 32Mb Cache, 7200 об, S-ATA-II)
> FileSystem - XFS
> 
> http://bugzilla.kernel.org/show_bug.cgi?id=12309#c234
> It is the same computer, only a kernel 2.6.28.5-default x86_64 (now 2.6.30-rc6
> x86_64)
> 
> Yes I consider that speed of copying became very low (comparison of this fact
> to operation of other operating systems confirms my supposition).

29.75MB/s doesn't look too bad for copying in the same disk.  It will depend on a lot of parameters including readahead, disk fragmentation, copy buffer size, filesystem implementation and so on.

> Yes I see
> that at the moment of copying responsiveness of system zero (the system sleeps,
> and the processor heats air by 90% iowait)

The lack of responsiveness could be caused by FS.  There was related fix for ext3 lately but in general XFS tends to perform better responsiveness-wise.  Also, iowait == idle w/ IOs in flight, so it won't be heating air.  90% iowait means that the cpu had nothing to do while waiting for io.

> And plus (kernel 2.6.30-rc x86_64)
> 
> time cp -r qt-x11-opensource-src-4.5.1 qt-x11-opensource-src-4.5.1-1
> real    5m51.075s
> user    0m0.147s
> sys     0m2.192s
> 
> 302.6Mb / 351s = 0.9 Mb/s
> 
> It too good speed? To me say that it is perfect other problem. But I do not see
> here other problem. After copying of each small file system that waits and only
> then starts to copy the following file.

I can't really understand what you tried to say above (were you trying to be sarcastic?), but yeah I'm on XFS too and small file ops feel quite slow.  I saw many benchmarks saying xfs is better performer than ext3 and it indeed seems more responsive with a lot of IOs in flight but after several months of use I can't really say it feels faster than ext3.  Please note that small file ops are dominated by seeking and flushing and may vary greatly depending on which mount option you're using, the filesystem layout and whatnot.

At any rate, doesn't look like anything low level.  cc'ing Andrew.  Andrew, can you please take this over?  I don't have much clue how to proceed and my crystal ball tells me this bug is gonna have a lot of confusion around it.  :-P
Comment 5 Yuriy Lalym 2009-05-20 21:23:04 UTC
(In reply to comment #4)
Is not present what sarcasm. I can is not correct that has translated that into English, it can be.
I cannot deny your words, to me is simple there is nothing to check up work ext3 in the same operations. Who can be will give results for a similar hard disk, well if is not present in couple of days we will close this question.
Comment 6 Alan 2009-05-21 16:27:00 UTC
This should probably be moved to memory management judging by the 2.6.30 work
Comment 7 Yuriy Lalym 2009-05-23 23:01:39 UTC
I use openSUSE therefore for me operation is natural
The Menu-> YAST-> dialogue of input of the superuser password-> input password -> YAST window

If copying or removal it is fulfilled looks so
The Menu-> YAST-> dialogue of input of the superuser password-> input password -> without the answer (inquiries leave in anywhere or the latency is so great that the system simply breaks inquiry)

After end of file operation all again works normally (under the first circuit).
And it not only YAST, and all that demands the superuser password. Well simple copying so to load system cannot.

http://www.rhic.bnl.gov/RCF/LiaisonMeeting/20061228/Other/xfs.html
http://www.debian-administration.org/articles/388

I do not see a special difference between xfs and ext3, and besides

>Debian Etch (kernel 2.6.15), distribution upgraded on April 18, 2006
>Processor : Intel Celeron 533
>RAM : 512MB RAM PC100

>Operations on a file tree (7500 files, 900 directories, 1.9GB)
>The initial copy of the tree was quicker for Ext3 (158.3 secs) and XFS (166.1)

File tree 1 (7443 files, 816 directories, 1.9GB) on my system

yura@suse:~> time cp -r 1 2

real    2m42.864s
user    0m0.081s
sys     0m3.424s

2*60+42=162s, I use Core 2 Duo and SATA-2 (no Intel Celeron 533)

http://linuxgazette.net/122/TWDT.html#piszcz

COPY 2.6.14.4 KERNEL SOURCE TREE
xfs - 44s
REMOVE KERNEL 2.6.14.4 SOURCE TREE
xfs - 9s

yura@suse:~> time cp -r /usr/src/linux-2.6.30-rc6 /home/yura/

real    6m21.969s
user    0m0.142s
sys     0m2.785s

yura@suse:~> time rm -fr /home/yura/linux-2.6.30-rc6

real    1m54.315s
user    0m0.020s
sys     0m1.619s
Comment 8 Yuriy Lalym 2009-05-24 01:42:43 UTC
yura@suse:~> dd if=/dev/zero of=test bs=1024 count=500
500+0 records in
500+0 records out
512000 bytes (512 kB) copied, 0.000914665 s, 560 MB/s

yura@suse:~> time cp test test1
real    0m0.057s
user    0m0.000s
sys     0m0.002s

0.5Мб * 10000 = 4.88Gb
0.057s * 10000 = 57s

whence
>time cp -r 1 2 (7443 files, 816 directories, 1.9GB)
>real    2m42.864s
>user    0m0.081s
>sys     0m3.424s

yura@suse:~> dd if=/dev/zero of=test2 bs=1G count=2
2+0 records in                                     
2+0 records out                                    
2147483648 bytes (2.1 GB) copied, 8.43014 s, 255 MB/s

yura@suse:~> time cp test2 test2-1
real    0m22.890s
user    0m0.036s 
sys     0m2.889s 

yura@suse:~> dd if=/dev/zero of=test4 bs=1G count=4
4+0 records in                                     
4+0 records out                                    
4294967296 bytes (4.3 GB) copied, 50.9457 s, 84.3 MB/s

yura@suse:~> time cp test4 test4-1 
real    0m49.013s
user    0m0.100s 
sys     0m6.210s 

yura@suse:~> dd if=/dev/zero of=test8 bs=1G count=8
8+0 records in                                     
8+0 records out                                    
8589934592 bytes (8.6 GB) copied, 89.543 s, 95.9 MB/s

yura@suse:~> time cp test8 test8-1 
real    3m37.498s
user    0m0.202s 
sys     0m13.607s

yura@suse:~> dd if=/dev/zero of=test16 bs=1G count=16
16+0 records in                                      
16+0 records out                                     
17179869184 bytes (17 GB) copied, 248.019 s, 69.3 MB/s

yura@suse:~> time cp test16 test16-1
real    9m11.836s
user    0m0.376s 
sys     0m26.328s

System time grows linearly, real-?

And

yura@suse:~> time rm test8 (8.6 GB)
real    0m0.001s
user    0m0.000s
sys     0m0.001s

yura@suse:~> time rm test16 (17 GB)
real    0m0.001s
user    0m0.000s
sys     0m0.001s

linux-2.6.30-rc6 (27841 files, 1737 directories)
0.001*(27841+1737) = 48s

whence
>time rm -fr /home/yura/linux-2.6.30-rc6
>real    1m54.315s

Than the system is engaged in intervals between removals of each file, and files not on 17Gb
Comment 9 Perlover 2009-05-24 07:50:17 UTC
Confirm bug

My OS is Fedora Core release 6
Kernel: 2.6.22.14-72.fc6
2 CPUs: Intel® Xeon® CPU 5130 @ 2.00GHz
HDDs: SAS 3.0 Gb/s, FUJITSU
RAID: Adaptec 4800SAS
RAID10

How to test:
# dd if=/dev/zero of=testfile.1gb bs=1M count=1000

In other terminal during a copying you should run:
# vmstat 1

I see for example:
r b swpd free buff cache si so bi bo in cs us sy id wa st
14 8 460 120716 280236 1509844 0 0 9 14 0 0 9 3 66 22
0
0 13 468 121936 279216 1550936 0 0 1368 47776 1927 4153 24 8 8 60
0
0 15 468 121516 280200 1551200 0 0 1408 3744 1726 2846 1 2 3 94
0
0 8 468 129804 280520 1545940 0 0 1612 4280 1854 4060 3 2 1 95
0
0 6 468 131388 281868 1546628 0 0 2140 3620 2020 4650 12 3 13 71
0
0 17 468 114220 282792 1571864 0 0 1208 3212 1647 2715 4 3 6 87
0
1 12 468 115356 283164 1570704 0 0 1420 18964 1718 2397 2 2 2 94
0
0 9 468 114320 283628 1570868 0 0 768 1204 1753 2831 3 1 0 96

iowait -> 80-90% during 'dd'
All other CPU's task work very very slow ...

AND (!!!), the output of 'dd' is:
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 112.086 seconds, 9.4 MB/s
                                                   ^^^^^^^^^

During some years i see a following behaviour: if server often uses a harddisk (for testing: 'dd' examples here) then iowait is stability 50-90% and many tasks are frozen during some seconds (10-20 and may be more at me). It's easy for testing through 'dd'. I cannot resolve this trouble by ionice for example - iowait is high even if i do a some i/o tasks ionice -c3 or ionice -c2 -n7 for example! So each server under kernel 2.6.18 and more (i read many topics) has this bug. A people in forums write that the kernel of 2.6.30-rc2 has bug too and that FreeBSD work quickly (mouse moving, video showing and some other CPU's tasks) during 'dd' testing unlike Linux ...

I don't know what arguments do you want for finding this bug! This bug to be since 2007 year ...

Please help! Here examples of my loaded server in some times (not DD - there only typical Mysql database & Mysql tasks & apache tasks):

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
-
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
13 14    120  68460 574784 1286748    0    0    13     1    0    0  9  3 66 22
0
 1 11    120  74564 576080 1286976    0    0  1560     0 1632 3641 34 10  0 57
0
 0 12    120  69988 577572 1287352    0    0  1904     0 1969 3696  5  2  0 93
0
 0 11    120  66916 578984 1287860    0    0  1900     0 1809 3615  6  2  0 92
0
 0 11    120  64960 580424 1288028    0    0  1668     0 1642 2188  1  1  0 97
0
 0 11    120  72764 576508 1286788    0    0  1668     0 1681 2198  3  2  0 96
0
 1 11    120  71424 577940 1287300    0    0  1604   332 1575 2152  2  1  0 97
0
 3 11    120  58852 579528 1289100    0    0  2000     0 1984 3286 44  7  0 49
0
 1 11    120  75104 581012 1287472    0    0  1608     0 2119 2839 39  7  0 55
0
 0 13    120  72160 582572 1287672    0    0  1908   120 1645 2366  7  1  0 92
0


[root@63 logs]# vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
-
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 5  9    120  95540 570248 1276840    0    0    13     1    0    0  9  3 66 22
0
 1  7    120  93996 571428 1277440    0    0  1772 33712 2024 4341 28  4 11 57
0
 0  7    120  97980 572528 1277884    0    0  1444   300 1568 2339 13  1 17 70
0
 0  7    120  99900 573532 1278468    0    0  1504     0 1513 2364  4  2  3 90
0
 1  5    120  98656 574484 1278540    0    0  1052   400 1629 1924  2  1  0 97
0
 1  3    120  97924 574932 1278916    0    0   480 21108 2276 1987 11  2 47 40
0
 1  4    120  87280 575264 1279040    0    0   432  3676 2456 2654 23  2 40 35
0
 1  5    120  95856 575668 1279140    0    0   780  4128 2249 3097 26  2 25 47
0

Here you can see stability high 'wa' field. When my tasks frozen during 10-20 seconds i see there 80-90% 'wa'.

Please can catch this bug!

Thanks
Comment 10 Yuriy Lalym 2009-05-24 08:09:12 UTC
I have forgotten to mention, if such small program as kcalc does not start for
1 second it not so well, well all right the system is occupied - for 3 seconds.
But if at a copying file of 17 gigabytes start time kcalc 20-30 seconds, it
already than not explainably. A disk 1000Gb from them 700Gb are free. Before
copyings I have fulfilled a defragmentation of all disk.
On operations with files 2 and 4 Gb we see that the disk works well and holds
speed declared Seagate.
Comment 11 Alex Efros 2009-05-25 01:44:22 UTC
Same is here. I'm ready to provide any required info, but looks like everything already provided both in BUG-12309 and here. On russian forums people noticed only some hardware affected by this bug, maybe because of this it's so hard to find and fix it. Anybody checked related code changes between 2.6.17 and 2.6.18 (when this bug was introduced) or it's too dirty work?
Comment 12 Yuriy Lalym 2009-05-25 16:59:20 UTC
(In reply to comment #6)
> This should probably be moved to memory management judging by the 2.6.30 work

Probably. But also it is possible, to task management, problems glibc and gcc, etc.

And it is possible in general what does not concern. Simply there is an iron which demands defined the driver and these of the driver in itself (separately taken) work, but in not which combination they start to clash with each other, we and observe consequences of this conflict. Therefore we on one computers have a problem, and on other it are not present.

But I cannot judge about it.
Comment 13 Yuriy Lalym 2009-05-26 20:56:40 UTC
You do not like a word iowait, will forget about it, as about a by-product.
 
Question.

> http://www.debian-administration.org/articles/388
> Processor : Intel Celeron 533
> RAM : 512MB RAM PC100
> Motherboard : ASUS P2B
> Hard drive : WD Caviar SE 160GB (EIDE 100, 7200 RPM, 8MB Cache)
> Controller : ATA/133 PCI (Silicon Image)
> Debian Etch (kernel 2.6.15), distribution upgraded on April 18, 2006

Processor : Core 2 Duo E8400
RAM : 8Gb DDR2-800
Motherboard : ASUS P5K
Hard drive : ST31000340AS Seagate 1000Gb (SATA-2, 7200 RPM, 32MB Cache)
Controller : JMicron AHCI NCQ
Kernel 2.6.30-rc7, enter with init 3
File system: xfs

> Operations on a large file (ISO image, 700MB)
> The initial copy of the large file took longer on Ext3 (38.2 secs) and
> ReiserFS (41.8) when compared to JFS and XFS (35.1 and 34.8).

time cp file-700Mb file-700Mb-1
real    0m8.421s ( < 34.8s It is very good; 34.8/8.421 = 4.13 !!!)
user    0m0.047s
sys     0m1.207s

> Operations on a file tree (7500 files, 900 directories, 1.9GB)
> The initial copy of the tree was quicker for Ext3 (158.3 secs) and XFS (166.1)
> ...
> However, the tree removal was about 2 times longer for Ext3 (22 secs) when
> compared to ReiserFS (8.2 secs), XFS (10.5 secs) and JFS (12.5 secs)!

I have prepared directory FileTree with such data (7500 files, 900 directories, 1.9 Gigabytes). Files of the different size from 100 byte to 6.6 Mbytes.

time cp -r FileTree FileTree-1
real    2m21.355s ( 2*60+21 = 141s, That it.
                    Here should be as a maximum 166.1/4.13 = 40.2s )
user    0m0.093s
sys     0m3.723s

time rm -fr FileTree-1
real    0m32.946s ( It already over any concepts.
                    Here should be as a maximum 10.5/4.13 = 2.5s )
user    0m0.007s
sys     0m0.817s
Comment 14 Yuriy Lalym 2009-05-26 21:18:12 UTC
> Directory listing and file search into the previous file tree
> The complete (recursive) directory listing of the tree was quicker for
> ReiserFS (1.4 secs) and XFS (1.8) when compared to Ext3 and JFS (2.5 and 3.1).

time ls -R FileTree
...............
real    0m4.680s
user    0m0.013s
sys     0m0.087s

It is horror.

And to it it is necessary to add very low responsiveness of system. I do not argue there are tasks where all resources need to be thrown on their performance. But there is also a weight of other applications where is inadmissible to wait while file copying.
Comment 15 Perlover 2009-06-09 09:57:22 UTC
Please to see post # 360
http://bugzilla.kernel.org/show_bug.cgi?id=12309
I think there found a some solution
I didn't test but may be anybody will try it ?
Comment 16 Perlover 2009-06-17 04:32:11 UTC
Did anybody here resolve this bug ?
I see a workaround only as an installing FreeBSD instead a Linux kernel version >= 2.6.18
Comment 17 Alan 2009-06-17 08:33:31 UTC
Hard disk seek time is long, hard disks only get decent I/O rates when handling continous streamed data. Those are hard physics limits.

There are things that feed into that - the size of the ext3 journal is one and configurable. In the ext3 case there were also performance problems with fsync() which shows up with current desktops badly as they tend to over use and misuse fsync a lot. 2.6.30 has some imprvements in that area.
Comment 18 Yuriy Lalym 2009-06-17 21:24:53 UTC
(In reply to comment #17)
> Hard disk seek time is long, hard disks only get decent I/O rates when handling
> continous streamed data. Those are hard physics limits.

Rather also it is rather convincing. I even have begun to doubt in the correctness.

Barracuda® 7200.11 Desktop Hard Drives
Designed with up to four platters and the only second-generation perpendicular recording technology .......

Model Number	          ST31000340AS    WD1600AAJS
Interface	          SATA 3.0Gb/s    EIDE 100 МБ/с
Cache	                  32 MBytes       8 MBytes
Capacity	          1 TB            160 GB

PERFORMANCE	 
Spindle Speed	          7,200 rpm       7,200 rpm
Average latency	          4.16 msec       4.20 ms
Average  seek time        ?               8.9 ms
Random read seek time	  <8.5 msec       ?
Random write seek time	  <9.5 msec       ?
Track-To-Track Seek Time  ?               2 ms

So where the reason of increase in time rm (and ls -R FileTree) in 3 times.
Comment 19 Tejun Heo 2009-06-18 01:05:17 UTC
7200.11 drives are notorious for various performance glitches.  Most seemed to be blamed on the firmware but only Seagate would know what actually went down there.  Looks like 7200.12 ironed out all those issues and works just fine.  I don't know what you're experiencing is this issue tho.  Are you saying that ST31000340AS is three times slower to delete files compared to WD1600AAJS on the same machine?
Comment 20 Perlover 2009-06-18 16:58:09 UTC
I think that i coped with a this bug !
I made some options of kernel and my server works stability and there are no frozen timeouts with high iowait already 10-12 hours!

Detailed info:
My kernel now is 2.6.22.14-72.fc6
Fedora Core 6

This the suggestion is not bug resolving (i think there is bug in kernel and it stays) but this is a workaround. I have read many topics and forums and stopped at these commands:

# echo 50 > /proc/sys/vm/vfs_cache_pressure
# echo deadline > /sys/block/DEVICE/queue/scheduler
#  # echo 1 > /sys/block/DEVICE/device/queue_depth
# echo 1024 > /sys/block/DEVICE/queue/nr_requests

The DEVICE is 'hda' or 'sda' for some HDDs. I didn't test queue_depth because for my HDDs (SAS SCSI + RAID10) this file is readonly (no there NCQ supporting as i think). But may be this command will help to you. I don't know.

I suggest anybody who have a frozen timeouts with high iowait to try this turning

I am very glad! Please to try this workaround. I didn't test 'dd' command but my heavy a HDD working has been freezing the server. Now i don't see this.
Comment 21 Yuriy Lalym 2009-06-27 13:41:49 UTC
(In reply to comment #19)
> 7200.11 drives are notorious for various performance glitches.  Most seemed to
> be blamed on the firmware but only Seagate would know what actually went down
> there.  Looks like 7200.12 ironed out all those issues and works just fine.  I
> don't know what you're experiencing is this issue tho.  Are you saying that
> ST31000340AS is three times slower to delete files compared to WD1600AAJS on
> the same machine?

1. In Windows there are no problems.
2. I perfectly understand that manufacturers of iron give the information, the driver and other rubbish, first of all to microsoft. You have done errors to you and to eliminate these errors.
3. Why I cannot open Firefox (I expect more than two minutes) more precisely when file operations are executed.

You have other super-super disk? You have a super heaped up file system? Perfectly. But I do not trust. Result to me a brand of a disk and tests of its operation - then it is possible to speak about something.
Comment 22 Tejun Heo 2009-06-27 22:58:13 UTC
Yuriy, I'm having a lot of trouble following what you try to communicate.  So, I suppose there's no significant performance difference between the two drives under Windows, right?  Hmmmm... there have been some ext3 latency issues which have been fixed during 2.6.30 devel cycle.  Can you please give a shot at 2.6.30?

Thanks.
Comment 23 Yuriy Lalym 2009-06-28 14:36:21 UTC
Kernel 2.6.30
Comment 24 Yuriy Lalym 2009-06-28 15:53:56 UTC
I would put a film, but he is rather big on a size. And where to lay out him that it was more conveniently to look I do not know.

Essence is in the following. Works ktorrent (5 megabit in a second, ~25 files, out). Ktorrent in a system tray. I press on an icon in the system tray of ktorren developed, and I see all list of distributions. I press on a dagger (to close a window) - ktorren convolves in a system tray.

I unpack linux-2.6.31-rc1.tar.bz2 (right-click in Dolphin).

Again I press on the icon of ktorren in a system tray and ktorren opened out (approximately after 3 seconds). But I see a simply white window 10-15 seconds. The list of distributions appears then. I press on a dagger. A window does not answer. Yet I wait 10-15 seconds and again press on a dagger the window of ktorren convolves however. At the next opening out I see a white window again.
Comment 25 developer 2009-09-06 15:24:04 UTC
(In reply to comment #20)
> [...]
> at these commands:
> # echo 50 > /proc/sys/vm/vfs_cache_pressure
> # echo deadline > /sys/block/DEVICE/queue/scheduler
> #  # echo 1 > /sys/block/DEVICE/device/queue_depth
> # echo 1024 > /sys/block/DEVICE/queue/nr_requests
> The DEVICE is 'hda' or 'sda' for some HDDs. I didn't test queue_depth because
> for my HDDs (SAS SCSI + RAID10) this file is readonly (no there NCQ supporting
> as i think). But may be this command will help to you. I don't know.
> I suggest anybody who have a frozen timeouts with high iowait to try this
> turning
> I am very glad! Please to try this workaround. I didn't test 'dd' command but
> my heavy a HDD working has been freezing the server. Now i don't see this.

I tried this work around and it seems that the iowaits are not dropping significantly. But the system is much more responsive with the high iowaits. The stalls are completely gone! Even if you leave NCQ on (not setting queue_depth to 1).

Thank you Perlover for the hint.

But still the issue remains...
Comment 26 Perlover 2009-09-11 08:02:17 UTC
> Thank you Perlover for the hint.
> 

Yes, these commands (i think only a setting of deadline scheduler will be enough) helped to me. I administrate 10 SCSI servers in the different datacenters and with a different hardwares. And all 10 servers now work fine! Now there are not a frozen servers. May be now an iowait is kept and to be around 50-100% sometimes but now there no a frozen stops of operation system (before was a regular 10-40 seconds stops of OS many times in day with 99% iowait). All works smoothly.
I didn't upgrade kernels, i use old 2.6.22.14-72.fc6 kernel version. And i will not try to upgrade because all 10 servers under high loading.
Comment 27 Alex Efros 2009-11-23 02:29:46 UTC
I'm finally tired of this bug - it's faster to boot windows, copy several films and boot linux again than just copy these films in linux. So I decided to change hardware, because AFAIK this bug happens only on some hardware. I've waited for several years until this bug will be fixed, but looks like only way to get full hdd i/o speed in modern linux is use different hardware. :(

The question is, WHICH hardware I should buy? It would be hard enough to come into shop and take one-by-one several motherboards and hdd until I will be lucky enough to find one which work at full speed.

Right now I've Core2Duo E6600 CPU (which I don't wanna change), so I need motherboard on LGA775 socket. My current motherboard is ASUS P5B-Deluxe (chipsets P965 and ICH8R, plus JMicron JMB363). My HDD is ST3320620AS Barracuda 7200.10 320GB. I'm not sure what exactly should be changed: motherboard, hdd or both. And I've no idea which hardware will works without that bug. Any recommendations?
Comment 28 Alan 2009-11-23 10:42:56 UTC
Not enough information to even guess if your specific problem is hardware. In fact you've provided almost no information at all to even tell whether the problem you report is related to this report (which itself appears to be a mix of different bugss, some fixed some maybe not - hard to tell due to lack of info).
Comment 29 Alex Efros 2009-11-23 11:44:55 UTC
I've provided no information because I have exactly same bug as explained in this ticket: slow hdd i/o (about 20MB/sec while copying from /dev/zero to hdd, and about 10MB/sec while copying from hdd to hdd on same ext3 fs), plus high iowait while i/o operations, and of course very slow system while copying large files (probably because of high iowait).
Comment 30 Alan 2009-11-23 11:56:23 UTC
That isn't the bug most of this ticket is reporting though.

Lets start from the beginning bit. Copying to/from /dev/zero. What drive, what controller, what does dmesg report, what does hdparm -t, what file system are you using and what exact kernel revision ?

On a decent core 2 duo I'd expect to see something like

hdparm -t /dev/sda

/dev/sda:
 Timing buffered disk reads:  274 MB in  3.00 seconds =  91.18 MB/sec

as a base point. That is an Intel ICH10 based board with a WD drive. Slightly newer but not vastly so.
Comment 31 Alex Efros 2009-11-23 12:31:42 UTC
I can add more information: I suffer from this bug for several years, from about 2.6.18-2.6.19 kernel.

One of my friends doesn't had that bug until he upgraded his hardware to ASUS P5KPL-AM SE with CPU Intel 5200.

Another friend doesn't have this bug even now, he use 2.6.25 kernel and his hardware is Core 2 Quad Q9300 on Gigabyte EP35C-DS3R, with HDD SAMSUNG HD103UJ.


Right now, on my system (kernel 2.6.28, and I have of course tried latests vanilla kernels too - with no luck):

# dd if=/dev/zero of=test1 bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 79.2394 s, 26.5 MB/s

And this is iowait values while executing dd command:
# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0 429788 261604 213220    0    0   101    67   39    4  4  2 93  1
 0  0      0 430536 261604 213240    0    0     0     0 2219 2456 19  1 80  0
 0  0      0 429540 261612 213808    0    0   575     0 2284 1992  7  3 88  3
 0  1      0 310756 261736 327460    0    0    60 113716 2399 2547 10 14 41 36
 0  1      0 293332 261752 344704    0    0     0 16384 2329 1542  6  4 44 47
 0  1      0 213680 261828 421764    0    0     0 78016 2400 1302  4 10 45 41
 0  2      0 190460 261852 446796    0    0     0 20484 2301 1201  4  5 22 68
 0  3      0 191520 261852 446808    0    0     0     0 2382 1820  5  4  0 92
 0  1      0  84684 261968 550340    0    0     8 108648 2335 1691  8 13 33 47
 0  1      0  60748 250248 586548    0    0     0 40960 2404 2223 15  6 34 45
 0  1      0  54180 231768 613944    0    0   128 36864 2389 1962  6  6 38 50
 0  3      0  49808 216020 637712    0    0    20 24580 2386 1637  4  7 20 70
 1  1      0  51564 216032 639092    0    0     0  3324 2379 1776  5  3  0 93
 0  1      0  52404 153896 708316    0    0     4 97656 2444 2638 15 15 28 43
 0  1      0  47240 129372 741900    0    0     0 45056 2303 3838 14  7 35 44
 0  1      0  54340 105140 764532    0    0     4 54448 2421 2197 13  9 39 41
 0  2      0  55812  86952 785592    0    0     0 36864 2367 1044  5  6 30 59
 0  3      0  49640  86988 792596    0    0     0 33152 2443 2901 10  7  0 84
 1  2      0  50504  87040 794124    0    0    12 37156 2329 1973  6  9 14 73
 0  3      0  49824  87080 797232    0    0     0 40960 2509 2561  6  7  0 87
 0  2      0  50256  87124 798252    0    0    24 13428 2453 1503  5  6 14 76
 2  1      0  55072  87156 795552    0    0     4  7088 2473 1326  4  5 45 46
 1  2      0  50712  87184 800392    0    0     4 21624 2644 2617 19  7 30 44
 0  2      0  48268  87196 803428    0    0     0 29124 2636 1133  7  2 46 45
 0  3      0  47920  87216 803532    0    0     0 24336 2601 3347  7  5 42 46
 0  3      0  56564  87224 796416    0    0     0 23412 2669 2766  7  6 33 54
 0  3      0  58280  87224 796392    0    0    68     0 2557 3037 17  8  0 75
 0  3      0  58232  87268 796492    0    0   104     0 2466 3076 16  4  0 80
 0  4      0  59392  87344 796480    0    0    76     0 2524 3708 20  5  0 76
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  2      0  50636  87484 807852    0    0    64 61508 2465 2460 10 10  4 76
 0  2      0  52516  87496 806876    0    0     0 25232 2483 1898 10  4 39 48
 0  2      0  52644  87496 806916    0    0     0     0 2641 1766  7  2 49 42
 0  2      0  52348  87496 807108    0    0     0   160 2620 1122  3  1 49 47
 0  3      0  53228  87496 807108    0    0     0     0 2491 1744  5  2 49 44
 0  2      0  57272  87600 806092    0    0    24 48020 2465 1722  4 10 47 40
 0  3      0  48992  87612 814304    0    0     0  5980 2526 1696  8  2 42 49
 0  3      0  56064  85352 810348    0    0     8 15036 2558 1535 15  3 34 48
 0  4      0  57768  83828 810872    0    0     0  8460 2553 1137  3  3 42 52
 1  1      0  54124  83848 814984    0    0     4 15760 2679 1598  5  3  5 88
 0  3      0  51100  82732 819376    0    0     0 24284 2607 1536  4  4 46 48
 0  3      0  69984  78712 805060    0    0     4 23048 2635 1258 13  3 25 60
 0  8      0  52248  78756 821672    0    0    96 22124 2642 3388 17  8  0 75
 0  4      0  52148  78776 822000    0    0   292  2972 2527 2713 18  6  0 76
 0  4      0  54868  78780 821984    0    0    32     0 2618 2962 10  4 25 62
 0  4      0  55708  78780 822016    0    0     0     0 2517 3060  8  1 44 47
 0  3      0  47380  72296 839272    0    0     0 15560 2649 2018  4  8 46 43
 2  3      0  52012  70824 835960    0    0    64 77860 2541 2106  9  9 18 64
 0  2      0  55880  68868 835860    0    0    12 20484 2451 3162 12  8 40 41
 0  3      0  56344  67276 837788    0    0   240 11748 2481 1384  5  3 46 46
 0  3      0  55128  65556 841540    0    0    24 16196 2507 1786 16  5 44 36
 1  3      0  52804  64456 845768    0    0     0 19748 2533 1482  5  4 45 47
 0  4      0  53572  62740 847640    0    0     4 24176 2649 1202  5  4 23 67
 0  3      0  51244  60244 852116    0    0     0 30704 2542 2561  8  6 12 74
 0  2      0  54456  59424 851808    0    0     4  8256 2532 2450  8  6  7 80
 0  2      0  55576  59424 851840    0    0     0     0 2575 2209 19  2  0 79
 0  3      0  56164  59424 851840    0    0     0     0 2471 3640 11  2  0 87
 2  1      0  48676  56748 866280    0    0     4 10872 2475 2757 17  9  2 72
 0  3      0  56768  54396 858564    0    0   124 76112 2519 1672  9  8 35 48
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  2      0  49508  53776 867584    0    0    20 16384 2469 1745  4  3 26 67
 0  2      0  48980  53032 869128    0    0     4 24608 2466 1533  3  5  0 93
 0  2      0  54564  52200 864996    0    0     0 20480 2436 1591  6  4  0 91
 0  2      0  51440  51780 868884    0    0     0 20672 2570 2602  7  7  0 86
 0  2      0  48976  51708 872124    0    0     0 16924 2462 1791 14  4 19 64
 0  2      0  53488  51376 868228    0    0    56 25276 2536 3062 11  7 36 47
 0  3      0  56644  51140 866520    0    0    32 28672 2411 2718  7  6 42 44
 1  2      0  52600  50792 871912    0    0     0 20912 2527 1773  3  5 46 47
 0  3      0  49816  50436 876080    0    0     0 32832 2414 1409  4  5 46 46
 1  2      0  56932  49776 870756    0    0   152 16484 2437 1917 12  6 42 42
 0  3      0  50008  49264 877636    0    0     0 27508 2398 2604 17  5 34 44
 1  3      0  49268  48660 879288    0    0     0 22032 2483 3657 15  4 33 47
 1  2      0  55920  48280 874544    0    0     0 15996 2588 1675 11  4 42 44
 0  2      0  49308  47588 882680    0    0     0 20576 2542 1738  6  3 43 48
 0  3      0  48116  47020 884992    0    0   428 32800 2402 1418  3  5 40 51
 0  2      0  49224  46772 884684    0    0   224 16384 2546 1709  3  4 48 46
 0  2      0  50120  46308 884128    0    0     0 20480 2527 2434  7  5 45 44
 1  2      0  52424  46112 884668    0    0     0 29096 2468 1398  4  5 44 46
 0  2      0  52424  45588 886340    0    0     0 28672 2391 1640 16  5 37 43
 2  2      0  52748  44236 888712    0    0     4 36864 2394 1378  4  7 45 45
 0  3      0  51596  43988 891304    0    0     0 33180 2526 1548  4  6 45 45
 0  3      0  51756  43216 892264    0    0     0 20608 2532 2179  4  4 47 45
 2  2      0  50040  42712 892708    0    0     4 16388 2458 2478  5  5 47 43
 2  2      0  50900  42080 893400    0    0     4 28700 2408 2882 22  7 36 34
 4  1      0  50544  41580 895708    0    0   360  8200 2407 3910 18  7 34 41
 1  1      0  51976  41580 895708    0    0     0     0 2419 4167 24  4 38 34
 0  1      0  53244  41580 895708    0    0     0     0 2526 1318  5  2 49 45
 0  0      0  54668  41588 895900    0    0   192  1592 2309 1955  5  2 81 13
 0  0      0  54228  41588 895900    0    0     0     0 2220  975  4  1 96  0
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  0      0  53808  41600 895944    0    0    44   220 2308 2316 10  4 77 10
 0  0      0  54708  41600 895944    0    0     0     0 2213 1551  8  1 92  0
 0  0      0  54756  41600 895944    0    0     0     0 2196 1205 12  1 86  0
 5  0      0  54628  41600 895944    0    0     0     0 2212 1578  7  1 92  0
 0  0      0  54636  41600 895944    0    0    20     0 2285 3183  9  6 85  0
 0  0      0  54740  41612 895956    0    0     0    56 2223 2116  7  3 90  0


next command:
# dd if=test1 of=test2 bs=1M
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 180.319 s, 11.6 MB/s

And this is iowait values while executing dd command:
# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  0      0  64640  41936 887224    0    0   101    73   47   11  5  2 93  1
 0  0      0  64180  41936 887256    0    0     0     0 2239 2403 11  3 86  0
 1  0      0  49624  40772 903164    0    0 22656     0 2359 2033  9  5 75 11
 0  1      0  53400  38688 902476    0    0 35584 20444 2569 2034  5  7 48 40
 0  1      0  51336  38252 905184    0    0 13568 12288 2451 1720  5  4 45 46
 0  1      0  54636  38228 901948    0    0  6912  8192 2595 1807 14  2 43 41
 0  3      0  52512  38236 904268    0    0  5376 13156 2555 2813 10  3 42 45
 1  4      0  50064  38096 907788    0    0  5376  4232 2564 2692 20  3 38 40
 0  4      0  47840  38100 909812    0    0  5376  4168 2556 1336  7  2 47 44
 1  4      0  52496  37212 905088    0    0  5248  4068 2582 2547  8  4 44 44
 1  3      0  50568  37216 907736    0    0  5504  4784 2588 1338  4  2 48 46
 0  4      0  47120  36956 911236    0    0  5248  4184 2655 4339  9  5 44 43
 0  4      0  53664  36452 905008    0    0  4736  3636 2536 1081  5  2 44 48
 0  3      0  51648  36176 908384    0    0  5632  2456 2605 1930 10  3 44 43
 0  3      0  50296  35916 910116    0    0  5504  5596 2550 1591 10  2 19 70
 0  3      0  51920  35664 908984    0    0  5120  4264 2585 2931 11  3 42 46
 0  3      0  48580  35676 912372    0    0  5632  4164 2597 1678  7  2 32 59
 0  4      0  53392  35296 907984    0    0  5120  3132 2556 1410  4  2 37 56
 4  4      0  51412  35048 910256    0    0  5120  4188 2559 1801  5  3 43 50
 0  3      0  50152  34856 912452    0    0  5120  6256 2564 2413 17  2 12 69
 0  4      0  49640  34864 913020    0    0  4740  3016 2532 3819 13  3 36 48
 0  5      0  48420  34724 915240    0    0  5120  6072 2538 2043 23  3 34 41
 2  5      0  46856  34472 917224    0    0  4864  4084 2531 1692  4  2 47 48
 0  5      0  53548  34352 911044    0    0  4868  4428 2567 2324  5  2 48 45
 0  5      0  51444  34104 913556    0    0  5376  3908 2558  698  2  2 49 47
 0  5      0  48848  33852 915820    0    0  5120 22692 2574 2145  8  3 46 43
 0  5      0  54304  33612 910164    0    0  5248  7560 2538  891  8  2 44 47
 0  2      0  47520  33616 916504    0    0  3456 29928 2629 2316  9  4 46 42
 0  2      0  54588  33248 910164    0    0  4352     0 2502 2474 12  3 47 39
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  2      0  48280  33256 917012    0    0  3840   216 2481 1474  8  2 49 42
 1  2      0  54732  32748 908008    0    0  3584     0 2460 2140  8  6 43 44
 0  2      0  49180  32756 916624    0    0  4608 14464 2510 4355 15  9 36 41
 1  1      0  51216  31764 918536    0    0 25348   188 2505 2478 11  6 45 39
 1  1      0  49680  30888 920896    0    0 21504 11328 2500 3981 19  6 41 34
 0  1      0  51012  30132 920688    0    0 15360 12288 2412 3577 22  5 41 32
 0  1      0  50352  29760 922588    0    0 12800 16384 2512 1374  9  4 44 44
 0  2      0  51964  29384 921700    0    0  5888 41920 2476 1471  5  2 44 50
 0  2      0  52580  29128 921636    0    0  2816     0 2418 1509  5  2 45 48
 0  1      0  48372  27896 928608    0    0 36356  3356 2494 2527  4  9 46 43
 4  1      0  52372  27272 925432    0    0 13568 16384 2477 3519 16  6 40 38
 0  1      0  47640  26648 931064    0    0 16128 16384 2477 2965  8  7 45 40
 0  2      0  50888  25884 929544    0    0 14848 48080 2413 1499  8  5 36 51
 2  2      0  54660  25632 926460    0    0  2560     0 2426 4070 14  5  0 82
 2  0      0  54020  23480 930876    0    0 40708  6132 2485 2200  5  8 46 42
 0  1      0  50724  23384 934772    0    0  9472 11456 2452 2097 10  3 42 45
 0  1      0  48044  22632 938392    0    0 17664 16384 2459 1979  5  5 43 46
 1  2      0  48104  22524 938312    0    0 12032 48080 2517 2377 15  4 40 40
 0  2      0  51756  22324 935620    0    0  3072     0 2393 3089 19  2 46 34
 0  1      0  50408  21808 939052    0    0 22016   136 2427 3106 17  6 40 37
 0  1      0  52892  20768 937852    0    0 27136 17456 2413 1281  4  6 47 44
 0  1      0  52564  20024 938944    0    0 16516 16384 2436 3218  7  6 43 44
 0  2      0  50848  19652 940668    0    0 13620 49112 2403 2427  8  6 29 58
 0  2      0  52716  19656 939980    0    0  3456     0 2433  715  1  2 47 50
 0  1      0  49648  19452 944264    0    0 34688   172 2528 3378 15  9 39 38
 0  1      0  48864  18956 945448    0    0 12032 12288 2515 3264 13  8 39 41
 2  1      0  52532  18228 942916    0    0 15108 12288 2448 1879 11  4 43 43
 0  2      0  48228  18000 946816    0    0 13824 54248 2406 2125  7  5 41 48
 0  2      0  51076  17964 944912    0    0  3072     0 2453 1777  4  3  0 93
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  1      0  51216  17560 946564    0    0 37760  3220 2590 2621  6  8 33 54
 0  1      0  47264  17316 950740    0    0 18180 16384 2420 1585  4  4 48 43
 1  1      0  51516  17056 946620    0    0 13952 16384 2413 2643 18  5 35 43
 0  1      0  47984  17068 949924    0    0 12548 12288 2438 3094 20  4 32 43
 0  3      0  47472  17080 950476    0    0  3636 38624 2413 2325 13  3 38 47
 0  3      0  48288  17084 949528    0    0  3072   124 2456 2436  4  3 46 46
 0  3      0  49564  17088 948884    0    0  3456     0 2527 2520  7  5 45 44
 0  1      0  51220  16832 948700    0    0 31288  3104 2660 2575  7  9 45 39
 0  1      0  51640  16548 948912    0    0 16256 16384 2395 1637  9  4 47 41
 0  2      0  53580  16548 946300    0    0  3584 40932 2420 1333  6  6 45 44
 1  1      0  51960  16400 949276    0    0  8960   108 2549 1775  6  3 50 41
 0  1      0  53956  16264 947664    0    0 23808  2728 2832 2738  9  6 46 40
 0  1      0  54016  16240 947368    0    0  7808  8192 2603 2266  5  3 44 48
 0  2      0  47368  16144 954308    0    0 10884 12288 2617 2109  5  3 46 47
 0  3      0  49712  16004 951652    0    0  6424 33112 2465 1665  5  5 45 45
 1  2      0  52440  16020 949580    0    0  6088     0 2635 2335  6  2 49 43
 2  2      0  51396  16048 950440    0    0 18052 10768 2694 3160 26  5 31 39
 0  1      0  51584  16064 950452    0    0 11016  8192 2754 3579  8  4 38 49
 0  1      0  52780  16076 949476    0    0 14208 16384 2685 2220 11  4 43 43
 0  2      0  47896  16068 955044    0    0 10624 24148 2560 1105  1  6 46 47
 0  1      0  51592  15708 952080    0    0 15008   100 2640 2295  3  4 45 47
 0  1      0  52972  14968 951024    0    0 27784  3852 2761 2287  9  6 42 43
 0  1      0  49392  15000 954764    0    0  5792  8192 2555 1936  6  2 44 47
 0  1      0  48928  14832 955084    0    0 12316 16384 2562 1899  8  4 38 51
 0  2      0  54908  14592 948648    0    0  5516 36868 2587 1431  7  4 28 62
 0  2      0  50632  14480 952824    0    0  6144  4096 2639 2211  7  3 47 44
 0  2      0  47036  14488 956568    0    0  5760  8192 2577 1511  5  2 49 45
 0  2      0  48008  14368 956472    0    0  3840  4220 2476 1369  3  2 49 45
 0  2      0  48324  14244 956448    0    0  7040  8192 2607 1747  6  3 49 42
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 4  3      0  51084  13996 954160    0    0  6784  6260 2632 1895 18  2 33 46
 1  3      0  49292  14004 956560    0    0  5380  4256 2586 3278 14  3  0 84
 2  2      0  49472  13956 955268    0    0 11764  5408 2505 3665 11  4 34 50
 2  1      0  48412  14000 954804    0    0 13736 11440 2565 1985 13  7 40 39
 2  1      0  52832  14008 952396    0    0 11008    84 2709 3244  9  8 44 40
 0  1      0  47784  14024 957180    0    0 20100  5072 2675 2111  4  5 47 44
 0  1      0  52112  13980 952832    0    0  7680  8192 2733 2427  7  4 46 44
 0  1      0  52804  13804 952200    0    0 11520 12288 2632 2022  8  4 47 42
 1  2      0  47816  13680 956464    0    0  6272 35896 2655 1631  8  3 41 49
 0  2      0  47488  13560 957428    0    0  4864     0 2601 1758  6  2 45 46
 0  2      0  48472  13444 957608    0    0  4224   760 2551 1690  4  2 49 45
 1  0      0  48220  13320 958068    0    0  9988     0 2618 2442  5  4 46 45
 0  1      0  50872  13164 955644    0    0 17152  5064 2688 2415  5  4 46 45
 1  2      0  54092  13068 951700    0    0  7812 41016 2516 1652  7  4 44 45
 3  2      0  48064  13076 958200    0    0  7584  8136 2720 2155 20  4 40 36
 0  2      0  47156  13084 959820    0    0  3580  4236 2441 2947 10  2 41 48
 0  1      0  52192  13056 956104    0    0 10624 12292 2515 3222 14  4 39 44
 0  1      0  50600  13036 957028    0    0  7956  4096 2669 2493  6  5 44 46
 0  3      0  49400  13048 958740    0    0  6920 10304 2700 2190  4  3 37 55
 0  3      0  53744  13072 954764    0    0  6220     0 2576 1135  7  2 47 44
 0  1      0  52404  13076 955776    0    0 10284    76 2590 2016  7  4 45 45
 0  1      0  52792  13088 954608    0    0 26100  3932 2789 2979 10  6 43 42
 0  1      0  48584  13104 958944    0    0 16384 16384 2503 1463  5  4 48 43
 0  2      0  49880  13112 957392    0    0  9472 44200 2463 1736  6  4 40 50
 0  1      0  53268  13124 954844    0    0  7552   136 2550 1627  4  3 45 48
 1  1      0  51000  13092 956952    0    0 33064   856 2784 2478  7  7 43 44
 0  1      0  48588  13096 959508    0    0  7680 12288 2642 2236  6  3 45 47
 0  2      0  50148  12972 956160    0    0  8596 41292 2595 3174 21  6 27 47
 0  2      0  53240  12972 954728    0    0  3584     0 2472 1775 14  2 42 42
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  1      0  51428  12476 957896    0    0 25604   124 2741 3998  9  6 40 45
 0  1      0  48284  12048 962492    0    0 18304 12112 2725 3184 13  5 37 45
 0  1      0  54168  11804 956016    0    0  9476 12288 2599 1513  2  3 48 48
 0  2      0  54632  11748 954856    0    0 11052 43188 2686 2457  9  4 38 49
 0  2      0  46708  11752 963160    0    0  4300     0 2491 1011  3  2 45 51
 0  1      0  53340  11536 958596    0    0 14592   136 2571 2167  8  5 42 45
 0  1      0  52252  11528 958880    0    0 26880 10060 2602 2136  8  6 49 38
 0  1      0  53828  11460 957488    0    0  9732 12288 2631 1930  3  3 49 46
 0  2      0  49020  11336 961692    0    0 10296 45236 2546 1973  6  4 42 48
 0  2      0  50796  11084 959896    0    0  7084     0 2705 1688  5  2 48 45
 0  2      0  48272  11008 962376    0    0  7552     0 2670 3058 10  5 43 43
 0  1      0  50056  11024 961372    0    0 17024   104 2609 2923 18  6 37 38
 0  1      0  47776  10972 963492    0    0 16384 12108 2587 2832  8  6 45 42
 0  3      0  48976  10952 961192    0    0  9636 46264 2490 2068 15  4 34 47
 0  2      0  48356  10952 962772    0    0  4444     4 2525 3121  9  2 28 60
 0  1      0  48008  10948 964488    0    0 16772    96 2625 2604 12  5 47 38
 0  2      0  52768  10996 959568    0    0 18068  3592 2617 1480  5  4 43 48
 1  1      0  51064  11028 961192    0    0  9756  8192 2523 2634  7  3 42 48
 0  2      0  47892  10868 963628    0    0 13440 51384 2506 1456  5  4 29 61
 0  2      0  48076  10864 963800    0    0  4352     0 2506 1663  9  2  0 89
 0  1      0  46840  10876 966088    0    0 31876   104 2658 1928  3  7 35 54
 0  1      0  47516  10668 964384    0    0 14464 49152 2582 1177  2  4 47 47
 0  2      0  47408  10684 964076    0    0  3820     0 2426 1880  8  2 25 66
 0  3      0  50776  10628 961460    0    0  6492 11264 2427 1346  5  3 46 47
 0  2      0  49664  10628 963424    0    0  5164     0 2535 1920  7  2 43 47
 0  1      0  49024  10216 964668    0    0 29444    96 2689 2106 14  7 44 36
 0  1      0  50720  10188 963104    0    0 15744  9216 2551 1325  4  4 49 44
 0  1      0  51316  10176 962420    0    0  5508  8192 2574 2701 16  3 44 38
 0  1      0  52828  10176 960684    0    0  8512  8192 2585 3111  8  3 49 39
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  2      0  47376  10184 965484    0    0  4736 39936 2620 2569 13  3 44 40
 0  2      0  53236  10180 960088    0    0  4736     0 2598 1145  6  2 47 44
 1  1      0  50140  10200 963712    0    0 24960   100 2811 2740  5  5 49 42
 0  1      0  52412  10220 961712    0    0 11972  7204 2831 2829  7  4 30 60
 0  1      0  53016  10184 960800    0    0 14344 12288 2490 2221  8  4 42 47
 0  4      0  52360  10240 959764    0    0  9868 45028 2570 2050  6  6 30 57
 1  2      0  51496  10252 958804    0    0  4260   112 2507 1362  4  3 41 53
 0  2      0  47324  10264 966612    0    0  6664   212 2662 3194 11  8 41 40
 0  2      0  49372  10276 963924    0    0  5516     4 2655 2908 10  6 40 45
 0  2      0  51076  10288 962384    0    0  5636 23652 2504 1683 15  3 39 43
 0  2      0  47040  10280 966840    0    0  5380     0 2623 2618  7  5 44 44
 0  2      0  47916  10296 966008    0    0  9292   176 2594 2560  6  5 46 43
 0  2      0  53552  10316 960840    0    0 11932     4 2517 3408 16  4 39 41
 0  1      0  52020  10352 962216    0    0 22928  9124 2635 3990 11  6 45 40
 0  2      0  52376  10380 960776    0    0 13712 50272 2539 2213 17  5 36 42
 0  2      0  52740  10384 960752    0    0  6152     0 2632 1420  5  2 47 45
 0  2      0  47256  10392 966960    0    0  6148     0 2641 2170  9  2 47 42
 1  3      0  51800  10456 963156    0    0  7492   208 2399 1551  7  2 42 49
 0  1      0  53360  10496 961396    0    0 30028  7072 2537 2540  8  6 44 42
 0  1      0  50200  10512 964180    0    0 13956 16384 2435 1941  8  4 46 43
 0  2      0  48560  10512 965496    0    0  4352 39012 2529 1615  5  3  2 91
 0  1      0  52716  10528 962136    0    0  7936    96 2497 2413  9  3  6 83
 1  1      0  53580  10396 960688    0    0 32020  1948 2670 2396 13  7 39 42
 0  1      0  52244  10416 962120    0    0 12684 16384 2597 2020  8  4 44 44
 0  1      0  52272  10424 961772    0    0  6664  8192 2714 2096  6  2 46 46
 0  3      0  55180  10424 957824    0    0  5124 37996 2578 1198 10  3 30 58
 0  2      0  51432  10440 961892    0    0  6152  2964 2664 2466 12  4 42 41
 0  2      0  47524  10448 965932    0    0  6148 12388 2586 2888 14  5 44 39
 1  2      0  49712  10460 964596    0    0  3716    96 2490 2149 14  2 41 43
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 3  2      0  52728  10476 961864    0    0  6536     0 2662 1852  7  2 50 41
 1  1      0  47340  10496 968208    0    0 19608 15612 2573 2264  9  5 43 44
 0  1      0  53072  10504 961972    0    0  9224  8192 2617 2952  8  6 43 43
 0  1      0  50656  10512 964312    0    0  8456 24852 2525 1745  6  4 49 42
 0  1      0  50316  10556 964096    0    0 20248     0 2549 2838  7  6 43 44
 1  1      0  49424  10524 965100    0    0 21536  4096 2468 2512  8  7 42 43
 0  3      0  52544  10508 962312    0    0 14556 26272 2620 1974  6  4 41 48
 0  5      0  53084  10520 960576    0    0  6396  3580 2729 2672 18  5 38 40
 0  3      0  50276  10544 965148    0    0  5588  3444 2681 1351  3  3 47 48
 0  3      0  50116  10544 965236    0    0     0 15864 2843 1831  4  1 49 45
 0  1      0  50472  10544 965236    0    0     0     8 3026 1396 13  2 42 44
 0  1      0  50440  10552 965236    0    0     0 16528 2467 3885 13  4 64 19
 1  0      0  51968  10560 965228    0    0     0    80 2574 3064 20  4 52 23
 1  0      0  52168  10560 965436    0    0   208     0 2236 1233  4  1 94  2
 2  0      0  52172  10560 965444    0    0     0     0 2224 2144  8  2 90  0
 0  0      0  50744  10592 967412    0    0  2012     0 2339 1769  8  1 78 14
 0  0      0  50492  10592 967424    0    0     0     0 2203 1556  5  1 95  0
 0  0      0  50492  10600 967416    0    0     0    56 2232 1226  4  1 95  0


As for me, results looks exactly like in most reports in this ticket.

As for hardware itself - it's ok, in Windows file copying speed is about 60MB/s.
Comment 32 Alex Efros 2009-11-23 12:33:18 UTC
As for hdparm - it works as expected, no problems here:
# hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   6292 MB in  2.00 seconds = 3150.38 MB/sec
 Timing buffered disk reads:  230 MB in  3.01 seconds =  76.33 MB/sec
Comment 33 Alan 2009-11-23 13:48:08 UTC
Right so the drive and hardware performance is just fine.

You didn't answer what file system you are using
Comment 34 Alex Efros 2009-11-23 14:25:17 UTC
> Right so the drive and hardware performance is just fine.

Probably I misunderstood what you mean. If you wanna say copying files
with 10MB/sec is just fine, and overall system slowdown while copying 
files also just fine, then I can't agree with you. Especially because
it's modern enough system, and kernels before 2.6.18 works much faster,
and windows also works at full speed without any interface slowdown.

> You didn't answer what file system you are using

Because that doesn't matter much. I've tried different file systems
(ext3, reiserfs and vfat/fat32), and different configurations of i/o
scheduler and vfs params as shown in Perlover's hint in this ticket's comment.
In all cases it's copy files very slow, and also interface slow while copying too.

In above examples I run dd on my root partition (ext3). Here is dd executed
on vfat(fat32) partition:

home /mnt/G # dd if=/dev/zero of=test1 bs=1M count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 70.333 s, 29.8 MB/s
home /mnt/G # dd if=test1 of=test2 bs=1M
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 355.844 s, 5.9 MB/s
home /mnt/G # time rm test1 test2 
real    0m0.785s
user    0m0.000s
sys     0m0.197s

The iowait in vmstat output while these tests on fat32 partition was same as
in above tests on ext3 partition.

All these tests was executed with settings recommended by Perlover above:

home ~ # cat /sys/block/sda/queue/nr_requests
1024
home ~ # cat /sys/block/sda/device/queue_depth
1
home ~ # cat /sys/block/sda/queue/scheduler
noop anticipatory [deadline] cfq 
home ~ # cat /proc/sys/vm/vfs_cache_pressure
50

Usually I use [cfq] and default values for others, but it makes no difference.
Comment 35 Alex Efros 2009-11-23 14:40:04 UTC
And here is dd output from my friend's system which doesn't have that bug
(Core 2 Quad Q9300 on Gigabyte EP35C-DS3R, with HDD SAMSUNG HD103UJ):

home /mnt/ext3 # dd if=/dev/zero of=test1 bs=1M count=8000
8000+0 records in
8000+0 records out
8388608000 bytes (8.4 GB) copied, 96.9127 s, 86.6 MB/s
home /mnt/ext3 # dd if=test1 of=test2 bs=1M
8000+0 records in
8000+0 records out
8388608000 bytes (8.4 GB) copied, 186.208 s, 45.0 MB/s
home /mnt/ext3 # hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   3730 MB in  2.00 seconds = 1865.60 MB/sec
 Timing buffered disk reads:  342 MB in  3.01 seconds = 113.76 MB/sec

And I wanna MY system to work on same speed!!! :-(
Comment 36 Jens Axboe 2009-11-23 18:30:43 UTC
Could you post your kernel .config?
Comment 37 Yuriy Lalym 2009-11-23 22:35:49 UTC
Hardware and FS (comment #3)
New kernel (openSUSE-11.2)
suse:/home/yura # uname -r
2.6.31.5-0.1-desktop

Ok

//////////////////////////////
kernel 2.6.30-rc6
//////////////////////////////
yura@suse:~> dd if=/dev/zero of=test1 bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 129.928 s, 80.7 MB/s

//////////////////////////////
2.6.31.5-0.1-desktop
//////////////////////////////
suse:/home/yura # dd if=/dev/zero of=test1 bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 116.863 s, 89.7 MB/s

############################################################################

//////////////////////////////
kernel 2.6.30-rc6
//////////////////////////////
yura@suse:~> time cp bigfile bigfile.cp

real    5m52.457s
user    0m0.343s
sys     0m21.356s

//////////////////////////////
2.6.31.5-0.1-desktop
//////////////////////////////
time cp test1 test2

real    4m21.466s
user    0m0.354s
sys     0m22.326s

//////////////////////////////
The difference 1m 31s (25.8%), from the sky has fallen
//////////////////////////////

############################################################################

//////////////////////////////
kernel 2.6.30-rc6
//////////////////////////////
time cp -r qt-x11-opensource-src-4.5.1 qt-x11-opensource-src-4.5.1-1
real    5m51.075s
user    0m0.147s
sys     0m2.192s

302.6Mb / 351s = 0.9 Mb/s

//////////////////////////////
2.6.31.5-0.1-desktop
//////////////////////////////
time cp -r qt-everywhere-opensource-src-4.6.0-rc1 qt-everywhere-opensource-src-4.6.0-rc12

real    8m1.028s
user    0m0.232s
sys     0m5.614s

In directiry qt-everywhere-opensource-src-4.6.0-rc1 32110 files - all size 1248222969
1190Mb/481s = 2.47 Mb/s (Scrolling in firefox brakes !!!???)

//////////////////////////////
Speed has increased in 2.5 times, at the expense of what?
//////////////////////////////

############################################################################

And I again wish to ask, and where these speeds have got to (http://linuxgazette.net/122/TWDT.html#piszcz)
Comment 38 Yuriy Lalym 2009-11-23 22:50:31 UTC
For absolute cleanliness it is necessary to consider, when  tested the kernel 2.6.30 disk was has been occupied on 10% - now on 60%
Comment 39 Yuriy Lalym 2009-11-23 23:54:26 UTC
1. A graphic interface works quickly, while the operations of files are not executed.
2. At copying of the big directory in a browser page scrolling jams. Start of the new application becomes very slow.
3. If I compare old tests of speed of work of a file system (found on spaces of the internet) with measured by me - there is a wild mismatch (my speeds much less).
4. I always see iowait-> 100%

If you consider it as different errors - probably. But I think that it is one error and its consequences. And I see improvements in a kernel 2.6.31 - but not on so much to remove this question.
Comment 40 Alan 2009-11-24 00:14:00 UTC
100% iowait is expected - your computer is faster than the disk. The rest is not.

How much memory does the system have and is it running a 32 or 64bit kernel ?


#35 is what I'd expect to see - copy performance is lower than raw disk performance because the disks then have to seek more and seeking is very slow (doubly so on a laptop drive). The #35 numbers look sensible, yours do not.

A very fragmented disk would see performance drop but the numbers look to bad for that to be the cause and you've also tried different file systems.

[I'm basically trying to work out what is different between your set up which has problems and your friends set up which does not]
Comment 41 Alex Efros 2009-11-24 02:34:19 UTC
Here is my .config:

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.28-hardened-r9
# Fri Jun 19 18:59:48 2009
#
# CONFIG_64BIT is not set
CONFIG_X86_32=y
# CONFIG_X86_64 is not set
CONFIG_X86=y
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/i386_defconfig"
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_FAST_CMPXCHG_LOCAL=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
# CONFIG_GENERIC_TIME_VSYSCALL is not set
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
# CONFIG_HAVE_CPUMASK_OF_CPU_MAP is not set
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
# CONFIG_ZONE_DMA32 is not set
CONFIG_ARCH_POPULATES_NODE_MAP=y
# CONFIG_AUDIT_ARCH is not set
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_X86_SMP=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_X86_32_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_KTIME_SCALAR=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"

#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
# CONFIG_POSIX_MQUEUE is not set
# CONFIG_BSD_PROCESS_ACCT is not set
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
# CONFIG_AUDIT is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=17
# CONFIG_CGROUPS is not set
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_GROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
# CONFIG_RT_GROUP_SCHED is not set
CONFIG_USER_SCHED=y
# CONFIG_CGROUP_SCHED is not set
# CONFIG_SYSFS_DEPRECATED_V2 is not set
# CONFIG_RELAY is not set
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
# CONFIG_IPC_NS is not set
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
# CONFIG_BLK_DEV_INITRD is not set
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
# CONFIG_COMPAT_BRK is not set
CONFIG_SLAB=y
# CONFIG_SLUB is not set
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
# CONFIG_MARKERS is not set
CONFIG_HAVE_OPROFILE=y
# CONFIG_KPROBES is not set
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_GENERIC_DMA_COHERENT=y
CONFIG_RT_MUTEXES=y
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
# CONFIG_LBD is not set
# CONFIG_BLK_DEV_IO_TRACE is not set
# CONFIG_LSF is not set
# CONFIG_BLK_DEV_BSG is not set
# CONFIG_BLK_DEV_INTEGRITY is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_CLASSIC_RCU=y
# CONFIG_FREEZER is not set

#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_VSMP is not set
# CONFIG_X86_RDC321X is not set
CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y
# CONFIG_PARAVIRT_GUEST is not set
# CONFIG_MEMTEST is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
CONFIG_MCORE2=y
# CONFIG_GENERIC_CPU is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CPU=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_ALIGNMENT_16=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=4
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_CYRIX_32=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR_32=y
CONFIG_CPU_SUP_TRANSMETA_32=y
CONFIG_CPU_SUP_UMC_32=y
# CONFIG_X86_DS is not set
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
# CONFIG_IOMMU_HELPER is not set
CONFIG_NR_CPUS=2
# CONFIG_SCHED_SMT is not set
CONFIG_SCHED_MC=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
# CONFIG_X86_MCE_P4THERMAL is not set
CONFIG_VM86=y
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_X86_REBOOTFIXUPS is not set
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_HIGHMEM=y
# CONFIG_ARCH_PHYS_ADDR_T_64BIT is not set
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_SPARSEMEM_STATIC=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_RESOURCES_64BIT is not set
# CONFIG_PHYS_ADDR_T_64BIT is not set
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_UNEVICTABLE_LRU=y
# CONFIG_HIGHPTE is not set
# CONFIG_X86_CHECK_BIOS_CORRUPTION is not set
CONFIG_X86_RESERVE_LOW_64K=y
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
# CONFIG_MTRR_SANITIZER is not set
CONFIG_X86_PAT=y
# CONFIG_EFI is not set
CONFIG_SECCOMP=y
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_SCHED_HRTICK=y
# CONFIG_KEXEC is not set
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x200000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x100000
# CONFIG_HOTPLUG_CPU is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

#
# Power management and ACPI options
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
# CONFIG_SUSPEND is not set
# CONFIG_HIBERNATION is not set
CONFIG_ACPI=y
# CONFIG_ACPI_PROCFS is not set
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_ACPI_SYSFS_POWER=y
CONFIG_ACPI_PROC_EVENT=y
# CONFIG_ACPI_AC is not set
# CONFIG_ACPI_BATTERY is not set
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_THERMAL=y
# CONFIG_ACPI_WMI is not set
# CONFIG_ACPI_ASUS is not set
# CONFIG_ACPI_TOSHIBA is not set
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
# CONFIG_ACPI_PCI_SLOT is not set
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
# CONFIG_ACPI_CONTAINER is not set
# CONFIG_ACPI_SBS is not set

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set
# CONFIG_CPU_IDLE is not set

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
# CONFIG_PCI_GOOLPC is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
CONFIG_PCIEAER=y
# CONFIG_PCIEASPM is not set
CONFIG_ARCH_SUPPORTS_MSI=y
# CONFIG_PCI_MSI is not set
CONFIG_PCI_LEGACY=y
CONFIG_HT_IRQ=y
CONFIG_ISA_DMA_API=y
# CONFIG_ISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set
# CONFIG_OLPC is not set
# CONFIG_PCCARD is not set
# CONFIG_HOTPLUG_PCI is not set

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
CONFIG_HAVE_AOUT=y
CONFIG_BINFMT_AOUT=y
# CONFIG_BINFMT_MISC is not set
CONFIG_HAVE_ATOMIC_IOMAP=y
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
# CONFIG_IP_FIB_TRIE is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
# CONFIG_INET_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
# CONFIG_IPV6 is not set
# CONFIG_NETLABEL is not set
# CONFIG_NETWORK_SECMARK is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_NETFILTER_ADVANCED=y

#
# Core Netfilter Configuration
#
# CONFIG_NETFILTER_NETLINK_QUEUE is not set
# CONFIG_NETFILTER_NETLINK_LOG is not set
CONFIG_NF_CONNTRACK=y
# CONFIG_NF_CT_ACCT is not set
# CONFIG_NF_CONNTRACK_MARK is not set
# CONFIG_NF_CONNTRACK_EVENTS is not set
# CONFIG_NF_CT_PROTO_DCCP is not set
CONFIG_NF_CT_PROTO_GRE=y
# CONFIG_NF_CT_PROTO_SCTP is not set
# CONFIG_NF_CT_PROTO_UDPLITE is not set
# CONFIG_NF_CONNTRACK_AMANDA is not set
CONFIG_NF_CONNTRACK_FTP=y
# CONFIG_NF_CONNTRACK_H323 is not set
CONFIG_NF_CONNTRACK_IRC=y
# CONFIG_NF_CONNTRACK_NETBIOS_NS is not set
CONFIG_NF_CONNTRACK_PPTP=y
# CONFIG_NF_CONNTRACK_SANE is not set
# CONFIG_NF_CONNTRACK_SIP is not set
# CONFIG_NF_CONNTRACK_TFTP is not set
# CONFIG_NF_CT_NETLINK is not set
# CONFIG_NETFILTER_TPROXY is not set
CONFIG_NETFILTER_XTABLES=y
# CONFIG_NETFILTER_XT_TARGET_CLASSIFY is not set
# CONFIG_NETFILTER_XT_TARGET_CONNMARK is not set
# CONFIG_NETFILTER_XT_TARGET_DSCP is not set
CONFIG_NETFILTER_XT_TARGET_MARK=y
# CONFIG_NETFILTER_XT_TARGET_NFLOG is not set
# CONFIG_NETFILTER_XT_TARGET_NFQUEUE is not set
# CONFIG_NETFILTER_XT_TARGET_RATEEST is not set
CONFIG_NETFILTER_XT_TARGET_TCPMSS=y
# CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP is not set
# CONFIG_NETFILTER_XT_MATCH_COMMENT is not set
# CONFIG_NETFILTER_XT_MATCH_CONNBYTES is not set
# CONFIG_NETFILTER_XT_MATCH_CONNLIMIT is not set
# CONFIG_NETFILTER_XT_MATCH_CONNMARK is not set
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
# CONFIG_NETFILTER_XT_MATCH_DCCP is not set
CONFIG_NETFILTER_XT_MATCH_DSCP=y
# CONFIG_NETFILTER_XT_MATCH_ESP is not set
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=y
CONFIG_NETFILTER_XT_MATCH_HELPER=y
CONFIG_NETFILTER_XT_MATCH_IPRANGE=y
# CONFIG_NETFILTER_XT_MATCH_LENGTH is not set
CONFIG_NETFILTER_XT_MATCH_LIMIT=y
CONFIG_NETFILTER_XT_MATCH_MAC=y
CONFIG_NETFILTER_XT_MATCH_MARK=y
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
CONFIG_NETFILTER_XT_MATCH_OWNER=y
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=y
# CONFIG_NETFILTER_XT_MATCH_QUOTA is not set
# CONFIG_NETFILTER_XT_MATCH_RATEEST is not set
# CONFIG_NETFILTER_XT_MATCH_REALM is not set
# CONFIG_NETFILTER_XT_MATCH_RECENT is not set
# CONFIG_NETFILTER_XT_MATCH_SCTP is not set
CONFIG_NETFILTER_XT_MATCH_STATE=y
# CONFIG_NETFILTER_XT_MATCH_STATISTIC is not set
# CONFIG_NETFILTER_XT_MATCH_STRING is not set
CONFIG_NETFILTER_XT_MATCH_TCPMSS=y
# CONFIG_NETFILTER_XT_MATCH_TIME is not set
# CONFIG_NETFILTER_XT_MATCH_U32 is not set
# CONFIG_IP_VS is not set

#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=y
CONFIG_NF_CONNTRACK_IPV4=y
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
# CONFIG_IP_NF_QUEUE is not set
CONFIG_IP_NF_IPTABLES=y
# CONFIG_IP_NF_MATCH_ADDRTYPE is not set
# CONFIG_IP_NF_MATCH_AH is not set
# CONFIG_IP_NF_MATCH_ECN is not set
# CONFIG_IP_NF_MATCH_TTL is not set
# CONFIG_IP_NF_MATCH_STEALTH is not set
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_TARGET_LOG=y
# CONFIG_IP_NF_TARGET_ULOG is not set
CONFIG_NF_NAT=y
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=y
# CONFIG_IP_NF_TARGET_NETMAP is not set
CONFIG_IP_NF_TARGET_REDIRECT=y
# CONFIG_NF_NAT_SNMP_BASIC is not set
CONFIG_NF_NAT_PROTO_GRE=y
CONFIG_NF_NAT_FTP=y
CONFIG_NF_NAT_IRC=y
# CONFIG_NF_NAT_TFTP is not set
# CONFIG_NF_NAT_AMANDA is not set
CONFIG_NF_NAT_PPTP=y
# CONFIG_NF_NAT_H323 is not set
# CONFIG_NF_NAT_SIP is not set
CONFIG_IP_NF_MANGLE=y
# CONFIG_IP_NF_TARGET_CLUSTERIP is not set
# CONFIG_IP_NF_TARGET_ECN is not set
# CONFIG_IP_NF_TARGET_TTL is not set
# CONFIG_IP_NF_RAW is not set
# CONFIG_IP_NF_SECURITY is not set
# CONFIG_IP_NF_ARPTABLES is not set
# CONFIG_IP_DCCP is not set
# CONFIG_IP_SCTP is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_NET_DSA is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
CONFIG_NET_SCHED=y

#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_MULTIQ=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_INGRESS=m

#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
# CONFIG_CLS_U32_PERF is not set
# CONFIG_CLS_U32_MARK is not set
CONFIG_NET_CLS_RSVP=m
# CONFIG_NET_CLS_RSVP6 is not set
# CONFIG_NET_CLS_FLOW is not set
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_IPT=m
# CONFIG_NET_ACT_NAT is not set
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
CONFIG_NET_ACT_SKBEDIT=m
# CONFIG_NET_CLS_IND is not set
CONFIG_NET_SCH_FIFO=y

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
# CONFIG_PHONET is not set
CONFIG_FIB_RULES=y
# CONFIG_WIRELESS is not set
# CONFIG_RFKILL is not set
CONFIG_NET_9P=y
# CONFIG_NET_9P_DEBUG is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_SYS_HYPERVISOR is not set
# CONFIG_CONNECTOR is not set
# CONFIG_MTD is not set
# CONFIG_PARPORT is not set
CONFIG_PNP=y
CONFIG_PNP_DEBUG_MESSAGES=y

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=y
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_CRYPTOLOOP=y
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
# CONFIG_BLK_DEV_RAM is not set
CONFIG_CDROM_PKTCDVD=y
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
# CONFIG_ATA_OVER_ETH is not set
# CONFIG_BLK_DEV_HD is not set
CONFIG_MISC_DEVICES=y
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_SGI_IOC4 is not set
# CONFIG_TIFM_CORE is not set
# CONFIG_TC1100_WMI is not set
# CONFIG_ICS932S401 is not set
# CONFIG_SONY_LAPTOP is not set
# CONFIG_THINKPAD_ACPI is not set
# CONFIG_INTEL_MENLOW is not set
# CONFIG_ENCLOSURE_SERVICES is not set
# CONFIG_HP_ILO is not set
# CONFIG_C2PORT is not set
CONFIG_HAVE_IDE=y
CONFIG_IDE=y

#
# Please see Documentation/ide/ide.txt for help/info on IDE drives
#
# CONFIG_BLK_DEV_IDE_SATA is not set
CONFIG_IDE_GD=y
CONFIG_IDE_GD_ATA=y
# CONFIG_IDE_GD_ATAPI is not set
CONFIG_BLK_DEV_IDECD=y
CONFIG_BLK_DEV_IDECD_VERBOSE_ERRORS=y
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDESCSI is not set
# CONFIG_BLK_DEV_IDEACPI is not set
# CONFIG_IDE_TASK_IOCTL is not set
CONFIG_IDE_PROC_FS=y

#
# IDE chipset support/bugfixes
#
# CONFIG_IDE_GENERIC is not set
# CONFIG_BLK_DEV_PLATFORM is not set
# CONFIG_BLK_DEV_CMD640 is not set
# CONFIG_BLK_DEV_IDEPNP is not set
CONFIG_BLK_DEV_IDEDMA_SFF=y

#
# PCI IDE chipsets support
#
CONFIG_BLK_DEV_IDEPCI=y
# CONFIG_IDEPCI_PCIBUS_ORDER is not set
# CONFIG_BLK_DEV_OFFBOARD is not set
CONFIG_BLK_DEV_GENERIC=y
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_CS5535 is not set
# CONFIG_BLK_DEV_HPT366 is not set
CONFIG_BLK_DEV_JMICRON=y
# CONFIG_BLK_DEV_SC1200 is not set
CONFIG_BLK_DEV_PIIX=y
# CONFIG_BLK_DEV_IT8213 is not set
# CONFIG_BLK_DEV_IT821X is not set
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_BLK_DEV_TC86C001 is not set
CONFIG_BLK_DEV_IDEDMA=y

#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
# CONFIG_SCSI_TGT is not set
# CONFIG_SCSI_NETLINK is not set
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=y
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=y
# CONFIG_CHR_DEV_SCH is not set

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
# CONFIG_SCSI_MULTI_LUN is not set
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m

#
# SCSI Transports
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
CONFIG_SCSI_LOWLEVEL=y
# CONFIG_ISCSI_TCP is not set
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_ARCMSR is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_MVSAS is not set
# CONFIG_SCSI_STEX is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLA_FC is not set
# CONFIG_SCSI_QLA_ISCSI is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set
# CONFIG_SCSI_SRP is not set
# CONFIG_SCSI_DH is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y
CONFIG_SATA_AHCI=y
# CONFIG_SATA_SIL24 is not set
CONFIG_ATA_SFF=y
# CONFIG_SATA_SVW is not set
CONFIG_ATA_PIIX=y
# CONFIG_SATA_MV is not set
# CONFIG_SATA_NV is not set
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_PROMISE is not set
# CONFIG_SATA_SX4 is not set
# CONFIG_SATA_SIL is not set
# CONFIG_SATA_SIS is not set
# CONFIG_SATA_ULI is not set
# CONFIG_SATA_VIA is not set
# CONFIG_SATA_VITESSE is not set
# CONFIG_SATA_INIC162X is not set
# CONFIG_PATA_ACPI is not set
# CONFIG_PATA_ALI is not set
# CONFIG_PATA_AMD is not set
# CONFIG_PATA_ARTOP is not set
# CONFIG_PATA_ATIIXP is not set
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
# CONFIG_PATA_CS5535 is not set
# CONFIG_PATA_CS5536 is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
CONFIG_ATA_GENERIC=y
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_IT8213 is not set
CONFIG_PATA_JMICRON=y
# CONFIG_PATA_TRIFLEX is not set
# CONFIG_PATA_MARVELL is not set
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_OLDPIIX is not set
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NINJA32 is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_NS87415 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_RADISYS is not set
# CONFIG_PATA_RZ1000 is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SERVERWORKS is not set
# CONFIG_PATA_PDC2027X is not set
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
# CONFIG_PATA_VIA is not set
# CONFIG_PATA_WINBOND is not set
# CONFIG_PATA_SCH is not set
CONFIG_MD=y
# CONFIG_BLK_DEV_MD is not set
CONFIG_BLK_DEV_DM=y
# CONFIG_DM_DEBUG is not set
CONFIG_DM_CRYPT=y
# CONFIG_DM_SNAPSHOT is not set
# CONFIG_DM_MIRROR is not set
# CONFIG_DM_ZERO is not set
# CONFIG_DM_MULTIPATH is not set
# CONFIG_DM_DELAY is not set
# CONFIG_DM_UEVENT is not set
# CONFIG_BLK_DEV_DM_BBR is not set
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#

#
# Enable only one of the two stacks, unless you know what you are doing
#
# CONFIG_FIREWIRE is not set
# CONFIG_IEEE1394 is not set
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
# CONFIG_IFB is not set
CONFIG_DUMMY=y
# CONFIG_BONDING is not set
# CONFIG_MACVLAN is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set
# CONFIG_VETH is not set
# CONFIG_NET_SB1000 is not set
# CONFIG_ARCNET is not set
# CONFIG_PHYLIB is not set
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_CASSINI is not set
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_NET_TULIP is not set
# CONFIG_HP100 is not set
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
# CONFIG_IBM_NEW_EMAC_NO_FLOW_CTRL is not set
# CONFIG_IBM_NEW_EMAC_MAL_CLR_ICINTSTAT is not set
# CONFIG_IBM_NEW_EMAC_MAL_COMMON_ERR is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_B44 is not set
# CONFIG_FORCEDETH is not set
# CONFIG_EEPRO100 is not set
# CONFIG_E100 is not set
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
CONFIG_8139TOO=m
# CONFIG_8139TOO_PIO is not set
# CONFIG_8139TOO_TUNE_TWISTER is not set
# CONFIG_8139TOO_8129 is not set
# CONFIG_8139_OLD_RX_RESET is not set
# CONFIG_R6040 is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_SC92031 is not set
# CONFIG_ATL2 is not set
CONFIG_NETDEV_1000=y
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_E1000E is not set
# CONFIG_IP1000 is not set
# CONFIG_IGB is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SIS190 is not set
CONFIG_SKGE=m
CONFIG_SKY2=m
# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set
# CONFIG_BNX2 is not set
# CONFIG_QLA3XXX is not set
# CONFIG_ATL1 is not set
# CONFIG_ATL1E is not set
# CONFIG_JME is not set
CONFIG_NETDEV_10000=y
# CONFIG_CHELSIO_T1 is not set
# CONFIG_CHELSIO_T3 is not set
# CONFIG_ENIC is not set
# CONFIG_IXGBE is not set
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set
# CONFIG_MYRI10GE is not set
# CONFIG_NETXEN_NIC is not set
# CONFIG_NIU is not set
# CONFIG_MLX4_EN is not set
# CONFIG_MLX4_CORE is not set
# CONFIG_TEHUTI is not set
# CONFIG_BNX2X is not set
# CONFIG_QLGE is not set
# CONFIG_SFC is not set
# CONFIG_TR is not set

#
# Wireless LAN
#
# CONFIG_WLAN_PRE80211 is not set
# CONFIG_WLAN_80211 is not set
# CONFIG_IWLWIFI_LEDS is not set

#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
CONFIG_PPP=y
# CONFIG_PPP_MULTILINK is not set
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=y
# CONFIG_PPP_SYNC_TTY is not set
CONFIG_PPP_DEFLATE=y
CONFIG_PPP_BSDCOMP=y
# CONFIG_PPP_MPPE is not set
# CONFIG_PPPOE is not set
# CONFIG_PPPOL2TP is not set
# CONFIG_SLIP is not set
CONFIG_SLHC=y
# CONFIG_NET_FC is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
# CONFIG_INPUT_POLLDEV is not set

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_BCM5974 is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=y
# CONFIG_INPUT_WISTRON_BTNS is not set
# CONFIG_INPUT_ATLAS_BTNS is not set
# CONFIG_INPUT_ATI_REMOTE is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_CM109 is not set
# CONFIG_INPUT_UINPUT is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
# CONFIG_DEVKMEM is not set
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_NOZOMI is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_CONSOLE is not set
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
# CONFIG_IPMI_HANDLER is not set
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_INTEL=y
# CONFIG_HW_RANDOM_AMD is not set
# CONFIG_HW_RANDOM_GEODE is not set
# CONFIG_HW_RANDOM_VIA is not set
# CONFIG_NVRAM is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set
# CONFIG_MWAVE is not set
# CONFIG_PC8736x_GPIO is not set
# CONFIG_NSC_GPIO is not set
# CONFIG_CS5535_GPIO is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_HPET is not set
# CONFIG_HANGCHECK_TIMER is not set
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_HELPER_AUTO=y

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
# CONFIG_I2C_AMD756 is not set
# CONFIG_I2C_AMD8111 is not set
CONFIG_I2C_I801=y
# CONFIG_I2C_ISCH is not set
# CONFIG_I2C_PIIX4 is not set
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set

#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_SIMTEC is not set

#
# External I2C/SMBus adapter drivers
#
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_TINY_USB is not set

#
# Graphics adapter I2C/DDC channel drivers
#
# CONFIG_I2C_VOODOO3 is not set

#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_PCA_PLATFORM is not set
# CONFIG_I2C_STUB is not set
# CONFIG_SCx200_ACB is not set

#
# Miscellaneous I2C Chip support
#
# CONFIG_DS1682 is not set
# CONFIG_AT24 is not set
# CONFIG_SENSORS_EEPROM is not set
# CONFIG_SENSORS_PCF8574 is not set
# CONFIG_PCF8575 is not set
# CONFIG_SENSORS_PCA9539 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_SENSORS_MAX6875 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set
# CONFIG_SPI is not set
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
# CONFIG_GPIOLIB is not set
# CONFIG_W1 is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_BATTERY_DS2760 is not set
# CONFIG_BATTERY_BQ27x00 is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=y
# CONFIG_SENSORS_ABITUGURU is not set
# CONFIG_SENSORS_ABITUGURU3 is not set
# CONFIG_SENSORS_AD7414 is not set
# CONFIG_SENSORS_AD7418 is not set
# CONFIG_SENSORS_ADM1021 is not set
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1026 is not set
# CONFIG_SENSORS_ADM1029 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ADM9240 is not set
# CONFIG_SENSORS_ADT7462 is not set
# CONFIG_SENSORS_ADT7470 is not set
# CONFIG_SENSORS_ADT7473 is not set
# CONFIG_SENSORS_K8TEMP is not set
# CONFIG_SENSORS_ASB100 is not set
# CONFIG_SENSORS_ATXP1 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_I5K_AMB is not set
# CONFIG_SENSORS_F71805F is not set
# CONFIG_SENSORS_F71882FG is not set
# CONFIG_SENSORS_F75375S is not set
# CONFIG_SENSORS_FSCHER is not set
# CONFIG_SENSORS_FSCPOS is not set
# CONFIG_SENSORS_FSCHMD is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_GL520SM is not set
CONFIG_SENSORS_CORETEMP=y
# CONFIG_SENSORS_IT87 is not set
# CONFIG_SENSORS_LM63 is not set
# CONFIG_SENSORS_LM75 is not set
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
# CONFIG_SENSORS_LM83 is not set
# CONFIG_SENSORS_LM85 is not set
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_LM92 is not set
# CONFIG_SENSORS_LM93 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_MAX6650 is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_PC87427 is not set
# CONFIG_SENSORS_SIS5595 is not set
# CONFIG_SENSORS_DME1737 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_SMSC47M192 is not set
# CONFIG_SENSORS_SMSC47B397 is not set
# CONFIG_SENSORS_ADS7828 is not set
# CONFIG_SENSORS_THMC50 is not set
# CONFIG_SENSORS_VIA686A is not set
# CONFIG_SENSORS_VT1211 is not set
# CONFIG_SENSORS_VT8231 is not set
# CONFIG_SENSORS_W83781D is not set
# CONFIG_SENSORS_W83791D is not set
# CONFIG_SENSORS_W83792D is not set
# CONFIG_SENSORS_W83793 is not set
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83L786NG is not set
# CONFIG_SENSORS_W83627HF is not set
CONFIG_SENSORS_W83627EHF=y
# CONFIG_SENSORS_HDAPS is not set
# CONFIG_SENSORS_LIS3LV02D is not set
# CONFIG_SENSORS_APPLESMC is not set
# CONFIG_HWMON_DEBUG_CHIP is not set
CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
# CONFIG_WATCHDOG is not set
CONFIG_SSB_POSSIBLE=y

#
# Sonics Silicon Backplane
#
# CONFIG_SSB is not set

#
# Multifunction device drivers
#
# CONFIG_MFD_CORE is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_PMIC_DA903X is not set
# CONFIG_MFD_WM8400 is not set
# CONFIG_MFD_WM8350_I2C is not set
# CONFIG_REGULATOR is not set

#
# Multimedia devices
#

#
# Multimedia core support
#
# CONFIG_VIDEO_DEV is not set
# CONFIG_DVB_CORE is not set
# CONFIG_VIDEO_MEDIA is not set

#
# Multimedia drivers
#
# CONFIG_DAB is not set

#
# Graphics support
#
# CONFIG_AGP is not set
# CONFIG_DRM is not set
# CONFIG_VGASTATE is not set
CONFIG_VIDEO_OUTPUT_CONTROL=m
# CONFIG_FB is not set
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set

#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_VGACON_SOFT_SCROLLBACK is not set
CONFIG_DUMMY_CONSOLE=y
CONFIG_SOUND=y
CONFIG_SOUND_OSS_CORE=y
CONFIG_SND=y
CONFIG_SND_TIMER=y
CONFIG_SND_PCM=y
CONFIG_SND_HWDEP=y
CONFIG_SND_RAWMIDI=y
CONFIG_SND_SEQUENCER=y
# CONFIG_SND_SEQ_DUMMY is not set
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=y
CONFIG_SND_PCM_OSS=y
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
# CONFIG_SND_DYNAMIC_MINORS is not set
CONFIG_SND_SUPPORT_OLD_API=y
# CONFIG_SND_VERBOSE_PROCFS is not set
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
CONFIG_SND_MPU401_UART=y
CONFIG_SND_AC97_CODEC=y
CONFIG_SND_DRIVERS=y
# CONFIG_SND_PCSP is not set
# CONFIG_SND_DUMMY is not set
# CONFIG_SND_VIRMIDI is not set
# CONFIG_SND_MTPAV is not set
# CONFIG_SND_SERIAL_U16550 is not set
CONFIG_SND_MPU401=y
# CONFIG_SND_AC97_POWER_SAVE is not set
CONFIG_SND_PCI=y
# CONFIG_SND_AD1889 is not set
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AW2 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CA0106 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_OXYGEN is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS5530 is not set
# CONFIG_SND_CS5535AUDIO is not set
# CONFIG_SND_DARLA20 is not set
# CONFIG_SND_GINA20 is not set
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
# CONFIG_SND_LAYLA24 is not set
# CONFIG_SND_MONA is not set
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
# CONFIG_SND_INDIGODJ is not set
CONFIG_SND_EMU10K1=y
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_FM801 is not set
# CONFIG_SND_HDA_INTEL is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
# CONFIG_SND_HIFIER is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
# CONFIG_SND_INTEL8X0 is not set
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_PCXHR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_SIS7019 is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_TRIDENT is not set
CONFIG_SND_VIA82XX=y
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VIRTUOSO is not set
# CONFIG_SND_VX222 is not set
# CONFIG_SND_YMFPCI is not set
CONFIG_SND_USB=y
# CONFIG_SND_USB_AUDIO is not set
# CONFIG_SND_USB_USX2Y is not set
# CONFIG_SND_USB_CAIAQ is not set
# CONFIG_SND_USB_US122L is not set
# CONFIG_SND_SOC is not set
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=y
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
# CONFIG_HID_DEBUG is not set
# CONFIG_HIDRAW is not set

#
# USB Input Devices
#
CONFIG_USB_HID=y
# CONFIG_HID_PID is not set
# CONFIG_USB_HIDDEV is not set

#
# Special HID drivers
#
CONFIG_HID_COMPAT=y
CONFIG_HID_A4TECH=y
CONFIG_HID_APPLE=y
CONFIG_HID_BELKIN=y
CONFIG_HID_BRIGHT=y
CONFIG_HID_CHERRY=y
CONFIG_HID_CHICONY=y
CONFIG_HID_CYPRESS=y
CONFIG_HID_DELL=y
CONFIG_HID_EZKEY=y
CONFIG_HID_GYRATION=y
CONFIG_HID_LOGITECH=y
# CONFIG_LOGITECH_FF is not set
# CONFIG_LOGIRUMBLEPAD2_FF is not set
CONFIG_HID_MICROSOFT=y
CONFIG_HID_MONTEREY=y
CONFIG_HID_PANTHERLORD=y
# CONFIG_PANTHERLORD_FF is not set
CONFIG_HID_PETALYNX=y
CONFIG_HID_SAMSUNG=y
CONFIG_HID_SONY=y
CONFIG_HID_SUNPLUS=y
# CONFIG_THRUSTMASTER_FF is not set
# CONFIG_ZEROPLUS_FF is not set
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set
# CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
CONFIG_USB_DEVICE_CLASS=y
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set
CONFIG_USB_MON=y
# CONFIG_USB_WUSB is not set
# CONFIG_USB_WUSB_CBAF is not set

#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
CONFIG_USB_EHCI_HCD=y
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1760_HCD is not set
# CONFIG_USB_OHCI_HCD is not set
CONFIG_USB_UHCI_HCD=y
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_WHCI_HCD is not set
# CONFIG_USB_HWA_HCD is not set

#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
# CONFIG_USB_PRINTER is not set
# CONFIG_USB_WDM is not set
# CONFIG_USB_TMC is not set

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may also be needed;
#

#
# see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=y
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_DPCM is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_STORAGE_ALAUDA is not set
# CONFIG_USB_STORAGE_ONETOUCH is not set
# CONFIG_USB_STORAGE_KARMA is not set
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
# CONFIG_USB_LIBUSUAL is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set

#
# USB port drivers
#
# CONFIG_USB_SERIAL is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_SEVSEG is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_BERRY_CHARGE is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGET is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
# CONFIG_USB_TEST is not set
# CONFIG_USB_ISIGHTFW is not set
# CONFIG_USB_VST is not set
# CONFIG_USB_GADGET is not set
# CONFIG_UWB is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
CONFIG_EDAC=y

#
# Reporting subsystems
#
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_MM_EDAC=y
# CONFIG_EDAC_AMD76X is not set
# CONFIG_EDAC_E7XXX is not set
# CONFIG_EDAC_E752X is not set
# CONFIG_EDAC_I82875P is not set
# CONFIG_EDAC_I82975X is not set
# CONFIG_EDAC_I3000 is not set
# CONFIG_EDAC_X38 is not set
# CONFIG_EDAC_I82860 is not set
# CONFIG_EDAC_R82600 is not set
# CONFIG_EDAC_I5000 is not set
# CONFIG_EDAC_I5100 is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
# CONFIG_RTC_DEBUG is not set

#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
CONFIG_RTC_INTF_DEV_UIE_EMUL=y
# CONFIG_RTC_DRV_TEST is not set

#
# I2C RTC drivers
#
# CONFIG_RTC_DRV_DS1307 is not set
# CONFIG_RTC_DRV_DS1374 is not set
# CONFIG_RTC_DRV_DS1672 is not set
# CONFIG_RTC_DRV_MAX6900 is not set
# CONFIG_RTC_DRV_RS5C372 is not set
# CONFIG_RTC_DRV_ISL1208 is not set
# CONFIG_RTC_DRV_X1205 is not set
# CONFIG_RTC_DRV_PCF8563 is not set
# CONFIG_RTC_DRV_PCF8583 is not set
# CONFIG_RTC_DRV_M41T80 is not set
# CONFIG_RTC_DRV_S35390A is not set
# CONFIG_RTC_DRV_FM3130 is not set
# CONFIG_RTC_DRV_RX8581 is not set

#
# SPI RTC drivers
#

#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=y
# CONFIG_RTC_DRV_DS1286 is not set
# CONFIG_RTC_DRV_DS1511 is not set
# CONFIG_RTC_DRV_DS1553 is not set
# CONFIG_RTC_DRV_DS1742 is not set
# CONFIG_RTC_DRV_STK17TA8 is not set
# CONFIG_RTC_DRV_M48T86 is not set
# CONFIG_RTC_DRV_M48T35 is not set
# CONFIG_RTC_DRV_M48T59 is not set
# CONFIG_RTC_DRV_BQ4802 is not set
# CONFIG_RTC_DRV_V3020 is not set

#
# on-CPU RTC drivers
#
CONFIG_DMADEVICES=y

#
# DMA Devices
#
# CONFIG_INTEL_IOATDMA is not set
# CONFIG_UIO is not set
# CONFIG_STAGING is not set

#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_FIRMWARE_MEMMAP=y
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
CONFIG_DMIID=y
# CONFIG_ISCSI_IBFT_FIND is not set

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
# CONFIG_EXT2_FS_POSIX_ACL is not set
# CONFIG_EXT2_FS_SECURITY is not set
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
# CONFIG_EXT3_FS_POSIX_ACL is not set
# CONFIG_EXT3_FS_SECURITY is not set
# CONFIG_EXT4_FS is not set
CONFIG_JBD=y
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=y
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not set
CONFIG_REISERFS_FS_XATTR=y
# CONFIG_REISERFS_FS_POSIX_ACL is not set
# CONFIG_REISERFS_FS_SECURITY is not set
# CONFIG_JFS_FS is not set
# CONFIG_FS_POSIX_ACL is not set
CONFIG_FILE_LOCKING=y
# CONFIG_XFS_FS is not set
# CONFIG_OCFS2_FS is not set
CONFIG_DNOTIFY=y
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_QUOTA is not set
# CONFIG_AUTOFS_FS is not set
# CONFIG_AUTOFS4_FS is not set
CONFIG_FUSE_FS=y

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=866
CONFIG_FAT_DEFAULT_IOCHARSET="utf8"
CONFIG_NTFS_FS=y
# CONFIG_NTFS_DEBUG is not set
# CONFIG_NTFS_RW is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_SYSCTL=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_CONFIGFS_FS=y

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_SQUASHFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_OMFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
CONFIG_NETWORK_FILESYSTEMS=y
# CONFIG_NFS_FS is not set
# CONFIG_NFSD is not set
# CONFIG_SMB_FS is not set
CONFIG_CIFS=y
# CONFIG_CIFS_STATS is not set
# CONFIG_CIFS_WEAK_PW_HASH is not set
# CONFIG_CIFS_XATTR is not set
# CONFIG_CIFS_DEBUG2 is not set
# CONFIG_CIFS_EXPERIMENTAL is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
CONFIG_9P_FS=y

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
# CONFIG_BSD_DISKLABEL is not set
# CONFIG_MINIX_SUBPARTITION is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
# CONFIG_LDM_PARTITION is not set
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
# CONFIG_KARMA_PARTITION is not set
# CONFIG_EFI_PARTITION is not set
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
# CONFIG_NLS_CODEPAGE_437 is not set
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
CONFIG_NLS_CODEPAGE_866=y
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
# CONFIG_NLS_ASCII is not set
# CONFIG_NLS_ISO8859_1 is not set
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
CONFIG_NLS_KOI8_R=y
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=y
# CONFIG_DLM is not set

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
# CONFIG_PRINTK_TIME is not set
CONFIG_ENABLE_WARN_DEPRECATED=y
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=1024
CONFIG_MAGIC_SYSRQ=y
CONFIG_UNUSED_SYMBOLS=y
# CONFIG_DEBUG_FS is not set
# CONFIG_HEADERS_CHECK is not set
# CONFIG_DEBUG_KERNEL is not set
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
CONFIG_STACKTRACE=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_FRAME_POINTER=y
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
CONFIG_LATENCYTOP=y
CONFIG_SYSCTL_SYSCALL_CHECK=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y

#
# Tracers
#
# CONFIG_SYSPROF_TRACER is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_DYNAMIC_PRINTK_DEBUG is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_STRICT_DEVMEM=y
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
# CONFIG_EARLY_PRINTK_DBGP is not set
# CONFIG_4KSTACKS is not set
CONFIG_DOUBLEFAULT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
# CONFIG_OPTIMIZE_INLINING is not set

#
# Security options
#

#
# Grsecurity
#
CONFIG_GRKERNSEC=y
# CONFIG_GRKERNSEC_LOW is not set
# CONFIG_GRKERNSEC_MEDIUM is not set
# CONFIG_GRKERNSEC_HIGH is not set
# CONFIG_GRKERNSEC_HARDENED_SERVER is not set
# CONFIG_GRKERNSEC_HARDENED_WORKSTATION is not set
CONFIG_GRKERNSEC_CUSTOM=y

#
# Address Space Protection
#
CONFIG_GRKERNSEC_KMEM=y
# CONFIG_GRKERNSEC_IO is not set
CONFIG_GRKERNSEC_PROC_MEMMAP=y
CONFIG_GRKERNSEC_BRUTE=y
CONFIG_GRKERNSEC_MODSTOP=y
CONFIG_GRKERNSEC_HIDESYM=y

#
# Role Based Access Control Options
#
CONFIG_GRKERNSEC_ACL_HIDEKERN=y
CONFIG_GRKERNSEC_ACL_MAXTRIES=3
CONFIG_GRKERNSEC_ACL_TIMEOUT=30

#
# Filesystem Protections
#
CONFIG_GRKERNSEC_PROC=y
CONFIG_GRKERNSEC_PROC_USER=y
CONFIG_GRKERNSEC_PROC_ADD=y
CONFIG_GRKERNSEC_LINK=y
CONFIG_GRKERNSEC_FIFO=y
CONFIG_GRKERNSEC_CHROOT=y
CONFIG_GRKERNSEC_CHROOT_MOUNT=y
CONFIG_GRKERNSEC_CHROOT_DOUBLE=y
CONFIG_GRKERNSEC_CHROOT_PIVOT=y
CONFIG_GRKERNSEC_CHROOT_CHDIR=y
CONFIG_GRKERNSEC_CHROOT_CHMOD=y
CONFIG_GRKERNSEC_CHROOT_FCHDIR=y
CONFIG_GRKERNSEC_CHROOT_MKNOD=y
CONFIG_GRKERNSEC_CHROOT_SHMAT=y
CONFIG_GRKERNSEC_CHROOT_UNIX=y
CONFIG_GRKERNSEC_CHROOT_FINDTASK=y
CONFIG_GRKERNSEC_CHROOT_NICE=y
CONFIG_GRKERNSEC_CHROOT_SYSCTL=y
CONFIG_GRKERNSEC_CHROOT_CAPS=y

#
# Kernel Auditing
#
# CONFIG_GRKERNSEC_AUDIT_GROUP is not set
# CONFIG_GRKERNSEC_EXECLOG is not set
CONFIG_GRKERNSEC_RESLOG=y
# CONFIG_GRKERNSEC_CHROOT_EXECLOG is not set
# CONFIG_GRKERNSEC_AUDIT_CHDIR is not set
# CONFIG_GRKERNSEC_AUDIT_MOUNT is not set
# CONFIG_GRKERNSEC_AUDIT_IPC is not set
CONFIG_GRKERNSEC_SIGNAL=y
CONFIG_GRKERNSEC_FORKFAIL=y
# CONFIG_GRKERNSEC_TIME is not set
CONFIG_GRKERNSEC_PROC_IPADDR=y
# CONFIG_GRKERNSEC_AUDIT_TEXTREL is not set

#
# Executable Protections
#
CONFIG_GRKERNSEC_EXECVE=y
CONFIG_GRKERNSEC_DMESG=y
# CONFIG_GRKERNSEC_TPE is not set

#
# Network Protections
#
CONFIG_GRKERNSEC_RANDNET=y
# CONFIG_GRKERNSEC_SOCKET is not set

#
# Sysctl support
#
CONFIG_GRKERNSEC_SYSCTL=y
CONFIG_GRKERNSEC_SYSCTL_ON=y

#
# Logging Options
#
CONFIG_GRKERNSEC_FLOODTIME=10
CONFIG_GRKERNSEC_FLOODBURST=4

#
# PaX
#
CONFIG_PAX=y

#
# PaX Control
#
# CONFIG_PAX_SOFTMODE is not set
CONFIG_PAX_EI_PAX=y
CONFIG_PAX_PT_PAX_FLAGS=y
CONFIG_PAX_NO_ACL_FLAGS=y
# CONFIG_PAX_HAVE_ACL_FLAGS is not set
# CONFIG_PAX_HOOK_ACL_FLAGS is not set

#
# Non-executable pages
#
CONFIG_PAX_NOEXEC=y
# CONFIG_PAX_PAGEEXEC is not set
CONFIG_PAX_SEGMEXEC=y
# CONFIG_PAX_EMUTRAMP is not set
CONFIG_PAX_MPROTECT=y
# CONFIG_PAX_NOELFRELOCS is not set
# CONFIG_PAX_KERNEXEC is not set

#
# Address Space Layout Randomization
#
CONFIG_PAX_ASLR=y
CONFIG_PAX_RANDKSTACK=y
CONFIG_PAX_RANDUSTACK=y
CONFIG_PAX_RANDMMAP=y

#
# Miscellaneous hardening features
#
# CONFIG_PAX_MEMORY_SANITIZE is not set
CONFIG_PAX_MEMORY_UDEREF=y
CONFIG_PAX_REFCOUNT=y
# CONFIG_KEYS is not set
CONFIG_SECURITY=y
# CONFIG_SECURITYFS is not set
# CONFIG_SECURITY_NETWORK is not set
# CONFIG_SECURITY_FILE_CAPABILITIES is not set
# CONFIG_SECURITY_ROOTPLUG is not set
CONFIG_SECURITY_DEFAULT_MMAP_MIN_ADDR=0
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_FIPS=y
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
# CONFIG_CRYPTO_GF128MUL is not set
# CONFIG_CRYPTO_NULL is not set
# CONFIG_CRYPTO_CRYPTD is not set
# CONFIG_CRYPTO_AUTHENC is not set
# CONFIG_CRYPTO_TEST is not set

#
# Authenticated Encryption with Associated Data
#
# CONFIG_CRYPTO_CCM is not set
# CONFIG_CRYPTO_GCM is not set
# CONFIG_CRYPTO_SEQIV is not set

#
# Block modes
#
CONFIG_CRYPTO_CBC=y
# CONFIG_CRYPTO_CTR is not set
# CONFIG_CRYPTO_CTS is not set
# CONFIG_CRYPTO_ECB is not set
# CONFIG_CRYPTO_LRW is not set
# CONFIG_CRYPTO_PCBC is not set
# CONFIG_CRYPTO_XTS is not set

#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
# CONFIG_CRYPTO_XCBC is not set

#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_CRC32C_INTEL=y
# CONFIG_CRYPTO_MD4 is not set
CONFIG_CRYPTO_MD5=y
# CONFIG_CRYPTO_MICHAEL_MIC is not set
# CONFIG_CRYPTO_RMD128 is not set
# CONFIG_CRYPTO_RMD160 is not set
# CONFIG_CRYPTO_RMD256 is not set
# CONFIG_CRYPTO_RMD320 is not set
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=y
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_TGR192 is not set
# CONFIG_CRYPTO_WP512 is not set

#
# Ciphers
#
CONFIG_CRYPTO_AES=y
# CONFIG_CRYPTO_AES_586 is not set
# CONFIG_CRYPTO_ANUBIS is not set
# CONFIG_CRYPTO_ARC4 is not set
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_CAMELLIA is not set
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
# CONFIG_CRYPTO_DES is not set
# CONFIG_CRYPTO_FCRYPT is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_SALSA20 is not set
# CONFIG_CRYPTO_SALSA20_586 is not set
# CONFIG_CRYPTO_SEED is not set
# CONFIG_CRYPTO_SERPENT is not set
# CONFIG_CRYPTO_TEA is not set
# CONFIG_CRYPTO_TWOFISH is not set
# CONFIG_CRYPTO_TWOFISH_586 is not set

#
# Compression
#
# CONFIG_CRYPTO_DEFLATE is not set
# CONFIG_CRYPTO_LZO is not set

#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=y
CONFIG_CRYPTO_HW=y
# CONFIG_CRYPTO_DEV_PADLOCK is not set
# CONFIG_CRYPTO_DEV_GEODE is not set
# CONFIG_CRYPTO_DEV_HIFN_795X is not set
CONFIG_HAVE_KVM=y
CONFIG_VIRTUALIZATION=y
# CONFIG_KVM is not set
# CONFIG_LGUEST is not set
# CONFIG_VIRTIO_PCI is not set
# CONFIG_VIRTIO_BALLOON is not set

#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_CRC_CCITT=y
CONFIG_CRC16=y
# CONFIG_CRC_T10DIF is not set
CONFIG_CRC_ITU_T=y
CONFIG_CRC32=y
# CONFIG_CRC7 is not set
CONFIG_LIBCRC32C=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_PLIST=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
Comment 42 Alex Efros 2009-11-24 07:15:47 UTC
BTW, on my friend's system, which doesn't have this bug,
I also see very high iowait and loadavg values while executing dd.
But these high iowait/loadavg values doesn't result in overall system slowdown:
interface works fast, and hdd works at full speed.

So, looks like high iowait/loadavg values can't be used at sign of this bug,
and we only have slower file copying speed than expected on this hdd,
slow gui (like switching from one virtual desktop to another in fluxbox)
while file copying, and slow application start while file copying. :(
Comment 43 Alex Efros 2009-11-24 08:25:42 UTC
I've just tried to use my friend's HDD in my computer.
I've copied my linux partition to his hdd and boot from it.

# hdparm -t /dev/sda
/dev/sda:
 Timing buffered disk reads:  334 MB in  3.00 seconds = 111.28 MB/sec

If I run 'dd if=/dev/zero of=test1 bs=1M count=8000' and will just wait
until it finish, without doing anything else on system, speed will be
about 100MB/sec.

If I run same command, but will actively use system while it's working
(switch virtual desktops, talk in pidgin, work in opera, start different
applications), then dd speed will be about 80MB/sec.

If I run 4 such dd in parallel, then iowait will be about 80%,
loadavg up to 6.5, BUT...... overall system slowdown will be 
_significantly_ less than previously (with my original HDD)!!!
Final speed of all 4 dd was about 20MB/sec.
Same overall system slowdown and dd speed I have if I run just
two dd for copy files like `dd if=test1 of=test11 bs=1M`.


I don't think changing HDD fixed the bug, because sometimes I've up to
2-4 seconds freeze while just typing messages in Pidgin (with dd running in
background), but... overall system slowdown is now feels acceptable,
it doesn't make system nearly _unusable_ while copying large files,
as previously.
Also, now file copying works on full HDD speed, instead of 3-5MB/sec as sometimes on another HDD.
So, looks like with this faster HDD the bug become less noticeable.
I'm going to buy Seagate Barracuda LP 2TB HDD and will test it too.


Here is example of `vmstat 1` output while running 4 dd as described above
(with loadavg about 6.5 and fast enough overall system):
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  6      0  52396  10004 1027216    0    0  1248 41448 2493 1838  8 11  0 82
 0  5      0  55224   9932 1023476    0    0   924 54596 2370 1430  6  9  0 84
 0  5      0  56372  10028 1026172    0    0   108 97340 2473 2192  9 20  0 71
 1  5      0  53344  10032 1027352    0    0   656 107936 2500 1421  4 17  6 73
 0  7      0  53408  10004 1026640    0    0   444 94228 2474 1188  2 15  0 83
 1  6      0  54352  10056 1024192    0    0     4 79420 2444 1248  5 13  0 82
 0  6      0  52632  10132 1025312    0    0     0 97412 2472 1388  6 16  0 78
 1  6      0  56612  10204 1024860    0    0     4 86984 2421 1756 14 14  0 72
 0  5      0  53356  10220 1027576    0    0    48 99416 2453 1701  6 17  0 76
 0  6      0  51568  10300 1027888    0    0   388 97372 2417 1402 10 15  0 76
 1  5      0  54756  10312 1025188    0    0   516 61192 2491 1858 10 11  0 79
 1  3      0  52720  10184 1024832    0    0   356 39288 2376 1740 10  9  3 78
 0  7      0  56432  10220 1025596    0    0     4 62236 2460 2165 13 20  1 67
 0  5      0  56216  10188 1025444    0    0   200 125564 2422 1585 10 15  6 70
 0  7      0  53828  10172 1026520    0    0   336 108372 2543 1612  7 17  0 77
 0  5      0  58336  10148 1019816    0    0     4 88924 2420 2264 11 18  5 67
 0  6      0  55488  10256 1018676    0    0   952 47832 2524 3149 15 17  0 69
 0  7      0  52896  10328 1023392    0    0  1012 42204 2439 2200 16 15  0 68
 1  8      0  58808  10300 1020324    0    0  1464 36340 2482 2557 18 11  0 72
 1  6      0  52328  10240 1026560    0    0   756 82652 2403 2273 15 12  1 73
 1  7      0  55792  10336 1022208    0    0  1616 52864 2489 2056 16 10  0 74
 1  6      0  52748  10324 1024844    0    0  1464 49324 2396 1085 12  8  0 80
 1  7      0  53292  10372 1026292    0    0   288 91920 2506 2564  6 21  0 73
 5  5      0  56404  10404 1021116    0    0    92 109964 2493 1500  5 17  0 78
 2  5      0  52840  10444 1023968    0    0   284 82828 2448 1821  7 15  0 78
 1  7      0  57172  10320 1018960    0    0     4 104736 2422 1144  3 16  0 82
 0  6      0  54712  10332 1019736    0    0   164 101156 2456 1267  9 16  0 76
 0  5      0  55364  10388 1021452    0    0  1464 42832 2492 1888  7 14  6 72
 2  5      0  57000  10332 1020748    0    0   168 108988 2554 2261  9 16 16 58
Comment 44 Tejun Heo 2009-11-24 08:44:23 UTC
Alex, that starts to sound like a faulty hard drive to me.  Drives developing problems which lead to significant slow down in seek time in partial areas of the drive definitely is possible.  Now that you have a spare hard drive, it would be interesting to create a filesystem where windows partition used to be and see whether the IO slow down is reproducible from there too.
Comment 45 Alan 2009-11-24 11:23:07 UTC
Which driver is actually being used for your problem disk ?

Some of the pausing may be the high memory balancing if you have a 32bit kernel (as you do) but over 4GB of RAM and the drive is being handled by a non AHCI interface (the old IDE interfaces can only access the low 4GB of addresses directly so must copy using the CPU for the rest).

Were the old and new disks plugged into the same controller ?
Comment 46 Alex Efros 2009-11-24 15:10:23 UTC
> Which driver is actually being used for your problem disk ?

How I can check this?

> Some of the pausing may be ... but over 4GB

I've only 2GB now.

> Were the old and new disks plugged into the same controller ?

Yep.

> Alex, that starts to sound like a faulty hard drive to me.

I've tested read and write speed on different partitions using
  dd if=/dev/zero of=test1 bs=1M count=3000
and
  dd if=test1 of=/dev/null bs=1M
commands, executed 3 times for each partition.

I've repeated these tests twice: first time I just wait until these
commands will finish, and second time I tried to use system while
these commands was running (I've switched virtual desktops, talk in pidgin,
used opera, work with mutt and start several light applications like mc,
cpan, and gqview).

And there some strange results.

# fdisk -l

Disk /dev/sda: 320.1 GB, 320072933376 bytes
255 heads, 63 sectors/track, 38913 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x15741573

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1           7       56196   16  Hidden FAT16
/dev/sda2   *           8          14       56227+   6  FAT16
/dev/sda3              15          21       56227+  16  Hidden FAT16
/dev/sda4              22       38913   312399990    f  W95 Ext'd (LBA)
/dev/sda5              22        1718    13631121    b  W95 FAT32
/dev/sda6            1719        4330    20980858+  83  Linux
/dev/sda7            4331        5636    10490413+   7  HPFS/NTFS
/dev/sda8            5637       12806    57592993+  83  Linux
/dev/sda9           12807       38913   209704446    b  W95 FAT32

The partitions tested are:
  sda5 (fat32)
  sda6 (ext3, linux root)
  sda8 (ext3)
  sda9 (fat32)
As you see, partitions are going in order, as they located on disk.
First two (sda5 and sda6) are both small enough and located near beginning
of disk, so they should work at maximum speed (in theory).

Test results (numbers are MB/sec):
        W1/W2  R1/R2
  sda5  47/38  68/55
  sda6  27/27  45/43
  sda8  50/42  64/54
  sda9  42/34  50/35

  the W1 column is write test without doing anything while testing,
  the W2 column is write test with actively used system while testing,
  the R1/R2 are same for read tests

As you see, there several strange things:
1) sda6 are inexplicably slow
2) sda6 work on same speed even if I actively use system while test,
   unlike all others

After that I tried to check linear read speed for both sda5 and sda6 partitions
by reading per 100 cylinders from sda in a loop (in hope to find some cylinders
located on sda6 which slower than others). But this test show stable speed
of 80 MB/sec (!!!) for all disk space under both sda5 and sda6:

# for s in $(seq 22 100 4330); do
    echo -n cylinders $s - $(( $s + 99 )) ": "
    dd if=/dev/sda of=/dev/null bs=8225280 count=100 skip=$s 2>&1 | grep -v records
  done

--- sda5
cylinders 22 - 121 : 822528000 bytes (823 MB) copied, 10.2647 s, 80.1 MB/s
cylinders 122 - 221 : 822528000 bytes (823 MB) copied, 10.2471 s, 80.3 MB/s
cylinders 222 - 321 : 822528000 bytes (823 MB) copied, 10.3837 s, 79.2 MB/s
cylinders 322 - 421 : 822528000 bytes (823 MB) copied, 10.2467 s, 80.3 MB/s
cylinders 422 - 521 : 822528000 bytes (823 MB) copied, 10.6297 s, 77.4 MB/s
cylinders 522 - 621 : 822528000 bytes (823 MB) copied, 10.458 s, 78.7 MB/s
cylinders 622 - 721 : 822528000 bytes (823 MB) copied, 10.2455 s, 80.3 MB/s
cylinders 722 - 821 : 822528000 bytes (823 MB) copied, 10.2497 s, 80.2 MB/s
cylinders 822 - 921 : 822528000 bytes (823 MB) copied, 10.2477 s, 80.3 MB/s
cylinders 922 - 1021 : 822528000 bytes (823 MB) copied, 10.5178 s, 78.2 MB/s
cylinders 1022 - 1121 : 822528000 bytes (823 MB) copied, 10.2711 s, 80.1 MB/s
cylinders 1122 - 1221 : 822528000 bytes (823 MB) copied, 10.2613 s, 80.2 MB/s
cylinders 1222 - 1321 : 822528000 bytes (823 MB) copied, 10.2796 s, 80.0 MB/s
cylinders 1322 - 1421 : 822528000 bytes (823 MB) copied, 10.4984 s, 78.3 MB/s
cylinders 1422 - 1521 : 822528000 bytes (823 MB) copied, 10.2779 s, 80.0 MB/s
cylinders 1522 - 1621 : 822528000 bytes (823 MB) copied, 10.2504 s, 80.2 MB/s
cylinders 1622 - 1721 : 822528000 bytes (823 MB) copied, 10.3638 s, 79.4 MB/s
---sda6
cylinders 1722 - 1821 : 822528000 bytes (823 MB) copied, 10.2751 s, 80.1 MB/s
cylinders 1822 - 1921 : 822528000 bytes (823 MB) copied, 10.5918 s, 77.7 MB/s
cylinders 1922 - 2021 : 822528000 bytes (823 MB) copied, 10.2925 s, 79.9 MB/s
cylinders 2022 - 2121 : 822528000 bytes (823 MB) copied, 10.252 s, 80.2 MB/s
cylinders 2122 - 2221 : 822528000 bytes (823 MB) copied, 10.2498 s, 80.2 MB/s
cylinders 2222 - 2321 : 822528000 bytes (823 MB) copied, 10.3093 s, 79.8 MB/s
cylinders 2322 - 2421 : 822528000 bytes (823 MB) copied, 10.2605 s, 80.2 MB/s
cylinders 2422 - 2521 : 822528000 bytes (823 MB) copied, 10.2508 s, 80.2 MB/s
cylinders 2522 - 2621 : 822528000 bytes (823 MB) copied, 10.4766 s, 78.5 MB/s
cylinders 2622 - 2721 : 822528000 bytes (823 MB) copied, 10.2503 s, 80.2 MB/s
cylinders 2722 - 2821 : 822528000 bytes (823 MB) copied, 10.3006 s, 79.9 MB/s
cylinders 2822 - 2921 : 822528000 bytes (823 MB) copied, 10.2754 s, 80.0 MB/s
cylinders 2922 - 3021 : 822528000 bytes (823 MB) copied, 10.3224 s, 79.7 MB/s
cylinders 3022 - 3121 : 822528000 bytes (823 MB) copied, 10.2507 s, 80.2 MB/s
cylinders 3122 - 3221 : 822528000 bytes (823 MB) copied, 10.2885 s, 79.9 MB/s
cylinders 3222 - 3321 : 822528000 bytes (823 MB) copied, 10.3702 s, 79.3 MB/s
cylinders 3322 - 3421 : 822528000 bytes (823 MB) copied, 10.327 s, 79.6 MB/s
cylinders 3422 - 3521 : 822528000 bytes (823 MB) copied, 10.2526 s, 80.2 MB/s
cylinders 3522 - 3621 : 822528000 bytes (823 MB) copied, 10.2801 s, 80.0 MB/s
cylinders 3622 - 3721 : 822528000 bytes (823 MB) copied, 10.2456 s, 80.3 MB/s
cylinders 3722 - 3821 : 822528000 bytes (823 MB) copied, 10.2472 s, 80.3 MB/s
cylinders 3822 - 3921 : 822528000 bytes (823 MB) copied, 10.2626 s, 80.1 MB/s
cylinders 3922 - 4021 : 822528000 bytes (823 MB) copied, 10.5369 s, 78.1 MB/s
cylinders 4022 - 4121 : 822528000 bytes (823 MB) copied, 10.2885 s, 79.9 MB/s
cylinders 4122 - 4221 : 822528000 bytes (823 MB) copied, 10.3256 s, 79.7 MB/s
cylinders 4222 - 4321 : 822528000 bytes (823 MB) copied, 10.36 s, 79.4 MB/s
cylinders 4322 - 4421 : 822528000 bytes (823 MB) copied, 10.4416 s, 78.8 MB/s

To make sure dd really read correct cylinders I've also read 1000 cylinders
at the middle and end of disk using same for loop
(just to see smaller numbers than 80MB/s):
...
cylinders 20000 - 20099 : 822528000 bytes (823 MB) copied, 11.8926 s, 69.2 MB/s
cylinders 20100 - 20199 : 822528000 bytes (823 MB) copied, 11.8949 s, 69.1 MB/s
cylinders 20200 - 20299 : 822528000 bytes (823 MB) copied, 11.9409 s, 68.9 MB/s
cylinders 20300 - 20399 : 822528000 bytes (823 MB) copied, 12.0738 s, 68.1 MB/s
cylinders 20400 - 20499 : 822528000 bytes (823 MB) copied, 12.1441 s, 67.7 MB/s
cylinders 20500 - 20599 : 822528000 bytes (823 MB) copied, 11.9761 s, 68.7 MB/s
cylinders 20600 - 20699 : 822528000 bytes (823 MB) copied, 11.9936 s, 68.6 MB/s
cylinders 20700 - 20799 : 822528000 bytes (823 MB) copied, 12.1596 s, 67.6 MB/s
cylinders 20800 - 20899 : 822528000 bytes (823 MB) copied, 12.5385 s, 65.6 MB/s
cylinders 20900 - 20999 : 822528000 bytes (823 MB) copied, 11.9722 s, 68.7 MB/s
...
cylinders 38000 - 38099 : 822528000 bytes (823 MB) copied, 18.7223 s, 43.9 MB/s
cylinders 38100 - 38199 : 822528000 bytes (823 MB) copied, 19.3013 s, 42.6 MB/s
cylinders 38200 - 38299 : 822528000 bytes (823 MB) copied, 20.1871 s, 40.7 MB/s
cylinders 38300 - 38399 : 822528000 bytes (823 MB) copied, 20.2306 s, 40.7 MB/s
cylinders 38400 - 38499 : 822528000 bytes (823 MB) copied, 20.2433 s, 40.6 MB/s
cylinders 38500 - 38599 : 822528000 bytes (823 MB) copied, 20.2426 s, 40.6 MB/s
cylinders 38600 - 38699 : 822528000 bytes (823 MB) copied, 20.3642 s, 40.4 MB/s
cylinders 38700 - 38799 : 822528000 bytes (823 MB) copied, 20.3394 s, 40.4 MB/s
cylinders 38800 - 38899 : 822528000 bytes (823 MB) copied, 20.3607 s, 40.4 MB/s
cylinders 38900 - 38999 : 109541376 bytes (110 MB) copied, 2.58861 s, 42.3 MB/s

So, HDD itself looks ok.
Comment 47 Tejun Heo 2009-11-24 15:30:32 UTC
Hmmm... Just one more thing.  Can you please run the following two commands in parallel for each partition and see whether there's any drastic difference?

# dd if=/dev/sdaN of=/dev/null bs=1M count=512 iflag=direct skip=0
# dd if=/dev/sdaN of=/dev/null bs=1M count=512 iflag=direct skip=8192
Comment 48 Alex Efros 2009-11-24 15:38:38 UTC
No, there no difference.

# for part in 5 6 8 9; do 
    echo --- /dev/sda$part
    dd if=/dev/sda$part of=/dev/null bs=1M count=512 iflag=direct skip=0 &
    dd if=/dev/sda$part of=/dev/null bs=1M count=512 iflag=direct skip=8192
    wait
  done
--- /dev/sda5
[1] 926
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 13.9199 s, 38.6 MB/s
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 14.7542 s, 36.4 MB/s
[1]+  Done                    dd if=/dev/sda$part of=/dev/null bs=1M count=512 iflag=direct skip=0
--- /dev/sda6
[1] 929
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 14.6602 s, 36.6 MB/s
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 14.8848 s, 36.1 MB/s
[1]+  Done                    dd if=/dev/sda$part of=/dev/null bs=1M count=512 iflag=direct skip=0
--- /dev/sda8
[1] 934
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 14.7573 s, 36.4 MB/s
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 15.1113 s, 35.5 MB/s
[1]+  Done                    dd if=/dev/sda$part of=/dev/null bs=1M count=512 iflag=direct skip=0
--- /dev/sda9
[1] 936
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 14.927 s, 36.0 MB/s
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 15.8174 s, 33.9 MB/s
[1]+  Done                    dd if=/dev/sda$part of=/dev/null bs=1M count=512 iflag=direct skip=0
Comment 49 Tejun Heo 2009-11-24 15:49:00 UTC
Hmmm... doesn't look like a hardware or driver problem.  Can you be persuaded into backing up sda6, creating a new filesystem and test it?  If you back up sda6 by dd'ing the whole partition, you'll be able to get back the same fs layout by dd'ing it back.
Comment 50 Alex Efros 2009-11-24 15:58:39 UTC
I prefer to use this way to copy partition instead:
  mount /dev/sda6 /mnt/old
  mount /dev/some /mnt/new
  tar cf - -C /mnt/old ./ | tar xpf - -C /mnt/new
This way I should also fix all possible disk fragmentation issues because
new partition is formatted and then tar create all files.

I've done this already several months ago - in hope to fix this slowdown
if it happens just because of high disk fragmentation (it's Gentoo, so
I expect much higher rate of new files creation/deletion while upgrading
and compiling system than in other distributions which may result in higher
disk fragmentation). This doesn't helps. But I can repeat this once again,
just to be sure.
Comment 51 Tejun Heo 2009-11-24 16:05:25 UTC
I was suggesting dd'ing so that the problem can be reproduced later if fs defragmentation turns out to be the issue.  Now I'm out of ideas.  If no one has further ideas, creating a fresh filesystem can test it without populating it might reveal what's going on a bit more.
Comment 52 Alan 2009-11-24 16:30:12 UTC
After boot dmesg will contain info for the various controllers/devices

You will find lines of a form like:

scsi0 : ahci
scsi1 : ahci

the left being the controller number the right the driver

and later on lines of the form

ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata2.00: ATA-8: WDC WD5000AAKS-75A7B2, 01.03B01, max UDMA/133
ata2.00: 976773168 sectors, multi 0: LBA48 NCQ (depth 31/32)
ata2.00: configured for UDMA/133

scsi 1:0:0:0: Direct-Access     ATA      WDC WD5000AAKS-7 01.0 PQ: 0 ANSI: 5
^^

showing each device. The second set of information gives you the scsi device, the first set then tell you which driver that matches.

Alan
Comment 53 Alex Efros 2009-11-24 21:22:20 UTC
> After boot dmesg will contain info for the various controllers/devices

ok, here it is:
...
Uniform Multi-Platform E-IDE driver
jmicron 0000:03:00.1: IDE controller (0x197b:0x2363 rev 0x02)
JMicron IDE 0000:03:00.1: enabling device (0000 -> 0001)
JMicron IDE 0000:03:00.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
jmicron 0000:03:00.1: 100% native mode on irq 17
JMicron IDE 0000:03:00.1: setting latency timer to 64
    ide0: BM-DMA at 0xb400-0xb407
    ide1: BM-DMA at 0xb408-0xb40f
Probing IDE interface ide0...
Switched to high resolution mode on CPU 1
Switched to high resolution mode on CPU 0
Probing IDE interface ide1...
ide0 at 0xbc00-0xbc07,0xb882 on irq 17
ide1 at 0xb800-0xb807,0xb482 on irq 17
ide-gd driver 1.18
ide-cd driver 5.00
Driver 'sd' needs updating - please use bus_type methods
Driver 'sr' needs updating - please use bus_type methods
ahci 0000:00:1f.2: version 3.0
ahci 0000:00:1f.2: PCI INT B -> GSI 19 (level, low) -> IRQ 19
ahci 0000:00:1f.2: AHCI 0001.0100 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
ahci 0000:00:1f.2: flags: 64bit ncq sntf ilck stag pm led clo pmp pio slum part ems 
ahci 0000:00:1f.2: setting latency timer to 64
scsi0 : ahci
scsi1 : ahci
scsi2 : ahci
scsi3 : ahci
scsi4 : ahci
scsi5 : ahci
ata1: SATA max UDMA/133 abar m2048@0xfebff800 port 0xfebff900 irq 19
ata2: SATA max UDMA/133 abar m2048@0xfebff800 port 0xfebff980 irq 19
ata3: SATA max UDMA/133 abar m2048@0xfebff800 port 0xfebffa00 irq 19
ata4: SATA max UDMA/133 abar m2048@0xfebff800 port 0xfebffa80 irq 19
ata5: SATA max UDMA/133 abar m2048@0xfebff800 port 0xfebffb00 irq 19
ata6: SATA max UDMA/133 abar m2048@0xfebff800 port 0xfebffb80 irq 19
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata1.00: ATA-7: ST3320620AS, 3.AAE, max UDMA/133
ata1.00: 625142448 sectors, multi 0: LBA48 NCQ (depth 31/32)
ata1.00: configured for UDMA/133
ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata2.00: ATAPI: HL-DT-ST DVDRAM GH20NS10, EL00, max UDMA/100
ata2.00: configured for UDMA/100
ata3: SATA link down (SStatus 0 SControl 300)
ata4: SATA link down (SStatus 0 SControl 300)
ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata5.00: ATAPI: Optiarc DVD RW AD-7203S, 1.09, max UDMA/100
ata5.00: configured for UDMA/100
ata6: SATA link down (SStatus 0 SControl 300)
scsi 0:0:0:0: Direct-Access     ATA      ST3320620AS      3.AA PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 625142448 512-byte hardware sectors: (320 GB/298 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 0:0:0:0: [sda] 625142448 512-byte hardware sectors: (320 GB/298 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 sda8 sda9 >
sd 0:0:0:0: [sda] Attached SCSI disk
sd 0:0:0:0: Attached scsi generic sg0 type 0
scsi 1:0:0:0: CD-ROM            HL-DT-ST DVDRAM GH20NS10  EL00 PQ: 0 ANSI: 5
sr0: scsi3-mmc drive: 48x/48x writer dvd-ram cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.20
sr 1:0:0:0: Attached scsi CD-ROM sr0
sr 1:0:0:0: Attached scsi generic sg1 type 5
scsi 4:0:0:0: CD-ROM            Optiarc  DVD RW AD-7203S  1.09 PQ: 0 ANSI: 5
sr1: scsi3-mmc drive: 94x/94x writer dvd-ram cd/rw xa/form2 cdda tray
sr 4:0:0:0: Attached scsi CD-ROM sr1
sr 4:0:0:0: Attached scsi generic sg2 type 5
ahci 0000:03:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ahci 0000:03:00.0: AHCI 0001.0000 32 slots 2 ports 3 Gbps 0x3 impl SATA mode
ahci 0000:03:00.0: flags: 64bit ncq pm led clo pmp pio slum part 
ahci 0000:03:00.0: setting latency timer to 64
scsi6 : ahci
scsi7 : ahci
ata7: SATA max UDMA/133 abar m8192@0xfe9fe000 port 0xfe9fe100 irq 16
ata8: SATA max UDMA/133 abar m8192@0xfe9fe000 port 0xfe9fe180 irq 16
ata7: SATA link down (SStatus 0 SControl 300)
ata8: SATA link down (SStatus 0 SControl 300)
...
Comment 54 Alex Efros 2009-11-26 05:05:58 UTC
I've tested new HDD - Seagate Barracuda LP 2TB:


# hdparm -t /dev/sdb

/dev/sdb:
 Timing buffered disk reads:  332 MB in  3.02 seconds = 110.10 MB/sec


I've created two partitions at beginning of HDD, first for 20GB and second for 200GB. I've copied (using tar) my linux root partition to first (20GB) partition on new HDD and boot from new HDD (old HDD was disconnected from motherboard).


# fdisk -l /dev/sdb

Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x59a8da18

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        2612    20980858+  83  Linux
/dev/sdb2            2613       28721   209720542+  83  Linux


It's read/write speed with and without some activity while testing are higher
when compare to my old HDD (see comment #46 for description of W1/W2 columns):

Test results (numbers are MB/sec):
        W1/W2  R1/R2
 OLD:
  sda5  47/38  68/55
  sda6  27/27  45/43
  sda8  50/42  64/54
  sda9  42/34  50/35
 NEW:
  sda2  71/65  112/97

BUT. Overall system slowdown is SAME as with my old HDD. Of course, there no
way to measure "slowdown" in some absolute numbers (after loadavg and iowait
numbers was proven to have nothing with that bug), so we talk only about my
subjective feeling. But the feeling is, replacing old 320GB HDD with new 
2TB HDD (both are Seagate's) doesn't fixes this issue.

That mean, this slowdown doesn't related to disk fragmentation, because
it still exists on new partition populated using `tar cf|tar xf` commands.


Another interesting thing: if new HDD connected as sdb, and I boot from my
old HDD and run all these dd-tests on new HDD - the speed of i/o in tests
are slightly increased and there NO SLOWDOWN at all!


As example, if I boot from new HDD and run these 4 commands:
  dd if=/dev/zero of=test[1234] bs=1M count=8000
the avg speed of each dd will be 15.5MB/sec (62MB/sec overall write speed),
but if I run same test on new HDD while booted from old HDD,
the avg speed of each dd will be 18.6MB/sec (74.4MB/sec overall write speed).

Another test, running 2 these commands on new HDD while boot from new HDD:
  dd if=test[12] of=test[12]_2 bs=1M
the avg speed of each dd will be 13MB/sec (26MB/sec overall copy speed)
but if I run same test on new HDD while booted from old HDD,
the avg speed of each dd will be 14.6MB/sec (29.2MB/sec overall copy speed).


Any ideas what to check next?
Comment 55 Mikko Rantalainen 2009-11-26 13:44:38 UTC
Some tweaks you could try (as root) to reduce latencies under load:

# decrease SATA NCQ pipeline depth (I'd suggest minimum of 2)
echo 3 > /sys/block/sda/device/queue_depth

# decrease queue pipeline depth (throughput may suffer with too small number)
echo 4 > /sys/block/sda/queue/nr_requests

# decrease maximum command buffer (too small number will reduce throughput)
echo 64 > /sys/block/sda/queue/max_sectors_kb

# use CFQ scheduler (with settings below)
echo cfq > /sys/block/sda/queue/scheduler

# tweak timeouts for reads (sync) and writes (in ms)
echo 80 > /sys/block/sda/queue/iosched/fifo_expire_sync
echo 800 > /sys/block/sda/queue/iosched/fifo_expire_async

# tweak timeouts for time per process (ms)
echo 60 > /sys/block/sda/queue/iosched/slice_sync
echo 40 > /sys/block/sda/queue/iosched/slice_async

# decrease pipeline depth
echo 2 > /sys/block/sda/queue/iosched/quantum
echo 1 > /sys/block/sda/queue/iosched/slice_async_rq

# tweak idle time for process to continue after sync request
# may require RT patches in kernel to not lose throughput with small values
# the idea is to have the time in ms from the moment the application
# has received last read data until the moment the same application
# asks for more data, too big value will idle for nothing, too small
# will cause other programs to move the HDD head to other track
echo 1 > /sys/block/sda/queue/iosched/slice_idle

You may have to tweak above numbers a bit to nicely fit your hardware. I'm pretty happy with the above settings with HDD ST3500410AS plus CPU Intel E8400 on Acer EQ45M motherboard. I'm currently running kernel "2.6.31-9-rt #152-Ubuntu SMP PREEMPT RT".

Smaller timeouts and shorter pipelines will reduce latency but can reduce throughput seriously.

One more thing you could try: disable real sync to disk, only sync to drive cache (not safe if power is lost worth to check if it makes a difference)
 mount -o nobarrier,remount /
(and possibly other mount points, too).
Comment 56 Alex Efros 2009-11-29 06:13:32 UTC
> Some tweaks you could try (as root) to reduce latencies under load:

Thanks, I've tried these settings (but doesn't try to tune them for my hardware).
This doesn't solve this issue. I think these settings slightly improve overall
situation, but it's hard to measure real effect. And even if slowdown become,
say, 10% less noticeable, the interface is still very slow and copy speed is
still too low.

I've tried another one HDD with same model as my friend's HDD tested in
comment #43 - Samsung HD103UJ 1TB. It also very fast - actually that just
mean it works at full speed, just like all these HDD do in Windows but not
in Linux - and interface slowdown is much less noticeable even with 4 dd
doing copy of 8GB files in background.

And, again, this Samsung's speed is comparable to another new and fast
HDD speed - Seagate Barracuda LP 2TB - but that Seagate on my system work
not at full speed and interface slowdown with that Seagate is same as with
my current HDD.

So, for me, replacing my old HDD with that Samsung HD103UJ solved this issue!
Of course interface slowdown is still noticeable, but it's very small.
And HDD works at full speed, just like 2-3 years ago with kernel <2.6.18! :)
Comment 57 Dan Hollocher 2010-02-13 20:14:58 UTC
Hello linux gurus, I have a very similar issue as Alex, and I also have similar hardware.  I only have a single IDE port on my MOBO: the hdd is master and the cd burner is slave.  Like Alex, my file copies are quite slow, but for me the main problem is that my computer will lock up while burning a cd.  I have a dual core, and one core will get completely io locked during cd burning, and sometimes both cores.  If I try to do absolutely anything (in gnome), like open a webpage, or maximize and minimize a window, both cores will iolock for a couple of minutes.


The cd is burning at _only_ 4x, under a MB/s, well under the slow 10MBs/s I get for file copy, but still enough to completely lock up gnome.  I thought that maybe my hdd and cd burner were fighting over each other, so I tried different things hoping to cache the ISO that I was burning: cat it to /dev/null to put it in cache, and mounting K3Bs workspace location to tmpfs.  Those didn't help at all.  But that leads me to believe that some sort of io bus is getting bent out of shape.  maybe?

I've tested in windows, and not only can I burn the same cd+iso without any noticeable slow down, but I can also defrag the drive at the same time.  (ie, I can open a program like open office while those other 2 things are going)  Given this, and the fact that I have had this issue for the 3 years that I have been using linux, I believe that this is not an issue of hardware breaking down.

I also can't check or enable DMA with hdparm.  Is that normal?
$ sudo hdparm -d /dev/sdb

/dev/sdb:
 HDIO_GET_DMA failed: Inappropriate ioctl for device
Comment 58 Max Pilipenko 2010-02-25 11:07:49 UTC
I confirm the slowdowns. Started noticing that somewhere since 2.6.24 in Debian Etch, then moved to Debian Lenny, used 2.6.30 from backports, and now a custom-compiled 2.6.32.6.
My custom-build kernel features "Low-latency desktop" enabled, and it improved overall system responsiveness, but high iowait and slow transfer speeds are still around.

CPU: Intel Core2 6300
Motherboard: Intel D975XBX
HDD: 400GB Seagate 7200.10 ST3400620AS (SATA-II, NCQ, 16 Mb cache)

hdparm -t gives a reasonable reading about 60-70 Mb/s transfer rate, but copying large files between directories, to a USB HDD or over a 1 Gbit network shows about 2-3 Mb/s and iowait around 90-100%

I tried connecting my hard drive through motherboard secondary SATA controller Silicon Image, Inc. SiI 3114, but there were no improvements.
Tried connecting the hard drive to a completely different dual-core AMD system but saw the same sluggishness.
Comment 59 Con Kolivas 2010-04-30 01:46:36 UTC
The reports describe the following:

Kernel Version: 	2.6.18 - 2.6.30 

Looking at the 2.6.18 changelog, the following rather significant change that can't be ignored occurred at precisely 2.6.18:

commit b17fd9bceb99610f6dc7998c9a4ed6b71520be2b
[PATCH] Make CFQ the default IO scheduler

Prior to that, the Anticipatory IO scheduler AS was the default IO scheduler, and it no longer exists in the current kernels. I've not found that the deadline IO scheduler performed well at all under these loads so switching between CFQ and deadline having similar performance is not entirely unexpected.

So, the way to determine whether this change was due to the change to CFQ _from_ AS is to go to the last kernel that still included a good working AS and try comparing them. Ideally if people can still run 2.6.18 kernels on their machine, to try their workloads with the AS scheduler compared to the CFQ scheduler of the time.
Comment 60 Alex Efros 2010-05-19 03:44:30 UTC
Just short status update. Last 6 months with new HDD make it clear: bug is still there and still affect my system.

BUT. Unlike with several previous HDDs I tried before, with this HDD I experience slowdown very very seldom, like 1-2 times per month. It always unusually high I/O activity on my system at that time (like checking torrent's hash PLUS copying large file by network), and even in these cases interface slowdown much less noticeable than previously - i.e. switching between applications/windows make take several seconds, but previously in similar cases (with unusually high I/O load) I experience complete interface freeze for _minutes_.

I still use same 2.6.28 kernel (because it latests "stable" kernel with hardened (PaX/GrSec) patches available in official Gentoo portage for now). So, except changing HDD nothing was changed on my system, neither software nor my usual working tasks.
Comment 61 Max Pilipenko 2010-06-14 13:52:39 UTC
(In reply to comment #58)
Now I'm using 64-bit Debian Lenny with 2.6.32-bpo.4-amd64 kernel from backports.org and a brand new SATA 1 TB WDC WD10EARS-00Y5B1 mounted as /home, and a SATA 80 GB SAMSUNG HD080HJ used as /

Copying 1.5 GB file from /home to /tmp gets me around 4 MB/s and around 100% iowait
Total DISK READ: 4.98 M/s | Total DISK WRITE: 5.00 M/s
  PID USER      DISK READ  DISK WRITE   SWAPIN    IO>    COMMAND
 3973 root        4.98 M/s    4.98 M/s  0.00 % 98.79 % mc

Same results show up when I try to copy files to my USB-HDD. In Windows the speed is like 20 MB/s, but in Linux it's 3-5 MB/s.

What happened to the kernel? This 'bug' is like 3-4 years old, and still no one can find and fix it! I stick to Windows now at my home PC because it's really fast when it comes to large files handling. :(

I can't use 2.6.18 btw, the motherboard is too new for that kernel.
Comment 62 Tejun Heo 2010-06-14 14:34:44 UTC
Max, what you're seeing is pretty different from what Alex is reporting, so please file a separate bug report and please include the output of "dd if=/dev/null of=tmpfile bs=1M count=1024" for cases where tmpfile is on the USB hdd and the harddrive.

Alex, under heavy memory load, sometimes stutters like that can happen.  1-2 times per month under heavy memory load + much shorter stutter is a very different situation from stalls which go on for minutes.  With the data present, I can't tell what your original problem was.  I'm thinking either a hardware problem or somehow different/slower timing interacting weirdly with IO scheduler.  It would be interesting if you can try the old hard drive w/ 2.6.34 + ext4 and see whether the problem is still reproducible.

Thanks.
Comment 63 Tejun Heo 2010-06-14 14:38:30 UTC
Dan, please file a separate bug report.  What you're seeing is completely different from the problem this bug report is about.
Comment 64 Yuriy Lalym 2011-05-25 05:32:36 UTC
(In reply to comment #3)
> (In reply to comment #1)
>
> CPU - Core 2 Duo E8400,
> RAM - 8 Gb
> HDD - S31000340AS (1000 Gb, 32Mb Cache, 7200 об, S-ATA-II)
> FileSystem - XFS
>
> time cp -r qt-x11-opensource-src-4.5.1 qt-x11-opensource-src-4.5.1-1
> real    5m51.075s
> user    0m0.147s
> sys     0m2.192s

openSUSE 11.4 ( + http://download.opensuse.org/repositories/openSUSE:/Tumbleweed/standard/)
The same computer
Kernel Version: 2.6.39-30-desktop
FileSystem - XFS

After Qt unpacking I fulfill reboot.

time cp -r qt-everywhere-opensource-src-4.8.0-tp qt-everywhere-opensource-src-4.8.0-tp1

real    0m35.823s
user    0m0.149s
sys     0m5.247s

time rm -fr qt-everywhere-opensource-src-4.8.0-tp2 

real    0m7.955s
user    0m0.096s
sys     0m3.084s

It (qt-everywhere-opensource-src-4.8.0-tp2) is created in last session that there was an absolute warranty that files lie on a disk, instead of in storage buffers. Removal of just created directory looks so

time rm -fr qt-everywhere-opensource-src-4.8.0-tp1

real    0m0.001s
user    0m0.000s
sys     0m0.001s

Even if not to consider that Qt-4.8.0 it is more on volume and an amount of files - we have the TENFOLD gain in speed of copying and removal. Responsiveness of system, thus, the excellent.

ALL OF YOU COULD MAKE THE IMPROBABLE.
Comment 65 Perlover 2012-04-26 13:12:54 UTC
Good day, anybody!

I found an optimal options against this and like this bug
Anybody please try a following options.

I found that my kernels 2.6* and 3.2.* and 3.3.* versions of my server has periodical freezings 4-15 secs. I found that this occur in writeback time (flushing to disk) in time when 30sec expires for expired dirty pages occur. I tried many variants of dirty_* options and found optimal these:

I can suggest two veriants
Here 1st and 2nd variants
The second variant commented
Only uncommment second lines and nothing

#######################################
# every 3 sec look up for dirty status
# It for smooting writebacking, may be 100 will be better
echo 300    > /proc/sys/vm/dirty_writeback_centisecs

# Only 100Kb data of dirty pages and writeback...
# It very important option :)
echo 102400 > /proc/sys/vm/dirty_background_bytes

# second variant - uncomment it - but you will have frozens but rarely 
# echo 225280000 > /proc/sys/vm/dirty_background_bytes

# my a frozens happen at time of expiring of dirty pages (default 30 sec)
# i increased it (it doesn't mean for 1st variant - it will never happen)
echo 864000 > /proc/sys/vm/dirty_expire_centisecs

# I increased limit for non background writebacking (it never happens i think)
echo 10     > /proc/sys/vm/dirty_ratio

#######################################

I like 1st variant - my system now works smooth 
I found that freezings occurs when dirty pages are written to disk.
You can see it by this:

watch -n1 grep -A 1 dirty /proc/vmstat

New kernel features from 3.2.* version (writeback throttling) will not help to me. Now i tested kernel 3.3.2-6 of FC16 and it have a troubles too. But these settings work for me!

I don't have any time for detailed description
But if you will test it and it will help i will ready to discus for it
Sorry for my English :)

Bye!
Perlover

Note You need to log in before you can comment on or make changes to this bug.