Bug 91691
Summary: | Extremely slow disk I/O, high iowait in 3.16 and newer kernels, unless blktrace is running | ||
---|---|---|---|
Product: | IO/Storage | Reporter: | Andreas Klauer (Andreas.Klauer) |
Component: | Block Layer | Assignee: | Jens Axboe (axboe) |
Status: | NEEDINFO --- | ||
Severity: | normal | CC: | dave, gfa |
Priority: | P1 | ||
Hardware: | x86-64 | ||
OS: | Linux | ||
Kernel Version: | 3.16 | Subsystem: | |
Regression: | No | Bisected commit-id: | |
Attachments: |
cpuinfo
dmesg lspci 3.15.10 btrace vda3 statistics 3.15.10 btrace vda statistics 3.16.6 btrace vda3 statistics 3.16.6 btrace vda statistics |
Description
Andreas Klauer
2015-01-21 16:17:36 UTC
Created attachment 164171 [details]
dmesg
Created attachment 164181 [details]
lspci
Would it be possible to use blktrace to log to a different machine, assuming that then still shows the slowdown? That might help us further pinpoint where things are going wrong. You can do that ala: (on other machine, same network): # blktrace -l then do blktrace -s other-machine /dev/vda If you have some short isolated sequence of IO you can do, that would help. It seems to behave the same over network as locally. But there seems to be a difference between running blktrace on /dev/vda as opposed to /dev/vda3. The result in 3.16.6, blktrace -s -h othermachine /dev/vda3: 512+0 records in 512+0 records out 536870912 bytes (537 MB) copied, 0.391734 s, 1.4 GB/s real 0m12.161s user 0m0.000s sys 0m0.397s server: connection from 37.120.172.165 server: end of run for 37.120.172.165:vda3 === vda3 === CPU 0: 440602 events, 20654 KiB data CPU 1: 1 events, 1 KiB data Total: 440603 events (dropped 0), 20654 KiB data The result in 3.16.6, blktrace -s -h othermachine /dev/vda: 512+0 records in 512+0 records out 536870912 bytes (537 MB) copied, 262.732 s, 2.0 MB/s real 4m22.735s user 0m0.000s sys 0m0.645s server: connection from 37.120.172.165 server: end of run for 37.120.172.165:vda === vda === CPU 0: 246610 events, 11560 KiB data CPU 1: 251797 events, 11804 KiB data Total: 498407 events (dropped 0), 23363 KiB data I'll attach some btrace statistics (for 3.15, 3.16 and vda, vda3) in case they are of help. Created attachment 164191 [details]
3.15.10 btrace vda3 statistics
Created attachment 164201 [details]
3.15.10 btrace vda statistics
Created attachment 164211 [details]
3.16.6 btrace vda3 statistics
Created attachment 164221 [details]
3.16.6 btrace vda statistics
|