Bug 155821

Summary: regresion in re-read operation by iozone ~10%
Product: IO/Storage Reporter: mkocka (mkocka)
Component: OtherAssignee: io_other
Status: NEW ---    
Severity: normal CC: hladky.jiri, jmikovic, kkolakow, regressions, spetrovi
Priority: P1    
Hardware: x86-64   
OS: Linux   
Kernel Version: 4.8.0-0.rc2.git0.1 Subsystem:
Regression: No Bisected commit-id:
Attachments: ibm-x3650 ext4
ibm-x3650 xfs
IBM x3750 M4 with ext4 on SAS SSD
IBM x3750 M4 with xfs on SAS SSD
iozone reread SAS HDD
iozone reread SATA SSD

Description mkocka@redhat.com 2016-09-02 13:38:34 UTC
Created attachment 231921 [details]
ibm-x3650 ext4

Hello 
we see regression in re-read operation with both ext4 and xfs file system in out of cache IOZONE test. 

I am adding the tests outputs to the attachments. 

The storage configuration is: 
for rambo-01 machine 1x 200GB SAS SSD
for ibm-x3650 LVM with 4x 300GB SAS HDD

Note: I compared two upstream kernels 4.8.0-0.rc2.git0.1 against 4.7.0-1 however, the regression is about 10%. How ever, when compared with RHEL-7.2 kernel, the regression is even bigger.
Comment 1 mkocka@redhat.com 2016-09-02 13:39:00 UTC
Created attachment 231931 [details]
ibm-x3650 xfs
Comment 2 mkocka@redhat.com 2016-09-02 13:39:53 UTC
Created attachment 231941 [details]
IBM x3750 M4 with ext4 on SAS SSD
Comment 3 mkocka@redhat.com 2016-09-02 13:40:07 UTC
Created attachment 231951 [details]
IBM x3750 M4 with xfs on SAS SSD
Comment 4 The Linux kernel's regression tracker (Thorsten Leemhuis) 2016-09-11 13:16:45 UTC
Did you report this to some list? LKML maybe? That might be the better approach to get attention…
Comment 5 mkocka@redhat.com 2016-10-24 12:19:50 UTC
Hello, 

I have devoted more attention to this problem. I rerun the iozone test on the both SATA SSD and SAS HDD drives with XFS file system. 
The regression is there for the re-read operation since the kernel-4.7.0-1, the 4.6.0-1 was OK, as you can see in attached images. 

SATA SSD drive, iozone re-read operation 

File size [MB] | kernel 4.6 [MB] | err [%] | kernel 4.7 [MB] | err [%] | difference [%]
----------------------------------------------------------------------------------------
    1.0 | 13269.10 | 5.0 | 10416.83 | 1.9 | -21.5
    2.0 | 13015.61 | 4.7 | 10024.62 | 0.2 | -23.0
    4.0 | 12745.33 | 4.5 | 10050.46 | 0.2 | -21.1
    8.0 | 12905.56 | 3.1 |  9934.37 | 1.4 | -23.0
   16.0 | 11294.55 | 6.7 |  9627.13 | 0.0 | -14.8
   32.0 |  7417.55 | 4.8 |  6454.81 | 1.8 | -13.0
   64.0 |  6789.36 | 3.6 |  5759.64 | 2.0 | -15.2
  128.0 |  6768.6  | 0.8 |  5930.25 | 1.7 | -12.4
  256.0 |  7199.7  | 0.0 |  6101.62 | 0.7 | -15.2
  512.0 |  7456.02 | 0.1 |  6234.62 | 0.4 | -16.4
 1024.0 |  7503.09 | 0.4 |  6230.33 | 0.2 | -17.0
 2048.0 |  7509.11 | 0.4 |  6177.36 | 0.1 | -17.7
 4096.0 |  7486.92 | 0.1 |  6219.58 | 0.5 | -16.9
 8192.0 |  6860.39 | 0.0 |  5760.61 | 0.3 | -16.0
16384.0 |  6702.57 | 0.1 |  5624.17 | 0.1 | -16.1
32768.0 |  1215.87 | 0.8 |   942.31 |12.8 | -22.5
65536.0 |   511.72 | 0.2 |   512.58 | 0.2 | 0.2

SATA SSD drive, iozone re-read operation

File size [MB] | kernel 4.6 [MB] | err [%] | kernel 4.7 [MB] | err [%] | difference [%]
----------------------------------------------------------------------------------------
    1.0 |  9281.34 | 29.5 | 9949.28 | 2.3 | 7.2
    2.0 |  8245.63 | 25.0 | 9732.59 | 2.1 | 18.0
    4.0 | 10183.30 |  2.7 | 9454.31 | 1.2 | -7.2
    8.0 | 10329.19 | 11.3 | 9093.59 | 4.7 | -12.0
   16.0 | 10356.85 | 11.1 | 8913.64 | 2.5 | -13.9
   32.0 |  7495.62 |  5.6 | 6217.42 | 2.4 | -17.1
   64.0 |  6608.68 |  7.6 | 5749.62 | 2.1 | -13.0
  128.0 |  7012.76 |  5.0 | 5927.56 | 1.1 | -15.5
  256.0 |  7203.37 |  1.7 | 6130.53 | 0.9 | -14.9
  512.0 |  7429.33 |  0.5 | 6219.21 | 0.4 | -16.3
 1024.0 |  7470.16 |  0.4 | 6247.00 | 0.3 | -16.4
 2048.0 |  7468.32 |  0.2 | 6209.81 | 0.8 | -16.9
 4096.0 |  7475.38 |  0.1 | 6065.46 | 1.8 | -18.9
 8192.0 |  6854.90 |  0.3 | 5687.97 | 0.2 | -17.0
16384.0 |  6678.79 |  0.3 | 5594.29 | 0.4 | -16.2
32768.0 |   444.30 |  0.7 |  483.13 |10.3 | 8.7
65536.0 |   304.49 |  0.0 |  250.57 | 0.0 | -17.7


The machine is : LENOVO System x3550 M5
Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
32768 MB DDR4 2133 MHz
I used numactl to pin iozone only to socket 0. Raid controller is connected to socket 0. 

The file system is XFS, but it is similar in EXT4 too, same regression.  

Please note, that the system have 32GB RAM, operation is doing mostly in-cache. 

We think that this can be caused by some virtual memory issue? 

I am going to post this also to kernel mailing list.
Comment 6 mkocka@redhat.com 2016-10-24 12:21:17 UTC
Created attachment 242491 [details]
iozone reread SAS HDD
Comment 7 mkocka@redhat.com 2016-10-24 12:21:38 UTC
Created attachment 242501 [details]
iozone reread SATA SSD