Created attachment 29062 [details] utility that helps to reproduce the issue I'm experirncing problems with performance of random IO operations of dm_crypt-encrypted devices. Sequential IO is good (using AES-NI gives almost no difference comparing with unencrypted device), but random IO is poor comparing with unencrypted devices. For benchmarking i've used simple utility, that reading small pieces of data from a raw disk device, in a random access pattern. The utility measures the number of seeks per second (reading operations per second), the device can do. The source code of utility is attached. Also avalilable via link http://www.linuxinsight.com/files/seeker.c I've tested 2 RAID arrays: 1. RAID 10 of 4 x Intel X25-E SSD drives (/dev/sdb) 2. RAID 10 of 8 x Seagate Savvio 15K 2.5" SAS drives (/dv/sdc) Both arrays is handled by Adaptec 5805Z RAID card. I've created encrypted mappings with: cryptsetup --cipher=aes-cbc-essiv:sha256 --key-size=256 create cfs1 /dev/sdb cryptsetup --cipher=aes-cbc-essiv:sha256 --key-size=256 create cfs2 /dev/sdc Testing results: 1. SSD array: /dev/sdb: 3320 seeks/second /dev/mapper/cfs1: 99 seeks/second 2. SAS array: /dev/sdc: 257 seeks/second /dev/mapper/cfs2: 85 seeks/second As you see the difference between clear and encrypted devices is huge, especially in SSD case. The current behavior makes almost impossible to use encrypted devices to handle databases. You know, the typical database workload is random IO operations. These tests done using kernel: 2.6.32-24-server #39-Ubuntu SMP x86_64 GNU/Linux Some months before i've investigated also other kernels and distributions. The behavior is very similar so I suppose vanilla kernel is also affected. If you need any additional info, a can provide it. Thank you.
Well, this is interesting. I don't think you will see difference in other distro/upstream kernel - encryption threads in dm-crypt are still the same. I tested quite quict system (with AES-NI) and over simpe SATA device it is seek slowdown acceptable (~10%). But you can easily isolate problem using zero-backed device: create "zero device" - "ideal disk" # dmsetup create x --table "0 312500000 zero" # ./seeker /dev/mapper/x ... Results: 275599 seeks/second, 0.00 ms random access time map dm-crypt over it (default is aes-cbc-essiv:sha256 / 256bit key) # echo xxx | cryptsetup create x_c /dev/mapper/x # ./seeker /dev/mapper/x_c Results: 46533 seeks/second, 0.02 ms random access time So the difference is quite significant. (I am going to test prepared changes in dm-crypt threads - maybe it can help here as side effect because it tries to keep work on current core.) But there will be always quite big overhead, let see if it can be improved.
Milan, you wrote at dm-devel: > I have some ideas how to improve it, but the change must follow patches which > are currently waiting for upstream inclusion. How do you think, how much time it will take? > (Actually the per-cpu thread patches helps in this case too as side effect.) Did you made benchmarking with patched dm-crypt? Could you please provide the results? Thank you.
(In reply to comment #2) > Did you made benchmarking with patched dm-crypt? > Could you please provide the results? Nothing extra important but slight improvement, actualy I run this synthetic test just to verify it is not worse:-) Unpatched 2.6.34 zero: Results: 210908 seeks/second, 0.00 ms random access time crypt over zero: Results: 14401 seeks/second, 0.07 ms random access time Patched 2.6.34 (per core patches for now in http://mbroz.fedorapeople.org/dm-crypt/) zero: Results: 214021 seeks/second, 0.00 ms random access time crypt over zero: Results: 16237 seeks/second, 0.06 ms random access time Running database over dm-crypt is not good idea anyway. But there is surely some possible optimisation.
(In reply to comment #3) > But there is surely some possible optimisation. Sure. Let's look one more time to my results of benchmarking SSD array (kernel 2.6.32): 1. /dev/sdb: Results: 2922 seeks/second, 0.34 ms random access time 2. Encrypted /dev/sdb: 97 seeks/second, 10.27 ms random access time 10 ms of each encrypted seek is: 1. 0.34 ms to seek and read 4K block from underlying /dev/sdb; 2. 0.02 ms to decrypt 4K block with AES-NI (or even less); 3. 9.34 the rest. What for? 9 ms in terms of Xeon X5680 is rather huge amount of time. Can you explain what kernel does this time? >Running database over dm-crypt is not good idea anyway. Why? Now I'm running ~10G database with ~200 simultaneous users. The database is on dm-encrypted SSD array. The system running 2.6.24 with realtime preemption patch (without RT-patch kernel performs much worse in random encrypted IO). RT-patched kernel 2.6.24 gives ~2400 seeks/second over encrypted SSD device (~4200 seeks without encryption). This is rather good performance of random encryption, isn't it. And this is rather old host without AES-NI. Now I'm working on new modern server, but with modern kernels I cannot get more than 100 seeks/second (more than 300 seeks on RT-patched kernels). :-( I'm ready to do more testing and provide more info, if you need it to make the issue more clear. Thank you.
It seems, this issue is somehow interrelated with system caching. If doing direct IO (with O_DIRCET flag) then random reads and writes from/to encrypted device performs much better. I've done tests comparing syncronous buffered random writes (O_SYNC) with non-buffered writes (O_DIRECT) using fio tool (http://freshmeat.net/projects/fio/) 1. SSD array 1.1 Not encrypted: Buffered IO: bw=78982KB/s, iops=9872 Direct IO: bw=83626KB/s, iops=10453 1.2 Encryped: Buffered IO: bw=2914KB/s, iops=364 Direct IO: bw=45714KB/s, iops=5714 2. SAS array 1.1 Not encrypted: Buffered IO: bw=20410KB/s, iops=2551 Direct IO: bw=19665KB/s, iops=2458 1.2 Encryped: Buffered IO: bw=1327KB/s, iops=165 Direct IO: bw=5338KB/s, iops=667 You see, in case of not encrypted devices buffered and non-biffered writes performs very similar. I case of encrypted devices we see significant performance flop with buffered writes. Job configuration file for fio is attached (fio-randwrite.conf).
Created attachment 34792 [details] job configuration file for fio
FYI - after some changes in 2.6.38, there is major rework of plug/unplug queue handling planned. https://lkml.org/lkml/2011/1/21/506 I did not yet test it over dmcrypt but I would expect this should help.