Bug 194967

Summary: database server hangs
Product: IO/Storage Reporter: Tomas Mozes (hydrapolic)
Component: Block LayerAssignee: other_other
Status: RESOLVED OBSOLETE    
Severity: normal CC: bugzilla.kernel.org
Priority: P1    
Hardware: x86-64   
OS: Linux   
Kernel Version: 4.4, 4.9 Subsystem:
Regression: No Bisected commit-id:
Attachments: sysrq-l.log
sysrq-m.log
sysrq-w.log
sysrq-w.log (case 2)

Description Tomas Mozes 2017-03-23 13:17:12 UTC
Hello,
we are observing system hangs after upgrading kernel from 4.1 to 4.4. This happens on a database server running MariaDB, hot backuping with Percona Xtrabackup and when doing rsync (mostly deletion on the server). Happens on three different servers, on both ext4 and xfs. Can be reproduced on kernel 4.4.55, 4.9.16, does not happen with kernel 4.1.39. This is a virtual machine (xen-domU) having disks on lvm.

There is nothing in dmesg, but server load goes up, cannot even do `lsof` or `cat` files. A hard system reboot of the virtual machine puts it back to normal.

https://bugs.launchpad.net/percona-xtrabackup/+bug/1625984
https://jira.mariadb.org/browse/MDEV-10852
Comment 1 Tomas Mozes 2017-03-23 13:17:57 UTC
Created attachment 255453 [details]
sysrq-l.log

echo l > /proc/sysrq-trigger
Comment 2 Tomas Mozes 2017-03-23 13:18:42 UTC
Created attachment 255455 [details]
sysrq-m.log

echo m > /proc/sysrq-trigger
Comment 3 Tomas Mozes 2017-03-23 13:19:26 UTC
Created attachment 255457 [details]
sysrq-w.log

echo w > /proc/sysrq-trigger
Comment 4 Tomas Mozes 2017-03-23 13:25:05 UTC
Sadly, this is not related to https://bugzilla.kernel.org/show_bug.cgi?id=119841
Comment 5 Tomas Mozes 2017-05-30 07:22:41 UTC
Created attachment 256765 [details]
sysrq-w.log (case 2)

sysrq-w.log (case 2)
Comment 6 Tomas Mozes 2017-05-30 07:23:59 UTC
Attached a new occurrence with kernel 4.4.67.
Comment 7 Tomas Mozes 2017-10-27 06:06:37 UTC
Cannot reproduce with kernel 4.12+. Maybe related to #193331.