Created attachment 162291 [details] panic log Since kernel version 3.16 we are getting kernel panics and the panic message *might* suggest that it's related to the raid controller we use. I did try the kernel versions 3.17, 3.18 and also and it keeps crashing. 3.15.10 runs totally fine though. I can reproduce the crash 100% with a simple DD command dd if=/dev/zero of=test bs=1G count=10 I've attached the backtrace and tw_cli configruation options. We have several cluster nodes so I can try patches or give additional information when necessary.
Created attachment 162301 [details] tw_cli raid config
I tried to mail "Adam Radford <linuxraid@lsi.com>" directly as stated in drivers/scsi/3w-sas.c but the mail address doesn't exist anymore.
Merlin, 1.) Can you enable CONFIG_DMA_API_DEBUG in your kernel conf and send complete /var/log/messages along with back trace. 2.) From existing logs I can figure out that crash happened at below place. static void intel_unmap_sg(struct device *dev, struct scatterlist *sglist, int nelems, enum dma_data_direction dir, struct dma_attrs *attrs) { intel_unmap(dev, sglist[0].dma_address); } sglist[0] is NULL and while accessing dma_address (which is 0x10th index as per below definition) struct scatterlist { #ifdef CONFIG_DEBUG_SG unsigned long sg_magic; #endif unsigned long page_link; unsigned int offset; unsigned int length; dma_addr_t dma_address; < -- 0x10th index #ifdef CONFIG_NEED_SG_DMA_LENGTH unsigned int dma_length; #endif }; Crash detail - [ 1005.899543] Hardware name: Supermicro X9DR7/E-(J)LN4F/X9DR7/E-(J)LN4F, BIOS 3.0a 09/10/2013 [ 1005.899971] task: ffffffff8160a460 ti: ffffffff815f4000 task.ti: ffffffff815f4000 [ 1005.900401] RIP: 0010:[<ffffffff813db181>] [<ffffffff813db181>] intel_unmap_sg+0x1/0x10 ~ kashyap
Hi Kashyap, 1. I enabled CONFIG_DMA_API_DEBUG , strangely enough after enabling it I couldn't even boot into the system anymore because it would crash when trying to access the root partition (after the kernel image was loaded). I think the problem was the initial xfs_repair that is done on bootup. So I took the same kernel and copied it to a USB device and booted from that on the same host into a system that also resides on the USB disk. Then I mounted /dev/sda (which is the first device on the raid controller which is the root partition). Then I tried to reproduce the panic with the simple dd that was used before without success. I then compiled the kernel sources with -j25 on the mounted root partition 2 times without any problems. Then I chrooted onto that partition and did the same and could finally reproduce the crash. I am not sure if that backtrace is more useful though. I have also attached the /var/log/messages but of course that crash wasn't saved in it. I also added my kernel config.
Created attachment 162641 [details] syslog before crash
Created attachment 162651 [details] backtrace with CONFIG_DMA_API_DEBUG enabled
Created attachment 162661 [details] kernel config
Hi Kashyap, is there any additional info you need or do you know a better way to create a useable backtrace for you?
I've been bitten by this as well, on a 3ware 9750. It's currently preventing from using recent kernels on the server in question - which is an issue, because while 3.16.1 is rock-solid for scsi it has an intermittent wireless oops, and 3.17.1 oopses out with this (same backtrace as the existing attachment).
Testing with 3.19 shows it's still there for me.
I can confirm it's still happening with 3.19 here. @Alex: What server hardware are you using? We are using Supermicro boards and Intel CPUs.
Created attachment 166681 [details] Hardware of a second machine with the issue Intel CPUs on an ASUS board - here's the output of lshw on the machine.
My old email id is not working now, so I have to update my email id to new avagotech domain. Sorry for delay in response. Does this mean issue does not happen with 3.16.1 but only happens with 3.17.1 (onwards) ? This is good data point. Can you share whole dmesg logs for both 3.16.1 and 3.17.1 ? Also are you using intel_iommu=on in your config ? Can you boot without enabling Intel IOMMU? New logs with ""CONFIG_DMA_API_DEBUG" does not help much. You may have to post complete logs. We may have some interesting data before crash. Thanks, Kashyap
@Kashyap For me it started with 3.16 on all our servers with 3ware/lsi/avagotech controllers. I rechecked in our internal ticket system. We are using IOMMU so I completely disabled it, recompiled the kernal but getting the same error. I also attached my kernel config before so maybe you can use that config and compile the kernel on your side. Running with that config I can reproduce the crash with a simple dd command: dd if=/dev/zero of=test bs=1G count=10
(In reply to merlin from comment #14) > @Kashyap > > For me it started with 3.16 on all our servers with 3ware/lsi/avagotech > controllers. > I rechecked in our internal ticket system. > > We are using IOMMU so I completely disabled it, recompiled the kernal but > getting the same error. > > I also attached my kernel config before so maybe you can use that config and > compile the kernel on your side. > Running with that config I can reproduce the crash with a simple dd command: > dd if=/dev/zero of=test bs=1G count=10 Can you share whole dmesg logs for the failure even if IOMMU is disabled ? I will try to reproduce at my setup as I was able to find same h/w. ~ Kashyap
Created attachment 173931 [details] panic_screenshot screenshot of kernel panic
I am unable to get a dmesg output since the system crashes on boot. It looks like it's happening when udev is initialized. Sorry for the screenshot but the system crashed before any network came up.
Created attachment 174171 [details] dmesg I finally managed to get a dmesg output with the crashing kernel. Especially this part might indicate the root of the problem? [Thu Apr 16 13:02:10 2015] ------------[ cut here ]------------ [Thu Apr 16 13:02:10 2015] WARNING: CPU: 6 PID: 1736 at lib/dma-debug.c:601 debug_dma_assert_idle+0x17c/0x1e0() [Thu Apr 16 13:02:10 2015] 3w-sas 0000:03:00.0: DMA-API: cpu touching an active dma mapped cacheline [cln=0x0000000101ff8e00] [Thu Apr 16 13:02:10 2015] Modules linked in: [Thu Apr 16 13:02:10 2015] CPU: 6 PID: 1736 Comm: mount Not tainted 4.0.0-gentoo #1 [Thu Apr 16 13:02:11 2015] Hardware name: Supermicro X9DR7/E-(J)LN4F/X9DR7/E-(J)LN4F, BIOS 3.0a 09/10/2013 [Thu Apr 16 13:02:11 2015] ffffffff8170e0f5 ffff88406050fcb8 ffffffff814c8778 000000000000009a [Thu Apr 16 13:02:11 2015] ffff88406050fd08 ffff88406050fcf8 ffffffff8107c250 ffff884060422e08 [Thu Apr 16 13:02:11 2015] ffff8840647fe5e0 0000000000000206 ffffea00e1bf9ce8 ffff884060422e08 [Thu Apr 16 13:02:11 2015] Call Trace: [Thu Apr 16 13:02:11 2015] [<ffffffff814c8778>] dump_stack+0x45/0x57 [Thu Apr 16 13:02:11 2015] [<ffffffff8107c250>] warn_slowpath_common+0x80/0xc0 [Thu Apr 16 13:02:11 2015] [<ffffffff8107c2d1>] warn_slowpath_fmt+0x41/0x50 [Thu Apr 16 13:02:11 2015] [<ffffffff812b9dfc>] debug_dma_assert_idle+0x17c/0x1e0 [Thu Apr 16 13:02:11 2015] [<ffffffff81114814>] do_wp_page+0xc4/0x810 [Thu Apr 16 13:02:11 2015] [<ffffffff8111744c>] handle_mm_fault+0xb3c/0x10f0 [Thu Apr 16 13:02:11 2015] [<ffffffff8111ca6d>] ? do_mmap_pgoff+0x34d/0x400 [Thu Apr 16 13:02:11 2015] [<ffffffff81070ec6>] __do_page_fault+0x146/0x390 [Thu Apr 16 13:02:11 2015] [<ffffffff8107114c>] do_page_fault+0xc/0x10 [Thu Apr 16 13:02:11 2015] [<ffffffff814cf252>] page_fault+0x22/0x30 [Thu Apr 16 13:02:11 2015] ---[ end trace 326b0530baa65706 ]--- I was still able to login with root after that and reproduce the crash with another backtrace. See the next attachment.
Created attachment 174181 [details] crash-backtrace
Created attachment 174421 [details] dmesg for the commit that git bisect landed on I'm getting a similar panic (attached). $lspci | grep 3ware 05:00.0 RAID bus controller: 3ware Inc 9750 SAS2/SATA-II RAID PCIe (rev 05) $uname -a Linux localhost 3.16.0-rc5+ #18 SMP PREEMPT Wed Apr 8 20:11:03 2015 x86_64 Intel(R) Xeon(R) CPU X5482 @ 3.20GHz GenuineIntel GNU/Linux root@localhost /usr/src/linux $git rev-parse HEAD 74665016086615bbaa3fa6f83af410a0a4e029ee The panic bisected to: 74665016086615bbaa3fa6f83af410a0a4e029ee is the first bad commit commit 74665016086615bbaa3fa6f83af410a0a4e029ee Author: Christoph Hellwig <hch@lst.de> Date: Wed Jan 22 15:29:29 2014 +0100 scsi: convert host_busy to atomic_t Avoid taking the host-wide host_lock to check the per-host queue limit. Instead we do an atomic_inc_return early on to grab our slot in the queue, and if necessary decrement it after finishing all checks. I was unable to trigger the panic by writing to the 3w array with: $dd if=/dev/zero of=test bs=1G count=10 or even: $dd if=/dev/zero of=test bs=1G count=100 Instead, it takes somewhere between a few hours and 2 days of writing to panic, so the good bisects were run for 3 days to make sure. Crash happens with and without ZFS modules loaded but most of the bisect was done with because I couldn't take the array out of the equation for a month. Both cases are attached along with .config and bisect log. I'll recompile without IOMMU and post the results, further suggestions appreciated.
Created attachment 174431 [details] git bisect log between 3.14.33 and 3.18.7
Created attachment 174441 [details] .config
Created attachment 174451 [details] dmesg for commit 'scsi: convert device_busy to atomic_t'
Created attachment 174461 [details] dmesg for panic without zfs modules
Just to clarify, I couldn't trigger this by writing to the array from /dev/zero at all (maybe because ZFS), all panics were triggered by writing real data from another array on the same controller.
Created attachment 174581 [details] dmesg for the commit that git bisect landed on, with IOMMU disabled
I am unable to reproduce this on any of the 4.x.y kernels.