Bug 35682 - killing requests for dead queue
Summary: killing requests for dead queue
Status: CLOSED CODE_FIX
Alias: None
Product: IO/Storage
Classification: Unclassified
Component: SCSI (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: linux-scsi@vger.kernel.org
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-05-23 10:34 UTC by Zbigniew
Modified: 2012-05-12 13:41 UTC (History)
5 users (show)

See Also:
Kernel Version: 2.6.39
Subsystem:
Regression: No
Bisected commit-id:


Attachments

Description Zbigniew 2011-05-23 10:34:01 UTC
When yesterday booted newly compiled 2.6.39 kernel up - directly after the compilation - there wasn't anything bad. But today in the morning system was unable to complete boot-up, staying at the message: "sde: 
killing requests for dead queue".

Maybe new kernel doesn't like my USB card reader? "cat /proc/scsi/scsi" shows:

Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: STM3320418AS     Rev: CC38
  Type:   Direct-Access                    ANSI  SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: HL-DT-ST Model: DVD-RAM GH22LS30 Rev: 1.01
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi4 Channel: 00 Id: 00 Lun: 00
  Vendor: Generic  Model: USB SD Reader    Rev: 1.00
  Type:   Direct-Access                    ANSI  SCSI revision: 00
Host: scsi4 Channel: 00 Id: 00 Lun: 01
  Vendor: Generic  Model: USB CF Reader    Rev: 1.01
  Type:   Direct-Access                    ANSI  SCSI revision: 00
Host: scsi4 Channel: 00 Id: 00 Lun: 02
  Vendor: Generic  Model: USB SM Reader    Rev: 1.02
  Type:   Direct-Access                    ANSI  SCSI revision: 00
Host: scsi4 Channel: 00 Id: 00 Lun: 03
  Vendor: Generic  Model: USB MS Reader    Rev: 1.03
  Type:   Direct-Access                    ANSI  SCSI revision: 00

No memory cards neither USB sticks were inserted during boot-up.

It seems, that issue discussed in the thread, which I've found at:

https://lkml.org/lkml/2011/4/11/175

...still hasn't been resolved. Well, at least not completely.
Comment 1 Michael Tokarev 2011-06-09 17:12:32 UTC
The same happens to me with 2.6.39.1 - when I boot with an USB stick inserted it hangs with the same message, but it boots fine w/o the stick attached.  I can hotplug it later and it works fine.  What's "interesting" is that I discovered the issue while trying to boot a new kernel from this USB stick, so in that case it obviously _is_ inserted during boot...

There's one more case which may be related -- see e.g. https://lkml.org/lkml/2011/6/3/33 -- that's how 2.6.38.8 were hanging here too, but it appears that these QUEUE_FLAG_DEAD changes has been incorporated into .39.1 already.
Comment 2 alust 2011-06-26 13:48:16 UTC
Hello,

I've run into this issue too. Kernel 2.6.39 compiled from linux-source-2.6.39 debian package.

Any time I insert a flash stick or boot with the stick inserted I will got 

"killing requests for dead queue" message that is followed by "lost interrupt" and "ide-atapi cmd 0x1e timed out".

Please let me know if any additioanal information is required.
Comment 3 rpansky 2011-06-27 18:33:18 UTC
I have encountered a similar issue. A panic happens immediately after a USB stick insertion (sometimes). The message "scsi: killing requests for dead queue" is also present. However, in some cases the message was not accompanied with panic.
Notably, I could not reproduce the problem with 2.6.39.2 although that message was there too.

I hereby provide a panic trace for 2.6.39.1

(Please note that "Tainted: W" is due to this probably harmless warning:
WARNING: at kernel/printk.c:293 do_syslog+0xbf/0x4f0()
Hardware name: GA-MA770-UD3
Attempt to access syslog with CAP_SYS_ADMIN but no CAP_SYSLOG (deprecated).
)

BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<ffffffff811a64eb>] blk_dequeue_request+0x1b/0x60
PGD 226d7f067 PUD 226e77067 PMD 0 
Oops: 0002 [#1] SMP 
last sysfs file: /sys/devices/virtual/bdi/8:48/uevent
CPU 3 
Modules linked in: usb_storage snd_seq snd_seq_device nfsd nfs lockd sunrpc bridge stp llc iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state iptable_filter xt_DSCP xt_dscp xt_owner xt_multiport xt_mark xt_connmark nf_conntrack ip_tables x_tables netconsole tun it87 hwmon_vid hwmon snd_hda_codec_realtek snd_hda_intel snd_hda_codec ohci_hcd snd_pcm r8169 snd_timer snd ehci_hcd soundcore usbcore snd_page_alloc i2c_core mii

Pid: 16, comm: ksoftirqd/3 Tainted: G        W   2.6.39.1 #1 Gigabyte Technology Co., Ltd. GA-MA770-UD3/GA-MA770-UD3
RIP: 0010:[<ffffffff811a64eb>]  [<ffffffff811a64eb>] blk_dequeue_request+0x1b/0x60
RSP: 0018:ffff88022e8afc98  EFLAGS: 00010046
RAX: ffff88022d503408 RBX: ffff88022d502e40 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff88022d502e40 RDI: ffff88022d502e40
RBP: ffff88022d502e40 R08: 0000000000000000 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000400 R12: ffff88022d502878
R13: ffff88022d502b40 R14: ffff880224d90040 R15: 0000000000000246
FS:  00007f0465c7c700(0000) GS:ffff88022fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000226d7e000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ksoftirqd/3 (pid: 16, threadinfo ffff88022e8ae000, task ffff88022e86e2d0)
Stack:
 ffffffff811a6ef9 ffff88022d502e40 ffffffff811a7d45 0000000000000808
 0000000000000000 0000000000000286 ffff88022d502878 ffff88022d502878
 ffffffff8125192c ffff880224d90000 ffff880224d93800 ffff880224d90000
Call Trace:
 [<ffffffff811a6ef9>] ? blk_start_request+0x9/0x40
 [<ffffffff811a7d45>] ? blk_peek_request+0xd5/0x1d0
 [<ffffffff8125192c>] ? scsi_request_fn+0x3ec/0x4a0
 [<ffffffff811a67e8>] ? blk_run_queue+0x28/0x50
 [<ffffffff81250df1>] ? scsi_run_queue+0xd1/0x270
 [<ffffffff810c500a>] ? kmem_cache_free+0x9a/0xb0
 [<ffffffff81251bfb>] ? scsi_next_command+0x3b/0x60
 [<ffffffff812527e7>] ? scsi_io_completion+0x2c7/0x5b0
 [<ffffffff811ad3cd>] ? blk_done_softirq+0x6d/0x80
 [<ffffffff8103f491>] ? __do_softirq+0x91/0x120
 [<ffffffff8103f5c5>] ? run_ksoftirqd+0xa5/0x150
 [<ffffffff8103f520>] ? __do_softirq+0x120/0x120
 [<ffffffff8103f520>] ? __do_softirq+0x120/0x120
 [<ffffffff81054136>] ? kthread+0x96/0xa0
 [<ffffffff8134be94>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff810540a0>] ? kthread_worker_fn+0x120/0x120
 [<ffffffff8134be90>] ? gs_change+0xb/0xb
Code: 02 00 00 5b c3 31 d2 e9 fb fe ff ff 0f 1f 40 00 48 8b 07 48 8b 4f 38 48 39 c7 74 46 48 83 7f 78 00 75 3b 48 8b 57 08 48 89 50 08 
 89 02 8b 47 40 48 89 3f 48 89 7f 08 f6 c4 40 74 1f 83 7f 44 
RIP  [<ffffffff811a64eb>] blk_dequeue_request+0x1b/0x60
 RSP <ffff88022e8afc98>
CR2: 0000000000000000
---[ end trace 9658ff6466e3ea44 ]---

Of some interest might also be the following (tainted by nvidia proprietary module) trace:

kernel BUG at block/blk-core.c:1935!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/virtual/bdi/8:48/uevent
CPU 0 
Modules linked in: usb_storage nfsd nfs lockd sunrpc bridge stp llc iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state iptable_filter xt_DSCP xt_dscp xt_owner xt_multiport xt_mark xt_connmark nf_conntrack ip_tables x_tables snd_seq snd_seq_device netconsole tun it87 hwmon_vid hwmon snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_pcm snd_timer snd ohci_hcd ehci_hcd r8169 soundcore usbcore snd_page_alloc i2c_core mii [last unloaded: nvidia]

Pid: 3, comm: ksoftirqd/0 Tainted: P        W   2.6.39.1 #1 Gigabyte Technology Co., Ltd. GA-MA770-UD3/GA-MA770-UD3
RIP: 0010:[<ffffffff811a651e>]  [<ffffffff811a651e>] blk_dequeue_request+0x4e/0x60
RSP: 0018:ffff88022e881c98  EFLAGS: 00010086
RAX: ffff88022daea2b0 RBX: ffff88022dae9ce8 RCX: 0000000000000002
RDX: 000000008102d46e RSI: ffff88022dae9720 RDI: ffff88022dae9ce8
RBP: 0000000000000286 R08: 0000000000000000 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000400 R12: ffff88022dae9720
R13: ffff88022dae9ce8 R14: ffff880224112840 R15: 0000000000000246
FS:  00007f6f93b08700(0000) GS:ffff88022fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fff561bc188 CR3: 00000002257ee000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process ksoftirqd/0 (pid: 3, threadinfo ffff88022e880000, task ffff88022e868ba0)
Stack:
 ffffffff811a6ef9 ffffffff8111e5ef ffffffff812513cb 0000000000000000
 0000000000000286 ffff88022dae9720 ffff88022dae9720 ffff880224112840
 ffffffff81251924 ffff88022fc11500 0000000000000086 ffff88022fc11500
Call Trace:
 [<ffffffff811a6ef9>] ? blk_start_request+0x9/0x40
 [<ffffffff8111e5ef>] ? proc_flush_task+0xff/0x1f0
 [<ffffffff812513cb>] ? scsi_kill_request+0x2b/0x1a0
 [<ffffffff81251924>] ? scsi_request_fn+0x3e4/0x4a0
 [<ffffffff811a67e8>] ? blk_run_queue+0x28/0x50
 [<ffffffff81250df1>] ? scsi_run_queue+0xd1/0x270
 [<ffffffff810c500a>] ? kmem_cache_free+0x9a/0xb0
 [<ffffffff81251bfb>] ? scsi_next_command+0x3b/0x60
 [<ffffffff812527e7>] ? scsi_io_completion+0x2c7/0x5b0
 [<ffffffff811ad3cd>] ? blk_done_softirq+0x6d/0x80
 [<ffffffff8103f491>] ? __do_softirq+0x91/0x120
 [<ffffffff8103f5c5>] ? run_ksoftirqd+0xa5/0x150
 [<ffffffff8103f520>] ? __do_softirq+0x120/0x120
 [<ffffffff8103f520>] ? __do_softirq+0x120/0x120
 [<ffffffff81054136>] ? kthread+0x96/0xa0
 [<ffffffff8134be94>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff810540a0>] ? kthread_worker_fn+0x120/0x120
 [<ffffffff8134be90>] ? gs_change+0xb/0xb
sd 6:0:0:0: [sdd] Assuming drive cache: write through
Code: 3f 48 89 7f 08 f6 c4 40 74 1f 83 7f 44 01 74 04 a8 40 74 15 83 e0 11 ff c8 0f 95 c0 83 e0 01 48 05 d8 00 00 00 ff 44 81 0c 
 sdd: sdd1
f3 c3 <0f> 0b eb fe 0f 0b eb fe 66 2e 0f 1f 84 00 00 00 00 00 8b 4f 40 
RIP  [<ffffffff811a651e>] blk_dequeue_request+0x4e/0x60
 RSP <ffff88022e881c98>
---[ end trace 1be76b42f53e92c7 ]---
Comment 4 rpansky 2011-07-19 12:23:14 UTC
I still cannot reproduce the panic with 2.6.39.3.
Comment 5 alust 2011-08-28 16:28:33 UTC
Yes, I also cannot reproduce this bug in 3.0.0

Note You need to log in before you can comment on or make changes to this bug.