Most recent kernel where this bug did *NOT* occur: occured always Distribution: Gentoo Hardware Environment: Via Epis 5000 board Software Environment: nothing special Problem Description: I reproducably get a kernel panic when plugging in a usb isdn modem, Billion tiny USB ISDN TA 128). It is said to work with the hfc_usb isdn drivers in the kernel, so I compiled them into it. When plugging the usb cable, I get this: Oops: 0000 [#1] PREEMPT Modules linked in: CPU: 0 EIP: 0060:[<00000000>] Not tainted VLI EFLAGS: 00010046 (2.6.20-rc5 #2) EIP is at 0x0 eax: d5ab6000 ebx: d5ab6a03 ecx: 00000000 edx: 00000112 esi: d5ab6000 edi: 00000005 ebp: c064beec esp: c064bedc ds: 007b es: 007b ss: 0068 Process khubd (pid: 131, ti=c064b000 task=d7dae050 task.ti=d7d97000) Stack: c03a6808 00ab6ac0 d5ab6ac0 d4e2db08 c064bf10 c03a74cc c04b5a49 d7ac1628 00000040 00000002 d4e2db08 d7a9b308 00000000 c064bf28 c034a82f 00000000 d7a9b41c d7aca5a0 d679a3e8 c064bf5c c035e462 c035e600 00000001 c064bf50 Call Trace: [<c010352a>] show_trace_log_lvl+0x1a/0x30 [<c01035fa>] show_stack_log_lvl+0x9a/0xd0 [<c0103846>] show_registers+0x1c6/0x340 [<c0103bc1>] die+0x171/0x230 [<c010f906>] do_page_fault+0x366/0x5b0 [<c04b5f74>] error_code+0x74/0x80 [<c03a74cc>] rx_complete+0x8c/0x150 [<c034a82f>] usb_hcd_giveback_urb+0x3f/0xb0 [<c035e462>] uhci_giveback_urb+0x82/0x1c0 [<c035e63c>] uhci_scan_qh+0x9c/0x240 [<c035e958>] uhci_scan_schedule+0x98/0x130 [<c035f757>] uhci_irq+0xd7/0x180 [<c034a8c5>] usb_hcd_irq+0x25/0x60 [<c0146330>] handle_IRQ_event+0x30/0x70 [<c01473dd>] handle_level_irq+0x7d/0x110 [<c0104fe7>] do_IRQ+0x87/0xd0 [<c010324e>] common_interrupt+0x2e/0x40 [<c0165420>] cache_alloc_debugcheck_after+0xc0/0x180 [<c0165afa>] __kmalloc_track_caller+0xaa/0x100 [<c03f133d>] __alloc_skb+0x4d/0x110 [<c028fff1>] kobject_uevent_env+0x2e1/0x4e0 [<c02901fa>] kobject_uevent+0xa/0x10 [<c03020cf>] device_del+0x8f/0x220 [<c030226b>] device_unregister+0xb/0x20 [<c0350a18>] usb_remove_ep_files+0x58/0x70 [<c035020a>] usb_create_sysfs_intf_files+0xca/0x100 [<c034ce43>] usb_set_configuration+0x313/0x470 [<c0353f16>] generic_probe+0x26/0x80 [<c034d32c>] usb_probe_device+0x4c/0x60 [<c03041f4>] really_probe+0x94/0x110 [<c0304329>] driver_probe_device+0x99/0xd0 [<c0304368>] __device_attach+0x8/0x10 [<c0303619>] bus_for_each_drv+0x49/0x70 [<c03043d6>] device_attach+0x66/0x90 [<c0303861>] bus_attach_device+0x21/0x50 [<c0301d49>] device_add+0x199/0x430 [<c0347233>] __usb_new_device+0x63/0xf0 [<c0347327>] usb_new_device+0x67/0xa0 [<c03482b3>] hub_port_connect_change+0x1c3/0x3d0 [<c034868d>] hub_events+0x1cd/0x3e0 [<c03488b5>] hub_thread+0x15/0x100 [<c012c494>] kthread+0x94/0xc0 [<c0103387>] kernel_thread_helper+0x7/0x10 ======================= Code: Bad EIP value. EIP: [<00000000>] 0x0 SS:ESP 0068:c064bedc <0>Kernel panic - not syncing: Fatal exception in interrupt Steps to reproduce: 1. plug in usb isdn model # ./scripts/ver_linux If some fields are empty or look unusual you may have an old version. Compare to the current minimal requirements in Documentation/Changes. Linux dojo 2.6.20-rc5 #2 PREEMPT Tue Jan 23 11:39:40 CET 2007 i686 VIA Samuel 2 CentaurHauls GNU/Linux Gnu C 3.4.6 Gnu make 3.81 binutils 2.16.1 util-linux 2.12r mount 2.12r module-init-tools 3.2.2 e2fsprogs 1.39 reiserfsprogs 3.6.19 Linux C Library > libc.2.4 Dynamic linker (ldd) 2.4 Procps 3.2.6 Net-tools 1.60 Kbd 1.12 Sh-utils 6.4 udev 103 Modules Loaded
Reply-To: akpm@linux-foundation.org whee.. Begin forwarded message: Date: Wed, 7 Feb 2007 12:15:21 -0800 From: bugme-daemon@bugzilla.kernel.org To: bugme-new@lists.osdl.org Subject: [Bugme-new] [Bug 7960] New: kernel panic when inserting usb isdn modem http://bugzilla.kernel.org/show_bug.cgi?id=7960 Summary: kernel panic when inserting usb isdn modem Kernel Version: 2.6.20-rc5 Status: NEW Severity: normal Owner: greg@kroah.com Submitter: cweiske@cweiske.de Most recent kernel where this bug did *NOT* occur: occured always Distribution: Gentoo Hardware Environment: Via Epis 5000 board Software Environment: nothing special Problem Description: I reproducably get a kernel panic when plugging in a usb isdn modem, Billion tiny USB ISDN TA 128). It is said to work with the hfc_usb isdn drivers in the kernel, so I compiled them into it. When plugging the usb cable, I get this: Oops: 0000 [#1] PREEMPT Modules linked in: CPU: 0 EIP: 0060:[<00000000>] Not tainted VLI EFLAGS: 00010046 (2.6.20-rc5 #2) EIP is at 0x0 eax: d5ab6000 ebx: d5ab6a03 ecx: 00000000 edx: 00000112 esi: d5ab6000 edi: 00000005 ebp: c064beec esp: c064bedc ds: 007b es: 007b ss: 0068 Process khubd (pid: 131, ti=c064b000 task=d7dae050 task.ti=d7d97000) Stack: c03a6808 00ab6ac0 d5ab6ac0 d4e2db08 c064bf10 c03a74cc c04b5a49 d7ac1628 00000040 00000002 d4e2db08 d7a9b308 00000000 c064bf28 c034a82f 00000000 d7a9b41c d7aca5a0 d679a3e8 c064bf5c c035e462 c035e600 00000001 c064bf50 Call Trace: [<c010352a>] show_trace_log_lvl+0x1a/0x30 [<c01035fa>] show_stack_log_lvl+0x9a/0xd0 [<c0103846>] show_registers+0x1c6/0x340 [<c0103bc1>] die+0x171/0x230 [<c010f906>] do_page_fault+0x366/0x5b0 [<c04b5f74>] error_code+0x74/0x80 [<c03a74cc>] rx_complete+0x8c/0x150 [<c034a82f>] usb_hcd_giveback_urb+0x3f/0xb0 [<c035e462>] uhci_giveback_urb+0x82/0x1c0 [<c035e63c>] uhci_scan_qh+0x9c/0x240 [<c035e958>] uhci_scan_schedule+0x98/0x130 [<c035f757>] uhci_irq+0xd7/0x180 [<c034a8c5>] usb_hcd_irq+0x25/0x60 [<c0146330>] handle_IRQ_event+0x30/0x70 [<c01473dd>] handle_level_irq+0x7d/0x110 [<c0104fe7>] do_IRQ+0x87/0xd0 [<c010324e>] common_interrupt+0x2e/0x40 [<c0165420>] cache_alloc_debugcheck_after+0xc0/0x180 [<c0165afa>] __kmalloc_track_caller+0xaa/0x100 [<c03f133d>] __alloc_skb+0x4d/0x110 [<c028fff1>] kobject_uevent_env+0x2e1/0x4e0 [<c02901fa>] kobject_uevent+0xa/0x10 [<c03020cf>] device_del+0x8f/0x220 [<c030226b>] device_unregister+0xb/0x20 [<c0350a18>] usb_remove_ep_files+0x58/0x70 [<c035020a>] usb_create_sysfs_intf_files+0xca/0x100 [<c034ce43>] usb_set_configuration+0x313/0x470 [<c0353f16>] generic_probe+0x26/0x80 [<c034d32c>] usb_probe_device+0x4c/0x60 [<c03041f4>] really_probe+0x94/0x110 [<c0304329>] driver_probe_device+0x99/0xd0 [<c0304368>] __device_attach+0x8/0x10 [<c0303619>] bus_for_each_drv+0x49/0x70 [<c03043d6>] device_attach+0x66/0x90 [<c0303861>] bus_attach_device+0x21/0x50 [<c0301d49>] device_add+0x199/0x430 [<c0347233>] __usb_new_device+0x63/0xf0 [<c0347327>] usb_new_device+0x67/0xa0 [<c03482b3>] hub_port_connect_change+0x1c3/0x3d0 [<c034868d>] hub_events+0x1cd/0x3e0 [<c03488b5>] hub_thread+0x15/0x100 [<c012c494>] kthread+0x94/0xc0 [<c0103387>] kernel_thread_helper+0x7/0x10 ======================= Code: Bad EIP value. EIP: [<00000000>] 0x0 SS:ESP 0068:c064bedc <0>Kernel panic - not syncing: Fatal exception in interrupt Steps to reproduce: 1. plug in usb isdn model # ./scripts/ver_linux If some fields are empty or look unusual you may have an old version. Compare to the current minimal requirements in Documentation/Changes. Linux dojo 2.6.20-rc5 #2 PREEMPT Tue Jan 23 11:39:40 CET 2007 i686 VIA Samuel 2 CentaurHauls GNU/Linux Gnu C 3.4.6 Gnu make 3.81 binutils 2.16.1 util-linux 2.12r mount 2.12r module-init-tools 3.2.2 e2fsprogs 1.39 reiserfsprogs 3.6.19 Linux C Library > libc.2.4 Dynamic linker (ldd) 2.4 Procps 3.2.6 Net-tools 1.60 Kbd 1.12 Sh-utils 6.4 udev 103 Modules Loaded ------- You are receiving this mail because: ------- You are on the CC list for the bug, or are watching someone who is.
It looks like you might have CONFIG_USB_MULTITHREADED_PROBE turned on. If you do, try turning it off. It is known to be buggy and to cause problems like this. Or just use 2.6.20, which no longer has that config option.
Another hint: see the FAQ for how to submit USB bugs. It is for example unclear what driver is involved here. I'll suspect HiSax...
@comment #2: multithreaded is not activated @comment #3: I wrote that I used hfc_usb drivers in my description. I will try 2.6.20 now.
Using 2.6.20 the trace is a bit different, but not much: Oops: 0000 [#1] PREEMPT Modules linked in: CPU: 0 EIP: 0060:[<00000000>] Not tainted VLI EFLAGS: 00010046 (2.6.20 #1) EIP is at 0x0 eax: ca8ac000 ebx: ca8aca03 ecx: 00000000 edx: 00000112 esi: ca8ac000 edi: 00000005 ebp: c064ceec esp: c064cedc ds: 007b es: 007b ss: 0068 Process khubd (pid: 132, ti=c064c000 task=d7d8b550 task.ti=d7d92000) Stack: c03a6348 008acac0 ca8acac0 d2664640 c064cf10 c03a700c c04b5789 d7ac2734 00000040 00000002 d2664640 d7a9c308 00000000 c064cf28 c034a33f 00000000 d7a9c41c d7aca5a0 caa493e8 c064cf5c c035df72 c035e100 00000001 c064cf50 Call Trace: [<c01034fa>] show_trace_log_lvl+0x1a/0x30 [<c01035ca>] show_stack_log_lvl+0x9a/0xd0 [<c0103816>] show_registers+0x1c6/0x340 [<c0103b91>] die+0x171/0x230 [<c010f706>] do_page_fault+0x366/0x5b0 [<c04b5cb4>] error_code+0x74/0x80 [<c03a700c>] rx_complete+0x8c/0x150 [<c034a33f>] usb_hcd_giveback_urb+0x3f/0xb0 [<c035df72>] uhci_giveback_urb+0x82/0x1c0 [<c035e14c>] uhci_scan_qh+0x9c/0x240 [<c035e468>] uhci_scan_schedule+0x98/0x130 [<c035f267>] uhci_irq+0xd7/0x180 [<c034a3d5>] usb_hcd_irq+0x25/0x60 [<c01460b0>] handle_IRQ_event+0x30/0x70 [<c014716d>] handle_level_irq+0x7d/0x110 [<c0104fb7>] do_IRQ+0x87/0xd0 [<c010321e>] common_interrupt+0x2e/0x40 [<c0114292>] complete+0x42/0x50 [<c04b260a>] klist_release+0x2a/0x40 [<c028fea4>] kref_put+0x34/0xa0 [<c04b2630>] klist_dec_and_del+0x10/0x20 [<c04b265b>] klist_del+0x1b/0x40 [<c0301c70>] device_del+0x20/0x220 [<c0301e7b>] device_unregister+0xb/0x20 [<c0350528>] usb_remove_ep_files+0x58/0x70 [<c034fd1a>] usb_create_sysfs_intf_files+0xca/0x100 [<c034c953>] usb_set_configuration+0x313/0x470 [<c0353a26>] generic_probe+0x26/0x80 [<c034ce3c>] usb_probe_device+0x4c/0x60 [<c0303e04>] really_probe+0x94/0x110 [<c0303f39>] driver_probe_device+0x99/0xd0 [<c0303f78>] __device_attach+0x8/0x10 [<c0303229>] bus_for_each_drv+0x49/0x70 [<c0303fe6>] device_attach+0x66/0x90 [<c0303471>] bus_attach_device+0x21/0x50 [<c0301959>] device_add+0x199/0x430 [<c0346d43>] __usb_new_device+0x63/0xf0 [<c0346e37>] usb_new_device+0x67/0xa0 [<c0347dc3>] hub_port_connect_change+0x1c3/0x3d0 [<c034819d>] hub_events+0x1cd/0x3e0 [<c03483c5>] hub_thread+0x15/0x100 [<c012c214>] kthread+0x94/0xc0 [<c0103357>] kernel_thread_helper+0x7/0x10 ======================= Code: Bad EIP value. EIP: [<00000000>] 0x0 SS:ESP 0068:c064cedc <0>Kernel panic - not syncing: Fatal exception in interrupt I can post the kernel configg if it's needed.
Alan, this looks like your area :)
It seems I forgot half of the mail when copying the text from the one I sent to the list to the bug tracker: # lsusb -v Bus 002 Device 001: ID 0000:0000 Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 1.10 bDeviceClass 9 Hub bDeviceSubClass 0 Unused bDeviceProtocol 0 Full speed (or root) hub bMaxPacketSize0 64 idVendor 0x0000 idProduct 0x0000 bcdDevice 2.06 iManufacturer 3 Linux 2.6.20-rc5 uhci_hcd iProduct 2 UHCI Host Controller iSerial 1 0000:00:11.3 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 25 bNumInterfaces 1 bConfigurationValue 1 iConfiguration 0 bmAttributes 0xe0 Self Powered Remote Wakeup MaxPower 0mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 9 Hub bInterfaceSubClass 0 Unused bInterfaceProtocol 0 Full speed (or root) hub iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0002 1x 2 bytes bInterval 255 Hub Descriptor: bLength 9 bDescriptorType 41 nNbrPorts 2 wHubCharacteristic 0x000a No power switching (usb 1.0) Per-port overcurrent protection bPwrOn2PwrGood 1 * 2 milli seconds bHubContrCurrent 0 milli Ampere DeviceRemovable 0x00 PortPwrCtrlMask 0x00 Hub Port Status: Port 1: 0000.0308 lowspeed power oc Port 2: 0000.0308 lowspeed power oc Bus 001 Device 001: ID 0000:0000 Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 1.10 bDeviceClass 9 Hub bDeviceSubClass 0 Unused bDeviceProtocol 0 Full speed (or root) hub bMaxPacketSize0 64 idVendor 0x0000 idProduct 0x0000 bcdDevice 2.06 iManufacturer 3 Linux 2.6.20-rc5 uhci_hcd iProduct 2 UHCI Host Controller iSerial 1 0000:00:11.2 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 25 bNumInterfaces 1 bConfigurationValue 1 iConfiguration 0 bmAttributes 0xe0 Self Powered Remote Wakeup MaxPower 0mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 9 Hub bInterfaceSubClass 0 Unused bInterfaceProtocol 0 Full speed (or root) hub iInterface 0 Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0002 1x 2 bytes bInterval 255 Hub Descriptor: bLength 9 bDescriptorType 41 nNbrPorts 2 wHubCharacteristic 0x000a No power switching (usb 1.0) Per-port overcurrent protection bPwrOn2PwrGood 1 * 2 milli seconds bHubContrCurrent 0 milli Ampere DeviceRemovable 0x00 PortPwrCtrlMask 0x00 Hub Port Status: Port 1: 0000.0300 lowspeed power Port 2: 0000.0300 lowspeed power
It looks like a bug in the hfc_usb driver (which isn't really my area, Greg!). Please do "objdump -d drivers/isdn/hisax/hfc_usb.o", extract from the output the portion corresponding to rx_complete, and attach it to this bug report. That will help identify exactly where the oops is occurring.
Created attachment 10352 [details] objdump of drivers/isdn/hisax/hfc_usb.o, rx_complete part
Created attachment 10357 [details] Make the HiSax driver use usb_kill_urb Try this patch. With luck it will fix the panic. On the other hand, from your log it looks like the driver is encountering some sort of error. So even if it doesn't oops, it still might not work.
Unfortunately I didn't have any luck; it still crashes. I will post a new trace this evening.
Created attachment 10375 [details] trace after patch #1
Created attachment 10421 [details] Extra HiSax debugging I still can't tell what's wrong. Try turning on CONFIG_HISAX_DEBUG, and apply the attached patch on top of the previous one. When you load the new driver module, set the module parameter "debug=2". Then attach the dmesg log showing what happens from the time you plug in (or turn on) the modem.
Created attachment 10707 [details] hisax debug output before panic when booting
I tried to load hisax using modprobe and the parameter debug=2, but it told me that debug is not a supported parameter. Further, how can I "live" view dmesg output before a panic? I hope the messages before the panic when booting do help.
I don't understand why it said debug was not a supported parameter. However you can avoid specifying the modprobe parameter by changing the source file. The line near the top that says: static u_int debug; should be changed to static u_int debug = 3; (Using 3 instead of 2 will provide even more useful information). Be sure to keep CONFIG_HISAX_DEBUG turned on. The most reliable way to get dmesg info before a panic is to use a serial console. But the messages you attached are fine, it's just that without debug=3 they don't contain enough to be useful.
Created attachment 10723 [details] hisax debug output before panic when booting debug=3 Seems as if debug=3 does not give us more information.
Created attachment 10724 [details] attaching the isdn modem to another computer The device does work when being attached to a different computer running 2.6.20 on an AMD processor. I begin to believe that the error is not the isdn module but the usb system driver for the epia board.
Created attachment 10731 [details] Extra HiSax debugging I don't think the USB subsystem is at fault. Attached is an updated patch that contains all the changes we've talked about so far, plus a couple of others. You'll have to undo all the earlier changes before applying it.
Created attachment 10806 [details] Extra Hisax debugging fixed Your all-in-one patch had a bug; this fixes it
Created attachment 10807 [details] hisax debug output before panic when booting, new patch applied Here is the serial console output when booting.
Created attachment 10808 [details] Extra HiSax debugging We're slowly making progress. Here's an updated version with the error fixed and a little more debugging added. I'm pretty sure the problem occurs in the state_handler() routine; this should tell us for certain. The reason you're not getting much output on your serial console is because you have the printk-level set too low, so debugging messages don't get printed. Before plugging in the device, do "echo 9 >/proc/sysrq-trigger".
Created attachment 10817 [details] hisax debug output before panic with log level 9 failing in statements. You were right, Alan.
Created attachment 10818 [details] Extra HiSax debugging I think the problem was that fifo->hfc was never initialized. This patch should fix that, along with adding a little more debugging info.
Created attachment 10819 [details] hisax debug output Didn't help
Created attachment 10834 [details] Extra HiSax debugging I was wrong about failing to initialize that field. Here's some more debugging; let's see what it shows. By the way, you should set your serial communications program to wrap long lines. Right now it truncates lines that are longer than 80 characters.
Created attachment 10847 [details] debug output I got no kernel panic this time. Can I try to get it working now, or was the patch just to check some assumptions?
You didn't get a crash, but the driver still isn't working. I added error checking that had been left out. The test near the start of hisax_register() in drivers/isdn/hisax/config.c failed, which means the kernel thinks you already have the maximum number of HiSax cards attached. Do you have any other HiSax devices attached? What does "grep HISAX .config" show for your configuration options?
Created attachment 10852 [details] grep HISAX .config I don't have any other hisax devices attached. I will try to set the max card number higher.
Created attachment 10853 [details] attaching with CARDS set to 4 When setting cards to 4, I see the attached output. The lines "HFC-S USB: INT-D-RX lst_urblen" are repeated in frenzy - until I unplug the usb isdn modem. Seems there is a loop somewhere.
Created attachment 10872 [details] HiSax fix Don't worry about all that output. It's supposed to be there for debugging. Here's the patch with all the debugging stuff removed, leaving only the code fixes. You can turn off CONFIG_HISAX_DEBUG and try it. In the end, I think a big part of the problem was that you had the max card value set to 1. For some reason the driver thinks that the first card will be one of the ones selected in the config, and the USB card wasn't the one it chose. So it thought you had some other type of card plus the USB card -- which was more than the maximum.
It looks good - the correct LED is activated and I don't get a panic :) I would say the bug as fixed. The dmesg output after attaching is now: usb 2-1: new full speed USB device using uhci_hcd and address 2 usb 2-1: configuration #1 chosen from 1 choice HFC-S USB: probing interface(0) actalt(0) minor(0) hfc_usb: probe of 2-1:1.0 failed with error -5 usb 2-1: device_add(2-1:1.0) --> -5 HFC-S USB: probing interface(1) actalt(0) minor(0) HFC-S USB: detected "Billion tiny USB ISDN TA 128" HiSax: Card 1 Protocol EDSS1 Id=hfc_usb0 (0) HiSax: DSS1 Rev. 2.32.2.3 HiSax: 2 channels added HiSax: MAX_WAITING_CALLS added HFC-S USB: starting intr IN fifo:5 HFC-S USB: starting intr IN fifo:1 HFC-S USB: starting intr IN fifo:3 HFC-S USB: starting ISO-chain for Fifo 4 HFC-S USB: starting ISO-chain for Fifo 0 HFC-S USB: starting ISO-chain for Fifo 2
Okay, I'll send the patch in to the people responsible for maintaining this driver.
Thanks for helping.
The patch has been accepted for 2.6.22. Closing the bug report.