Reply-To: akpm@linux-foundation.org

> On Wed, 14 Mar 2007 17:49:58 -0700 bugme-daemon@bugzilla.kernel.org wrote:
> http://bugzilla.kernel.org/show_bug.cgi?id=8198

Looking at the full dmesg I'm thinking that eth1394 might be the
culprit here.

Reply-To: stefanr@s5r6.in-berlin.de

Andrew Morton wrote:
>> On Wed, 14 Mar 2007 17:49:58 -0700 bugme-daemon@bugzilla.kernel.org wrote:
>> http://bugzilla.kernel.org/show_bug.cgi?id=8198
> 
> Looking at the full dmesg I'm thinking that eth1394 might be the
> culprit here.

I have doubts.

BUG: unable to handle kernel paging request at virtual address 6b6b6e6b
EIP is at module_put+0x20/0x52

How can this happen?


Since Linux 2.6.19.y seems to work for the reporter, we had these two
commits between .19 and .20 touching eth1394:

"slab: remove kmem_cache_t"
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blobdiff;f=drivers/ieee1394/eth1394.c;h=27d6c642415dd16af7d94bc81150c2137c1652ac;hp=31e5cc49d61a01899cea3e6d5220809f2b889be7;hb=e18b890bb0881bbab6f4f1a6cd20d9c60d66b003;hpb=441e143e95f5aa1e04026cb0aa71c801ba53982f
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blobdiff;f=drivers/ieee1394/eth1394.c;h=27d6c642415dd16af7d94bc81150c2137c1652ac;hp=31e5cc49d61a01899cea3e6d5220809f2b889be7

"ieee1394: Consolidate driver registering"
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=ed30c26ee809b060f437fd9f1b3453e1d0375e32

The latter actually added the link
/sys/bus/ieee1394/drivers/eth1394/module -> ../../../../module/eth139
i.e. to /sys/module/eth139.

OpenSUSE have a patch in their 10.2 kernel (2.6.18.2-34) which adds the
symlink too. I haven't heard of a similar bug report yet. Having the
symlink is obviously correct. Why a process walking sysfs triggers an
oops (suspectedly) after that change is anyones guess. I tend to think
it is another sysfs race.

Reply-To: stefanr@s5r6.in-berlin.de

I wrote:
> module/eth139

module/eth1394 of course

Also, it is not unlikely that Ubuntu added that symlink earlier than
mainline.

Fedora, which is what the reporter uses, has another bug report of a
hard-to-trigger sysfs bug; this one caused by haldeamon:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=188140

The obvious thing to try would be not loading eth1394 in 2.6.20.2, and then
seeing if I can still trigger the oops. I can do that...

It took slightly more invocations of "winecfg" than I was expecting, but
"Voila!". The eth1394 module has not been loaded, but still sysfs explodes under
udevd.

Linux version 2.6.20.2 (chris@volcano.underworld) (gcc version 4.1.1 20070105
(Red Hat 4.1.1-51)) #1 SMP PREEMPT Fri Mar 9 22:37:26 GMT 2007
BIOS-provided physical RAM map:
sanitize start
sanitize end
copy_e820_map() start: 0000000000000000 size: 00000000000a0000 end:
00000000000a0000 type: 1
copy_e820_map() type is E820_RAM
copy_e820_map() start: 00000000000f0000 size: 0000000000010000 end:
0000000000100000 type: 2
copy_e820_map() start: 0000000000100000 size: 000000007fe75000 end:
000000007ff75000 type: 1
copy_e820_map() type is E820_RAM
copy_e820_map() start: 000000007ff75000 size: 0000000000002000 end:
000000007ff77000 type: 4
copy_e820_map() start: 000000007ff77000 size: 0000000000021000 end:
000000007ff98000 type: 3
copy_e820_map() start: 000000007ff98000 size: 0000000000068000 end:
0000000080000000 type: 2
copy_e820_map() start: 00000000fec00000 size: 0000000000090000 end:
00000000fec90000 type: 2
copy_e820_map() start: 00000000fee00000 size: 0000000000010000 end:
00000000fee10000 type: 2
copy_e820_map() start: 00000000ffb00000 size: 0000000000500000 end:
0000000100000000 type: 2
 BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000007ff75000 (usable)
 BIOS-e820: 000000007ff75000 - 000000007ff77000 (ACPI NVS)
 BIOS-e820: 000000007ff77000 - 000000007ff98000 (ACPI data)
 BIOS-e820: 000000007ff98000 - 0000000080000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec90000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
 BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
1151MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000fe710
Entering add_active_range(0, 0, 524149) 0 entries of 256 used
Zone PFN ranges:
  DMA             0 ->     4096
  Normal       4096 ->   229376
  HighMem    229376 ->   524149
early_node_map[1] active PFN ranges
    0:        0 ->   524149
On node 0 totalpages: 524149
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 4064 pages, LIFO batch:0
  Normal zone: 1760 pages used for memmap
  Normal zone: 223520 pages, LIFO batch:31
  HighMem zone: 2302 pages used for memmap
  HighMem zone: 292471 pages, LIFO batch:31
DMI 2.3 present.
ACPI: RSDP (v000 DELL                                  ) @ 0x000febc0
ACPI: RSDT (v001 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd4f1
ACPI: FADT (v001 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd529
ACPI: SSDT (v001   DELL    st_ex 0x00001000 MSFT 0x0100000d) @ 0xfffefafa
ACPI: MADT (v001 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd59d
ACPI: BOOT (v001 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd621
ACPI: ASF! (v016 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd649
ACPI: DSDT (v001   DELL    dt_ex 0x00001000 MSFT 0x0100000d) @ 0x00000000
ACPI: PM-Timer IO Port: 0x808
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:2 APIC version 20
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x06] enabled)
Processor #6 15:2 APIC version 20
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled)
Processor #1 15:2 APIC version 20
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x07] enabled)
Processor #7 15:2 APIC version 20
ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x09] address[0xfec80000] gsi_base[24])
IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI 24-47
ACPI: IOAPIC (id[0x0a] address[0xfec80800] gsi_base[48])
IOAPIC[2]: apic_id 10, version 32, address 0xfec80800, GSI 48-71
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Enabling APIC mode:  Flat.  Using 3 I/O APICs
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at 88000000 (gap: 80000000:7ec00000)
Detected 2658.217 MHz processor.
Built 1 zonelists.  Total pages: 520055
Kernel command line: ro root=LABEL=/ nmi_watchdog=1 elevator=cfq
console=ttyS0,115200n8 console=tty0 acpi_pm_good
mapped APIC to ffffd000 (fee00000)
mapped IOAPIC to ffffc000 (fec00000)
mapped IOAPIC to ffffb000 (fec80000)
mapped IOAPIC to ffffa000 (fec80800)
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c0345000 soft=c033d000
PID hash table entries: 4096 (order: 12, 16384 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 2074480k/2096596k available (1484k kernel code, 20840k reserved, 579k
data, 192k init, 1179092k highmem)
virtual kernel memory layout:
    fixmap  : 0xffedf000 - 0xfffff000   (1152 kB)
    pkmap   : 0xff800000 - 0xffc00000   (4096 kB)
    vmalloc : 0xf8800000 - 0xff7fe000   ( 111 MB)
    lowmem  : 0xc0000000 - 0xf8000000   ( 896 MB)
      .init : 0xc0308000 - 0xc0338000   ( 192 kB)
      .data : 0xc027300b - 0xc0303c54   ( 579 kB)
      .text : 0xc0100000 - 0xc027300b   (1484 kB)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 5320.36 BogoMIPS
(lpj=10640722)Mount-cache hash table entries: 512
CPU: After generic identify, caps: bfebfbff 00000000 00000000 00000000 00004400
00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 0
CPU: After all inits, caps: bfebfbff 00000000 00000000 00003080 00004400
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU0: Intel P4/Xeon Extended MCE MSRs (12) available
CPU0: Thermal monitoring enabled
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 10k freed
ACPI: Core revision 20060707
CPU0: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Booting processor 1/1 eip 2000
CPU 1 irqstacks, hard=c0346000 soft=c033e000
Initializing CPU#1
Calibrating delay using timer specific routine.. 5316.58 BogoMIPS
(lpj=10633168)CPU: After generic identify, caps: bfebfbff 00000000 00000000
00000000 00004400 00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 0
CPU: After all inits, caps: bfebfbff 00000000 00000000 00003080 00004400
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel P4/Xeon Extended MCE MSRs (12) available
CPU1: Thermal monitoring enabled
CPU1: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Booting processor 2/6 eip 2000
CPU 2 irqstacks, hard=c0347000 soft=c033f000
Initializing CPU#2
Calibrating delay using timer specific routine.. 5316.63 BogoMIPS
(lpj=10633276)CPU: After generic identify, caps: bfebfbff 00000000 00000000
00000000 00004400 00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 3
CPU: After all inits, caps: bfebfbff 00000000 00000000 00003080 00004400
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#2.
CPU2: Intel P4/Xeon Extended MCE MSRs (12) available
CPU2: Thermal monitoring enabled
CPU2: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Booting processor 3/7 eip 2000
CPU 3 irqstacks, hard=c0348000 soft=c0340000
Initializing CPU#3
Calibrating delay using timer specific routine.. 5316.61 BogoMIPS
(lpj=10633234)CPU: After generic identify, caps: bfebfbff 00000000 00000000
00000000 00004400 00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 3
CPU: After all inits, caps: bfebfbff 00000000 00000000 00003080 00004400
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#3.
CPU3: Intel P4/Xeon Extended MCE MSRs (12) available
CPU3: Thermal monitoring enabled
CPU3: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Total of 4 processors activated (21270.20 BogoMIPS).
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
checking TSC synchronization across 4 CPUs: passed.
Brought up 4 CPUs
migration_cost=104,637
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: PCI BIOS revision 2.10 entry at 0xfbdf2, last bus=5
PCI: Using configuration type 1
Setting up standard PCI resources
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI: Probing PCI hardware (bus 00)
ACPI: Assume root bridge [\_SB_.PCI0] bus is 0
PCI quirk: region 0800-087f claimed by ICH4 ACPI/GPIO/TCO
PCI quirk: region 0880-08bf claimed by ICH4 GPIO
Boot video device is 0000:01:00.0
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1.PCI2._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1.PCI3._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI4._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *9 10 11 12 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 *10 11 12 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 *11 12 15)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 13 devices
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
pnp: 00:0c: ioport range 0x800-0x85f could not be reserved
pnp: 00:0c: ioport range 0xc00-0xc7f has been reserved
pnp: 00:0c: ioport range 0x860-0x8ff could not be reserved
PCI: Bridge: 0000:00:01.0
  IO window: e000-efff
  MEM window: ff800000-ff9fffff
  PREFETCH window: e8000000-f7ffffff
PCI: Bridge: 0000:02:1d.0
  IO window: d000-dfff
  MEM window: ff600000-ff7fffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:02:1f.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:02.0
  IO window: d000-dfff
  MEM window: ff400000-ff7fffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
  IO window: c000-cfff
  MEM window: ff200000-ff3fffff
  PREFETCH window: 88000000-880fffff
PCI: Setting latency timer of device 0000:00:1e.0 to 64
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
TCP established hash table entries: 131072 (order: 8, 1572864 bytes)
TCP bind hash table entries: 65536 (order: 7, 786432 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
checking if image is initramfs... it is
Freeing initrd memory: 897k freed
Simple Boot Flag value 0x87 read from CMOS RAM was invalid
Simple Boot Flag at 0x7a set to 0x1
Machine check exception polling timer started.
highmem bounce pool size: 64 pages
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler cfq registered (default)
Real Time Clock Driver v1.12ac
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:0a: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH4: IDE controller at PCI slot 0000:00:1f.1
PCI: Enabling device 0000:00:1f.1 (0005 -> 0007)
ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 16
ICH4: chipset revision 1
ICH4: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:DMA
Probing IDE interface ide0...
hda: IC35L090AVV207-0, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hdc: HL-DT-STDVD-ROM GDR8162B, ATAPI CD/DVD-ROM drive
hdd: SONY CD-RW CRX216E, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 512KiB
hda: 156250000 sectors (80000 MB) w/1821KiB Cache, CHS=16383/255/63, UDMA(100)
hda: cache flushes supported
 hda: hda1 hda2 hda3 hda4 < hda5 hda6 hda7 >
PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
md: raid1 personality registered for level 1
device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised:
dm-devel@redhat.cominput: AT Translated Set 2 keyboard as /class/input/input0
TCP cubic registered
NET: Registered protocol family 1
Testing NMI watchdog ... OK.
Starting balanced_irq
Using IPI Shortcut mode
Freeing unused kernel memory: 192k freed
Time: tsc clocksource has been installed.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Linux agpgart interface v0.101 (c) Dave Jones
input: PC Speaker as /class/input/input1
agpgart: Detected an Intel E7505 Chipset.
agpgart: AGP aperture is 128M @ 0xe0000000
hdc: ATAPI 48X DVD-ROM drive, 256kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
hdd: ATAPI 48X CD-ROM CD-R/RW drive, 2048kB Cache, UDMA(33)
SCSI subsystem initialized
ACPI: PCI Interrupt 0000:00:1f.3[B] -> GSI 17 (level, low) -> IRQ 17
EDAC MC: Ver: 2.0.1 Mar  9 2007
libata version 2.00 loaded.
Intel(R) PRO/1000 Network Driver - version 7.3.15-k2-NAPI
Copyright (c) 1999-2006 Intel Corporation.
ACPI: PCI Interrupt 0000:03:0e.0[A] -> GSI 24 (level, low) -> IRQ 18
input: ImPS/2 Generic Wheel Mouse as /class/input/input2
USB Universal Host Controller Interface driver v3.0
ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 19
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:1d.0: irq 19, io base 0x0000ff80
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
ieee1394: Initialized config rom entry `ip1394'
ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 20
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1d.1: irq 20, io base 0x0000ff60
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
e1000: 0000:03:0e.0: e1000_probe: (PCI-X:100MHz:64-bit) 00:0d:56:0d:e7:3f
ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 16
PCI: Setting latency timer of device 0000:00:1d.2 to 64
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1d.2: irq 16, io base 0x0000ff40
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
sata_sil 0000:05:0e.0: version 2.0
ACPI: PCI Interrupt 0000:05:0e.0[A] -> GSI 22 (level, low) -> IRQ 21
ata1: SATA max UDMA/100 cmd 0xF8848C80 ctl 0xF8848C8A bmdma 0xF8848C00 irq 21
ata2: SATA max UDMA/100 cmd 0xF8848CC0 ctl 0xF8848CCA bmdma 0xF8848C08 irq 21
scsi0 : sata_sil
usb 1-2: new full speed USB device using uhci_hcd and address 2
ata1: SATA link down (SStatus 0 SControl 310)
scsi1 : sata_sil
usb 1-2: configuration #1 chosen from 1 choice
Linux video capture interface: v2.00
pwc: Philips webcam module version 10.0.12 loaded.
pwc: Supports Philips PCA645/646, PCVC675/680/690, PCVC720[40]/730/740/750 &
PCVC830/840.
pwc: Also supports the Askey VC010, various Logitech Quickcams, Samsung MPC-C10
and MPC-C30,
pwc: the Creative WebCam 5 & Pro Ex, SOTEC Afina Eye and Visionite VCS-UC300 and
VCS-UM100.
ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata2.00: ATA-7, max UDMA/133, 488397168 sectors: LBA48 NCQ (depth 0/32)
ata2.00: ata2: dev 0 multi count 16
ata2.00: configured for UDMA/100
scsi 1:0:0:0: Direct-Access     ATA      HDT722525DLA380  V44O PQ: 0 ANSI: 5
ACPI: PCI Interrupt 0000:00:1d.7[D] -> GSI 23 (level, low) -> IRQ 22
PCI: Setting latency timer of device 0000:00:1d.7 to 64
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 4
ehci_hcd 0000:00:1d.7: debug port 1
PCI: cache line size of 128 is not supported by device 0000:00:1d.7
ehci_hcd 0000:00:1d.7: irq 22, io mem 0xffa20000
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 6 ports detected
usbcore: registered new interface driver snd-usb-audio
pwc: Logitech QuickCam Zoom (new model) USB webcam detected.
pwc: Registered as /dev/video0.
usbcore: registered new interface driver Philips webcam
scsi 1:0:0:0: Attached scsi generic sg0 type 0
SCSI device sda: 488397168 512-byte hdwr sectors (250059 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
usb 1-2: USB disconnect, address 2
SCSI device sda: 488397168 512-byte hdwr sectors (250059 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
 sda:<6>ACPI: PCI Interrupt 0000:05:0d.0[A] -> GSI 21 (level, low) -> IRQ 23
Installing spdif_bug patch: Audigy 2 ZS [SB0353]
 sda1 sda2 sda3
sd 1:0:0:0: Attached scsi disk sda
ACPI: PCI Interrupt 0000:05:0c.0[A] -> GSI 20 (level, low) -> IRQ 24
ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[24]  MMIO=[ff2ff800-ff2fffff]  Max
Packet=[2048]  IR/IT contexts=[4/8]
ACPI: PCI Interrupt 0000:05:0d.2[B] -> GSI 22 (level, low) -> IRQ 21
ohci1394: fw-host1: OHCI-1394 1.1 (PCI): IRQ=[21]  MMIO=[ff2ff000-ff2ff7ff]  Max
Packet=[2048]  IR/IT contexts=[4/8]
usb 1-2: new full speed USB device using uhci_hcd and address 3
Non-volatile memory driver v1.2
parport: PnPBIOS parport detected.
parport0: PC-style at 0x378 (0x778), irq 7, dma 1 [PCSPP,TRISTATE,COMPAT,ECP,DMA]
lp0: using parport0 (interrupt-driven).
usb 1-2: configuration #1 chosen from 1 choice
pwc: Logitech QuickCam Zoom (new model) USB webcam detected.
pwc: Registered as /dev/video0.
input: Power Button (FF) as /class/input/input3
ACPI: Power Button (FF) [PWRF]
input: Power Button (CM) as /class/input/input4
ACPI: Power Button (CM) [VBTN]
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
ieee1394: Host added: ID:BUS[0-00:1023]  GUID[87ffffffffffff00]
ieee1394: The root node is not cycle master capable; selecting a new root node
and resetting...
ieee1394: Node added: ID:BUS[1-00:1023]  GUID[0050c500001076df]
ieee1394: Host added: ID:BUS[1-01:1023]  GUID[00023c00a1037111]
ieee1394: raw1394: /dev/raw1394 device initialized
video1394: Installed video1394 module
EXT3 FS on hda2, internal journal
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda5, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-0, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda3, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 2040212k swap on /dev/hda7.  Priority:-1 extents:1 across:2040212k
IA-32 Microcode Update Driver: v1.14a <tigran@aivazian.fsnet.co.uk>
NET: Registered protocol family 17
e1000: eth0: e1000_watchdog: NIC Link is Up 100 Mbps Full Duplex
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
Mobile IPv6
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period
p4-clockmod: P4/Xeon(TM) CPU On-Demand Clock Modulation available
Time: acpi_pm clocksource has been installed.
usbcore: deregistering interface driver Philips webcam
pwc: Philips webcam module removed.
pwc: Philips webcam module version 10.0.12 loaded.
pwc: Supports Philips PCA645/646, PCVC675/680/690, PCVC720[40]/730/740/750 &
PCVC830/840.
pwc: Also supports the Askey VC010, various Logitech Quickcams, Samsung MPC-C10
and MPC-C30,
pwc: the Creative WebCam 5 & Pro Ex, SOTEC Afina Eye and Visionite VCS-UC300 and
VCS-UM100.
pwc: Logitech QuickCam Zoom (new model) USB webcam detected.
pwc: Registered as /dev/video0.
usbcore: registered new interface driver Philips webcam
[drm] Initialized drm 1.1.0 20060810
ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 16 (level, low) -> IRQ 19
[drm] Initialized radeon 1.25.0 20060524 on minor 0
agpgart: Found an AGP 3.0 compliant device at 0000:00:00.0.
agpgart: Putting AGP V3 device at 0000:00:00.0 into 8x mode
agpgart: Putting AGP V3 device at 0000:01:00.0 into 8x mode
[drm] Setting GART location based on new memory map
[drm] Loading R200 Microcode
[drm] writeback test succeeded in 1 usecs
BUG: unable to handle kernel paging request at virtual address 6b6b6b6b
 printing eip:
6b6b6b6b
*pde = 00000000
Oops: 0000 [#1]
PREEMPT SMP
Modules linked in: snd_rtctimer radeon drm pwc eeprom cpufreq_ondemand
p4_clockmod speedstep_lib nfsd exportfs ipv6 autofs4 nfs lockd sunrpc af_packet
firmware_class binfmt_misc video1394 raw1394 video thermal processor fan button
ac lp parport_pc parport nvram snd_emu10k1_synth snd_emux_synth snd_seq_virmidi
snd_seq_midi_emul sd_mod sg compat_ioctl32 videodev snd_usb_audio snd_usb_lib
v4l2_common v4l1_compat snd_emu10k1 snd_rawmidi snd_ac97_codec ac97_bus
snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss snd_mixer_oss
ohci1394 snd_pcm ieee1394 snd_seq_device sata_sil ehci_hcd uhci_hcd snd_timer
e1000 snd_page_alloc serio_raw libata e7xxx_edac snd_util_mem snd_hwdep psmouse
edac_mc i2c_i801 snd soundcore scsi_mod ide_cd intel_agp pcspkr i2c_core agpgart
cdrom usbcore ext3 jbd
CPU:    2
EIP:    0060:[<6b6b6b6b>]    Not tainted VLI
EFLAGS: 00010202   (2.6.20.2 #1)
EIP is at 0x6b6b6b6b
eax: ed602144   ebx: 6b6b6b6b   ecx: ed296000   edx: f6866f30
esi: ed6021cc   edi: c02eca2c   ebp: ed6021cc   esp: f7a88f48
ds: 007b   es: 007b   ss: 0068
Process udevd (pid: 8953, ti=f7a88000 task=f7fc1560 task.ti=f7a88000)
Stack: c01e7ebd f6d65edc ed2f0554 c0183954 00000080 bfd029a4 c014f197 f6866f30
       f6d65ef0 ed2becb4 bfd029a4 c01838dc 00000080 c0150a85 f7a88fa0 ed2becb4
       ed2becb4 fffffff7 00000003 f7a88000 c0150edb f7a88fa0 00000000 00000000
Call Trace:
 [<c01e7ebd>] dev_attr_show+0x18/0x1b
 [<c0183954>] sysfs_read_file+0x78/0x115
 [<c014f197>] do_sys_open+0xb9/0xc3
 [<c01838dc>] sysfs_read_file+0x0/0x115
 [<c0150a85>] vfs_read+0x88/0x134
 [<c0150edb>] sys_read+0x41/0x67
 [<c0102bde>] sysenter_past_esp+0x5f/0x85
 [<c0270033>] __sched_text_start+0x613/0x971
 =======================
Code:  Bad EIP value.
EIP: [<6b6b6b6b>] 0x6b6b6b6b SS:ESP 0068:f7a88f48

Here's an oops against 2.6.20.3 with the following patches applied:

gregkh-driver-sysfs-crash-debugging.patch
detect-atomic-counter-underflows.patch

Linux version 2.6.20.3 (chris@volcano.underworld) (gcc version 4.1.1 20070105
(Red Hat 4.1.1-51)) #1 SMP PREEMPT Sat Mar 17 13:21:18 GMT 2007
BIOS-provided physical RAM map:
sanitize start
sanitize end
copy_e820_map() start: 0000000000000000 size: 00000000000a0000 end:
00000000000a0000 type: 1
copy_e820_map() type is E820_RAM
copy_e820_map() start: 00000000000f0000 size: 0000000000010000 end:
0000000000100000 type: 2
copy_e820_map() start: 0000000000100000 size: 000000007fe75000 end:
000000007ff75000 type: 1
copy_e820_map() type is E820_RAM
copy_e820_map() start: 000000007ff75000 size: 0000000000002000 end:
000000007ff77000 type: 4
copy_e820_map() start: 000000007ff77000 size: 0000000000021000 end:
000000007ff98000 type: 3
copy_e820_map() start: 000000007ff98000 size: 0000000000068000 end:
0000000080000000 type: 2
copy_e820_map() start: 00000000fec00000 size: 0000000000090000 end:
00000000fec90000 type: 2
copy_e820_map() start: 00000000fee00000 size: 0000000000010000 end:
00000000fee10000 type: 2
copy_e820_map() start: 00000000ffb00000 size: 0000000000500000 end:
0000000100000000 type: 2
 BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000007ff75000 (usable)
 BIOS-e820: 000000007ff75000 - 000000007ff77000 (ACPI NVS)
 BIOS-e820: 000000007ff77000 - 000000007ff98000 (ACPI data)
 BIOS-e820: 000000007ff98000 - 0000000080000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec90000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
 BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
1151MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000fe710
Entering add_active_range(0, 0, 524149) 0 entries of 256 used
Zone PFN ranges:
  DMA             0 ->     4096
  Normal       4096 ->   229376
  HighMem    229376 ->   524149
early_node_map[1] active PFN ranges
    0:        0 ->   524149
On node 0 totalpages: 524149
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 4064 pages, LIFO batch:0
  Normal zone: 1760 pages used for memmap
  Normal zone: 223520 pages, LIFO batch:31
  HighMem zone: 2302 pages used for memmap
  HighMem zone: 292471 pages, LIFO batch:31
DMI 2.3 present.
ACPI: RSDP (v000 DELL                                  ) @ 0x000febc0
ACPI: RSDT (v001 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd4f1
ACPI: FADT (v001 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd529
ACPI: SSDT (v001   DELL    st_ex 0x00001000 MSFT 0x0100000d) @ 0xfffefafa
ACPI: MADT (v001 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd59d
ACPI: BOOT (v001 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd621
ACPI: ASF! (v016 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd649
ACPI: DSDT (v001   DELL    dt_ex 0x00001000 MSFT 0x0100000d) @ 0x00000000
ACPI: PM-Timer IO Port: 0x808
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:2 APIC version 20
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x06] enabled)
Processor #6 15:2 APIC version 20
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled)
Processor #1 15:2 APIC version 20
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x07] enabled)
Processor #7 15:2 APIC version 20
ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x09] address[0xfec80000] gsi_base[24])
IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI 24-47
ACPI: IOAPIC (id[0x0a] address[0xfec80800] gsi_base[48])
IOAPIC[2]: apic_id 10, version 32, address 0xfec80800, GSI 48-71
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Enabling APIC mode:  Flat.  Using 3 I/O APICs
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at 88000000 (gap: 80000000:7ec00000)
Detected 2658.145 MHz processor.
Built 1 zonelists.  Total pages: 520055
Kernel command line: ro root=LABEL=/ nmi_watchdog=1 elevator=cfq
console=ttyS0,115200n8 console=tty0 acpi_pm_good
mapped APIC to ffffd000 (fee00000)
mapped IOAPIC to ffffc000 (fec00000)
mapped IOAPIC to ffffb000 (fec80000)
mapped IOAPIC to ffffa000 (fec80800)
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c0348000 soft=c0340000
PID hash table entries: 4096 (order: 12, 16384 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 2074464k/2096596k available (1493k kernel code, 20856k reserved, 581k
data, 192k init, 1179092k highmem)
virtual kernel memory layout:
    fixmap  : 0xffedf000 - 0xfffff000   (1152 kB)
    pkmap   : 0xff800000 - 0xffc00000   (4096 kB)
    vmalloc : 0xf8800000 - 0xff7fe000   ( 111 MB)
    lowmem  : 0xc0000000 - 0xf8000000   ( 896 MB)
      .init : 0xc030b000 - 0xc033b000   ( 192 kB)
      .data : 0xc027559b - 0xc0306c54   ( 581 kB)
      .text : 0xc0100000 - 0xc027559b   (1493 kB)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 5320.49 BogoMIPS
(lpj=10640984)Mount-cache hash table entries: 512
CPU: After generic identify, caps: bfebfbff 00000000 00000000 00000000 00004400
00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 0
CPU: After all inits, caps: bfebfbff 00000000 00000000 00003080 00004400
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU0: Intel P4/Xeon Extended MCE MSRs (12) available
CPU0: Thermal monitoring enabled
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 10k freed
ACPI: Core revision 20060707
CPU0: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Booting processor 1/1 eip 2000
CPU 1 irqstacks, hard=c0349000 soft=c0341000
Initializing CPU#1
Calibrating delay using timer specific routine.. 5316.58 BogoMIPS
(lpj=10633172)CPU: After generic identify, caps: bfebfbff 00000000 00000000
00000000 00004400 00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 0
CPU: After all inits, caps: bfebfbff 00000000 00000000 00003080 00004400
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel P4/Xeon Extended MCE MSRs (12) available
CPU1: Thermal monitoring enabled
CPU1: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Booting processor 2/6 eip 2000
CPU 2 irqstacks, hard=c034a000 soft=c0342000
Initializing CPU#2
Calibrating delay using timer specific routine.. 5316.62 BogoMIPS
(lpj=10633255)CPU: After generic identify, caps: bfebfbff 00000000 00000000
00000000 00004400 00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 3
CPU: After all inits, caps: bfebfbff 00000000 00000000 00003080 00004400
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#2.
CPU2: Intel P4/Xeon Extended MCE MSRs (12) available
CPU2: Thermal monitoring enabled
CPU2: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Booting processor 3/7 eip 2000
CPU 3 irqstacks, hard=c034b000 soft=c0343000
Initializing CPU#3
Calibrating delay using timer specific routine.. 5316.65 BogoMIPS
(lpj=10633314)CPU: After generic identify, caps: bfebfbff 00000000 00000000
00000000 00004400 00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 3
CPU: After all inits, caps: bfebfbff 00000000 00000000 00003080 00004400
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#3.
CPU3: Intel P4/Xeon Extended MCE MSRs (12) available
CPU3: Thermal monitoring enabled
CPU3: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Total of 4 processors activated (21270.36 BogoMIPS).
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
checking TSC synchronization across 4 CPUs: passed.
Brought up 4 CPUs
migration_cost=142,629
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: PCI BIOS revision 2.10 entry at 0xfbdf2, last bus=5
PCI: Using configuration type 1
Setting up standard PCI resources
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI: Probing PCI hardware (bus 00)
ACPI: Assume root bridge [\_SB_.PCI0] bus is 0
PCI quirk: region 0800-087f claimed by ICH4 ACPI/GPIO/TCO
PCI quirk: region 0880-08bf claimed by ICH4 GPIO
Boot video device is 0000:01:00.0
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1.PCI2._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1.PCI3._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI4._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *9 10 11 12 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 *10 11 12 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 *11 12 15)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 13 devices
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
pnp: 00:0c: ioport range 0x800-0x85f could not be reserved
pnp: 00:0c: ioport range 0xc00-0xc7f has been reserved
pnp: 00:0c: ioport range 0x860-0x8ff could not be reserved
PCI: Bridge: 0000:00:01.0
  IO window: e000-efff
  MEM window: ff800000-ff9fffff
  PREFETCH window: e8000000-f7ffffff
PCI: Bridge: 0000:02:1d.0
  IO window: d000-dfff
  MEM window: ff600000-ff7fffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:02:1f.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:02.0
  IO window: d000-dfff
  MEM window: ff400000-ff7fffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
  IO window: c000-cfff
  MEM window: ff200000-ff3fffff
  PREFETCH window: 88000000-880fffff
PCI: Setting latency timer of device 0000:00:1e.0 to 64
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
TCP established hash table entries: 131072 (order: 8, 1572864 bytes)
TCP bind hash table entries: 65536 (order: 7, 786432 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
checking if image is initramfs... it is
Freeing initrd memory: 898k freed
Simple Boot Flag value 0x87 read from CMOS RAM was invalid
Simple Boot Flag at 0x7a set to 0x1
Machine check exception polling timer started.
highmem bounce pool size: 64 pages
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler cfq registered (default)
Real Time Clock Driver v1.12ac
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:0a: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH4: IDE controller at PCI slot 0000:00:1f.1
PCI: Enabling device 0000:00:1f.1 (0005 -> 0007)
ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 16
ICH4: chipset revision 1
ICH4: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:DMA
Probing IDE interface ide0...
hda: IC35L090AVV207-0, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hdc: HL-DT-STDVD-ROM GDR8162B, ATAPI CD/DVD-ROM drive
hdd: SONY CD-RW CRX216E, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 512KiB
hda: 156250000 sectors (80000 MB) w/1821KiB Cache, CHS=16383/255/63, UDMA(100)
hda: cache flushes supported
 hda: hda1 hda2 hda3 hda4 < hda5 hda6 hda7 >
PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
md: raid1 personality registered for level 1
device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised:
dm-devel@redhat.cominput: AT Translated Set 2 keyboard as /class/input/input0
TCP cubic registered
NET: Registered protocol family 1
Testing NMI watchdog ... OK.
Starting balanced_irq
Using IPI Shortcut mode
Freeing unused kernel memory: 192k freed
Time: tsc clocksource has been installed.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Linux agpgart interface v0.101 (c) Dave Jones
agpgart: Detected an Intel E7505 Chipset.
agpgart: AGP aperture is 128M @ 0xe0000000
input: PC Speaker as /class/input/input1
ACPI: PCI Interrupt 0000:00:1f.3[B] -> GSI 17 (level, low) -> IRQ 17
hdc: ATAPI 48X DVD-ROM drive, 256kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
EDAC MC: Ver: 2.0.1 Mar 17 2007
hdd: ATAPI 48X CD-ROM CD-R/RW drive, 2048kB Cache, UDMA(33)
Intel(R) PRO/1000 Network Driver - version 7.3.15-k2-NAPI
Copyright (c) 1999-2006 Intel Corporation.
ACPI: PCI Interrupt 0000:03:0e.0[A] -> GSI 24 (level, low) -> IRQ 18
SCSI subsystem initialized
input: ImPS/2 Generic Wheel Mouse as /class/input/input2
libata version 2.00 loaded.
ACPI: PCI Interrupt 0000:00:1d.7[D] -> GSI 23 (level, low) -> IRQ 19
PCI: Setting latency timer of device 0000:00:1d.7 to 64
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:1d.7: debug port 1
PCI: cache line size of 128 is not supported by device 0000:00:1d.7
ehci_hcd 0000:00:1d.7: irq 19, io mem 0xffa20000
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 6 ports detected
e1000: 0000:03:0e.0: e1000_probe: (PCI-X:100MHz:64-bit) 00:0d:56:0d:e7:3f
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
ieee1394: Initialized config rom entry `ip1394'
USB Universal Host Controller Interface driver v3.0
sata_sil 0000:05:0e.0: version 2.0
ACPI: PCI Interrupt 0000:05:0e.0[A] -> GSI 22 (level, low) -> IRQ 20
ata1: SATA max UDMA/100 cmd 0xF8934C80 ctl 0xF8934C8A bmdma 0xF8934C00 irq 20
ata2: SATA max UDMA/100 cmd 0xF8934CC0 ctl 0xF8934CCA bmdma 0xF8934C08 irq 20
scsi0 : sata_sil
ata1: SATA link down (SStatus 0 SControl 310)
scsi1 : sata_sil
ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata2.00: ATA-7, max UDMA/133, 488397168 sectors: LBA48 NCQ (depth 0/32)
ata2.00: ata2: dev 0 multi count 16
ata2.00: configured for UDMA/100
scsi 1:0:0:0: Direct-Access     ATA      HDT722525DLA380  V44O PQ: 0 ANSI: 5
ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 21
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1d.0: irq 21, io base 0x0000ff80
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
scsi 1:0:0:0: Attached scsi generic sg0 type 0
SCSI device sda: 488397168 512-byte hdwr sectors (250059 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
SCSI device sda: 488397168 512-byte hdwr sectors (250059 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
 sda: sda1 sda2 sda3
sd 1:0:0:0: Attached scsi disk sda
ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 22
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1d.1: irq 22, io base 0x0000ff60
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 16
PCI: Setting latency timer of device 0000:00:1d.2 to 64
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:1d.2: irq 16, io base 0x0000ff40
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
ACPI: PCI Interrupt 0000:05:0c.0[A] -> GSI 20 (level, low) -> IRQ 23
usb 2-2: new full speed USB device using uhci_hcd and address 2
ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[23]  MMIO=[ff2ff800-ff2fffff]  Max
Packet=[2048]  IR/IT contexts=[4/8]
ACPI: PCI Interrupt 0000:05:0d.2[B] -> GSI 22 (level, low) -> IRQ 20
ohci1394: fw-host1: OHCI-1394 1.1 (PCI): IRQ=[20]  MMIO=[ff2ff000-ff2ff7ff]  Max
Packet=[2048]  IR/IT contexts=[4/8]
ACPI: PCI Interrupt 0000:05:0d.0[A] -> GSI 21 (level, low) -> IRQ 24
Installing spdif_bug patch: Audigy 2 ZS [SB0353]
usb 2-2: configuration #1 chosen from 1 choice
Linux video capture interface: v2.00
pwc: Philips webcam module version 10.0.12 loaded.
pwc: Supports Philips PCA645/646, PCVC675/680/690, PCVC720[40]/730/740/750 &
PCVC830/840.
pwc: Also supports the Askey VC010, various Logitech Quickcams, Samsung MPC-C10
and MPC-C30,
pwc: the Creative WebCam 5 & Pro Ex, SOTEC Afina Eye and Visionite VCS-UC300 and
VCS-UM100.
ieee1394: Host added: ID:BUS[0-00:1023]  GUID[87ffffffffffff00]
usbcore: registered new interface driver snd-usb-audio
pwc: Logitech QuickCam Zoom (new model) USB webcam detected.
pwc: Registered as /dev/video0.
usbcore: registered new interface driver Philips webcam
eth1394: eth1: IEEE-1394 IPv4 over 1394 Ethernet (fw-host0)
eth1394: eth2: IEEE-1394 IPv4 over 1394 Ethernet (fw-host1)
ieee1394: Node added: ID:BUS[1-00:1023]  GUID[0050c500001076df]
ieee1394: Host added: ID:BUS[1-01:1023]  GUID[00023c00a1037111]
Non-volatile memory driver v1.2
ieee1394: raw1394: /dev/raw1394 device initialized
video1394: Installed video1394 module
parport: PnPBIOS parport detected.
parport0: PC-style at 0x378 (0x778), irq 7, dma 1 [PCSPP,TRISTATE,COMPAT,ECP,DMA]
lp0: using parport0 (interrupt-driven).
input: Power Button (FF) as /class/input/input3
ACPI: Power Button (FF) [PWRF]
input: Power Button (CM) as /class/input/input4
ACPI: Power Button (CM) [VBTN]
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
EXT3 FS on hda2, internal journal
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda5, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-0, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda3, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 2040212k swap on /dev/hda7.  Priority:-1 extents:1 across:2040212k
IA-32 Microcode Update Driver: v1.14a <tigran@aivazian.fsnet.co.uk>
NET: Registered protocol family 17
e1000: eth0: e1000_watchdog: NIC Link is Up 100 Mbps Full Duplex
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
Mobile IPv6
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period
p4-clockmod: P4/Xeon(TM) CPU On-Demand Clock Modulation available
Time: acpi_pm clocksource has been installed.
usbcore: deregistering interface driver Philips webcam
pwc: Philips webcam module removed.
pwc: Philips webcam module version 10.0.12 loaded.
pwc: Supports Philips PCA645/646, PCVC675/680/690, PCVC720[40]/730/740/750 &
PCVC830/840.
pwc: Also supports the Askey VC010, various Logitech Quickcams, Samsung MPC-C10
and MPC-C30,
pwc: the Creative WebCam 5 & Pro Ex, SOTEC Afina Eye and Visionite VCS-UC300 and
VCS-UM100.
pwc: Logitech QuickCam Zoom (new model) USB webcam detected.
pwc: Registered as /dev/video0.
usbcore: registered new interface driver Philips webcam
[drm] Initialized drm 1.1.0 20060810
ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 16 (level, low) -> IRQ 21
[drm] Initialized radeon 1.25.0 20060524 on minor 0
agpgart: Found an AGP 3.0 compliant device at 0000:00:00.0.
agpgart: Putting AGP V3 device at 0000:00:00.0 into 8x mode
agpgart: Putting AGP V3 device at 0000:01:00.0 into 8x mode
[drm] Setting GART location based on new memory map
[drm] Loading R200 Microcode
[drm] writeback test succeeded in 1 usecs
BUG: unable to handle kernel paging request at virtual address 6b6b6e6b
 printing eip:
c0130653
*pde = 00000000
Oops: 0002 [#1]
PREEMPT SMP
last sysfs file: /class/usb_endpoint/usbdev2.2_ep84/dev
Modules linked in: snd_rtctimer snd_seq_midi radeon drm pwc eeprom
cpufreq_ondemand p4_clockmod speedstep_lib nfsd exportfs ipv6 autofs4 nfs lockd
sunrpc af_packet firmware_class binfmt_misc video thermal processor fan button
ac lp parport_pc parport video1394 raw1394 nvram eth1394 compat_ioctl32 videodev
v4l2_common v4l1_compat snd_usb_audio snd_usb_lib snd_emu10k1_synth
snd_emux_synth snd_seq_virmidi snd_seq_midi_emul sd_mod sg snd_emu10k1
snd_rawmidi snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event
snd_seq snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_device ohci1394 uhci_hcd
sata_sil snd_timer ieee1394 ehci_hcd libata snd_page_alloc snd_util_mem
snd_hwdep scsi_mod snd soundcore serio_raw e1000 e7xxx_edac psmouse edac_mc
ide_cd i2c_i801 pcspkr intel_agp agpgart i2c_core cdrom usbcore ext3 jbd
CPU:    3
EIP:    0060:[<c0130653>]    Not tainted VLI
EFLAGS: 00010202   (2.6.20.3 #1)
EIP is at module_put+0x20/0x52
eax: 6b6b6e6b   ebx: 6b6b6b6b   ecx: 00000001   edx: eccee000
esi: ed6cfb88   edi: 6b6b6b6b   ebp: ed7637dc   esp: ecceef58
ds: 007b   es: 007b   ss: 0068
Process udevd (pid: 6655, ti=eccee000 task=f7b5fa90 task.ti=eccee000)
Stack: f54a89d0 c0184563 00000010 f596ee88 ed2867f0 c01517e7 00000000 00000000
       f7834730 f596ee88 f7e86b1c 00000000 f596ee88 c014f308 00000003 00000003
       f7e86b1c f7e86b9c c0150301 00000003 00000007 00000003 eccee000 c0102bde
Call Trace:
 [<c0184563>] sysfs_release+0x2d/0x4c
 [<c01517e7>] __fput+0x96/0x13c
 [<c014f308>] filp_close+0x51/0x58
 [<c0150301>] sys_close+0x70/0xa7
 [<c0102bde>] sysenter_past_esp+0x5f/0x85
 [<c0270033>] unix_dgram_recvmsg+0x1b7/0x1c8
 =======================
Code: 00 89 f0 83 c4 0c 5b 5e 5f 5d c3 53 89 c3 85 c0 74 49 b8 01 00 00 00 e8 2b
44 fe ff e8 1b 64 07 00 c1 e0 07 8d 84 18 80 01 00 00 <ff> 08 83 3b 02 75 0b 8b
83 88 05 00 00 e8 75 40 fe ff b8 01 00
EIP: [<c0130653>] module_put+0x20/0x52 SS:ESP 0068:ecceef58
 <6>note: udevd[6655] exited with preempt_count 1
BUG: scheduling while atomic: udevd/0x10000001/6655
 [<c0271fd6>] __sched_text_start+0x56/0x9a2
 [<c011cef6>] irq_exit+0x37/0x42
 [<c010d1e3>] smp_apic_timer_interrupt+0x70/0x79
 [<c01036ac>] apic_timer_interrupt+0x28/0x30
 [<c0114b0d>] __cond_resched+0x12/0x2c
 [<c0272f26>] cond_resched+0x26/0x31
 [<c0140a4a>] unmap_vmas+0x3d3/0x4df
 [<c014326d>] exit_mmap+0x7e/0x10a
 [<c0116c7d>] mmput+0x37/0xab
 [<c011b562>] do_exit+0x1c9/0x71b
 [<c011007b>] sys_vm86+0x8d/0x21d
 [<c010410c>] die+0x1f7/0x21c
 [<c011181a>] do_page_fault+0x442/0x510
 [<c01113d8>] do_page_fault+0x0/0x510
 [<c027494c>] error_code+0x7c/0x84
 [<c0130653>] module_put+0x20/0x52
 [<c0184563>] sysfs_release+0x2d/0x4c
 [<c01517e7>] __fput+0x96/0x13c
 [<c014f308>] filp_close+0x51/0x58
 [<c0150301>] sys_close+0x70/0xa7
 [<c0102bde>] sysenter_past_esp+0x5f/0x85
 [<c0270033>] unix_dgram_recvmsg+0x1b7/0x1c8
 =======================

AN INTERESTING DEVELOPMENT :-)

My original /etc/rc.d/rc.local file was running the following commands:

rmmod pwc || true
modprobe pwc

This was because the pwc module used to have trouble initialising correctly on a
cold boot. Then I wrote a udev rule to run setpwc, but I left the commands in
rc.local. I have now removed these commands, and tried triggering the bug again.
And this time, there is an important difference - the poison value is 0x5a
instead of 0x6b.

Linux version 2.6.20.3 (chris@volcano.underworld) (gcc version 4.1.1 20070105
(Red Hat 4.1.1-51)) #1 SMP PREEMPT Sat Mar 17 13:21:18 GMT 2007
BIOS-provided physical RAM map:
sanitize start
sanitize end
copy_e820_map() start: 0000000000000000 size: 00000000000a0000 end:
00000000000a0000 type: 1
copy_e820_map() type is E820_RAM
copy_e820_map() start: 00000000000f0000 size: 0000000000010000 end:
0000000000100000 type: 2
copy_e820_map() start: 0000000000100000 size: 000000007fe75000 end:
000000007ff75000 type: 1
copy_e820_map() type is E820_RAM
copy_e820_map() start: 000000007ff75000 size: 0000000000002000 end:
000000007ff77000 type: 4
copy_e820_map() start: 000000007ff77000 size: 0000000000021000 end:
000000007ff98000 type: 3
copy_e820_map() start: 000000007ff98000 size: 0000000000068000 end:
0000000080000000 type: 2
copy_e820_map() start: 00000000fec00000 size: 0000000000090000 end:
00000000fec90000 type: 2
copy_e820_map() start: 00000000fee00000 size: 0000000000010000 end:
00000000fee10000 type: 2
copy_e820_map() start: 00000000ffb00000 size: 0000000000500000 end:
0000000100000000 type: 2
 BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000007ff75000 (usable)
 BIOS-e820: 000000007ff75000 - 000000007ff77000 (ACPI NVS)
 BIOS-e820: 000000007ff77000 - 000000007ff98000 (ACPI data)
 BIOS-e820: 000000007ff98000 - 0000000080000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec90000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
 BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
1151MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000fe710
Entering add_active_range(0, 0, 524149) 0 entries of 256 used
Zone PFN ranges:
  DMA             0 ->     4096
  Normal       4096 ->   229376
  HighMem    229376 ->   524149
early_node_map[1] active PFN ranges
    0:        0 ->   524149
On node 0 totalpages: 524149
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 4064 pages, LIFO batch:0
  Normal zone: 1760 pages used for memmap
  Normal zone: 223520 pages, LIFO batch:31
  HighMem zone: 2302 pages used for memmap
  HighMem zone: 292471 pages, LIFO batch:31
DMI 2.3 present.
ACPI: RSDP (v000 DELL                                  ) @ 0x000febc0
ACPI: RSDT (v001 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd4f1
ACPI: FADT (v001 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd529
ACPI: SSDT (v001   DELL    st_ex 0x00001000 MSFT 0x0100000d) @ 0xfffefafa
ACPI: MADT (v001 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd59d
ACPI: BOOT (v001 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd621
ACPI: ASF! (v016 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd649
ACPI: DSDT (v001   DELL    dt_ex 0x00001000 MSFT 0x0100000d) @ 0x00000000
ACPI: PM-Timer IO Port: 0x808
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:2 APIC version 20
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x06] enabled)
Processor #6 15:2 APIC version 20
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled)
Processor #1 15:2 APIC version 20
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x07] enabled)
Processor #7 15:2 APIC version 20
ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x09] address[0xfec80000] gsi_base[24])
IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI 24-47
ACPI: IOAPIC (id[0x0a] address[0xfec80800] gsi_base[48])
IOAPIC[2]: apic_id 10, version 32, address 0xfec80800, GSI 48-71
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Enabling APIC mode:  Flat.  Using 3 I/O APICs
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at 88000000 (gap: 80000000:7ec00000)
Detected 2658.137 MHz processor.
Built 1 zonelists.  Total pages: 520055
Kernel command line: ro root=LABEL=/ nmi_watchdog=1 elevator=cfq
console=ttyS0,115200n8 console=tty0 acpi_pm_good
mapped APIC to ffffd000 (fee00000)
mapped IOAPIC to ffffc000 (fec00000)
mapped IOAPIC to ffffb000 (fec80000)
mapped IOAPIC to ffffa000 (fec80800)
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c0348000 soft=c0340000
PID hash table entries: 4096 (order: 12, 16384 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 2074464k/2096596k available (1493k kernel code, 20856k reserved, 581k
data, 192k init, 1179092k highmem)
virtual kernel memory layout:
    fixmap  : 0xffedf000 - 0xfffff000   (1152 kB)
    pkmap   : 0xff800000 - 0xffc00000   (4096 kB)
    vmalloc : 0xf8800000 - 0xff7fe000   ( 111 MB)
    lowmem  : 0xc0000000 - 0xf8000000   ( 896 MB)
      .init : 0xc030b000 - 0xc033b000   ( 192 kB)
      .data : 0xc027559b - 0xc0306c54   ( 581 kB)
      .text : 0xc0100000 - 0xc027559b   (1493 kB)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 5320.39 BogoMIPS
(lpj=10640784)Mount-cache hash table entries: 512
CPU: After generic identify, caps: bfebfbff 00000000 00000000 00000000 00004400
00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 0
CPU: After all inits, caps: bfebfbff 00000000 00000000 00003080 00004400
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU0: Intel P4/Xeon Extended MCE MSRs (12) available
CPU0: Thermal monitoring enabled
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 10k freed
ACPI: Core revision 20060707
CPU0: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Booting processor 1/1 eip 2000
CPU 1 irqstacks, hard=c0349000 soft=c0341000
Initializing CPU#1
Calibrating delay using timer specific routine.. 5316.60 BogoMIPS
(lpj=10633204)CPU: After generic identify, caps: bfebfbff 00000000 00000000
00000000 00004400 00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 0
CPU: After all inits, caps: bfebfbff 00000000 00000000 00003080 00004400
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel P4/Xeon Extended MCE MSRs (12) available
CPU1: Thermal monitoring enabled
CPU1: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Booting processor 2/6 eip 2000
CPU 2 irqstacks, hard=c034a000 soft=c0342000
Initializing CPU#2
Calibrating delay using timer specific routine.. 5316.66 BogoMIPS
(lpj=10633320)CPU: After generic identify, caps: bfebfbff 00000000 00000000
00000000 00004400 00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 3
CPU: After all inits, caps: bfebfbff 00000000 00000000 00003080 00004400
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#2.
CPU2: Intel P4/Xeon Extended MCE MSRs (12) available
CPU2: Thermal monitoring enabled
CPU2: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Booting processor 3/7 eip 2000
CPU 3 irqstacks, hard=c034b000 soft=c0343000
Initializing CPU#3
Calibrating delay using timer specific routine.. 5316.67 BogoMIPS
(lpj=10633358)CPU: After generic identify, caps: bfebfbff 00000000 00000000
00000000 00004400 00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 3
CPU: After all inits, caps: bfebfbff 00000000 00000000 00003080 00004400
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#3.
CPU3: Intel P4/Xeon Extended MCE MSRs (12) available
CPU3: Thermal monitoring enabled
CPU3: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Total of 4 processors activated (21270.33 BogoMIPS).
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
checking TSC synchronization across 4 CPUs: passed.
Brought up 4 CPUs
migration_cost=99,633
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: PCI BIOS revision 2.10 entry at 0xfbdf2, last bus=5
PCI: Using configuration type 1
Setting up standard PCI resources
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI: Probing PCI hardware (bus 00)
ACPI: Assume root bridge [\_SB_.PCI0] bus is 0
PCI quirk: region 0800-087f claimed by ICH4 ACPI/GPIO/TCO
PCI quirk: region 0880-08bf claimed by ICH4 GPIO
Boot video device is 0000:01:00.0
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1.PCI2._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1.PCI3._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI4._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *9 10 11 12 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 *10 11 12 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 *11 12 15)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 13 devices
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
pnp: 00:0c: ioport range 0x800-0x85f could not be reserved
pnp: 00:0c: ioport range 0xc00-0xc7f has been reserved
pnp: 00:0c: ioport range 0x860-0x8ff could not be reserved
PCI: Bridge: 0000:00:01.0
  IO window: e000-efff
  MEM window: ff800000-ff9fffff
  PREFETCH window: e8000000-f7ffffff
PCI: Bridge: 0000:02:1d.0
  IO window: d000-dfff
  MEM window: ff600000-ff7fffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:02:1f.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:02.0
  IO window: d000-dfff
  MEM window: ff400000-ff7fffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
  IO window: c000-cfff
  MEM window: ff200000-ff3fffff
  PREFETCH window: 88000000-880fffff
PCI: Setting latency timer of device 0000:00:1e.0 to 64
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
TCP established hash table entries: 131072 (order: 8, 1572864 bytes)
TCP bind hash table entries: 65536 (order: 7, 786432 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
checking if image is initramfs... it is
Freeing initrd memory: 898k freed
Simple Boot Flag value 0x87 read from CMOS RAM was invalid
Simple Boot Flag at 0x7a set to 0x1
Machine check exception polling timer started.
highmem bounce pool size: 64 pages
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler cfq registered (default)
Real Time Clock Driver v1.12ac
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:0a: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH4: IDE controller at PCI slot 0000:00:1f.1
PCI: Enabling device 0000:00:1f.1 (0005 -> 0007)
ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 16
ICH4: chipset revision 1
ICH4: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:DMA
Probing IDE interface ide0...
hda: IC35L090AVV207-0, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hdc: HL-DT-STDVD-ROM GDR8162B, ATAPI CD/DVD-ROM drive
hdd: SONY CD-RW CRX216E, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 512KiB
hda: 156250000 sectors (80000 MB) w/1821KiB Cache, CHS=16383/255/63, UDMA(100)
hda: cache flushes supported
 hda: hda1 hda2 hda3 hda4 < hda5 hda6 hda7 >
PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
md: raid1 personality registered for level 1
device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised:
dm-devel@redhat.comTCP cubic registered
NET: Registered protocol family 1
Testing NMI watchdog ... OK.
Starting balanced_irq
Using IPI Shortcut mode
input: AT Translated Set 2 keyboard as /class/input/input0
Time: tsc clocksource has been installed.
Freeing unused kernel memory: 192k freed
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Linux agpgart interface v0.101 (c) Dave Jones
agpgart: Detected an Intel E7505 Chipset.
agpgart: AGP aperture is 128M @ 0xe0000000
ACPI: PCI Interrupt 0000:00:1f.3[B] -> GSI 17 (level, low) -> IRQ 17
input: PC Speaker as /class/input/input1
hdc: ATAPI 48X DVD-ROM drive, 256kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
EDAC MC: Ver: 2.0.1 Mar 17 2007
hdd: ATAPI 48X CD-ROM CD-R/RW drive, 2048kB Cache, UDMA(33)
SCSI subsystem initialized
libata version 2.00 loaded.
Intel(R) PRO/1000 Network Driver - version 7.3.15-k2-NAPI
Copyright (c) 1999-2006 Intel Corporation.
ACPI: PCI Interrupt 0000:03:0e.0[A] -> GSI 24 (level, low) -> IRQ 18
USB Universal Host Controller Interface driver v3.0
ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 19
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:1d.0: irq 19, io base 0x0000ff80
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
input: ImPS/2 Generic Wheel Mouse as /class/input/input2
ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 20
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1d.1: irq 20, io base 0x0000ff60
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
e1000: 0000:03:0e.0: e1000_probe: (PCI-X:100MHz:64-bit) 00:0d:56:0d:e7:3f
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
sata_sil 0000:05:0e.0: version 2.0
ACPI: PCI Interrupt 0000:05:0e.0[A] -> GSI 22 (level, low) -> IRQ 21
ata1: SATA max UDMA/100 cmd 0xF88BEC80 ctl 0xF88BEC8A bmdma 0xF88BEC00 irq 21
ata2: SATA max UDMA/100 cmd 0xF88BECC0 ctl 0xF88BECCA bmdma 0xF88BEC08 irq 21
scsi0 : sata_sil
ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 16
PCI: Setting latency timer of device 0000:00:1d.2 to 64
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1d.2: irq 16, io base 0x0000ff40
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
usb 1-2: new full speed USB device using uhci_hcd and address 2
ata1: SATA link down (SStatus 0 SControl 310)
scsi1 : sata_sil
ieee1394: Initialized config rom entry `ip1394'
usb 1-2: configuration #1 chosen from 1 choice
Linux video capture interface: v2.00
ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata2.00: ATA-7, max UDMA/133, 488397168 sectors: LBA48 NCQ (depth 0/32)
ata2.00: ata2: dev 0 multi count 16
ata2.00: configured for UDMA/100
scsi 1:0:0:0: Direct-Access     ATA      HDT722525DLA380  V44O PQ: 0 ANSI: 5
ACPI: PCI Interrupt 0000:05:0c.0[A] -> GSI 20 (level, low) -> IRQ 22
pwc: Philips webcam module version 10.0.12 loaded.
pwc: Supports Philips PCA645/646, PCVC675/680/690, PCVC720[40]/730/740/750 &
PCVC830/840.
pwc: Also supports the Askey VC010, various Logitech Quickcams, Samsung MPC-C10
and MPC-C30,
pwc: the Creative WebCam 5 & Pro Ex, SOTEC Afina Eye and Visionite VCS-UC300 and
VCS-UM100.
scsi 1:0:0:0: Attached scsi generic sg0 type 0
ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[22]  MMIO=[ff2ff800-ff2fffff]  Max
Packet=[2048]  IR/IT contexts=[4/8]
ACPI: PCI Interrupt 0000:05:0d.2[B] -> GSI 22 (level, low) -> IRQ 21
SCSI device sda: 488397168 512-byte hdwr sectors (250059 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
SCSI device sda: 488397168 512-byte hdwr sectors (250059 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
 sda: sda1 sda2 sda3
sd 1:0:0:0: Attached scsi disk sda
ohci1394: fw-host1: OHCI-1394 1.1 (PCI): IRQ=[21]  MMIO=[ff2ff000-ff2ff7ff]  Max
Packet=[2048]  IR/IT contexts=[4/8]
ACPI: PCI Interrupt 0000:00:1d.7[D] -> GSI 23 (level, low) -> IRQ 23
ACPI: PCI Interrupt 0000:05:0d.0[A] -> GSI 21 (level, low) -> IRQ 24
PCI: Setting latency timer of device 0000:00:1d.7 to 64
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 4
ehci_hcd 0000:00:1d.7: debug port 1
PCI: cache line size of 128 is not supported by device 0000:00:1d.7
ehci_hcd 0000:00:1d.7: irq 23, io mem 0xffa20000
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 6 ports detected
Installing spdif_bug patch: Audigy 2 ZS [SB0353]
usb 1-2: USB disconnect, address 2
usbcore: registered new interface driver snd-usb-audio
usbcore: registered new interface driver Philips webcam
usb 1-2: new full speed USB device using uhci_hcd and address 3
Non-volatile memory driver v1.2
parport: PnPBIOS parport detected.
parport0: PC-style at 0x378 (0x778), irq 7, dma 1 [PCSPP,TRISTATE,COMPAT,ECP,DMA]
usb 1-2: configuration #1 chosen from 1 choice
pwc: Logitech QuickCam Zoom (new model) USB webcam detected.
pwc: Registered as /dev/video0.
lp0: using parport0 (interrupt-driven).
ieee1394: Host added: ID:BUS[0-00:1023]  GUID[87ffffffffffff00]
eth1394: eth1: IEEE-1394 IPv4 over 1394 Ethernet (fw-host0)
eth1394: eth2: IEEE-1394 IPv4 over 1394 Ethernet (fw-host1)
input: Power Button (FF) as /class/input/input3
ACPI: Power Button (FF) [PWRF]
input: Power Button (CM) as /class/input/input4
ACPI: Power Button (CM) [VBTN]
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
ieee1394: Node added: ID:BUS[1-00:1023]  GUID[0050c500001076df]
ieee1394: Host added: ID:BUS[1-01:1023]  GUID[00023c00a1037111]
ieee1394: raw1394: /dev/raw1394 device initialized
video1394: Installed video1394 module
EXT3 FS on hda2, internal journal
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda5, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-0, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda3, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 2040212k swap on /dev/hda7.  Priority:-1 extents:1 across:2040212k
IA-32 Microcode Update Driver: v1.14a <tigran@aivazian.fsnet.co.uk>
NET: Registered protocol family 17
e1000: eth0: e1000_watchdog: NIC Link is Up 100 Mbps Full Duplex
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
Mobile IPv6
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period
p4-clockmod: P4/Xeon(TM) CPU On-Demand Clock Modulation available
Time: acpi_pm clocksource has been installed.
[drm] Initialized drm 1.1.0 20060810
ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 16 (level, low) -> IRQ 19
[drm] Initialized radeon 1.25.0 20060524 on minor 0
agpgart: Found an AGP 3.0 compliant device at 0000:00:00.0.
agpgart: Putting AGP V3 device at 0000:00:00.0 into 8x mode
agpgart: Putting AGP V3 device at 0000:01:00.0 into 8x mode
[drm] Setting GART location based on new memory map
[drm] Loading R200 Microcode
[drm] writeback test succeeded in 1 usecs
BUG: unable to handle kernel paging request at virtual address 6b6b6e6b
 printing eip:
c0130653
*pde = 00000000
Oops: 0002 [#1]
PREEMPT SMP
last sysfs file: /class/usb_endpoint/usbdev1.3_ep84/dev
Modules linked in: snd_rtctimer snd_seq_midi radeon drm eeprom cpufreq_ondemand
p4_clockmod speedstep_lib nfsd exportfs ipv6 autofs4 nfs lockd sunrpc af_packet
firmware_class binfmt_misc video1394 raw1394 video thermal processor fan button
ac eth1394 lp parport_pc parport nvram snd_emu10k1_synth snd_emux_synth
snd_seq_virmidi snd_seq_midi_emul sd_mod sg pwc compat_ioctl32 videodev
snd_usb_audio snd_usb_lib v4l2_common v4l1_compat snd_emu10k1 ohci1394
snd_rawmidi snd_ac97_codec ieee1394 ac97_bus snd_seq_dummy snd_seq_oss
snd_seq_midi_event snd_seq serio_raw snd_pcm_oss snd_mixer_oss ehci_hcd sata_sil
snd_pcm uhci_hcd snd_seq_device snd_timer snd_page_alloc snd_util_mem snd_hwdep
snd e1000 libata soundcore psmouse scsi_mod e7xxx_edac edac_mc ide_cd pcspkr
i2c_i801 i2c_core intel_agp agpgart cdrom usbcore ext3 jbd
CPU:    3
EIP:    0060:[<c0130653>]    Not tainted VLI
EFLAGS: 00010202   (2.6.20.3 #1)
EIP is at module_put+0x20/0x52
eax: 6b6b6e6b   ebx: 6b6b6b6b   ecx: 00000001   edx: f7fe9000
esi: ed64510c   edi: 6b6b6b6b   ebp: ed247e9c   esp: f7fe9f58
ds: 007b   es: 007b   ss: 0068
Process udevd (pid: 4589, ti=f7fe9000 task=ed0eea90 task.ti=f7fe9000)
Stack: f5bf2b4c c0184563 00000010 ed200564 ed115c88 c01517e7 00000000 00000000
       f7ff2208 ed200564 ed2cee34 00000000 ed200564 c014f308 00000003 00000003
       ed2cee34 ed2ceeb4 c0150301 00000003 00000007 00000003 f7fe9000 c0102bde
Call Trace:
 [<c0184563>] sysfs_release+0x2d/0x4c
 [<c01517e7>] __fput+0x96/0x13c
 [<c014f308>] filp_close+0x51/0x58
 [<c0150301>] sys_close+0x70/0xa7
 [<c0102bde>] sysenter_past_esp+0x5f/0x85
 [<c0270033>] unix_dgram_recvmsg+0x1b7/0x1c8
 =======================
Code: 00 89 f0 83 c4 0c 5b 5e 5f 5d c3 53 89 c3 85 c0 74 49 b8 01 00 00 00 e8 2b
44 fe ff e8 1b 64 07 00 c1 e0 07 8d 84 18 80 01 00 00 <ff> 08 83 3b 02 75 0b 8b
83 88 05 00 00 e8 75 40 fe ff b8 01 00
EIP: [<c0130653>] module_put+0x20/0x52 SS:ESP 0068:f7fe9f58
 <6>note: udevd[4589] exited with preempt_count 1
BUG: scheduling while atomic: udevd/0x10000001/4589
 [<c0271fd6>] __sched_text_start+0x56/0x9a2
 [<c011cef6>] irq_exit+0x37/0x42
 [<c010d1e3>] smp_apic_timer_interrupt+0x70/0x79
 [<c01036ac>] apic_timer_interrupt+0x28/0x30
 [<c0114b0d>] __cond_resched+0x12/0x2c
 [<c0272f26>] cond_resched+0x26/0x31
 [<c0140a4a>] unmap_vmas+0x3d3/0x4df
 [<c014326d>] exit_mmap+0x7e/0x10a
 [<c0116c7d>] mmput+0x37/0xab
 [<c011b562>] do_exit+0x1c9/0x71b
 [<c011007b>] sys_vm86+0x8d/0x21d
 [<c010410c>] die+0x1f7/0x21c
 [<c011181a>] do_page_fault+0x442/0x510
 [<c01113d8>] do_page_fault+0x0/0x510
 [<c027494c>] error_code+0x7c/0x84
 [<c0130653>] module_put+0x20/0x52
 [<c0184563>] sysfs_release+0x2d/0x4c
 [<c01517e7>] __fput+0x96/0x13c
 [<c014f308>] filp_close+0x51/0x58
 [<c0150301>] sys_close+0x70/0xa7
 [<c0102bde>] sysenter_past_esp+0x5f/0x85
 [<c0270033>] unix_dgram_recvmsg+0x1b7/0x1c8
 =======================
BUG: unable to handle kernel paging request at virtual address 5a5a5a5a
 printing eip:
5a5a5a5a
*pde = 00000000
Oops: 0000 [#2]
PREEMPT SMP
last sysfs file: /class/usb_endpoint/usbdev1.3_ep84/dev
Modules linked in: snd_rtctimer snd_seq_midi radeon drm eeprom cpufreq_ondemand
p4_clockmod speedstep_lib nfsd exportfs ipv6 autofs4 nfs lockd sunrpc af_packet
firmware_class binfmt_misc video1394 raw1394 video thermal processor fan button
ac eth1394 lp parport_pc parport nvram snd_emu10k1_synth snd_emux_synth
snd_seq_virmidi snd_seq_midi_emul sd_mod sg pwc compat_ioctl32 videodev
snd_usb_audio snd_usb_lib v4l2_common v4l1_compat snd_emu10k1 ohci1394
snd_rawmidi snd_ac97_codec ieee1394 ac97_bus snd_seq_dummy snd_seq_oss
snd_seq_midi_event snd_seq serio_raw snd_pcm_oss snd_mixer_oss ehci_hcd sata_sil
snd_pcm uhci_hcd snd_seq_device snd_timer snd_page_alloc snd_util_mem snd_hwdep
snd e1000 libata soundcore psmouse scsi_mod e7xxx_edac edac_mc ide_cd pcspkr
i2c_i801 i2c_core intel_agp agpgart cdrom usbcore ext3 jbd
CPU:    0
EIP:    0060:[<5a5a5a5a>]    Not tainted VLI
EFLAGS: 00010206   (2.6.20.3 #1)
EIP is at 0x5a5a5a5a
eax: ed645084   ebx: 5a5a5a5a   ecx: ed2f5000   edx: f7ba445c
esi: ed64510c   edi: c02efa2c   ebp: ed64510c   esp: f755df48
ds: 007b   es: 007b   ss: 0068
Process udevd (pid: 4624, ti=f755d000 task=ecc40560 task.ti=f755d000)
Stack: c01e8d1d f636c134 f5461d34 c0184499 00000080 bfc990c4 c014007b f7ba445c
       f636c148 ecc3c7d4 bfc990c4 c0184421 00000080 c0150fed f755dfa0 ecc3c7d4
       ecc3c7d4 fffffff7 00000003 f755d000 c0151443 f755dfa0 00000000 00000000
Call Trace:
 [<c01e8d1d>] dev_attr_show+0x18/0x1b
 [<c0184499>] sysfs_read_file+0x78/0x115
 [<c014007b>] pmd_clear_bad+0x25/0x33
 [<c0184421>] sysfs_read_file+0x0/0x115
 [<c0150fed>] vfs_read+0x88/0x134
 [<c0151443>] sys_read+0x41/0x67
 [<c0102bde>] sysenter_past_esp+0x5f/0x85
 [<c0270033>] unix_dgram_recvmsg+0x1b7/0x1c8
 =======================
Code:  Bad EIP value.
EIP: [<5a5a5a5a>] 0x5a5a5a5a SS:ESP 0068:f755df48

$ /sbin/lsusb
Bus 004 Device 001: ID 0000:0000
Bus 002 Device 001: ID 0000:0000
Bus 003 Device 001: ID 0000:0000
Bus 001 Device 003: ID 046d:08b4 Logitech, Inc. QuickCam Zoom
Bus 001 Device 001: ID 0000:0000

According to my oopsen, the following sysfs files have been implicated by this bug:

/class/net/eth2/statistics/collisions
/class/usb_endpoint/usbdev1.3_ep84/dev
/class/usb_endpoint/usbdev1.3_ep82/dev
/class/usb_endpoint/usbdev2.2_ep84/dev

So as an experiment, I have tried wandering around the /sys directory and
opening files, and then looking at the module reference counts via lsmod.

Apparently, I can hold an open file descriptor on
/sys/devices/pci0000:00/0000:00:1d.0/usb1/1-2/1-2:1.0/usbdev1.3_ep82/dev while
the pwc module still has a reference count of zero! *And* I can do this in
2.6.19.7! I'm thinking that the only reason I can't trigger this bug in 2.6.19.x
is because I've given up too soon. (Maybe it's also harder to trigger for some
reason...?)

It's a similar story with the eth2/statistics/collisions file; I can hold this
file open while the eth1394 module keeps a reference count of zero too.

Should just entering these directories increase the relevant modules' reference
counts?

Created attachment 10821 [details]
Possible patch against multiple-threaded access of ep_class

Seeing as my machine has 2 hyperthreaded physical CPUs, giving a total of 4
logical CPUs, and that I have compiled my kernel with PREEMPT and with usbcore
as a module, what are the chances that multiple threads will call
init_endpoint_class() simultaneously when the usbcore module loads?

I have applied this patch to my kernel, and haven't been able to trigger the
problem again yet... (Which probably means it'll happen 10 seconds after I
press the "submit" button, but anyay.)

Well, the handling of endpoint.c::ep_class really looks strange. I'm not sure
which context creates the class, but I presume there are different contexts
which may destroy it.

One bigger and two minor remarks to your patch:

 - destroy_endpoint_class() takes the mutex while it drops the reference. If it
was the last reference, release_endpoint_class() will be called while the mutex
is held. release_endpoint_class() wants to take the mutex too. -> Recursive
loccking. release_endpoint_class() does not have to take the mutex since it is
only called from destroy_endpoint_class().

 - Maybe an existing subsystem mutex or rwsem could be reused.

 - It could be implemented locklessly if ep_class would be a static struct (not
just a pointer) and if the kref would be replaced by an atomic_t reference
counter (or if the kref.refcount would be manipulated directly, which is
probably not desired).

Still, the ep_class oddity you found does not explain the other sysfs oopses
(eth1394's collsion file etc.).

I'm not surprised that there are issues with the patch. My main concern was
discovering whether it affected the problem in any way. And it does seem to.

As to the eth1394 collision oops, it's possible that there are 2 bugs... ;-).

Created attachment 10824 [details]
2nd attempt at USB endpoint memory corruption bug

OK, I think that "lockless" is good... I'll test this new patch over the next
few days.

Hmm, thinking about this... Using a mutex forced other users to wait until the
ep_class was properly initialised. Doesn't this "atomic increment" approach mean
that a second thread could be trying to use the ep_class while a previous thread
is still in the process of constructing it? Maybe we need a mutex after all,
except make it private to the init_endpoint_class() function instead?

Yes. Looks like you are right that callers of init_endpoint_class, i.e.
usb_create_ep_files, have to sleep until the class is ready or until they can
create the class on their own.

See the minor_lock endpoint_get_minor? Maybe this lock should be replaced by a
common one for both the class creation and teardown and the allocation of minor
device numbers.

But besides, making ep_class a static struct seems worthwhile. Or even simpler,
explode that struct into its two components. I don't see a use for that type
anymore after it was made static.

Created attachment 10825 [details]
3rd attempt at fixing memory corruption with USB endpoints

This patch could almost certainly be improved upon, but it should at least work
and is what I am currently testing.

Created attachment 10826 [details]
4th attempt at fixing memory corruption with USB endpoints

Try that again, except remembering to decrement the atomic counter on the error
path this time.

What are you doing here?  a atomic_t is the same thing as a kref, so you aren't
doing anything different here at all with that (becides not being acceptable at
all.)

And no, the pwc module reference will not be incremented if these files are
open, _AND_ these files do nothing right now, so this is not the problem you are
having at all.

So, I suggest looking elsewhere, as this is the wrong path to be taking...

> What are you doing here?

Is that a rhetorical question?

> a atomic_t is the same thing as a kref, so you aren't doing anything different
> here at all with that (becides not being acceptable at all.)

For my purposes, crash = unacceptable, no-crash = acceptable. If you would like
to avoid having multiple threads creating different ep_class structures at the
same time differently then feel free. But to date, this patch fixes my oops.

> And no, the pwc module reference will not be incremented if these files are
> open, _AND_ these files do nothing right now, so this is not the problem you
> are having at all.

I abandoned this train of thought; it has no relevance to any of the above patches.

Note, accesses to endpoint.c::usb_endpoint_major and apparently also to
endpoint_idr are not properly serialized either.  I suggest that the existing
minor_lock is replaced by a mutex which is global to endpoint.c and is used to
protect ep_class, usb_endpoint_major, and endpoint_idr.  (Sorry, I am too busy
otherwise to send a proper patch.)

On Mon, Mar 19, 2007 at 01:41:48AM -0700, bugme-daemon@bugzilla.kernel.org wrote:
> > a atomic_t is the same thing as a kref, so you aren't doing anything
> different
> > here at all with that (becides not being acceptable at all.)
> 
> For my purposes, crash = unacceptable, no-crash = acceptable. If you would
> like
> to avoid having multiple threads creating different ep_class structures at
> the
> same time differently then feel free. But to date, this patch fixes my oops.

How are multiple threads creating different ep_class structures?  There
is only one core USB thread that does this.

Also a static ep_class is not acceptable as it must be a dynamic
structure, not a static one.

Do you have a test case to reproduce this problem?

It's not for me to care about, but:
  - ep_class is only a container for a pointer to a struct class, thus ep_class
can be static.
  - init_endpoint_class has one caller; this caller has two callers. Are they
happening in the single same context? (usb_create_sysfs_dev_files and
usb_create_sysfs_intf_files)
  - Is creation of ep_class guaranteedly serialized versus deletion?

On Mon, Mar 19, 2007 at 08:32:31AM -0700, bugme-daemon@bugzilla.kernel.org wrote:
>   - ep_class is only a container for a pointer to a struct class, thus
>   ep_class
> can be static.

But this has nothing to do with the real reasons you are trying to track
down here.  Try not to change too many things at once, especially ones
that will not change anything.

>   - init_endpoint_class has one caller; this caller has two callers. Are they
> happening in the single same context? (usb_create_sysfs_dev_files and
> usb_create_sysfs_intf_files)

Yes.

>   - Is creation of ep_class guaranteedly serialized versus deletion?

That is what the kref ensures.

> Try not to change too many things at once,

OK.

>> - Is creation of ep_class guaranteedly serialized versus deletion?
>
> That is what the kref ensures.

No. The kref does not protect
    if (ep_class != NULL) {
        ...
versus
    if (ep_class)
        ...

OK, so I added a printk() statement to endpoint.c:

--- endpoint.c.orig     2007-03-18 22:32:52.000000000 +0000
+++ endpoint.c  2007-03-19 18:46:45.000000000 +0000
@@ -220,6 +220,8 @@
                goto exit;
        }

+       printk(KERN_INFO "Allocating ep_class on CPU %d\n", get_cpu());
+
        ep_class = kmalloc(sizeof(*ep_class), GFP_KERNEL);
        if (!ep_class) {
                result = -ENOMEM;

If this function were being accessed by more than one thread of execution then
that message should appear:

20_map() start: 00000000ffb00000 size: 0000000000500000 end: 0000000100000000
type: 2
 BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000007ff75000 (usable)
 BIOS-e820: 000000007ff75000 - 000000007ff77000 (ACPI NVS)
 BIOS-e820: 000000007ff77000 - 000000007ff98000 (ACPI data)
 BIOS-e820: 000000007ff98000 - 0000000080000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec90000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
 BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
1151MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000fe710
Entering add_active_range(0, 0, 524149) 0 entries of 256 used
Zone PFN ranges:
  DMA             0 ->     4096
  Normal       4096 ->   229376
  HighMem    229376 ->   524149
early_node_map[1] active PFN ranges
    0:        0 ->   524149
On node 0 totalpages: 524149
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 4064 pages, LIFO batch:0
  Normal zone: 1760 pages used for memmap
  Normal zone: 223520 pages, LIFO batch:31
  HighMem zone: 2302 pages used for memmap
  HighMem zone: 292471 pages, LIFO batch:31
DMI 2.3 present.
ACPI: RSDP (v000 DELL                                  ) @ 0x000febc0
ACPI: RSDT (v001 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd4f1
ACPI: FADT (v001 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd529
ACPI: SSDT (v001   DELL    st_ex 0x00001000 MSFT 0x0100000d) @ 0xfffefafa
ACPI: MADT (v001 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd59d
ACPI: BOOT (v001 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd621
ACPI: ASF! (v016 DELL    WS 650  0x00000009 ASL  0x00000061) @ 0x000fd649
ACPI: DSDT (v001   DELL    dt_ex 0x00001000 MSFT 0x0100000d) @ 0x00000000
ACPI: PM-Timer IO Port: 0x808
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
Processor #0 15:2 APIC version 20
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x06] enabled)
Processor #6 15:2 APIC version 20
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled)
Processor #1 15:2 APIC version 20
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x07] enabled)
Processor #7 15:2 APIC version 20
ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x09] address[0xfec80000] gsi_base[24])
IOAPIC[1]: apic_id 9, version 32, address 0xfec80000, GSI 24-47
ACPI: IOAPIC (id[0x0a] address[0xfec80800] gsi_base[48])
IOAPIC[2]: apic_id 10, version 32, address 0xfec80800, GSI 48-71
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Enabling APIC mode:  Flat.  Using 3 I/O APICs
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at 88000000 (gap: 80000000:7ec00000)
Detected 2658.163 MHz processor.
Built 1 zonelists.  Total pages: 520055
Kernel command line: ro root=LABEL=/ nmi_watchdog=1 elevator=cfq
console=ttyS0,115200n8 console=tty0 acpi_pm_good
mapped APIC to ffffd000 (fee00000)
mapped IOAPIC to ffffc000 (fec00000)
mapped IOAPIC to ffffb000 (fec80000)
mapped IOAPIC to ffffa000 (fec80800)
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c0348000 soft=c0340000
PID hash table entries: 4096 (order: 12, 16384 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 2074464k/2096596k available (1493k kernel code, 20856k reserved, 581k
data, 192k init, 1179092k highmem)
virtual kernel memory layout:
    fixmap  : 0xffedf000 - 0xfffff000   (1152 kB)
    pkmap   : 0xff800000 - 0xffc00000   (4096 kB)
    vmalloc : 0xf8800000 - 0xff7fe000   ( 111 MB)
    lowmem  : 0xc0000000 - 0xf8000000   ( 896 MB)
      .init : 0xc030b000 - 0xc033b000   ( 192 kB)
      .data : 0xc027559b - 0xc0306c54   ( 581 kB)
      .text : 0xc0100000 - 0xc027559b   (1493 kB)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 5320.38 BogoMIPS
(lpj=10640776)Mount-cache hash table entries: 512
CPU: After generic identify, caps: bfebfbff 00000000 00000000 00000000 00004400
00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 0
CPU: After all inits, caps: bfebfbff 00000000 00000000 00003080 00004400
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU0: Intel P4/Xeon Extended MCE MSRs (12) available
CPU0: Thermal monitoring enabled
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 10k freed
ACPI: Core revision 20060707
CPU0: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Booting processor 1/1 eip 2000
CPU 1 irqstacks, hard=c0349000 soft=c0341000
Initializing CPU#1
Calibrating delay using timer specific routine.. 5316.59 BogoMIPS
(lpj=10633183)CPU: After generic identify, caps: bfebfbff 00000000 00000000
00000000 00004400 00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 0
CPU: After all inits, caps: bfebfbff 00000000 00000000 00003080 00004400
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#1.
CPU1: Intel P4/Xeon Extended MCE MSRs (12) available
CPU1: Thermal monitoring enabled
CPU1: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Booting processor 2/6 eip 2000
CPU 2 irqstacks, hard=c034a000 soft=c0342000
Initializing CPU#2
Calibrating delay using timer specific routine.. 5316.61 BogoMIPS
(lpj=10633221)CPU: After generic identify, caps: bfebfbff 00000000 00000000
00000000 00004400 00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 3
CPU: After all inits, caps: bfebfbff 00000000 00000000 00003080 00004400
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#2.
CPU2: Intel P4/Xeon Extended MCE MSRs (12) available
CPU2: Thermal monitoring enabled
CPU2: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Booting processor 3/7 eip 2000
CPU 3 irqstacks, hard=c034b000 soft=c0343000
Initializing CPU#3
Calibrating delay using timer specific routine.. 5316.62 BogoMIPS
(lpj=10633254)CPU: After generic identify, caps: bfebfbff 00000000 00000000
00000000 00004400 00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Physical Processor ID: 3
CPU: After all inits, caps: bfebfbff 00000000 00000000 00003080 00004400
00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#3.
CPU3: Intel P4/Xeon Extended MCE MSRs (12) available
CPU3: Thermal monitoring enabled
CPU3: Intel(R) Xeon(TM) CPU 2.66GHz stepping 09
Total of 4 processors activated (21270.21 BogoMIPS).
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
checking TSC synchronization across 4 CPUs: passed.
Brought up 4 CPUs
migration_cost=135,634
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: PCI BIOS revision 2.10 entry at 0xfbdf2, last bus=5
PCI: Using configuration type 1
Setting up standard PCI resources
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI: Probing PCI hardware (bus 00)
ACPI: Assume root bridge [\_SB_.PCI0] bus is 0
PCI quirk: region 0800-087f claimed by ICH4 ACPI/GPIO/TCO
PCI quirk: region 0880-08bf claimed by ICH4 GPIO
Boot video device is 0000:01:00.0
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1.PCI2._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1.PCI3._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI4._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *9 10 11 12 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 *10 11 12 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 *11 12 15)
ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 *11 12 15)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 13 devices
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
pnp: 00:0c: ioport range 0x800-0x85f could not be reserved
pnp: 00:0c: ioport range 0xc00-0xc7f has been reserved
pnp: 00:0c: ioport range 0x860-0x8ff could not be reserved
PCI: Bridge: 0000:00:01.0
  IO window: e000-efff
  MEM window: ff800000-ff9fffff
  PREFETCH window: e8000000-f7ffffff
PCI: Bridge: 0000:02:1d.0
  IO window: d000-dfff
  MEM window: ff600000-ff7fffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:02:1f.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:02.0
  IO window: d000-dfff
  MEM window: ff400000-ff7fffff
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
  IO window: c000-cfff
  MEM window: ff200000-ff3fffff
  PREFETCH window: 88000000-880fffff
PCI: Setting latency timer of device 0000:00:1e.0 to 64
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
TCP established hash table entries: 131072 (order: 8, 1572864 bytes)
TCP bind hash table entries: 65536 (order: 7, 786432 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
checking if image is initramfs... it is
Freeing initrd memory: 898k freed
Simple Boot Flag value 0x87 read from CMOS RAM was invalid
Simple Boot Flag at 0x7a set to 0x1
Machine check exception polling timer started.
highmem bounce pool size: 64 pages
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler cfq registered (default)
Real Time Clock Driver v1.12ac
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:0a: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH4: IDE controller at PCI slot 0000:00:1f.1
PCI: Enabling device 0000:00:1f.1 (0005 -> 0007)
ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 16
ICH4: chipset revision 1
ICH4: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:DMA
Probing IDE interface ide0...
hda: IC35L090AVV207-0, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
Probing IDE interface ide1...
hdc: HL-DT-STDVD-ROM GDR8162B, ATAPI CD/DVD-ROM drive
hdd: SONY CD-RW CRX216E, ATAPI CD/DVD-ROM drive
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 512KiB
hda: 156250000 sectors (80000 MB) w/1821KiB Cache, CHS=16383/255/63, UDMA(100)
hda: cache flushes supported
 hda: hda1 hda2 hda3 hda4 < hda5 hda6 hda7 >
PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
md: raid1 personality registered for level 1
device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised:
dm-devel@redhat.cominput: AT Translated Set 2 keyboard as /class/input/input0
TCP cubic registered
NET: Registered protocol family 1
Testing NMI watchdog ... OK.
Starting balanced_irq
Using IPI Shortcut mode
Freeing unused kernel memory: 192k freed
Time: tsc clocksource has been installed.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
ACPI: PCI Interrupt 0000:00:1f.3[B] -> GSI 17 (level, low) -> IRQ 17
hdc: ATAPI 48X DVD-ROM drive, 256kB Cache, UDMA(33)
Uniform CD-ROM driver Revision: 3.20
hdd: ATAPI 48X CD-ROM CD-R/RW drive, 2048kB Cache, UDMA(33)
input: PC Speaker as /class/input/input1
Linux agpgart interface v0.101 (c) Dave Jones
agpgart: Detected an Intel E7505 Chipset.
agpgart: AGP aperture is 128M @ 0xe0000000
SCSI subsystem initialized
EDAC MC: Ver: 2.0.1 Mar 17 2007
input: ImPS/2 Generic Wheel Mouse as /class/input/input2
libata version 2.00 loaded.
sata_sil 0000:05:0e.0: version 2.0
ACPI: PCI Interrupt 0000:05:0e.0[A] -> GSI 22 (level, low) -> IRQ 18
ata1: SATA max UDMA/100 cmd 0xF88ACC80 ctl 0xF88ACC8A bmdma 0xF88ACC00 irq 18
ata2: SATA max UDMA/100 cmd 0xF88ACCC0 ctl 0xF88ACCCA bmdma 0xF88ACC08 irq 18
scsi0 : sata_sil
USB Universal Host Controller Interface driver v3.0
ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 19
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1
uhci_hcd 0000:00:1d.0: irq 19, io base 0x0000ff80
Allocating ep_class on CPU 0
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
BUG: scheduling while atomic: modprobe/0x00000001/829
 [<c0271fd6>] __sched_text_start+0x56/0x9a2
 [<f886b3c5>] usb_control_msg+0xd1/0xdb [usbcore]
 [<c0120a3a>] lock_timer_base+0x15/0x2f
 [<c0120b44>] __mod_timer+0x92/0x9c
 [<c027307f>] schedule_timeout+0x70/0x8d
 [<c0120315>] process_timeout+0x0/0x5
 [<c0120cec>] msleep+0x1a/0x1f
 [<f8868331>] hub_probe+0x515/0x5e0 [usbcore]
 [<f886cd44>] usb_probe_interface+0x5d/0x7f [usbcore]
 [<c01eaf6a>] really_probe+0x7f/0x103
 [<c01eb032>] driver_probe_device+0x44/0xa5
 [<c0271dcd>] klist_next+0x58/0x6d
 [<c01eb093>] __device_attach+0x0/0x5
 [<c01ea553>] bus_for_each_drv+0x32/0x58
 [<c01eb0f8>] device_attach+0x50/0x63
 [<c01eb093>] __device_attach+0x0/0x5
 [<c01ea4d2>] bus_attach_device+0x1e/0x3e
 [<c01e9762>] device_add+0x336/0x491
 [<f886bacb>] usb_set_configuration+0x34c/0x3c6 [usbcore]
 [<f8871c42>] generic_probe+0x182/0x1c0 [usbcore]
 [<f886ca04>] usb_probe_device+0x32/0x37 [usbcore]
 [<c01eaf6a>] really_probe+0x7f/0x103
 [<c01eb032>] driver_probe_device+0x44/0xa5
 [<c0274706>] _spin_unlock+0xd/0x21
 [<c0271dcd>] klist_next+0x58/0x6d
 [<c01eb093>] __device_attach+0x0/0x5
 [<c01ea553>] bus_for_each_drv+0x32/0x58
 [<c01eb0f8>] device_attach+0x50/0x63
 [<c01eb093>] __device_attach+0x0/0x5
 [<c01ea4d2>] bus_attach_device+0x1e/0x3e
 [<c01e9762>] device_add+0x336/0x491
 [<f8867afc>] __usb_new_device+0xd2/0x11a [usbcore]
 [<f886c12a>] usb_get_device_descriptor+0x72/0x7c [usbcore]
 [<f8869cc6>] usb_add_hcd+0x472/0x56e [usbcore]
 [<f8872110>] usb_hcd_pci_probe+0x1ea/0x282 [usbcore]
 [<c0185732>] sysfs_create_link+0x128/0x13e
 [<c01ab569>] pci_device_probe+0x36/0x57
 [<c01eaf6a>] really_probe+0x7f/0x103
 [<c01eb10b>] __driver_attach+0x0/0x7f
 [<c01eb032>] driver_probe_device+0x44/0xa5
 [<c0274706>] _spin_unlock+0xd/0x21
 [<c01eb10b>] __driver_attach+0x0/0x7f
 [<c01eb154>] __driver_attach+0x49/0x7f
 [<c01ea618>] bus_for_each_dev+0x33/0x55
 [<c01eae24>] driver_attach+0x16/0x18
 [<c01eb10b>] __driver_attach+0x0/0x7f
 [<c01ea8e3>] bus_add_driver+0x5e/0x15f
 [<c01ab6cf>] __pci_register_driver+0x6f/0x9b
 [<f88b1078>] uhci_hcd_init+0x78/0x99 [uhci_hcd]
 [<c0131eaa>] sys_init_module+0x169f/0x17e6
 [<c02747d6>] _read_unlock_irq+0xe/0x22
 [<c0150805>] do_sync_read+0xc7/0x10a
 [<c011164f>] do_page_fault+0x277/0x510
 [<c0102bde>] sysenter_past_esp+0x5f/0x85
 =======================
Intel(R) PRO/1000 Network Driver - version 7.3.15-k2-NAPI
Copyright (c) 1999-2006 Intel Corporation.
ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 20
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1d.1: irq 20, io base 0x0000ff60
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
BUG: scheduling while atomic: modprobe/0x00000001/829
 [<c0271fd6>] __sched_text_start+0x56/0x9a2
 [<f886b3c5>] usb_control_msg+0xd1/0xdb [usbcore]
 [<c0120a3a>] lock_timer_base+0x15/0x2f
 [<c0120b44>] __mod_timer+0x92/0x9c
 [<c027307f>] schedule_timeout+0x70/0x8d
 [<c0120315>] process_timeout+0x0/0x5
 [<c0120cec>] msleep+0x1a/0x1f
 [<f8868331>] hub_probe+0x515/0x5e0 [usbcore]
 [<f886cd44>] usb_probe_interface+0x5d/0x7f [usbcore]
 [<c01eaf6a>] really_probe+0x7f/0x103
 [<c01eb032>] driver_probe_device+0x44/0xa5
 [<c0271dcd>] klist_next+0x58/0x6d
 [<c01eb093>] __device_attach+0x0/0x5
 [<c01ea553>] bus_for_each_drv+0x32/0x58
 [<c01eb0f8>] device_attach+0x50/0x63
 [<c01eb093>] __device_attach+0x0/0x5
 [<c01ea4d2>] bus_attach_device+0x1e/0x3e
 [<c01e9762>] device_add+0x336/0x491
 [<f886bacb>] usb_set_configuration+0x34c/0x3c6 [usbcore]
 [<f8871c42>] generic_probe+0x182/0x1c0 [usbcore]
 [<f886ca04>] usb_probe_device+0x32/0x37 [usbcore]
 [<c01eaf6a>] really_probe+0x7f/0x103
 [<c01eb032>] driver_probe_device+0x44/0xa5
 [<c0271dcd>] klist_next+0x58/0x6d
 [<c01eb093>] __device_attach+0x0/0x5
 [<c01ea553>] bus_for_each_drv+0x32/0x58
 [<c01eb0f8>] device_attach+0x50/0x63
 [<c01eb093>] __device_attach+0x0/0x5
 [<c01ea4d2>] bus_attach_device+0x1e/0x3e
 [<c01e9762>] device_add+0x336/0x491
 [<f8867afc>] __usb_new_device+0xd2/0x11a [usbcore]
 [<f886c12a>] usb_get_device_descriptor+0x72/0x7c [usbcore]
 [<f8869cc6>] usb_add_hcd+0x472/0x56e [usbcore]
 [<f8872110>] usb_hcd_pci_probe+0x1ea/0x282 [usbcore]
 [<c0185732>] sysfs_create_link+0x128/0x13e
 [<c01ab569>] pci_device_probe+0x36/0x57
 [<c01eaf6a>] really_probe+0x7f/0x103
 [<c01eb10b>] __driver_attach+0x0/0x7f
 [<c01eb032>] driver_probe_device+0x44/0xa5
 [<c01eb10b>] __driver_attach+0x0/0x7f
 [<c01eb154>] __driver_attach+0x49/0x7f
 [<c01ea618>] bus_for_each_dev+0x33/0x55
 [<c01eae24>] driver_attach+0x16/0x18
 [<c01eb10b>] __driver_attach+0x0/0x7f
 [<c01ea8e3>] bus_add_driver+0x5e/0x15f
 [<c01ab6cf>] __pci_register_driver+0x6f/0x9b
 [<f88b1078>] uhci_hcd_init+0x78/0x99 [uhci_hcd]
 [<c0131eaa>] sys_init_module+0x169f/0x17e6
 [<c02747d6>] _read_unlock_irq+0xe/0x22
 [<c0150805>] do_sync_read+0xc7/0x10a
 [<c011164f>] do_page_fault+0x277/0x510
 [<c0102bde>] sysenter_past_esp+0x5f/0x85
 =======================
ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 16
PCI: Setting latency timer of device 0000:00:1d.2 to 64
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1d.2: irq 16, io base 0x0000ff40
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
BUG: scheduling while atomic: modprobe/0x00000001/829
 [<c0271fd6>] __sched_text_start+0x56/0x9a2
 [<f886b3c5>] usb_control_msg+0xd1/0xdb [usbcore]
 [<c0120a3a>] lock_timer_base+0x15/0x2f
 [<c027476f>] _spin_unlock_irqrestore+0xf/0x23
 [<c0120b44>] __mod_timer+0x92/0x9c
 [<c027307f>] schedule_timeout+0x70/0x8d
 [<c0120315>] process_timeout+0x0/0x5
 [<c0120cec>] msleep+0x1a/0x1f
 [<f8868331>] hub_probe+0x515/0x5e0 [usbcore]
 [<f886cd44>] usb_probe_interface+0x5d/0x7f [usbcore]
 [<c01eaf6a>] really_probe+0x7f/0x103
 [<c01eb032>] driver_probe_device+0x44/0xa5
 [<c0274706>] _spin_unlock+0xd/0x21
 [<c0271dcd>] klist_next+0x58/0x6d
 [<c01eb093>] __device_attach+0x0/0x5
 [<c01ea553>] bus_for_each_drv+0x32/0x58
 [<c01eb0f8>] device_attach+0x50/0x63
 [<c01eb093>] __device_attach+0x0/0x5
 [<c01ea4d2>] bus_attach_device+0x1e/0x3e
 [<c01e9762>] device_add+0x336/0x491
 [<f886bacb>] usb_set_configuration+0x34c/0x3c6 [usbcore]
 [<f8871c42>] generic_probe+0x182/0x1c0 [usbcore]
 [<f886ca04>] usb_probe_device+0x32/0x37 [usbcore]
 [<c01eaf6a>] really_probe+0x7f/0x103
 [<c01eb032>] driver_probe_device+0x44/0xa5
 [<c0274706>] _spin_unlock+0xd/0x21
 [<c0271dcd>] klist_next+0x58/0x6d
 [<c01eb093>] __device_attach+0x0/0x5
 [<c01ea553>] bus_for_each_drv+0x32/0x58
 [<c01eb0f8>] device_attach+0x50/0x63
 [<c01eb093>] __device_attach+0x0/0x5
 [<c01ea4d2>] bus_attach_device+0x1e/0x3e
 [<c01e9762>] device_add+0x336/0x491
 [<f8867afc>] __usb_new_device+0xd2/0x11a [usbcore]
 [<f886c12a>] usb_get_device_descriptor+0x72/0x7c [usbcore]
 [<f8869cc6>] usb_add_hcd+0x472/0x56e [usbcore]
 [<f8872110>] usb_hcd_pci_probe+0x1ea/0x282 [usbcore]
 [<c0185732>] sysfs_create_link+0x128/0x13e
 [<c01ab569>] pci_device_probe+0x36/0x57
 [<c01eaf6a>] really_probe+0x7f/0x103
 [<c01eb10b>] __driver_attach+0x0/0x7f
 [<c01eb032>] driver_probe_device+0x44/0xa5
 [<c01eb10b>] __driver_attach+0x0/0x7f
 [<c01eb154>] __driver_attach+0x49/0x7f
 [<c01ea618>] bus_for_each_dev+0x33/0x55
 [<c01eae24>] driver_attach+0x16/0x18
 [<c01eb10b>] __driver_attach+0x0/0x7f
 [<c01ea8e3>] bus_add_driver+0x5e/0x15f
 [<c01ab6cf>] __pci_register_driver+0x6f/0x9b
 [<f88b1078>] uhci_hcd_init+0x78/0x99 [uhci_hcd]
 [<c0131eaa>] sys_init_module+0x169f/0x17e6
 [<c02747d6>] _read_unlock_irq+0xe/0x22
 [<c0150805>] do_sync_read+0xc7/0x10a
 [<f88bd1e0>] find_driver+0x3b/0xec [snd_seq_device]
 [<f88bd0f0>] init_device+0x54/0x82 [snd_seq_device]
 [<f88bd208>] find_driver+0x63/0xec [snd_seq_device]
 [<f88b9000>] ehci_hcd_init+0x0/0xf [ehci_hcd]
 [<c011164f>] do_page_fault+0x277/0x510
 [<f88bd230>] find_driver+0x8b/0xec [snd_seq_device]
 [<c0102bde>] sysenter_past_esp+0x5f/0x85
 =======================
ata1: SATA link down (SStatus 0 SControl 310)
scsi1 : sata_sil
BUG: scheduling while atomic: modprobe/0x00000001/829
 [<c0271fd6>] __sched_text_start+0x56/0x9a2
 [<c0112380>] __wake_up_common+0x32/0x55
 [<c02742e0>] __down+0xc2/0xd4
 [<c01146c3>] default_wake_function+0x0/0xc
 [<c01eb10b>] __driver_attach+0x0/0x7f
 [<c0274093>] __down_failed+0x7/0xc
 [<c01eb142>] __driver_attach+0x37/0x7f
 [<c01ea618>] bus_for_each_dev+0x33/0x55
 [<c01eae24>] driver_attach+0x16/0x18
 [<c01eb10b>] __driver_attach+0x0/0x7f
 [<c01ea8e3>] bus_add_driver+0x5e/0x15f
 [<c01ab6cf>] __pci_register_driver+0x6f/0x9b
 [<f88b1078>] uhci_hcd_init+0x78/0x99 [uhci_hcd]
 [<c0131eaa>] sys_init_module+0x169f/0x17e6
 [<c02747d6>] _read_unlock_irq+0xe/0x22
 [<c0150805>] do_sync_read+0xc7/0x10a
 [<f88bd1e0>] find_driver+0x3b/0xec [snd_seq_device]
 [<f88bd0f0>] init_device+0x54/0x82 [snd_seq_device]
 [<f88bd208>] find_driver+0x63/0xec [snd_seq_device]
 [<f88b9000>] ehci_hcd_init+0x0/0xf [ehci_hcd]
 [<c011164f>] do_page_fault+0x277/0x510
 [<f88bd230>] find_driver+0x8b/0xec [snd_seq_device]
 [<c0102bde>] sysenter_past_esp+0x5f/0x85
 =======================
usb 1-2: new full speed USB device using uhci_hcd and address 2
ieee1394: Initialized config rom entry `ip1394'
usb 1-2: configuration #1 chosen from 1 choice
ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata2.00: ATA-7, max UDMA/133, 488397168 sectors: LBA48 NCQ (depth 0/32)
ata2.00: ata2: dev 0 multi count 16
ata2.00: configured for UDMA/100
scsi 1:0:0:0: Direct-Access     ATA      HDT722525DLA380  V44O PQ: 0 ANSI: 5
note: modprobe[829] exited with preempt_count 1
ACPI: PCI Interrupt 0000:03:0e.0[A] -> GSI 24 (level, low) -> IRQ 21
ACPI: PCI Interrupt 0000:00:1d.7[D] -> GSI 23 (level, low) -> IRQ 22
PCI: Setting latency timer of device 0000:00:1d.7 to 64
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 4
ehci_hcd 0000:00:1d.7: debug port 1
PCI: cache line size of 128 is not supported by device 0000:00:1d.7
ehci_hcd 0000:00:1d.7: irq 22, io mem 0xffa20000
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 6 ports detected
usb 1-2: USB disconnect, address 2
scsi 1:0:0:0: Attached scsi generic sg0 type 0
SCSI device sda: 488397168 512-byte hdwr sectors (250059 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
SCSI device sda: 488397168 512-byte hdwr sectors (250059 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: write cache: enabled, read cache: enabled, doesn't support DPO
or FUA
 sda:<6>Linux video capture interface: v2.00
 sda1 sda2 sda3
sd 1:0:0:0: Attached scsi disk sda
e1000: 0000:03:0e.0: e1000_probe: (PCI-X:100MHz:64-bit) 00:0d:56:0d:e7:3f
pwc: Philips webcam module version 10.0.12 loaded.
pwc: Supports Philips PCA645/646, PCVC675/680/690, PCVC720[40]/730/740/750 &
PCVC830/840.
pwc: Also supports the Askey VC010, various Logitech Quickcams, Samsung MPC-C10
and MPC-C30,
pwc: the Creative WebCam 5 & Pro Ex, SOTEC Afina Eye and Visionite VCS-UC300 and
VCS-UM100.
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
ACPI: PCI Interrupt 0000:05:0d.0[A] -> GSI 21 (level, low) -> IRQ 23
Installing spdif_bug patch: Audigy 2 ZS [SB0353]
ACPI: PCI Interrupt 0000:05:0c.0[A] -> GSI 20 (level, low) -> IRQ 24
ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[24]  MMIO=[ff2ff800-ff2fffff]  Max
Packet=[2048]  IR/IT contexts=[4/8]
ACPI: PCI Interrupt 0000:05:0d.2[B] -> GSI 22 (level, low) -> IRQ 18
usbcore: registered new interface driver snd-usb-audio
usbcore: registered new interface driver Philips webcam
ohci1394: fw-host1: OHCI-1394 1.1 (PCI): IRQ=[18]  MMIO=[ff2ff000-ff2ff7ff]  Max
Packet=[2048]  IR/IT contexts=[4/8]
usb 1-2: new full speed USB device using uhci_hcd and address 3
Non-volatile memory driver v1.2
parport: PnPBIOS parport detected.
parport0: PC-style at 0x378 (0x778), irq 7, dma 1 [PCSPP,TRISTATE,COMPAT,ECP,DMA]
usb 1-2: configuration #1 chosen from 1 choice
pwc: Logitech QuickCam Zoom (new model) USB webcam detected.
pwc: Registered as /dev/video0.
lp0: using parport0 (interrupt-driven).
input: Power Button (FF) as /class/input/input3
ACPI: Power Button (FF) [PWRF]
input: Power Button (CM) as /class/input/input4
ACPI: Power Button (CM) [VBTN]
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
ieee1394: Host added: ID:BUS[0-00:1023]  GUID[87ffffffffffff00]
eth1394: eth1: IEEE-1394 IPv4 over 1394 Ethernet (fw-host0)
eth1394: eth2: IEEE-1394 IPv4 over 1394 Ethernet (fw-host1)
ieee1394: The root node is not cycle master capable; selecting a new root node
and resetting...
ieee1394: Node added: ID:BUS[1-00:1023]  GUID[0050c500001076df]
ieee1394: Host added: ID:BUS[1-01:1023]  GUID[00023c00a1037111]
ieee1394: raw1394: /dev/raw1394 device initialized
video1394: Installed video1394 module
EXT3 FS on hda2, internal journal
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda5, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-0, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda3, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on dm-1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 2040212k swap on /dev/hda7.  Priority:-1 extents:1 across:2040212k
IA-32 Microcode Update Driver: v1.14a <tigran@aivazian.fsnet.co.uk>
NET: Registered protocol family 17
e1000: eth0: e1000_watchdog: NIC Link is Up 100 Mbps Full Duplex
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
Mobile IPv6
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period
p4-clockmod: P4/Xeon(TM) CPU On-Demand Clock Modulation available
Time: acpi_pm clocksource has been installed.
[drm] Initialized drm 1.1.0 20060810
ACPI: PCI Interrupt 0000:01:00.0[A] -> GSI 16 (level, low) -> IRQ 19
[drm] Initialized radeon 1.25.0 20060524 on minor 0
agpgart: Found an AGP 3.0 compliant device at 0000:00:00.0.
agpgart: Putting AGP V3 device at 0000:00:00.0 into 8x mode
agpgart: Putting AGP V3 device at 0000:01:00.0 into 8x mode
[drm] Setting GART location based on new memory map
[drm] Loading R200 Microcode
[drm] writeback test succeeded in 1 usecs
BUG: unable to handle kernel paging request at virtual address 6b6b6b6b
 printing eip:
6b6b6b6b
*pde = 00000000
Oops: 0000 [#1]
PREEMPT SMP
last sysfs file: /class/usb_endpoint/usbdev1.3_ep84/dev
Modules linked in: snd_seq_midi radeon drm eeprom cpufreq_ondemand p4_clockmod
speedstep_lib nfsd exportfs ipv6 autofs4 nfs lockd sunrpc af_packet
firmware_class binfmt_misc video1394 raw1394 eth1394 video thermal processor fan
button ac lp parport_pc parport nvram snd_emu10k1_synth snd_emux_synth
snd_seq_virmidi snd_seq_midi_emul pwc compat_ioctl32 videodev sd_mod v4l2_common
sg v4l1_compat snd_usb_audio snd_usb_lib snd_emu10k1 snd_rawmidi snd_ac97_codec
ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq ohci1394
snd_pcm_oss ieee1394 snd_mixer_oss snd_pcm ehci_hcd snd_seq_device snd_timer
e1000 uhci_hcd sata_sil libata e7xxx_edac snd_page_alloc snd_util_mem snd_hwdep
edac_mc snd soundcore scsi_mod serio_raw psmouse intel_agp agpgart pcspkr ide_cd
i2c_i801 i2c_core cdrom usbcore ext3 jbd
CPU:    3
EIP:    0060:[<6b6b6b6b>]    Not tainted VLI
EFLAGS: 00010202   (2.6.20.3 #1)
EIP is at 0x6b6b6b6b
eax: e4c6fd8c   ebx: 6b6b6b6b   ecx: e5d80000   edx: de4df328
esi: e4c6fe14   edi: c02efa2c   ebp: e4c6fe14   esp: dea07f48
ds: 007b   es: 007b   ss: 0068
Process udevd (pid: 4056, ti=dea07000 task=dccaf560 task.ti=dea07000)
Stack: c01e8d1d f7f3b5f4 ed46f9ec c0184499 00000080 bfe281c4 c014f6ff de4df328
       f7f3b608 e5d8a7d4 bfe281c4 c0184421 00000080 c0150fed dea07fa0 e5d8a7d4
       e5d8a7d4 fffffff7 00000003 dea07000 c0151443 dea07fa0 00000000 00000000
Call Trace:
 [<c01e8d1d>] dev_attr_show+0x18/0x1b
 [<c0184499>] sysfs_read_file+0x78/0x115
 [<c014f6ff>] do_sys_open+0xb9/0xc3
 [<c0184421>] sysfs_read_file+0x0/0x115
 [<c0150fed>] vfs_read+0x88/0x134
 [<c0151443>] sys_read+0x41/0x67
 [<c0102bde>] sysenter_past_esp+0x5f/0x85
 [<c0270033>] unix_dgram_recvmsg+0x1b7/0x1c8
 =======================
Code:  Bad EIP value.
EIP: [<6b6b6b6b>] 0x6b6b6b6b SS:ESP 0068:dea07f48

And the log shows that the message appears only once, but that the bug still
happens. So you are correct - this cannot be the explanation. Those "scheduling
while atomic" messages worry me, though. Can they have been caused by my one
little printk() statement?

The second interesting point to note is that the sysfs file in question is
/class/usb_endpoint/usbdev1.3_ep84/dev. However, according to ls, this file is
not in the directory?!?!

total 0
0 drwxr-xr-x 13 root root 0 Mar 19 19:12 .
0 drwxr-xr-x 29 root root 0 Mar 19 19:05 ..
0 drwxr-xr-x  3 root root 0 Mar 19 19:05 usbdev1.1_ep00
0 drwxr-xr-x  3 root root 0 Mar 19 19:05 usbdev1.1_ep81
0 drwxr-xr-x  3 root root 0 Mar 19 19:05 usbdev1.3_ep00
0 drwxr-xr-x  3 root root 0 Mar 19 19:05 usbdev1.3_ep82
0 drwxr-xr-x  3 root root 0 Mar 19 19:05 usbdev1.3_ep85
0 drwxr-xr-x  3 root root 0 Mar 19 19:05 usbdev2.1_ep00
0 drwxr-xr-x  3 root root 0 Mar 19 19:05 usbdev2.1_ep81
0 drwxr-xr-x  3 root root 0 Mar 19 19:05 usbdev3.1_ep00
0 drwxr-xr-x  3 root root 0 Mar 19 19:05 usbdev3.1_ep81
0 drwxr-xr-x  3 root root 0 Mar 19 19:05 usbdev4.1_ep00
0 drwxr-xr-x  3 root root 0 Mar 19 19:05 usbdev4.1_ep81

I tested this again with the original usbcore.ko, with the same results:

BUG: unable to handle kernel paging request at virtual address 6b6b6b6b
 printing eip:
6b6b6b6b
*pde = 00000000
Oops: 0000 [#1]
PREEMPT SMP
last sysfs file: /class/usb_endpoint/usbdev2.2_ep84/dev
Modules linked in: snd_rtctimer snd_seq_midi radeon drm eeprom cpufreq_ondemand
p4_clockmod speedstep_lib nfsd exportfs ipv6 autofs4 nfs lockd sunrpc af_packet
firmware_class binfmt_misc video thermal processor fan button ac lp video1394
raw1394 parport_pc parport nvram eth1394 pwc compat_ioctl32 videodev v4l2_common
v4l1_compat snd_usb_audio snd_usb_lib snd_emu10k1_synth snd_emux_synth
snd_seq_virmidi snd_seq_midi_emul sd_mod sg snd_emu10k1 snd_rawmidi
snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event ohci1394
snd_seq ieee1394 snd_pcm_oss snd_mixer_oss snd_pcm uhci_hcd sata_sil libata
ehci_hcd serio_raw psmouse snd_seq_device snd_timer snd_page_alloc snd_util_mem
snd_hwdep snd scsi_mod e1000 soundcore e7xxx_edac edac_mc ide_cd pcspkr i2c_i801
i2c_core intel_agp agpgart cdrom usbcore ext3 jbd
CPU:    0
EIP:    0060:[<6b6b6b6b>]    Not tainted VLI
EFLAGS: 00010202   (2.6.20.3 #1)
EIP is at 0x6b6b6b6b
eax: ed327874   ebx: 6b6b6b6b   ecx: ecda9000   edx: f780cdfc
esi: ed3278fc   edi: c02efa2c   ebp: ed3278fc   esp: ed3f5f48
ds: 007b   es: 007b   ss: 0068
Process udevd (pid: 4137, ti=ed3f5000 task=f7f9e560 task.ti=ed3f5000)
Stack: c01e8d1d f5daef74 f5b8fa94 c0184499 00000080 bfca5794 c014f6ff f780cdfc
       f5daef88 f544542c bfca5794 c0184421 00000080 c0150fed ed3f5fa0 f544542c
       f544542c fffffff7 00000003 ed3f5000 c0151443 ed3f5fa0 00000000 00000000
Call Trace:
 [<c01e8d1d>] dev_attr_show+0x18/0x1b
 [<c0184499>] sysfs_read_file+0x78/0x115
 [<c014f6ff>] do_sys_open+0xb9/0xc3
 [<c0184421>] sysfs_read_file+0x0/0x115
 [<c0150fed>] vfs_read+0x88/0x134
 [<c0151443>] sys_read+0x41/0x67
 [<c0102bde>] sysenter_past_esp+0x5f/0x85
 [<c0270033>] unix_dgram_recvmsg+0x1b7/0x1c8
 =======================
Code:  Bad EIP value.
EIP: [<6b6b6b6b>] 0x6b6b6b6b SS:ESP 0068:ed3f5f48

total 0
0 drwxr-xr-x 13 root root 0 Mar 19 19:17 .
0 drwxr-xr-x 29 root root 0 Mar 19 19:18 ..
0 drwxr-xr-x  3 root root 0 Mar 19 19:17 usbdev1.1_ep00
0 drwxr-xr-x  3 root root 0 Mar 19 19:17 usbdev1.1_ep81
0 drwxr-xr-x  3 root root 0 Mar 19 19:17 usbdev2.1_ep00
0 drwxr-xr-x  3 root root 0 Mar 19 19:17 usbdev2.1_ep81
0 drwxr-xr-x  3 root root 0 Mar 19 19:17 usbdev2.2_ep00
0 drwxr-xr-x  3 root root 0 Mar 19 19:17 usbdev2.2_ep82
0 drwxr-xr-x  3 root root 0 Mar 19 19:17 usbdev2.2_ep85
0 drwxr-xr-x  3 root root 0 Mar 19 19:17 usbdev3.1_ep00
0 drwxr-xr-x  3 root root 0 Mar 19 19:17 usbdev3.1_ep81
0 drwxr-xr-x  3 root root 0 Mar 19 19:17 usbdev4.1_ep00
0 drwxr-xr-x  3 root root 0 Mar 19 19:17 usbdev4.1_ep81

But having said that, some of my oopsen have been on usbdev?.?_ep82 so I'm not
sure what's going wrong. It might have been usbdev?.?_ep82's turn to be missing
from /sys when those oopsen happened - I have no way of knowing. In fact, at
this stage, I'm not even sure exactly what wine is doing to trigger this!

On Mon, Mar 19, 2007 at 11:28:21AM -0700, bugme-daemon@bugzilla.kernel.org wrote:
> >> - Is creation of ep_class guaranteedly serialized versus deletion?
> >
> > That is what the kref ensures.
> 
> No. The kref does not protect
>     if (ep_class != NULL) {
>         ...
> versus
>     if (ep_class)
>         ...

But for USB devices, that does not matter.

They are serialized, both in creation and destruction.

Please do not paste in large log messages into the bug, attach the
information instead, it makes it almost impossible to determine what is
going on.

And do you have a simple test case for this?  If so, can you use
'git bisect' to try to find the problem commit?

On Mon, Mar 19, 2007 at 12:40:11PM -0700, bugme-daemon@bugzilla.kernel.org wrote:
> OK, so I added a printk() statement to endpoint.c:
> 
> --- endpoint.c.orig     2007-03-18 22:32:52.000000000 +0000
> +++ endpoint.c  2007-03-19 18:46:45.000000000 +0000
> @@ -220,6 +220,8 @@
>                 goto exit;
>         }
> 
> +       printk(KERN_INFO "Allocating ep_class on CPU %d\n", get_cpu());
> +
>         ep_class = kmalloc(sizeof(*ep_class), GFP_KERNEL);
>         if (!ep_class) {
>                 result = -ENOMEM;

This will only be called once, when the usb core is initialized.  Not on
a per-device basis at all.

So again, I don't think this class is the problem here, especially as
the files associated with it aren't used for anything yet.

No, I don't have a simple test case. I can't trigger this problem at will. If I
could, I'd never have been able to convince myself that my serialisation patch
was effective.

Anyway, I have noticed that the endpoint link that is named in the oops message
does not appear in the ls listing for /sys/class/usb_endpoint, even before the
oops happens. Could this be significant?

Endpoint addresses 82 and 85 belong to the pwc driver, while endpoint address 84
belongs to snd-usb-audio. (The pwc webcam has a built-in microphone.) Could the
pwc and snd-usb-audio modules be tripping over each other when they create links
in the /sys/class/usb_endpoint directory?

Under 2.6.19.7, the /sys/drivers/pci-etc-etc directory for snd-usb-audio doesn't
include a usbdev?.?_ep84 directory.

I:  If#= 1 Alt= 0 #EPs= 0 Cls=01(audio) Sub=01 Prot=00 Driver=snd-usb-audio
I:  If#= 2 Alt= 0 #EPs= 0 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
I:  If#= 2 Alt= 1 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
E:  Ad=84(I) Atr=01(Isoc) MxPS=  90 Ivl=1ms
I:  If#= 2 Alt= 2 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
E:  Ad=84(I) Atr=01(Isoc) MxPS=  46 Ivl=1ms
I:  If#= 2 Alt= 3 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
E:  Ad=84(I) Atr=01(Isoc) MxPS=  25 Ivl=1ms
I:  If#= 2 Alt= 4 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
E:  Ad=84(I) Atr=01(Isoc) MxPS=  16 Ivl=1ms

Under 2.6.20.3:

$ ls -als
total 0
0 drwxr-xr-x 13 root root 0 Mar 19 20:43 .
0 drwxr-xr-x 29 root root 0 Mar 19 20:32 ..
0 drwxr-xr-x  3 root root 0 Mar 19 20:32 usbdev1.1_ep00
0 drwxr-xr-x  3 root root 0 Mar 19 20:32 usbdev1.1_ep81
0 drwxr-xr-x  3 root root 0 Mar 19 20:32 usbdev1.3_ep00
0 drwxr-xr-x  3 root root 0 Mar 19 20:33 usbdev1.3_ep82
0 drwxr-xr-x  3 root root 0 Mar 19 20:32 usbdev1.3_ep85
0 drwxr-xr-x  3 root root 0 Mar 19 20:32 usbdev2.1_ep00
0 drwxr-xr-x  3 root root 0 Mar 19 20:32 usbdev2.1_ep81
0 drwxr-xr-x  3 root root 0 Mar 19 20:32 usbdev3.1_ep00
0 drwxr-xr-x  3 root root 0 Mar 19 20:32 usbdev3.1_ep81
0 drwxr-xr-x  3 root root 0 Mar 19 20:32 usbdev4.1_ep00
0 drwxr-xr-x  3 root root 0 Mar 19 20:32 usbdev4.1_ep81

Shouldn't there be a usbdev1.3_ep84 here too? And in this directory too?
ls -als 1-2\:1.1/
total 0
0 drwxr-xr-x 3 root root    0 Mar 19 20:32 .
0 drwxr-xr-x 6 root root    0 Mar 19 20:32 ..
0 -r--r--r-- 1 root root 4096 Mar 19 20:32 bAlternateSetting
0 -r--r--r-- 1 root root 4096 Mar 19 20:32 bInterfaceClass
0 -r--r--r-- 1 root root 4096 Mar 19 20:32 bInterfaceNumber
0 -r--r--r-- 1 root root 4096 Mar 19 20:32 bInterfaceProtocol
0 -r--r--r-- 1 root root 4096 Mar 19 20:32 bInterfaceSubClass
0 -r--r--r-- 1 root root 4096 Mar 19 20:32 bNumEndpoints
0 lrwxrwxrwx 1 root root    0 Mar 19 20:31 bus -> ../../../../../../bus/usb
0 lrwxrwxrwx 1 root root    0 Mar 19 20:32 driver ->
../../../../../../bus/usb/drivers/snd-usb-audio
0 -r--r--r-- 1 root root 4096 Mar 19 20:32 modalias
0 drwxr-xr-x 2 root root    0 Mar 19 20:32 power
0 lrwxrwxrwx 1 root root    0 Mar 19 20:32 sound:audio1 ->
../../../../../../class/sound/audio1
0 lrwxrwxrwx 1 root root    0 Mar 19 20:32 sound:controlC1 ->
../../../../../../class/sound/controlC1
0 lrwxrwxrwx 1 root root    0 Mar 19 20:32 sound:dsp1 ->
../../../../../../class/sound/dsp1
0 lrwxrwxrwx 1 root root    0 Mar 19 20:32 sound:mixer1 ->
../../../../../../class/sound/mixer1
0 lrwxrwxrwx 1 root root    0 Mar 19 20:32 sound:pcmC1D0c ->
../../../../../../class/sound/pcmC1D0c 0 lrwxrwxrwx 1 root root    0 Mar 19
20:32 subsystem -> ../../../../../../bus/usb
0 --w------- 1 root root 4096 Mar 19 20:32 uevent

/proc/bus/usb/devices shows us that ep84 exists, so where has it gone? When this
kernel eventually BUGs, I'm betting that it will be complain about ep84...

And I'm right:
BUG: unable to handle kernel paging request at virtual address 6b6b6b6b
 printing eip:
6b6b6b6b
*pde = 00000000
Oops: 0000 [#1]
PREEMPT SMP
last sysfs file: /class/usb_endpoint/usbdev1.3_ep84/dev
...

> So again, I don't think this class is the problem here, especially as
> the files associated with it aren't used for anything yet.

It's your patch that's saying otherwise:
gregkh-driver-sysfs-crash-debugging.patch

Why could your patch be claiming that the last sysfs file is sometimes:

/class/usb_endpoint/usbdev1.3_ep84/dev

considering that ep84 apparently only exists when If#2 has bAlternateSetting > 0?

I:  If#= 2 Alt= 0 #EPs= 0 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
I:  If#= 2 Alt= 1 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
E:  Ad=84(I) Atr=01(Isoc) MxPS=  90 Ivl=1ms
I:  If#= 2 Alt= 2 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
E:  Ad=84(I) Atr=01(Isoc) MxPS=  46 Ivl=1ms
I:  If#= 2 Alt= 3 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
E:  Ad=84(I) Atr=01(Isoc) MxPS=  25 Ivl=1ms
I:  If#= 2 Alt= 4 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
E:  Ad=84(I) Atr=01(Isoc) MxPS=  16 Ivl=1ms

On Mon, Mar 19, 2007 at 06:11:25PM -0700, bugme-daemon@bugzilla.kernel.org wrote:
> > So again, I don't think this class is the problem here, especially as
> > the files associated with it aren't used for anything yet.
> 
> It's your patch that's saying otherwise:
> gregkh-driver-sysfs-crash-debugging.patch
> 
> Why could your patch be claiming that the last sysfs file is sometimes:
> 
> /class/usb_endpoint/usbdev1.3_ep84/dev

That merely shows the last sysfs file that was opened, it does not
pinpoint the location of any crash.  It is there as a hint for
debugging, there is a good reason it is not in the mainline kernel tree
:)

And udev is the program that opened that file up, and then closed it.
Does your oops show that by reading this file the crash happens?

> considering that ep84 apparently only exists when If#2 has bAlternateSetting
> > 0?
> 
> I:  If#= 2 Alt= 0 #EPs= 0 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
> I:  If#= 2 Alt= 1 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
> E:  Ad=84(I) Atr=01(Isoc) MxPS=  90 Ivl=1ms
> I:  If#= 2 Alt= 2 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
> E:  Ad=84(I) Atr=01(Isoc) MxPS=  46 Ivl=1ms
> I:  If#= 2 Alt= 3 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
> E:  Ad=84(I) Atr=01(Isoc) MxPS=  25 Ivl=1ms
> I:  If#= 2 Alt= 4 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
> E:  Ad=84(I) Atr=01(Isoc) MxPS=  16 Ivl=1ms

The pwc driver is not bound to this device, this is a sound device, right?

>> /class/usb_endpoint/usbdev1.3_ep84/dev

> That merely shows the last sysfs file that was opened, it does not
> pinpoint the location of any crash.

True, except:
a) That file does not exist within the /sys/class/usb_endpoint directory
structure, so how is udevd opening it? The ep84 endpoint is listed here as being
 on If#2, with Alt > 0. The class entry in /sys for If#2 has bAlternateSetting = 0.

> I:  If#= 2 Alt= 0 #EPs= 0 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
> I:  If#= 2 Alt= 1 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
> E:  Ad=84(I) Atr=01(Isoc) MxPS=  90 Ivl=1ms
> I:  If#= 2 Alt= 2 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
> E:  Ad=84(I) Atr=01(Isoc) MxPS=  46 Ivl=1ms
> I:  If#= 2 Alt= 3 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
> E:  Ad=84(I) Atr=01(Isoc) MxPS=  25 Ivl=1ms
> I:  If#= 2 Alt= 4 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
> E:  Ad=84(I) Atr=01(Isoc) MxPS=  16 Ivl=1ms

> And udev is the program that opened that file up, and then closed it.

The kernel is crashing reading a sysfs file, yes. It's not too unreasonable to
think that sysfs file might be the last one opened, particularly when that sysfs
file does not exist.

> Does your oops show that by reading this file the crash happens?

One prime example (I have posted several):

last sysfs file: /class/usb_endpoint/usbdev1.3_ep84/dev
Modules linked in: snd_seq_midi radeon drm eeprom cpufreq_ondemand p4_clockmod
speedstep_lib nfsd exportfs ipv6 autofs4 nfs lockd sunrpc af_packet
firmware_class binfmt_misc video1394 raw1394 eth1394 video thermal processor fan
button ac lp parport_pc parport nvram snd_emu10k1_synth snd_emux_synth
snd_seq_virmidi snd_seq_midi_emul pwc compat_ioctl32 videodev sd_mod v4l2_common
sg v4l1_compat snd_usb_audio snd_usb_lib snd_emu10k1 snd_rawmidi snd_ac97_codec
ac97_bus snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq ohci1394
snd_pcm_oss ieee1394 snd_mixer_oss snd_pcm ehci_hcd snd_seq_device snd_timer
e1000 uhci_hcd sata_sil libata e7xxx_edac snd_page_alloc snd_util_mem snd_hwdep
edac_mc snd soundcore scsi_mod serio_raw psmouse intel_agp agpgart pcspkr ide_cd
i2c_i801 i2c_core cdrom usbcore ext3 jbd
CPU:    3
EIP:    0060:[<6b6b6b6b>]    Not tainted VLI
EFLAGS: 00010202   (2.6.20.3 #1)
EIP is at 0x6b6b6b6b
eax: e4c6fd8c   ebx: 6b6b6b6b   ecx: e5d80000   edx: de4df328
esi: e4c6fe14   edi: c02efa2c   ebp: e4c6fe14   esp: dea07f48
ds: 007b   es: 007b   ss: 0068
Process udevd (pid: 4056, ti=dea07000 task=dccaf560 task.ti=dea07000)
Stack: c01e8d1d f7f3b5f4 ed46f9ec c0184499 00000080 bfe281c4 c014f6ff de4df328
       f7f3b608 e5d8a7d4 bfe281c4 c0184421 00000080 c0150fed dea07fa0 e5d8a7d4
       e5d8a7d4 fffffff7 00000003 dea07000 c0151443 dea07fa0 00000000 00000000
Call Trace:
 [<c01e8d1d>] dev_attr_show+0x18/0x1b
 [<c0184499>] sysfs_read_file+0x78/0x115
 [<c014f6ff>] do_sys_open+0xb9/0xc3
 [<c0184421>] sysfs_read_file+0x0/0x115
 [<c0150fed>] vfs_read+0x88/0x134
 [<c0151443>] sys_read+0x41/0x67
 [<c0102bde>] sysenter_past_esp+0x5f/0x85
 [<c0270033>] unix_dgram_recvmsg+0x1b7/0x1c8
 =======================

> The pwc driver is not bound to this device, this is a sound device, right?

Correct - this webcam has a built in microphone, so If#0 and If#1 are bound by
the pwc driver, and If#2 is bound by snd-usb-audio.

>> /class/usb_endpoint/usbdev1.3_ep84/dev

When I say that this file does not exist, I mean that the entire directory
/class/usb_endpoint/usbdev1.3_ep84/ does not exist, although the corresponding
ep82 and ep85 ones do.

Another interesting observation: under 2.6.19.x,
/class/usb_endpoint/usbdev1.3_ep84 is a symlink to something in the
/sys/devices/ directory structure. In 2.6.20.x, this symlink has become a
directory and the symlink in in /sys/devices/... etc instead.

(Grr, it's been a rough morning, and I'm being careless with cut-and-paste)

To clarify:
> Another interesting observation: under 2.6.19.x,
> /class/usb_endpoint/usbdev1.3_ep84 is a symlink to something in the ...

There is no symlink "usbdev1.3_ep84" in 2.6.19.x; I am merely trying to say that
the /class/usb_endpoint/ directory contains symlinks under 2.6.19.x, and
directories under 2.6.20.x.

Created attachment 10936 [details]
dmesg output from 2.6.20.4 oops

The bug still exists in 2.6.20.4. This time, I removed the firmware_class
module by hand as well, just in case it was leaving anything in /sys that it
shouldn't after the init scripts had loaded and  unloaded the microcode module.


One other point: I ran "winecfg" so many times that I thought the bug wasn't
going to happen... until I entered the "Audio" tab. Not that entering the
"Audio" tab is always sufficient, but in this particular case the kernel oopsed
immediately afterwards. My "winecfg" is set to use "ALSA" for audio. On
2.6.19.7, it also triggers the loading of the snd_seq_midi and snd_rtctimer
modules.

Created attachment 10937 [details]
config file for 2.6.20.4 kernel

This may or may not be useful...

Created attachment 11144 [details]
dmesg output from 2.6.20.7 oops

I don't know if this output is any more useful than any of the others. And I
can't see where the "single bit error" comes from when the output is full of
"use after free" poison. However, this much at least is different:

EIP: [<6b6b6b6b>] 0x6b6b6b6b SS:ESP 0068:ec9eaf48
 <3>Slab corruption: start=eceea92c, len=512
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c0226642>](kfree_skbmem+0xb/0x9c)
080: 6b 6b 6b 6b 6a 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
Single bit error detected. Probably bad RAM.
Run memtest86+ or a similar memory test tool.
Prev obj: start=eceea720, len=512
Redzone: 0x170fc2a5/0x170fc2a5.
Last user: [<f886eda5>](usb_create_ep_files+0xf1/0x2c2 [usbcore])
000: f8 fd 1b c2 e0 ee 4e f7 01 00 00 00 00 00 00 00
010: 30 a7 ee ec 30 a7 ee ec f4 91 1e c0 96 90 1e c0
Next obj: start=eceeab38, len=512
Redzone: 0x5a2cf071/0x5a2cf071.
Last user: [<c0226642>](kfree_skbmem+0xb/0x9c)
000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b

Again, it's claiming that the last sysfs file accessed was the non-existent
/class/usb_endpoint/usbdev1.3_ep84/dev, which I simply cannot explain because
this endpoint belongs to If# = 2, Alt > 0:

I:  If#= 2 Alt= 0 #EPs= 0 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
I:  If#= 2 Alt= 1 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
E:  Ad=84(I) Atr=01(Isoc) MxPS=  90 Ivl=1ms
I:  If#= 2 Alt= 2 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
E:  Ad=84(I) Atr=01(Isoc) MxPS=  46 Ivl=1ms
I:  If#= 2 Alt= 3 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
E:  Ad=84(I) Atr=01(Isoc) MxPS=  25 Ivl=1ms
I:  If#= 2 Alt= 4 #EPs= 1 Cls=01(audio) Sub=02 Prot=00 Driver=snd-usb-audio
E:  Ad=84(I) Atr=01(Isoc) MxPS=  16 Ivl=1ms

Could the ALSA USB audio driver be changing the "Alt" on the fly somehow,
causing usbcore to need to create the extra endpoint in /sys?

Memtest has just succesfully checked out all 2 GB of memory.

I am seeing this same issue (yep, a "me too").  I upgraded a motherboard and
kernel about 3 months ago (at the same time) and so do not have real good
information on timeframe or kernel version that caused this to start showing up
for me.  It did only show up for me at about 3 months ago though.  I've upgrade
from FC6 to F7test3 in a attempt to solve the issue but no luck.

The main things I see are the "BUG: unable to handle kernel paging request at
virtual address 5a5a5a5a5a" message printed to all my terminal windows about
once every 4 hours.  I've seen the address change form time to time... Examples
include "5a5a0231", "5a0063763", and "657474f9".

In my case, the printout always says "last sysfs file:
/class/usb_endpoint/usbdev2.4_ep81/dev".

If I change into /sysclass/usb_endpoint/usbdev2.4_ep81" then I see no files,
strangely enough, from an "ls".  If I instead change into
"/sysclass/usb_endpoint" and then use something like "mc" to change into
"usbdev2.4_ep81" then I can see the files.

If I next view the file "usbdev2.4_ep81/dev" with a specific program then I can
immediately cause the same crash in udevd.  At that point, something seems to
re-create the directory and files with a new timestamp.

The program that causes the crash to always occur is the directory browser I
wrote and is at http://clist.sf.net .  Basically, it does a "less" on the file
and then does a stat() call afterwords on the directory to see if anything
changed.  "mc" doesn't seem to be able to view the "dev" file and so doesn't
reproduce the crash.  I would think any program that views and then stat()'s
would crash though.

Since its so easy for me to reproduce the crash, I'd be happy to do some more
debugging but I could really us some pointers on what to try.

Oooerr! Reproducible on demand! Hooray! Do you have access to any 2.6.19.x
kernels please, just to confirm that this bug was introduced in 2.6.20.x?

Also, what is this USB device that has the dodgy endpoint?

And is your machine SMP? Hyperthreaded? Intel/AMD?

I'll see if I can dig up an older kernel and reproduce it.  The "easy" way would
be to get a older Fedora RPM but they are patched so heavily probably it will be
hard to correlate.

I have an Intel P4 3GHz that shows up as 2 CPU's in /proc/cpuinfo.

I am having a hard time reading the output of either "lsusb -v" or
/proc/bus/usb/devices.  So I'll just list my whole output in hope it helps.  I
seem "81" multiple times... All 81's seem to be "hubs" but I can't tell which (I
have 1 or 2 internal to motherboard and 1 external).

T:  Bus=05 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#=  1 Spd=480 MxCh= 8
B:  Alloc=  0/800 us ( 0%), #Int=  1, #Iso=  0
D:  Ver= 2.00 Cls=09(hub  ) Sub=00 Prot=01 MxPS=64 #Cfgs=  1
P:  Vendor=0000 ProdID=0000 Rev= 2.06
S:  Manufacturer=Linux 2.6.20-1.3094.fc7 ehci_hcd
S:  Product=EHCI Host Controller
S:  SerialNumber=0000:00:1d.7
C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=  0mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=09(hub  ) Sub=00 Prot=00 Driver=hub
E:  Ad=81(I) Atr=03(Int.) MxPS=   4 Ivl=256ms

T:  Bus=05 Lev=01 Prnt=01 Port=01 Cnt=01 Dev#=  2 Spd=480 MxCh= 4
D:  Ver= 2.00 Cls=09(hub  ) Sub=00 Prot=01 MxPS=64 #Cfgs=  1
P:  Vendor=0409 ProdID=0059 Rev= 1.00
C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=100mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=09(hub  ) Sub=00 Prot=00 Driver=hub
E:  Ad=81(I) Atr=03(Int.) MxPS=   1 Ivl=256ms

T:  Bus=04 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#=  1 Spd=12  MxCh= 2
B:  Alloc=  0/900 us ( 0%), #Int=  0, #Iso=  0
D:  Ver= 1.10 Cls=09(hub  ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=0000 ProdID=0000 Rev= 2.06
S:  Manufacturer=Linux 2.6.20-1.3094.fc7 uhci_hcd
S:  Product=UHCI Host Controller
S:  SerialNumber=0000:00:1d.3
C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=  0mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=09(hub  ) Sub=00 Prot=00 Driver=hub
E:  Ad=81(I) Atr=03(Int.) MxPS=   2 Ivl=255ms

T:  Bus=04 Lev=01 Prnt=01 Port=00 Cnt=01 Dev#=  2 Spd=12  MxCh= 0
D:  Ver= 1.10 Cls=00(>ifc ) Sub=00 Prot=00 MxPS= 8 #Cfgs=  1
P:  Vendor=058f ProdID=9360 Rev= 1.00
S:  Manufacturer= 
S:  Product=USB Reader
S:  SerialNumber=2004888
C:* #Ifs= 1 Cfg#= 1 Atr=80 MxPwr=100mA
I:* If#= 0 Alt= 0 #EPs= 2 Cls=08(stor.) Sub=06 Prot=50 Driver=(none)
E:  Ad=01(O) Atr=02(Bulk) MxPS=  64 Ivl=0ms
E:  Ad=82(I) Atr=02(Bulk) MxPS=  64 Ivl=0ms

T:  Bus=03 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#=  1 Spd=12  MxCh= 2
B:  Alloc=  0/900 us ( 0%), #Int=  0, #Iso=  0
D:  Ver= 1.10 Cls=09(hub  ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=0000 ProdID=0000 Rev= 2.06
S:  Manufacturer=Linux 2.6.20-1.3094.fc7 uhci_hcd
S:  Product=UHCI Host Controller
S:  SerialNumber=0000:00:1d.2
C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=  0mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=09(hub  ) Sub=00 Prot=00 Driver=hub
E:  Ad=81(I) Atr=03(Int.) MxPS=   2 Ivl=255ms

T:  Bus=02 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#=  1 Spd=12  MxCh= 2
B:  Alloc=  0/900 us ( 0%), #Int=  1, #Iso=  0
D:  Ver= 1.10 Cls=09(hub  ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=0000 ProdID=0000 Rev= 2.06
S:  Manufacturer=Linux 2.6.20-1.3094.fc7 uhci_hcd
S:  Product=UHCI Host Controller
S:  SerialNumber=0000:00:1d.1
C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=  0mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=09(hub  ) Sub=00 Prot=00 Driver=hub
E:  Ad=81(I) Atr=03(Int.) MxPS=   2 Ivl=255ms

T:  Bus=02 Lev=01 Prnt=01 Port=00 Cnt=01 Dev#=  2 Spd=12  MxCh= 4
D:  Ver= 1.00 Cls=09(hub  ) Sub=00 Prot=00 MxPS= 8 #Cfgs=  1
P:  Vendor=0451 ProdID=1446 Rev= 1.00
C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=100mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=09(hub  ) Sub=00 Prot=00 Driver=hub
E:  Ad=81(I) Atr=03(Int.) MxPS=   1 Ivl=255ms

T:  Bus=02 Lev=02 Prnt=02 Port=00 Cnt=01 Dev#=  3 Spd=12  MxCh= 0
D:  Ver= 2.00 Cls=00(>ifc ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=04b8 ProdID=0803 Rev= 1.00
S:  Manufacturer=EPSON
S:  Product=USB2.0 MFP
S:  SerialNumber=M15P10404020840570
C:* #Ifs= 2 Cfg#= 1 Atr=c0 MxPwr=  2mA
I:* If#= 0 Alt= 0 #EPs= 2 Cls=08(stor.) Sub=06 Prot=50 Driver=(none)
E:  Ad=01(O) Atr=02(Bulk) MxPS=  64 Ivl=0ms
E:  Ad=82(I) Atr=02(Bulk) MxPS=  64 Ivl=0ms
I:* If#= 1 Alt= 0 #EPs= 2 Cls=07(print) Sub=01 Prot=02 Driver=usblp
E:  Ad=04(O) Atr=02(Bulk) MxPS=  64 Ivl=0ms
E:  Ad=85(I) Atr=02(Bulk) MxPS=  64 Ivl=0ms

T:  Bus=02 Lev=02 Prnt=02 Port=03 Cnt=02 Dev#=  4 Spd=12  MxCh= 0
D:  Ver= 1.10 Cls=ff(vend.) Sub=00 Prot=ff MxPS= 8 #Cfgs=  1
P:  Vendor=04a9 ProdID=2220 Rev= 1.00
S:  Manufacturer=Canon
S:  Product=CanoScan
C:* #Ifs= 1 Cfg#= 1 Atr=a0 MxPwr=500mA
I:* If#= 0 Alt= 0 #EPs= 3 Cls=ff(vend.) Sub=00 Prot=ff Driver=(none)
E:  Ad=81(I) Atr=03(Int.) MxPS=   1 Ivl=16ms
E:  Ad=82(I) Atr=02(Bulk) MxPS=  64 Ivl=0ms
E:  Ad=03(O) Atr=02(Bulk) MxPS=  64 Ivl=0ms

T:  Bus=01 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#=  1 Spd=12  MxCh= 2
B:  Alloc=  0/900 us ( 0%), #Int=  0, #Iso=  0
D:  Ver= 1.10 Cls=09(hub  ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=0000 ProdID=0000 Rev= 2.06
S:  Manufacturer=Linux 2.6.20-1.3094.fc7 uhci_hcd
S:  Product=UHCI Host Controller
S:  SerialNumber=0000:00:1d.0
C:* #Ifs= 1 Cfg#= 1 Atr=e0 MxPwr=  0mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=09(hub  ) Sub=00 Prot=00 Driver=hub
E:  Ad=81(I) Atr=03(Int.) MxPS=   2 Ivl=255ms

If it's always "usbdev2.4_ep81/dev" then that's endpoint 81 on bus 2, device 4:

T:  Bus=02 Lev=02 Prnt=02 Port=03 Cnt=02 Dev#=  4 Spd=12  MxCh= 0
D:  Ver= 1.10 Cls=ff(vend.) Sub=00 Prot=ff MxPS= 8 #Cfgs=  1
P:  Vendor=04a9 ProdID=2220 Rev= 1.00
S:  Manufacturer=Canon
S:  Product=CanoScan
C:* #Ifs= 1 Cfg#= 1 Atr=a0 MxPwr=500mA
I:* If#= 0 Alt= 0 #EPs= 3 Cls=ff(vend.) Sub=00 Prot=ff Driver=(none)
E:  Ad=81(I) Atr=03(Int.) MxPS=   1 Ivl=16ms
E:  Ad=82(I) Atr=02(Bulk) MxPS=  64 Ivl=0ms
E:  Ad=03(O) Atr=02(Bulk) MxPS=  64 Ivl=0ms

So it's a completely different USB device to mine.

Ahhh, thanks for helping decode that.

Yes, its always "usbdev2.4_ep81/dev". Now that rings a bell since its my
scanner.  I installed "scanbuttond" package relatively recently.  I didn't
correlate the two events (installing deamon and seeing udevd problems) but maybe
they are related.

So it could be related to this program scanning for button presses while udevd
is accessing the "usbdev2.4_ep81/dev".

I only know that scanbuttond makes use of libusb... I do not know if its
constantly polling the scanner or if its sent events when a button is pressed.

What we really need is to narrow down the "bug-triggering" event to something
really simple. Can you slice up your clist program to create the simplest
possible executable that will cause the oops?

When you trigger the oops using clist, does the oops report still say that udevd
is responsible? I'm hoping not - we might be able to patch this problem if we
can identify the buggy sysfs operations.

I notice that you have a P4 SMP machine too. Is this a single physical CPU with
hyperthreading enabled?

I cannot trigger this bug on my machine using clist. I can browse the
/sys/class/usb_endpoint directory and see exactly the same files as using ls.
The bug is still present though, as enough bashing with winecfg eventually shows:

BUG: unable to handle kernel paging request at virtual address 6b6b6deb
 printing eip:
c013066f
*pde = 00000000
Oops: 0002 [#1]
PREEMPT SMP
last sysfs file: /class/usb_endpoint/usbdev1.3_ep84/dev
Modules linked in: snd_rtctimer snd_seq_midi zd1211rw ieee80211softmac ieee80211
ieee80211_crypt radeon drm eeprom cpufreq_ondemand p4_clockmod speedstep_lib
nfsd exportfs ipv6 autofs4 nfs lockd sunrpc af_packet firmware_class binfmt_misc
video1394 raw1394 eth1394 video pwc thermal compat_ioctl32 processor videodev
v4l2_common fan v4l1_compat button ac lp parport_pc parport nvram snd_usb_audio
snd_usb_lib snd_emu10k1_synth snd_emux_synth snd_seq_virmidi snd_seq_midi_emul
sd_mod sg snd_emu10k1 snd_rawmidi snd_ac97_codec ac97_bus snd_seq_dummy snd_seq_oss
snd_seq_midi_event ohci1394 snd_seq ieee1394 snd_pcm_oss snd_mixer_oss snd_pcm
ehci_hcd uhci_hcd e1000 sata_sil snd_seq_device snd_timer e7xxx_edac serio_raw
libata snd_page_alloc snd_util_mem snd_hwdep edac_mc psmouse snd soundcore
scsi_mod intel_agp i2c_i801 ide_cd agpgart pcspkr cdrom i2c_core usbcore ext3 jbd
CPU:    2
EIP:    0060:[<c013066f>]    Not tainted VLI
EFLAGS: 00010206   (2.6.20.10 #1)
EIP is at module_put+0x20/0x52
eax: 6b6b6deb   ebx: 6b6b6b6b   ecx: 00000001   edx: ec8d3000
esi: ed647d14   edi: 6b6b6b6b   ebp: eca6b62c   esp: ec8d3f58
ds: 007b   es: 007b   ss: 0068
Process udevd (pid: 5455, ti=ec8d3000 task=f7b42a90 task.ti=ec8d3000)
Stack: ed57b4c4 c01845e7 00000010 ed054b7c f5977978 c0151837 00000000 00000000
       f7831730 ed054b7c f75b5ca8 00000000 ed054b7c c014f358 00000003 00000003
       f75b5ca8 f75b5d28 c0150351 00000003 00000007 00000003 ec8d3000 c0102bde
Call Trace:
 [<c01845e7>] sysfs_release+0x2d/0x4c
 [<c0151837>] __fput+0x96/0x13c
 [<c014f358>] filp_close+0x51/0x58
 [<c0150351>] sys_close+0x70/0xa7
 [<c0102bde>] sysenter_past_esp+0x5f/0x85
 [<c0270033>] unix_create+0x14/0x63
 =======================
Code: 00 89 f0 83 c4 0c 5b 5e 5f 5d c3 53 89 c3 85 c0 74 49 b8 01 00 00 00 e8 0f
44 fe ff e8 7f 64 07 00 c1 e0 07 8d 84 18 80 01 00 00 <ff> 08 83 3b 02 75 0b 8b
83 88 05 00 00 e8 59 40 fe ff b8 01 00
EIP: [<c013066f>] module_put+0x20/0x52 SS:ESP 0068:ec8d3f58
 <6>note: udevd[5455] exited with preempt_count 1

Are you seeing this problem with latest kernels?

The address patters that you see is for the slab cache allocation debug:
/* ...and for poisoning */
#define POISON_INUSE    0x5a    /* for use-uninitialised poisoning */
#define POISON_FREE     0x6b    /* for use-after-free poisoning */
#define POISON_END      0xa5    /* end-byte of poisoning */

Maybe it is worthwhile enabling memory allocation debugging in the kernel, and locking debug as well.

Which kernel version is known to have the fix?

2.6.21.2, specifically this patch:

Tejun Heo (3):
...
driver-core: don't free devt_attr till the device is released
...