Bug 10868

Summary: Oops on loading ipaq module since 2.6.26, prevents use of device
Product: Drivers Reporter: Adam Williamson (adamw)
Component: USBAssignee: Greg Kroah-Hartman (greg)
Status: CLOSED CODE_FIX    
Severity: normal CC: ahnert, bunk, kernel-f
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 2.6.26 Subsystem:
Regression: Yes Bisected commit-id:
Bug Depends on:    
Bug Blocks: 10492    
Attachments: The oops (and surrounding, relevant info) from /var/log/messages

Description Adam Williamson 2008-06-05 17:39:21 UTC
Latest working kernel version: 2.6.25.4
Earliest failing kernel version: 2.6.26rc4
Distribution: Mandriva Linux
Hardware Environment: i586
Software Environment:
Problem Description:

When testing Windows Mobile 2003 device synchronization, I noticed it was working fine on one system but not another. I then noticed the working system was running 2.6.24 and the broken was running 2.6.26, so went to narrow down the issue.

After testing, I see that when I plug in the device under 2.6.26rc4, a kernel oops shows up in the logs. lshal output is rather different, synce-trayicon doesn't see the device, and synce-matchmaker complains about not being able to find a HAL property (it's looking at the wrong HAL device). Booting a 2.6.24 kernel, everything is fine. I then tried kernel 2.6.25.4, and again, everything is fine: no oops, synce-trayicon sees the device, so does synce-matchmaker.

This is with stock kernel.org kernel (packaged in Mandriva as kernel-linus), not Mandriva's normal distro-patched kernel. So it's definitely your bug. :)

I will attach the oops from /var/log/messages . I don't know what else might be useful - please advise and I'll try to help. I see only four changes to ipaq.c since October 2007, so hopefully it should be easy to pin this down.

Steps to reproduce:

Plug in an iPaq 1715 - USB ID 03f0:1016 - with kernel 2.6.26.
Comment 1 Adam Williamson 2008-06-05 17:41:43 UTC
Created attachment 16409 [details]
The oops (and surrounding, relevant info) from /var/log/messages
Comment 2 Anonymous Emailer 2008-06-05 18:28:31 UTC
Reply-To: akpm@linux-foundation.org


(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Thu,  5 Jun 2008 17:39:21 -0700 (PDT) bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=10868
> 
>            Summary: Oops on loading ipaq module since 2.6.26, prevents use
>                     of device
>            Product: Drivers
>            Version: 2.5
>      KernelVersion: 2.6.26
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: USB
>         AssignedTo: greg@kroah.com
>         ReportedBy: awilliamson@mandriva.com
> 
> 
> Latest working kernel version: 2.6.25.4
> Earliest failing kernel version: 2.6.26rc4

I'll ask Rafael to add this to the post-2.6.25 regression list.

> Distribution: Mandriva Linux
> Hardware Environment: i586
> Software Environment:
> Problem Description:
> 
> When testing Windows Mobile 2003 device synchronization, I noticed it was
> working fine on one system but not another. I then noticed the working system
> was running 2.6.24 and the broken was running 2.6.26, so went to narrow down
> the issue.
> 
> After testing, I see that when I plug in the device under 2.6.26rc4, a kernel
> oops shows up in the logs. lshal output is rather different, synce-trayicon
> doesn't see the device, and synce-matchmaker complains about not being able
> to
> find a HAL property (it's looking at the wrong HAL device). Booting a 2.6.24
> kernel, everything is fine. I then tried kernel 2.6.25.4, and again,
> everything
> is fine: no oops, synce-trayicon sees the device, so does synce-matchmaker.
> 
> This is with stock kernel.org kernel (packaged in Mandriva as kernel-linus),
> not Mandriva's normal distro-patched kernel. So it's definitely your bug. :)
> 
> I will attach the oops from /var/log/messages . I don't know what else might
> be
> useful - please advise and I'll try to help. I see only four changes to
> ipaq.c
> since October 2007, so hopefully it should be easy to pin this down.
> 
> Steps to reproduce:
> 
> Plug in an iPaq 1715 - USB ID 03f0:1016 - with kernel 2.6.26.

> usb 3-3: new full speed USB device using ohci_hcd and address 2
> usb 3-3: configuration #1 chosen from 1 choice
> usb 3-3: New USB device found, idVendor=03f0, idProduct=1016
> usb 3-3: New USB device strings: Mfr=0, Product=0, SerialNumber=0
> usbcore: registered new interface driver usbserial
> usbserial: USB Serial support registered for generic
> usbcore: registered new interface driver usbserial_generic
> usbserial: USB Serial Driver core
> usbserial: USB Serial support registered for PocketPC PDA
> ipaq: USB PocketPC PDA driver v0.5
> ipaq 3-3:1.0: PocketPC PDA converter detected
> usb 3-3: PocketPC PDA converter now attached to ttyUSB0
> usb 3-3: PocketPC PDA converter now attached to ttyUSB1
> usbcore: registered new interface driver ipaq
> PPP generic driver version 2.4.2
> BUG: unable to handle kernel NULL pointer dereference at 0000003c
> IP: [<e0eec8bd>] :ipaq:ipaq_open+0x1c4/0x2fb
> *pde = 00000000
> Oops: 0002 [#1]
> Modules linked in: ppp_generic slhc ipaq usbserial nvidia(P) fuse af_packet
> snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_pcm_oss
> snd_mixer_oss ipv6 binfmt_misc loop reiserfs dm_mirror dm_log dm_mod floppy
> cpufreq_ondemand freq_table cpufreq_conservative cpufreq_powersave
> cpufreq_nforce2 evdev snd_mpu401 snd_cs4232 snd_opl3_lib snd_hwdep
> snd_cs4231_lib snd_mpu401_uart snd_rawmidi parport_pc parport snd_seq_device
> ns558 gameport rtc_cmos rtc_core rtc_lib snd_pcsp sg skge usbkbd arc4 ecb
> crypto_blkcipher osst st thermal processor button shpchp pci_hotplug usbhid
> usbmouse ff_memless snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm snd_timer
> zd1211rw firmware_class snd soundcore snd_page_alloc nvidia_agp i2c_nforce2
> agpgart mac80211 i2c_core cfg80211 forcedeth hid ide_generic amd74xx ide_core
> sbp2 ohci1394 ieee1394 usb_storage sata_sil pata_amd libata dock sd_mod
> scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd usbcore [last unloaded:
> nf_conntrack]
> 
> Pid: 7409, comm: pppd Tainted: P          (2.6.26-0.rc4.1mdv #1)

The nvidia driver is loaded, but it's unlikely to have caused this.

> EIP: 0060:[<e0eec8bd>] EFLAGS: 00010246 CPU: 0
> EIP is at ipaq_open+0x1c4/0x2fb [ipaq]
> EAX: 00000000 EBX: cf552f40 ECX: c016e158 EDX: dda79000
> ESI: 00000000 EDI: dda51000 EBP: cf66fe3c ESP: cf66fe1c
>  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
> Process pppd (pid: 7409, ti=cf66e000 task=cf658280 task.ti=cf66e000)
> Stack: cf552f54 df01d740 00000100 00000064 dda51000 00000000 dda51000
> df01d740
>        cf66fe5c e0ec8cd6 cf575480 df96a000 dda51024 ffffffed e0ec8bf7
>        cf575480
>        cf66fe84 c022549d 00000802 00000000 0bc00001 00000001 df96a000
>        00000000
> Call Trace:
>  [<e0ec8cd6>] ? serial_open+0xdf/0x13d [usbserial]
>  [<e0ec8bf7>] ? serial_open+0x0/0x13d [usbserial]
>  [<c022549d>] ? tty_open+0x184/0x272
>  [<c0173364>] ? chrdev_open+0x130/0x147
>  [<c016f967>] ? __dentry_open+0x103/0x1f0
>  [<c016fadb>] ? nameidata_to_filp+0x1f/0x33
>  [<c0173234>] ? chrdev_open+0x0/0x147
>  [<c017a8ba>] ? do_filp_open+0x338/0x6b1
>  [<c016f733>] ? get_unused_fd_flags+0xc1/0xcb
>  [<c016f77d>] ? do_sys_open+0x40/0xbc
>  [<c016f5c5>] ? filp_close+0x50/0x5a
>  [<c016f83b>] ? sys_open+0x1e/0x26
>  [<c010392f>] ? sysenter_past_esp+0x78/0xd9
>  =======================
> Code: 00 00 00 85 c0 89 87 94 00 00 00 75 16 89 d0 e8 d4 10 28 df c7 87 84 00
> 00 00 00 00 00 00 e9 11 01 00 00 8b 87 8c 00 00 00 31 f6 <89> 50 3c 8b 87 94
> 00 00 00 8b 97 9c 00 00 00 89 42 3c 8b 87 8c
> EIP: [<e0eec8bd>] ipaq_open+0x1c4/0x2fb [ipaq] SS:ESP 0068:cf66fe1c
> ---[ end trace 592c3c8911178cf3 ]---
> padlock: VIA PadLock Hash Engine not detected.
> PPP MPPE Compression module registered
> PPP BSD Compression module registered
> PPP Deflate Compression module registered
> mppe_decomp_init[0]: unknown key length
> 

Somewhere in the middle of the large ipaq_open() and you're running a
vendor-packaged kernel and we haven't done much at all to ipaq.c since
2.6.25.  Tricky.
Comment 3 Adam Williamson 2008-06-05 19:53:46 UTC
On Thu, 2008-06-05 at 18:28 -0700, Andrew Morton wrote:
> > 
> > Pid: 7409, comm: pppd Tainted: P          (2.6.26-0.rc4.1mdv #1)
> 
> The nvidia driver is loaded, but it's unlikely to have caused this.

I agree. I can re-run the test without it if you *really* like.

> Somewhere in the middle of the large ipaq_open() and you're running a
> vendor-packaged kernel and we haven't done much at all to ipaq.c since
> 2.6.25.  Tricky.

As I specifically mentioned, the kernel I am running is a vanilla
upstream kernel. We introduced the kernel-linus package specifically to
aid testing in cases like this; it contains a completely clean build of
the kernel.org kernel with no patches or modifications at all. It was
specifically introduced in order to make it easy to determine if a bug
is in upstream kernel, or in our patches.
Comment 4 Alan 2008-06-06 03:14:49 UTC
> > This is with stock kernel.org kernel (packaged in Mandriva as
> kernel-linus),
> > not Mandriva's normal distro-patched kernel. So it's definitely your bug.
> :)

No.. there is Nvidia stuff loaded

> > Pid: 7409, comm: pppd Tainted: P          (2.6.26-0.rc4.1mdv #1)
> 
> The nvidia driver is loaded, but it's unlikely to have caused this.

Easy way to find out - is it reliably repeatable without that loaded. I
suspect yes.

> Somewhere in the middle of the large ipaq_open() and you're running a
> vendor-packaged kernel and we haven't done much at all to ipaq.c since
> 2.6.25.  Tricky.

Needs whoever can repeat it to pin it down further. Obvious checks would
be that port->read_urb/write_urb/tty are not NULL on entry and stick some
printks in to see how far down it gets
Comment 5 Adam Williamson 2008-06-06 09:33:10 UTC
On Fri, 2008-06-06 at 10:59 +0100, Alan Cox wrote:
> > > This is with stock kernel.org kernel (packaged in Mandriva as
> kernel-linus),
> > > not Mandriva's normal distro-patched kernel. So it's definitely your bug.
> :)
> 
> No.. there is Nvidia stuff loaded

That's built from DKMS.

> > > Pid: 7409, comm: pppd Tainted: P          (2.6.26-0.rc4.1mdv #1)
> > 
> > The nvidia driver is loaded, but it's unlikely to have caused this.
> 
> Easy way to find out - is it reliably repeatable without that loaded. I
> suspect yes.

So do I, so does Andrew, but fine, I'll do it, give me ten minutes...

> Needs whoever can repeat it to pin it down further. Obvious checks would
> be that port->read_urb/write_urb/tty are not NULL on entry and stick some
> printks in to see how far down it gets

I am not a hacker and know nothing about the kernel code base, so I'm
going to need instructions at more of an "upper management" level, I'm
afraid :)
Comment 6 Alan 2008-06-06 13:41:23 UTC
> That's built from DKMS.

I don't care if it was built from DKMS or hand assembled out of dead
chickens ;)

> > Needs whoever can repeat it to pin it down further. Obvious checks would
> > be that port->read_urb/write_urb/tty are not NULL on entry and stick some
> > printks in to see how far down it gets
> 
> I am not a hacker and know nothing about the kernel code base, so I'm
> going to need instructions at more of an "upper management" level, I'm
> afraid :)

If I send you a patch to apply, test and report what it says is that
fine ?

Alan
Comment 7 Adam Williamson 2008-06-06 13:48:42 UTC
On Fri, 2008-06-06 at 21:25 +0100, Alan Cox wrote:
> > That's built from DKMS.
> 
> I don't care if it was built from DKMS or hand assembled out of dead
> chickens ;)

Sure, but just pointing out that the kernel itself is fine, it's me
who's polluted it with NVIDIA code. ;)

> > > Needs whoever can repeat it to pin it down further. Obvious checks would
> > > be that port->read_urb/write_urb/tty are not NULL on entry and stick some
> > > printks in to see how far down it gets
> > 
> > I am not a hacker and know nothing about the kernel code base, so I'm
> > going to need instructions at more of an "upper management" level, I'm
> > afraid :)
> 
> If I send you a patch to apply, test and report what it says is that
> fine ?

Yeah, I can do that. I'll get the no-NVIDIA test done soon, just been
busy with other work for the last few hours.
Comment 8 Adam Williamson 2008-06-06 14:21:51 UTC
On Fri, 2008-06-06 at 21:25 +0100, Alan Cox wrote:
> > That's built from DKMS.
> 
> I don't care if it was built from DKMS or hand assembled out of dead
> chickens ;)

Okay, to absolutely no-one's surprise ;), the oops happens without
nvidia loaded. Attached.
Comment 9 Alan 2008-06-07 15:08:11 UTC
On Fri, 06 Jun 2008 14:19:25 -0700
Adam Williamson <awilliamson@mandriva.com> wrote:

> On Fri, 2008-06-06 at 21:25 +0100, Alan Cox wrote:
> > > That's built from DKMS.
> > 
> > I don't care if it was built from DKMS or hand assembled out of dead
> > chickens ;)
> 
> Okay, to absolutely no-one's surprise ;), the oops happens without
> nvidia loaded. Attached.

See what this reveals

ipaq: Debugging scan

From: Alan Cox <alan@redhat.com>

Trying to chase down a bug
---

 drivers/usb/serial/ipaq.c |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)


diff --git a/drivers/usb/serial/ipaq.c b/drivers/usb/serial/ipaq.c
index ea924dc..0b13c4e 100644
--- a/drivers/usb/serial/ipaq.c
+++ b/drivers/usb/serial/ipaq.c
@@ -609,6 +609,8 @@ static int ipaq_open(struct usb_serial_port *port, struct file *filp)
 	priv->free_len = 0;
 	INIT_LIST_HEAD(&priv->queue);
 	INIT_LIST_HEAD(&priv->freelist);
+	
+	printk("Serial %p tty %p\n", serial, port->tty);
 
 	for (i = 0; i < URBDATA_QUEUE_MAX / PACKET_SIZE; i++) {
 		pkt = kmalloc(sizeof(struct ipaq_packet), GFP_KERNEL);
@@ -636,6 +638,7 @@ static int ipaq_open(struct usb_serial_port *port, struct file *filp)
 	port->tty->raw = 1;
 	port->tty->real_raw = 1;
 
+	printk("bulkin %p bulkout %p\n", port->bulk_in_buffer, port->bulk_out_buffer);
 	/*
 	 * Lose the small buffers usbserial provides. Make larger ones.
 	 */
@@ -653,6 +656,7 @@ static int ipaq_open(struct usb_serial_port *port, struct file *filp)
 		port->bulk_in_buffer = NULL;
 		goto enomem;
 	}
+	printk("rdu wtu %p %p\n", port->read_urb, port->write_urb);
 	port->read_urb->transfer_buffer = port->bulk_in_buffer;
 	port->write_urb->transfer_buffer = port->bulk_out_buffer;
 	port->read_urb->transfer_buffer_length = URBDATA_SIZE;
@@ -669,6 +673,7 @@ static int ipaq_open(struct usb_serial_port *port, struct file *filp)
 	 */
 
 	while (retries--) {
+		printk("Pokety poke\n");
 		result = usb_control_msg(serial->dev,
 				usb_sndctrlpipe(serial->dev, 0), 0x22, 0x21,
 				0x1, 0, NULL, 0, 100);
@@ -683,13 +688,14 @@ static int ipaq_open(struct usb_serial_port *port, struct file *filp)
 		    result);
 		goto error;
 	}
-
+	printk("Filling\n");
 	/* Start reading from the device */
 	usb_fill_bulk_urb(port->read_urb, serial->dev,
 		      usb_rcvbulkpipe(serial->dev, port->bulk_in_endpointAddress),
 		      port->read_urb->transfer_buffer, port->read_urb->transfer_buffer_length,
 		      ipaq_read_bulk_callback, port);
 
+	printk("and submit\n");
 	result = usb_submit_urb(port->read_urb, GFP_KERNEL);
 	if (result) {
 		err("%s - failed submitting read urb, error %d", __func__, result);
Comment 10 Adam Williamson 2008-06-07 21:06:28 UTC
On Sat, 2008-06-07 at 22:52 +0100, Alan Cox wrote:
> On Fri, 06 Jun 2008 14:19:25 -0700
> Adam Williamson <awilliamson@mandriva.com> wrote:
> 
> > On Fri, 2008-06-06 at 21:25 +0100, Alan Cox wrote:
> > > > That's built from DKMS.
> > > 
> > > I don't care if it was built from DKMS or hand assembled out of dead
> > > chickens ;)
> > 
> > Okay, to absolutely no-one's surprise ;), the oops happens without
> > nvidia loaded. Attached.
> 
> See what this reveals

Roger - I'll try and get to it tomorrow, but if not, Monday.
Comment 11 Rafael J. Wysocki 2008-06-08 05:56:57 UTC
Regressions list annotation:
Handled-By : Alan Cox <alan@redhat.com>
Comment 12 Adam Williamson 2008-06-09 18:49:19 UTC
On Sat, 2008-06-07 at 22:52 +0100, Alan Cox wrote:
> On Fri, 06 Jun 2008 14:19:25 -0700
> Adam Williamson <awilliamson@mandriva.com> wrote:
> 
> > On Fri, 2008-06-06 at 21:25 +0100, Alan Cox wrote:
> > > > That's built from DKMS.
> > > 
> > > I don't care if it was built from DKMS or hand assembled out of dead
> > > chickens ;)
> > 
> > Okay, to absolutely no-one's surprise ;), the oops happens without
> > nvidia loaded. Attached.
> 
> See what this reveals
> 
> ipaq: Debugging scan

Okay, output with the patch is attached. Thanks for your help.
Comment 13 Alan 2008-06-14 14:43:10 UTC
Still waiting for the attachment ?
Comment 14 Rafael J. Wysocki 2008-06-14 15:05:24 UTC
On Saturday, 14 of June 2008, Alan Cox wrote:
> On Sat, 14 Jun 2008 22:12:04 +0200 (CEST)
> "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> 
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.25.  Please verify if it still should be listed.
> > 
> > 
> > Bug-Entry   : http://bugzilla.kernel.org/show_bug.cgi?id=10868
> > Subject             : Oops on loading ipaq module since 2.6.26, prevents
> use of device
> > Submitter   : Adam Williamson <awilliamson@mandriva.com>
> > Date                : 2008-06-05 17:39 (10 days old)
> > Handled-By  : Alan Cox <alan@redhat.com>
> 
> Still waiting for the actual attached result of the test patch to debug
> this further. Guess it will miss 2.6.26
Comment 15 Alan 2008-06-16 04:01:26 UTC
On Mon, 09 Jun 2008 18:45:48 -0700
Adam Williamson <awilliamson@mandriva.com> wrote:

> bulkin 00000000 bulkout 00000000
> rdu wtu 00000000 00000000


Ok found the email in the junk folder. It blows up because the driver
tries to modify port->read_urb and port->write_urb. That means it's not
one of the serial side changes I did but something on the USB side that
has changed.

Greg can no doubt continue from here.

    printk("rdu wtu %p %p\n", port->read_urb, port->write_urb);
        port->read_urb->transfer_buffer = port->bulk_in_buffer;
        port->write_urb->transfer_buffer = port->bulk_out_buffer;
        port->read_urb->transfer_buffer_length = URBDATA_SIZE;
        port->bulk_out_size = port->write_urb->transfer_buffer_length =
URBDATA

I don't actually see how this can occur as port->read_urb is set up in
the attach method and then never touched. 

Alan
Comment 16 Adam Williamson 2008-06-16 09:09:43 UTC
On Mon, 2008-06-16 at 11:43 +0100, Alan Cox wrote:
> On Mon, 09 Jun 2008 18:45:48 -0700
> Adam Williamson <awilliamson@mandriva.com> wrote:
> 
> > bulkin 00000000 bulkout 00000000
> > rdu wtu 00000000 00000000
> 
> 
> Ok found the email in the junk folder. It blows up because the driver
> tries to modify port->read_urb and port->write_urb. That means it's not
> one of the serial side changes I did but something on the USB side that
> has changed.
> 
> Greg can no doubt continue from here.

OK, so you need anything more from me yet? Just checking :)
Comment 17 Alan 2008-06-16 11:33:27 UTC
On Mon, 16 Jun 2008 09:08:21 -0700
Adam Williamson <awilliamson@mandriva.com> wrote:

> On Mon, 2008-06-16 at 11:43 +0100, Alan Cox wrote:
> > On Mon, 09 Jun 2008 18:45:48 -0700
> > Adam Williamson <awilliamson@mandriva.com> wrote:
> > 
> > > bulkin 00000000 bulkout 00000000
> > > rdu wtu 00000000 00000000
> > 
> > 
> > Ok found the email in the junk folder. It blows up because the driver
> > tries to modify port->read_urb and port->write_urb. That means it's not
> > one of the serial side changes I did but something on the USB side that
> > has changed.
> > 
> > Greg can no doubt continue from here.
> 
> OK, so you need anything more from me yet? Just checking :)

I don't.

Alan
Comment 18 Rafael J. Wysocki 2008-06-22 13:37:42 UTC
On Sunday, 22 of June 2008, Alan Cox wrote:
> On Sun, 22 Jun 2008 19:54:46 +0200 (CEST)
> "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> 
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.25.  Please verify if it still should be listed.
> > 
> > 
> > Bug-Entry   : http://bugzilla.kernel.org/show_bug.cgi?id=10868
> > Subject             : Oops on loading ipaq module since 2.6.26, prevents
> use of device
> > Submitter   : Adam Williamson <awilliamson@mandriva.com>
> > Date                : 2008-06-05 17:39 (18 days old)
> > Handled-By  : Alan Cox <alan@redhat.com>
> 
> Handed over to Greg as it's not a tty layer bug but something in USB
> (actually from the traces I think its a broken gcc/build but we shall)

Not-Handled-By : Alan Cox <alan@redhat.com>
Comment 19 Rafael J. Wysocki 2008-06-22 14:05:48 UTC
On Sunday, 22 of June 2008, Arjan van de Ven wrote:
> On Sun, 22 Jun 2008 14:00:44 -0700
> Arjan van de Ven <arjan@infradead.org> wrote:
> 
> > On Sun, 22 Jun 2008 21:07:06 +0100
> > Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> > 
> > > On Sun, 22 Jun 2008 19:54:46 +0200 (CEST)
> > > "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > > 
> > > > This message has been generated automatically as a part of a
> > > > report of recent regressions.
> > > > 
> > > > The following bug entry is on the current list of known
> > > > regressions from 2.6.25.  Please verify if it still should be
> > > > listed.
> > > > 
> > > > 
> > > > Bug-Entry       :
> > > > http://bugzilla.kernel.org/show_bug.cgi?id=10868
> > > > Subject         : Oops on loading ipaq module since
> > > > 2.6.26, prevents use of device Submitter        : Adam Williamson
> > > > <awilliamson@mandriva.com> Date         : 2008-06-05 17:39
> > > > (18 days old) Handled-By        : Alan Cox <alan@redhat.com>
> > > 
> > > Handed over to Greg as it's not a tty layer bug but something in USB
> > > (actually from the traces I think its a broken gcc/build but we
> > > shall)
> > 
> > http://www.kerneloops.org/search.php?search=ipaq_open
> > 
> > has a whole slew of different ones.... surprisingly they're almost all
> > with pppd...
> > 
> 
> btw this also shows that this isn't a 2.6.25 regression... 
Comment 20 Rafael J. Wysocki 2008-06-22 14:09:30 UTC
Dropping from the list of recent regressions.
Comment 21 Anonymous Emailer 2008-06-22 14:23:16 UTC
Reply-To: bunk@stusta.de

> > 
> > > On Sun, 22 Jun 2008 19:54:46 +0200 (CEST)
> > > "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > > 
> > > > This message has been generated automatically as a part of a
> > > > report of recent regressions.
> > > > 
> > > > The following bug entry is on the current list of known
> > > > regressions from 2.6.25.  Please verify if it still should be
> > > > listed.
> > > > 
> > > > 
> > > > Bug-Entry       :
> > > > http://bugzilla.kernel.org/show_bug.cgi?id=10868
> > > > Subject         : Oops on loading ipaq module since
> > > > 2.6.26, prevents use of device Submitter        : Adam Williamson
> > > > <awilliamson@mandriva.com> Date         : 2008-06-05 17:39
> > > > (18 days old) Handled-By        : Alan Cox <alan@redhat.com>
> > > 
> > > Handed over to Greg as it's not a tty layer bug but something in USB
> > > (actually from the traces I think its a broken gcc/build but we
> > > shall)
> > 
> > http://www.kerneloops.org/search.php?search=ipaq_open
> > 
> > has a whole slew of different ones.... surprisingly they're almost all
> > with pppd...
> 
> btw this also shows that this isn't a 2.6.25 regression... 

That hardly shows it.

And Adam said quite explicitely that 2.6.25 worked for him.

cu
Adrian
Comment 22 Rafael J. Wysocki 2008-06-22 14:27:07 UTC
Sorry, I should have checked the original report before dropping the bug from the list.
Comment 23 Adam Williamson 2008-06-22 17:36:50 UTC
Just to clean this up from my perspective: this is, for me at least, a demonstrably reproducible regression: I boot a (vanilla) 2.6.25 kernel and it fails, I boot a (vanilla) 2.6.26 kernel and it fails. I can reproduce this all day.

Some kinda compiler issue is a possibility I suppose: the kernel in question *is* built with our packaged gcc and probably with our normal optimizations. If there's any specific information you'd like about the exact gcc version, build and configuration, do ask, and I'll find it out.

And yes, it's not surprising that this usually happens with pppd, because as Alan Cox emailed, that's the normal use of this driver. Virtually anyone trying to actually do anything with the ipaq module will be using pppd.
Comment 24 Adrian Bunk 2008-06-22 23:36:27 UTC
(In reply to comment #23)
> Some kinda compiler issue is a possibility I suppose: the kernel in question
> *is* built with our packaged gcc and probably with our normal optimizations.
> If
> there's any specific information you'd like about the exact gcc version,
> build
> and configuration, do ask, and I'll find it out.

I doubt it is really a gcc issue, but to rule this out you can try it.

To try whether a different gcc makes a difference do:
  make CC=gcc-4.1
or
  make CC=/path/to/gcc
with a different gcc version installed.
Comment 25 Rafael J. Wysocki 2008-06-29 11:06:14 UTC
References : http://marc.info/?t=121287647600002&r=1&w=4
Handled-By : Oliver Neukum <oliver@neukum.org>
Comment 26 Greg Kroah-Hartman 2008-07-02 14:23:46 UTC
wait, you say 2.6.25 also fails?  What kernel version works for you for this driver?
Comment 27 Adam Williamson 2008-07-02 14:40:33 UTC
No, sorry, that line in comment #23 was a typo. 2.6.25 works, as per my original report. #23 should have read "I boot a (vanilla) 2.6.25 kernel and it
works, I boot a (vanilla) 2.6.26 kernel and it fails. I can reproduce this all
day.

in case you're not in CC - I'm working on this with Oliver Neukum ATM.
Comment 28 Adrian Bunk 2008-07-03 01:58:35 UTC
On Thu, Jul 03, 2008 at 10:05:57AM +0200, Oliver Neukum wrote:
> Am Mittwoch 02 Juli 2008 23:41:40 schrieb Adam Williamson:
> > On Wed, 2008-07-02 at 23:33 +0200, Oliver Neukum wrote:
> > 
> > > This is odd. Your device shows one interface with one endpoint bulk and
> > > bulk out respectively. Yet two ports are created. Odd.
> > 
> > OK. If you mean /dev/ttyUSB0 and /dev/ttyUSB1 are always both created
> > when the device is plugged in - yep. This is the case in the working
> 
> Now this is very hard to explain. From the code in 2.6.25 it is clear that
> only ttyUSB0 will be created. Please verify that indeed you get ttyUSB0
> and ttyUSB1 with the kernel working for you.
> 
> > kernel too. From what I've seen in howtos and the like, this seems to be
> > the case for most such devices. Well, let me know what else you need
> > from me. :)
> 
> As far as I can tell somebody changed the ipaq driver in 2.6.26-rc6. I cannot
> find the exact patch that did it in Greg's directory. As it causes a
> regression
> here's a reversal.


It was changed by:


commit e1879b19b0abdb387e4aeb0b935a486cc75042fb
Author: Matthias Geissert <matthias.geissert@web.de>
Date:   Thu Mar 6 22:00:33 2008 +0100

    USB: ipaq: fix devices having more than one endpoint
    
    The ipaq module  supports devices with one endpoint only. Some devices,
    e.g. Yakumo Delta 300, have more than one endpoint.
    
    This patch fixes support for devices having up to 2 endpoints which used
    to work on older kernel versions.
    
    Signed-off-by: Matthias Geissert <matthias.geissert@web.de>
    Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>

diff --git a/drivers/usb/serial/ipaq.c b/drivers/usb/serial/ipaq.c
index 9b38a08..17f2a53 100644
--- a/drivers/usb/serial/ipaq.c
+++ b/drivers/usb/serial/ipaq.c
@@ -571,9 +571,9 @@ static struct usb_serial_driver ipaq_device = {
 	.usb_driver = 		&ipaq_driver,
 	.id_table =		ipaq_id_table,
 	.num_interrupt_in =	NUM_DONT_CARE,
-	.num_bulk_in =		1,
-	.num_bulk_out =		1,
-	.num_ports =		1,
+	.num_bulk_in =		NUM_DONT_CARE,
+	.num_bulk_out =		NUM_DONT_CARE,
+	.num_ports =		2,
 	.open =			ipaq_open,
 	.close =		ipaq_close,
 	.attach =		ipaq_startup,


>       Regards
>               Oliver
> 
> Signed-off-by: Oliver Neukum <oneukum@suse.de>
> 
> ---
> 
> --- linux-2.6.26-greg/drivers/usb/serial/ipaq.alt.c   2008-07-03
> 09:01:37.000000000 +0200
> +++ linux-2.6.26-greg/drivers/usb/serial/ipaq.c       2008-07-03
> 09:01:47.000000000 +0200
> @@ -570,7 +570,7 @@ static struct usb_serial_driver ipaq_dev
>       .description =          "PocketPC PDA",
>       .usb_driver =           &ipaq_driver,
>       .id_table =             ipaq_id_table,
> -     .num_ports =            2,
> +     .num_ports =            1,
>       .open =                 ipaq_open,
>       .close =                ipaq_close,
>       .attach =               ipaq_startup,
> 

cu
Adrian
Comment 29 Anonymous Emailer 2008-07-03 07:08:23 UTC
Reply-To: oliver@neukum.org

Am Donnerstag 03 Juli 2008 10:57:33 schrieb Adrian Bunk:
> On Thu, Jul 03, 2008 at 10:05:57AM +0200, Oliver Neukum wrote:
 
> > As far as I can tell somebody changed the ipaq driver in 2.6.26-rc6. I
> cannot
> > find the exact patch that did it in Greg's directory. As it causes a
> regression
> > here's a reversal.
> 
> 
> It was changed by:

Thanks, did you do git magic?

> commit e1879b19b0abdb387e4aeb0b935a486cc75042fb
> Author: Matthias Geissert <matthias.geissert@web.de>
> Date:   Thu Mar 6 22:00:33 2008 +0100
> 
>     USB: ipaq: fix devices having more than one endpoint
>     
>     The ipaq module  supports devices with one endpoint only. Some devices,
>     e.g. Yakumo Delta 300, have more than one endpoint.
>     
>     This patch fixes support for devices having up to 2 endpoints which used
>     to work on older kernel versions.
>     
>     Signed-off-by: Matthias Geissert <matthias.geissert@web.de>
>     Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>
> 
> diff --git a/drivers/usb/serial/ipaq.c b/drivers/usb/serial/ipaq.c
> index 9b38a08..17f2a53 100644
> --- a/drivers/usb/serial/ipaq.c
> +++ b/drivers/usb/serial/ipaq.c
> @@ -571,9 +571,9 @@ static struct usb_serial_driver ipaq_device = {
>       .usb_driver =           &ipaq_driver,
>       .id_table =             ipaq_id_table,
>       .num_interrupt_in =     NUM_DONT_CARE,
> -     .num_bulk_in =          1,
> -     .num_bulk_out =         1,
> -     .num_ports =            1,
> +     .num_bulk_in =          NUM_DONT_CARE,
> +     .num_bulk_out =         NUM_DONT_CARE,

This is good.

> +     .num_ports =            2,

This is fatal.

The patch I sent reverts only this part, so this issue is solved.

	Regards
		Oliver
Comment 30 Adrian Bunk 2008-07-03 07:21:04 UTC
On Thu, Jul 03, 2008 at 04:08:44PM +0200, Oliver Neukum wrote:
> Am Donnerstag 03 Juli 2008 10:57:33 schrieb Adrian Bunk:
> > On Thu, Jul 03, 2008 at 10:05:57AM +0200, Oliver Neukum wrote:
>  
> > > As far as I can tell somebody changed the ipaq driver in 2.6.26-rc6. I
> cannot
> > > find the exact patch that did it in Greg's directory. As it causes a
> regression
> > > here's a reversal.
> > 
> > 
> > It was changed by:
> 
> Thanks, did you do git magic?

I wouldn't call that "magic":

$ git-log v2.6.25.. drivers/usb/serial/ipaq.c

>...
> > --- a/drivers/usb/serial/ipaq.c
> > +++ b/drivers/usb/serial/ipaq.c
> > @@ -571,9 +571,9 @@ static struct usb_serial_driver ipaq_device = {
> >     .usb_driver =           &ipaq_driver,
> >     .id_table =             ipaq_id_table,
> >     .num_interrupt_in =     NUM_DONT_CARE,
> > -   .num_bulk_in =          1,
> > -   .num_bulk_out =         1,
> > -   .num_ports =            1,
> > +   .num_bulk_in =          NUM_DONT_CARE,
> > +   .num_bulk_out =         NUM_DONT_CARE,
> 
> This is good.

These fields are removed by a later commit in 2.6.26...  8-)

> > +   .num_ports =            2,
> 
> This is fatal.
> 
> The patch I sent reverts only this part, so this issue is solved.
> 
>       Regards
>               Oliver

cu
Adrian
Comment 31 Anonymous Emailer 2008-07-03 07:26:22 UTC
Reply-To: oliver@neukum.org

Am Donnerstag 03 Juli 2008 16:20:30 schrieb Adrian Bunk:
> On Thu, Jul 03, 2008 at 04:08:44PM +0200, Oliver Neukum wrote:
> > Am Donnerstag 03 Juli 2008 10:57:33 schrieb Adrian Bunk:

> > > --- a/drivers/usb/serial/ipaq.c
> > > +++ b/drivers/usb/serial/ipaq.c
> > > @@ -571,9 +571,9 @@ static struct usb_serial_driver ipaq_device = {
> > >   .usb_driver =           &ipaq_driver,
> > >   .id_table =             ipaq_id_table,
> > >   .num_interrupt_in =     NUM_DONT_CARE,
> > > - .num_bulk_in =          1,
> > > - .num_bulk_out =         1,
> > > - .num_ports =            1,
> > > + .num_bulk_in =          NUM_DONT_CARE,
> > > + .num_bulk_out =         NUM_DONT_CARE,
> > 
> > This is good.
> 
> These fields are removed by a later commit in 2.6.26...  8-)

Well, NUM_DONT_CARE is default, so it makes no difference, but a comment
would have been nice.

	Regards
		Oliver
Comment 32 Adrian Bunk 2008-07-03 08:00:33 UTC
On Thu, Jul 03, 2008 at 04:26:48PM +0200, Oliver Neukum wrote:
> Am Donnerstag 03 Juli 2008 16:20:30 schrieb Adrian Bunk:
> > On Thu, Jul 03, 2008 at 04:08:44PM +0200, Oliver Neukum wrote:
> > > Am Donnerstag 03 Juli 2008 10:57:33 schrieb Adrian Bunk:
> 
> > > > --- a/drivers/usb/serial/ipaq.c
> > > > +++ b/drivers/usb/serial/ipaq.c
> > > > @@ -571,9 +571,9 @@ static struct usb_serial_driver ipaq_device = {
> > > >         .usb_driver =           &ipaq_driver,
> > > >         .id_table =             ipaq_id_table,
> > > >         .num_interrupt_in =     NUM_DONT_CARE,
> > > > -       .num_bulk_in =          1,
> > > > -       .num_bulk_out =         1,
> > > > -       .num_ports =            1,
> > > > +       .num_bulk_in =          NUM_DONT_CARE,
> > > > +       .num_bulk_out =         NUM_DONT_CARE,
> > > 
> > > This is good.
> > 
> > These fields are removed by a later commit in 2.6.26...  8-)
> 
> Well, NUM_DONT_CARE is default, so it makes no difference, but a comment
> would have been nice.

They were removed from struct usb_serial_driver in 
include/linux/usb/serial.h since the checks they were used for caused 
more problems than they solved.

But you are right that the effects of the removal are the same as 
NUM_DONT_CARE.

>       Regards
>               Oliver

cu
Adrian
Comment 33 Adrian Bunk 2008-07-03 23:59:56 UTC
fixed by commit 4edb966b375dfbabfc96b580a164c5ae90584aa0
Comment 34 Anonymous Emailer 2008-07-07 11:11:57 UTC
Reply-To: matthias.geissert@web.de

Am Donnerstag, 3. Juli 2008 16:08:44 schrieb Oliver Neukum:


> >     .num_interrupt_in =     NUM_DONT_CARE,
> > -   .num_bulk_in =          1,
> > -   .num_bulk_out =         1,
> > -   .num_ports =            1,
> > +   .num_bulk_in =          NUM_DONT_CARE,
> > +   .num_bulk_out =         NUM_DONT_CARE,
>
> This is good.
>
> > +   .num_ports =            2,
>
> This is fatal.

I checked what you said with kernel 2.6.26 rc9.  I set num_ports to 5. It 
worked quite well until I tried to connect to a non-existing endpoint. 

However, the problem is that the Yakumo Delta needs to connect to the 2nd 
endpoint. You can connect to the first one but you don't get any data. 

Is there any good way to tell the ipaq driver to use the 2nd endpoint? Maybe 
one could provide a different struct usb_serial_driver ipaq_device depending 
on the usb id or an option which tells the driver to use 2 endpoints.

Regards,
matthias  
Comment 35 Anonymous Emailer 2008-07-07 12:18:50 UTC
Reply-To: oliver@neukum.org

Am Montag 07 Juli 2008 20:16:02 schrieb Matthias Geissert:
> Am Donnerstag, 3. Juli 2008 16:08:44 schrieb Oliver Neukum:
> 
> 
> > >   .num_interrupt_in =     NUM_DONT_CARE,
> > > - .num_bulk_in =          1,
> > > - .num_bulk_out =         1,
> > > - .num_ports =            1,
> > > + .num_bulk_in =          NUM_DONT_CARE,
> > > + .num_bulk_out =         NUM_DONT_CARE,
> >
> > This is good.
> >
> > > + .num_ports =            2,
> >
> > This is fatal.
> 
> I checked what you said with kernel 2.6.26 rc9.  I set num_ports to 5. It 
> worked quite well until I tried to connect to a non-existing endpoint. 
> 
> However, the problem is that the Yakumo Delta needs to connect to the 2nd 
> endpoint. You can connect to the first one but you don't get any data. 
> 
> Is there any good way to tell the ipaq driver to use the 2nd endpoint? Maybe 
> one could provide a different struct usb_serial_driver ipaq_device depending 
> on the usb id or an option which tells the driver to use 2 endpoints.

If only these devices have a second input endpoint, we can detect that
in attach().

	Regards
		Oliver
Comment 36 Frank de Lange 2008-11-08 14:52:28 UTC
This bug - or its sibling - is alive and kicking in 2.6.28-rc3:

on connecting an HTC Prophet (aka Qtek S200 aka (lsusb:) ID 0bb4:0bce High Tech Computer Corp. Vario MDA):



[50361.774063] BUG: unable to handle kernel NULL pointer dereference at 0000003c
[50361.774078] IP: [<f0a11923>] ipaq_open+0x1bb/0x31a [ipaq]
[50361.774117] *pde = 00000000 
[50361.774126] Oops: 0002 [#8] PREEMPT 
[50361.774134] last sysfs file: /sys/devices/pci0000:00/0000:00:1d.1/usb2/2-2/2-2:1.0/usb_endpoint/usbdev2.13_ep81/dev
[50361.774143] Modules linked in: nls_iso8859_1 nls_cp437 vfat fat usb_storage libusual af_packet wlan_scan_sta ath_rate_sample ath_pci wlan ath_hal(P) ppp_generic slhc rndis_host cdc_ether usbnet savage drm binfmt_misc tun vboxdrv nfsd auth_rpcgss exportfs ppdev autofs4 ipv6 speedstep_lib cpufreq_stats cpufreq_conservative cpufreq_userspace cpufreq_powersave sbs sbshc nfs lockd sunrpc iptable_filter ip_tables x_tables configfs lp ipaq usbserial pcmcia video output nsc_ircc parport_pc parport irda crc_ccitt snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss psmouse snd_pcm serio_raw yenta_socket rsrc_nonstatic snd_seq_dummy pcmcia_core snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer intel_agp snd_seq_device pcspkr agpgart evdev iTCO_wdt iTCO_vendor_support snd soundcore snd_page_alloc sr_mod cdrom sd_mod sg ata_piix libata scsi_mod e100 mii uhci_hcd usbcore fuse
[50361.774294] 
[50361.774302] Pid: 5109, comm: pppd Tainted: P      D    (2.6.28-rc3-t23-200811072117 #22) 26474MG
[50361.774311] EIP: 0060:[<f0a11923>] EFLAGS: 00010246 CPU: 0
[50361.774334] EIP is at ipaq_open+0x1bb/0x31a [ipaq]
[50361.774340] EAX: 00000000 EBX: dbdfd460 ECX: f0a11901 EDX: c18e8000
[50361.774347] ESI: 00000000 EDI: dbd5b400 EBP: dbd54e38 ESP: dbd54e14
[50361.774354]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
[50361.774362] Process pppd (pid: 5109, ti=dbd54000 task=c18f2fc0 task.ti=dbd54000)
[50361.774368] Stack:
[50361.774372]  dbf40000 dbdfd474 dbdf7180 00000100 00000064 dbd5b400 f0a18640 dbd5b400
[50361.774387]  dbdf7180 dbd54e60 f09f6dd9 dbd65800 dbf40000 dbd5b43c dbd5b404 00000001
[50361.774402]  dbf40000 ee7536c0 ffffffed dbd54e8c c02258ad dbd65800 dbce7e6c 00000802
[50361.774417] Call Trace:
[50361.774428]  [<f09f6dd9>] ? serial_open+0x117/0x176 [usbserial]
[50361.774452]  [<c02258ad>] ? tty_open+0x25b/0x34a
[50361.774468]  [<c016b649>] ? chrdev_open+0x15b/0x172
[50361.774479]  [<c0167e49>] ? __dentry_open+0x11c/0x203
[50361.774493]  [<c0167fb7>] ? nameidata_to_filp+0x1f/0x33
[50361.774504]  [<c016b4ee>] ? chrdev_open+0x0/0x172
[50361.774512]  [<c01725b7>] ? do_filp_open+0x33b/0x659
[50361.774526]  [<c012ddb9>] ? autoremove_wake_function+0x0/0x33
[50361.774539]  [<c0179823>] ? alloc_fd+0x5b/0xdf
[50361.774551]  [<c0167c54>] ? do_sys_open+0x42/0xbc
[50361.774560]  [<c016a0d2>] ? fput+0x16/0x18
[50361.774571]  [<c0167d10>] ? sys_open+0x1e/0x26
[50361.774580]  [<c0103945>] ? sysenter_do_call+0x12/0x25
[50361.774592] Code: 70 40 3b c0 e8 63 51 75 cf 8b 57 68 85 c0 89 47 78 75 13 89 d0 e8 4f 50 75 cf c7 47 68 00 00 00 00 e9 22 01 00 00 8b 47 70 31 f6 <89> 50 3c 8b 47 78 8b 97 80 00 00 00 89 42 3c 8b 47 70 c7 40 44 
[50361.774662] EIP: [<f0a11923>] ipaq_open+0x1bb/0x31a [ipaq] SS:ESP 0068:dbd54e14
[50361.774691] ---[ end trace b129b1efb3f40048 ]---


This kernel is tainted by the presence of Madwifi, the oops happens with ath5k as well (but ath5k does not work as it should so...). The oops is as reliable as Big Ben.
Comment 37 Adam Williamson 2008-11-08 15:10:34 UTC
Unfortunately I can't confirm or deny this here, as my WM2003 test device is dead :(. If I get another I will check back.
Comment 38 ahnert 2009-07-19 19:00:12 UTC
This bug seems still be alive and kicking in 2.6.30.
I did connect an HTC Prophet and the kernel starts to oops.

lsusb:
Bus 002 Device 002: ID 0bb4:0bce High Tech Computer Corp. Vario MDA

kernel log:
usb 2-8: new full speed USB device using ohci_hcd and address 2
usb 2-8: configuration #1 chosen from 1 choice
usbcore: registered new interface driver usbserial
USB Serial support registered for generic
usbcore: registered new interface driver usbserial_generic
usbserial: USB Serial Driver core
USB Serial support registered for PocketPC PDA
ipaq 2-8:1.0: PocketPC PDA converter detected
usb 2-8: PocketPC PDA converter now attached to ttyUSB0
ipaq 2-8:1.1: PocketPC PDA converter detected
usb 2-8: PocketPC PDA converter now attached to ttyUSB1
usbcore: registered new interface driver ipaq
ipaq: v0.5:USB PocketPC PDA driver
usbcore: registered new interface driver cdc_ether
usbcore: registered new interface driver rndis_host
usbcore: registered new interface driver rndis_wlan
PPP generic driver version 2.4.2
BUG: unable to handle kernel NULL pointer dereference at 0000003c
IP: [<f8b41cd9>] ipaq_open+0x209/0x650 [ipaq]
*pde = 00000000 
Oops: 0002 [#1] PREEMPT SMP 
last sysfs file: /sys/devices/virtual/ppp/ppp/dev
Modules linked in: ppp_generic slhc rndis_wlan rndis_host cdc_ether usbnet mii ipaq usbserial aes_i586 aes_generic fuse ipv6 arc4 ecb snd_hda_codec_nvhdmi usb_storage zd1211rw mac80211 cfg80211 snd_hda_codec_idt ohci_hcd snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_hda_intel nvidia(P) snd_seq_device i2c_nforce2 agpgart snd_hda_codec snd_hwdep snd_pcm snd_timer snd soundcore ehci_hcd sg wmi i2c_core snd_page_alloc usbcore psmouse forcedeth pcspkr serio_raw evdev thermal processor fan button battery ac rtc_cmos rtc_core rtc_lib reiserfs sr_mod cdrom sd_mod pata_acpi pata_amd ata_generic ahci libata scsi_mod

Pid: 3895, comm: pppd Tainted: P           (2.6.30-ARCH #1) GF7100/7050PVT-M3
EIP: 0060:[<f8b41cd9>] EFLAGS: 00210286 CPU: 1
EIP is at ipaq_open+0x209/0x650 [ipaq]
EAX: 00000000 EBX: f5bac000 ECX: f525e800 EDX: f5baf000
ESI: f6afa000 EDI: 00001000 EBP: 00000000 ESP: f3c63da8
 DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Process pppd (pid: 3895, ti=f3c62000 task=f5b7c800 task.ti=f3c62000)
Stack:
 22222222 22222222 22222222 22222222 c539602e c539602e f6afa014 00000100
 f525e800 f5b74800 00000064 f5142ba0 f8e5c00a 00000000 c539602e f8b48f00
 f5142ba0 f5b74800 f525e800 f8a97acc c02e4ed8 c539602e f43ab600 c539602e
Call Trace:
 [<f8e5c00a>] ? usb_autopm_do_interface+0x8a/0x100 [usbcore]
 [<f8a97acc>] ? serial_open+0x1ac/0x260 [usbserial]
 [<c02e4ed8>] ? tty_ldisc_setup+0x88/0x90
 [<c02dfe66>] ? tty_open+0x186/0x460
 [<c01d48da>] ? chrdev_open+0x10a/0x1e0
 [<c01ce7bd>] ? __dentry_open+0xfd/0x2c0
 [<c01d47d0>] ? chrdev_open+0x0/0x1e0
 [<c01ceaad>] ? nameidata_to_filp+0x6d/0x80
 [<c01ded73>] ? do_filp_open+0x223/0x8b0
 [<c0129ccc>] ? flush_tlb_page+0x4c/0xe0
 [<c0129455>] ? ptep_set_access_flags+0x75/0x80
 [<c01e9a2d>] ? alloc_fd+0xcd/0x120
 [<c01ce4f9>] ? do_sys_open+0x69/0x110
 [<c01ce648>] ? sys_open+0x38/0x60
 [<c0103c73>] ? sysenter_do_call+0x12/0x28
Code: c3 0f 85 d0 02 00 00 8b 4c 24 20 85 db 89 99 ac 00 00 00 0f 84 32 04 00 00 8b 54 24 20 8b 82 a4 00 00 00 89 d1 8b 92 9c 00 00 00 <89> 50 3c 8b 81 b4 00 00 00 8b 91 ac 00 00 00 89 50 3c 8b 81 a4 
EIP: [<f8b41cd9>] ipaq_open+0x209/0x650 [ipaq] SS:ESP 0068:f3c63da8
CR2: 000000000000003c
---[ end trace c25388a15e92b51c

I hope I can help.