Bug 10451

Summary: Oops when trying to mount remote CIFS share
Product: File System Reporter: Olivier Berger (oberger)
Component: CIFSAssignee: Steve French (sfrench)
Status: RESOLVED CODE_FIX    
Severity: high CC: bunk, gege, olivier.berger, shirishp
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 2.6.25.3 Subsystem:
Regression: Yes Bisected commit-id:
Attachments: debug data from comment #6
Fix for mount oops

Description Olivier Berger 2008-04-13 23:09:15 UTC
Latest working kernel version: 2.6.22 ?
Earliest failing kernel version: 2.6.24 ?
Distribution: Debian (testing)
Hardware Environment: 
The remote "server" is a Iomega "NAS" Network hard drive 500 Gb
Software Environment: std Debian kernels

Problem Description:
Kernel oops when mounting a remote share with command similar to : 
mount //server/stuff /mnt/server -t cifs -o ip=192.168.4.14,user=xxx,pass=xxx

Steps to reproduce:
Mounting it like above cmdline, we get traces like (obtained on latest non-std test Debian kernel package for 2.6.25-rc8, but similar traces on earlier ones) :

[  183.990770] BUG: unable to handle kernel NULL pointer dereference at 00000010
[  183.990770] IP: [<f0fd99f8>] :cifs:cifs_strfromUCS_le+0x50/0x5f
[  183.990770] *pde = 00000000 
[  183.990770] Oops: 0002 [#1] SMP 
[  183.990770] Modules linked in: nls_utf8 cifs nls_base binfmt_misc radeon drm nfsd auth_rpcgss exportfs ppdev lp ac battery microcode firmware_class nfs lockd nfs_acl sunrpc ipv6 rfcomm l2cap bluetooth reiserfs dm_crypt crypto_blkcipher fuse smsc47m1 eeprom firewire_sbp2 loop evdev joydev usbhid hid ff_memless usblp parport_pc parport snd_intel8x0 snd_ac97_codec ac97_bus psmouse snd_bt87x pcspkr serio_raw snd_pcm_oss snd_mixer_oss rtc snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event container snd_seq snd_timer snd_seq_device snd button i2c_sis96x soundcore snd_page_alloc i2c_core sis_agp agpgart shpchp pci_hotplug ext3 jbd mbcache dm_mirror dm_snapshot dm_mod sg sr_mod cdrom sd_mod sis5513 ide_pci_generic ide_core floppy ehci_hcd firewire_ohci firewire_core crc_itu_t ohci_hcd tulip pata_sis ata_generic usbcore libata scsi_mod dock thermal processor fan
[  183.990770] 
[  183.990770] Pid: 4322, comm: mount.cifs Not tainted (2.6.25-rc8-686 #1)
[  183.990770] EIP: 0060:[<f0fd99f8>] EFLAGS: 00210246 CPU: 0
[  183.990770] EIP is at cifs_strfromUCS_le+0x50/0x5f [cifs]
[  183.990770] EAX: 00000010 EBX: 00000000 ECX: 00000000 EDX: dfb7132e
[  183.990770] ESI: 0000000b EDI: 00000000 EBP: 00000010 ESP: dfb4dccc
[  183.990770]  DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[  183.990770] Process mount.cifs (pid: 4322, ti=dfb4c000 task=dfb1b310 task.ti=dfb4c000)
[  183.990770] Stack: dfb765c0 00000000 dfb7132e 00000000 0000000b dfb7132d dfacfe80 f0fdc914 
[  183.990770]        f0cdd4c0 00000001 00000000 00000000 ef992800 f0fd712a 00000000 dfb7132e 
[  183.990770]        df9aebc0 dfb58300 44ac3674 8576505f 6a921331 47b2bed6 5fbcea37 6c58df6b 
[  183.990770] Call Trace:
[  183.990770]  [<f0fdc914>] CIFS_SessSetup+0x4c8/0x728 [cifs]
[  183.990770]  [<f0fd712a>] SendReceive+0x3bf/0x3d2 [cifs]
[  183.990770]  [<f0fca5a4>] cifs_setup_session+0x124/0xaec [cifs]
[  183.990770]  [<c01db711>] sprintf+0x1d/0x20
[  183.990770]  [<f0fcd871>] cifs_mount+0x1841/0x1e92 [cifs]
[  183.990770]  [<f0fcd8f4>] cifs_mount+0x18c4/0x1e92 [cifs]
[  183.990770]  [<c01599b3>] get_page_from_freelist+0x2fe/0x37b
[  183.990770]  [<c01da7c2>] strlcpy+0x10/0x3a
[  183.990770]  [<f0fc1589>] cifs_get_sb+0x94/0x166 [cifs]
[  183.990770]  [<c0174154>] vfs_kern_mount+0x7b/0xed
[  183.990770]  [<c0174204>] do_kern_mount+0x2f/0xb4
[  183.990770]  [<c0185999>] do_new_mount+0x55/0x89
[  183.990770]  [<c0185b66>] do_mount+0x199/0x1b8
[  183.990770]  [<c0183fc2>] copy_mount_options+0x26/0x109
[  183.990770]  [<c0185bf2>] sys_mount+0x6d/0xa7
[  183.990770]  [<c01048cc>] sysenter_past_esp+0x6d/0xa5
[  183.990770]  [<c02a0000>] tpacket_rcv+0x369/0x3b4
[  183.990770]  =======================
[  183.990770] Code: b9 06 00 00 00 ff 56 08 85 c0 7e 04 01 c3 eb 07 8b 04 24 43 c6 00 3f 47 3b 7c 24 04 7d 0d 8b 54 24 08 66 8b 04 7a 66 85 c0 75 c7 <c6> 44 1d 00 00 89 d8 83 c4 0c 5b 5e 5f 5d c3 55 31 ed 57 89 cf 
[  183.990770] EIP: [<f0fd99f8>] cifs_strfromUCS_le+0x50/0x5f [cifs] SS:ESP 0068:dfb4dccc
[  183.990770] ---[ end trace c210fe14b4e416ce ]---

More details in the Debian BTS : http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=463402
Comment 1 Olivier Berger 2008-05-23 14:18:09 UTC
FYI, I tested with Debian kernel 2.6.25-2-686, and still crashing (more details at http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=463402).
Comment 2 Olivier Berger 2008-05-23 14:46:54 UTC
Actually tested on Debian kernel 2.6.25-2-686 (2.6.25-3) which contains kernel 2.6.25.3.
Comment 3 Shirish Pargaonkar 2009-01-31 16:49:57 UTC
When the oops happens, what are the contents in the file
 /proc/fs/cifs/DebugData?
Comment 4 Steve French 2009-02-03 08:12:04 UTC
Looking at this today in trying to cleanout old bugs that had fallen through the cracks - we need more information since it does not appear that these fields can be null in the cifs_strfromUCSle function and it is not obvious to me or other developers at first glance why/what is wrong in decode_unicode_ssetup in fs/cifs/sess.c.   Would you turn on debugging and attach (or forward via email) the trace.

you can do:

1) "dmesg -c" (to clear the message log)
2) "echo 7 > /proc/fs/cifs/cifsFYI" (to enable cifs debugging)
3) mount
4) "echo 0 > /proc/fs/cifs/cifsFYI" (to turn off cifs debugging messages)
5) dmesg > error-log-from-mount

and email or attach the error-log-from-mount

If you are comfortable building your own kernel, then it would also be helpful  to enable memory debugging in the "kernel hacking" part of the kernel configuration:  e.g. some set of debug slab memory allocations, debug vm, compile kernel with frame pointers, debug page memory, check for stack overflows, write protect read only structures
Comment 5 Steve French 2009-02-03 08:15:44 UTC
To verify whether the NAS appliance has sent a corrupt SMB SessionSetup response, it would also be helpful if we could get a network trace of the failed mount.

See 

http://wiki.samba.org/index.php/Capture_Packets

for information on how to take a network trace if you are not familiar with it
Comment 6 Michael Geiger 2009-02-08 03:37:04 UTC
Hello, I have the same problem here!
Kernel: 2.6.28.2, running on an AMD 64 X2   (SuSE 11.1 64bit, vanilla kernel)

I tried to mount a NAS device (Raidsonic ICY BOX IB-NAS900) with this fstab entry:
//elenas/musik /mnt/ele cifs noauto,credentials=/etc/fstab.cred.ele,uid=share,gid=users,file_mode=0640,dir_mode=0750,noacl,iocharset=iso8859-1 0 0

To help you with this bug I'll attach bug-10451-gege.tgz including these files:
boot.txt        dmesg from system boot
config.gz       Kernel config
debugdata.txt   content from /proc/fs/cifs/DebugData
mount-oops.txt  output from dmesg (as suggested in comment #4)
net.dump        tcpdump from mount attempt


Michael
Comment 7 Michael Geiger 2009-02-08 03:38:19 UTC
Created attachment 20164 [details]
debug data from comment #6
Comment 8 Steve French 2009-02-16 10:16:43 UTC
The OS and the Network OS fields in the SMB response are empty (one character of null), which is quite unusual.  This may be the reason that session setup has problems parsing this response.
Comment 9 Steve French 2009-02-16 13:04:36 UTC
by changing a test version of Samba server slightly (to return a null OS field) I was able to reproduce the problem.
Comment 10 Steve French 2009-02-16 14:33:08 UTC
Created attachment 20269 [details]
Fix for mount oops

When the length of the OS or NOS field was null we were not allocating any memory for them and thus would oops when we set the serverNOS[0] to null

Fix attached. Will try to push this upstream ASAP if it tests out ok for you
Comment 11 Steve French 2009-02-16 18:18:24 UTC
Pushed into cifs-2.6.git and cc:stable
Comment 12 Michael Geiger 2009-02-17 07:25:04 UTC
Tested with 2.6.28.5, your patch solved the problem!
Thank you!!
Comment 13 Olivier Berger 2009-03-01 23:06:38 UTC
Tested with 2.6.26 and it solves the problem successfully.

Many thanks indeed