Bug 11389
Summary: | getting call trace while copying the large ext3 data to ext4 mounted filesystem | ||
---|---|---|---|
Product: | File System | Reporter: | Rishikesh K Rajak (risrajak) |
Component: | ext4 | Assignee: | fs_ext4 (fs_ext4) |
Status: | CLOSED PATCH_ALREADY_AVAILABLE | ||
Severity: | high | CC: | iranna.ankad, tytso |
Priority: | P1 | ||
Hardware: | All | ||
OS: | Linux | ||
Kernel Version: | 2.6.27-rc3 plus 2.6.27-rc3-ext4-1 ( even on plain 2.6.27-rc3 als | Subsystem: | |
Regression: | --- | Bisected commit-id: |
Description
Rishikesh K Rajak
2008-08-21 02:36:29 UTC
sorry, the kernel version is 2.6.27-rc3. Here is the uname output: Linux e326 2.6.27-rc3 #3 SMP Thu Aug 21 10:36:10 IST 2008 i686 athlon i386 GNU/Linux Note: This kernel is 2.6.27-rc3 + ftp://ftp.kernel.org/pub/linux/kernel/people/tytso/ext4-patches/2.6.27-rc3-ext4-1/broken-out.tar.bz2 - Rishi here is the fsck output for this particular case: [root@e326 sbin]# ./fsck.ext4dev -fnv /dev/sdb5 e2fsck 1.41.0 (10-Jul-2008) Warning! /dev/sdb5 is mounted. Warning: skipping journal recovery because doing a read-only filesystem check. Pass 1: Checking inodes, blocks, and sizes Inode 3147295, i_blocks is 105788, should be 105768. Fix? no Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Block bitmap differences: -(50356371--50356372) -(50449009--50449016) Fix? no Free blocks count wrong (68960931, counted=68987599). Fix? no /dev/sdb5: ********** WARNING: Filesystem still has errors ********** 26027 inodes used (0.58%) 5942 non-contiguous inodes (22.8%) # of inodes with ind/dind/tind blocks: 5922/918/1 2721033 blocks used (3.80%) 0 bad blocks 0 large files 20204 regular files 4030 directories 148 character device files 35 block device files 4 fifos 2 links 1587 symbolic links (1514 fast symbolic links) 10 sockets -------- 26020 files [root@e326 sbin]# This problem was reported last week, and already fixed in the ext4-patch-queue, and released in the 2.6.27-rc3-ext4-1 and 2.6.26-ext4-7 patchsets. I will be pushing the fix to Linus shortly. Oh, sorry, I didn't see your comment saying you were using the 2.6.27-rc3-ext4-1 patchset. Note that a key part of how to replicate this is almost certainly: EXT4-fs warning (device sdb5): ext4_fill_super: extents feature not enabled on this filesystem, use tune2fs. Given that you claimed you created the filesystem with: ./mkfs.ext4dev /dev/sdb5 ... that implies you didn't install the mke2fs.conf file from e2fsprogs 1.41.0, as instructed by the Ext2 HOWTO man page. Can you send the output of dumpe2fs -h of the filesystem? This also implies that "2> mount /dev/sdb5 /mnt/" almost certainly needed to be "mount -t ext4dev /dev/sdb5 /mnt", unless you don't have the ext3 filesystem built in your kernel. Using 2.6.27-rc3 + the 2.6.27-rc3-ext4-1 patchset applied, I can't reproduce the problem. I tried it with "mke2fs -t ext3; tune2fs -E test_fs" and then mounted an ext3 filesystem using ext4dev. I tried it with "mke2fs -t ext4dev -O ^extents" (since your kernel messages indicated that you didn't have extents enabled). And I tried it with "mke2fs -t ext4dev", which turned on the full set of ext4 features assuming that you properly installed the mke2fs.conf file. I was not able to get any crashes. Can you send us your mke2fs.conf file, and the output of dumpe2fs on the filesystem, so we can see what features were enabled. Also, how much memory do you have on your system? Hi Ted, Memory is 2GB of the machine and here is the output of mke2fs.conf what i have installed. [root@e326 ~]# cat /etc/mke2fs.conf [defaults] base_features = sparse_super,filetype,resize_inode,dir_index blocksize = 4096 inode_ratio = 8192 [fs_types] small = { blocksize = 1024 inode_ratio = 4096 } floppy = { blocksize = 1024 } news = { inode_ratio = 4096 } largefile = { inode_ratio = 1048576 } largefile4 = { inode_ratio = 4194304 } [root@e326 ~]# Meantime i am also trying to reproduce on some other i386 machine here. BTW i hope you are executing the same steps what i have given in the bug report. Steps to reproduce: 1> ./mkfs.ext4dev /dev/sdb5 2> mount /dev/sdb5 /mnt/ 3> cp -a / /mnt/ 4> After copying almost 1.6GB of data, it produces the call trace as above. Here i am trying to copy all the root filesystem content which is almost of 16GB on my machine and i have partitioned the ext4 filesystem with 34 GB of memory so there is no point of shortage of memory. [root@e326 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sdb5 68G 1.5G 63G 3% /mnt [root@e326 ~]# [root@e326 mnt]# ls autobench bin boot dev EXT2 EXT4 home lib lost+found misc mnt root SAMBA1 SAMBA3 srv sys tftpboot var I am not sure why it is generating call trace and after that i am not able to execute any command inside /mnt directory, it seems this bug is blocker for me and i am able to reproduce it evertime. I will be very happy if you want more detail. -Rishi Given your mke2fs.conf file, the sequence which you claim to be using can't possibly work. "mkfs.ext4dev /dev/sdb5" with your mke2fs.conf file will create an ext2 filesystem. It won't have the extents feature enabled, and it won't even have the journal feature enabled. If you then follow that up with "mount /dev/sdb5 /mnt", it must have been mounted as an ext2 filesystem. I've replicated that using your mke2fs.conf file, and that's what happens. This is why I asked you for the dumpe2fs -h file. If that is all you are literally typing, it can't possibly be testing the ext4dev filesystem, and it can't be possibly generating a stack trace with ext4 --- with the mke2fs.conf you have reported, and with the mount command you have claimed, it is guaranteed that you will create a filesystem that won't be supported by ext4. Can you also double check and confirm exactly what kernel you are using? I've tried multiple variants because your exact sequence of commands CAN'T POSSIBLY WORK AS AN EXT4 FILESYSTEM. If you want to convince me otherwise, please try sending me the output of this script: ---- cut here ------ #!/bin/sh -vx ./mkfs.ext4dev /dev/sdb5 dumpe2fs -h /dev/sdb5 blkid /dev/sdb5 mount /dev/sdb5 /mnt/ cat /proc/mounts ---- cut here ---- I did think of one possible way that this might have allowed the mkfs.ext4dev to work. Exactly how are you compiling e2fsprogs? Are you giving any options to the configure script? Most normal compiles don't, but it could be that you gave it an option which ended up forcing mke2fs to look for the mke2fs.conf file somewhere else, such as /usr/local/etc/mke2fs.conf. And if the file doesn't exist, it will fall back to a compiled-in default that will do something sane. But aside from that caveat, I can't think of any way you could be creating a valid ext4 filesystem with that mke2fs.conf file being installed in /etc/mke2fs.conf. yes, you are absolutely right, i am giving the option as prefix, while compiling e2fsprgs, and sorry for not noticing this in early stage, here is the mke2fs.conf file which is basically generated by e2fsprogs. [defaults] base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr blocksize = 4096 inode_size = 256 inode_ratio = 16384 [fs_types] ext3 = { features = has_journal } ext4 = { features = has_journal,extents,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize inode_size = 256 } ext4dev = { features = has_journal,extents,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize inode_size = 256 options = test_fs=1 } small = { blocksize = 1024 inode_size = 128 inode_ratio = 4096 } floppy = { blocksize = 1024 inode_size = 128 inode_ratio = 8192 } news = { inode_ratio = 4096 } largefile = { inode_ratio = 1048576 blocksize = -1 } largefile4 = { inode_ratio = 4194304 blocksize = -1 } hurd = { blocksize = 4096 inode_size = 128 } Can you just let me know with this config is it something wrong ? meanwhile i will try overwrite thos mke2fs.conf with /etc/mke2fs.conf and will try to regenrate the bug. -Rishi Well, if that's the mke2fs.conf you're using, that doesn't explain this error message which was in your bug report: EXT4-fs warning (device sdb5): ext4_fill_super: extents feature not enabled on this filesystem, use tune2fs. I will again ask you for the output of dumpe2fs, so I can see exactly what filesystem features are enabled on the filesystem in question. Here it is: [root@e326 sbin]# dumpe2fs -h /dev/sdb5 dumpe2fs 1.41-WIP (17-Jun-2008) Filesystem volume name: <none> Last mounted on: <not available> Filesystem UUID: dc85cb8f-438c-4a5a-a66e-18162e7f6469 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super Filesystem flags: signed_directory_hash test_filesystem Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 4480512 Block count: 71681964 Reserved block count: 3584098 Free blocks: 70501332 Free inodes: 4480501 First block: 1 Block size: 1024 Fragment size: 1024 Reserved GDT blocks: 256 Blocks per group: 8192 Fragments per group: 8192 Inodes per group: 512 Inode blocks per group: 128 Filesystem created: Fri Aug 22 07:46:54 2008 Last mount time: Fri Aug 22 07:47:47 2008 Last write time: Fri Aug 22 07:47:47 2008 Mount count: 1 Maximum mount count: 30 Last checked: Fri Aug 22 07:46:54 2008 Check interval: 15552000 (6 months) Next check after: Wed Feb 18 07:46:54 2009 Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode: 8 Default directory hash: tea Directory Hash Seed: 2c71215b-caf6-48db-9cee-a2105c734bad Journal backup: inode blocks Journal size: 32M [root@e326 sbin]# Hi Ted, Can you execute this step and see it is reproducible on your setup or not ? 1> ./mke2fs -E test_fs -j -I 256 -b 1024 /dev/sdb5 2> mount -t ext4dev /dev/sdb5 /mnt/ 3> go to /mnt directory 4> execute " cp -a / ." command 5> After copying of 1.6G you should get the call trace in dmesg. Note: I have also overwritten the /root/ext4-testcases/e2fsprogs-1.41.0/build/tmp/etc/mke2fs.conf to /etc/mke2fs.conf Here is the dumpe2fs output: [root@e326 sbin]# ./dumpe2fs -h /dev/sdb5 dumpe2fs 1.41.0 (10-Jul-2008) Filesystem volume name: <none> Last mounted on: <not available> Filesystem UUID: 04502aed-71d9-44da-bf86-f3e29f4c06e0 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super Filesystem flags: signed_directory_hash test_filesystem Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 4480512 Block count: 71681964 Reserved block count: 3584098 Free blocks: 68962148 Free inodes: 4454483 First block: 1 Block size: 1024 Fragment size: 1024 Reserved GDT blocks: 256 Blocks per group: 8192 Fragments per group: 8192 Inodes per group: 512 Inode blocks per group: 128 Filesystem created: Fri Aug 22 08:42:36 2008 Last mount time: Fri Aug 22 08:49:55 2008 Last write time: Fri Aug 22 08:49:55 2008 Mount count: 1 Maximum mount count: 38 Last checked: Fri Aug 22 08:42:36 2008 Check interval: 15552000 (6 months) Next check after: Wed Feb 18 08:42:36 2009 Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode: 8 Default directory hash: tea Directory Hash Seed: ecf9dcf0-d56e-4346-b81b-6690eaafa54f Journal backup: inode blocks Journal size: 32M [root@e326 sbin]# -Rishi Note: This bug is also reproducible with plane 2.6.27-rc3 kernel. Here is the bug report. Also i am giving the dumpe2fs below. Ted, Please let me know if my execution of steps are wrong somehwere the steps i am executing are: 1> ./mke2fs -E test_fs -j -I 256 -b 1024 /dev/sdb5 2> mount -t ext4dev /dev/sdb5 /mnt/ 3> go to /mnt directory 4> execute " cp -a / ." command 5> After copying of 1.6G you should get the call trace in dmesg. Call trace output: ------------[ cut here ]------------ kernel BUG at fs/jbd2/transaction.c:984! invalid opcode: 0000 [#1] SMP Modules linked in: ext4dev jbd2 crc16 ipv6 autofs4 hidp rfcomm l2cap bluetooth sunrpc dm_mirror dm_log dm_multipath dm_mod sbs sbshc battery ac parport_pc lp parport sg sr_mod ide_cd_mod cdrom serio_raw button tg3 libphy rtc_cmos rtc_core rtc_lib i2c_piix4 pcspkr i2c_core k8temp hwmon usb_storage sata_svw libata mptspi mptscsih scsi_transport_spi mptbase sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd Pid: 8978, comm: pdflush Not tainted (2.6.27-rc3-no-ext4-1 #3) EIP: 0060:[<f8ca26c8>] EFLAGS: 00010246 CPU: 0 EIP is at jbd2_journal_dirty_metadata+0x8e/0x113 [jbd2] EAX: 00000000 EBX: f0c70630 ECX: f0c70630 EDX: f701a600 ESI: d6bd0f74 EDI: f62d73b0 EBP: e359bc10 ESP: e359bbfc DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 Process pdflush (pid: 8978, ti=e359b000 task=f7d5bb40 task.ti=e359b000) Stack: f52f50c0 f701a600 f0c70630 f62d73b0 f0c70630 e359bc28 f8cc7bc1 f8cd1a1c f0c70630 0350ed0d e359bcc0 e359bd18 f8cb6f74 c048b80d 034f800a e359bcd4 f7426900 e359bd68 00000008 d6a26c98 00161c0c d6a26bec f62d73b0 e359bcb4 Call Trace: [<f8cc7bc1>] ? __ext4_journal_dirty_metadata+0x19/0x3c [ext4dev] [<f8cb6f74>] ? ext4_get_blocks_handle+0x680/0xab7 [ext4dev] [<c048b80d>] ? __find_get_block+0x147/0x151 [<f8cb748e>] ? ext4_get_blocks_wrap+0xe3/0x1ce [ext4dev] [<f8cb8fb6>] ? ext4_da_get_block_write+0x6b/0x16f [ext4dev] [<f8cb6492>] ? mpage_da_map_blocks+0x9a/0x364 [ext4dev] [<f8cba4e3>] ? ext4_da_writepages+0x13e/0x1b3 [ext4dev] [<c0629685>] ? schedule+0x73d/0x78b [<f8cb8f4b>] ? ext4_da_get_block_write+0x0/0x16f [ext4dev] [<f8cba3a5>] ? ext4_da_writepages+0x0/0x1b3 [ext4dev] [<c0456f7e>] ? do_writepages+0x23/0x34 [<c0487f1f>] ? __writeback_single_inode+0x128/0x24b [<c0457ae1>] ? pdflush+0x0/0x198 [<c04570a8>] ? get_dirty_limits+0x17/0x271 [<c0488341>] ? generic_sync_sb_inodes+0x190/0x251 [<c048840a>] ? sync_sb_inodes+0x8/0xa [<c04886cb>] ? writeback_inodes+0x51/0x91 [<c0457ae1>] ? pdflush+0x0/0x198 [<c04576bd>] ? background_writeout+0x73/0x9e [<c0457be1>] ? pdflush+0x100/0x198 [<c045764a>] ? background_writeout+0x0/0x9e [<c0434bf4>] ? kthread+0x3b/0x63 [<c0434bb9>] ? kthread+0x0/0x63 [<c04044cb>] ? kernel_thread_helper+0x7/0x10 ======================= Code: 90 8b 03 25 00 00 20 00 85 c0 75 f3 f0 0f ba 2b 15 19 c0 85 c0 75 ea 83 7e 0c 00 75 16 c7 46 0c 01 00 00 00 8b 47 04 85 c0 7f 04 <0f> 0b eb fe 48 89 47 04 8b 45 ec 39 46 18 75 12 83 7e 08 01 75 EIP: [<f8ca26c8>] jbd2_journal_dirty_metadata+0x8e/0x113 [jbd2] SS:ESP 0068:e359bbfc ---[ end trace bc1fde78e76f1de4 ]--- [root@e326 sbin]# ./dumpe2fs -h /dev/sdb5 dumpe2fs 1.41.0 (10-Jul-2008) Filesystem volume name: <none> Last mounted on: <not available> Filesystem UUID: 6d392a92-1257-43bd-acfa-28beba856e5e Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super Filesystem flags: signed_directory_hash test_filesystem Default mount options: (none) Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 4480512 Block count: 71681964 Reserved block count: 3584098 Free blocks: 68667276 Free inodes: 4453217 First block: 1 Block size: 1024 Fragment size: 1024 Reserved GDT blocks: 256 Blocks per group: 8192 Fragments per group: 8192 Inodes per group: 512 Inode blocks per group: 128 Filesystem created: Fri Aug 22 12:20:57 2008 Last mount time: Fri Aug 22 12:22:24 2008 Last write time: Fri Aug 22 12:22:24 2008 Mount count: 1 Maximum mount count: 30 Last checked: Fri Aug 22 12:20:57 2008 Check interval: 15552000 (6 months) Next check after: Wed Feb 18 12:20:57 2009 Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode: 8 Default directory hash: tea Directory Hash Seed: 5ac54fb6-1174-4f94-8f18-619e6c0a02d8 Journal backup: inode blocks Journal size: 32M [root@e326 sbin]# [root@e326 sbin]# uname -a Linux e326 2.6.27-rc3-no-ext4-1 #3 SMP Fri Aug 22 11:30:31 IST 2008 i686 athlon i386 GNU/Linux -Rishi OK, so I just tried to replicate using the mke2fs you gave in comment #13, with no luck. I was able to copy 6 gigabytes of data without any error before I got bored and gave up. How big is /dev/sdb5? Is this a failure in a disk full situation? Also, I'm really not interested about whether the bug exists in unmodified 2.6.27-rc3; we made enough changes in the relevent code that it's much more interesting if it occurs in with the 2.6.27-rc3-ext4-1 patchset --- especially since I'm about to push it to Linus, and I can't replicate the problem with the patchset. Are you ***sure*** you had the patchset enabled? Or is there something else unusual with your system? Hi Ted, I have tested today with 2.6.27-rc3 + http://repo.or.cz/w/ext4-patch-queue.git + 80639d3eefff3bb033d656e2dfe1b5a16b404180 , which has been given by aneesh. With this kernel i am not able to reproduce this bug, so i think now for me everything is working fine till i go more ahead from here :) I am closing this bug also. -Rishi |