Bug 11389

sorry, the kernel version is 2.6.27-rc3.

Here is the uname output:
Linux e326 2.6.27-rc3 #3 SMP Thu Aug 21 10:36:10 IST 2008 i686 athlon i386 GNU/Linux

Note: This kernel is 2.6.27-rc3 + ftp://ftp.kernel.org/pub/linux/kernel/people/tytso/ext4-patches/2.6.27-rc3-ext4-1/broken-out.tar.bz2 

- Rishi

here is the fsck output for this particular case:

[root@e326 sbin]# ./fsck.ext4dev -fnv /dev/sdb5
e2fsck 1.41.0 (10-Jul-2008)
Warning!  /dev/sdb5 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Inode 3147295, i_blocks is 105788, should be 105768.  Fix? no

Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -(50356371--50356372) -(50449009--50449016)
Fix? no

Free blocks count wrong (68960931, counted=68987599).
Fix? no


/dev/sdb5: ********** WARNING: Filesystem still has errors **********


   26027 inodes used (0.58%)
    5942 non-contiguous inodes (22.8%)
         # of inodes with ind/dind/tind blocks: 5922/918/1
 2721033 blocks used (3.80%)
       0 bad blocks
       0 large files

   20204 regular files
    4030 directories
     148 character device files
      35 block device files
       4 fifos
       2 links
    1587 symbolic links (1514 fast symbolic links)
      10 sockets
--------
   26020 files
[root@e326 sbin]#

This problem was reported last week, and already fixed in the ext4-patch-queue, and released in the 2.6.27-rc3-ext4-1 and 2.6.26-ext4-7 patchsets.  I will be pushing the fix to Linus shortly.

Oh, sorry, I didn't see your comment saying you were using the 2.6.27-rc3-ext4-1 patchset.

Note that a key part of how to replicate this is almost certainly:

EXT4-fs warning (device sdb5): ext4_fill_super: extents feature not enabled on
this filesystem, use tune2fs.

Given that you claimed you created the filesystem with:

./mkfs.ext4dev /dev/sdb5

... that implies you didn't install the mke2fs.conf file from e2fsprogs 1.41.0, as instructed by the Ext2 HOWTO man page.

Can you send the output of dumpe2fs -h of the filesystem?

This also implies that "2> mount /dev/sdb5 /mnt/" almost certainly needed to be "mount -t ext4dev /dev/sdb5 /mnt", unless you don't have the ext3 filesystem built in your kernel.

Using 2.6.27-rc3 + the 2.6.27-rc3-ext4-1 patchset applied, I can't reproduce the problem.  I tried it with "mke2fs -t ext3; tune2fs -E test_fs" and then mounted an ext3 filesystem using ext4dev.   I tried it with "mke2fs -t ext4dev -O ^extents" (since your kernel messages indicated that you didn't have extents enabled).   And I tried it with "mke2fs -t ext4dev", which turned on the full set of ext4 features assuming that you properly installed the mke2fs.conf file.  I was not able to get any crashes.   Can you send us your mke2fs.conf file, and the output of dumpe2fs on the filesystem, so we can see what features were enabled.  Also, how much memory do you have on your system?

Hi Ted,

Memory is 2GB of the machine and here is the output of mke2fs.conf what i have installed.

[root@e326 ~]# cat /etc/mke2fs.conf
[defaults]
        base_features = sparse_super,filetype,resize_inode,dir_index
        blocksize = 4096
        inode_ratio = 8192

[fs_types]
        small = {
                blocksize = 1024
                inode_ratio = 4096
        }
        floppy = {
                blocksize = 1024
        }
        news = {
                inode_ratio = 4096
        }
        largefile = {
                inode_ratio = 1048576
        }
        largefile4 = {
                inode_ratio = 4194304
        }
[root@e326 ~]#

Meantime i am also trying to reproduce on some other i386 machine here. BTW i hope you are executing the same steps what i have given in the bug report.

Steps to reproduce:
1> ./mkfs.ext4dev /dev/sdb5
2> mount /dev/sdb5 /mnt/
3> cp -a / /mnt/
4> After copying almost 1.6GB of data, it produces the call trace as above.

Here i am trying to copy all the root filesystem content which is almost of 16GB on my machine and i have partitioned the ext4 filesystem with 34 GB of memory so there is no point of shortage of memory.

[root@e326 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb5              68G  1.5G   63G   3% /mnt
[root@e326 ~]#

[root@e326 mnt]# ls
autobench  bin  boot  dev  EXT2  EXT4  home  lib  lost+found  misc  mnt  root  SAMBA1  SAMBA3  srv  sys  tftpboot  var


I am not sure why it is generating call trace and after that i am not able to execute any command inside /mnt directory, it seems this bug is blocker for me and i am able to reproduce it evertime.

I will be very happy if you want more detail.

-Rishi

 

Given your mke2fs.conf file, the sequence which you claim to be using can't possibly work.  "mkfs.ext4dev /dev/sdb5" with your mke2fs.conf file will create an ext2 filesystem.  It won't have the extents feature enabled, and it won't even have the journal feature enabled.  If you then follow that up with "mount /dev/sdb5 /mnt", it must have been mounted as an ext2 filesystem.   I've replicated that using your mke2fs.conf file, and that's what happens.

This is why I asked you for the dumpe2fs -h file.   If that is all you are literally typing, it can't possibly be testing the ext4dev filesystem, and it can't be possibly generating a stack trace with ext4 --- with the mke2fs.conf you have reported, and with the mount command you have claimed, it is guaranteed that you will create a filesystem that won't be supported by ext4.

Can you also double check and confirm exactly what kernel you are using?  I've tried multiple variants because your exact sequence of commands CAN'T POSSIBLY WORK AS AN EXT4 FILESYSTEM.

If you want to convince me otherwise, please try sending me the output of this script:

---- cut here ------
#!/bin/sh -vx
./mkfs.ext4dev /dev/sdb5
dumpe2fs -h /dev/sdb5
blkid /dev/sdb5
mount /dev/sdb5 /mnt/
cat /proc/mounts
---- cut here ----

I did think of one possible way that this might have allowed the mkfs.ext4dev to work.  Exactly how are you compiling e2fsprogs?   Are you giving any options to the configure script?   Most normal compiles don't, but it could be that you gave it an option which ended up forcing mke2fs to look for the mke2fs.conf file somewhere else, such as /usr/local/etc/mke2fs.conf.   And if the file doesn't exist, it will fall back to a compiled-in default that will do something sane.  But aside from that caveat, I can't think of any way you could be creating a valid ext4 filesystem with that mke2fs.conf file being installed in /etc/mke2fs.conf.

yes, you are absolutely right, i am giving the option as prefix, while compiling e2fsprgs, and sorry for not noticing this in early stage, here is the mke2fs.conf file which is basically generated by e2fsprogs.

[defaults]
        base_features = sparse_super,filetype,resize_inode,dir_index,ext_attr
        blocksize = 4096
        inode_size = 256
        inode_ratio = 16384

[fs_types]
        ext3 = {
                features = has_journal
        }
        ext4 = {
                features = has_journal,extents,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize
                inode_size = 256
        }
        ext4dev = {
                features = has_journal,extents,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize
                inode_size = 256
                options = test_fs=1
        }
        small = {
                blocksize = 1024
                inode_size = 128
                inode_ratio = 4096
        }
        floppy = {
                blocksize = 1024
                inode_size = 128
                inode_ratio = 8192
        }
        news = {
                inode_ratio = 4096
        }
        largefile = {
                inode_ratio = 1048576
                blocksize = -1
        }
        largefile4 = {
                inode_ratio = 4194304
                blocksize = -1
        }
        hurd = {
             blocksize = 4096
             inode_size = 128
        }

Can you just let me know with this config is it something wrong ? meanwhile i will try overwrite thos mke2fs.conf with /etc/mke2fs.conf and will try to regenrate the bug.

-Rishi

Well, if that's the mke2fs.conf you're using, that doesn't explain this error message which was in your bug report:

EXT4-fs warning (device sdb5): ext4_fill_super: extents feature not enabled on
this filesystem, use tune2fs.

I will again ask you for the output of dumpe2fs, so I can see exactly what filesystem features are enabled on the filesystem in question.

Here it is:

[root@e326 sbin]# dumpe2fs -h /dev/sdb5
dumpe2fs 1.41-WIP (17-Jun-2008)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          dc85cb8f-438c-4a5a-a66e-18162e7f6469
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super
Filesystem flags:         signed_directory_hash test_filesystem
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              4480512
Block count:              71681964
Reserved block count:     3584098
Free blocks:              70501332
Free inodes:              4480501
First block:              1
Block size:               1024
Fragment size:            1024
Reserved GDT blocks:      256
Blocks per group:         8192
Fragments per group:      8192
Inodes per group:         512
Inode blocks per group:   128
Filesystem created:       Fri Aug 22 07:46:54 2008
Last mount time:          Fri Aug 22 07:47:47 2008
Last write time:          Fri Aug 22 07:47:47 2008
Mount count:              1
Maximum mount count:      30
Last checked:             Fri Aug 22 07:46:54 2008
Check interval:           15552000 (6 months)
Next check after:         Wed Feb 18 07:46:54 2009
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      2c71215b-caf6-48db-9cee-a2105c734bad
Journal backup:           inode blocks
Journal size:             32M

[root@e326 sbin]#

Hi Ted,

Can you execute this step and see it is reproducible on your setup or not ?

1> ./mke2fs -E test_fs -j -I 256 -b 1024  /dev/sdb5
2> mount -t ext4dev /dev/sdb5 /mnt/
3> go to /mnt directory
4> execute " cp -a / ." command
5> After copying of 1.6G you should get the call trace in dmesg.

Note: I have also overwritten the /root/ext4-testcases/e2fsprogs-1.41.0/build/tmp/etc/mke2fs.conf to /etc/mke2fs.conf 


Here is the dumpe2fs output:
[root@e326 sbin]# ./dumpe2fs -h /dev/sdb5
dumpe2fs 1.41.0 (10-Jul-2008)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          04502aed-71d9-44da-bf86-f3e29f4c06e0
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super
Filesystem flags:         signed_directory_hash test_filesystem
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              4480512
Block count:              71681964
Reserved block count:     3584098
Free blocks:              68962148
Free inodes:              4454483
First block:              1
Block size:               1024
Fragment size:            1024
Reserved GDT blocks:      256
Blocks per group:         8192
Fragments per group:      8192
Inodes per group:         512
Inode blocks per group:   128
Filesystem created:       Fri Aug 22 08:42:36 2008
Last mount time:          Fri Aug 22 08:49:55 2008
Last write time:          Fri Aug 22 08:49:55 2008
Mount count:              1
Maximum mount count:      38
Last checked:             Fri Aug 22 08:42:36 2008
Check interval:           15552000 (6 months)
Next check after:         Wed Feb 18 08:42:36 2009
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      ecf9dcf0-d56e-4346-b81b-6690eaafa54f
Journal backup:           inode blocks
Journal size:             32M

[root@e326 sbin]#

-Rishi

Note: This bug is also reproducible with plane 2.6.27-rc3 kernel. Here is the bug report. Also i am giving the dumpe2fs below.

Ted, Please let me know if my execution of steps are wrong somehwere the steps i am executing are:

1> ./mke2fs -E test_fs -j -I 256 -b 1024  /dev/sdb5
2> mount -t ext4dev /dev/sdb5 /mnt/
3> go to /mnt directory
4> execute " cp -a / ." command
5> After copying of 1.6G you should get the call trace in dmesg.

Call trace output:

------------[ cut here ]------------
kernel BUG at fs/jbd2/transaction.c:984!
invalid opcode: 0000 [#1] SMP
Modules linked in: ext4dev jbd2 crc16 ipv6 autofs4 hidp rfcomm l2cap bluetooth sunrpc dm_mirror dm_log dm_multipath dm_mod sbs sbshc battery ac parport_pc lp parport sg sr_mod ide_cd_mod cdrom serio_raw button tg3 libphy rtc_cmos rtc_core rtc_lib i2c_piix4 pcspkr i2c_core k8temp hwmon usb_storage sata_svw libata mptspi mptscsih scsi_transport_spi mptbase sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd

Pid: 8978, comm: pdflush Not tainted (2.6.27-rc3-no-ext4-1 #3)
EIP: 0060:[<f8ca26c8>] EFLAGS: 00010246 CPU: 0
EIP is at jbd2_journal_dirty_metadata+0x8e/0x113 [jbd2]
EAX: 00000000 EBX: f0c70630 ECX: f0c70630 EDX: f701a600
ESI: d6bd0f74 EDI: f62d73b0 EBP: e359bc10 ESP: e359bbfc
 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process pdflush (pid: 8978, ti=e359b000 task=f7d5bb40 task.ti=e359b000)
Stack: f52f50c0 f701a600 f0c70630 f62d73b0 f0c70630 e359bc28 f8cc7bc1 f8cd1a1c
       f0c70630 0350ed0d e359bcc0 e359bd18 f8cb6f74 c048b80d 034f800a e359bcd4
       f7426900 e359bd68 00000008 d6a26c98 00161c0c d6a26bec f62d73b0 e359bcb4
Call Trace:
 [<f8cc7bc1>] ? __ext4_journal_dirty_metadata+0x19/0x3c [ext4dev]
 [<f8cb6f74>] ? ext4_get_blocks_handle+0x680/0xab7 [ext4dev]
 [<c048b80d>] ? __find_get_block+0x147/0x151
 [<f8cb748e>] ? ext4_get_blocks_wrap+0xe3/0x1ce [ext4dev]
 [<f8cb8fb6>] ? ext4_da_get_block_write+0x6b/0x16f [ext4dev]
 [<f8cb6492>] ? mpage_da_map_blocks+0x9a/0x364 [ext4dev]
 [<f8cba4e3>] ? ext4_da_writepages+0x13e/0x1b3 [ext4dev]
 [<c0629685>] ? schedule+0x73d/0x78b
 [<f8cb8f4b>] ? ext4_da_get_block_write+0x0/0x16f [ext4dev]
 [<f8cba3a5>] ? ext4_da_writepages+0x0/0x1b3 [ext4dev]
 [<c0456f7e>] ? do_writepages+0x23/0x34
 [<c0487f1f>] ? __writeback_single_inode+0x128/0x24b
 [<c0457ae1>] ? pdflush+0x0/0x198
 [<c04570a8>] ? get_dirty_limits+0x17/0x271
 [<c0488341>] ? generic_sync_sb_inodes+0x190/0x251
 [<c048840a>] ? sync_sb_inodes+0x8/0xa
 [<c04886cb>] ? writeback_inodes+0x51/0x91
 [<c0457ae1>] ? pdflush+0x0/0x198
 [<c04576bd>] ? background_writeout+0x73/0x9e
 [<c0457be1>] ? pdflush+0x100/0x198
 [<c045764a>] ? background_writeout+0x0/0x9e
 [<c0434bf4>] ? kthread+0x3b/0x63
 [<c0434bb9>] ? kthread+0x0/0x63
 [<c04044cb>] ? kernel_thread_helper+0x7/0x10
 =======================
Code: 90 8b 03 25 00 00 20 00 85 c0 75 f3 f0 0f ba 2b 15 19 c0 85 c0 75 ea 83 7e 0c 00 75 16 c7 46 0c 01 00 00 00 8b 47 04 85 c0 7f 04 <0f> 0b eb fe 48 89 47 04 8b 45 ec 39 46 18 75 12 83 7e 08 01 75
EIP: [<f8ca26c8>] jbd2_journal_dirty_metadata+0x8e/0x113 [jbd2] SS:ESP 0068:e359bbfc
---[ end trace bc1fde78e76f1de4 ]---







[root@e326 sbin]# ./dumpe2fs -h /dev/sdb5
dumpe2fs 1.41.0 (10-Jul-2008)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          6d392a92-1257-43bd-acfa-28beba856e5e
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super
Filesystem flags:         signed_directory_hash test_filesystem
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              4480512
Block count:              71681964
Reserved block count:     3584098
Free blocks:              68667276
Free inodes:              4453217
First block:              1
Block size:               1024
Fragment size:            1024
Reserved GDT blocks:      256
Blocks per group:         8192
Fragments per group:      8192
Inodes per group:         512
Inode blocks per group:   128
Filesystem created:       Fri Aug 22 12:20:57 2008
Last mount time:          Fri Aug 22 12:22:24 2008
Last write time:          Fri Aug 22 12:22:24 2008
Mount count:              1
Maximum mount count:      30
Last checked:             Fri Aug 22 12:20:57 2008
Check interval:           15552000 (6 months)
Next check after:         Wed Feb 18 12:20:57 2009
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   tea
Directory Hash Seed:      5ac54fb6-1174-4f94-8f18-619e6c0a02d8
Journal backup:           inode blocks
Journal size:             32M

[root@e326 sbin]#


[root@e326 sbin]# uname -a
Linux e326 2.6.27-rc3-no-ext4-1 #3 SMP Fri Aug 22 11:30:31 IST 2008 i686 athlon i386 GNU/Linux


-Rishi

OK, so I just tried to replicate using the mke2fs you gave in comment #13, with no luck.  I was able to copy 6 gigabytes of data without any error before I got bored and gave up.

How big is /dev/sdb5?   Is this a failure in a disk full situation?   

Also, I'm really not interested about whether the bug exists in unmodified 2.6.27-rc3; we made enough changes in the relevent code that it's much more interesting if it occurs in with the 2.6.27-rc3-ext4-1 patchset --- especially since I'm about to push it to Linus, and I can't replicate the problem with the patchset.  Are you ***sure*** you had the patchset enabled?  Or is there something else unusual with your system?

Hi Ted,

I have tested today with 2.6.27-rc3 + http://repo.or.cz/w/ext4-patch-queue.git + 80639d3eefff3bb033d656e2dfe1b5a16b404180 , which has been given by aneesh. With this kernel i am not able to reproduce this bug, so i think now for me everything is working fine till i go more ahead from here :)

I am closing this bug also.

-Rishi