Bug 12594 - filesystem corruption while running bonnie
Summary: filesystem corruption while running bonnie
Status: CLOSED OBSOLETE
Alias: None
Product: File System
Classification: Unclassified
Component: ext3 (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: fs_ext3@kernel-bugs.osdl.org
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-02-01 06:59 UTC by Folkert van Heusden
Modified: 2012-05-30 12:45 UTC (History)
1 user (show)

See Also:
Kernel Version: 2.6.28.1
Subsystem:
Regression: No
Bisected commit-id:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Description Folkert van Heusden 2009-02-01 06:59:50 UTC 
mkdir /mnt/tmp /tmp/tmp ; mount -t tmpfs /dev/null /mnt
chown folkert /mnt/tmp /tmp/tmp

and then in parallel:
while [ 1 ] ; do bonnie++ -d /tmp/tmp -s 128 -r 64 -x 1 -u folkert ; done
while [ 1 ] ; do bonnie++ -d /mnt/tmp -s 128 -r 64 -x 1 -u folkert ; done

then after say 15 minutes i run the reboot command (normal via init) during the fsck (I always do a touch /forcefsck) I get:
fsck 1.41.3 (12-Oct-2008)
/dev/mapper/VGODR-tmp contains a file system with errors, check forced.
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (1) has invalid depth (2)
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (1) not referenced
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (2) has invalid depth (2)
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (2) not referenced
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (3) has invalid depth (2)
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (3) not referenced
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (4) has invalid depth (2)
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (4) not referenced
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (5) has invalid depth (2)
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (5) not referenced
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (6) has invalid depth (2)
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (6) not referenced
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (7) has invalid depth (2)
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (7) not referenced
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (8) has invalid depth (2)
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (8) has bad max hash
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (8) not referenced
...
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (523) not referenced
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (524) has invalid depth (2)
/dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (524) not referenced
/dev/mapper/VGODR-tmp: Invalid HTREE directory inode 69634 (/tmp/Bonnie.3813). 

I've got a gzip'ed dump of /tmp available on request.
Comment 1 Eric Sandeen 2009-02-01 08:14:58 UTC 
bugme-daemon@bugzilla.kernel.org wrote:

> mkdir /mnt/tmp /tmp/tmp ; mount -t tmpfs /dev/null /mnt
> chown folkert /mnt/tmp /tmp/tmp
> 
> and then in parallel:
> while [ 1 ] ; do bonnie++ -d /tmp/tmp -s 128 -r 64 -x 1 -u folkert ; done
> while [ 1 ] ; do bonnie++ -d /mnt/tmp -s 128 -r 64 -x 1 -u folkert ; done
> 
> then after say 15 minutes i run the reboot command (normal via init) during
> the
> fsck (I always do a touch /forcefsck) I get:
> fsck 1.41.3 (12-Oct-2008)
> /dev/mapper/VGODR-tmp contains a file system with errors, check forced.

Need to look at system logs before this to see why the fs has errors, I
think.

-Eric
Comment 2 Anonymous Emailer 2009-02-01 12:50:00 UTC 
Reply-To: akpm@linux-foundation.org


(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Sun,  1 Feb 2009 06:59:52 -0800 (PST) bugme-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=12594
> 
>            Summary: filesystem corruption while running bonnie
>            Product: File System
>            Version: 2.5
>      KernelVersion: 2.6.28.1
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: ext3
>         AssignedTo: fs_ext3@kernel-bugs.osdl.org
>         ReportedBy: folkert@vanheusden.com
> 
> 
> mkdir /mnt/tmp /tmp/tmp ; mount -t tmpfs /dev/null /mnt
> chown folkert /mnt/tmp /tmp/tmp
> 
> and then in parallel:
> while [ 1 ] ; do bonnie++ -d /tmp/tmp -s 128 -r 64 -x 1 -u folkert ; done
> while [ 1 ] ; do bonnie++ -d /mnt/tmp -s 128 -r 64 -x 1 -u folkert ; done
> 
> then after say 15 minutes i run the reboot command (normal via init) during
> the
> fsck (I always do a touch /forcefsck) I get:
> fsck 1.41.3 (12-Oct-2008)
> /dev/mapper/VGODR-tmp contains a file system with errors, check forced.
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (1) has
> invalid depth (2)
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (1) not
> referenced
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (2) has
> invalid depth (2)
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (2) not
> referenced
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (3) has
> invalid depth (2)
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (3) not
> referenced
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (4) has
> invalid depth (2)
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (4) not
> referenced
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (5) has
> invalid depth (2)
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (5) not
> referenced
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (6) has
> invalid depth (2)
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (6) not
> referenced
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (7) has
> invalid depth (2)
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (7) not
> referenced
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (8) has
> invalid depth (2)
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (8) has
> bad
> max hash
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (8) not
> referenced
> ...
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (523) not
> referenced
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (524) has
> invalid depth (2)
> /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (524) not
> referenced
> /dev/mapper/VGODR-tmp: Invalid HTREE directory inode 69634
> (/tmp/Bonnie.3813). 
> 
> I've got a gzip'ed dump of /tmp available on request.

htree went bad.  It's strange that exercising tmpfs would do this.

I always get worried when I see device-mapper in the picture.  It has a
history..  Perhaps you can describe your device-mapper setup?
Comment 3 Folkert van Heusden 2009-02-04 04:11:09 UTC 
> > http://bugzilla.kernel.org/show_bug.cgi?id=12594
> > 
> >            Summary: filesystem corruption while running bonnie
> >            Product: File System
> >            Version: 2.5
> >      KernelVersion: 2.6.28.1
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: ext3
> >         AssignedTo: fs_ext3@kernel-bugs.osdl.org
> >         ReportedBy: folkert@vanheusden.com
> > 
> > 
> > mkdir /mnt/tmp /tmp/tmp ; mount -t tmpfs /dev/null /mnt
> > chown folkert /mnt/tmp /tmp/tmp
> > 
> > and then in parallel:
> > while [ 1 ] ; do bonnie++ -d /tmp/tmp -s 128 -r 64 -x 1 -u folkert ; done
> > while [ 1 ] ; do bonnie++ -d /mnt/tmp -s 128 -r 64 -x 1 -u folkert ; done
> > 
> > then after say 15 minutes i run the reboot command (normal via init) during
> the
> > fsck (I always do a touch /forcefsck) I get:
> > fsck 1.41.3 (12-Oct-2008)
> > /dev/mapper/VGODR-tmp contains a file system with errors, check forced.
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (1) has
> > invalid depth (2)
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (1) not
> > referenced
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (2) has
> > invalid depth (2)
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (2) not
> > referenced
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (3) has
> > invalid depth (2)
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (3) not
> > referenced
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (4) has
> > invalid depth (2)
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (4) not
> > referenced
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (5) has
> > invalid depth (2)
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (5) not
> > referenced
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (6) has
> > invalid depth (2)
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (6) not
> > referenced
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (7) has
> > invalid depth (2)
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (7) not
> > referenced
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (8) has
> > invalid depth (2)
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (8) has
> bad
> > max hash
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (8) not
> > referenced
> > ...
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (523)
> not
> > referenced
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (524)
> has
> > invalid depth (2)
> > /dev/mapper/VGODR-tmp: Problem in HTREE directory inode 69634: node (524)
> not
> > referenced
> > /dev/mapper/VGODR-tmp: Invalid HTREE directory inode 69634
> (/tmp/Bonnie.3813). 
> > 
> > I've got a gzip'ed dump of /tmp available on request.
> 
> htree went bad.  It's strange that exercising tmpfs would do this.
> 
> I always get worried when I see device-mapper in the picture.  It has a
> history..  Perhaps you can describe your device-mapper setup?

(Sorry for the delayed response, had the flu)

The system is an LVM setup with /home, /, /tmp, /var, /usr and swap in
seperate LVM slices.


Folkert van Heusden
Comment 4 Folkert van Heusden 2009-02-05 12:00:13 UTC 
> > htree went bad.  It's strange that exercising tmpfs would do this.
> > 
> > I always get worried when I see device-mapper in the picture.  It has a
> > history..  Perhaps you can describe your device-mapper setup?
> 
> (Sorry for the delayed response, had the flu)
> 
> The system is an LVM setup with /home, /, /tmp, /var, /usr and swap in
> seperate LVM slices.

Some more details:
odr:/home/folkert# vgdisplay 
  --- Volume group ---
  VG Name               VGODR
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  7
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               6
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               37.03 GB
  PE Size               4.00 MB
  Total PE              9479
  Alloc PE / Size       9479 / 37.03 GB
  Free  PE / Size       0 / 0   
  VG UUID               dlQony-04VM-KDwW-2Qul-c3gG-qr6L-faKDMH


odr:/home/folkert# lvdisplay 
  --- Logical volume ---
  LV Name                /dev/VGODR/root
  VG Name                VGODR
  LV UUID                dpFEsy-myWC-YBhp-bw1t-lakV-rKwe-Pu2lDL
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                276.00 MB
  Current LE             69
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:0
   
  --- Logical volume ---
  LV Name                /dev/VGODR/usr
  VG Name                VGODR
  LV UUID                xJdCNc-k3iv-czxs-rUdd-tIi1-0Vab-Xk13U3
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                4.77 GB
  Current LE             1221
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:1
   
  --- Logical volume ---
  LV Name                /dev/VGODR/var
  VG Name                VGODR
  LV UUID                lbqzC7-NDIC-t9rT-Eqo2-1iQS-EadB-tnHkPq
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                2.86 GB
  Current LE             733
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:2
   
  --- Logical volume ---
  LV Name                /dev/VGODR/swap_1
  VG Name                VGODR
  LV UUID                olCdBr-asFX-3K32-zXQ8-6ee0-ou0M-PYbnro
  LV Write Access        read/write
  LV Status              available
  # open                 2
  LV Size                1.38 GB
  Current LE             354
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:3
   
  --- Logical volume ---
  LV Name                /dev/VGODR/tmp
  VG Name                VGODR
  LV UUID                nsFpW0-eUtE-F8nL-a4rN-80vQ-6mo2-EVbtph
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                392.00 MB
  Current LE             98
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:4
   
  --- Logical volume ---
  LV Name                /dev/VGODR/home
  VG Name                VGODR
  LV UUID                0l3HlK-rrsR-WWLZ-ZMip-bmHE-tRD2-sp7DAk
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                27.36 GB
  Current LE             7004
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:5
   


Folkert van Heusden
Note You need to log in before you can comment on or make changes to this bug.