Bug 13964

Summary: ext4: panic causes lost data in git
Product: File System Reporter: Jesse Brandeburg (jbrandeb)
Component: ext4Assignee: fs_ext4 (fs_ext4)
Status: CLOSED WILL_NOT_FIX    
Severity: high CC: alan, devzero, sandeen, tytso
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 2.6.31-rc5 Subsystem:
Regression: No Bisected commit-id:

Description Jesse Brandeburg 2009-08-11 18:58:35 UTC 
When working within a local git tree, I committed a change, compiled and reloaded my driver, which caused a panic.

After reboot my .git/objects/xx/xxxx (latest commit) had empty files.  This doesn't seem to be an acceptable loss of data.  I've since started mounting /home with data=ordered in the hope it would help.

I've mounted my LVM / and /home partitions with the default options

here is the output of tune2fs on /home
[root@jbrandeb-hc jbrandeb]# tune2fs -l /dev/mapper/vg_jbrandebhc-HomeVol
tune2fs 1.41.4 (27-Jan-2009)
Filesystem volume name:   <none>
Last mounted on:          /home
Filesystem UUID:          696e2e7d-467d-47b3-992e-a28dcf9b1ef8
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              8454144
Block count:              33792000
Reserved block count:     1689600
Free blocks:              32742879
Free inodes:              8386475
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1015
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Tue Jun 23 09:50:14 2009
Last mount time:          Tue Aug 11 11:55:14 2009
Last write time:          Tue Aug 11 11:55:14 2009
Mount count:              1
Maximum mount count:      -1
Last checked:             Tue Aug 11 11:55:06 2009
Check interval:           0 (<none>)
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      25ca440e-1707-4084-ac05-3addd26ad952
Journal backup:           inode blocks

Any other information will be gladly provided.
Comment 1 Jesse Brandeburg 2009-08-11 19:02:48 UTC 
related info, I'll try the alloc_on_commit option
http://www.h-online.com/open/Ext4-data-loss-explanations-and-workarounds--/news/112892
Comment 2 Roland Kletzing 2009-08-28 19:34:16 UTC 
i also have seen this issue, but i can`t find alloc_on_commit option in latest .git kernel.  hasn`t the appropriate patch been merged?
Comment 3 Theodore Tso 2009-08-29 02:47:23 UTC 
The alloc_on_commit patch had a number of deadlock problems, and I've never had time to try to work them out.

For now, the recommended solution for git is to put the following in your .gitconfig file:

[core]
     	fsyncobjectfiles = yes

This will cause git to force an fsync() when writing object files, which is the only portable way of guaranteeing the object files will be written after a crash.  This is needed for safety reasons for ext4, xfs, and btrfs filesystems.
Comment 4 Roland Kletzing 2009-08-29 13:06:46 UTC 
ok, but what is the solution for all the apps which don`t have issues with ext3 but start having issues with ext4? fix all the apps? that would take a long time...   alloc_on_commit really sounds like a good compatibility switch....
Comment 5 Jesse Brandeburg 2010-05-25 17:56:58 UTC 
I was unable to continue using ext4 for development since it was not reliable through panic.  Was there ever any progress on this issue?  Due to this bug I have been telling everyone I work with to not use ext4.

It is a pretty easy bug to reproduce, isn't there something we can do about it?
Comment 6 Eric Sandeen 2010-05-25 18:02:53 UTC 
Jesse, are you still running with data=writeback (as comment nr. 1 seemed to indicate, since you said you would -try- data=ordered...) - data=writeback is just not safe at all for anybody, anywhere, any time, unless you plan to recreate all data post-crash.

With default mount options (which includes data=ordered) do you still see this problem on upstream kernels?

In the end, though, you have to realize that buffered data which is not synced to disk -will- be lost on a crash.  This is true for all filesystems, though the window of "opportunity" may differ....