Bug 12543

Summary: ext4_da_writepages error in 2.6.28.1 after a disk error
Product: File System Reporter: Andrew Baptist (abaptist)
Component: ext4Assignee: fs_ext4 (fs_ext4)
Status: CLOSED PATCH_ALREADY_AVAILABLE    
Severity: normal CC: diegocg
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 2.6.28.1 Subsystem:
Regression: --- Bisected commit-id:

Description Andrew Baptist 2009-01-26 08:56:38 UTC 
Distribution: 
custom stripped down kernel parameters

Hardware Environment: 
4 disk SATA system -

Software Environment:
Problem Description:
While running tests over the weekend one of the drives reported errors "Buffer I/O error on device". After this error, a series of "emerg" level stack traces were printed with the following stack:
Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: Pid: 13486, comm: pdflush Not tainted 2.6.28.1.cs.1 #1

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: Call Trace:

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c10e9168>] ext4_da_writepages+0x2d8/0x310

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c12d5b0b>] schedule+0x22b/0x880

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c10626ab>] do_writepages+0x2b/0x50

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c109ccb9>] __writeback_single_inode+0x89/0x300

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c109d2ee>] generic_sync_sb_inodes+0x1ce/0x2b0

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c109d738>] writeback_inodes+0x88/0xb0

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c10630f4>] wb_kupdate+0x84/0xf0

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c1063520>] pdflush+0x0/0x1a0

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c106360b>] pdflush+0xeb/0x1a0

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c1063070>] wb_kupdate+0x0/0xf0

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c1039b42>] kthread+0x42/0x70

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c1039b00>] kthread+0x0/0x70

Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel:  [<c1004aab>] kernel_thread_helper+0x7/0x1c

Steps to reproduce:
The syslog before these errors started was the following:
Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: Buffer I/O error on device sdd1, logical block 121667584

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: lost page write due to I/O error on sdd1

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: ext4_abort called.

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: EXT4-fs error (device sdd1): ext4_journal_start_sb: Detected aborted journal

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: Remounting filesystem read-only

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: EXT4-fs error (device sdd1) in ext4_delete_inode: Journal has aborted

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: EXT4-fs error (device sdd1) in ext4_create: IO failure

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: JBD2: I/O error detected when updating journal superblock for sdd1:8.

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: lost page write due to I/O error on sdd1

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK

Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: end_request: I/O error, dev sdd, sector 26623

 
One note is that the filesystem was NOT remounted read-only even though it claimed to be (at least by looking at mount).
Comment 1 Diego Calleja 2009-01-27 10:19:15 UTC 
IMHO I'd say there's a broken sector, have you passed a badblocks test? (but ext4 should have handled the failure gracefully)
Comment 2 Andrew Baptist 2009-01-27 10:26:05 UTC 
The disk was failing, really the purpose of this bug was just to have ext4 handle this gracefully. The continuous emerg level messages printed to console (about 2/sec) make it very hard to work with (at least with default of emerg going to *)
Comment 3 Theodore Tso 2009-01-27 11:08:12 UTC 
Having ext4 handling this gracefully is already in mainline.   See commit:

2a21e37e4: ext4: tone down ext4_da_writepages warnings


    ext4: tone down ext4_da_writepages warnings
    
    If the filesystem has errors, ext4_da_writepages() will return a *lot*
    of errors, including lots and lots of stack dumps.  While it's true
    that we are dropping user data on the floor, which is unfortunate, the
    stack dumps aren't helpful, and they tend to obscure the true original
    root cause of the problem.  So in the case where the filesystem has
    aborted, return an EROFS right away.
    
    Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>

It is in 2.6.29-rc1 and I plan to send it for the 2.6.28.y release, just because it is *so* annoying.