Distribution: custom stripped down kernel parameters Hardware Environment: 4 disk SATA system - Software Environment: Problem Description: While running tests over the weekend one of the drives reported errors "Buffer I/O error on device". After this error, a series of "emerg" level stack traces were printed with the following stack: Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: Pid: 13486, comm: pdflush Not tainted 2.6.28.1.cs.1 #1 Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: Call Trace: Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: [<c10e9168>] ext4_da_writepages+0x2d8/0x310 Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: [<c12d5b0b>] schedule+0x22b/0x880 Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: [<c10626ab>] do_writepages+0x2b/0x50 Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: [<c109ccb9>] __writeback_single_inode+0x89/0x300 Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: [<c109d2ee>] generic_sync_sb_inodes+0x1ce/0x2b0 Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: [<c109d738>] writeback_inodes+0x88/0xb0 Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: [<c10630f4>] wb_kupdate+0x84/0xf0 Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: [<c1063520>] pdflush+0x0/0x1a0 Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: [<c106360b>] pdflush+0xeb/0x1a0 Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: [<c1063070>] wb_kupdate+0x0/0xf0 Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: [<c1039b42>] kthread+0x42/0x70 Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: [<c1039b00>] kthread+0x0/0x70 Jan 26 15:06:13 qwchi-mbxss-pd52.cleversafelabs.com kernel: [<c1004aab>] kernel_thread_helper+0x7/0x1c Steps to reproduce: The syslog before these errors started was the following: Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: Buffer I/O error on device sdd1, logical block 121667584 Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: lost page write due to I/O error on sdd1 Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: ext4_abort called. Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: EXT4-fs error (device sdd1): ext4_journal_start_sb: Detected aborted journal Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: Remounting filesystem read-only Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: EXT4-fs error (device sdd1) in ext4_delete_inode: Journal has aborted Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: EXT4-fs error (device sdd1) in ext4_create: IO failure Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: JBD2: I/O error detected when updating journal superblock for sdd1:8. Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: lost page write due to I/O error on sdd1 Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: sd 3:0:0:0: [sdd] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK,SUGGEST_OK Jan 26 08:12:21 qwchi-mbxss-pd52.cleversafelabs.com kernel: end_request: I/O error, dev sdd, sector 26623 One note is that the filesystem was NOT remounted read-only even though it claimed to be (at least by looking at mount).
IMHO I'd say there's a broken sector, have you passed a badblocks test? (but ext4 should have handled the failure gracefully)
The disk was failing, really the purpose of this bug was just to have ext4 handle this gracefully. The continuous emerg level messages printed to console (about 2/sec) make it very hard to work with (at least with default of emerg going to *)
Having ext4 handling this gracefully is already in mainline. See commit: 2a21e37e4: ext4: tone down ext4_da_writepages warnings ext4: tone down ext4_da_writepages warnings If the filesystem has errors, ext4_da_writepages() will return a *lot* of errors, including lots and lots of stack dumps. While it's true that we are dropping user data on the floor, which is unfortunate, the stack dumps aren't helpful, and they tend to obscure the true original root cause of the problem. So in the case where the filesystem has aborted, return an EROFS right away. Signed-off-by: "Theodore Ts'o" <tytso@mit.edu> It is in 2.6.29-rc1 and I plan to send it for the 2.6.28.y release, just because it is *so* annoying.