Two hours ago my database server stopped working. I turned out to be a problem with ext4: pa ffff88063643bd00: logic 1024, phys. 2677760, len 1024 EXT4-fs error (device dm-3): ext4_mb_release_inode_pa: free 777, pa_free 778 Aborting journal on device dm-3:8. Remounting filesystem read-only The problem was fixed by: # fsck /dev/VG0/sql fsck from util-linux-ng 2.16.1 e2fsck 1.41.9 (22-Aug-2009) /dev/mapper/VG0-sql_tmp: recovering journal /dev/mapper/VG0-sql_tmp contains a file system with errors, check forced. Pass 1: Checking inodes, blocks, and sizes Deleted inode 66 has zero dtime. Fix<y>? yes Inodes that were part of a corrupted orphan linked list found. Fix<y>? yes Inode 67 was part of the orphaned inode list. FIXED. Inode 68 was part of the orphaned inode list. FIXED. Inode 69 was part of the orphaned inode list. FIXED. Inode 2975 was part of the orphaned inode list. FIXED. Deleted inode 3731 has zero dtime. Fix<y>? yes Inode 3732 was part of the orphaned inode list. FIXED. Inode 3733 was part of the orphaned inode list. FIXED. Inode 3734 was part of the orphaned inode list. FIXED. Inode 3735 was part of the orphaned inode list. FIXED. Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Block bitmap differences: -2678117 Fix<y>? yes Inode bitmap differences: -(66--69) -2975 -(3731--3735) Fix<y>? yes Free inodes count wrong for group #0 (4457, counted=4467). Fix<y>? yes Free inodes count wrong (2093400, counted=2093410). Fix<y>? yes /dev/mapper/VG0-sql_tmp: ***** FILE SYSTEM WAS MODIFIED ***** /dev/mapper/VG0-sql_tmp: 3742/2097152 files (19.4% non-contiguous), 963461/8388608 blocks Before I started the fsck I had made a copy of that fs, so I'm still able to provide a dump if necessary.
My fs has just crashed again: pa ffff88063643b410: logic 1024, phys. 8079360, len 1024 EXT4-fs error (device dm-3): ext4_mb_release_inode_pa: free 1000, pa_free 1001 Aborting journal on device dm-3:8. Remounting filesystem read-only # fsck -f /var/sql fsck from util-linux-ng 2.16.1 e2fsck 1.41.9 (22-Aug-2009) /dev/mapper/VG0-sql: recovering journal Clearing orphaned inode 2975 (uid=60, gid=60, mode=0100600, size=0) Clearing orphaned inode 69 (uid=60, gid=60, mode=0100600, size=0) Clearing orphaned inode 68 (uid=60, gid=60, mode=0100600, size=0) Clearing orphaned inode 67 (uid=60, gid=60, mode=0100600, size=20) Clearing orphaned inode 66 (uid=60, gid=60, mode=0100600, size=0) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Block bitmap differences: -8080068 Fix<y>? yes /dev/mapper/VG0-sql: ***** FILE SYSTEM WAS MODIFIED ***** /dev/mapper/VG0-sql: 6099/2097152 files (22.8% non-contiguous), 1069002/8388608 blocks
Closing as obsolete - ext4 has changed extensively since 2.6.30 If this is still seen with modern kernels please update the bug