Just had a Raspberry Pi crash, and upon reboot fsck.ext4 was unable to repair the filesystem. Moving the SD card to a PC and running fsck there showed the same behaviour. I am not sure what exactly you need to know so please advise if you need anything else. The reason for logging this bug is that I can't work out why fsck.ext4 is telling me "unable to set superblock" when I can successfully mount the filesystem and write to it without problem, however I can't boot the Raspberry Pi from it because it tries to run fsck on every boot which fails, falling back to single user mode instead of booting properly. I also don't understand why the old journal apparently remains even after cleanly unmounting the filesystem, making it look like there are still errors and causing fsck to keep failing. $ fsck.ext4 /dev/sdb2 e2fsck 1.43.4 (31-Jan-2017) /dev/sdb2: recovering journal Superblock needs_recovery flag is clear, but journal has data. Run journal anyway<y>? yes fsck.ext4: unable to set superblock flags on /dev/sdb2 /dev/sdb2: ********** WARNING: Filesystem still has errors ********** $ fsck.ext4 /dev/sdb2 e2fsck 1.43.4 (31-Jan-2017) /dev/sdb2: recovering journal Superblock needs_recovery flag is clear, but journal has data. Run journal anyway<y>? no Clear journal<y>? yes fsck.ext4: unable to set superblock flags on /dev/sdb2 /dev/sdb2: ********** WARNING: Filesystem still has errors ********** $ fsck.ext4 /dev/sdb2 e2fsck 1.43.4 (31-Jan-2017) /dev/sdb2: recovering journal Superblock needs_recovery flag is clear, but journal has data. Run journal anyway<y>? no Clear journal<y>? no fsck.ext4: unable to set superblock flags on /dev/sdb2 /dev/sdb2: ********** WARNING: Filesystem still has errors ********** $ mkfs.ext4 -n /dev/sdb2 mke2fs 1.43.4 (31-Jan-2017) /dev/sdb2 contains a ext4 file system last mounted on / on Sun Aug 13 19:58:40 2017 Proceed anyway? (y,N) y Creating filesystem with 3885568 4k blocks and 972944 inodes Filesystem UUID: b0aabda8-ecd4-49a7-8d49-847fdf66b795 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208 $ fsck.ext4 -b 98304 /dev/sdb2 e2fsck 1.43.4 (31-Jan-2017) Superblock needs_recovery flag is clear, but journal has data. Recovery flag not set in backup superblock, so running journal anyway. /dev/sdb2: recovering journal Superblock needs_recovery flag is clear, but journal has data. Recovery flag not set in backup superblock, so running journal anyway. Superblock needs_recovery flag is clear, but journal has data. Recovery flag not set in backup superblock, so running journal anyway. fsck.ext4: unable to set superblock flags on /dev/sdb2 /dev/sdb2: ***** FILE SYSTEM WAS MODIFIED ***** /dev/sdb2: ********** WARNING: Filesystem still has errors ********** $ fsck.ext4 -b 163840 /dev/sdb2 e2fsck 1.43.4 (31-Jan-2017) Superblock needs_recovery flag is clear, but journal has data. Recovery flag not set in backup superblock, so running journal anyway. /dev/sdb2: recovering journal Superblock needs_recovery flag is clear, but journal has data. Recovery flag not set in backup superblock, so running journal anyway. Superblock needs_recovery flag is clear, but journal has data. Recovery flag not set in backup superblock, so running journal anyway. fsck.ext4: unable to set superblock flags on /dev/sdb2 /dev/sdb2: ***** FILE SYSTEM WAS MODIFIED ***** /dev/sdb2: ********** WARNING: Filesystem still has errors ********** $ mkdir temp $ mount /dev/sdb2 temp $ ls temp total 80K drwxr-xr-x 17 root root 4.0K Jul 26 11:29 . drwxr-x--- 29 root root 4.0K Aug 16 18:05 .. lrwxrwxrwx 1 root root 7 Apr 1 11:34 bin -> usr/bin drwxr-xr-x 2 root root 4.0K Mar 11 2016 boot drwxr-xr-x 2 root root 4.0K Mar 1 2016 dev drwxr-xr-x 45 root root 4.0K Aug 16 15:44 etc drwxr-xr-x 3 root root 4.0K Mar 11 2016 home lrwxrwxrwx 1 root root 7 Apr 1 11:34 lib -> usr/lib drwx------ 2 root root 16K Mar 11 2016 lost+found drwxr-xr-x 2 root root 4.0K Oct 6 2015 mnt drwxr-xr-x 3 root root 4.0K Mar 1 2016 opt dr-xr-xr-x 2 root root 4.0K Mar 1 2016 proc drwxr-x--- 8 root root 4.0K Aug 16 15:44 root drwxr-xr-x 2 root root 4.0K Mar 1 2016 run lrwxrwxrwx 1 root root 7 Apr 1 11:34 sbin -> usr/bin drwxr-xr-x 4 root root 4.0K Oct 6 2015 srv dr-xr-xr-x 2 root root 4.0K Mar 1 2016 sys drwxrwxrwt 2 root root 4.0K Mar 1 2016 tmp drwxr-xr-x 8 root root 4.0K Aug 16 15:44 usr drwxr-xr-x 12 root root 4.0K Jul 26 11:29 var $ umount temp $ fsck.ext4 /dev/sdb2 e2fsck 1.43.4 (31-Jan-2017) /dev/sdb2: recovering journal Superblock needs_recovery flag is clear, but journal has data. Run journal anyway<y>? no Clear journal<y>? yes fsck.ext4: unable to set superblock flags on /dev/sdb2 /dev/sdb2: ********** WARNING: Filesystem still has errors ********** $ dmesg | tail [881102.628229] scsi host7: usb-storage 3-1:1.0 [881103.655218] scsi 7:0:0:0: Direct-Access Generic STORAGE DEVICE 0821 PQ: 0 ANSI: 6 [881103.655493] sd 7:0:0:0: Attached scsi generic sg2 type 0 [881104.004041] sd 7:0:0:0: [sdb] 31291392 512-byte logical blocks: (16.0 GB/14.9 GiB) [881104.004576] sd 7:0:0:0: [sdb] Write Protect is off [881104.004579] sd 7:0:0:0: [sdb] Mode Sense: 23 00 00 00 [881104.005161] sd 7:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA [881104.008549] sdb: sdb1 sdb2 [881104.010750] sd 7:0:0:0: [sdb] Attached SCSI removable disk [881151.920150] EXT4-fs (sdb2): recovery complete [881151.920669] EXT4-fs (sdb2): mounted filesystem with ordered data mode. Opts: (null) [881178.944509] EXT4-fs (sdb2): recovery complete [881178.944973] EXT4-fs (sdb2): mounted filesystem with ordered data mode. Opts: (null) [881658.714126] EXT4-fs (sdb2): recovery complete [881658.714609] EXT4-fs (sdb2): mounted filesystem with ordered data mode. Opts: (null) [882732.745664] EXT4-fs (sdb2): recovery complete [882732.746169] EXT4-fs (sdb2): mounted filesystem with ordered data mode. Opts: (null) $ strace fsck.ext4 -y /dev/sdb2 ... write(1, "Run ", 4Run ) = 4 write(1, "journal", 7journal) = 7 write(1, " anyway", 7 anyway) = 7 write(1, "? yes\n\n", 7? yes ) = 7 pwrite64(4, "\266\3\0\0\306\3\0\0\326\3\0\0\21#\227\24$\1\4\0\0\0\0\0\0\0\0\0R\v\343\273"..., 4096, 4096) = 4096 pwrite64(4, "\300;9\230\0\0\0\4\0\0\0\0\0\0\20\0\0\0\200\0\0\0\0\1\0\0\221o\0\0@,"..., 4096, 6580559872) = 4096 fsync(4) = 0 lseek(4, 1072, SEEK_SET) = 1072 write(4, "\247\375\223Y", 4) = 4 lseek(4, 1400, SEEK_SET) = 1400 write(4, "E\321", 2) = 2 fsync(4) = 0 close(4) = 0 open("/dev/sdb2", O_RDWR|O_EXCL) = 4 getuid() = 0 geteuid() = 0 getgid() = 0 getegid() = 0 prctl(PR_GET_DUMPABLE) = 1 (SUID_DUMP_USER) fstat(4, {st_mode=S_IFBLK|0660, st_rdev=makedev(8, 18), ...}) = 0 ioctl(4, BLKDISCARDZEROES, [0]) = 0 ioctl(4, BLKROGET, [0]) = 0 uname({sysname="Linux", nodename="korath.teln.shikadi.net", ...}) = 0 pread64(4, "\220\330\16\0\0J;\0\346\366\2\0\233\217,\0\312\311\r\0\0\0\0\0\2\0\0\0\2\0\0\0"..., 1024, 1024) = 1024 pread64(4, "\266\3\0\0\306\3\0\0\326\3\0\0\21#\227\24$\1\4\0\0\0\0\0\0\0\0\0R\v\343\273"..., 4096, 4096) = 4096 pread64(4, "\0\0\0\0\0\0\0\0\330\241\342V\330\241\342V\330\241\342V\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 4022272) = 4096 pread64(4, "\300;9\230\0\0\0\4\0\0\0\0\0\0\20\0\0\0\200\0\0\0\0\1\0\0\221o\0\0@,"..., 4096, 6580559872) = 4096 write(2, "fsck.ext4", 9fsck.ext4) = 9 write(2, ": ", 2: ) = 2 write(2, "unable to set superblock flags o"..., 44unable to set superblock flags on /dev/sdb2 ) = 44 ...
On Wed, Aug 16, 2017 at 08:14:26AM +0000, bugzilla-daemon@bugzilla.kernel.org wrote: > > Just had a Raspberry Pi crash, and upon reboot fsck.ext4 was unable to repair > the filesystem. Moving the SD card to a PC and running fsck there showed the > same behaviour. I am not sure what exactly you need to know so please advise > if you need anything else. This is a e2fsprogs bugs, not a kernel bug, but it's fine to discuss it here. Can you send me the output of "dumpe2fs /dev/sdb2", "debugfs -R 'stat <8>' /dev/sdb2", and "debugfs -R logdump /dev/sdb2"? Also could you run: debugfs -R 'dump_inode <8> /tmp/journal.dat' /dev/sdb2 and then send me a copy of /tmp/journal.dat? I'm guessing the journal must be corrupted in some interesting way, since what should have happened if it was just a case of the journal data being present but the superblock flags not being set sould have been: e2fsck 1.43.5 (04-Aug-2017) /tmp/foo.img: recovering journal Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Free blocks count wrong (26854, counted=24490). Fix? yes Free inodes count wrong (8181, counted=7281). Fix? yes /tmp/foo.img: ***** FILE SYSTEM WAS MODIFIED ***** /tmp/foo.img: 911/8192 files (0.0% non-contiguous), 8278/32768 blocks Cheers, - Ted
Just to follow up on this, after performing a couple of troubleshooting steps suggested by Ted, it became apparent that the SD card containing the ext4 filesystem has become read only. Writes succeed without error, but the original data is always returned unmodified. For the benefit of anyone else finding their way here, I was able to verify the problem by making a copy of the filesystem on another device and running fsck on the copy, which succeeded. To avoid needing a lot of disk space for the copy, I used e2image to copy the metadata into a sparse file: $ e2image -r /dev/sdb2 sdb2.e2i e2image 1.43.4 (31-Jan-2017) $ fsck.ext4 sdb2.e2i e2fsck 1.43.4 (31-Jan-2017) sdb2.e2i: recovering journal Setting free inodes count to 902960 (was 903626) Setting free blocks count to 2918014 (was 2920347) sdb2.e2i: clean, 69984/972944 files, 967554/3885568 blocks This only took 180MB of on-disk space instead of the full 15GB. So the underlying cause of this bug is a hardware failure on the SD card. It appears to have gone into some sort of failsafe read-only mode, but unfortunately it doesn't reject write operations, it just discards them. This means the host (and also the user, thanks to caching) does not realise there is a problem unless, like fsck, a program does a verify-after-write and reveals the issue.