Hi folks, yesterday I deleted a great amount of data (didn't count files, so no numbers here). After 2 hours of slow deleting, the filesystem hung. In dmesq I found the following: ################### XFS: Internal error XFS_WANT_CORRUPTED_GOTO at line 1503 of file /build/linux-eKuxrT/linux-3.2.60/fs/xfs/xfs_alloc.c. Caller 0xffffffffa025e106 Pid: 7342, comm: rm Not tainted 3.2.0-4-amd64 #1 Debian 3.2.60-1+deb7u3 Call Trace: [<ffffffffa025bd72>] ? xfs_free_ag_extent+0xec/0x66d [xfs] [<ffffffffa025e106>] ? xfs_free_extent+0xb7/0xf3 [xfs] [<ffffffffa025ad21>] ? kmem_zone_alloc+0x27/0x71 [xfs] [<ffffffffa0269c77>] ? xfs_bmap_finish+0xf3/0x162 [xfs] [<ffffffffa027e9e5>] ? xfs_itruncate_extents+0xe4/0x1ce [xfs] [<ffffffffa027eb4f>] ? xfs_itruncate_data+0x80/0xea [xfs] [<ffffffffa0259102>] ? xfs_inactive+0x1a0/0x3ae [xfs] [<ffffffff8110d6a5>] ? end_writeback+0x6a/0x77 [<ffffffffa0255102>] ? xfs_fs_evict_inode+0xad/0xb9 [xfs] [<ffffffff8110d748>] ? evict+0x96/0x148 [<ffffffff81105ea1>] ? do_unlinkat+0x107/0x156 [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b XFS (dm-0): xfs_do_force_shutdown(0x8) called from line 3729 of file /build/linux-eKuxrT/linux-3.2.60/fs/xfs/xfs_bmap.c. Return address = 0xffffffffa0269cac XFS (dm-0): Corruption of in-memory data detected. Shutting down filesystem XFS (dm-0): Please umount the filesystem and rectify the problem(s) XFS (dm-0): xfs_log_force: error 5 returned. [... goes on and on] XFS (dm-0): xfs_log_force: error 5 returned. XFS (dm-0): xfs_do_force_shutdown(0x1) called from line 1033 of file /build/linux-eKuxrT/linux-3.2.60/fs/xfs/xfs_buf.c. Return address = 0xffffffffa024a8a4 XFS (dm-0): xfs_log_force: error 5 returned. XFS (dm-0): xfs_log_force: error 5 returned. ############# I couldn't mount the filesystem again. After using "xfs_repair" and removed errors, I could luckily mount again. This is a backup system so... I kind of have some doupts about the stability right now. This is the fstab entry: # <file system> <mount point> <type> <options> <dump> <pass> UUID=blabla /backup1 xfs rw 0 0 I you need further information please dont't hesitate to ask! Best regards Carsten
xfs_info /backup1 Metadaten =/dev/mapper/backup-backup1 isize=256 agcount=44, agsize=16777024 blks = sectsz=512 attr=0 Daten = bsize=4096 Blöcke=732419072, imaxpct=25 = sunit=0 swidth=0 blks Benennung =Version 2 bsize=4096 ascii-ci=0 Protokoll =Intern bsize=4096 Blöcke=32768, Version=1 = sectsz=512 sunit=0 blks, lazy-count=0 Echtzeit =keine extsz=65536 Blöcke=0, rtextents=0
aptitude show xfsprogs Version: 3.1.7+b1
cat /proc/fs/xfs/stat extent_alloc 89948 6078084 2584345 60444138 abt 0 0 0 0 blk_map 123392621 2496918 2564504 86437 2570976 128625230 0 bmbt 0 0 0 0 dir 33428199 3267791 13923554 6029692 trans 811 27743862 16026 ig 0 5317644 0 25036369 0 25036339 735093 log 175932 11039888 7380 9834 6130 push_ail 27760705 0 364143 2435105 415904423 645178 10442731 0 0 3909 xstrat 16099 0 rw 243093 771451 attr 20866 0 0 0 icluster 2551833 1570830 15464697 vnodes 4269930939 0 0 0 25036357 25036357 25036357 0 buf 166711819 3521744 262932224 490028 314741302 3050555 0 5325414 2974679 abtb2 2677256 39207532 494370 167658 0 0 244397 19950 65765 150938 1530 151 1530 151 230583838 abtc2 5033161 72486018 2638069 2311314 0 0 183759 6985 67103 166185 1556 134 1556 134 1054809610 bmbt2 67050 461129 12722 29132 0 0 2611 575 3215 548 15 44 15 44 2603931 ibt2 3377207 46420161 0 26530 0 0 294066 10564 7699 5368 0 116 0 116 4289627 xpc 24592424960 18175772994 33741973667 debug 0
Since I have to, I am continuing deleting files. The RAM is pretty flooded, ~1.9GB RAM are used. 2GB RAM are available... I hope XFS is not hanging after some time just because there is no RAM left... slabtop output: 951124 948082 99% 0,94K 237781 4 951124K xfs_inode 950976 948288 99% 0,21K 52832 18 211328K xfs_ili 86376 79396 91% 0,31K 7198 12 28792K xfs_buf 72821 59479 81% 0,55K 10403 7 41612K radix_tree_node 23364 5640 24% 0,06K 396 59 1584K size-64 14200 10407 73% 0,19K 710 20 2840K dentry 11988 11960 99% 0,14K 444 27 1776K sysfs_dir_cache 11872 9881 83% 0,03K 106 112 424K size-32 8954 6648 74% 0,10K 242 37 968K buffer_head 8790 3219 36% 0,12K 293 30 1172K size-128 5005 2635 52% 0,05K 65 77 260K anon_vma_chain 4340 4333 99% 0,54K 620 7 2480K inode_cache 4092 3474 84% 0,17K 186 22 744K vm_area_struct 3120 1648 52% 0,19K 156 20 624K size-192 2560 2332 91% 0,75K 512 5 2048K ext3_inode_cache 2183 1634 74% 0,06K 37 59 148K anon_vma 2085 1455 69% 0,25K 139 15 556K filp 1890 1781 94% 0,60K 315 6 1260K proc_inode_cache 1224 1200 98% 1,00K 306 4 1224K size-1024 1173 699 59% 0,22K 69 17 276K xfs_buf_item 1008 689 68% 0,02K 7 144 28K dm_target_io 968 943 97% 0,50K 121 8 484K size-512 960 719 74% 0,19K 48 20 192K bio-0 952 931 97% 2,00K 476 2 1904K size-2048 920 692 75% 0,04K 10 92 40K dm_io 901 790 87% 0,07K 17 53 68K Acpi-Operand 870 567 65% 0,25K 58 15 232K skbuff_head_cache 680 218 32% 0,19K 34 20 136K cred_jar 648 629 97% 0,64K 108 6 432K shmem_inode_cache
You started iterating over the filesystem so you probably found a problem that had been on disk for a long time. THere's pretty much nothing we can do here - you've already fixed the corruption, and the kernel is so old it doesn't have any of the instrumentation that might have told us in more detail what the corruption on disk was. As there is nothing we can analyse here, please close the bug. -Dave.