Hello, Have huge write activities (mounted with space cache space_cache=v2 and compress=zlib): [641107.094468] ------------[ cut here ]------------ [641107.094477] WARNING: CPU: 0 PID: 21454 at fs/btrfs/extent-tree.c:6944 __btrfs_free_extent.isra.73+0x287/0xcc0 [641107.094478] BTRFS: Transaction aborted (error -28) [641107.094479] Modules linked in: [641107.094482] CPU: 0 PID: 21454 Comm: btrfs-transacti Not tainted 4.11.0-rc4-xxxx-std-ipv6-64 #1 [641107.094483] Hardware name: Supermicro X10DRH/X10DRH-iT, BIOS 2.0 12/17/2015 [641107.094484] Call Trace: [641107.094490] dump_stack+0x4d/0x65 [641107.094495] __warn+0xc7/0xf0 [641107.094497] warn_slowpath_fmt+0x4a/0x50 [641107.094498] __btrfs_free_extent.isra.73+0x287/0xcc0 [641107.094501] __btrfs_run_delayed_refs+0x600/0x1030 [641107.094504] btrfs_run_delayed_refs+0x7f/0x290 [641107.094508] ? __btrfs_run_delayed_items+0x1ae/0x1f0 [641107.094512] btrfs_commit_transaction+0x3bb/0x870 [641107.094513] ? start_transaction+0x96/0x400 [641107.094516] transaction_kthread+0x169/0x1a0 [641107.094518] kthread+0xf7/0x130 [641107.094520] ? btrfs_cleanup_transaction+0x460/0x460 [641107.094521] ? kthread_park+0x90/0x90 [641107.094524] ret_from_fork+0x29/0x40 [641107.094526] ---[ end trace 94546cd5a4c0836e ]--- [641107.094604] BTRFS: error (device sda3) in __btrfs_free_extent:6944: errno=-28 No space left [641107.094606] BTRFS info (device sda3): forced readonly [641107.094610] BTRFS: error (device sda3) in btrfs_run_delayed_refs:2961: errno=-28 No space left [641107.094613] BTRFS warning (device sda3): Skipping commit of aborted transaction. [641107.094614] BTRFS: error (device sda3) in cleanup_transaction:1878: errno=-28 No space left
Up, it's very important for the 4.11 release
[165882.505661] ------------[ cut here ]------------ [165882.505671] WARNING: CPU: 4 PID: 15187 at fs/btrfs/extent-tree.c:6944 __btrfs_free_extent.isra.73+0x26e/0xc90 [165882.505672] BTRFS: Transaction aborted (error -28) [165882.505673] Modules linked in: [165882.505677] CPU: 4 PID: 15187 Comm: btrfs-cleaner Not tainted 4.11.0-rc8-xxxx-std-ipv6-64 #1 [165882.505678] Hardware name: Supermicro X10DRH/X10DRH-iT, BIOS 2.0 12/17/2015 [165882.505679] Call Trace: [165882.505684] dump_stack+0x4d/0x65 [165882.505687] __warn+0xc7/0xf0 [165882.505689] warn_slowpath_fmt+0x46/0x50 [165882.505691] __btrfs_free_extent.isra.73+0x26e/0xc90 [165882.505695] ? btrfs_merge_delayed_refs+0x5c/0x670 [165882.505698] __btrfs_run_delayed_refs+0x5e0/0xfe0 [165882.505701] btrfs_run_delayed_refs+0x7f/0x290 [165882.505705] ? free_extent_buffer+0x46/0x80 [165882.505708] btrfs_should_end_transaction+0x43/0x60 [165882.505710] btrfs_drop_snapshot+0x3ed/0x7a0 [165882.505712] btrfs_clean_one_deleted_snapshot+0xaf/0xe0 [165882.505714] cleaner_kthread+0x130/0x180 [165882.505718] kthread+0xff/0x140 [165882.505720] ? btree_invalidatepage+0xa0/0xa0 [165882.505722] ? kthread_park+0x90/0x90 [165882.505726] ret_from_fork+0x29/0x40 [165882.505728] ---[ end trace 0b3074c99353f217 ]--- [165882.505730] BTRFS: error (device sda3) in __btrfs_free_extent:6944: errno=-28 No space left [165882.505732] BTRFS info (device sda3): forced readonly [165882.505734] BTRFS: error (device sda3) in btrfs_run_delayed_refs:2961: errno=-28 No space left [165882.505738] BTRFS: error (device sda3) in btrfs_update_root:149: errno=-28 No space left [165882.505740] BTRFS: error (device sda3) in btrfs_drop_snapshot:9116: errno=-28 No space left [165882.505796] BTRFS: error (device sda3) in btrfs_drop_snapshot:9190: errno=-28 No space left With 4.11 release, with space_cache=v2
Seems like its similar stuff is documented under FAQ: https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space Can you check these commands and share its output? (It may help btrfs devs) $sudo btrfs fi show sudo btrfs fi df /mount/point
+ from https://btrfs.wiki.kernel.org/index.php/Mount_options enospc_debug: Used for debugging problems with "no space left on device". Usually prints more debugging messages to the syslog, otherwise can be used without performance degradation.
Hi, Thanks for your time, the output is: Label: '/home' uuid: 208d1b5a-aca5-4a70-9405-01fa5871397f Total devices 1 FS bytes used 22.06TiB devid 1 size 60.01TiB used 22.97TiB path /dev/sda3 Data, single: total=22.22TiB, used=21.70TiB System, DUP: total=32.00MiB, used=2.95MiB Metadata, DUP: total=383.50GiB, used=379.39GiB GlobalReserve, single: total=512.00MiB, used=183.94MiB Cheers,
thanks for the output. How often you get this error message? If its frequent, we can add above mount option (enospc_debug) & see whether it produces verbose message than current one?
(If I'm not wrong) I think you exactly running into this: https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_on_device.22_errors.2C_but_df_says_I.27ve_got_lots_of_space "Note that the Metadata used value is fairly close (75% or more) to the Metadata total value"" >Metadata, DUP: total=383.50GiB, used=379.39GiB I guess 'fi balance' may be needed. Lets wait for btrfs-dev confirmation on this.
Hi, It's at each 5TB written. But I don't wish drop my data just to do this. Then for now hard to reproduce. I need wait some week. My case don't seam similar because I have only 38% of space used, 22.97TiB/60.01TiB Cheers,
I think you are referring to 38% from output of 'btrfs fi show'. I was referring to the output of 'btrfs fi df'. Where you can see meta-data is almost close to total size. If I understand correctly, even if you have lot of data space left, low meta-data space can create this error message.
I had always see this on all my btrfs volume (meta-data and data near to but full with btrfs fi df, on my other 80 servers too), and same on my local computer. You understand correctly.
Created attachment 256595 [details] dmesg
Created attachment 256621 [details] dmesg dmesg after no free space
Have a look at this blog.http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html If I'm not wrong, you need to run 'balance' with appropriate dusage value.
1) why it's no do automatically? 2) No, I had test all the value, and each balance it's 2 months...
(In reply to alpha_one_x86 from comment #14) > 1) why it's no do automatically? > 2) No, I had test all the value, and each balance it's 2 months... 2 months! thats a lot of time. Yes. I agree with you on (1) there should be some kind of automatic option. (but I dont know about complexity of such option) . Lets keep this bug open for dev comments.