Bug 198787 - Btrfs errors leading to readonly filesystem
Summary: Btrfs errors leading to readonly filesystem
Status: RESOLVED OBSOLETE
Alias: None
Product: File System
Classification: Unclassified
Component: btrfs (show other bugs)
Hardware: x86-64 Linux
: P1 high
Assignee: Josef Bacik
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-02-14 17:56 UTC by Alex Adriaanse
Modified: 2022-10-06 21:24 UTC (History)
1 user (show)

See Also:
Kernel Version: 4.13.13-1, 4.14.7, 4.14.13
Subsystem:
Regression: No
Bisected commit-id:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Description Alex Adriaanse 2018-02-14 17:56:34 UTC 
We've had Btrfs filesystems go into read-only mode on three different VMs over the past week. In each of the instances there were a lot of reads+writes taking place when this happened.

All three servers were running Debian Stretch on AWS/EBS with kernels 4.13.13-1~bpo9+1, 4.14.7-1~bpo9+1, and 4.14.13-1~bpo9+1 respectively from the Debian stretch-backports repo. The kernel messages for all three report the device not having space left, yet df and "btrfs filesystem usage" shows plenty of available space. After rebooting the servers the filesystem are writable for a bit, but shortly after become read-only with kernel errors showing up again. In at least one of the instances I ran "btrfs check", which showed FS corruption. I ended up migrating all the data to a brand new filesystem and eliminate the original (corrupted) filesystem to get rid of the read-only issues.

Every 15 minutes we have a cronjob creating new snapshots and deleting old ones. We also have a nightly cronjob that runs the following commands:

    btrfs balance start -dusage=10 /srv/volumes
    btrfs balance start -dusage=20 /srv/volumes
    btrfs balance start -dusage=40,limit=100 /srv/volumes


Details for instance #1:
No btrfs balance was running at the time.

[4176777.571884] BTRFS: Transaction aborted (error -28)
[4176777.571900] BTRFS: error (device xvdc) in btrfs_run_delayed_refs:3009: errno=-28 No space left
[4176777.571908] ------------[ cut here ]------------
[4176777.571936] WARNING: CPU: 3 PID: 386 at /build/linux-0jFc0i/linux-4.13.13/fs/btrfs/extent-tree.c:3009 btrfs_run_delayed_refs+0x25c/0x270 [btrfs]
[4176777.571936] Modules linked in: xt_nat xt_tcpudp veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c crc32c_generic br_netfilter bridge stp llc intel_rapl sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ppdev intel_rapl_perf serio_raw parport_pc parport evdev ip_tables x_tables autofs4 btrfs xor raid6_pq ata_generic crc32c_intel cirrus ttm aesni_intel drm_kms_helper ata_piix aes_x86_64 crypto_simd xen_blkfront i2c_piix4 libata psmouse cryptd glue_helper drm ena scsi_mod button
[4176777.571965] CPU: 3 PID: 386 Comm: btrfs-cleaner Not tainted 4.13.0-0.bpo.1-amd64 #1 Debian 4.13.13-1~bpo9+1
[4176777.571966] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[4176777.571967] task: ffff9fdb7ba2b0c0 task.stack: ffffb780442e0000
[4176777.571982] RIP: 0010:btrfs_run_delayed_refs+0x25c/0x270 [btrfs]
[4176777.571983] RSP: 0018:ffffb780442e3d30 EFLAGS: 00010286
[4176777.571984] RAX: 0000000000000026 RBX: ffff9fd6ea555690 RCX: 0000000000000000
[4176777.571985] RDX: 0000000000000000 RSI: ffff9fdb80acde98 RDI: ffff9fdb80acde98
[4176777.571986] RBP: ffff9fd7bbc37368 R08: 0000000000000001 R09: 0000000000009a5c
[4176777.571987] R10: 0000000000001000 R11: 0000000000009a5c R12: ffff9fdb7be6e000
[4176777.571988] R13: ffff9fd6ea555690 R14: 00000000000d92d8 R15: ffff9fd9ccf591df
[4176777.571989] FS:  0000000000000000(0000) GS:ffff9fdb80ac0000(0000) knlGS:0000000000000000
[4176777.571990] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[4176777.571991] CR2: 00007f063547a0c8 CR3: 0000000291a45000 CR4: 00000000001406e0
[4176777.571993] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[4176777.571994] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[4176777.571995] Call Trace:
[4176777.572014]  ? btrfs_commit_transaction+0x46/0x940 [btrfs]
[4176777.572027]  ? btrfs_block_rsv_check+0x54/0x60 [btrfs]
[4176777.572041]  ? should_end_transaction+0x34/0x50 [btrfs]
[4176777.572054]  ? __btrfs_end_transaction+0x24f/0x2e0 [btrfs]
[4176777.572067]  ? btrfs_drop_snapshot+0x4a4/0x8f0 [btrfs]
[4176777.572081]  ? btrfs_clean_one_deleted_snapshot+0xbe/0x110 [btrfs]
[4176777.572095]  ? cleaner_kthread+0x12e/0x180 [btrfs]
[4176777.572099]  ? kthread+0xfc/0x130
[4176777.572112]  ? run_one_async_done+0xb0/0xb0 [btrfs]
[4176777.572114]  ? kthread_create_on_node+0x70/0x70
[4176777.572117]  ? ret_from_fork+0x25/0x30
[4176777.572118] Code: c7 c6 40 a8 66 c0 48 89 df 44 89 04 24 e8 76 a1 09 00 44 8b 04 24 eb 86 44 89 c6 48 c7 c7 08 13 67 c0 44 89 04 24 e8 2f 38 70 cb <0f> ff 44 8b 04 24 eb c4 66 90 66 2e 0f 1f 84 00 00 00 00 00 0f
[4176777.572141] ---[ end trace a3dee9b0ebbded7e ]---


Details for instance #2:
"btrfs balance start -dusage=20 /srv/volumes" was running when the error occurred.

[2514499.221432] BTRFS: Transaction aborted (error -28)
[2514499.221457] ------------[ cut here ]------------
[2514499.221466] BTRFS: error (device xvdc) in btrfs_run_delayed_refs:3089: errno=-28 No space left
[2514499.221469] BTRFS: error (device xvdc) in btrfs_run_delayed_refs:3089: errno=-28 No space left
[2514499.221471] BTRFS info (device xvdc): forced readonly
[2514499.222776] BUG: unable to handle kernel NULL pointer dereference at           (null)
[2514499.222806] IP: __del_reloc_root+0x40/0x100 [btrfs]
[2514499.222807] PGD 227cef067 P4D 227cef067 PUD 31804d067 PMD 0
[2514499.222811] Oops: 0000 [#1] SMP
[2514499.222812] Modules linked in: dm_mod xt_nat xt_tcpudp veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c crc32c_generic br_netfilter bridge stp llc intel_rapl sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ppdev intel_rapl_perf serio_raw parport_pc parport evdev ip_tables x_tables autofs4 btrfs xor zstd_decompress zstd_compress xxhash raid6_pq ata_generic crc32c_intel ata_piix cirrus libata ttm aesni_intel aes_x86_64 drm_kms_helper xen_blkfront crypto_simd cryptd psmouse glue_helper i2c_piix4 drm scsi_mod ena button
[2514499.222845] CPU: 3 PID: 3622 Comm: btrfs Not tainted 4.14.0-0.bpo.2-amd64 #1 Debian 4.14.7-1~bpo9+1
[2514499.222845] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[2514499.222847] task: ffffa0763cb54080 task.stack: ffffb45dc7858000
[2514499.222865] RIP: 0010:__del_reloc_root+0x40/0x100 [btrfs]
[2514499.222867] RSP: 0018:ffffb45dc785ba68 EFLAGS: 00010246
[2514499.222868] RAX: 0000000000000000 RBX: 0000000000000000 RCX: fffffb031aae555f
[2514499.222869] RDX: 0000000000000001 RSI: ffffa0758633ddc0 RDI: ffffa077f9f9ed70
[2514499.222869] RBP: ffffb45dc785ba90 R08: 0000000000000000 R09: ffffa07606c7e8c1
[2514499.222870] R10: ffffb45dc785b840 R11: 00000000000003b6 R12: ffffa07b3779e000
[2514499.222871] R13: ffffa077f9f9ed70 R14: ffffa0783a4da800 R15: ffffa077f9f9e800
[2514499.222873] FS:  00007f67031898c0(0000) GS:ffffa07b40ac0000(0000) knlGS:0000000000000000
[2514499.222874] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2514499.222875] CR2: 0000000000000000 CR3: 00000002cfe02005 CR4: 00000000001606e0
[2514499.222877] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[2514499.222878] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[2514499.222879] Call Trace:
[2514499.222897]  free_reloc_roots+0x27/0x70 [btrfs]
[2514499.222914]  merge_reloc_roots+0x187/0x260 [btrfs]
[2514499.222931]  relocate_block_group+0x297/0x600 [btrfs]
[2514499.222947]  btrfs_relocate_block_group+0x17c/0x230 [btrfs]
[2514499.222966]  btrfs_relocate_chunk+0x3e/0xc0 [btrfs]
[2514499.222984]  btrfs_balance+0xa3b/0x12a0 [btrfs]
[2514499.223002]  btrfs_ioctl_balance+0x36d/0x380 [btrfs]
[2514499.223019]  btrfs_ioctl+0x1170/0x24e0 [btrfs]
[2514499.223023]  ? lru_cache_add_active_or_unevictable+0x36/0xb0
[2514499.223026]  ? __handle_mm_fault+0x694/0x1140
[2514499.223031]  do_vfs_ioctl+0xa3/0x600
[2514499.223033]  ? do_vfs_ioctl+0xa3/0x600
[2514499.223036]  ? __do_page_fault+0x280/0x4f0
[2514499.223039]  SyS_ioctl+0x79/0x90
[2514499.223042]  system_call_fast_compare_end+0xc/0x97
[2514499.223043] RIP: 0033:0x7f6702211e07
[2514499.223044] RSP: 002b:00007fffae3223c8 EFLAGS: 00000202 ORIG_RAX: 0000000000000010
[2514499.223046] RAX: ffffffffffffffda RBX: 00007f67024cab00 RCX: 00007f6702211e07
[2514499.223047] RDX: 00007fffae322458 RSI: 00000000c4009420 RDI: 0000000000000003
[2514499.223048] RBP: 0000000000008041 R08: 0000000000c23000 R09: 0000000000000000
[2514499.223049] R10: 0000000000000541 R11: 0000000000000202 R12: 0000000000000000
[2514499.223050] R13: 00007f67024cab58 R14: 0000000000000000 R15: 0000000000008060
[2514499.223052] Code: 4c 8b a7 f0 01 00 00 4d 8b bc 24 40 14 00 00 4d 8d af 70 05 00 00 4c 89 ef e8 dd 48 d7 d8 49 8b 9f 68 05 00 00 49 8b 06 48 85 db <48> 8b 10 75 0e e9 ab 00 00 00 48 8b 5b 10 48 85 db 74 15 48 3b
[2514499.223097] RIP: __del_reloc_root+0x40/0x100 [btrfs] RSP: ffffb45dc785ba68
[2514499.223097] CR2: 0000000000000000
[2514499.223100] ---[ end trace d9be58739256d7f7 ]---


Details for instance #3:
"btrfs balance start -dusage 10 /srv/volumes" was running when the error occurred.

df -h /srv/volumes:
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdc      1000G  395G  598G  40% /srv/volumes

btrfs filesystem usage /srv/volumes:
Overall:
    Device size:		1000.00GiB
    Device allocated:		 738.05GiB
    Device unallocated:		 261.95GiB
    Device missing:		     0.00B
    Used:			 394.15GiB
    Free (estimated):		 598.49GiB	(min: 598.49GiB)
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		 512.00MiB	(used: 121.08MiB)

Data,single: Size:708.01GiB, Used:371.47GiB
   /dev/xvdc	 708.01GiB

Metadata,single: Size:30.01GiB, Used:22.68GiB
   /dev/xvdc	  30.01GiB

System,single: Size:32.00MiB, Used:96.00KiB
   /dev/xvdc	  32.00MiB

Unallocated:
   /dev/xvdc	 261.95GiB

[418428.322838] BTRFS: Transaction aborted (error -28)
[418428.322872] ------------[ cut here ]------------
[418428.327580] WARNING: CPU: 0 PID: 27622 at /build/linux-3RM5ap/linux-4.14.13/fs/btrfs/extent-tree.c:3089 btrfs_run_delayed_refs+0x259/0x270 [btrfs]
[418428.338873] Modules linked in: xt_nat xt_tcpudp veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c crc32c_generic br_netfilter bridge stp llc intel_rapl sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ppdev intel_rapl_perf serio_raw parport_pc parport evdev ip_tables x_tables autofs4 btrfs xor zstd_decompress zstd_compress xxhash raid6_pq ata_generic crc32c_intel cirrus ttm xen_blkfront aesni_intel ata_piix libata aes_x86_64 crypto_simd drm_kms_helper cryptd i2c_piix4 glue_helper psmouse ena drm scsi_mod button
[418428.381740] CPU: 0 PID: 27622 Comm: btrfs Not tainted 4.14.0-0.bpo.3-amd64 #1 Debian 4.14.13-1~bpo9+1
[418428.388349] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[418428.393670] task: ffff9286500aa080 task.stack: ffffbcc94830c000
[418428.398636] RIP: 0010:btrfs_run_delayed_refs+0x259/0x270 [btrfs]
[418428.404494] RSP: 0018:ffffbcc94830f988 EFLAGS: 00010282
[418428.409254] RAX: 0000000000000026 RBX: ffff9280a950f168 RCX: 0000000000000000
[418428.414595] RDX: 0000000000000000 RSI: ffff928880a166f8 RDI: ffff928880a166f8
[418428.421059] RBP: ffff928239762968 R08: 0000000000000001 R09: 0000000000003297
[418428.426895] R10: 0000000000000001 R11: 0000000000003297 R12: ffff9288767d8000
[418428.432917] R13: ffff9280a950f168 R14: 000000000013e102 R15: ffff9280a67491df
[418428.438693] FS:  00007f6aa28958c0(0000) GS:ffff928880a00000(0000) knlGS:0000000000000000
[418428.445125] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[418428.450201] CR2: 00007f03921af000 CR3: 000000000e74a006 CR4: 00000000001606f0
[418428.456211] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[418428.462038] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[418428.467361] Call Trace:
[418428.470559]  btrfs_commit_transaction+0x46/0x950 [btrfs]
[418428.475313]  ? btrfs_block_rsv_check+0x54/0x60 [btrfs]
[418428.479711]  ? should_end_transaction+0x34/0x50 [btrfs]
[418428.484393]  ? __btrfs_end_transaction+0x24f/0x2e0 [btrfs]
[418428.488672]  btrfs_drop_snapshot+0x4a4/0x8d0 [btrfs]
[418428.492890]  merge_reloc_roots+0xf4/0x260 [btrfs]
[418428.497191]  relocate_block_group+0x29a/0x610 [btrfs]
[418428.501322]  btrfs_relocate_block_group+0x17b/0x230 [btrfs]
[418428.505842]  btrfs_relocate_chunk+0x38/0xb0 [btrfs]
[418428.510171]  btrfs_balance+0xa15/0x1250 [btrfs]
[418428.514259]  btrfs_ioctl_balance+0x368/0x380 [btrfs]
[418428.518847]  btrfs_ioctl+0x1170/0x24e0 [btrfs]
[418428.522556]  ? mem_cgroup_try_charge+0x86/0x1a0
[418428.526391]  ? __handle_mm_fault+0x640/0x10e0
[418428.530670]  ? do_vfs_ioctl+0x9f/0x600
[418428.534348]  do_vfs_ioctl+0x9f/0x600
[418428.537761]  ? handle_mm_fault+0xc6/0x1b0
[418428.542084]  ? __do_page_fault+0x289/0x500
[418428.546381]  SyS_ioctl+0x74/0x80
[418428.550699]  system_call_fast_compare_end+0xc/0x6f
[418428.555460] RIP: 0033:0x7f6aa191de07
[418428.559742] RSP: 002b:00007ffcd77adb28 EFLAGS: 00000206
[418428.559745] Code: c7 c6 20 68 6a c0 48 89 df 44 89 04 24 e8 39 b6 09 00 44 8b 04 24 eb 86 44 89 c6 48 c7 c7 a8 d4 6a c0 44 89 04 24 e8 c2 a2 2c c7 <0f> ff 44 8b 04 24 eb c4 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00
[418428.579966] ---[ end trace 126bacfc38b0f09a ]---
[418428.584989] BTRFS: error (device xvdc) in btrfs_run_delayed_refs:3089: errno=-28 No space left
[418428.592776] BTRFS info (device xvdc): forced readonly
[418428.598673] BUG: unable to handle kernel NULL pointer dereference at           (null)
[418428.604488] IP: __del_reloc_root+0x3c/0x100 [btrfs]
[418428.608762] PGD 80000005860b5067 P4D 80000005860b5067 PUD 5860a3067 PMD 0
[418428.614394] Oops: 0000 [#1] SMP PTI
[418428.617873] Modules linked in: xt_nat xt_tcpudp veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c crc32c_generic br_netfilter bridge stp llc intel_rapl sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ppdev intel_rapl_perf serio_raw parport_pc parport evdev ip_tables x_tables autofs4 btrfs xor zstd_decompress zstd_compress xxhash raid6_pq ata_generic crc32c_intel cirrus ttm xen_blkfront aesni_intel ata_piix libata aes_x86_64 crypto_simd drm_kms_helper cryptd i2c_piix4 glue_helper psmouse ena drm scsi_mod button
[418428.660718] CPU: 0 PID: 27622 Comm: btrfs Tainted: G        W       4.14.0-0.bpo.3-amd64 #1 Debian 4.14.13-1~bpo9+1
[418428.669395] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[418428.675114] task: ffff9286500aa080 task.stack: ffffbcc94830c000
[418428.681038] RIP: 0010:__del_reloc_root+0x3c/0x100 [btrfs]
[418428.686452] RSP: 0018:ffffbcc94830fab0 EFLAGS: 00010282
[418428.691798] RAX: 0000000000000000 RBX: ffff9281a9822a40 RCX: ffffebeb0b48a51f
[418428.698128] RDX: 0000000000000001 RSI: ffff9286b1f76a40 RDI: ffff92823a565570
[418428.703814] RBP: ffff9280a6749000 R08: ffff928349b85580 R09: ffff9281a9822a41
[418428.709881] R10: ffff92874fd1d640 R11: ffff92887bc12040 R12: ffff9288767d8000
[418428.716130] R13: ffff92823a565570 R14: ffff92823a565000 R15: 00000000ffffffe2
[418428.722571] FS:  00007f6aa28958c0(0000) GS:ffff928880a00000(0000) knlGS:0000000000000000
[418428.729708] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[418428.735153] CR2: 0000000000000000 CR3: 000000000e74a006 CR4: 00000000001606f0
[418428.741871] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[418428.748143] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[418428.755007] Call Trace:
[418428.758723]  free_reloc_roots+0x22/0x60 [btrfs]
[418428.763646]  merge_reloc_roots+0x184/0x260 [btrfs]
[418428.768611]  relocate_block_group+0x29a/0x610 [btrfs]
[418428.774172]  btrfs_relocate_block_group+0x17b/0x230 [btrfs]
[418428.779861]  btrfs_relocate_chunk+0x38/0xb0 [btrfs]
[418428.784865]  btrfs_balance+0xa15/0x1250 [btrfs]
[418428.789802]  btrfs_ioctl_balance+0x368/0x380 [btrfs]
[418428.794984]  btrfs_ioctl+0x1170/0x24e0 [btrfs]
[418428.800091]  ? mem_cgroup_try_charge+0x86/0x1a0
[418428.804717]  ? __handle_mm_fault+0x640/0x10e0
[418428.809607]  ? do_vfs_ioctl+0x9f/0x600
[418428.813829]  do_vfs_ioctl+0x9f/0x600
[418428.818310]  ? handle_mm_fault+0xc6/0x1b0
[418428.823007]  ? __do_page_fault+0x289/0x500
[418428.827340]  SyS_ioctl+0x74/0x80
[418428.831251]  system_call_fast_compare_end+0xc/0x6f
[418428.835904] RIP: 0033:0x7f6aa191de07
[418428.840139] RSP: 002b:00007ffcd77adb28 EFLAGS: 00000206
[418428.840142] Code: 8b a7 f0 01 00 00 4d 8b b4 24 40 14 00 00 4d 8d ae 70 05 00 00 4c 89 ef e8 c2 09 81 c7 49 8b 9e 68 05 00 00 48 8b 45 00 48 85 db <48> 8b 10 75 0e e9 ad 00 00 00 48 8b 5b 10 48 85 db 74 11 48 3b
[418428.860188] RIP: __del_reloc_root+0x3c/0x100 [btrfs] RSP: ffffbcc94830fab0
[418428.866692] CR2: 0000000000000000
[418428.870945] ---[ end trace 126bacfc38b0f09b ]---

Is there something we need to change in how we're using Btrfs? We've been having a lot of stability issues with it for the past couple of years, especially ever since we started putting heavy I/O loads on it. This is just the most recent issue we've come across.
Comment 1 Alex Adriaanse 2018-02-15 00:28:27 UTC 
We just got hit by a fourth instance. btrfs balance was not running at the time.

df -h /srv/volumes:
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvdc       300G  216G   84G  73% /srv/volumes

btrfs filesystem usage /srv/volumes:
Overall:
    Device size:		 300.00GiB
    Device allocated:		 217.03GiB
    Device unallocated:		  82.97GiB
    Device missing:		     0.00B
    Used:			 214.74GiB
    Free (estimated):		  83.70GiB	(min: 83.70GiB)
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		 512.00MiB	(used: 0.00B)

Data,single: Size:199.00GiB, Used:198.27GiB
   /dev/xvdc	 199.00GiB

Metadata,single: Size:18.00GiB, Used:16.47GiB
   /dev/xvdc	  18.00GiB

System,single: Size:32.00MiB, Used:48.00KiB
   /dev/xvdc	  32.00MiB

Unallocated:
   /dev/xvdc	  82.97GiB


[456096.832054] BTRFS: Transaction aborted (error -28)
[456096.832084] ------------[ cut here ]------------
[456096.835169] WARNING: CPU: 1 PID: 31862 at /build/linux-3RM5ap/linux-4.14.13/fs/btrfs/extent-tree.c:2274 __btrfs_inc_extent_ref.isra.58+0x272/0x280 [btrfs]
[456096.843143] Modules linked in: dm_mod xt_nat xt_tcpudp veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c br_netfilter bridge stp llc ppdev intel_rapl sb_edac crct10dif_pclmul crc32_pclmul ghash_clmulni_intel intel_rapl_perf evdev cirrus ttm serio_raw drm_kms_helper drm parport_pc parport button ip_tables x_tables autofs4 btrfs crc32c_generic xor zstd_decompress zstd_compress xxhash raid6_pq ata_generic crc32c_intel xen_blkfront aesni_intel aes_x86_64 crypto_simd cryptd glue_helper psmouse ata_piix libata scsi_mod ena i2c_piix4
[456096.875821] CPU: 1 PID: 31862 Comm: btrfs Not tainted 4.14.0-0.bpo.3-amd64 #1 Debian 4.14.13-1~bpo9+1
[456096.880994] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[456096.884946] task: ffff908a79142000 task.stack: ffffb038c3d1c000
[456096.889050] RIP: 0010:__btrfs_inc_extent_ref.isra.58+0x272/0x280 [btrfs]
[456096.893207] RSP: 0018:ffffb038c3d1f890 EFLAGS: 00010286
[456096.896767] RAX: 0000000000000026 RBX: 0000000000000000 RCX: 0000000000000000
[456096.901131] RDX: 0000000000000000 RSI: ffff908c7fe566f8 RDI: ffff908c7fe566f8
[456096.905557] RBP: ffff90894b8bf0f0 R08: 0000000000000001 R09: 0000000000007409
[456096.910157] R10: 0000000000000001 R11: 0000000000007409 R12: 00000000ffffffe4
[456096.914758] R13: ffff908c7b53f4d0 R14: 0000000000003078 R15: ffff908c7878e000
[456096.919046] FS:  00007fdcbfda68c0(0000) GS:ffff908c7fe40000(0000) knlGS:0000000000000000
[456096.923903] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[456096.927894] CR2: 00007f26f0e14000 CR3: 000000028cd28004 CR4: 00000000001606e0
[456096.932347] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[456096.936800] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[456096.941276] Call Trace:
[456096.943637]  __btrfs_run_delayed_refs+0x9e7/0x12a0 [btrfs]
[456096.947343]  btrfs_run_delayed_refs+0x7a/0x270 [btrfs]
[456096.950944]  create_pending_snapshot+0x5e3/0xf00 [btrfs]
[456096.954525]  ? create_pending_snapshots+0x95/0xc0 [btrfs]
[456096.958337]  ? btrfs_wait_pending_ordered+0x76/0x80 [btrfs]
[456096.962491]  create_pending_snapshots+0x95/0xc0 [btrfs]
[456096.966252]  btrfs_commit_transaction+0x3b1/0x950 [btrfs]
[456096.970080]  ? start_transaction+0x89/0x410 [btrfs]
[456096.973945]  btrfs_mksubvol+0x4a2/0x4f0 [btrfs]
[456096.977672]  ? remove_wait_queue+0x60/0x60
[456096.980913]  btrfs_ioctl_snap_create_transid+0x18b/0x1a0 [btrfs]
[456096.985056]  btrfs_ioctl_snap_create_v2+0x11d/0x170 [btrfs]
[456096.988915]  btrfs_ioctl+0x64d/0x24e0 [btrfs]
[456096.992077]  ? mem_cgroup_try_charge+0x86/0x1a0
[456096.995332]  ? __handle_mm_fault+0x640/0x10e0
[456096.998693]  ? do_vfs_ioctl+0x9f/0x600
[456097.001539]  do_vfs_ioctl+0x9f/0x600
[456097.004440]  ? handle_mm_fault+0xc6/0x1b0
[456097.007546]  ? __do_page_fault+0x289/0x500
[456097.010538]  SyS_ioctl+0x74/0x80
[456097.013278]  system_call_fast_compare_end+0xc/0x6f
[456097.016611] RIP: 0033:0x7fdcbee2ee07
[456097.019531] RSP: 002b:00007ffdb9f65ec8 EFLAGS: 00000202
[456097.019533] Code: c4 48 83 c4 18 e9 68 ff ff ff 41 bc f4 ff ff ff e9 4d fe ff ff 0f 0b e8 3d 3b a7 e2 44 89 e6 48 c7 c7 a8 e4 4a c0 e8 29 04 ad e2 <0f> ff eb 81 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 45 85
[456097.033855] ---[ end trace cdc156ddd643cca5 ]---
[456097.037360] BTRFS: error (device xvdc) in __btrfs_inc_extent_ref:2274: errno=-28 No space left
[456097.043801] BTRFS info (device xvdc): forced readonly
[456097.048164] BTRFS: error (device xvdc) in btrfs_run_delayed_refs:3089: errno=-28 No space left
[456097.054375] BTRFS: error (device xvdc) in create_pending_snapshot:1625: errno=-28 No space left
[456097.060721] BTRFS warning (device xvdc): Skipping commit of aborted transaction.
[456097.066005] BTRFS: error (device xvdc) in cleanup_transaction:1873: errno=-28 No space left
[456097.311006] BTRFS error (device xvdc): pending csums is 16384

btrfs-snap output at the exact same time as the kernel error:
ERROR: cannot snapshot '/srv/volumes/<subvol>': No space left on device
Create a readonly snapshot of '/srv/volumes/<subvol>' in '/srv/volumes/<subvol>/.snapshot/15m_2018-02-14_23:48:57'
Comment 2 David Sterba 2022-10-06 21:24:36 UTC 
This is a semi-automated bugzilla cleanup, report is against an old kernel version. If the problem still happens, please open a new bug. Thanks.
Note You need to log in before you can comment on or make changes to this bug.