Bug 74101 - "Out of space" reported when there's lots of non-allocated space
Summary: "Out of space" reported when there's lots of non-allocated space
Status: NEW
Alias: None
Product: File System
Classification: Unclassified
Component: btrfs (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Josef Bacik
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-04-14 21:08 UTC by Jeff Mitchell
Modified: 2020-03-27 15:28 UTC (History)
8 users (show)

See Also:
Kernel Version: 3.13
Tree: Mainline
Regression: No


Attachments

Description Jeff Mitchell 2014-04-14 21:08:05 UTC
Although the device has plenty of non-allocated space (after extending the partition and running a resize command), I am getting out of space errors. Rebalancing does not help. I was asked in #btrfs to report this on the bugtracker as it appears to be a legitimate bug.

# btrfs fi show
Label: none  uuid: c553ada1-031f-48f8-a497-cc5e1f913619
	Total devices 1 FS bytes used 2.79GiB
	devid    1 size 20.00GiB used 4.61GiB path /dev/sda2

Btrfs v3.12

# btrfs fi df /
Data, single: total=3.25GiB, used=2.54GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=664.12MiB, used=257.11MiB

# uname -a
Linux repo 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

I have a btrfs-image dump of the filesystem, but it's above the maximum file size limit for the bugtracker.
Comment 1 Justin Alan Ryan 2015-07-15 17:36:50 UTC
Have you tried restoring the btrfs-image dump to another volume, or seeing if you can reproduce this with a fresh filesystem by following steps you recall leading to it?
Comment 2 Roman Kapusta 2016-06-14 09:22:08 UTC
I'm hitting same problem here, lot of unallocated space, but cannot be used.

First I have created lvm partition (l1) of similar size as second physical disk (l2) and formatted with btrfs with mirroring (RADI1): 
devid    2 size 1465136512.00KiB used 1339064320.00KiB path /dev/mapper/l2

Then when my free space was below 100 GB, I added new physical disk (l3):
devid    3 size 244196544.00KiB used 119537664.00KiB path /dev/mapper/l3

and extend lvm partition (l1) to around size of l2+l3:
devid    1 size 1709701120.00KiB used 1458601984.00KiB path /dev/mapper/l1

Current state: disk is full

# df -k /media/storage/
Filesystem      1K-blocks       Used Available Use% Mounted on
/dev/mapper/l1 1709517096 1457855616 126644080  93% /media/storage

# btrfs fi show --kbytes /media/storage/
Label: 'storage'  uuid: 912c98c3-5f1f-4c92-b7c5-710d6761154d
	Total devices 3 FS bytes used 1457394816.00KiB
	devid    1 size 1709701120.00KiB used 1458601984.00KiB path /dev/mapper/l1
	devid    2 size 1465136512.00KiB used 1339064320.00KiB path /dev/mapper/l2
	devid    3 size 244196544.00KiB used 119537664.00KiB path /dev/mapper/l3

# btrfs fi df --kbytes /media/storage/
Data, RAID1: total=1455423488.00KiB, used=1454843392.00KiB
System, RAID1: total=32768.00KiB, used=224.00KiB
Metadata, RAID1: total=3145728.00KiB, used=2480528.00KiB

I tried to resize individual disks:
# btrfs fi resize 1:max /media/storage/
# btrfs fi resize 2:max /media/storage/
# btrfs fi resize 3:max /media/storage/
dmesg:
[171983.005478] BTRFS info (device dm-18): resizing devid 1
[171983.005492] BTRFS info (device dm-18): new size for /dev/mapper/l1 is 1750733946880
[171990.403990] BTRFS info (device dm-18): resizing devid 2
[171990.404003] BTRFS info (device dm-18): new size for /dev/mapper/l2 is 1500299812864
[171994.630144] BTRFS info (device dm-18): resizing devid 3
[171994.630156] BTRFS info (device dm-18): new size for /dev/mapper/l3 is 250057252864

I tried rebalance and waited more then 24 hours to finish:
# btrfs balance start /media/storage/

Nothing helped, my kernel version is 4.4.12-200.fc22.x86_64, btrfs-progs version 4.3.1-1.fc22.x86_64.
It probably does not have impact but partitions are encrypted with LUKS
Comment 3 Kevin B. 2016-08-11 01:06:07 UTC
I've also come across this issue and it has started to become consistent (happened a few times over the course of weeks and now nearly once a day for the past few days).  I'll try deleting/compressing some files to see if that reduces the frequency.

I've tried balance and defrag.  I'm not sure if either fully finished because i started them at night and when I check in the evening for see progress, the computer is locked up/unusuable.

There are (4) drives are in RAID 5 configuration using a "Adaptec Series 6 - ASR-6805" controller.

uname -a:
Linux localhost.localdomain 4.6.4-201.fc23.x86_64 #1 SMP Tue Jul 12 11:43:59 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

df:
Filesystem       1K-blocks       Used  Available Use% Mounted on
devtmpfs          16437844          0   16437844   0% /dev
tmpfs             16448452      54488   16393964   1% /dev/shm
tmpfs             16448452       1840   16446612   1% /run
tmpfs             16448452          0   16448452   0% /sys/fs/cgroup
/dev/sda4      11695209472 5525759712 6159218512  48% /
tmpfs             16448452      28052   16420400   1% /tmp
/dev/sda2          3966144     256604    3488356   7% /boot
/dev/sda4      11695209472 5525759712 6159218512  48% /home
/dev/sda1          2043984      15884    2028100   1% /boot/efi
tmpfs              3289692          0    3289692   0% /run/user/0
tmpfs              3289692         24    3289668   1% /run/user/1000

btrfs fi df /:
Data, single: total=5.19TiB, used=5.13TiB
System, DUP: total=64.00MiB, used=580.00KiB
Metadata, DUP: total=15.00GiB, used=9.93GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

btrfs fi show /dev/sda4:
Label: 'fedora'  uuid: 09cfb441-2242-40dc-999f-e50ec6cad82a
        Total devices 1 FS bytes used 5.14TiB
        devid    1 size 10.89TiB used 5.22TiB path /dev/sda4
Comment 4 suncuss.exe 2017-03-09 16:37:45 UTC
I also run into this issue and it is becoming consistent.
btrfs balance will solve it temporarily, but it will become full again short after.

uname -a:
Linux localhost.localdomain 4.5.0-040500rc6-generic #201602281230 SMP Sun Feb 28 17:33:02 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

btrfs fi df /home:
Data, RAID0: total=5.62TiB, used=5.61TiB
System, RAID1: total=32.00MiB, used=400.00KiB
Metadata, RAID1: total=195.00GiB, used=22.89GiB
GlobalReserve, single: total=512.00MiB, used=156.27MiB

btrfs fi show /home:
Label: none  uuid: 6ceeca45-8d5d-4c48-adc0-211b65b2807e
        Total devices 4 FS bytes used 5.63TiB
        devid    1 size 1.82TiB used 1.50TiB path /dev/sde
        devid    2 size 1.82TiB used 1.50TiB path /dev/sdf
        devid    3 size 1.82TiB used 1.50TiB path /dev/sdg
        devid    4 size 1.82TiB used 1.50TiB path /dev/sdi


btrfs fi usage /home:
Overall:
    Device size:                   7.28TiB
    Device allocated:              6.00TiB
    Device unallocated:            1.28TiB
    Device missing:                  0.00B
    Used:                          5.66TiB
    Free (estimated):              1.28TiB      (min: 658.13GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 149.05MiB)

Data,RAID0: Size:5.62TiB, Used:5.61TiB
   /dev/sde        1.40TiB
   /dev/sdf        1.40TiB
   /dev/sdg        1.40TiB
   /dev/sdi        1.40TiB

Metadata,RAID1: Size:195.00GiB, Used:22.89GiB
   /dev/sde       98.00GiB
   /dev/sdf       97.00GiB
   /dev/sdg       97.00GiB
   /dev/sdi       98.00GiB

System,RAID1: Size:32.00MiB, Used:400.00KiB
   /dev/sde       32.00MiB
   /dev/sdi       32.00MiB

Unallocated:
   /dev/sde      326.99GiB
   /dev/sdf      328.02GiB
   /dev/sdg      328.02GiB
   /dev/sdi      326.99GiB
Comment 5 jojopost62 2020-02-21 10:29:17 UTC
I am experimenting with BTRFS on my NAS using OMV and I think I just encountered this bug.
I have 2 16 TB drives (data striped, metadata mirrored) in use. 
What confuses me is that my Samba network drive under Windows suddenly did not show the correct total storage size. Also the OMV overview did not show "Total: 32 TB, Used: 16 TB" anymore, but only "Total: 16 TB, Used: 16 TB". But as you can see below, btrfs fi still showed a device size 29.11Tib.

After a complete rebalance, the displays were normal again and OMV also showed Total 32 TB again.



df and usage output before rebalance:

# btrfs filesystem df -h /srv/dev-disk-by-label-Storage
Data, RAID0: total=14.28TiB, used=14.28TiB
System, RAID1: total=8.00MiB, used=1.00MiB
Metadata, RAID1: total=16.00GiB, used=15.92GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


# btrfs fi usage /srv/dev-disk-by-label-Storage
Overall:
    Device size:                  29.11TiB
    Device allocated:             14.31TiB
    Device unallocated:           14.79TiB
    Device missing:                  0.00B
    Used:                         14.31TiB
    Free (estimated):             14.80TiB      (min: 7.40TiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)

Data,RAID0: Size:14.28TiB, Used:14.28TiB
   /dev/sdb        7.14TiB
   /dev/sdc        7.14TiB

Metadata,RAID1: Size:16.00GiB, Used:15.92GiB
   /dev/sdb       16.00GiB
   /dev/sdc       16.00GiB

System,RAID1: Size:8.00MiB, Used:1.00MiB
   /dev/sdb        8.00MiB
   /dev/sdc        8.00MiB

Unallocated:
   /dev/sdb        7.39TiB
   /dev/sdc        7.39TiB



df and usage output after rebalance:

# btrfs filesystem df -h /srv/dev-disk-by-label-Storage
Data, RAID0: total=14.32TiB, used=14.32TiB
System, RAID1: total=32.00MiB, used=1.02MiB
Metadata, RAID1: total=16.00GiB, used=15.28GiB
GlobalReserve, single: total=512.00MiB, used=0.00B


# btrfs fi usage /srv/dev-disk-by-label-Storage
Overall:
    Device size:                  29.11TiB
    Device allocated:             14.35TiB
    Device unallocated:           14.75TiB
    Device missing:                  0.00B
    Used:                         14.35TiB
    Free (estimated):             14.76TiB      (min: 7.38TiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)

Data,RAID0: Size:14.32TiB, Used:14.32TiB
   /dev/sdb        7.16TiB
   /dev/sdc        7.16TiB

Metadata,RAID1: Size:16.00GiB, Used:15.28GiB
   /dev/sdb       16.00GiB
   /dev/sdc       16.00GiB

System,RAID1: Size:32.00MiB, Used:1.02MiB
   /dev/sdb       32.00MiB
   /dev/sdc       32.00MiB

Unallocated:
   /dev/sdb        7.38TiB
   /dev/sdc        7.38TiB


# btrfs fi show /srv/dev-disk-by-label-Storage
Label: 'Storage'  uuid: 6ddb9a58-84bd-4d93-a513-f47d9cd71fd5
        Total devices 2 FS bytes used 14.33TiB
        devid    1 size 14.55TiB used 7.17TiB path /dev/sdb
        devid    2 size 14.55TiB used 7.17TiB path /dev/sdc
Comment 6 Ricardo Cescon 2020-03-19 07:50:54 UTC
Hello,

can't use the 11TB from /dev/sdf
How to fix it?

root@pm-arc:/home/ubuntu# btrfs fi usage /mnt/arc/
WARNING: RAID56 detected, not implemented
Overall:
    Device size:                  76.40TiB
    Device allocated:            180.16GiB
    Device unallocated:           76.22TiB
    Device missing:                  0.00B
    Used:                        179.11GiB
    Free (estimated):                0.00B      (min: 8.00EiB)
    Data ratio:                       0.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)

Data,RAID5: Size:51.69TiB, Used:51.66TiB
   /dev/sda       12.70TiB
   /dev/sdb       12.70TiB
   /dev/sdc       12.70TiB
   /dev/sdd       12.70TiB
   /dev/sde       12.70TiB
   /dev/sdf      912.02GiB

Metadata,RAID1: Size:90.06GiB, Used:89.55GiB
   /dev/sda       35.00GiB
   /dev/sdb       35.00GiB
   /dev/sdc       34.06GiB
   /dev/sdd       35.00GiB
   /dev/sde       35.00GiB
   /dev/sdf        6.06GiB

System,RAID1: Size:20.00MiB, Used:3.02MiB
   /dev/sdd       20.00MiB
   /dev/sdf       20.00MiB

Unallocated:
   /dev/sda        1.00MiB
   /dev/sdb       21.00MiB
   /dev/sdc        1.00MiB
   /dev/sdd       33.00MiB
   /dev/sde       65.00MiB
   /dev/sdf       11.84TiB
Comment 7 John Freeman 2020-03-27 15:28:23 UTC
Got same issue on 3.6 TB storage, no raid, no deduplication except default metadata DUP mode.

Overall:
    Device size:                   3.55TiB
    Device allocated:              2.50TiB
    Device unallocated:            1.05TiB
    Device missing:                  0.00B
    Used:                          2.50TiB
    Free (estimated):              1.05TiB      (min: 539.60GiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 32.00KiB)

Data,single: Size:2.48TiB, Used:2.48TiB
   /dev/md66       2.48TiB

Metadata,DUP: Size:7.50GiB, Used:6.99GiB
   /dev/md66      15.00GiB

System,DUP: Size:32.00MiB, Used:304.00KiB
   /dev/md66      64.00MiB

Unallocated:
   /dev/md66       1.05TiB

df : 

Data, single: total=2.48TiB, used=2.48TiB
System, DUP: total=32.00MiB, used=320.00KiB
Metadata, DUP: total=7.50GiB, used=6.99GiB
GlobalReserve, single: total=512.00MiB, used=32.00KiB

Stuck on 2.50 TB then "no free space left on device"

BUT it's status after rebalance, during problem there was strange metadata status

Metadata, DUP: total=7.50GiB, used=7.49GiB

Probably there is fail in btrfs architecture and it's NEEDED TO BE REGULARY REBALANCED in cron. Metadata full => no space left even if there is tons of free space.

Note You need to log in before you can comment on or make changes to this bug.