Bug 98491

Summary:	BTRFS. Different device usage information with RAID1 volumes.
Product:	File System	Reporter:	olegon.ru (olegon)
Component:	btrfs	Assignee:	Josef Bacik (josef)
Status:	RESOLVED OBSOLETE
Severity:	high	CC:	dsterba, olegon
Priority:	P1
Hardware:	x86-64
OS:	Linux
Kernel Version:	3.19.7-200.fc21.x86_64	Subsystem:
Regression:	No	Bisected commit-id:

Description olegon.ru 2015-05-17 17:11:02 UTC

Nowadays, I don't know about correctness of my RAID1. Misunderstanding with device usage.
Confused by different DATA size in RAID1 devices.
btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt changed device usage, but not realign values.
I unmounted and checked devices by check command with error = 0 message.

btrfs fi df -kh .
Data, RAID1: total=847.00GiB, used=842.37GiB
System, RAID1: total=32.00MiB, used=160.00KiB
Metadata, RAID1: total=4.00GiB, used=2.48GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

btrfs fi show 
Label: 'exchange'  uuid: 052d18f5-8996-4935-a474-30886714a980
	Total devices 1 FS bytes used 288.72GiB
	devid    1 size 922.19GiB used 291.04GiB path /dev/sdc2

Label: 'storage'  uuid: b8dddbed-911d-457f-9101-897a13c215bd
	Total devices 3 FS bytes used 844.85GiB
>>>     devid    1 size 1.82TiB used 564.03GiB path /dev/sdb
>>>     devid    2 size 1.82TiB used 564.00GiB path /dev/sde
>>>     devid    3 size 1.83TiB used 574.03GiB path /dev/sdc1

btrfs device usage .
/dev/sdb, ID: 1
   Device size:             1.82TiB
>>>   Data,RAID1:            561.00GiB
   Metadata,RAID1:          3.00GiB
   System,RAID1:           32.00MiB
   Unallocated:             1.27TiB

/dev/sdc1, ID: 3
   Device size:             1.83TiB
>>>   Data,RAID1:            571.00GiB
   Metadata,RAID1:          3.00GiB
   System,RAID1:           32.00MiB
   Unallocated:             1.27TiB

/dev/sde, ID: 2
   Device size:             1.82TiB
>>>   Data,RAID1:            562.00GiB
   Metadata,RAID1:          2.00GiB
   Unallocated:             1.27TiB


blkid | grep btrfs
/dev/sdb: LABEL="storage" UUID="b8dddbed-911d-457f-9101-897a13c215bd" UUID_SUB="4e2e90ad-d18c-4e89-93ff-6fcbc85e3aae" TYPE="btrfs"
/dev/sdc1: LABEL="storage" UUID="b8dddbed-911d-457f-9101-897a13c215bd" UUID_SUB="6fcd7643-d586-4c25-862b-0fde7f278536" TYPE="btrfs" PARTUUID="40bbcc22-5c44-417e-8daf-4537cb4ba2df"
/dev/sdc2: LABEL="exchange" UUID="052d18f5-8996-4935-a474-30886714a980" UUID_SUB="7052acc4-ef49-46b0-b082-0dbb5103e34f" TYPE="btrfs" PARTUUID="36714c06-ca8f-4668-8d02-60e7e24b136a"
/dev/sde: LABEL="storage" UUID="b8dddbed-911d-457f-9101-897a13c215bd" UUID_SUB="76c5d5b6-4c5d-4b89-b58f-85e876c465ec" TYPE="btrfs"

mount | grep /d/d
/dev/sdb on /d/d type btrfs (rw,relatime,compress-force=zlib,space_cache,autodefrag)

Comment 1 olegon.ru 2015-05-17 18:35:39 UTC

btrfs fi usage .
Overall:
    Device size:		   5.47TiB
    Device allocated:		   1.66TiB
    Device unallocated:		   3.80TiB
    Device missing:		     0.00B
    Used:			   1.65TiB
    Free (estimated):		   1.91TiB	(min: 1.91TiB)
    Data ratio:			      2.00
    Metadata ratio:		      2.00
    Global reserve:		 512.00MiB	(used: 0.00B)

Data,RAID1: Size:847.00GiB, Used:842.37GiB
   /dev/sdb	 561.00GiB
   /dev/sdc1	 571.00GiB
   /dev/sde	 562.00GiB

Metadata,RAID1: Size:4.00GiB, Used:2.48GiB
   /dev/sdb	   3.00GiB
   /dev/sdc1	   3.00GiB
   /dev/sde	   2.00GiB

System,RAID1: Size:32.00MiB, Used:160.00KiB
   /dev/sdb	  32.00MiB
   /dev/sdc1	  32.00MiB

Unallocated:
   /dev/sdb	   1.27TiB
   /dev/sdc1	   1.27TiB
   /dev/sde	   1.27TiB

Comment 2 olegon.ru 2015-05-17 18:41:56 UTC

Devices sdb and sde is the same model of hard disk
smartctl -a /dev/sdb >f ; smartctl -a /dev/sdb >s ; diff f s
shows no differences

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital AV-GP (AF)
Device Model:     WDC WD20EURS-63S48Y0
Serial Number:    WD-WMAZA7578083
LU WWN Device Id: 5 0014ee 2b16830ee
Firmware Version: 51.0AB51
User Capacity:    2 000 398 934 016 bytes [2,00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:    Sun May 17 21:39:46 2015 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Comment 3 olegon.ru 2015-05-21 14:35:29 UTC

btrfs balance start -v -dusage=99 /d/d
and
btrfs balance start -v /d/d
gave me

/dev/sdb, ID: 1
   Device size:             1.82TiB
   Data,RAID1:            560.00GiB
   Metadata,RAID1:          2.00GiB
   Unallocated:             1.27TiB

/dev/sdc1, ID: 3
   Device size:             1.83TiB
   Data,RAID1:            567.00GiB
   Metadata,RAID1:          3.00GiB
   System,RAID1:           32.00MiB
   Unallocated:             1.27TiB

/dev/sde, ID: 2
   Device size:             1.82TiB
   Data,RAID1:            557.00GiB
   Metadata,RAID1:          3.00GiB
   System,RAID1:           32.00MiB
   Unallocated:             1.27TiB

so, metadata and data is different across three raid1 volumes again...

Comment 4 olegon.ru 2015-05-22 05:25:56 UTC

If this RAID0 with RAID1 label? o_O
Please, review this bug ASAP, or I'll reformat drives...

Comment 5 David Sterba 2015-05-22 12:15:28 UTC

The numbers look ok with respect to what raid1 means on btrfs: there always 2 copies of the given data, not N like for md-raid1 does.

Therefore:

* from 'btrfs fi df' output

Metadata, RAID1: total=4.00GiB, used=2.48GiB

there are 4G of total space (in logical units), that has to be saved twice, which gives 8G

* from 'btrfs fi usage' output, comment 3:

sdb: 2G
sdc1: 3G
sde: 3G

sums to 8G as expected.

Comment 6 olegon.ru 2015-05-22 13:09:53 UTC

Sorry, I don't know about this behaviour of btrfs... All my collegues don't knows too... Sorry for question, but the only way to get 3 copies is to remove one drive at sync it by rsync? Or any property of btrfs is able to do this?

Comment 7 olegon.ru 2015-05-24 05:12:12 UTC

Sorry, resolved by one deleted drive.