Bug 214719
Summary: | ntfs3: disk usage of file not updated on overwrite by smaller file | ||
---|---|---|---|
Product: | File System | Reporter: | Ganapathi Kamath (hgkamath) |
Component: | Other | Assignee: | fs_other |
Status: | NEW --- | ||
Severity: | normal | CC: | kari.argillander |
Priority: | P1 | ||
Hardware: | All | ||
OS: | Linux | ||
Kernel Version: | 5.15.0-0.rc3.20211001git4de593fb965f.30.fc35.x86_64 | Subsystem: | |
Regression: | No | Bisected commit-id: |
Description
Ganapathi Kamath
2021-10-14 13:26:23 UTC
Emphasizing that this is with the NEW in-kernel paragon-NTFS3 driver and not the older fuseblk driver. I had to rpmbuild my own kernel using the fedora-rawhide srpm after enabling relevant kernel-configs setting for NTFS3. I could not find maintainer email-id in CC-field in bugzilla.kernel.org So I emailed MAINTAINERS NTFS3 FILESYSTEM M: Konstantin Komarov <almaz.alexandrovich@paragon-software.com> L: ntfs3@lists.linux.dev S: Supported W: http://www.paragon-software.com/ T: git https://github.com/Paragon-Software-Group/linux-ntfs3.git F: Documentation/filesystems/ntfs3.rst F: fs/ntfs3/ > Namjae Jeon: Can you check if below change fix your issue ? Sungjong find
> root cause and provide the change to me.
diff --git a/fs/exfat/inode.c b/fs/exfat/inode.c
index ca37d4344361..1c7aa1ea4724 100644
--- a/fs/exfat/inode.c
+++ b/fs/exfat/inode.c
@@ -604,7 +604,7 @@ static int exfat_fill_inode(struct inode *inode, struct exfat_dir_entry *info)
exfat_save_attr(inode, info->attr);
inode->i_blocks = ((i_size_read(inode) + (sbi->cluster_size - 1)) &
- ~(sbi->cluster_size - 1)) >> inode->i_blkbits;
+ ~((loff_t)sbi->cluster_size - 1)) >> inode->i_blkbits;
inode->i_mtime = info->mtime;
inode->i_ctime = info->mtime;
ei->i_crtime = info->crtime;
The issue of wrong disk-usage/allocated-block-size seems to be resolved. Whether the block allocation is spec compliant/optimal is another issue/question. which I can't comment as I don't know if right/wrong. I have not read/dug into the ExFAT spec. Below, in addition to logs, I have collected some extra troubleshooting. In the qemu case I noted that there is a half-a-cluster of extra block-allocation. See comments inline. Btw, took me 3.5 hrs to build new kernel rpms. I installed, booted into kernel with the patch. I neglected to keep their rpm-s of the older previous kernel-5.15.0-rc3 rpms prior to your patch, so I can't go back to the exact same custom-built-kernel without rebuilding. The next older kernel I have installed is fedora-34 5.14.9-300 [root@sirius gana]# mount /dev/sda16 /mnt/a16 [root@sirius gana]# rm /mnt/a16/test1.bin rm: remove regular file '/mnt/a16/test1.bin'? y [root@sirius gana]# umount /mnt/a16 [root@sirius gana]# mount /dev/sda16 /mnt/a16 [root@sirius gana]# [root@sirius gana]# dd if=/dev/zero of=/mnt/a16/test1.bin bs=1M count=6000 6000+0 records in 6000+0 records out 6291456000 bytes (6.3 GB, 5.9 GiB) copied, 17.3153 s, 363 MB/s [root@sirius gana]# ls -ls /mnt/a16/test1.bin 6144000 -rwxr-xr-x. 1 root root 6291456000 Oct 15 19:25 /mnt/a16/test1.bin [root@sirius gana]# stat /mnt/a16/test1.bin File: /mnt/a16/test1.bin Size: 6291456000 Blocks: 12288000 IO Block: 131072 regular file Device: 10300h/66304d Inode: 155 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:dosfs_t:s0 Access: 2021-10-15 19:25:30.000000000 +0530 Modify: 2021-10-15 19:25:48.800000000 +0530 Change: 2021-10-15 19:25:48.800000000 +0530 Birth: 2021-10-15 19:25:31.490000000 +0530 [root@sirius gana]# [root@sirius gana]# [root@sirius gana]# umount /mnt/a16 [root@sirius gana]# mount /dev/sda16 /mnt/a16 [root@sirius gana]# [root@sirius gana]# ls -ls /mnt/a16/test1.bin 6144000 -rwxr-xr-x. 1 root root 6291456000 Oct 15 19:25 /mnt/a16/test1.bin [root@sirius gana]# stat /mnt/a16/test1.bin File: /mnt/a16/test1.bin Size: 6291456000 Blocks: 12288000 IO Block: 131072 regular file Device: 10300h/66304d Inode: 156 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:dosfs_t:s0 Access: 2021-10-15 19:25:30.000000000 +0530 Modify: 2021-10-15 19:25:48.800000000 +0530 Change: 2021-10-15 19:25:48.800000000 +0530 Birth: 2021-10-15 19:25:31.490000000 +0530 [root@sirius gana]# [root@sirius gana]# echo $((6291456000/512)) 512 byte blocks = $((6291456000/1024)) kB 12288000 512 byte blocks = 6144000 kB [root@sirius gana]# # seems like fixed ! Blocks count is preserved on unmount [root@sirius gana]# # also note, that in this case allocation is perfect, no extra unused blocks. [root@sirius gana]# [root@sirius gana]# umount /mnt/a16 [root@sirius gana]# mount /dev/sda16 /mnt/a16 [root@sirius gana]# rm /mnt/a16/test1.bin rm: remove regular file '/mnt/a16/test1.bin'? y [root@sirius gana]# umount /mnt/a16 [root@sirius gana]# mount /dev/sda16 /mnt/a16 [root@sirius gana]# qemu-img create -f qcow2 /mnt/a16/test1.bin 6G Formatting '/mnt/a16/test1.bin', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=6442450944 lazy_refcounts=off refcount_bits=16 [root@sirius gana]# ls -ls /mnt/a16/test1.bin 256 -rwxr-xr-x. 1 root root 196704 Oct 15 19:27 /mnt/a16/test1.bin [root@sirius gana]# stat /mnt/a16/test1.bin File: /mnt/a16/test1.bin Size: 196704 Blocks: 512 IO Block: 131072 regular file Device: 10300h/66304d Inode: 158 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:dosfs_t:s0 Access: 2021-10-15 19:27:56.000000000 +0530 Modify: 2021-10-15 19:27:57.990000000 +0530 Change: 2021-10-15 19:27:57.990000000 +0530 Birth: 2021-10-15 19:27:57.930000000 +0530 [root@sirius gana]# echo $((512*512)) " Bytes, " $((512*512-196704)) " Bytes extra" 262144 Bytes, 65440 Bytes extra [root@sirius gana]# qemu-img create -f qcow2 -o preallocation=falloc /mnt/a16/test1.bin 6G Formatting '/mnt/a16/test1.bin', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=falloc compression_type=zlib size=6442450944 lazy_refcounts=off refcount_bits=16 [root@sirius gana]# ls -ls /mnt/a16/test1.bin 6292736 -rwxr-xr-x. 1 root root 6443696128 Oct 15 19:28 /mnt/a16/test1.bin [root@sirius gana]# stat /mnt/a16/test1.bin File: /mnt/a16/test1.bin Size: 6443696128 Blocks: 12585472 IO Block: 131072 regular file Device: 10300h/66304d Inode: 158 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:dosfs_t:s0 Access: 2021-10-15 19:28:44.000000000 +0530 Modify: 2021-10-15 19:28:45.510000000 +0530 Change: 2021-10-15 19:28:45.510000000 +0530 Birth: 2021-10-15 19:27:57.930000000 +0530 [root@sirius gana]# echo $((12585472*512)) " Bytes, " $((12585472*512-6443696128)) " Bytes extra" 6443761664 Bytes, 65536 Bytes extra [root@sirius gana]# echo $((6*1024*1024*1024)) 6442450944 [root@sirius gana]# echo $((6443696128/131072)) 49161 [root@sirius gana]# echo $((6443696128-49161*131072)) 65536 [root@sirius gana]# # seems like 128 blocks are extra allocated for no reason. [root@sirius gana]# [root@sirius gana]# [root@sirius gana]# umount /mnt/a16 [root@sirius gana]# mount /dev/sda16 /mnt/a16 [root@sirius gana]# [root@sirius gana]# ls -ls /mnt/a16/test1.bin 6292736 -rwxr-xr-x. 1 root root 6443696128 Oct 15 19:28 /mnt/a16/test1.bin [root@sirius gana]# stat /mnt/a16/test1.bin File: /mnt/a16/test1.bin Size: 6443696128 Blocks: 12585472 IO Block: 131072 regular file Device: 10300h/66304d Inode: 159 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:dosfs_t:s0 Access: 2021-10-15 19:28:44.000000000 +0530 Modify: 2021-10-15 19:28:45.510000000 +0530 Change: 2021-10-15 19:28:45.510000000 +0530 Birth: 2021-10-15 19:27:57.930000000 +0530 [root@sirius gana]# [root@sirius gana]# # well, [root@sirius gana]# # (1) the allocated size greater than file-size, thats a good thing [root@sirius gana]# # and (2) block count is preserved during unmount and remount The below are extra troubleshoorting [root@sirius gana]# dump.exfat /dev/sda16 exfatprogs version : 1.1.2 -------------- Dump Boot sector region -------------- Volume Length(sectors): 305061888 FAT Offset(sector offset): 2048 FAT Length(sectors): 9472 Cluster Heap Offset (sector offset): 12288 Cluster Count: 1191600 Root Cluster (cluster offset): 5 Volume Serial: 0xf13ff267 Sector Size Bits: 9 Sector per Cluster bits: 8 ----------------- Dump Root entries ----------------- Volume entry type: 0x83 Volume label: CPERF0 Volume label character count: 6 Bitmap entry type: 0x81 Bitmap start cluster: 2 Bitmap size: 148950 Upcase table entry type: 0x82 Upcase table start cluster: 4 Upcase table size: 5836 ---------------- Show the statistics ---------------- Cluster size: 131072 Total Clusters: 1191600 Free Clusters: 537402 The dump.exfat says exfat clustersize is 131072 (twice of 65536) What happens on btrfs ? BTRFS does support sparse. [root@sirius gana]# rm -f /opt/tmp/test* [root@sirius gana]# [root@sirius gana]# qemu-img create -f qcow2 /opt/tmp/test1.bin 6G Formatting '/opt/tmp/test1.bin', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=6442450944 lazy_refcounts=off refcount_bits=16 [root@sirius gana]# ls -ls /opt/tmp/test1.bin 196 -rw-r--r--. 1 root root 196704 Oct 15 19:57 /opt/tmp/test1.bin [root@sirius gana]# stat /opt/tmp/test1.bin File: /opt/tmp/test1.bin Size: 196704 Blocks: 392 IO Block: 4096 regular file Device: 22h/34d Inode: 1844292 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Context: unconfined_u:object_r:usr_t:s0 Access: 2021-10-15 19:57:48.865554058 +0530 Modify: 2021-10-15 19:57:48.859554007 +0530 Change: 2021-10-15 19:57:48.859554007 +0530 Birth: 2021-10-15 19:57:48.836553809 +0530 [root@sirius gana]# [root@sirius gana]# echo $((392*512)) " Bytes, " $((392*512-196704)) " Bytes extra" 200704 Bytes, 4000 Bytes extra [root@sirius gana]# [root@sirius gana]# qemu-img create -f qcow2 -o preallocation=falloc /opt/tmp/test1.bin 6G Formatting '/opt/tmp/test1.bin', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=falloc compression_type=zlib size=6442450944 lazy_refcounts=off refcount_bits=16 [root@sirius gana]# ls -ls /opt/tmp/test1.bin 6292612 -rw-r--r--. 1 root root 6443696128 Oct 15 19:59 /opt/tmp/test1.bin [root@sirius gana]# stat /opt/tmp/test1.bin File: /opt/tmp/test1.bin Size: 6443696128 Blocks: 12585224 IO Block: 4096 regular file Device: 22h/34d Inode: 1844292 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Context: unconfined_u:object_r:usr_t:s0 Access: 2021-10-15 19:59:45.364560032 +0530 Modify: 2021-10-15 19:59:45.355559953 +0530 Change: 2021-10-15 19:59:45.355559953 +0530 Birth: 2021-10-15 19:57:48.836553809 +0530 [root@sirius gana]# echo $((12585224*512)) " Bytes, " $((12585224*512-6443696128)) " Bytes extra" 6443634688 Bytes, -61440 Bytes extra [root@sirius gana]# find /opt/tmp -type f -printf "%S\t%p\n" | gawk '$1 <1.0 {print}' 0.99999 /opt/tmp/test1.bin [root@sirius gana]# [root@sirius gana]# qemu-img create -f qcow2 -o preallocation=full /opt/tmp/test1.bin 6G Formatting '/opt/tmp/test1.bin', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=full compression_type=zlib size=6442450944 lazy_refcounts=off refcount_bits=16 [root@sirius gana]# ls -ls /opt/tmp/test1.bin 6292612 -rw-r--r--. 1 root root 6443696128 Oct 15 20:02 /opt/tmp/test1.bin [root@sirius gana]# stat /opt/tmp/test1.bin File: /opt/tmp/test1.bin Size: 6443696128 Blocks: 12585224 IO Block: 4096 regular file Device: 22h/34d Inode: 1844292 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Context: unconfined_u:object_r:usr_t:s0 Access: 2021-10-15 20:02:56.627193628 +0530 Modify: 2021-10-15 20:02:56.617193505 +0530 Change: 2021-10-15 20:02:56.617193505 +0530 Birth: 2021-10-15 19:57:48.836553809 +0530 [root@sirius gana]# echo $((12585224*512)) " Bytes, " $((12585224*512-6443696128)) " Bytes extra" 6443634688 Bytes, -61440 Bytes extra [root@sirius gana]# find /opt/tmp -type f -printf "%S\t%p\n" | gawk '$1 <1.0 {print}' 0.99999 /opt/tmp/test1.bin [root@sirius gana]# [root@sirius gana]# # File size bigger than blocks allocated. [root@sirius gana]# # Both prealloc schemes 'falloc' and 'full' seem to be a bit sparse. [root@sirius gana]# # Which, is strange, as 'falloc' and 'full' preallocations are not meant to be sparse. [root@sirius gana]# # But, this is btrfs. What happens if 1 character is added to a large file on exfat? [root@sirius gana]# rm /mnt/a16/test1.bin rm: remove regular file '/mnt/a16/test1.bin'? y [root@sirius gana]# dd if=/dev/zero of=/mnt/a16/test1.bin bs=1M count=6000 6000+0 records in 6000+0 records out 6291456000 bytes (6.3 GB, 5.9 GiB) copied, 11.1701 s, 563 MB/s [root@sirius gana]# #printf "Q" >> /mnt/a16/test1.bin [root@sirius gana]# ls -ls /mnt/a16/test1.bin 6144000 -rwxr-xr-x. 1 root root 6291456000 Oct 15 19:45 /mnt/a16/test1.bin [root@sirius gana]# stat /mnt/a16/test1.bin File: /mnt/a16/test1.bin Size: 6291456000 Blocks: 12288000 IO Block: 131072 regular file Device: 10300h/66304d Inode: 271 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:dosfs_t:s0 Access: 2021-10-15 19:45:12.000000000 +0530 Modify: 2021-10-15 19:45:23.320000000 +0530 Change: 2021-10-15 19:45:23.320000000 +0530 Birth: 2021-10-15 19:45:12.150000000 +0530 [root@sirius gana]# printf "Q" >> /mnt/a16/test1.bin [root@sirius gana]# ls -ls /mnt/a16/test1.bin 6144128 -rwxr-xr-x. 1 root root 6291456001 Oct 15 19:46 /mnt/a16/test1.bin [root@sirius gana]# stat /mnt/a16/test1.bin File: /mnt/a16/test1.bin Size: 6291456001 Blocks: 12288256 IO Block: 131072 regular file Device: 10300h/66304d Inode: 271 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:dosfs_t:s0 Access: 2021-10-15 19:45:12.000000000 +0530 Modify: 2021-10-15 19:46:02.410000000 +0530 Change: 2021-10-15 19:46:02.410000000 +0530 Birth: 2021-10-15 19:45:12.150000000 +0530 root@sirius gana]# root@sirius gana]# # adding 1 extra character allocates 128 blocks = 65536 bytes = half a cluster Not that I am complaining. I don't know if this is correct or not. Why is the allocation unit 65536 = 64 Kb = 128 blocks, when clustersize is 131072 = 128 Kb = 256 blocks? The below link also says default cluster size is 128Kb for volume-sizes of 32 GB to 256 TB https://support.microsoft.com/en-us/topic/default-cluster-size-for-ntfs-fat-and-exfat-9772e6f1-e31a-00d7-e18f-73169155af95 ouch sorry, ignore previous comment. Not for this bug, my mistake, sorry. This bug still exists. Status: RESOLVED FIXED https://lore.kernel.org/ntfs3/aaf41f35-b702-b391-1cff-de4688b3bb65@paragon-software.com/ |