Hi Jens! I will write an mail to linux-kernel, linux-m68k and possibly linux-ide as well. With my 2 TB mixed Amiga/Linux backup disk, which I partioned under AmigaOS 4.0/1 with Media Toolbox I get the following in Linux: Jun 17 07:28:14 merkaba kernel: [30852.968978] sata_sil24 0000:05:00.0: enabling device (0000 -> 0003) Jun 17 07:28:14 merkaba kernel: [30852.969401] scsi9 : sata_sil24 Jun 17 07:28:14 merkaba kernel: [30852.969533] ata10: SATA max UDMA/100 host m128@0xf1c02000 port 0xf1c00000 irq 19 Jun 17 07:28:16 merkaba kernel: [30855.163712] ata10: SATA link up 3.0 Gbps (SStatus 123 SControl 0) Jun 17 07:28:16 merkaba kernel: [30855.165014] ata10.00: ATA-8: Hitachi HDS5C3020ALA632, ML6OA580, max UDMA/133 Jun 17 07:28:16 merkaba kernel: [30855.165017] ata10.00: 3907029168 sectors, multi 16: LBA48 NCQ (depth 31/32) Jun 17 07:28:16 merkaba kernel: [30855.166378] ata10.00: configured for UDMA/100 Jun 17 07:28:16 merkaba kernel: [30855.166477] scsi 9:0:0:0: Direct-Access ATA Hitachi HDS5C302 ML6O PQ: 0 ANSI: 5 Jun 17 07:28:16 merkaba kernel: [30855.166653] sd 9:0:0:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB) Jun 17 07:28:16 merkaba kernel: [30855.166699] sd 9:0:0:0: [sdb] Write Protect is off Jun 17 07:28:16 merkaba kernel: [30855.166702] sd 9:0:0:0: [sdb] Mode Sense: 00 3a 00 00 Jun 17 07:28:16 merkaba kernel: [30855.166726] sd 9:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jun 17 07:28:16 merkaba kernel: [30855.200936] sdb: RDSK (512) sdb1 (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb 4) Jun 17 07:28:16 merkaba kernel: [30855.200943] sdb: p1 size 18446744072560312368 extends beyond EOD, enabling native capacity Jun 17 07:28:16 merkaba kernel: [30855.201344] sdb: RDSK (512) sdb1 (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb 4) Jun 17 07:28:16 merkaba kernel: [30855.201347] sdb: p1 size 18446744072560312368 extends beyond EOD, truncated Jun 17 07:28:16 merkaba kernel: [30855.201398] sdb: p2 start 18446744072560314432 is beyond EOD, truncated Jun 17 07:28:16 merkaba kernel: [30855.201400] sdb: p3 start 18446744073189460080 is beyond EOD, truncated Jun 17 07:28:16 merkaba kernel: [30855.201570] sd 9:0:0:0: [sdb] Attached SCSI disk The first partition seems to be way to big: merkaba:~#1> amiga-fdisk -l /dev/sdc Disk /dev/sdc: 3 heads, 16 sectors, 81396441 cylinders, RDB: 0 Logical Cylinders from 43 to 81396440, 24576 bytes/Cylinder Device Boot Mount Begin End Size Pri BBlks System /dev/sdc1 * 43 65536043 1572864024 0 0 Linux native /dev/sdc2 * 65536044 78643244 314572824 0 0 [unknown] /dev/sdc3 * 78643245 81396440 66076704 0 0 Amiga FFS Int. (sdc2 is JXFS, a new filesystem in AmigaOS 4 that is not known to Linux yet) In cat /proc/partitions I get: merkaba:~> cat /proc/partitions major minor #blocks name […] 8 16 1953514584 sdb 8 17 1953513552 sdb1 merkaba:~> Thus the Debian Linux Kernel 3.4.1 I am using here, truncates the first oversized partition and has no space for the other, too, which therefore are inaccessible under Linux. I didn´t notice this initially, but it happened from the beginning, I have an old amiga-fdisk listing that is exactly the same. Amiga Mediatoolbox has a different oppinion on the partition layout: LVM aka sdc1: 1500 GByte, 43 to 65536043 cyl, size 65536001 Zylinder BAK aka sdc2: 300 GByte, 65536044 to 78643244 cyl, size 13107201 Zylinder TAUSCH2 aka sdc3: 63,016 GByte, 78643245 to 81396440 cyl, didn´t note the size here, but as far as I remember was okay as well But it seems to be confused about the whole size of the disk as well: Logical sizes: Blocks per cylinder: 48 Cylinders: 81396441 Sectors: -397938128 Blocksize: 512 The sectors seem overflowed. So it might be a problem with RDB and 2TB disks and nothing to do with Linux. But still on AmigaOS 4.1 I can access the two Amiga partitions after the Linux partition. I have another 500GB disk for backup up my Sam440ep AmigaOS 4.1 "Amiga", I plan to repartition the 2 TB disk as GPT anyway, but since MediaToolBox in AmigaOS 4.1 has a different meaning about the partioning and this can cause serious data loss, I think its good to look at it. I had a BTRFS filesystem that had some checksum errors. Maybe thats somehow related to this issue and AmigaOS and/or Linux overwrote something it shouldn´t have touched. I will report a bug with AmigaOS 4.1 developers as well to get more details.
Okay, detailed kernel version: merkaba:~> cat /proc/version Linux version 3.4-trunk-amd64 (Debian 3.4.1-1~experimental.1) (debian-kernel@lists.debian.org) (gcc version 4.6.3 (Debian 4.6.3-1) ) #1 SMP Wed Jun 6 10:34:53 CEST 2012
Created attachment 73771 [details] mediatoolbox binary profile of RDB, should be just a binary copy A short intro on the Amiga RDB is at http://en.wikipedia.org/wiki/Amiga_Rigid_Disk_Block More detailed in the link there: http://lclevy.free.fr/adflib/adf_info.html#p6 Seems to be a binary copy of the RDB: martin@merkaba:~/Computer/Festplatten/Steigerwald/Amiga> hd profile-binary.img | head 00000000 52 44 53 4b 00 00 00 80 ef 69 b5 58 00 00 00 07 |RDSK.....i.X....| 00000010 00 00 02 00 00 00 00 17 ff ff ff ff 00 00 00 01 |................| 00000020 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| * 00000040 04 da 02 d9 00 00 00 10 00 00 00 03 00 00 00 01 |................| 00000050 04 da 02 d9 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000080 00 00 00 00 00 00 08 0f 00 00 00 2b 04 da 02 d8 |...........+....| 00000090 00 00 00 30 00 00 00 00 00 00 00 03 00 00 00 00 |...0............|
Here the relevant information in mountlist format: LVM aka sdc1: Device = usbdisk.device Unit = 0 Flags = 0 Surfaces = 3 SectorsPerBlock = 1 BlocksPerTrack = 16 Reserved = 2 LowCyl = 43 HighCyl = 65536043 Buffers = 100 BufMemType = 1 MaxTransfer = 0x7fffffff Mask = 0x7ffffffc DosType = 0x4c4e5800 GlobVec = -1 # BAK aka sdc2: Device = usbdisk.device Unit = 0 Flags = 0 Surfaces = 3 SectorsPerBlock = 1 BlocksPerTrack = 16 Reserved = 2 LowCyl = 65536044 HighCyl = 78643244 Buffers = 5000 BufMemType = 1 MaxTransfer = 0x7fffffff Mask = 0x7ffffffc DosType = 0x4a584604 GlobVec = -1 # TAUSCH2 aka sdc3: Device = usbdisk.device Unit = 0 Flags = 0 Surfaces = 3 SectorsPerBlock = 4 BlocksPerTrack = 16 Reserved = 2 LowCyl = 78643245 HighCyl = 81396440 Buffers = 2000 BufMemType = 1 MaxTransfer = 0x7fffffff Mask = 0x7ffffffc DosType = 0x444f5303 GlobVec = -1 # From the Amiga side this looks correct. Even amiga-fdisk prints start and end cylinders correctly: merkaba:~> amiga-fdisk -l /dev/sdc Disk /dev/sdc: 3 heads, 16 sectors, 81396441 cylinders, RDB: 0 Logical Cylinders from 43 to 81396440, 24576 bytes/Cylinder Device Boot Mount Begin End Size Pri BBlks System /dev/sdc1 * 43 65536043 1572864024 0 0 Linux native /dev/sdc2 * 65536044 78643244 314572824 0 0 [unknown] /dev/sdc3 * 78643245 81396440 66076704 0 0 Amiga FFS Int. Its just the size of the first partitions that seems miscalculated, cause 65536043 - 43 is 65536000, not 1572864024 So are Linux kernel and amiga-fdisk overflowing there?
Created attachment 73781 [details] screenshot of mediatoolbox showing the RDB informations All seems good except for the sectors count which seems to be overflowing. Whether thats just a cosmetic problem or related to this issue I do not know yet. I will report this with AmigaOS developers to find out more.
Created attachment 73791 [details] screenshot of mediatoolbox´s view of the partitions with the first one selected MediaToolbox shows correct size of the first partition.
Raising importance to high. Granted not many people might use Amiga RDB disks with Linux, but this seems to be a *serious* data loss issue. Cause I now found out that the LVM I created on the disk is of the size of the whole disk, while at least 300 GB is for the Amiga backup partitions. AmigaOS 4.1 sees the Amiga backup partition. So the backups from Amiga and Linux may overwrite each other. This might explain the BTRFS checksum errors as well. merkaba:~> pvdisplay /dev/sdb1 --- Physical volume --- PV Name /dev/sdb1 VG Name steigerwald PV Size 1,82 TiB / not usable 4,08 MiB Allocatable yes PE Size 4,00 MiB Total PE 476931 Free PE 105731 Allocated PE 371200 PV UUID ZXMECC-JAir-lX8Q-rLhS-W1cS-quwz-b3FXWp merkaba:~> vgdisplay /dev/sdb1 Volume group "sdb1" not found merkaba:~#5> vgdisplay steigerwald --- Volume group --- VG Name steigerwald System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 7 VG Access read/write VG Status resizable MAX LV 0 Cur LV 5 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 1,82 TiB PE Size 4,00 MiB Total PE 476931 Alloc PE / Size 371200 / 1,42 TiB Free PE / Size 105731 / 413,01 GiB VG UUID uhjjE1-0yrD-Ch1A-d9qL-P5jY-UDXE-io4bpi And basically thats explainable by: merkaba:~> cat /proc/partitions major minor #blocks name […] 8 16 1953514584 sdb 8 17 1953513552 sdb1 Actually I see two issues in there: 1) Linux seems to miscalculate the size of the first partitions grossly. 2) It then *truncates* it to the whole disk and risks serious *data loss* instead of refusing to do *anything* with the disk. I will report another bug on issue 2. If a partition is out of bounds I think the safest choice is to leave it untouched completely. Just truncating and using it is asking for serious trouble. I will scrub the BTRFS partitions and fsck the other partitions to test whether the Amiga backup which I aborted after seeing this, to check for errors. Then I will redo the disk as GPT as I need a reliable backup.
Am Montag, 18. Juni 2012 schrieb jdow: > On 2012/06/17 14:20, Martin Steigerwald wrote: > > Am Sonntag, 17. Juni 2012 schrieb jdow: > >> On 2012/06/17 09:36, Geert Uytterhoeven wrote: > >>> On Sun, Jun 17, 2012 at 2:58 PM, Martin Steigerwald > > > > <Martin@lichtvoll.de> wrote: > >>>> Am Sonntag, 17. Juni 2012 schrieb jdow: > >>>> | JXFS 64 bit file system > >>>> | > >>>> | With AmigaOS 4.x a new file system has been introduced called > >>>> | JXFS. It is a totally new 64 bit file system that supports > >>>> | partitions up to 16 TB in size. It is a modern journalling file > >>>> | system, which means that it reduces data loss if data writes to > >>>> | the disk are interrupted. It is the fastest and most reliable > >>>> | file system ever created for AmigaOS. > >>>> > >>>> http://www.amigaos.net/content/1/features > >>>> > >>>> Well I asked AmigaOS 4 developers about this issue as well. Lets > >>>> see what they say about 2 TB limits. > >>> > >>> 16 TB = 2 TB * 8. Perhaps they increased the block size from 512 to > >>> 4096? > >>> > >>> block/partitions/amiga.c reads the block size from > >>> RigidDiskBlock.rdb_BlockBytes, > >>> but after conversion to 512-byte blocks, all further calculations > >>> are done on "int", so it will overflow for disks larger than 2 > >>> TiB. > >>> > >>> Note that in your profile-binary.img, the field is 0x200, i.e. 512 > >>> bytes per block, > >>> so I'll have to get a deeper look into your RDB first... > > > > […] > > > >> Note that the work I did on the Linux RDB code eons ago took full > >> cognizance of the physical and virtual block sizes. > > > > […] > > > >> I've asked Martin for a digital copy of his RDBs and what he thinks > >> the partition(s) should look like. I should also be told whether > >> the disk is supposed to be solely Amiga OSs or not. I gather it's > >> not. > > > > Its all in the bug report. profile-binary.img should be it. > > > > Its small so I attach it. > > > > Meanwhile I try to get the currently supported maximum disk size out > > from the OS 4 developers. Maybe JXFS is playing tricks that other > > filesystems do not play or simply has a different implementation. > > > > Thanks, > > I've sent Jens a proposed fix changing these lines in amiga.c. > ===8<--- was > struct PartitionBlock *pb; > int start_sect, nr_sects, blk, part, res = 0; > int blksize = 1; /* Multiplier for disk block size */ > ===8<--- becomes changing line 338 into lines 338-339 > struct PartitionBlock *pb; > sector_t start_sect, nr_sects; > int blk, part, res = 0; > int blksize = 1; /* Multiplier for disk block size */ > ===8<--- > > (I'm working without proper tools at the moment or I'd make a real > diff. Sorry.) > > This fix may not be complete. The RDBs contain provisions for > describing the physical disk block size and how many physical blocks > are used in a file system's logical blocks. This MAY not work quite > right large physical block sizes. Way better: dmesg: Jun 19 21:19:09 merkaba kernel: [ 7891.819552] ata8: SATA link up 3.0 Gbps (SStatus 123 SControl 0) Jun 19 21:19:09 merkaba kernel: [ 7891.821306] ata8.00: ATA-8: Hitachi HDS5C3020ALA632, ML6OA580, max UDMA/133 Jun 19 21:19:09 merkaba kernel: [ 7891.821315] ata8.00: 3907029168 sectors, multi 0: LBA48 NCQ (depth 31/32) Jun 19 21:19:09 merkaba kernel: [ 7891.823252] ata8.00: configured for UDMA/100 Jun 19 21:19:09 merkaba kernel: [ 7891.823589] scsi 7:0:0:0: Direct-Access ATA Hitachi HDS5C302 ML6O PQ: 0 ANSI: 5 Jun 19 21:19:09 merkaba kernel: [ 7891.824126] sd 7:0:0:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB) Jun 19 21:19:09 merkaba kernel: [ 7891.824440] sd 7:0:0:0: [sdb] Write Protect is off Jun 19 21:19:09 merkaba kernel: [ 7891.824452] sd 7:0:0:0: [sdb] Mode Sense: 00 3a 00 00 Jun 19 21:19:09 merkaba kernel: [ 7891.824531] sd 7:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jun 19 21:19:09 merkaba kernel: [ 7891.843284] sdb: RDSK (512) sdb1 (LNX^@)(res 2 spb 1) sdb2 (JXF^D)(res 2 spb 1) sdb3 (DOS^C)(res 2 spb 4) Jun 19 21:19:09 merkaba kernel: [ 7891.844055] sd 7:0:0:0: [sdb] Attached SCSI disk cat /proc/partitions: major minor #blocks name […] 8 16 1953514584 sdb 8 17 1572864024 sdb1 8 18 314572824 sdb2 8 19 66076704 sdb3 1572864024 + 314572824 + 66076704 = 1953513552 which is below the complete size of the disk. Partition start analysis: merkaba:~> file -sk /dev/sdb1 /dev/sdb1: sticky LVM2 PV (Linux Logical Volume Manager), UUID: ZXMECC-JAir-lX8Q-rLhS-W1cS-quwz-b3FXWp, size: 2000397877248 merkaba:~> file -sk /dev/sdb2 /dev/sdb2: sticky data merkaba:~> file -sk /dev/sdb3 /dev/sdb3: sticky Amiga Inter FFS disk merkaba:~> hd /dev/sdb2 | head 00000000 4a 58 46 04 11 1a 0f 0c 00 00 00 00 00 00 00 00 |JXF.............| 00000010 00 00 00 00 00 11 00 00 3e db 3d 54 40 00 00 00 |........>.=T@...| 00000020 00 00 00 00 00 00 00 00 00 00 01 77 00 10 80 00 |...........w....| 00000030 00 00 01 c2 00 10 e0 00 25 80 00 30 00 00 02 00 |........%..0....| 00000040 00 00 00 30 00 00 00 00 00 00 00 00 00 00 00 00 |...0............| 00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000060 00 00 00 00 00 00 00 00 00 00 08 00 00 00 00 00 |................| 00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00000200 54 52 4f 4b ab ad b0 b1 00 00 00 01 00 00 00 00 |TROK............| I don´t know the on-disk format for JXFS, but this could be it. amiga-fdisk looks the same as before: merkaba:~> amiga-fdisk -l /dev/sdb Disk /dev/sdb: 3 heads, 16 sectors, 81396441 cylinders, RDB: 0 Logical Cylinders from 43 to 81396440, 24576 bytes/Cylinder Device Boot Mount Begin End Size Pri BBlks System /dev/sdb1 * 43 65536043 1572864024 0 0 Linux native /dev/sdb2 * 65536044 78643244 314572824 0 0 [unknown] /dev/sdb3 * 78643245 81396440 66076704 0 0 Amiga FFS Int. But obviously its right anyway: It seems to display the size in blocks, not in cylinders. Media Toolbox says: LVM aka sdc1: 1500 GByte, 43 to 65536043 cyl, size 65536001 Zylinder BAK aka sdc2: 300 GByte, 65536044 to 78643244 cyl, size 13107201 Zylinder TAUSCH2 aka sdc3: 63,016 GByte, 78643245 to 81396440 cyl, didn´t note the size here, but as far as I remember was okay as well with Blocks per cylinder: 48 Cylinders: 81396441 So thats: 65536001 * 48 / 2 = 1572864024 13107201 * 48 / 2 = 314572824 ( 81396440 - 78643245 ) * 48 / 2 = 66076680 (I verified from a windowshot I made that 81396440 - 78643245 = 2753195 is indeed displayed by Media Toolbox as size of the partition) So this is looking pretty good. Many thanks, Joanne for your detective work and the fix. Tested with: martin@merkaba:~> cat /proc/version Linux version 3.5.0-rc3-fix-bug-43511+ (martin@merkaba) (gcc version 4.7.0 (Debian 4.7.1-1) ) #1 SMP Tue Jun 19 14:31:56 CEST 2012 Tested-By: Martin Steigerwald <martin@lichtvoll.de> Reviewed-By: Martin Steigerwald <martin@lichtvoll.de> Patch from Joanne in diff format: martin@merkaba:~/Computer/Merkaba/Kernel/linux-2.6> git diff HEAD~ | cat diff --git a/block/partitions/amiga.c b/block/partitions/amiga.c index 70cbf44..42d36f9 100644 --- a/block/partitions/amiga.c +++ b/block/partitions/amiga.c @@ -29,7 +29,8 @@ int amiga_partition(struct parsed_partitions *state) unsigned char *data; struct RigidDiskBlock *rdb; struct PartitionBlock *pb; - int start_sect, nr_sects, blk, part, res = 0; + sector_t start_sect, nr_sects; + int blk, part, res = 0; int blksize = 1; /* Multiplier for disk block size */ int slot = 1; char b[BDEVNAME_SIZE]; Thanks, -- Martin 'Helios' Steigerwald - http://www.Lichtvoll.de GPG: 03B0 0D6C 0040 0710 4AFA B82F 991B EAAC A599 84C7
Link to the thread with above message from today: Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in AmigaOS 4.1 https://lkml.org/lkml/2012/6/17/6
As the fix did not make it into Linux kernel back then, I provided no patch for review, and Michael Schmitz works on a new patch, I am reopening this. I am lowering the importance, as to what I learned so far, even tough Media Toolbox allowed me to create this, I bet it is still quite unusual.
This fix meanwhile made it in: [1/3] block: fix signed int overflow in Amiga partition support commit: fc3d092c6bb48d5865fec15ed5b333c12f36288c [2/3] block: change all __u32 annotations to __be32 in affs_hardblocks.h commit: 95a55437dc49fb3342c82e61f5472a71c63d9ed0 [3/3] block: add overflow checks for Amiga partition support commit: b6f3f28f604ba3de4724ad82bea6adb1300c0b5f https://lore.kernel.org/linux-m68k/29a4b7210abd927e78b9c576f9283403f494e555.camel@physik.fu-berlin.de/T/#m0eea1b4b33ff8d96ebfe35546bce3a67256ee166 However there appears to be an issue with a "-1" as partition size discussed here: [FSL P50x0] [PASEMI] The Access to partitions on disks with an Amiga partition table doesn't work anymore after the block updates 2023-06-23 https://lore.kernel.org/linux-m68k/a113cb83-9f82-fd39-f132-41ba4c259265@gmail.com/T/#t It may have been used as an end of partition list marker. This issue is triggered by fixing this bug. It is a different issue, however not (yet) reported here, so keeping this bug report open for a little longer for now.
(In reply to Martin Steigerwald from comment #8) > Link to the thread with above message from today: > > Partitions: Amiga RDB partition on 2 TB disk way too big, while OK in > AmigaOS 4.1 > https://lkml.org/lkml/2012/6/17/6 https://lore.kernel.org/linux-m68k/201206170841.20222.Martin@lichtvoll.de/
The overflow fix and the fix for the issue with signedness went in both to vanilla and to various stable kernels. So this case can finally be closed. Many thanks to everyone involved, especially to Michael.