Bug 14021 - hfsplus caused data loss
Summary: hfsplus caused data loss
Status: RESOLVED CODE_FIX
Alias: None
Product: File System
Classification: Unclassified
Component: HFS/HFSPLUS (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Christoph Hellwig
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-08-20 02:17 UTC by Rogério Brito
Modified: 2010-11-04 07:40 UTC (History)
1 user (show)

See Also:
Kernel Version: 2.6.31-rc5
Tree: Mainline
Regression: Yes


Attachments

Description Rogério Brito 2009-08-20 02:17:19 UTC
Hi.

I am trying to package a new version of Apple's own fsck/mkfs for HFS+ filesystems so that they can be used in Linux as a way to transfer data among computers, but I had quite a surprise when I was using the hfsplus module.

I just created a 100MB file with nulls (dd if=/dev/null ...) and I created an HFS+ filesystem on it.

Then, I loop mounted it and tried to use it a little bit (in particular, applying patches with quilt on a source tree). The commands spit some errors about not being able to create links (I never had that problem before) and the directory where I was became empty!

Furthermore, here is a quite, quite strange directory listing:

,----
| rbrito@chagas:/media/usb7$ ls -lAF
| ls: hfsprogs_332.14-7.diff.gz: No such file or directory
| ls: hfsprogs_332.14-7.dsc: No such file or directory
| ls: hfsprogs_332.14.orig.tar.gz: No such file or directory
| ls: hfsprogs_332.18-1.diff.gz: No such file or directory
| ls: hfsprogs_332.18-1.dsc: No such file or directory
| ls: hfsprogs_332.18-1_amd64.changes: No such file or directory
| ls: hfsprogs_332.18-1_amd64.deb: No such file or directory
| ls: hfsprogs_332.18.orig.tar.gz: No such file or directory
| total 1636
| drwxr-xr-x 1 rbrito rbrito     39 Aug 17 08:13 hfsprogs-332.14/
| drwxr-xr-x 1 rbrito rbrito     39 Aug 17 08:13 hfsprogs-332.14/
| drwxr-xr-x 1 rbrito rbrito     45 Aug 17 15:30 hfsprogs-332.18/
| -rw-r--r-- 1 rbrito rbrito  35609 Aug 17 08:13 hfsprogs_332.14-7.diff.gz
| -rw-r--r-- 1 rbrito rbrito   1193 Aug 17 08:13 hfsprogs_332.14-7.dsc
| -rw-r--r-- 1 rbrito rbrito 714035 Aug 17 08:13 hfsprogs_332.14.orig.tar.gz
| -rw-r--r-- 1 rbrito rbrito  35342 Aug 17 15:26 hfsprogs_332.18-1.diff.gz
| -rw-r--r-- 1 rbrito rbrito    954 Aug 17 15:26 hfsprogs_332.18-1.dsc
| -rw-r--r-- 1 rbrito rbrito   2148 Aug 17 15:26 hfsprogs_332.18-1_amd64.changes
| -rw-r--r-- 1 rbrito rbrito 135398 Aug 17 15:26 hfsprogs_332.18-1_amd64.deb
| -rw-r--r-- 1 rbrito rbrito 732449 Aug 17 08:35 hfsprogs_332.18.orig.tar.gz
| rbrito@chagas:/media/usb7$
`----

I am using kernel 2.6.31-rc5-1rb.pre6 (that is, rc5 with git updates up to one or two days before rc6).

There are no messages in the dmesg, besides these:

,----
| [30991.501804] loop: module loaded
| [30991.513337] hfs: create hidden dir...
| [39897.867830] hfs: create hidden dir...
| [39960.061622] hfs: splitting index node...
`----

No stack traces, no nothing. Oh, I still have the disk image that I created, if it is of any interest.

Please let me know if I can provide any further information.


Regards, Rogério Brito.
Comment 1 Andrew Morton 2009-08-20 22:02:45 UTC
(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Thu, 20 Aug 2009 02:17:21 GMT
bugzilla-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=14021
> 
>            Summary: hfsplus caused data loss
>            Product: File System
>            Version: 2.5
>     Kernel Version: 2.6.31-rc5
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: HFS/HFSPLUS
>         AssignedTo: zippel@linux-m68k.org
>         ReportedBy: rbrito@ime.usp.br
>         Regression: Yes
> 
> 
> Hi.
> 
> I am trying to package a new version of Apple's own fsck/mkfs for HFS+
> filesystems so that they can be used in Linux as a way to transfer data among
> computers, but I had quite a surprise when I was using the hfsplus module.
> 
> I just created a 100MB file with nulls (dd if=/dev/null ...) and I created an
> HFS+ filesystem on it.
> 
> Then, I loop mounted it and tried to use it a little bit (in particular,
> applying patches with quilt on a source tree). The commands spit some errors
> about not being able to create links (I never had that problem before) and
> the
> directory where I was became empty!
> 
> Furthermore, here is a quite, quite strange directory listing:
> 
> ,----
> | rbrito@chagas:/media/usb7$ ls -lAF
> | ls: hfsprogs_332.14-7.diff.gz: No such file or directory
> | ls: hfsprogs_332.14-7.dsc: No such file or directory
> | ls: hfsprogs_332.14.orig.tar.gz: No such file or directory
> | ls: hfsprogs_332.18-1.diff.gz: No such file or directory
> | ls: hfsprogs_332.18-1.dsc: No such file or directory
> | ls: hfsprogs_332.18-1_amd64.changes: No such file or directory
> | ls: hfsprogs_332.18-1_amd64.deb: No such file or directory
> | ls: hfsprogs_332.18.orig.tar.gz: No such file or directory
> | total 1636
> | drwxr-xr-x 1 rbrito rbrito     39 Aug 17 08:13 hfsprogs-332.14/
> | drwxr-xr-x 1 rbrito rbrito     39 Aug 17 08:13 hfsprogs-332.14/
> | drwxr-xr-x 1 rbrito rbrito     45 Aug 17 15:30 hfsprogs-332.18/
> | -rw-r--r-- 1 rbrito rbrito  35609 Aug 17 08:13 hfsprogs_332.14-7.diff.gz
> | -rw-r--r-- 1 rbrito rbrito   1193 Aug 17 08:13 hfsprogs_332.14-7.dsc
> | -rw-r--r-- 1 rbrito rbrito 714035 Aug 17 08:13 hfsprogs_332.14.orig.tar.gz
> | -rw-r--r-- 1 rbrito rbrito  35342 Aug 17 15:26 hfsprogs_332.18-1.diff.gz
> | -rw-r--r-- 1 rbrito rbrito    954 Aug 17 15:26 hfsprogs_332.18-1.dsc
> | -rw-r--r-- 1 rbrito rbrito   2148 Aug 17 15:26
> hfsprogs_332.18-1_amd64.changes
> | -rw-r--r-- 1 rbrito rbrito 135398 Aug 17 15:26 hfsprogs_332.18-1_amd64.deb
> | -rw-r--r-- 1 rbrito rbrito 732449 Aug 17 08:35 hfsprogs_332.18.orig.tar.gz
> | rbrito@chagas:/media/usb7$
> `----
> 
> I am using kernel 2.6.31-rc5-1rb.pre6 (that is, rc5 with git updates up to
> one
> or two days before rc6).
> 
> There are no messages in the dmesg, besides these:
> 
> ,----
> | [30991.501804] loop: module loaded
> | [30991.513337] hfs: create hidden dir...
> | [39897.867830] hfs: create hidden dir...
> | [39960.061622] hfs: splitting index node...
> `----
> 
> No stack traces, no nothing. Oh, I still have the disk image that I created,
> if
> it is of any interest.
> 
> Please let me know if I can provide any further information.
> 

Gee.  Nobody really does much maintenance work on hfs/hfsplus any more.
I cc'ed the ppc guys as I expect that most HFS users are over there.

It seems like a pretty gross failure - others should be hitting it.

I wonder if it's a weird interaction with the loop driver.
Comment 2 Rogério Brito 2009-08-20 22:17:15 UTC
Hi, Andrew.

On Aug 20 2009, Andrew Morton wrote:
> On Thu, 20 Aug 2009 02:17:21 GMT
> bugzilla-daemon@bugzilla.kernel.org wrote:
> > ,----
> > | [30991.501804] loop: module loaded
> > | [30991.513337] hfs: create hidden dir...
> > | [39897.867830] hfs: create hidden dir...
> > | [39960.061622] hfs: splitting index node...
> > `----
> > 
> > No stack traces, no nothing. Oh, I still have the disk image that I
> > created, if it is of any interest.
>
> Gee.  Nobody really does much maintenance work on hfs/hfsplus any more.

Right.

I think that I could run some stress tests (say, like the exhaustive
regression tests that the ntfs-3g developers use), just to get some
basic functionality on HFS+ working.

> I cc'ed the ppc guys as I expect that most HFS users are over there.

Thanks once again, Andrew.

> It seems like a pretty gross failure - others should be hitting it.
> 
> I wonder if it's a weird interaction with the loop driver.

Just to make it clear: the reason why I loop-mounted the filesystem was
just to work with a low-risk situation and submit the image to the
recently "ported" fsck, just to get a little bit more confidence that I
didn't mess with the userspace side of things.

But, then, this surprise.

I had some minor inconsistencies with the HFS+ driver in the past, but
they were fixed by running the fsck.hfsplus that I packaged and I didn't
bother reporting it before. My mistake. :-(


Thanks, Rogério Brito.
Comment 3 Rogério Brito 2009-08-20 22:17:23 UTC
Hi, Andrew.

On Aug 20 2009, Andrew Morton wrote:
> On Thu, 20 Aug 2009 02:17:21 GMT
> bugzilla-daemon@bugzilla.kernel.org wrote:
> > ,----
> > | [30991.501804] loop: module loaded
> > | [30991.513337] hfs: create hidden dir...
> > | [39897.867830] hfs: create hidden dir...
> > | [39960.061622] hfs: splitting index node...
> > `----
> > 
> > No stack traces, no nothing. Oh, I still have the disk image that I
> > created, if it is of any interest.
>
> Gee.  Nobody really does much maintenance work on hfs/hfsplus any more.

Right.

I think that I could run some stress tests (say, like the exhaustive
regression tests that the ntfs-3g developers use), just to get some
basic functionality on HFS+ working.

> I cc'ed the ppc guys as I expect that most HFS users are over there.

Thanks once again, Andrew.

> It seems like a pretty gross failure - others should be hitting it.
> 
> I wonder if it's a weird interaction with the loop driver.

Just to make it clear: the reason why I loop-mounted the filesystem was
just to work with a low-risk situation and submit the image to the
recently "ported" fsck, just to get a little bit more confidence that I
didn't mess with the userspace side of things.

But, then, this surprise.

I had some minor inconsistencies with the HFS+ driver in the past, but
they were fixed by running the fsck.hfsplus that I packaged and I didn't
bother reporting it before. My mistake. :-(


Thanks, Rogério Brito.
Comment 4 Anonymous Emailer 2009-08-21 06:20:06 UTC
Reply-To: flar@allandria.com

On Thu, Aug 20, 2009 at 03:02:42PM -0700, Andrew Morton wrote:
> 
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> On Thu, 20 Aug 2009 02:17:21 GMT
> bugzilla-daemon@bugzilla.kernel.org wrote:
> 
> > http://bugzilla.kernel.org/show_bug.cgi?id=14021
> > 
> >            Summary: hfsplus caused data loss
> >            Product: File System
> >            Version: 2.5
> >     Kernel Version: 2.6.31-rc5
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: HFS/HFSPLUS
> >         AssignedTo: zippel@linux-m68k.org
> >         ReportedBy: rbrito@ime.usp.br
> >         Regression: Yes
> > 
> > 
> > Hi.
> > 
> > I am trying to package a new version of Apple's own fsck/mkfs for HFS+
> > filesystems so that they can be used in Linux as a way to transfer data
> among
> > computers, but I had quite a surprise when I was using the hfsplus module.
> > 
> > I just created a 100MB file with nulls (dd if=/dev/null ...) and I created
> an
> > HFS+ filesystem on it.
> > 
> > Then, I loop mounted it and tried to use it a little bit (in particular,
> > applying patches with quilt on a source tree). The commands spit some
> errors
> > about not being able to create links (I never had that problem before) and
> the
> > directory where I was became empty!
> > 
> > Furthermore, here is a quite, quite strange directory listing:
> > 
> > ,----
> > | rbrito@chagas:/media/usb7$ ls -lAF
> > | ls: hfsprogs_332.14-7.diff.gz: No such file or directory
> > | ls: hfsprogs_332.14-7.dsc: No such file or directory
> > | ls: hfsprogs_332.14.orig.tar.gz: No such file or directory
> > | ls: hfsprogs_332.18-1.diff.gz: No such file or directory
> > | ls: hfsprogs_332.18-1.dsc: No such file or directory
> > | ls: hfsprogs_332.18-1_amd64.changes: No such file or directory
> > | ls: hfsprogs_332.18-1_amd64.deb: No such file or directory
> > | ls: hfsprogs_332.18.orig.tar.gz: No such file or directory
> > | total 1636
> > | drwxr-xr-x 1 rbrito rbrito     39 Aug 17 08:13 hfsprogs-332.14/
> > | drwxr-xr-x 1 rbrito rbrito     39 Aug 17 08:13 hfsprogs-332.14/
> > | drwxr-xr-x 1 rbrito rbrito     45 Aug 17 15:30 hfsprogs-332.18/
> > | -rw-r--r-- 1 rbrito rbrito  35609 Aug 17 08:13 hfsprogs_332.14-7.diff.gz
> > | -rw-r--r-- 1 rbrito rbrito   1193 Aug 17 08:13 hfsprogs_332.14-7.dsc
> > | -rw-r--r-- 1 rbrito rbrito 714035 Aug 17 08:13
> hfsprogs_332.14.orig.tar.gz
> > | -rw-r--r-- 1 rbrito rbrito  35342 Aug 17 15:26 hfsprogs_332.18-1.diff.gz
> > | -rw-r--r-- 1 rbrito rbrito    954 Aug 17 15:26 hfsprogs_332.18-1.dsc
> > | -rw-r--r-- 1 rbrito rbrito   2148 Aug 17 15:26
> > hfsprogs_332.18-1_amd64.changes
> > | -rw-r--r-- 1 rbrito rbrito 135398 Aug 17 15:26
> hfsprogs_332.18-1_amd64.deb
> > | -rw-r--r-- 1 rbrito rbrito 732449 Aug 17 08:35
> hfsprogs_332.18.orig.tar.gz
> > | rbrito@chagas:/media/usb7$
> > `----
> > 
> > I am using kernel 2.6.31-rc5-1rb.pre6 (that is, rc5 with git updates up to
> one
> > or two days before rc6).
> > 
> > There are no messages in the dmesg, besides these:
> > 
> > ,----
> > | [30991.501804] loop: module loaded
> > | [30991.513337] hfs: create hidden dir...
> > | [39897.867830] hfs: create hidden dir...
> > | [39960.061622] hfs: splitting index node...
> > `----
> > 
> > No stack traces, no nothing. Oh, I still have the disk image that I
> created, if
> > it is of any interest.
> > 
> > Please let me know if I can provide any further information.
> > 
> 
> Gee.  Nobody really does much maintenance work on hfs/hfsplus any more.
> I cc'ed the ppc guys as I expect that most HFS users are over there.
> 
> It seems like a pretty gross failure - others should be hitting it.
> 
> I wonder if it's a weird interaction with the loop driver.

I can explain the ls output in a very generic sense. This is what you
get if the lookup fails on a name returned by a readdir. I suspect
that creating hard links is failing in some way. In particular, the
"hidden dir" mentioned in the log is used to save the real files for
hard links.

As a quick explanation, HFS+ doesn't have a real concept of hard
links. Apple hacked it in after the fact. The way it works is that
there is a special hidden directory, and the real catalog entries
that match the real names are just references to the hidden file.
Both HFS and HFS+ have no separate notion of directory entries and
inode data. The metadata is stored in records keyed by file name.

Hard links were added after I stopped working on the HFS+ code, so
I don't know it in detail. I know it works for reading hard links,
but I never actually tried creating more links.

	Brad Boyer
	flar@allandria.com
Comment 5 Benjamin Herrenschmidt 2009-08-21 07:29:11 UTC
On Thu, 2009-08-20 at 15:02 -0700, Andrew Morton wrote:
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).

Is Roman Zippel still around ? (I added him to the CC list). AFAIK, He
maintains HFS and HFS+ (and was the last one to do any major work on
them).

Cheers,
Ben.

> On Thu, 20 Aug 2009 02:17:21 GMT
> bugzilla-daemon@bugzilla.kernel.org wrote:
> 
> > http://bugzilla.kernel.org/show_bug.cgi?id=14021
> > 
> >            Summary: hfsplus caused data loss
> >            Product: File System
> >            Version: 2.5
> >     Kernel Version: 2.6.31-rc5
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: HFS/HFSPLUS
> >         AssignedTo: zippel@linux-m68k.org
> >         ReportedBy: rbrito@ime.usp.br
> >         Regression: Yes
> > 
> > 
> > Hi.
> > 
> > I am trying to package a new version of Apple's own fsck/mkfs for HFS+
> > filesystems so that they can be used in Linux as a way to transfer data
> among
> > computers, but I had quite a surprise when I was using the hfsplus module.
> > 
> > I just created a 100MB file with nulls (dd if=/dev/null ...) and I created
> an
> > HFS+ filesystem on it.
> > 
> > Then, I loop mounted it and tried to use it a little bit (in particular,
> > applying patches with quilt on a source tree). The commands spit some
> errors
> > about not being able to create links (I never had that problem before) and
> the
> > directory where I was became empty!
> > 
> > Furthermore, here is a quite, quite strange directory listing:
> > 
> > ,----
> > | rbrito@chagas:/media/usb7$ ls -lAF
> > | ls: hfsprogs_332.14-7.diff.gz: No such file or directory
> > | ls: hfsprogs_332.14-7.dsc: No such file or directory
> > | ls: hfsprogs_332.14.orig.tar.gz: No such file or directory
> > | ls: hfsprogs_332.18-1.diff.gz: No such file or directory
> > | ls: hfsprogs_332.18-1.dsc: No such file or directory
> > | ls: hfsprogs_332.18-1_amd64.changes: No such file or directory
> > | ls: hfsprogs_332.18-1_amd64.deb: No such file or directory
> > | ls: hfsprogs_332.18.orig.tar.gz: No such file or directory
> > | total 1636
> > | drwxr-xr-x 1 rbrito rbrito     39 Aug 17 08:13 hfsprogs-332.14/
> > | drwxr-xr-x 1 rbrito rbrito     39 Aug 17 08:13 hfsprogs-332.14/
> > | drwxr-xr-x 1 rbrito rbrito     45 Aug 17 15:30 hfsprogs-332.18/
> > | -rw-r--r-- 1 rbrito rbrito  35609 Aug 17 08:13 hfsprogs_332.14-7.diff.gz
> > | -rw-r--r-- 1 rbrito rbrito   1193 Aug 17 08:13 hfsprogs_332.14-7.dsc
> > | -rw-r--r-- 1 rbrito rbrito 714035 Aug 17 08:13
> hfsprogs_332.14.orig.tar.gz
> > | -rw-r--r-- 1 rbrito rbrito  35342 Aug 17 15:26 hfsprogs_332.18-1.diff.gz
> > | -rw-r--r-- 1 rbrito rbrito    954 Aug 17 15:26 hfsprogs_332.18-1.dsc
> > | -rw-r--r-- 1 rbrito rbrito   2148 Aug 17 15:26
> > hfsprogs_332.18-1_amd64.changes
> > | -rw-r--r-- 1 rbrito rbrito 135398 Aug 17 15:26
> hfsprogs_332.18-1_amd64.deb
> > | -rw-r--r-- 1 rbrito rbrito 732449 Aug 17 08:35
> hfsprogs_332.18.orig.tar.gz
> > | rbrito@chagas:/media/usb7$
> > `----
> > 
> > I am using kernel 2.6.31-rc5-1rb.pre6 (that is, rc5 with git updates up to
> one
> > or two days before rc6).
> > 
> > There are no messages in the dmesg, besides these:
> > 
> > ,----
> > | [30991.501804] loop: module loaded
> > | [30991.513337] hfs: create hidden dir...
> > | [39897.867830] hfs: create hidden dir...
> > | [39960.061622] hfs: splitting index node...
> > `----
> > 
> > No stack traces, no nothing. Oh, I still have the disk image that I
> created, if
> > it is of any interest.
> > 
> > Please let me know if I can provide any further information.
> > 
> 
> Gee.  Nobody really does much maintenance work on hfs/hfsplus any more.
> I cc'ed the ppc guys as I expect that most HFS users are over there.
> 
> It seems like a pretty gross failure - others should be hitting it.
> 
> I wonder if it's a weird interaction with the loop driver.
> 
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev
Comment 6 Andrew Morton 2009-08-21 07:37:15 UTC
On Fri, 21 Aug 2009 17:29:01 +1000 Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

> On Thu, 2009-08-20 at 15:02 -0700, Andrew Morton wrote:
> > (switched to email.  Please respond via emailed reply-to-all, not via the
> > bugzilla web interface).
> 
> Is Roman Zippel still around ? (I added him to the CC list). AFAIK, He
> maintains HFS and HFS+ (and was the last one to do any major work on
> them).

I don't think I've heard from Roman since January of this year.
Comment 7 Christoph Hellwig 2010-10-01 06:34:28 UTC
If you still have the image, can you make is accessible to me?
Comment 9 Christoph Hellwig 2010-10-28 12:50:49 UTC
The fix above is now in mainline.
Comment 10 Rogério Brito 2010-11-04 07:40:18 UTC
Dear Chistoph,

On Thu, Oct 28, 2010 at 10:50,  <bugzilla-daemon@bugzilla.kernel.org> wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=14021
> --- Comment #9 from Christoph Hellwig <hch@lst.de>  2010-10-28 12:50:49 ---
> The fix above is now in mainline.

Thank you very much for being very speedy fixing this bug in the new
version of the kernel. I had some trips between countries and I barely
had access to my e-mail.

And a second thank you for the patch that you sent me on Debian's BTS.
I will integrate it in the next revision of the package (which, BTW, I
plan on uploading to the archive in this weekend.


Thanks once again,

Note You need to log in before you can comment on or make changes to this bug.