Bug 217074

I ran into this as well.  I believe it is the result of trying to use mdadm on a kernel with BLOCK_LEGACY_AUTOLOAD disabled.

Hi Tim,

Thanks for your feedback, this may well be the case as I have it disabled, indeed:

grep BLOCK_LEGACY_AUTOLOAD /etc/kernels/kernel-config-6.1.12-gentoo-x86_64 
# CONFIG_BLOCK_LEGACY_AUTOLOAD is not set

I did notice the change when I was configuring and trying the 6.1.12 kernel, so I've manually loaded 'md_mod' and 'raid0' after the system had loaded. And this did not help either, thus the bug report. 'raid0' depends on 'md_mod' and 'md_mod' has no other dependencies according to modinfo (or these are compiled into the kernel in my case)...

As the cadence of rebooting this system is not so frequent, I cannot check if this was caused by CONFIG_BLOCK_LEGACY_AUTOLOAD. Even if it is fixed by enabling the block legacy autoload, I think it should also work by manually loading the kernel modules?

Cheers,
-Nikolay

I ran into this as well, also upon upgrading from 5.15.88 to 6.1.12, only in my case it's an mdraid *raid1* array that I cannot assemble, so the issue is not specific to raid0.

While comparing kernel configs, I too suspected CONFIG_DEVTMPFS_SAFE=y, as I intentionally enabled that option while building my new kernel. It's good to know that that's not the issue. (Thanks, Nikolay.)

My kernel does not have any block-layer code compiled as modules — it's all built in — so I would not think that CONFIG_BLOCK_LEGACY_AUTOLOAD would be the culprit, though I too have that option disabled in my new kernel. Given the error that occurs (ENXIO) and where it occurs in the strace (opening a device node for the mdraid device), that option does look highly suspect. I'll turn it on and try again during my maintenance window tonight.

Indeed, enabling CONFIG_BLOCK_LEGACY_AUTOLOAD allowed mdadm to assemble my arrays. Does mdadm need to be updated so as not to depend on legacy behavior?

mdadm can be told not to depend on this legacy behavior by adding the line

 CREATE names=yes

to /etc/mdadm.conf.
This works since mdadm-4.1.

Maybe we need to make that the default now.  It works on any kernel since 2.6.29.

@Neil Brown: Are you sure about that? Here's what the mdadm.conf(5) man page has to say about that option:

names=yes
Since Linux 2.6.29 it has been possible to create md devices with a name like md_home rather than just a number, like md3. mdadm will use the numeric alternative by default as other tools that interact with md arrays may expect only numbers. If names=yes is given in mdadm.conf then mdadm will use a name when appropriate. If names=no is given, then non-numeric md device names will not be used even if the default changes in a future release of mdadm.

This option seems irrelevant to the issue at hand here. I'm not letting mdadm choose the name for my md device node; rather, I am specifying it. I don't use mdadm.conf. My command line is:

mdadm --assemble /dev/md/root \
  --uuid=ab1712b4:d3c0c78b:2bc4e96b:4dc8c563

The issue is that mdadm creates the /dev/md/root block special device node with major number 9, minor number 127, and then attempts to open it, and the kernel complains, "whoa, no such device exists." The kernel doesn't care about the device node's path in the file system; it cares about the 9:127, and that's going to be the same regardless of whether I'm picking the path name myself or I'm letting mdadm pick it, with or without the "names=yes" option.

Yes I am sure.  Possibly the documentation for mdadm.conf could be improved.

There are two different sorts names.  Note that you almost acknowledged this by writing "name for my md device node" while the documentation only talks about names for "md devices", not for "md device nodes".

There are
1/ there are names in /dev or /dev/md/ (device nodes)
2/ there are names that appear in /proc/mdstat and in /sys/block/ (devices)

names=yes is about the second sort of names.

By default, mdadm creates devices by using mknod to create something in /dev with the appropriate major/minor numbers.  The kernel will transparently create a device with a names like "md%d" based on the minor number.

With "names=yes", mdadm instead creates devices by writing e.g. md_root or md27 into /sys/module/md_mod/parameters/new_array.  The kernel will then create an array with exactly that name.

The same thing appears in /dev/md/ in both cases, but without names=yes, /dev/md/root will be a symlink to /dev/md127 (or similar) while with names=yes /dev/md/root will be a symlink to /dev/md_root (I think - it is a while since I've worked on this and I might not have all the details correct).

@Neil Brown: Thank you for the clarification! So it seems that mdadm should not be relying on the legacy behavior in any case. Without "names=yes" (or with "names=no"), mdadm should still be writing "md127" into /sys/module/md_mod/parameters/new_array before it attempts to open the new device node that it created. Presently mdadm is relying on the kernel to create the md device on the fly as a legacy side effect of opening a device node for a non-existent md device.

I don't particularly like the idea of having /dev/md_root, as I am more familiar with having numbered device nodes in /dev and having human-named nodes (possibly as symlinks) in subdirectories of /dev. Regardless, how would I enable "names=yes" when invoking "mdadm --assemble" with an explicit array specification (i.e., no mdadm.conf)?

Incidentally, I suspect you might be wrong about what needs to be written into "new_array" since the kernel would need to know not only the name for the new md device but also the device number. Or, alternatively, if the mechanism is designed to have the kernel dynamically pick the device number, then it makes less/no sense for mdadm to name the md device with a number since it wouldn't find out the device number until after it has passed a name to "new_array". Maybe that's why the new mechanism isn't used in the case when the device node is to be numbered rather than named?

The only way is with mdadm.conf.

Maybe:
  echo CREATE names=yes > /etc/mdadm.conf ; mdadm --assemble .... ; rm /etc/mdadm.conf
??

When the names=yes functionality was added, the new_array parameter was new and so it had to be opt-in.  I agree that the code needs to be different now.

I had never imagined that any would LIKE the numeric names.... I guess we could keep them if you really like them, but why?

> Incidentally, I suspect you might be wrong about what needs to be written
> into "new_array" since the kernel would need to know not only the name for
> the new md device but also the device number.

Allow me to quote the comment that I placed in the source code - so when I forgot how all this worked, I could easily go back and find out:

	/*
	 * val must be "md_*" or "mdNNN".
	 * For "md_*" we allocate an array with a large free minor number, and
	 * set the name to val.  val must not already be an active name.
	 * For "mdNNN" we allocate an array with the minor number NNN
	 * which must not already be in use.
	 */

Hopefully that also helps you understand how it works.

(In reply to Neil Brown from comment #10)
> I had never imagined that any would LIKE the numeric names....

It's just familiarity. I can adjust, though then I'd question the sanity of having /dev/md/root be a symlink to /dev/md_root, as that seems redundant.

Should I open a new bug on mdadm to discuss how to fix its behavior? In particular, there appears to be an abstraction breakdown when specifying array members on the command line, as the command line takes an argument to specify the device node path name, but there's no way to specify the array name, which need not be similar to the node name. Ideally, I would want to pass an array name like "XXX" (rather than a device node path like "/dev/md/XXX") to "mdadm --assemble" and have it write that name to "new_array" and create /dev/mdNN and /dev/md/XXX->../mdNN, where NN is the device minor number chosen by the kernel for the new array. If "new_array" doesn't exist (perhaps because sysfs isn't mounted at /sys), then mdadm could fall back to the present behavior (which fails if the kernel does not have CONFIG_BLOCK_LEGACY_AUTOLOAD). Perhaps, to maintain backward compatibility, mdadm should continue accepting an md device node path name and should detect if it is of the form "/dev/md_XXX" or "/dev/md/XXX" and attempt to write "XXX" to "new_array" first before falling back to the present behavior. (If the device node path is specified as "/dev/mdNN", then the "new_array" mechanism would not be used, and the current behavior would be used exclusively. The man page should mention the need for CONFIG_BLOCK_LEGACY_AUTOLOAD when specifying a device node path name in the numeric form.)

> Should I open a new bug on mdadm to discuss how to fix its behavior?

That is unlikely to help.  This bugzilla isn't used by most kernel developers.  We mostly prefer email lists.  I still get notification when someone flags "MD" even though I'm not the maintainer any more.  I don't know if anyone else does.

The best place to discuss this more broadly is to send an email to linux-raid@vger.kernel.org.  You don't need to subscribe or anything like that, just send an email.

> the command line takes an argument to specify the device node path name, but
> there's no way to specify the array name, which need not be similar to the
> node name.

Not entirely true.  While the device name and the device node name could be completely unrelated, mdadm does not support that.  If the device name is "md_FOO", then the name of the array is "FOO" and you get "/dev/md/FOO" linked to "/dev/md_FOO".

You can assemble an array as 
  mdadm -A FOO /dev/sd[abc]

 and that will be treated just like 
  mdadm -A /dev/md/FOO /dev/sd[abc]

 and the device name will be "md_FOO" - if names are supported.

I agree that testing for the presence of parameters/new_array would be sensible - we cannot default to names=yes when that doesn't exist.

I'm actually hoping that we can revert the dependency on CONFIG_BLOCK_LEGACY_AUTOLOAD until mdadm has better support for that option.  I'd rather not mention it in the man page unless we really have to.

See also
 https://lore.kernel.org/linux-raid/a13cd3b5-cc41-bf2f-c8ac-e031ad0d5dd7@leemhuis.info/

and my reply to the thread.

It turns out that I mis-diagnosed this problem somewhat.

mdadm 4.1 already supports using parameters/new_array when it exsits.  So on a kernel with CONFIG_BLOCK_LEGACY_AUTOLOAD

 mdadm -A /dev/md0 /dev/sd[bc]

will work.  EXCEPT there is one case where it doesn't use new_array.  This happens when it isn't given a name that it trusts - typically because the "name" field in the metadata says that this array belongs to some other computer (which is unfortunately quite common because arrays are created before the correct host name is set).  In this case mdadm typically chooses /dev/md127.

In this case new_array isn't used and mdadm fails.

I've just posted a patch for mdadm to fix this.

https://lore.kernel.org/linux-raid/167875238571.8008.9808655454439667586@noble.neil.brown.name/T/#u

The patch is one line:

--- a/mdopen.c
+++ b/mdopen.c
@@ -370,6 +370,7 @@ int create_mddev(char *dev, char *name, int autof, int trustworthy,
 		}
 		if (block_udev)
 			udev_block(devnm);
+		create_named_array(devnm);
 	}
 
 	sprintf(devname, "/dev/%s", devnm);


I expect the stable kernels will soon get a patch to force CONFIG_BLOCK_LEGACY_AUTOLOAD on when md is enabled.  Hopefully the next release of mdadm will have this fully fixed.

Thanks for reporting this!

(In reply to Neil Brown from comment #14)
> EXCEPT there is one case where it doesn't use new_array.  This
> happens when it isn't given a name that it trusts - typically because the
> "name" field in the metadata says that this array belongs to some other
> computer (which is unfortunately quite common because arrays are created
> before the correct host name is set).

Ahh, nice catch. In my case it's the opposite scenario: the array was created with the correct hostname set, but I'm assembling it in an early boot environment (like an initramfs but not implemented as one), so the hostname hasn't been set yet at the time the array is assembled.

Thanks for the fix and for the information.