Created attachment 304860 [details]
It needs to build at least 2 different RAIDs(eg. RAID0 and RAID10, RAID5 and RAID10) and then you will see below error repeatly(need to use serial console to see it)
[ 205.360738] systemd-shutdown: Stopping MD devices.
[ 205.366384] systemd-shutdown: sd-device-enumerator: Scan all dirs
[ 205.373327] systemd-shutdown: sd-device-enumerator: Scanning /sys/bus
[ 205.380427] systemd-shutdown: sd-device-enumerator: Scanning /sys/class
[ 205.388257] systemd-shutdown: Stopping MD /dev/md127 (9:127).
[ 205.394880] systemd-shutdown: Failed to sync MD block device /dev/md127, ignoring: Input/output error
[ 205.404975] md: md127 stopped.
[ 205.470491] systemd-shutdown: Stopping MD /dev/md126 (9:126).
[ 205.770179] md: md126: resync interrupted.
[ 205.776258] md126: detected capacity change from 1900396544 to 0
[ 205.783349] md: md126 stopped.
[ 205.862258] systemd-shutdown: Stopping MD /dev/md125 (9:125).
[ 205.862435] md: md126 stopped.
[ 205.868376] systemd-shutdown: Failed to sync MD block device /dev/md125, ignoring: Input/output error
[ 205.872845] block device autoloading is deprecated and will be removed.
[ 205.880955] md: md125 stopped.
[ 205.934349] systemd-shutdown: Stopping MD /dev/md124p2 (259:7).
[ 205.947707] systemd-shutdown: Could not stop MD /dev/md124p2: Device or resource busy
[ 205.957004] systemd-shutdown: Stopping MD /dev/md124p1 (259:6).
[ 205.964177] systemd-shutdown: Could not stop MD /dev/md124p1: Device or resource busy
[ 205.973155] systemd-shutdown: Stopping MD /dev/md124 (9:124).
[ 205.979789] systemd-shutdown: Could not stop MD /dev/md124: Device or resource busy
[ 205.988475] systemd-shutdown: Not all MD devices stopped, 4 left.
Didn't encounter this issue with v5.19 kernel, and reproduce the issue after v6.0.
After bisected the kernel and found below commit intorduces the issue
12a6caf27324 md: only delete entries from all_mddevs when the disk is freed
I'm not sure the issue should be fixed in systemd or md driver, please help to check what went wrong, thanks.
Created attachment 304861 [details]
minicom reboot hang log
The issue is reproducible with IMSM metadata on productional environment, affect multiple platforms.
Reproduction ratio: around 20%.
Base system functionality is damaged can we consider bumping it to high?
+ Adding MD raid folks.
Christoph  on linux-regressions list posted a proposed workqueue flushing
diff. Can you please test it?
(In reply to Bagas Sanjaya from comment #4)
> Hi AceLan,
> Christoph  on linux-regressions list posted a proposed workqueue flushing
> diff. Can you please test it?
Oops, Hannes Reinecke had posted the diff instead.
Created attachment 304919 [details]
I am off work, please expect delay in responses.
I found the clue of the problem:
@@ -8280,8 +8289,7 @@ static void *md_seq_next(struct seq_file *seq, void *v, loff_t *pos)
next_mddev = list_entry(tmp, struct mddev, all_mddevs);
- mddev = next_mddev;
- tmp = mddev->all_mddevs.next;
+ tmp = next_mddev->all_mddevs.next;
We are continuously decrementing "mddev->active" for failed array and as a result we cannot delete it (we wants 0).
It passed if me hits the moment when kernel doesn't call mddev_put() for failed array in md_seq_next.
It should lower reproduction ratio a lot or it may be no longer reproducible.
AceLan could you test this on your side?
I can't reproduce the issue with this patch.
The issue has been fixed by the commit, thanks.