Bug 191991

Summary:	nvme: Kernel panic - not synching:VFS:Unable to mount root fs on unknown-block(0,0)
Product:	Drivers	Reporter:	Sven Köhler (sven.koehler)
Component:	Other	Assignee:	drivers_other
Status:	NEW ---
Severity:	normal	CC:	tomi.p.sarvela
Priority:	P1
Hardware:	All
OS:	Linux
Kernel Version:	4.9.0	Subsystem:
Regression:	No	Bisected commit-id:
Attachments:	kconfig to reproduce the nvme issue Intel Skull Canyon NUT dmidecode

Description Sven Köhler 2017-01-05 22:17:21 UTC

Today, I changed my kernel config and afterwards couldn't boot anymore.
I changed from CONFIG_DRM=y and CONFIG_DRM_I915=y to CONFIG_DRM=m and CONFIG_DRM_I915=m and suddenly the kernel can't mount the root fs anymore. If I change it back, the kernel can mount the root fs.

I had CONFIG_BLK_DEV_NVME=y in all cases.

I sense a terrible race condition. Could it be the case that the nvme devices don't show up in time? Maybe having the intel driver as part of the kernel slows down the boot process enough to have the nvme devices show up before the kernel tries to mount the root fs. Then again, this is just speculation.

The kernel command line is: root=/dev/nvme0n1p3 ro init=/usr/lib/systemd/systemd quiet resume=PARTUUID=34e3dbac-9fc6-4aa8-9960-f44a2f58cd91


I do not use an initramfs or initrd. I specify the root device and have the appropriate drivers in the kernel (nvme, see above).

Comment 1 Tomi Sarvela 2017-07-04 10:40:39 UTC

Can confirm.

My config has almost everything else compiled into kernel except i915 and some  drivers (snd, net). kconfig attached. Base system is up-to-date Ubuntu 16.04, but the issue is just as reproducable with 17.04.

Newest kernels work well with initramdisk, system boots up. Without initramdisk the root partition (or even partition table) is not found at all.

Kernel is booted through grub2 or UEFI shell, no difference in results. Command line just the: root=/dev/nvme0n1p2

Bisected the issue with Intel Skull Canyon NUC, with NVMe boot disk on DRM-Tip tree.

GOOD commit 93ffa9a479ffb65d045e74e141346e7f107fcde1
x86/init: Add i8042 state to the platform data

FIRST BAD commit d79e141c1c6ea7cb70c169971d522b88c8d5b419
Input: i8042 - Trust firmware a bit more when probing on X86

Reverting the following lines made the NUC boot again from NVMe without initrd:

+++ b/drivers/input/serio/i8042-x86ia64io.h
@@ -983,7 +983,11 @@ static int __init i8042_pnp_init(void)
 #if defined(__ia64__)
                return -ENODEV;
 #else
-               pr_info("PNP: No PS/2 controller found. Probing ports directly.\n");
+               pr_info("PNP: No PS/2 controller found.\n");
+               if (x86_platform.legacy.i8042 !=
+                               X86_LEGACY_I8042_EXPECTED_PRESENT)
+                       return -ENODEV;
+               pr_info("Probing ports directly.\n");
                return 0;
 #endif

Comment 2 Tomi Sarvela 2017-07-04 10:41:14 UTC

Created attachment 257345 [details]
kconfig to reproduce the nvme issue

Comment 3 Tomi Sarvela 2017-07-04 10:41:56 UTC

Created attachment 257347 [details]
Intel Skull Canyon NUT dmidecode

Comment 4 Tomi Sarvela 2017-07-04 11:45:13 UTC

Additional information for debugging: disabling CONFIG_SERIO (and keyboards, mice and all that) made the good commit go bad. Bisected back to 4.5.0-rc1, no good commits found which would boot NVMe without initrd.

Comment 5 Tomi Sarvela 2017-07-04 11:54:46 UTC

Guess: portpoking in ser i/o area causes enough delay for nvme to get up and work

Comment 6 Tomi Sarvela 2017-07-04 13:13:04 UTC

Cleaner solution: kernel boot option 'rootwait'