Created attachment 304107 [details] dmesg log for igb The 6.3-rc1 and later release candidates are hanging during boot on our Dell PowerEdge R620 servers with Intel I350 nics (igb). After bisecting from v6.2 to v6.3-rc1, I isolated the problem to: > [6fffbc7ae1373e10b989afe23a9eeb9c49fe15c3] PCI: Honor firmware's device > disabled status > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c > index 1779582fb500..b1d80c1d7a69 100644 > --- a/drivers/pci/probe.c > +++ b/drivers/pci/probe.c > @@ -1841,6 +1841,8 @@ int pci_setup_device(struct pci_dev *dev) > > pci_set_of_node(dev); > pci_set_acpi_fwnode(dev); > + if (dev->dev.fwnode && !fwnode_device_is_available(dev->dev.fwnode)) > + return -ENODEV; > > pci_dev_assign_slot(dev); > I have verified that reverting 6fffbc7ae1373e10b989afe23a9eeb9c49fe15c3 resolves the issue on v6.3-rc4. I have attached the dmesg log from v6.3.0-rc1.
Created attachment 304108 [details] Output from acpidump
Created attachment 304109 [details] Output from lspci -vv
Created attachment 304115 [details] Full dmesg with dyndbg="file drivers/acpi/* +p"
Not sure how this is related by I've discovered that AWS Xen instances also crash on boot with [ 3.635376] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver [ 3.635379] ixgbevf: Copyright (c) 2009 - 2018 Intel Corporation. [ 3.673083] Invalid max_queues (4), will use default max: 2. [ 3.680817] BUG: kernel NULL pointer dereference, address: 0000000000000004 [ 3.687649] #PF: supervisor read access in kernel mode [ 3.692673] #PF: error_code(0x0000) - not-present page [ 3.697690] PGD 0 P4D 0 [ 3.700522] Oops: 0000 [#1] PREEMPT SMP PTI [ 3.704778] CPU: 1 PID: 62 Comm: xenwatch Not tainted 5.14.0+ #58 [ 3.710675] Hardware name: Xen HVM domU, BIOS 4.11.amazon 08/24/2006 [ 3.715333] ixgbevf 0000:00:03.0: 06:5f:49:6d:1b:92 [ 3.717220] RIP: 0010:get_free_entries+0xc3/0x2f0 [ 3.717228] Code: e8 48 83 c4 10 5b 5d 41 5c 41 5d 41 5e 41 5f c3 cc cc cc cc 48 8b 05 7c 7f 34 02 41 89 de 31 d2 44 8b 3d 90 7f 34 02 45 29 e6 <8b> 48 04 41 8d 44 0e ff f7 f1 45 8d 2c 07 41 89 c4 e8 17 fe ff ff [ 3.717231] RSP: 0018:ffffb340402c3d48 EFLAGS: 00010002 [ 3.717235] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffffb340402c3df8 [ 3.717236] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffa05fb184 [ 3.717237] RBP: 0000000000000286 R08: ffffd6f9020f3880 R09: 0000000000000000 [ 3.717239] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 3.718023] ixgbevf 0000:00:03.0: MAC: 1 [ 3.718768] R13: 0000000000083ce2 R14: 0000000000000001 R15: 0000000000000000 [ 3.718770] FS: 0000000000000000(0000) GS:ffff90266fa40000(0000) knlGS:0000000000000000 [ 3.720480] ixgbevf 0000:00:03.0: Intel(R) 82599 Virtual Function [ 3.721255] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3.721258] CR2: 0000000000000004 CR3: 0000000033152006 CR4: 00000000001706e0 [ 3.721260] Call Trace: [ 3.721262] <TASK> [ 3.721265] gnttab_grant_foreign_access+0x1c/0x70 [ 3.822046] xenbus_grant_ring+0x56/0x120 [ 3.826044] setup_blkring+0x1c8/0x460 [xen_blkfront] [ 3.830977] talk_to_blkback+0xb8/0x910 [xen_blkfront] [ 3.835967] ? xenbus_dev_request_and_reply+0x80/0x80 [ 3.840848] xenwatch_thread+0x94/0x180 [ 3.844670] ? cpuacct_percpu_seq_show+0x10/0x10 [ 3.849264] kthread+0xe0/0x100 [ 3.852553] ? kthread_complete_and_exit+0x20/0x20 [ 3.857311] ret_from_fork+0x22/0x30 [ 3.860972] </TASK> [ 3.863405] Modules linked in: xen_blkfront(+) serio_raw ghash_clmulni_intel(+) ixgbevf dm_mirror dm_region_hash dm_log dm_mod [ 3.873844] CR2: 0000000000000004 [ 3.877518] ---[ end trace 5fa245c60a20f54e ]--- [ 3.877520] RIP: 0010:get_free_entries+0xc3/0x2f0 [ 3.877528] Code: e8 48 83 c4 10 5b 5d 41 5c 41 5d 41 5e 41 5f c3 cc cc cc cc 48 8b 05 7c 7f 34 02 41 89 de 31 d2 44 8b 3d 90 7f 34 02 45 29 e6 <8b> 48 04 41 8d 44 0e ff f7 f1 45 8d 2c 07 41 89 c4 e8 17 fe ff ff [ 3.877530] RSP: 0018:ffffb340402c3d48 EFLAGS: 00010002 [ 3.877532] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffffb340402c3df8 [ 3.877534] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffa05fb184 [ 3.877535] RBP: 0000000000000286 R08: ffffd6f9020f3880 R09: 0000000000000000 [ 3.877536] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 3.877538] R13: 0000000000083ce2 R14: 0000000000000001 R15: 0000000000000000 [ 3.877539] FS: 0000000000000000(0000) GS:ffff90266fa40000(0000) knlGS:0000000000000000 [ 3.877541] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3.877542] CR2: 0000000000000004 CR3: 0000000033152006 CR4: 00000000001706e0 [ 3.877544] Kernel panic - not syncing: Fatal exception [ 3.944455] Kernel Offset: 0x1cc00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) and I've bisected the problem to the same 6fffbc7ae1373e10b989afe23a9eeb9c49fe15c3.