Created attachment 298517 [details]
Serial Log

When enabling VMD in BIOS setup, the host OS cannot boot successfully with the following error message:

[   13.577850] DMAR: DRHD: handling fault status reg 2
[   13.577852] DMAR: [INTR-REMAP] Request device [0x00:0x00.5] fault index 0xa00 [fault reason 0x25] Blocked a compatibility format interrupt request

*** Hardware Info ***
CPU: Ice Lake
VMD: 8086:28c0
    # lspci -s 0000:00.5 -nn
    0000:00:00.5 RAID bus controller [0104]: Intel Corporation Volume Management 
    Device NVMe RAID Controller [8086:28c0] (rev 04)

*** Detail Info ***
`git bisect` points the following offending patch (commit: ee81ee84f873):

commit ee81ee84f8739e584c9ccf113ba3c796187b7080
Author: Jon Derrick <jonathan.derrick@intel.com>
Date:   Wed Feb 10 09:13:15 2021 -0700

    PCI: vmd: Disable MSI-X remapping when possible

    VMD will retransmit child device MSI-X using its own MSI-X table and
    requester-id. This limits the number of MSI-X available to the whole
    child device domain to the number of VMD MSI-X interrupts.

    Some VMD devices have a mode where this remapping can be disabled,
    allowing child device interrupts to bypass processing with the VMD MSI-X
    domain interrupt handler and going straight the child device interrupt
    handler, allowing for better performance and scaling. The requester-id
    still gets changed to the VMD endpoint's requester-id, and the interrupt
    remapping handlers have been updated to properly set IRTE for child
    device interrupts to the VMD endpoint's context.

    Some VMD platforms have existing production BIOS which rely on MSI-X
    remapping and won't explicitly program the MSI-X remapping bit. This
    re-enables MSI-X remapping on unload.

    Link: https://lore.kernel.org/r/20210210161315.316097-3-jonathan.derrick@intel.com
    Signed-off-by: Jon Derrick <jonathan.derrick@intel.com>
    Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
    Reviewed-by: Krzysztof Wilczyński <kw@linux.com>
    Acked-by: Joerg Roedel <jroedel@suse.de>


*** Debugging Info ***
1. Reverting ee81ee84f873 on top of 5.14-rc7 can fix the issue.

2. The issue occurs when the VMD MSI remapping is disabled by invoking "vmd_set_msi_remapping(vmd, false)." However, IOMMU hardware blocks the compatibility format interrupt request because Interrupt Remapping Enable Status (IRES) and Extended Interrupt Mode Enable (EIME) are enabled. Please refer to section "5.1.4 Interrupt-Remapping Hardware Operation" in Intel VT-d spec. The following patch can fix the issue:

  diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
  index e3fcdfec58b3..dd0c57e5e9a5 100644
  --- a/drivers/pci/controller/vmd.c
  +++ b/drivers/pci/controller/vmd.c
  @@ -863,8 +863,7 @@ static const struct pci_device_id vmd_ids[] = {
                  .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP,},
          {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_28C0),
                  .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW |
  -                               VMD_FEAT_HAS_BUS_RESTRICTIONS |
  -                               VMD_FEAT_CAN_BYPASS_MSI_REMAP,},
  +                               VMD_FEAT_HAS_BUS_RESTRICTIONS,},
          {PCI_DEVICE(PCI_VENDOR_ID_INTEL, 0x467f),
                  .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP |
                                  VMD_FEAT_HAS_BUS_RESTRICTIONS |
  
  I know VMD_FEAT_CAN_BYPASS_MSI_REMAP is a new feature in Ice Lake's VMD controller, so option 3 might be the better solution.
 
3. The following patch checks if the IOMMU enables the interrupt remapping. If so, VMD still enables interrupt remapping irrespective of VMD_FEAT_CAN_BYPASS_MSI_REMAP.

  # git diff
  diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c
  index e3fcdfec58b3..db72932d049f 100644
  --- a/drivers/pci/controller/vmd.c
  +++ b/drivers/pci/controller/vmd.c
  @@ -6,6 +6,7 @@

   #include <linux/device.h>
   #include <linux/interrupt.h>
  +#include <linux/iommu.h>
   #include <linux/irq.h>
   #include <linux/kernel.h>
   #include <linux/module.h>
  @@ -710,7 +711,8 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned 
  long features)
           * acceptable because the guest is usually CPU-limited and MSI
           * remapping doesn't become a performance bottleneck.
           */
  -       if (!(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
  +       if (iommu_capable(vmd->dev->dev.bus, IOMMU_CAP_INTR_REMAP) ||
  +           !(features & VMD_FEAT_CAN_BYPASS_MSI_REMAP) ||
              offset[0] || offset[1]) {
                  ret = vmd_alloc_irqs(vmd);
                  if (ret)

  The test passes with/without the "intremap=off" kernel parameter.

  Jon, I can submit this patch for upstream review if you think this is the valid patch. If you have other solution for this, please let me know. I can test your patch.