Bug 64821

Hi,

This problem was discussed here:

http://marc.info/?t=138369090600001&r=1&w=2

Essentially, today the pciehp assumes that any HP controller that does not implement the following 4 HP elements, but yet advertises support for "command completed" notifications, will not generate command completed notifications on the slot control register:

/*
 * Controller doesn't notify of command completion if the "No
 * Command Completed Support" bit is set in Slot Capability
 * register or the controller supports none of power
 * controller, attention led, power led and EMI.
 */
if (NO_CMD_CMPL(ctrl) ||
    !(POWER_CTRL(ctrl) | ATTN_LED(ctrl) | PWR_LED(ctrl) | EMI(ctrl)))
    ctrl->no_cmd_complete = 1;

However, this is apparently not according to the PCI Express specs:

Bjorn Helgaas <bhelgaas@google.com> wrote:
> I'm not sure this "cmd completed" is actually spurious.  Spec section
> 7.8.10 is very clear that any write to Slot Control must cause a
> hot-plug command to be generated (if the port is hot-plug capable).

The current behaviour causes a problem for the HP controllers that do not implement the above mentioned HP elements, but yet generate "command Completed" notifications for pcie_disable_notification():

    pciehp_probe
       pcie_init
         readl(SLTCAP)
         if (NO_CMD_CMPL || !(POWER_CTRL | ATTN_LED | PWR_LED | EMI))  # true
           ctrl->no_cmd_complete = 1     # set (condition above is true)
         writew(SLTSTA, 0x1f)            # clear ABP PFD MRLSC PDC CC
         pcie_disable_notification
           pcie_write_cmd(0, PDCE | ABPE | MRLSCE | PFDE | HPIE | CCIE | DLLSCE)
             readw(SLTSTA)               # CC == 0 (was cleared above)
             writew(SLTCTL)
             ...
             # no waiting here because no_cmd_complete == 1
             ...
             hardware sets SLTSTA.CC
             ...
       pcie_init_notification
         pcie_enable_notification
           pcie_write_cmd
             readw(SLTSTA)               # CC == 1 (was set by HW above)
             if (SLTSTA.CC)              # true
               if (!NO_CMD_CMPL))        # true
                 dbg("Unexpected CMD_COMPLETED. ...") <---- Problem 1 
                 ctrl->no_cmd_complete = 0
             writew(SLTCTL)
             ...
             # this time we wait because no_cmd_complete == 0
             ...
             pcie_wait_cmd(..., poll == 0)# now no_cmd_complete == 0
               wait_event_timeout(..., !ctrl->cmd_busy,...) #cmd_busy is never cleared <-- Problem 2
                 Timeout & print(""Command not completed in 1000 msec\n"")

Why cmd_busy is not cleared?
  -> because the pcie_isr is never invoked
     -> because IRQ is not generated
        -> since the CC bit was never cleared.

I verified that setting ctrl->no_cmd_complete = 0 solves the problem.

I see 2 problems in the current code:

1) In case of spurious "command completed" interrupt detected in pcie_write_cmd(), it is never cleared:
        if (slot_status & PCI_EXP_SLTSTA_CC) {
           /* Not cleared here. Unless we cleared it, further interrupts will not be generated */
	}

2) 1 second delay in probe and following 2 error messages needs to get rid of:
 Unexpected CMD_COMPLETED. Need to wait for command completed event.
 Command not completed in 1000 msec


Here is the lspci -vv and relevent dmesg output:

PCI bridge: Integrated Device Technology, Inc. Device 807a (rev 02)
 (prog-if 00 [Normal decode])
 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
 Stepping- SERR- FastB2B- DisINTx+
 Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
 Bus: primary=02, secondary=50, subordinate=5f, sec-latency=0
 I/O behind bridge: 00003000-00003fff
 Memory behind bridge: 8c000000-8fffffff
 Prefetchable memory behind bridge: 00000000b0600000-00000000b07fffff
 Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
        PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Express (v2) Downstream Port (Slot+), MSI 00
         DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                 ExtTag+ RBE+ FLReset-
         DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                 RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                 MaxPayload 128 bytes, MaxReadReq 128 bytes
         DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
         LnkCap: Port #3, Speed 5GT/s, Width x4, ASPM L0s L1, Latency
 L0 <4us, L1 <4us
                 ClockPM- Surprise+ LLActRep+ BwNot+
         LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
                 ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
         LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk-
 DLActive- BWMgmt- ABWMgmt-
         SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise-
                 Slot #3, PowerLimit 0.000W; Interlock- NoCompl-
         SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet+ CmdCplt+ HPIrq+ LinkChg+
                 Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
         SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt+ PresDet- Interlock-
                 Changed: MRL- PresDet- LinkState-
         DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-,
 OBFF Not Supported ARIFwd+
         DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-,
 OBFF Disabled ARIFwd-
         LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-,
 Selectable De-emphasis: -6dB
                  Transmit Margin: Normal Operating Range,
 EnterModifiedCompliance- ComplianceSOS-
                  Compliance De-emphasis: -6dB
         LnkSta2: Current De-emphasis Level: -6dB,
 EqualizationComplete-, EqualizationPhase1-
                  EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [c0] Power Management version 3
         Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
 PME(D0+,D1-,D2-,D3hot+,D3cold+)
         Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
 Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
         Address: 00000000ff041740  Data: 0003
 Capabilities: [100 v1] Advanced Error Reporting
         UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
         CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
         AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
 Capabilities: [200 v1] Virtual Channel
         Caps:   LPEVC=0 RefClk=100ns PATEntryBits=4
         Arb:    Fixed- WRR32- WRR64- WRR128-
        Ctrl:   ArbSelect=Fixed
         Status: InProgress-
         VC0:    Caps:   PATOffset=04 MaxTimeSlots=1 RejSnoopTrans-
                 Arb:    Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
                Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                 Status: NegoPending- InProgress-
                 Port Arbitration Table <?>
 Capabilities: [320 v1] Access Control Services
         ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+
 EgressCtrl+ DirectTrans+
         ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd-
 EgressCtrl- DirectTrans-
 Capabilities: [330 v1] #12
 Kernel driver in use: pcieport

 In addition, here is what the pciehp driver spews out:

 Hotplug Controller:
   Seg/Bus/Dev/Func/IRQ : 0000:02:03.0 IRQ 21
   Vendor ID            : 0x111d
   Device ID            : 0x807a
   Subsystem ID         : 0x0000
   Subsystem Vendor ID  : 0x0000
   PCIe Cap offset      : 0x40
   PCI resource [7]     : [io  0x13000-0x13fff]
   PCI resource [8]     : [mem 0xc0c000000-0xc0fffffff]
   PCI resource [9]     : [mem 0xc30600000-0xc307fffff 64bit pref]
 Slot Capabilities      : 0x00180040
   Physical Slot Number : 3
   Attention Button     :  no
   Power Controller     :  no
   MRL Sensor           :  no
   Attention Indicator  :  no
   Power Indicator      :  no
   Hot-Plug Surprise    :  no
   EMI Present          :  no
   Command Completed    : yes
 Slot Status            : 0x0010
 Slot Control           : 0x0000
 Link Active Reporting supported
 HPC vendor_id 111d device_id 807a ss_vid 0 ss_did 0
 Registering domain:bus:dev=0000:50:00 sun=3
 Unexpected CMD_COMPLETED. Need to wait for command completed event.
 Command not completed in 1000 msec


Thanks,

Rajat