Created attachment 307113 [details] bnxt_re cmd error After 6.3 kernel include bnxt_re driver for bnxt_en. The kernel show CallTrace when bootup: Tested on 6.12-rc5 today, there is still some error, but CallTrace is gone. [ 5.531490] bnxt_en 0000:82:00.0: QPLIB: cmdq[0x8]=0xf status 0x3 [ 5.531497] bnxt_en 0000:82:00.0 bnxt_re0: Failed to register fence-MR [ 5.531615] bnxt_en 0000:82:00.0 bnxt_re0: Failed to create Fence-MR [ 5.531694] bnxt_en 0000:82:00.0: QPLIB: cmdq[0xa]=0x9 status 0x3 [ 5.531702] bnxt_en 0000:82:00.0 bnxt_re0: Failed to create HW CQ [ 5.531706] infiniband bnxt_re0: Couldn't create ib_mad CQ [ 5.531709] infiniband bnxt_re0: Couldn't open port 1 [ 5.531872] infiniband bnxt_re0: Device registered with IB successfully It could be related to firmware. Log attached.
Created attachment 307114 [details] 6.11 kernel CallTrace Nov 01 00:07:41 ubuntu kernel: bnxt_en 0000:82:00.0: QPLIB: bnxt_re_is_fw_stalled: FW STALL Detected. cmdq[0xe]=0x3 waited (41254 > 40000) msec a> Nov 01 00:07:41 ubuntu kernel: bnxt_en 0000:82:00.0 bnxt_re0: Failed to modify HW QP Nov 01 00:07:41 ubuntu kernel: infiniband bnxt_re0: Couldn't change QP1 state to INIT: -110 Nov 01 00:07:41 ubuntu kernel: infiniband bnxt_re0: Couldn't start port Nov 01 00:07:41 ubuntu kernel: bnxt_en 0000:82:00.0 bnxt_re0: Failed to destroy HW QP Nov 01 00:07:41 ubuntu kernel: ------------[ cut here ]------------ Nov 01 00:07:41 ubuntu kernel: WARNING: CPU: 36 PID: 2191 at drivers/infiniband/core/cq.c:322 ib_free_cq+0x10b/0x160 [ib_core] Nov 01 00:07:41 ubuntu kernel: Modules linked in: rfcomm cmac algif_hash algif_skcipher af_alg nvme_fabrics bnep intel_rapl_msr snd_hda_codec_rea> Nov 01 00:07:41 ubuntu kernel: usb_storage hid amdxcp drm_exec gpu_sched drm_buddy video i2c_algo_bit drm_suballoc_helper drm_ttm_helper ttm ixg> Nov 01 00:07:41 ubuntu kernel: CPU: 36 UID: 0 PID: 2191 Comm: systemd-udevd Not tainted 6.11.0-061100-generic #202409151536 Nov 01 00:07:41 ubuntu kernel: Hardware name: LENOVO ThinkStation P8/105E, BIOS S0GKT21A 07/26/2024 Nov 01 00:07:41 ubuntu kernel: RIP: 0010:ib_free_cq+0x10b/0x160 [ib_core] Nov 01 00:07:41 ubuntu kernel: Code: ba 02 00 65 ff 0d ed 7c ee 3d 0f 85 70 ff ff ff 0f 1f 44 00 00 e9 66 ff ff ff 83 f8 03 0f 84 3a ff ff ff 0f > Nov 01 00:07:41 ubuntu kernel: RSP: 0018:ff4eb0d48905b848 EFLAGS: 00010202 Nov 01 00:07:41 ubuntu kernel: RAX: 0000000000000002 RBX: ff303a857a600000 RCX: 0000000000000000 Nov 01 00:07:41 ubuntu kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff303a84e4f2f000 Nov 01 00:07:41 ubuntu kernel: RBP: ff4eb0d48905b8a8 R08: 0000000000000000 R09: 0000000000000000 Nov 01 00:07:41 ubuntu kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00000000ffffff92 Nov 01 00:07:41 ubuntu kernel: R13: 00000000000000b0 R14: ff303a84f48be000 R15: ff303a84f48be8f8 Nov 01 00:07:41 ubuntu kernel: FS: 000077dc7425d8c0(0000) GS:ff303a87e9200000(0000) knlGS:0000000000000000 Nov 01 00:07:41 ubuntu kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Nov 01 00:07:41 ubuntu kernel: CR2: 0000738e6caeadd0 CR3: 00000001264da004 CR4: 0000000000f71ef0 Nov 01 00:07:41 ubuntu kernel: PKRU: 55555554 Nov 01 00:07:41 ubuntu kernel: Call Trace: Nov 01 00:07:41 ubuntu kernel: <TASK> Nov 01 00:07:41 ubuntu kernel: ? srso_alias_return_thunk+0x5/0xfbef5 Nov 01 00:07:41 ubuntu kernel: ? show_trace_log_lvl+0x273/0x310 Nov 01 00:07:41 ubuntu kernel: ? show_trace_log_lvl+0x273/0x310 Nov 01 00:07:41 ubuntu kernel: ? ib_mad_init_device+0x5c/0xd0 [ib_core] Nov 01 00:07:41 ubuntu kernel: ? show_regs.part.0+0x22/0x30 Nov 01 00:07:41 ubuntu kernel: ? show_regs.cold+0x8/0x10 Nov 01 00:07:41 ubuntu kernel: ? ib_free_cq+0x10b/0x160 [ib_core] Nov 01 00:07:41 ubuntu kernel: ? __warn.cold+0xa7/0x101 Nov 01 00:07:41 ubuntu kernel: ? ib_free_cq+0x10b/0x160 [ib_core] Nov 01 00:07:41 ubuntu kernel: ? report_bug+0x114/0x160 Nov 01 00:07:41 ubuntu kernel: ? handle_bug+0x51/0xa0 Nov 01 00:07:41 ubuntu kernel: ? exc_invalid_op+0x18/0x80 Nov 01 00:07:41 ubuntu kernel: ? asm_exc_invalid_op+0x1b/0x20 Nov 01 00:07:41 ubuntu kernel: ? ib_free_cq+0x10b/0x160 [ib_core] Nov 01 00:07:41 ubuntu kernel: ? srso_alias_return_thunk+0x5/0xfbef5 Nov 01 00:07:41 ubuntu kernel: ? ib_mad_port_open+0x393/0x450 [ib_core] Nov 01 00:07:41 ubuntu kernel: ib_mad_init_device+0x5c/0xd0 [ib_core] Nov 01 00:07:41 ubuntu kernel: ? srso_alias_return_thunk+0x5/0xfbef5 Nov 01 00:07:41 ubuntu kernel: add_client_context+0x115/0x1c0 [ib_core] Nov 01 00:07:41 ubuntu kernel: enable_device_and_get+0xe8/0x1e0 [ib_core] Nov 01 00:07:41 ubuntu kernel: ib_register_device+0xf4/0x180 [ib_core] Nov 01 00:07:41 ubuntu kernel: bnxt_re_ib_init+0x141/0x160 [bnxt_re] Nov 01 00:07:41 ubuntu kernel: bnxt_re_probe+0x14c/0x1b0 [bnxt_re] Nov 01 00:07:41 ubuntu kernel: ? __pfx_bnxt_re_probe+0x10/0x10 [bnxt_re] Nov 01 00:07:41 ubuntu kernel: auxiliary_bus_probe+0x49/0x90 Nov 01 00:07:41 ubuntu kernel: ? driver_sysfs_add+0x66/0xd0 Nov 01 00:07:41 ubuntu kernel: really_probe+0xf6/0x370 Nov 01 00:07:41 ubuntu kernel: ? pm_runtime_barrier+0x55/0xa0 Nov 01 00:07:41 ubuntu kernel: __driver_probe_device+0x8c/0x140 Nov 01 00:07:41 ubuntu kernel: driver_probe_device+0x24/0xd0 Nov 01 00:07:41 ubuntu kernel: __driver_attach+0xe4/0x210 Nov 01 00:07:41 ubuntu kernel: ? __pfx___driver_attach+0x10/0x10 Nov 01 00:07:41 ubuntu kernel: bus_for_each_dev+0x8c/0xf0 Nov 01 00:07:41 ubuntu kernel: driver_attach+0x1e/0x30 Nov 01 00:07:41 ubuntu kernel: bus_add_driver+0x14e/0x240 Nov 01 00:07:41 ubuntu kernel: driver_register+0x73/0xf0 Nov 01 00:07:41 ubuntu kernel: __auxiliary_driver_register+0x73/0xf0 Nov 01 00:07:41 ubuntu kernel: ? __pfx_bnxt_re_mod_init+0x10/0x10 [bnxt_re] Nov 01 00:07:41 ubuntu kernel: bnxt_re_mod_init+0x3e/0xff0 [bnxt_re] Nov 01 00:07:41 ubuntu kernel: ? __pfx_bnxt_re_mod_init+0x10/0x10 [bnxt_re] Nov 01 00:07:41 ubuntu kernel: do_one_initcall+0x5b/0x330 Nov 01 00:07:41 ubuntu kernel: do_init_module+0x97/0x280 Nov 01 00:07:41 ubuntu kernel: load_module+0x64d/0x750 Nov 01 00:07:41 ubuntu kernel: __do_sys_init_module+0x19e/0x1d0 Nov 01 00:07:41 ubuntu kernel: __x64_sys_init_module+0x1a/0x30 Nov 01 00:07:41 ubuntu kernel: x64_sys_call+0x1586/0x22b0 Nov 01 00:07:41 ubuntu kernel: do_syscall_64+0x7e/0x170 Nov 01 00:07:41 ubuntu kernel: ? srso_alias_return_thunk+0x5/0xfbef5 Nov 01 00:07:41 ubuntu kernel: ? vfs_read+0x2a0/0x380 Nov 01 00:07:41 ubuntu kernel: ? srso_alias_return_thunk+0x5/0xfbef5 Nov 01 00:07:41 ubuntu kernel: ? ksys_read+0x71/0x100 Nov 01 00:07:41 ubuntu kernel: ? srso_alias_return_thunk+0x5/0xfbef5 Nov 01 00:07:41 ubuntu kernel: ? syscall_exit_to_user_mode+0x4e/0x250 Nov 01 00:07:41 ubuntu kernel: ? srso_alias_return_thunk+0x5/0xfbef5 Nov 01 00:07:41 ubuntu kernel: ? do_syscall_64+0x8a/0x170 Nov 01 00:07:41 ubuntu kernel: ? srso_alias_return_thunk+0x5/0xfbef5 Nov 01 00:07:41 ubuntu kernel: ? exc_page_fault+0x96/0x1c0 Nov 01 00:07:41 ubuntu kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e Nov 01 00:07:41 ubuntu kernel: RIP: 0033:0x77dc74126bde Nov 01 00:07:41 ubuntu kernel: Code: 48 8b 0d 55 32 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 > Nov 01 00:07:41 ubuntu kernel: RSP: 002b:00007ffc48e754d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000af Nov 01 00:07:41 ubuntu kernel: RAX: ffffffffffffffda RBX: 0000576973b18670 RCX: 000077dc74126bde Nov 01 00:07:41 ubuntu kernel: RDX: 000077dc74482441 RSI: 0000000000064439 RDI: 0000576973bac3f0 Nov 01 00:07:41 ubuntu kernel: RBP: 0000576973bac3f0 R08: 27d4eb2f165667c5 R09: 85ebca77c2b2ae63 Nov 01 00:07:41 ubuntu kernel: R10: 0000000000002e81 R11: 0000000000000246 R12: 000077dc74482441 Nov 01 00:07:41 ubuntu kernel: R13: 0000576973b18e90 R14: 0000576973b27380 R15: 0000576973b06300 Nov 01 00:07:41 ubuntu kernel: </TASK> Nov 01 00:07:41 ubuntu kernel: ---[ end trace 0000000000000000 ]--- Nov 01 00:07:41 ubuntu kernel: bnxt_en 0000:82:00.0 bnxt_re0: Free MW failed: 0xffffff92 Nov 01 00:07:41 ubuntu kernel: infiniband bnxt_re0: Couldn't open port 1 Nov 01 00:07:41 ubuntu kernel: infiniband bnxt_re0: Device registered with IB successfully Nov 01 00:08:22 ubuntu kernel: bnxt_en 0000:82:00.1: QPLIB: bnxt_re_is_fw_stalled: FW STALL Detected. cmdq[0xe]=0x3 waited (40842 > 40000) msec a> Nov 01 00:08:22 ubuntu kernel: bnxt_en 0000:82:00.1 bnxt_re1: Failed to modify HW QP Nov 01 00:08:22 ubuntu kernel: infiniband bnxt_re1: Couldn't change QP1 state to INIT: -110 Nov 01 00:08:22 ubuntu kernel: infiniband bnxt_re1: Couldn't start port Nov 01 00:08:22 ubuntu kernel: bnxt_en 0000:82:00.1 bnxt_re1: Failed to destroy HW QP Nov 01 00:08:22 ubuntu kernel: bnxt_en 0000:82:00.1 bnxt_re1: Free MW failed: 0xffffff92 Nov 01 00:08:22 ubuntu kernel: infiniband bnxt_re1: Couldn't open port 1
Hi Pengyu Ma, Thank you for the report. We will take a look at this. Could you provide following outputs: 1. lspci -s 0000:82:00.0 -vvv 2. lspci -s 0000:82:00.0 -xxx 3. ethtool -i ethX : to know the firmware version you are using. Regards, Kalesh
$ sudo lspci -s 0000:82:00.0 -vvv 82:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01) Subsystem: Lenovo BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 86 IOMMU group: 24 Region 0: Memory at 4266c010000 (64-bit, prefetchable) [size=64K] Region 2: Memory at 4266bf00000 (64-bit, prefetchable) [size=1M] Region 4: Memory at 4266c022000 (64-bit, prefetchable) [size=8K] Expansion ROM at d0e80000 [disabled] [size=512K] Capabilities: [48] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- Capabilities: [50] Vital Product Data Product Name: Broadcom NX-E PCIe 10Gb 2-Port Base-T Ethernet Adapter Read-only fields: [PN] Part number: SN30L27797 [SN] Serial number: L0FG27N01VM [FN] FRU: 00YK535 [V0] Vendor specific: 214.0.286.17 [RV] Reserved: checksum good, 83 byte(s) reserved End Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [a0] MSI-X: Enable+ Count=74 Masked- Vector table: BAR=4 offset=00000000 PBA: BAR=4 offset=000004a0 Capabilities: [ac] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75.000W DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+ FLReset- MaxPayload 512 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend- LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s (ok), Width x8 (ok) TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+ 10BitTagComp- 10BitTagReq- OBFF Via WAKE#, ExtFmt- EETLPPrefix- EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- TPHComp- ExtTPHComp- AtomicOpsCap: 32bit- 64bit- 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Via WAKE#, AtomicOpsCtl: ReqEn+ LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+ EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO+ CmpltAbrt- UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr- AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn+ ECRCChkCap+ ECRCChkEn+ MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- HeaderLog: 04008001 8000220f 82020000 00000000 Capabilities: [13c v1] Device Serial Number 00-62-0b-ff-fe-26-11-70 Capabilities: [150 v1] Power Budgeting <?> Capabilities: [160 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=020 <?> Capabilities: [1b0 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 1 ARICtl: MFVC- ACS-, Function Group: 0 Capabilities: [230 v1] Transaction Processing Hints Interrupt vector mode supported Device specific mode supported Steering table in MSI-X table Capabilities: [300 v1] Secondary PCI Express LnkCtl3: LnkEquIntrruptEn- PerformEqu- LaneErrStat: 0 Capabilities: [200 v1] Precision Time Measurement PTMCap: Requester:+ Responder:- Root:- PTMClockGranularity: Unimplemented PTMControl: Enabled:- RootSelected:- PTMEffectiveGranularity: Unknown Kernel driver in use: bnxt_en Kernel modules: bnxt_en
$ sudo lspci -s 0000:82:00.0 -xxx 82:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57416 NetXtreme-E Dual-Media 10G RDMA Ethernet Controller (rev 01) 00: e4 14 d8 16 06 04 10 00 01 00 00 02 00 00 80 00 10: 0c 00 01 6c 26 04 00 00 0c 00 f0 6b 26 04 00 00 20: 0c 20 02 6c 26 04 00 00 00 00 00 00 aa 17 60 41 30: 00 00 e8 d0 48 00 00 00 00 00 00 00 ff 01 00 00 40: 00 00 00 00 00 00 00 00 01 50 03 c8 08 20 00 64 50: 03 58 c4 80 00 00 00 78 05 a0 86 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0: 11 ac 49 80 04 00 00 00 a4 04 00 00 10 00 02 00 b0: a2 8d 2c 11 57 2d 19 00 83 c0 44 00 40 00 83 10 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 1f 08 08 00 40 60 00 00 0e 00 00 00 01 00 1f 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
$ sudo ethtool -i enp130s0f0np0 driver: bnxt_en version: 6.11.0-061100-generic firmware-version: 214.4.9.9/pkg 214.0.286.17 expansion-rom-version: bus-info: 0000:82:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: no
Hi Pengyu Ma, Thank you for providing additional information. This is a BCM57416 device. I see that the FW version is quite old and this is an expected failure after the commit "a9a457f338e7 ("RDMA/bnxt_re: Update HW interface headers")". Broadcom RoCE solution was not GAed for this FW version and to use RoCE in production, you will have to update to a latest FW. Are you okay to update the FW to the latest versions? You can find the firmware location here: https://www.broadcom.com/support/download-search?pg=Ethernet+Connectivity,+Switching,+and+PHYs&pf=Ethernet+Connectivity,+Switching,+and+PHYs&pn=&pa=&po=&dk=&pl=&l=false Regards, Kalesh
@Kalesh A P, Thanks for your great support. I will try to do it.