Bug 196197

Can you bisect between 4.10 and 4.11?

Hello,

Sorry, I've read the git dokumentation half the day, anyway I don't understand how bisect works and how to use it.

Is the following what you need ?

# git bisect log 
git bisect start
# bad: [c1ae3cfa0e89fa1a7ecc4c99031f5e9ae99d9201] Linux 4.11-rc1
git bisect bad c1ae3cfa0e89fa1a7ecc4c99031f5e9ae99d9201
# good: [7089db84e356562f8ba737c29e472cc42d530dbc] Linux 4.10-rc8
git bisect good 7089db84e356562f8ba737c29e472cc42d530dbc


What I have done is downloading from
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/
the original kernel sources, compiled them and tested them
4.12-rc7 (bad)
4.11-rc1 (bad)
4.10-rc8 (good)

The bug arises with 4.11 and still persists in 4.12, so as I assumed the step from 4.10 to 4.11 should be eamined.

Thanks,
Andreas

(In reply to Andreas Brogle from comment #2)
> # git bisect log 
> git bisect start
> # bad: [c1ae3cfa0e89fa1a7ecc4c99031f5e9ae99d9201] Linux 4.11-rc1
> git bisect bad c1ae3cfa0e89fa1a7ecc4c99031f5e9ae99d9201
> # good: [7089db84e356562f8ba737c29e472cc42d530dbc] Linux 4.10-rc8
> git bisect good 7089db84e356562f8ba737c29e472cc42d530dbc

That's just the start of bisecting. Now you need to keep testing the kernel built from the next commit checked out by git bisect. If the ring test failure happens, run "git bisect bad". If it doesn't happen and the GPU works normally after bootup, run "git bisect good". If anything else happens, e.g. the compiled kernel doesn't boot at all, run "git bisect skip". After repeating this a number (normally about tens) of times, git bisect will either specify the first bad commit or the minimum range of candidates.

Hello Michel,
needed a little bit help for the last step.
I am glad to give you now the precise information you need.

Last bisect good:
7089db84e356562f8ba737c29e472cc42d530dbc

First bisect bad:
ca2dea434d10e3a676482fdf6df17d20cdb3e907

Thanks, Andreas

(In reply to Andreas Brogle from comment #4)
> Hello Michel,
> needed a little bit help for the last step.
> I am glad to give you now the precise information you need.
> 
> Last bisect good:
> 7089db84e356562f8ba737c29e472cc42d530dbc
> 
> First bisect bad:
> ca2dea434d10e3a676482fdf6df17d20cdb3e907
> 
> Thanks, Andreas

Sorry,

Last bisect good:
caa59428971d5ad81d19512365c9ba580d83268c

First bisect bad:
ca2dea434d10e3a676482fdf6df17d20cdb3e907


root@a1:/work/linux-4.12-rc7# git bisect log
git bisect start
# bad: [c1ae3cfa0e89fa1a7ecc4c99031f5e9ae99d9201] Linux 4.11-rc1
git bisect bad c1ae3cfa0e89fa1a7ecc4c99031f5e9ae99d9201
# good: [7089db84e356562f8ba737c29e472cc42d530dbc] Linux 4.10-rc8
git bisect good 7089db84e356562f8ba737c29e472cc42d530dbc
# good: [caa59428971d5ad81d19512365c9ba580d83268c] Merge tag 'staging-4.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
git bisect good caa59428971d5ad81d19512365c9ba580d83268c
# bad: [ca2dea434d10e3a676482fdf6df17d20cdb3e907] Merge tag 'juno-fixes-4.11' of git://git.kernel.org/pub/scm/linux/kernel/git/sudeep.holla/linux into next/late
git bisect bad ca2dea434d10e3a676482fdf6df17d20cdb3e907

That's not the end of the bisection yet. Every time you run "git bisect good/bad/skip", it will check out a new commit that you need to compile and test, and print the estimated number of tests remaining. After the latter reaches 0, it should either print details of the first bad commit, or the minimal list of candidates.

Hello Michel,

is that what you need ?
The last 7 turns caused a kernel panic, all the 4.10.0-rc1 versions. I declared them as bad with git bisect. Hence I can't say if that ring 0 bug does exist or not for those versions.


root@a1:/work/linux-4.12-rc7# git bisect bad
0fc1223f0e77a748f7040562faaa7027f7db71ca is the first bad commit
commit 0fc1223f0e77a748f7040562faaa7027f7db71ca
Author: Rajat Jain <rajatja@google.com>
Date:   Mon Jan 2 22:34:10 2017 -0800

    PCI/ASPM: Add L1 substate capability structure register definitions
    
    Add L1 substate capability structure register definitions for use in
    subsequent patches.  See the PCIe r3.1 spec, sec 7.33.
    
    [bhelgaas: add PCIe spec reference]
    Signed-off-by: Rajat Jain <rajatja@google.com>
    Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>

:040000 040000 fd3e716ea9d6ed71cea541a3cc3a999e92c188ca 8a22f20457cd2bdd9f6f3fe8ad43ecba629a95bd M	include




root@a1:/work/linux-4.12-rc7# git bisect log
git bisect start
# bad: [c1ae3cfa0e89fa1a7ecc4c99031f5e9ae99d9201] Linux 4.11-rc1
git bisect bad c1ae3cfa0e89fa1a7ecc4c99031f5e9ae99d9201
# good: [7089db84e356562f8ba737c29e472cc42d530dbc] Linux 4.10-rc8
git bisect good 7089db84e356562f8ba737c29e472cc42d530dbc
# good: [caa59428971d5ad81d19512365c9ba580d83268c] Merge tag 'staging-4.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
git bisect good caa59428971d5ad81d19512365c9ba580d83268c
# bad: [ca2dea434d10e3a676482fdf6df17d20cdb3e907] Merge tag 'juno-fixes-4.11' of git://git.kernel.org/pub/scm/linux/kernel/git/sudeep.holla/linux into next/late
git bisect bad ca2dea434d10e3a676482fdf6df17d20cdb3e907
# good: [28cbc335d272f293c4368abd4ac2e17e36805b79] Merge tag 'sound-4.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
git bisect good 28cbc335d272f293c4368abd4ac2e17e36805b79
# bad: [54fff785db6e44208478ae3b0e5c56b853b3e10d] Merge tag 'armsoc-defconfig' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect bad 54fff785db6e44208478ae3b0e5c56b853b3e10d
# good: [190c3ee06a0f0660839785b7ad8a830e832d9481] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
git bisect good 190c3ee06a0f0660839785b7ad8a830e832d9481
# bad: [af8999f672421776417977101c3e1f334414c065] Merge tag 'armsoc-fixes-nc' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect bad af8999f672421776417977101c3e1f334414c065
# bad: [b2e6d3055d5545b97533d4e8376fa848639d9951] Merge branch 'pci/host-exynos' into next
git bisect bad b2e6d3055d5545b97533d4e8376fa848639d9951
# bad: [0dc49eb2a25a6b48e235803eeb2b52677f577ca5] Merge branch 'pci/resource' into next
git bisect bad 0dc49eb2a25a6b48e235803eeb2b52677f577ca5
# bad: [63ab93f021ecd815d848c3e9d3e326fa9628e5a9] Merge branch 'pci/enumeration' into next
git bisect bad 63ab93f021ecd815d848c3e9d3e326fa9628e5a9
# bad: [8f417ca9ebfa8701a41db64f5ed9cbb01b8e4219] Merge branch 'pci/aspm' into next
git bisect bad 8f417ca9ebfa8701a41db64f5ed9cbb01b8e4219
# bad: [f1f0366dd6be9624f7d355b72cc909ab821eb4c0] PCI/ASPM: Calculate and save the L1.2 timing parameters
git bisect bad f1f0366dd6be9624f7d355b72cc909ab821eb4c0
# bad: [b2103ccbb67e3ef0f7a75d21c989f9614ddbcaca] PCI/ASPM: Add support for L1 substates
git bisect bad b2103ccbb67e3ef0f7a75d21c989f9614ddbcaca
# bad: [0fc1223f0e77a748f7040562faaa7027f7db71ca] PCI/ASPM: Add L1 substate capability structure register definitions
git bisect bad 0fc1223f0e77a748f7040562faaa7027f7db71ca
# first bad commit: [0fc1223f0e77a748f7040562faaa7027f7db71ca] PCI/ASPM: Add L1 substate capability structure register definitions


Greetings, Andreas

Hello,
is there a chance that this bug will be fixed ?
Or would it be easier to install another graphics card ?
Greetings, Andreas

Hello,

same problem with a ATI Radeon HD3450 graphics card (RV620).
Maybe the problem is mainboard specific, it is a TYAN S3992-E.

Life must go on.
Have installed a NVIDIA card now. That is working fine with the nouveau driver.

Greetings, Andreas

Got the opportunity to test the RV630 XT [Radeon HD 2600 XT] card at an other mainboard. It works as it should. So the problem seems to be mainboard specific.

TYAN Thunder h2000M S3992-E
Expansion Slots 
•  Two (2) PCI Express x16 slots (each w/ x8 signal)

Somebody in some forum suspected the radeon driver has a problem with the "x8 signal" and leads to "*ERROR* radeon: ring 0 test failed". Maybe this can help.
But why this problem is araising with kernel 4.11 an newer ?

More or less same problem with an ASUS M4A88T-M/USB3 with an integrated RS880 (Radeon HD 4250).

All kernels <= 4.9.* work fine, all kernels >= 4.10.* exibit frequent kernel errors such as:
[drm:r600_ring_test [radeon]] *ERROR* radeon: ring 0 test failed (scratch(0x****)=0xCAFEDEAD)
Video modesetting fails, dri is not working.

Before finding this ticket (and all the similar ones) I began suspecting my mobo or power supply were broken. But then I downgraded the kernel from 4.13.* to 4.9.* and all is always fine again.

Please, fix it!

(In reply to Andreas Brogle from comment #7)
> The last 7 turns caused a kernel panic, all the 4.10.0-rc1 versions. I
> declared them as bad with git bisect.

In cases like this, where it's not possible to test a commit for the problem being bisected, one needs to run

 git bisect skip

instead of "bad".


> 0fc1223f0e77a748f7040562faaa7027f7db71ca is the first bad commit
> commit 0fc1223f0e77a748f7040562faaa7027f7db71ca
> Author: Rajat Jain <rajatja@google.com>
> Date:   Mon Jan 2 22:34:10 2017 -0800
> 
>     PCI/ASPM: Add L1 substate capability structure register definitions

Does this commit boot and exhibit the problem? If so, does the previous commit boot and not exhibit the problem?

Hello,

here is the complete bisection process. Testing needs about a week.
All versions 4.10.0-rc1 lead to a kernel panic. So there are 60 commits that have to be SKIPped.


git bisect start
# bad: [c1ae3cfa0e89fa1a7ecc4c99031f5e9ae99d9201] Linux 4.11-rc1
git bisect bad c1ae3cfa0e89fa1a7ecc4c99031f5e9ae99d9201
# good: [c470abd4fde40ea6a0846a2beab642a578c0b8cd] Linux 4.10
git bisect good c470abd4fde40ea6a0846a2beab642a578c0b8cd
# good: [5bcbe22ca47da04cda3a858cef67f55b550c1d13] Merge branch 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6
git bisect good 5bcbe22ca47da04cda3a858cef67f55b550c1d13
# bad: [da2f27e9e615d1c799c9582b15262458da61fddc] netfilter: nf_conntrack_sip: fix wrong memory initialisation
git bisect bad da2f27e9e615d1c799c9582b15262458da61fddc
# bad: [195849ea13549dc2390b7db7d80a22d76f73dfae] Merge tag 'armsoc-dt' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect bad 195849ea13549dc2390b7db7d80a22d76f73dfae
# bad: [6ae52c65e01991426e87bb0fb8a2ac9e032db7b1] Merge tag 'armsoc-soc' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect bad 6ae52c65e01991426e87bb0fb8a2ac9e032db7b1
# good: [e4cc4fa7cca9ac7d7c3abea7d6c90db1c519d6c6] IB/mlx5: Enable QP creation with cvlan offload
git bisect good e4cc4fa7cca9ac7d7c3abea7d6c90db1c519d6c6
# skip: [c4d052ce970ea98e9e1cc72461ba3b7a25397657] Merge branch 'pci/virtualization' into next
git bisect skip c4d052ce970ea98e9e1cc72461ba3b7a25397657
# skip: [8b9c156262f9ebaa88736b93263c1af5ae5fa43e] Merge branch 'pci/virtualization' into next
git bisect skip 8b9c156262f9ebaa88736b93263c1af5ae5fa43e
# skip: [4d94b4a239eec860701f0d2e07123a0ed5283a22] ARM: OMAP1: USB: delete redundant CPU class checks
git bisect skip 4d94b4a239eec860701f0d2e07123a0ed5283a22
# good: [a329b0cac2e3f345f2161ed7cbfb0f2ce11226d9] Merge tag 'imx-cleanup-4.11' of git://git.kernel.org/pub/scm/linux/kernel/git/shawnguo/linux into next/soc
git bisect good a329b0cac2e3f345f2161ed7cbfb0f2ce11226d9
# skip: [f08bf55e690ebdac28402955ef953e11ff478668] Merge branch 'pci/host-versatile' into next
git bisect skip f08bf55e690ebdac28402955ef953e11ff478668
# skip: [e7cd7ef58e1fedb09b16720919869a81d7a2b867] PCI: exynos: Support the PHY generic framework
git bisect skip e7cd7ef58e1fedb09b16720919869a81d7a2b867
# skip: [c5c4d3a3f4a8c830cb514eb469a1025df2df0379] PCI: thunder-pem: Add support for cn81xx and cn83xx SoCs
git bisect skip c5c4d3a3f4a8c830cb514eb469a1025df2df0379
# good: [c0d32456091a81b0880e302d8c38c736a4dddab0] Merge tag 'omap-for-v4.11/fixes-not-urgent-signed' of git://git.kernel.org/pub/scm/linux/kernel/git/tmlind/linux-omap into next/fixes-non-critical
git bisect good c0d32456091a81b0880e302d8c38c736a4dddab0
# skip: [bcea623c65d5ad3b2f4e51908cffc9423175a10b] Merge branch 'pci/host-rockchip' into next
git bisect skip bcea623c65d5ad3b2f4e51908cffc9423175a10b
# skip: [906c142634def5dd3dda533d5d20bd3c1c19d1fb] Merge branch 'pci/hotplug' into next
git bisect skip 906c142634def5dd3dda533d5d20bd3c1c19d1fb
# good: [d07d1d70ce1ad1c525f51f459ce36ca49ec2bf48] IB/umem: Update on demand page (ODP) support
git bisect good d07d1d70ce1ad1c525f51f459ce36ca49ec2bf48
# good: [e65ade77e87fd206bb78fae32ab7b604b01fade0] Merge branch 'mlx4-misc-fixes'
git bisect good e65ade77e87fd206bb78fae32ab7b604b01fade0
# skip: [68094b4b8d72360f9a90804c203e4292c6662fdd] Merge branch 'pci/host-iproc' into next
git bisect skip 68094b4b8d72360f9a90804c203e4292c6662fdd
# skip: [442ec4c04d1235f8c664a74004dae54a7a574d18] PCI: dwc: all: Split struct pcie_port into host-only and core structures
git bisect skip 442ec4c04d1235f8c664a74004dae54a7a574d18
# skip: [95ed41df129ca8f76eb190534d20f4a6dcd37213] dt-bindings: video: exynos7-decon: Remove obsolete samsung,power-domain property
git bisect skip 95ed41df129ca8f76eb190534d20f4a6dcd37213
# skip: [5f334db665173facf2213854408bb5fa2445d0b3] PCI: dwc: designware: Parse "num-lanes" property in dw_pcie_setup_rc()
git bisect skip 5f334db665173facf2213854408bb5fa2445d0b3
# skip: [57a66aafda62aacd15f0d1f4f260cc0efdf1d0e4] ARM: davinci: da8xx-dt: add OF_DEV_AUXDATA entry for SATA
git bisect skip 57a66aafda62aacd15f0d1f4f260cc0efdf1d0e4
# skip: [63ab93f021ecd815d848c3e9d3e326fa9628e5a9] Merge branch 'pci/enumeration' into next
git bisect skip 63ab93f021ecd815d848c3e9d3e326fa9628e5a9
# good: [f95d7a46bc5722767c30ee223c8b67dd0f2e8793] netfilter: ctnetlink: Fix regression in CTA_HELP processing
git bisect good f95d7a46bc5722767c30ee223c8b67dd0f2e8793
# skip: [e34e38bf891bd7bf7c9305ff1d2e1b29a77dd159] Merge branch 'pci/host-imx6' into next
git bisect skip e34e38bf891bd7bf7c9305ff1d2e1b29a77dd159
# skip: [602d38bc65aa2926d1ecd290a348e87aa8c21290] PCI: dra7xx: Simplify probe code with devm_gpiod_get_optional()
git bisect skip 602d38bc65aa2926d1ecd290a348e87aa8c21290
# skip: [4c2ae32d4c6ccd63819f36702f8455a5b08eaca8] Merge branch 'pci/vpd' into next
git bisect skip 4c2ae32d4c6ccd63819f36702f8455a5b08eaca8
# good: [1564dd1d2f876dcc288fae94a5179baccc89de8f] Merge tag 'samsung-soc-4.11-2' of git://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux into next/soc
git bisect good 1564dd1d2f876dcc288fae94a5179baccc89de8f
# skip: [ab5fe4f4d31ee27df9b8d7720da710d955d55737] PCI: dra7xx: Add support to force RC to work in GEN1 mode
git bisect skip ab5fe4f4d31ee27df9b8d7720da710d955d55737
# skip: [1a55761392b8a4703039614f3ef6275f88dad4c3] Merge branch 'pci/host-altera' into next
git bisect skip 1a55761392b8a4703039614f3ef6275f88dad4c3
# good: [5172d5a5a95ff0278135896fe29c2edc4b946900] Merge tag 'mvebu-soc-4.11-2' of git://git.infradead.org/linux-mvebu into next/soc
git bisect good 5172d5a5a95ff0278135896fe29c2edc4b946900
# skip: [5dcda98a3b7cfc0025ca49a700765ac9ba78a6a7] Merge branch 'pci/host-hv' into next
git bisect skip 5dcda98a3b7cfc0025ca49a700765ac9ba78a6a7
# skip: [e2dc4f225b765c2a12e5eaa7bc09994e67d13741] Merge branch 'pci/host-layerscape' into next
git bisect skip e2dc4f225b765c2a12e5eaa7bc09994e67d13741
# skip: [00bacfbfaf508940fd5fba22372a0935c4a01358] ARM: davinci: da850: model the SATA refclk
git bisect skip 00bacfbfaf508940fd5fba22372a0935c4a01358
# skip: [d9520971589b008faa8df31aa2bb3d1e4f1c4534] Merge branch 'pci/host-hisi' into next
git bisect skip d9520971589b008faa8df31aa2bb3d1e4f1c4534
# skip: [a142f4d3e5c54db5e942fa6ee5f3dc0e8c83207b] PCI/ASPM: Add comment about L1 substate latency
git bisect skip a142f4d3e5c54db5e942fa6ee5f3dc0e8c83207b
# skip: [ec6bd78a09d9967c4fcec53e7fabfaabd4f0e367] PCI: xilinx: Configure PCIe MPS settings
git bisect skip ec6bd78a09d9967c4fcec53e7fabfaabd4f0e367
# skip: [3278478084747c02725ba804d672235f2ba56bbc] PCI: exynos: Refactor to make it easier to support other SoCs
git bisect skip 3278478084747c02725ba804d672235f2ba56bbc
# skip: [caf3f562e1161a86bd48a4c4c33af89d3693c658] PCI: imx6: Remove redundant "Link never came up" message
git bisect skip caf3f562e1161a86bd48a4c4c33af89d3693c658
# skip: [ad8ec41afa98615a4154eee0121bcf8276695327] Documentation: samsung-phy: Add exynos-pcie-phy binding
git bisect skip ad8ec41afa98615a4154eee0121bcf8276695327
# skip: [87b336d003d47876e376d943be3c9d35152f3b86] PCI/DPC: Decode extended reasons
git bisect skip 87b336d003d47876e376d943be3c9d35152f3b86
# skip: [b90dc392212d1153a12eea15cbc6eae352a3c989] PCI: dwc: designware: Move register defines to designware header file
git bisect skip b90dc392212d1153a12eea15cbc6eae352a3c989
# skip: [63d182abd71cb47cee4adb8dd2afd71d987794d5] PCI: rpadlpar: Remove unnecessary return statement
git bisect skip 63d182abd71cb47cee4adb8dd2afd71d987794d5
# skip: [9bcf0a6fdc5062e451cd6f1ab39045e142a5938f] PCI: dwc: all: Use platform_set_drvdata() to save private data
git bisect skip 9bcf0a6fdc5062e451cd6f1ab39045e142a5938f
# skip: [25e77388e1ab63e11e21d94a994eca227472aeed] PCI: Remove res_to_dev_res() debug message
git bisect skip 25e77388e1ab63e11e21d94a994eca227472aeed
# skip: [948b7620c15411444167a62cfc14cdd4b0e44682] PCI/MSI: Return -ENOSPC if pci_enable_msi_range() can't get enough vectors
git bisect skip 948b7620c15411444167a62cfc14cdd4b0e44682
# skip: [e8e8dd6d20fa55ff974460ad742fcbf35b301062] [media] cobalt: use pci_irq_allocate_vectors()
git bisect skip e8e8dd6d20fa55ff974460ad742fcbf35b301062
# skip: [c3cf2c61ddc1410424da0ea87717edf16fc296c5] PCI/MSI: Document pci_alloc_irq_vectors(), deprecate pci_enable_msi()
git bisect skip c3cf2c61ddc1410424da0ea87717edf16fc296c5
# skip: [afc9595ea4770f0157ae06fb3acedff703e169b6] PCI: rockchip: Disable RC's ASPM L0s based on DT "aspm-no-l0s"
git bisect skip afc9595ea4770f0157ae06fb3acedff703e169b6
# good: [a748d60df32ec5da31626a140be1795eefa04282] IB/mlx4: Take source GID by index from HW GID table
git bisect good a748d60df32ec5da31626a140be1795eefa04282
# skip: [9e38637a0832f15552ae130622808831f515107c] Merge branch 'pci/host-xilinx' into next
git bisect skip 9e38637a0832f15552ae130622808831f515107c
# skip: [33be632b8443b6ac74aa293504f430604fb9abeb] PCI: Add ACS quirk for Qualcomm QDF2400 and QDF2432
git bisect skip 33be632b8443b6ac74aa293504f430604fb9abeb
# skip: [b98a7f7509ed7a031ed750be11b72b7ca1c95513] Merge branch 'pci/host-thunder' into next
git bisect skip b98a7f7509ed7a031ed750be11b72b7ca1c95513
# good: [d9d84780f17cd9e25ddde10eb40357b96b7d6874] vfio: fix a typo in comment of function vfio_pin_pages
git bisect good d9d84780f17cd9e25ddde10eb40357b96b7d6874
# skip: [b2e6d3055d5545b97533d4e8376fa848639d9951] Merge branch 'pci/host-exynos' into next
git bisect skip b2e6d3055d5545b97533d4e8376fa848639d9951
# skip: [3bb0356bb1b5a2c4434b4a3267df6dd9927ca5bf] Merge branch 'pci/host-designware' into next
git bisect skip 3bb0356bb1b5a2c4434b4a3267df6dd9927ca5bf
# skip: [ebe85a44aad47bb5f08c5cbd939eb5e40956c73b] PCI: dra7xx: Enable MSI and legacy interrupts simultaneously
git bisect skip ebe85a44aad47bb5f08c5cbd939eb5e40956c73b
# skip: [dda718926c0fe091fd5c7ed3f88c9542d8bf67cf] Merge branch 'pci/host-mvebu' into next
git bisect skip dda718926c0fe091fd5c7ed3f88c9542d8bf67cf
# skip: [92004a064875e45f0ceec72f6962e258e59f0682] PCI: exynos: Remove duplicated code
git bisect skip 92004a064875e45f0ceec72f6962e258e59f0682
# skip: [e5c3b3e9f012b8862753a04f03c6e27344332718] PCI: iproc: Use of_device_get_match_data() to simplify probe
git bisect skip e5c3b3e9f012b8862753a04f03c6e27344332718
# good: [c6057e101a2d09d24d84a3d2d701fd5e71f5e26a] ARM: ux500: remove deleted file from Makefile
git bisect good c6057e101a2d09d24d84a3d2d701fd5e71f5e26a
# skip: [cf0adb8e281b69801fb8faef18c14443d9d41d3c] phy: phy-exynos-pcie: Add support for Exynos PCIe PHY
git bisect skip cf0adb8e281b69801fb8faef18c14443d9d41d3c
# skip: [4f69bd16df1a38c32d5d207a96d1d86df4808f87] PCI: Increase VPD access timeout to 125ms
git bisect skip 4f69bd16df1a38c32d5d207a96d1d86df4808f87
# skip: [862290f9e23c39051e59bf12ce65707a8ec8b911] PCI/MSI: Check that we have a legacy interrupt line before using it
git bisect skip 862290f9e23c39051e59bf12ce65707a8ec8b911
# skip: [013dd3d5e1835c2cfa9c824e61465b61509afa54] PCI: rockchip: Add system PM support
git bisect skip 013dd3d5e1835c2cfa9c824e61465b61509afa54
# skip: [34f80c7ddfffe262bf04fb03e198e64de4cec7fc] Documentation: binding: Modify the exynos5440 PCIe binding
git bisect skip 34f80c7ddfffe262bf04fb03e198e64de4cec7fc
# skip: [b2103ccbb67e3ef0f7a75d21c989f9614ddbcaca] PCI/ASPM: Add support for L1 substates
git bisect skip b2103ccbb67e3ef0f7a75d21c989f9614ddbcaca
# skip: [19ce01cc8cbc314d73db9755715a8f6e8ad59a7f] PCI: dwc: all: Rename cfg_read/cfg_write to read/write
git bisect skip 19ce01cc8cbc314d73db9755715a8f6e8ad59a7f
# skip: [950bf6388bc22c2749b8b66c501df1462639d6bd] PCI: Move DesignWare IP support to new drivers/pci/dwc/ directory
git bisect skip 950bf6388bc22c2749b8b66c501df1462639d6bd
# skip: [e4e7d59704d485f272061cea9057798dda3cfd99] PCI/MSI: Update MSI/MSI-X bits in PCIEBUS-HOWTO
git bisect skip e4e7d59704d485f272061cea9057798dda3cfd99
# skip: [5b0948dfe138f0837699f46f5877f4f81c252dac] PCI: Lock each enable/disable num_vfs operation in sysfs
git bisect skip 5b0948dfe138f0837699f46f5877f4f81c252dac
# skip: [81efbaddd687f1b478c15665716fd545e2e4401e] PCI/MSI: Fix msi_desc->affinity memory leak when freeing MSI IRQs
git bisect skip 81efbaddd687f1b478c15665716fd545e2e4401e
# skip: [4a9b0933bdfcd85da840284bf5a0eb17b654b9c2] PCI: hv: Use device serial number as PCI domain
git bisect skip 4a9b0933bdfcd85da840284bf5a0eb17b654b9c2
# skip: [feb85d9b1c47ea8dbcae559ff127b433fdb245b7] PCI: dwc: Split pcie-designware.c into host and core files
git bisect skip feb85d9b1c47ea8dbcae559ff127b433fdb245b7
# skip: [1acf8bca9cdb2443a8707ff0afc3aadbfb5669f4] PCI: acpiphp_ibm: Make ibm_apci_table_attr __ro_after_init
git bisect skip 1acf8bca9cdb2443a8707ff0afc3aadbfb5669f4
# skip: [7a2b3f024b8b724257d82f76690e68c69d797efd] PCI: dwc: Remove dependency of designware on CONFIG_PCI
git bisect skip 7a2b3f024b8b724257d82f76690e68c69d797efd
# skip: [ce709f86501a013e941e9986cb072eae375ddf3e] PCI: Add Broadcom Northstar2 PAXC quirk for device class and MPSS
git bisect skip ce709f86501a013e941e9986cb072eae375ddf3e
# skip: [42d87e3ffbd53c4514bccc0f24e40d6231a567f5] Merge branch 'pci/msi' into next
git bisect skip 42d87e3ffbd53c4514bccc0f24e40d6231a567f5
# skip: [b88a10765471cff87025794fbe1ccaa96c7945a8] Merge branch 'v4.11/fixes-non-critical' into v4.11/soc
git bisect skip b88a10765471cff87025794fbe1ccaa96c7945a8
# skip: [2bdd584a7555d5ac3341b61e751a6ac807a9814a] Merge branch 'pci/enumeration' into next
git bisect skip 2bdd584a7555d5ac3341b61e751a6ac807a9814a
# skip: [7faebda21d573a6889bab1e0100ed4092a5a4716] PCI: rockchip: Use readl_poll_timeout() instead of open-coding it
git bisect skip 7faebda21d573a6889bab1e0100ed4092a5a4716
# skip: [3ec2574e31b5aebeeed52ea784bb6d7960788202] Merge branch 'pci/msi' into next
git bisect skip 3ec2574e31b5aebeeed52ea784bb6d7960788202
# skip: [4788316f743539076712d2b80b6cd289458fe2be] PCI: hisi: Rename config space accessors to remove "acpi"
git bisect skip 4788316f743539076712d2b80b6cd289458fe2be
# skip: [0dc49eb2a25a6b48e235803eeb2b52677f577ca5] Merge branch 'pci/resource' into next
git bisect skip 0dc49eb2a25a6b48e235803eeb2b52677f577ca5
# skip: [8ba0f6caa41c0786b964dcf75ab74bb05c726ec3] ARM: davinci: da850: add con_id for the SATA clock
git bisect skip 8ba0f6caa41c0786b964dcf75ab74bb05c726ec3
# skip: [56195e9d1d68a9c201587cb55b7fdf6d43fb31b6] Merge branch 'pci/host-rcar' into next
git bisect skip 56195e9d1d68a9c201587cb55b7fdf6d43fb31b6
# skip: [60db3a4d8cc9073cf56264785197ba75ee1caca4] PCI: Enable PCIe Extended Tags if supported
git bisect skip 60db3a4d8cc9073cf56264785197ba75ee1caca4
# skip: [9d534265bdcd47659c9fd080fff4c005512c4770] Merge branch 'pci/host-xgene' into next
git bisect skip 9d534265bdcd47659c9fd080fff4c005512c4770
# skip: [2681c0e7ff70a6d41a81527f95f2edabafea4ace] PCI: exynos: Use the bitops BIT() macro to build bitmasks
git bisect skip 2681c0e7ff70a6d41a81527f95f2edabafea4ace
# good: [f14cc3b13d8f3ceee862f8365d37ba214630126a] Merge tag 'vfio-v4.11-rc1' of git://github.com/awilliam/linux-vfio
git bisect good f14cc3b13d8f3ceee862f8365d37ba214630126a
# skip: [f1d722b607d610b66785f7f00d2e1d457260647c] PCI: rockchip: Fix rockchip_pcie_probe() error path to free resource list
git bisect skip f1d722b607d610b66785f7f00d2e1d457260647c
# good: [cdbe33d0f82d68ff74f05502a4c26e65ec7e90bb] IB/mlx5: Fix configuration of port capabilities
git bisect good cdbe33d0f82d68ff74f05502a4c26e65ec7e90bb
# skip: [3adfb572f2978a980b250a9e1a56f84f3a031001] PCI/MSI: Return failure when msix_setup_entries() fails
git bisect skip 3adfb572f2978a980b250a9e1a56f84f3a031001
# skip: [9c9b1bc25291e275b04f758f2549c81e092954f5] ARM: davinci: add skeleton for pdata-quirks
git bisect skip 9c9b1bc25291e275b04f758f2549c81e092954f5
# skip: [a3035886852cd859e371550fe726da08b85c67d0] ARM: davinci: remove BUG_ON() from da850_register_sata()
git bisect skip a3035886852cd859e371550fe726da08b85c67d0
# skip: [8ca6e0a75a5145458e8a680edf2394375f2129da] PCI: altera: Extract TLP completion status correctly
git bisect skip 8ca6e0a75a5145458e8a680edf2394375f2129da
# skip: [47feb41888bc82ba5c9268c344775adc483d6de7] PCI/MSI: Remove unused pci_msi_create_default_irq_domain()
git bisect skip 47feb41888bc82ba5c9268c344775adc483d6de7
# skip: [6dc2c04fd9868a5ad00b402935021d6f3ff27b17] PCI: layerscape: Use of_device_get_match_data() to simplify probe
git bisect skip 6dc2c04fd9868a5ad00b402935021d6f3ff27b17
# good: [ccaba0621a8577ce6b84cf625f2b9a4c6466d999] Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf
git bisect good ccaba0621a8577ce6b84cf625f2b9a4c6466d999
# skip: [ca0ef7fecc881a7e8d11db2d8852ab580cd29e03] Merge branch 'pci/aer' into next
git bisect skip ca0ef7fecc881a7e8d11db2d8852ab580cd29e03
# bad: [af8999f672421776417977101c3e1f334414c065] Merge tag 'armsoc-fixes-nc' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
git bisect bad af8999f672421776417977101c3e1f334414c065
# skip: [8f417ca9ebfa8701a41db64f5ed9cbb01b8e4219] Merge branch 'pci/aspm' into next
git bisect skip 8f417ca9ebfa8701a41db64f5ed9cbb01b8e4219
# skip: [a2ec1996098c7da0593a0981190316025301eab1] PCI: hisi: Add DT almost-ECAM support for Hip06/Hip07 host controllers
git bisect skip a2ec1996098c7da0593a0981190316025301eab1
# skip: [3674cc49da9a8fc55bf1dec2ab03a66c77f2dcdf] PCI/portdrv: Use pci_irq_alloc_vectors()
git bisect skip 3674cc49da9a8fc55bf1dec2ab03a66c77f2dcdf
# skip: [8267b07526cabe2e2afc834a138ece8644af87ed] PCI: rcar: Add compatible string for r8a7796
git bisect skip 8267b07526cabe2e2afc834a138ece8644af87ed
# skip: [40f67fb2c384fe12741aa35010d62bfe8c98286c] PCI: dwc: designware: Get device pointer at the start of dw_pcie_host_init()
git bisect skip 40f67fb2c384fe12741aa35010d62bfe8c98286c
# bad: [60e8d3e11645a1b9c4197d9786df3894332c1685] Merge tag 'pci-v4.11-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci
git bisect bad 60e8d3e11645a1b9c4197d9786df3894332c1685
# skip: [4d4836ab70d3c59bc934d244f0cccdd035c1ead8] PCI: iproc: Configure PCIe MPS settings
git bisect skip 4d4836ab70d3c59bc934d244f0cccdd035c1ead8
# skip: [314fc854f50317931fb4dfaab431695ab886e8de] PCI: dwc: designware: Fix style errors in pcie-designware.c
git bisect skip 314fc854f50317931fb4dfaab431695ab886e8de
# skip: [ff1677e231651205e7e19770a677057dea05cb70] PCI: rcar: Use of_device_get_match_data() to simplify probe
git bisect skip ff1677e231651205e7e19770a677057dea05cb70
# skip: [af3a2ab5daa0b8bf188a039c122a670cdc8b9544] Merge branch 'pci/dpc' into next
git bisect skip af3a2ab5daa0b8bf188a039c122a670cdc8b9544
# skip: [e75377404726be171d66c154f8ea1e6cf840811d] amd-xgbe: Update PCI support to use new IRQ functions
git bisect skip e75377404726be171d66c154f8ea1e6cf840811d
# skip: [a0560209f10cb55fbfcdd11ef85cfb444f8bea89] PCI: dwc: Add CONFIG_PCIE_DW_HOST to enable PCI dwc host
git bisect skip a0560209f10cb55fbfcdd11ef85cfb444f8bea89
# skip: [e3538f402453beca2d83002910cfe13b43d8a95b] PCI: exynos: Remove unnecessary local variables
git bisect skip e3538f402453beca2d83002910cfe13b43d8a95b
# skip: [11a61a8602812c024d8c404193ce1654ee3b8f08] PCI: dwc: Use PTR_ERR_OR_ZERO to simplify code
git bisect skip 11a61a8602812c024d8c404193ce1654ee3b8f08
# skip: [cdcb33f9824429a926b971bf041a6cec238f91ff] PCI: Avoid possible deadlock on pci_lock and p->pi_lock
git bisect skip cdcb33f9824429a926b971bf041a6cec238f91ff
# skip: [699c4cec238731a4c466f73fe6e9e45ab6f49a41] PCI/MSI: Remove pci_msi_domain_{alloc,free}_irqs()
git bisect skip 699c4cec238731a4c466f73fe6e9e45ab6f49a41
# skip: [21b7245034aa83c80fb8d0011e18d597a62a91f1] PCI: imx6: Fix a typo in error message
git bisect skip 21b7245034aa83c80fb8d0011e18d597a62a91f1
# skip: [e94888d23736cec51ba851f6e798d0eeb9ef5f41] PCI: rcar: Return -ENODEV from host bridge probe when no card present
git bisect skip e94888d23736cec51ba851f6e798d0eeb9ef5f41
# good: [190c3ee06a0f0660839785b7ad8a830e832d9481] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
git bisect good 190c3ee06a0f0660839785b7ad8a830e832d9481
# skip: [792e0a6814b63b120c6cfddf79a309046c6e840a] PCI: hisi: Use of_device_get_match_data() to simplify probe
git bisect skip 792e0a6814b63b120c6cfddf79a309046c6e840a
# skip: [4fe0395550aeb6709ea5332f46de3644aef7d328] PCI/MSI: Remove pci_enable_msi_{exact,range}()
git bisect skip 4fe0395550aeb6709ea5332f46de3644aef7d328
# skip: [7184f5b451cf3dc61de79091d235b5d2bba2782d] PCI: Add ACS quirk for Intel Union Point
git bisect skip 7184f5b451cf3dc61de79091d235b5d2bba2782d
# skip: [0b351c986a5ffedb502632c1b27690c9730d79c4] PCI: rockchip: Mark PM functions as __maybe_unused
git bisect skip 0b351c986a5ffedb502632c1b27690c9730d79c4
# skip: [656795c8873f93956a031d5db6fb08cf6168ebb0] Merge branch 'for-linus' into pci/host-designware
git bisect skip 656795c8873f93956a031d5db6fb08cf6168ebb0
# skip: [fed678145d02d08d75825d9f0854fad93cffd1a0] PCI: Remove duplicate check for positive return value from probe() functions
git bisect skip fed678145d02d08d75825d9f0854fad93cffd1a0
# skip: [60e2e2fbafdd1285ae1b4ad39ded41603e0c74d0] PCI: hv: Fix wslot_to_devfn() to fix warnings on device removal
git bisect skip 60e2e2fbafdd1285ae1b4ad39ded41603e0c74d0
# skip: [aeda9adebab8b5bdd90576e3065a1f3f948279ad] PCI/ASPM: Configure L1 substate settings
git bisect skip aeda9adebab8b5bdd90576e3065a1f3f948279ad
# skip: [2fd260f03b6a365bad48522f3948463928f91c2c] PCI/AER: Remove unused .link_reset() callback
git bisect skip 2fd260f03b6a365bad48522f3948463928f91c2c
# skip: [3fb5561879d71b5b80ddb48b3e7e5fa18c696d2a] PCI: generic: Call pci_fixup_irqs() only on ARM
git bisect skip 3fb5561879d71b5b80ddb48b3e7e5fa18c696d2a
# skip: [1ded56df3247d358390ae6dc09ccee620262ac5f] PCI: xgene: Fix double free on init error
git bisect skip 1ded56df3247d358390ae6dc09ccee620262ac5f
# skip: [cd3e2eb8905d14fe28a2fc75362b8ecec16f0fb6] PCI: Sort the list of devices with D3 delay quirk by ID
git bisect skip cd3e2eb8905d14fe28a2fc75362b8ecec16f0fb6
# skip: [0fc1223f0e77a748f7040562faaa7027f7db71ca] PCI/ASPM: Add L1 substate capability structure register definitions
git bisect skip 0fc1223f0e77a748f7040562faaa7027f7db71ca
# skip: [5800790a925b0aefb621ae3da86668c3a2867750] PCI: rockchip: Set vendor ID from local core config space
git bisect skip 5800790a925b0aefb621ae3da86668c3a2867750
# skip: [d6da7d90fad8e34afdebeadbb08484ea4c98a792] PCI: exynos: Replace the *_blk/*_phy/*_elb accessors
git bisect skip d6da7d90fad8e34afdebeadbb08484ea4c98a792
# skip: [72f2ff0deb870145a5a2d24cd75b4f9936159a62] PCI: Disable MSI for HiSilicon Hip06/Hip07 Root Ports
git bisect skip 72f2ff0deb870145a5a2d24cd75b4f9936159a62
# skip: [abdbf4d635a9a8c956bb9757a9d4f08c2abe1f97] PCI/DPC: Wait for Root Port busy to clear
git bisect skip abdbf4d635a9a8c956bb9757a9d4f08c2abe1f97
# skip: [1f6c4501c667a6bf25996f04c9cae49a88d90d01] PCI: dra7xx: Group PHY API invocations
git bisect skip 1f6c4501c667a6bf25996f04c9cae49a88d90d01
# skip: [a71280722eeba8f1afa51ad6656028dcb96e110b] PCI: imx6: Remove LTSSM disable workaround
git bisect skip a71280722eeba8f1afa51ad6656028dcb96e110b
# skip: [f1f0366dd6be9624f7d355b72cc909ab821eb4c0] PCI/ASPM: Calculate and save the L1.2 timing parameters
git bisect skip f1f0366dd6be9624f7d355b72cc909ab821eb4c0
# skip: [4e0a90b381bd8bddf1644591dc585cf4c6ea652e] PCI: exynos: Rename all pointer names from "exynos_pcie" to "ep"
git bisect skip 4e0a90b381bd8bddf1644591dc585cf4c6ea652e
# skip: [b5a0a9b59c8185aebcd9a717e2e6258b58c72c06] PCI/ASPM: Read and set up L1 substate capabilities
git bisect skip b5a0a9b59c8185aebcd9a717e2e6258b58c72c06
# skip: [8ed81ec82a8c57c3a6ad69b4c4d3e4801163c256] PCI: mvebu: Change delay after reset to the PCIe spec mandated 100ms
git bisect skip 8ed81ec82a8c57c3a6ad69b4c4d3e4801163c256
# skip: [26b54be568ad8611ebc62f02f9a5d5328e7b3392] PCI: xilinx-nwl: Remove mask for messages not supported by AXI
git bisect skip 26b54be568ad8611ebc62f02f9a5d5328e7b3392
# skip: [977509f7c5c6fb992ffcdf4291051af343b91645] PCI: Apply _HPX settings only to relevant devices
git bisect skip 977509f7c5c6fb992ffcdf4291051af343b91645
# skip: [53762ba810398c11efaf65f9a45d992125e86dcf] PCI/ACPI: Fix bus range comparison in pci_mcfg_lookup()
git bisect skip 53762ba810398c11efaf65f9a45d992125e86dcf
# skip: [70bc1b684b49781c882fec44023b37b7b45fe359] PCI: versatile: Configure PCIe MPS settings
git bisect skip 70bc1b684b49781c882fec44023b37b7b45fe359
# skip: [7da7a1a66e7700903bbc9ed09256fc95da34d43d] PCI: xgene: Configure PCIe MPS settings
git bisect skip 7da7a1a66e7700903bbc9ed09256fc95da34d43d
# skip: [d9bf28e2650fe3eeefed7e34841aea07d10c6543] PCI: mvebu: Handle changes to the bridge windows while enabled
git bisect skip d9bf28e2650fe3eeefed7e34841aea07d10c6543
# skip: [dadf17334f3820e2f8c561011706b6fb99bf9860] PCI/MSI: Fix msi_capability_init() kernel-doc warnings
git bisect skip dadf17334f3820e2f8c561011706b6fb99bf9860
# only skipped commits left to test
# possible first bad commit: [60e8d3e11645a1b9c4197d9786df3894332c1685] Merge tag 'pci-v4.11-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci
# possible first bad commit: [c4d052ce970ea98e9e1cc72461ba3b7a25397657] Merge branch 'pci/virtualization' into next
# possible first bad commit: [42d87e3ffbd53c4514bccc0f24e40d6231a567f5] Merge branch 'pci/msi' into next
# possible first bad commit: [2bdd584a7555d5ac3341b61e751a6ac807a9814a] Merge branch 'pci/enumeration' into next
# possible first bad commit: [9e38637a0832f15552ae130622808831f515107c] Merge branch 'pci/host-xilinx' into next
# possible first bad commit: [9d534265bdcd47659c9fd080fff4c005512c4770] Merge branch 'pci/host-xgene' into next
# possible first bad commit: [f08bf55e690ebdac28402955ef953e11ff478668] Merge branch 'pci/host-versatile' into next
# possible first bad commit: [b98a7f7509ed7a031ed750be11b72b7ca1c95513] Merge branch 'pci/host-thunder' into next
# possible first bad commit: [bcea623c65d5ad3b2f4e51908cffc9423175a10b] Merge branch 'pci/host-rockchip' into next
# possible first bad commit: [56195e9d1d68a9c201587cb55b7fdf6d43fb31b6] Merge branch 'pci/host-rcar' into next
# possible first bad commit: [dda718926c0fe091fd5c7ed3f88c9542d8bf67cf] Merge branch 'pci/host-mvebu' into next
# possible first bad commit: [e2dc4f225b765c2a12e5eaa7bc09994e67d13741] Merge branch 'pci/host-layerscape' into next
# possible first bad commit: [68094b4b8d72360f9a90804c203e4292c6662fdd] Merge branch 'pci/host-iproc' into next
# possible first bad commit: [e34e38bf891bd7bf7c9305ff1d2e1b29a77dd159] Merge branch 'pci/host-imx6' into next
# possible first bad commit: [5dcda98a3b7cfc0025ca49a700765ac9ba78a6a7] Merge branch 'pci/host-hv' into next
# possible first bad commit: [d9520971589b008faa8df31aa2bb3d1e4f1c4534] Merge branch 'pci/host-hisi' into next
# possible first bad commit: [b2e6d3055d5545b97533d4e8376fa848639d9951] Merge branch 'pci/host-exynos' into next
# possible first bad commit: [1a55761392b8a4703039614f3ef6275f88dad4c3] Merge branch 'pci/host-altera' into next
# possible first bad commit: [3bb0356bb1b5a2c4434b4a3267df6dd9927ca5bf] Merge branch 'pci/host-designware' into next
# possible first bad commit: [7a2b3f024b8b724257d82f76690e68c69d797efd] PCI: dwc: Remove dependency of designware on CONFIG_PCI
# possible first bad commit: [a0560209f10cb55fbfcdd11ef85cfb444f8bea89] PCI: dwc: Add CONFIG_PCIE_DW_HOST to enable PCI dwc host
# possible first bad commit: [feb85d9b1c47ea8dbcae559ff127b433fdb245b7] PCI: dwc: Split pcie-designware.c into host and core files
# possible first bad commit: [314fc854f50317931fb4dfaab431695ab886e8de] PCI: dwc: designware: Fix style errors in pcie-designware.c
# possible first bad commit: [5f334db665173facf2213854408bb5fa2445d0b3] PCI: dwc: designware: Parse "num-lanes" property in dw_pcie_setup_rc()
# possible first bad commit: [442ec4c04d1235f8c664a74004dae54a7a574d18] PCI: dwc: all: Split struct pcie_port into host-only and core structures
# possible first bad commit: [40f67fb2c384fe12741aa35010d62bfe8c98286c] PCI: dwc: designware: Get device pointer at the start of dw_pcie_host_init()
# possible first bad commit: [19ce01cc8cbc314d73db9755715a8f6e8ad59a7f] PCI: dwc: all: Rename cfg_read/cfg_write to read/write
# possible first bad commit: [9bcf0a6fdc5062e451cd6f1ab39045e142a5938f] PCI: dwc: all: Use platform_set_drvdata() to save private data
# possible first bad commit: [b90dc392212d1153a12eea15cbc6eae352a3c989] PCI: dwc: designware: Move register defines to designware header file
# possible first bad commit: [11a61a8602812c024d8c404193ce1654ee3b8f08] PCI: dwc: Use PTR_ERR_OR_ZERO to simplify code
# possible first bad commit: [1f6c4501c667a6bf25996f04c9cae49a88d90d01] PCI: dra7xx: Group PHY API invocations
# possible first bad commit: [ebe85a44aad47bb5f08c5cbd939eb5e40956c73b] PCI: dra7xx: Enable MSI and legacy interrupts simultaneously
# possible first bad commit: [ab5fe4f4d31ee27df9b8d7720da710d955d55737] PCI: dra7xx: Add support to force RC to work in GEN1 mode
# possible first bad commit: [602d38bc65aa2926d1ecd290a348e87aa8c21290] PCI: dra7xx: Simplify probe code with devm_gpiod_get_optional()
# possible first bad commit: [950bf6388bc22c2749b8b66c501df1462639d6bd] PCI: Move DesignWare IP support to new drivers/pci/dwc/ directory
# possible first bad commit: [656795c8873f93956a031d5db6fb08cf6168ebb0] Merge branch 'for-linus' into pci/host-designware
# possible first bad commit: [e7cd7ef58e1fedb09b16720919869a81d7a2b867] PCI: exynos: Support the PHY generic framework
# possible first bad commit: [34f80c7ddfffe262bf04fb03e198e64de4cec7fc] Documentation: binding: Modify the exynos5440 PCIe binding
# possible first bad commit: [cf0adb8e281b69801fb8faef18c14443d9d41d3c] phy: phy-exynos-pcie: Add support for Exynos PCIe PHY
# possible first bad commit: [ad8ec41afa98615a4154eee0121bcf8276695327] Documentation: samsung-phy: Add exynos-pcie-phy binding
# possible first bad commit: [81efbaddd687f1b478c15665716fd545e2e4401e] PCI/MSI: Fix msi_desc->affinity memory leak when freeing MSI IRQs
# possible first bad commit: [8ca6e0a75a5145458e8a680edf2394375f2129da] PCI: altera: Extract TLP completion status correctly
# possible first bad commit: [5800790a925b0aefb621ae3da86668c3a2867750] PCI: rockchip: Set vendor ID from local core config space
# possible first bad commit: [33be632b8443b6ac74aa293504f430604fb9abeb] PCI: Add ACS quirk for Qualcomm QDF2400 and QDF2432
# possible first bad commit: [4a9b0933bdfcd85da840284bf5a0eb17b654b9c2] PCI: hv: Use device serial number as PCI domain
# possible first bad commit: [cd3e2eb8905d14fe28a2fc75362b8ecec16f0fb6] PCI: Sort the list of devices with D3 delay quirk by ID
# possible first bad commit: [4c2ae32d4c6ccd63819f36702f8455a5b08eaca8] Merge branch 'pci/vpd' into next
# possible first bad commit: [8b9c156262f9ebaa88736b93263c1af5ae5fa43e] Merge branch 'pci/virtualization' into next
# possible first bad commit: [0dc49eb2a25a6b48e235803eeb2b52677f577ca5] Merge branch 'pci/resource' into next
# possible first bad commit: [3ec2574e31b5aebeeed52ea784bb6d7960788202] Merge branch 'pci/msi' into next
# possible first bad commit: [906c142634def5dd3dda533d5d20bd3c1c19d1fb] Merge branch 'pci/hotplug' into next
# possible first bad commit: [63ab93f021ecd815d848c3e9d3e326fa9628e5a9] Merge branch 'pci/enumeration' into next
# possible first bad commit: [af3a2ab5daa0b8bf188a039c122a670cdc8b9544] Merge branch 'pci/dpc' into next
# possible first bad commit: [8f417ca9ebfa8701a41db64f5ed9cbb01b8e4219] Merge branch 'pci/aspm' into next
# possible first bad commit: [ca0ef7fecc881a7e8d11db2d8852ab580cd29e03] Merge branch 'pci/aer' into next
# possible first bad commit: [e4e7d59704d485f272061cea9057798dda3cfd99] PCI/MSI: Update MSI/MSI-X bits in PCIEBUS-HOWTO
# possible first bad commit: [c3cf2c61ddc1410424da0ea87717edf16fc296c5] PCI/MSI: Document pci_alloc_irq_vectors(), deprecate pci_enable_msi()
# possible first bad commit: [a142f4d3e5c54db5e942fa6ee5f3dc0e8c83207b] PCI/ASPM: Add comment about L1 substate latency
# possible first bad commit: [aeda9adebab8b5bdd90576e3065a1f3f948279ad] PCI/ASPM: Configure L1 substate settings
# possible first bad commit: [f1f0366dd6be9624f7d355b72cc909ab821eb4c0] PCI/ASPM: Calculate and save the L1.2 timing parameters
# possible first bad commit: [b5a0a9b59c8185aebcd9a717e2e6258b58c72c06] PCI/ASPM: Read and set up L1 substate capabilities
# possible first bad commit: [b2103ccbb67e3ef0f7a75d21c989f9614ddbcaca] PCI/ASPM: Add support for L1 substates
# possible first bad commit: [0fc1223f0e77a748f7040562faaa7027f7db71ca] PCI/ASPM: Add L1 substate capability structure register definitions
# possible first bad commit: [3278478084747c02725ba804d672235f2ba56bbc] PCI: exynos: Refactor to make it easier to support other SoCs
# possible first bad commit: [948b7620c15411444167a62cfc14cdd4b0e44682] PCI/MSI: Return -ENOSPC if pci_enable_msi_range() can't get enough vectors
# possible first bad commit: [3674cc49da9a8fc55bf1dec2ab03a66c77f2dcdf] PCI/portdrv: Use pci_irq_alloc_vectors()
# possible first bad commit: [862290f9e23c39051e59bf12ce65707a8ec8b911] PCI/MSI: Check that we have a legacy interrupt line before using it
# possible first bad commit: [f1d722b607d610b66785f7f00d2e1d457260647c] PCI: rockchip: Fix rockchip_pcie_probe() error path to free resource list
# possible first bad commit: [21b7245034aa83c80fb8d0011e18d597a62a91f1] PCI: imx6: Fix a typo in error message
# possible first bad commit: [60e2e2fbafdd1285ae1b4ad39ded41603e0c74d0] PCI: hv: Fix wslot_to_devfn() to fix warnings on device removal
# possible first bad commit: [fed678145d02d08d75825d9f0854fad93cffd1a0] PCI: Remove duplicate check for positive return value from probe() functions
# possible first bad commit: [abdbf4d635a9a8c956bb9757a9d4f08c2abe1f97] PCI/DPC: Wait for Root Port busy to clear
# possible first bad commit: [87b336d003d47876e376d943be3c9d35152f3b86] PCI/DPC: Decode extended reasons
# possible first bad commit: [699c4cec238731a4c466f73fe6e9e45ab6f49a41] PCI/MSI: Remove pci_msi_domain_{alloc,free}_irqs()
# possible first bad commit: [47feb41888bc82ba5c9268c344775adc483d6de7] PCI/MSI: Remove unused pci_msi_create_default_irq_domain()
# possible first bad commit: [26b54be568ad8611ebc62f02f9a5d5328e7b3392] PCI: xilinx-nwl: Remove mask for messages not supported by AXI
# possible first bad commit: [3adfb572f2978a980b250a9e1a56f84f3a031001] PCI/MSI: Return failure when msix_setup_entries() fails
# possible first bad commit: [7da7a1a66e7700903bbc9ed09256fc95da34d43d] PCI: xgene: Configure PCIe MPS settings
# possible first bad commit: [60db3a4d8cc9073cf56264785197ba75ee1caca4] PCI: Enable PCIe Extended Tags if supported
# possible first bad commit: [2fd260f03b6a365bad48522f3948463928f91c2c] PCI/AER: Remove unused .link_reset() callback
# possible first bad commit: [3fb5561879d71b5b80ddb48b3e7e5fa18c696d2a] PCI: generic: Call pci_fixup_irqs() only on ARM
# possible first bad commit: [72f2ff0deb870145a5a2d24cd75b4f9936159a62] PCI: Disable MSI for HiSilicon Hip06/Hip07 Root Ports
# possible first bad commit: [4788316f743539076712d2b80b6cd289458fe2be] PCI: hisi: Rename config space accessors to remove "acpi"
# possible first bad commit: [ce709f86501a013e941e9986cb072eae375ddf3e] PCI: Add Broadcom Northstar2 PAXC quirk for device class and MPSS
# possible first bad commit: [70bc1b684b49781c882fec44023b37b7b45fe359] PCI: versatile: Configure PCIe MPS settings
# possible first bad commit: [ec6bd78a09d9967c4fcec53e7fabfaabd4f0e367] PCI: xilinx: Configure PCIe MPS settings
# possible first bad commit: [4d4836ab70d3c59bc934d244f0cccdd035c1ead8] PCI: iproc: Configure PCIe MPS settings
# possible first bad commit: [92004a064875e45f0ceec72f6962e258e59f0682] PCI: exynos: Remove duplicated code
# possible first bad commit: [2681c0e7ff70a6d41a81527f95f2edabafea4ace] PCI: exynos: Use the bitops BIT() macro to build bitmasks
# possible first bad commit: [e3538f402453beca2d83002910cfe13b43d8a95b] PCI: exynos: Remove unnecessary local variables
# possible first bad commit: [d6da7d90fad8e34afdebeadbb08484ea4c98a792] PCI: exynos: Replace the *_blk/*_phy/*_elb accessors
# possible first bad commit: [8ed81ec82a8c57c3a6ad69b4c4d3e4801163c256] PCI: mvebu: Change delay after reset to the PCIe spec mandated 100ms
# possible first bad commit: [a2ec1996098c7da0593a0981190316025301eab1] PCI: hisi: Add DT almost-ECAM support for Hip06/Hip07 host controllers
# possible first bad commit: [792e0a6814b63b120c6cfddf79a309046c6e840a] PCI: hisi: Use of_device_get_match_data() to simplify probe
# possible first bad commit: [5b0948dfe138f0837699f46f5877f4f81c252dac] PCI: Lock each enable/disable num_vfs operation in sysfs
# possible first bad commit: [4f69bd16df1a38c32d5d207a96d1d86df4808f87] PCI: Increase VPD access timeout to 125ms
# possible first bad commit: [e5c3b3e9f012b8862753a04f03c6e27344332718] PCI: iproc: Use of_device_get_match_data() to simplify probe
# possible first bad commit: [6dc2c04fd9868a5ad00b402935021d6f3ff27b17] PCI: layerscape: Use of_device_get_match_data() to simplify probe
# possible first bad commit: [ff1677e231651205e7e19770a677057dea05cb70] PCI: rcar: Use of_device_get_match_data() to simplify probe
# possible first bad commit: [1ded56df3247d358390ae6dc09ccee620262ac5f] PCI: xgene: Fix double free on init error
# possible first bad commit: [a71280722eeba8f1afa51ad6656028dcb96e110b] PCI: imx6: Remove LTSSM disable workaround
# possible first bad commit: [cdcb33f9824429a926b971bf041a6cec238f91ff] PCI: Avoid possible deadlock on pci_lock and p->pi_lock
# possible first bad commit: [0b351c986a5ffedb502632c1b27690c9730d79c4] PCI: rockchip: Mark PM functions as __maybe_unused
# possible first bad commit: [7faebda21d573a6889bab1e0100ed4092a5a4716] PCI: rockchip: Use readl_poll_timeout() instead of open-coding it
# possible first bad commit: [d9bf28e2650fe3eeefed7e34841aea07d10c6543] PCI: mvebu: Handle changes to the bridge windows while enabled
# possible first bad commit: [4e0a90b381bd8bddf1644591dc585cf4c6ea652e] PCI: exynos: Rename all pointer names from "exynos_pcie" to "ep"
# possible first bad commit: [7184f5b451cf3dc61de79091d235b5d2bba2782d] PCI: Add ACS quirk for Intel Union Point
# possible first bad commit: [4fe0395550aeb6709ea5332f46de3644aef7d328] PCI/MSI: Remove pci_enable_msi_{exact,range}()
# possible first bad commit: [e75377404726be171d66c154f8ea1e6cf840811d] amd-xgbe: Update PCI support to use new IRQ functions
# possible first bad commit: [e8e8dd6d20fa55ff974460ad742fcbf35b301062] [media] cobalt: use pci_irq_allocate_vectors()
# possible first bad commit: [c5c4d3a3f4a8c830cb514eb469a1025df2df0379] PCI: thunder-pem: Add support for cn81xx and cn83xx SoCs
# possible first bad commit: [afc9595ea4770f0157ae06fb3acedff703e169b6] PCI: rockchip: Disable RC's ASPM L0s based on DT "aspm-no-l0s"
# possible first bad commit: [53762ba810398c11efaf65f9a45d992125e86dcf] PCI/ACPI: Fix bus range comparison in pci_mcfg_lookup()
# possible first bad commit: [dadf17334f3820e2f8c561011706b6fb99bf9860] PCI/MSI: Fix msi_capability_init() kernel-doc warnings
# possible first bad commit: [caf3f562e1161a86bd48a4c4c33af89d3693c658] PCI: imx6: Remove redundant "Link never came up" message
# possible first bad commit: [1acf8bca9cdb2443a8707ff0afc3aadbfb5669f4] PCI: acpiphp_ibm: Make ibm_apci_table_attr __ro_after_init
# possible first bad commit: [63d182abd71cb47cee4adb8dd2afd71d987794d5] PCI: rpadlpar: Remove unnecessary return statement
# possible first bad commit: [8267b07526cabe2e2afc834a138ece8644af87ed] PCI: rcar: Add compatible string for r8a7796
# possible first bad commit: [e94888d23736cec51ba851f6e798d0eeb9ef5f41] PCI: rcar: Return -ENODEV from host bridge probe when no card present
# possible first bad commit: [013dd3d5e1835c2cfa9c824e61465b61509afa54] PCI: rockchip: Add system PM support
# possible first bad commit: [977509f7c5c6fb992ffcdf4291051af343b91645] PCI: Apply _HPX settings only to relevant devices
# possible first bad commit: [25e77388e1ab63e11e21d94a994eca227472aeed] PCI: Remove res_to_dev_res() debug message
# skip: [dadf17334f3820e2f8c561011706b6fb99bf9860] PCI/MSI: Fix msi_capability_init() kernel-doc warnings
git bisect skip dadf17334f3820e2f8c561011706b6fb99bf9860

Concerning myself with git I randomly discovered that.

That is the last GOOD commit :
# good: [190c3ee06a0f0660839785b7ad8a830e832d9481] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net

That is the first BAD commit :
# bad: [60e8d3e11645a1b9c4197d9786df3894332c1685] Merge tag 'pci-v4.11-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci


git rev-list v4.11-rc1
...
d939cdfde34f50b95254b375f498447c82190b3e
60e8d3e11645a1b9c4197d9786df3894332c1685 (BAD)
190c3ee06a0f0660839785b7ad8a830e832d9481 (GOOD)
e93b9865251a0503d83fd570e7d5a7c8bc351715
...

Both commits are as close as they can be in the commits revision history.
Is this helpful ?

Hmm, seems like a lucky punch to mee. No chance if the two commits would have been vice versa ;-)

# skip: [d939cdfde34f50b95254b375f498447c82190b3e] md/linear: shutup lockdep warnning
git bisect skip d939cdfde34f50b95254b375f498447c82190b3e

git rev-list v4.11-rc1
...
d939cdfde34f50b95254b375f498447c82190b3e (Kernel Panic)
60e8d3e11645a1b9c4197d9786df3894332c1685 (BAD)
190c3ee06a0f0660839785b7ad8a830e832d9481 (GOOD)
e93b9865251a0503d83fd570e7d5a7c8bc351715
...

(In reply to Andreas Brogle from comment #14)
> 60e8d3e11645a1b9c4197d9786df3894332c1685 (BAD)
> 190c3ee06a0f0660839785b7ad8a830e832d9481 (GOOD)

There are only PCI subsystem changes between those two commits. Please reassign the product/component of this report accordingly.

What do you suggest ?
Who is the maintainer for fixing this bug ?

Reassigning to PCI and marking as a regression (v4.10 works, v4.11 fails).

Can you please test a current kernel, e.g., v4.14 or even v4.15-rc8?  The v4.11 kernel you're using is quite old, and it's possible this has been fixed since then.  If that's the case, we would just need to decide whether a v4.11 stable kernel needs the fix, and if so, backport the existing fix.

*** Bug 198239 has been marked as a duplicate of this bug. ***

Hello Bjorn,

the bisection process I'm have been running in December 2017 started with a kernel 4.15. Aside this I'm using Debian Sid with recent kernel, daily upgraded. 
The bug exists since v4.11 and up, with every kernel, without any exceptions.

Sincerely, Andreas

OK, great.  I didn't see any reference to v4.15 above, so I couldn't tell that it was also broken.

You should be able to bisect between those two commits to narrow it down further:

60e8d3e11645 ("Merge tag 'pci-v4.11-changes...)
190c3ee06a0f ("Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net

Hello,

bisection is in progress, starting with
> 60e8d3e11645 ("Merge tag 'pci-v4.11-changes...)
BAD
> 190c3ee06a0f ("Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
GOOD

There are 60 commits left to test. But I'm afraid it will bring no further information. The first 6 kernels between the starting points all lead to a kernel panic and had to be skipped. The same problem I have had in Dezember.
I'm doing that testing again, bit it would need presumably some weeks.

What are the details of the kernel panic?

It's surprising that 6 kernels chosen by git bisect would all panic.  Do you have notes about which commits those were?

Created attachment 273811 [details]
serial console dump

Hello,

every commit between the two I have started leads to a kernel panic. Meanwhile there are more than 50 tested.

The boot log, read by serial console is attached.

Created attachment 273813 [details]
Commits that lead to a kernel panic

All the skipped commits lead to a kernel bug, see attachment.

Oof, stop booting all those kernels.  I think that's wasting your time.  We need to figure out what's causing the panic first, because I think there's something else going on there.

Just to double-check, all the "git bisect skip <SHA1>" kernels cause a panic, and all the panics look like the one in comment #24?  E.g., all these:

  1a55761392b8 ("Merge branch 'pci/host-altera' into next")
  3ec2574e31b5 ("Merge branch 'pci/msi' into next")
  21b7245034aa ("PCI: imx6: Fix a typo in error message")

cause a panic?

Created attachment 273819 [details]
1a55761392b8 ("Merge branch 'pci/host-altera' into next")

Boot log file attached.

Created attachment 273821 [details]
3ec2574e31b5 ("Merge branch 'pci/msi' into next")

Created attachment 273823 [details]
21b7245034aa ("PCI: imx6: Fix a typo in error message")

Created attachment 273909 [details]
git bisect log

Here is the complete log file of the bisection process.
There are no commits left to test. 
All the skipped commits lead to kernel panic
RIP: kobject_get+0x10/0x80 RSP: ffffc90003127d20
Is that the relevant line ?

A script has done the job for me. I only had to press the reset button from time to time, aside doing some boring work on an other machine.

I think the panics are likely this one: https://patchwork.kernel.org/patch/9488241/

Let's start with 1a55761392b8 ("Merge branch 'pci/host-altera' into next") since that's one kernel that panics for you and you've already attached the dmesg log for it.  Can you build that kernel again, confirm that it still panics, and attach the .config file here?  Maybe we can verify that you're seeing the MCE issue.  If so, maybe we can figure out a config change or simple patch that will enable us to get better information out of the bisect.

Created attachment 273931 [details]
.config file

Here is the .config file.

Tried
CONFIG_HOTPLUG_CPU=n
Kernel Panic and Hardware Reset Needed
same as before

Tried 
disabling SMP
Kernel Panic and CTRL-ALT-DEL reboot possible

Can you turn off CONFIG_X86_MCE_AMD in 1a55761392b8 ("Merge branch 'pci/host-altera' into next")?

That shouldn't panic, so maybe you'll be able to bisect between:

  60e8d3e11645 ("Merge tag 'pci-v4.11-changes...)
  190c3ee06a0f ("Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net

Hello,

CONFIG_X86_MCE_AMD=n was the key to avoid kernel panic.

git bisect now has run and finished regularly.


Here is the git bisect log :

git bisect start
# bad: [1a55761392b8a4703039614f3ef6275f88dad4c3] Merge branch 'pci/host-altera' into next
git bisect bad 1a55761392b8a4703039614f3ef6275f88dad4c3
# good: [190c3ee06a0f0660839785b7ad8a830e832d9481] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
git bisect good 190c3ee06a0f0660839785b7ad8a830e832d9481
# good: [a782b5f986c3fa1cfa7f2b57941200c6a5809242] PCI: designware: Check for iATU unroll only on platforms that use ATU
git bisect good a782b5f986c3fa1cfa7f2b57941200c6a5809242
# bad: [3ec2574e31b5aebeeed52ea784bb6d7960788202] Merge branch 'pci/msi' into next
git bisect bad 3ec2574e31b5aebeeed52ea784bb6d7960788202
# bad: [63ab93f021ecd815d848c3e9d3e326fa9628e5a9] Merge branch 'pci/enumeration' into next
git bisect bad 63ab93f021ecd815d848c3e9d3e326fa9628e5a9
# good: [8f417ca9ebfa8701a41db64f5ed9cbb01b8e4219] Merge branch 'pci/aspm' into next
git bisect good 8f417ca9ebfa8701a41db64f5ed9cbb01b8e4219
# bad: [60db3a4d8cc9073cf56264785197ba75ee1caca4] PCI: Enable PCIe Extended Tags if supported
git bisect bad 60db3a4d8cc9073cf56264785197ba75ee1caca4
# good: [53762ba810398c11efaf65f9a45d992125e86dcf] PCI/ACPI: Fix bus range comparison in pci_mcfg_lookup()
git bisect good 53762ba810398c11efaf65f9a45d992125e86dcf
# good: [cdcb33f9824429a926b971bf041a6cec238f91ff] PCI: Avoid possible deadlock on pci_lock and p->pi_lock
git bisect good cdcb33f9824429a926b971bf041a6cec238f91ff
# first bad commit: [60db3a4d8cc9073cf56264785197ba75ee1caca4] PCI: Enable PCIe Extended Tags if supported

Great!  I should have figured out the CONFIG_X86_MCE_AMD connection long ago; sorry about that.

So apparently 60db3a4d8cc9 ("PCI: Enable PCIe Extended Tags if supported") is the culprit.  We thought Extended Tags were safe to enable by default whenever they're supported, but maybe there's some wrinkle there.  Adding Sinan, the author of that commit.

Commit 60db3a4d8cc9 appeared in v4.11, which matches your experience, Andreas.

However, Ivan (comment #11) is seeing issues with v4.10.  That must be a different problem.

Can you attach the complete "lspci -vvxxx" output as root?

(In reply to Bjorn Helgaas from comment #35)

> However, Ivan (comment #11) is seeing issues with v4.10.  That must be a
> different problem.

See #198239.

For me, nuking splashutils and fb_con_decor solved the problem.

I don't know if they worked as a "trigger" for this "PCIe Extended Tags" bug (or any other bug), anyway all is working fine for me and CONFIG_X86_MCE_AMD=y in my kernel.

Maybe, we have another buggy platform.

"Andreas Brogle 2017-07-21 12:38:40 UTC
Got the opportunity to test the RV630 XT [Radeon HD 2600 XT] card at an other mainboard. It works as it should. So the problem seems to be mainboard specific."

Enabling extended tags should be safe according to the spec. We have already found some exceptions.

Just for clarity, can you go to 4.15 kernel and just revert 

"first bad commit: [60db3a4d8cc9073cf56264785197ba75ee1caca4] PCI: Enable PCIe Extended Tags if supported"

to see if the problem goes away?

Making sure that we are chasing the right guy.

(In reply to Ivan Iraci from comment #36)
> (In reply to Bjorn Helgaas from comment #35)
> 
> > However, Ivan (comment #11) is seeing issues with v4.10.  That must be a
> > different problem.
> 
> See #198239.
> 
> For me, nuking splashutils and fb_con_decor solved the problem.
> 
> I don't know if they worked as a "trigger" for this "PCIe Extended Tags" bug
> (or any other bug), anyway all is working fine for me and
> CONFIG_X86_MCE_AMD=y in my kernel.

Nuking splashutils and fb_con_decor sounds like a workaround, not a real solution.  We would really like to get the the root cause of the problem so you (and others!) don't have to discover and use the workaround.

If you're able to help debug that, it would be great.  I would start by opening a new bugzilla (so it doesn't get confused with *this* one), reproduce the problem, and attach the complete dmesg and "lspci -vv" logs.  If you can identify an old working kernel and a new broken one, that would be great.  If you could bisect and identify a specific commit, that would be even better (but it's a lot of work).

Created attachment 273949 [details]
lspci.log

See attachment.

Thanks, Andreas

(In reply to Bjorn Helgaas from comment #38)

> If you're able to help debug that, it would be great.  I would start by
> opening a new bugzilla (so it doesn't get confused with *this* one),
> reproduce the problem, and attach the complete dmesg and "lspci -vv" logs. 
> If you can identify an old working kernel and a new broken one, that would
> be great.  If you could bisect and identify a specific commit, that would be
> even better (but it's a lot of work).

I am really sorry, but that one is a production workstation and this bug already costed me several days of work, I totally cannot afford other downtimes. If I had a spare PC with a similar setup I would happily help you.

(In reply to Sinan Kaya from comment #37)

> Just for clarity, can you go to 4.15 kernel and just revert 
> 
> "first bad commit: [60db3a4d8cc9073cf56264785197ba75ee1caca4] PCI: Enable
> PCIe Extended Tags if supported"
> 
> to see if the problem goes away?

Hello Sinan,

I'm willing to do any tests to get rid of that bug.
But I'm confused. Meanwhile there are 4 people writing to this thread and it's not clear who is meaned. Do you want Ivan or me to start a new bisection process with latest kernel 4.15 ?

Greetings, Andreas

I can work with anyone that can reproduce the issue. I just want to see if Extended Tags is the real reason for the failure. Go ahead and test without the patch on a recent kernel. It looked liked all kernels between 4.11 and 4.15 were broken.

Andreas, sorry if I confused the issue.  The problem Ivan is seeing occurs on v4.10, which does not enable extended tags, so it is different from what you're seeing.  My proposal was to treat Ivan's issue as a different bugzilla and a new debugging process.

You (Andreas) do not need to bisect any more because you already identified the cause of the problem *you* are seeing.  Sinan is asking if you can test v4.15 with 60db3a4d8cc9 reverted.  That *should* work correctly.  If so, we can extend the existing quirk [1] to cover your system as well.  The quirk currently covers Broadcom HT2100 devices, and you have Broadcom HT2000 devices, so it's not surprising that they have similar problems.

[1] http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=62ce94a7a5a5

(In reply to Bjorn Helgaas from comment #43)

> Andreas, sorry if I confused the issue.  The problem Ivan is seeing occurs
> on v4.10, which does not enable extended tags, so it is different from what
> you're seeing.  My proposal was to treat Ivan's issue as a different
> bugzilla and a new debugging process.

In the last year and an half (maybe two) fbsplash showed segfaults from time to time in dmesg. Starting from Kernel 4.n (n = 10 approx) I think that some incompatibilities between the kernel, fbsplash and my hardware exploded far beyond the segfaults that showed up in dmesg. Alas, I understood that the 0xCAFEDEAD was correlated to (or triggered by) fbsplash too late.

Anyway, fbsplash now is defunct (I disintalled it and it doesn't even compile with recent libs) and no one else is reporting similar problems.

So any effort to investigate my issue would be a waste of time, since any kernel is working fine for me now.

Created attachment 273953 [details]
bisect log file

Hello Sinan,

did a fresh bisection process with the same result, log file attached.

This is what I have started with :
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
git bisect reset
git bisect start
git bisect good v4.10-rc8

Hi Andreas,

I didn't mean that you need to do a bisect. 

try this:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
git checkout v4.15
git revert 60db3a4d8cc9073cf56264785197ba75ee1caca4

test this kernel and see if it works.

Bjorn and I think we understand the problem. Just need confirmation.

Sinan

(In reply to Sinan Kaya from comment #46)

Hi Sinan,

> git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> git checkout v4.15
> git revert 60db3a4d8cc9073cf56264785197ba75ee1caca4

I'm capitulating about that git revert.
Get some errors because pathes are not merged (sorry can't translate exactly those german error messages).
What to before git revert ?

Andreas

Ok, let's try a brute force approach. Comment out this line:

http://elixir.free-electrons.com/linux/latest/source/drivers/pci/probe.c#L1884

(In reply to Sinan Kaya from comment #48)
> Ok, let's try a brute force approach. Comment out this line:
> 
> http://elixir.free-electrons.com/linux/latest/source/drivers/pci/probe.
> c#L1884

Tested, looks good.

OK. Thanks for confirming. Bjorn contacted AMD guys for help yesterday. Will let you know how to proceed. It is a bug in AMD's root port hardware. We have blacklisted some versions already. Your system seems to be in the same category but not blacklisted.

Hello,

as we have localized the problem, will it be fixed ?

Bjorn,
Do you want to go ahead and blacklist all these Broadcom HTxxx root ports?
Sinan

> Do you want to go ahead and blacklist all these Broadcom HTxxx root ports?

Yes, that seems like the only choice we have.  I did ask some people at Broadcom and submitted a query via the formal Broadcom support channel, but I'm not optimistic.

Separately, I wonder if we should also add some way to disable extended tags via a kernel parameter.  I'm not enthusiastic about adding a whole collection of "disable this", "disable that" flags, but it's a pretty big burden for folks to have to bisect to find issues like this.  Maybe we should have a "pci=safe" or "pci=conservative" or something that could be the first step in debugging PCI-related issues.  I guess if that made a difference, the next step would have to be disabling individual features to narrow it down, so maybe we'd end up with a collection of more specific parameters, too.  Do you have any ideas?

how about having feature enables in Kconfig instead for these that are set by default but can be individually disabled?

	pci_configure_mps(dev);
	pci_configure_extended_tags(dev, NULL);
	pci_configure_relaxed_ordering(dev);
	pci_configure_ltr(dev);

That looks like a good set of potentially troublesome features.  I was thinking more of a kernel boot parameter that could be used without having to rebuild the kernel.

we could overload "pci=pcie_bus_safe" and skip everything except mps.

(In reply to Bjorn Helgaas from comment #53)
> > Do you want to go ahead and blacklist all these Broadcom HTxxx root ports?
> 
> Yes, that seems like the only choice we have.  I did ask some people at
> Broadcom and submitted a query via the formal Broadcom support channel, but
> I'm not optimistic.

My "official channels" query to Broadcom (service request CQSUP00060698) got closed after 3 weeks with "this is the wrong place to ask, go somewhere else (specifically, to the 'customer support portal under "myBroadcom" after you login').  I can't be bothered to try to figure that out.

Please blacklist them all.

posted a patch with subject "PCI: quirk HT1100 & HT2000 and one HT2100 Root Ports for Extended Tags"

Bjorn merged the change and also sent to stable for backporting. I believe this can be closed now.

The change is http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=1b30dfd376e2, which appeared in v4.17-rc1.

I'll close this, but please reopen if you continue to see this issue with v4.17-rc1 or later.

Hello Bjorn and Sinan,

I can confirm that this bug is fixed with kernel v4.17-rc3.

Thank you both very much for fixing it.

Greetings, Andreas

Glad to hear. Sorry for the inconvenience.

I think I've got a similar problem with the kernel 5.3.0-46-generic. 


Hardware:

Device-1: Intel Haswell-ULT Integrated Graphics driver: i915 v: kernel 
  Device-2: AMD Sun XT [Radeon HD 8670A/8670M/8690M / R5 M330 / M430 / R7 
  M520] 

kern.log:
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36878.943052] [drm] PCIE gen 2 link speeds already enabled
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36878.957040] [drm] PCIE GART of 2048M enabled (table at 0x0000000000040000).
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36878.957149] radeon 0000:06:00.0: WB enabled
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36878.957152] radeon 0000:06:00.0: fence driver on ring 0 use gpu addr 0x0000000080000c00 and cpu addr 0x00000000b6b15b20
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36878.957154] radeon 0000:06:00.0: fence driver on ring 1 use gpu addr 0x0000000080000c04 and cpu addr 0x0000000074d2a471
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36878.957155] radeon 0000:06:00.0: fence driver on ring 2 use gpu addr 0x0000000080000c08 and cpu addr 0x00000000b655a957
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36878.957157] radeon 0000:06:00.0: fence driver on ring 3 use gpu addr 0x0000000080000c0c and cpu addr 0x0000000074105431
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36878.957158] radeon 0000:06:00.0: fence driver on ring 4 use gpu addr 0x0000000080000c10 and cpu addr 0x000000006f86e7fe
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36879.675253] [drm:r600_ring_test [radeon]] *ERROR* radeon: ring 0 test failed (scratch(0x850C)=0xCAFEDEAD)
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36879.675288] [drm:si_resume [radeon]] *ERROR* si startup failed on resume
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.594254] ACPI: EC: interrupt blocked
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.633962] ACPI: Preparing to enter system sleep state S3
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.637262] ACPI: EC: event blocked
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.637263] ACPI: EC: EC stopped
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.637264] PM: Saving platform NVS memory
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.637279] Disabling non-boot CPUs ...
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.638996] smpboot: CPU 1 is now offline
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.639698] IRQ 49: no longer affine to CPU2
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.640706] smpboot: CPU 2 is now offline
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.641340] IRQ 23: no longer affine to CPU3
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.641344] IRQ 45: no longer affine to CPU3
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.641346] IRQ 47: no longer affine to CPU3
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.642359] smpboot: CPU 3 is now offline
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.643727] ACPI: Low-level resume complete
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.643838] ACPI: EC: EC started
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.643840] PM: Restoring platform NVS memory
Apr  6 12:07:00 tobias-ThinkPad-S5-S540 kernel: [36880.644400] Enabling non-boot CPUs ...