Bug 54411

Summary: kernel panic when parallel remove pci device triggered by sysfs/pci interface
Product: Drivers Reporter: Gu Zheng (guz.fnst)
Component: PCIAssignee: drivers_pci (drivers_pci)
Status: CLOSED CODE_FIX    
Severity: normal CC: alan, bjorn, myron.stowe, yinghai
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 3.8.rc6 Subsystem:
Regression: No Bisected commit-id:
Attachments: lspci info
test patch from Yinghai
booting dmesg with Yinghai's patch on kernel 3.8
panic dmesg info with concurrent removal on linus's 0226 tree with Yinghai's patch

Description Gu Zheng 2013-02-25 02:29:23 UTC
Created attachment 94021 [details]
lspci info

When we used a test script:
echo -n 1 > /sys/bus/pci/devices/0000\:10\:00.0/remove ; echo -n 1 >  /sys/bus/pci/devices/0000\:1a\:01.0/remove
to test parallel remove routines triggered by sysfs/pci interface, the kernel panicked.

[  328.037479] general protection fault: 0000 [#1] SMP 
[  328.096991] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc sunrpc binfmt_misc dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun uinput iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm crc32c_intel microcode pcspkr lpc_ich mfd_core sg i2c_i801 i2c_core ioatdma i7core_edac edac_core e1000e igb dca ptp pps_core sd_mod crc_t10dif megaraid_sas mptsas mptscsih mptbase scsi_transport_sas scsi_mod
[  328.697122] CPU 6 
[  328.719040] Pid: 6, comm: kworker/u:0 Tainted: G        W    3.8.0-rc6-aspm-pcie-fix+ #58 FUJITSU-SV PRIMEQUEST 1800E/SB
[  328.851117] RIP: 0010:[<ffffffff813928f8>]  [<ffffffff813928f8>] pcie_aspm_exit_link_state+0x38/0x190
[  328.961428] RSP: 0018:ffff8807bde17c48  EFLAGS: 00010202
[  329.024874] RAX: ffff8807bb4a1290 RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000006
[  329.110125] RDX: 0000000000000006 RSI: ffff8807bde1afc8 RDI: 0000000000000246
[  329.195371] RBP: ffff8807bde17c68 R08: 0000000000000001 R09: 0000000000000001
[  329.280619] R10: 0000000000000003 R11: 0000000000000001 R12: ffff8807bb49b3d8
[  329.365869] R13: ffff8807bb49b3d8 R14: ffffffff82126d80 R15: ffff8807bde17d58
[  329.451127] FS:  0000000000000000(0000) GS:ffff8807c2600000(0000) knlGS:0000000000000000
[  329.547796] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  329.616431] CR2: ffffffffff600400 CR3: 0000000001c0c000 CR4: 00000000000007e0
[  329.701687] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  329.786935] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  329.872185] Process kworker/u:0 (pid: 6, threadinfo ffff8807bde16000, task ffff8807bde1a680)
[  329.973006] Stack:
[  329.997000]  0000000000000006 ffff8807bb49b3d8 0000000000000000 ffff8807bb49b3d8
[  330.085822]  ffff8807bde17c88 ffffffff81380f42 2222222222222222 ffff8807bb49b3d8
[  330.174627]  ffff8807bde17cb8 ffffffff81380fb4 0000000000000000 ffff8807bb49b3d8
[  330.263427] Call Trace:
[  330.292616]  [<ffffffff81380f42>] pci_stop_dev+0xb2/0xd0
[  330.356064]  [<ffffffff81380fb4>] pci_stop_bus_device+0x54/0x60
[  330.426778]  [<ffffffff81381156>] pci_stop_and_remove_bus_device+0x16/0x30
[  330.508919]  [<ffffffff8138894b>] remove_callback+0x2b/0x40
[  330.575487]  [<ffffffff8125a82a>] sysfs_schedule_callback_work+0x1a/0x80
[  330.655551]  [<ffffffff81091b81>] process_one_work+0x241/0x5f0
[  330.725228]  [<ffffffff81091b0f>] ? process_one_work+0x1cf/0x5f0
[  330.796981]  [<ffffffff8125a810>] ? sysfs_schedule_callback+0x210/0x210
[  330.876002]  [<ffffffff81093d3b>] worker_thread+0x12b/0x3f0
[  330.942567]  [<ffffffff81093c10>] ? manage_workers+0x180/0x180
[  331.012243]  [<ffffffff81099f9e>] kthread+0xee/0x100
[  331.071546]  [<ffffffff810e1839>] ? __lock_release+0x129/0x190
[  331.141223]  [<ffffffff81099eb0>] ? __init_kthread_worker+0x70/0x70
[  331.216099]  [<ffffffff816b8aec>] ret_from_fork+0x7c/0xb0
[  331.280585]  [<ffffffff81099eb0>] ? __init_kthread_worker+0x70/0x70
[  331.355453] Code: 89 65 f0 4c 89 6d f8 66 66 66 66 90 31 c0 49 89 fc 48 c7 c7 35 ee a3 81 e8 70 83 31 00 49 8b 44 24 10 48 8b 58 38 48 85 db 74 48 <80> 7b 4a 00 74 42 48 83 bb 88 00 00 00 00 74 38 31 c0 48 c7 c7 
[  331.587982] RIP  [<ffffffff813928f8>] pcie_aspm_exit_link_state+0x38/0x190
[  331.670227]  RSP <ffff8807bde17c48>
[  331.711935] ---[ end trace 359d14e0593f23af ]---
[  331.767128] Kernel panic - not syncing: Fatal exception
[  331.829701] ------------[ cut here ]------------
[  331.884839] WARNING: at arch/x86/kernel/smp.c:123 native_smp_send_reschedule+0x5c/0x60()
[  331.981506] Hardware name: PRIMEQUEST 1800E
[  332.031449] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc sunrpc binfmt_misc dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun uinput iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm crc32c_intel microcode pcspkr lpc_ich mfd_core sg i2c_i801 i2c_core ioatdma i7core_edac edac_core e1000e igb dca ptp pps_core sd_mod crc_t10dif megaraid_sas mptsas mptscsih mptbase scsi_transport_sas scsi_mod
[  332.631448] Pid: 6, comm: kworker/u:0 Tainted: G      D W    3.8.0-rc6-aspm-pcie-fix+ #58
[  332.729156] Call Trace:
[  332.758334]  <IRQ>  [<ffffffff8106dd5f>] warn_slowpath_common+0x7f/0xc0
[  332.837472]  [<ffffffff8106ddba>] warn_slowpath_null+0x1a/0x20
[  332.907144]  [<ffffffff8103db0c>] native_smp_send_reschedule+0x5c/0x60
[  332.985129]  [<ffffffff810bc027>] trigger_load_balance+0x357/0x4f0
[  333.058957]  [<ffffffff810aab76>] scheduler_tick+0x116/0x150
[  333.126557]  [<ffffffff8108093e>] update_process_times+0x6e/0x90
[  333.198305]  [<ffffffff810d8359>] tick_sched_handle+0x39/0x80
[  333.266939]  [<ffffffff810d8584>] tick_sched_timer+0x54/0x90
[  333.334541]  [<ffffffff8109f613>] __run_hrtimer+0x83/0x320
[  333.400060]  [<ffffffff810d8530>] ? tick_nohz_handler+0xc0/0xc0
[  333.470773]  [<ffffffff8109fb56>] hrtimer_interrupt+0x106/0x280
[  333.541489]  [<ffffffff810b3fe7>] ? irqtime_account_irq+0xe7/0x100
[  333.615316]  [<ffffffff816ba949>] smp_apic_timer_interrupt+0x69/0x99
[  333.691221]  [<ffffffff816b9872>] apic_timer_interrupt+0x72/0x80
[  333.762968]  <EOI>  [<ffffffff816aab60>] ? panic+0x1a6/0x1ee
[  333.830680]  [<ffffffff816aab5c>] ? panic+0x1a2/0x1ee
[  333.891012]  [<ffffffff81071ca8>] ? kmsg_dump+0x1d8/0x2a0
[  333.955492]  [<ffffffff81071af6>] ? kmsg_dump+0x26/0x2a0
[  334.018937]  [<ffffffff81071c90>] ? kmsg_dump+0x1c0/0x2a0
[  334.083424]  [<ffffffff816b022c>] oops_end+0xdc/0xf0
[  334.142717]  [<ffffffff8101aa8b>] die+0x5b/0x90
[  334.196816]  [<ffffffff816afe0c>] do_general_protection+0xdc/0x160
[  334.270643]  [<ffffffff816af2a3>] ? restore_args+0x30/0x30
[  334.336165]  [<ffffffff816af518>] general_protection+0x28/0x30
[  334.405839]  [<ffffffff813928f8>] ? pcie_aspm_exit_link_state+0x38/0x190
[  334.485897]  [<ffffffff813928ea>] ? pcie_aspm_exit_link_state+0x2a/0x190
[  334.565955]  [<ffffffff81380f42>] pci_stop_dev+0xb2/0xd0
[  334.629398]  [<ffffffff81380fb4>] pci_stop_bus_device+0x54/0x60
[  334.700114]  [<ffffffff81381156>] pci_stop_and_remove_bus_device+0x16/0x30
[  334.782248]  [<ffffffff8138894b>] remove_callback+0x2b/0x40
[  334.848807]  [<ffffffff8125a82a>] sysfs_schedule_callback_work+0x1a/0x80
[  334.928863]  [<ffffffff81091b81>] process_one_work+0x241/0x5f0
[  334.998539]  [<ffffffff81091b0f>] ? process_one_work+0x1cf/0x5f0
[  335.070290]  [<ffffffff8125a810>] ? sysfs_schedule_callback+0x210/0x210
[  335.149311]  [<ffffffff81093d3b>] worker_thread+0x12b/0x3f0
[  335.215870]  [<ffffffff81093c10>] ? manage_workers+0x180/0x180
[  335.285544]  [<ffffffff81099f9e>] kthread+0xee/0x100
[  335.344837]  [<ffffffff810e1839>] ? __lock_release+0x129/0x190
[  335.414511]  [<ffffffff81099eb0>] ? __init_kthread_worker+0x70/0x70
[  335.489379]  [<ffffffff816b8aec>] ret_from_fork+0x7c/0xb0
[  335.553860]  [<ffffffff81099eb0>] ? __init_kthread_worker+0x70/0x70
[  335.628727] ---[ end trace 359d14e0593f23b0 ]---

*test script*
echo -n 1 > /sys/bus/pci/devices/0000\:10\:00.0/remove ; echo -n 1 >  /sys/bus/pci/devices/0000\:1a\:01.0/remove

*pci topology tree*
+-09.0-[10-1e]----00.0-[11-1e]--+-00.0-[12-18]----00.0-[13-18]--+-00.0-[14]--+-00.0
             |                               |                               |            \-00.1
             |                               |                               +-01.0-[15]--+-00.0
             |                               |                               |            \-00.1
             |                               |                               +-02.0-[16]----00.0
             |                               |                               +-03.0-[17]----00.0
             |                               |                               \-04.0-[18]--
             |                               \-01.0-[19-1e]----00.0-[1a-1e]--+-00.0-[1b]--
             |                                                               +-01.0-[1c]--+-00.0
             |                                                               |            \-00.1
             |                                                               +-02.0-[1d]--
             |                                                               \-03.0-[1e]--

$ lspci -vs 10:00.0
10:00.0 PCI bridge: Integrated Device Technology, Inc. Device 807f (rev 02) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=10, secondary=11, subordinate=1e, sec-latency=0
        I/O behind bridge: 00001000-00005fff
        Memory behind bridge: 92a00000-937fffff
        Prefetchable memory behind bridge: 0000000092200000-00000000929fffff
        Capabilities: <access denied>
        Kernel driver in use: pcieport
        Kernel modules: shpchp

$ lspci -vs 1a:01.0
1a:01.0 PCI bridge: Integrated Device Technology, Inc. Device 807f (rev 02) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Bus: primary=1a, secondary=1c, subordinate=1c, sec-latency=0
        I/O behind bridge: 00001000-00001fff
        Memory behind bridge: 92e00000-930fffff
        Prefetchable memory behind bridge: 0000000092600000-00000000927fffff
        Capabilities: <access denied>
        Kernel driver in use: pcieport
        Kernel modules: shpchp
Comment 1 Gu Zheng 2013-02-25 02:35:08 UTC
The pci topology tree seems broken, the 1a:01.0 device is downstream from the 10:00.0 bridge.
Comment 2 Bjorn Helgaas 2013-02-25 21:33:07 UTC
I think this is a general object lifetime issue that really has
nothing to do with ASPM except that ASPM happens to be the victim.

You're doing this:

    echo -n 1 > /sys/bus/pci/devices/0000\:10\:00.0/remove ; echo -n 1
>  /sys/bus/pci/devices/0000\:1a\:01.0/remove

The 1a:01.0 device is downstream from the 10:00.0 bridge.  The sysfs
interface remove_store() uses device_schedule_callback() to schedule
the remove for later.  I think what's happening is that we schedule
remove_callback() for both devices before 10:00.0 has been removed,
like this:

    # echo -n 1 > /sys/bus/pci/devices/0000\:10\:00.0/remove
    remove_store  # for 10:00.0
      device_schedule_callback(10:00.0, remove_callback)
        sysfs_schedule_callback
          kobject_get
          queue_work
    # echo -n 1 >  /sys/bus/pci/devices/0000\:1a\:01.0/remove
    remove_store  # for 1a:01.0
      device_schedule_callback(1a:01.0, remove_callback)
        sysfs_schedule_callback
          kobject_get
          queue_work

Note that we acquire a reference on each pci_dev before queuing the work item.

Later, we run the callbacks, starting with 10:00.0.  This calls
remove_callback() to perform the remove:

    remove_callback(10:00.0)
      mutex_lock(&pci_remove_rescan_mutex)
      pci_stop_and_remove_bus_device(pdev)
      mutex_unlock(&pci_remove_rescan_mutex)

This will stop and remove the subtree below 10:00.0, but it does not
actually free the pci_dev for 1a:01.0 because we increased its ref
count in sysfs_schedule_callback.  So after completing
remove_callback(10:00.0), we run the second callback for 1a:01.0.

The remove for 1a:01.0 calls pcie_aspm_exit_link_state() from
pci_stop_dev().  This is where we blow up because, according to your
debugging, pdev->bus->self is no longer valid.

The PCI core did this removal wrong.  If we have a valid pci_dev
pointer, as we do in pcie_aspm_exit_link_state(), the whole object
ought to be valid.  But the PCI core deallocated the struct pci_bus
for bus 0000:1a too soon.

My guess is that when we build a pci_dev, we need to increase the ref
count on the pci_bus where that pci_dev lives.  That way we can keep
around all the buses and bridges leading from the root to the device
in question.
Comment 3 Bjorn Helgaas 2013-02-25 21:35:33 UTC
Created attachment 94091 [details]
test patch from Yinghai

Please test this patch and confirm whether or not it fixes the crash, Gu.
Comment 4 Gu Zheng 2013-02-26 10:21:02 UTC
Sorry for my mistake, the original was on kernel 3.8.rc6!
We test it on kernel 3.8 release today, and the kernel panics too.
*dmesg*
[  418.775140]  ioatdma i7core_edac edac_core sg e1000e igb dca ptp pps_core sd_mod crc_t10dif megaraid_sas mptsas mptscsih mptbase scsi_transport_sas scsi_mod
[  418.946462] CPU 4 
[  418.968377] Pid: 512, comm: kworker/u:2 Tainted: G        W    3.8.0 #2 FUJITSU-SV PRIMEQUEST 1800E/SB
[  419.081763] RIP: 0010:[<ffffffff8137972e>]  [<ffffffff8137972e>] pci_bus_read_config_word+0x5e/0x90
[  419.189965] RSP: 0018:ffff8807b0a37c08  EFLAGS: 00010046
[  419.253409] RAX: 6b6b6b6b6b6b6b6b RBX: ffff8807bb4a1290 RCX: 0000000000000002
[  419.338658] RDX: 00000000000000c4 RSI: 0000000000000008 RDI: ffff8807bb4a1290
[  419.423925] RBP: ffff8807b0a37c48 R08: ffff8807b0a37c24 R09: 6db5c22da55960d0
[  419.509175] R10: 0000000000000000 R11: 000000000003ecd0 R12: ffff8807b0a37c66
[  419.594425] R13: 0000000000000282 R14: ffffffff82126d40 R15: 0000000000000000
[  419.679675] FS:  0000000000000000(0000) GS:ffff8807c2200000(0000) knlGS:0000000000000000
[  419.776343] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  419.844981] CR2: 00007ffa898a54f8 CR3: 0000000001c0c000 CR4: 00000000000007e0
[  419.930236] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  420.015484] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  420.100736] Process kworker/u:2 (pid: 512, threadinfo ffff8807b0a36000, task ffff8807b30bcd00)
[  420.203632] Stack:
[  420.227623]  ffff8807000000c4 ffffffff00000008 ffffffff813851ef 0000000000992000
[  420.316421]  ffff8807b0a37c98 ffff8807bb49b3d8 0000000000000000 0000000000000000
[  420.405233]  ffff8807b0a37c88 ffffffff8138044b ffff8807b0a37c88 0000000000000246
[  420.494137] Call Trace:
[  420.523326]  [<ffffffff813851ef>] ? remove_callback+0x1f/0x40
[  420.591984]  [<ffffffff8138044b>] pci_pme_active+0x4b/0x1c0
[  420.658545]  [<ffffffff8137d8e7>] pci_stop_bus_device+0x57/0xb0
[  420.729259]  [<ffffffff8137dab6>] pci_stop_and_remove_bus_device+0x16/0x30
[  420.811392]  [<ffffffff813851fb>] remove_callback+0x2b/0x40
[  420.877955]  [<ffffffff81257a56>] sysfs_schedule_callback_work+0x26/0x70
[  420.958017]  [<ffffffff810919ae>] process_one_work+0x20e/0x5c0
[  421.027691]  [<ffffffff8109193f>] ? process_one_work+0x19f/0x5c0
[  421.099441]  [<ffffffff81257a30>] ? sysfs_schedule_callback+0x210/0x210
[  421.178461]  [<ffffffff81093a4e>] worker_thread+0x12e/0x370
[  421.245020]  [<ffffffff81093920>] ? manage_workers+0x180/0x180
[  421.314697]  [<ffffffff81099b8e>] kthread+0xee/0x100
[  421.373992]  [<ffffffff810e0f09>] ? __lock_release+0x129/0x190
[  421.443671]  [<ffffffff81099aa0>] ? __init_kthread_worker+0x70/0x70
[  421.518544]  [<ffffffff816b2dac>] ret_from_fork+0x7c/0xb0
[  421.583031]  [<ffffffff81099aa0>] ? __init_kthread_worker+0x70/0x70
[  421.657894] Code: 89 75 c8 c7 45 dc 00 00 00 00 e8 4e ef 32 00 49 89 c5 48 8b 83 b8 00 00 00 4c 8d 45 dc b9 02 00 00 00 8b 55 c0 8b 75 c8 48 89 df <ff> 10 8b 55 dc 4c 89 ee 48 c7 c7 c0 67 cb 81 89 45 c8 66 41 89 
[  421.890306] RIP  [<ffffffff8137972e>] pci_bus_read_config_word+0x5e/0x90
[  421.970475]  RSP <ffff8807b0a37c08>
[  422.012121] ---[ end trace 403f76cf31f1bcb1 ]---
[  422.067263] Kernel panic - not syncing: Fatal exception
[  422.129761] ------------[ cut here ]------------
[  422.184902] WARNING: at arch/x86/kernel/smp.c:123 native_smp_send_reschedule+0x5c/0x60()
[  422.281566] Hardware name: PRIMEQUEST 1800E
[  422.331508] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc sunrpc binfmt_misc dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun uinput iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm crc32c_intel microcode pcspkr i2c_i801 i2c_core lpc_ich mfd_core ioatdma i7core_edac edac_core sg e1000e igb dca ptp pps_core sd_mod crc_t10dif megaraid_sas mptsas mptscsih mptbase scsi_transport_sas scsi_mod
[  422.931477] Pid: 512, comm: kworker/u:2 Tainted: G      D W    3.8.0 #2
[  423.010495] Call Trace:
[  423.039672]  <IRQ>  [<ffffffff8106dc9f>] warn_slowpath_common+0x7f/0xc0
[  423.118816]  [<ffffffff8106dcfa>] warn_slowpath_null+0x1a/0x20
[  423.188490]  [<ffffffff8103daac>] native_smp_send_reschedule+0x5c/0x60
[  423.266475]  [<ffffffff810bbaf7>] trigger_load_balance+0x357/0x4f0
[  423.340303]  [<ffffffff810aa706>] scheduler_tick+0x116/0x150
[  423.407901]  [<ffffffff8108076e>] update_process_times+0x6e/0x90
[  423.479649]  [<ffffffff810d7b89>] tick_sched_handle+0x39/0x80
[  423.548286]  [<ffffffff810d7db4>] tick_sched_timer+0x54/0x90
[  423.615885]  [<ffffffff8109f203>] __run_hrtimer+0x83/0x320
[  423.681406]  [<ffffffff810d7d60>] ? tick_nohz_handler+0xc0/0xc0
[  423.752119]  [<ffffffff8109f746>] hrtimer_interrupt+0x106/0x280
[  423.822836]  [<ffffffff810b3b47>] ? irqtime_account_irq+0xe7/0x100
[  423.896661]  [<ffffffff816b4c19>] smp_apic_timer_interrupt+0x69/0x99
[  423.972565]  [<ffffffff816b3b32>] apic_timer_interrupt+0x72/0x80
[  424.044313]  <EOI>  [<ffffffff816a95b3>] ? retint_restore_args+0x13/0x13
[  424.124487]  [<ffffffff816a4eef>] ? panic+0x1a6/0x1ee
[  424.184815]  [<ffffffff816a4eeb>] ? panic+0x1a2/0x1ee
[  424.245144]  [<ffffffff81071be8>] ? kmsg_dump+0x1d8/0x2a0
[  424.309628]  [<ffffffff81071a36>] ? kmsg_dump+0x26/0x2a0
[  424.373073]  [<ffffffff81071bd0>] ? kmsg_dump+0x1c0/0x2a0
[  424.437557]  [<ffffffff816aa56c>] oops_end+0xdc/0xf0
[  424.496850]  [<ffffffff8101aa8b>] die+0x5b/0x90
[  424.550949]  [<ffffffff816aa14c>] do_general_protection+0xdc/0x160
[  424.624778]  [<ffffffff816a95e3>] ? restore_args+0x30/0x30
[  424.690298]  [<ffffffff816a9858>] general_protection+0x28/0x30
[  424.759973]  [<ffffffff8137972e>] ? pci_bus_read_config_word+0x5e/0x90
[  424.837956]  [<ffffffff81379712>] ? pci_bus_read_config_word+0x42/0x90
[  424.915935]  [<ffffffff813851ef>] ? remove_callback+0x1f/0x40
[  424.984574]  [<ffffffff8138044b>] pci_pme_active+0x4b/0x1c0
[  425.051134]  [<ffffffff8137d8e7>] pci_stop_bus_device+0x57/0xb0
[  425.121845]  [<ffffffff8137dab6>] pci_stop_and_remove_bus_device+0x16/0x30
[  425.203979]  [<ffffffff813851fb>] remove_callback+0x2b/0x40
[  425.270541]  [<ffffffff81257a56>] sysfs_schedule_callback_work+0x26/0x70
[  425.350598]  [<ffffffff810919ae>] process_one_work+0x20e/0x5c0
[  425.420273]  [<ffffffff8109193f>] ? process_one_work+0x19f/0x5c0
[  425.492023]  [<ffffffff81257a30>] ? sysfs_schedule_callback+0x210/0x210
[  425.571044]  [<ffffffff81093a4e>] worker_thread+0x12e/0x370
[  425.637602]  [<ffffffff81093920>] ? manage_workers+0x180/0x180
[  425.707277]  [<ffffffff81099b8e>] kthread+0xee/0x100
[  425.766571]  [<ffffffff810e0f09>] ? __lock_release+0x129/0x190
[  425.836246]  [<ffffffff81099aa0>] ? __init_kthread_worker+0x70/0x70
[  425.911112]  [<ffffffff816b2dac>] ret_from_fork+0x7c/0xb0
[  425.975594]  [<ffffffff81099aa0>] ? __init_kthread_worker+0x70/0x70
[  426.050459] ---[ end trace 403f76cf31f1bcb2 ]---

I think this because Rafael added pci_pme_active() into pci_stop_dev() in this commit: 
commit 249bfb83cf8ba658955f0245ac3981d941f746ee
Author: Rafael J. Wysocki <rjw@sisk.pl>
Date:   Mon Feb 11 20:49:49 2013 +0100
PCI/PM: Clean up PME state when removing a device

so the panic point moves forward into pci_pme_active()->pci_bus_read_config_word(), the issue is the same.
Comment 5 Gu Zheng 2013-02-26 10:43:29 UTC
(In reply to comment #3)
> Created an attachment (id=94091) [details]
> test patch from Yinghai
> 
> Please test this patch and confirm whether or not it fixes the crash, Gu.

OK, I'll test Yinghai's patch on kernel 3.8, and confirm whether or not it works later.
Comment 6 Gu Zheng 2013-02-27 06:24:26 UTC
(In reply to comment #3)
> Created an attachment (id=94091) [details]
> test patch from Yinghai
> 
> Please test this patch and confirm whether or not it fixes the crash, Gu.
Hi Bjorn,
    I test Yinghai's patch on kernel 3.8 right now, but it does not work. Besides, it leads into other issues.
We get lots of WARNING message on booting, like this:

[   16.421207] ------------[ cut here ]------------
[   16.476356] WARNING: at include/linux/kref.h:42 kobject_get+0x32/0x40()
[   16.555381] Hardware name: PRIMEQUEST 1800E
[   16.605330] Modules linked in:
[   16.641893] Pid: 1, comm: swapper/0 Tainted: G        W    3.8.0+ #6
[   16.717805] Call Trace:
[   16.746995]  [<ffffffff8106dc9f>] warn_slowpath_common+0x7f/0xc0
[   16.818755]  [<ffffffff8106dcfa>] warn_slowpath_null+0x1a/0x20
[   16.888435]  [<ffffffff81355d12>] kobject_get+0x32/0x40
[   16.950847]  [<ffffffff81467339>] get_device+0x19/0x20
[   17.012221]  [<ffffffff8137afcc>] pci_device_add+0xbc/0xd0
[   17.077752]  [<ffffffff81694ea8>] pci_scan_single_device+0xa8/0xc0
[   17.151584]  [<ffffffff8137c754>] pci_scan_slot+0x54/0x160
[   17.217114]  [<ffffffff8137c89f>] pci_scan_child_bus+0x3f/0x170
[   17.287833]  [<ffffffff8137cd32>] pci_scan_bridge+0x362/0x670
[   17.356479]  [<ffffffff8137c11c>] ? pci_read_bridge_bases+0xec/0x1b0
[   17.432386]  [<ffffffff8137c91c>] pci_scan_child_bus+0xbc/0x170
[   17.503109]  [<ffffffff8137cd32>] pci_scan_bridge+0x362/0x670
[   17.571749]  [<ffffffff8137c11c>] ? pci_read_bridge_bases+0xec/0x1b0
[   17.647663]  [<ffffffff8137c91c>] pci_scan_child_bus+0xbc/0x170
[   17.718380]  [<ffffffff8137cd32>] pci_scan_bridge+0x362/0x670
[   17.787026]  [<ffffffff81694e64>] ? pci_scan_single_device+0x64/0xc0
[   17.862933]  [<ffffffff8137d38e>] ? pci_create_root_bus+0x34e/0x410
[   17.937810]  [<ffffffff8137c91c>] pci_scan_child_bus+0xbc/0x170
[   18.008527]  [<ffffffff815560cb>] pci_acpi_scan_root+0x33b/0x390
[   18.080285]  [<ffffffff813c6a79>] acpi_pci_root_add+0x33d/0x448
[   18.151003]  [<ffffffff81259d6d>] ? sysfs_do_create_link+0xed/0x220
[   18.225884]  [<ffffffff810a21b7>] ? __blocking_notifier_call_chain+0xc7/0xd0
[   18.310101]  [<ffffffff813c1542>] acpi_device_probe+0x50/0x18a
[   18.379782]  [<ffffffff81259ed3>] ? sysfs_create_link+0x13/0x20
[   18.450500]  [<ffffffff8146baec>] really_probe+0x6c/0x320
[   18.514992]  [<ffffffff8146bde7>] driver_probe_device+0x47/0xa0
[   18.585708]  [<ffffffff8146beeb>] __driver_attach+0xab/0xb0
[   18.652278]  [<ffffffff8146be40>] ? driver_probe_device+0xa0/0xa0
[   18.725071]  [<ffffffff81469bbc>] bus_for_each_dev+0x6c/0xa0
[   18.792680]  [<ffffffff8146b7ae>] driver_attach+0x1e/0x20
[   18.857166]  [<ffffffff8146b188>] bus_add_driver+0x218/0x2a0
[   18.924774]  [<ffffffff81f46adb>] ? find_dock_and_bay+0x87/0x87
[   18.995490]  [<ffffffff8146c494>] driver_register+0x74/0x160
[   19.063098]  [<ffffffff81f46adb>] ? find_dock_and_bay+0x87/0x87
[   19.133815]  [<ffffffff813c2ac0>] acpi_bus_register_driver+0x43/0x45
[   19.209729]  [<ffffffff81f46afe>] acpi_pci_root_init+0x23/0x32
[   19.279409]  [<ffffffff81002042>] do_one_initcall+0x42/0x180
[   19.347015]  [<ffffffff81f10667>] do_basic_setup+0x9d/0xbb
[   19.412541]  [<ffffffff81f1090a>] ? kernel_init_freeable+0x285/0x285
[   19.488454]  [<ffffffff81f1088f>] kernel_init_freeable+0x20a/0x285
[   19.562287]  [<ffffffff816927c0>] ? rest_init+0x180/0x180
[   19.626779]  [<ffffffff816927ce>] kernel_init+0xe/0xf0
[   19.688150]  [<ffffffff816b2dac>] ret_from_fork+0x7c/0xb0
[   19.752641]  [<ffffffff816927c0>] ? rest_init+0x180/0x180
[   19.817130] ---[ end trace c2a7ddb90bc44488 ]---
[   19.874767] pci 0000:02:00.0: PCI bridge to [bus 03-09]
[   19.937194] pci 0000:02:00.0:   bridge window [io  0x7000-0xafff]
[   19.937202] pci 0000:02:00.0:   bridge window [mem 0x94200000-0x945fffff]
[   19.937352] pci 0000:04:00.0: [111d:8071] type 01 class 0x060400
[   19.937566] pci 0000:04:00.0: PME# supported from D0 D3hot D3cold

The complete dmesg info is reported as an attachment.

And when I just run script:
"echo -n 1 >  /sys/bus/pci/devices/0000\:1a\:01.0/remove"
just to remove device 1a:01.0 only, not parallel removal, the kernel *panics*.
The net console output is below:

[ 1587.937658] general protection fault: 0000 [#1] SMP 
[ 1587.997171] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc sunrpc binfmt_misc dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun uinput iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm crc32c_intel microcode pcspkr sg lpc_ich mfd_core i2c_i801 i2c_core ioatdma i7core_edac edac_core e1000e igb dca ptp pps_core sd_mod crc_t10dif megaraid_sas mptsas mptscsih mptbase scsi_transport_sas scsi_mod
[ 1588.597458] CPU 0 
[ 1588.619375] Pid: 6, comm: kworker/u:0 Tainted: G        W    3.8.0+ #6 FUJITSU-SV PRIMEQUEST 1800E/SB
[ 1588.731726] RIP: 0010:[<ffffffff814679b9>]  [<ffffffff814679b9>] device_get_devnode+0x39/0x130
[ 1588.834742] RSP: 0018:ffff8807bde17b38  EFLAGS: 00010202
[ 1588.898187] RAX: 6b6b6b6b6b6b6b6b RBX: ffff8807bb4a1ce8 RCX: 0000000000000006
[ 1588.983438] RDX: ffff8807bde17b70 RSI: 0000000000000000 RDI: ffff8807bb4a1ce8
[ 1589.068686] RBP: ffff8807bde17b58 R08: 0000000000000001 R09: 0000000000000001
[ 1589.153935] R10: 0000000000000003 R11: 0000000000020840 R12: 0000000000000000
[ 1589.239185] R13: ffff8807bb4a1bd8 R14: ffff8807bde17b70 R15: 0000000000000000
[ 1589.324434] FS:  0000000000000000(0000) GS:ffff8807c1a00000(0000) knlGS:0000000000000000
[ 1589.367161] irq 18: nobody cared (try booting with the "irqpoll" option)
[ 1589.367165] Pid: 0, comm: swapper/13 Tainted: G        W    3.8.0+ #6
[ 1589.367166] Call Trace:
[ 1589.367181]  <IRQ>  [<ffffffff81121e9d>] __report_bad_irq+0x3d/0xe0
[ 1589.367186]  [<ffffffff81122096>] note_interrupt+0x156/0x210
[ 1589.367191]  [<ffffffff8111f70f>] handle_irq_event_percpu+0xdf/0x3a0
[ 1589.367196]  [<ffffffff8111fa18>] handle_irq_event+0x48/0x70
[ 1589.367200]  [<ffffffff811229ae>] ? handle_fasteoi_irq+0x1e/0xf0
[ 1589.367205]  [<ffffffff811229ea>] handle_fasteoi_irq+0x5a/0xf0
[ 1589.367215]  [<ffffffff810196ac>] handle_irq+0x5c/0x150
[ 1589.367224]  [<ffffffff816adc56>] ? atomic_notifier_call_chain+0x16/0x20
[ 1589.367230]  [<ffffffff816b4b2d>] do_IRQ+0x5d/0xe0
[ 1589.367235]  [<ffffffff816a94f2>] common_interrupt+0x72/0x72
[ 1589.367246]  <EOI>  [<ffffffff81525d95>] ? cpuidle_wrap_enter+0x55/0xa0
[ 1589.367250]  [<ffffffff81525d91>] ? cpuidle_wrap_enter+0x51/0xa0
[ 1589.367255]  [<ffffffff81525df0>] cpuidle_enter_tk+0x10/0x20
[ 1589.367259]  [<ffffffff815257d7>] cpuidle_enter_state+0x17/0x50
[ 1589.367263]  [<ffffffff8152614d>] cpuidle_idle_call+0xcd/0x290
[ 1589.367271]  [<ffffffff81020535>] cpu_idle+0xe5/0x140
[ 1589.367275]  [<ffffffff8169db0e>] start_secondary+0xdd/0xdf
[ 1589.367278] handlers:
[ 1589.367285] [<ffffffff8149d8c0>] usb_hcd_irq
[ 1589.367288] [<ffffffff8149d8c0>] usb_hcd_irq
[ 1589.367291] [<ffffffff8149d8c0>] usb_hcd_irq
[ 1589.367305] [<ffffffffa0060c90>] mpt_interrupt [mptbase]
[ 1589.367313] [<ffffffffa0060c90>] mpt_interrupt [mptbase]
[ 1589.367315] Disabling IRQ #18
[ 1591.129933] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1591.198566] CR2: 00000032836aae90 CR3: 0000000001c0c000 CR4: 00000000000007f0
[ 1591.283815] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1591.369066] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 1591.454315] Process kworker/u:0 (pid: 6, threadinfo ffff8807bde16000, task ffff8807bde1a680)
[ 1591.555135] Stack:
[ 1591.579127]  ffff8807bb4a1ce8 ffff8807bb49b470 ffff8807bb4a1bd8 ffff8807bb49b3d8
[ 1591.667932]  ffff8807bde17c18 ffffffff81470a99 ffffffff81691e9d 0000000000000000
[ 1591.756736]  ffff8807bb57be00 ffffffff81467500 ffff8807bde17b98 ffffffff81355cdd
[ 1591.845530] Call Trace:
[ 1591.874719]  [<ffffffff81470a99>] devtmpfs_delete_node+0x69/0x110
[ 1591.947516]  [<ffffffff81691e9d>] ? klist_put+0x5d/0xb0
[ 1592.009925]  [<ffffffff81467500>] ? device_shutdown+0x180/0x180
[ 1592.080649]  [<ffffffff81355cdd>] ? kobject_release+0xd/0x10
[ 1592.148249]  [<ffffffff81355a2c>] ? kobject_put+0x2c/0x60
[ 1592.212734]  [<ffffffff81467377>] ? put_device+0x17/0x20
[ 1592.276183]  [<ffffffff81467512>] ? klist_children_put+0x12/0x20
[ 1592.347936]  [<ffffffff81691ea8>] ? klist_put+0x68/0xb0
[ 1592.410350]  [<ffffffff814683d8>] device_del+0x1a8/0x1e0
[ 1592.473800]  [<ffffffff81468432>] device_unregister+0x22/0x60
[ 1592.542442]  [<ffffffff8137da43>] pci_remove_bus+0x53/0x60
[ 1592.607966]  [<ffffffff8137da9b>] pci_remove_bus_device+0x4b/0x70
[ 1592.680763]  [<ffffffff8137dade>] pci_stop_and_remove_bus_device+0x1e/0x30
[ 1592.762906]  [<ffffffff8138521b>] remove_callback+0x2b/0x40
[ 1592.829473]  [<ffffffff81257a56>] sysfs_schedule_callback_work+0x26/0x70
[ 1592.909535]  [<ffffffff810919ae>] process_one_work+0x20e/0x5c0
[ 1592.979213]  [<ffffffff8109193f>] ? process_one_work+0x19f/0x5c0
[ 1593.050965]  [<ffffffff81257a30>] ? sysfs_schedule_callback+0x210/0x210
[ 1593.129986]  [<ffffffff81093a4e>] worker_thread+0x12e/0x370
[ 1593.196551]  [<ffffffff81093920>] ? manage_workers+0x180/0x180
[ 1593.266229]  [<ffffffff81099b8e>] kthread+0xee/0x100
[ 1593.325527]  [<ffffffff810e0f09>] ? __lock_release+0x129/0x190
[ 1593.395208]  [<ffffffff81099aa0>] ? __init_kthread_worker+0x70/0x70
[ 1593.470080]  [<ffffffff816b2dac>] ret_from_fork+0x7c/0xb0
[ 1593.534570]  [<ffffffff81099aa0>] ? __init_kthread_worker+0x70/0x70
[ 1593.609439] Code: 64 24 08 4c 89 6c 24 10 4c 89 74 24 18 66 66 66 66 90 48 c7 02 00 00 00 00 48 8b 47 58 48 89 fb 49 89 f4 49 89 d6 48 85 c0 74 37 <48> 8b 40 18 48 85 c0 74 2e ff d0 48 85 c0 49 89 c5 49 89 06 74 
[ 1593.841895] RIP  [<ffffffff814679b9>] device_get_devnode+0x39/0x130
[ 1593.916873]  RSP <ffff8807bde17b38>
[ 1593.959187] ---[ end trace c2a7ddb90bc4449f ]---
[ 1594.014488] Kernel panic - not syncing: Fatal exception
[ 1594.077073] ------------[ cut here ]------------
[ 1594.132216] WARNING: at arch/x86/kernel/smp.c:123 native_smp_send_reschedule+0x5c/0x60()
[ 1594.228887] Hardware name: PRIMEQUEST 1800E
[ 1594.278834] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc sunrpc binfmt_misc dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun uinput iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm crc32c_intel microcode pcspkr sg lpc_ich mfd_core i2c_i801 i2c_core ioatdma i7core_edac edac_core e1000e igb dca ptp pps_core sd_mod crc_t10dif megaraid_sas mptsas mptscsih mptbase scsi_transport_sas scsi_mod
[ 1594.879031] Pid: 6, comm: kworker/u:0 Tainted: G      D W    3.8.0+ #6
[ 1594.957011] Call Trace:
[ 1594.986191]  <IRQ>  [<ffffffff8106dc9f>] warn_slowpath_common+0x7f/0xc0
[ 1595.065340]  [<ffffffff8106dcfa>] warn_slowpath_null+0x1a/0x20
[ 1595.135018]  [<ffffffff8103daac>] native_smp_send_reschedule+0x5c/0x60
[ 1595.213003]  [<ffffffff810bbaf7>] trigger_load_balance+0x357/0x4f0
[ 1595.286838]  [<ffffffff810aa706>] scheduler_tick+0x116/0x150
[ 1595.354446]  [<ffffffff8108076e>] update_process_times+0x6e/0x90
[ 1595.426199]  [<ffffffff810d7b89>] tick_sched_handle+0x39/0x80
[ 1595.494840]  [<ffffffff810d7db4>] tick_sched_timer+0x54/0x90
[ 1595.562441]  [<ffffffff8109f203>] __run_hrtimer+0x83/0x320
[ 1595.627966]  [<ffffffff810d7d60>] ? tick_nohz_handler+0xc0/0xc0
[ 1595.698680]  [<ffffffff8109f746>] hrtimer_interrupt+0x106/0x280
[ 1595.769397]  [<ffffffff810b3b47>] ? irqtime_account_irq+0xe7/0x100
[ 1595.843231]  [<ffffffff816b4c19>] smp_apic_timer_interrupt+0x69/0x99
[ 1595.919140]  [<ffffffff816b3b32>] apic_timer_interrupt+0x72/0x80
[ 1595.990892]  <EOI>  [<ffffffff816a4f0f>] ? panic+0x1a6/0x1ee
[ 1596.058619]  [<ffffffff816a4f0b>] ? panic+0x1a2/0x1ee
[ 1596.118957]  [<ffffffff81071be8>] ? kmsg_dump+0x1d8/0x2a0
[ 1596.183446]  [<ffffffff81071a36>] ? kmsg_dump+0x26/0x2a0
[ 1596.246899]  [<ffffffff81071bd0>] ? kmsg_dump+0x1c0/0x2a0
[ 1596.311389]  [<ffffffff816aa56c>] oops_end+0xdc/0xf0
[ 1596.370689]  [<ffffffff8101aa8b>] die+0x5b/0x90
[ 1596.424793]  [<ffffffff816aa14c>] do_general_protection+0xdc/0x160
[ 1596.498628]  [<ffffffff816a95e3>] ? restore_args+0x30/0x30
[ 1596.564156]  [<ffffffff816a9858>] general_protection+0x28/0x30
[ 1596.633837]  [<ffffffff814679b9>] ? device_get_devnode+0x39/0x130
[ 1596.706634]  [<ffffffff81470a99>] devtmpfs_delete_node+0x69/0x110
[ 1596.779430]  [<ffffffff81691e9d>] ? klist_put+0x5d/0xb0
[ 1596.841843]  [<ffffffff81467500>] ? device_shutdown+0x180/0x180
[ 1596.912564]  [<ffffffff81355cdd>] ? kobject_release+0xd/0x10
[ 1596.980166]  [<ffffffff81355a2c>] ? kobject_put+0x2c/0x60
[ 1597.044650]  [<ffffffff81467377>] ? put_device+0x17/0x20
[ 1597.108094]  [<ffffffff81467512>] ? klist_children_put+0x12/0x20
[ 1597.179847]  [<ffffffff81691ea8>] ? klist_put+0x68/0xb0
[ 1597.242261]  [<ffffffff814683d8>] device_del+0x1a8/0x1e0
[ 1597.305711]  [<ffffffff81468432>] device_unregister+0x22/0x60
[ 1597.374350]  [<ffffffff8137da43>] pci_remove_bus+0x53/0x60
[ 1597.439877]  [<ffffffff8137da9b>] pci_remove_bus_device+0x4b/0x70
[ 1597.512674]  [<ffffffff8137dade>] pci_stop_and_remove_bus_device+0x1e/0x30
[ 1597.594814]  [<ffffffff8138521b>] remove_callback+0x2b/0x40
[ 1597.661382]  [<ffffffff81257a56>] sysfs_schedule_callback_work+0x26/0x70
[ 1597.741445]  [<ffffffff810919ae>] process_one_work+0x20e/0x5c0
[ 1597.811125]  [<ffffffff8109193f>] ? process_one_work+0x19f/0x5c0
[ 1597.882877]  [<ffffffff81257a30>] ? sysfs_schedule_callback+0x210/0x210
[ 1597.961898]  [<ffffffff81093a4e>] worker_thread+0x12e/0x370
[ 1598.028463]  [<ffffffff81093920>] ? manage_workers+0x180/0x180
[ 1598.098144]  [<ffffffff81099b8e>] kthread+0xee/0x100
[ 1598.157436]  [<ffffffff810e0f09>] ? __lock_release+0x129/0x190
[ 1598.227119]  [<ffffffff81099aa0>] ? __init_kthread_worker+0x70/0x70
[ 1598.301989]  [<ffffffff816b2dac>] ret_from_fork+0x7c/0xb0
[ 1598.366476]  [<ffffffff81099aa0>] ? __init_kthread_worker+0x70/0x70
[ 1598.441343] ---[ end trace c2a7ddb90bc444a0 ]---
Comment 7 Gu Zheng 2013-02-27 06:25:53 UTC
Created attachment 94161 [details]
booting dmesg with Yinghai's patch on kernel 3.8
Comment 8 Yinghai Lu 2013-02-27 06:45:32 UTC
I think you should try that patch on top current linus's tree, of 2013-02-26
or v3.9-rc1.
Comment 9 Gu Zheng 2013-03-01 03:16:43 UTC
Created attachment 94251 [details]
panic dmesg info with concurrent removal on linus's 0226 tree with Yinghai's patch

commitid d895cb1af15c04c522a25c79cc429076987c089b