Created attachment 94021 [details] lspci info When we used a test script: echo -n 1 > /sys/bus/pci/devices/0000\:10\:00.0/remove ; echo -n 1 > /sys/bus/pci/devices/0000\:1a\:01.0/remove to test parallel remove routines triggered by sysfs/pci interface, the kernel panicked. [ 328.037479] general protection fault: 0000 [#1] SMP [ 328.096991] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc sunrpc binfmt_misc dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun uinput iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm crc32c_intel microcode pcspkr lpc_ich mfd_core sg i2c_i801 i2c_core ioatdma i7core_edac edac_core e1000e igb dca ptp pps_core sd_mod crc_t10dif megaraid_sas mptsas mptscsih mptbase scsi_transport_sas scsi_mod [ 328.697122] CPU 6 [ 328.719040] Pid: 6, comm: kworker/u:0 Tainted: G W 3.8.0-rc6-aspm-pcie-fix+ #58 FUJITSU-SV PRIMEQUEST 1800E/SB [ 328.851117] RIP: 0010:[<ffffffff813928f8>] [<ffffffff813928f8>] pcie_aspm_exit_link_state+0x38/0x190 [ 328.961428] RSP: 0018:ffff8807bde17c48 EFLAGS: 00010202 [ 329.024874] RAX: ffff8807bb4a1290 RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000006 [ 329.110125] RDX: 0000000000000006 RSI: ffff8807bde1afc8 RDI: 0000000000000246 [ 329.195371] RBP: ffff8807bde17c68 R08: 0000000000000001 R09: 0000000000000001 [ 329.280619] R10: 0000000000000003 R11: 0000000000000001 R12: ffff8807bb49b3d8 [ 329.365869] R13: ffff8807bb49b3d8 R14: ffffffff82126d80 R15: ffff8807bde17d58 [ 329.451127] FS: 0000000000000000(0000) GS:ffff8807c2600000(0000) knlGS:0000000000000000 [ 329.547796] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 329.616431] CR2: ffffffffff600400 CR3: 0000000001c0c000 CR4: 00000000000007e0 [ 329.701687] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 329.786935] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 329.872185] Process kworker/u:0 (pid: 6, threadinfo ffff8807bde16000, task ffff8807bde1a680) [ 329.973006] Stack: [ 329.997000] 0000000000000006 ffff8807bb49b3d8 0000000000000000 ffff8807bb49b3d8 [ 330.085822] ffff8807bde17c88 ffffffff81380f42 2222222222222222 ffff8807bb49b3d8 [ 330.174627] ffff8807bde17cb8 ffffffff81380fb4 0000000000000000 ffff8807bb49b3d8 [ 330.263427] Call Trace: [ 330.292616] [<ffffffff81380f42>] pci_stop_dev+0xb2/0xd0 [ 330.356064] [<ffffffff81380fb4>] pci_stop_bus_device+0x54/0x60 [ 330.426778] [<ffffffff81381156>] pci_stop_and_remove_bus_device+0x16/0x30 [ 330.508919] [<ffffffff8138894b>] remove_callback+0x2b/0x40 [ 330.575487] [<ffffffff8125a82a>] sysfs_schedule_callback_work+0x1a/0x80 [ 330.655551] [<ffffffff81091b81>] process_one_work+0x241/0x5f0 [ 330.725228] [<ffffffff81091b0f>] ? process_one_work+0x1cf/0x5f0 [ 330.796981] [<ffffffff8125a810>] ? sysfs_schedule_callback+0x210/0x210 [ 330.876002] [<ffffffff81093d3b>] worker_thread+0x12b/0x3f0 [ 330.942567] [<ffffffff81093c10>] ? manage_workers+0x180/0x180 [ 331.012243] [<ffffffff81099f9e>] kthread+0xee/0x100 [ 331.071546] [<ffffffff810e1839>] ? __lock_release+0x129/0x190 [ 331.141223] [<ffffffff81099eb0>] ? __init_kthread_worker+0x70/0x70 [ 331.216099] [<ffffffff816b8aec>] ret_from_fork+0x7c/0xb0 [ 331.280585] [<ffffffff81099eb0>] ? __init_kthread_worker+0x70/0x70 [ 331.355453] Code: 89 65 f0 4c 89 6d f8 66 66 66 66 90 31 c0 49 89 fc 48 c7 c7 35 ee a3 81 e8 70 83 31 00 49 8b 44 24 10 48 8b 58 38 48 85 db 74 48 <80> 7b 4a 00 74 42 48 83 bb 88 00 00 00 00 74 38 31 c0 48 c7 c7 [ 331.587982] RIP [<ffffffff813928f8>] pcie_aspm_exit_link_state+0x38/0x190 [ 331.670227] RSP <ffff8807bde17c48> [ 331.711935] ---[ end trace 359d14e0593f23af ]--- [ 331.767128] Kernel panic - not syncing: Fatal exception [ 331.829701] ------------[ cut here ]------------ [ 331.884839] WARNING: at arch/x86/kernel/smp.c:123 native_smp_send_reschedule+0x5c/0x60() [ 331.981506] Hardware name: PRIMEQUEST 1800E [ 332.031449] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc sunrpc binfmt_misc dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun uinput iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm crc32c_intel microcode pcspkr lpc_ich mfd_core sg i2c_i801 i2c_core ioatdma i7core_edac edac_core e1000e igb dca ptp pps_core sd_mod crc_t10dif megaraid_sas mptsas mptscsih mptbase scsi_transport_sas scsi_mod [ 332.631448] Pid: 6, comm: kworker/u:0 Tainted: G D W 3.8.0-rc6-aspm-pcie-fix+ #58 [ 332.729156] Call Trace: [ 332.758334] <IRQ> [<ffffffff8106dd5f>] warn_slowpath_common+0x7f/0xc0 [ 332.837472] [<ffffffff8106ddba>] warn_slowpath_null+0x1a/0x20 [ 332.907144] [<ffffffff8103db0c>] native_smp_send_reschedule+0x5c/0x60 [ 332.985129] [<ffffffff810bc027>] trigger_load_balance+0x357/0x4f0 [ 333.058957] [<ffffffff810aab76>] scheduler_tick+0x116/0x150 [ 333.126557] [<ffffffff8108093e>] update_process_times+0x6e/0x90 [ 333.198305] [<ffffffff810d8359>] tick_sched_handle+0x39/0x80 [ 333.266939] [<ffffffff810d8584>] tick_sched_timer+0x54/0x90 [ 333.334541] [<ffffffff8109f613>] __run_hrtimer+0x83/0x320 [ 333.400060] [<ffffffff810d8530>] ? tick_nohz_handler+0xc0/0xc0 [ 333.470773] [<ffffffff8109fb56>] hrtimer_interrupt+0x106/0x280 [ 333.541489] [<ffffffff810b3fe7>] ? irqtime_account_irq+0xe7/0x100 [ 333.615316] [<ffffffff816ba949>] smp_apic_timer_interrupt+0x69/0x99 [ 333.691221] [<ffffffff816b9872>] apic_timer_interrupt+0x72/0x80 [ 333.762968] <EOI> [<ffffffff816aab60>] ? panic+0x1a6/0x1ee [ 333.830680] [<ffffffff816aab5c>] ? panic+0x1a2/0x1ee [ 333.891012] [<ffffffff81071ca8>] ? kmsg_dump+0x1d8/0x2a0 [ 333.955492] [<ffffffff81071af6>] ? kmsg_dump+0x26/0x2a0 [ 334.018937] [<ffffffff81071c90>] ? kmsg_dump+0x1c0/0x2a0 [ 334.083424] [<ffffffff816b022c>] oops_end+0xdc/0xf0 [ 334.142717] [<ffffffff8101aa8b>] die+0x5b/0x90 [ 334.196816] [<ffffffff816afe0c>] do_general_protection+0xdc/0x160 [ 334.270643] [<ffffffff816af2a3>] ? restore_args+0x30/0x30 [ 334.336165] [<ffffffff816af518>] general_protection+0x28/0x30 [ 334.405839] [<ffffffff813928f8>] ? pcie_aspm_exit_link_state+0x38/0x190 [ 334.485897] [<ffffffff813928ea>] ? pcie_aspm_exit_link_state+0x2a/0x190 [ 334.565955] [<ffffffff81380f42>] pci_stop_dev+0xb2/0xd0 [ 334.629398] [<ffffffff81380fb4>] pci_stop_bus_device+0x54/0x60 [ 334.700114] [<ffffffff81381156>] pci_stop_and_remove_bus_device+0x16/0x30 [ 334.782248] [<ffffffff8138894b>] remove_callback+0x2b/0x40 [ 334.848807] [<ffffffff8125a82a>] sysfs_schedule_callback_work+0x1a/0x80 [ 334.928863] [<ffffffff81091b81>] process_one_work+0x241/0x5f0 [ 334.998539] [<ffffffff81091b0f>] ? process_one_work+0x1cf/0x5f0 [ 335.070290] [<ffffffff8125a810>] ? sysfs_schedule_callback+0x210/0x210 [ 335.149311] [<ffffffff81093d3b>] worker_thread+0x12b/0x3f0 [ 335.215870] [<ffffffff81093c10>] ? manage_workers+0x180/0x180 [ 335.285544] [<ffffffff81099f9e>] kthread+0xee/0x100 [ 335.344837] [<ffffffff810e1839>] ? __lock_release+0x129/0x190 [ 335.414511] [<ffffffff81099eb0>] ? __init_kthread_worker+0x70/0x70 [ 335.489379] [<ffffffff816b8aec>] ret_from_fork+0x7c/0xb0 [ 335.553860] [<ffffffff81099eb0>] ? __init_kthread_worker+0x70/0x70 [ 335.628727] ---[ end trace 359d14e0593f23b0 ]--- *test script* echo -n 1 > /sys/bus/pci/devices/0000\:10\:00.0/remove ; echo -n 1 > /sys/bus/pci/devices/0000\:1a\:01.0/remove *pci topology tree* +-09.0-[10-1e]----00.0-[11-1e]--+-00.0-[12-18]----00.0-[13-18]--+-00.0-[14]--+-00.0 | | | \-00.1 | | +-01.0-[15]--+-00.0 | | | \-00.1 | | +-02.0-[16]----00.0 | | +-03.0-[17]----00.0 | | \-04.0-[18]-- | \-01.0-[19-1e]----00.0-[1a-1e]--+-00.0-[1b]-- | +-01.0-[1c]--+-00.0 | | \-00.1 | +-02.0-[1d]-- | \-03.0-[1e]-- $ lspci -vs 10:00.0 10:00.0 PCI bridge: Integrated Device Technology, Inc. Device 807f (rev 02) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=10, secondary=11, subordinate=1e, sec-latency=0 I/O behind bridge: 00001000-00005fff Memory behind bridge: 92a00000-937fffff Prefetchable memory behind bridge: 0000000092200000-00000000929fffff Capabilities: <access denied> Kernel driver in use: pcieport Kernel modules: shpchp $ lspci -vs 1a:01.0 1a:01.0 PCI bridge: Integrated Device Technology, Inc. Device 807f (rev 02) (prog-if 00 [Normal decode]) Flags: bus master, fast devsel, latency 0 Bus: primary=1a, secondary=1c, subordinate=1c, sec-latency=0 I/O behind bridge: 00001000-00001fff Memory behind bridge: 92e00000-930fffff Prefetchable memory behind bridge: 0000000092600000-00000000927fffff Capabilities: <access denied> Kernel driver in use: pcieport Kernel modules: shpchp
The pci topology tree seems broken, the 1a:01.0 device is downstream from the 10:00.0 bridge.
I think this is a general object lifetime issue that really has nothing to do with ASPM except that ASPM happens to be the victim. You're doing this: echo -n 1 > /sys/bus/pci/devices/0000\:10\:00.0/remove ; echo -n 1 > /sys/bus/pci/devices/0000\:1a\:01.0/remove The 1a:01.0 device is downstream from the 10:00.0 bridge. The sysfs interface remove_store() uses device_schedule_callback() to schedule the remove for later. I think what's happening is that we schedule remove_callback() for both devices before 10:00.0 has been removed, like this: # echo -n 1 > /sys/bus/pci/devices/0000\:10\:00.0/remove remove_store # for 10:00.0 device_schedule_callback(10:00.0, remove_callback) sysfs_schedule_callback kobject_get queue_work # echo -n 1 > /sys/bus/pci/devices/0000\:1a\:01.0/remove remove_store # for 1a:01.0 device_schedule_callback(1a:01.0, remove_callback) sysfs_schedule_callback kobject_get queue_work Note that we acquire a reference on each pci_dev before queuing the work item. Later, we run the callbacks, starting with 10:00.0. This calls remove_callback() to perform the remove: remove_callback(10:00.0) mutex_lock(&pci_remove_rescan_mutex) pci_stop_and_remove_bus_device(pdev) mutex_unlock(&pci_remove_rescan_mutex) This will stop and remove the subtree below 10:00.0, but it does not actually free the pci_dev for 1a:01.0 because we increased its ref count in sysfs_schedule_callback. So after completing remove_callback(10:00.0), we run the second callback for 1a:01.0. The remove for 1a:01.0 calls pcie_aspm_exit_link_state() from pci_stop_dev(). This is where we blow up because, according to your debugging, pdev->bus->self is no longer valid. The PCI core did this removal wrong. If we have a valid pci_dev pointer, as we do in pcie_aspm_exit_link_state(), the whole object ought to be valid. But the PCI core deallocated the struct pci_bus for bus 0000:1a too soon. My guess is that when we build a pci_dev, we need to increase the ref count on the pci_bus where that pci_dev lives. That way we can keep around all the buses and bridges leading from the root to the device in question.
Created attachment 94091 [details] test patch from Yinghai Please test this patch and confirm whether or not it fixes the crash, Gu.
Sorry for my mistake, the original was on kernel 3.8.rc6! We test it on kernel 3.8 release today, and the kernel panics too. *dmesg* [ 418.775140] ioatdma i7core_edac edac_core sg e1000e igb dca ptp pps_core sd_mod crc_t10dif megaraid_sas mptsas mptscsih mptbase scsi_transport_sas scsi_mod [ 418.946462] CPU 4 [ 418.968377] Pid: 512, comm: kworker/u:2 Tainted: G W 3.8.0 #2 FUJITSU-SV PRIMEQUEST 1800E/SB [ 419.081763] RIP: 0010:[<ffffffff8137972e>] [<ffffffff8137972e>] pci_bus_read_config_word+0x5e/0x90 [ 419.189965] RSP: 0018:ffff8807b0a37c08 EFLAGS: 00010046 [ 419.253409] RAX: 6b6b6b6b6b6b6b6b RBX: ffff8807bb4a1290 RCX: 0000000000000002 [ 419.338658] RDX: 00000000000000c4 RSI: 0000000000000008 RDI: ffff8807bb4a1290 [ 419.423925] RBP: ffff8807b0a37c48 R08: ffff8807b0a37c24 R09: 6db5c22da55960d0 [ 419.509175] R10: 0000000000000000 R11: 000000000003ecd0 R12: ffff8807b0a37c66 [ 419.594425] R13: 0000000000000282 R14: ffffffff82126d40 R15: 0000000000000000 [ 419.679675] FS: 0000000000000000(0000) GS:ffff8807c2200000(0000) knlGS:0000000000000000 [ 419.776343] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 419.844981] CR2: 00007ffa898a54f8 CR3: 0000000001c0c000 CR4: 00000000000007e0 [ 419.930236] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 420.015484] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 420.100736] Process kworker/u:2 (pid: 512, threadinfo ffff8807b0a36000, task ffff8807b30bcd00) [ 420.203632] Stack: [ 420.227623] ffff8807000000c4 ffffffff00000008 ffffffff813851ef 0000000000992000 [ 420.316421] ffff8807b0a37c98 ffff8807bb49b3d8 0000000000000000 0000000000000000 [ 420.405233] ffff8807b0a37c88 ffffffff8138044b ffff8807b0a37c88 0000000000000246 [ 420.494137] Call Trace: [ 420.523326] [<ffffffff813851ef>] ? remove_callback+0x1f/0x40 [ 420.591984] [<ffffffff8138044b>] pci_pme_active+0x4b/0x1c0 [ 420.658545] [<ffffffff8137d8e7>] pci_stop_bus_device+0x57/0xb0 [ 420.729259] [<ffffffff8137dab6>] pci_stop_and_remove_bus_device+0x16/0x30 [ 420.811392] [<ffffffff813851fb>] remove_callback+0x2b/0x40 [ 420.877955] [<ffffffff81257a56>] sysfs_schedule_callback_work+0x26/0x70 [ 420.958017] [<ffffffff810919ae>] process_one_work+0x20e/0x5c0 [ 421.027691] [<ffffffff8109193f>] ? process_one_work+0x19f/0x5c0 [ 421.099441] [<ffffffff81257a30>] ? sysfs_schedule_callback+0x210/0x210 [ 421.178461] [<ffffffff81093a4e>] worker_thread+0x12e/0x370 [ 421.245020] [<ffffffff81093920>] ? manage_workers+0x180/0x180 [ 421.314697] [<ffffffff81099b8e>] kthread+0xee/0x100 [ 421.373992] [<ffffffff810e0f09>] ? __lock_release+0x129/0x190 [ 421.443671] [<ffffffff81099aa0>] ? __init_kthread_worker+0x70/0x70 [ 421.518544] [<ffffffff816b2dac>] ret_from_fork+0x7c/0xb0 [ 421.583031] [<ffffffff81099aa0>] ? __init_kthread_worker+0x70/0x70 [ 421.657894] Code: 89 75 c8 c7 45 dc 00 00 00 00 e8 4e ef 32 00 49 89 c5 48 8b 83 b8 00 00 00 4c 8d 45 dc b9 02 00 00 00 8b 55 c0 8b 75 c8 48 89 df <ff> 10 8b 55 dc 4c 89 ee 48 c7 c7 c0 67 cb 81 89 45 c8 66 41 89 [ 421.890306] RIP [<ffffffff8137972e>] pci_bus_read_config_word+0x5e/0x90 [ 421.970475] RSP <ffff8807b0a37c08> [ 422.012121] ---[ end trace 403f76cf31f1bcb1 ]--- [ 422.067263] Kernel panic - not syncing: Fatal exception [ 422.129761] ------------[ cut here ]------------ [ 422.184902] WARNING: at arch/x86/kernel/smp.c:123 native_smp_send_reschedule+0x5c/0x60() [ 422.281566] Hardware name: PRIMEQUEST 1800E [ 422.331508] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc sunrpc binfmt_misc dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun uinput iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm crc32c_intel microcode pcspkr i2c_i801 i2c_core lpc_ich mfd_core ioatdma i7core_edac edac_core sg e1000e igb dca ptp pps_core sd_mod crc_t10dif megaraid_sas mptsas mptscsih mptbase scsi_transport_sas scsi_mod [ 422.931477] Pid: 512, comm: kworker/u:2 Tainted: G D W 3.8.0 #2 [ 423.010495] Call Trace: [ 423.039672] <IRQ> [<ffffffff8106dc9f>] warn_slowpath_common+0x7f/0xc0 [ 423.118816] [<ffffffff8106dcfa>] warn_slowpath_null+0x1a/0x20 [ 423.188490] [<ffffffff8103daac>] native_smp_send_reschedule+0x5c/0x60 [ 423.266475] [<ffffffff810bbaf7>] trigger_load_balance+0x357/0x4f0 [ 423.340303] [<ffffffff810aa706>] scheduler_tick+0x116/0x150 [ 423.407901] [<ffffffff8108076e>] update_process_times+0x6e/0x90 [ 423.479649] [<ffffffff810d7b89>] tick_sched_handle+0x39/0x80 [ 423.548286] [<ffffffff810d7db4>] tick_sched_timer+0x54/0x90 [ 423.615885] [<ffffffff8109f203>] __run_hrtimer+0x83/0x320 [ 423.681406] [<ffffffff810d7d60>] ? tick_nohz_handler+0xc0/0xc0 [ 423.752119] [<ffffffff8109f746>] hrtimer_interrupt+0x106/0x280 [ 423.822836] [<ffffffff810b3b47>] ? irqtime_account_irq+0xe7/0x100 [ 423.896661] [<ffffffff816b4c19>] smp_apic_timer_interrupt+0x69/0x99 [ 423.972565] [<ffffffff816b3b32>] apic_timer_interrupt+0x72/0x80 [ 424.044313] <EOI> [<ffffffff816a95b3>] ? retint_restore_args+0x13/0x13 [ 424.124487] [<ffffffff816a4eef>] ? panic+0x1a6/0x1ee [ 424.184815] [<ffffffff816a4eeb>] ? panic+0x1a2/0x1ee [ 424.245144] [<ffffffff81071be8>] ? kmsg_dump+0x1d8/0x2a0 [ 424.309628] [<ffffffff81071a36>] ? kmsg_dump+0x26/0x2a0 [ 424.373073] [<ffffffff81071bd0>] ? kmsg_dump+0x1c0/0x2a0 [ 424.437557] [<ffffffff816aa56c>] oops_end+0xdc/0xf0 [ 424.496850] [<ffffffff8101aa8b>] die+0x5b/0x90 [ 424.550949] [<ffffffff816aa14c>] do_general_protection+0xdc/0x160 [ 424.624778] [<ffffffff816a95e3>] ? restore_args+0x30/0x30 [ 424.690298] [<ffffffff816a9858>] general_protection+0x28/0x30 [ 424.759973] [<ffffffff8137972e>] ? pci_bus_read_config_word+0x5e/0x90 [ 424.837956] [<ffffffff81379712>] ? pci_bus_read_config_word+0x42/0x90 [ 424.915935] [<ffffffff813851ef>] ? remove_callback+0x1f/0x40 [ 424.984574] [<ffffffff8138044b>] pci_pme_active+0x4b/0x1c0 [ 425.051134] [<ffffffff8137d8e7>] pci_stop_bus_device+0x57/0xb0 [ 425.121845] [<ffffffff8137dab6>] pci_stop_and_remove_bus_device+0x16/0x30 [ 425.203979] [<ffffffff813851fb>] remove_callback+0x2b/0x40 [ 425.270541] [<ffffffff81257a56>] sysfs_schedule_callback_work+0x26/0x70 [ 425.350598] [<ffffffff810919ae>] process_one_work+0x20e/0x5c0 [ 425.420273] [<ffffffff8109193f>] ? process_one_work+0x19f/0x5c0 [ 425.492023] [<ffffffff81257a30>] ? sysfs_schedule_callback+0x210/0x210 [ 425.571044] [<ffffffff81093a4e>] worker_thread+0x12e/0x370 [ 425.637602] [<ffffffff81093920>] ? manage_workers+0x180/0x180 [ 425.707277] [<ffffffff81099b8e>] kthread+0xee/0x100 [ 425.766571] [<ffffffff810e0f09>] ? __lock_release+0x129/0x190 [ 425.836246] [<ffffffff81099aa0>] ? __init_kthread_worker+0x70/0x70 [ 425.911112] [<ffffffff816b2dac>] ret_from_fork+0x7c/0xb0 [ 425.975594] [<ffffffff81099aa0>] ? __init_kthread_worker+0x70/0x70 [ 426.050459] ---[ end trace 403f76cf31f1bcb2 ]--- I think this because Rafael added pci_pme_active() into pci_stop_dev() in this commit: commit 249bfb83cf8ba658955f0245ac3981d941f746ee Author: Rafael J. Wysocki <rjw@sisk.pl> Date: Mon Feb 11 20:49:49 2013 +0100 PCI/PM: Clean up PME state when removing a device so the panic point moves forward into pci_pme_active()->pci_bus_read_config_word(), the issue is the same.
(In reply to comment #3) > Created an attachment (id=94091) [details] > test patch from Yinghai > > Please test this patch and confirm whether or not it fixes the crash, Gu. OK, I'll test Yinghai's patch on kernel 3.8, and confirm whether or not it works later.
(In reply to comment #3) > Created an attachment (id=94091) [details] > test patch from Yinghai > > Please test this patch and confirm whether or not it fixes the crash, Gu. Hi Bjorn, I test Yinghai's patch on kernel 3.8 right now, but it does not work. Besides, it leads into other issues. We get lots of WARNING message on booting, like this: [ 16.421207] ------------[ cut here ]------------ [ 16.476356] WARNING: at include/linux/kref.h:42 kobject_get+0x32/0x40() [ 16.555381] Hardware name: PRIMEQUEST 1800E [ 16.605330] Modules linked in: [ 16.641893] Pid: 1, comm: swapper/0 Tainted: G W 3.8.0+ #6 [ 16.717805] Call Trace: [ 16.746995] [<ffffffff8106dc9f>] warn_slowpath_common+0x7f/0xc0 [ 16.818755] [<ffffffff8106dcfa>] warn_slowpath_null+0x1a/0x20 [ 16.888435] [<ffffffff81355d12>] kobject_get+0x32/0x40 [ 16.950847] [<ffffffff81467339>] get_device+0x19/0x20 [ 17.012221] [<ffffffff8137afcc>] pci_device_add+0xbc/0xd0 [ 17.077752] [<ffffffff81694ea8>] pci_scan_single_device+0xa8/0xc0 [ 17.151584] [<ffffffff8137c754>] pci_scan_slot+0x54/0x160 [ 17.217114] [<ffffffff8137c89f>] pci_scan_child_bus+0x3f/0x170 [ 17.287833] [<ffffffff8137cd32>] pci_scan_bridge+0x362/0x670 [ 17.356479] [<ffffffff8137c11c>] ? pci_read_bridge_bases+0xec/0x1b0 [ 17.432386] [<ffffffff8137c91c>] pci_scan_child_bus+0xbc/0x170 [ 17.503109] [<ffffffff8137cd32>] pci_scan_bridge+0x362/0x670 [ 17.571749] [<ffffffff8137c11c>] ? pci_read_bridge_bases+0xec/0x1b0 [ 17.647663] [<ffffffff8137c91c>] pci_scan_child_bus+0xbc/0x170 [ 17.718380] [<ffffffff8137cd32>] pci_scan_bridge+0x362/0x670 [ 17.787026] [<ffffffff81694e64>] ? pci_scan_single_device+0x64/0xc0 [ 17.862933] [<ffffffff8137d38e>] ? pci_create_root_bus+0x34e/0x410 [ 17.937810] [<ffffffff8137c91c>] pci_scan_child_bus+0xbc/0x170 [ 18.008527] [<ffffffff815560cb>] pci_acpi_scan_root+0x33b/0x390 [ 18.080285] [<ffffffff813c6a79>] acpi_pci_root_add+0x33d/0x448 [ 18.151003] [<ffffffff81259d6d>] ? sysfs_do_create_link+0xed/0x220 [ 18.225884] [<ffffffff810a21b7>] ? __blocking_notifier_call_chain+0xc7/0xd0 [ 18.310101] [<ffffffff813c1542>] acpi_device_probe+0x50/0x18a [ 18.379782] [<ffffffff81259ed3>] ? sysfs_create_link+0x13/0x20 [ 18.450500] [<ffffffff8146baec>] really_probe+0x6c/0x320 [ 18.514992] [<ffffffff8146bde7>] driver_probe_device+0x47/0xa0 [ 18.585708] [<ffffffff8146beeb>] __driver_attach+0xab/0xb0 [ 18.652278] [<ffffffff8146be40>] ? driver_probe_device+0xa0/0xa0 [ 18.725071] [<ffffffff81469bbc>] bus_for_each_dev+0x6c/0xa0 [ 18.792680] [<ffffffff8146b7ae>] driver_attach+0x1e/0x20 [ 18.857166] [<ffffffff8146b188>] bus_add_driver+0x218/0x2a0 [ 18.924774] [<ffffffff81f46adb>] ? find_dock_and_bay+0x87/0x87 [ 18.995490] [<ffffffff8146c494>] driver_register+0x74/0x160 [ 19.063098] [<ffffffff81f46adb>] ? find_dock_and_bay+0x87/0x87 [ 19.133815] [<ffffffff813c2ac0>] acpi_bus_register_driver+0x43/0x45 [ 19.209729] [<ffffffff81f46afe>] acpi_pci_root_init+0x23/0x32 [ 19.279409] [<ffffffff81002042>] do_one_initcall+0x42/0x180 [ 19.347015] [<ffffffff81f10667>] do_basic_setup+0x9d/0xbb [ 19.412541] [<ffffffff81f1090a>] ? kernel_init_freeable+0x285/0x285 [ 19.488454] [<ffffffff81f1088f>] kernel_init_freeable+0x20a/0x285 [ 19.562287] [<ffffffff816927c0>] ? rest_init+0x180/0x180 [ 19.626779] [<ffffffff816927ce>] kernel_init+0xe/0xf0 [ 19.688150] [<ffffffff816b2dac>] ret_from_fork+0x7c/0xb0 [ 19.752641] [<ffffffff816927c0>] ? rest_init+0x180/0x180 [ 19.817130] ---[ end trace c2a7ddb90bc44488 ]--- [ 19.874767] pci 0000:02:00.0: PCI bridge to [bus 03-09] [ 19.937194] pci 0000:02:00.0: bridge window [io 0x7000-0xafff] [ 19.937202] pci 0000:02:00.0: bridge window [mem 0x94200000-0x945fffff] [ 19.937352] pci 0000:04:00.0: [111d:8071] type 01 class 0x060400 [ 19.937566] pci 0000:04:00.0: PME# supported from D0 D3hot D3cold The complete dmesg info is reported as an attachment. And when I just run script: "echo -n 1 > /sys/bus/pci/devices/0000\:1a\:01.0/remove" just to remove device 1a:01.0 only, not parallel removal, the kernel *panics*. The net console output is below: [ 1587.937658] general protection fault: 0000 [#1] SMP [ 1587.997171] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc sunrpc binfmt_misc dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun uinput iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm crc32c_intel microcode pcspkr sg lpc_ich mfd_core i2c_i801 i2c_core ioatdma i7core_edac edac_core e1000e igb dca ptp pps_core sd_mod crc_t10dif megaraid_sas mptsas mptscsih mptbase scsi_transport_sas scsi_mod [ 1588.597458] CPU 0 [ 1588.619375] Pid: 6, comm: kworker/u:0 Tainted: G W 3.8.0+ #6 FUJITSU-SV PRIMEQUEST 1800E/SB [ 1588.731726] RIP: 0010:[<ffffffff814679b9>] [<ffffffff814679b9>] device_get_devnode+0x39/0x130 [ 1588.834742] RSP: 0018:ffff8807bde17b38 EFLAGS: 00010202 [ 1588.898187] RAX: 6b6b6b6b6b6b6b6b RBX: ffff8807bb4a1ce8 RCX: 0000000000000006 [ 1588.983438] RDX: ffff8807bde17b70 RSI: 0000000000000000 RDI: ffff8807bb4a1ce8 [ 1589.068686] RBP: ffff8807bde17b58 R08: 0000000000000001 R09: 0000000000000001 [ 1589.153935] R10: 0000000000000003 R11: 0000000000020840 R12: 0000000000000000 [ 1589.239185] R13: ffff8807bb4a1bd8 R14: ffff8807bde17b70 R15: 0000000000000000 [ 1589.324434] FS: 0000000000000000(0000) GS:ffff8807c1a00000(0000) knlGS:0000000000000000 [ 1589.367161] irq 18: nobody cared (try booting with the "irqpoll" option) [ 1589.367165] Pid: 0, comm: swapper/13 Tainted: G W 3.8.0+ #6 [ 1589.367166] Call Trace: [ 1589.367181] <IRQ> [<ffffffff81121e9d>] __report_bad_irq+0x3d/0xe0 [ 1589.367186] [<ffffffff81122096>] note_interrupt+0x156/0x210 [ 1589.367191] [<ffffffff8111f70f>] handle_irq_event_percpu+0xdf/0x3a0 [ 1589.367196] [<ffffffff8111fa18>] handle_irq_event+0x48/0x70 [ 1589.367200] [<ffffffff811229ae>] ? handle_fasteoi_irq+0x1e/0xf0 [ 1589.367205] [<ffffffff811229ea>] handle_fasteoi_irq+0x5a/0xf0 [ 1589.367215] [<ffffffff810196ac>] handle_irq+0x5c/0x150 [ 1589.367224] [<ffffffff816adc56>] ? atomic_notifier_call_chain+0x16/0x20 [ 1589.367230] [<ffffffff816b4b2d>] do_IRQ+0x5d/0xe0 [ 1589.367235] [<ffffffff816a94f2>] common_interrupt+0x72/0x72 [ 1589.367246] <EOI> [<ffffffff81525d95>] ? cpuidle_wrap_enter+0x55/0xa0 [ 1589.367250] [<ffffffff81525d91>] ? cpuidle_wrap_enter+0x51/0xa0 [ 1589.367255] [<ffffffff81525df0>] cpuidle_enter_tk+0x10/0x20 [ 1589.367259] [<ffffffff815257d7>] cpuidle_enter_state+0x17/0x50 [ 1589.367263] [<ffffffff8152614d>] cpuidle_idle_call+0xcd/0x290 [ 1589.367271] [<ffffffff81020535>] cpu_idle+0xe5/0x140 [ 1589.367275] [<ffffffff8169db0e>] start_secondary+0xdd/0xdf [ 1589.367278] handlers: [ 1589.367285] [<ffffffff8149d8c0>] usb_hcd_irq [ 1589.367288] [<ffffffff8149d8c0>] usb_hcd_irq [ 1589.367291] [<ffffffff8149d8c0>] usb_hcd_irq [ 1589.367305] [<ffffffffa0060c90>] mpt_interrupt [mptbase] [ 1589.367313] [<ffffffffa0060c90>] mpt_interrupt [mptbase] [ 1589.367315] Disabling IRQ #18 [ 1591.129933] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1591.198566] CR2: 00000032836aae90 CR3: 0000000001c0c000 CR4: 00000000000007f0 [ 1591.283815] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1591.369066] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1591.454315] Process kworker/u:0 (pid: 6, threadinfo ffff8807bde16000, task ffff8807bde1a680) [ 1591.555135] Stack: [ 1591.579127] ffff8807bb4a1ce8 ffff8807bb49b470 ffff8807bb4a1bd8 ffff8807bb49b3d8 [ 1591.667932] ffff8807bde17c18 ffffffff81470a99 ffffffff81691e9d 0000000000000000 [ 1591.756736] ffff8807bb57be00 ffffffff81467500 ffff8807bde17b98 ffffffff81355cdd [ 1591.845530] Call Trace: [ 1591.874719] [<ffffffff81470a99>] devtmpfs_delete_node+0x69/0x110 [ 1591.947516] [<ffffffff81691e9d>] ? klist_put+0x5d/0xb0 [ 1592.009925] [<ffffffff81467500>] ? device_shutdown+0x180/0x180 [ 1592.080649] [<ffffffff81355cdd>] ? kobject_release+0xd/0x10 [ 1592.148249] [<ffffffff81355a2c>] ? kobject_put+0x2c/0x60 [ 1592.212734] [<ffffffff81467377>] ? put_device+0x17/0x20 [ 1592.276183] [<ffffffff81467512>] ? klist_children_put+0x12/0x20 [ 1592.347936] [<ffffffff81691ea8>] ? klist_put+0x68/0xb0 [ 1592.410350] [<ffffffff814683d8>] device_del+0x1a8/0x1e0 [ 1592.473800] [<ffffffff81468432>] device_unregister+0x22/0x60 [ 1592.542442] [<ffffffff8137da43>] pci_remove_bus+0x53/0x60 [ 1592.607966] [<ffffffff8137da9b>] pci_remove_bus_device+0x4b/0x70 [ 1592.680763] [<ffffffff8137dade>] pci_stop_and_remove_bus_device+0x1e/0x30 [ 1592.762906] [<ffffffff8138521b>] remove_callback+0x2b/0x40 [ 1592.829473] [<ffffffff81257a56>] sysfs_schedule_callback_work+0x26/0x70 [ 1592.909535] [<ffffffff810919ae>] process_one_work+0x20e/0x5c0 [ 1592.979213] [<ffffffff8109193f>] ? process_one_work+0x19f/0x5c0 [ 1593.050965] [<ffffffff81257a30>] ? sysfs_schedule_callback+0x210/0x210 [ 1593.129986] [<ffffffff81093a4e>] worker_thread+0x12e/0x370 [ 1593.196551] [<ffffffff81093920>] ? manage_workers+0x180/0x180 [ 1593.266229] [<ffffffff81099b8e>] kthread+0xee/0x100 [ 1593.325527] [<ffffffff810e0f09>] ? __lock_release+0x129/0x190 [ 1593.395208] [<ffffffff81099aa0>] ? __init_kthread_worker+0x70/0x70 [ 1593.470080] [<ffffffff816b2dac>] ret_from_fork+0x7c/0xb0 [ 1593.534570] [<ffffffff81099aa0>] ? __init_kthread_worker+0x70/0x70 [ 1593.609439] Code: 64 24 08 4c 89 6c 24 10 4c 89 74 24 18 66 66 66 66 90 48 c7 02 00 00 00 00 48 8b 47 58 48 89 fb 49 89 f4 49 89 d6 48 85 c0 74 37 <48> 8b 40 18 48 85 c0 74 2e ff d0 48 85 c0 49 89 c5 49 89 06 74 [ 1593.841895] RIP [<ffffffff814679b9>] device_get_devnode+0x39/0x130 [ 1593.916873] RSP <ffff8807bde17b38> [ 1593.959187] ---[ end trace c2a7ddb90bc4449f ]--- [ 1594.014488] Kernel panic - not syncing: Fatal exception [ 1594.077073] ------------[ cut here ]------------ [ 1594.132216] WARNING: at arch/x86/kernel/smp.c:123 native_smp_send_reschedule+0x5c/0x60() [ 1594.228887] Hardware name: PRIMEQUEST 1800E [ 1594.278834] Modules linked in: ip6table_filter ip6_tables ebtable_nat ebtables nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle iptable_filter ip_tables bridge stp llc sunrpc binfmt_misc dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun uinput iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm crc32c_intel microcode pcspkr sg lpc_ich mfd_core i2c_i801 i2c_core ioatdma i7core_edac edac_core e1000e igb dca ptp pps_core sd_mod crc_t10dif megaraid_sas mptsas mptscsih mptbase scsi_transport_sas scsi_mod [ 1594.879031] Pid: 6, comm: kworker/u:0 Tainted: G D W 3.8.0+ #6 [ 1594.957011] Call Trace: [ 1594.986191] <IRQ> [<ffffffff8106dc9f>] warn_slowpath_common+0x7f/0xc0 [ 1595.065340] [<ffffffff8106dcfa>] warn_slowpath_null+0x1a/0x20 [ 1595.135018] [<ffffffff8103daac>] native_smp_send_reschedule+0x5c/0x60 [ 1595.213003] [<ffffffff810bbaf7>] trigger_load_balance+0x357/0x4f0 [ 1595.286838] [<ffffffff810aa706>] scheduler_tick+0x116/0x150 [ 1595.354446] [<ffffffff8108076e>] update_process_times+0x6e/0x90 [ 1595.426199] [<ffffffff810d7b89>] tick_sched_handle+0x39/0x80 [ 1595.494840] [<ffffffff810d7db4>] tick_sched_timer+0x54/0x90 [ 1595.562441] [<ffffffff8109f203>] __run_hrtimer+0x83/0x320 [ 1595.627966] [<ffffffff810d7d60>] ? tick_nohz_handler+0xc0/0xc0 [ 1595.698680] [<ffffffff8109f746>] hrtimer_interrupt+0x106/0x280 [ 1595.769397] [<ffffffff810b3b47>] ? irqtime_account_irq+0xe7/0x100 [ 1595.843231] [<ffffffff816b4c19>] smp_apic_timer_interrupt+0x69/0x99 [ 1595.919140] [<ffffffff816b3b32>] apic_timer_interrupt+0x72/0x80 [ 1595.990892] <EOI> [<ffffffff816a4f0f>] ? panic+0x1a6/0x1ee [ 1596.058619] [<ffffffff816a4f0b>] ? panic+0x1a2/0x1ee [ 1596.118957] [<ffffffff81071be8>] ? kmsg_dump+0x1d8/0x2a0 [ 1596.183446] [<ffffffff81071a36>] ? kmsg_dump+0x26/0x2a0 [ 1596.246899] [<ffffffff81071bd0>] ? kmsg_dump+0x1c0/0x2a0 [ 1596.311389] [<ffffffff816aa56c>] oops_end+0xdc/0xf0 [ 1596.370689] [<ffffffff8101aa8b>] die+0x5b/0x90 [ 1596.424793] [<ffffffff816aa14c>] do_general_protection+0xdc/0x160 [ 1596.498628] [<ffffffff816a95e3>] ? restore_args+0x30/0x30 [ 1596.564156] [<ffffffff816a9858>] general_protection+0x28/0x30 [ 1596.633837] [<ffffffff814679b9>] ? device_get_devnode+0x39/0x130 [ 1596.706634] [<ffffffff81470a99>] devtmpfs_delete_node+0x69/0x110 [ 1596.779430] [<ffffffff81691e9d>] ? klist_put+0x5d/0xb0 [ 1596.841843] [<ffffffff81467500>] ? device_shutdown+0x180/0x180 [ 1596.912564] [<ffffffff81355cdd>] ? kobject_release+0xd/0x10 [ 1596.980166] [<ffffffff81355a2c>] ? kobject_put+0x2c/0x60 [ 1597.044650] [<ffffffff81467377>] ? put_device+0x17/0x20 [ 1597.108094] [<ffffffff81467512>] ? klist_children_put+0x12/0x20 [ 1597.179847] [<ffffffff81691ea8>] ? klist_put+0x68/0xb0 [ 1597.242261] [<ffffffff814683d8>] device_del+0x1a8/0x1e0 [ 1597.305711] [<ffffffff81468432>] device_unregister+0x22/0x60 [ 1597.374350] [<ffffffff8137da43>] pci_remove_bus+0x53/0x60 [ 1597.439877] [<ffffffff8137da9b>] pci_remove_bus_device+0x4b/0x70 [ 1597.512674] [<ffffffff8137dade>] pci_stop_and_remove_bus_device+0x1e/0x30 [ 1597.594814] [<ffffffff8138521b>] remove_callback+0x2b/0x40 [ 1597.661382] [<ffffffff81257a56>] sysfs_schedule_callback_work+0x26/0x70 [ 1597.741445] [<ffffffff810919ae>] process_one_work+0x20e/0x5c0 [ 1597.811125] [<ffffffff8109193f>] ? process_one_work+0x19f/0x5c0 [ 1597.882877] [<ffffffff81257a30>] ? sysfs_schedule_callback+0x210/0x210 [ 1597.961898] [<ffffffff81093a4e>] worker_thread+0x12e/0x370 [ 1598.028463] [<ffffffff81093920>] ? manage_workers+0x180/0x180 [ 1598.098144] [<ffffffff81099b8e>] kthread+0xee/0x100 [ 1598.157436] [<ffffffff810e0f09>] ? __lock_release+0x129/0x190 [ 1598.227119] [<ffffffff81099aa0>] ? __init_kthread_worker+0x70/0x70 [ 1598.301989] [<ffffffff816b2dac>] ret_from_fork+0x7c/0xb0 [ 1598.366476] [<ffffffff81099aa0>] ? __init_kthread_worker+0x70/0x70 [ 1598.441343] ---[ end trace c2a7ddb90bc444a0 ]---
Created attachment 94161 [details] booting dmesg with Yinghai's patch on kernel 3.8
I think you should try that patch on top current linus's tree, of 2013-02-26 or v3.9-rc1.
Created attachment 94251 [details] panic dmesg info with concurrent removal on linus's 0226 tree with Yinghai's patch commitid d895cb1af15c04c522a25c79cc429076987c089b