Bug 40092 - RCU stall in linux-3.0.0
Summary: RCU stall in linux-3.0.0
Status: RESOLVED OBSOLETE
Alias: None
Product: Process Management
Classification: Unclassified
Component: Scheduler (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Paul E. McKenney
URL:
Keywords:
: 40082 (view as bug list)
Depends on:
Blocks: 36912
  Show dependency tree
 
Reported: 2011-07-25 21:44 UTC by Philip Armstrong
Modified: 2012-08-24 15:44 UTC (History)
7 users (show)

See Also:
Kernel Version: 3.0.0
Tree: Mainline
Regression: Yes


Attachments
dmesg output (82.72 KB, text/plain)
2011-07-25 21:44 UTC, Philip Armstrong
Details
dmesg output with rcu_tree.rcu_cpu_stall_supress=1 (86.16 KB, text/plain)
2011-07-26 08:34 UTC, Philip Armstrong
Details
RCU oops with rcu_tree.rcu_cpu_stall_suppress=1 (74.96 KB, text/plain)
2011-07-26 17:59 UTC, Philip Armstrong
Details
Mutiple RCU oops with CONFIG_RCU_FAST_NO_HZ=y, CONFIG_FRAME_POINTER=y (97.93 KB, text/plain)
2011-07-26 18:32 UTC, Philip Armstrong
Details

Description Philip Armstrong 2011-07-25 21:44:33 UTC
Created attachment 66622 [details]
dmesg output

[1.] One line summary of the problem:
Kernel boot hangs, usually for about a minute, sometimes apparently for good.
[2.] Full description of the problem/report:
Since about 2.6.38, boot times have been getting steadily longer on my machine, with occasionally the boot appearing to hang altogether.

linux-3.0.0 reports an RCU stall, which seems to be associated with the hang in the dmesg, which is attached.

Hopefully "scheduler" is the right place to report this: I'm sure which section RCU bugs fall under in general, but this particular one hits tick_sched_timer in the backtrace so it seems a reasonable first stab. Apologies if I've got it wrong.

[3.] Keywords (i.e., modules, networking, kernel):
RCU, kernel

[4.] Kernel version (from /proc/version):
Linux version 3.0.0-1-amd64 (Debian 3.0.0-1) (ben@decadent.org.uk) (gcc version 4.5.3 (Debian 4.5.3-4) ) #1 SMP Sun Jul 24 02:24:44 UTC 2011

[5.] Output of Oops.. message (if applicable) with symbolic information 
     resolved (see Documentation/oops-tracing.txt)
See dmesg attached.

[6.] A small shell script or example program which triggers the
     problem (if possible)
Happens on boot.

[7.] Environment
[7.1.] Software (add the output of the ver_linux script here)
Linux xanthus 3.0.0-1-amd64 #1 SMP Sun Jul 24 02:24:44 UTC 2011 x86_64 GNU/Linux
 
Gnu C                  4.6.1
Gnu make               3.81
binutils               2.21.53.20110720
1.11
util-linux             2.17.2
mount                  support
module-init-tools      3.12
e2fsprogs              1.41.12
PPP                    2.4.5
Linux C Library        2.13
Dynamic linker (ldd)   2.13
Procps                 3.2.8
Net-tools              1.60
Kbd                    1.15.2
Sh-utils               8.5
Modules Loaded         ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables cpufreq_stats cpufreq_powersave ppdev cpufreq_conservative lp cpufreq_userspace powernow_k8 mperf bridge stp bnep rfcomm bluetooth rfkill binfmt_misc fuse ext2 dm_snapshot it87 hwmon_vid loop firewire_sbp2 kvm_amd kvm snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcm radeon ttm drm_kms_helper snd_seq snd_timer snd_seq_device drm snd edac_core sp5100_tco i2c_algo_bit soundcore tpm_tis tpm i2c_piix4 k10temp i2c_core button processor edac_mce_amd tpm_bios snd_page_alloc pcspkr evdev power_supply parport_pc wmi parport thermal_sys serio_raw ext4 mbcache jbd2 crc16 usbhid hid dm_mirror dm_region_hash dm_log dm_mod sg sr_mod sd_mod cdrom crc_t10dif ata_generic ohci_hcd ahci libahci pata_atiixp firewire_ohci firewire_core crc_itu_t libata ehci_hcd scsi_mod usbcore r8169 mii

[7.2.] Processor information (from /proc/cpuinfo):
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 16
model		: 5
model name	: AMD Athlon(tm) II X4 630 Processor
stepping	: 2
cpu MHz		: 800.000
cache size	: 512 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt npt lbrv svm_lock nrip_save
bogomips	: 5626.04
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate
(+ 3 more cpus the same)

[7.3.] Module information (from /proc/modules):
ip6table_filter 12540 0 - Live 0xffffffffa0555000
ip6_tables 22263 1 ip6table_filter, Live 0xffffffffa054a000
ebtable_nat 12580 0 - Live 0xffffffffa0545000
ebtables 26250 1 ebtable_nat, Live 0xffffffffa0539000
ipt_MASQUERADE 12635 3 - Live 0xffffffffa052f000
iptable_nat 12928 1 - Live 0xffffffffa0534000
nf_nat 18306 2 ipt_MASQUERADE,iptable_nat, Live 0xffffffffa0529000
nf_conntrack_ipv4 18176 4 iptable_nat,nf_nat, Live 0xffffffffa051a000
nf_defrag_ipv4 12483 1 nf_conntrack_ipv4, Live 0xffffffffa0515000
xt_state 12503 1 - Live 0xffffffffa0520000
nf_conntrack 57034 5 ipt_MASQUERADE,iptable_nat,nf_nat,nf_conntrack_ipv4,xt_state, Live 0xffffffffa0506000
ipt_REJECT 12502 2 - Live 0xffffffffa0501000
xt_tcpudp 12570 4 - Live 0xffffffffa04fc000
iptable_filter 12536 1 - Live 0xffffffffa04f7000
ip_tables 22131 2 iptable_nat,iptable_filter, Live 0xffffffffa04e7000
x_tables 19024 10 ip6table_filter,ip6_tables,ebtables,ipt_MASQUERADE,iptable_nat,xt_state,ipt_REJECT,xt_tcpudp,iptable_filter,ip_tables, Live 0xffffffffa04f1000
cpufreq_stats 12862 0 - Live 0xffffffffa04e2000
cpufreq_powersave 12454 0 - Live 0xffffffffa04dd000
ppdev 12763 0 - Live 0xffffffffa04d8000
cpufreq_conservative 13147 0 - Live 0xffffffffa04c8000
lp 17270 0 - Live 0xffffffffa04d2000
cpufreq_userspace 12576 0 - Live 0xffffffffa04cd000
powernow_k8 17688 0 - Live 0xffffffffa03d6000
mperf 12453 1 powernow_k8, Live 0xffffffffa04c3000
bridge 66612 0 - Live 0xffffffffa04b1000
stp 12435 1 bridge, Live 0xffffffffa0395000
bnep 17615 2 - Live 0xffffffffa03d0000
rfcomm 33848 0 - Live 0xffffffffa04a7000
bluetooth 114376 10 bnep,rfcomm, Live 0xffffffffa048a000
rfkill 19080 3 bluetooth, Live 0xffffffffa0385000
binfmt_misc 13040 1 - Live 0xffffffffa0280000
fuse 62334 3 - Live 0xffffffffa0479000
ext2 63732 1 - Live 0xffffffffa0468000
dm_snapshot 32809 3 - Live 0xffffffffa038b000
it87 30739 0 - Live 0xffffffffa037c000
hwmon_vid 12388 1 it87, Live 0xffffffffa029b000
loop 22711 0 - Live 0xffffffffa025a000
firewire_sbp2 18062 0 - Live 0xffffffffa0271000
kvm_amd 51180 0 - Live 0xffffffffa03c2000
kvm 268970 1 kvm_amd, Live 0xffffffffa0425000
snd_hda_codec_hdmi 26321 1 - Live 0xffffffffa0293000
snd_hda_codec_realtek 278027 1 - Live 0xffffffffa03e0000
snd_hda_intel 26140 2 - Live 0xffffffffa0234000
snd_hda_codec 72699 3 snd_hda_codec_hdmi,snd_hda_codec_realtek,snd_hda_intel, Live 0xffffffffa03af000
snd_hwdep 13186 1 snd_hda_codec, Live 0xffffffffa0217000
snd_pcm 68104 3 snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec, Live 0xffffffffa039d000
radeon 648372 2 - Live 0xffffffffa02dc000
ttm 52979 1 radeon, Live 0xffffffffa0285000
drm_kms_helper 27216 1 radeon, Live 0xffffffffa0278000
snd_seq 45208 0 - Live 0xffffffffa0264000
snd_timer 22581 2 snd_pcm,snd_seq, Live 0xffffffffa021c000
snd_seq_device 13137 1 snd_seq, Live 0xffffffffa0207000
drm 163231 4 radeon,ttm,drm_kms_helper, Live 0xffffffffa02b3000
snd 52823 13 snd_hda_codec_hdmi,snd_hda_codec_realtek,snd_hda_intel,snd_hda_codec,snd_hwdep,snd_pcm,snd_seq,snd_timer,snd_seq_device, Live 0xffffffffa024c000
edac_core 35343 0 - Live 0xffffffffa01f4000
sp5100_tco 12900 0 - Live 0xffffffffa01e2000
i2c_algo_bit 12850 1 radeon, Live 0xffffffffa01e8000
soundcore 13152 1 snd, Live 0xffffffffa023e000
tpm_tis 13152 0 - Live 0xffffffffa0247000
tpm 17819 1 tpm_tis, Live 0xffffffffa01bd000
i2c_piix4 12536 0 - Live 0xffffffffa02a8000
k10temp 12611 0 - Live 0xffffffffa02ae000
i2c_core 23909 5 radeon,drm_kms_helper,drm,i2c_algo_bit,i2c_piix4, Live 0xffffffffa02a1000
button 12930 0 - Live 0xffffffffa01ef000
processor 27942 1 powernow_k8, Live 0xffffffffa022c000
edac_mce_amd 17103 0 - Live 0xffffffffa0226000
tpm_bios 12944 1 tpm, Live 0xffffffffa0212000
snd_page_alloc 13043 2 snd_hda_intel,snd_pcm, Live 0xffffffffa01dd000
pcspkr 12579 0 - Live 0xffffffffa01d5000
evdev 17558 7 - Live 0xffffffffa01b7000
power_supply 13475 1 radeon, Live 0xffffffffa01ae000
parport_pc 22395 1 - Live 0xffffffffa01ce000
wmi 13243 0 - Live 0xffffffffa020d000
parport 31929 3 ppdev,lp,parport_pc, Live 0xffffffffa01fe000
thermal_sys 17949 1 processor, Live 0xffffffffa01c4000
serio_raw 12846 0 - Live 0xffffffffa013e000
ext4 313917 2 - Live 0xffffffffa0160000
mbcache 13066 2 ext2,ext4, Live 0xffffffffa00ef000
jbd2 62574 1 ext4, Live 0xffffffffa014f000
crc16 12343 2 bluetooth,ext4, Live 0xffffffffa00d5000
usbhid 40516 0 - Live 0xffffffffa0144000
hid 73172 1 usbhid, Live 0xffffffffa012b000
dm_mirror 17643 0 - Live 0xffffffffa00e9000
dm_region_hash 13506 1 dm_mirror, Live 0xffffffffa00af000
dm_log 13577 2 dm_mirror,dm_region_hash, Live 0xffffffffa005e000
dm_mod 63164 22 dm_snapshot,dm_mirror,dm_log, Live 0xffffffffa011a000
sg 25985 0 - Live 0xffffffffa00e1000
sr_mod 21811 0 - Live 0xffffffffa00da000
sd_mod 36259 4 - Live 0xffffffffa00cb000
cdrom 35093 1 sr_mod, Live 0xffffffffa00bc000
crc_t10dif 12348 1 sd_mod, Live 0xffffffffa00c6000
ata_generic 12479 0 - Live 0xffffffffa0013000
ohci_hcd 26576 0 - Live 0xffffffffa00b4000
ahci 25089 3 - Live 0xffffffffa0070000
libahci 22767 1 ahci, Live 0xffffffffa0065000
pata_atiixp 12697 0 - Live 0xffffffffa004d000
firewire_ohci 35553 0 - Live 0xffffffffa0054000
firewire_core 48371 2 firewire_sbp2,firewire_ohci, Live 0xffffffffa0040000
crc_itu_t 12347 1 firewire_core, Live 0xffffffffa0039000
libata 149043 4 ata_generic,ahci,libahci,pata_atiixp, Live 0xffffffffa00f4000
ehci_hcd 40090 0 - Live 0xffffffffa00a4000
scsi_mod 162442 5 firewire_sbp2,sg,sr_mod,sd_mod,libata, Live 0xffffffffa007b000
usbcore 124242 4 usbhid,ohci_hcd,ehci_hcd, Live 0xffffffffa0019000
r8169 42411 0 - Live 0xffffffffa0007000
mii 12675 1 r8169, Live 0xffffffffa0000000


[7.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem)
$ cat /proc/ioports 
0000-0cf7 : PCI Bus 0000:00
  0000-001f : dma1
  0020-0021 : pic1
  0040-0043 : timer0
  0050-0053 : timer1
  0060-0060 : keyboard
  0064-0064 : keyboard
  0070-0073 : rtc0
  0080-008f : dma page reg
  00a0-00a1 : pic2
  00c0-00df : dma2
  00f0-00ff : fpu
  0170-0177 : 0000:00:14.1
    0170-0177 : pata_atiixp
  01f0-01f7 : 0000:00:14.1
    01f0-01f7 : pata_atiixp
  0220-0225 : pnp 00:01
  0228-022f : pnp 00:02
    022d-022e : it87
      022d-022e : it87
  0238-023f : pnp 00:02
  0290-0294 : pnp 00:01
  0376-0376 : 0000:00:14.1
    0376-0376 : pata_atiixp
  0378-037a : parport0
  03c0-03df : vga+
  03f6-03f6 : 0000:00:14.1
    03f6-03f6 : pata_atiixp
  03f8-03ff : serial
  040b-040b : pnp 00:02
  04d0-04d1 : pnp 00:01
  04d6-04d6 : pnp 00:02
  0b00-0b0f : pnp 00:02
    0b00-0b07 : piix4_smbus
  0b10-0b1f : pnp 00:02
  0b20-0b3f : pnp 00:02
  0c00-0c01 : pnp 00:02
  0c14-0c14 : pnp 00:02
  0c50-0c52 : pnp 00:02
  0c6c-0c6d : pnp 00:02
  0c6f-0c6f : pnp 00:02
  0cd0-0cd1 : pnp 00:02
  0cd2-0cd3 : pnp 00:02
  0cd4-0cdf : pnp 00:02
0cf8-0cff : PCI conf1
0d00-ffff : PCI Bus 0000:00
  4000-40fe : pnp 00:02
    4000-4003 : ACPI PM1a_EVT_BLK
    4004-4005 : ACPI PM1a_CNT_BLK
    4008-400b : ACPI PM_TMR
    4010-4015 : ACPI CPU throttle
    4020-4027 : ACPI GPE0_BLK
    4050-4050 : ACPI PM2_CNT_BLK
  4100-411f : pnp 00:02
  4210-4217 : pnp 00:02
  c000-cfff : PCI Bus 0000:03
  d000-dfff : PCI Bus 0000:02
    de00-deff : 0000:02:00.0
      de00-deff : r8169
  e000-efff : PCI Bus 0000:01
    ee00-eeff : 0000:01:00.0
  fa00-fa0f : 0000:00:14.1
    fa00-fa0f : pata_atiixp
  fb00-fb0f : 0000:00:11.0
    fb00-fb0f : ahci
  fc00-fc03 : 0000:00:11.0
    fc00-fc03 : ahci
  fd00-fd07 : 0000:00:11.0
    fd00-fd07 : ahci
  fe00-fe03 : 0000:00:11.0
    fe00-fe03 : ahci
  ff00-ff07 : 0000:00:11.0
    ff00-ff07 : ahci

$ cat /proc/iomem 
00000000-0000ffff : reserved
00010000-000983ff : System RAM
00098400-0009ffff : reserved
000a0000-000bffff : PCI Bus 0000:00
000c0000-000dffff : PCI Bus 0000:00
  000c0000-000cf7ff : Video ROM
000f0000-000fffff : reserved
  000f0000-000fffff : System ROM
00100000-cfd80fff : System RAM
  01000000-0133f2e2 : Kernel code
  0133f2e3-0168a07f : Kernel data
  0171b000-017f7fff : Kernel bss
  c4000000-c7ffffff : GART
cfd81000-cfddffff : RAM buffer
cfde0000-cfde2fff : ACPI Non-volatile Storage
cfde3000-cfdeffff : ACPI Tables
cfdf0000-cfdfffff : reserved
  cfdf0000-cfdfffff : pnp 00:0b
cfe00000-cfefffff : RAM buffer
cff00000-febfffff : PCI Bus 0000:00
  d0000000-dfffffff : PCI Bus 0000:01
    d0000000-dfffffff : 0000:01:00.0
  e0000000-efffffff : PCI MMCONFIG 0000 [bus 00-ff]
    e0000000-efffffff : reserved
      e0000000-efffffff : pnp 00:0a
  fdb00000-fdbfffff : PCI Bus 0000:02
    fdb00000-fdb0ffff : 0000:02:00.0
    fdbe0000-fdbeffff : 0000:02:00.0
      fdbe0000-fdbeffff : r8169
    fdbff000-fdbfffff : 0000:02:00.0
      fdbff000-fdbfffff : r8169
  fdc00000-fdcfffff : PCI Bus 0000:03
  fdd00000-fddfffff : PCI Bus 0000:03
    fddf8000-fddfbfff : 0000:03:0e.0
    fddff000-fddff7ff : 0000:03:0e.0
      fddff000-fddff7ff : firewire_ohci
  fde00000-fdefffff : PCI Bus 0000:02
  fdf00000-fdffffff : PCI Bus 0000:01
    fdf00000-fdf1ffff : 0000:01:00.0
    fdfc0000-fdfdffff : 0000:01:00.0
    fdffc000-fdffffff : 0000:01:00.1
      fdffc000-fdffffff : ICH HD audio
  fe024000-fe027fff : 0000:00:14.2
    fe024000-fe027fff : ICH HD audio
  fe028000-fe028fff : 0000:00:14.5
    fe028000-fe028fff : ohci_hcd
  fe029000-fe0290ff : 0000:00:13.2
    fe029000-fe0290ff : ehci_hcd
  fe02a000-fe02afff : 0000:00:13.1
    fe02a000-fe02afff : ohci_hcd
  fe02b000-fe02bfff : 0000:00:13.0
    fe02b000-fe02bfff : ohci_hcd
  fe02c000-fe02c0ff : 0000:00:12.2
    fe02c000-fe02c0ff : ehci_hcd
  fe02d000-fe02dfff : 0000:00:12.1
    fe02d000-fe02dfff : ohci_hcd
  fe02e000-fe02efff : 0000:00:12.0
    fe02e000-fe02efff : ohci_hcd
  fe02f000-fe02f3ff : 0000:00:11.0
    fe02f000-fe02f3ff : ahci
fec00000-ffffffff : reserved
  fec00000-fec003ff : IOAPIC 0
  fed00000-fed003ff : HPET 0
  fee00000-fee00fff : Local APIC
    fee00400-fee00fff : pnp 00:02
  fff80000-fffeffff : pnp 00:0b
  ffff0000-ffffffff : pnp 00:0b
100000000-12fffffff : System RAM


[7.5.] PCI information ('lspci -vvv' as root)
root@xanthus:/home/phil# lspci -vvv
00:00.0 Host bridge: Advanced Micro Devices [AMD] RS780 Host Bridge
	Subsystem: Advanced Micro Devices [AMD] RS780 Host Bridge
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
	Latency: 32
	Region 3: Memory at <ignored> (64-bit, non-prefetchable)
	Capabilities: [c4] HyperTransport: Slave or Primary Interface
		Command: BaseUnitID=0 UnitCnt=12 MastHost- DefDir- DUL-
		Link Control 0: CFlE- CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0 IsocEn+ LSEn- ExtCTL- 64b-
		Link Config 0: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn-
		Link Control 1: CFlE- CST- CFE- <LkFail+ Init- EOC+ TXO+ <CRCErr=0 IsocEn- LSEn- ExtCTL- 64b-
		Link Config 1: MLWI=8bit DwFcIn- MLWO=8bit DwFcOut- LWI=8bit DwFcInEn- LWO=8bit DwFcOutEn-
		Revision ID: 3.00
		Link Frequency 0: [b]
		Link Error 0: <Prot+ <Ovfl- <EOC- CTLTm-
		Link Frequency Capability 0: 200MHz+ 300MHz- 400MHz+ 500MHz- 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz- 1.4GHz- 1.6GHz- Vend-
		Feature Capability: IsocFC- LDTSTOP+ CRCTM- ECTLT- 64bA- UIDRD-
		Link Frequency 1: 200MHz
		Link Error 1: <Prot- <Ovfl- <EOC- CTLTm-
		Link Frequency Capability 1: 200MHz- 300MHz- 400MHz- 500MHz- 600MHz- 800MHz- 1.0GHz- 1.2GHz- 1.4GHz- 1.6GHz- Vend-
		Error Handling: PFlE- OFlE- PFE- OFE- EOCFE- RFE- CRCFE- SERRFE- CF- RE- PNFE- ONFE- EOCNFE- RNFE- CRCNFE- SERRNFE-
		Prefetchable memory behind bridge Upper: 00-00
		Bus Number: 00
	Capabilities: [54] HyperTransport: UnitID Clumping
	Capabilities: [40] HyperTransport: Retry Mode
	Capabilities: [9c] HyperTransport: #1a
	Capabilities: [f8] HyperTransport: #1c

00:02.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge (ext gfx port 0) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000e000-0000efff
	Memory behind bridge: fdf00000-fdffffff
	Prefetchable memory behind bridge: 00000000d0000000-00000000dfffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep+ BwNot+
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #2, PowerLimit 75.000W; Interlock- NoCompl+
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis- ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee0f00c  Data: 4141
	Capabilities: [b0] Subsystem: Advanced Micro Devices [AMD] Device 9600
	Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [110 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Kernel driver in use: pcieport

00:0a.0 PCI bridge: Advanced Micro Devices [AMD] RS780 PCI to PCI bridge (PCIE port 5) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000d000-0000dfff
	Memory behind bridge: fde00000-fdefffff
	Prefetchable memory behind bridge: 00000000fdb00000-00000000fdbfffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #10, PowerLimit 25.000W; Interlock- NoCompl+
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet+ LinkState+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis- ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -3.5dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee0f00c  Data: 4149
	Capabilities: [b0] Subsystem: Advanced Micro Devices [AMD] Device 9600
	Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [110 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Kernel driver in use: pcieport

00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode] (prog-if 01 [AHCI 1.0])
	Subsystem: Giga-byte Technology Device b002
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 22
	Region 0: I/O ports at ff00 [size=8]
	Region 1: I/O ports at fe00 [size=4]
	Region 2: I/O ports at fd00 [size=8]
	Region 3: I/O ports at fc00 [size=4]
	Region 4: I/O ports at fb00 [size=16]
	Region 5: Memory at fe02f000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [60] Power Management version 2
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [70] SATA HBA v1.0 InCfgSpace
	Kernel driver in use: ahci

00:12.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 Controller (prog-if 10 [OHCI])
	Subsystem: Giga-byte Technology Device 5004
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at fe02e000 (32-bit, non-prefetchable) [size=4K]
	Kernel driver in use: ohci_hcd

00:12.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller (prog-if 10 [OHCI])
	Subsystem: Giga-byte Technology Device 5004
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at fe02d000 (32-bit, non-prefetchable) [size=4K]
	Kernel driver in use: ohci_hcd

00:12.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller (prog-if 20 [EHCI])
	Subsystem: Giga-byte Technology Device 5004
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 17
	Region 0: Memory at fe02c000 (32-bit, non-prefetchable) [size=256]
	Capabilities: [c0] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
		Bridge: PM- B3+
	Capabilities: [e4] Debug port: BAR=1 offset=00e0
	Kernel driver in use: ehci_hcd

00:13.0 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI0 Controller (prog-if 10 [OHCI])
	Subsystem: Giga-byte Technology Device 5004
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 18
	Region 0: Memory at fe02b000 (32-bit, non-prefetchable) [size=4K]
	Kernel driver in use: ohci_hcd

00:13.1 USB Controller: ATI Technologies Inc SB700 USB OHCI1 Controller (prog-if 10 [OHCI])
	Subsystem: Giga-byte Technology Device 5004
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 18
	Region 0: Memory at fe02a000 (32-bit, non-prefetchable) [size=4K]
	Kernel driver in use: ohci_hcd

00:13.2 USB Controller: ATI Technologies Inc SB700/SB800 USB EHCI Controller (prog-if 20 [EHCI])
	Subsystem: Giga-byte Technology Device 5004
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 19
	Region 0: Memory at fe029000 (32-bit, non-prefetchable) [size=256]
	Capabilities: [c0] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
		Bridge: PM- B3+
	Capabilities: [e4] Debug port: BAR=1 offset=00e0
	Kernel driver in use: ehci_hcd

00:14.0 SMBus: ATI Technologies Inc SBx00 SMBus Controller (rev 3a)
	Subsystem: Giga-byte Technology Device 4385
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [b0] HyperTransport: MSI Mapping Enable- Fixed+
	Kernel driver in use: piix4_smbus

00:14.1 IDE interface: ATI Technologies Inc SB700/SB800 IDE Controller (prog-if 8a [Master SecP PriP])
	Subsystem: Giga-byte Technology Device 5002
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 16
	Region 0: I/O ports at 01f0 [size=8]
	Region 1: I/O ports at 03f4 [size=1]
	Region 2: I/O ports at 0170 [size=8]
	Region 3: I/O ports at 0374 [size=1]
	Region 4: I/O ports at fa00 [size=16]
	Capabilities: [70] MSI: Enable- Count=1/1 Maskable- 64bit-
		Address: 00000000  Data: 0000
	Kernel driver in use: pata_atiixp

00:14.2 Audio device: ATI Technologies Inc SBx00 Azalia (Intel HDA)
	Subsystem: Giga-byte Technology Device a102
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at fe024000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Kernel driver in use: HDA Intel

00:14.3 ISA bridge: ATI Technologies Inc SB700/SB800 LPC host controller
	Subsystem: ATI Technologies Inc SB700/SB800 LPC host controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0

00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (prog-if 01 [Subtractive decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 64
	Bus: primary=00, secondary=03, subordinate=03, sec-latency=64
	I/O behind bridge: 0000c000-0000cfff
	Memory behind bridge: fdd00000-fddfffff
	Prefetchable memory behind bridge: fdc00000-fdcfffff
	Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-

00:14.5 USB Controller: ATI Technologies Inc SB700/SB800 USB OHCI2 Controller (prog-if 10 [OHCI])
	Subsystem: Giga-byte Technology Device 5004
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32, Cache Line Size: 64 bytes
	Interrupt: pin C routed to IRQ 18
	Region 0: Memory at fe028000 (32-bit, non-prefetchable) [size=4K]
	Kernel driver in use: ohci_hcd

00:18.0 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor HyperTransport Configuration
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [80] HyperTransport: Host or Secondary Interface
		Command: WarmRst+ DblEnd- DevNum=0 ChainSide- HostHide+ Slave- <EOCErr- DUL-
		Link Control: CFlE- CST- CFE- <LkFail- Init+ EOC- TXO- <CRCErr=0 IsocEn+ LSEn+ ExtCTL- 64b-
		Link Config: MLWI=16bit DwFcIn- MLWO=16bit DwFcOut- LWI=16bit DwFcInEn- LWO=16bit DwFcOutEn-
		Revision ID: 3.00
		Link Frequency: [b]
		Link Error: <Prot- <Ovfl- <EOC- CTLTm-
		Link Frequency Capability: 200MHz+ 300MHz- 400MHz+ 500MHz- 600MHz+ 800MHz+ 1.0GHz+ 1.2GHz+ 1.4GHz- 1.6GHz- Vend-
		Feature Capability: IsocFC+ LDTSTOP+ CRCTM- ECTLT- 64bA+ UIDRD- ExtRS- UCnfE-

00:18.1 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Address Map
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:18.2 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor DRAM Controller
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:18.3 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Miscellaneous Control
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Capabilities: [f0] Secure device <?>
	Kernel driver in use: k10temp

00:18.4 Host bridge: Advanced Micro Devices [AMD] Family 10h Processor Link Control
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

01:00.0 VGA compatible controller: ATI Technologies Inc Juniper [Radeon HD 5750 Series] (prog-if 00 [VGA controller])
	Subsystem: PC Partner Limited Device e138
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 43
	Region 0: Memory at d0000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at fdfc0000 (64-bit, non-prefetchable) [size=128K]
	Region 4: I/O ports at ee00 [size=256]
	[virtual] Expansion ROM at fdf00000 [disabled] [size=128K]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee0f00c  Data: 4179
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Kernel driver in use: radeon

01:00.1 Audio device: ATI Technologies Inc Juniper HDMI Audio [Radeon HD 5700 Series]
	Subsystem: PC Partner Limited Device aa58
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 44
	Region 0: Memory at fdffc000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee0f00c  Data: 4181
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Kernel driver in use: HDA Intel

02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 02)
	Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 42
	Region 0: I/O ports at de00 [size=256]
	Region 2: Memory at fdbff000 (64-bit, prefetchable) [size=4K]
	Region 4: Memory at fdbe0000 (64-bit, prefetchable) [size=64K]
	[virtual] Expansion ROM at fdb00000 [disabled] [size=64K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/2 Maskable- 64bit+
		Address: 00000000fee0f00c  Data: 4151
	Capabilities: [70] Express (v1) Endpoint, MSI 01
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <8us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 4096 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <64us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [b0] MSI-X: Enable- Count=2 Masked-
		Vector table: BAR=4 offset=00000000
		PBA: BAR=4 offset=00000800
	Capabilities: [d0] Vital Product Data
		Unknown small resource type 00, will not decode more.
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [140 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [160 v1] Device Serial Number 12-34-56-78-12-34-56-78
	Kernel driver in use: r8169

03:0e.0 FireWire (IEEE 1394): Texas Instruments TSB43AB23 IEEE-1394a-2000 Controller (PHY/Link) (prog-if 10 [OHCI])
	Subsystem: Giga-byte Technology GA-EP45-DS5 Motherboard
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 32 (500ns min, 1000ns max), Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 22
	Region 0: Memory at fddff000 (32-bit, non-prefetchable) [size=2K]
	Region 1: Memory at fddf8000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [44] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME+
	Kernel driver in use: firewire_ohci
Comment 1 Philip Armstrong 2011-07-26 08:08:12 UTC
*** Bug 40082 has been marked as a duplicate of this bug. ***
Comment 2 Philip Armstrong 2011-07-26 08:33:33 UTC
> You can suppress the RCU CPU stall checking by adding
> "rcu_tree.rcu_cpu_stall_suppress=1" to the kernel boot parameters.
> If you do this, the stall would likely still be happening, but
> RCU would refrain from reporting it.  Still, this is useful in case
> the problem is in the RCU stall checking itself.

OK, I've tried this & received another oops at boot. (dmesg attached if it makes any difference.

>From the look of the stack traces, I would guess that you have frame
>pointers disabled.  If so, could you please try reproducing this with
>frame pointers enabled?
>Could you please also try rebuilding the kernel with CONFIG_RCU_FAST_NO_HZ=n? 

Will do.
Comment 3 Philip Armstrong 2011-07-26 08:34:21 UTC
Created attachment 66642 [details]
dmesg output with rcu_tree.rcu_cpu_stall_supress=1
Comment 4 Paul E. McKenney 2011-07-26 13:15:40 UTC
On Tue, Jul 26, 2011 at 09:12:39AM +0100, Phil Armstrong wrote:
> On 26/07/11 01:15, Paul E. McKenney wrote:
> >Hello, Phil,
> >
> >Bugzilla seems less than responsive, so I am falling back on email.
> 
> It seems to have recovered again now...

I CCed it so that this message should show up automatically.

> >You can suppress the RCU CPU stall checking by adding
> >"rcu_tree.rcu_cpu_stall_suppress=1" to the kernel boot parameters.
> >If you do this, the stall would likely still be happening, but
> >RCU would refrain from reporting it.  Still, this is useful in case
> >the problem is in the RCU stall checking itself.
> 
> Will try this.
> 
> Looking at a boot log from this morning, I got the delayed boot but
> without the RCU stall check oops, so perhaps there are really two
> problems :(

Or perhaps the stall is happening and delaying your boot whether or not
RCU is complaining about it.

Ah, and notice that CPU 3 is not responding to the NMI
until a full minute after the stall was detected in
https://bugzilla.kernel.org/attachment.cgi?id=66642:

	[    3.976349]  [<ffffffff8132f594>] ? start_secondary+0x1de/0x1e5
	[   67.293217] NMI backtrace for cpu 3
	[   67.293217] CPU 3 

But wait...  We also have the following, earlier in that same
dmesg:

	[   67.293213]  [<ffffffff8100827e>] ? cpu_idle+0xa2/0xc4
	[   67.293213]  [<ffffffff8132f594>] ? start_secondary+0x1de/0x1e5
	[    3.976363] NMI backtrace for cpu 0
	[    3.976363] CPU 0 

Different CPUs seem to have rather different ideas about what time
it is.  Which could cause the system to wait a minute, if one CPU
waited for an absolute time set by the other CPU.  The different
ideas about time can also provoke RCU CPU stalls.

And you got a stall despite rcu_tree.rcu_cpu_stall_suppress=1.
Not sure why that would be.  Could you please double-check this?

If worst comes to worst, I can send you a patch that hard-disables
CPU stall checking.

							Thanx, Paul

> > From the look of the stack traces, I would guess that you have frame
> >pointers disabled.  If so, could you please try reproducing this with
> >frame pointers enabled?
> >Could you please also try rebuilding the kernel with
> CONFIG_RCU_FAST_NO_HZ=n?
> 
> Will do.
> 
> Phil
Comment 5 Philip Armstrong 2011-07-26 14:59:31 UTC
Looks like I mis-spelt "suppress" in the kernel command line for the second dmesg, so scratch the second report. I'll try again tonight with the correct option. Sorry about that.

Having left the machine in question downloading the full git clone I'll try a frame-pointer enabled build.
Comment 6 Philip Armstrong 2011-07-26 16:00:36 UTC
Comment on attachment 66642 [details]
dmesg output with rcu_tree.rcu_cpu_stall_supress=1

Ignore this, the kernel command line is wrong.
Comment 7 Philip Armstrong 2011-07-26 17:57:55 UTC
OK. I think I got the kernel command line correct, but I'm still getting an oops from the RCU code. See the attached dmesg output: syslog-3.0.0-no_rcu_stall2.txt
Comment 8 Philip Armstrong 2011-07-26 17:59:06 UTC
Created attachment 66762 [details]
RCU oops with  rcu_tree.rcu_cpu_stall_suppress=1
Comment 9 Philip Armstrong 2011-07-26 18:32:15 UTC
Created attachment 66782 [details]
Mutiple RCU oops with CONFIG_RCU_FAST_NO_HZ=y, CONFIG_FRAME_POINTER=y
Comment 10 Philip Armstrong 2011-07-26 19:56:34 UTC
Setting CONFIG_RCU_FAST_NO_HZ=n results in a kernel that boots to

[   0.503236] Switched to NOHz mode on CPU#3
[   0.503241] Switched to NOHz mode on CPU#1
[   0.503247] Switched to NOHz mode on CPU#2
[   0.503250] Switched to NOHz mode on CPU#0

and then hangs. After fifteen minutes I poked the keyboard & it promptly jumped ahead again, before hanging for good at:

[  92.899293] NET: Registered protocol family 1

I gave up after another half an hour. Note that the time on the log (92s) bears no relation to wall clock time at all. I'll try this kernel again & then give up on it!
Comment 11 john stultz 2011-07-26 20:05:41 UTC
Does nohz=off change the system behavior?

Also, does tapping frequently on the keyboard (ie: ctrl key) during boot make the issue go away?
Comment 12 Philip Armstrong 2011-07-26 20:10:12 UTC
Tapping on the keyboard does seem to make the boot go quicker. I'll try a few times & see whether it kills the RCU oops: they don't appear every boot.
Comment 13 Philip Armstrong 2011-07-26 20:18:57 UTC
nohz=off eliminates the delays for both CONFIG_RCU_FAST_NO_HZ=y and allows the CONFIG_RCU_FAST_NO_HZ=n kernel to boot successfully.
Comment 14 Philip Armstrong 2011-07-26 20:27:55 UTC
Tapping on the keyboard doesn't seem to help with CONFIG_RCU_FAST_NO_HZ=y. I also have a kernel dmesg from that boot where the timestamps appear to go backwards as the oops is polled from the different CPUs. Odd.
Comment 15 john stultz 2011-07-26 20:34:33 UTC
Ok. If nohz=off resolves it, it sounds like it then a problem with the oneshot clockevent mode. Possibly the device that triggers the timer irq is halting in a low-power mode.

Usually keyboard interrupts force the system out of low-power, allowing the system to continue more normally. That said, it does seem configuration dependent so there may be a few things going on.

Could you also try "nolapic_timer" ?
Comment 16 Philip Armstrong 2011-07-26 20:42:47 UTC
nolapic_timer also fixes both kernels. It also eliminates some pauses (which could be curtailed by a keyboard press) in the shutdown process with showed up even with nohz=off.
Comment 17 john stultz 2011-07-26 21:10:58 UTC
Philip: Thanks for the debugging and very quick feedback! 

Thomas: Any thoughts on how to address this?
Comment 18 Rafael J. Wysocki 2011-08-14 19:16:08 UTC
Thomas?
Comment 19 Rafael J. Wysocki 2011-08-15 17:41:42 UTC
On Monday, August 15, 2011, Paul E. McKenney wrote:
> On Sun, Aug 14, 2011 at 09:02:27PM +0200, Rafael J. Wysocki wrote:
...
> > Bug-Entry   : http://bugzilla.kernel.org/show_bug.cgi?id=40092
> > Subject             : RCU stall in linux-3.0.0
> > Submitter   : Philip Armstrong <phil@kantaka.co.uk>
> > Date                : 2011-07-25 21:44 (21 days old)
> 
> This one was due the the CPU's idea of the current time getting way
> out of sync (as in more than a minute's worth of disagreement, which is
> pretty impressive when you think about it).  So what happened is that
> one CPU decided that the current grace period was a full minute old
> immediately, and thus started screaming piteously.  John Stultz provided
> a configuration workaround, and he and Thomas Gleixner would be working
> out the long-term fix.

Note You need to log in before you can comment on or make changes to this bug.