Bug 12490 - ath5k related kernel panic in 2.6.29-rc1
Summary: ath5k related kernel panic in 2.6.29-rc1
Status: RESOLVED PATCH_ALREADY_AVAILABLE
Alias: None
Product: Drivers
Classification: Unclassified
Component: network-wireless (show other bugs)
Hardware: All Linux
: P1 normal
Assignee: Bob Copeland
URL:
Keywords:
Depends on:
Blocks: 12398
  Show dependency tree
 
Reported: 2009-01-19 08:37 UTC by Rafael J. Wysocki
Modified: 2009-09-03 20:48 UTC (History)
6 users (show)

See Also:
Kernel Version: 2.6.29-rc1
Subsystem:
Regression: Yes
Bisected commit-id:


Attachments
ath5k: gracefully handle rate == NULL (1.83 KB, patch)
2009-01-19 15:05 UTC, Bob Copeland
Details | Diff
ath5k: gracefully handle rate == NULL (1.69 KB, patch)
2009-01-20 13:16 UTC, Bob Copeland
Details | Diff
don't allow STAs with unsupported rates (1.74 KB, patch)
2009-01-25 09:20 UTC, Bob Copeland
Details | Diff
remerge of "don't allow STAs with unsupported rates" against 2.6.29-rc2 (1.71 KB, patch)
2009-01-25 13:50 UTC, Sergey S. Kostyliov
Details | Diff
debug the rate tables (1.50 KB, patch)
2009-01-27 20:26 UTC, Bob Copeland
Details | Diff
more rates debugging (1.33 KB, patch)
2009-02-16 15:14 UTC, Bob Copeland
Details | Diff
full dmesg (24.24 KB, application/octet-stream)
2009-02-22 13:43 UTC, Sergey S. Kostyliov
Details
My config (52.10 KB, application/octet-stream)
2009-02-26 11:03 UTC, Sergey S. Kostyliov
Details
Bob's config (56.49 KB, text/plain)
2009-03-01 07:23 UTC, Bob Copeland
Details

Description Rafael J. Wysocki 2009-01-19 08:37:16 UTC
Subject    : ath5k related kernel panic in 2.6.29-rc1
Submitter  : "Sergey S. Kostyliov" <rathamahata@gmail.com>
Date       : 2009-01-12 7:38
References : http://marc.info/?l=linux-kernel&m=123174591509586&w=4
Handled-By : Bob Copeland <me@bobcopeland.com>

This entry is being used for tracking a regression from 2.6.28.  Please don't
close it until the problem is fixed in the mainline.
Comment 1 Bob Copeland 2009-01-19 15:05:48 UTC
Created attachment 19893 [details]
 ath5k: gracefully handle rate == NULL

Hi, this fixes the BUG at the end, but the preceeding warnings indicate the root cause which is some sort of rate controller bug.  Can you replicate it easily?
Comment 2 Sergey S. Kostyliov 2009-01-20 03:00:55 UTC
I'll check that as soon as I return home from my work - in about 6 hours or so...
Comment 3 Sergey S. Kostyliov 2009-01-20 07:12:24 UTC
Unfortunately the patch doesn't apply neither to 2.6.28-rc{1,2} nor to the current git:


rathamahata@autonomist /usr/local/src/linux-2.6-test $ cat ../ath5k.patch | patch -p1
patching file drivers/net/wireless/ath5k/base.c
Hunk #1 FAILED at 1196.
Hunk #2 succeeded at 1936 (offset -32 lines).
Hunk #3 succeeded at 1969 (offset -32 lines).
1 out of 3 hunks FAILED -- saving rejects to file drivers/net/wireless/ath5k/base.c.rej
rathamahata@autonomist /usr/local/src/linux-2.6-test $ cat drivers/net/wireless/ath5k/base.c.rej
***************
*** 1196,1201 ****
                        PCI_DMA_TODEVICE);

        rate = ieee80211_get_tx_rate(sc->hw, info);

        if (info->flags & IEEE80211_TX_CTL_NO_ACK)
                flags |= AR5K_TXDESC_NOACK;
--- 1196,1205 ----
                        PCI_DMA_TODEVICE);

        rate = ieee80211_get_tx_rate(sc->hw, info);
+       if (!rate) {
+               ret = -EINVAL;
+               goto err_unmap;
+       }

        if (info->flags & IEEE80211_TX_CTL_NO_ACK)
                flags |= AR5K_TXDESC_NOACK;
rathamahata@autonomist /usr/local/src/linux-2.6-test $
Comment 4 Sergey S. Kostyliov 2009-01-20 07:14:33 UTC
> Unfortunately the patch doesn't apply neither to 2.6.28-rc{1,2} nor to the
current git:
Typo. s/2.6.28-rc/2.6.29-rc/
Comment 5 Bob Copeland 2009-01-20 13:11:51 UTC
Sorry, it was against other changes I had in my tree.

Note this will not fix the warnings, just the BUG at the end.
Comment 6 Bob Copeland 2009-01-20 13:16:06 UTC
Created attachment 19911 [details]
ath5k: gracefully handle rate == NULL

Sorry, it was against other changes I had in my tree.  Here's a respin.  Note this will not fix the warnings, just the BUG at the end.  The rest seems to be a rate controller bug, unfortunately I won't be much help for that
Comment 7 Sergey S. Kostyliov 2009-01-21 12:38:01 UTC
> 
> Sorry, it was against other changes I had in my tree.  Here's a respin.  Note
> this will not fix the warnings, just the BUG at the end.  The rest seems to
> be
> a rate controller bug, unfortunately I won't be much help for that
> 

Yes, the last patch fixes indeed the BUG for me. And yes, at least "WARNING: at net/mac80211/tx.c:567" is still here - but  this warning doesn't add any visible effect besides warnings itself for me.

Thank you!
Comment 8 Sergey S. Kostyliov 2009-01-21 13:10:42 UTC
In minutes after I wrote previous comment I had been hit by another rate related BUG_ON():

------------[ cut here ]------------                                       
kernel BUG at net/mac80211/rate.c:239!                                     
invalid opcode: 0000 [#1] PREEMPT SMP                                      
last sysfs file: /sys/devices/LNXSYSTM:00/device:00/PNP0A03:00/device:1c/PNP0C0A:00/power_supply/BAT1/charge_full
CPU 1                                                                                                            
Modules linked in:                                                                                               
Pid: 3548, comm: firefox-bin Not tainted 2.6.29-rc2 #1                                                           
RIP: 0010:[<ffffffff806295e2>]  [<ffffffff806295e2>] rate_control_get_rate+0xb2/0xc0                             
RSP: 0000:ffff88003d88b8d8  EFLAGS: 00010286                                                                     
RAX: 00000000ffffffff RBX: ffff88003c14b630 RCX: ffff88003d88b894                                                
RDX: 0000000000000000 RSI: ffff88003c14b639 RDI: ffff88003d88b894                                                
RBP: ffff88003ccb0680 R08: 0000000000000058 R09: 0000000000000001                                                
R10: ffff88003c09c240 R11: 0000000000000046 R12: ffff88003d94e000                                                
R13: ffff88003d88b9f8 R14: ffff88003ccb9a00 R15: ffff880030c3a098                                                
FS:  00007f0bd2608750(0000) GS:ffff88003d805b00(0000) knlGS:0000000000000000                                     
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033                                                                
CR2: 00007f0bbcb77fa0 CR3: 000000002dcb9000 CR4: 00000000000006e0                                                
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000                                                
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400                                                
Process firefox-bin (pid: 3548, threadinfo ffff880034d80000, task ffff88003cdda160)                              
Stack:                                                                                                           
 ffff88003d88b9f8 ffff88003c14b600 ffff88003c14b630 ffffffff8062f4cd                                             
 7fffffffffffffff ffffffff80852460 ffff88003c14b600 7fffffff0000ffff                                             
 ffffffff80852460 00000042971bc179 ffffffff8025f324 0000000000000286                                             
Call Trace:                                                                                                      
 <IRQ> <0> [<ffffffff8062f4cd>] ? invoke_tx_handlers+0x52d/0xe30                                                 
 [<ffffffff8025f324>] ? tick_handle_oneshot_broadcast+0xc4/0x100                                                 
 [<ffffffff8062ea66>] ? __ieee80211_tx_prepare+0x196/0x310                                                       
 [<ffffffff80593900>] ? pskb_expand_head+0x110/0x180                                                             
 [<ffffffff806310ca>] ? ieee80211_master_start_xmit+0x20a/0x4c0                                                  
 [<ffffffff805ad156>] ? __qdisc_run+0x226/0x270                                                                  
 [<ffffffff8059be70>] ? dev_queue_xmit+0x2a0/0x5b0                                                               
 [<ffffffff80630c63>] ? ieee80211_subif_start_xmit+0x3b3/0x610                                                   
 [<ffffffff80240008>] ? disable_nonboot_cpus+0xf8/0x140                                                          
 [<ffffffff805ad156>] ? __qdisc_run+0x226/0x270                                                                  
 [<ffffffff8059be70>] ? dev_queue_xmit+0x2a0/0x5b0                                                               
 [<ffffffff805bfdae>] ? ip_queue_xmit+0x22e/0x430                                                                
 [<ffffffff802baf63>] ? pollwake+0x43/0x50                                                                       
 [<ffffffff80238aa0>] ? default_wake_function+0x0/0x10                                                           
 [<ffffffff805d363f>] ? tcp_transmit_skb+0x4bf/0x760                                                             
 [<ffffffff805d2683>] ? tcp_rcv_state_process+0xb03/0xc00                                                        
 [<ffffffff805d8e11>] ? tcp_v4_do_rcv+0xb1/0x210                                                                 
 [<ffffffff805d95e2>] ? tcp_v4_rcv+0x672/0x710                                                                   
 [<ffffffff805badc5>] ? ip_local_deliver_finish+0xb5/0x160                                                       
 [<ffffffff805ba811>] ? ip_rcv_finish+0x121/0x340                                                                
 [<ffffffff80212900>] ? nommu_map_single+0x0/0xa0                                                                
 [<ffffffff8059ac1b>] ? netif_receive_skb+0x28b/0x370                                                            
 [<ffffffff8059b260>] ? process_backlog+0x70/0xc0                                                                
 [<ffffffff80599301>] ? net_rx_action+0x121/0x1e0                                                                
 [<ffffffff80244403>] ? __do_softirq+0x73/0x130                                                                  
 [<ffffffff8020caac>] ? call_softirq+0x1c/0x30                                                                   
 [<ffffffff8020e475>] ? do_softirq+0x35/0x80                                                                     
 [<ffffffff802440ad>] ? irq_exit+0x8d/0xb0                                                                       
 [<ffffffff8020e6d3>] ? do_IRQ+0x83/0x110                                                                        
 [<ffffffff8020c353>] ? ret_from_intr+0x0/0xa                                                                    
 <EOI> <0>Code: 08 48 83 c1 03 48 83 f9 0f 75 db 80 7b 08 00 78 1a 48 83 c4 08 5b 5d c3 90 48 8b 47 08 4c 89 c6 48 8b 7f 10 ff 50 40 66 90 eb af <0f> 0b eb fe 66 2e 0f 1f 84 00 00 00 00 00 48 83 c7 18 48 c7 c6                                         
RIP  [<ffffffff806295e2>] rate_control_get_rate+0xb2/0xc0                                                                    
 RSP <ffff88003d88b8d8>                                                                                                      
---[ end trace 28d34917ba21fca1 ]---                                                                                         
Kernel panic - not syncing: Fatal exception in interrupt
Comment 9 Bob Copeland 2009-01-21 13:33:23 UTC
Well, that's unfriendly.  Setting idx to -1 must trigger the BUG later which is a WARN_ON elsewhere.  I suppose it would help to know:

- Rate controller you are using? (=contents of /debug/ieee80211/phy0/rc/name if you have mac80211 debugging enabled)
- Anything special about your setup, like which bands you are using (802.11a/b/g), which ones your AP allows, etc.

Since you seem to be able to easily reproduce it, would you mind testing debug patches?

Also does specifying a fixed rate help? "iwconfig wlan0 rate 1M fixed"
Comment 10 Sergey S. Kostyliov 2009-01-22 11:25:51 UTC
(In reply to comment #9)
> Well, that's unfriendly.  Setting idx to -1 must trigger the BUG later which
> is
> a WARN_ON elsewhere.  I suppose it would help to know:
> 
> - Rate controller you are using? (=contents of /debug/ieee80211/phy0/rc/name
> if

minstrel

> you have mac80211 debugging enabled)
> - Anything special about your setup, like which bands you are using
> (802.11a/b/g), which ones your AP allows, etc.

That is just two ards in Ad-Hoc mode setup. Another one is usb rndis in my desktop:
wlan0     IEEE802.11bg  ESSID:"wahhab"
          Mode:Ad-Hoc  Frequency:2.457 GHz  Cell: F6:1B:A8:DF:5C:81
          Bit Rate=54 Mb/s   Tx-Power=14 dBm
          RTS thr=2347 B   Fragment thr=2346 B
          Link Quality=93/100  Signal level=-36 dBm
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:0  Invalid misc:0   Missed beacon:0

Sorry, I don't know how to get any other information.


> 
> Since you seem to be able to easily reproduce it, would you mind testing
> debug
> patches?

Yes, I would like to test debug patches.

> 
> Also does specifying a fixed rate help? "iwconfig wlan0 rate 1M fixed"
> 

It seems that does help.
Comment 11 Bob Copeland 2009-01-25 09:20:36 UTC
Created attachment 19982 [details]
don't allow STAs with unsupported rates

Okay, please try this patch from Christian Lamparter.
Comment 12 Sergey S. Kostyliov 2009-01-25 13:50:18 UTC
Created attachment 19985 [details]
remerge of  "don't allow STAs with unsupported rates" against 2.6.29-rc2

"don't allow STAs with unsupported rates" (id=19982) patch doesn't apply to 2.6.29-rc2 or current git as is. So I remerged it by hands (that was trivial). Remerged patch is attached - just was the record.

I've just rebooted my notebook with the remerged patch applied.

So far I haven't been able to reproduce any crash. But there are new warnings in my dmesg:
------------[ cut here ]------------                                                                                         
WARNING: at net/mac80211/rc80211_minstrel.c:69 minstrel_tx_status+0x9f/0x120()                                               
Hardware name: Aspire 5100                                                                                                   
Modules linked in:                                                                                                           
Pid: 3433, comm: firefox-bin Not tainted 2.6.29-rc2 #1                                                                       
Call Trace:                                                                                                                  
 <IRQ>  [<ffffffff8023e71a>] warn_slowpath+0xea/0x160                                                                        
 [<ffffffff802239fa>] ack_apic_level+0x1a/0x160                                                                              
 [<ffffffff80273172>] handle_fasteoi_irq+0xa2/0x100                                                                          
 [<ffffffff8024414c>] irq_exit+0x4c/0xb0                                                                                     
 [<ffffffff8020e6d3>] do_IRQ+0x83/0x110                                                                                      
 [<ffffffff802b7b33>] pollwake+0x43/0x50                                                                                     
 [<ffffffff8020c353>] ret_from_intr+0x0/0xa                                                                                  
 [<ffffffff806408c4>] _spin_unlock_irqrestore+0x4/0x30                                                                       
 [<ffffffff805c572e>] tcp_rcv_established+0x7ae/0x920                                                                        
 [<ffffffff805ccdad>] tcp_v4_do_rcv+0xdd/0x210                                                                               
 [<ffffffff8025c398>] getnstimeofday+0x58/0xe0                                                                               
 [<ffffffff80627fbf>] minstrel_tx_status+0x9f/0x120                                                                          
 [<ffffffff80610663>] ieee80211_tx_status+0x103/0x4c0                                                                        
 [<ffffffff804b0392>] ath5k_tasklet_tx+0x202/0x3a0                                                                           
 [<ffffffff80243919>] tasklet_action+0x59/0xd0                                                                               
 [<ffffffff802444e3>] __do_softirq+0x73/0x130                                                                                
 [<ffffffff8020caac>] call_softirq+0x1c/0x30                                                                                 
 [<ffffffff8020e475>] do_softirq+0x35/0x80                                                                                   
 [<ffffffff8024418d>] irq_exit+0x8d/0xb0                                                                                     
 [<ffffffff8020e6d3>] do_IRQ+0x83/0x110                                                                                      
 [<ffffffff8020c353>] ret_from_intr+0x0/0xa                                                                                  
 <EOI> <4>---[ end trace 83ece8f886d377df ]---                                                                               
------------[ cut here ]------------                                                                                         
WARNING: at net/mac80211/rc80211_minstrel.c:69 minstrel_tx_status+0x9f/0x120()                                               
Hardware name: Aspire 5100                                                                                                   
Modules linked in:                                                                                                           
Pid: 3444, comm: firefox-bin Tainted: G        W  2.6.29-rc2 #1                                                              
Call Trace:                                                                                                                  
 <IRQ>  [<ffffffff8023e71a>] warn_slowpath+0xea/0x160                                                                        
 [<ffffffff80624e9d>] ieee80211_subif_start_xmit+0x3bd/0x620                                                                 
 [<ffffffff8025173d>] queue_work_on+0x4d/0x60                                                                                
 [<ffffffff805a0f17>] __qdisc_run+0xd7/0x270                                                                                 
 [<ffffffff803da213>] cpumask_next_and+0x23/0x40                                                                             
 [<ffffffff8023656a>] find_busiest_group+0x28a/0xa00                                                                         
 [<ffffffff804a6256>] ath5k_hw_get_isr+0x266/0x3e0                                                                           
 [<ffffffff804a5c34>] ath5k_hw_is_intr_pending+0x14/0x20                                                                     
 [<ffffffff80627fbf>] minstrel_tx_status+0x9f/0x120                                                                          
 [<ffffffff80610663>] ieee80211_tx_status+0x103/0x4c0                                                                        
 [<ffffffff804b0392>] ath5k_tasklet_tx+0x202/0x3a0                                                                           
 [<ffffffff80243919>] tasklet_action+0x59/0xd0                                                                               
 [<ffffffff802444e3>] __do_softirq+0x73/0x130                                                                                
 [<ffffffff8020caac>] call_softirq+0x1c/0x30                                                                                 
 [<ffffffff8020e475>] do_softirq+0x35/0x80                                                                                   
 [<ffffffff8024418d>] irq_exit+0x8d/0xb0                                                                                     
 [<ffffffff8020e6d3>] do_IRQ+0x83/0x110                                                                                      
 [<ffffffff8020c353>] ret_from_intr+0x0/0xa                                                                                  
 <EOI> <4>---[ end trace 83ece8f886d377e0 ]---                                                                               
------------[ cut here ]------------                                                                                         
WARNING: at net/mac80211/rc80211_minstrel.c:69 minstrel_tx_status+0x9f/0x120()                                               
Hardware name: Aspire 5100                                                                                                   
Modules linked in:                                                                                                           
Pid: 0, comm: swapper Tainted: G        W  2.6.29-rc2 #1
Call Trace:
 <IRQ>  [<ffffffff8023e71a>] warn_slowpath+0xea/0x160
 [<ffffffff80624e9d>] ieee80211_subif_start_xmit+0x3bd/0x620
 [<ffffffff80238aaf>] try_to_wake_up+0xff/0x1a0
 [<ffffffff803da213>] cpumask_next_and+0x23/0x40
 [<ffffffff803da213>] cpumask_next_and+0x23/0x40
 [<ffffffff8025c398>] getnstimeofday+0x58/0xe0
 [<ffffffff8023656a>] find_busiest_group+0x28a/0xa00
 [<ffffffff8025c398>] getnstimeofday+0x58/0xe0
 [<ffffffff80258570>] ktime_get_ts+0x30/0x60
 [<ffffffff802585ac>] ktime_get+0xc/0x50
 [<ffffffff80220805>] lapic_next_event+0x15/0x20
 [<ffffffff8025fc36>] tick_dev_program_event+0x36/0xb0
 [<ffffffff80627fbf>] minstrel_tx_status+0x9f/0x120
 [<ffffffff80610663>] ieee80211_tx_status+0x103/0x4c0
 [<ffffffff8020c4e3>] apic_timer_interrupt+0x13/0x20
 [<ffffffff804b0392>] ath5k_tasklet_tx+0x202/0x3a0
 [<ffffffff80243919>] tasklet_action+0x59/0xd0
 [<ffffffff802444e3>] __do_softirq+0x73/0x130
 [<ffffffff8020caac>] call_softirq+0x1c/0x30
 [<ffffffff8020e475>] do_softirq+0x35/0x80
 [<ffffffff8024418d>] irq_exit+0x8d/0xb0
 [<ffffffff8020e6d3>] do_IRQ+0x83/0x110
 [<ffffffff8020c353>] ret_from_intr+0x0/0xa
 <EOI> <4>---[ end trace 83ece8f886d377e1 ]---
Comment 13 Bob Copeland 2009-01-25 14:29:08 UTC
(In reply to comment #12)
> Created an attachment (id=19985) [details]
> remerge of  "don't allow STAs with unsupported rates" against 2.6.29-rc2
> 
> "don't allow STAs with unsupported rates" (id=19982) patch doesn't apply to
> 2.6.29-rc2 or current git as is. So I remerged it by hands (that was
> trivial).
> Remerged patch is attached - just was the record. 

Hmm oops I rebased the patch on .28.  Anyway, I think it only works for AP mode, not adhoc, so another one should be on the way soon.
Comment 14 Sergey S. Kostyliov 2009-01-25 15:45:31 UTC
(In reply to comment #13)
...
> Hmm oops I rebased the patch on .28.  Anyway, I think it only works for AP
> mode, not adhoc, so another one should be on the way soon.
> 

Yes, it looks like your theory (about adhoc) have just been proved by another oops:

------------[ cut here ]------------                                 
Kernel BUG at ffffffff806235a2 [verbose debug info unavailable]      
invalid opcode: 0000 [#1] PREEMPT SMP                                
last sysfs file: /sys/devices/LNXSYSTM:00/device:00/PNP0A03:00/device:1c/PNP0C0A:00/power_supply/BAT1/charge_full
CPU 0                                                                                                            
Modules linked in:                                                                                               
Pid: 3491, comm: firefox-bin Tainted: G        W  2.6.29-rc2 #2                                                  
RIP: 0010:[<ffffffff806235a2>]  [<ffffffff806235a2>] rate_control_get_rate+0xb2/0xc0                             
RSP: 0018:ffff880031c99908  EFLAGS: 00010282                                                                     
RAX: 00000000ffffffff RBX: ffff880031dd9b40 RCX: ffff880031c998c4                                                
RDX: 0000000000000000 RSI: ffff880031dd9b49 RDI: ffff880031c998c4                                                
RBP: ffff88003cca9680 R08: 0000000000000058 R09: 0000000000000001                                                
R10: ffff88003c083ea0 R11: 0000000000000025 R12: ffff88003b422000                                                
R13: ffff880031c99a28 R14: ffff88003cc29a00 R15: ffff88002e5eec84                                                
FS:  00007ff1152ff950(0000) GS:ffffffff80852040(0000) knlGS:0000000000000000                                     
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b                                                                
CR2: 00007ff10ba35c58 CR3: 00000000348a8000 CR4: 00000000000006e0                                                
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000                                                
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400                                                
Process firefox-bin (pid: 3491, threadinfo ffff880031c98000, task ffff88003cd010b0)                              
Stack:                                                                                                           
 ffff880031c99a28 ffff880031dd9b10 ffff880031dd9b40 ffffffff806295ed                                             
 0000000000000000 ffff88003cc2cb90 ffff880031dd9b10 ffffffff00005aa1                                             
 ffff880031dd8480 ffff88003ccbab60 ffff88003cc2cb98 0000000000000282                                             
Call Trace:                                                                                                      
 [<ffffffff806295ed>] ? invoke_tx_handlers+0x52d/0xe30                                                           
 [<ffffffff80628b76>] ? __ieee80211_tx_prepare+0x196/0x320                                                       
 [<ffffffff8058d6c0>] ? pskb_expand_head+0x110/0x180                                                             
 [<ffffffff8062b1fa>] ? ieee80211_master_start_xmit+0x20a/0x4d0                                                  
 [<ffffffff805a6f36>] ? __qdisc_run+0x226/0x270                                                                  
 [<ffffffff80595c50>] ? dev_queue_xmit+0x2a0/0x5b0                                                               
 [<ffffffff8062ad8d>] ? ieee80211_subif_start_xmit+0x3bd/0x620                                                   
 [<ffffffff802b6992>] ? do_sys_poll+0x372/0x430                                                                  
 [<ffffffff805a6f36>] ? __qdisc_run+0x226/0x270                                                                  
 [<ffffffff80595c50>] ? dev_queue_xmit+0x2a0/0x5b0                                                               
 [<ffffffff805b9b8e>] ? ip_queue_xmit+0x22e/0x430                                                                
 [<ffffffff802a3655>] ? kmem_cache_alloc+0x65/0xa0                                                               
 [<ffffffff805b1921>] ? __ip_route_output_key+0x191/0xb00                                                        
 [<ffffffff805cd47f>] ? tcp_transmit_skb+0x4bf/0x760                                                             
 [<ffffffff805cdfa4>] ? tcp_connect+0x364/0x440                                                                  
 [<ffffffff805d440c>] ? tcp_v4_connect+0x41c/0x4f0                                                               
 [<ffffffff805e10c8>] ? inet_stream_connect+0x228/0x2d0                                                          
 [<ffffffff802a3655>] ? kmem_cache_alloc+0x65/0xa0                                                               
 [<ffffffff802a825e>] ? fget_light+0xce/0xe0                                                                     
 [<ffffffff80585616>] ? sys_connect+0x86/0xe0                                                                    
 [<ffffffff802a4fff>] ? fd_install+0x2f/0x60                                                                     
 [<ffffffff80585a34>] ? sock_map_fd+0x54/0x90                                                                    
 [<ffffffff802a82ee>] ? fget+0x7e/0xa0                                                                           
 [<ffffffff8020ba1b>] ? system_call_fastpath+0x16/0x1b                                                           
Code: 08 48 83 c1 03 48 83 f9 0f 75 db 80 7b 08 00 78 1a 48 83 c4 08 5b 5d c3 90 48 8b 47 08 4c 89 c6 48 8b 7f 10 ff 50 40 66 90 eb af <0f> 0b 0f 1f 40 00 eb fa 66 0f 1f 44 00 00 48 83 c7 18 48 c7 c6                                                   
RIP  [<ffffffff806235a2>] rate_control_get_rate+0xb2/0xc0                                                                    
 RSP <ffff880031c99908>                                                                                                      
---[ end trace f15625ddd27732c4 ]---                                                                                         
Kernel panic - not syncing: Fatal exception in interrupt
Comment 15 Bob Copeland 2009-01-27 20:26:14 UTC
Created attachment 20022 [details]
debug the rate tables

Ok, not 100% sure the AP case is similar -- since ibss always adds the default rates.  Here's some debugging statements to help pin it down - it should dump hopefully useful info when the warn_on triggers.
Comment 16 Rafael J. Wysocki 2009-02-03 15:05:04 UTC
On Tuesday 20 January 2009, Bob Copeland wrote:
> On Mon, Jan 19, 2009 at 10:32:14PM +0100, Rafael J. Wysocki wrote:
> > This message has been generated automatically as a part of a report
> > of recent regressions.
> > 
> > The following bug entry is on the current list of known regressions
> > from 2.6.28.  Please verify if it still should be listed and let me know
> > (either way).
> 
> Yeah I think it should, though it seems to be a rate controller bug and
> not specific to ath5k.
Comment 17 Sergey S. Kostyliov 2009-02-03 23:47:34 UTC
Below is another oops from kernel with all patches applied:

------------[ cut here ]------------                                                                                          
Kernel BUG at ffffffff80622db2 [verbose debug info unavailable]                                                               
invalid opcode: 0000 [#1] PREEMPT SMP                                                                                         
last sysfs file: /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors                                             
CPU 1                                                                                                                         
Modules linked in:                                                                                                            
Pid: 3540, comm: pdnsd Not tainted 2.6.29-rc2 #3                                                                              
RIP: 0010:[<ffffffff80622db2>]  [<ffffffff80622db2>] rate_control_get_rate+0xb2/0xc0                                          
RSP: 0018:ffff880024067858  EFLAGS: 00010286                                                                                  
RAX: 00000000ffffffff RBX: ffff88003c8fe730 RCX: ffff880024067814                                                             
RDX: 0000000000000000 RSI: ffff88003c8fe739 RDI: ffff880024067814                                                             
RBP: ffff88003d948680 R08: 0000000000000058 R09: 0000000000000001                                                             
R10: ffff88003bd1fde0 R11: 0000000000000078 R12: ffff88003c8fe700                                                             
R13: ffff880024067978 R14: ffff88003c8fe730 R15: ffff88003ccf9a00                                                             
FS:  00007f2955423950(0000) GS:ffff88003d805b00(0000) knlGS:0000000000000000                                                  
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033                                                                             
CR2: 00007f0c3ff2f008 CR3: 000000003b1fd000 CR4: 00000000000006e0                                                             
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000                                                             
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400                                                             
Process pdnsd (pid: 3540, threadinfo ffff880024066000, task ffff88002a8bb7a0)                                                 
Stack:                                                                                                                        
 ffff880024067978 0000000000000000 ffff88002eea7c30 ffffffff80628dfd                                                          
 00000000ffffffff ffff88003ccfcb90 ffff88003c8fe700 ffffffff00000020                                                          
 ffff88003c8fe930 ffff88003d944a40 ffff88003ccfcb98 0000000000000282                                                          
Call Trace:                                                                                                                   
 [<ffffffff80628dfd>] ? invoke_tx_handlers+0x52d/0xed0                                                                        
 [<ffffffff80628386>] ? __ieee80211_tx_prepare+0x196/0x320                                                                    
 [<ffffffff8058d0d0>] ? pskb_expand_head+0x110/0x180                                                                          
 [<ffffffff8062aaab>] ? ieee80211_master_start_xmit+0x20b/0x4e0                                                               
 [<ffffffff80231648>] ? update_curr+0xc8/0x130                                                                                
 [<ffffffff805a6916>] ? __qdisc_run+0x226/0x270                                                                               
 [<ffffffff80595640>] ? dev_queue_xmit+0x2a0/0x5b0                                                                            
 [<ffffffff8062a63d>] ? ieee80211_subif_start_xmit+0x3bd/0x620                                                                
 [<ffffffff8062e69b>] ? csum_partial_copy_from_user+0x6b/0x100                                                                
 [<ffffffff805a6916>] ? __qdisc_run+0x226/0x270                                                                               
 [<ffffffff80595640>] ? dev_queue_xmit+0x2a0/0x5b0                                                                            
 [<ffffffff805b8f2c>] ? ip_push_pending_frames+0x2dc/0x440                                                                    
 [<ffffffff805d7ae4>] ? udp_push_pending_frames+0x134/0x3e0                                                                   
 [<ffffffff805d9539>] ? udp_sendmsg+0x329/0x700                                                                               
 [<ffffffff805857bb>] ? sock_sendmsg+0xcb/0x100                                                                               
 [<ffffffff805b1301>] ? __ip_route_output_key+0x191/0xb00                                                                     
 [<ffffffff80254f60>] ? autoremove_wake_function+0x0/0x30                                                                     
 [<ffffffff805b1c89>] ? ip_route_output_flow+0x19/0x70                                                                        
 [<ffffffff802a815e>] ? fget_light+0xce/0xe0                                                                                  
 [<ffffffff80584d62>] ? sockfd_lookup_light+0x22/0x90                                                                         
 [<ffffffff80585c79>] ? sys_sendto+0x119/0x180                                                                                
 [<ffffffff80585095>] ? sys_connect+0xd5/0xe0                                                                                 
 [<ffffffff802a4eff>] ? fd_install+0x2f/0x60                                                                                  
 [<ffffffff8020ba1b>] ? system_call_fastpath+0x16/0x1b                                                                        
Code: 08 48 83 c1 03 48 83 f9 0f 75 db 80 7b 08 00 78 1a 48 83 c4 08 5b 5d c3 90 48 8b 47 08 4c 89 c6 48 8b 7f 10 ff 50 40 66 90 eb af <0f> 0b 0f 1f 40 00 eb fa 66 0f 1f 44 00 00 48 83 c7 18 48 c7 c6                                                     
RIP  [<ffffffff80622db2>] rate_control_get_rate+0xb2/0xc0                                                                     
 RSP <ffff880024067858>                                                                                                       
---[ end trace 7b744176834faae3 ]---                                                                                          
Kernel panic - not syncing: Fatal exception in interrupt
Comment 18 Sergey S. Kostyliov 2009-02-04 01:33:13 UTC
And below is warning produced with "rate 1M fixed"

------------[ cut here ]------------                                                                                         
WARNING: at net/mac80211/tx.c:567 invoke_tx_handlers+0xe9a/0xed0()                                                           
Hardware name: Aspire 5100                                                                                                   
Modules linked in:                                                                                                           
Pid: 0, comm: swapper Not tainted 2.6.29-rc2 #3                                                                              
Call Trace:                                                                                                                  
 <IRQ>  [<ffffffff8023e5ca>] warn_slowpath+0xea/0x160                                                                        
 [<ffffffff80258420>] ktime_get_ts+0x30/0x60                                                                                 
 [<ffffffff8025c238>] getnstimeofday+0x58/0xe0                                                                               
 [<ffffffff80258420>] ktime_get_ts+0x30/0x60                                                                                 
 [<ffffffff8025845c>] ktime_get+0xc/0x50                                                                                     
 [<ffffffff8025c238>] getnstimeofday+0x58/0xe0                                                                               
 [<ffffffff80258420>] ktime_get_ts+0x30/0x60                                                                                 
 [<ffffffff804a9a06>] ath5k_hw_get_isr+0x266/0x3e0                                                                           
 [<ffffffff804a93e4>] ath5k_hw_is_intr_pending+0x14/0x20                                                                     
 [<ffffffff804b1e97>] ath5k_intr+0xd7/0x280                                                                                  
 [<ffffffff804a971d>] ath5k_hw_start_tx_dma+0xfd/0x140                                                                       
 [<ffffffff8062976a>] invoke_tx_handlers+0xe9a/0xed0                                                                         
 [<ffffffff80628386>] __ieee80211_tx_prepare+0x196/0x320                                                                     
 [<ffffffff8058d0d0>] pskb_expand_head+0x110/0x180                                                                           
 [<ffffffff8062aaab>] ieee80211_master_start_xmit+0x20b/0x4e0                                                                
 [<ffffffff805a6916>] __qdisc_run+0x226/0x270                                                                                
 [<ffffffff80595640>] dev_queue_xmit+0x2a0/0x5b0                                                                             
 [<ffffffff8062a63d>] ieee80211_subif_start_xmit+0x3bd/0x620                                                                 
 [<ffffffff805a6916>] __qdisc_run+0x226/0x270                                                                                
 [<ffffffff80595640>] dev_queue_xmit+0x2a0/0x5b0                                                                             
 [<ffffffff805b954e>] ip_queue_xmit+0x22e/0x430                                                                              
 [<ffffffff806259cb>] ieee80211_deliver_skb+0xbb/0x1b0                                                                       
 [<ffffffff80625ec9>] ieee80211_invoke_rx_handlers+0x409/0x15d0                                                              
 [<ffffffff80238a20>] default_wake_function+0x0/0x10                                                                         
 [<ffffffff805cce1f>] tcp_transmit_skb+0x4bf/0x760                                                                           
 [<ffffffff805caf9a>] tcp_rcv_established+0x79a/0x920                                                                        
 [<ffffffff805d260d>] tcp_v4_do_rcv+0xdd/0x210                                                                               
 [<ffffffff805d2db2>] tcp_v4_rcv+0x672/0x710                                                                                 
 [<ffffffff805b4575>] ip_local_deliver_finish+0xb5/0x160                                                                     
 [<ffffffff805b3fd1>] ip_rcv_finish+0x121/0x340                                                                              
 [<ffffffff802128b0>] nommu_map_single+0x0/0xa0                                                                              
 [<ffffffff805943fb>] netif_receive_skb+0x28b/0x370                                                                          
 [<ffffffff80594a40>] process_backlog+0x70/0xc0
 [<ffffffff80592ae1>] net_rx_action+0x121/0x1e0
 [<ffffffff80244393>] __do_softirq+0x73/0x130
 [<ffffffff80223940>] ack_apic_level+0x40/0x100
 [<ffffffff8020caac>] call_softirq+0x1c/0x30
 [<ffffffff8020e475>] do_softirq+0x35/0x80
 [<ffffffff8024403d>] irq_exit+0x8d/0xb0
 [<ffffffff8020e6d3>] do_IRQ+0x83/0x110
 [<ffffffff8020c353>] ret_from_intr+0x0/0xa
 <EOI> <4>---[ end trace 62822c38c6c1ba2f ]---
rc_bug: bad rate 1 56 (n_b 12)
rc_bug: rate 0 0
rc_bug: rate 1 56
rc_bug: rate 2 0
rc_bug: rate 3 0
rc_bug: rate 4 -1
rc_bug: bad rate 1 56 (n_b 12)
rc_bug: rate 0 0
rc_bug: rate 1 56
rc_bug: rate 2 0
rc_bug: rate 3 0
rc_bug: rate 4 -1
rc_bug: bad rate 1 38 (n_b 12)
rc_bug: rate 0 0
rc_bug: rate 1 38
rc_bug: rate 2 0
rc_bug: rate 3 0
rc_bug: rate 4 -1
rc_bug: bad rate 1 55 (n_b 12)
rc_bug: rate 0 0
rc_bug: rate 1 55
rc_bug: rate 2 0
rc_bug: rate 3 0
rc_bug: rate 4 -1
rc_bug: bad rate 1 110 (n_b 12)
rc_bug: rate 0 0
rc_bug: rate 1 110
rc_bug: rate 2 0
rc_bug: rate 3 0
rc_bug: rate 4 -1
ath5k phy0: unsupported jumbo
Comment 19 Bob Copeland 2009-02-04 13:58:20 UTC
Wow, the rate indexes look like rates themselves.  Hmm...
Comment 20 Bob Copeland 2009-02-16 14:58:38 UTC
Ok I still have no idea what is going on, I thought perhaps we are using bitrate value instead of rate index somewhere, but I didn't see such a thing.  The minstrel maintainer is looking into this too.  In the meantime I cooked up another debug patch which will at least debug the rate tables we send/recv.
Comment 21 Bob Copeland 2009-02-16 15:14:19 UTC
Created attachment 20270 [details]
more rates debugging

This will add tons of debug output, but hopefully will show what happens when the rc selects out of bound rate indexes.
Comment 22 Sergey S. Kostyliov 2009-02-22 13:43:57 UTC
Created attachment 20323 [details]
full dmesg

Here is full dmesg for 2.6.29-rc5 with all patches applied.
Comment 23 Bob Copeland 2009-02-23 21:41:54 UTC
Great, thanks.  So the debug log pretty much absolves ath5k of blame.  But the rate index that minstrel is choosing is nonsensical, it looks like memory corruption perhaps.  Can you post your config?  And can you try some of the slab debugging options?
Comment 24 Rafael J. Wysocki 2009-02-25 14:55:16 UTC
On Tuesday 24 February 2009, Bob Copeland wrote:
> On Mon, Feb 23, 2009 at 10:48:05PM +0100, Rafael J. Wysocki wrote:
> > Bug-Entry   : http://bugzilla.kernel.org/show_bug.cgi?id=12490
> > Subject             : ath5k related kernel panic in 2.6.29-rc1
> > Submitter   : Sergey S. Kostyliov <rathamahata@gmail.com>
> > Date                : 2009-01-12 7:38 (43 days old)
> > References  : http://marc.info/?l=linux-kernel&m=123174591509586&w=4
> > Handled-By  : Bob Copeland <me@bobcopeland.com>
> 
> Still unsolved, looks like some kind of memory corruption.
Comment 25 Sergey S. Kostyliov 2009-02-26 11:03:35 UTC
Created attachment 20376 [details]
My config

u post your config?
(In reply to comment #23)
> Great, thanks.  So the debug log pretty much absolves ath5k of blame.  But
> the
> rate index that minstrel is choosing is nonsensical, it looks like memory
> corruption perhaps.  Can you post your config?
Sure, here it is.

> And can you try some of the slab debugging options?
I'll try...
Comment 26 Bob Copeland 2009-02-26 11:14:05 UTC
I notice you don't have CONFIG_MODVERSIONS enabled -- can you try that and report back?  It's under "Enable loadable module support -> Module versioning support".  Also the slab/slub debugging options are under the "Kernel Hacking" menu.
Comment 27 Sergey S. Kostyliov 2009-02-26 13:01:21 UTC
Do I still need all debug patches mentioned early to be applied?
Comment 28 Bob Copeland 2009-02-26 13:13:07 UTC
Nope, you can drop all of them.
Comment 29 Sergey S. Kostyliov 2009-02-26 14:50:52 UTC
Ok. I have enabled CONFIG_MODVERSIONS and slub debuging. I.e.:
rathamahata@autonomist /usr/local/src/linux-2.6.29-rc6 $ grep CONFIG_MODVERSIONS .config
CONFIG_MODVERSIONS=y
rathamahata@autonomist /usr/local/src/linux-2.6.29-rc6 $ grep SLUB_DEBUG .config
CONFIG_SLUB_DEBUG=y
CONFIG_SLUB_DEBUG_ON=y
rathamahata@autonomist /usr/local/src/linux-2.6.29-rc6

Is there any other options I should turn on?

Below is full last 2.6.29-rc6 dmesg including an oops;

Linux version 2.6.29-rc6 (rathamahata@autonomist.ath.cx) (gcc version 4.3.3 (Gentoo 4.3.3 p1.0, pie-10.1.5) ) #1 SMP PREEMPT Fri Feb 27 00:41:58 MSK 2009
Command line: BOOT_IMAGE=/boot/linux-2.6.29-rc6 root=/dev/sda2 rootflags=commit=11 elevator=deadline netconsole=6666@192.168.168.2/eth0,6667@192.168.168.1/00:13:8f:e0:34:f5
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009dc00 (usable)
 BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000d0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000003de80000 (usable)
 BIOS-e820: 000000003de80000 - 000000003de97000 (ACPI data)
 BIOS-e820: 000000003de97000 - 000000003df00000 (ACPI NVS)
 BIOS-e820: 000000003df00000 - 0000000040000000 (reserved)
 BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
DMI present.
last_pfn = 0x3de80 max_arch_pfn = 0x100000000
x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
init_memory_mapping: 0000000000000000-000000003de80000
last_map_addr: 3de80000 end: 3de80000
ACPI: RSDP 000F7D80, 0014 (r0 ACRSYS)
ACPI: RSDT 3DE8E70B, 0038 (r1 ACRSYS ACRPRDCT  6040000  LTP        0)
ACPI: FACP 3DE96C04, 0074 (r1 ATI    Bowfin    6040000 ATI     F4240)
FADT: X_PM1a_EVT_BLK.bit_width (16) does not match PM1_EVT_LEN (4)
FADT: X_PM1b_EVT_BLK.bit_width (16) does not match PM1_EVT_LEN (4)
ACPI: DSDT 3DE8E743, 84C1 (r1   Acer  Navarro  6040000 MSFT  3000000)
ACPI: FACS 3DE97FC0, 0040
ACPI: SLIC 3DE96C78, 0176 (r1 ACRSYS ACRPRDCT  6040000 LOHR        0)
ACPI: APIC 3DE96DEE, 0054 (r1 PTLTD  	 APIC    6040000  LTP        0)
ACPI: MCFG 3DE96E42, 003C (r1 PTLTD    MCFG    6040000  LTP        0)
ACPI: SSDT 3DE96E7E, 0182 (r1 PTLTD  POWERNOW  6040000  LTP        1)
(5 early reservations) ==> bootmem [0000000000 - 003de80000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #2 [0000200000 - 00008aaff4]    TEXT DATA BSS ==> [0000200000 - 00008aaff4]
  #3 [000009dc00 - 0000100000]    BIOS reserved ==> [000009dc00 - 0000100000]
  #4 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
found SMP MP-table at [ffff8800000f7e40] 000f7e40
Zone PFN ranges:
  DMA      0x00000000 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000000 -> 0x0000009d
    0: 0x00000100 -> 0x0003de80
SB4X0 revision 0x83
Ignoring ACPI timer override.
If you got timer trouble try acpi_use_timer_override
Detected use of extended apic ids on hypertransport bus
ACPI: PM-Timer IO Port: 0x8008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 0, address 0xfec00000, GSI 0-23
Using ACPI (MADT) for SMP configuration information
SMP: Allowing 2 CPUs, 0 hotplug CPUs
PM: Registered nosave memory: 000000000009d000 - 000000000009e000
PM: Registered nosave memory: 000000000009e000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 00000000000d0000
PM: Registered nosave memory: 00000000000d0000 - 0000000000100000
Allocating PCI resources starting at 50000000 (gap: 40000000:a0000000)
NR_CPUS:2 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
PERCPU: Allocating 49152 bytes of per cpu data
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 248192
Kernel command line: BOOT_IMAGE=/boot/linux-2.6.29-rc6 root=/dev/sda2 rootflags=commit=11 elevator=deadline netconsole=6666@192.168.168.2/eth0,6667@192.168.168.1/00:13:8f:e0:34:f5
Initializing CPU#0
Preemptible RCU implementation.
PID hash table entries: 4096 (order: 12, 32768 bytes)
Fast TSC calibration using PIT
Detected 1595.711 MHz processor.
Console: colour VGA+ 80x25
console [tty0] enabled
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
Checking aperture...
No AGP bridge found
Node 0: aperture @ cff0000000 size 32 MB
Aperture beyond 4GB. Ignoring.
Memory: 990376k/1014272k available (4464k kernel code, 396k absent, 22912k reserved, 1648k data, 328k init)
SLUB: Genslabs=12, HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
Calibrating delay loop (skipped), value calculated using timer frequency.. 3191.42 BogoMIPS (lpj=1595711)
Mount-cache hash table entries: 256
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
using C1E aware idle routine
ACPI: Core revision 20081204
Setting APIC routing to flat
..TIMER: vector=0x30 apic1=0 pin1=0 apic2=-1 pin2=-1
CPU0: AMD Turion(tm) 64 X2 Mobile Technology TL-52 stepping 02
Booting processor 1 APIC 0x1 ip 0x6000
Initializing CPU#1
Calibrating delay using timer specific routine.. 3191.40 BogoMIPS (lpj=1595704)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 1
x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
CPU1: <6>System has AMD C1E enabled
Switch to broadcast mode on CPU1
AMD Turion(tm) 64 X2 Mobile Technology TL-52 stepping 02
Brought up 2 CPUs
Total of 2 processors activated (6382.83 BogoMIPS).
Switch to broadcast mode on CPU0
net_namespace: 936 bytes
NET: Registered protocol family 16
TOM: 0000000040000000 aka 1024M
ACPI: bus type pci registered
PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 7
PCI: MCFG area at e0000000 reserved in E820
PCI: Using MMCONFIG at e0000000 - e07fffff
PCI: Using configuration type 1 for base access
bio: create slab <bio-0> at 0
ACPI: BIOS _OSI(Linux) query ignored
ACPI: Interpreter enabled
ACPI: (supports S0 S3 S4 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: EC: non-query interrupt received, switching to interrupt mode
ACPI: EC: GPE = 0x10, I/O: command/status = 0x66, data = 0x62
ACPI: EC: driver started in interrupt mode
ACPI: No dock devices found.
ACPI: PCI Root Bridge [PCI0] (0000:00)
pci 0000:00:04.0: PME# supported from D0 D3hot D3cold
pci 0000:00:04.0: PME# disabled
pci 0000:00:05.0: PME# supported from D0 D3hot D3cold
pci 0000:00:05.0: PME# disabled
pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot
pci 0000:00:13.2: PME# disabled
HPET not enabled in BIOS. You might try hpet=force boot option
pci 0000:00:14.2: PME# supported from D0 D3hot D3cold
pci 0000:00:14.2: PME# disabled
pci 0000:02:00.0: PME# supported from D3hot
pci 0000:02:00.0: PME# disabled
pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
pci 0000:06:01.0: PME# supported from D1 D2 D3hot D3cold
pci 0000:06:01.0: PME# disabled
pci 0000:06:04.0: PME# supported from D0 D1 D2 D3hot D3cold
pci 0000:06:04.0: PME# disabled
pci 0000:06:04.1: PME# supported from D0 D1 D2 D3hot
pci 0000:06:04.1: PME# disabled
pci 0000:06:04.2: PME# supported from D0 D1 D2 D3hot
pci 0000:06:04.2: PME# disabled
pci 0000:06:04.3: PME# supported from D0 D1 D2 D3hot
pci 0000:06:04.3: PME# disabled
pci 0000:06:04.4: PME# supported from D0 D1 D2 D3hot
pci 0000:06:04.4: PME# disabled
pci 0000:00:14.4: transparent bridge
ACPI: PCI Interrupt Link [LNKA] (IRQs 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKB] (IRQs 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKC] (IRQs 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKD] (IRQs 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKE] (IRQs 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKU] (IRQs 3 4 5 7) *0, disabled.
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
pci 0000:00:04.0: BAR 7: can't allocate resource
pci 0000:00:04.0: BAR 8: can't allocate resource
pci 0000:00:04.0: BAR 9: can't allocate resource
pci 0000:00:05.0: BAR 7: can't allocate resource
pci 0000:00:05.0: BAR 8: can't allocate resource
pci 0000:02:00.0: BAR 0: can't allocate resource
cfg80211: Using static regulatory domain info
cfg80211: Regulatory domain: US
	(start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
	(2402000 KHz - 2472000 KHz @ 40000 KHz), (600 mBi, 2700 mBm)
	(5170000 KHz - 5190000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
	(5190000 KHz - 5210000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
	(5210000 KHz - 5230000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
	(5230000 KHz - 5330000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
	(5735000 KHz - 5835000 KHz @ 40000 KHz), (600 mBi, 3000 mBm)
cfg80211: Calling CRDA for country: US
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp 00:09: mem resource (0x0-0xfff) overlaps 0000:00:12.0 BAR 6 (0x0-0x7ffff), disabling
pnp 00:09: mem resource (0x0-0xfff) overlaps 0000:01:05.0 BAR 6 (0x0-0x1ffff), disabling
pnp 00:09: mem resource (0x0-0xfff) overlaps 0000:02:00.0 BAR 0 (0x0-0xffff), disabling
pnp: PnP ACPI: found 10 devices
ACPI: ACPI bus type pnp unregistered
system 00:01: iomem range 0xfec00000-0xfec00fff has been reserved
system 00:01: iomem range 0xfee00000-0xfee00fff has been reserved
system 00:08: ioport range 0x1080-0x1080 has been reserved
system 00:08: ioport range 0x40b-0x40b has been reserved
system 00:08: ioport range 0x4d0-0x4d1 has been reserved
system 00:08: ioport range 0x4d6-0x4d6 has been reserved
system 00:08: ioport range 0xc00-0xc01 has been reserved
system 00:08: ioport range 0xc14-0xc14 has been reserved
system 00:08: ioport range 0xc50-0xc52 has been reserved
system 00:08: ioport range 0xc6c-0xc6c has been reserved
system 00:08: ioport range 0xc6f-0xc6f has been reserved
system 00:08: ioport range 0xcd4-0xcd5 has been reserved
system 00:08: ioport range 0xcd6-0xcd7 has been reserved
system 00:08: ioport range 0xcd8-0xcdf has been reserved
system 00:08: ioport range 0x8000-0x805f has been reserved
system 00:08: ioport range 0xf40-0xf47 has been reserved
system 00:08: ioport range 0x280-0x293 has been reserved
system 00:08: ioport range 0x87f-0x87f has been reserved
system 00:09: iomem range 0xe0000-0xfffff could not be reserved
system 00:09: iomem range 0xfff00000-0xffffffff could not be reserved
pci 0000:00:01.0: PCI bridge, secondary bus 0000:01
pci 0000:00:01.0:   IO window: 0x9000-0x9fff
pci 0000:00:01.0:   MEM window: 0xfc000000-0xfc0fffff
pci 0000:00:01.0:   PREFETCH window: 0x000000f8000000-0x000000fbffffff
pci 0000:00:04.0: PCI bridge, secondary bus 0000:02
pci 0000:00:04.0:   IO window: disabled
pci 0000:00:04.0:   MEM window: 0x54000000-0x540fffff
pci 0000:00:04.0:   PREFETCH window: disabled
pci 0000:00:05.0: PCI bridge, secondary bus 0000:04
pci 0000:00:05.0:   IO window: disabled
pci 0000:00:05.0:   MEM window: disabled
pci 0000:00:05.0:   PREFETCH window: disabled
pci 0000:06:04.0: CardBus bridge, secondary bus 0000:07
pci 0000:06:04.0:   IO window: 0x00a400-0x00a4ff
pci 0000:06:04.0:   IO window: 0x00a800-0x00a8ff
pci 0000:06:04.0:   PREFETCH window: 0x50000000-0x53ffffff
pci 0000:06:04.0:   MEM window: 0x58000000-0x5bffffff
pci 0000:00:14.4: PCI bridge, secondary bus 0000:06
pci 0000:00:14.4:   IO window: 0xa000-0xafff
pci 0000:00:14.4:   MEM window: 0xfc200000-0xfc2fffff
Switched to NOHz mode on CPU #0
Switched to NOHz mode on CPU #1
pci 0000:00:14.4:   PREFETCH window: 0x00000050000000-0x00000053ffffff
pci 0000:00:04.0: enabling device (0000 -> 0002)
pci 0000:06:04.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
TCP established hash table entries: 131072 (order: 9, 2097152 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
NET: Registered protocol family 1
HugeTLB registered 2 MB page size, pre-allocated 0 pages
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
SGI XFS with security attributes, large block/inode numbers, no debug enabled
msgmni has been set to 1935
alg: No test for stdrng (krng)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered (default)
io scheduler cfq registered
pci 0000:00:00.0: MSI quirk detected; MSI disabled
ACPI: AC Adapter [ACAD] (on-line)
ACPI: Battery Slot [BAT1] (battery present)
input: Power Button (FF) as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
ACPI: Power Button (FF) [PWRF]
input: Lid Switch as /devices/LNXSYSTM:00/device:00/PNP0C0D:00/input/input1
ACPI: Lid Switch [LID]
input: Power Button (CM) as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input2
ACPI: Power Button (CM) [PWRB]
input: Sleep Button (CM) as /devices/LNXSYSTM:00/device:00/PNP0C0E:00/input/input3
ACPI: Sleep Button (CM) [SLPB]
ACPI: processor limited to max C-state 1
processor ACPI_CPU:00: registered as cooling_device0
processor ACPI_CPU:01: registered as cooling_device1
ACPI: Invalid active0 threshold
thermal LNXTHERM:01: registered as thermal_zone0
ACPI: Thermal Zone [THRM] (58 C)
lp: driver loaded but no devices found
Linux agpgart interface v0.103
[drm] Initialized drm 1.1.0 20060810
pci 0000:01:05.0: power state changed by ACPI to D0
pci 0000:01:05.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
[drm] Initialized radeon 1.29.0 20080528 on minor 0
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
PPP generic driver version 2.4.2
PPP Deflate Compression module registered
PPP BSD Compression module registered
PPP MPPE Compression module registered
NET: Registered protocol family 24
8139too Fast Ethernet driver 0.9.28
8139too 0000:06:01.0: power state changed by ACPI to D0
8139too 0000:06:01.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21
eth0: RealTek RTL8139 at 0xffffc20000032000, 00:16:d4:ce:03:15, IRQ 21
ath5k 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ath5k 0000:02:00.0: registered as 'phy0'
wmaster0 (ath5k): not using net_device_ops yet
wlan0 (ath5k): not using net_device_ops yet
ath5k phy0: Atheros AR2425 chip found (MAC: 0xe2, PHY: 0x70)
netconsole: local port 6666
netconsole: local IP 192.168.168.2
netconsole: interface eth0
netconsole: remote port 6667
netconsole: remote IP 192.168.168.1
netconsole: remote ethernet address 00:13:8f:e0:34:f5
netconsole: device eth0 not up yet, forcing it
eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
netconsole: carrier detect appears untrustworthy, waiting 4 seconds
console [netcon0] enabled
netconsole: network logging started
Driver 'sd' needs updating - please use bus_type methods
Driver 'sr' needs updating - please use bus_type methods
sata_sil 0000:00:12.0: enabling device (0005 -> 0007)
sata_sil 0000:00:12.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
scsi0 : sata_sil
scsi1 : sata_sil
ata1: SATA max UDMA/100 mmio m512@0xfc507000 tf 0xfc507080 irq 22
ata2: SATA max UDMA/100 mmio m512@0xfc507000 tf 0xfc5070c0 irq 22
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata1.00: ATA-7: Hitachi HTS541612J9SA00, SBDOC70P, max UDMA/100
ata1.00: 234441648 sectors, multi 16: LBA48 NCQ (depth 0/32)
ata1.00: configured for UDMA/100
isa bounce pool size: 16 pages
scsi 0:0:0:0: Direct-Access     ATA      Hitachi HTS54161 SBDO PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 234441648 512-byte hardware sectors: (120 GB/111 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 0:0:0:0: [sda] 234441648 512-byte hardware sectors: (120 GB/111 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sda: sda1 sda2 sda3
sd 0:0:0:0: [sda] Attached SCSI disk
sd 0:0:0:0: Attached scsi generic sg0 type 0
ata2: SATA link down (SStatus 0 SControl 310)
pata_atiixp 0000:00:14.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16
scsi2 : pata_atiixp
scsi3 : pata_atiixp
ata3: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0x8420 irq 14
ata4: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0x8428 irq 15
ata3.00: ATAPI: Slimtype DVD A  DS8A1P, CA11, max UDMA/33
ata3.00: configured for UDMA/33
scsi 2:0:0:0: CD-ROM            Slimtype DVD A  DS8A1P    CA11 PQ: 0 ANSI: 5
sr0: scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.20
sr 2:0:0:0: Attached scsi generic sg1 type 5
ieee1394: raw1394: /dev/raw1394 device initialized
usbmon: debugfs is not available
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci_hcd 0000:00:13.2: PCI INT A -> GSI 19 (level, low) -> IRQ 19
ehci_hcd 0000:00:13.2: EHCI Host Controller
ehci_hcd 0000:00:13.2: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:13.2: irq 19, io mem 0xfc506000
ehci_hcd 0000:00:13.2: USB 2.0 started, EHCI 1.00
usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: EHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.29-rc6 ehci_hcd
usb usb1: SerialNumber: 0000:00:13.2
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 8 ports detected
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
ohci_hcd 0000:00:13.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
ohci_hcd 0000:00:13.0: OHCI Host Controller
ohci_hcd 0000:00:13.0: new USB bus registered, assigned bus number 2
ohci_hcd 0000:00:13.0: irq 19, io mem 0xfc504000
usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: OHCI Host Controller
usb usb2: Manufacturer: Linux 2.6.29-rc6 ohci_hcd
usb usb2: SerialNumber: 0000:00:13.0
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 4 ports detected
ohci_hcd 0000:00:13.1: PCI INT A -> GSI 19 (level, low) -> IRQ 19
ohci_hcd 0000:00:13.1: OHCI Host Controller
ohci_hcd 0000:00:13.1: new USB bus registered, assigned bus number 3
ohci_hcd 0000:00:13.1: irq 19, io mem 0xfc505000
usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb3: Product: OHCI Host Controller
usb usb3: Manufacturer: Linux 2.6.29-rc6 ohci_hcd
usb usb3: SerialNumber: 0000:00:13.1
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 4 ports detected
usbcore: registered new interface driver usblp
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
PNP: PS/2 Controller [PNP0303:KBC0,PNP0f13:MSS0] at 0x60,0x64 irq 1,12
i8042.c: Detected active multiplexing controller, rev 1.1.
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX0 port at 0x60,0x64 irq 12
serio: i8042 AUX1 port at 0x60,0x64 irq 12
serio: i8042 AUX2 port at 0x60,0x64 irq 12
serio: i8042 AUX3 port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
rtc_cmos 00:04: RTC can wake from S4
rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one month, 114 bytes nvram
i2c /dev entries driver
piix4_smbus 0000:00:14.0: SMBus Host Controller at 0x8410, revision 0
k8temp 0000:00:18.3: Temperature readouts might be wrong - check erratum #141
cpuidle: using governor ladder
cpuidle: using governor menu
input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input4
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
sdhci-pci 0000:06:04.2: SDHCI controller found [1524:0550] (rev 1)
sdhci-pci 0000:06:04.2: PCI INT B -> GSI 23 (level, low) -> IRQ 23
Registered led device: mmc0::
mmc0: SDHCI controller on PCI [0000:06:04.2] using PIO
sdhci-pci 0000:06:04.4: SDHCI controller found [1524:0551] (rev 1)
sdhci-pci 0000:06:04.4: enabling device (0000 -> 0002)
sdhci-pci 0000:06:04.4: PCI INT B -> GSI 23 (level, low) -> IRQ 23
Registered led device: mmc1::
mmc1: SDHCI controller on PCI [0000:06:04.4] using PIO
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
Advanced Linux Sound Architecture Driver Version 1.0.18a.
HDA Intel 0000:00:14.2: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ALSA device list:
  #0: HDA ATI SB at 0xfc500000 irq 16
TCP cubic registered
NET: Registered protocol family 17
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
powernow-k8: Found 1 AMD Turion(tm) 64 X2 Mobile Technology TL-52 processors (2 cpu cores) (version 2.20.00)
powernow-k8:    0 : fid 0x8 (1600 MHz), vid 0x13
powernow-k8:    1 : fid 0x0 (800 MHz), vid 0x1e
usb 1-4: new high speed USB device using ehci_hcd and address 2
usb 1-4: New USB device found, idVendor=5986, idProduct=0100
usb 1-4: New USB device strings: Mfr=0, Product=1, SerialNumber=0
usb 1-4: Product: USB2.0 Camera
usb 1-4: configuration #1 chosen from 1 choice
Synaptics Touchpad, model: 1, fw: 6.2, id: 0x1280b1, caps: 0xa04713/0x204000
input: SynPS/2 Synaptics TouchPad as /devices/platform/i8042/serio4/input/input5
rtc_cmos 00:04: setting system clock to 2009-02-26 22:27:45 UTC (1235687265)
EXT4-fs: INFO: recovery required on readonly filesystem.
EXT4-fs: write access will be enabled during recovery.
EXT4-fs: barriers enabled
usb 2-4: new low speed USB device using ohci_hcd and address 2
usb 2-4: New USB device found, idVendor=046d, idProduct=c019
usb 2-4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 2-4: Product: USB Optical Mouse
usb 2-4: Manufacturer: Logitech
usb 2-4: configuration #1 chosen from 1 choice
input: Logitech USB Optical Mouse as /devices/pci0000:00/0000:00:13.0/usb2/2-4/2-4:1.0/input/input6
generic-usb 0003:046D:C019.0001: input: USB HID v1.11 Mouse [Logitech USB Optical Mouse] on usb-0000:00:13.0-4/input0
EXT4-fs: delayed allocation enabled
EXT4-fs: file extents enabled
EXT4-fs: mballoc enabled
EXT4-fs: sda2: orphan cleanup on readonly fs
EXT4-fs: sda2: 2 orphan inodes deleted
EXT4-fs: recovery complete.
kjournald2 starting: pid 1249, dev sda2:8, commit interval 11 seconds
EXT4-fs: mounted filesystem sda2 with ordered data mode
VFS: Mounted root (ext4 filesystem) readonly on device 8:2.
Freeing unused kernel memory: 328k freed
EXT4-fs warning: maximal mount count reached, running e2fsck is recommended
EXT4 FS on sda2, internal journal on sda2:8
EXT4-fs: barriers enabled
kjournald2 starting: pid 2587, dev sda3:8, commit interval 11 seconds
EXT4-fs warning: maximal mount count reached, running e2fsck is recommended
EXT4 FS on sda3, internal journal on sda3:8
EXT4-fs: delayed allocation enabled
EXT4-fs: file extents enabled
EXT4-fs: mballoc enabled
EXT4-fs: recovery complete.
EXT4-fs: mounted filesystem sda3 with ordered data mode
Adding 2008084k swap on /dev/sda1.  Priority:-1 extents:1 across:2008084k 
BUG: unable to handle kernel NULL pointer dereference at 0000000000000006
IP: [<ffffffff804c0430>] ath5k_tx+0x280/0x5f0
PGD 369c5067 PUD 369c4067 PMD 0 
Oops: 0000 [#1] PREEMPT SMP 
last sysfs file: /sys/devices/LNXSYSTM:00/device:00/PNP0A03:00/device:1c/PNP0C0A:00/power_supply/BAT1/charge_full
CPU 1 
Modules linked in:
Pid: 0, comm: swapper Tainted: G        W  2.6.29-rc6 #1 Aspire 5100     
RIP: 0010:[<ffffffff804c0430>]  [<ffffffff804c0430>] ath5k_tx+0x280/0x5f0
RSP: 0018:ffff88003d8cb680  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffffffff804b6260 RCX: 0000000000007fff
RDX: ffff8800807bc000 RSI: 0000000000000046 RDI: ffff880034434640
RBP: ffff88003d8cb770 R08: 0000000000000011 R09: 0000000000000010
R10: ffffffff80886ae0 R11: ffff88003d8cb3b0 R12: ffff88003d9aec38
R13: ffff880034434640 R14: ffff88003cded030 R15: ffff88003ce51900
FS:  00007f9b28eab750(0000) GS:ffff88003d8067d0(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000006 CR3: 00000000369f6000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffff88003d8c6000, task ffff88003d882340)
Stack:
 ffff88003a81bc20 0000000000000001 ffff88003d8cb7d0 ffff880034434640
 ffff88003d8cb7a0 ffffffff8063a8c4 ffffffff80803bb0 0000000000000282
 ffff880000000011 ffff88003ce502a0 ffff88003ce54b90 ffff88003ce54b80
Call Trace:
 <IRQ> <0> [<ffffffff8063a8c4>] ? invoke_tx_handlers+0x784/0xe40
 [<ffffffff804b6260>] ? ath5k_hw_setup_4word_tx_desc+0x0/0x260
 [<ffffffff8063b5be>] __ieee80211_tx+0x4e/0x150
 [<ffffffff8063c34f>] ieee80211_master_start_xmit+0x24f/0x520
 [<ffffffff805a4ecd>] dev_hard_start_xmit+0x24d/0x2d0
 [<ffffffff805b6bae>] __qdisc_run+0x22e/0x280
 [<ffffffff805a5318>] dev_queue_xmit+0x288/0x590
 [<ffffffff8063be81>] ieee80211_subif_start_xmit+0x3e1/0x660
 [<ffffffff802a6f7d>] ? check_object+0x27d/0x290
 [<ffffffff805a4ecd>] dev_hard_start_xmit+0x24d/0x2d0
 [<ffffffff80597f81>] ? sock_wmalloc+0x31/0x70
 [<ffffffff805b6bae>] __qdisc_run+0x22e/0x280
 [<ffffffff805a5318>] dev_queue_xmit+0x288/0x590
 [<ffffffff805ca317>] ip_finish_output+0x1b7/0x320
 [<ffffffff805ca50d>] ip_output+0x8d/0xf0
 [<ffffffff805c9210>] ip_local_out+0x20/0x30
 [<ffffffff805c9501>] ip_push_pending_frames+0x2e1/0x450
 [<ffffffff805c98b6>] ip_send_reply+0x246/0x270
 [<ffffffff805e1c49>] tcp_v4_send_reset+0xf9/0x180
 [<ffffffff8059f08c>] ? __skb_checksum_complete+0xc/0x10
 [<ffffffff805e3892>] tcp_v4_rcv+0x292/0x6f0
 [<ffffffff805c4b51>] ip_local_deliver_finish+0xc1/0x170
 [<ffffffff805c4c8d>] ip_local_deliver+0x8d/0xa0
 [<ffffffff805c459b>] ip_rcv_finish+0x12b/0x340
 [<ffffffff805c4a03>] ip_rcv+0x253/0x2e0
 [<ffffffff805a3fe2>] netif_receive_skb+0x282/0x360
 [<ffffffff804c15ac>] ? ath5k_tasklet_rx+0x41c/0x620
 [<ffffffff805a45c0>] napi_gro_receive+0x60/0x70
 [<ffffffff805a464a>] process_backlog+0x7a/0xc0
 [<ffffffff805a263d>] net_rx_action+0x13d/0x200
 [<ffffffff802453a4>] __do_softirq+0x74/0x130
 [<ffffffff8020cbac>] call_softirq+0x1c/0x30
 [<ffffffff8020e4fd>] do_softirq+0x3d/0x80
 [<ffffffff8024503d>] irq_exit+0x8d/0xb0
 [<ffffffff8020e766>] do_IRQ+0x86/0x110
 [<ffffffff8020c453>] ret_from_intr+0x0/0xa
 <EOI> <0> [<ffffffff802136f2>] ? default_idle+0x42/0x50
 [<ffffffff802138f6>] ? c1e_idle+0xa6/0x100
 [<ffffffff8025a931>] ? atomic_notifier_call_chain+0x11/0x20
 [<ffffffff8020aaf2>] ? cpu_idle+0x62/0xc0
 [<ffffffff80652590>] ? start_secondary+0x163/0x1b3
Code: 78 ff ff ff 48 8b 50 28 0f b6 41 04 48 8b 4c c2 18 48 0f be c6 48 8d 14 85 00 00 00 00 48 c1 e0 04 48 29 d0 48 03 41 08 4c 89 ef <44> 0f b7 60 06 41 8b 9f 18 34 00 00 44 89 85 50 ff ff ff e8 48 
RIP  [<ffffffff804c0430>] ath5k_tx+0x280/0x5f0
 RSP <ffff88003d8cb680>
CR2: 0000000000000006
---[ end trace bb001332080d1386 ]---
Kernel panic - not syncing: Fatal exception in interrupt
Comment 30 Bob Copeland 2009-03-01 07:20:49 UTC
So the modversions hint was because it looked like some of the structure accesses were off by a few bytes which can happen if the headers you compile against are different from the ones modules are compiled against, but now I see you're not using a modular kernel.

Hmm, as far as I can tell this is the same as the original (get_tx_rate returns null).  There wasn't any WARN in dmesg?  The taint flags indicate that there should have been a warning. 

BTW it might also help subsequent investigations to turn on debug info -- my cross compiler generates much different code than yours so I'd probably need your objdump -S, but that is almost useless without debug info.  That is under "Kernel Hacking->Compile the kernel with debug info".  Also, what version of compiler are you using?

I'll upload my config for reference, but it is 32 bit x86.
Comment 31 Bob Copeland 2009-03-01 07:23:13 UTC
Created attachment 20397 [details]
Bob's config

My config, for reference.
Comment 32 Sergey S. Kostyliov 2009-03-01 08:18:04 UTC

(In reply to comment #30)
....
> Hmm, as far as I can tell this is the same as the original (get_tx_rate
> returns null).  There wasn't any WARN in dmesg?  The taint flags indicate
> that there should have been a warning.

I'm pretty sure there was warning but it was lost due to the some interaction with netconsole. It seems to me that netconsole doesn't send WARN until loglevel is high enough. In my case loglevel is set by init scripts. And it looks like warnings(s) had happened before particular init script was executed. Unfortunately neither warnings(s) nor oops was written to local file system at all. And I don't know how to properly solve that situation.


> 
> BTW it might also help subsequent investigations to turn on debug info -- my
> cross compiler generates much different code than yours so I'd probably need
> your objdump -S, but that is almost useless without debug info.  That is
> under
> "Kernel Hacking->Compile the kernel with debug info".  Also, what version of
> compiler are you using?
> 
> I'll upload my config for reference, but it is 32 bit x86.

Ok. I'll try to change all debug options accordingly.
Comment 33 Sergey S. Kostyliov 2009-03-01 08:47:59 UTC
(In reply to comment #30)
...
>  Also, what version of compiler are you using?
Sorry. I forgot to answer that part:
rathamahata@autonomist ~ $ gcc -dumpversion
4.3.3
rathamahata@autonomist ~ $
Comment 34 Sergey S. Kostyliov 2009-03-01 13:31:46 UTC
So, I recompiled 2.6.29-rc6 with CONFIG_DEBUG_INFO=y and several other debug options enabled.
rathamahata@autonomist /usr/local/src/linux-2.6.29-rc6 $ grep DEBUG .config | egrep -v '^#'
CONFIG_SLUB_DEBUG=y
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_ATH5K_DEBUG=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_SHIRQ=y
CONFIG_SCHED_DEBUG=y
CONFIG_SLUB_DEBUG_ON=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_SPINLOCK_SLEEP=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_VM=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_NOTIFIERS=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_DEBUG_RODATA=y
rathamahata@autonomist /usr/local/src/linux-2.6.29-rc6 $

I have just rebooted that kernel with the same oops. And again unfortunately I was unable to catch warning that precedes oops via netconsole.
Comment 35 Sergey S. Kostyliov 2009-03-02 00:10:01 UTC
I have finally succeeded in getting both warning and oops. Below is full dmesg:

Linux version 2.6.29-rc6 (rathamahata@autonomist.ath.cx) (gcc version 4.3.3 (Gentoo 4.3.3 p1.0, pie-10.1.5) ) #2 SMP PREEMPT Sun Mar 1 20:50:14 MSK 2009
Command line: BOOT_IMAGE=/boot/linux-2.6.29-rc6 root=/dev/sda2 rootflags=commit=11 elevator=deadline netconsole=6666@192.168.168.2/eth0,6667@192.168.168.1/00:13:8f:e0:34:f5
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009dc00 (usable)
 BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000d0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000003de80000 (usable)
 BIOS-e820: 000000003de80000 - 000000003de97000 (ACPI data)
 BIOS-e820: 000000003de97000 - 000000003df00000 (ACPI NVS)
 BIOS-e820: 000000003df00000 - 0000000040000000 (reserved)
 BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
 BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
DMI present.
last_pfn = 0x3de80 max_arch_pfn = 0x100000000
x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
init_memory_mapping: 0000000000000000-000000003de80000
last_map_addr: 3de80000 end: 3de80000
ACPI: RSDP 000F7D80, 0014 (r0 ACRSYS)
ACPI: RSDT 3DE8E70B, 0038 (r1 ACRSYS ACRPRDCT  6040000  LTP        0)
ACPI: FACP 3DE96C04, 0074 (r1 ATI    Bowfin    6040000 ATI     F4240)
FADT: X_PM1a_EVT_BLK.bit_width (16) does not match PM1_EVT_LEN (4)
FADT: X_PM1b_EVT_BLK.bit_width (16) does not match PM1_EVT_LEN (4)
ACPI: DSDT 3DE8E743, 84C1 (r1   Acer  Navarro  6040000 MSFT  3000000)
ACPI: FACS 3DE97FC0, 0040
ACPI: SLIC 3DE96C78, 0176 (r1 ACRSYS ACRPRDCT  6040000 LOHR        0)
ACPI: APIC 3DE96DEE, 0054 (r1 PTLTD  	 APIC    6040000  LTP        0)
ACPI: MCFG 3DE96E42, 003C (r1 PTLTD    MCFG    6040000  LTP        0)
ACPI: SSDT 3DE96E7E, 0182 (r1 PTLTD  POWERNOW  6040000  LTP        1)
(5 early reservations) ==> bootmem [0000000000 - 003de80000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #2 [0000200000 - 00008d1c34]    TEXT DATA BSS ==> [0000200000 - 00008d1c34]
  #3 [000009dc00 - 0000100000]    BIOS reserved ==> [000009dc00 - 0000100000]
  #4 [00008d2000 - 0000ac2000]          PGTABLE ==> [00008d2000 - 0000ac2000]
found SMP MP-table at [ffff8800000f7e40] 000f7e40
Zone PFN ranges:
  DMA      0x00000000 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000000 -> 0x0000009d
    0: 0x00000100 -> 0x0003de80
SB4X0 revision 0x83
Ignoring ACPI timer override.
If you got timer trouble try acpi_use_timer_override
Detected use of extended apic ids on hypertransport bus
ACPI: PM-Timer IO Port: 0x8008
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 0, address 0xfec00000, GSI 0-23
Using ACPI (MADT) for SMP configuration information
SMP: Allowing 2 CPUs, 0 hotplug CPUs
PM: Registered nosave memory: 000000000009d000 - 000000000009e000
PM: Registered nosave memory: 000000000009e000 - 00000000000a0000
PM: Registered nosave memory: 00000000000a0000 - 00000000000d0000
PM: Registered nosave memory: 00000000000d0000 - 0000000000100000
Allocating PCI resources starting at 50000000 (gap: 40000000:a0000000)
NR_CPUS:2 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
PERCPU: Allocating 49152 bytes of per cpu data
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 247658
Kernel command line: BOOT_IMAGE=/boot/linux-2.6.29-rc6 root=/dev/sda2 rootflags=commit=11 elevator=deadline netconsole=6666@192.168.168.2/eth0,6667@192.168.168.1/00:13:8f:e0:34:f5
Initializing CPU#0
Preemptible RCU implementation.
PID hash table entries: 4096 (order: 12, 32768 bytes)
Fast TSC calibration using PIT
Detected 1596.111 MHz processor.
Console: colour VGA+ 80x25
console [tty0] enabled
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
Checking aperture...
No AGP bridge found
Node 0: aperture @ cdf0000000 size 32 MB
Aperture beyond 4GB. Ignoring.
Memory: 988240k/1014272k available (4520k kernel code, 396k absent, 25064k reserved, 1717k data, 328k init)
SLUB: Genslabs=12, HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
Calibrating delay loop (skipped), value calculated using timer frequency.. 3192.22 BogoMIPS (lpj=1596111)
Mount-cache hash table entries: 256
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
using C1E aware idle routine
ACPI: Core revision 20081204
Setting APIC routing to flat
..TIMER: vector=0x30 apic1=0 pin1=0 apic2=-1 pin2=-1
CPU0: AMD Turion(tm) 64 X2 Mobile Technology TL-52 stepping 02
Booting processor 1 APIC 0x1 ip 0x6000
Initializing CPU#1
Calibrating delay using timer specific routine.. 3191.35 BogoMIPS (lpj=1595676)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 1
x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
CPU1: <6>System has AMD C1E enabled
Switch to broadcast mode on CPU1
AMD Turion(tm) 64 X2 Mobile Technology TL-52 stepping 02
Brought up 2 CPUs
Total of 2 processors activated (6383.57 BogoMIPS).
Switch to broadcast mode on CPU0
net_namespace: 968 bytes
NET: Registered protocol family 16
TOM: 0000000040000000 aka 1024M
ACPI: bus type pci registered
PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 7
PCI: MCFG area at e0000000 reserved in E820
PCI: Using MMCONFIG at e0000000 - e07fffff
PCI: Using configuration type 1 for base access
bio: create slab <bio-0> at 0
ACPI: BIOS _OSI(Linux) query ignored
ACPI: Interpreter enabled
ACPI: (supports S0 S3 S4 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: EC: non-query interrupt received, switching to interrupt mode
ACPI: EC: GPE = 0x10, I/O: command/status = 0x66, data = 0x62
ACPI: EC: driver started in interrupt mode
ACPI: No dock devices found.
ACPI: PCI Root Bridge [PCI0] (0000:00)
pci 0000:00:04.0: PME# supported from D0 D3hot D3cold
pci 0000:00:04.0: PME# disabled
pci 0000:00:05.0: PME# supported from D0 D3hot D3cold
pci 0000:00:05.0: PME# disabled
pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot
pci 0000:00:13.2: PME# disabled
HPET not enabled in BIOS. You might try hpet=force boot option
pci 0000:00:14.2: PME# supported from D0 D3hot D3cold
pci 0000:00:14.2: PME# disabled
pci 0000:02:00.0: PME# supported from D3hot
pci 0000:02:00.0: PME# disabled
pci 0000:02:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
pci 0000:06:01.0: PME# supported from D1 D2 D3hot D3cold
pci 0000:06:01.0: PME# disabled
pci 0000:06:04.0: PME# supported from D0 D1 D2 D3hot D3cold
pci 0000:06:04.0: PME# disabled
pci 0000:06:04.1: PME# supported from D0 D1 D2 D3hot
pci 0000:06:04.1: PME# disabled
pci 0000:06:04.2: PME# supported from D0 D1 D2 D3hot
pci 0000:06:04.2: PME# disabled
pci 0000:06:04.3: PME# supported from D0 D1 D2 D3hot
pci 0000:06:04.3: PME# disabled
pci 0000:06:04.4: PME# supported from D0 D1 D2 D3hot
pci 0000:06:04.4: PME# disabled
pci 0000:00:14.4: transparent bridge
ACPI: PCI Interrupt Link [LNKA] (IRQs 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKB] (IRQs 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKC] (IRQs 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKD] (IRQs 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKE] (IRQs 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 10 11) *0, disabled.
ACPI: PCI Interrupt Link [LNKU] (IRQs 3 4 5 7) *0, disabled.
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
pci 0000:00:04.0: BAR 7: can't allocate resource
pci 0000:00:04.0: BAR 8: can't allocate resource
pci 0000:00:04.0: BAR 9: can't allocate resource
pci 0000:00:05.0: BAR 7: can't allocate resource
pci 0000:00:05.0: BAR 8: can't allocate resource
pci 0000:02:00.0: BAR 0: can't allocate resource
cfg80211: Using static regulatory domain info
cfg80211: Regulatory domain: US
	(start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
	(2402000 KHz - 2472000 KHz @ 40000 KHz), (600 mBi, 2700 mBm)
	(5170000 KHz - 5190000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
	(5190000 KHz - 5210000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
	(5210000 KHz - 5230000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
	(5230000 KHz - 5330000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
	(5735000 KHz - 5835000 KHz @ 40000 KHz), (600 mBi, 3000 mBm)
cfg80211: Calling CRDA for country: US
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp 00:09: mem resource (0x0-0xfff) overlaps 0000:00:12.0 BAR 6 (0x0-0x7ffff), disabling
pnp 00:09: mem resource (0x0-0xfff) overlaps 0000:01:05.0 BAR 6 (0x0-0x1ffff), disabling
pnp 00:09: mem resource (0x0-0xfff) overlaps 0000:02:00.0 BAR 0 (0x0-0xffff), disabling
pnp: PnP ACPI: found 10 devices
ACPI: ACPI bus type pnp unregistered
system 00:01: iomem range 0xfec00000-0xfec00fff has been reserved
system 00:01: iomem range 0xfee00000-0xfee00fff has been reserved
system 00:08: ioport range 0x1080-0x1080 has been reserved
system 00:08: ioport range 0x40b-0x40b has been reserved
system 00:08: ioport range 0x4d0-0x4d1 has been reserved
system 00:08: ioport range 0x4d6-0x4d6 has been reserved
system 00:08: ioport range 0xc00-0xc01 has been reserved
system 00:08: ioport range 0xc14-0xc14 has been reserved
system 00:08: ioport range 0xc50-0xc52 has been reserved
system 00:08: ioport range 0xc6c-0xc6c has been reserved
system 00:08: ioport range 0xc6f-0xc6f has been reserved
system 00:08: ioport range 0xcd4-0xcd5 has been reserved
system 00:08: ioport range 0xcd6-0xcd7 has been reserved
system 00:08: ioport range 0xcd8-0xcdf has been reserved
system 00:08: ioport range 0x8000-0x805f has been reserved
system 00:08: ioport range 0xf40-0xf47 has been reserved
system 00:08: ioport range 0x280-0x293 has been reserved
system 00:08: ioport range 0x87f-0x87f has been reserved
system 00:09: iomem range 0xe0000-0xfffff could not be reserved
system 00:09: iomem range 0xfff00000-0xffffffff could not be reserved
pci 0000:00:01.0: PCI bridge, secondary bus 0000:01
pci 0000:00:01.0:   IO window: 0x9000-0x9fff
pci 0000:00:01.0:   MEM window: 0xfc000000-0xfc0fffff
pci 0000:00:01.0:   PREFETCH window: 0x000000f8000000-0x000000fbffffff
pci 0000:00:04.0: PCI bridge, secondary bus 0000:02
pci 0000:00:04.0:   IO window: disabled
pci 0000:00:04.0:   MEM window: 0x54000000-0x540fffff
pci 0000:00:04.0:   PREFETCH window: disabled
pci 0000:00:05.0: PCI bridge, secondary bus 0000:04
pci 0000:00:05.0:   IO window: disabled
pci 0000:00:05.0:   MEM window: disabled
pci 0000:00:05.0:   PREFETCH window: disabled
pci 0000:06:04.0: CardBus bridge, secondary bus 0000:07
Switched to NOHz mode on CPU #0
Switched to NOHz mode on CPU #1
pci 0000:06:04.0:   IO window: 0x00a400-0x00a4ff
pci 0000:06:04.0:   IO window: 0x00a800-0x00a8ff
pci 0000:06:04.0:   PREFETCH window: 0x50000000-0x53ffffff
pci 0000:06:04.0:   MEM window: 0x58000000-0x5bffffff
pci 0000:00:14.4: PCI bridge, secondary bus 0000:06
pci 0000:00:14.4:   IO window: 0xa000-0xafff
pci 0000:00:14.4:   MEM window: 0xfc200000-0xfc2fffff
pci 0000:00:14.4:   PREFETCH window: 0x00000050000000-0x00000053ffffff
pci 0000:00:04.0: enabling device (0000 -> 0002)
pci 0000:06:04.0: PCI INT A -> GSI 20 (level, low) -> IRQ 20
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
TCP established hash table entries: 131072 (order: 9, 2097152 bytes)
TCP bind hash table entries: 65536 (order: 9, 2097152 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
NET: Registered protocol family 1
HugeTLB registered 2 MB page size, pre-allocated 0 pages
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
SGI XFS with security attributes, large block/inode numbers, no debug enabled
msgmni has been set to 1931
alg: No test for stdrng (krng)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered (default)
io scheduler cfq registered
pci 0000:00:00.0: MSI quirk detected; MSI disabled
ACPI: AC Adapter [ACAD] (on-line)
ACPI: Battery Slot [BAT1] (battery present)
input: Power Button (FF) as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
ACPI: Power Button (FF) [PWRF]
input: Lid Switch as /devices/LNXSYSTM:00/device:00/PNP0C0D:00/input/input1
ACPI: Lid Switch [LID]
input: Power Button (CM) as /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input2
ACPI: Power Button (CM) [PWRB]
input: Sleep Button (CM) as /devices/LNXSYSTM:00/device:00/PNP0C0E:00/input/input3
ACPI: Sleep Button (CM) [SLPB]
ACPI: processor limited to max C-state 1
processor ACPI_CPU:00: registered as cooling_device0
processor ACPI_CPU:01: registered as cooling_device1
ACPI: Invalid active0 threshold
thermal LNXTHERM:01: registered as thermal_zone0
ACPI: Thermal Zone [THRM] (66 C)
lp: driver loaded but no devices found
Linux agpgart interface v0.103
[drm] Initialized drm 1.1.0 20060810
pci 0000:01:05.0: power state changed by ACPI to D0
pci 0000:01:05.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
[drm] Initialized radeon 1.29.0 20080528 on minor 0
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
PPP generic driver version 2.4.2
PPP Deflate Compression module registered
PPP BSD Compression module registered
PPP MPPE Compression module registered
NET: Registered protocol family 24
8139too Fast Ethernet driver 0.9.28
8139too 0000:06:01.0: power state changed by ACPI to D0
8139too 0000:06:01.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21
eth0: RealTek RTL8139 at 0xffffc20000032000, 00:16:d4:ce:03:15, IRQ 21
ath5k 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ath5k 0000:02:00.0: registered as 'phy0'
wmaster0 (ath5k): not using net_device_ops yet
wlan0 (ath5k): not using net_device_ops yet
ath5k phy0: Atheros AR2425 chip found (MAC: 0xe2, PHY: 0x70)
netconsole: local port 6666
netconsole: local IP 192.168.168.2
netconsole: interface eth0
netconsole: remote port 6667
netconsole: remote IP 192.168.168.1
netconsole: remote ethernet address 00:13:8f:e0:34:f5
netconsole: device eth0 not up yet, forcing it
eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
netconsole: carrier detect appears untrustworthy, waiting 4 seconds
console [netcon0] enabled
netconsole: network logging started
Driver 'sd' needs updating - please use bus_type methods
Driver 'sr' needs updating - please use bus_type methods
sata_sil 0000:00:12.0: enabling device (0005 -> 0007)
sata_sil 0000:00:12.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
scsi0 : sata_sil
scsi1 : sata_sil
ata1: SATA max UDMA/100 mmio m512@0xfc507000 tf 0xfc507080 irq 22
ata2: SATA max UDMA/100 mmio m512@0xfc507000 tf 0xfc5070c0 irq 22
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata1.00: ATA-7: Hitachi HTS541612J9SA00, SBDOC70P, max UDMA/100
ata1.00: 234441648 sectors, multi 16: LBA48 NCQ (depth 0/32)
ata1.00: configured for UDMA/100
isa bounce pool size: 16 pages
scsi 0:0:0:0: Direct-Access     ATA      Hitachi HTS54161 SBDO PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 234441648 512-byte hardware sectors: (120 GB/111 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 0:0:0:0: [sda] 234441648 512-byte hardware sectors: (120 GB/111 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sda: sda1 sda2 sda3
sd 0:0:0:0: [sda] Attached SCSI disk
sd 0:0:0:0: Attached scsi generic sg0 type 0
ata2: SATA link down (SStatus 0 SControl 310)
pata_atiixp 0000:00:14.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16
scsi2 : pata_atiixp
scsi3 : pata_atiixp
ata3: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0x8420 irq 14
ata4: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0x8428 irq 15
ata3.00: ATAPI: Slimtype DVD A  DS8A1P, CA11, max UDMA/33
ata3.00: configured for UDMA/33
scsi 2:0:0:0: CD-ROM            Slimtype DVD A  DS8A1P    CA11 PQ: 0 ANSI: 5
sr0: scsi3-mmc drive: 24x/24x writer dvd-ram cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.20
sr 2:0:0:0: Attached scsi generic sg1 type 5
ieee1394: raw1394: /dev/raw1394 device initialized
usbmon: debugfs is not available
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci_hcd 0000:00:13.2: PCI INT A -> GSI 19 (level, low) -> IRQ 19
ehci_hcd 0000:00:13.2: EHCI Host Controller
ehci_hcd 0000:00:13.2: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:13.2: irq 19, io mem 0xfc506000
ehci_hcd 0000:00:13.2: USB 2.0 started, EHCI 1.00
usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: EHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.29-rc6 ehci_hcd
usb usb1: SerialNumber: 0000:00:13.2
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 8 ports detected
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
ohci_hcd 0000:00:13.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
ohci_hcd 0000:00:13.0: OHCI Host Controller
ohci_hcd 0000:00:13.0: new USB bus registered, assigned bus number 2
ohci_hcd 0000:00:13.0: irq 19, io mem 0xfc504000
usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: OHCI Host Controller
usb usb2: Manufacturer: Linux 2.6.29-rc6 ohci_hcd
usb usb2: SerialNumber: 0000:00:13.0
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 4 ports detected
ohci_hcd 0000:00:13.1: PCI INT A -> GSI 19 (level, low) -> IRQ 19
ohci_hcd 0000:00:13.1: OHCI Host Controller
ohci_hcd 0000:00:13.1: new USB bus registered, assigned bus number 3
ohci_hcd 0000:00:13.1: irq 19, io mem 0xfc505000
usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb3: Product: OHCI Host Controller
usb usb3: Manufacturer: Linux 2.6.29-rc6 ohci_hcd
usb usb3: SerialNumber: 0000:00:13.1
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 4 ports detected
usbcore: registered new interface driver usblp
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
PNP: PS/2 Controller [PNP0303:KBC0,PNP0f13:MSS0] at 0x60,0x64 irq 1,12
i8042.c: Detected active multiplexing controller, rev 1.1.
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX0 port at 0x60,0x64 irq 12
serio: i8042 AUX1 port at 0x60,0x64 irq 12
serio: i8042 AUX2 port at 0x60,0x64 irq 12
serio: i8042 AUX3 port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
rtc_cmos 00:04: RTC can wake from S4
rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one month, 114 bytes nvram
i2c /dev entries driver
piix4_smbus 0000:00:14.0: SMBus Host Controller at 0x8410, revision 0
k8temp 0000:00:18.3: Temperature readouts might be wrong - check erratum #141
cpuidle: using governor ladder
cpuidle: using governor menu
input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input4
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
sdhci-pci 0000:06:04.2: SDHCI controller found [1524:0550] (rev 1)
sdhci-pci 0000:06:04.2: PCI INT B -> GSI 23 (level, low) -> IRQ 23
Registered led device: mmc0::
mmc0: SDHCI controller on PCI [0000:06:04.2] using PIO
sdhci-pci 0000:06:04.4: SDHCI controller found [1524:0551] (rev 1)
sdhci-pci 0000:06:04.4: enabling device (0000 -> 0002)
sdhci-pci 0000:06:04.4: PCI INT B -> GSI 23 (level, low) -> IRQ 23
Registered led device: mmc1::
mmc1: SDHCI controller on PCI [0000:06:04.4] using PIO
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
Advanced Linux Sound Architecture Driver Version 1.0.18a.
HDA Intel 0000:00:14.2: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ALSA device list:
  #0: HDA ATI SB at 0xfc500000 irq 16
TCP cubic registered
NET: Registered protocol family 17
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
powernow-k8: Found 1 AMD Turion(tm) 64 X2 Mobile Technology TL-52 processors (2 cpu cores) (version 2.20.00)
powernow-k8:    0 : fid 0x8 (1600 MHz), vid 0x13
powernow-k8:    1 : fid 0x0 (800 MHz), vid 0x1e
usb 1-4: new high speed USB device using ehci_hcd and address 2
usb 1-4: New USB device found, idVendor=5986, idProduct=0100
usb 1-4: New USB device strings: Mfr=0, Product=1, SerialNumber=0
usb 1-4: Product: USB2.0 Camera
usb 1-4: configuration #1 chosen from 1 choice
Synaptics Touchpad, model: 1, fw: 6.2, id: 0x1280b1, caps: 0xa04713/0x204000
input: SynPS/2 Synaptics TouchPad as /devices/platform/i8042/serio4/input/input5
rtc_cmos 00:04: setting system clock to 2009-03-02 07:47:08 UTC (1235980028)
EXT4-fs: INFO: recovery required on readonly filesystem.
EXT4-fs: write access will be enabled during recovery.
EXT4-fs: barriers enabled
EXT4-fs: delayed allocation enabled
EXT4-fs: file extents enabled
EXT4-fs: mballoc enabled
EXT4-fs: recovery complete.
kjournald2 starting: pid 1243, dev sda2:8, commit interval 11 seconds
EXT4-fs: mounted filesystem sda2 with ordered data mode
VFS: Mounted root (ext4 filesystem) readonly on device 8:2.
debug: unmapping init memory ffffffff80836000..ffffffff80888000
Write protecting the kernel read-only data: 5836k
usb 2-4: new low speed USB device using ohci_hcd and address 2
usb 2-4: New USB device found, idVendor=046d, idProduct=c019
usb 2-4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 2-4: Product: USB Optical Mouse
usb 2-4: Manufacturer: Logitech
usb 2-4: configuration #1 chosen from 1 choice
input: Logitech USB Optical Mouse as /devices/pci0000:00/0000:00:13.0/usb2/2-4/2-4:1.0/input/input6
generic-usb 0003:046D:C019.0001: input: USB HID v1.11 Mouse [Logitech USB Optical Mouse] on usb-0000:00:13.0-4/input0
SysRq : Changing Loglevel
Loglevel set to 8
EXT4-fs warning: maximal mount count reached, running e2fsck is recommended
EXT4 FS on sda2, internal journal on sda2:8
EXT4-fs: barriers enabled
kjournald2 starting: pid 2532, dev sda3:8, commit interval 11 seconds
EXT4-fs warning: maximal mount count reached, running e2fsck is recommended
EXT4 FS on sda3, internal journal on sda3:8
EXT4-fs: delayed allocation enabled
EXT4-fs: file extents enabled
EXT4-fs: mballoc enabled
EXT4-fs: recovery complete.
EXT4-fs: mounted filesystem sda3 with ordered data mode
Adding 2008084k swap on /dev/sda1.  Priority:-1 extents:1 across:2008084k 
wlan0: deauthenticating by local choice (reason=3)
ath5k phy0: bf=ffff88003c96f480 bf_skb=(null)
ath5k phy0: bf=ffff88003c96f480 bf_skb=(null)
[drm] Setting GART location based on new memory map
[drm] Loading R300 Microcode
[drm] Num pipes: 4
[drm] writeback test succeeded in 1 usecs
------------[ cut here ]------------
WARNING: at net/mac80211/tx.c:567 invoke_tx_handlers+0xdd9/0xe40()
Hardware name: Aspire 5100     
Modules linked in:
Pid: 3213, comm: pdnsd Not tainted 2.6.29-rc6 #2
Call Trace:
 [<ffffffff80243280>] warn_slowpath+0xd0/0x130
 [<ffffffff8029c820>] ? do_brk+0x360/0x3e0
 [<ffffffff8020c50e>] ? common_interrupt+0xe/0x13
 [<ffffffff80647c99>] invoke_tx_handlers+0xdd9/0xe40
 [<ffffffff805a8da5>] ? skb_release_data+0x85/0xd0
 [<ffffffff8066653d>] ? _spin_unlock_irqrestore+0x1d/0x50
 [<ffffffff8064698d>] ? __ieee80211_tx_prepare+0x19d/0x310
 [<ffffffff80649087>] ieee80211_master_start_xmit+0x227/0x510
 [<ffffffff805b169d>] dev_hard_start_xmit+0x24d/0x2d0
 [<ffffffff805c34ee>] __qdisc_run+0x23e/0x290
 [<ffffffff805b1ae0>] dev_queue_xmit+0x290/0x5a0
 [<ffffffff80648bd7>] ieee80211_subif_start_xmit+0x3d7/0x660
 [<ffffffff805b169d>] dev_hard_start_xmit+0x24d/0x2d0
 [<ffffffff805c34ee>] __qdisc_run+0x23e/0x290
 [<ffffffff805b1ae0>] dev_queue_xmit+0x290/0x5a0
 [<ffffffff805b849c>] neigh_resolve_output+0x10c/0x2d0
 [<ffffffff805d6c7a>] ip_finish_output+0x14a/0x320
 [<ffffffff805d6ee7>] ip_output+0x97/0xf0
 [<ffffffff805d5be0>] ip_local_out+0x20/0x30
 [<ffffffff805d5ed1>] ip_push_pending_frames+0x2e1/0x450
 [<ffffffff805f59ad>] udp_push_pending_frames+0x13d/0x3e0
 [<ffffffff805f7459>] udp_sendmsg+0x319/0x740
 [<ffffffff80249459>] ? local_bh_enable_ip+0x59/0xc0
 [<ffffffff805f83af>] ? arp_bind_neighbour+0x6f/0xb0
 [<ffffffff805fdd75>] inet_sendmsg+0x45/0x80
 [<ffffffff805a12df>] sock_sendmsg+0xdf/0x110
 [<ffffffff8025a7a0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff802b43de>] ? fget_light+0xce/0xe0
 [<ffffffff805a0915>] ? sockfd_lookup_light+0x25/0x80
 [<ffffffff805a171a>] sys_sendto+0xea/0x120
 [<ffffffff805fddee>] ? inet_dgram_connect+0x3e/0x80
 [<ffffffff805a0bdd>] ? sys_connect+0x9d/0xa0
 [<ffffffff802b1222>] ? fd_install+0x52/0x60
 [<ffffffff8020bbdb>] system_call_fastpath+0x16/0x1b
---[ end trace 5e4271fb8fc22a18 ]---
------------[ cut here ]------------
kernel BUG at net/mac80211/rate.c:239!
invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
last sysfs file: /sys/devices/LNXSYSTM:00/device:00/PNP0A03:00/device:1c/PNP0C0A:00/power_supply/BAT1/charge_full
CPU 1 
Modules linked in:
Pid: 0, comm: swapper Tainted: G        W  2.6.29-rc6 #2 Aspire 5100     
RIP: 0010:[<ffffffff806413ca>]  [<ffffffff806413ca>] rate_control_get_rate+0xba/0xc0
RSP: 0018:ffff88003d8b7630  EFLAGS: 00010286
RAX: 00000000ffffffff RBX: ffff88003a0a73f0 RCX: ffff88003d8b75ec
RDX: 0000000000000000 RSI: ffff88003a0a73f9 RDI: ffff88003d8b75ec
RBP: ffff88003d8b7640 R08: 0000000000000058 R09: 0000000000000001
R10: ffff88003b49ebd0 R11: 0000000000000060 R12: ffff88003d82c7e0
R13: ffff88003a0a73f0 R14: ffff88003d8b7770 R15: ffff88003c9a9b30
FS:  00007fc2db5b86f0(0000) GS:ffff88003d8047d0(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00007fa6caf3a034 CR3: 0000000000155000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffff88003d8b0000, task ffff88003d88b300)
Stack:
 0000000000000000 ffff88003a0a73c0 ffff88003d8b7740 ffffffff806473a2
 ffffffff80827c50 ffffffff805a8da5 ffff88003d8b76a0 ffff88003a0a73c0
 000088003c5955f8 ffff88003b509e84 ffff88003c5955a0 0000000000000286
Call Trace:
 <IRQ> <0> [<ffffffff806473a2>] invoke_tx_handlers+0x4e2/0xe40
 [<ffffffff805a8da5>] ? skb_release_data+0x85/0xd0
 [<ffffffff8066653d>] ? _spin_unlock_irqrestore+0x1d/0x50
 [<ffffffff8064698d>] ? __ieee80211_tx_prepare+0x19d/0x310
 [<ffffffff80649087>] ieee80211_master_start_xmit+0x227/0x510
 [<ffffffff805b169d>] dev_hard_start_xmit+0x24d/0x2d0
 [<ffffffff805c34ee>] __qdisc_run+0x23e/0x290
 [<ffffffff805b1ae0>] dev_queue_xmit+0x290/0x5a0
 [<ffffffff80648bd7>] ieee80211_subif_start_xmit+0x3d7/0x660
 [<ffffffff8064b753>] ? ieee80211_led_tx+0x33/0x50
 [<ffffffff805b169d>] dev_hard_start_xmit+0x24d/0x2d0
 [<ffffffff805c34ee>] __qdisc_run+0x23e/0x290
 [<ffffffff805b1ae0>] dev_queue_xmit+0x290/0x5a0
 [<ffffffff805d6cf7>] ip_finish_output+0x1c7/0x320
 [<ffffffff805d6ee7>] ip_output+0x97/0xf0
 [<ffffffff805d5be0>] ip_local_out+0x20/0x30
 [<ffffffff805d64c4>] ip_queue_xmit+0x214/0x410
 [<ffffffff802ad8b0>] ? init_object+0x50/0x90
 [<ffffffff802ae9a8>] ? __slab_alloc+0x148/0x720
 [<ffffffff805eabee>] ? tcp_send_ack+0x2e/0x120
 [<ffffffff805ea855>] tcp_transmit_skb+0x505/0x7a0
 [<ffffffff805eac97>] tcp_send_ack+0xd7/0x120
 [<ffffffff805e79cc>] __tcp_ack_snd_check+0x5c/0xb0
 [<ffffffff805e8803>] tcp_rcv_established+0x803/0x9a0
 [<ffffffff805f01c5>] tcp_v4_do_rcv+0xd5/0x210
 [<ffffffff805f0964>] tcp_v4_rcv+0x664/0x6f0
 [<ffffffff805d14b1>] ip_local_deliver_finish+0xc1/0x170
 [<ffffffff805d15ed>] ip_local_deliver+0x8d/0xa0
 [<ffffffff805d0efb>] ip_rcv_finish+0x12b/0x340
 [<ffffffff805d1363>] ip_rcv+0x253/0x2e0
 [<ffffffff805b078a>] netif_receive_skb+0x28a/0x360
 [<ffffffff8025de89>] ? ktime_get_ts+0x59/0x60
 [<ffffffff805b0d60>] napi_gro_receive+0x60/0x70
 [<ffffffff805b0df2>] process_backlog+0x82/0xd0
 [<ffffffff805af239>] net_rx_action+0x139/0x200
 [<ffffffff8024967d>] __do_softirq+0x6d/0x130
 [<ffffffff8020cc6c>] call_softirq+0x1c/0x30
 [<ffffffff8020e58d>] do_softirq+0x3d/0x80
 [<ffffffff8024936d>] irq_exit+0x7d/0xa0
 [<ffffffff8020e842>] do_IRQ+0xd2/0x1d0
 [<ffffffff8020c513>] ret_from_intr+0x0/0xa
 <EOI> <0> [<ffffffff80213842>] ? default_idle+0x42/0x50
 [<ffffffff80213aa5>] ? c1e_idle+0xa5/0x100
 [<ffffffff8025ef71>] ? atomic_notifier_call_chain+0x11/0x20
 [<ffffffff8020ab3f>] ? cpu_idle+0x5f/0xc0
 [<ffffffff8065f8bc>] ? start_secondary+0x157/0x1ab
Code: 48 83 c1 03 48 83 f9 0f 75 d9 80 7b 08 00 78 1b 5b 41 5c c9 c3 0f 1f 40 00 48 8b 47 08 4c 89 c6 48 8b 7f 10 ff 50 40 66 90 eb aa <0f> 0b eb fe 66 90 55 48 83 c7 18 48 c7 c6 20 14 64 80 48 89 e5 
RIP  [<ffffffff806413ca>] rate_control_get_rate+0xba/0xc0
 RSP <ffff88003d8b7630>
---[ end trace 5e4271fb8fc22a19 ]---
Kernel panic - not syncing: Fatal exception in interrupt
Comment 36 Bob Copeland 2009-03-15 20:27:19 UTC
Well, these warnings are the same as in the original report.  So back to square 1.  I'm still unable to reproduce this here.  Can you try building with pid controller instead of MAC80211_RC_DEFAULT_PID and see if you still get oopses?  Also if it's easy to reproduce, you can try bisecting it from 2.6.28.
Comment 37 Bob Copeland 2009-04-06 22:39:16 UTC
I don't suppose this got magically fixed in 2.6.30-rc did it?  There was one patch which fixed an unmapping error, and another which fixed a skb_free error, either of which ostensibly could cause random memory corruption.

There are some weird things in your dmesg:

pci 0000:02:00.0: BAR 0: can't allocate resource
[...]
pnp 00:09: mem resource (0x0-0xfff) overlaps 0000:02:00.0 BAR 0 (0x0-0xffff),
disabling

but I don't think those really matter, iomap still succeeded in the driver.
Comment 38 Rafael J. Wysocki 2009-04-07 21:32:06 UTC
References : http://lkml.org/lkml/2009/4/6/527
Comment 39 Sasha Medvedev 2009-04-12 20:55:43 UTC
I have system crush after few hours of work, when wireless rate algorithm set to auto. When I use "iwconfig wlan0 rate 12M" system doesn't crush. but today I have received kmesg log about few Gb whith following messages:

WARNING: at net/mac80211/rc80211_minstrel.c:69 minstrel_tx_status+0xa2/0x120 [mac80211]()
Hardware name: 965P-S3
Modules linked in: isofs nls_cp1251 nls_utf8 smbfs nls_base ppp_deflate zlib_deflate zlib_inflate bsd_comp ppp_async crc_ccitt ppp_generic slhc af_packet coretemp it87 hwmon_vid hwmon radeon drm agpgart fuse hid_microsoft usbhid hid usb_storage tuner tea5767 tda8290 tuner_xc2028 xc5000 tda9887 tuner_simple tuner_types mt20xx tea5761 arc4 ecb snd_ice1724 snd_rawmidi snd_seq_device snd_ice17xx_ak4xxx snd_ac97_codec ac97_bus snd_ak4xxx_adda saa7134 floppy snd_ak4114 snd_pcm ath5k i2c_i801 ir_common snd_timer snd_page_alloc v4l2_common mac80211 snd_pt2258 videodev videobuf_dma_sg snd_i2c snd videobuf_core cfg80211 soundcore tveeprom sky2 ehci_hcd uhci_hcd usbcore rtc_cmos rtc_core 8250_pnp rtc_lib 8250 serial_core evdev unix
Pid: 0, comm: swapper Tainted: G        W  2.6.29-gentoo-r1-N1 #1
Call Trace:
 [<c012b3d7>] warn_slowpath+0x87/0xe0
 [<f8dcb6ff>] __ieee80211_rx+0x1af/0x6d0 [mac80211]
 [<f8dd5832>] minstrel_tx_status+0xa2/0x120 [mac80211]
 [<f8db8e9c>] ieee80211_tx_status+0xfc/0x4d0 [mac80211]
 [<f8e27cf7>] ath5k_tasklet_tx+0x1c7/0x360 [ath5k]
 [<c0148948>] clocksource_get_next+0x38/0x40
 [<c0147312>] update_wall_time+0x492/0x8d0
 [<c012ff70>] tasklet_action+0x50/0xd0
 [<c01306c7>] __do_softirq+0x97/0x160
 [<c0117753>] ack_apic_level+0x73/0x290
 [<c01307cd>] do_softirq+0x3d/0x50
 [<c0130a15>] irq_exit+0x75/0xa0
 [<c0105828>] do_IRQ+0x48/0x90
 [<c0143e78>] hrtimer_start+0x18/0x20
 [<c01039e7>] common_interrupt+0x27/0x2c
 [<c010a3ac>] mwait_idle+0x4c/0x60
 [<c0102335>] cpu_idle+0x65/0xa0
---[ end trace ea41fd0ecb4f71fc ]---

Is this related to this bug?
Comment 40 Sasha Medvedev 2009-04-13 09:12:08 UTC
I apply patch from http://lkml.org/lkml/2009/4/6/527. My wifi network rate became permanetly 1Mb/s in auto mod. Setting constant rate works fine.
Comment 41 Bob Copeland 2009-04-21 15:41:38 UTC
Yeah sounds like the same issue.  The patch detects the condition and sets it to lowest rate (usually 1 mb/s) rather than crash.  Can you try capturing a scan of your network using "iw dev wlan0 scan trigger; sleep 10; iw dev wlan0 scan dump > scan.txt"?  I want to see if it has something to do with available AP rates.
Comment 42 Sasha Medvedev 2009-04-22 09:00:13 UTC
"iw dev wlan0 scan trigger" returns
"command failed: Operation not supported (-95)".
But "iwlist scan" returns
wlan0     Scan completed :
          Cell 01 - Address: 4A:A3:B1:4A:9B:4F
                    ESSID:"BUBLENET2"
                    Mode:Ad-Hoc
                    Channel:11
                    Frequency:2.462 GHz (Channel 11)
                    Quality=74/100  Signal level:-56 dBm  Noise level=-82 dBm
                    Encryption key:off
                    IE: Unknown: 00094255424C454E455432
                    IE: Unknown: 010882848B962430486C
                    IE: Unknown: 03010B
                    IE: Unknown: 06020000
                    IE: Unknown: 2A0100
                    IE: Unknown: 2F0100
                    IE: Unknown: 32040C121860
                    IE: Unknown: DD09001018020010000000
                    Bit Rates:1 Mb/s; 2 Mb/s; 5.5 Mb/s; 11 Mb/s; 18 Mb/s
                              24 Mb/s; 36 Mb/s; 54 Mb/s; 6 Mb/s; 9 Mb/s
                              12 Mb/s; 48 Mb/s
                    Extra:tsf=000000003406c707
                    Extra: Last beacon: 10ms ago

I have gentoo-2.6.29-r1 kernel and the following card:
 lspci -s 05:01.0 -vvx
05:01.0 Ethernet controller: Atheros Communications Inc. AR2413 802.11bg NIC (rev 01)
	Subsystem: Atheros Communications Inc. TP-Link TL-WN510G Wireless CardBus Adapter
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 168 (2500ns min, 7000ns max), Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 19
	Region 0: Memory at f8100000 (32-bit, non-prefetchable) [size=64K]
	Capabilities: <access denied>
	Kernel driver in use: ath5k
	Kernel modules: ath5k
00: 8c 16 1a 00 06 00 90 02 01 00 00 02 08 a8 00 00
10: 00 00 10 f8 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 01 50 00 00 8c 16 52 10
30: 00 00 00 00 44 00 00 00 00 00 00 00 0f 01 0a 1c

Here is part of my kernel config

CONFIG_WIRELESS=y
CONFIG_CFG80211=m
# CONFIG_CFG80211_REG_DEBUG is not set
CONFIG_NL80211=y
CONFIG_WIRELESS_OLD_REGULATORY=y
CONFIG_WIRELESS_EXT=y
# CONFIG_WIRELESS_EXT_SYSFS is not set
CONFIG_LIB80211=m
# CONFIG_LIB80211_DEBUG is not set
CONFIG_MAC80211=m

#
# Rate control algorithm selection
#
CONFIG_MAC80211_RC_MINSTREL=y
# CONFIG_MAC80211_RC_DEFAULT_PID is not set
CONFIG_MAC80211_RC_DEFAULT_MINSTREL=y
CONFIG_MAC80211_RC_DEFAULT="minstrel"
CONFIG_MAC80211_MESH=y
CONFIG_MAC80211_LEDS=y
# CONFIG_MAC80211_DEBUG_MENU is not set
# CONFIG_WIMAX is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set

and some dmesg output

ath5k 0000:05:01.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
ath5k 0000:05:01.0: registered as 'phy0'
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
saa7130/34: v4l2 driver version 0.2.14 loaded
wmaster0 (ath5k): not using net_device_ops yet
phy0: Selected rate control algorithm 'minstrel'
wlan0 (ath5k): not using net_device_ops yet
ath5k phy0: Atheros AR2413 chip found (MAC: 0x78, PHY: 0x45)
-----------------
wlan0: deauthenticating by local choice (reason=3)
ath5k phy0: bf=f65121c0 bf_skb=(null)
wlan0: Trigger new scan to find an IBSS to join
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
wlan0: Trigger new scan to find an IBSS to join
__ratelimit: 2 callbacks suppressed
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
ath5k phy0: bf=f65121c0 bf_skb=(null)
wlan0: Creating new IBSS network, BSSID 4a:a3:b1:4a:9b:4f
wlan0: deauthenticating by local choice (reason=3)

By the way I receive system hang on 2.6.29 kernel with constant rate 12Mb, so just now I'm using 2.6.27 kernel, which have no auto rate.
Comment 43 John W. Linville 2009-06-01 17:27:44 UTC
Bob, did you get a chance to look at this?
Comment 44 Bob Copeland 2009-06-01 18:35:35 UTC
Yeah, bug 13000 sounds like the same and has a promising lead (AP with weird rate sets), I just now created a setup to test it.  Also there were a couple of patches in 2.6.30 which may have a positive effect (Jiri's bugfix for minstrel that broke every single band card did fix some memory corruption, and mine to flush the queues when changing channels can keep it from reporting bogus rates).

http://lkml.org/lkml/2009/4/7/636 is a band-aid patch but someone reported it still crashed with it.
Comment 45 Bob Copeland 2009-06-05 14:13:03 UTC
Ok, I could reproduce this at last.  Patch is here:
http://lkml.org/lkml/2009/6/5/269
Comment 46 Rafael J. Wysocki 2009-06-07 12:09:38 UTC
Patch : http://patchwork.kernel.org/patch/28210/
Comment 47 Nick Kossifidis 2009-09-03 15:50:51 UTC
Bug is still there, i reproduced it on a VIA C7 with 2 cards in ad-hoc mode @ fixed rate (54M).

After i handled the NULL rate on base.c (ath5k_txbuf_setup) by returning -EINVAL and drop the packet i further debuged this and found that if i set the rate after the 2 cards associate (iwconfig wlan0 rate 54M) then ieee80211_get_tx_rate doesn't return NULL anymore and everything works fine. So it seems that fixed rate info is lost after the cards associate.

Also ieee80211_get_tx_rate returns NULL (if i don't set the rate after association) only for data frames, beacons work fine.

Don't have time to debug this further, hope it helps.
Comment 48 Bob Copeland 2009-09-03 20:38:16 UTC
Well, must be more than one bug at play then.  Can you verify that my fix to minstrel is in your kernel?
Comment 49 Nick Kossifidis 2009-09-03 20:48:24 UTC
Yup, i used the current wireless-testing and checked for your fix.

Note You need to log in before you can comment on or make changes to this bug.