ATA part looks very suspicious.  It's an optiarc drive attached to ahci.

 ata5: illegal qc_active transition (00000001->00000003)
 ata5.00: exception Emask 0x2 SAct 0x0 SErr 0x0 action 0x6 frozen
 sr 4:0:0:0: [sr0] CDB: cdb[0]=0x46: 46 01 00 00 00 00 00 01 64 00
 ata5.00: cmd a0/00:00:00:64:01/00:00:00:00:00/a0 tag 0 pio 16740 in
          res 50/00:03:00:08:00/00:00:00:00:00/a0 Emask 0x2 (HSM violation)
 ata5.00: status: { DRDY }
 ata5: hard resetting link
 ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
 ata5.00: configured for UDMA/100
 ata5: EH complete

libata never issues more than one non-NCQ commands concurrently and the log accordingly shows only one in-flight command.  The above can only happen if somehow bit 1 is magically set in PORT_CMD_ISSUE register of the controller.  Can you please turn on CONFIG_PRINTK_TIME and attach the resulting log?  Also, is the drive accessible after the above failure?

Now I hit this issue again, it looked like fs was read-only few seconds after problems (cause I tried write dmesg) and then completly disappeared (ls,dmesg,cd not worked). I again hit before that issue information it was disconnected from wifi).

It happend (again) after running konqueror, so it might be somehow connected to wifi (because once it does when I run Kopete).

But maybe it's because I noticed, my DVD-RW have some problem with burning CD and when I tried burn it, my whole lappy started vibrate and rotation of cd wasn't right. But this problem not appeared anytime when I tried burn. Maybe is DVD-RW magically damaged.

I'm afraid I not be able get more logs, because it's very random and it doesn't allow me save logs... I add time printk feature and hope I can write to paper data from log...

If you have a second machine, the best way would be setting up netconsole or serial console.

Created attachment 24232 [details]
photo_of_screen.jpg

I have only ability use ssh over WiFi (HTC Dream). I have only picture, of my screen with dmesg (not complete ;-( ). Next time I try get more info.

Now I hit this comment - http://lkml.org/lkml/2009/12/19/82 and I think it's possible that I have same problem with 2.6.32 with ahci driver. Now testing with pci=nomsi, I hope it help.

Created attachment 24236 [details]
photo_of_screen2.jpg

I'm not sure, if it is same issue, but behavior is similiar:
I tried run 40 instances of konqueror and get this. Of course dmesg wasn't saved. It's configuration with pci=nomsi.

Unfortunately, from the log, I can't tell what went wrong.  The ATA part which ended up on the last screen doesn't say anything about why and what went wrong.  Can you plug in a USB stick and save dmesg output there?  ie. keep "while true; do dmesg -c >> /mnt/USB_STICK/dmesg.out; sleep 1; done" running.  You log will most likely be on the usb stick after disk checks out.

Created attachment 24244 [details]
picture_random_driver_damage.jpg

So, I tried (even with sync option) and look like there is no connection between sata subsystem, because it telling something about problem with usb. Look like something inside kernel start damaging actually used structures etc. maybe WiFi driver, but he fail after that problem, so probably he have something damaged too.

pci=nomsi of cource doesn't affect this issue.

And I forgot - this state I'm able avoid mostly with running lot of konquerors, but when compiling glibc/KDE 4.4 or anything else, it's stable. Maybe it's because konqueror run lot of proccess and kernel can't handle it (my guess)

Without kernel log, it's really difficult to tell what's going on.  Any chance you can extract log off that machine (serial/netconsole, usb stick or separate disk, ssh session which pipes kernel messages off the machine, whatever...)?

serial/netconsole impossible, usb stick failed (same way as my hdd was disconnected with lot of errors), separate disk end probably same way, ssh is probably correct solution - I try, when I get near to some pc)

so, 2.6.33-rc2 solved my problem (I tried running of xx konquerors + kopete and it worked).

Problem probably remains on 2.6.32

If 2.6.33-rc2 fixes your problem I don't think libata would have much to do with it.  AFAICT, there's no libata change which could cause that type of behavior difference.  It looks like something more fundamental is broken.

So I'm afraid I hit this problem with 2.6.33-rc2 (exactly this problem) again :-( I try do screenshot next time more quickly.

On Wednesday 30 December 2009, okias wrote:
> Yes, it still valid - even on 2.6.33-rc2
> 
> 2009/12/29 Rafael J. Wysocki <rjw@sisk.pl>:
> > This message has been generated automatically as a part of a report
> > of regressions introduced between 2.6.31 and 2.6.32.
> >
> > The following bug entry is on the current list of known regressions
> > introduced between 2.6.31 and 2.6.32.  Please verify if it still should
> > be listed and let me know (either way).
> >
> >
> > Bug-Entry       : http://bugzilla.kernel.org/show_bug.cgi?id=14817
> > Subject         : When is system under load, then freeze/HD fail
> > Submitter       : okias <d.okias@gmail.com>
> > Date            : 2009-12-15 11:12 (15 days old)

I don't think this problem is specific to or caused by ata.  Something at lower level seems broken.  The only possibly related change would be enabling of FPDMA AA but I don't think it's related.  Just in case, does kernel paramter "libata.force=noncq" make any difference?

I think it would be best to try bisection and find out which commit between 31 and 32 caused the problem but the problem can't be reproduce reliably, right?

it's not related with ATA at all. Even if I did "while
true; do dmesg -c >> /mnt/USB_STICK/dmesg.out; sleep 1; done" it writed errors related with USB stack (and log wasn't writed on usb stick), so something much more lowlevel is broken. Another problem is, there is really hard reproduce it on 2.6.33-rc2 (I didn't hit that problem for few days).

Created attachment 24446 [details]
first_part

Please look at it - 2.6.33-rc2-git

Created attachment 24447 [details]
second_part.jpg

cc'ing HPET people and Andrew.  The system is failing in weird ways after triggering a WARN in HPET path.  Any ideas?

cc x86 people, johnstul.

That hpet warning is quite common and the frequency seems to have increased around 2.6.32.  I don't think I've seen an explanation of what would cause it to come out, nor why it might have become more common.  Having it first come out after 9000 seconds uptime seems a bit alarming.

I doubt if an HPET glitch would cause ata to fall over though.  Perhaps something more upstream has failed (interrupts?, bus config?) and that then caused both hpet and ata to fail.

Can you still reproduce this with current mainline kernels?

It seems to be ok.