Bug 63101

Can you bisect?

Going back to commit 95167aad6761ec8a0fc76506ed00439483208ee1

- GPU lockup only, somewhat recoverable error, gpu locked in highest power state.
lots of errors like 
[  523.602078] radeon 0000:00:01.0: ffff8800034ebc00 pin failed
And
[  546.612391] WARNING: CPU: 0 PID: 2891 at drivers/gpu/drm/drm_mm.c:257 ttm_bo_man_put_node+0x23/0x3d [ttm]()

No other errors, i can even launch other games (tried DoD). The CPU is severely throttled (this is with 4 threads compile, the CPU is rated at 3.2 GHz):

# cpufreq-aperf -o
CPU	Average freq(KHz)	Time in C0	Time in Cx	C0 percentage
000	1536000			00 sec 996 ms	00 sec 003 ms	99
001	1536000			00 sec 984 ms	00 sec 015 ms	98
002	1536000			00 sec 970 ms	00 sec 029 ms	97
003	1536000			00 sec 985 ms	00 sec 014 ms	98

probably because of the GPU being in the highest state.
Attached dmesg. I continue bisecting.

Created attachment 111221 [details]
dmesg of 95167aad6761ec8a0fc76506ed00439483208ee1

X crashed right after i posted the previous comment. Attached xorg log with the crash.

Created attachment 111231 [details]
x crashlog

3.12 RC4 made my system laggy but RC5 seems has fixed the problem.

I run the latest git, updated until commit 83f11a9cf2578b104c0daf18fc9c7d33c3d6d53a and i still have this issue.

The problem was introduced somewhere between

a2ac07fe292ea41296049dfdbfeed203e2467ee7 (v3.12-rc3-71-ga2ac07f)
and
0bfdbf0e79ab20394c932f27f6d3a34b757035ef (v3.12-rc3-276-g0bfdbf0).
I ran in some issue when bisecting, i was thrown back to between rc1 and rc2 when checking out certain commits which were shown as rc3+ in the commit log.

Ok, i finished bisecting.
According to git, 6d15ee492809d38bd62237b6d0f6a81d4dd12d15 (v3.12-rc3-267-g6d15ee4) is the first bad commit.

Reverted 6d15ee492809d38bd62237b6d0f6a81d4dd12d15, bit didnt help, i still get the lockup.

Build v3.12-rc3-265-gafe05d4 (until commit afe05d41e2c25ca3e047f9c7e5341bda553a932f) is working.

This issue has to do with the radeon driver, because if i dont append radeon.dpm=1 to the kernel boot options, TF2 launches even with latest 3.12 rc5+ kernel (but it has dismal performance).

Also, the CPU voltage is correct without the dpm option - between 0.9-1.34v, instead of 0.65-1.22 as it is with dpm.

(In reply to Kertesz Laszlo from comment #10)
> This issue has to do with the radeon driver, because if i dont append
> radeon.dpm=1 to the kernel boot options, TF2 launches even with latest 3.12
> rc5+ kernel (but it has dismal performance).

If it was dpm specific you should have mentioned that.  Make sure your kernel has this patch:
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4076a65544e2de310cbf4eaadb13ee15bbfaaf4f

(In reply to Alex Deucher from comment #11)
> (In reply to Kertesz Laszlo from comment #10)
> > This issue has to do with the radeon driver, because if i dont append
> > radeon.dpm=1 to the kernel boot options, TF2 launches even with latest 3.12
> > rc5+ kernel (but it has dismal performance).
> 
> If it was dpm specific you should have mentioned that.  Make sure your
> kernel has this patch:
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/
> ?id=4076a65544e2de310cbf4eaadb13ee15bbfaaf4f

Sorry about that. I have the dpm option there since it appeared, i totally forgot about it - now i started the computer in single user mode which doesnt have that option then i proceeded to runlevel 2. This way i saw this.

I have that patch, i have the linus kernel cloned from git (now at version v3.12-rc5-123-g04919af). I even re cloned it recently.

Also please take a look at bug #62861 - it is related to the dpm code too.

I reverted commit 4076a65544e2de310cbf4eaadb13ee15bbfaaf4f, but still crashes with dpm.

Reverting this commit changed back the CPU voltage handling as it was before.

It seems that the CPU power management is best for APUs in its latest form - if dpm is not used or with 4076a65544e2de310cbf4eaadb13ee15bbfaaf4f reverted and dpm activated, the CPU is scaled with voltages between 0.9 and 1.34, using all frequencies, but if all cores are used, after a while the CPU becomes throttled and the effective frequencies go down to around ~3 GHz or even lower, in some cases even ~ 2GHz or lower (temps go as high as 60C withut dpm and ~56C with dpm).

With the new code we have ~0.65 - 1.22 volts, the frequencies are maxed out at 3.168GHz (rarely 3.2) coupled with a lower max temperature(max 51C, mostly 49), but the frequencies are stable, i can compile a kernel with 4 threads and there is absolutely no throttling. Only thing is missing its the highest voltage level (1.34) which enables turbo mode.

I did some more testing. Turns out that i can get into the game if i set the texture quality to "low". But it still crashes afterwards at some point - usually after ~20 or so minutes.

Sometimes io have tons of messages like this:

Oct 22 13:06:00 laca-desktop kernel: [  561.631586] [TTM] Buffer eviction failed
Oct 22 13:06:00 laca-desktop kernel: [  561.631594] radeon 0000:00:01.0: ffff88005c4e8400 pin failed
Oct 22 13:06:00 laca-desktop kernel: [  561.631596] [drm:radeon_crtc_page_flip] *ERROR* failed to pin new rbo buffer before flip

or

Oct 22 13:06:00 laca-desktop kernel: [  561.743974] radeon 0000:00:01.0: ffff88005c4e8400 pin failed
Oct 22 13:06:00 laca-desktop kernel: [  561.743977] [drm:radeon_crtc_page_flip] *ERROR* failed to pin new rbo buffer before flip


With the latest git kernel after the crash gpu lockup message the computer is still somewhat usable, the GPU is throttled to max and opengl apps fail with "Bus Error" (both tf2 and glxgears report this error). Sometimes after a while the computer freezes if left in this state.

Created attachment 113871 [details]
dmesg, kernel next-20131108 until it froze

Tried the next-20131108 kernel.
Still crashes, but now the GPU softresets, sometimes even 5 or so times in a row. If it resets multiple times, usually freezes the X server, only the mouse is moving. Sometimes it resets 1 or 2 times first then the X server is usable (it resets afterwards out of the blue too sometimes, but the X session isnt affected).
But if i try to launch TF2 again, it locks the X server and sometimes freezes the whole computer (logs dont show anything). Most times i was able to reboot with the sysrq keys.

PS. Weird thing, i once was able to play TF2 with the 3.12 final kernel too. Then it was perfectly playable, launched it a few times in a row and had no issues whatsoever. I did nothing out of the ordinary, the session was running for a day or 2 after which i launched it, previously i compiled stuff, tested stuff in virtual machines, watched movies, browsed etc.
However after reboot i couldnt reproduce this. No software (mesa etc) was changed, but after reboot, it crashed as before.

It seems that it does work after a day or so uptime. I had 1 day and 6 or 8 hours uptime and i could play the game just fine. Previously i did use programs that used quite some memory (eclipse, virtual machines, browsers).

Any ideas why this might be happening?

Created attachment 114241 [details]
disable various dpm features

You might try disabling trinity dpm features to see if you can narrow down what's causing the stability problems.  Try the attached patch and see if it helps, and if so, try enabling additional features to narrow down the problem.

Tested with 3.12.0-next-20131111, started the computer and launched Steam and TF2 right away.

Activating enable_nbps_policy to enable_gfx_clock_gating worked well.

The enable_mg_clock_gating, enable_gfx_dynamic_mgpg, override_dynamic_mgpg options
gave gpu lockups and softresets (same as the above attachment), mostly after i entered the map, but sometimes after the reset the game was resumed and could play without issues.
Combining them with uvd_dpm seems to be the killer factor though.

This is what seems to work well so far (uvd_dpm seems to not create problems if the mgpg and mg options are disabled:

pi->enable_bapm = false;
	pi->enable_nbps_policy = true; /*mychange*/
	pi->enable_sclk_ds = true;
	pi->enable_gfx_power_gating = true;
	pi->enable_gfx_clock_gating = true;
	pi->enable_mg_clock_gating = false;
	pi->enable_gfx_dynamic_mgpg = false; /* ??? */
	pi->override_dynamic_mgpg = false;
	pi->enable_auto_thermal_throttling = true;
	pi->voltage_drop_in_dce = false; /* need to restructure dpm/modeset interaction */
	pi->uvd_dpm = true; /* ??? */

Enabling any of enable_mg_clock_gating, enable_gfx_dynamic_mgpg, override_dynamic_mgpg and uvd_dpm will cause the GPU softreset, TF2 to crash and most times an unusable desktop, sometimes with garbled image (composed of the real desktop tiles with some of the TF2 tiles. Exactly the behavior described with the unmodified 3.12.0-next-20131111 kernel.

Created attachment 114671 [details]
adjust dpm features for stability

This patch should do the trick.  With this it sounds like dpm is finally stable on your system.

The 

pi->uvd_dpm = true; /* ??? */

line above wasnt a typo. I was trying to say that 

- ANY of the enable_mg_clock_gating, enable_gfx_dynamic_mgpg, override_dynamic_mgpg options enabled caused softresets which most of the time were recovered IF uvd_dpm was false.
- But if ANY of the above 3 options was true AND uvd_dpm true, i got the worst kind of behavior which most times rendereed the desktop unusable. 

- If the 3 options were false AND uvd_dpm true, it works.

I have the kernel (next-20131113) compiled with that exact piece of code i pasted in my previous comment right now and everything works fine so far, all games i tried, hardware decoding with xbmc and the webgl demos from chromeexperiments (on Seamonkey and Firefox, Chrome seems to hate mesa/radeon).

Created attachment 114681 [details]
adjust dpm features for stability v2

Thanks for clarifying.  New patch attached.