Bug 199749

Summary: amdgpu on Ryzen 2400G freeze randomly
Product: Drivers Reporter: muelladdi
Component: Video(DRI - non Intel)Assignee: drivers_video-dri
Status: NEW ---    
Severity: normal CC: andrey.grodzovsky, chewi, pmenzel+bugzilla.kernel.org, song.fc
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 4.17 RC2 Subsystem:
Regression: No Bisected commit-id:
Attachments: Trace process
Trace process 2
Kernel log from serial port when it freezes

Description muelladdi 2018-05-17 09:21:39 UTC
System video freezes randomly during the day while working on the system normally (no 3D).
Logfile shows only:

May 17 11:18:00 ws01 kernel: [11831.268044] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, last signaled seq=2081253, last emitted seq=2081256
May 17 11:18:00 ws01 kernel: [11831.268051] [drm] No hardware hang detected. Did some blocks stall?


If I can assist any further, please tell.
Comment 1 Andrey Grodzovsky 2018-06-12 21:27:50 UTC
Can you load kernel with grub command line amdgpu.vm_update_mode=3 to force CPU VM update mode and see if this makes the issue go away ?


Andrey
Comment 2 muelladdi 2018-06-13 07:17:17 UTC
Added the grub cmdline and will investigate
Comment 3 notsyncing 2018-06-16 15:47:54 UTC
Hello, I think I'm experiencing the same problem here. My Ryzen 5 2400G system freezes often, especially under some high cpu and disk activity, even after the "Typical Current Idle" UEFI workaround. Sometimes I could reboot with sysrq, most of the times I need a hard reset. The freezes never leave a message in kernel log until today, after 2 months I built my machine. 


6月 16 23:29:00 sfc-DESKTOP kernel: [drm] No hardware hang detected. Did some blocks stall?
6月 16 23:29:00 sfc-DESKTOP kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, last signaled seq=1217227, last emitted seq=1217229
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0:   at page 0x000000011241a000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0:   at page 0x0000000112414000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0:   at page 0x0000000112416000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0:   at page 0x0000000112410000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0:   at page 0x0000000112412000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0:   at page 0x000000011241f000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0:   at page 0x0000000112425000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0:   at page 0x0000000112427000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0:   at page 0x0000000112429000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00101031
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0:   at page 0x0000000112421000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)

The HDD led still flashes after this freeze, but in the previous freezes, it won't flash at all.

using kernel 4.17.0-041700-generic on Ubuntu 18.04, mesa 18.1.1-0~b~padoka0. CPU runs at stock 3.6GHz, memory 16G DDR4, running at 2133MHz, no overclocking.
Comment 4 James Le Cuirot 2018-06-16 16:29:24 UTC
Not sure if this is related to bug #199653, which concerns freezing on the 2500U and 2700U. It hasn't received any attention from AMD or other kernels devs but there is more information that could potentially be useful. I tried amdgpu.vm_update_mode=3 but that didn't help.
Comment 5 Andrey Grodzovsky 2018-06-16 18:24:58 UTC
Those two bugs are unrelated.
Comment 6 Andrey Grodzovsky 2018-06-16 18:31:40 UTC
(In reply to James Le Cuirot from comment #4)
> Not sure if this is related to bug #199653, which concerns freezing on the
> 2500U and 2700U. It hasn't received any attention from AMD or other kernels
> devs but there is more information that could potentially be useful. I tried
> amdgpu.vm_update_mode=3 but that didn't help.

Are you seeing sdma0 timeout message when the system freezes like muelladi above ? I expect 
amdgpu.vm_update_mode=3 to help only in that case.
Comment 7 James Le Cuirot 2018-06-16 18:59:20 UTC
(In reply to Andrey Grodzovsky from comment #6)
> Are you seeing sdma0 timeout message when the system freezes like muelladi
> above ? I expect 
> amdgpu.vm_update_mode=3 to help only in that case.

I haven't been able to get any information as I have been unable to access the system following these freezes. Judging by the output from the other reporter, they do indeed seem unrelated. Sorry for the noise but some attention on that issue would be hugely appreciated. It's so bad, I've considered selling the laptop.
Comment 8 notsyncing 2018-06-17 09:37:00 UTC
Today I tried the amdgpu.vm_update_mode=3 option, and my computer still freezed. This time no log was recorded.
Comment 9 notsyncing 2018-06-17 09:41:47 UTC
I found I'm able to reproduce the freeze when I'm compiling two Android ROMs at the same time while browsing or something else. It will freeze in at most 3 hours. I have a 300W PSU, and 2 SSD + 1 HDD storages. Android sources are stored in the HDD. This time I can reboot with sysrq, seems not a PSU fault.
Comment 10 muelladdi 2018-06-18 06:16:38 UTC
So far I have not had any more freezes during normal, non-3D work.
Comment 11 Michel Dänzer 2018-06-18 08:55:22 UTC
Other Raven Ridge users have reported that updating to the current microcode files from https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/amdgpu has fixed stability issues.

Make sure your system BIOS and CPU microcode are up to date as well.
Comment 12 notsyncing 2018-06-21 13:34:45 UTC
I updated to the current microcode files at linux-firmware.git. Today I replicated the same workload, and my computer freezed in just 5 minutes! Even sysrq does not work. I have to do a hard reset.

Still, no log was recorded.

I have an Asrock AB350M-Pro4 motherboard, with latest UEFI version L4.82.
Comment 13 Michel Dänzer 2018-06-21 16:23:29 UTC
(In reply to notsyncing from comment #12)
> I updated to the current microcode files at linux-firmware.git.

Did you update the microcode files in the initrd as well?
Comment 14 notsyncing 2018-06-22 13:29:46 UTC
(In reply to Michel Dänzer from comment #13)
> Did you update the microcode files in the initrd as well?

I just copied the files to /lib/firmware. I'll try with update-initramfs.
Comment 15 notsyncing 2018-06-24 06:12:21 UTC
Still freezed under two Android source compilation + 2 intellij idea + 10 firefox tabs + EVE online playing after 3 hours. Sysrq does not work, need hard reset. No log was recorded.
Comment 16 Andrey Grodzovsky 2018-06-25 14:30:34 UTC
What kernel version are you using ?

We can try and figure out what was the last commands in HW before you experienced the page fault.

You can clone and install our register analyzer from here - https://cgit.freedesktop.org/amd/umr/

Then launch your X with ENV variable GALLIUM_DDEBUG=always to dump all the 3D commands into files in ~/ddebug_dumps/

Run your workload.

After you again experience the GPU page fault please provide the following outputs 

sudo umr -lb
sudo umr -O verbose,follow_ib -R gfx[.]
sudo umr -O bits -wa
sudo umr -O many,bits  -r*.*.mmGRBM_STATUS
sudo umr -O many,bits  -r *.*.HEADER_DUMP
sudo umr -O many,bits  -r *.*.CP_EOP
Comment 17 notsyncing 2018-06-26 12:03:58 UTC
(In reply to Andrey Grodzovsky from comment #16)
> Then launch your X with ENV variable GALLIUM_DDEBUG=always to dump all the
> 3D commands into files in ~/ddebug_dumps/

Would you mind telling me how to add this variable? I googled and cannot find any information. Should I add it to /etc/X11/Xsession or something else? Thanks very much!

I'm on kernel 4.17.2-041702-generic, Ubuntu 18.04, mesa 18.1.1-1ubuntu1~18.04.0~ppa1
Comment 18 notsyncing 2018-06-26 12:04:36 UTC
BTW, I'm using KDE plasma 5.13 with sddm.
Comment 19 Andrey Grodzovsky 2018-06-26 12:40:17 UTC
(In reply to notsyncing from comment #18)
> BTW, I'm using KDE plasma 5.13 with sddm.

You just prepend this before command to start your graphic stack. You need to manually run your graphic stack from command line and add this before. E.G. for me on Ubuntu I will disable graphics launch on boot from GRUB. Then from terminal I will run
GALLIUM_DDEBUG=always service lightdm start
Comment 20 Andrey Grodzovsky 2018-06-26 12:40:27 UTC
(In reply to notsyncing from comment #18)
> BTW, I'm using KDE plasma 5.13 with sddm.

You just prepend this before command to start your graphic stack. You need to manually run your graphic stack from command line and add this before. E.G. for me on Ubuntu I will disable graphics launch on boot from GRUB. Then from terminal I will run
GALLIUM_DDEBUG=always service lightdm start
Comment 21 notsyncing 2018-06-26 14:58:04 UTC
(In reply to Andrey Grodzovsky from comment #20)
> You just prepend this before command to start your graphic stack. You need
> to manually run your graphic stack from command line and add this before.
> E.G. for me on Ubuntu I will disable graphics launch on boot from GRUB. Then
> from terminal I will run
> GALLIUM_DDEBUG=always service lightdm start

I compiled and installed umr and executed "GALLIUM_DDEBUG=always service sddm start" on a root tty, and no "ddebug_dumps" directory was found in /root or in my home directory after desktop started. Is that normal?
Comment 22 Andrey Grodzovsky 2018-06-26 16:55:23 UTC
(In reply to notsyncing from comment #21)
> (In reply to Andrey Grodzovsky from comment #20)
> > You just prepend this before command to start your graphic stack. You need
> > to manually run your graphic stack from command line and add this before.
> > E.G. for me on Ubuntu I will disable graphics launch on boot from GRUB.
> Then
> > from terminal I will run
> > GALLIUM_DDEBUG=always service lightdm start
> 
> I compiled and installed umr and executed "GALLIUM_DDEBUG=always service
> sddm start" on a root tty, and no "ddebug_dumps" directory was found in
> /root or in my home directory after desktop started. Is that normal?

I checked myself, you need to  use this variable with specific graphic application you are running, can you pinpoint what graphic work is going on while you have this faults ?
Comment 23 notsyncing 2018-06-27 11:44:58 UTC
In fact there is no specified graphic work. I just put an Android source compilation running and go to bed, besides only firefox and dolphin running. The next morning, I found the machine has already freezed. Still no log. The log I posted is just a lucky one.

Would you mind telling me if the umr had some system-wide debugging methods? Or should I make a netconsole to see if there was anything?
Comment 24 Andrey Grodzovsky 2018-06-27 13:39:06 UTC
(In reply to notsyncing from comment #23)
> In fact there is no specified graphic work. I just put an Android source
> compilation running and go to bed, besides only firefox and dolphin running.
> The next morning, I found the machine has already freezed. Still no log. The
> log I posted is just a lucky one.
> 
> Would you mind telling me if the umr had some system-wide debugging methods?
> Or should I make a netconsole to see if there was anything?

UMR is system wide any way, it's memory/registers/HW debugging tool. you can provide the UMR outputs I asked before once the freeze happened assuming you still have SSH access (which seems like you don't).

Since the memory faults you experience are clearly due to some graphic rendering activity maybe you could try to isolate the app which triggers it, repeat what you do but close both firefox and dolphin and check if this still happens. If not try to find which of them was causing this and then we can run it with MESA debug flags.
Comment 25 Andrey Grodzovsky 2018-06-29 19:59:23 UTC
Created attachment 277059 [details]
Trace process
Comment 26 Andrey Grodzovsky 2018-06-29 20:03:18 UTC
Created attachment 277061 [details]
Trace process 2

Attached 2 patches if applied to your kernel should tell which process caused the VM_FAULT please also launch your kernel from GRUB with following parameter - 
amdgpu.vm_fault_stop=2
Comment 27 notsyncing 2018-06-30 03:09:38 UTC
After 3 days, I managed to reproduce it again with 2 android compilation and firefox with kernel parameter mem=4096m (I have 16GB memory). I found that it's easier to reproduce when the memory is full.

sfc@sfc-DESKTOP:~$ sudo umr -lb
        raven1.gfx91
        raven1.vcn10
        raven1.dcn10
        raven1.nbio70
        raven1.sdma041
        raven1.hdp40
        raven1.oss40
        raven1.mmhub91
        raven1.mp100
sfc@sfc-DESKTOP:~$ sudo umr -O verbose,follow_ib -R gfx[.]
error: Unknown option [follow_ib]
sfc@sfc-DESKTOP:~$ sudo umr -O bits -wa
No active waves!
sfc@sfc-DESKTOP:~$ sudo umr -O many,bits  -r*.*.mmGRBM_STATUS
[ERROR]: Unknown option <-r*.*.mmGRBM_STATUS>
sfc@sfc-DESKTOP:~$ sudo umr -O many,bits  -r *.*.HEADER_DUMP
gfx91.mmCP_MEC_ME1_HEADER_DUMP => 0xc0000e00
        .HEADER_DUMP[0:31]                                               == 3221229056 (0xc0000e00)
gfx91.mmCP_MEC_ME2_HEADER_DUMP => 0xdef0def0
        .HEADER_DUMP[0:31]                                               == 3740327664 (0xdef0def0)
gfx91.mmCP_ME_HEADER_DUMP => 0xc0004200
        .ME_HEADER_DUMP[0:31]                                            == 3221242368 (0xc0004200)
gfx91.mmCP_PFP_HEADER_DUMP => 0xffff1000
        .PFP_HEADER_DUMP[0:31]                                           == 4294905856 (0xffff1000)
gfx91.mmCP_CE_HEADER_DUMP => 0xffff1000
        .CE_HEADER_DUMP[0:31]                                            == 4294905856 (0xffff1000)
sfc@sfc-DESKTOP:~$ sudo umr -O many,bits  -r *.*.CP_EOP
gfx91.mmCP_EOPQ_WAIT_TIME => 0x0000052c
        .WAIT_TIME[0:9]                                                  ==      300 (0x0000012c)
        .SCALE_COUNT[10:17]                                              ==        1 (0x00000001)
gfx91.mmCP_EOP_DONE_ADDR_LO => 0x00609000
        .ADDR_LO[2:31]                                                   ==  1582080 (0x00182400)
gfx91.mmCP_EOP_DONE_ADDR_HI => 0x000000f5
        .ADDR_HI[0:15]                                                   ==      245 (0x000000f5)
gfx91.mmCP_EOP_DONE_DATA_LO => 0x000008e4
        .DATA_LO[0:31]                                                   ==     2276 (0x000008e4)
gfx91.mmCP_EOP_DONE_DATA_HI => 0x00000000
        .DATA_HI[0:31]                                                   ==        0 (0x00000000)
gfx91.mmCP_EOP_LAST_FENCE_LO => 0x000008e4
        .LAST_FENCE_LO[0:31]                                             ==     2276 (0x000008e4)
gfx91.mmCP_EOP_LAST_FENCE_HI => 0x00000000
        .LAST_FENCE_HI[0:31]                                             ==        0 (0x00000000)
gfx91.mmCP_EOP_DONE_EVENT_CNTL => 0x00038060
        .WBINV_TC_OP[0:6]                                                ==       96 (0x00000060)
        .WBINV_ACTION_ENA[12:17]                                         ==       56 (0x00000038)
        .CACHE_POLICY[25:25]                                             ==        0 (0x00000000)
        .EXECUTE[28:28]                                                  ==        0 (0x00000000)
gfx91.mmCP_EOP_DONE_DATA_CNTL => 0x40010000
        .DST_SEL[16:17]                                                  ==        1 (0x00000001)
        .INT_SEL[24:26]                                                  ==        0 (0x00000000)
        .DATA_SEL[29:31]                                                 ==        2 (0x00000002)
gfx91.mmCP_EOP_DONE_CNTX_ID => 0x00000000
        .CNTX_ID[0:31]                                                   ==        0 (0x00000000)

ddebug_dumps:

---
Command: /usr/lib/firefox/firefox 
Driver vendor: X.Org
Device vendor: AMD
Device name: AMD RAVEN (DRM 3.25.0, 4.17.2-041702-generic, LLVM 6.0.0)

Remainder of driver log:
---

I tried netconsole and could not get it working. I bought a serial converter and it's on the way. When it delivered, I will try getting the log from serial port. Then I could try your patches because currently the logs did not get persisted at all. Thanks for your patches.
Comment 28 notsyncing 2018-06-30 03:10:06 UTC
btw, these umr commands are executed after reboot.
Comment 29 notsyncing 2018-06-30 04:17:36 UTC
I just upgraded to mesa 18.1.3 and kernel 4.17.3. I ran firefox with GALLIUM_DDEBUG after reboot. It produces these after I opened some tabs and firefox stopped responsing.

---
Gallium debugger active. Logging all calls.
Hang detection timeout is 1000ms.
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000000
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000001
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000002
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000003
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000004
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000005
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000006
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000007
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000008
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000009
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000010
dd: can't create a directory (13)
dd: can't create a directory (13)
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000011
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000012
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000013
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000014
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000015
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000016
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000017
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000018
GPU hang detected, collecting information...

Draw #   driver  prev BOP  TOP  BOP  dump file
-------------------------------------------------------------
8         YES      NO      NO   NO   dd: can't create a directory (13)
fopen failed

Done.
Sandbox: seccomp sandbox violation: pid 5057, tid 5268, syscall 162, args 140631883336544 7 140633020008624 140632174061559 7 140632344285188.
dd: Aborting the process...
[Parent 4748, Gecko_IOThread] WARNING: pipe error (72): 连接被对方重设: file /build/firefox-m9FtQy/firefox-60.0.2+build1/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 353
[Parent 4748, Gecko_IOThread] WARNING: pipe error (113): 连接被对方重设: file /build/firefox-m9FtQy/firefox-60.0.2+build1/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 353

###!!! [Parent][MessageChannel] Error: (msgtype=0x15007F,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv


---

The umr commands are:

---
sfc@sfc-DESKTOP:~$ sudo umr -lb
        raven1.gfx91
        raven1.vcn10
        raven1.dcn10
        raven1.nbio70
        raven1.sdma041
        raven1.hdp40
        raven1.oss40
        raven1.mmhub91
        raven1.mp100
sfc@sfc-DESKTOP:~$ sudo umr -O verbose,follow_ib -R gfx[.]
error: Unknown option [follow_ib]
sfc@sfc-DESKTOP:~$ sudo umr -O bits -wa
No active waves!
sfc@sfc-DESKTOP:~$ sudo umr -O many,bits  -r*.*.mmGRBM_STATUS
[ERROR]: Unknown option <-r*.*.mmGRBM_STATUS>
sfc@sfc-DESKTOP:~$ sudo umr -O many,bits  -r *.*.HEADER_DUMP
gfx91.mmCP_MEC_ME1_HEADER_DUMP => 0xc0000e00
        .HEADER_DUMP[0:31]                                               == 3221229056 (0xc0000e00)
gfx91.mmCP_MEC_ME2_HEADER_DUMP => 0xdef0def0
        .HEADER_DUMP[0:31]                                               == 3740327664 (0xdef0def0)
gfx91.mmCP_ME_HEADER_DUMP => 0xc0004200
        .ME_HEADER_DUMP[0:31]                                            == 3221242368 (0xc0004200)
gfx91.mmCP_PFP_HEADER_DUMP => 0xffff1000
        .PFP_HEADER_DUMP[0:31]                                           == 4294905856 (0xffff1000)
gfx91.mmCP_CE_HEADER_DUMP => 0xffff1000
        .CE_HEADER_DUMP[0:31]                                            == 4294905856 (0xffff1000)
sfc@sfc-DESKTOP:~$ sudo umr -O many,bits  -r *.*.CP_EOP
gfx91.mmCP_EOPQ_WAIT_TIME => 0x0000052c
        .WAIT_TIME[0:9]                                                  ==      300 (0x0000012c)
        .SCALE_COUNT[10:17]                                              ==        1 (0x00000001)
gfx91.mmCP_EOP_DONE_ADDR_LO => 0x00609000
        .ADDR_LO[2:31]                                                   ==  1582080 (0x00182400)
gfx91.mmCP_EOP_DONE_ADDR_HI => 0x000000f5
        .ADDR_HI[0:15]                                                   ==      245 (0x000000f5)
gfx91.mmCP_EOP_DONE_DATA_LO => 0x00001d01
        .DATA_LO[0:31]                                                   ==     7425 (0x00001d01)
gfx91.mmCP_EOP_DONE_DATA_HI => 0x00000000
        .DATA_HI[0:31]                                                   ==        0 (0x00000000)
gfx91.mmCP_EOP_LAST_FENCE_LO => 0x00001d01
        .LAST_FENCE_LO[0:31]                                             ==     7425 (0x00001d01)
gfx91.mmCP_EOP_LAST_FENCE_HI => 0x00000000
        .LAST_FENCE_HI[0:31]                                             ==        0 (0x00000000)
gfx91.mmCP_EOP_DONE_EVENT_CNTL => 0x00038060
        .WBINV_TC_OP[0:6]                                                ==       96 (0x00000060)
        .WBINV_ACTION_ENA[12:17]                                         ==       56 (0x00000038)
        .CACHE_POLICY[25:25]                                             ==        0 (0x00000000)
        .EXECUTE[28:28]                                                  ==        0 (0x00000000)
gfx91.mmCP_EOP_DONE_DATA_CNTL => 0x40010000
        .DST_SEL[16:17]                                                  ==        1 (0x00000001)
        .INT_SEL[24:26]                                                  ==        0 (0x00000000)
        .DATA_SEL[29:31]                                                 ==        2 (0x00000002)
gfx91.mmCP_EOP_DONE_CNTX_ID => 0x00000000
        .CNTX_ID[0:31]                                                   ==        0 (0x00000000)
---


No GPU hang log in dmesg.
Comment 30 notsyncing 2018-07-01 06:57:26 UTC
Created attachment 277101 [details]
Kernel log from serial port when it freezes

Finally got logs from serial port when freezed. Seems my problem has nothing to do with amdgpu. Maybe I should file a new bug.
Comment 31 James Le Cuirot 2018-07-01 08:56:41 UTC
(In reply to notsyncing from comment #30)
> Finally got logs from serial port when freezed. Seems my problem has nothing
> to do with amdgpu. Maybe I should file a new bug.

I may be off the mark but that looks more like bug #196683. Have you tried adjusting "Power Supply Idle Control" in the BIOS (if you have it) or using zenstates.py to disable the C6 package state?
Comment 32 notsyncing 2018-07-01 09:00:55 UTC
I've set that option to "Typical Current Idle" and still freezes. The logs in 196683 points to RCU, which seems not my case. I suspect it's due to the zram. Now I'm trying to reproduce it with zram disabled.
Comment 33 notsyncing 2018-07-14 06:45:17 UTC
Now about half a month passed, and my machine is running fine with zram disabled. No more freezes. Seems my problem is related to zram, not this bug, while the "ring gfx timeout" error never happened again. Thanks for everyone who gave me advice.
Comment 34 Andrey Grodzovsky 2018-08-02 20:02:57 UTC
notsyncing, can you close this ticket then ?
Comment 35 notsyncing 2018-08-04 04:46:54 UTC
(In reply to Andrey Grodzovsky from comment #34)
> notsyncing, can you close this ticket then ?

This is not my ticket.