Bug 214447

Created attachment 298855 [details]
kernel .config (kernel 5.14.5, AMD Ryzen 9 5950X)

Created attachment 298857 [details]
output of kmemleak (kernel 5.14.5, AMD Ryzen 9 5950X)

Looks like a missing dma_fence_put() somewhere.

Can you bisect this? Should be rather trivial to pin point the patch which introduced this.

(In reply to Christian König from comment #3)
> Looks like a missing dma_fence_put() somewhere.
> 
> Can you bisect this? Should be rather trivial to pin point the patch which
> introduced this.
Ok, I will try to bisect.

Also the problem must have been introduced recently. Have been running the machine with Kernel 5.13.18 for a few hours now and the leak did not show up.

The bisect revealed this commit. The NULL dereference surely got fixed as I found out during bisecting but with the side effect of this memory leak it seems.

 # git bisect good
f18f58012ee894039cd59ee8c889bf499d7a3943 is the first bad commit
commit f18f58012ee894039cd59ee8c889bf499d7a3943
Author: Mikel Rychliski <mikel@mikelr.com>
Date:   Thu Jun 24 00:51:20 2021 -0400

    drm/radeon: Fix NULL dereference when updating memory stats
    
    radeon_ttm_bo_destroy() is attempting to access the resource object to
    update memory counters. However, the resource object is already freed when
    ttm calls this function via the destroy callback. This causes an oops when
    a bo is freed:
    
            BUG: kernel NULL pointer dereference, address: 0000000000000010
            RIP: 0010:radeon_ttm_bo_destroy+0x2c/0x100 [radeon]
            Call Trace:
             radeon_bo_unref+0x1a/0x30 [radeon]
             radeon_gem_object_free+0x33/0x50 [radeon]
             drm_gem_object_release_handle+0x69/0x70 [drm]
             drm_gem_handle_delete+0x62/0xa0 [drm]
             ? drm_mode_destroy_dumb+0x40/0x40 [drm]
             drm_ioctl_kernel+0xb2/0xf0 [drm]
             drm_ioctl+0x30a/0x3c0 [drm]
             ? drm_mode_destroy_dumb+0x40/0x40 [drm]
             radeon_drm_ioctl+0x49/0x80 [radeon]
             __x64_sys_ioctl+0x8e/0xd0
    
    Avoid the issue by updating the counters in the delete_mem_notify callback
    instead. Also, fix memory statistic updating in radeon_bo_move() to
    identify the source type correctly. The source type needs to be saved
    before the move, because the moved from object may be altered by the move.
    
    Fixes: bfa3357ef9ab ("drm/ttm: allocate resource object instead of embedding it v2")
    Signed-off-by: Mikel Rychliski <mikel@mikelr.com>
    Reviewed-by: Christian König <christian.koenig@amd.com>
    Signed-off-by: Christian König <christian.koenig@amd.com>
    Link: https://patchwork.freedesktop.org/patch/msgid/20210624045121.15643-1-mikel@mikelr.com

 drivers/gpu/drm/radeon/radeon_object.c | 29 ++++++++++++-----------------
 drivers/gpu/drm/radeon/radeon_object.h |  2 +-
 drivers/gpu/drm/radeon/radeon_ttm.c    | 13 ++++++++++---
 3 files changed, 23 insertions(+), 21 deletions(-)

Created attachment 298869 [details]
bisect.log

Mhm, that bisect result doesn't really make much sense.

Can you try to revert the change and so double check if that helps or not?

As I tried to revert the change I realized I can no longer reproduce this issue. In the meantime I did nothing specual but to update some system packages... Anyway, vanilla v5.14.6 and v5.15-rc2 run just fine without showing memory leaks (left the machine for >1 hr which normally delivered plenty of leaks).

So I'll mark this as obsolete. In case I should see this again I will re-open the bug and have a closer look at the circumstances it appears.

Reopening as I can replicate the bug on my Talos II with another Radeon card () on kernel 5.14.9 and latest 5.15-rc4.

On the Talos II I was also able to verify that reverting f18f58012ee894039cd59ee8c889bf499d7a3943 (on top of v5.15-rc4) fixes the memory leak. But of course you get the NULL dereference back.

[..]
unreferenced object 0xc00020000ea7fb00 (size 128):
  comm "X", pid 543, jiffies 4295307011 (age 9605.800s)
  hex dump (first 32 bytes):
    c0 00 00 00 29 86 9a b0 c0 08 00 00 1f 28 81 38  ....)........(.8
    00 00 01 4d 80 92 a3 83 c0 00 20 00 0a 97 77 98  ...M...... ...w.
  backtrace:
    [<c00800001f13f568>] .radeon_fence_emit+0x38/0x130 [radeon]
    [<c00800001f261800>] .evergreen_copy_dma+0x390/0x4a0 [radeon]
    [<c00800001f141a68>] .radeon_bo_move+0x438/0x640 [radeon]
    [<c00800001ed2857c>] .ttm_bo_handle_move_mem+0xbc/0x220 [ttm]
    [<c00800001ed2a254>] .ttm_bo_validate+0xe4/0x1a0 [ttm]
    [<c00800001f143c98>] .radeon_bo_fault_reserve_notify+0x178/0x2c0 [radeon]
    [<c00800001f15fa78>] .radeon_gem_fault+0x98/0x130 [radeon]
    [<c000000000292d58>] .__do_fault+0x58/0x120
    [<c0000000002999cc>] .__handle_mm_fault+0x105c/0x1900
    [<c00000000029a3a8>] .handle_mm_fault+0x138/0x330
    [<c00000000005cd50>] .___do_page_fault+0x5a0/0xa30
    [<c00000000005d20c>] .do_page_fault+0x2c/0xc0
    [<c0000000000088dc>] data_access_common_virt+0x19c/0x1f0
unreferenced object 0xc000200004ba6900 (size 128):
  comm "X", pid 543, jiffies 4296163097 (age 6752.330s)
  hex dump (first 32 bytes):
    c0 00 00 00 29 86 9a b0 c0 08 00 00 1f 28 81 38  ....)........(.8
    00 00 03 e5 e9 9d fd fe c0 00 20 00 0a 97 77 98  .......... ...w.
  backtrace:
    [<c00800001f13f568>] .radeon_fence_emit+0x38/0x130 [radeon]
    [<c00800001f261800>] .evergreen_copy_dma+0x390/0x4a0 [radeon]
    [<c00800001f141a68>] .radeon_bo_move+0x438/0x640 [radeon]
    [<c00800001ed2857c>] .ttm_bo_handle_move_mem+0xbc/0x220 [ttm]
    [<c00800001ed2a254>] .ttm_bo_validate+0xe4/0x1a0 [ttm]
    [<c00800001f143c98>] .radeon_bo_fault_reserve_notify+0x178/0x2c0 [radeon]
    [<c00800001f15fa78>] .radeon_gem_fault+0x98/0x130 [radeon]
    [<c000000000292d58>] .__do_fault+0x58/0x120
    [<c0000000002999cc>] .__handle_mm_fault+0x105c/0x1900
    [<c00000000029a3a8>] .handle_mm_fault+0x138/0x330
    [<c00000000005cd50>] .___do_page_fault+0x5a0/0xa30
    [<c00000000005d20c>] .do_page_fault+0x2c/0xc0
    [<c0000000000088dc>] data_access_common_virt+0x19c/0x1f0
unreferenced object 0xc000000029864e00 (size 128):
  comm "X", pid 543, jiffies 4297018853 (age 3899.817s)
  hex dump (first 32 bytes):
    c0 00 00 00 29 86 9a b0 c0 08 00 00 1f 28 81 38  ....)........(.8
    00 00 06 7e 11 07 45 5e c0 00 20 00 0a 97 77 98  ...~..E^.. ...w.
  backtrace:
    [<c00800001f13f568>] .radeon_fence_emit+0x38/0x130 [radeon]
    [<c00800001f261800>] .evergreen_copy_dma+0x390/0x4a0 [radeon]
    [<c00800001f141a68>] .radeon_bo_move+0x438/0x640 [radeon]
    [<c00800001ed2857c>] .ttm_bo_handle_move_mem+0xbc/0x220 [ttm]
    [<c00800001ed2a254>] .ttm_bo_validate+0xe4/0x1a0 [ttm]
    [<c00800001f143c98>] .radeon_bo_fault_reserve_notify+0x178/0x2c0 [radeon]
    [<c00800001f15fa78>] .radeon_gem_fault+0x98/0x130 [radeon]
    [<c000000000292d58>] .__do_fault+0x58/0x120
    [<c0000000002999cc>] .__handle_mm_fault+0x105c/0x1900
    [<c00000000029a3a8>] .handle_mm_fault+0x138/0x330
    [<c00000000005cd50>] .___do_page_fault+0x5a0/0xa30
    [<c00000000005d20c>] .do_page_fault+0x2c/0xc0
    [<c0000000000088dc>] data_access_common_virt+0x19c/0x1f0

Created attachment 299135 [details]
kernel dmesg (kernel 5.14.9, Talos II Secure Workstation)

Created attachment 299137 [details]
kernel .config (kernel 5.14.9, Talos II Secure Workstation)

Card on the Talos II is a passive cooled HD 6670.

 # lspci -v -s 0000:01:00.0
0000:01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Turks XT [Radeon HD 6670/7670] (prog-if 00 [VGA controller])
	Subsystem: PC Partner Limited / Sapphire Technology Turks XT [Radeon HD 6670/7670]
	Device tree node: /sys/firmware/devicetree/base/pciex@600c3c0000000/pci@0/vga@0
	Flags: bus master, fast devsel, latency 0, IRQ 77, NUMA node 0, IOMMU group 0
	Memory at 6000000000000 (64-bit, prefetchable) [size=256M]
	Memory at 600c000000000 (64-bit, non-prefetchable) [size=128K]
	I/O ports at <unassigned> [disabled]
	Expansion ROM at 600c000020000 [disabled] [size=128K]
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Legacy Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Kernel driver in use: radeon
	Kernel modules: radeon

Created attachment 299231 [details]
kernel dmesg (kernel 5.15-rc5, AMD Ryzen 9 5950X)

I was also able to replicate the bug on the hardware which I originally discovered it. Probably I mixed something up with the booted kernels the time I closed the bug... As of v5.15-rc5 it was easy to get the leak, even when no monitor is connected to the card.

And I can conform that reverting f18f58012ee894039cd59ee8c889bf499d7a3943 (on top of v5.15-rc5) fixes the leak with the rv515 card on the Ryzen 9 too. But you also get the NULL pointer dereference back.

Created attachment 299233 [details]
output of kmemleak (kernel 5.15-rc5, AMD Ryzen 9 5950X)

Created attachment 299235 [details]
kernel .config (kernel 5.15-rc5, AMD Ryzen 9 5950X)

Thanks for the logs! I think I've found the root cause of this!.

It's just that the bisect is unfortunately incorrect.The bug happens really at random during memory eviction and so very hard to reproduce.

Created attachment 299275 [details]
Potential fix

Please give the attache patch a try.

(In reply to Christian König from comment #16)
> Created attachment 299275 [details]
> Potential fix
> 
> Please give the attache patch a try.
This patch indeed did the trick, many thanks! Uptime with patched v5.15-rc6 is >12 hrs now without any memleak, where the unpatched kernel would show dozens.

The patch also fixed the leak from bug #214029. There with patched v5.14.13, also >12 hrs uptime without any leak up to now. Only machine left to test would be my Talos II where I'apply the patch now.

The fix landed in kernel 5.15, 5.14.16 and affected LTS kernels.

Closing.