Bug 217457

Summary:	Persistent rt_sigreturn segfaults on KVM VMs after upgrade to 5.15
Product:	Linux	Reporter:	Theodor Milkov (tm)
Component:	Kernel	Assignee:	Virtual assignee for kernel bugs (linux-kernel)
Status:	RESOLVED OBSOLETE
Severity:	normal	CC:	bagasdotme, qhu, rockyramchandani02
Priority:	P3
Hardware:	All
OS:	Linux
Kernel Version:		Subsystem:
Regression:	No	Bisected commit-id:

Description Theodor Milkov 2023-05-18 08:43:13 UTC

I'm experiencing sporadic but persistent segmentation faults on the KVM VMs I manage. These faults began appearing after upgrading from Linux Kernel 4.x to 5.15.59. I further upgraded to 5.15.91 and transitioned the userspace from Debian 10 (buster) to Debian 11 (bullseye), yet the issues persist. Notably, the libc has also changed in the process as seen in the following error logs:

post.sh[21952]: bad frame in rt_sigreturn frame:000072db65961bb8 ip:6c25f82a9a5d sp:72db65962168 orax:ffffffffffffffff in libc-2.28.so[6c25f8294000+147000]

cron[7626]: bad frame in rt_sigreturn frame:000073ddebeb6ff8 ip:72ad9f44d594 sp:73ddebeb75a8 orax:ffffffffffffffff in libc-2.28.so[72ad9f3a9000+147000]

cron[64687]: bad frame in rt_sigreturn frame:000073265764b038 ip:67c7b5a0f14a sp:73265764b5f0 orax:ffffffffffffffff in libc-2.31.so[67c7b596f000+159000]

worker.py[54568]: bad frame in rt_sigreturn frame:000078eef6591cf8 ip:6c9f9b2a604e sp:78eef6592298 orax:ffffffffffffffff in libpthread-2.31.so[6c9f9b29a000+10000]

The segmentation faults occur 1-3 times daily across approximately 1000 VMs running on hundreds of (supermicro, intel cpu) bare-metal servers. Currently, there's no reliable way for me to reproduce the issue. I initially considered this bug - https://www.spinics.net/lists/linux-tip-commits/msg61293.html - as a possible cause, but judging from the comments it likely isn't.

The best approximation to a reproducer I have is a Python script that initiates several child processes and continuously sends them a sigusr1 signal. Still, it takes a few hours to trigger the issue even when running this script on several hundred VMs.

Switching to the 6.x kernel isn't immediately feasible as these are production systems with specific requirements. The transition is planned but will likely take several months.

I'm looking for suggestions on how to more reliably reproduce this problem. Then I could try different old and new kernels and maybe narrow it down.

Comment 1 Bagas Sanjaya 2023-05-18 13:45:36 UTC

(In reply to Theodor Milkov from comment #0)
> I'm experiencing sporadic but persistent segmentation faults on the KVM VMs
> I manage. These faults began appearing after upgrading from Linux Kernel 4.x
> to 5.15.59. I further upgraded to 5.15.91 and transitioned the userspace
> from Debian 10 (buster) to Debian 11 (bullseye), yet the issues persist.
> Notably, the libc has also changed in the process as seen in the following
> error logs:
> 

I guess before upgrading you use v4.19 (from your distro), right?

Comment 2 Bagas Sanjaya 2023-05-18 13:49:13 UTC

(In reply to Theodor Milkov from comment #0)
> Switching to the 6.x kernel isn't immediately feasible as these are
> production systems with specific requirements. The transition is planned but
> will likely take several months.
> 

If you already have testing system (which should be identical to your production ones), can you try latest mainline there?

Comment 3 Theodor Milkov 2023-05-18 15:19:04 UTC

We have compiled both our previous 4.14 kernel and our current 5.15 kernel ourselves, without module support. These kernels are used in the guest systems. Meanwhile, the host systems are using the 5.10 kernel from the distribution (Debian). The problem started right after upgrading guest kernel to 5.15.59.

I am currently looking into trying 6.1 kernel on a subset of approximately 150 guest systems. I will update you on the results in the coming days.

Comment 4 Rocky 2023-05-22 08:42:26 UTC

@Theodor Milkov, is there an equivalent core dump generated for your program/script? probably under /var/log/core.

If yes, can you provide excerpts from its gdb analysis?
PS - https://stackoverflow.com/questions/5115613/core-dump-file-analysis

Comment 5 Theodor Milkov 2023-05-22 15:09:38 UTC

The only way I currently have to trigger this segmentation fault is with the following Python code: https://gist.github.com/z-image/762691ee7a67ffdeb88318c47d9ebf0c

The actual code is much lengthier, as it involves monitoring /proc/stat & /proc/pressure and regulating the rate of os.kill() to avoid overloading the servers. These are, after all, production machines. However, the most crucial component is this.

I've managed to obtain a core dump, but, as anticipated, it doesn't contain much useful information. This is because a) this is the non-debug version of Python 3, and b) the issue lies deeper within the kernel.

Here's the backtrace:

(gdb) bt full
#0  0x000060f0e5c4904e in __libc_read (fd=5, buf=0x60f0e5701620, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:26
        resultvar = 0
        __arg3 = <optimized out>
        _a2 = <optimized out>
        sc_ret = <optimized out>
        __arg1 = <optimized out>
        _a3 = <optimized out>
        sc_cancel_oldtype = <optimized out>
        resultvar = <optimized out>
        resultvar = <optimized out>
        __arg2 = <optimized out>
        _a1 = <optimized out>
#1  0x0000000000561ec7 in _Py_read ()
No symbol table info available.

I may try with python3-dbg if you think that is going to be more useful.

Comment 6 Theodor Milkov 2023-10-04 14:34:38 UTC

I'd like to report that the persistent rt_sigreturn segmentation faults on KVM VMs are no longer reproducible after upgrading to kernel version 5.15.131. It appears the issue was fixed in one of the updates post 5.15.91.