Bug 218484

Summary: virtual memory exhausted in 6.6.16 on i586 after minmax patchset
Product: Linux Reporter: Giuseppe Ghibò (ghibo)
Component: KernelAssignee: Virtual assignee for kernel bugs (linux-kernel)
Status: RESOLVED CODE_FIX    
Severity: normal CC: regressions
Priority: P3    
Hardware: i386   
OS: Linux   
Kernel Version: Subsystem:
Regression: No Bisected commit-id:

Description Giuseppe Ghibò 2024-02-12 10:36:13 UTC
After upgrading to version 6.6.16, the kernel compilation on a i586 arch (on a 32bit chroot in a 64bit host) fails with a message:

virtual memory exhausted: Cannot allocate memory

this happens even lowering the number of parallel compilation threads. On a x86_64 arch the same problem doesn't occur. It's not clear whether some weird recursion is triggered that exhausts the memory, but it seems that the problem is caused by the patchset 'minmax' added to the 6.6.16 version, in particular it seems caused by these patches:

- minmax-allow-min-max-clamp-if-the-arguments-have-the-same-signedness.patch
- minmax-fix-indentation-of-__cmp_once-and-__clamp_once.patch
- minmax-allow-comparisons-of-int-against-unsigned-char-short.patch
- minmax-relax-check-to-allow-comparison-between-unsigned-arguments-and-signed-constants.patch

Reverting those patches fixes the memory exhaustion problem during compilation.
Comment 1 The Linux kernel's regression tracker (Thorsten Leemhuis) 2024-02-12 13:30:38 UTC
(In reply to Giuseppe Ghibò from comment #0)

> Reverting those patches fixes the memory exhaustion problem during
> compilation.

The important thing to know would be: does the same problem occur with mainline (e.g. 6.8-rc4) as well. Could you test this please?
Comment 2 Giuseppe Ghibò 2024-02-12 15:35:55 UTC
From a quick test the same problem doesn't occur in 6.8-rc4.
Comment 3 The Linux kernel's regression tracker (Thorsten Leemhuis) 2024-02-12 17:22:34 UTC
Forwarded by mail:
https://lore.kernel.org/all/f9f89284-0f48-4971-ad8d-86938a82fafc@leemhuis.info/
Comment 4 The Linux kernel's regression tracker (Thorsten Leemhuis) 2024-02-13 15:02:16 UTC
(In reply to The Linux kernel's regression tracker (Thorsten Leemhuis) from comment #3)
> Forwarded by mail:
> https://lore.kernel.org/all/f9f89284-0f48-4971-ad8d-86938a82fafc@leemhuis.
> info/

Please check that thread: could you confirm that the problem happens when compiling drivers/media/pci/solo6x10/solo6x10-offsets.h? Or is it something else?
Comment 5 The Linux kernel's regression tracker (Thorsten Leemhuis) 2024-02-13 15:18:28 UTC
(In reply to The Linux kernel's regression tracker (Thorsten Leemhuis) from comment #4)
> Please check that thread: could you confirm [...]

Likely not needed anymore, but can't hurt if you have a minute.
Comment 6 Giuseppe Ghibò 2024-02-13 15:48:45 UTC
(In reply to The Linux kernel's regression tracker (Thorsten Leemhuis) from comment #4)
> (In reply to The Linux kernel's regression tracker (Thorsten Leemhuis) from
> comment #3)
> > Forwarded by mail:
> > https://lore.kernel.org/all/f9f89284-0f48-4971-ad8d-86938a82fafc@leemhuis.
> > info/
> 
> Please check that thread: could you confirm that the problem happens when
> compiling drivers/media/pci/solo6x10/solo6x10-offsets.h? Or is it something
> else?

Yes, sort of. I'll try to build with CONFIG_VIDEO_SOLO6X10 disabled and the minmax patchset applied to see whether there are other missed.
Comment 7 Giuseppe Ghibò 2024-02-13 17:13:44 UTC
With CONFIG_VIDEO_SOLO6X10 disabled it compiles smooth without memory exaustion errors.

Now trying with the patch:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/media/pci/solo6x10/solo6x10-offsets.h?id=31e97d7c9ae3de072d7b424b2cf706a03ec10720

Pretty weird that the max() used in solo6x10-offsets.h like max(min(...,...),...)  would cause that big problem; Though the min() or max() macro has always been fragile (because of being macros, at least when defined with the ternary operator, I wonder if we can extract a minimal test case.
Comment 8 Giuseppe Ghibò 2024-02-13 19:33:11 UTC
Confirming with the media-solo6x10-replace-max-a-min-b-c-by-clamp-b-a-c.patch the  above problem doesn't occur anymore.
Comment 9 Giuseppe Ghibò 2024-05-06 11:33:10 UTC
Closing this, as a fix was included in 6.6.17.