Bug 17852 - Build: detection for working -fstack-protector wrongly says "stack protector enabled but no compiler support" if using hardened gcc due to use of -mcmodel=kernel without -D__KERNEL__
Build: detection for working -fstack-protector wrongly says "stack protector ...
Status: RESOLVED CODE_FIX
Product: Other
Classification: Unclassified
Component: Other
All Linux
: P1 low
Assigned To: other_other
https://bugs.gentoo.org/show_bug.cgi?...
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-09-05 18:30 UTC by bugs-kernel@spamgourmet.com
Modified: 2010-09-23 02:29 UTC (History)
1 user (show)

See Also:
Kernel Version: v2.6.35.4
Tree: Mainline
Regression: No


Attachments
Patch by Magnus Granberg to add KBUILD_CPPFLAGS to the SSP test so that it does not build with -fPIE (685 bytes, patch)
2010-09-11 04:28 UTC, bugs-kernel@spamgourmet.com
Details | Diff

Description bugs-kernel@spamgourmet.com 2010-09-05 18:30:15 UTC
This has been confirmed in v2.6.35.4.  From source inspection, it looks like v2.6.36-rc3 is also affected.

This was reported to the Gentoo bug tracker as <https://bugs.gentoo.org/show_bug.cgi?id=312335>.  Gentoo offers a hardened gcc 4.4 that automatically enables PIC and SSP for user code, unless instructed otherwise.  It determines user vs. kernel by the presence or absence of -D__KERNEL__ on the command line.  Unfortunately, the way that scripts/gcc-x86_64-has-stack-protector.sh calls gcc causes it to fail with an error message:

$ echo "int foo(void) { char X[200]; return 3; }" | gcc -S -xc -c -O0 -mcmodel=kernel -fstack-protector - -o -
<stdin>:1: error: code model kernel does not support PIC mode
<stdin>:1: error: code model 'kernel' not supported in the 64 bit mode
$ echo $?
1

Adding in -D__KERNEL__ causes hardened gcc to recognize that PIC is inappropriate, and allows it to complete successfully:

$ echo "int foo(void) { char X[200]; return 3; }" | gcc -S -xc -c -O0 -mcmodel=kernel -fstack-protector - -o - -D__KERNEL__ | grep -q '%gs'
$ echo $?
0

Magnus Granberg posted a proposed patch as <https://bugs.gentoo.org/attachment.cgi?id=225963> that adds $(KBUILD_CPPFLAGS) to the arch/x86/Makefile call to scripts/gcc-x86_$(BITS)-has-stack-protector.sh.  With this patch, the test invocation of gcc includes -D__KERNEL__, thereby suppressing the PIC behavior and causing gcc to produce the output expected by the script.  I hope that this solution is generic enough that it can be included in the mainline kernel.  It should not have any effect on a non-hardened gcc, since the test input includes no files and makes no explicit references to any preprocessor symbols.
Comment 1 Andrew Morton 2010-09-08 22:00:30 UTC
(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Sun, 5 Sep 2010 18:30:20 GMT
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=17852
> 
>                URL: https://bugs.gentoo.org/show_bug.cgi?id=312335
>            Summary: Build: detection for working -fstack-protector wrongly
>                     says "stack protector enabled but no compiler support"
>                     if using hardened gcc due to use of -mcmodel=kernel
>                     without -D__KERNEL__

Send us a patch.  Suitable recipients are

Ingo Molnar <mingo@elte.hu>
Thomas Gleixner <tglx@linutronix.de>
"H. Peter Anvin" <hpa@zytor.com>
linux-kernel@vger.kernel.org
Andrew Morton <akpm@linux-foundation.org>
Comment 2 bugs-kernel@spamgourmet.com 2010-09-11 04:28:27 UTC
Created attachment 29552 [details]
Patch by Magnus Granberg to add KBUILD_CPPFLAGS to the SSP test so that it does not build with -fPIE

> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).

Unfortunately, I cannot do that.  I never received an e-mail for this because I am not subscribed to e-mail notifications, and spamgourmet would eat them even if I was subscribed.

> Send us a patch.

The patch from Magnus Granberg is attached.
Comment 3 Anthony Basile 2010-09-13 10:00:44 UTC
(In reply to comment #1)
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> On Sun, 5 Sep 2010 18:30:20 GMT
> bugzilla-daemon@bugzilla.kernel.org wrote:
> 
> > https://bugzilla.kernel.org/show_bug.cgi?id=17852
> > 
> >                URL: https://bugs.gentoo.org/show_bug.cgi?id=312335
> >            Summary: Build: detection for working -fstack-protector wrongly
> >                     says "stack protector enabled but no compiler support"
> >                     if using hardened gcc due to use of -mcmodel=kernel
> >                     without -D__KERNEL__
> 
> Send us a patch.  Suitable recipients are
> 
> Ingo Molnar <mingo@elte.hu>
> Thomas Gleixner <tglx@linutronix.de>
> "H. Peter Anvin" <hpa@zytor.com>
> linux-kernel@vger.kernel.org
> Andrew Morton <akpm@linux-foundation.org>

I did email in the patch to the above about a month ago and it never made it through.  I will try again.
Comment 4 bugs-kernel@spamgourmet.com 2010-09-23 02:29:40 UTC
The fix was accepted by hpa as 08c2b394b98786ebb067e2a54d08f1f6f0d247da and appeared in mainline as part of v2.6.36-rc4.

Note You need to log in before you can comment on or make changes to this bug.