Bug 219089

Created attachment 306611 [details]
Kernel configuration used to reproduce the problem

I've encountered a problem with certain 32-bit ARM kernel builds freezing or panic-ing during early boot, during "ftrace_init" execution. I debugged the problem and found the most likely root cause. This bug is created to document the findings and serve as a reference point.

PROBLEM: The kernel boot freezes (or causes kernel panic) during the early boot. The "earlycon" kernel argument needs to be used to see kernel log messages before the freeze. The problem happens in the following call chain:
start_kernel -> ftrace_init -> ftrace_process_locs -> ftrace_update_code. Note, the kernel is build with Thumb instructions (CONFIG_THUMB2_KERNEL=y).

The "ftrace_update_code" has a loop that handles all ftrace call locations. The problem happens when it updates the ftrace location (by calling "ftrace_nop_initialize") for the entry for:
 ***  _raw_spin_unlock_irqrestore  ***

The problem ONLY shows-up when the ftrace location for "_raw_spin_unlock_irqrestore" is NOT 4-byte aligned and 4 bytes at this location straddle the instruction cache line (0x20) boundaries. I.e. the address ends on on:
   0x1e, 0x3e, 0x5e, ... 0xfe.
Per observation, the location is 2 bytes after the start of the function, so the problem happens when the "_raw_spin_unlock_irqrestore" address (as can be seen in System.map) ends on:
  0x1c, 0x3c, 0x5c, ..., 0xfc.

ROOT CAUSE: The "ftrace_nop_initalize" goes into the arch-specific code. For 32-bit ARM, it eventually calls "__patch_text_real" defined in "arch/arm/kernel/patch.c". After the function modifies the instruction, it looks like this:
    if (waddr != addr) {
        flush_kernel_vmap_range(waddr, twopage ? size / 2 : size);
        patch_unmap(FIX_TEXT_POKE0, &flags);
    }

    flush_icache_range((uintptr_t)(addr),
               (uintptr_t)(addr) + size);

The "patch_unmap" calls the above-mentioned "_raw_spin_unlock_irqrestore", and hereby lies the problem.  If it's patching the "_raw_spin_unlock_irqrestore" itself and invokes the function BEFORE flushing the icache, there is a possibility for the CPU to see an invaild instruction created by the combination of the updated and non-updated one.

REPRODUCING THE PROBLEM:

Hardware: I used single-board computers "Banana PI M1" and "Orange Pi One". I belive it should reproducible on any Cortex-A7 CPU, or likely on any 32-bit ARM CPU.

System: Used an Armbian installation (www.armbian.com), replacing the kernel with custom-built one (and disabling the initrd).

Kernel: Used the attached configuration (upgraded to mainline) to build it. Reproduced with the mainline (6.10.y) kernel, as well as 6.9.y, 6.6.y, and 6.1.y. However, needed to modify the code in order to shift the "_raw_spin_unlock_irqrestore" to the "trigger" location as per above description. Used asm("nop; nop; ..."); statements to insert nop instructions in the code, using the below approach. A stack trace from a panic on 6.10 kernel is attached.

Shifting the "_raw_spin_unlock_irqrestore":
 * Add 4 "nop" instructions at a time to the end of "try_to_generate_entropy" function in the 
   "drivers/char/random.c", until "_raw_spin_unlock_irqrestore" address
   ends on -x8, or -xC, where "x" is odd. E.g. ...1c, ...3c, ...5c, etc.
    E.g. asm("nop;nop;nop;nop; ");
 * If it ends on 8, add 2 more "nop" instructions to one of the lock functions
   inside the "__lock_text_start" section: see the System.map on which one
   comes first/earlier.