Created attachment 120971 [details]
Backtrace using crash/vmcore

Moving to drivers/wireless as this seems to be the driver not properly checking if it got a short frame with no iv block and then trying to remove data that was not present.

I suspect that txdesc->iv_len has somehow wrong value, but not sure how this could happen.

Is this bug reproducible ? Could you provide vmcore file for download somewhere ?

I do not know how to reproduce the crash. I have been having sporadic crashes, usually while the machine is unattended, for the past month or so. Since I got kdump working and submitted this bug, I have removed the hardware from the machine, but if it would be useful, I can use the device again and see if I can capture another vmcore.

I have sent the original vmcore corresponding to the trace above to Stanislaw privately.

Thanks, I'm looking in vmcore now, but analyzing memory dump can be hard, so it can take some time ...

crash> struct sk_buff ffff88081a045f80 | head -20
struct sk_buff {
  next = 0x0, 
  prev = 0x0, 
  tstamp = {
    tv64 = 0
  }, 
  sk = 0xffff8807c7fde880, 
  dev = 0xffff8808039b42a0, 
  cb = "P\000\200@\001\002\000\000\000\000\a(\000\000\000\000\000\000\000\000\000\000\000\000\252\252\003\000\000\000\b\000E\000\001\254lZ@\000\004\021T\256\300\250\003\226", 
  _skb_refdst = 7784309262464843759, 
  sp = 0x49544f4ef6959801, 
  len = 706762971, 
  data_len = 1414809632, 
  mac_len = 12112, 
  hdr_len = 11825, 
  {
    csum = 1208618289, 
    {
      csum_start = 3377, 
      csum_offset = 18442
crash> rd ffff88081a045f80 30
ffff88081a045f80:  0000000000000000 0000000000000000   ................
ffff88081a045f90:  0000000000000000 ffff8807c7fde880   ................
ffff88081a045fa0:  ffff8808039b42a0 0000020140800050   .B......P..@....
ffff88081a045fb0:  0000000028070000 0000000000000000   ...(............
ffff88081a045fc0:  000800000003aaaa 00405a6cac010045   ........E...lZ@.
ffff88081a045fd0:  9603a8c0ae541104 6c076c07faffffef   ..T..........l.l
ffff88081a045fe0:  49544f4ef6959801 545448202a2058db   ....NOTI.X * HTT
ffff88081a045ff0:  480a0d312e312f50 393332203a74736f   P/1.1..Host: 239
ffff88081a046000:  3535322e3535322e 3039313a3035322e   .255.255.250:190
ffff88081a046010:  65686361430a0d30 6c6f72746e6f432d   0..Cache-Control
ffff88081a046020:  67612d78616d203a 0000000000313d65   : max-age=1.....
ffff88081a046030:  0000000000000000 0000000000000000   ................
ffff88081a046040:  0000003e00600074 000002c00000020c   t.`.>...........
ffff88081a046050:  ffff8805399ca4f8 ffff8805399ca536   ...9....6..9....
ffff88081a046060:  0000000100000500 cccccccccccccccc   ................

sk_buff structure is corrupted by network packet (there is HTTP data where actual skb len & data_len values should be).

Alex, please install fedora kernel-debug, try run it for some time and check if it detect some problems. If not, I will provide you kernel compiled with CONFIG_DEBUG_PAGEALLOC , which is even more intensive memory corruption debug method, than are used in kernel-debug, but it slow down performance of the kernel vastly.

If this is software bug, it should be detectable by above methods, but this could be also firmware bug or DMA settings bug, which are not easy detectable. rt2x00usb driver does not set DMA mappings directly, it is done by usb host driver, which one are you using ("lsusb -t" should show that) ?

Thank you Stanislaw. I have downloaded kernel-debug, and will run that exclusively for a while.

Some other info: 

I've been having sporadic crashes since performing a hardware upgrade at the start of December (new motherboard, CPU, RAM and SSD, but same graphics card and rt2800usb device). Previous machine was fairly stable, hardly any unexplained crashes. One other change is that the boot process on the new hardware is via UEFI rather than legacy BIOS. The machine is not overclocked in any way and memory timing is at default settings. Unfortunately the 32G of RAM is not ECC, and non-Xeon Haswell CPUs don't seem to support ECC RAM. Initial memtest86 with the new hardware detected no errors after 12 hours. 

I've had slub_debug=FZPU on the kernel command line since 2013-12-23. I've had kdump/crashkernel enabled since 2014-01-04, shortly before capturing this crash. The rt2800usb has been physically removed from the computer since 2014-01-04.

The frequency of crashes does seem to have decreased since 2014-01-04, but I have had one more crash with IP in __slab_alloc called from __alloc_skb, unix_stream_sendmsg, reported at:
https://bugzilla.redhat.com/show_bug.cgi?id=1051476

As you can see, Dave Jones suspected a bitflip, so I ran memtest86 again for another 20 hours, with no errors detected. I've also run single and parallel instances of memtester from userspace for another ~24 hours in total, with no errors detected.

---

lsusb -t

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
    |__ Port 5: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/14p, 480M
    |__ Port 5: Dev 2, If 0, Class=Audio, Driver=snd-usb-audio, 12M
    |__ Port 5: Dev 2, If 1, Class=Audio, Driver=snd-usb-audio, 12M
    |__ Port 5: Dev 2, If 2, Class=Audio, Driver=snd-usb-audio, 12M
    |__ Port 6: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 4, If 0, Class=Hub, Driver=hub/2p, 480M
            |__ Port 1: Dev 7, If 0, Class=Mass Storage, Driver=usb-storage, 480M
            |__ Port 2: Dev 8, If 0, Class=Wireless, Driver=btusb, 12M
            |__ Port 2: Dev 8, If 1, Class=Wireless, Driver=btusb, 12M
        |__ Port 2: Dev 10, If 0, Class=Vendor Specific Class, Driver=rt2800usb, 480M
        |__ Port 3: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 4: Dev 6, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
        |__ Port 4: Dev 6, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M

---

lspci -v

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
	Subsystem: Gigabyte Technology Co., Ltd Device 5000
	Flags: bus master, fast devsel, latency 0
	Capabilities: [e0] Vendor Specific Information: Len=0c <?>

00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000e000-0000efff
	Memory behind bridge: e0000000-f00fffff
	Capabilities: [88] Subsystem: Gigabyte Technology Co., Ltd Device 5000
	Capabilities: [80] Power Management version 3
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [a0] Express Root Port (Slot+), MSI 00
	Capabilities: [100] Virtual Channel
	Capabilities: [140] Root Complex Link
	Capabilities: [d94] #19
	Kernel driver in use: pcieport

00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 04) (prog-if 30 [XHCI])
	Subsystem: Gigabyte Technology Co., Ltd Device 5007
	Flags: bus master, medium devsel, latency 0, IRQ 43
	Memory at f0200000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: [70] Power Management version 2
	Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
	Kernel driver in use: xhci_hcd

00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
	Subsystem: Gigabyte Technology Co., Ltd Device 1c3a
	Flags: bus master, fast devsel, latency 0, IRQ 45
	Memory at f0218000 (64-bit, non-prefetchable) [size=16]
	Capabilities: [50] Power Management version 3
	Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Kernel driver in use: mei_me

00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04)
	Subsystem: Gigabyte Technology Co., Ltd Device a002
	Flags: bus master, fast devsel, latency 0, IRQ 47
	Memory at f0210000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 2
	Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
	Capabilities: [100] Virtual Channel
	Kernel driver in use: snd_hda_intel

00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	Capabilities: [40] Express Root Port (Slot-), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Device 5001
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
	I/O behind bridge: 0000d000-0000dfff
	Memory behind bridge: f0100000-f01fffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Device 5001
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d4) (prog-if 01 [Subtractive decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=04, subordinate=05, sec-latency=0
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Device 5001
	Capabilities: [a0] Power Management version 3

00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 04)
	Subsystem: Gigabyte Technology Co., Ltd Device 5001
	Flags: bus master, medium devsel, latency 0
	Capabilities: [e0] Vendor Specific Information: Len=0c <?>
	Kernel driver in use: lpc_ich

00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04) (prog-if 01 [AHCI 1.0])
	Subsystem: Gigabyte Technology Co., Ltd Device b005
	Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 42
	I/O ports at f070 [size=8]
	I/O ports at f060 [size=4]
	I/O ports at f050 [size=8]
	I/O ports at f040 [size=4]
	I/O ports at f020 [size=32]
	Memory at f0216000 (32-bit, non-prefetchable) [size=2K]
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [70] Power Management version 3
	Capabilities: [a8] SATA HBA v1.0
	Kernel driver in use: ahci

00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04)
	Subsystem: Gigabyte Technology Co., Ltd Device 5001
	Flags: medium devsel, IRQ 18
	Memory at f0215000 (64-bit, non-prefetchable) [size=256]
	I/O ports at f000 [size=32]

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 6770] (prog-if 00 [VGA controller])
	Subsystem: Gigabyte Technology Co., Ltd Device 220c
	Flags: bus master, fast devsel, latency 0, IRQ 44
	Memory at e0000000 (64-bit, prefetchable) [size=256M]
	Memory at f0020000 (64-bit, non-prefetchable) [size=128K]
	I/O ports at e000 [size=256]
	Expansion ROM at f0000000 [disabled] [size=128K]
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Legacy Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Kernel driver in use: radeon

01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series]
	Subsystem: Gigabyte Technology Co., Ltd Device aa58
	Flags: bus master, fast devsel, latency 0, IRQ 48
	Memory at f0040000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Legacy Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Kernel driver in use: snd_hda_intel

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
	Subsystem: Gigabyte Technology Co., Ltd Motherboard
	Flags: bus master, fast devsel, latency 0, IRQ 46
	I/O ports at d000 [size=256]
	Memory at f0104000 (64-bit, non-prefetchable) [size=4K]
	Memory at f0100000 (64-bit, prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [70] Express Endpoint, MSI 01
	Capabilities: [b0] MSI-X: Enable- Count=4 Masked-
	Capabilities: [d0] Vital Product Data
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Virtual Channel
	Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00
	Kernel driver in use: r8169

04:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 41) (prog-if 01 [Subtractive decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=04, secondary=05, subordinate=05, sec-latency=32
	Capabilities: [90] Power Management version 2
	Capabilities: [a0] Subsystem: Gigabyte Technology Co., Ltd Device 8892

---

cat /proc/interrupts

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
  0:        134          0          0          0          0          0          0          0  IR-IO-APIC-edge      timer
  1:         15          0          0          0          0          0          0          0  IR-IO-APIC-edge      i8042
  8:          1          0          0          0          0          0          0          0  IR-IO-APIC-edge      rtc0
  9:          0          0          0          0          0          0          0          0  IR-IO-APIC-fasteoi   acpi
 40:          0          0          0          0          0          0          0          0  DMAR_MSI-edge      dmar0
 42:      17331          0          0          0          0          0     183103          0  IR-PCI-MSI-edge      ahci
 43:     212972          0          0          0          0          0          0          0  IR-PCI-MSI-edge      xhci_hcd
 44:        157          0          0          0          0          0          0      51950  IR-PCI-MSI-edge      radeon
 45:         13          0          0          0          0          0          0          0  IR-PCI-MSI-edge      mei_me
 46:      95727          0          0          0          0          0          0          0  IR-PCI-MSI-edge      p4p1
 47:       1579          0          0          0          0          0          0          0  IR-PCI-MSI-edge      snd_hda_intel
 48:       7224          0          0          0          0          0          0          0  IR-PCI-MSI-edge      snd_hda_intel
NMI:      49106     715007      30487      59315      14622      39729     216281      36009   Non-maskable interrupts
LOC:     351949     451316     237676     336484     221566     270926     554846     464766   Local timer interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
PMI:      49106     715007      30487      59315      14622      39729     216281      36009   Performance monitoring interrupts
IWI:      16272      10259       7591      10770       7830      14900       9828      23661   IRQ work interrupts
RTR:          0          0          0          0          0          0          0          0   APIC ICR read retries
RES:      15533       8333       5785       5419      30757      32330      21055      21429   Rescheduling interrupts
CAL:       3892       2352       2214       2289        989        883        828        835   Function call interrupts
TLB:      12772      14182       5244      13896      12839      10366       6443      10613   TLB shootdowns
TRM:        113        113        113        113        113        113        113        113   Thermal event interrupts
THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
MCE:          0          0          0          0          0          0          0          0   Machine check exceptions
MCP:         26         26         26         26         26         26         26         26   Machine check polls
ERR:          0
MIS:          0

---

Once again, thanks for your help.

OK, I have a new error message using 3.12.7-300.fc20.x86_64+debug, after more than 34 hours uptime and many more error-free loops of memtester:

=============================================================================
BUG vm_area_struct (Not tainted): Poison overwritten
-----------------------------------------------------------------------------

INFO: 0xffff8802ca728039-0xffff8802ca728039. First byte 0x4b instead of 0x6b
INFO: Allocated in dup_mm+0x230/0x710 age=112633 cpu=0 pid=369
        __slab_alloc+0x3eb/0x4fe
        kmem_cache_alloc+0x294/0x340
        dup_mm+0x230/0x710
        copy_process.part.23+0x12d4/0x1890
        do_fork+0xce/0x450
        SyS_clone+0x16/0x20
        stub_clone+0x69/0x90
INFO: Freed in remove_vma+0x76/0x80 age=109627 cpu=5 pid=28464
        __slab_free+0x3a/0x382
        kmem_cache_free+0x356/0x370
        remove_vma+0x76/0x80
        exit_mmap+0xf4/0x170
        mmput+0x7f/0x110
        do_exit+0x2a5/0xcd0
        do_group_exit+0x4c/0xc0
        SyS_exit_group+0x14/0x20
        system_call_fastpath+0x16/0x1b
INFO: Slab 0xffffea000b29ca00 objects=32 used=32 fp=0x          (null) flags=0x5ff00000004080
INFO: Object 0xffff8802ca728000 @offset=0 fp=0xffff8802ca72aa00

Object ffff8802ca728000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802ca728010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802ca728020: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802ca728030: 6b 6b 6b 6b 6b 6b 6b 6b 6b 4b 6b 6b 6b 6b 6b 6b  kkkkkkkkkKkkkkkk
Object ffff8802ca728040: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802ca728050: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802ca728060: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802ca728070: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802ca728080: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802ca728090: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802ca7280a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Object ffff8802ca7280b0: 6b 6b 6b 6b 6b 6b 6b a5                          kkkkkkk.
Redzone ffff8802ca7280b8: bb bb bb bb bb bb bb bb                          ........
Padding ffff8802ca7281f8: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
CPU: 7 PID: 5195 Comm: Xorg Tainted: G    B        3.12.7-300.fc20.x86_64+debug #1
Hardware name: Gigabyte Technology Co., Ltd. Z87-HD3/Z87-HD3, BIOS F6 08/03/2013
 ffff8802ca728000 ffff8807ec29bbd8 ffffffff81749d2b ffff88081d019200
 ffff8807ec29bc18 ffffffff811d40fd 0000000000000008 ffff880200000001
 ffff8802ca72803a ffff88081d019200 000000000000006b ffff8802ca728000
Call Trace:
 [<ffffffff81749d2b>] dump_stack+0x54/0x74
 [<ffffffff811d40fd>] print_trailer+0x14d/0x200
 [<ffffffff811d42ef>] check_bytes_and_report+0xcf/0x110
 [<ffffffff811d5177>] check_object+0x1d7/0x250
 [<ffffffff811b28e8>] ? mmap_region+0x348/0x5d0
 [<ffffffff817474b2>] alloc_debug_processing+0x76/0x118
 [<ffffffff817480ed>] __slab_alloc+0x3eb/0x4fe
 [<ffffffffa00171e9>] ? drm_gem_object_lookup+0x29/0x160 [drm]
 [<ffffffff811b28e8>] ? mmap_region+0x348/0x5d0
 [<ffffffff811d6cc4>] kmem_cache_alloc+0x294/0x340
 [<ffffffff811b28e8>] ? mmap_region+0x348/0x5d0
 [<ffffffff811b28e8>] mmap_region+0x348/0x5d0
 [<ffffffff811b2ed0>] do_mmap_pgoff+0x360/0x3f0
 [<ffffffff8119cf50>] vm_mmap_pgoff+0x90/0xc0
 [<ffffffff811b1423>] SyS_mmap_pgoff+0x1d3/0x270
 [<ffffffff8101e7e2>] SyS_mmap+0x22/0x30
 [<ffffffff8175d029>] system_call_fastpath+0x16/0x1b
FIX vm_area_struct: Restoring 0xffff8802ca728039-0xffff8802ca728039=0x6b

FIX vm_area_struct: Marking all objects used

---

Possibly related: afterwards I ran slabinfo --validate, and saw this message:

SLUB: vm_area_struct 1086 slabs counted but counter=1087

Is there enough info here to figure out who scribbled on the slab, or is it another mysterious bitflip? Process 369 is systemd-udevd, but process 28464 had already exited by the time I looked.

I had one previous "BUG" message from the same boot, "MAX_LOCKDEP_ENTRIES too low!", but it didn't seem worthy of a report. Saved details in case anyone is interested. I've also seen various stack depth reports that made me nervous, the latest being:

btrfs (2954) used greatest stack depth: 1856 bytes left

If i understand correctly, that means ~78% of the stack was used. Is 22% an adequate safety margin, or am I at risk of stack overflow? I don't use LVM or network mounts, so there shouldn't be that much layering involved.

This is bit flip again (from 0x6b to 0x4b). According to that the problems start to happen after replacing motherboard this looks most likely some hardware (or firmware) fault. There is still possibility of kernel bug, because with H/W you start to use different drivers, but usually kernel/driver memory corruption bugs are not single bit flip, but some memory override. 

I lunched kernel build with DEBUG_PAGEALLOC here:
http://koji.fedoraproject.org/koji/taskinfo?taskID=6434128
When driver or some kernel subsystem start to write to memory which do not belongs to it, that kernel will crash. That allow detect actual broken module. You can use debug_guardpage_minorder=1 (or 2) to increase protected area, hence increase probability to find faulty driver. Please try that kernel, possibly with rt2x00 driver to increase chance to catch corruption.

If it will not detect a fault, or kernel will crash in old way, that will probably mean that problem lies in H/W. In such case you can divide memory into two half then use only first and then second half, and see which one half do not cause corruption.

Other than that you can blacklist various modules you are using (i.e. visualization, networking, sound, ... ) and see if that prevent memory corruption.

Thank you for the DEBUG_PAGEALLOC kernel, which I've been running.

Here is another instance of slab corruption.

=============================================================================
BUG kmalloc-32 (Not tainted): Redzone overwritten
-----------------------------------------------------------------------------

Disabling lock debugging due to kernel taint
INFO: 0xffff880551f33dab-0xffff880551f33dab. First byte 0x8c instead of 0xcc
INFO: Allocated in radeon_fence_emit+0x2d/0x1c0 [radeon] age=119 cpu=3 pid=15902
        __slab_alloc+0x3eb/0x4fe
        kmem_cache_alloc_trace+0x2a8/0x360
        radeon_fence_emit+0x2d/0x1c0 [radeon]
        radeon_ib_schedule+0x222/0x2b0 [radeon]
        radeon_cs_ioctl+0x929/0xbb0 [radeon]
        drm_ioctl+0x512/0x650 [drm]
        do_vfs_ioctl+0x305/0x530
        SyS_ioctl+0x81/0xa0
        system_call_fastpath+0x16/0x1b
INFO: Freed in radeon_semaphore_free+0x55/0x70 [radeon] age=119 cpu=3 pid=15902
        __slab_free+0x3a/0x382
        kfree+0x2c0/0x2d0
        radeon_semaphore_free+0x55/0x70 [radeon]
        radeon_ib_schedule+0x1ce/0x2b0 [radeon]
        radeon_cs_ioctl+0x929/0xbb0 [radeon]
        drm_ioctl+0x512/0x650 [drm]
        do_vfs_ioctl+0x305/0x530
        SyS_ioctl+0x81/0xa0
        system_call_fastpath+0x16/0x1b
INFO: Slab 0xffffea001547cc80 objects=22 used=12 fp=0xffff880551f32438 flags=0x5ff00000004081
INFO: Object 0xffff880551f33d88 @offset=7560 fp=0xffff880551f325a0

Bytes b4 ffff880551f33d78: 4f e5 cd 01 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  O.......ZZZZZZZZ
Object ffff880551f33d88: 00 80 5f 0d 08 88 ff ff 00 00 00 00 6b 6b 6b 6b  .._.........kkkk
Object ffff880551f33d98: 00 00 00 00 00 00 00 00 00 00 00 00 6b 6b 6b a5  ............kkk.
Redzone ffff880551f33da8: cc cc cc 8c cc cc cc cc                          ........
Padding ffff880551f33ee8: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
CPU: 3 PID: 15902 Comm: mplayer Tainted: G    B        3.12.8-300.bz68171.fc20.x86_64+debug #1
Hardware name: Gigabyte Technology Co., Ltd. Z87-HD3/Z87-HD3, BIOS F6 08/03/2013
 ffff880551f33d88 ffff88057b37dae0 ffffffff8174a25b ffff880819004480
 ffff88057b37db20 ffffffff811d452d 0000000000000008 ffff880500000001
 ffff880551f33dac ffff880819004480 00000000000000cc ffff880551f33d88
Call Trace:
 [<ffffffff8174a25b>] dump_stack+0x54/0x74
 [<ffffffff811d452d>] print_trailer+0x14d/0x200
 [<ffffffff811d471f>] check_bytes_and_report+0xcf/0x110
 [<ffffffff811d5562>] check_object+0x192/0x250
 [<ffffffff81747b3d>] free_debug_processing+0xb9/0x22a
 [<ffffffff817534e6>] ? _raw_spin_unlock_irqrestore+0x36/0x70
 [<ffffffffa009e7e4>] ? radeon_fence_unref+0x34/0x40 [radeon]
 [<ffffffffa009e7e4>] ? radeon_fence_unref+0x34/0x40 [radeon]
 [<ffffffff81747ce8>] __slab_free+0x3a/0x382
 [<ffffffff81391e4e>] ? debug_check_no_obj_freed+0x14e/0x250
 [<ffffffff811d6e1c>] ? kfree+0xbc/0x2d0
 [<ffffffffa009e7e4>] ? radeon_fence_unref+0x34/0x40 [radeon]
 [<ffffffff811d7020>] kfree+0x2c0/0x2d0
 [<ffffffffa009e7e4>] radeon_fence_unref+0x34/0x40 [radeon]
 [<ffffffffa009ed5e>] radeon_sync_obj_unref+0xe/0x10 [radeon]
 [<ffffffffa006f87a>] ttm_bo_wait+0x13a/0x190 [ttm]
 [<ffffffffa00a15fe>] radeon_bo_wait+0x9e/0x140 [radeon]
 [<ffffffffa00b3c52>] radeon_gem_busy_ioctl+0x52/0x130 [radeon]
 [<ffffffffa0014e82>] drm_ioctl+0x512/0x650 [drm]
 [<ffffffff8130e1d5>] ? avc_has_perm+0x25/0x350
 [<ffffffff81310773>] ? inode_has_perm.isra.48+0x53/0x80
 [<ffffffff8120c7e5>] do_vfs_ioctl+0x305/0x530
 [<ffffffff81310dab>] ? selinux_file_ioctl+0x5b/0x110
 [<ffffffff8120ca91>] SyS_ioctl+0x81/0xa0
 [<ffffffff8175d569>] system_call_fastpath+0x16/0x1b
FIX kmalloc-32: Restoring 0xffff880551f33dab-0xffff880551f33dab=0xcc

=============================================================================
BUG kmalloc-32 (Tainted: G    B       ): Redzone overwritten
-----------------------------------------------------------------------------

INFO: 0xffff880551f33da8-0xffff880551f33daf. First byte 0xcc instead of 0xbb
INFO: Allocated in radeon_fence_emit+0x2d/0x1c0 [radeon] age=998 cpu=3 pid=15902
        __slab_alloc+0x3eb/0x4fe
        kmem_cache_alloc_trace+0x2a8/0x360
        radeon_fence_emit+0x2d/0x1c0 [radeon]
        radeon_ib_schedule+0x222/0x2b0 [radeon]
        radeon_cs_ioctl+0x929/0xbb0 [radeon]
        drm_ioctl+0x512/0x650 [drm]
        do_vfs_ioctl+0x305/0x530
        SyS_ioctl+0x81/0xa0
        system_call_fastpath+0x16/0x1b
INFO: Freed in radeon_semaphore_free+0x55/0x70 [radeon] age=998 cpu=3 pid=15902
        __slab_free+0x3a/0x382
        kfree+0x2c0/0x2d0
        radeon_semaphore_free+0x55/0x70 [radeon]
        radeon_ib_schedule+0x1ce/0x2b0 [radeon]
        radeon_cs_ioctl+0x929/0xbb0 [radeon]
        drm_ioctl+0x512/0x650 [drm]
        do_vfs_ioctl+0x305/0x530
        SyS_ioctl+0x81/0xa0
        system_call_fastpath+0x16/0x1b
INFO: Slab 0xffffea001547cc80 objects=22 used=21 fp=0xffff880551f32e10 flags=0x5ff00000004080
INFO: Object 0xffff880551f33d88 @offset=7560 fp=0xffff880551f325a0

Bytes b4 ffff880551f33d78: 08 e9 cd 01 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a  ........ZZZZZZZZ
Object ffff880551f33d88: 00 80 5f 0d 08 88 ff ff 00 00 00 00 6b 6b 6b 6b  .._.........kkkk
Object ffff880551f33d98: 00 00 00 00 00 00 00 00 00 00 00 00 6b 6b 6b a5  ............kkk.
Redzone ffff880551f33da8: cc cc cc cc cc cc cc cc                          ........
Padding ffff880551f33ee8: 5a 5a 5a 5a 5a 5a 5a 5a                          ZZZZZZZZ
CPU: 1 PID: 16782 Comm: dinoshade Tainted: G    B        3.12.8-300.bz68171.fc20.x86_64+debug #1
Hardware name: Gigabyte Technology Co., Ltd. Z87-HD3/Z87-HD3, BIOS F6 08/03/2013
 ffff880551f33d88 ffff880566ead8e8 ffffffff8174a25b ffff880819004480
 ffff880566ead928 ffffffff811d452d 0000000000000008 ffff880500000001
 ffff880551f33db0 ffff880819004480 00000000000000bb ffff880551f33d88
Call Trace:
 [<ffffffff8174a25b>] dump_stack+0x54/0x74
 [<ffffffff811d452d>] print_trailer+0x14d/0x200
 [<ffffffff811d471f>] check_bytes_and_report+0xcf/0x110
 [<ffffffff811d5562>] check_object+0x192/0x250
 [<ffffffffa00fcf5c>] ? radeon_semaphore_create+0x2c/0xe0 [radeon]
 [<ffffffff817479e2>] alloc_debug_processing+0x76/0x118
 [<ffffffff8174861d>] __slab_alloc+0x3eb/0x4fe
 [<ffffffffa00fcf5c>] ? radeon_semaphore_create+0x2c/0xe0 [radeon]
 [<ffffffffa00fdacb>] ? radeon_sa_bo_new+0x27b/0x480 [radeon]
 [<ffffffff811d7e98>] kmem_cache_alloc_trace+0x2a8/0x360
 [<ffffffffa00fcf5c>] ? radeon_semaphore_create+0x2c/0xe0 [radeon]
 [<ffffffffa00fcf5c>] radeon_semaphore_create+0x2c/0xe0 [radeon]
 [<ffffffffa00b4700>] radeon_ib_get+0x50/0x110 [radeon]
 [<ffffffffa00b749d>] radeon_cs_ioctl+0x82d/0xbb0 [radeon]
 [<ffffffffa0014e82>] drm_ioctl+0x512/0x650 [drm]
 [<ffffffff81310773>] ? inode_has_perm.isra.48+0x53/0x80
 [<ffffffff8120c7e5>] do_vfs_ioctl+0x305/0x530
 [<ffffffff81310dab>] ? selinux_file_ioctl+0x5b/0x110
 [<ffffffff8120ca91>] SyS_ioctl+0x81/0xa0
 [<ffffffff8175d569>] system_call_fastpath+0x16/0x1b
FIX kmalloc-32: Restoring 0xffff880551f33da8-0xffff880551f33daf=0xbb

FIX kmalloc-32: Marking all objects used

---

My main question: does this look like another hardware bitflip?

I guess it does, if I'm reading these correctly - if the second reported "redzone overwritten" is just a false alarm, a consequence of SLUB getting confused as to what the redzone should be after the first redzone corruption was repaired.

I still don't have a good way to reproduce these errors. I guess I'll just have to try memtest86 again.

> My main question: does this look like another hardware bitflip?
Yes.

Sorry for not mension that before, it is better to run DEBUG_PAGEALLOC kernel on non-debug fedora kernel variant, as then it have more chance to catch driver corruption.

Anyway, problem unfortunately looks more like hardware issue and removing components (i.e stop using some cpu features or some other hardware by blacklisting modules, remove half of physical memory modules, etc.) can be better strategy to figure what cause corruption.

Other users reported corruption on Gigabyte *87* boards as well, so this really looks like h/w problem.

Some people report that using only 2 DIMM slots helped for them:
http://www.tonymacx86.com/general-help/121042-solved-4-dimm-crashing-freezing-ga-z87.html

I think this bug can be closed as duplicate of 
https://bugzilla.kernel.org/show_bug.cgi?id=64521

*** This bug has been marked as a duplicate of bug 64521 ***