Hello. I have a PC with Intel N100 (with PCI device 8086:461c, DID_ADL_N_SKU4 in igen6_edac.c) with a BIOS/UEFI that allows me to enable In-Band ECC (IBECC). ECC polling seems operational in PassMark Memtest86 V11.0 Free version (scenario A below). Interrupt seems not working in Linux, but it seems able to detect the error when "modprobe igen6_edac" (scenario B and C below). Also "rmmod igen6_edac" can trigger segmentation fault (scenario D below). I am not familiar with this area, but I am not sure if this is a kernel bug or BIOS bug. If there is a BIOS bug causing this, is there any chance to workaround this by using polling instead? Thanks in advance. I am happy to test different version of kernel or enable CONFIG_EDAC_DEBUG to collect more logs if needed. Preparation 1. Install Ubuntu 24.04 Desktop 2. Install kernel from mainline PPA, at https://kernel.ubuntu.com/mainline/v6.11/amd64/ which is based on 059f3ce2575f0de9821a9f4e3765e75698e197b6 3. Add "blacklist igen6_edac" to /etc/modprobe.d/ 4. Enter BIOS and enable In-band ECC support In-Band ECC Support: Enabled In-Band ECC Operation Mode: 2 (make all requests protected and ignore range checks) IBECC Error Injection Control: Inject Correctable Error on insertion counter Error Injaction Insertion Count: 251658240 (0xf000000) "Machine Check" settings in BIOS remain enabled (default) Reproduce steps: scenario A 1. Power off PC 2. Power on 3. Boot PassMark Memtest86 V11.0 Free version with "ECC Polling" enabled (default), it will report correctable IBECC error about once every 4 seconds. Reproduce steps: scenario B 1. Power off PC 2. Power on 3. Boot Linux 4. Run "modprobe igen6_edac" It will print the following message without memory error (this is expected) [ 13.746402] resource: resource sanity check: requesting [mem 0x00000000fedc0000-0x00000000fedcffff], which spans more than pnp 00:04 [mem 0xfedc0000-0xfedc7fff] [ 13.746417] caller igen6_probe+0x192/0x590 [igen6_edac] mapping multiple BARs [ 13.746487] EDAC MC0: Giving out device to module igen6_edac controller Intel_client_SoC MC#0: DEV 0000:00:00.0 (INTERRUPT) [ 13.746577] EDAC igen6: v2.5.1 "ras-mc-ctl --error-count" report zero for both CE and UE (this is expected) If I run "memtester 1M 100" will not trigger memory error (although it probably should, refer to scenario C for comparison) Reproduce steps: scenario C 1. Power off PC 2. Power on 3. Boot Linux 4. Run "memtester 1M 100" 5. Run "modprobe igen6_edac" It will generate the following kernel message with memory error (this is expected) [ 38.197670] resource: resource sanity check: requesting [mem 0x00000000fedc0000-0x00000000fedcffff], which spans more than pnp 00:04 [mem 0xfedc0000-0xfedc7fff] [ 38.197690] caller igen6_probe+0x192/0x590 [igen6_edac] mapping multiple BARs [ 38.197785] EDAC MC0: Giving out device to module igen6_edac controller Intel_client_SoC MC#0: DEV 0000:00:00.0 (INTERRUPT) [ 38.197954] EDAC igen6: v2.5.1 [ 38.198671] EDAC igen6 MC0: HANDLING IBECC MEMORY ERROR [ 38.198678] EDAC igen6 MC0: ADDR 0x372d95c40 [ 38.198715] EDAC MC0: 1 CE on MC#0_Chan#0_DIMM#0 (channel:0 slot:0 page:0x372d95 offset:0xc40 grain:64 syndrome:0x1) "ras-mc-ctl --error-count" report 1 for correctable error (this is epxected) However, running "memtester 1M 100" again will not trigger more memory error (although it probably should, so I suspect interrupt seems not working) Reproduce steps: scenario D Running "rmmod igen6_edac" will print "Segmentation fault" with the following kernel message [ 14.340068] EDAC MC0: Giving out device to module igen6_edac controller Intel_client_SoC MC#0: DEV 0000:00:00.0 (INTERRUPT) [ 14.340156] EDAC igen6: v2.5.1 [ 31.062124] EDAC MC: Removed device 0 for igen6_edac Intel_client_SoC MC#0: DEV 0000:00:00.0 [ 31.062190] ------------[ cut here ]------------ [ 31.062194] kernel BUG at mm/slub.c:553! [ 31.062215] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 31.062226] CPU: 2 UID: 0 PID: 2242 Comm: rmmod Not tainted 6.11.0-061100-generic #202409151536 [ 31.062236] Hardware name: AZW MINI S/MINI S, BIOS ADLNV105 12/12/2023 [ 31.062241] RIP: 0010:kfree+0x2be/0x2e0 [ 31.062260] Code: e7 e8 d6 c7 ff ff e9 89 fe ff ff 4d 89 f1 41 b8 01 00 00 00 48 89 d9 48 89 da 4c 89 e6 4c 89 ef e8 77 f8 ff ff e9 6a fe ff ff <0f> 0b 48 85 c0 0f 84 79 ff ff ff e9 06 ff ff ff 48 8b 15 0b ec fc [ 31.062268] RSP: 0018:ffffaa17c1043960 EFLAGS: 00010246 [ 31.062277] RAX: ffff985306ad3800 RBX: ffff985306ad3800 RCX: ffff985306ad3c00 [ 31.062283] RDX: 0000000000232002 RSI: ffffffffc1e5670f RDI: ffff985306ad3800 [ 31.062288] RBP: ffffaa17c10439a0 R08: 0000000000000000 R09: 0000000000000000 [ 31.062293] R10: 0000000000000000 R11: 0000000000000000 R12: fffff113441ab400 [ 31.062297] R13: ffff98530004a400 R14: ffffffffc1e5670f R15: ffffffffc1e594c8 [ 31.062302] FS: 00007b99fe7d1080(0000) GS:ffff985650300000(0000) knlGS:0000000000000000 [ 31.062309] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 31.062314] CR2: 00007b99fde543e0 CR3: 0000000105ffa000 CR4: 0000000000f50ef0 [ 31.062321] PKRU: 55555554 [ 31.062325] Call Trace: [ 31.062330] <TASK> [ 31.062336] ? show_trace_log_lvl+0x1be/0x310 [ 31.062352] ? show_trace_log_lvl+0x1be/0x310 [ 31.062371] ? igen6_remove+0x6f/0x90 [igen6_edac] [ 31.062384] ? show_regs.part.0+0x22/0x30 [ 31.062390] ? __die_body.cold+0x8/0x10 [ 31.062397] ? __die+0x2a/0x40 [ 31.062402] ? die+0x2f/0x60 [ 31.062410] ? do_trap+0xc8/0x110 [ 31.062421] ? do_error_trap+0x71/0xb0 [ 31.062431] ? kfree+0x2be/0x2e0 [ 31.062442] ? exc_invalid_op+0x52/0x80 [ 31.062454] ? kfree+0x2be/0x2e0 [ 31.062463] ? asm_exc_invalid_op+0x1b/0x20 [ 31.062474] ? igen6_remove+0x6f/0x90 [igen6_edac] [ 31.062483] ? igen6_remove+0x6f/0x90 [igen6_edac] [ 31.062490] ? kfree+0x2be/0x2e0 [ 31.062502] igen6_remove+0x6f/0x90 [igen6_edac] [ 31.062510] pci_device_remove+0x3e/0xb0 [ 31.062523] device_remove+0x40/0x80 [ 31.062535] device_release_driver_internal+0x206/0x270 [ 31.062544] driver_detach+0x4a/0xa0 [ 31.062551] bus_remove_driver+0x83/0x110 [ 31.062562] driver_unregister+0x2f/0x60 [ 31.062569] pci_unregister_driver+0x40/0x90 [ 31.062580] igen6_exit+0x10/0x560 [igen6_edac] [ 31.062588] __do_sys_delete_module.isra.0+0x19f/0x2e0 [ 31.062600] __x64_sys_delete_module+0x12/0x20 [ 31.062607] x64_sys_call+0x1104/0x22b0 [ 31.062617] do_syscall_64+0x7e/0x170 [ 31.062624] ? __fput_sync+0x1c/0x30 [ 31.062635] ? syscall_exit_to_user_mode+0x4e/0x250 [ 31.062645] ? do_syscall_64+0x8a/0x170 [ 31.062652] ? filemap_map_pages+0x34f/0x570 [ 31.062663] ? vfs_read+0x16a/0x380 [ 31.062673] ? do_read_fault+0xfd/0x1d0 [ 31.062684] ? do_fault+0x1ae/0x300 [ 31.062694] ? handle_pte_fault+0x12a/0x1c0 [ 31.062701] ? __handle_mm_fault+0x3d5/0x7a0 [ 31.062709] ? __count_memcg_events+0x86/0x160 [ 31.062719] ? count_memcg_events.constprop.0+0x2a/0x50 [ 31.062727] ? handle_mm_fault+0x1bb/0x2d0 [ 31.062734] ? do_user_addr_fault+0x5e9/0x7e0 [ 31.062744] ? irqentry_exit_to_user_mode+0x43/0x250 [ 31.062754] ? irqentry_exit+0x43/0x50 [ 31.062763] ? exc_page_fault+0x96/0x1c0 [ 31.062772] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 31.062780] RIP: 0033:0x7b99fdf2ac9b [ 31.062788] Code: 73 01 c3 48 8b 0d 7d 81 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 4d 81 0d 00 f7 d8 64 89 01 48 [ 31.062794] RSP: 002b:00007ffc8884ea88 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0 [ 31.062802] RAX: ffffffffffffffda RBX: 00005bdb5457e720 RCX: 00007b99fdf2ac9b [ 31.062807] RDX: 0000000000000000 RSI: 0000000000000800 RDI: 00005bdb5457e788 [ 31.062812] RBP: 00007ffc8884eab0 R08: 1999999999999999 R09: 0000000000000000 [ 31.062817] R10: 00007b99fdfb1fc0 R11: 0000000000000206 R12: 0000000000000000 [ 31.062822] R13: 00007ffc8884ed00 R14: 00005bdb5457e720 R15: 0000000000000000 [ 31.062831] </TASK> [ 31.062835] Modules linked in: igen6_edac(-) rfcomm snd_seq_dummy snd_hrtimer cmac algif_hash algif_skcipher af_alg qrtr bnep snd_sof_pci_intel_tgl snd_sof_pci_intel_cnl snd_sof_intel_hda_generic soundwire_intel soundwire_cadence snd_sof_intel_hda_common snd_sof_intel_hda_mlink snd_sof_intel_hda snd_sof_pci snd_hda_codec_hdmi snd_sof_xtensa_dsp snd_sof snd_hda_codec_realtek xe snd_hda_codec_generic snd_sof_utils snd_hda_scodec_component snd_soc_hdac_hda snd_soc_acpi_intel_match soundwire_generic_allocation snd_soc_acpi soundwire_bus drm_gpuvm drm_exec gpu_sched drm_suballoc_helper drm_ttm_helper snd_soc_avs snd_soc_hda_codec snd_hda_ext_core snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine binfmt_misc snd_hda_intel intel_rapl_msr snd_intel_dspcfg intel_rapl_common snd_intel_sdw_acpi snd_hda_codec snd_hda_core snd_hwdep x86_pkg_temp_thermal snd_pcm intel_powerclamp iwlmvm coretemp cmdlinepart snd_seq_midi spi_nor snd_seq_midi_event nls_iso8859_1 kvm_intel mtd snd_rawmidi ee1004 i915 mac80211 kvm libarc4 snd_seq [ 31.062997] crct10dif_pclmul polyval_clmulni polyval_generic btusb ghash_clmulni_intel drm_buddy sha256_ssse3 ttm sha1_ssse3 btrtl snd_seq_device aesni_intel btintel snd_timer crypto_simd iwlwifi drm_display_helper cryptd snd cec i2c_i801 btbcm rapl rc_core btmtk spi_intel_pci i2c_mux soundcore intel_cstate wmi_bmof spi_intel i2c_smbus cfg80211 bluetooth mei_me mei i2c_algo_bit intel_pmc_core intel_vsec pmt_telemetry acpi_tad pmt_class acpi_pad input_leds joydev mac_hid sch_fq_codel msr parport_pc ppdev lp parport efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq libcrc32c hid_generic usbhid hid spi_pxa2xx_platform dw_dmac dw_dmac_core 8250_dw spi_pxa2xx_core crc32_pclmul r8169 intel_lpss_pci ahci realtek libahci intel_lpss xhci_pci idma64 xhci_pci_renesas video wmi pinctrl_alderlake [ 31.063215] ---[ end trace 0000000000000000 ]--- [ 33.510109] RIP: 0010:kfree+0x2be/0x2e0 [ 33.510118] Code: e7 e8 d6 c7 ff ff e9 89 fe ff ff 4d 89 f1 41 b8 01 00 00 00 48 89 d9 48 89 da 4c 89 e6 4c 89 ef e8 77 f8 ff ff e9 6a fe ff ff <0f> 0b 48 85 c0 0f 84 79 ff ff ff e9 06 ff ff ff 48 8b 15 0b ec fc [ 33.510121] RSP: 0018:ffffaa17c1043960 EFLAGS: 00010246 [ 33.510123] RAX: ffff985306ad3800 RBX: ffff985306ad3800 RCX: ffff985306ad3c00 [ 33.510125] RDX: 0000000000232002 RSI: ffffffffc1e5670f RDI: ffff985306ad3800 [ 33.510126] RBP: ffffaa17c10439a0 R08: 0000000000000000 R09: 0000000000000000 [ 33.510127] R10: 0000000000000000 R11: 0000000000000000 R12: fffff113441ab400 [ 33.510129] R13: ffff98530004a400 R14: ffffffffc1e5670f R15: ffffffffc1e594c8 [ 33.510130] FS: 00007b99fe7d1080(0000) GS:ffff985650300000(0000) knlGS:0000000000000000 [ 33.510132] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 33.510134] CR2: 00007b99fde543e0 CR3: 0000000105ffa000 CR4: 0000000000f50ef0 [ 33.510135] PKRU: 55555554
Hello. I dig into the code and created the following patch (based on 6.11.5 05b1367d372aca98a4e09c1a0e7ff0b9d721b2bc), and it seems to resolve the segmentation fault and to support polling. The segmentation fault happens because During modprobe: 1. In igen6_probe(), igen6_pvt will be allocated with kzalloc() 2. In igen6_register_mci(), mci->pvt_info will point to &igen6_pvt->imc[mc] During rmmod: 1. In mci_release() in edac_mc.c, it will kfree(mci->pvt_info) 2. In igen6_remove(), it will kfree(igen6_pvt); And that caused double kfree on the same memory address. My proposal is to set mci->pvt_info to NULL to avoid double-kfree. The other part of the patch is to support polling. I have test it on my machine (N100, with PCI device 8086:461c, DID_ADL_N_SKU4) and it seems to work as expected. I also tried to "modprobe igen6_edac edac_op_state=0" and "rmmod" repeatedly and did not observe any issue. Thank you. diff --git a/drivers/edac/igen6_edac.c b/drivers/edac/igen6_edac.c index 189a2fc29e74..5027070410a5 100644 --- a/drivers/edac/igen6_edac.c +++ b/drivers/edac/igen6_edac.c @@ -1170,6 +1170,19 @@ static int igen6_pci_setup(struct pci_dev *pdev, u64 *mchbar) return -ENODEV; } +static void igen6_check(struct mem_ctl_info *mci) +{ + struct igen6_imc *imc = mci->pvt_info; + + /* errsts_clear() isn't NMI-safe. Delay it in the IRQ context */ + u64 ecclog = ecclog_read_and_clear(imc); + if (!ecclog) + return; + if (!ecclog_gen_pool_add(imc->mc, ecclog)) + irq_work_queue(&ecclog_irq_work); + +} + static int igen6_register_mci(int mc, u64 mchbar, struct pci_dev *pdev) { struct edac_mc_layer layers[2]; @@ -1211,6 +1224,9 @@ static int igen6_register_mci(int mc, u64 mchbar, struct pci_dev *pdev) mci->edac_cap = EDAC_FLAG_SECDED; mci->mod_name = EDAC_MOD_STR; mci->dev_name = pci_name(pdev); + if (edac_op_state == EDAC_OPSTATE_POLL) { + mci->edac_check = igen6_check; + } mci->pvt_info = &igen6_pvt->imc[mc]; imc = mci->pvt_info; @@ -1245,6 +1261,7 @@ static int igen6_register_mci(int mc, u64 mchbar, struct pci_dev *pdev) imc->mci = mci; return 0; fail3: + mci->pvt_info = NULL; kfree(mci->ctl_name); fail2: edac_mc_free(mci); @@ -1269,6 +1286,7 @@ static void igen6_unregister_mcis(void) edac_mc_del_mc(mci->pdev); kfree(mci->ctl_name); + mci->pvt_info = NULL; edac_mc_free(mci); iounmap(imc->window); } @@ -1448,7 +1466,9 @@ static int __init igen6_init(void) if (owner && strncmp(owner, EDAC_MOD_STR, sizeof(EDAC_MOD_STR))) return -EBUSY; - edac_op_state = EDAC_OPSTATE_NMI; + if (edac_op_state == EDAC_OPSTATE_INVAL) { + edac_op_state = EDAC_OPSTATE_NMI; + } rc = pci_register_driver(&igen6_driver); if (rc) @@ -1472,3 +1492,6 @@ module_exit(igen6_exit); MODULE_LICENSE("GPL v2"); MODULE_AUTHOR("Qiuxu Zhuo"); MODULE_DESCRIPTION("MC Driver for Intel client SoC using In-Band ECC"); + +module_param(edac_op_state, int, 0444); +MODULE_PARM_DESC(edac_op_state, "EDAC Error Reporting state: 0=Poll,1=NMI. Default=1");