Created attachment 300582 [details] system log about emmc issue We have encountered the following problems in the operation of our equipment. --------------------------------- [2660450.914500] mmc0: Timeout waiting for hardware interrupt. [2660450.918756] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x400e00 [2660524.052290] <LWLog:Info:> [LinkNotify] Type=33[Link Down] Subype=0 Id=533109 Value=0 SessionId=0 [2660525.067924] <LWLog:Info:> [LinkNotify] Type=32[Link UP] Subype=0 Id=533109 Value=104 SessionId=0 [2660650.961923] INFO: task xfsaild/dm-0:420 blocked for more than 120 seconds. [2660650.961927] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [2660650.961930] xfsaild/dm-0 D ffff880175516000 0 420 2 0x00000000 [2660650.961935] ffff880035c13d58 0000000000000046 ffff880035d48fb0 ffff880035c13fd8 [2660650.961939] ffff880035c13fd8 ffff880035c13fd8 ffff880035d48fb0 ffff8801756b0e00 [2660650.961942] 0000000000000000 ffff880035d48fb0 ffff880035d57128 ffff880175516000 [2660650.961945] Call Trace: [2660650.961955] [<ffffffff8168c7f9>] schedule+0x29/0x70 [2660650.961999] [<ffffffffa02523ed>] _xfs_log_force+0x1bd/0x2b0 [xfs] [2660650.962023] [<ffffffff810c54e0>] ? wake_up_state+0x20/0x20 [2660650.962049] [<ffffffffa0252506>] xfs_log_force+0x26/0x80 [xfs] [2660650.962077] [<ffffffffa025ddd0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] [2660650.962113] [<ffffffffa025df2a>] xfsaild+0x15a/0x660 [xfs] [2660650.962139] [<ffffffffa025ddd0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs] [2660650.962143] [<ffffffff810b0a4f>] kthread+0xcf/0xe0 [2660650.962146] [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 [2660650.962150] [<ffffffff81697758>] ret_from_fork+0x58/0x90 [2660650.962153] [<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140 [2660650.962158] INFO: task auditd:674 blocked for more than 120 seconds. [2660650.962160] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [2660650.962161] auditd D ffffffff8168a850 0 674 1 0x00000000 [2660650.962165] ffff88000001fc00 0000000000000086 ffff8801755cbec0 ffff88000001ffd8 [2660650.962168] ffff88000001ffd8 ffff88000001ffd8 ffff8801755cbec0 ffff88017fc96c40 [2660650.962171] 0000000000000000 7fffffffffffffff ffff88017ff985e8 ffffffff8168a850 [2660650.962174] Call Trace: [2660650.962178] [<ffffffff8168a850>] ? bit_wait+0x50/0x50 [2660650.962181] [<ffffffff8168c7f9>] schedule+0x29/0x70 [2660650.962184] [<ffffffff8168a239>] schedule_timeout+0x239/0x2c0 [2660650.962188] [<ffffffff8130d9ab>] ? deadline_add_request+0x5b/0x70 [2660650.962193] [<ffffffff810eb0dc>] ? ktime_get_ts64+0x4c/0xf0 [2660650.962196] [<ffffffff8168a850>] ? bit_wait+0x50/0x50 [2660650.962199] [<ffffffff8168bd9e>] io_schedule_timeout+0xae/0x130 [2660650.962202] [<ffffffff8168be38>] io_schedule+0x18/0x20 [2660650.962205] [<ffffffff8168a861>] bit_wait_io+0x11/0x50 [2660650.962208] [<ffffffff8168a385>] __wait_on_bit+0x65/0x90 [2660650.962212] [<ffffffff81180221>] wait_on_page_bit+0x81/0xa0 [2660650.962215] [<ffffffff810b1be0>] ? wake_bit_function+0x40/0x40 [2660650.962218] [<ffffffff81180351>] __filemap_fdatawait_range+0x111/0x190 [2660650.962222] [<ffffffff8118d94e>] ? do_writepages+0x1e/0x40 [2660650.962225] [<ffffffff811803e4>] filemap_fdatawait_range+0x14/0x30 [2660650.962228] [<ffffffff81182666>] filemap_write_and_wait_range+0x56/0x90 [2660650.962253] [<ffffffffa02332b6>] xfs_file_fsync+0x66/0x1e0 [xfs] [2660650.962258] [<ffffffff81230695>] do_fsync+0x65/0xa0 [2660650.962261] [<ffffffff81230960>] SyS_fsync+0x10/0x20 [2660650.962264] [<ffffffff81697809>] system_call_fastpath+0x16/0x1b [2661050.917711] mmc0: Card stuck in programming state! mmcblk0 card_busy_detect [2661050.917727] blk_update_request: I/O error, dev mmcblk0, sector 35661352 [2661050.917731] blk_update_request: I/O error, dev mmcblk0, sector 35661360 [2661050.917734] blk_update_request: I/O error, dev mmcblk0, sector 35661368 [2661050.917736] blk_update_request: I/O error, dev mmcblk0, sector 35661376 [2661050.917739] blk_update_request: I/O error, dev mmcblk0, sector 35661384 [2661050.917742] blk_update_request: I/O error, dev mmcblk0, sector 35661392 [2661050.917744] blk_update_request: I/O error, dev mmcblk0, sector 35661400 [2661050.917747] blk_update_request: I/O error, dev mmcblk0, sector 35661408 [2661050.917750] blk_update_request: I/O error, dev mmcblk0, sector 35661416 [2661050.917752] blk_update_request: I/O error, dev mmcblk0, sector 35661424 [2661050.917827] Buffer I/O error on dev dm-0, logical block 3431109, lost async page write [2661050.917836] Buffer I/O error on dev dm-0, logical block 3431110, lost async page write [2661050.917839] Buffer I/O error on dev dm-0, logical block 3431111, lost async page write [2661050.917841] Buffer I/O error on dev dm-0, logical block 3431112, lost async page write [2661050.917844] Buffer I/O error on dev dm-0, logical block 3431113, lost async page write [2661050.917846] Buffer I/O error on dev dm-0, logical block 3431114, lost async page write [2661050.917848] Buffer I/O error on dev dm-0, logical block 3431115, lost async page write [2661050.917851] Buffer I/O error on dev dm-0, logical block 3431116, lost async page write [2661050.917853] Buffer I/O error on dev dm-0, logical block 3431117, lost async page write [2661050.917855] Buffer I/O error on dev dm-0, logical block 3431118, lost async page write [2661050.926374] mmc0: cache flush error -110 [2661050.934646] mmc0: cache flush error -110 [2661060.961667] mmc0: Timeout waiting for hardware interrupt. [2661060.963680] ------------[ cut here ]------------ [2661060.963697] WARNING: at drivers/mmc/host/sdhci.c:1004 sdhci_send_command+0x355/0x3b0 [sdhci]() [2661060.963699] Modules linked in: lwconn(OE) sm4(OE) igb(OE) ixgbe(OE) bridge stp llc sha512_ssse3 sha512_generic drbg ansi_cprng intel_powerclamp coretemp iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw ipmi_ssif gf128mul glue_helper qat_c3xxx ablk_helper i2c_i801 cryptd pcspkr intel_qat authenc i2c_algo_bit ipmi_msghandler i2c_ismt shpchp i2c_core ip_tables xfs libcrc32c mmc_block sdhci_pci crct10dif_pclmul crct10dif_common dca sdhci ptp mmc_core crc32c_intel pps_core ahci libahci libata fjes dm_mirror dm_region_hash dm_log dm_mod cdc_ether(OE) GobiNet(OE) qmi_wwan_q(OE) cdc_wdm usbnet(OE) mii option(OE) usb_wwan(OE) nf_conntrack [last unloaded: igb] [2661060.963743] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G W OE ------------ 3.10.0-514.26.2.el7.x86_64 #1 [2661060.963745] Hardware name: Default string Default string/LW3002-XH, BIOS 5.13 (G168-027) 11/16/2020 [2661060.963747] 0000000000000000 cd6e6ca033004373 ffff88017fd03d28 ffffffff81687133 [2661060.963751] ffff88017fd03d60 ffffffff81085cb0 ffff8800365da500 ffff8800365da500 [2661060.963754] 0000000000000003 ffff880175525930 ffff8800365da500 ffff88017fd03d70 [2661060.963757] Call Trace: [2661060.963759] <IRQ> [<ffffffff81687133>] dump_stack+0x19/0x1b [2661060.963770] [<ffffffff81085cb0>] warn_slowpath_common+0x70/0xb0 [2661060.963773] [<ffffffff81085dfa>] warn_slowpath_null+0x1a/0x20 [2661060.963778] [<ffffffffa015ff45>] sdhci_send_command+0x355/0x3b0 [sdhci] [2661060.963783] [<ffffffffa01600b0>] sdhci_finish_data+0x110/0x370 [sdhci] [2661060.963788] [<ffffffffa0160310>] ? sdhci_finish_data+0x370/0x370 [sdhci] [2661060.963793] [<ffffffffa016038d>] sdhci_timeout_timer+0x7d/0xc0 [sdhci] [2661060.963796] [<ffffffff81095eb6>] call_timer_fn+0x36/0x110 [2661060.963801] [<ffffffffa0160310>] ? sdhci_finish_data+0x370/0x370 [sdhci] [2661060.963804] [<ffffffff81098ba7>] run_timer_softirq+0x237/0x340 [2661060.963808] [<ffffffff8108f63f>] __do_softirq+0xef/0x280 [2661060.963812] [<ffffffff8169929c>] call_softirq+0x1c/0x30 [2661060.963816] [<ffffffff8102d365>] do_softirq+0x65/0xa0 [2661060.963819] [<ffffffff8108f9d5>] irq_exit+0x115/0x120 [2661060.963822] [<ffffffff81699f15>] smp_apic_timer_interrupt+0x45/0x60 [2661060.963824] [<ffffffff8169845d>] apic_timer_interrupt+0x6d/0x80 [2661060.963826] <EOI> [<ffffffff81514eb2>] ? cpuidle_enter_state+0x52/0xc0 [2661060.963832] [<ffffffff81514ff9>] cpuidle_idle_call+0xd9/0x210 [2661060.963836] [<ffffffff8103512e>] arch_cpu_idle+0xe/0x30 [2661060.963840] [<ffffffff810e82f5>] cpu_startup_entry+0x245/0x290 [2661060.963844] [<ffffffff8104f0da>] start_secondary+0x1ba/0x230 [2661060.963847] ---[ end trace 42ca8c72f9e7a97b ]--- [2661060.966757] mmcblk0: error -110 sending stop command, original cmd response 0x0, card status 0x400e00 [2661309.490931] <LWLog:Info:> [LinkNotify] Type=33[Link Down] Subype=0 Id=533109 Value=0 SessionId=0 [2661310.515420] <LWLog:Info:> [LinkNotify] Type=32[Link UP] Subype=0 Id=533109 Value=105 SessionId=0 [2661660.965885] mmc0: Card stuck in programming state! mmcblk0 card_busy_detect [2661660.965899] blk_update_request: 84 callbacks suppressed [2661660.965902] blk_update_request: I/O error, dev mmcblk0, sector 34660233 [2661660.965904] blk_update_request: I/O error, dev mmcblk0, sector 34660241 [2661660.965906] blk_update_request: I/O error, dev mmcblk0, sector 34660249 [2661660.965939] XFS (dm-0): metadata I/O error: block 0x1938f89 ("xlog_iodone") error 5 numblks 64 [2661660.974849] XFS (dm-0): xfs_do_force_shutdown(0x2) called from line 1203 of file fs/xfs/xfs_log.c. Return address = 0xffffffffa0250590 [2661660.975005] XFS (dm-0): Log I/O Error Detected. Shutting down filesystem [2661660.975086] XFS (dm-0): xfs_log_force: error -5 returned. [2661660.975124] XFS (dm-0): xfs_log_force: error -5 returned. [2661660.982191] XFS (dm-0): Please umount the filesystem and rectify the problem(s) [2661661.025893] XFS (dm-0): xfs_log_force: error -5 returned. -------------------------