[11763.234840] sd 9:0:4:0: device_block, handle(0x000d) [11771.230286] sd 9:0:4:0: device_unblock and setting to running, handle(0x000d) [11771.237895] sd 9:0:4:0: [sde] tag#7508 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=8s [11771.252976] sd 9:0:4:0: [sde] tag#7508 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00 [11771.261127] I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0 [11771.269711] sd 9:0:4:0: [sde] tag#7515 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=8s [11771.285980] sd 9:0:4:0: [sde] tag#7515 CDB: Write(10) 2a 00 2f e5 6a 80 00 00 68 00 [11771.294540] I/O error, dev sde, sector 803564160 op 0x1:(WRITE) flags 0x8800 phys_seg 13 prio class 0 [11771.311956] sd 9:0:4:0: [sde] tag#7516 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK cmd_age=8s [11771.330320] sd 9:0:4:0: [sde] tag#7516 CDB: Write(10) 2a 00 2f e5 6a e8 00 00 18 00 [11771.339884] I/O error, dev sde, sector 803564264 op 0x1:(WRITE) flags 0x8800 phys_seg 3 prio class 0 [11771.359290] I/O error, dev sde, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 0 [11771.369582] device-mapper: thin metadata: pre-commit callback failed [11771.379885] device-mapper: thin: 253:6: metadata operation 'dm_pool_commit_metadata' failed: error = -5 [11771.401546] device-mapper: thin: 253:6: aborting current metadata transaction [11771.414737] sd 9:0:4:0: [sde] Synchronizing SCSI cache [11771.425364] sd 9:0:4:0: [sde] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK [11771.438864] ------------[ cut here ]------------ [11771.447026] mpt3sas_cm1: mpt3sas_transport_port_remove: removed: sas_addr(0x5003048029c4e103) [11771.457023] workqueue: WQ_MEM_RECLAIM dm-thin:do_worker [dm_thin_pool] is flushing !WQ_MEM_RECLAIM events:flush_cpu_slab [11771.467588] mpt3sas_cm1: removing handle(0x000d), sas_addr(0x5003048029c4e103) [11771.467588] WARNING: CPU: 13 PID: 2980 at kernel/workqueue.c:2660 check_flush_dependency+0x110/0x130 [11771.488363] mpt3sas_cm1: enclosure logical id(0x5003048029c4e13f), slot(3) [11771.498753] Modules linked in: rpcsec_gss_krb5(E) iscsi_target_mod(E) target_core_mod(E) 8021q(E) garp(E) mrp(E) stp(E) [11771.519513] mpt3sas_cm1: enclosure level(0x0000), connector name( ) [11771.530228] llc(E) bonding(E) tls(E) rfkill(E) raid1(E) dm_raid(E) raid456(E) [11771.553284] mpt3sas_cm1: handle(0xd) sas_address(0x5003048029c4e103) port_type(0x1) [11771.563219] async_raid6_recov(E) async_memcpy(E) async_pq(E) async_xor(E) async_tx(E) xor(E) intel_rapl_msr(E) intel_rapl_common(E) raid6_pq(E) dm_thin_pool(E) dm_persistent_data(E) sb_edac(E) dm_bio_prison(E) dm_bufio(E) x86_pkg_temp_thermal(E) intel_powerclamp(E) coretemp(E) iTCO_wdt(E) kvm_intel(E) intel_pmc_bxt(E) iTCO_vendor_support(E) kvm(E) irqbypass(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) rapl(E) intel_cstate(E) intel_uncore(E) pcspkr(E) nvmet_rdma(E) mei_me(E) ses(E) enclosure(E) joydev(E) lpc_ich(E) mei(E) nvmet(E) i2c_i801(E) i2c_smbus(E) ioatdma(E) ipmi_ssif(E) vfat(E) fat(E) binfmt_misc(E) rpcrdma(E) nfsd(E) auth_rpcgss(E) nfs_acl(E) lockd(E) grace(E) sunrpc(E) xfs(E) libcrc32c(E) ib_umad(E) rdma_ucm(E) ib_ipoib(E) rdma_cm(E) iw_cm(E) ib_cm(E) mlx4_ib(E) ib_uverbs(E) ib_core(E) mlx4_en(E) sd_mod(E) sg(E) isci(E) nvme(E) ahci(E) mpt3sas(E) libsas(E) nvme_core(E) libahci(E) igb(E) raid_class(E) libata(E) t10_pi(E) scsi_transport_sas(E) mlx4_core(E) [11771.585141] crc32c_intel(E) dca(E) i2c_algo_bit(E) wmi(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E) [11771.720632] CPU: 13 PID: 2980 Comm: kworker/u272:1 Tainted: G E 5.16.16-1.el8.elrepo.x86_64 #1 [11771.743305] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.06.0007.082420181029 08/24/2018 [11771.767137] Workqueue: dm-thin do_worker [dm_thin_pool] [11771.778326] RIP: 0010:check_flush_dependency+0x110/0x130 [11771.789446] Code: ff ff 48 8b 50 18 48 8d 8b b0 00 00 00 49 89 e8 48 81 c6 b0 00 00 00 48 c7 c7 a0 7c 72 b2 c6 05 8a 60 a8 01 01 e8 7f 49 8b 00 <0f> 0b e9 0a ff ff ff 80 3d 78 60 a8 01 00 75 95 e9 41 ff ff ff 66 [11771.822923] RSP: 0018:ffffb50d89887bf0 EFLAGS: 00010086 [11771.834076] RAX: 0000000000000000 RBX: ffff8f5b40059000 RCX: 0000000000000027 [11771.845189] RDX: 0000000000000027 RSI: ffff8f724ed60780 RDI: ffff8f724ed60788 [11771.856260] RBP: ffffffffb1714cc0 R08: 0000000000000000 R09: c0000000fffeffff [11771.867236] R10: 0000000000000001 R11: ffffb50d89887a08 R12: ffff8f5c261b1980 [11771.878117] R13: 0000000000000001 R14: ffff8f724ed6ff00 R15: 0000000000000088 [11771.889009] FS: 0000000000000000(0000) GS:ffff8f724ed40000(0000) knlGS:0000000000000000 [11771.899920] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [11771.910435] CR2: 00007f337bc98d50 CR3: 0000000cc116a004 CR4: 00000000000606e0 [11771.920910] Call Trace: [11771.931121] [11771.942342] __flush_work+0x9a/0x1f0 [11771.952126] ? __queue_work+0x141/0x3f0 [11771.961803] ? __flush_cpu_slab+0x60/0x60 [11771.971344] flush_all_cpus_locked+0xf8/0x120 [11771.980743] __kmem_cache_shutdown+0x2b/0x190 [11771.989945] ? dm_bufio_client_destroy+0x161/0x1e0 [dm_bufio] [11771.999041] kmem_cache_destroy+0x45/0x100 [11772.007896] dm_bufio_client_destroy+0x195/0x1e0 [dm_bufio] [11772.016639] dm_block_manager_destroy+0x11/0x20 [dm_persistent_data] [11772.025264] dm_pool_abort_metadata+0x60/0xa0 [dm_thin_pool] [11772.033719] metadata_operation_failed+0x59/0xb0 [dm_thin_pool] [11772.042024] commit+0x48/0xe0 [dm_thin_pool] [11772.050110] ? get_next_thin+0x67/0x90 [dm_thin_pool] [11772.058049] do_worker+0x7dc/0x860 [dm_thin_pool] [11772.065807] process_one_work+0x225/0x400 [11772.073336] worker_thread+0x2d/0x3c0 [11772.080619] ? process_one_work+0x400/0x400 [11772.087758] kthread+0x156/0x180 [11772.094724] ? set_kthread_struct+0x40/0x40 [11772.101598] ret_from_fork+0x1f/0x30 [11772.108271] [11772.114729] ---[ end trace 93a5544354cacbb3 ]--- [11772.122484] device-mapper: thin: 253:6: switching pool to read-only mode [11772.230068] mpt3sas_cm1: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) [11779.978533] scsi 9:0:19:0: Direct-Access ATA Samsung SSD 850 3B6Q PQ: 0 ANSI: 6 [11779.985101] scsi 9:0:19:0: SATA: handle(0x000d), sas_addr(0x5003048029c4e103), phy(3), device_name(0x0000000000000000) [11779.999067] scsi 9:0:19:0: enclosure logical id (0x5003048029c4e13f), slot(3) [11780.005899] scsi 9:0:19:0: enclosure level(0x0000), connector name( ) [11780.012743] scsi 9:0:19:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y) [11780.019627] scsi 9:0:19:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1) [11780.027891] sd 9:0:19:0: Attached scsi generic sg5 type 0 [11780.028015] sd 9:0:19:0: Power-on or device reset occurred [11780.042081] end_device-9:0:12: add: handle(0x000d), sas_addr(0x5003048029c4e103) [11780.042698] sd 9:0:19:0: [sds] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB) [11780.057428] sd 9:0:19:0: [sds] Write Protect is off [11780.064404] sd 9:0:19:0: [sds] Mode Sense: 9b 00 10 08 [11780.064801] sd 9:0:19:0: [sds] Write cache: enabled, read cache: enabled, supports DPO and FUA [11780.097735] sds: sds1 [11780.128703] sd 9:0:19:0: [sds] Attached SCSI disk