Bug 101891
| Summary: | mvsas prep failed, NULL pointer dereference in mvs_slot_task_free+0x5/0x1f0 [mvsas] | ||
|---|---|---|---|
| Product: | SCSI Drivers | Reporter: | Dāvis (davispuh) |
| Component: | Other | Assignee: | scsi_drivers-other |
| Status: | RESOLVED CODE_FIX | ||
| Severity: | normal | CC: | davispuh, satyasrinivasp, turbo |
| Priority: | P1 | ||
| Hardware: | x86-64 | ||
| OS: | Linux | ||
| Kernel Version: | 4.1.2 | Subsystem: | |
| Regression: | No | Bisected commit-id: | |
|
Description
Dāvis
2015-07-23 21:34:36 UTC
Some more call traces ------------[ cut here ]------------ kernel: WARNING: CPU: 4 PID: 6442 at fs/sysfs/group.c:224 sysfs_remove_group+0xa1/0xb0() kernel: sysfs group ffffffff8189de80 not found for kobject 'end_device-8:0' kernel: Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_tcpudp ip6t_rpfilter kernel: aesni_intel rc_core snd_hda_codec_realtek aes_x86_64 lrw gf128mul videobuf2_dma_sg glue_helper kernel: CPU: 4 PID: 6442 Comm: kworker/u16:12 Tainted: P R D O 4.1.2-2-ARCH #1 kernel: Hardware name: Gigabyte Technology Co., Ltd. GA-990FXA-UD3/GA-990FXA-UD3, BIOS FFe 11/08/2013 kernel: Workqueue: scsi_wq_8 sas_destruct_devices [libsas] kernel: 0000000000000000 00000000fff093ac ffff88008071bbf8 ffffffff81585c8e kernel: 0000000000000000 ffff88008071bc50 ffff88008071bc38 ffffffff81078c9a kernel: ffff88008071bc68 0000000000000000 ffffffff8189de80 ffff880222550810 kernel: Call Trace: kernel: [<ffffffff81585c8e>] dump_stack+0x4c/0x6e kernel: [<ffffffff81078c9a>] warn_slowpath_common+0x8a/0xc0 kernel: [<ffffffff81078d25>] warn_slowpath_fmt+0x55/0x70 kernel: [<ffffffff81258b9c>] ? kernfs_find_and_get_ns+0x4c/0x60 kernel: [<ffffffff8125c351>] sysfs_remove_group+0xa1/0xb0 kernel: [<ffffffff813fc4c7>] dpm_sysfs_remove+0x57/0x60 kernel: [<ffffffff813ef828>] device_del+0x58/0x270 kernel: [<ffffffff813ef2f7>] ? put_device+0x17/0x20 kernel: [<ffffffff813efa62>] device_unregister+0x22/0x80 kernel: [<ffffffff812b5a00>] bsg_unregister_queue+0x60/0xc0 kernel: [<ffffffffa003a7dc>] sas_rphy_remove+0x4c/0x80 [scsi_transport_sas] kernel: [<ffffffffa003a826>] sas_rphy_delete+0x16/0x30 [scsi_transport_sas] kernel: [<ffffffffa0114a95>] sas_destruct_devices+0x65/0x90 [libsas] kernel: [<ffffffff8109193b>] process_one_work+0x14b/0x470 kernel: [<ffffffff81091ca8>] worker_thread+0x48/0x4c0 kernel: [<ffffffff81091c60>] ? process_one_work+0x470/0x470 kernel: [<ffffffff810977f8>] kthread+0xd8/0xf0 kernel: [<ffffffff81097720>] ? kthread_worker_fn+0x170/0x170 kernel: [<ffffffff8158b962>] ret_from_fork+0x42/0x70 kernel: [<ffffffff81097720>] ? kthread_worker_fn+0x170/0x170 kernel: ---[ end trace 18b7a6f928680375 ]--- kernel: ------------[ cut here ]------------ kernel: WARNING: CPU: 4 PID: 6442 at fs/sysfs/group.c:224 sysfs_remove_group+0xa1/0xb0() kernel: sysfs group ffffffff8189de80 not found for kobject 'end_device-8:1' kernel: Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_tcpudp ip6t_rpfilter ip kernel: aesni_intel rc_core snd_hda_codec_realtek aes_x86_64 lrw gf128mul videobuf2_dma_sg glue_helper a kernel: CPU: 4 PID: 6442 Comm: kworker/u16:12 Tainted: P R D W O 4.1.2-2-ARCH #1 kernel: Hardware name: Gigabyte Technology Co., Ltd. GA-990FXA-UD3/GA-990FXA-UD3, BIOS FFe 11/08/2013 kernel: Workqueue: scsi_wq_8 sas_destruct_devices [libsas] kernel: 0000000000000000 00000000fff093ac ffff88008071bc38 ffffffff81585c8e kernel: 0000000000000000 ffff88008071bc90 ffff88008071bc78 ffffffff81078c9a kernel: ffff88008071bc78 0000000000000000 ffffffff8189de80 ffff88022254c810 kernel: Call Trace: kernel: [<ffffffff81585c8e>] dump_stack+0x4c/0x6e kernel: [<ffffffff81078c9a>] warn_slowpath_common+0x8a/0xc0 kernel: [<ffffffff81078d25>] warn_slowpath_fmt+0x55/0x70 kernel: [<ffffffff81258b9c>] ? kernfs_find_and_get_ns+0x4c/0x60 kernel: [<ffffffff8125c351>] sysfs_remove_group+0xa1/0xb0 kernel: [<ffffffff813fc4c7>] dpm_sysfs_remove+0x57/0x60 kernel: [<ffffffff813ef828>] device_del+0x58/0x270 kernel: [<ffffffffa003a7ec>] sas_rphy_remove+0x5c/0x80 [scsi_transport_sas] kernel: [<ffffffffa003a826>] sas_rphy_delete+0x16/0x30 [scsi_transport_sas] kernel: [<ffffffffa0114a95>] sas_destruct_devices+0x65/0x90 [libsas] kernel: [<ffffffff8109193b>] process_one_work+0x14b/0x470 kernel: [<ffffffff81091ca8>] worker_thread+0x48/0x4c0 kernel: [<ffffffff81091c60>] ? process_one_work+0x470/0x470 kernel: [<ffffffff810977f8>] kthread+0xd8/0xf0 kernel: [<ffffffff81097720>] ? kthread_worker_fn+0x170/0x170 kernel: [<ffffffff8158b962>] ret_from_fork+0x42/0x70 kernel: [<ffffffff81097720>] ? kthread_worker_fn+0x170/0x170 kernel: ---[ end trace 18b7a6f92868037c ]--- kernel: ------------[ cut here ]------------ kernel: WARNING: CPU: 4 PID: 6442 at fs/sysfs/group.c:224 sysfs_remove_group+0xa1/0xb0() kernel: sysfs group ffffffff8189de80 not found for kobject 'end_device-8:2' kernel: Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_tcpudp ip6t_rpfilter ip kernel: aesni_intel rc_core snd_hda_codec_realtek aes_x86_64 lrw gf128mul videobuf2_dma_sg glue_helper a kernel: CPU: 4 PID: 6442 Comm: kworker/u16:12 Tainted: P R D W O 4.1.2-2-ARCH #1 kernel: Hardware name: Gigabyte Technology Co., Ltd. GA-990FXA-UD3/GA-990FXA-UD3, BIOS FFe 11/08/2013 kernel: Workqueue: scsi_wq_8 sas_destruct_devices [libsas] kernel: 0000000000000000 00000000fff093ac ffff88008071bb88 ffffffff81585c8e kernel: 0000000000000000 ffff88008071bbe0 ffff88008071bbc8 ffffffff81078c9a kernel: ffff88008071bbc8 0000000000000000 ffffffff8189de80 ffff88022254d838 kernel: Call Trace: kernel: [<ffffffff81585c8e>] dump_stack+0x4c/0x6e kernel: [<ffffffff81078c9a>] warn_slowpath_common+0x8a/0xc0 kernel: [<ffffffff81078d25>] warn_slowpath_fmt+0x55/0x70 kernel: [<ffffffff81258b9c>] ? kernfs_find_and_get_ns+0x4c/0x60 kernel: [<ffffffff8125c351>] sysfs_remove_group+0xa1/0xb0 kernel: [<ffffffff813fc4c7>] dpm_sysfs_remove+0x57/0x60 kernel: [<ffffffff813ef828>] device_del+0x58/0x270 kernel: [<ffffffff813eed79>] ? device_remove_file+0x19/0x20 kernel: [<ffffffff813f920e>] attribute_container_class_device_del+0x1e/0x30 kernel: [<ffffffff813f9422>] transport_remove_classdev+0x52/0x60 kernel: [<ffffffff813f93d0>] ? transport_add_class_device+0x40/0x40 kernel: [<ffffffff813f8d7c>] attribute_container_device_trigger+0xdc/0xf0 kernel: [<ffffffff813f9385>] transport_remove_device+0x15/0x20 kernel: [<ffffffffa003a7e4>] sas_rphy_remove+0x54/0x80 [scsi_transport_sas] kernel: [<ffffffffa003a826>] sas_rphy_delete+0x16/0x30 [scsi_transport_sas] kernel: [<ffffffffa0114a95>] sas_destruct_devices+0x65/0x90 [libsas] kernel: [<ffffffff8109193b>] process_one_work+0x14b/0x470 kernel: [<ffffffff81091ca8>] worker_thread+0x48/0x4c0 kernel: [<ffffffff81091c60>] ? process_one_work+0x470/0x470 kernel: [<ffffffff810977f8>] kthread+0xd8/0xf0 kernel: [<ffffffff81097720>] ? kthread_worker_fn+0x170/0x170 kernel: [<ffffffff8158b962>] ret_from_fork+0x42/0x70 kernel: [<ffffffff81097720>] ? kthread_worker_fn+0x170/0x170 kernel: ---[ end trace 18b7a6f92868037e ]--- kernel: ------------[ cut here ]------------ kernel: WARNING: CPU: 4 PID: 6442 at fs/sysfs/group.c:224 sysfs_remove_group+0xa1/0xb0() kernel: sysfs group ffffffff8189de80 not found for kobject 'end_device-8:2' kernel: Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_tcpudp ip6t_rpfilter ip kernel: aesni_intel rc_core snd_hda_codec_realtek aes_x86_64 lrw gf128mul videobuf2_dma_sg glue_helper a kernel: CPU: 4 PID: 6442 Comm: kworker/u16:12 Tainted: P R D W O 4.1.2-2-ARCH #1 kernel: Hardware name: Gigabyte Technology Co., Ltd. GA-990FXA-UD3/GA-990FXA-UD3, BIOS FFe 11/08/2013 kernel: Workqueue: scsi_wq_8 sas_destruct_devices [libsas] kernel: 0000000000000000 00000000fff093ac ffff88008071bb88 ffffffff81585c8e kernel: 0000000000000000 ffff88008071bbe0 ffff88008071bbc8 ffffffff81078c9a kernel: ffff88008071bbc8 0000000000000000 ffffffff8189de80 ffff88022254dc38 kernel: Call Trace: kernel: [<ffffffff81585c8e>] dump_stack+0x4c/0x6e kernel: [<ffffffff81078c9a>] warn_slowpath_common+0x8a/0xc0 kernel: [<ffffffff81078d25>] warn_slowpath_fmt+0x55/0x70 kernel: [<ffffffff81258b9c>] ? kernfs_find_and_get_ns+0x4c/0x60 kernel: [<ffffffff8125c351>] sysfs_remove_group+0xa1/0xb0 kernel: [<ffffffff813fc4c7>] dpm_sysfs_remove+0x57/0x60 kernel: [<ffffffff813ef828>] device_del+0x58/0x270 kernel: [<ffffffff813eed79>] ? device_remove_file+0x19/0x20 kernel: [<ffffffff813f920e>] attribute_container_class_device_del+0x1e/0x30 kernel: [<ffffffff813f9422>] transport_remove_classdev+0x52/0x60 kernel: [<ffffffff813f93d0>] ? transport_add_class_device+0x40/0x40 kernel: [<ffffffff813f8d7c>] attribute_container_device_trigger+0xdc/0xf0 kernel: [<ffffffff813f9385>] transport_remove_device+0x15/0x20 kernel: [<ffffffffa003a7e4>] sas_rphy_remove+0x54/0x80 [scsi_transport_sas] kernel: [<ffffffffa003a826>] sas_rphy_delete+0x16/0x30 [scsi_transport_sas] kernel: [<ffffffffa0114a95>] sas_destruct_devices+0x65/0x90 [libsas] kernel: [<ffffffff8109193b>] process_one_work+0x14b/0x470 kernel: [<ffffffff81091ca8>] worker_thread+0x48/0x4c0 kernel: [<ffffffff81091c60>] ? process_one_work+0x470/0x470 kernel: [<ffffffff810977f8>] kthread+0xd8/0xf0 kernel: [<ffffffff81097720>] ? kthread_worker_fn+0x170/0x170 kernel: [<ffffffff8158b962>] ret_from_fork+0x42/0x70 kernel: [<ffffffff81097720>] ? kthread_worker_fn+0x170/0x170 kernel: ---[ end trace 18b7a6f92868037f ]--- kernel: ------------[ cut here ]------------ kernel: WARNING: CPU: 0 PID: 5845 at fs/sysfs/group.c:224 sysfs_remove_group+0xa1/0xb0() kernel: sysfs group ffffffff8189de80 not found for kobject '8:0:0:0' kernel: Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_tcpudp ip6t_rpfilter ip kernel: aesni_intel rc_core snd_hda_codec_realtek aes_x86_64 lrw gf128mul videobuf2_dma_sg glue_helper a kernel: CPU: 0 PID: 5845 Comm: rmmod Tainted: P R D W O 4.1.2-2-ARCH #1 kernel: Hardware name: Gigabyte Technology Co., Ltd. GA-990FXA-UD3/GA-990FXA-UD3, BIOS FFe 11/08/2013 kernel: 0000000000000000 0000000015da1c4c ffff880100a9bbb8 ffffffff81585c8e kernel: 0000000000000000 ffff880100a9bc10 ffff880100a9bbf8 ffffffff81078c9a kernel: ffff880100a9bbf8 0000000000000000 ffffffff8189de80 ffff8802224b5810 kernel: Call Trace: kernel: [<ffffffff81585c8e>] dump_stack+0x4c/0x6e kernel: [<ffffffff81078c9a>] warn_slowpath_common+0x8a/0xc0 kernel: [<ffffffff81078d25>] warn_slowpath_fmt+0x55/0x70 kernel: [<ffffffff81258b9c>] ? kernfs_find_and_get_ns+0x4c/0x60 kernel: [<ffffffff8125c351>] sysfs_remove_group+0xa1/0xb0 kernel: [<ffffffff813fc4c7>] dpm_sysfs_remove+0x57/0x60 kernel: [<ffffffff813ef828>] device_del+0x58/0x270 kernel: [<ffffffff813efa62>] device_unregister+0x22/0x80 kernel: [<ffffffff812b5a00>] bsg_unregister_queue+0x60/0xc0 kernel: [<ffffffffa008c501>] __scsi_remove_device+0xb1/0xe0 [scsi_mod] kernel: [<ffffffffa008a964>] scsi_forget_host+0x64/0x70 [scsi_mod] kernel: [<ffffffffa007f4e9>] scsi_remove_host+0x79/0x160 [scsi_mod] kernel: [<ffffffffa020c5fe>] mvs_pci_remove+0x4e/0xe0 [mvsas] kernel: [<ffffffff81311e2f>] pci_device_remove+0x3f/0xc0 kernel: [<ffffffff813f3d87>] __device_release_driver+0x87/0x120 kernel: [<ffffffff813f4898>] driver_detach+0xc8/0xd0 kernel: [<ffffffff813f39e9>] bus_remove_driver+0x59/0xe0 kernel: [<ffffffff813f5130>] driver_unregister+0x30/0x70 kernel: [<ffffffff8131179d>] pci_unregister_driver+0x2d/0xa0 kernel: [<ffffffffa0215969>] mvs_exit+0x10/0x6a7 [mvsas] kernel: [<ffffffff811006df>] SyS_delete_module+0x1cf/0x280 kernel: [<ffffffff8158b56e>] system_call_fastpath+0x12/0x71 kernel: ---[ end trace 18b7a6f928680389 ]--- kernel: ------------[ cut here ]------------ kernel: WARNING: CPU: 0 PID: 5845 at fs/sysfs/group.c:224 sysfs_remove_group+0xa1/0xb0() kernel: sysfs group ffffffff8189de80 not found for kobject '8:0:0:0' kernel: Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_tcpudp ip6t_rpfilter ip kernel: aesni_intel rc_core snd_hda_codec_realtek aes_x86_64 lrw gf128mul videobuf2_dma_sg glue_helper a kernel: CPU: 0 PID: 5845 Comm: rmmod Tainted: P R D W O 4.1.2-2-ARCH #1 kernel: Hardware name: Gigabyte Technology Co., Ltd. GA-990FXA-UD3/GA-990FXA-UD3, BIOS FFe 11/08/2013 kernel: 0000000000000000 0000000015da1c4c ffff880100a9bb08 ffffffff81585c8e kernel: 0000000000000000 ffff880100a9bb60 ffff880100a9bb48 ffffffff81078c9a kernel: ffff880100a9bb48 0000000000000000 ffffffff8189de80 ffff8802224b4820 kernel: Call Trace: kernel: [<ffffffff81585c8e>] dump_stack+0x4c/0x6e kernel: [<ffffffff81078c9a>] warn_slowpath_common+0x8a/0xc0 kernel: [<ffffffff81078d25>] warn_slowpath_fmt+0x55/0x70 kernel: [<ffffffff81258b9c>] ? kernfs_find_and_get_ns+0x4c/0x60 kernel: [<ffffffff8125c351>] sysfs_remove_group+0xa1/0xb0 kernel: [<ffffffff813fc4c7>] dpm_sysfs_remove+0x57/0x60 kernel: [<ffffffff813ef828>] device_del+0x58/0x270 kernel: [<ffffffffa0008ba5>] sd_remove+0x55/0xc0 [sd_mod] kernel: [<ffffffff813f3d87>] __device_release_driver+0x87/0x120 kernel: [<ffffffff813f3e43>] device_release_driver+0x23/0x30 kernel: [<ffffffff813f36c8>] bus_remove_device+0x108/0x180 kernel: [<ffffffff813ef911>] device_del+0x141/0x270 kernel: [<ffffffffa008c51d>] __scsi_remove_device+0xcd/0xe0 [scsi_mod] kernel: [<ffffffffa008a964>] scsi_forget_host+0x64/0x70 [scsi_mod] kernel: [<ffffffffa007f4e9>] scsi_remove_host+0x79/0x160 [scsi_mod] kernel: [<ffffffffa020c5fe>] mvs_pci_remove+0x4e/0xe0 [mvsas] kernel: [<ffffffff81311e2f>] pci_device_remove+0x3f/0xc0 kernel: [<ffffffff813f3d87>] __device_release_driver+0x87/0x120 kernel: [<ffffffff813f4898>] driver_detach+0xc8/0xd0 kernel: [<ffffffff813f39e9>] bus_remove_driver+0x59/0xe0 kernel: [<ffffffff813f5130>] driver_unregister+0x30/0x70 kernel: [<ffffffff8131179d>] pci_unregister_driver+0x2d/0xa0 kernel: [<ffffffffa0215969>] mvs_exit+0x10/0x6a7 [mvsas] kernel: [<ffffffff811006df>] SyS_delete_module+0x1cf/0x280 kernel: [<ffffffff8158b56e>] system_call_fastpath+0x12/0x71 kernel: ---[ end trace 18b7a6f92868038c ]--- kernel: ------------[ cut here ]------------ kernel: WARNING: CPU: 0 PID: 5845 at fs/sysfs/group.c:224 sysfs_remove_group+0xa1/0xb0() kernel: sysfs group ffffffff8189de80 not found for kobject 'sdj1' kernel: Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_tcpudp ip6t_rpfilter ip kernel: aesni_intel rc_core snd_hda_codec_realtek aes_x86_64 lrw gf128mul videobuf2_dma_sg glue_helper a kernel: CPU: 0 PID: 5845 Comm: rmmod Tainted: P R D W O 4.1.2-2-ARCH #1 kernel: Hardware name: Gigabyte Technology Co., Ltd. GA-990FXA-UD3/GA-990FXA-UD3, BIOS FFe 11/08/2013 kernel: 0000000000000000 0000000015da1c4c ffff880100a9ba98 ffffffff81585c8e kernel: 0000000000000000 ffff880100a9baf0 ffff880100a9bad8 ffffffff81078c9a kernel: ffff880100a9bad8 0000000000000000 ffffffff8189de80 ffff880222555038 kernel: Call Trace: kernel: [<ffffffff81585c8e>] dump_stack+0x4c/0x6e kernel: [<ffffffff81078c9a>] warn_slowpath_common+0x8a/0xc0 kernel: [<ffffffff81078d25>] warn_slowpath_fmt+0x55/0x70 kernel: [<ffffffff81258b9c>] ? kernfs_find_and_get_ns+0x4c/0x60 kernel: [<ffffffff8125c351>] sysfs_remove_group+0xa1/0xb0 kernel: [<ffffffff813fc4c7>] dpm_sysfs_remove+0x57/0x60 kernel: [<ffffffff813ef828>] device_del+0x58/0x270 kernel: [<ffffffff812aebec>] delete_partition+0x4c/0x80 kernel: [<ffffffff812ad140>] del_gendisk+0xd0/0x240 kernel: [<ffffffffa0008bb1>] sd_remove+0x61/0xc0 [sd_mod] kernel: [<ffffffff813f3d87>] __device_release_driver+0x87/0x120 kernel: [<ffffffff813f3e43>] device_release_driver+0x23/0x30 kernel: [<ffffffff813f36c8>] bus_remove_device+0x108/0x180 kernel: [<ffffffff813ef911>] device_del+0x141/0x270 kernel: [<ffffffffa008c51d>] __scsi_remove_device+0xcd/0xe0 [scsi_mod] kernel: [<ffffffffa008a964>] scsi_forget_host+0x64/0x70 [scsi_mod] kernel: [<ffffffffa007f4e9>] scsi_remove_host+0x79/0x160 [scsi_mod] kernel: [<ffffffffa020c5fe>] mvs_pci_remove+0x4e/0xe0 [mvsas] kernel: [<ffffffff81311e2f>] pci_device_remove+0x3f/0xc0 kernel: [<ffffffff813f3d87>] __device_release_driver+0x87/0x120 kernel: [<ffffffff813f4898>] driver_detach+0xc8/0xd0 kernel: [<ffffffff813f39e9>] bus_remove_driver+0x59/0xe0 kernel: [<ffffffff813f5130>] driver_unregister+0x30/0x70 kernel: [<ffffffff8131179d>] pci_unregister_driver+0x2d/0xa0 kernel: [<ffffffffa0215969>] mvs_exit+0x10/0x6a7 [mvsas] kernel: [<ffffffff811006df>] SyS_delete_module+0x1cf/0x280 kernel: [<ffffffff8158b56e>] system_call_fastpath+0x12/0x71 kernel: ---[ end trace 18b7a6f92868038d ]--- kernel: ------------[ cut here ]------------ kernel: WARNING: CPU: 1 PID: 5845 at fs/sysfs/group.c:224 sysfs_remove_group+0xa1/0xb0() kernel: sysfs group ffffffff81859160 not found for kobject 'sdj' kernel: Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_tcpudp ip6t_rpfilter ip kernel: aesni_intel rc_core snd_hda_codec_realtek aes_x86_64 lrw gf128mul videobuf2_dma_sg glue_helper a kernel: CPU: 1 PID: 5845 Comm: rmmod Tainted: P R D W O 4.1.2-2-ARCH #1 kernel: Hardware name: Gigabyte Technology Co., Ltd. GA-990FXA-UD3/GA-990FXA-UD3, BIOS FFe 11/08/2013 kernel: 0000000000000000 0000000015da1c4c ffff880100a9baf8 ffffffff81585c8e kernel: 0000000000000000 ffff880100a9bb50 ffff880100a9bb38 ffffffff81078c9a kernel: ffff880226087100 0000000000000000 ffffffff81859160 ffff8802224b4c80 kernel: Call Trace: kernel: [<ffffffff81585c8e>] dump_stack+0x4c/0x6e kernel: [<ffffffff81078c9a>] warn_slowpath_common+0x8a/0xc0 kernel: [<ffffffff81078d25>] warn_slowpath_fmt+0x55/0x70 kernel: [<ffffffff81258b9c>] ? kernfs_find_and_get_ns+0x4c/0x60 kernel: [<ffffffff8125c351>] sysfs_remove_group+0xa1/0xb0 kernel: [<ffffffff8113d7f4>] blk_trace_remove_sysfs+0x14/0x20 kernel: [<ffffffff8129eed5>] blk_unregister_queue+0x65/0xa0 kernel: [<ffffffff812ad19e>] del_gendisk+0x12e/0x240 kernel: [<ffffffffa0008bb1>] sd_remove+0x61/0xc0 [sd_mod] kernel: [<ffffffff813f3d87>] __device_release_driver+0x87/0x120 kernel: [<ffffffff813f3e43>] device_release_driver+0x23/0x30 kernel: [<ffffffff813f36c8>] bus_remove_device+0x108/0x180 kernel: [<ffffffff813ef911>] device_del+0x141/0x270 kernel: [<ffffffffa008c51d>] __scsi_remove_device+0xcd/0xe0 [scsi_mod] kernel: [<ffffffffa008a964>] scsi_forget_host+0x64/0x70 [scsi_mod] kernel: [<ffffffffa007f4e9>] scsi_remove_host+0x79/0x160 [scsi_mod] kernel: [<ffffffffa020c5fe>] mvs_pci_remove+0x4e/0xe0 [mvsas] kernel: [<ffffffff81311e2f>] pci_device_remove+0x3f/0xc0 kernel: [<ffffffff813f3d87>] __device_release_driver+0x87/0x120 kernel: [<ffffffff813f4898>] driver_detach+0xc8/0xd0 kernel: [<ffffffff813f39e9>] bus_remove_driver+0x59/0xe0 kernel: [<ffffffff813f5130>] driver_unregister+0x30/0x70 kernel: [<ffffffff8131179d>] pci_unregister_driver+0x2d/0xa0 kernel: [<ffffffffa0215969>] mvs_exit+0x10/0x6a7 [mvsas] kernel: [<ffffffff811006df>] SyS_delete_module+0x1cf/0x280 kernel: [<ffffffff8158b56e>] system_call_fastpath+0x12/0x71 kernel: ---[ end trace 18b7a6f92868038f ]--- kernel: sd 8:0:0:0: [sdj] Synchronizing SCSI cache kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 kernel: IP: [<ffffffffa020e1ea>] mvs_task_exec.isra.3+0x2a/0xe90 [mvsas] kernel: PGD 1663e9067 PUD 153765067 PMD 0 kernel: Oops: 0000 [#2] PREEMPT SMP kernel: Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_tcpudp ip6t_rpfilter ip kernel: aesni_intel rc_core snd_hda_codec_realtek aes_x86_64 lrw gf128mul videobuf2_dma_sg glue_helper a kernel: CPU: 1 PID: 5845 Comm: rmmod Tainted: P R D W O 4.1.2-2-ARCH #1 kernel: Hardware name: Gigabyte Technology Co., Ltd. GA-990FXA-UD3/GA-990FXA-UD3, BIOS FFe 11/08/2013 kernel: task: ffff880118008a30 ti: ffff880100a98000 task.ti: ffff880100a98000 kernel: RIP: 0010:[<ffffffffa020e1ea>] [<ffffffffa020e1ea>] mvs_task_exec.isra.3+0x2a/0xe90 [mvsas] kernel: RSP: 0018:ffff880100a9b758 EFLAGS: 00010092 kernel: RAX: 0000000000000000 RBX: ffff8802253bc4f8 RCX: ffff880100b5c718 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880100b5c700 kernel: RBP: ffff880100a9b7f8 R08: ffffffffa011201d R09: ffff880100b5c700 kernel: R10: 000000000003002f R11: 000000000000002f R12: 0000000000000000 kernel: R13: ffff8802253bc000 R14: ffff880100b5c700 R15: ffff880100b5c700 kernel: FS: 00007fac1bb7b700(0000) GS:ffff88022ec40000(0000) knlGS:0000000000000000 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b kernel: CR2: 0000000000000018 CR3: 000000014a767000 CR4: 00000000000407e0 kernel: Stack: kernel: 000000000000000a 0000000015da1c4c ffff880100a9b798 ffffffff81a52e87 kernel: ffffffffffffffff ffffffff81713cb4 ffff880100a9b7b8 ffffffff812581cc kernel: ffff8802230b2348 0000000000000086 ffff880100000000 0000000000000000 kernel: Call Trace: kernel: [<ffffffff812581cc>] ? kernfs_path_locked+0x3c/0x90 kernel: [<ffffffffa011201d>] ? sas_alloc_task+0x1d/0x40 [libsas] kernel: [<ffffffffa020f832>] mvs_queue_command+0x12/0x20 [mvsas] kernel: [<ffffffffa011aec1>] sas_ata_qc_issue+0x1b1/0x260 [libsas] kernel: [<ffffffffa0263950>] ata_qc_issue+0x170/0x3e0 [libata] kernel: [<ffffffff810e4c83>] ? internal_add_timer+0x63/0x80 kernel: [<ffffffffa0268680>] ? ata_scsi_em_message_show+0x40/0x40 [libata] kernel: [<ffffffffa026a3b6>] ata_scsi_translate+0xb6/0x1d0 [libata] kernel: [<ffffffffa026ccaf>] ata_sas_queuecmd+0x9f/0x250 [libata] kernel: [<ffffffffa01199cf>] sas_queuecommand+0x18f/0x1f0 [libsas] kernel: [<ffffffffa00842d3>] scsi_dispatch_cmd+0xc3/0x1c0 [scsi_mod] kernel: [<ffffffffa008748d>] scsi_request_fn+0x2dd/0x590 [scsi_mod] kernel: [<ffffffff812992e7>] __blk_run_queue+0x37/0x50 kernel: [<ffffffff812a1565>] blk_execute_rq_nowait+0xb5/0x180 kernel: [<ffffffff8129a91f>] ? get_request+0x2ef/0x7f0 kernel: [<ffffffff812a16bb>] blk_execute_rq+0x8b/0x150 kernel: [<ffffffff8129aea6>] ? blk_get_request+0x86/0xf0 kernel: [<ffffffffa0084531>] scsi_execute+0x141/0x1f0 [scsi_mod] kernel: [<ffffffffa008468e>] scsi_execute_req_flags+0x8e/0x100 [scsi_mod] kernel: [<ffffffffa00085f7>] sd_sync_cache+0xa7/0x1a0 [sd_mod] kernel: [<ffffffffa0008aa2>] sd_shutdown+0x72/0x120 [sd_mod] kernel: [<ffffffffa0008bb9>] sd_remove+0x69/0xc0 [sd_mod] kernel: [<ffffffff813f3d87>] __device_release_driver+0x87/0x120 kernel: [<ffffffff813f3e43>] device_release_driver+0x23/0x30 kernel: [<ffffffff813f36c8>] bus_remove_device+0x108/0x180 kernel: [<ffffffff813ef911>] device_del+0x141/0x270 kernel: [<ffffffffa008c51d>] __scsi_remove_device+0xcd/0xe0 [scsi_mod] kernel: [<ffffffffa008a964>] scsi_forget_host+0x64/0x70 [scsi_mod] kernel: [<ffffffffa007f4e9>] scsi_remove_host+0x79/0x160 [scsi_mod] kernel: [<ffffffffa020c5fe>] mvs_pci_remove+0x4e/0xe0 [mvsas] kernel: [<ffffffff81311e2f>] pci_device_remove+0x3f/0xc0 kernel: [<ffffffff813f3d87>] __device_release_driver+0x87/0x120 kernel: [<ffffffff813f4898>] driver_detach+0xc8/0xd0 kernel: [<ffffffff813f39e9>] bus_remove_driver+0x59/0xe0 kernel: [<ffffffff813f5130>] driver_unregister+0x30/0x70 kernel: [<ffffffff8131179d>] pci_unregister_driver+0x2d/0xa0 kernel: [<ffffffffa0215969>] mvs_exit+0x10/0x6a7 [mvsas] kernel: [<ffffffff811006df>] SyS_delete_module+0x1cf/0x280 kernel: [<ffffffff8158b56e>] system_call_fastpath+0x12/0x71 kernel: Code: 00 66 66 66 66 90 55 48 89 e5 41 57 41 56 41 55 41 54 49 89 ff 53 48 83 ec 78 48 8b 07 48 8 kernel: RIP [<ffffffffa020e1ea>] mvs_task_exec.isra.3+0x2a/0xe90 [mvsas] kernel: RSP <ffff880100a9b758> kernel: CR2: 0000000000000018 kernel: ---[ end trace 18b7a6f928680391 ]--- kernel: sas: Enter sas_scsi_recover_host busy: 1 failed: 1 kernel: sas: trying to find task 0xffff880100b5c700 kernel: sas: sas_scsi_find_task: aborting task 0xffff880100b5c700 kernel: drivers/scsi/mvsas/mv_sas.c 1487:Device has removed kernel: sas: sas_scsi_find_task: querying task 0xffff880100b5c700 kernel: drivers/scsi/mvsas/mv_sas.c 1470:mvs_query_task:rc= 5 kernel: sas: sas_scsi_find_task: task 0xffff880100b5c700 failed to abort kernel: sas: task 0xffff880100b5c700 is not at LU: I_T recover kernel: sas: I_T nexus reset for dev 0000000000000000 kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000085 kernel: IP: [<ffffffffa02111bd>] mvs_I_T_nexus_reset+0x1d/0xe0 [mvsas] kernel: PGD 0 kernel: Oops: 0000 [#3] PREEMPT SMP kernel: Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast xt_tcpudp ip6t_rpfilter ip kernel: aesni_intel rc_core snd_hda_codec_realtek aes_x86_64 lrw gf128mul videobuf2_dma_sg glue_helper a kernel: CPU: 0 PID: 230 Comm: scsi_eh_8 Tainted: P R D W O 4.1.2-2-ARCH #1 kernel: Hardware name: Gigabyte Technology Co., Ltd. GA-990FXA-UD3/GA-990FXA-UD3, BIOS FFe 11/08/2013 kernel: task: ffff88007f84bd20 ti: ffff88007fc14000 task.ti: ffff88007fc14000 kernel: RIP: 0010:[<ffffffffa02111bd>] [<ffffffffa02111bd>] mvs_I_T_nexus_reset+0x1d/0xe0 [mvsas] kernel: RSP: 0018:ffff88007fc17d28 EFLAGS: 00010296 kernel: RAX: ffffffffa02111a0 RBX: ffff880222c91e00 RCX: 000000000000002d kernel: RDX: 0000000000000000 RSI: 0000000000000282 RDI: ffff880222c91e00 kernel: RBP: ffff88007fc17d58 R08: 000000000000000a R09: 0000000000000a42 kernel: R10: 000000000003f190 R11: 0000000000000a42 R12: 0000000000000000 kernel: R13: ffff880100b5c708 R14: ffff8802230ef000 R15: ffff880100b5c700 kernel: FS: 00007f833dffa700(0000) GS:ffff88022ec00000(0000) knlGS:0000000000000000 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b kernel: CR2: 0000000000000085 CR3: 000000021c800000 CR4: 00000000000407f0 kernel: Stack: kernel: ffff88007fc17d78 ffff880222c91e00 ffff8802230ef000 ffff880100b5c708 kernel: ffff8802230ef000 ffff880100b5c700 ffff88007fc17e28 ffffffffa0119cd8 kernel: ffff88007fc17d90 ffff88007f841088 ffff880222c92a00 ffff880222c92a00 kernel: Call Trace: kernel: [<ffffffffa0119cd8>] sas_scsi_recover_host+0x2a8/0xc20 [libsas] kernel: [<ffffffffa0083afc>] scsi_error_handler+0xfc/0x580 [scsi_mod] kernel: [<ffffffff81587212>] ? __schedule+0x362/0xa30 kernel: [<ffffffffa0083a00>] ? scsi_eh_get_sense+0x190/0x190 [scsi_mod] kernel: [<ffffffff810977f8>] kthread+0xd8/0xf0 kernel: [<ffffffff81097720>] ? kthread_worker_fn+0x170/0x170 kernel: [<ffffffff8158b962>] ret_from_fork+0x42/0x70 kernel: [<ffffffff81097720>] ? kthread_worker_fn+0x170/0x170 kernel: Code: e8 c9 78 e6 e0 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 57 41 56 41 55 41 5 kernel: RIP [<ffffffffa02111bd>] mvs_I_T_nexus_reset+0x1d/0xe0 [mvsas] kernel: RSP <ffff88007fc17d28> kernel: CR2: 0000000000000085 kernel: ---[ end trace 18b7a6f928680392 ]--- (In reply to Dāvis from comment #0) > Got this call trace, it caused any attempts to access those disks hang > (couldn't even kill those processes, eg. ls). > Using HighPoint RocketRAID 2760A controller. > > kernel: mvsas 0000:07:00.0: mvsas prep failed[0]! > kernel: sas: Enter sas_scsi_recover_host busy: 1 failed: 1 > kernel: sas: trying to find task 0xffff880213ac6a00 > kernel: sas: sas_scsi_find_task: aborting task 0xffff880213ac6a00 > kernel: BUG: unable to handle kernel NULL pointer dereference at > 0000000000000010 > kernel: IP: [<ffffffffa020dfa5>] mvs_slot_task_free+0x5/0x1f0 [mvsas] > kernel: PGD 1ee973067 PUD 1ee974067 PMD 0 > kernel: Oops: 0000 [#1] PREEMPT SMP > kernel: Modules linked in: fuse nf_conntrack_netbios_ns > nf_conntrack_broadcast xt_tcpudp ip6t_rpfilter ip > kernel: aesni_intel rc_core snd_hda_codec_realtek aes_x86_64 lrw gf128mul > videobuf2_dma_sg glue_helper a > kernel: CPU: 3 PID: 227 Comm: scsi_eh_7 Tainted: P O > 4.1.2-2-ARCH #1 > kernel: Hardware name: Gigabyte Technology Co., Ltd. > GA-990FXA-UD3/GA-990FXA-UD3, BIOS FFe 11/08/2013 > kernel: task: ffff88007f849e90 ti: ffff880223184000 task.ti: ffff880223184000 > kernel: RIP: 0010:[<ffffffffa020dfa5>] [<ffffffffa020dfa5>] > mvs_slot_task_free+0x5/0x1f0 [mvsas] > kernel: RSP: 0018:ffff880223187d00 EFLAGS: 00010a13 > kernel: RAX: 2e8ba2e8ba2e8ba3 RBX: ffff880213ac6a00 RCX: a2e8bb8b9cb3907b > kernel: RDX: 0000000000000000 RSI: ffff880213ac6a00 RDI: ffff880222440000 > kernel: RBP: ffff880223187d58 R08: 000000000000000a R09: 0000000000000607 > kernel: R10: 00000000000213fc R11: 0000000000000607 R12: 0000000000000005 > kernel: R13: ffff880222a59000 R14: ffff880222440000 R15: ffff880213ac6a08 > kernel: FS: 00007fdddc839880(0000) GS:ffff88022ecc0000(0000) > knlGS:0000000000000000 > kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > kernel: CR2: 0000000000000010 CR3: 00000001ee978000 CR4: 00000000000407e0 > kernel: Stack: > kernel: ffffffffa0210bde ffff880200000018 ffff880223187d68 ffff880223187d28 > kernel: 00000000a5257e12 ffff88007f840208 0000000000000005 ffff880223187db0 > kernel: ffff880213ac6a08 ffff8802230ef000 ffff880213ac6a00 ffff880223187e28 > kernel: Call Trace: > kernel: [<ffffffffa0210bde>] ? mvs_abort_task+0x1ce/0x230 [mvsas] > kernel: [<ffffffffa0119eab>] sas_scsi_recover_host+0x47b/0xc20 [libsas] > kernel: [<ffffffffa0083afc>] scsi_error_handler+0xfc/0x580 [scsi_mod] > kernel: [<ffffffff81587212>] ? __schedule+0x362/0xa30 > kernel: [<ffffffffa0083a00>] ? scsi_eh_get_sense+0x190/0x190 [scsi_mod] > kernel: [<ffffffff810977f8>] kthread+0xd8/0xf0 > kernel: [<ffffffff81097720>] ? kthread_worker_fn+0x170/0x170 > kernel: [<ffffffff8158b962>] ret_from_fork+0x42/0x70 > kernel: [<ffffffff81097720>] ? kthread_worker_fn+0x170/0x170 > kernel: Code: 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 87 b0 00 00 00 89 f6 > 48 89 e5 f0 48 0f b3 30 5d c > kernel: RIP [<ffffffffa020dfa5>] mvs_slot_task_free+0x5/0x1f0 [mvsas] > kernel: RSP <ffff880223187d00> > kernel: CR2: 0000000000000010 > kernel: ---[ end trace 18b7a6f928680374 ]--- It didn't used to happen before, but now today got it again. Seems it's quite reproducible as my usage was pretty similar, basically heavy I/O, rsync and compiling. Also seems there's no way to get disks back but just reboot as removing kernel modules fail (not even with force). I narrowed it down to this section of mvs_abort_task function (drivers/scsi/mvsas/mv_sas.c)
} else if (task->task_proto & SAS_PROTOCOL_SATA ||
task->task_proto & SAS_PROTOCOL_STP) {
if (SAS_SATA_DEV == dev->dev_type) {
struct mvs_slot_info *slot = task->lldd_task;
u32 slot_idx = (u32)(slot - mvi->slot_info);
mv_dprintk("mvs_abort_task() mvi=%p task=%p "
"slot=%p slot_idx=x%x\n",
mvi, task, slot, slot_idx);
task->task_state_flags |= SAS_TASK_STATE_ABORTED;
mvs_slot_task_free(mvi, task, slot, slot_idx);
rc = TMF_RESP_FUNC_COMPLETE;
goto out;
}
}
Basically this line "u32 slot_idx = (u32)(slot - mvi->slot_info)".
I think (slot - mvi->slot_info) returns 0x10 and that's why
(there's no "mvs_abort_task()" in journal so it crashes before that.
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: sas: Enter sas_scsi_recover_host busy: 1 failed: 1
kernel: sas: trying to find task 0xffff8801fff87500
kernel: sas: sas_scsi_find_task: aborting task 0xffff8801fff87500
kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
kernel: IP: [<ffffffffa017afa5>] mvs_slot_task_free+0x5/0x1f0 [mvsas]
kernel: PGD 0
kernel: Oops: 0000 [#1] PREEMPT SMP
kernel: Modules linked in: nls_iso8859_4 nls_cp775 vfat fat fuse nvidia(PO) xt_CHECKSUM ipt_MASQUERADE nf_nat_masq
kernel: serio_raw pcspkr fam15h_power snd_hda_codec_realtek snd_hda_codec_hdmi snd_hda_codec_generic snd_hda_inte
kernel:
kernel: CPU: 3 PID: 222 Comm: scsi_eh_7 Tainted: P O 4.1.5-ARCH-dirty #2
kernel: Hardware name: Gigabyte Technology Co., Ltd. GA-990FXA-UD3/GA-990FXA-UD3, BIOS FFe 11/08/2013
kernel: task: ffff880222718000 ti: ffff88007fc9c000 task.ti: ffff88007fc9c000
kernel: RIP: 0010:[<ffffffffa017afa5>] [<ffffffffa017afa5>] mvs_slot_task_free+0x5/0x1f0 [mvsas]
kernel: RSP: 0018:ffff88007fc9fd00 EFLAGS: 00010a13
kernel: RAX: 2e8ba2e8ba2e8ba3 RBX: ffff8801fff87500 RCX: 45d175ba2d18107b
kernel: RDX: 0000000000000000 RSI: ffff8801fff87500 RDI: ffff88007fb80000
kernel: RBP: ffff88007fc9fd58 R08: 000000000000000a R09: 000000000000060d
kernel: R10: 0000000000020cd8 R11: 000000000000060d R12: ffff88007fb836a0
kernel: R13: ffff8800ce394e00 R14: ffff88007fb80000 R15: ffff8801fff87508
kernel: FS: 00007f0720ffe700(0000) GS:ffff88022ecc0000(0000) knlGS:0000000000000000
kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
kernel: CR2: 0000000000000010 CR3: 0000000224182000 CR4: 00000000000406e0
kernel: Stack:
kernel: ffffffffa017dce2 ffff880000000018 ffff88007fc9fd68 ffff88007fc9fd28
kernel: 0000000020e55177 ffff88022536f208 0000000000000005 ffff88007fc9fdb0
kernel: ffff8801fff87508 ffff8800ce321000 ffff8801fff87500 ffff88007fc9fe28
kernel: Call Trace:
kernel: [<ffffffffa017dce2>] ? mvs_abort_task+0x272/0x2b0 [mvsas]
kernel: [<ffffffffa030aeab>] sas_scsi_recover_host+0x47b/0xc20 [libsas]
kernel: [<ffffffffa00dfb0c>] scsi_error_handler+0xfc/0x580 [scsi_mod]
kernel: [<ffffffff81588152>] ? __schedule+0x372/0xa30
kernel: [<ffffffffa00dfa10>] ? scsi_eh_get_sense+0x190/0x190 [scsi_mod]
kernel: [<ffffffff81097818>] kthread+0xd8/0xf0
kernel: [<ffffffff81097740>] ? kthread_worker_fn+0x170/0x170
kernel: [<ffffffff8158c8a2>] ret_from_fork+0x42/0x70
kernel: [<ffffffff81097740>] ? kthread_worker_fn+0x170/0x170
Code: 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 87 b0 00 00 00 89 f6 48 89 e5 f0 48 0f b3 30 5d c3 0f 1f
80 00 00 00 00 66 66 66 66 90 <48> 83 7a 10 00 0f 84 60 01 00 00 55 48
kernel: Code: 84 00 00 00 00 00 66 66 66 66 90 55 48 8b 87 b0 00 00 00 89 f6 48 89 e5 f0 48 0f b3 30 5d c3 0f 1f 8
kernel: RIP [<ffffffffa017afa5>] mvs_slot_task_free+0x5/0x1f0 [mvsas]
kernel: RSP <ffff88007fc9fd00>
kernel: CR2: 0000000000000010
kernel: ---[ end trace 93debf717bb54039 ]---
(In reply to Dāvis from comment #3) > I narrowed it down to this section of mvs_abort_task function > (drivers/scsi/mvsas/mv_sas.c) > > } else if (task->task_proto & SAS_PROTOCOL_SATA || > task->task_proto & SAS_PROTOCOL_STP) { > if (SAS_SATA_DEV == dev->dev_type) { > struct mvs_slot_info *slot = task->lldd_task; > u32 slot_idx = (u32)(slot - mvi->slot_info); > mv_dprintk("mvs_abort_task() mvi=%p task=%p " > "slot=%p slot_idx=x%x\n", > mvi, task, slot, slot_idx); > task->task_state_flags |= SAS_TASK_STATE_ABORTED; > mvs_slot_task_free(mvi, task, slot, slot_idx); > rc = TMF_RESP_FUNC_COMPLETE; > goto out; > } > > } > > > Basically this line "u32 slot_idx = (u32)(slot - mvi->slot_info)". > I think (slot - mvi->slot_info) returns 0x10 and that's why > (there's no "mvs_abort_task()" in journal so it crashes before that. > Sorry for being idiot, that line doesn't cause any pointer dereference and neither does previous line. It's just so obvious, compiler reordered instructions so that mvs_slot_task_free is executed before mv_dprintk is called and that's why it's not in journal. Even as title I wrote NULL pointer dereference in mvs_slot_task_free and that's exactly where had to look. So anyway when in mvs_task_prep and if pci_pool_alloc fails then task->lldd_task is NULL as can see task->lldd_task = NULL; slot->n_elem = n_elem; slot->slot_tag = tag; slot->buf = pci_pool_alloc(mvi->dma_pool, GFP_ATOMIC, &slot->buf_dma); if (!slot->buf) goto err_out_tag; then later it's aborted with mvs_abort_task and there mvs_slot_task_free is called with (slot = task->lldd_task) which is NULL and in mvs_slot_task_free { if (!slot->task) return; happens this NULL pointer dereference because slot is NULL. There's 2 ways to fix this, either check if slot is NULL before calling mvs_slot_task_free or just inside it check it. I went for second option as it seems easier and won't have to always check before calling. Here's a patch, haven't tested it yet but I think it will fix this and it's compiling right now so I'll let know once I'll have tested it. diff --git a/drivers/scsi/mvsas/mv_sas.c b/drivers/scsi/mvsas/mv_sas.c index 454536c..9c78074 100644 --- a/drivers/scsi/mvsas/mv_sas.c +++ b/drivers/scsi/mvsas/mv_sas.c @@ -887,6 +887,8 @@ static void mvs_slot_free(struct mvs_info *mvi, u32 rx_desc) static void mvs_slot_task_free(struct mvs_info *mvi, struct sas_task *task, struct mvs_slot_info *slot, u32 slot_idx) { + if (!slot) + return; if (!slot->task) return; if (!sas_protocol_ata(task->task_proto)) Success, patch indeed fixed it :)
Now instead of crash I get this ↓, but everything seems to be working and no need for reboot.
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: mvsas 0000:07:00.0: mvsas prep failed[0]!
kernel: sas: Enter sas_scsi_recover_host busy: 19 failed: 19
kernel: sas: trying to find task 0xffff8801c9599100
kernel: sas: sas_scsi_find_task: aborting task 0xffff8801c9599100
kernel: sas: sas_scsi_find_task: task 0xffff8801c9599100 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff8801c9599100 is aborted
kernel: sas: trying to find task 0xffff8801c9599500
kernel: sas: sas_scsi_find_task: aborting task 0xffff8801c9599500
kernel: sas: sas_scsi_find_task: task 0xffff8801c9599500 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff8801c9599500 is aborted
kernel: sas: trying to find task 0xffff8801c9599900
kernel: sas: sas_scsi_find_task: aborting task 0xffff8801c9599900
kernel: sas: sas_scsi_find_task: task 0xffff8801c9599900 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff8801c9599900 is aborted
kernel: sas: trying to find task 0xffff8801ba22a500
kernel: sas: sas_scsi_find_task: aborting task 0xffff8801ba22a500
kernel: sas: sas_scsi_find_task: task 0xffff8801ba22a500 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff8801ba22a500 is aborted
kernel: sas: trying to find task 0xffff88000f686300
kernel: sas: sas_scsi_find_task: aborting task 0xffff88000f686300
kernel: sas: sas_scsi_find_task: task 0xffff88000f686300 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff88000f686300 is aborted
kernel: sas: trying to find task 0xffff88000f687f00
kernel: sas: sas_scsi_find_task: aborting task 0xffff88000f687f00
kernel: sas: sas_scsi_find_task: task 0xffff88000f687f00 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff88000f687f00 is aborted
kernel: sas: trying to find task 0xffff88000f687c00
kernel: sas: sas_scsi_find_task: aborting task 0xffff88000f687c00
kernel: sas: sas_scsi_find_task: task 0xffff88000f687c00 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff88000f687c00 is aborted
kernel: sas: trying to find task 0xffff88000f686e00
kernel: sas: sas_scsi_find_task: aborting task 0xffff88000f686e00
kernel: sas: sas_scsi_find_task: task 0xffff88000f686e00 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff88000f686e00 is aborted
kernel: sas: trying to find task 0xffff88000f686a00
kernel: sas: sas_scsi_find_task: aborting task 0xffff88000f686a00
kernel: sas: sas_scsi_find_task: task 0xffff88000f686a00 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff88000f686a00 is aborted
kernel: sas: trying to find task 0xffff88000f687d00
kernel: sas: sas_scsi_find_task: aborting task 0xffff88000f687d00
kernel: sas: sas_scsi_find_task: task 0xffff88000f687d00 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff88000f687d00 is aborted
kernel: sas: trying to find task 0xffff88000f686f00
kernel: sas: sas_scsi_find_task: aborting task 0xffff88000f686f00
kernel: sas: sas_scsi_find_task: task 0xffff88000f686f00 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff88000f686f00 is aborted
kernel: sas: trying to find task 0xffff88000f687500
kernel: sas: sas_scsi_find_task: aborting task 0xffff88000f687500
kernel: sas: sas_scsi_find_task: task 0xffff88000f687500 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff88000f687500 is aborted
kernel: sas: trying to find task 0xffff88000f687000
kernel: sas: sas_scsi_find_task: aborting task 0xffff88000f687000
kernel: sas: sas_scsi_find_task: task 0xffff88000f687000 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff88000f687000 is aborted
kernel: sas: trying to find task 0xffff8800024f1900
kernel: sas: sas_scsi_find_task: aborting task 0xffff8800024f1900
kernel: sas: sas_scsi_find_task: task 0xffff8800024f1900 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff8800024f1900 is aborted
kernel: sas: trying to find task 0xffff8800024f0d00
kernel: sas: sas_scsi_find_task: aborting task 0xffff8800024f0d00
kernel: sas: sas_scsi_find_task: task 0xffff8800024f0d00 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff8800024f0d00 is aborted
kernel: sas: trying to find task 0xffff88007c8ec600
kernel: sas: sas_scsi_find_task: aborting task 0xffff88007c8ec600
kernel: sas: sas_scsi_find_task: task 0xffff88007c8ec600 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff88007c8ec600 is aborted
kernel: sas: trying to find task 0xffff88007c8ecf00
kernel: sas: sas_scsi_find_task: aborting task 0xffff88007c8ecf00
kernel: sas: sas_scsi_find_task: task 0xffff88007c8ecf00 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff88007c8ecf00 is aborted
kernel: sas: trying to find task 0xffff88007c8ec700
kernel: sas: sas_scsi_find_task: aborting task 0xffff88007c8ec700
kernel: sas: sas_scsi_find_task: task 0xffff88007c8ec700 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff88007c8ec700 is aborted
kernel: sas: trying to find task 0xffff880079672700
kernel: sas: sas_scsi_find_task: aborting task 0xffff880079672700
kernel: sas: sas_scsi_find_task: task 0xffff880079672700 is aborted
kernel: sas: sas_eh_handle_sas_errors: task 0xffff880079672700 is aborted
kernel: sas: ata11: end_device-5:6: cmd error handler
kernel: sas: ata12: end_device-5:7: cmd error handler
kernel: sas: ata5: end_device-5:0: dev error handler
kernel: sas: ata6: end_device-5:1: dev error handler
kernel: sas: ata7: end_device-5:2: dev error handler
kernel: sas: ata8: end_device-5:3: dev error handler
kernel: sas: ata9: end_device-5:4: dev error handler
kernel: sas: ata10: end_device-5:5: dev error handler
kernel: sas: ata11: end_device-5:6: dev error handler
kernel: ata11.00: exception Emask 0x0 SAct 0x3fff78 SErr 0x0 action 0x6 frozen
kernel: sas: ata12: end_device-5:7: dev error handler
kernel: ata12.00: exception Emask 0x0 SAct 0x100 SErr 0x0 action 0x6 frozen
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/10:00:70:5b:78/00:00:12:00:00/40 tag 3 ncq 8192 in
res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/10:00:60:72:78/00:00:12:00:00/40 tag 4 ncq 8192 in
res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/10:00:10:b9:78/00:00:12:00:00/40 tag 5 ncq 8192 in
res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/20:00:48:bb:78/00:00:12:00:00/40 tag 6 ncq 16384 in
res 40/00:0c:b8:5e:aa/00:00:0e:00:00/40 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/20:00:10:d6:79/00:00:12:00:00/40 tag 8 ncq 16384 in
res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/20:00:f0:ed:79/00:00:12:00:00/40 tag 9 ncq 16384 in
res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/10:00:e0:2f:7a/00:00:12:00:00/40 tag 10 ncq 8192 in
res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/10:00:e8:8e:7a/00:00:12:00:00/40 tag 11 ncq 8192 in
res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/10:00:a0:a8:7a/00:00:12:00:00/40 tag 12 ncq 8192 in
res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/10:00:20:e8:7a/00:00:12:00:00/40 tag 13 ncq 8192 in
res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/10:00:b0:12:7c/00:00:12:00:00/40 tag 14 ncq 8192 in
res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/10:00:50:26:7c/00:00:12:00:00/40 tag 15 ncq 8192 in
res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/10:00:a8:d2:7c/00:00:12:00:00/40 tag 16 ncq 8192 in
res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/10:00:c0:ec:7c/00:00:12:00:00/40 tag 17 ncq 8192 in
res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/08:00:98:6f:70/00:00:12:00:00/40 tag 18 ncq 4096 in
res 40/00:64:b8:7e:aa/00:00:0e:00:00/40 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/08:00:b8:3c:73/00:00:12:00:00/40 tag 19 ncq 4096 in
res 40/00:70:a0:40:8e/00:00:29:01:00/40 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/08:00:68:57:73/00:00:12:00:00/40 tag 20 ncq 4096 in
res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11.00: failed command: READ FPDMA QUEUED
kernel: ata11.00: cmd 60/08:00:f8:89:73/00:00:12:00:00/40 tag 21 ncq 4096 in
res 40/00:ff:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
kernel: ata11.00: status: { DRDY }
kernel: ata11: hard resetting link
kernel: ata12.00: failed command: WRITE FPDMA QUEUED
kernel: ata12.00: cmd 61/c0:00:40:33:50/00:00:0d:00:00/40 tag 8 ncq 98304 out
res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout)
kernel: ata12.00: status: { DRDY }
kernel: ata12: hard resetting link
kernel: sas: sas_form_port: phy2 belongs to port6 already(1)!
kernel: sas: sas_form_port: phy3 belongs to port7 already(1)!
kernel: /mnt/linux/drivers/scsi/mvsas/mv_sas.c 1439:mvs_I_T_nexus_reset for
kernel: /mnt/linux/drivers/scsi/mvsas/mv_sas.c 1439:mvs_I_T_nexus_reset for
kernel: ata12.00: configured for UDMA/133
kernel: ata12.00: device reported invalid CHS sector 0
kernel: ata12: EH complete
kernel: ata11.00: configured for UDMA/133
kernel: ata11.00: device reported invalid CHS sector 0
kernel: ata11.00: device reported invalid CHS sector 0
kernel: ata11.00: device reported invalid CHS sector 0
kernel: ata11.00: device reported invalid CHS sector 0
kernel: ata11.00: device reported invalid CHS sector 0
kernel: ata11.00: device reported invalid CHS sector 0
kernel: ata11.00: device reported invalid CHS sector 0
kernel: ata11.00: device reported invalid CHS sector 0
kernel: ata11.00: device reported invalid CHS sector 0
kernel: ata11.00: device reported invalid CHS sector 0
kernel: ata11.00: device reported invalid CHS sector 0
kernel: ata11.00: device reported invalid CHS sector 0
kernel: ata11.00: device reported invalid CHS sector 0
kernel: ata11.00: device reported invalid CHS sector 0
kernel: ata11.00: device reported invalid CHS sector 0
kernel: ata11: EH complete
kernel: sas: --- Exit sas_scsi_recover_host: busy: 0 failed: 0 tries: 1
Forgive an ignoramus, but those last lines doesn't look to good
kernel: ata11.00: device reported invalid CHS sector 0
I have a problem that is very much like yours, but my stack traces are different so I'm unsure if we have the same problem. I'm going to rebuild my kernel as well with your fix and see if it helps me as well.
(In reply to Turbo Fredriksson from comment #6) > Forgive an ignoramus, but those last lines doesn't look to good > > kernel: ata11.00: device reported invalid CHS sector 0 > > I have a problem that is very much like yours, but my stack traces are > different so I'm unsure if we have the same problem. I'm going to rebuild my > kernel as well with your fix and see if it helps me as well. My fix is only for "NULL pointer dereference in mvs_slot_task_free" in mvsas driver. If you use hardware with different driver then this fix won't change anything for you. And even if you've such hardware and use this driver then you might have hit different bug, you really should have posted stack trace, logs, etc... As for those other messages, I've no clue what they actually mean. But as I understand under heavy I/O load pci_pool_alloc fails, so those tasks are aborted and that probably prevents kernel from accessing disks and disk reset is issued. Then it can access all disks again and everything keeps working. |