My motherboard is Supermicro X10DRG-Q (details in attached output of dmidecode). It has 2 Xeon E5-2620 v4 (details in attached lscpu output). Two Titan X 2016 GPUs are inserted into PCIe slots (see nvidia-smi output). After enabling of the peer to peer access between those two cards, execution of cudaMemcpyPeer() hangs and dmesg shows: [16193.612535] DMAR: DRHD: handling fault status reg 602 [16193.617662] DMAR: [DMA Write] Request device [82:00.0] fault addr 387fc000c000 [fault reason 05] PTE Write access is not set [16193.661857] DMAR: DRHD: handling fault status reg 702 [16193.666976] DMAR: [DMA Write] Request device [82:00.0] fault addr f8139000 [fault reason 05] PTE Write access is not set (edited) I am using CoreOS, and the whole stuff happens inside a docker container running with -device /dev/nvidiactl --device /dev/nvidia0 --device /dev/nvidia1 --device /dev/nvidia-uvm --privileged --security-opt seccomp=unconfined The addition of intel_iommu=igfx_off to kernel command line cures the problem and peer to peer works perfectly.
Created attachment 245361 [details] dmidecode -t 2
Created attachment 245371 [details] lscpu
Created attachment 245381 [details] lspci -knnv
Created attachment 245391 [details] nvidia-smi proto -m
Created attachment 245401 [details] cat /proc/cmdline Added intel_iommu=off
Created attachment 245411 [details] uname -a