Last known good version: 4.9.8
Software: Arch Linux, XFCE, compton, nouveau (using DRI3)
Hardware: NVIDIA Corporation GK107 [GeForce GT 740] (rev a1)
The behavior *and* workaround for this bug are 100% identical to https://bugs.freedesktop.org/show_bug.cgi?id=97916
Quoting its behavior because they explained it way better than I could:
With default settings (no xorg.conf), dual monitors can only be used in "clone" mode. When trying to switch to "extend" mode, the right-hand display freezes and continues to display a snapshot of the cloned display contents at the time of the switch. Using Option "DRI" "2" in xorg.conf is a workaround.
But the cause is clearly different, because of 3 major differences from that bug report:
1. I started having this issue when I upgraded the kernel to 4.9.10 from 4.9.8, rather than when it started for him: 4.7.4. And yes, I already tested to confirm that this behavior changes between those two versions of Linux, and not with any other upgrade or downgrade to any of my other packages (not even nouveau, mesa, or xorg).
2. When X and XFCE first run, this behavior doesn't happen at all (assuming it wasn't happening before I shut it down). My dual-monitor setup still works properly. However, once I disconnect that monitor--or if it was disconnected when I rebooted, causing my system to remember that on boot--this behavior starts, and does not go away until I disable and enable the monitor in XFCE's Display preferences, which then restores the proper behavior until the next disconnect. It specifically has to be XFCE's Display preferences. Adding a regular, simple (using nothing but two Monitor sections and Identifiers matching the monitor names) dual-monitor .conf file to xorg.conf.d/ actually made it worse by causing this behavior to start immediately at boot no matter what, and for some reason xrandr couldn't turn the monitor back on after running xrandr --output <output> --off (though XFCE's preferences COULD turn it back on).
3. I still have this behavior when compton is disabled and I'm not running any compositor at all. I specifically have to either downgrade the kernel or run compton with the --paint-on-overlay option to fix it.
3.5. I have not yet tested for whether switching to DRI2 is another workaround like it is for that bug. It's been a long night and I'm taking a break.
OK, this is getting really weird. Despite working over quite a few monitor disconnections for the past few hours, the --paint-on-overlay option is suddenly no longer fixing it. I *have* to turn the display on and off in XFCE's settings now.
I've just confirmed that switching to DRI2 doesn't fix it, either.
Hello? I still can't upgrade my kernel until this gets fixed.
I can't just never upgrade my kernel ever again...
Please use bugs.freedesktop.org. That's what all the graphics people look at. That's where the proper categories live. I just happened to notice this when scanning dri-devel.
You mention that something broke in 4.9.10 (or .9) -- have you tried other kernels? 4.10.x? 4.11.x? Various bugs in nouveau get fixed, etc.
Lastly, you may consider getting help on irc.freenode.net #nouveau - that may be the fastest way to work out the issues.
I believe part of one of the problems was a bug in xf86-video-nouveau which was fixed in 1.0.14: https://cgit.freedesktop.org/nouveau/xf86-video-nouveau/commit/?id=924083938c8f209d8f6ff472caf8692a644f7e78
Unfortunately based on your description, you could be running into bugs in any number of pieces of software, and getting logs would be the most productive thing to do. But not here -- please use bugs.freedesktop.org, as suggested by the nouveau reporting bugs page, https://nouveau.freedesktop.org/wiki/Bugs/
Well, that's some obscure yet vital information. Why does this site even have the option to report bugs in a category that nobody pays attention to, rather than directing a potential bug reporter to the right place? And before you say that nouveau would've pointed me to the right spot, keep in mind I have no reason yet to suspect this is even a bug with nouveau (though it is a possibility), and have a lot of reason to suspect it's a bug with the kernel, since upgrading and downgrading my kernel changes its behavior.
Here's the new bug report in the proper place: https://bugs.freedesktop.org/show_bug.cgi?id=101372