Bug 86121

Created attachment 153451 [details]
dmesg log (good - 7babfd7f)

Extracted from attachment#153311 [details] (timestamps removed).

Created attachment 153461 [details]
dmesg log (bad - ce027dac5)

Extracted from attachment#153311 [details] (timestamps removed).

Diffs that look relevant (-good +bad):

@@ -692,9 +692,11 @@
 ACPI: PCI Interrupt Link [LNKH] (IRQs 1 3 4 5 6 7 11 12 14 15) *0, disabled.
 ACPI: Enabled 3 GPEs in block 00 to 3F
 ACPI : EC: GPE = 0x17, I/O: command/status = 0x66, data = 0x62
+vgaarb: setting as boot device: PCI:0000:00:02.0
 vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
 vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none
 vgaarb: loaded
+vgaarb: overriding boot device: PCI:0000:01:00.0
 vgaarb: bridge control possible 0000:01:00.0
 vgaarb: no bridge control possible 0000:00:02.0
 SCSI subsystem initialized
@@ -801,7 +803,6 @@
 UDP hash table entries: 4096 (order: 5, 131072 bytes)
 UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes)
 NET: Registered protocol family 1
-pci 0000:00:02.0: Boot video device
 pci 0000:00:14.0: enabling device (0000 -> 0002)
 pci 0000:00:14.0: can't derive routing for PCI INT A
 pci 0000:00:14.0: PCI INT A: no GSI

Could you please provide lspci for your system as well as the content of /sys/bus/pci/devices/*/boot_vga in both good (3.16.4) and bad case?
Looking at your logs you have two GPUs in there, one being integrated Intel graphics I guess and the other one the NVIDIA discrete GPU.

According to your kernel logs (lines added by "bad") your boot framebuffer is on the discrete NVIDIA GPU while the good case just states the intel GPU would be boot GPU.

Your binary blob is not telling anything useful though. Could you share your Xorg log for both cases (it may contain more information)?


Note, your kernel config is enabling both CONFIG_FB_EFI and CONFIG_X86_SYSFB.

Please try 3.16.4 with CONFIG_FB_EFI=y and CONFIG_X86_SYSFB unset - you should get the same result than the bad config. Unsetting both should get you back to working state (probably for you working state is boot_vga=1 for intel GPU and bad state is boot_vga=1 for NVIDIA GPU).

Note, you might need to access your system via ssh/network if Xorg does not start properly while not having any framebuffer console.

Check if loading i915 makes any difference.

Created attachment 153871 [details]
'lspci -v' of the system

As you suspected the boot_vga for 00:02.0 (Intel) is 1 and for 01:00.0 (NVIDIA) it is 0 in the good case and vice versa in the bad case. In both cases everything else works just fine except the backlight control. I'm running the cases again and attach the Xorg logs shortly. One remark is that I have explicitly set the BusID to PCI:1:0:0 in the xorg.conf to get the X running correctly with NVIDIA GPU. I will do one run with bad and good cases without BusID set and attach the Xorg logs too.

I would expect that NVIDIA binary driver might have better chances selecting/using the GPU with 3.16.5 (without mentioning BusID).

Looking at your lspci i915 is loaded and active (though I did't see it in your kernel logs).
Does loading it (and having X able to switch between GPUs using GMUX or however the optimus equivalent is called on Apple systems) improve or influence the way your backlight controls work?

Did you check for registration of backlights on kernel side (see /sys/class/backlight/) and the eventual backlight properties reported by X via randr?

It would be good to know how backlight shows up with 3.16.4 and how it does with 3.16.5 (and how exactly it fails there).

Created attachment 154021 [details]
more details on backlight control registration

I did some rmmod / modprobe steps with both the good case (3.16.4+-7babfd7f) and new bad case (3.16.6-acfaf475) and checked out what demsg gives out and what is seen under /sys/class/backlight/

In the good case after removing the apple_gmux module, generic ACPI backlight controls were loaded for both Intel and NVIDIA cards (see the 06-sys-class-backlight.log in the 3.16.4+ case) and using the 'acpi_video0' I was also able to control the backlight just fine. Reloading the apple_gmux gave back the gmux_backlight control, which then continued to work correctly.

In the bad case removing the apple_gmux module left the /sys/class/backlight empty. Reloading it spitted out 'gmux device not present' even though it was
succesfully reported in the intial boot up (apple_gmux: Found gmux version 3.2.19 [indexed])

Looking at attachment #154021 [details] it seems that some ACPI/VIDEO and GMUX behavior changes are triggered though what/why exactly is hard to guess.


Did you try running 3.16.4 with CONFIG_X86_SYSFB unset (also unsetting CONFIG_FB_SIMPLE) and CONFIG_FB_EFI=y? If this config works the same as 3.16.5 at least the patch identified in bisection did not inherently break something but just surface/force a problematic configuration.



In order to get a better idea of what effectively happens, please report all the resources associated to GMUX, PNP device 'APP000B' (dumping /proc/iomem and /proc/ioports should be sufficient if APP000B can be identified there).


Also insert pr_debug() call into gmux_is_indexed() function in drivers/platform/x86/apple_gmux.c and let all pr_debug() calls in there reach dmesg by making sure CONFIG_DYNAMIC_DEBUG=y and following guidelines in https://www.kernel.org/doc/Documentation/dynamic-debug-howto.txt
Probably this should do it:
 echo 'file apple_gmux.c +p' > $debugfs/dynamic_debug/control

The interesting details in gmux_is_indexed() are (especially when loading apple_gmux):
  gmux_data->iostart
  inb(gmux_data->iostart + 0xcc)
  inb(gmux_data->iostart + 0xcd)
at start of function and
  val in case it's different from 0x55aa

From the error printed by apple_gmux on second modprobe it seems gmux state does not match expected state anymore (e.g. val differs from 0x55aa).


Why the acpi_video0 backlight device (and it seems the whole ACPI VIDEO interface) does not work/show up as expected would be good to understand as well (a dump of ACPI DSDT might give us a clue on this):
  cat /sys/firmware/acpi/tables/DSDT > dsdt.dat
and eventually decompile it:
  iasl -d dsdt.dat
to produce a dsdt.dsl file.

Created attachment 155101 [details]
3.16.4 vs. 3.16.5 with only EFIFB and with EFIFB + SYSFB

Looks like you were right that 3.16.4 using only the EFIFB framebuffer doesn't work, but using EFIFB + SYSFB does. With 3.16.5 backlight doesn't work either way.  In the failing cases it seems that gmux_is_indexed() gets value 0xffff from inb(gmux_data->iostart + {0xcc,0xcd}) when trying to load the module second time.

I hope that the dumps in the attachment gives some clue what might be wrong. I'm happy to provide any further information that could help.

(In reply to Petri Hodju from comment #9)
> Created attachment 155101 [details]
> 3.16.4 vs. 3.16.5 with only EFIFB and with EFIFB + SYSFB
> 
> Looks like you were right that 3.16.4 using only the EFIFB framebuffer
> doesn't work, but using EFIFB + SYSFB does. With 3.16.5 backlight doesn't
> work either way.  In the failing cases it seems that gmux_is_indexed() gets
> value 0xffff from inb(gmux_data->iostart + {0xcc,0xcd}) when trying to load
> the module second time.
> 
> I hope that the dumps in the attachment gives some clue what might be wrong.
> I'm happy to provide any further information that could help.

Thanks for these details and confirming my assumption that my (two) patches just force the problematic setup and are not changing the behavior originally performed by efifb (when it is not disabled by SYSFB).


From the inb readings in apple_gmux initialization when it fails, it seems something (probably vgaarb) disabled/blocked IO for the gmux (0x0700)

The 0xffff readings of inb indicate that there is no one (anymore) at the 0x0700+0xcc IO port being read and probably vgaarb has somehow disabled the corresponding device or disabled IO on parent PCI bridge.

Comparing lspci -vvv could reveal a difference between different boot stages (or between godd/bad kernels). If you can capture this lspci output prior to loading any graphics/gmux module and after loading them, differences might point out what is needed to get/keep the IO port active.


I will have to study the DSDT in the hope to find some hint on why things go wrong (at least ACPI backlight devices not showing up properly after rmmod apple_gmux), though I guess it's related to the IO port!


If you can get your system to actively switch between both GPUs (with 3.16.4 and using SYSFB or disabling EFIFB) it would be interesting to see if/how switching the GPU affects ability to operate backlight and access gmux IO port. When doing so, check kernel log for further messages about vgaarb changing settings.
I expect it could degrade experience on 3.16.4 (like causing apple_gmux to not initialize anymore) or start working on 3.16.5.

Created attachment 161641 [details]
dmesg and lspci logs after boot and suspend/resume

One more observation on this. I used now 3.18.0 and after boot the apple_gmux backlight control is loaded but not working. I then do suspend/resume and after wakeup the backlight is working. There is difference in the lspci -vvv before and after suspend.

(In reply to Petri Hodju from comment #11)
> Created attachment 161641 [details]
> dmesg and lspci logs after boot and suspend/resume
> 
> One more observation on this. I used now 3.18.0 and after boot the
> apple_gmux backlight control is loaded but not working. I then do
> suspend/resume and after wakeup the backlight is working. There is
> difference in the lspci -vvv before and after suspend.

The biggest difference is that VGA bits on the bridge have changed, thus "reverting" the vgaarb GPU switch triggered by gmux while loading nvidia driver.

So on your system backlight does not work when VGA-arbitration enables VGA routing to your nvidia GPU on the bridge the nvidia GPU is connected to.

The proper fix for you would probably be to inhibit vga arbitration actions (enabling VGA on bridge to nvidia).


On your system, could you check what happens, step by step, checking
backlight function via backlight node under sysfs:
- boot (blacklisting i915 and nouveau)
  backlight should be working
- load gmux
  backlight should be working
- load i915
  backlight should be working
- load nvidia
  backlight should stop working (unless only when X loads)

Redo the same steps using nouveau instead of nvidia.

Created attachment 162591 [details]
loading gmux/i915/nvidia & gmux/i915/nouveau

Please find attached logs on loading the modules in the order you requested. I did blacklist nouveau, i915, nvidia and apple_gmux and booted directly in console mode avoiding DE to be loaded.

The end results are mostly as you expected. The only noteworthy difference is that straight after boot the /sys/class/backlight/ was empty. After loading the gmux the backlight started to work. Also after loading i915 the backlight was working. After loading nvidia the backlight stopped to work as you expected. Using nouveau instead of nvidia kept the backlight working.

Then I removed gmux from the blacklist and booted to the console. In this case the gmux was visible under the /sys/class/backlight but was not working.

The fact that using nouveau does not kill the backlight controls and also does not trigger any vga-arbitration switches is good news.
This probably means that the backlight is in fact controlled by Intel GPU though announced to OS via gmux PNP device (not via legacy VGA IO but still via Intel GPU's IO)

A quick search for VGA arbitration and nvidia revealed the following link:
  https://devtalk.nvidia.com/default/topic/545560/vfio-vga-arbitration-lock/

Maybe that patch would help, though I haven't looked at the sources of nvidia driver.

I'm thinking about letting gmux lock vga_default_device as owning device so nvidia would fail at switching things around. This way I should avoid affecting any third-parties (desktops) that need to do this kind of arbitration with some IGP.

Hopefully I can more or less test that on my AMD APU with secondary NVIDIA GPU and using nvidia blob.

Created attachment 162861 [details]
Lock IO+MEM of boot_vga in apple-gmux

Petri, could you test this patch (just compile-tested here)?

I'm interested in dmesg for loading nvidia after apple-gmux with this patch applied.

Possible outcomes I envision:
- no change
- it prevents decode changes toward nvidia GPU and thus keeps backlight working
- loading/initialization of nvidia never completes (could happen if nvidia calls vga_get() instead of vga_tryget())

For dmesg output to be more verbose, please follow guidelines in Documentation/dynamic-debug-howto.txt to enable dynamic debugging and have it generate debug output for both vgaarb and apple-gmux (that is, files: drivers/platform/x86/apple-gmux.c and drivers/gpu/vga/vgaarb.c).
Enable dynamic debug before loading gmux.

Created attachment 165781 [details]
dmesg logs with IO+MEM locked to boot GPU

Sorry about the delay here. With the patch things took a step back I'm afraid. Loading of the apple_gmux fails now with iostart[{0xcc,0xcd}] giving out 0xffff. It's like you mentioned earlier that looks like the intel side IO is shutdown by someone after the vga arbitration on boot changes to use Nvidia instead of Intel.

Created attachment 166231 [details]
Lock IO+MEM of (first) IO decoding GPU

(In reply to Petri Hodju from comment #16)
> Created attachment 165781 [details]
> dmesg logs with IO+MEM locked to boot GPU
> 
> Sorry about the delay here. With the patch things took a step back I'm
> afraid. Loading of the apple_gmux fails now with iostart[{0xcc,0xcd}] giving
> out 0xffff. It's like you mentioned earlier that looks like the intel side
> IO is shutdown by someone after the vga arbitration on boot changes to use
> Nvidia instead of Intel.

Oops, I passed the wrong pdev to vga_tryget()!
In our case boot_vga is the dGPU (nvidia) and I wanted to pass the iGP (intel).

Attached is new patch which should get that right. With this patch we just ensure gmux continues working but we don't fix things in case of detection failure.

Created attachment 166511 [details]
working backlight

Seems like that was it! Now the gmux backlight control keeps working on my system even after loading the nvidia module in console mode. Thanks for your effort on this! Please find attached dmesg logs showing the progress while loading: apple_gmux -> i915 -> nvidia. The patch applied cleanly over 3.19, from which the logs are captured.

Good news.

Could you also verify that rmmod and modprobe of apple-gmux a few times in a row works? If so I will send it to Bjorn and CC maintainers as well as dri-devel to get some more testing and then get it applied.

If you modprobe nvidia while apple-gmux is not loaded trying to load it at a later time (even after unloading nvidia) would probably fail. As I don't know what device handles backlight IO for all Apple Macs I'm reluctant to randomly try locking IO for graphics devices (especially as it might confuse running GPU drivers)

Created attachment 167391 [details]
loading apple_gmux multiple times

Things seem to work as you expected. When loading / unloading the apple_gmux multiple times in a row, the gmux is able to lock to the correct I/O and the backlight keeps functioning correctly. The backlight also keeps working if the gmux is loaded at the time nvidia gets loaded. As you expected, if I unload the gmux and then load nvidia, the backlight stops functioning and won't recover even by unloading nvidia and loading the gmux again. This behaviour is understandable as you described. I've attached here dmesg logs to show how things worked out.

Bruno, if you sent this patch to me, I lost it.  Can you resend it and make sure linux-pci is cc'd?

Bjorn, I've sent it to Darren Hart as he is the maintainer of apple_gmux (you were CCed).

Resent, 8bit encoded for Darren and linux-pci CCed.

4eebd5a4e726 ("apple-gmux: lock iGP IO to protect from vgaarb changes") appeared in v4.1 and mentions this bugzilla.  I'm assuming that change fixed this bug, so I'm closing it.  Please reopen if this is incorrect.

http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4eebd5a4e726