Stephen reported that on 2.6.30, closing the lid of an HP/Compaq 6910p laptop causes an oops (see attachment). This worked correctly in 2.6.29.4, so this is a regression. If Stephen unloads the "button" module, the laptop locks up instead of oopsing when closing the lid. Reported at http://lkml.org/lkml/2009/7/9/100
Created attachment 22295 [details] oops log
> I've not had time to go back to it but with 2.6.30 I get an oops > when I close the lid on my HP Compaq 6910p laptop. I opened this bugzilla for the oops: http://bugzilla.kernel.org/show_bug.cgi?id=13751 Other reports that may be related: http://lkml.indiana.edu/hypermail/linux/kernel/0906.2/02139.html https://bugs.launchpad.net/ubuntu/hardy/+source/hotkey-setup/+bug/157691 I see floundering and workarounds in the reports above, but no real solution yet. The oops is a page fault on the IP, which means we branched into the weeds, possibly by following a bad function pointer: BUG: unable to handle kernel paging request at ffff88007f223c30 IP: [<ffff88007f223c30>] 0xffff88007f223c30 One possible cause is a module that doesn't clean up properly when it is unloaded. Can you reproduce the problem with CONFIG_MODULE_UNLOAD turned off? I doubt this is a button driver problem, but attached is a patch that reverts the driver to the 2.6.29 version. Can you try it out? Bjorn
Created attachment 22296 [details] revert button driver to 2.6.29 version Here's the patch mentioned above.
With the patch it no longer oops or hangs up. I applied it to 2.6.30.1 though. Is it worth checking with 2.6.30? I can unload the module and reload it fine (although I notice it doesn't reuse the same /class/input/input## number). There is one difference between what I was doing and what I did recently. When I first found this problem it was when I was putting my laptop in my docking station, so it was attached to an external display (and that is why I was closing the lid). When doing the check this morning with unloading the module and now with the patch it isn't in the docking station. If that could be relevant I could try to get to my office tomorrow to check.
BTW, I just read the links from the first post where the author has teh same laptop as I have. My video device has a different number than the ones references in the posts; [root@antonia ~]# cat /proc/acpi/video/C098/DOS DOS setting: <0> I didn't try with disabling the UNLOAD and using the normal 2.6.30[.1] button.c. Is it worth doing that still? I can reproduce this problem from a fresh boot so I doubt anything has been unloaded.
If you can reproduce the problem in 2.6.30.1, and reverting to the 2.6.29 button driver makes a difference, we can start looking at that driver. But my guess is that the docking station and external display are necessary to reproduce the problem, and we changed too many things at once (hardware config, kernel version, and button driver). We need to start by reproducing the oops. Then I'd like to know whether (a) using the 2.6.29 button driver makes a difference, and (b) whether using CONFIG_MODULE_UNLOAD=n with the normal kernel makes any difference.
Yeah, looks like it needs to be in my docking station. It doesn't happen with 2.6.30 without being plugged in. I even plugged in an external monitor (which should be the same, it is just a port replicator actually, not a docking station) but that didn't trigger it. Hopefully I'll get by my office tomorrow or it might be a couple of weeks before I'm back there.
Will you please enable "CONFIG_I915_KMS" in kernel configuration and see whether the issue still happens when closing the LID? Thanks.
Sorry that it should be "CONFIG_DRM_I915_KMS". Thanks.
Created attachment 22301 [details] Oops using 2.6.29's button.c in 2.6.30.1.
Okay, in my docking station I can again get the problem. However, now it manifests as just a hang. Using the 2.6.31-rc2 kernel I get the same hang. Turning on the CONFIG_DRM_I915_KMS option removes this hang. However, the x-server goes a bit crazy when I log out. Lots of small horizontal lines. When I use 2.6.30.1 I get the hang. When I patched it for the older version of button.c I get an Ooops again when closing the lid, attached (button2.oops). So I think 2.6.31-rc2 + that option should work (although now I've booted into it for a second time I have some gconf problem... may related to just all the crashing).
Created attachment 22302 [details] print names of unloaded modules Thanks for doing all this testing. Using CONFIG_DRM_I915_KMS=y might be another workaround, but we still need to fix the hangs and oopses that happen when it's disabled. Hangs are hard to debug, so the most interesting thing to me is the oops that happens with the older button.c -- that gives us real data to work with. Both oopses show an invalid IP, which may be the result of memory corruption. Can you set the following config options: CONFIG_DEBUG_PAGEALLOC=y CONFIG_PAGE_POISONING=y CONFIG_FRAME_POINTER=y CONFIG_DEBUG_NOTIFIERS=y CONFIG_MODULE_UNLOAD=n (note this one is *disabled*, not enabled) Before closing the lid, please also collect /proc/modules and /proc/kallsyms, e.g., # cat /proc/modules > modules # cat /proc/kallsyms > kallsyms If the problem goes away when CONFIG_MODULE_UNLOAD=n, that's a good clue that some module's unload is buggy. So if you can't reproduce the oops with CONFIG_MODULE_UNLOAD=n, turn it back on and try it again with this patch, and maybe we can get a list of modules to look at.
The kms should be enabled on the HP6910p laptop if the video driver is loaded. In fact we see this issue on another HP laptop. The content in ACPI intel opregion is not initialized correctly while loading the video driver if the box is booted without KMS. And when the LID is closed, it will trigger SMI operation and print the following oops. >BUG: unable to handle kernel paging request at ffff88007f2f0001 And when the system is booted with KMS mode, the content in ACPI intel opregion will be initialized correctly. And then SMI will be skipped. In such case the kernel panic doesn't happen again. When SMI is triggered, it can't be controlled by the OS and it is transparent to OS. In such case we can do nothing to fix them. So IMO this box had better be booted with KMS enabled. Thanks.
> The content in ACPI intel opregion is not initialized correctly while > loading the video driver if the box is booted without KMS. So *why* isn't that opregion initialized correctly with a non-KMS driver? Is that something that cannot be fixed for some reason? Using KMS is not an option for everybody. Björn: do you still need the testing from comment #12 done given Yakui Zhao's comments?
Hi, Frans If the box is booted with KMS disabled, the system don't know the number and the type of display output device. In such case we can't know how to fill the content in intel opregion. So KMS is required on the HP6910p laptop. When it is booted with KMS enabled, the content in ACPI opregion can be initialized correctly. Thanks.
this is a duplicate of bug 11259. *** This bug has been marked as a duplicate of bug 11259 ***
Zhang, are you sure this is the same bug? I didn't have the problem with 2.6.29.6 (or earlier), only with 2.6.30 or later. As I only see this when my laptop is in the docking station I'll need to wait till I'm back at work to see if the patch in other bug helps.
The patch from Matthew is shipped in the 2.6.30-rc1. >commit 74a365b3f354fafc537efa5867deb7a9fadbfe27 Author: Matthew Garrett <mjg59@srcf.ucam.org> Date: Thu Mar 19 21:35:39 2009 +0000 ACPI: Populate DIDL before registering ACPI video device on Intel In this patch only when the box is booted with KMS enabled, it will register the ACPI video driver. Otherwise the ACPI video driver won't be bound with the ACPI video device. But the above dependency is too strict. And on many laptops there is no ACPI backlight I/F if the box is booted without KMS. It is not convenient. So Matthew sent another patch to fix this issue. This is shipped in the 2.6.30-rc4. >commit d770e3cfe5a274a343d896b2cc1646af85646fbc Author: Matthew Garrett <mjg59@srcf.ucam.org> Date: Wed Apr 15 21:46:36 2009 +0100 drm/i915: Register ACPI video even when not modesetting In this patch it will register the ACPI video driver even when the box is booted with KMS disabled. And if KMS is not enabled on HP6910 laptop, the ACPI opregion won't be initialized correctly. And it will trigger the OOPS as happened on the laptop in bug 11259. Thanks.
On Monday 20 July 2009, you wrote: > >commit 74a365b3f354fafc537efa5867deb7a9fadbfe27 > ACPI: Populate DIDL before registering ACPI video device on Intel > In this patch only when the box is booted with KMS enabled, it will > register the ACPI video driver. Otherwise the ACPI video driver > won't be bound with the ACPI video device. > > >commit d770e3cfe5a274a343d896b2cc1646af85646fbc > drm/i915: Register ACPI video even when not modesetting > In this patch it will register the ACPI video driver even when the > box is booted with KMS disabled. And if KMS is not enabled on HP6910 > laptop, the ACPI opregion won't be initialized correctly. And it > will trigger the OOPS as happened on the laptop in bug 11259. Neither of these patches fixes the oops for my HP 2510p (without KMS). I first reported it with 2.6.28 and can still reproduce it with 2.6.30.1. Since the workaround of setting /proc/acpi/video/C09A/DOS to 7 works well for me (and because ACPI in general confuses the hell out of me), I'm not all that motivated to persue this myself. However, I am quite willing to test patches.
If this is really the same as bug 11259, the resolution seems to require the user to enable KMS. I don't think that's acceptable, because it means users who don't enable KMS still see the oops/crash/hang, and we still have to filter out those bug reports, so I'm reopening this. Stephen, both of the patches Zhang mentioned in comment #18 are in 2.6.30, so you've already tested them. As a fool willing to rush in where angels fear to tread, I'm still willing to work on this. So if anybody collects the information from comment #12, I'll look at it. Please add "CONFIG_X86_CHECK_BIOS_CORRUPTION=y" to the comment #12 config settings. I am intrigued by Len's idea (comment #15 in bug 11259) that there might be a dependency on having AML run on CPU 0. All the oopses attached to 11259 and 13751 are on CPU 1, and acpi_ev_asynch_execute_gpe_method() appears in two backtraces.
Created attachment 22417 [details] Lid closure oops on HP 2510p (2.6.30.1) Hi Björn, Here's a recent oops I got on my HP 2510p notebook (while not docked). I'll build a kernel with the config you suggested and post the results sometime in the next few days. Cheers, FJP
(In reply to comment #20) > If this is really the same as bug 11259, the resolution seems to require the > user to enable KMS. I don't think that's acceptable, because it means users > who don't enable KMS still see the oops/crash/hang, and we still have to > filter > out those bug reports, so I'm reopening this. For most laptops there is no such issue and it is unnecessary to enable KMS. But it is required only on the several HP laptops(6910, 6710s). As mentioned in bug 11259, the SMI will be triggered if the ACPI opregion is not initialized correctly on HP6910p laptop. As SMI is transparent to OS, we can't know what happened in SMI. So to avoid triggering the SMI operation, it will be better that the content in ACPI opregion is initialized correctly. And this depends on the KMS. Thanks. > > Stephen, both of the patches Zhang mentioned in comment #18 are in 2.6.30, so > you've already tested them. > > As a fool willing to rush in where angels fear to tread, I'm still willing to > work on this. So if anybody collects the information from comment #12, I'll > look at it. Please add "CONFIG_X86_CHECK_BIOS_CORRUPTION=y" to the comment > #12 > config settings. > > I am intrigued by Len's idea (comment #15 in bug 11259) that there might be a > dependency on having AML run on CPU 0. All the oopses attached to 11259 and > 13751 are on CPU 1, and acpi_ev_asynch_execute_gpe_method() appears in two > backtraces.
Created attachment 22431 [details] Oops, kernel log, /proc/{modules,kallsyms} for Hp 2510p Here's the data for my system (HP 2510p). Included are the oopses (from netconsole), the kernel log for that boot, and the contents of the requested /proc/ files (from a previous boot of the same kernel, but I hope that does not matter). I had to log the oops using netconsole as the system hangs solid (no SysRq) after opening the lid and the kern.log is either not updated at all or contains only partial info. Note my comment in the oopses: they look to be divided over "when the lid is closed" and "when the lid is opened again". The system is at least partially alive for a bit after reopening the lid. I did make one weird discovery: closing the lid produces no errors if wireless is not in use. Because I had to use netconsole, I initially did ifup eth0 ifdown wlan0 and then <close lid> <open lid> two times and the system was fine. The oopses in the attachment were obtained with both wlan0 and eth0 up (as my wlan0 could not be used for netconsole: no polling). Comments: CONFIG_DEBUG_PAGEALLOC=y : set CONFIG_PAGE_POISONING=y : not set (depends !ARCH_SUPPORTS_DEBUG_PAGEALLOC) CONFIG_FRAME_POINTER=y : set CONFIG_DEBUG_NOTIFIERS=y : set CONFIG_MODULE_UNLOAD=n : set to "y"; I've not (yet) tested with "n" CONFIG_X86_CHECK_BIOS_CORRUPTION=y : set
Created attachment 22480 [details] 2.6.30.2 config file to reproduce I reproduced this problem on a 2510p with BIOS F.0A (1/25/2008) running Lenny (Debian 5.0.2) and a 2.6.30.2 kernel with the attached .config (based on one from Frans). I did not need a docking station or external monitor, though I did have a USB mouse attached. It's possible that having the wireless NIC up made the crash more likely, though it did happen with it down. The non-free firmware-iwlwifi package is required to get the wireless NIC to work.
Created attachment 22482 [details] patch to bind ACPI workqueue threads to CPU 0 In my limited testing, this patch was enough to avoid the crash. My testing showed that the crash happens when we run acpi_ev_notify_dispatch() or acpi_ev_asynch_execute_gpe_method() on CPU 1, so I suspect there's some AML that assumes it is running on CPU 0. There are several ways we might work around this, and I don't know what is best. This patch binds singlethread workqueues such as kacpid, kacpi_notify, and kacpi_hotplug to the current CPU when the workqueue is created. That might be nice for some drivers because a workqueue thread can be placed close to a device, but other applications might not want the thread bound to a CPU at all. Another approach would be to use set_cpus_allowed() to switch to CPU 0 inside acpi_os_execute_deferred(). That avoids the crash, too, but means more work on every deferred execution. And it would be nice to have a visible indication of this requirement so we don't accidentally break it in the future. I'll attach DMI information that we could use for some sort of quirk.
Created attachment 22483 [details] patch to use set_cpus_allowed() to force deferred AML to CPU 0 Here's an ugly patch to use set_cpus_allowed() to temporarily switch the workqueue thread to CPU 0. Still has a bunch of debug in it, so this is just to show the idea.
Created attachment 22484 [details] DMI/SMBIOS information for HP 2510p Here's dmidecode info we might use to make a 2510p-specific quirk. Stephen, if you could try out one of these patches and post your results and the corresponding 6910p dmidecode info, that would be great.
I'll add the dmidecode output in a moment. I tried the patch from #26 and it no longer has problems, it does generate this output (which varies slightly) when closing the lid but I guess that is due to the patch (I didn't look at it in detail); CPU 0 __acpi_os_execute queuing acpi_ev_asynch_execute_gpe_method+0x0/0x14c to kacpid CPU 0 acpi_os_execute_deferred running acpi_ev_asynch_execute_gpe_method+0x0/0x14c CPU 0 __acpi_os_execute queuing acpi_ev_notify_dispatch+0x0/0x60 to kacpi_notify CPU 0 __acpi_os_execute queuing acpi_ev_notify_dispatch+0x0/0x60 to kacpi_notify CPU 0 acpi_os_execute_deferred running acpi_ev_notify_dispatch+0x0/0x60 CPU 0 acpi_os_execute_deferred running acpi_ev_notify_dispatch+0x0/0x60 CPU 0 __acpi_os_execute queuing acpi_ev_notify_dispatch+0x0/0x60 to kacpi_notify CPU 0 __acpi_os_execute queuing acpi_ev_asynch_enable_gpe+0x0/0x1e to kacpi_notify CPU 0 acpi_os_execute_deferred running acpi_ev_notify_dispatch+0x0/0x60 CPU 0 acpi_os_execute_deferred running acpi_ev_asynch_enable_gpe+0x0/0x1e CPU 1 __acpi_os_execute queuing acpi_ev_asynch_execute_gpe_method+0x0/0x14c to k acpid CPU 0 acpi_os_execute_deferred running acpi_ev_asynch_execute_gpe_method+0x0/0x14c CPU 0 __acpi_os_execute queuing acpi_ev_notify_dispatch+0x0/0x60 to kacpi_notify CPU 0 __acpi_os_execute queuing acpi_ev_asynch_enable_gpe+0x0/0x1e to kacpi_notify CPU 0 acpi_os_execute_deferred running acpi_ev_notify_dispatch+0x0/0x60 CPU 0 __acpi_os_execute queuing acpi_ev_notify_dispatch+0x0/0x60 to kacpi_notify CPU 0 acpi_os_execute_deferred running acpi_ev_asynch_enable_gpe+0x0/0x1e CPU 0 acpi_os_execute_deferred running acpi_ev_notify_dispatch+0x0/0x60 CPU 1 __acpi_os_execute queuing acpi_ev_asynch_execute_gpe_method+0x0/0x14c to kacpid CPU 0 acpi_os_execute_deferred running acpi_ev_asynch_execute_gpe_method+0x0/0x14c CPU 0 __acpi_os_execute queuing acpi_ev_notify_dispatch+0x0/0x60 to kacpi_notify CPU 0 acpi_os_execute_deferred running acpi_ev_notify_dispatch+0x0/0x60 CPU 0 __acpi_os_execute queuing acpi_ev_notify_dispatch+0x0/0x60 to kacpi_notify CPU 0 acpi_os_execute_deferred running acpi_ev_notify_dispatch+0x0/0x60 CPU 0 __acpi_os_execute queuing acpi_ev_notify_dispatch+0x0/0x60 to kacpi_notify CPU 0 __acpi_os_execute queuing acpi_ev_asynch_enable_gpe+0x0/0x1e to kacpi_notify CPU 0 acpi_os_execute_deferred running acpi_ev_notify_dispatch+0x0/0x60 CPU 0 acpi_os_execute_deferred running acpi_ev_asynch_enable_gpe+0x0/0x1e
Created attachment 22519 [details] DMI/SMBIOS information for HP Compaq 6910p
Created attachment 22528 [details] bind ACPI workqueues to CPU 0 Posted to linux-acpi: http://marc.info/?l=linux-acpi&m=124890447020617&w=2 I think this is the same bug reported here: http://bugzilla.kernel.org/show_bug.cgi?id=11259 https://bugs.launchpad.net/bugs/157171 https://bugs.launchpad.net/bugs/157691 http://lkml.org/lkml/2008/6/16/157 http://lkml.org/lkml/2009/1/12/170 http://lkml.org/lkml/2009/6/19/311 http://lkml.org/lkml/2009/7/9/100
On Saturday 01 August 2009, Bjorn Helgaas wrote: > On Wednesday 29 July 2009 06:59:59 pm Zhang Rui wrote: > > On Thu, 2009-07-30 at 05:54 +0800, Bjorn Helgaas wrote: > > > On some machines, a software-initiated SMI causes corruption unless the > > > SMI runs on CPU 0. An SMI can be initiated by any AML, but typically > it's > > > done in GPE-related methods that are run via workqueues, so we can avoid > > > the known corruption cases by binding the workqueues to CPU 0. > > > > > > References: > > > http://bugzilla.kernel.org/show_bug.cgi?id=13751 > > > https://bugs.launchpad.net/bugs/157171 > > > https://bugs.launchpad.net/bugs/157691 > > > > > > Signed-off-by: Bjorn Helgaas <bjorn.helgaas@hp.com> > > > > Acked-by: Zhang Rui <rui.zhang@intel.com> > > In addition to the reports above, I think it's likely this patch > will fix the problems reported below: > > http://bugzilla.kernel.org/show_bug.cgi?id=13412 > http://bugzilla.kernel.org/show_bug.cgi?id=11259 > http://bugzilla.kernel.org/show_bug.cgi?id=12328 > http://bugzilla.kernel.org/show_bug.cgi?id=12106 > > I think we should consider this patch for 2.6.31. > > (Rafael, 13751 is on your "2.6.29 -> 2.6.30" regression list. > I actually think it's been around much longer than that, but > there seem to be many things that affect whether it manifests.)
Dropping from the list of post-2.6.29 regressions.
*** Bug 13412 has been marked as a duplicate of this bug. ***
74b5820808215f65b70b05a099d6d3c969b82689 ACPI: bind workqueues to CPU 0 to avoid SMI corruption shipped in v2.6.31-rc6 closed.
I meant to post earlier to confirm that 2.6.31-rc6 does fix it. Thanks!
Could this have come back in the later 2.6.32 series? I've been running fedora 12 recently and between their last two kernels my laptop hangs if I close the screen. The two kernels are; kernel-2.6.31.12-174.2.22.fc12.x86_64 kernel-2.6.32.9-67.fc12.x86_64 Before Christmas I was running vanilla 2.6.32.2 and it didn't have this problem. Nothing ends up in /var/log/messages this time. What is best to do? Try a vanilla kernel? Turn on the debugging output as described above?
right, please try the latest vanilla kernel and see if the problem still exists.