Bug 9772

Summary: 2.6.24-rc8 + patches: CPU hot removal while CPU is online leaves system in bad state
Product: ACPI Reporter: Daniel Arai (arai)
Component: Config-HotplugAssignee: Zhang Rui (rui.zhang)
Status: CLOSED CODE_FIX    
Severity: normal CC: achiang, acpi-bugzilla, akataria, astarikovskiy, bunk, trenn
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: 2.6.24-rc8 + patch for bug 2884 Subsystem:
Regression: --- Bisected commit-id:
Attachments: 01-patch-force-offline-processor
02-patch-create-sysfs-link
debug patch
Patch: hot remove processor in a kernel thread instead of work struct
01-flush-kacpi_notify_wq-before-removing-notify-handler
02-fix-a-deadlock-when-deivce-is-hot-removed
03-force-offline-cpu
04-processor-create-sysfs-link
fix a typo in patch 03-force-offline-cpu
Fixes some checkpatch.pl warnings introduced by patch in comment #23

Description Daniel Arai 2008-01-17 15:25:14 UTC
Latest working kernel version: Unknown
Earliest failing kernel version: Unknown
Distribution: sles10
Hardware Environment: x86_64
Software Environment:
Problem Description:
I've applied the patches attached to bug 2884 in kernel bugzilla so that I can work around the deadlock that occurs when you write to the eject node for an ACPI object.

The kernel doesn't properly handle the case of ejecting a CPU that is still online.  Doing so leaves the CPU online.  It still shows up in /proc/cpuinfo.  And the /sys/devices/system/cpuX node still exists.  But the kernel goes ahead and calls the ACPI eject method anyway, which means the hardware is free to be removed at that point.  And it cleans up the /sys nodes for the ACPI tree, so it's not possible to request another ejection.  This leads to system instability as things gradually hang as they attempt to interact with the CPU that has gone away.

Writing to the "eject" node for an online CPU should probably result in the ejection request being ignored because the CPU is still online.  The /sys/ tree node for the ACPI device should be left intact, so another ejection can be requested after the CPU has been taken offline.   The write to the eject sys node should probably fail with an error, but that's optional.  

Alternatively, the kernel could automatically offline an online CPU before ejecting it.  I think it's a matter of taste which behavior you prefer.

acpi_processor_remove looks like it returns -EINVAL if the CPU is online (see acpi_processor_handle_eject), but I don't think this return value is ever looked at, so the eject request isn't ever stopped.  At least as far as I can see.

Steps to reproduce:
echo 1 > /sys/devices/LNXSYSTM:00/device:00/ACPI0007:01/eject while CPU is still online.  Watch your shell hang.  Wait a few minutes and watch the whole system hang.
Comment 1 Alex Chiang 2008-01-23 13:08:29 UTC
I've been looking at this area too, as I'm starting to investigate CPU hotplug on ia64.

I'm not really sure what to make of the check in acpi_processor_handle_eject() that returns -EINVAL if the CPU is online. Given the fact that Documentation/cpu-hotplug.txt talks about receiving uevents to drive offlining, and the fact that the implementation of acpi_processor_hotplug_notify() really doesn't do much other than emit uevents, it seems that we have a chicken and egg problem here.

Ugh, it also seems that acpi_processor_handle_eject() calls arch_unregister_cpu(), which removes the /sys/devices/system/cpu/cpuX/online attribute, so a udev script can't do the right thing.

One possible solution would be for acpi_processor_hotplug_notify() to poke a finger into the base cpu driver and call cpu_down() when it receives an eject request, but that would negate the whole point of emitting uevents and letting userspace take care of it.

The other possibility is to move the call to arch_unregister_cpu into the cpu_down path. This still leaves open the question of what to do with acpi_unmap_lsapic(), but it might help untangle this mess.

Thoughts?
Comment 2 Zhang Rui 2008-01-23 23:58:43 UTC
Hi, Daniel,
In order to hot remove a CPU,
Please echo 0 > /sys/devices/system/cpu/cpuX/online first before echo 1 > /sys/bus/acpi/drivers/processor/ACPI0007:0X/eject.
Is this work for you?

Alex,
(In reply to comment #1)
> Ugh, it also seems that acpi_processor_handle_eject() calls
> arch_unregister_cpu(), which removes the /sys/devices/system/cpu/cpuX/online
> attribute, so a udev script can't do the right thing.
IMO, for cpu hot removal, it should work like this:
users prepare to hot removal a CPU, notification ACPI_NOTIFY_EJECT_REQUEST is generated. acpi_processor_hotplug_notify send a KOBJ_OFFLINE uevent.
Userspace should echo 0 > /sys/devices/system/.../cpuX/online to offline the processor, and then echo 1 > /sys/bus/acpi/.../ACPI0007:0x/eject to remove the processor from Linux device tree, including the acpi dev and sysdev.

> One possible solution would be for acpi_processor_hotplug_notify() to poke a
> finger into the base cpu driver and call cpu_down() when it receives an eject
> request, but that would negate the whole point of emitting uevents and
> letting userspace take care of it.
IMO, it works, but you're right that user space should take care of this.

> The other possibility is to move the call to arch_unregister_cpu into the
> cpu_down path. 
Then what about the acpi device node for processor? maybe we need a platform hook here?
Comment 3 Daniel Arai 2008-01-24 09:31:47 UTC
There's a couple of problems.

/sys/devices/system/cpu/cpuX/online doesn't always correspond to /sys/bus/acpi/drivers/processor/ACPI0007:0X.  Especially if you remove and then re-add CPUS, you can get a situation where CPU3 corresponds to ACPI0007:08, or something similar.  I sent mail to lkml and linux-acpi describing the issue, but nobody responded.  So you can't always offline the correct CPU before ejecting it.

In my testing, I'm always careful to offline the CPU before ejecting it.

It just seems really bad to make it this easy to accidentally hang the whole system.  Ejecting a CPU really needs to fail if the CPU is online, or else it needs to automatically offline the CPU.  Failing if the CPU is online seems to make the most sense.
Comment 4 Alex Chiang 2008-01-24 12:16:37 UTC
Hrm, this is odd. I don't seem to have an eject file in sysfs:

[root@canola ACPI0007:02]# pwd
/sys/bus/acpi/devices/LNXSYSTM:00/device:00/ACPI0007:02

[root@canola ACPI0007:02]# ls -F
bus@  driver@  hid  modalias  path  subsystem@  uevent

[root@canola ACPI0007:02]# uname -a
Linux canola.achiang 2.6.23-0.224.rc9.git6 #1 SMP Wed Oct 17 13:32:26 EDT 2007 ia64 ia64 ia64 GNU/Linux

I do have both cpu hotplug and acpi cpu hotplug kconfig'ed in...
Comment 5 Daniel Arai 2008-01-24 13:44:56 UTC
Does your firmware have an _EJ0 method for the processor objects?  You should be able to tell by dumping the DSDT or whatever.
Comment 6 Zhang Rui 2008-01-24 16:57:53 UTC
(In reply to comment #5)
> Does your firmware have an _EJ0 method for the processor objects?  You should
> be able to tell by dumping the DSDT or whatever.

Yes. the "eject" attribute is created for the devices that have a _EJ0 methods.
And IMO, for a processor that supports hot-plug,
the _EJ0 method is mandatory, isn't it?

Alex.
Please attach the acpidump output.
the latest pmtools can be downloaded here:
http://www.kernel.org/pub/linux/kernel/people/lenb/acpi/utils/
Comment 7 Zhang Rui 2008-01-24 17:02:58 UTC
(In reply to comment #3)
> There's a couple of problems.
So it really works for you, right?
ie, "echo 0 > online" first and "echo 1 > eject" then can hot remove a processor successfully.

> /sys/devices/system/cpu/cpuX/online doesn't always correspond to
> /sys/bus/acpi/drivers/processor/ACPI0007:0X.  Especially if you remove and
> then
> re-add CPUS, you can get a situation where CPU3 corresponds to ACPI0007:08,
> or
> something similar.
Yes. Hmm, can we create a sysfs link here, from cpu sysdev to acpi device node?

> Ejecting a CPU really needs to fail if the CPU is online, or else it
> needs to automatically offline the CPU.  Failing if the CPU is online seems >
> to make the most sense.

Yes. you're right. I'll look into this.
Comment 8 Daniel Arai 2008-01-24 17:15:21 UTC
Yes, it works if I offline the CPU first, then echo 1 into the eject node.

I think what you need is a symlink from the ACPI device node to the CPU sysdev.  When you get the eject request, you're given the ACPI device node, and can't figure out what the CPU number is (unless they just happen to match.)
Comment 9 Zhang Rui 2008-01-27 19:08:19 UTC
Created attachment 14618 [details]
01-patch-force-offline-processor

this patch forces offline the cpu if it's still online when poking the "eject" attribute.
Daniel, if the patch works, you don't need to poke the "online" attribute before echo 1 > eject. would you please give it a try. :)
Comment 10 Zhang Rui 2008-01-27 19:19:15 UTC
Created attachment 14619 [details]
02-patch-create-sysfs-link

this patch creates a sysfs link from the acpi device node to sysdev node.
Comment 11 Daniel Arai 2008-01-29 10:27:04 UTC
Both patches seem to work as advertised.  Thanks!

I still need to do some stress testing, but things now look a lot better than they did.

I'm not sure what the protocol here is.  Do I mark the bug as fixed with CODE_FIX, or do I not do that until the patch is actually in the kernel?
Comment 12 Daniel Arai 2008-01-29 16:23:51 UTC
I take it back.  The force-offline-processor patch doesn't seem to be working quite right yet.

The processor goes offline, but the kernel never seems to invoke the _EJ0 method for the CPU object, and the ACPI0007:01 sysnode tree isn't cleaned up.  Invoking the eject method again causes the command to complete, but my bash shell locks up.  I'm able to start a reboot via control-alt-delete, but I get messages on the console from init that say that various pids seem to hang, and the reboot process hangs.  I have to power cycle the machine.

With the other patch, I can at least fix up my /sbin/hotplug script so that it always offlines the correct CPU before issuing the eject though, which is nice!
Comment 13 Zhang Rui 2008-01-29 23:27:48 UTC
(In reply to comment #12)
> The processor goes offline, but the kernel never seems to invoke the _EJ0
> method for the CPU object, and the ACPI0007:01 sysnode tree isn't cleaned up. 

Wreid, can you attach the dmesg output and "tree ACPI0007:01" after eject the processor for the first time?
Comment 14 Daniel Arai 2008-01-30 16:37:52 UTC
So the behavior is strange.

If I power on with 2 CPUs, then echo 1 > .../ACPI0007:01/eject with CPU 1 online, things work.

If I power on with 4 CPUs, then I can remove cpus 3 and 2 via echo 1 > ACPI0007:0X/eject with the CPUs online, and things work.  But after doing this, I try to eject CPU 1 with it online via echo 1 > .../ACPI0007:01/eject, and nothing happens, aside from CPU 1 going offline.

Here's what I see in dmesg:

CPU 3 is now offline
CPU 2 is now offline
CPU 1 is now offline
SMP alternatives: switching to UP code



My SLES 10 OS doesn't have a "tree" command in root's PATH.  I can try to find the source for tree and compile it if that's really necessary.  Here's what ls -l ACPI0007:01 shows:

total 0
lrwxrwxrwx 1 root root    0 Jan 30 09:06 bus -> ../../../../bus/acpi
--w------- 1 root root    0 Jan 30 09:07 eject
-r--r--r-- 1 root root 4096 Jan 30 09:06 hid
-r--r--r-- 1 root root 4096 Jan 30 09:06 modalias
-r--r--r-- 1 root root 4096 Jan 30 09:06 path
drwxr-xr-x 2 root root    0 Jan 30 09:06 power
lrwxrwxrwx 1 root root    0 Jan 30 09:06 subsystem -> ../../../../bus/acpi
lrwxrwxrwx 1 root root    0 Jan 30 09:06 sysdev -> ../../../../devices/system/cpu/cpu1
-rw-r--r-- 1 root root 4096 Jan 30 09:06 uevent


Onlining CPU 1 once I'm in this state doesn't work.
Comment 15 Zhang Rui 2008-02-03 22:53:41 UTC
(In reply to comment #14)
> I try to eject CPU 1 with it online via echo 1 > .../ACPI0007:01/eject, and
> nothing happens, aside from CPU 1 going offline.
> Here's what I see in dmesg:
> CPU 3 is now offline
> CPU 2 is now offline
> CPU 1 is now offline
> SMP alternatives: switching to UP code
what can you see if you disable CPU1 with only two CPUs enabled?

> My SLES 10 OS doesn't have a "tree" command in root's PATH.  I can try to
> find
> the source for tree and compile it if that's really necessary.  Here's what
> ls
no, you don't need to do this. the following info is enough. :)

> -l ACPI0007:01 shows:
> total 0
> lrwxrwxrwx 1 root root    0 Jan 30 09:06 bus -> ../../../../bus/acpi
> --w------- 1 root root    0 Jan 30 09:07 eject
> -r--r--r-- 1 root root 4096 Jan 30 09:06 hid
> -r--r--r-- 1 root root 4096 Jan 30 09:06 modalias
> -r--r--r-- 1 root root 4096 Jan 30 09:06 path
> drwxr-xr-x 2 root root    0 Jan 30 09:06 power
> lrwxrwxrwx 1 root root    0 Jan 30 09:06 subsystem -> ../../../../bus/acpi
> lrwxrwxrwx 1 root root    0 Jan 30 09:06 sysdev ->
> ../../../../devices/system/cpu/cpu1
> -rw-r--r-- 1 root root 4096 Jan 30 09:06 uevent
device_unregister() is not executed correctly?
please try the debug patch I attached, so that we can find out why these sys attributes still exist.
Please echo 0x04 > /sys/module/acpi/parameters/debug_layer and
echo 0x8800001f > /sys/module/acpi/parameters/debug_level
before eject the cpu and attach the dmesg output
Comment 16 Zhang Rui 2008-02-03 22:54:16 UTC
Created attachment 14701 [details]
debug patch
Comment 17 Daniel Arai 2008-02-04 16:44:45 UTC
I was able to get one hang to happen, but I applied your debug patch, turned on ACPI debugging in my kernel config, and now I can't get the failure to happen at all.  

Dmesg output after the failed ejection is a bit insightful, though this was from before I had debugging turned on or the additional information:

events/0      R  running task        0     7      2
events/1      D ffffffff805ff778     0     8      2
 ffff81000144fbe0 0000000000000046 ffff81000fa26800 ffff81000ffd6040
 ffff81000ffd6040 ffffffff8022f327 ffff81000fa26800 ffff8100015317c0
 000000000000000f 0000000000000007 ffff81000fa26800 0000000000000001
Call Trace:
 [<ffffffff8022f327>] update_curr_load+0x87/0xa0
 [<ffffffff80231a46>] try_to_wake_up+0x66/0x370
 [<ffffffff80447572>] wait_for_completion+0xa2/0xf0
 [<ffffffff80231d50>] default_wake_function+0x0/0x10
 [<ffffffff8024e0f8>] kthread_stop+0x58/0x80
 [<ffffffff8024a4c9>] cleanup_workqueue_thread+0x19/0x30
 [<ffffffff8024a57f>] workqueue_cpu_callback+0x9f/0x150
 [<ffffffff8044bdc5>] notifier_call_chain+0x45/0x90
 [<ffffffff8025773f>] _cpu_down+0x1df/0x2c0
 [<ffffffff80257854>] cpu_down+0x34/0x60
 [<ffffffff80357f25>] acpi_processor_remove+0x63/0x140
 [<ffffffff8035282d>] acpi_device_remove+0x53/0x79
 [<ffffffff80390452>] __device_release_driver+0x82/0xc0
 [<ffffffff80390978>] device_release_driver+0x38/0x60
 [<ffffffff8035291c>] acpi_bus_remove+0x39/0x72
 [<ffffffff803529cf>] acpi_bus_trim+0x7a/0x106
 [<ffffffff80351433>] acpi_bus_data_handler+0x0/0x1
 [<ffffffff80352a91>] acpi_bus_hot_remove_device+0x36/0xc1
 [<ffffffff80338957>] acpi_os_execute_notify+0x0/0x2c
 [<ffffffff8033897a>] acpi_os_execute_notify+0x23/0x2c
 [<ffffffff8024a22f>] run_workqueue+0x7f/0x110
 [<ffffffff8024ad60>] worker_thread+0x0/0x130
 [<ffffffff8024ae25>] worker_thread+0xc5/0x130
 [<ffffffff8024e5b0>] autoremove_wake_function+0x0/0x30
 [<ffffffff8024ad60>] worker_thread+0x0/0x130
 [<ffffffff8024ad60>] worker_thread+0x0/0x130
 [<ffffffff8024e1eb>] kthread+0x4b/0x80
 [<ffffffff8020cd38>] child_rip+0xa/0x12
 [<ffffffff8024e1a0>] kthread+0x0/0x80
 [<ffffffff8020cd2e>] child_rip+0x0/0x12

Maybe I just didn't wait long enough, and it would have eventually completed?

I'll update if I can get this problem to reproduce.
Comment 18 Daniel Arai 2008-02-05 10:49:03 UTC
Oh, one other thing.

acpi_processor_remove needs

processor_device_array[pr->id] = NULL; 
after
processors[pr->id] = NULL;

or else acpi_processor_start goes bad in the buggy BIOS check if you remove a CPU then add one back in.
Comment 19 Daniel Arai 2008-02-06 15:27:40 UTC
I was able to get the hang to occur.  It's intermittent, but seems to happen more when I've got 2 CPUs than when I have more, and it definitely doesn't happen all the time.  Maybe it has to do with one of the threads running on CPU 1, and not migrating to CPU 0 when it gets offlined, or something?

Here's the dmesg output.  It doesn't look super useful:

IA-32 Microcode Update Driver: v1.14a <tigran@aivazian.fsnet.co.uk>
acpi_bus_remove
acpi_device_remove
acpi_processor_remove
acpi_processor_handle_eject
CPU 3 is now offline
acpi_processor_remove_fs
acpi_device_unregister
acpi_bus_remove
acpi_device_remove
acpi_processor_remove
acpi_processor_handle_eject
CPU 2 is now offline
acpi_processor_remove_fs
acpi_device_unregister
acpi_bus_remove
acpi_device_remove
acpi_processor_remove
acpi_processor_handle_eject
CPU 1 is now offline
SMP alternatives: switching to UP code


The kernel thread is stuck waiting with the same stack I reported up in comment #17.
Comment 20 Zhang Rui 2008-02-24 19:39:07 UTC
Created attachment 14969 [details]
Patch: hot remove processor in a kernel thread instead of work struct

please try this patch on top of the 01-patch-force-offline-proceesor and see if it helps.
but the problem is that Len has dropped this patch set as Alex and Thomas are working on something "simpler"...
So we'd better add them in the cc list, :)
Comment 21 Daniel Arai 2008-03-04 15:16:32 UTC
The latest patch looks pretty good to me.  I was able to do about 20 hot removes/hot adds before I got bored.

I can help out with testing the simpler solution when patches are available.
Comment 22 Zhang Rui 2008-04-01 20:04:43 UTC
Created attachment 15558 [details]
01-flush-kacpi_notify_wq-before-removing-notify-handler
Comment 23 Zhang Rui 2008-04-01 20:05:22 UTC
Created attachment 15559 [details]
02-fix-a-deadlock-when-deivce-is-hot-removed
Comment 24 Zhang Rui 2008-04-01 20:05:47 UTC
Created attachment 15560 [details]
03-force-offline-cpu
Comment 25 Zhang Rui 2008-04-01 20:06:10 UTC
Created attachment 15561 [details]
04-processor-create-sysfs-link
Comment 26 Zhang Rui 2008-04-01 20:07:04 UTC
Please try this patch set on a clean 2.6.25 kernel. :)
Comment 27 Alok Kataria 2008-04-08 11:57:52 UTC
Tested these new set of patches from Zhang on a 64bit kernel on top of 2.6.25-rc8.

For 32bit kernels tried these on 2.6.25-rc8-x86 tree as the 32bit ACPI boot code is fixed in the X86 tree with the smpboot integration patches submitted by Glauber. 

I was able to eject CPUs from the system and it worked as expected.
Please let me know if you need any more help on this to puch them upstream.

Thanks,
Alok
Comment 28 Zhang Rui 2008-04-08 18:33:12 UTC
Created attachment 15682 [details]
fix a typo in patch 03-force-offline-cpu
Comment 29 Zhang Rui 2008-04-08 18:39:47 UTC
IMO, patch in comment #22 #23 and #28 are ready for in 2.6.26. :)

For the patch in comment #25, we don't need this patch if the patch 03-force-offline-cpu is applied because we don't need this sysfs link to poke the "online" attribute under cpu sysdev node before poking the "eject" file.
Alok, what do you think?
Comment 30 Alok Kataria 2008-04-08 20:37:15 UTC
(In reply to comment #29)
> IMO, patch in comment #22 #23 and #28 are ready for in 2.6.26. :)
> 
> For the patch in comment #25, we don't need this patch if the patch
> 03-force-offline-cpu is applied because we don't need this sysfs link to poke
> the "online" attribute under cpu sysdev node before poking the "eject" file.
> Alok, what do you think?
> 

Hi Zhang,

IMO it would useful when we are re-adding the cpu to the system. In that case we can have scripts in userspace to bring the cpu online as soon as that cpu is added to the system. 
Now we can have a similar feature in kernel which implicitly brings the cpu online as soon as its added, but then I think we loose the flexibility for the user to decide if he wants the cpu online as soon as its added to the system.

So IMHO lets keep that patch, unless anybody disagrees.
 
Comment 31 Len Brown 2008-04-09 19:13:17 UTC
I'm concerned that if the access to the eject file completes
before the actual eject, that it may be possible for user-space
to get confused.  While this may be a little academic, it sounds
like the kind of thing where there will be a race.

Simpler for the user if the eject is guaranteed to be complete
when the access to the file returns.

So instead of deferring the actual eject so the return can complete,
how about if we execute the eject and defer removing the file until
the user access is complete?  That way when we return to the user,
we are guaranteed that the eject is done.
Comment 32 Zhang Rui 2008-04-10 19:11:54 UTC
Hi, Len,
I made a double check and found that we can't defer removing the file  until the user access is complete.
As ejecting a device requires to remove the ACPI device node from Linux device tree, the "eject" file must be removed before calling device_unregister, because it is an attribute of the device node.
So I still think the proposed patches works. and they can be shipped in acpi-test for now.
Comment 33 Len Brown 2008-04-28 23:38:38 UTC
patch in comment #22 applied

patch in comment #23, 28, 25 applied to acpi-test

re: patch in comment #23:
dos2unix: converting file 9772-23.patch to UNIX format ...
~/src/acpi
scripts/checkpatch.pl
WARNING: Use #include <linux/signal.h> instead of <asm/signal.h>
#26: FILE: drivers/acpi/scan.c:9:
+#include <asm/signal.h>

ERROR: Don't use kernel_thread(): see Documentation/feature-removal-schedule.txt
#117: FILE: drivers/acpi/scan.c:174:
+       ret = kernel_thread(acpi_bus_hot_remove_device,
Comment 34 Len Brown 2008-04-30 20:21:43 UTC
patch in comment #22 shipped in linux-2.6.25-git16
Comment 35 Alok Kataria 2008-05-27 15:29:04 UTC
Created attachment 16300 [details]
Fixes some checkpatch.pl warnings introduced by patch in comment #23
Comment 36 Alok Kataria 2008-05-27 15:31:51 UTC
(In reply to comment #33)
> patch in comment #22 applied
> 
> patch in comment #23, 28, 25 applied to acpi-test
> 
> re: patch in comment #23:
> dos2unix: converting file 9772-23.patch to UNIX format ...
> ~/src/acpi
> scripts/checkpatch.pl
> WARNING: Use #include <linux/signal.h> instead of <asm/signal.h>
> #26: FILE: drivers/acpi/scan.c:9:
> +#include <asm/signal.h>
> 
> ERROR: Don't use kernel_thread(): see
> Documentation/feature-removal-schedule.txt
> #117: FILE: drivers/acpi/scan.c:174:
> +       ret = kernel_thread(acpi_bus_hot_remove_device,
> 

Hi Len/Zhang,

While going through the mainline tree i noticed that these patches 23, 28, 25
are still not in the mainline. Was wondering if this checkpatch warning the
reason for holding the patches back, or is there some other issue with the patches ?
I have cooked up a patch to fix these warnings with checkpatch, in comment #35. Please have a look. If there is any other issue with it, please do let
me know.

Thanks,
Alok
Comment 37 Zhang Rui 2008-06-02 18:13:33 UTC
len, i think they can be shipped to acpi-test first.
Comment 38 Len Brown 2008-06-13 10:06:13 UTC
> patch in comment #23, 28, 25 applied to acpi-test

These patches were checked into acpi-test on April 28th.
They were not low risk enough to qualify for a fast track into
the 2.6.26 merge window, which closed on may 3rd.
Thus they have always been targeted for 2.6.27.

they have been present in acpi-test, linux-next
and every mm release since then:
2.6.26-rc2-mm1
2.6.26-rc5-mm1
2.6.26-rc5-mm2
2.6.26-rc5-mm3

you may have pulled acpi-test at a time when it was being
re-based and the patches above (and others) where thus
temporarily not present.

Thanks for the patch in comment #35.
I've applied it to this branch and we will propose it
for the 2.6.27 merge window.
Comment 39 Adrian Bunk 2008-07-16 15:41:45 UTC
fixed in Linus' tree