Recently I've started noticing that kernel does some very unpleasant and crucial things to my HDD when I'm powering off my Linux. While Windows simply powers off the HDD (together) with the rest of my PC, Linux instead makes HDD spindles stop, then my HDD tries to spin up again, then the power is finally retracted *when* my HDD has just started rotating again. The Linux poweroff sequence produces terribly unpleasant sounds. It's killing me. I will now try to alter the poweroff sequence calling: # sync # /sbin/hdparm -Y /dev/sda # halt -n -p -w (don't sync, we've done it earlier) instead of # halt -p -w I'm not sure Linux won't try to interfere with my way of powering off my HDD :(
This trick didn't help. Right after putting HDD to sleep, the kernel then spins it up again doing the rest of usual unmerciful stuff.
Weird, we haven't had that problem for a very long time now. Can you please attach the output of "hdparm -I /dev/sda" and "dmidecode"? Also, if you do "shutdown -h -H now", does the disk stay spinned down?
Created attachment 37742 [details] hdparm, dmidecode (In reply to comment #2) > Weird, we haven't had that problem for a very long time now. Can you please > attach the output of "hdparm -I /dev/sda" and "dmidecode"? Also, if you do > "shutdown -h -H now", does the disk stay spinned down? Yes, shutdown -h -H now works properly and the disk stays spinned down.
(cc'ing Len Brown, hi!) Which means that it's the system BIOS which is issuing the commands, probably another standby. Some drives spin up when it receives a standby while spun down, to spin down again. Before passing control to acpi poweroff, the kernel spins down the drive (and it should), the acpi poweroff receives control and issues an extra standby causing the double spin down. It's possible that the BIOS is behaving differently depending on which operating system is in use. Maybe it assumes anything other than Windows is dos and thus it's BIOS's job to spin down the drives. Can you please check whether this is a regression? Maybe something has changed on ACPI side, but ultimately, this is BIOS doing something it isn't supposed to do. Thanks.
Which kernel version would you suggest to test?
Oops, sorry about the double posting. bugzilla was being laggy. Anyways, as for the kernel version to try, I have no idea. That part of libata hasn't changed for years now. But why did you report that it's possibly a post 2.6.33 regression? If there are reasons to believe the problem appeared around 2.6.33, testing kernel before that would be a good idea. Thanks.
Two observations: 1) kernel 2.6.33.5 shuts down my HDD properly. 2) _any_ kernel shuts down my HDD properly when I put my PC to sleep mode (i.e. suspend to RAM).
I won't do regression testing because I don't want to kill my HDD :) So this bug clearly looks like a regression. I'm now thinking what to do as I don't like the current state of affairs.
Yeah, it is a regression. I don't think it is from libata tho. Nothing on that front has changed for quite some time now. Drives don't usually die that easily, so it would be nice if you could bisect it. As Len doesn't seem responsive, let's try Thomas. Thomas? Any ideas?
Quick summary: Disk is spin down, up and (hard?) off again when the machine is powered off. This behavior is not seen with S3. shutdown -h -H now works around the problem. 2.6.33 did not have the problem. Which kernel are you using? For which one are you sure that they are affected?
(In reply to comment #11) > Quick summary: Disk is spin down, up and (hard?) off again when the machine > is > powered off. This behavior is not seen with S3. > shutdown -h -H now works around the problem. > 2.6.33 did not have the problem. > Which kernel are you using? For which one are you sure that they are > affected? Right. Between spin down and up there's such a small interval (I think it's less than 0.5sec) HDD doesn't fully stop but I hear it stopping and spinning up almost simultaneously. I'm using vanilla kernels, (at least) both 2.6.35.x and 2.6.36.x have this problem.
(In reply to comment #10) > Yeah, it is a regression. I don't think it is from libata tho. Nothing on > that front has changed for quite some time now. Drives don't usually die > that > easily, so it would be nice if you could bisect it. As Len doesn't seem > responsive, let's try Thomas. Thomas? Any ideas? How can I disable libata code responsible for powering down HDDs on shut down (I'm running manually compiled vanilla kernel, so patching is very easy for me)? I've examined kernel code to no avail, my C skill are rather superficial. With my "super" BIOS ACPI shutdown call is sufficient. Meanwhile I'm using GRUB's `halt` command after reboot for powering off this PC.
You can disable it by... echo 0 > /sys/class/scsi_disk/h:c:i:l/manage_start_stop
How can I help solve this bug without bisecting the kernel? Is it possible to disassemble ACPI code in my motherboard BIOS to see if the Linux ACPI codepath is indeed different from the Windows codepath?
ASRock declines to even notice this bug report, so I'm closing it as "INVALID". Besides I've disposed of this motherboard.
It's kind of a shocker, but I now own a completely different motherboard made by ASUS (based on UEFI) and the bug has now returned. I'm desperate what to do since Windows doesn't have this bug. Any help will be highly appreciated.
That's weird. Which distro are you using? Maybe shutdown(8) is being nosy and spinning off the harddrives before telling the kernel to power off. That behavior should be long gone now tho.
Created attachment 48892 [details] 2.6.38-rc5 configuration I'm using Fedora 14 i686 with all updates installed. I will try to capture on video the shutdown process.
I see, so nothing wonky. I'm running out of ideas. Thomas, could this have been caused by ACPI changes in the kernel?
Well, in theory, it could, but then it would be a bit more reproducible (eg. on more than one machine). Artem, would it be possible to test an openSUSE live CD on your machine?
(In reply to comment #21) > Well, in theory, it could, but then it would be a bit more reproducible > (eg. on more than one machine). > > Artem, would it be possible to test an openSUSE live CD on your machine? I doubt OpenSuse ISO can make any difference, but I will try if you give me the download link. Here's dmesg -n8 log: sd 1:0:0:0 [sda] Synchronizing SCSI cache sd 1:0:0:0 [sda] Stopping disk --- (HDD spins down and stops) e100 0000:08:00.0: PME# enabled pci 0000:07:00.0: wake-up capability enabled by ACPI xhci_hcd 0000:07:00.0: PCI INT A disabled xhci_hcd 0000:03:00.0: PCI INT A disabled ehci_hcd 0000:00:1d.0: PCI INT A disabled ehci_hcd 0000:00:1a.0: PCI INT A disabled e1000e 0000:00:19.0: PCI INT A disabled e1000e 0000:00:19.0: PME# enabled e1000e 0000:00:19.0: wake-up capability enabled by ACPI ACPI: preparing to enter sleep state S5 Disabling non-boot CPUs ... Broke affinity for irq 1 Broke affinity for irq 40 Broke affinity for irq 19 Power down. acpi_power_off called --- (HDD spins up, down and stops for the second time) Here's the captured video (use mplayer/VLC to watch it): http://www.mediafire.com/?5d08nwyxbi6q0ap
BTW, this bug never occurs when I put the computer into suspend2ram mode.
Also, there's one really weird thing worth mentioning: if I shut down the PC immediately after I've turned it on, there's no such a problem under any kernel version. I'm removing "regression" keyword from this bug report as I'm totally unsure if it's really a regression.
I tried running vanilla 2.6.32 (without "stable"/"longterm" patches as they may contain some ACPI fixes which make 2.6.32 bheaviour similar to the later kernels) but it doesn't quite work: every second application crashes with "Floating point exception" - I'm seeing this problem for the first time in my life. I've no idea what to do :( BTW, how can I insert a delay (say 5 seconds) between powering off of SATA disks and the final shutdown system call? Another question: I suppose Linux kernel uses ACPI call for powering off the system. Is it possible to use some old-fashioned APM call?
The floating point exception problem could be the floating point context switching bug we had w/ preemption. Turn off preemption and see whether it goes away. The power off function is kernel/sys.c::kernel_power_off(). Inserting msleep() before calling machine_power_off() should do the trick. Yeah, I think you can still do that if you compile in APM support. Take a look at Documentation/kernel-parameters.txt. Good luck.
It turns out APM is not an option for me because it doesn't let me use three out of four cores of my CPU. Also APM and ACPI cannot coexist. Now about adding a delay. I added 'ssleep(15);' before 'machine_power_off();' in kernel/sys.c and here's what I've found out: the kernel correctly powers down my HDD in 75% cases, in other 25% cases I get my HDD spindle respinned back *right after* it was put to halt (so ACPI x86 power off call has nothing to do with this issue). I mean this glitch happens even before ssleep(15); machine_power_off(); That probably means that SATA HDDs power down signal doesn't always work correctly here. From hdparm manual I see there are at least two (maybe more) ways of shutting down the HDD: -y Put drive in standby mode -Y Put drive to sleep Is it possible the change/tune libata power down call?
Hmmm.. I don't think changing that behavior is a good idea. It has been like that forever and AFAIK it's also what other operating systems are using. We need to find out what exactly going on with your setup. Given the very low frequency about related problems (you're basically the only one reporting this problem since years), I don't think the problem is wide spread. Maybe we should print out whether and which commands are being issued to the disk after standbynow.
Created attachment 52282 [details] verbose-on-standbynow.patch Can you please apply this patch and report the output during shutdown? You might want to insert delays so that you can capture the output before power goes out or set up serial or netconsole. Thanks.
OK, here's what what I see with this patch (right off the bat I once again had this issue): [sda] Synchronizing SCSI cache [sda] Stopping disk ata1.00: ata_qc_issue: e0/00:00:00:00:00/00:00:00:00:00/a0 ata1.00: ata_qc_issue: 60/08:00:99:c6:cc/00:00:03:00:00/40
The 0xE0 is STANDBYNOW1 and 0x60 is FPDMA_READ, so someone is issuing read on the disk after the disk is shutdown. Unlike suspend path, the poweroff path doesn't plug the queue explicitly. It assumes that the system is put in idle state. So, the question is, who's issuing that READ? Can you please print out comm->task and pid? Thanks.
(In reply to comment #31) > So, the question is, who's issuing that READ? Can you please print out > comm->task and pid? That should have been current->comm and pid.
(In reply to comment #31) > The 0xE0 is STANDBYNOW1 and 0x60 is FPDMA_READ, so someone is issuing read on > the disk after the disk is shutdown. Unlike suspend path, the poweroff path > doesn't plug the queue explicitly. It assumes that the system is put in idle > state. > > So, the question is, who's issuing that READ? Can you please print out > comm->task and pid? > > Thanks. I'd gladly do that if you told me how :)
Created attachment 53522 [details] Screenshot Here's a screenshot that I captured when I once again faced this issue. It looks like the "swapper" process tries to flush data when the system is being shut down. P.S. I modified your patch and added printk("Kernel Thread::Current Process is \"%s \" (pid: %i) \n", current->comm,current->pid); after ata_dev_printk(); so now it looks this way: --- libata-core.c.orig 2011-03-15 06:20:32.000000000 +0500 +++ libata-core.c 2011-03-31 05:05:25.023328722 +0600 @@ -5028,6 +5028,18 @@ return; } + if (qc->dev->flags & ATA_DFLAG_PRINT_CMDS) { + struct ata_taskfile *cmd = &qc->tf; + ata_dev_printk(qc->dev, KERN_INFO, + "ata_qc_issue: %02x/%02x:%02x:%02x:%02x:%02x/%02x:%02x:%02x:%02x:%02x/%02x\n", + cmd->command, cmd->feature, cmd->nsect, + cmd->lbal, cmd->lbam, cmd->lbah, + cmd->hob_feature, cmd->hob_nsect, + cmd->hob_lbal, cmd->hob_lbam, cmd->hob_lbah, + cmd->device); + printk("Kernel Thread::Current Process is \"%s \" (pid: %i) \n", current->comm,current->pid); + } + ap->ops->qc_prep(qc); qc->err_mask |= ap->ops->qc_issue(qc);
This whole situation is very weird, since I have SWAP support disabled altogether in my kernel configuration: [root@localhost linux-2.6.38]# grep -i swap .config # CONFIG_SWAP is not set CONFIG_X86_BSWAP=y
(In reply to comment #35) > This whole situation is very weird, since I have SWAP support disabled > altogether in my kernel configuration: > "swapper" is just the historical name for the init thread, process #1. Add a call to dump_stack(); to find out why swapper got here.
err, the thread before the init thread, that is. I guess it has PID 0.
Created attachment 53532 [details] Screenshot with dump_stack(); (In reply to comment #36) > (In reply to comment #35) > > This whole situation is very weird, since I have SWAP support disabled > > altogether in my kernel configuration: > > > > "swapper" is just the historical name for the init thread, process #1. > > Add a call to > > dump_stack(); > > to find out why swapper got here.
It seems like I traced ata_qc_issue, not this process. I have no idea how I have to patch the kernel to see the swapper thread stack.
I guess in libata layer we can block all processes from waking/writing to the HDD after stopping the disk. Tejun Heo, can you create a patch for that? I've figured out it's the "swapper" process which breaks the perfect picture here, but I have no idea where to go and how else I can help in solving this bug report. I suspect this problem is in software, not in hardware.
Sorry about the delay. I've been traveling. We still don't know what's going on and I don't think we should just plug the problem by making libata reject the commands without knowing where they're coming from. To know where they're coming from, you should record who's issuing the requests rather than who ends up executing those requests (ata_qc_issue() is in the execution path). Another thing is that this is the only report on this problem, so this definitely is not a wide spread problem. I'll prep a patch to record and print out the issuer. Thank you.
Created attachment 55072 [details] verbose-on-spindown.patch Can you please apply the attached patch and post the resulting kernel log after the spindown/up event? It should point us to who's issuing the commands causing re-spinups. Thanks.
(In reply to comment #42) > > Can you please apply the attached patch and post the resulting kernel log > after > the spindown/up event? It should point us to who's issuing the commands > causing re-spinups. I will post the results as soon as I get them.
Created attachment 55302 [details] verbose-on-spindown.patch: result Here we go, I've caught it two times already.
That's pid 1, init, issuing read after the disk is put to sleep. Something is completely coo coo with your setup. Can you please try different distro (even f15b) and see whether the problem is still there? I don't think the kernel should be working around this. init shouldn't be accessing the filesystem after telling the kernel to shutdown. Thanks.
I don't think this problem can be reproduced with F15, because F14 and F15 have different init applications, namely upstart on F14 and systemd on F15. I'm kinda perplexed as to why I'm the only person who experiences this problem. Either other people don't notice it (which seems totally likely because usually people don't care much about sounds their HDD produces), or it happens under some special circumstances but then again I'm reproducing it easily. I have subscribed Linus who also uses Fedora 14, but it seems like he's not interested in this particular userspace bug. I will bug a report on RH's bugzilla (https://bugzilla.redhat.com/show_bug.cgi?id=699351), let's see what they'll say. (In reply to comment #45) > That's pid 1, init, issuing read after the disk is put to sleep. Something > is > completely coo coo with your setup. Can you please try different distro > (even > f15b) and see whether the problem is still there? I don't think the kernel > should be working around this. init shouldn't be accessing the filesystem > after telling the kernel to shutdown. Tejun Heo, You've already said that you don't endorse hacking libata to stop user processes from accessing storage after disk stop on shutdown/suspend, but can you create such a patch specifically for me?
In the meantime I've added killall -SIGSTOP init before the final `halt` invocation. Let's see if it helps. Theoretically it should solve my problem.
I'm sorry for flooding, I actually meant: kill -SIGSTOP 1 I also mark this bug as resolved/invalid, because actually it's not the kernel bug at all - thanks Tejun Heo!
I'm fairly sure we would be seeing many more reports if the problem were widespread. We had similar issues on shutdown previously and enough people noticed the problem at the time. It's not only the sound, also the emergency unload count is incremented depending on the drive when it happens. As for kernel patch, it would probably be easier to add "echo 1 > /sys/block/sdX/device/delete" during shutdown somewhere (this will also spin down the drive). Good luck.
Your solution will not likely work - as soon as I delete a HDD, all filesystems linked to it will become unavailable so I won't be able to run `halt`. :) If there was an equivalent to the `halt` command e.g. by echoing something to /sys or /proc that would be doable.
Artem, there was a similar report which Kay handled and the culprit turned out to be /sbin/hotplug. If CONFIG_UEVENT_HELPER_PATH is configured or /proc/sys/kernel/hotplug is set, the kernel will try to execute the binary on every uevent. During system shutdown, there are a lot of those events and the kernel ends up trying to read and execute the binary generating IO requests on the root FS. Can you please check whether hotplug helper is enabled? Thanks.
Here it is: $zcat /proc/config.gz | grep CONFIG_UEVENT_HELPER_PATH CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug" $cat /proc/sys/kernel/hotplug [empty]
It's fixed by: http://git.kernel.org/?p=linux/kernel/git/gregkh/driver-core-2.6.git;a=commitdiff;h=b50fa7c8077c625919b1e0a75fc37b825f024518 Indeed it was a kernel issue ;)