Bug 35352
| Summary: | WARNING: at arch/x86/mm/highmem_32.c:81 __kunmap_atomic, when doing swapoff | ||
|---|---|---|---|
| Product: | Memory Management | Reporter: | Witold Baryluk (witold.baryluk+kernel) |
| Component: | Other | Assignee: | Hugh Dickins (hughd) |
| Status: | CLOSED CODE_FIX | ||
| Severity: | normal | CC: | florian, hughd, ngupta |
| Priority: | P1 | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Kernel Version: | 2.6.39-rc7-00897-ga2b9c1f | Subsystem: | |
| Regression: | Yes | Bisected commit-id: | |
| Attachments: |
kernel config
swapoff BUG |
||
|
Description
Witold Baryluk
2011-05-19 01:24:05 UTC
I don't see any zram related function in any of the bracktraces, so cannot see how this might be related to zram. Do you see these warnings even when zram is not used -- i.e. you swapoff/swapon disk swaps only?
> it will quickly BUG somewhere, and after that
freeze system).
Can you post backtrace from the BUG message?
In fact swapoff is not needed probably.
I disabled almost everything on my system, this given me lots of memory, added swapon -p 10 /dev/zram0 , then loggin into gnome, started opera webbrowser, and make -j2 on kernel. After a while, i got
[ 3307.247895] ------------[ cut here ]------------
[ 3307.247913] WARNING: at arch/x86/mm/highmem_32.c:81 __kunmap_atomic+0xdc/0xf0()
[ 3307.247917] Hardware name: 2669UYD
[ 3307.247919] Modules linked in: lib80211_crypt_wep vboxnetadp vboxnetflt ebtable_nat ebtables acpi_cpufreq mperf uinput blowfish xcbc rmd160 nfsd hdaps tp_smapi thinkpad_ec radeonfb fb_ddc radeon ttm drm_kms_helper drm ipw2200 libipw intel_agp i2c_algo_bit i2c_i801 intel_gtt rng_core cfbfillrect cfbcopyarea agpgart cfbimgblt raid10 raid1 raid0 linear md_mod vboxdrv sr_mod cdrom [last unloaded: scsi_wait_scan]
[ 3307.247991] Pid: 3236, comm: zfs-fuse Tainted: G W 2.6.39-rc7-thinkpad-t43-00897-ga2b9c1f-dirty #60
[ 3307.247994] Call Trace:
[ 3307.248003] [<c1771dcf>] ? printk+0x1d/0x1f
[ 3307.248010] [<c1056822>] warn_slowpath_common+0x72/0xa0
[ 3307.248015] [<c103633c>] ? __kunmap_atomic+0xdc/0xf0
[ 3307.248020] [<c103633c>] ? __kunmap_atomic+0xdc/0xf0
[ 3307.248025] [<c1056870>] warn_slowpath_null+0x20/0x30
[ 3307.248030] [<c103633c>] __kunmap_atomic+0xdc/0xf0
[ 3307.248038] [<c14f61f1>] zram_read+0x1c1/0x250
[ 3307.248044] [<c14f6c38>] zram_make_request+0x88/0xd0
[ 3307.248050] [<c130be96>] generic_make_request.part.49+0x206/0x7b0
[ 3307.248058] [<c1090bd1>] ? mark_held_locks+0x51/0x70
[ 3307.248064] [<c1136c0d>] ? kmem_cache_alloc+0xbd/0x1f0
[ 3307.248070] [<c108d94f>] ? trace_hardirqs_off_caller+0x1f/0x130
[ 3307.248076] [<c130c47d>] generic_make_request+0x3d/0x50
[ 3307.248080] [<c130c4f4>] submit_bio+0x64/0xf0
[ 3307.248087] [<c117b020>] ? bio_alloc_bioset+0x40/0xc0
[ 3307.248094] [<c1129ef0>] ? end_swap_bio_write+0x90/0x90
[ 3307.248099] [<c112d3e0>] ? map_swap_page+0x10/0x20
[ 3307.248104] [<c1129e12>] ? get_swap_bio+0x32/0x80
[ 3307.248109] [<c112a058>] swap_readpage+0x38/0x50
[ 3307.248114] [<c112a534>] read_swap_cache_async+0xb4/0x1b0
[ 3307.248119] [<c112a695>] swapin_readahead+0x65/0x90
[ 3307.248126] [<c111a1ae>] do_swap_page+0x11e/0x5f0
[ 3307.248132] [<c111beb6>] handle_pte_fault+0x1a6/0x240
[ 3307.248138] [<c111c118>] handle_mm_fault+0x138/0x260
[ 3307.248144] [<c178dd53>] do_page_fault+0x113/0x480
[ 3307.248150] [<c11184e5>] ? might_fault+0x95/0xa0
[ 3307.248155] [<c178b53f>] ? error_code+0x5b/0x64
[ 3307.248159] [<c178dc40>] ? vmalloc_fault+0x190/0x190
[ 3307.248165] [<c178dc40>] ? vmalloc_fault+0x190/0x190
[ 3307.248169] [<c178b543>] error_code+0x5f/0x64
[ 3307.248176] [<c178007b>] ? acpi_processor_get_info+0x1d0/0x242
[ 3307.248181] [<c178dc40>] ? vmalloc_fault+0x190/0x190
[ 3307.248185] ---[ end trace 339a5accf47eb713 ]---
Of course this WARNING, is different, but is probably releated.
System is still working and i have
Every 2,0s: free;cat /proc/swaps Thu May 19 03:47:20 2011
# free
total used free shared buffers cached
Mem: 2049232 1895852 153380 0 28472 263340
-/+ buffers/cache: 1604040 445192
Swap: 6598648 46852 6551796
# cat /proc/swaps
Filename Type Size Used Priority
/dev/dm-4 partition 6074364 0 -1
/dev/zram0 partition 524284 46852 10
#
I'm attaching my kernel config to bugzilla.
Created attachment 58392 [details]
kernel config
(In reply to comment #1) > I don't see any zram related function in any of the bracktraces, so cannot > see > how this might be related to zram. Do you see these warnings even when zram > is > not used -- i.e. you swapoff/swapon disk swaps only? No, when zram is not used I have no problem. In fact I was using compcache some time ago, but today I migrated my boot scripts to use zram, and hit this problem. So it is zram releated for sure. > > > it will quickly BUG somewhere, and after that > freeze system). > > Can you post backtrace from the BUG message? Will try. Created attachment 58402 [details]
swapoff BUG
(In reply to comment #1) > > it will quickly BUG somewhere, and after that > freeze system). > > Can you post backtrace from the BUG message? Attached as https://bugzilla.kernel.org/attachment.cgi?id=58402 Can you try this patch:
diff --git a/drivers/staging/zram/zram_drv.c b/drivers/staging/zram/zram_drv.c
index aab4ec4..69414ac 100644
--- a/drivers/staging/zram/zram_drv.c
+++ b/drivers/staging/zram/zram_drv.c
@@ -198,8 +198,8 @@ static void handle_uncompressed_page(struct zram *zram,
zram->table[index].offset;
memcpy(user_mem, cmem, PAGE_SIZE);
- kunmap_atomic(user_mem, KM_USER0);
kunmap_atomic(cmem, KM_USER1);
+ kunmap_atomic(user_mem, KM_USER0);
flush_dcache_page(page);
}
@@ -256,8 +256,8 @@ static void zram_read(struct zram *zram, struct bio *bio)
xv_get_object_size(cmem) - sizeof(*zheader),
user_mem, &clen);
- kunmap_atomic(user_mem, KM_USER0);
kunmap_atomic(cmem, KM_USER1);
+ kunmap_atomic(user_mem, KM_USER0);
/* Should NEVER happen. Return bio error if it does. */
if (unlikely(ret != LZO_E_OK)) {
I was able to reproduce BUG without using zram (but zram is still loaded in memory, hoever never referenced or used since boot). Started with no swap. # swapon /dev/mapper/sredniczarny-swap_1 [ 386.764012] Adding 5025788k swap on /dev/mapper/sredniczarny-swap_1. Priority:-1 extents:1 across:5025788k # run python with simple script to consume whole memory + 200MB of swap. suspend it, to stabilize memory consumption. # swapon -p 10 /dev/mapper/sredniczarny-swap_2 [ 557.585833] Adding 819196k swap on /dev/mapper/sredniczarny-swap_2. Priority:10 extents:1 across:819196k # no problem, new swap is empty, after a moment it have maybe few kilobytes. # swapoff /dev/mapper/sredniczarny-swap_1 # immiedietly I see [ 567.788448] ------------[ cut here ]------------ [ 567.793230] WARNING: at arch/x86/mm/highmem_32.c:81 __kunmap_atomic+0xdc/0xf0() [ 567.797995] Hardware name: 2669UYD [ 567.802736] Modules linked in: vboxnetadp vboxnetflt ebtable_nat ebtables acpi_cpufreq mperf uinput blowfish xcbc rmd160 nfsd hdaps tp_smapi thinkpad_ec radeonfb fb_ddc radeon ttm ipw2200 drm_kms_helper drm libipw i2c_i801 inte l_agp intel_gtt rng_core i2c_algo_bit cfbfillrect agpgart cfbcopyarea cfbimgblt raid10 raid1 raid0 linear md_mod vboxdrv sr_mod cdrom [last unloaded: scsi_wait_scan] [ 567.818118] Pid: 7162, comm: swapoff Tainted: G W 2.6.39-rc7-thinkpad-t43-00897-ga2b9c1f-dirty #60 [ 567.823448] Call Trace: [ 567.828609] [<c1771dcf>] ? printk+0x1d/0x1f [ 567.833849] [<c1056822>] warn_slowpath_common+0x72/0xa0 [ 567.839052] [<c103633c>] ? __kunmap_atomic+0xdc/0xf0 [ 567.844227] [<c103633c>] ? __kunmap_atomic+0xdc/0xf0 [ 567.849290] [<c1056870>] warn_slowpath_null+0x20/0x30 [ 567.854435] [<c103633c>] __kunmap_atomic+0xdc/0xf0 [ 567.859581] [<c110fb3e>] shmem_unuse_inode+0x29e/0x380 [ 567.864755] [<c110fceb>] shmem_unuse+0xcb/0x120 [ 567.869878] [<c112c638>] try_to_unuse+0x238/0x4a0 [ 567.875004] [<c178e10d>] ? sub_preempt_count.part.170+0x4d/0x90 [ 567.880155] [<c178e174>] ? sub_preempt_count+0x24/0x50 [ 567.885232] [<c112d55c>] sys_swapoff+0x16c/0x450 [ 567.890317] [<c1791a98>] sysenter_do_call+0x12/0x38 [ 567.895299] ---[ end trace 497f82cdd9c98e59 ]--- [ 567.900335] BUG: unable to handle kernel paging request at fffb9000 [ 567.905411] IP: [<c110c191>] shmem_swp_set+0x11/0x30 [ 567.910310] *pdpt = 0000000001c8c001 *pde = 0000000001c91067 *pte = 0000000000000000 [ 567.910310] Oops: 0002 [#1] PREEMPT SMP [ 567.910310] last sysfs file: /sys/devices/virtual/block/dm-4/dm/name [ 567.910310] Modules linked in: vboxnetadp vboxnetflt ebtable_nat ebtables acpi_cpufreq mperf uinput blowfish xcbc rmd160 nfsd hdaps tp_smapi thinkpad_ec radeonfb fb_ddc radeon ttm ipw2200 drm_kms_helper drm libipw i2c_i801 inte l_agp intel_gtt rng_core i2c_algo_bit cfbfillrect agpgart cfbcopyarea cfbimgblt raid10 raid1 raid0 linear md_mod vboxdrv sr_mod cdrom [last unloaded: scsi_wait_scan] [ 567.910310] [ 567.910310] Pid: 7162, comm: swapoff Tainted: G W 2.6.39-rc7-thinkpad-t43-00897-ga2b9c1f-dirty #60 IBM 2669UYD/2669UYD [ 567.910310] EIP: 0060:[<c110c191>] EFLAGS: 00010296 CPU: 0 [ 567.910310] EIP is at shmem_swp_set+0x11/0x30 [ 567.910310] EAX: f7683480 EBX: ffffffff ECX: 00000000 EDX: fffb9000 [ 567.910310] ESI: fffb9000 EDI: 00000001 EBP: f2917eb4 ESP: f2917eb0 [ 567.910310] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 [ 567.910310] Process swapoff (pid: 7162, ti=f2916000 task=f3f36a20 task.ti=f2916000) [ 567.910310] Stack: [ 567.910310] 00000056 f2917ee8 c110fb1d 00000000 00000000 f6ca0620 00000056 fffba000 [ 567.910310] f7683670 f7683480 00000010 f6ca0620 ee911c7c f6ca0620 f2917f08 c110fceb [ 567.910310] 000000d0 00000000 00000056 00000056 f8c5d056 f72cf1c0 f2917f7c c112c638 [ 567.910310] Call Trace: [ 567.910310] [<c110fb1d>] shmem_unuse_inode+0x27d/0x380 [ 567.910310] [<c110fceb>] shmem_unuse+0xcb/0x120 [ 567.910310] [<c112c638>] try_to_unuse+0x238/0x4a0 [ 567.910310] [<c178e10d>] ? sub_preempt_count.part.170+0x4d/0x90 [ 567.910310] [<c178e174>] ? sub_preempt_count+0x24/0x50 [ 567.910310] [<c112d55c>] sys_swapoff+0x16c/0x450 [ 567.910310] [<c1791a98>] sysenter_do_call+0x12/0x38 [ 567.910310] Code: fc 89 ec 5d c3 90 8d 74 26 00 3b b0 e4 00 00 00 76 ce eb b4 8d b6 00 00 00 00 55 89 e5 53 3e 8d 74 26 00 83 f9 01 19 db 83 cb 01 [ 567.910310] 0a 89 d1 01 58 30 83 c0 3c 29 c1 83 f9 3f 76 0a 89 d0 e8 27 [ 567.910310] EIP: [<c110c191>] shmem_swp_set+0x11/0x30 SS:ESP 0068:f2917eb0 [ 567.910310] CR2: 00000000fffb9000 [ 567.910310] ---[ end trace 497f82cdd9c98e5a ]--- [ 567.910310] note: swapoff[7162] exited with preempt_count 2 [ 568.085856] BUG: sleeping function called from invalid context at kernel/rwsem.c:21 [ 568.091544] in_atomic(): 1, irqs_disabled(): 0, pid: 7162, name: swapoff [ 568.097207] INFO: lockdep is turned off. [ 568.102906] Pid: 7162, comm: swapoff Tainted: G D W 2.6.39-rc7-thinkpad-t43-00897-ga2b9c1f-dirty #60 [ 568.108775] Call Trace: [ 568.114560] [<c103ea52>] __might_sleep.part.159+0xb2/0xd0 [ 568.120449] [<c103ead2>] __might_sleep+0x62/0x90 [ 568.126291] [<c17895e0>] down_read+0x20/0x85 [ 568.132056] [<c1773de5>] ? audit_free_context+0xa8/0xcd [ 568.137889] [<c105ae80>] exit_mm+0x30/0x120 [ 568.143686] [<c105b09b>] do_exit+0x12b/0x440 [ 568.149411] [<c178c034>] oops_end+0x94/0xd0 [ 568.154956] [<c1771236>] no_context+0xc3/0xcb [ 568.160319] [<c1771366>] __bad_area_nosemaphore+0x128/0x130 [ 568.165481] [<c10803df>] ? sched_clock_local+0xcf/0x1d0 [ 568.170609] [<c1770c97>] ? pte_offset_kernel+0x1e/0x2d [ 568.175629] [<c178da57>] ? spurious_fault+0x87/0xe0 [ 568.180713] [<c178dabe>] ? vmalloc_fault+0xe/0x190 [ 568.185727] [<c1771385>] bad_area_nosemaphore+0x17/0x19 [ 568.190724] [<c178defc>] do_page_fault+0x2bc/0x480 [ 568.195796] [<c1080725>] ? local_clock+0x65/0x70 [ 568.200828] [<c178e174>] ? sub_preempt_count+0x24/0x50 [ 568.205928] [<c108dcc2>] ? put_lock_stats.isra.20+0x22/0x30 [ 568.210993] [<c178b53f>] ? error_code+0x5b/0x64 [ 568.216050] [<c178dc40>] ? vmalloc_fault+0x190/0x190 [ 568.221132] [<c178dc40>] ? vmalloc_fault+0x190/0x190 [ 568.226125] [<c178b543>] error_code+0x5f/0x64 [ 568.231128] [<c11200d8>] ? vm_stat_account+0x28/0x50 [ 568.236108] [<c178dc40>] ? vmalloc_fault+0x190/0x190 [ 568.241124] [<c110c191>] ? shmem_swp_set+0x11/0x30 [ 568.246106] [<c110fb1d>] shmem_unuse_inode+0x27d/0x380 [ 568.251081] [<c110fceb>] shmem_unuse+0xcb/0x120 [ 568.256010] [<c112c638>] try_to_unuse+0x238/0x4a0 [ 568.260882] [<c178e10d>] ? sub_preempt_count.part.170+0x4d/0x90 [ 568.265734] [<c178e174>] ? sub_preempt_count+0x24/0x50 [ 568.270534] [<c112d55c>] sys_swapoff+0x16c/0x450 [ 568.275187] [<c1791a98>] sysenter_do_call+0x12/0x38 [ 568.279672] BUG: scheduling while atomic: swapoff/7162/0x10000003 [ 568.284026] INFO: lockdep is turned off. [ 568.288323] Modules linked in: vboxnetadp vboxnetflt ebtable_nat ebtables acpi_cpufreq mperf uinput blowfish xcbc rmd160 nfsd hdaps tp_smapi thinkpad_ec radeonfb fb_ddc radeon ttm ipw2200 drm_kms_helper drm libipw i2c_i801 intel_agp intel_gtt rng_core i2c_algo_bit cfbfillrect agpgart cfbcopyarea cfbimgblt raid10 raid1 raid0 linear md_mod vboxdrv sr_mod cdrom [last unloaded: scsi_wait_scan] [ 568.302477] Pid: 7162, comm: swapoff Tainted: G D W 2.6.39-rc7-thinkpad-t43-00897-ga2b9c1f-dirty #60 [ 568.307311] Call Trace: [ 568.312122] [<c1771657>] __schedule_bug+0x74/0x7b [ 568.316895] [<c1787fa0>] schedule+0x6f0/0x730 [ 568.321595] [<c111a775>] ? zap_pte_range+0xf5/0x3b0 [ 568.326257] [<c111abf6>] ? unmap_page_range+0x1c6/0x230 [ 568.330895] [<c178e0c9>] ? sub_preempt_count.part.170+0x9/0x90 [ 568.335555] [<c104f987>] __cond_resched+0x17/0x30 [ 568.340158] [<c178829d>] _cond_resched+0x2d/0x40 [ 568.344720] [<c111b20c>] unmap_vmas+0x18c/0x310 [ 568.349252] [<c1121997>] exit_mmap+0xb7/0x190 [ 568.353852] [<c10541d9>] mmput.part.24+0x39/0x100 [ 568.358479] [<c10542ce>] mmput+0x2e/0x40 [ 568.363076] [<c105af45>] exit_mm+0xf5/0x120 [ 568.367625] [<c105b09b>] do_exit+0x12b/0x440 [ 568.372181] [<c178c034>] oops_end+0x94/0xd0 [ 568.376731] [<c1771236>] no_context+0xc3/0xcb [ 568.381141] [<c1771366>] __bad_area_nosemaphore+0x128/0x130 [ 568.385481] [<c10803df>] ? sched_clock_local+0xcf/0x1d0 [ 568.389784] [<c1770c97>] ? pte_offset_kernel+0x1e/0x2d [ 568.394049] [<c178da57>] ? spurious_fault+0x87/0xe0 [ 568.398350] [<c178dabe>] ? vmalloc_fault+0xe/0x190 [ 568.402608] [<c1771385>] bad_area_nosemaphore+0x17/0x19 [ 568.406818] [<c178defc>] do_page_fault+0x2bc/0x480 [ 568.410979] [<c1080725>] ? local_clock+0x65/0x70 [ 568.415183] [<c178e174>] ? sub_preempt_count+0x24/0x50 [ 568.419230] [<c108dcc2>] ? put_lock_stats.isra.20+0x22/0x30 [ 568.423251] [<c178b53f>] ? error_code+0x5b/0x64 [ 568.427210] [<c178dc40>] ? vmalloc_fault+0x190/0x190 [ 568.431231] [<c178dc40>] ? vmalloc_fault+0x190/0x190 [ 568.435172] [<c178b543>] error_code+0x5f/0x64 [ 568.439092] [<c11200d8>] ? vm_stat_account+0x28/0x50 [ 568.443043] [<c178dc40>] ? vmalloc_fault+0x190/0x190 [ 568.446981] [<c110c191>] ? shmem_swp_set+0x11/0x30 [ 568.450936] [<c110fb1d>] shmem_unuse_inode+0x27d/0x380 [ 568.454924] [<c110fceb>] shmem_unuse+0xcb/0x120 [ 568.458852] [<c112c638>] try_to_unuse+0x238/0x4a0 [ 568.462777] [<c178e10d>] ? sub_preempt_count.part.170+0x4d/0x90 [ 568.466726] [<c178e174>] ? sub_preempt_count+0x24/0x50 [ 568.470673] [<c112d55c>] sys_swapoff+0x16c/0x450 [ 568.474586] [<c1791a98>] sysenter_do_call+0x12/0x38 manual safe reboot sequence initiated [ 578.490645] SysRq : Emergency Sync. [ 578.511835] Emergency Sync complete ... In replay to Nitin Gupta > Can you try this patch: > ... Well, I can, but because it looks to be generic problem, I do not think it will change anything. However bugtrace from Comment #2, indicates it could be zram, or different bug. I think this problem is due to shmem_unuse_inode() not doing kmap_atomic/kunmap_atomic in strictly nested manner: https://lkml.org/lkml/2010/8/19/282 zram has the same issue which is fixed by patch in comment #7 You're exactly right: I made a fix to shmem's swapoff in -rc7, quite forgetting this new rule of kmap nesting - so my part of it is a regression in 2.6.39 (and the bug much easier for a highmem-user to reproduce than the original bug it was in the course of fixing). I have a fix, I'll carry on testing it just a little longer, then rush it to Linus and add it here. No disrespect intended to Witold and Nitin: but since GregKH already has the offending patch on review for 2.6.38-stable and 2.6.33-longterm, I'd like to correct it as soon as possible. Pray that I'm not introducing a further regression with my fix! Many thanks for your report and analysis. Commit 778dd893ae78 "tmpfs: fix race between umount and swapoff" forgot the new rules for strict atomic kmap nesting, causing WARNING: at arch/x86/mm/highmem_32.c:81 from __kunmap_atomic(), then BUG: unable to handle kernel paging request at fffb9000 from shmem_swp_set() when shmem_unuse_inode() is handling swapoff with highmem in use. My disgrace again. See https://bugzilla.kernel.org/show_bug.cgi?id=35352 Reported-by: Witold Baryluk <baryluk@smp.if.uj.edu.pl> Signed-off-by: Hugh Dickins <hughd@google.com> Cc: stable@kernel.org --- mm/shmem.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) --- 2.6.39/mm/shmem.c 2011-05-18 21:06:34.000000000 -0700 +++ linux/mm/shmem.c 2011-05-20 13:57:20.778870967 -0700 @@ -916,11 +916,12 @@ static int shmem_unuse_inode(struct shme if (size > ENTRIES_PER_PAGE) size = ENTRIES_PER_PAGE; offset = shmem_find_swp(entry, ptr, ptr+size); + shmem_swp_unmap(ptr); if (offset >= 0) { shmem_dir_unmap(dir); + ptr = shmem_swp_map(subdir); goto found; } - shmem_swp_unmap(ptr); } } lost1: A patch referencing this bug report has been merged in v3.0-rc1: commit e6c9366b2adb52cba64b359b3050200743c7568c Author: Hugh Dickins <hughd@google.com> Date: Fri May 20 15:47:33 2011 -0700 tmpfs: fix highmem swapoff crash regression |