Bug 27052

Summary: man sched_rr_get_interval says the interval depend on nice, but it's incorrect.
Product: Documentation Reporter: KOSAKI Motohiro (kosaki.motohiro)
Component: man-pagesAssignee: documentation_man-pages (documentation_man-pages)
Status: RESOLVED OBSOLETE    
Severity: normal CC: alan, avi, dj, mtosatti, tai.kernel
Priority: P1    
Hardware: All   
OS: Linux   
Kernel Version: Subsystem:
Regression: No Bisected commit-id:
Attachments: KVM: MMU: fix rmap_remove on non present sptes
update sp->gfns on pte update path
debug rmap remove
debug sp->gfns validity
kvm-debug-spte-gfn-2.patch

Description KOSAKI Motohiro 2011-01-19 05:04:48 UTC 
man sched_rr_get_interval says SCHED_RR quantum depend on nice value.

----------------------------------------------------------------------------------
   Linux Notes
       POSIX  does  not  specify  any  mechanism  for controlling the size of the
       round-robin time quantum.  However, Linux provides a (non-portable) method
       of  doing  this.  The quantum can be controlled by adjusting the process's
       nice value (see setpriority(2)).  Assigning a negative (i.e.,  high)  nice
       value  results  in a longer quantum; assigning a positive (i.e., low) nice
       value results in a shorter quantum.  The default quantum is  0.1  seconds;
       the degree to which changing the nice value affects the quantum has varied
       somewhat across kernel versions.
----------------------------------------------------------------------------------

But current code doesn't depent on it. it always use 0.1 seconds.
Hehe, The code comments is misleading too. It's not knob. ;)

----------------------------------------------------------------------------------
/*
 * These are the 'tuning knobs' of the scheduler:
 *
 * default timeslice is 100 msecs (used only for SCHED_RR tasks).
 * Timeslices get refilled after they expire.
 */
#define DEF_TIMESLICE           (100 * HZ / 1000)

(snip)

static void task_tick_rt(struct rq *rq, struct task_struct *p, int queued)
{
        update_curr_rt(rq);

        watchdog(rq, p);

        /*
         * RR tasks need a special form of timeslice management.
         * FIFO tasks have no timeslices.
         */
        if (p->policy != SCHED_RR)
                return;

        if (--p->rt.time_slice)
                return;

        p->rt.time_slice = DEF_TIMESLICE;

        /*
         * Requeue to the end of queue if we are not the only element
         * on the queue:
         */
        if (p->rt.run_list.prev != p->rt.run_list.next) {
                requeue_task_rt(rq, p, 0);
                set_tsk_need_resched(p);
        }
}

(snip)

static unsigned int get_rr_interval_rt(struct rq *rq, struct task_struct *task)
{
        /*
         * Time slice is 0 for SCHED_FIFO tasks
         */
        if (task->policy == SCHED_RR)
                return DEF_TIMESLICE;
        else
                return 0;
}

----------------------------------------------------------------------------------

I think the man page only describe O(1) scheduler.
Comment 1 prochazka 2011-01-19 11:05:05 UTC 
cpuinfo and cmdline : 

/usr/local/bin/qemu -name R005 -vga std -net tap,vlan=0,name=interne,ifname=vmtap5 -net nic,vlan=0,macaddr=ac:de:48:3f:74:73,model=rtl8139 -localtime -usb -usbdevice tablet -vnc 10.98.98.1:105 -monitor tcp:127.0.0.1:10105,server,nowait,nodelay -m 256 -pidfile /var/run/qemu/R005.pid -net vde,port=55,vlan=5,sock=/tmpsafe/neoswitch_bridge,name=externe -net nic,vlan=5,macaddr=ac:de:48:15:c2:f3,model=rtl8139 -mem-prealloc -mem-path /hugepages -rtc base=localtime -drive file=/mnt/vdisk/images/VM-R005.1294325971.722755,index=0,media=disk,snapshot=on,cache=writeback -drive file=/swapfile-guest/swap1,if=ide,index=1,media=disk,snapshot=on,boot=off -fda fat:floppy:/mnt/vdisk/diskconf/R005


DEV-10.98.98.1:~# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping        : 6
cpu MHz         : 2493.297
cache size      : 6144 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dts tpr_shadow vnmi flexpriority
bogomips        : 4986.59
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping        : 6
cpu MHz         : 2493.297
cache size      : 6144 KB
physical id     : 1
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 4
initial apicid  : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dts tpr_shadow vnmi flexpriority
bogomips        : 4987.73
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 2
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping        : 6
cpu MHz         : 2493.297
cache size      : 6144 KB
physical id     : 0
siblings        : 4
core id         : 1
cpu cores       : 4
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dts tpr_shadow vnmi flexpriority
bogomips        : 4987.66
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 3
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping        : 6
cpu MHz         : 2493.297
cache size      : 6144 KB
physical id     : 1
siblings        : 4
core id         : 1
cpu cores       : 4
apicid          : 5
initial apicid  : 5
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dts tpr_shadow vnmi flexpriority
bogomips        : 4987.67
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 4
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping        : 6
cpu MHz         : 2493.297
cache size      : 6144 KB
physical id     : 0
siblings        : 4
core id         : 2
cpu cores       : 4
apicid          : 2
initial apicid  : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dts tpr_shadow vnmi flexpriority
bogomips        : 4987.65
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 5
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping        : 6
cpu MHz         : 2493.297
cache size      : 6144 KB
physical id     : 1
siblings        : 4
core id         : 2
cpu cores       : 4
apicid          : 6
initial apicid  : 6
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dts tpr_shadow vnmi flexpriority
bogomips        : 4987.68
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 6
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping        : 6
cpu MHz         : 2493.297
cache size      : 6144 KB
physical id     : 0
siblings        : 4
core id         : 3
cpu cores       : 4
apicid          : 3
initial apicid  : 3
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dts tpr_shadow vnmi flexpriority
bogomips        : 4987.66
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:

processor       : 7
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping        : 6
cpu MHz         : 2493.297
cache size      : 6144 KB
physical id     : 1
siblings        : 4
core id         : 3
cpu cores       : 4
apicid          : 7
initial apicid  : 7
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm dts tpr_shadow vnmi flexpriority
bogomips        : 4987.68
clflush size    : 64
cache_alignment : 64
address sizes   : 38 bits physical, 48 bits virtual
power management:
Comment 2 prochazka 2011-01-19 11:26:16 UTC 
it seems without hugepage, i can not reproduce this bugs

(  -mem-prealloc -mem-path /hugepages  )

/usr/local/bin/qemu -name R005 -vga std -net
tap,vlan=0,name=interne,ifname=vmtap5 -net
nic,vlan=0,macaddr=ac:de:48:3f:74:73,model=rtl8139 -localtime -usb -usbdevice
tablet -vnc 10.98.98.1:105 -monitor tcp:127.0.0.1:10105,server,nowait,nodelay
-m 256 -pidfile /var/run/qemu/R005.pid -net
vde,port=55,vlan=5,sock=/tmpsafe/neoswitch_bridge,name=externe -net
nic,vlan=5,macaddr=ac:de:48:15:c2:f3,model=rtl8139 -rtc base=localtime -drive
file=/mnt/vdisk/images/VM-R005.1294325971.722755,index=0,media=disk,snapshot=on,cache=writeback
-drive
file=/swapfile-guest/swap1,if=ide,index=1,media=disk,snapshot=on,boot=off -fda
fat:floppy:/mnt/vdisk/diskconf/R005
Comment 3 prochazka 2011-01-19 11:34:21 UTC 
Sorry, 
witout hugepage, bug is alway here : 






rmap_remove: ffff8802455bfff8 0->BUG
------------[ cut here ]------------
kernel BUG at arch/x86/kvm/mmu.c:695!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
CPU 2
Modules linked in: kvm_intel kvm

Pid: 28761, comm: qemu Not tainted 2.6.37 #3 MS-9192-01S/Express5800/120Rj-2 [N8100-1407E]
RIP: 0010:[<ffffffffa001c5ae>]  [<ffffffffa001c5ae>] drop_spte+0x1de/0x1f0 [kvm]
RSP: 0018:ffff88078db35a18  EFLAGS: 00010292
RAX: 000000000000002b RBX: ffff8802455bfff8 RCX: 000000000003ffff
RDX: ffffffff81d970c8 RSI: 0000000000000082 RDI: 0000000000000246
RBP: ffff88078db35a28 R08: 00000000000106f1 R09: 0000000000000000
R10: 0000000000000000 R11: 000000000000000f R12: ffff8801cd2c8000
R13: 00000000010147fc R14: ffff88078da98000 R15: ffff88078db35a84
FS:  00007f4085c02710(0000) GS:ffff8800cfc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 000000001806107a CR3: 00000007641ee000 CR4: 00000000000026e0
DR0: 0000000000000001 DR1: 0000000000000002 DR2: 0000000000000001
DR3: 000000000000000a DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process qemu (pid: 28761, threadinfo ffff88078db34000, task ffff8801b8264000)
Stack:
 0000000000000ff8 ffff88077abdb280 ffff88078db35ab8 ffffffffa0021075
 0000000400000001 000000000000add2 00000000006d5f42 0000000000000001
 0000000000000000 ffffea0000000001 ffff88078db35a78 0000001f010031ed
Call Trace:
 [<ffffffffa0021075>] paging32_sync_page+0xe5/0x1c0 [kvm]
 [<ffffffffa001cd6a>] __kvm_sync_page+0x5a/0xb0 [kvm]
 [<ffffffffa001f6a9>] mmu_sync_children+0x249/0x350 [kvm]
 [<ffffffffa00217ca>] ? kvm_mmu_pte_write+0x29a/0xaa0 [kvm]
 [<ffffffffa00238aa>] ? seg_base+0x1a/0x30 [kvm]
 [<ffffffffa001d4b2>] ? mmu_free_roots+0xc2/0x180 [kvm]
 [<ffffffffa0020305>] ? kvm_mmu_get_page+0x4b5/0x710 [kvm]
 [<ffffffffa001f878>] mmu_sync_roots+0xc8/0x160 [kvm]
 [<ffffffffa00205e0>] kvm_mmu_load+0x80/0x420 [kvm]
 [<ffffffffa00177e5>] kvm_arch_vcpu_ioctl_run+0xc95/0xe20 [kvm]
 [<ffffffff81029a61>] ? native_load_tr_desc+0x11/0x20
 [<ffffffffa0015fc0>] ? kvm_arch_vcpu_load+0x50/0x140 [kvm]
 [<ffffffffa0005811>] kvm_vcpu_ioctl+0x561/0x860 [kvm]
 [<ffffffff819aeeac>] ? schedule+0x31c/0x990
 [<ffffffffa0004980>] ? kvm_vm_ioctl+0x0/0x3e0 [kvm]
 [<ffffffff81131087>] do_vfs_ioctl+0xa7/0x560
 [<ffffffff810840fe>] ? sys_futex+0xce/0x170
 [<ffffffff8113158f>] sys_ioctl+0x4f/0x80
 [<ffffffff81003042>] system_call_fastpath+0x16/0x1b
Code: e1 0f 0b eb fe 48 89 de 48 c7 c7 4e ab 03 a0 31 c0 e8 2a 20 99 e1 0f 0b eb fe 48 89 de 48 c7 c7 33 ab 03 a0 31 c0 e8 15 20 99 e1 <0f> 0b eb fe 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89
RIP  [<ffffffffa001c5ae>] drop_spte+0x1de/0x1f0 [kvm]
 RSP <ffff88078db35a18>
---[ end trace 75c63c522243bec6 ]---
rmap_remove: ffff8807d245fff8 0->BUG
------------[ cut here ]------------
kernel BUG at arch/x86/kvm/mmu.c:695!
invalid opcode: 0000 [#2] SMP
last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
CPU 4
Modules linked in: kvm_intel kvm

Pid: 17775, comm: qemu Tainted: G      D     2.6.37 #3 MS-9192-01S/Express5800/120Rj-2 [N8100-1407E]
RIP: 0010:[<ffffffffa001c5ae>]  [<ffffffffa001c5ae>] drop_spte+0x1de/0x1f0 [kvm]
RSP: 0018:ffff88002646ba18  EFLAGS: 00010292
RAX: 000000000000002b RBX: ffff8807d245fff8 RCX: 000000000003ffff
RDX: ffffffff81d970c8 RSI: 0000000000000082 RDI: 0000000000000246
RBP: ffff88002646ba28 R08: 0000000000011256 R09: 0000000000000000
R10: 0000000000000000 R11: 000000000000000f R12: ffff88002645c000
R13: 00000000098d67fc R14: ffff8800264e0000 R15: ffff88002646ba84
FS:  00007ff5b0c75710(0000) GS:ffff8800cfd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00000000e2248000 CR3: 0000000026435000 CR4: 00000000000026e0
DR0: 00000000000000a0 DR1: 0000000000000000 DR2: 0000000000000003
DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process qemu (pid: 17775, threadinfo ffff88002646a000, task ffff88005d4dc000)
Stack:
 0000000000000ff8 ffff8801b7ef10a0 ffff88002646bab8 ffffffffa0021075
 0000000000000001 000000000001045c 0000000000228e71 0000000000000001
 0000000000000000 ffffea0000000001 ffff88002646ba78 00000008010031ed
Call Trace:
 [<ffffffffa0021075>] paging32_sync_page+0xe5/0x1c0 [kvm]
 [<ffffffffa001cd6a>] __kvm_sync_page+0x5a/0xb0 [kvm]
 [<ffffffffa001f6a9>] mmu_sync_children+0x249/0x350 [kvm]
 [<ffffffffa0029671>] ? x86_emulate_insn+0x1e41/0x6350 [kvm]
 [<ffffffffa00238aa>] ? seg_base+0x1a/0x30 [kvm]
 [<ffffffffa001d4b2>] ? mmu_free_roots+0xc2/0x180 [kvm]
 [<ffffffffa0020305>] ? kvm_mmu_get_page+0x4b5/0x710 [kvm]
 [<ffffffffa001f878>] mmu_sync_roots+0xc8/0x160 [kvm]
 [<ffffffffa00205e0>] kvm_mmu_load+0x80/0x420 [kvm]
 [<ffffffffa00177e5>] kvm_arch_vcpu_ioctl_run+0xc95/0xe20 [kvm]
 [<ffffffffa0015fc0>] ? kvm_arch_vcpu_load+0x50/0x140 [kvm]
 [<ffffffffa0005811>] kvm_vcpu_ioctl+0x561/0x860 [kvm]
 [<ffffffff818278c8>] ? sys_sendto+0x138/0x140
 [<ffffffff81131087>] do_vfs_ioctl+0xa7/0x560
 [<ffffffff8113158f>] sys_ioctl+0x4f/0x80
 [<ffffffff81003042>] system_call_fastpath+0x16/0x1b
Code: e1 0f 0b eb fe 48 89 de 48 c7 c7 4e ab 03 a0 31 c0 e8 2a 20 99 e1 0f 0b eb fe 48 89 de 48 c7 c7 33 ab 03 a0 31 c0 e8 15 20 99 e1 <0f> 0b eb fe 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89
RIP  [<ffffffffa001c5ae>] drop_spte+0x1de/0x1f0 [kvm]
 RSP <ffff88002646ba18>
---[ end trace 75c63c522243bec7 ]---
DEV-10.98.98.1:~#
Comment 4 Marcelo Tosatti 2011-01-20 17:28:40 UTC 
Nicolas,

This should be fixed by the attached patch, queued for 2.6.36-stable.
Comment 5 Marcelo Tosatti 2011-01-20 17:30:38 UTC 
Created attachment 44522 [details]
KVM: MMU: fix rmap_remove on non present sptes

KVM: MMU: fix rmap_remove on non present sptes
Comment 6 prochazka 2011-01-20 19:45:49 UTC 
hello, 
I do not understand, patch seems to be already apply on 2.6.37 kernel tree, 
and my test are based on this release.

NP.
Comment 7 Marcelo Tosatti 2011-01-21 03:27:36 UTC 
Nicolas,

My bad. Can you please try the following patch.
Comment 8 Marcelo Tosatti 2011-01-21 03:29:36 UTC 
Created attachment 44552 [details]
update sp->gfns on pte update path
Comment 9 prochazka 2011-01-25 08:49:48 UTC 
seems to be closed, 
but our tests are not exhaustive
Regards, 
NP.
Comment 10 Marcelo Tosatti 2011-01-25 13:09:07 UTC 
Nicolas,

It would be good if you can test with hugepages enabled again.

Thanks for the detailed report.
Comment 11 Avi Kivity 2011-01-25 13:35:08 UTC 
Don't understand the patch.  We just update gfns[index], but we don't update the other rmap stuff?  How did it get inconsistent?
Comment 12 Marcelo Tosatti 2011-01-25 13:55:05 UTC 
It gets inconsistent if mmu_set_spte updates an already present spte with a 
different gfn. For example:

- unsync page spte N gfn A
- guest updates gpte that spte N shadows with gfn B
- page fault updates sptes N with gfn B, via mmu_set_spte
- sp->gfns[] contains stale data
Comment 13 Marcelo Tosatti 2011-01-25 17:07:58 UTC 
Nicolas,

Can you please back out the first patch and apply this debug one instead?
Comment 14 Marcelo Tosatti 2011-01-25 17:08:42 UTC 
Created attachment 45162 [details]
debug rmap remove
Comment 15 prochazka 2011-01-25 17:29:00 UTC 
previous test is with hugepages.
I try to test but i can reproduce only to one server, and this server is not mine.

Regards, 
Nicolas.
Comment 16 prochazka 2011-01-28 06:58:03 UTC 
Et voilà : 

Jan 28 01:28:18 bergson25412 rmap_remove:  ffff88011ce3fff8 1->BUG
Jan 28 01:28:18 bergson25412 ------------[ cut here ]------------
Jan 28 01:28:18 bergson25412 kernel BUG at arch/x86/kvm/mmu.c:701!
Jan 28 01:28:18 bergson25412 invalid opcode: 0000 [#1] SMP
Jan 28 01:28:18 bergson25412 last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
Jan 28 01:28:18 bergson25412 CPU 3
Jan 28 01:28:18 bergson25412 Modules linked in: kvm_intel kvm
Jan 28 01:28:18 bergson25412
Jan 28 01:28:18 bergson25412 Pid: 14479, comm: qemu Not tainted 2.6.37 #3 MS-9192-01S/Express5800/120Rh-1 [N8100-xxxxF]
Jan 28 01:28:18 bergson25412 RIP: 0010:[<ffffffffa001c5a9>]  [<ffffffffa001c5a9>] drop_spte+0x1d9/0x220 [kvm]
Jan 28 01:28:18 bergson25412 RSP: 0018:ffff880186bb7a08  EFLAGS: 00010292
Jan 28 01:28:18 bergson25412 RAX: 000000000000002c RBX: ffff88011ce3fff8 RCX: 000000000003ffff
Jan 28 01:28:18 bergson25412 RDX: ffffffff81d970c8 RSI: 0000000000000082 RDI: 0000000000000246
Jan 28 01:28:18 bergson25412 RBP: ffff880186bb7a28 R08: 000000000000e958 R09: 0000000000000000
Jan 28 01:28:18 bergson25412 R10: 0000000000000000 R11: 000000000000000f R12: ffff8801c5b8bf00
Jan 28 01:28:18 bergson25412 R13: ffff880186d6c000 R14: 0000000000007a03 R15: ffff880186bb7a84
Jan 28 01:28:18 bergson25412 FS:  00007f8e0fc6d710(0000) GS:ffff8800cfcc0000(0000) knlGS:0000000000000000
Jan 28 01:28:18 bergson25412 CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
Jan 28 01:28:18 bergson25412 CR2: 00000000000c72ed CR3: 00000001aa8ff000 CR4: 00000000000026e0
Jan 28 01:28:18 bergson25412 DR0: 0000000000000045 DR1: 0000000000000000 DR2: 0000000000000000
Jan 28 01:28:18 bergson25412 DR3: 0000000000000005 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 28 01:28:18 bergson25412 Process qemu (pid: 14479, threadinfo ffff880186bb6000, task ffff8801a85fc000)
Jan 28 01:28:18 bergson25412 Stack:
Jan 28 01:28:18 bergson25412 0000000000000ff8 ffff8801c5b8bf00 000000000c820ffc ffff8801973cb490
Jan 28 01:28:18 bergson25412 ffff880186bb7ab8 ffffffffa00210a5 0000000000000001 000000000000c830
Jan 28 01:28:18 bergson25412 00000000003e6830 0000000000000001 0000000000000000 ffffea0000000001
Jan 28 01:28:18 bergson25412 Call Trace:
Jan 28 01:28:18 bergson25412 [<ffffffffa00210a5>] paging32_sync_page+0xe5/0x1c0 [kvm]
Jan 28 01:28:18 bergson25412 [<ffffffffa001cd9a>] __kvm_sync_page+0x5a/0xb0 [kvm]
Jan 28 01:28:18 bergson25412 [<ffffffffa001f6d9>] mmu_sync_children+0x249/0x350 [kvm]
Jan 28 01:28:18 bergson25412 [<ffffffffa002391a>] ? seg_base+0x1a/0x30 [kvm]
Jan 28 01:28:18 bergson25412 [<ffffffffa001d4e2>] ? mmu_free_roots+0xc2/0x180 [kvm]
Jan 28 01:28:18 bergson25412 [<ffffffffa0020335>] ? kvm_mmu_get_page+0x4b5/0x710 [kvm]
Jan 28 01:28:18 bergson25412 [<ffffffffa001f8a8>] mmu_sync_roots+0xc8/0x160 [kvm]
Jan 28 01:28:18 bergson25412 [<ffffffffa0020610>] kvm_mmu_load+0x80/0x420 [kvm]
Jan 28 01:28:18 bergson25412 [<ffffffffa00177e5>] kvm_arch_vcpu_ioctl_run+0xc95/0xe20 [kvm]
Jan 28 01:28:18 bergson25412 [<ffffffffa0015fc0>] ? kvm_arch_vcpu_load+0x50/0x140 [kvm]
Jan 28 01:28:18 bergson25412 [<ffffffffa0005811>] kvm_vcpu_ioctl+0x561/0x860 [kvm]
Jan 28 01:28:18 bergson25412 [<ffffffff81131087>] do_vfs_ioctl+0xa7/0x560
Jan 28 01:28:18 bergson25412 [<ffffffff81122b39>] ? fput+0x29/0x280
Jan 28 01:28:18 bergson25412 [<ffffffff810840fe>] ? sys_futex+0xce/0x170
Jan 28 01:28:18 bergson25412 [<ffffffff8113158f>] sys_ioctl+0x4f/0x80
Jan 28 01:28:18 bergson25412 [<ffffffff81003042>] system_call_fastpath+0x16/0x1b
Jan 28 01:28:18 bergson25412 Code: 48 89 10 eb c4 48 89 de 48 c7 c7 08 93 03 a0 31 c0 e8 2f 20 99 e1 0f 0b eb fe 48 89 de 48 c
7 c7 de ab 03 a0 31 c0 e8 1a 20 99 e1 <0f> 0b eb fe 41 0f b6 4c 24 28 4c 89 f6 83 e1 0f 48 89 da 48 c7
Jan 28 01:28:18 bergson25412 RIP  [<ffffffffa001c5a9>] drop_spte+0x1d9/0x220 [kvm]
Jan 28 01:28:18 bergson25412 RSP <ffff880186bb7a08>
Jan 28 01:28:18 bergson25412 ---[ end trace ca5feff4c31aee0f ]---


Regards, 
Nicolas
Comment 17 Marcelo Tosatti 2011-02-04 21:11:55 UTC 
Nicolas,

Please try kvm-debug-spte-gfn.patch on top of plain 2.6.37 (without the previous two patches).

Thanks
Comment 18 Marcelo Tosatti 2011-02-04 21:13:09 UTC 
Created attachment 46242 [details]
debug sp->gfns validity
Comment 19 prochazka 2011-02-08 09:46:04 UTC 
An other log without patch #18 / #17 ( as #16 )  
I try with #18

Feb  8 00:16:37 bergson18494 rmap_remove: gfn=0 index=511 level=1
Feb  8 00:16:37 bergson18494 rmap_remove: ffff88027ed4fff8 0->BUG
Feb  8 00:16:37 bergson18494 ------------[ cut here ]------------
Feb  8 00:16:37 bergson18494 kernel BUG at arch/x86/kvm/mmu.c:696!
Feb  8 00:16:37 bergson18494 invalid opcode: 0000 [#2] SMP 
Feb  8 00:16:37 bergson18494 last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
Feb  8 00:16:37 bergson18494 CPU 5 
Feb  8 00:16:37 bergson18494 Modules linked in: kvm_intel kvm bnx2
Feb  8 00:16:37 bergson18494 
Feb  8 00:16:37 bergson18494 Pid: 9127, comm: qemu Tainted: G      D     2.6.37 #4 0P010H/PowerEdge M600
Feb  8 00:16:37 bergson18494 RIP: 0010:[<ffffffffa00305e4>]  [<ffffffffa00305e4>] drop_spte+0x214/0x220 [kvm]
Feb  8 00:16:37 bergson18494 RSP: 0018:ffff88014e145a58  EFLAGS: 00010296
Feb  8 00:16:37 bergson18494 RAX: 000000000000002b RBX: ffff88027ed4fff8 RCX: 000000000003ffff
Feb  8 00:16:37 bergson18494 RDX: ffffffff81d550c8 RSI: 0000000000000086 RDI: 0000000000000246
Feb  8 00:16:37 bergson18494 RBP: ffff88014e145a78 R08: 000000000002476c R09: 0000000000000000
Feb  8 00:16:37 bergson18494 R10: 0000000000000000 R11: 000000000000000f R12: ffff880109549640
Feb  8 00:16:37 bergson18494 R13: ffff88026ae40000 R14: 0000000000000000 R15: ffff88027ed4fff8
Feb  8 00:16:37 bergson18494 FS:  00007fdecdc7a710(0000) GS:ffff8800bfd40000(0000) knlGS:0000000000000000
Feb  8 00:16:37 bergson18494 CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
Feb  8 00:16:37 bergson18494 CR2: 000000002df52c1f CR3: 0000000107572000 CR4: 00000000000426e0
Feb  8 00:16:37 bergson18494 DR0: 0000000000000001 DR1: 0000000000000002 DR2: 0000000000000001
Feb  8 00:16:37 bergson18494 DR3: 000000000000000a DR6: 00000000ffff0ff0 DR7: 0000000000000400
Feb  8 00:16:37 bergson18494 Process qemu (pid: 9127, threadinfo ffff88014e144000, task ffff880222e6f000)
Feb  8 00:16:37 bergson18494 Stack:
Feb  8 00:16:37 bergson18494 ffff880109549640 ffff88026ae40000 ffff88014e145ae8 0000000000000000
Feb  8 00:16:37 bergson18494 ffff88014e145ad8 ffffffffa00308cf ffff8801095496e0 ffff88014e145ae8
Feb  8 00:16:37 bergson18494 ffff88026ae40000 ffffea00000001ff ffff88014e145ad8 ffff880222e60000
Feb  8 00:16:37 bergson18494 Call Trace:
Feb  8 00:16:37 bergson18494 [<ffffffffa00308cf>] kvm_mmu_prepare_zap_page+0x8f/0x2f0 [kvm]
Feb  8 00:16:37 bergson18494 [<ffffffffa0031627>] __kvm_mmu_free_some_pages+0x47/0x80 [kvm]
Feb  8 00:16:37 bergson18494 [<ffffffffa00378c8>] paging32_page_fault+0x6e8/0x720 [kvm]
Feb  8 00:16:37 bergson18494 [<ffffffffa003d6e1>] ? x86_emulate_insn+0x1e41/0x6350 [kvm]
Feb  8 00:16:37 bergson18494 [<ffffffffa003791a>] ? seg_base+0x1a/0x30 [kvm]
Feb  8 00:16:37 bergson18494 [<ffffffffa0033331>] kvm_mmu_page_fault+0x21/0x80 [kvm]
Feb  8 00:16:37 bergson18494 [<ffffffffa00758bd>] handle_exception+0x30d/0x380 [kvm_intel]
Feb  8 00:16:37 bergson18494 [<ffffffffa00759c9>] vmx_handle_exit+0x99/0x2f0 [kvm_intel]
Feb  8 00:16:37 bergson18494 [<ffffffffa002b166>] kvm_arch_vcpu_ioctl_run+0x616/0xe20 [kvm]
Feb  8 00:16:37 bergson18494 [<ffffffffa0019811>] kvm_vcpu_ioctl+0x561/0x860 [kvm]
Feb  8 00:16:37 bergson18494 [<ffffffff8100398e>] ? apic_timer_interrupt+0xe/0x20
Feb  8 00:16:37 bergson18494 [<ffffffff81131087>] do_vfs_ioctl+0xa7/0x560
Feb  8 00:16:37 bergson18494 [<ffffffff81122b18>] ? fput+0x8/0x280
Feb  8 00:16:37 bergson18494 [<ffffffff810840fe>] ? sys_futex+0xce/0x170
Feb  8 00:16:37 bergson18494 [<ffffffff8113158f>] sys_ioctl+0x4f/0x80
Feb  8 00:16:37 bergson18494 [<ffffffff81003042>] system_call_fastpath+0x16/0x1b
Feb  8 00:16:37 bergson18494 Code: 48 89 da 48 c7 c7 d8 d2 04 a0 49 2b 54 24 30 31 c0 48 c1 fa 03 e8 31 21 97 e1 48 89 de 48 c7 c7 c3 eb 04 a0 31 c0 e8 20 21 97 e1 <0f> 0b eb fe 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 55 41 54 53 
Feb  8 00:16:37 bergson18494 RIP  [<ffffffffa00305e4>] drop_spte+0x214/0x220 [kvm]
Feb  8 00:16:37 bergson18494 RSP <ffff88014e145a58>
Feb  8 00:16:37 bergson18494 ---[ end trace 432bd4d0bcdd90c6 ]---
Comment 20 prochazka 2011-02-08 10:50:13 UTC 
gfn=fffffffffffff001 idx=511 lvl=1 d=0 spte=0
sp->gfns[511] = fffffffffffff001
sp->gfns[511] = fffffffffffff001
sp->gfns[511] = fffffffffffff001
sp->gfns[510] = fffffffffffff001
sp->gfns[509] = fffffffffffff001
sp->gfns[508] = fffffffffffff001
sp->gfns[507] = fffffffffffff001
------------[ cut here ]------------
kernel BUG at arch/x86/kvm/mmu.c:727!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
CPU 2 
Modules linked in: kvm_intel kvm bnx2 [last unloaded: kvm]

Pid: 29624, comm: qemu Not tainted 2.6.37 #4 0P010H/PowerEdge M600
RIP: 0010:[<ffffffffa009d7eb>]  [<ffffffffa009d7eb>] drop_spte+0x2ab/0x2b0 [kvm]
RSP: 0018:ffff8802be535a08  EFLAGS: 00010246
RAX: 0000000000000027 RBX: ffff8802ca8dbff8 RCX: 000000000003ffff
RDX: ffffffff81d550c8 RSI: 0000000000000082 RDI: 0000000000000246
RBP: ffff8802be535a28 R08: 000000000001705a R09: 0000000000000000
R10: 0000000000000000 R11: 000000000000000f R12: fffffffffffff001
R13: ffff8802c47098c0 R14: ffff8802be744000 R15: ffff8802be535a84
FS:  00007f2090ae7710(0000) GS:ffff8800bfc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 0000000001110000 CR3: 00000002c228a000 CR4: 00000000000426e0
DR0: 0000000000000090 DR1: 00000000000000a4 DR2: 00000000000000ff
DR3: 000000000000000f DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process qemu (pid: 29624, threadinfo ffff8802be534000, task ffff8807fe060000)
Stack:
 0000000000000ff8 ffff8802c47098c0 0000000009a7effc ffff8802fb9a3490
 ffff8802be535ab8 ffffffffa00a22a5 0000000000000001 000000000000fa03
 0000000000455c03 ffff880200000001 ffff880200000000 ffffffff00000001
Call Trace:
 [<ffffffffa00a22a5>] paging32_sync_page+0xe5/0x1c0 [kvm]
 [<ffffffffa009df9a>] __kvm_sync_page+0x5a/0xb0 [kvm]
 [<ffffffffa00a08d9>] mmu_sync_children+0x249/0x350 [kvm]
 [<ffffffff819a529e>] ? _raw_spin_lock+0xe/0x20
 [<ffffffffa00af073>] ? pic_unlock+0x23/0xb0 [kvm]
 [<ffffffffa00aef6a>] ? pic_update_irq+0x7a/0xf0 [kvm]
 [<ffffffffa00af388>] ? picdev_write+0x158/0x3d0 [kvm]
 [<ffffffff8100398e>] ? apic_timer_interrupt+0xe/0x20
 [<ffffffffa00a1535>] ? kvm_mmu_get_page+0x4b5/0x710 [kvm]
 [<ffffffffa00a0aa8>] mmu_sync_roots+0xc8/0x160 [kvm]
 [<ffffffffa00a1810>] kvm_mmu_load+0x80/0x420 [kvm]
 [<ffffffffa00987e5>] kvm_arch_vcpu_ioctl_run+0xc95/0xe20 [kvm]
 [<ffffffffa0086811>] kvm_vcpu_ioctl+0x561/0x860 [kvm]
 [<ffffffff8112176a>] ? do_readv_writev+0x1aa/0x1f0
 [<ffffffff81131087>] do_vfs_ioctl+0xa7/0x560
 [<ffffffff810840fe>] ? sys_futex+0xce/0x170
 [<ffffffff8113158f>] sys_ioctl+0x4f/0x80
 [<ffffffff81003042>] system_call_fastpath+0x16/0x1b
Code: 0f 4c 89 e6 41 c0 e8 05 48 89 da 41 83 e0 01 49 2b 55 30 48 c7 c7 38 a5 0b a0 48 c1 fa 03 e8 21 4f 90 e1 48 89 df e8 f5 fb ff ff <0f> 0b eb fe 90 55 48 89 e5 41 55 41 54 53 48 83 ec 08 0f 1f 44 
RIP  [<ffffffffa009d7eb>] drop_spte+0x2ab/0x2b0 [kvm]
 RSP <ffff8802be535a08>
---[ end trace 37175e4ac115e78c ]---
Comment 21 prochazka 2011-02-10 08:25:35 UTC 
Dear all,
Do you have more information about this bug submit ?

Regards, 
Nicolas Prochazka.
Comment 22 Marcelo Tosatti 2011-02-10 13:36:25 UTC 
Problem description:

Present spte is dropped while syncing 32-bit level 1 shadow page. But sp->gfns[index] contains uninitialized value (0 or fffffffffffff001), so gfn->rmap conversion in rmap_remove fails.

However, debug patch from comment #18 verifies that on present spte instantiation, via mmu_set_spte, sp->gfns[] is initialized correctly.

From bug instances of comments 19 and 20, index == 511.
Comment 23 Marcelo Tosatti 2011-02-10 13:50:08 UTC 
Nicolas,

On comment #2 you mention the bug could not be reproduced, but in comment #3 you report it without hugepages enabled. So, were you using hugepages or not, 
in the reports #18 and #19?

Another thing, what is the last kernel version that works reliably under this workload?
Comment 24 prochazka 2011-02-10 14:14:25 UTC 
I can now reproduce it under this circonstance on different server 

- Windows XP guest SP2  : guest OS seems to be important, other XP sp3 works fine
- connect with vnc to this guest and connect with RDP on other 
( 5 or 6 guests ) .

kernel : 2.6.37 
qemu-kvm with hugepages option for #18 #19 . 

/usr/local/bin/qemu -name XP_013 -vga std -net tap,vlan=0,name=interne,ifname=vmtap28 -net nic,vlan=0,macaddr=ac:de:48:88:e2:92,model=e1000 -cpu host -localtime -usb -usbdevice tablet -vnc 10.98.98.13:135 -monitor tcp:127.0.0.1:10135,server,nowait,nodelay -m 512 -pidfile /var/run/qemu/XP_013.pid -net vde,port=85,vlan=5,sock=/tmpsafe/neoswitch_bridge,name=externe -net nic,vlan=5,macaddr=ac:de:48:7b:9e:ec,model=e1000 -mem-prealloc -mem-path /hugepages -rtc base=localtime -drive file=/mnt/vdisk/images/VM-XP_013.1297326902.381783,index=0,media=disk,snapshot=on,cache=unsafe -drive file=/swapfile-guest/swap1,if=ide,index=1,media=disk,snapshot=on,boot=off -fda fat:floppy:/mnt/vdisk/diskconf/XP_013

Last Kernel that works reliably : 2.6.34  ( I do not test with kernel between 2.6.34 and 2.6.37 ) 


I just reproduce bug, with kernel 2.6.38rc4  + without hugepage 
( kvm module from 2.6.38rc4 tree) 


general protection fault: 0000 [#4] SMP 
last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
CPU 0 
Modules linked in: kvm_intel kvm bnx2

Pid: 15886, comm: qemu Tainted: G      D     2.6.38-rc4 #1 0P010H/PowerEdge M600
RIP: 0010:[<ffffffffa00319a5>]  [<ffffffffa00319a5>] drop_spte+0xd5/0x1f0 [kvm]
RSP: 0018:ffff8804d6cd5b88  EFLAGS: 00010246
RAX: ffffc9001a2d2ff8 RBX: ffff88049dbc7c00 RCX: 0000880529dd6460
RDX: 0000000000000000 RSI: 0000880529dd6460 RDI: ffff8807e30ba000
RBP: ffff8804d6cd5b98 R08: 0000000000000000 R09: dead000000200200
R10: dead000000100100 R11: 0000000000000000 R12: ffff8804d6efc000
R13: ffff8804d6cd5c08 R14: 0000000000000000 R15: ffff88049dbc7c00
FS:  00007f9b43455740(0000) GS:ffff8800bfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000056ab000 CR3: 00000004d6cfd000 CR4: 00000000000426e0
DR0: 00000000000000a0 DR1: 0000000000000000 DR2: 0000000000000003
DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process qemu (pid: 15886, threadinfo ffff8804d6cd4000, task ffff88050f22c000)
Stack:
 ffff8804a5027f00 ffff8804d6efc000 ffff8804d6cd5bf8 ffffffffa0031e7f
 00000000fffffff5 0000000000000000 ffff8804d6cd5be8 ffffffff00000180
 0000000000000000 ffff8804d6efc000 ffff8804a50276e0 ffff8804d6cd5c08
Call Trace:
 [<ffffffffa0031e7f>] kvm_mmu_prepare_zap_page+0x8f/0x2f0 [kvm]
 [<ffffffffa00327aa>] kvm_mmu_zap_all+0x4a/0x90 [kvm]
 [<ffffffffa0026496>] kvm_arch_flush_shadow+0x16/0x30 [kvm]
 [<ffffffffa0018c43>] __kvm_set_memory_region+0x2c3/0x810 [kvm]
 [<ffffffff81075e28>] ? hrtimer_start+0x18/0x20
 [<ffffffffa00473b7>] ? create_pit_timer+0xb7/0xd0 [kvm]
 [<ffffffffa00474a3>] ? pit_load_count+0xd3/0x120 [kvm]
 [<ffffffffa0047852>] ? kvm_pit_load_count+0x22/0x60 [kvm]
 [<ffffffffa00191d3>] kvm_set_memory_region+0x43/0x70 [kvm]
 [<ffffffffa001921d>] kvm_vm_ioctl_set_memory_region+0x1d/0x30 [kvm]
 [<ffffffffa0019a55>] kvm_vm_ioctl+0x1e5/0x3e0 [kvm]
 [<ffffffff811368d3>] do_vfs_ioctl+0xa3/0x540
 [<ffffffff81083afe>] ? sys_futex+0xce/0x170
 [<ffffffff81136dbf>] sys_ioctl+0x4f/0x80
 [<ffffffff81002f82>] system_call_fastpath+0x16/0x1b
Code: 50 38 48 63 f6 48 8b 34 f2 0f b6 50 28 83 e2 0f eb b8 0f 1f 40 00 48 83 e6 fe 0f 84 d9 00 00 00 45 31 c0 0f 1f 00 48 89 f1 31 d2 <48> 8b 39 48 85 ff 74 10 48 39 fb 74 26 ff c2 48 83 c1 08 83 fa 
RIP  [<ffffffffa00319a5>] drop_spte+0xd5/0x1f0 [kvm]
 RSP <ffff8804d6cd5b88>
---[ end trace a0f93d7b4fb495a7 ]---
general protection fault: 0000 [#5] SMP 
last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
CPU 5 
Modules linked in: kvm_intel kvm bnx2

Pid: 30332, comm: bash Tainted: G      D     2.6.38-rc4 #1 0P010H/PowerEdge M600
RIP: 0010:[<ffffffff81140b68>]  [<ffffffff81140b68>] dup_fd+0x168/0x300
RSP: 0018:ffff8805fbd03da0  EFLAGS: 00010202
RAX: 00000000000007f8 RBX: ffff8807e94179c0 RCX: bfffffffffffffff
RDX: 00008807e3ef5480 RSI: 00000000000000ff RDI: 0000000000000800
RBP: ffff8805fbd03e00 R08: ffff8804f2c20280 R09: 0000000000000003
R10: 0000000000000001 R11: 4000000000000000 R12: ffff8804bf071000
R13: ffff8804f2c20540 R14: ffff8807dac23800 R15: 0000000000000100
FS:  00007fb0a6a11700(0000) GS:ffff8800bfd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000bf3000 CR3: 00000007116cf000 CR4: 00000000000426e0
DR0: 0000000000000003 DR1: 00000000000000b0 DR2: 0000000000000001
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process bash (pid: 30332, threadinfo ffff8805fbd02000, task ffff880715cd1000)
Stack:
 ffff880500000005 0000000100000282 0000000000000020 ffff8806fa7dca40
 ffff8807feaceec8 ffff8807feacef40 00007fb0a6a119d0 ffff8807db5f7000
 0000000000000000 0000000001200011 00007fb0a6a119d0 0000000000000000
Call Trace:
 [<ffffffff8104fd52>] copy_process+0xa02/0x1200
 [<ffffffff810505b3>] do_fork+0x63/0x340
 [<ffffffff819b2bee>] ? _raw_spin_lock+0xe/0x20
 [<ffffffff81124477>] ? fd_install+0x67/0x90
 [<ffffffff8112f1b0>] ? do_pipe_flags+0xb0/0x100
 [<ffffffff8100c598>] sys_clone+0x28/0x30
 [<ffffffff81003223>] stub_clone+0x13/0x20
 [<ffffffff81002f82>] ? system_call_fastpath+0x16/0x1b
Code: 4c 89 c2 e8 1b 35 23 00 45 85 ff 74 77 41 8d 47 ff 31 f6 48 8d 3c c5 08 00 00 00 41 ba 01 00 00 00 31 c0 eb 1a 66 0f 1f 44 00 00 <f0> 48 ff 42 30 49 89 14 04 ff c6 48 83 c0 08 48 39 f8 74 3c 49 
RIP  [<ffffffff81140b68>] dup_fd+0x168/0x300
 RSP <ffff8805fbd03da0>
---[ end trace a0f93d7b4fb495a8 ]---
general protection fault: 0000 [#6] SMP 
last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
CPU 5 
Modules linked in: kvm_intel kvm bnx2

Pid: 30332, comm: bash Tainted: G      D     2.6.38-rc4 #1 0P010H/PowerEdge M600
RIP: 0010:[<ffffffff81124549>]  [<ffffffff81124549>] filp_close+0x19/0x90
RSP: 0018:ffff8805fbd03b28  EFLAGS: 00010286
RAX: ffff8807dac23ff8 RBX: 0000000000000003 RCX: ffff8806fa7dc180
RDX: 0000000000000000 RSI: ffff8807feaceec0 RDI: 00008807e3ef5480
RBP: ffff8805fbd03b48 R08: 0000000000000000 R09: 0000000000000000
R10: ffff8807e5659d90 R11: 0000000000000000 R12: 0000000000000001
R13: ffff8806fa7dca40 R14: ffff8807feaceec0 R15: 00000000000000ff
FS:  0000000000000000(0000) GS:ffff8800bfd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000bf3000 CR3: 0000000001d61000 CR4: 00000000000426e0
DR0: 0000000000000003 DR1: 00000000000000b0 DR2: 0000000000000001
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process bash (pid: 30332, threadinfo ffff8805fbd02000, task ffff880715cd1000)
Stack:
 0000000000000099 0000000000000003 0000000000000001 ffff8806fa7dca40
 ffff8805fbd03b98 ffffffff81053890 ffff880715cd1000 0000000000000000
 0000000000000000 ffff8807feaceec0 ffff880715cd14b4 ffff880715cd1000
Call Trace:
 [<ffffffff81053890>] put_files_struct+0xd0/0xf0
 [<ffffffff81053904>] exit_files+0x54/0x70
 [<ffffffff810552ae>] do_exit+0x14e/0x800
 [<ffffffff8100726f>] oops_end+0xaf/0xf0
 [<ffffffff810074bb>] die+0x5b/0x90
 [<ffffffff81004df2>] do_general_protection+0x162/0x170
 [<ffffffff819b3335>] general_protection+0x25/0x30
 [<ffffffff81140b68>] ? dup_fd+0x168/0x300
 [<ffffffff8104fd52>] copy_process+0xa02/0x1200
 [<ffffffff810505b3>] do_fork+0x63/0x340
 [<ffffffff819b2bee>] ? _raw_spin_lock+0xe/0x20
 [<ffffffff81124477>] ? fd_install+0x67/0x90
 [<ffffffff8112f1b0>] ? do_pipe_flags+0xb0/0x100
 [<ffffffff8100c598>] sys_clone+0x28/0x30
 [<ffffffff81003223>] stub_clone+0x13/0x20
 [<ffffffff81002f82>] ? system_call_fastpath+0x16/0x1b
Code: 5d e8 4c 8b 65 f0 4c 8b 6d f8 c9 c3 66 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 4c 89 6d f8 0f 1f 44 00 00 <48> 8b 47 30 48 89 fb 49 89 f4 48 85 c0 74 4d 48 8b 47 20 48 85 
RIP  [<ffffffff81124549>] filp_close+0x19/0x90
 RSP <ffff8805fbd03b28>
---[ end trace a0f93d7b4fb495a9 ]---
Fixing recursive fault but reboot is needed!
BUG: unable to handle kernel paging request at ffffed7fffffffd8
IP: [<ffffffffa0031f12>] kvm_mmu_prepare_zap_page+0x122/0x2f0 [kvm]
PGD 0 
Oops: 0000 [#7] SMP 
last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
CPU 1 
Modules linked in: kvm_intel kvm bnx2

Pid: 17293, comm: qemu Tainted: G      D     2.6.38-rc4 #1 0P010H/PowerEdge M600
RIP: 0010:[<ffffffffa0031f12>]  [<ffffffffa0031f12>] kvm_mmu_prepare_zap_page+0x122/0x2f0 [kvm]
RSP: 0018:ffff8804a51838e8  EFLAGS: 00010206
RAX: 00000000000001ff RBX: ffff8804f2f04c80 RCX: 0000037fffffffc8
RDX: ffffea0000000000 RSI: ffff880463d77ff8 RDI: ffff880463d77ff0
RBP: ffff8804a5183938 R08: ffff8804df999808 R09: dead000000200200
R10: dead000000100100 R11: 0000000000000000 R12: ffff8804a50cc000
R13: ffff8804a51839e8 R14: 0000000000000002 R15: ffff880463d77ff8
FS:  00007f364c568710(0000) GS:ffff8800bfc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: ffffed7fffffffd8 CR3: 00000004a50ad000 CR4: 00000000000426e0
DR0: 0000000000000001 DR1: 0000000000000002 DR2: 0000000000000001
DR3: 000000000000000a DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process qemu (pid: 17293, threadinfo ffff8804a5182000, task ffff8804bf3ba000)
Stack:
 ffff8804a51839a8 ffffffffa0033ed5 ffff880400000006 00000000000001fe
 000000000000000f ffff8804f2f04c80 ffff8804f2f04c90 0000000000000000
 0000000000000080 ffff8804f2ef3700 ffff8804a5183a38 ffffffffa003791c
Call Trace:
 [<ffffffffa0033ed5>] ? paging32_walk_addr_generic+0x375/0x5c0 [kvm]
 [<ffffffffa003791c>] kvm_mmu_pte_write+0x22c/0xa90 [kvm]
 [<ffffffffa0016f42>] ? kvm_write_guest_page+0x72/0xd0 [kvm]
 [<ffffffffa0027106>] emulator_write_phys+0x56/0x70 [kvm]
 [<ffffffffa002718b>] emulator_write_emulated_onepage+0x6b/0x190 [kvm]
 [<ffffffffa0027333>] emulator_write_emulated+0x83/0xa0 [kvm]
 [<ffffffffa001f28d>] ? emulator_get_cached_segment_base+0x1d/0x20 [kvm]
 [<ffffffffa00272b0>] ? emulator_write_emulated+0x0/0xa0 [kvm]
 [<ffffffffa003fa0d>] x86_emulate_insn+0x20fd/0x6390 [kvm]
 [<ffffffffa003b64b>] ? x86_decode_insn+0x74b/0xcd0 [kvm]
 [<ffffffffa003ab00>] ? em_mov+0x0/0x20 [kvm]
 [<ffffffffa0028eb8>] x86_emulate_instruction+0xb8/0x3d0 [kvm]
 [<ffffffffa0034ff1>] kvm_mmu_page_fault+0x71/0x90 [kvm]
 [<ffffffffa007cb24>] handle_exception+0x324/0x390 [kvm_intel]
 [<ffffffffa007cc1e>] vmx_handle_exit+0x8e/0x2b0 [kvm_intel]
 [<ffffffffa002c546>] kvm_arch_vcpu_ioctl_run+0x526/0xe70 [kvm]
 [<ffffffffa001a6f2>] kvm_vcpu_ioctl+0x502/0x650 [kvm]
 [<ffffffff81063ed1>] ? dequeue_signal+0x41/0x170
 [<ffffffff81061b1f>] ? copy_siginfo_to_user+0xff/0x1f0
 [<ffffffff811368d3>] do_vfs_ioctl+0xa3/0x540
 [<ffffffff81083afe>] ? sys_futex+0xce/0x170
 [<ffffffff81136dbf>] sys_ioctl+0x4f/0x80
 [<ffffffff81002f82>] system_call_fastpath+0x16/0x1b
Code: 75 d8 0f 0b eb fe 0f 1f 00 48 ba 00 f0 ff ff ff ff 0f 00 4c 89 fe 48 21 d1 48 ba 00 00 00 00 00 ea ff ff 48 c1 e9 0c 48 6b c9 38 <48> 8b 7c 11 10 89 45 c8 e8 61 f5 ff ff 48 8b 0d ea 6c 02 00 8b 
RIP  [<ffffffffa0031f12>] kvm_mmu_prepare_zap_page+0x122/0x2f0 [kvm]
 RSP <ffff8804a51838e8>
CR2: ffffed7fffffffd8
---[ end trace a0f93d7b4fb495aa ]---
Comment 25 prochazka 2011-02-10 14:16:51 UTC 
cmdline 
/usr/local/bin/qemu -name Soins_003 -vga std -net tap,vlan=0,name=interne,ifname=vmtap5 -net nic,vlan=0,macaddr=ac:de:48:1d:e8:2c,model=e1000 -cpu host -localtime -usb -usbdevice tablet -vnc 10.98.98.19:120 -monitor tcp:127.0.0.1:10120,server,nowait,nodelay -m 512 -pidfile /var/run/qemu/Soins_003.pid -net vde,port=70,vlan=5,sock=/tmpsafe/neoswitch_bridge,name=externe -net nic,vlan=5,macaddr=ac:de:48:8c:cc:e0,model=e1000 -rtc base=localtime -drive file=/mnt/vdisk/images/VM-Soins_003.1296578833.637768,index=0,media=disk,snapshot=on,cache=unsafe -drive file=/swapfile-guest/swap1,if=ide,index=1,media=disk,snapshot=on,boot=off -fda fat:floppy:/mnt/vdisk/diskconf/Soins_003

KSM and transparent hugepage is activated on this kernel.

Regards, 
Nicolas
Comment 26 Marcelo Tosatti 2011-02-10 16:57:17 UTC 
Nicolas,

New debug patch attached. Please try it on top of clean 2.6.37.
Comment 27 Marcelo Tosatti 2011-02-10 16:57:59 UTC 
Created attachment 47152 [details]
kvm-debug-spte-gfn-2.patch
Comment 28 prochazka 2011-02-11 15:43:22 UTC 
Marcelo, 
debug info
Regards, 
Nicolas

general protection fault: 0000 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
CPU 0 
Modules linked in: kvm_intel kvm [last unloaded: kvm]

Pid: 8532, comm: run Not tainted 2.6.37 #3 0P010H/PowerEdge M600
RIP: 0010:[<ffffffff8113a3e8>]  [<ffffffff8113a3e8>] dup_fd+0x168/0x300
RSP: 0018:ffff8807e3df3d90  EFLAGS: 00010206
RAX: 00000000000007f8 RBX: ffff88022be64580 RCX: bfffffffffffffff
RDX: 00008807fe5ed0c0 RSI: 00000000000000ff RDI: 0000000000000800
RBP: ffff8807e3df3df0 R08: ffff88022c413a80 R09: 0000000000000003
R10: 0000000000000001 R11: 4000000000000000 R12: ffff8807fea2f000
R13: ffff88022c413e80 R14: ffff8807e7a9d000 R15: 0000000000000100
FS:  00007ffaa073f700(0000) GS:ffff8800bfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000006d6878 CR3: 00000007e79eb000 CR4: 00000000000426e0
DR0: 0000000000000001 DR1: 0000000000000002 DR2: 0000000000000001
DR3: 000000000000000a DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process run (pid: 8532, threadinfo ffff8807e3df2000, task ffff8807fe78a000)
Stack:
 ffff880200000000 0000000100000282 0000000000000020 ffff8807e7e3e440
 ffff8807fe93ac08 ffff8807fe93ac80 00007ffaa073f9d0 ffff88022d35e000
 0000000000000000 0000000001200011 00007ffaa073f9d0 0000000000000000
Call Trace:
 [<ffffffff8104fdba>] copy_process+0x9fa/0x11e0
 [<ffffffff81031e78>] ? do_page_fault+0x1b8/0x480
 [<ffffffff81050621>] do_fork+0x81/0x3f0
 [<ffffffff810634cd>] ? do_sigaction+0x13d/0x1e0
 [<ffffffff819b11c5>] ? _raw_spin_lock_irq+0x15/0x20
 [<ffffffff81062eb1>] ? sigprocmask+0x91/0x110
 [<ffffffff8100c498>] sys_clone+0x28/0x30
 [<ffffffff810032e3>] stub_clone+0x13/0x20
 [<ffffffff81003042>] ? system_call_fastpath+0x16/0x1b
Code: 4c 89 c2 e8 5b 83 23 00 45 85 ff 74 77 41 8d 47 ff 31 f6 48 8d 3c c5 08 00 00 00 41 ba 01 00 00 00 31 c0 eb 1a 66 0f 1f 44 00 00 <f0> 48 ff 42 30 49 89 14 04 ff c6 48 83 c0 08 48 39 f8 74 3c 49 
RIP  [<ffffffff8113a3e8>] dup_fd+0x168/0x300
 RSP <ffff8807e3df3d90>
---[ end trace 89621523c6ce6788 ]---
general protection fault: 0000 [#2] SMP 
last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
CPU 0 
Modules linked in: kvm_intel kvm [last unloaded: kvm]

Pid: 8532, comm: run Tainted: G      D     2.6.37 #3 0P010H/PowerEdge M600
RIP: 0010:[<ffffffff8111f5f9>]  [<ffffffff8111f5f9>] filp_close+0x19/0x90
RSP: 0018:ffff8807e3df3b18  EFLAGS: 00010286
RAX: ffff8807e7a9d7f8 RBX: 0000000000000003 RCX: ffff8807e7e3ec80
RDX: 0000000000000000 RSI: ffff8807fe93ac00 RDI: 00008807fe5ed0c0
RBP: ffff8807e3df3b38 R08: 0000000000000000 R09: 0000000000000000
R10: ffff8807e7c4b790 R11: 0000000000000000 R12: 0000000000000001
R13: ffff8807e7e3e440 R14: ffff8807fe93ac00 R15: 00000000000000ff
FS:  0000000000000000(0000) GS:ffff8800bfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000006d6878 CR3: 0000000001d87000 CR4: 00000000000426e0
DR0: 0000000000000001 DR1: 0000000000000002 DR2: 0000000000000001
DR3: 000000000000000a DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process run (pid: 8532, threadinfo ffff8807e3df2000, task ffff8807fe78a000)
Stack:
 0000000000000077 0000000000000003 0000000000000001 ffff8807e7e3e440
 ffff8807e3df3b88 ffffffff810539e0 ffff8807fe78a000 0000000000000000
 0000000000000000 ffff8807fe93ac00 ffff8807fe78a4b4 ffff8807fe78a000
Call Trace:
 [<ffffffff810539e0>] put_files_struct+0xd0/0xf0
 [<ffffffff81053a54>] exit_files+0x54/0x70
 [<ffffffff8105540e>] do_exit+0x14e/0x800
 [<ffffffff8100740f>] oops_end+0xaf/0xf0
 [<ffffffff8100765b>] die+0x5b/0x90
 [<ffffffff81004e72>] do_general_protection+0x162/0x170
 [<ffffffff819b18b5>] general_protection+0x25/0x30
 [<ffffffff8113a3e8>] ? dup_fd+0x168/0x300
 [<ffffffff8104fdba>] copy_process+0x9fa/0x11e0
 [<ffffffff81031e78>] ? do_page_fault+0x1b8/0x480
 [<ffffffff81050621>] do_fork+0x81/0x3f0
 [<ffffffff810634cd>] ? do_sigaction+0x13d/0x1e0
 [<ffffffff819b11c5>] ? _raw_spin_lock_irq+0x15/0x20
 [<ffffffff81062eb1>] ? sigprocmask+0x91/0x110
 [<ffffffff8100c498>] sys_clone+0x28/0x30
 [<ffffffff810032e3>] stub_clone+0x13/0x20
 [<ffffffff81003042>] ? system_call_fastpath+0x16/0x1b
Code: 5d e8 4c 8b 65 f0 4c 8b 6d f8 c9 c3 66 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 4c 89 6d f8 0f 1f 44 00 00 <48> 8b 47 30 48 89 fb 49 89 f4 48 85 c0 74 4d 48 8b 47 20 48 85 
RIP  [<ffffffff8111f5f9>] filp_close+0x19/0x90
 RSP <ffff8807e3df3b18>
---[ end trace 89621523c6ce6789 ]---
Fixing recursive fault but reboot is needed!
gfn=dead01ff idx=511 lvl=1 d=0 spte=0 old_spte=fffffffff001
sp->gfns[511] = dead01ff
sp->gfns[511] = dead01ff
sp->gfns[511] = dead01ff
sp->gfns[510] = dead01fe
sp->gfns[509] = dead01fd
sp->gfns[508] = dead01fc
sp->gfns[507] = dead01fb
------------[ cut here ]------------
kernel BUG at arch/x86/kvm/mmu.c:727!
invalid opcode: 0000 [#3] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/0000:02:01.0/0000:05:00.0/0000:06:00.0/irq
CPU 4 
Modules linked in: kvm_intel kvm [last unloaded: kvm]

Pid: 8790, comm: qemu Tainted: G      D     2.6.37 #3 0P010H/PowerEdge M600
RIP: 0010:[<ffffffffa008a810>]  [<ffffffffa008a810>] drop_spte+0x2d0/0x2e0 [kvm]
RSP: 0018:ffff88029c8cb9e8  EFLAGS: 00010246
RAX: 000000000000001f RBX: ffff88022d29fff8 RCX: 000000000003ffff
RDX: ffffffff81d970c8 RSI: 0000000000000086 RDI: 0000000000000246
RBP: ffff88029c8cba28 R08: 000000000001764e R09: 0000000000000000
R10: 0000000000000000 R11: 000000000000000d R12: 00000000dead01ff
R13: ffff880273745320 R14: ffff880293324000 R15: 0000fffffffff001
FS:  00007fe7fcb59710(0000) GS:ffff8800bfd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 000000007ca29600 CR3: 000000026a538000 CR4: 00000000000426e0
DR0: 00000000000000a0 DR1: 0000000000000000 DR2: 0000000000000003
DR3: 00000000000000b0 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process qemu (pid: 8790, threadinfo ffff88029c8ca000, task ffff8807e2b66000)
Stack:
 0000fffffffff001 ffff8801d6914148 ffff88029c8cba28 0000000000000ff8
 ffff880273745320 00000000027237fc ffff88026e95b490 ffff88029c8cba84
 ffff88029c8cbab8 ffffffffa008f2f5 0000000400000001 00000000000032ae
Call Trace:
 [<ffffffffa008f2f5>] paging32_sync_page+0xe5/0x1c0 [kvm]
 [<ffffffffa008afca>] __kvm_sync_page+0x5a/0xb0 [kvm]
 [<ffffffffa008d909>] mmu_sync_children+0x249/0x350 [kvm]
 [<ffffffffa008e58d>] ? kvm_mmu_get_page+0x4dd/0x730 [kvm]
 [<ffffffffa008dad8>] mmu_sync_roots+0xc8/0x160 [kvm]
 [<ffffffffa008e860>] kvm_mmu_load+0x80/0x420 [kvm]
 [<ffffffffa00857e5>] kvm_arch_vcpu_ioctl_run+0xc95/0xe20 [kvm]
 [<ffffffffa0073811>] kvm_vcpu_ioctl+0x561/0x860 [kvm]
 [<ffffffff81063e74>] ? __dequeue_signal+0xf4/0x190
 [<ffffffff810642c1>] ? dequeue_signal+0x41/0x170
 [<ffffffff81131087>] do_vfs_ioctl+0xa7/0x560
 [<ffffffff810840fe>] ? sys_futex+0xce/0x170
 [<ffffffff8113158f>] sys_ioctl+0x4f/0x80
 [<ffffffff81003042>] system_call_fastpath+0x16/0x1b
Code: 05 48 89 da 41 83 e0 01 49 2b 55 30 48 c7 c7 78 75 0a a0 48 c1 fa 03 4c 89 3c 24 4c 8b 0b e8 bb 3d 92 e1 48 89 df e8 d0 fb ff ff <0f> 0b eb fe 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 
RIP  [<ffffffffa008a810>] drop_spte+0x2d0/0x2e0 [kvm]
 RSP <ffff88029c8cb9e8>
---[ end trace 89621523c6ce678a ]---
general protection fault: 0000 [#4] SMP 
last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
CPU 7 
Modules linked in: kvm_intel kvm [last unloaded: kvm]

Pid: 1, comm: init Tainted: G      D     2.6.37 #3 0P010H/PowerEdge M600
RIP: 0010:[<ffffffff8113a3e8>]  [<ffffffff8113a3e8>] dup_fd+0x168/0x300
RSP: 0018:ffff88080014fd90  EFLAGS: 00010206
RAX: 00000000000007f8 RBX: ffff8807e7e35340 RCX: bfffffffffffffff
RDX: 00008807e7da63c0 RSI: 00000000000000ff RDI: 0000000000000800
RBP: ffff88080014fdf0 R08: ffff8802d67dc980 R09: 0000000000000003
R10: 0000000000000001 R11: 4000000000000000 R12: ffff8807e7aa6000
R13: ffff8802d67dc540 R14: ffff8807fe6d8800 R15: 0000000000000100
FS:  00007f5239c31700(0000) GS:ffff8800bfdc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000006d7bc8 CR3: 00000007fe6a3000 CR4: 00000000000426e0
DR0: 0000000000000001 DR1: 0000000000000002 DR2: 0000000000000001
DR3: 000000000000000a DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process init (pid: 1, threadinfo ffff88080014e000, task ffff88083f82a000)
Stack:
 ffff880200000007 0000000100000282 0000000000000020 ffff8807fe419600
 ffff880800150008 ffff880800150080 00007f5239c319d0 ffff8802fa5eb000
 0000000000000000 0000000001200011 00007f5239c319d0 0000000000000000
Call Trace:
 [<ffffffff8104fdba>] copy_process+0x9fa/0x11e0
 [<ffffffff81031e78>] ? do_page_fault+0x1b8/0x480
 [<ffffffff81050621>] do_fork+0x81/0x3f0
 [<ffffffff819b11c5>] ? _raw_spin_lock_irq+0x15/0x20
 [<ffffffff810634cd>] ? do_sigaction+0x13d/0x1e0
 [<ffffffff819b11c5>] ? _raw_spin_lock_irq+0x15/0x20
 [<ffffffff81062eb1>] ? sigprocmask+0x91/0x110
 [<ffffffff8100c498>] sys_clone+0x28/0x30
 [<ffffffff810032e3>] stub_clone+0x13/0x20
 [<ffffffff81003042>] ? system_call_fastpath+0x16/0x1b
Code: 4c 89 c2 e8 5b 83 23 00 45 85 ff 74 77 41 8d 47 ff 31 f6 48 8d 3c c5 08 00 00 00 41 ba 01 00 00 00 31 c0 eb 1a 66 0f 1f 44 00 00 <f0> 48 ff 42 30 49 89 14 04 ff c6 48 83 c0 08 48 39 f8 74 3c 49 
RIP  [<ffffffff8113a3e8>] dup_fd+0x168/0x300
 RSP <ffff88080014fd90>
---[ end trace 89621523c6ce678b ]---
general protection fault: 0000 [#5] SMP 
last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
CPU 7 
Modules linked in: kvm_intel kvm [last unloaded: kvm]

Pid: 1, comm: init Tainted: G      D     2.6.37 #3 0P010H/PowerEdge M600
RIP: 0010:[<ffffffff8111f5f9>]  [<ffffffff8111f5f9>] filp_close+0x19/0x90
RSP: 0018:ffff88080014fb18  EFLAGS: 00010286
RAX: ffff8807fe6d8ff8 RBX: 0000000000000003 RCX: ffff8807fe419700
RDX: 0000000000000000 RSI: ffff880800150000 RDI: 00008807e7da63c0
RBP: ffff88080014fb38 R08: 0000000000000004 R09: 0000000000000000
R10: ffff8800bfeaef28 R11: 0000000000000000 R12: 0000000000000001
R13: ffff8807fe419600 R14: ffff880800150000 R15: 00000000000000ff
FS:  0000000000000000(0000) GS:ffff8800bfdc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000006d7bc8 CR3: 0000000001d87000 CR4: 00000000000426e0
DR0: 0000000000000001 DR1: 0000000000000002 DR2: 0000000000000001
DR3: 000000000000000a DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process init (pid: 1, threadinfo ffff88080014e000, task ffff88083f82a000)
Stack:
 0000000000000078 0000000000000003 0000000000000001 ffff8807fe419600
 ffff88080014fb88 ffffffff810539e0 ffff88083f82a000 0000000000000000
 0000000000000000 ffff880800150000 ffff88083f82a4b4 ffff88083f82a000
Call Trace:
 [<ffffffff810539e0>] put_files_struct+0xd0/0xf0
 [<ffffffff81053a54>] exit_files+0x54/0x70
 [<ffffffff8105540e>] do_exit+0x14e/0x800
 [<ffffffff8100740f>] oops_end+0xaf/0xf0
 [<ffffffff8100765b>] die+0x5b/0x90
 [<ffffffff81004e72>] do_general_protection+0x162/0x170
 [<ffffffff819b18b5>] general_protection+0x25/0x30
 [<ffffffff8113a3e8>] ? dup_fd+0x168/0x300
 [<ffffffff8104fdba>] copy_process+0x9fa/0x11e0
 [<ffffffff81031e78>] ? do_page_fault+0x1b8/0x480
 [<ffffffff81050621>] do_fork+0x81/0x3f0
 [<ffffffff819b11c5>] ? _raw_spin_lock_irq+0x15/0x20
 [<ffffffff810634cd>] ? do_sigaction+0x13d/0x1e0
 [<ffffffff819b11c5>] ? _raw_spin_lock_irq+0x15/0x20
 [<ffffffff81062eb1>] ? sigprocmask+0x91/0x110
 [<ffffffff8100c498>] sys_clone+0x28/0x30
 [<ffffffff810032e3>] stub_clone+0x13/0x20
 [<ffffffff81003042>] ? system_call_fastpath+0x16/0x1b
Code: 5d e8 4c 8b 65 f0 4c 8b 6d f8 c9 c3 66 0f 1f 44 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 4c 89 6d f8 0f 1f 44 00 00 <48> 8b 47 30 48 89 fb 49 89 f4 48 85 c0 74 4d 48 8b 47 20 48 85 
RIP  [<ffffffff8111f5f9>] filp_close+0x19/0x90
 RSP <ffff88080014fb18>
---[ end trace 89621523c6ce678c ]---
Fixing recursive fault but reboot is needed!
Comment 29 Marcelo Tosatti 2011-02-11 21:33:17 UTC 
gfn=dead01ff idx=511 lvl=1 d=0 spte=0 old_spte=fffffffff001

The last spte is 0xfffffffff001, it should be 0xfffffffffffff001 (two bytes at
end of pgtable zeroed). Something is screwing up.

Nicolas, can you please enable CONFIG_SLUB_DEBUG=y and boot with slub_debug=FZPU kernel option.
Comment 30 prochazka 2011-02-12 17:06:02 UTC 
hi marcelo, 
I enable CONFIG_SLUB_DEBUG=y and boot with slub_debug=FZPU, 
but now I cannot reproduce bug

I have a lot of : BUG kmalloc-1024: Object padding overwritten

example : 
Feb 12 16:26:49 bergson11630 
Feb 12 16:26:49 bergson11630 =============================================================================
Feb 12 16:26:49 bergson11630 BUG kmalloc-1024: Object padding overwritten
Feb 12 16:26:49 bergson11630 -----------------------------------------------------------------------------
Feb 12 16:26:49 bergson11630 
Feb 12 16:26:49 bergson11630 INFO: 0xffff8802f8588446-0xffff8802f8588447. First byte 0x0 instead of 0x5a
Feb 12 16:26:49 bergson11630 INFO: Allocated in sock_alloc_send_pskb+0x1d4/0x330 age=1 cpu=1 pid=20244
Feb 12 16:26:49 bergson11630 INFO: Freed in pskb_expand_head+0xcc/0x1f0 age=1 cpu=1 pid=20244
Feb 12 16:26:49 bergson11630 INFO: Slab 0xffffea000a6535c0 objects=29 used=16 fp=0xffff8802f8588000 flags=0x2000000000040c1
Feb 12 16:26:49 bergson11630 INFO: Object 0xffff8802f8588000 @offset=0 fp=0xffff8802f858a240
Feb 12 16:26:49 bergson11630 
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588000:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588010:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588020:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588030:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588040:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588050:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588060:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588070:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588080:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588090:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85880a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85880b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85880c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85880d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85880e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85880f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588100:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588110:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588120:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588130:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588140:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588150:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588160:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588170:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588180:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588190:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85881a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85881b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85881c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85881d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85881e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85881f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588200:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588210:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588220:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588230:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588240:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588250:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588260:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588270:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588280:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588290:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85882a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85882b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85882c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85882d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85882e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85882f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588300:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588310:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588320:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588330:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588340:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588350:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588360:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588370:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588380:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f8588390:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85883a0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85883b0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85883c0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85883d0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85883e0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Feb 12 16:26:49 bergson11630 Object 0xffff8802f85883f0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk�
Feb 12 16:26:49 bergson11630 Redzone 0xffff8802f8588400:  bb bb bb bb bb bb bb bb                         ��������        
Feb 12 16:26:49 bergson11630 Padding 0xffff8802f8588440:  5a 5a 5a 5a 5a 5a 00 00                         ZZZZZZ..        
Feb 12 16:26:49 bergson11630 Pid: 20244, comm: qemu Not tainted 2.6.37 #4
Feb 12 16:26:49 bergson11630 Call Trace:
Feb 12 16:26:49 bergson11630 <IRQ>  [<ffffffff81113b13>] print_trailer+0x103/0x160
Feb 12 16:26:49 bergson11630 [<ffffffff81114222>] check_bytes_and_report+0x102/0x130
Feb 12 16:26:49 bergson11630 [<ffffffff811142fc>] check_object+0xac/0x260
Feb 12 16:26:49 bergson11630 [<ffffffff8183427e>] ? pskb_expand_head+0x5e/0x1f0
Feb 12 16:26:49 bergson11630 [<ffffffff81114d10>] alloc_debug_processing+0x110/0x1a0
Feb 12 16:26:49 bergson11630 [<ffffffff8111632f>] __slab_alloc+0x36f/0x3f0
Feb 12 16:26:49 bergson11630 [<ffffffff8183427e>] ? pskb_expand_head+0x5e/0x1f0
Feb 12 16:26:49 bergson11630 [<ffffffff81116b77>] __kmalloc+0xe7/0x1b0
Feb 12 16:26:49 bergson11630 [<ffffffff8183427e>] pskb_expand_head+0x5e/0x1f0
Feb 12 16:26:49 bergson11630 [<ffffffff81936b08>] nf_bridge_copy_header+0x78/0x180
Feb 12 16:26:49 bergson11630 [<ffffffff8192f220>] br_dev_queue_push_xmit+0x80/0xa0
Feb 12 16:26:49 bergson11630 [<ffffffff8193505a>] br_nf_dev_queue_xmit+0x2a/0xa0
Feb 12 16:26:49 bergson11630 [<ffffffff81935368>] br_nf_post_routing+0x298/0x2f0
Feb 12 16:26:49 bergson11630 [<ffffffff8187ec8c>] nf_iterate+0x6c/0xb0
Feb 12 16:26:49 bergson11630 [<ffffffff8192f1a0>] ? br_dev_queue_push_xmit+0x0/0xa0
Feb 12 16:26:49 bergson11630 [<ffffffff8187ed7e>] nf_hook_slow+0xae/0x110
Feb 12 16:26:49 bergson11630 [<ffffffff8192f1a0>] ? br_dev_queue_push_xmit+0x0/0xa0
Feb 12 16:26:49 bergson11630 [<ffffffff8192f240>] ? br_forward_finish+0x0/0x60
Feb 12 16:26:49 bergson11630 [<ffffffff8192f283>] br_forward_finish+0x43/0x60
Feb 12 16:26:49 bergson11630 [<ffffffff81935528>] br_nf_forward_finish+0x168/0x170
Feb 12 16:26:49 bergson11630 [<ffffffff81935a28>] br_nf_forward_ip+0x328/0x430
Feb 12 16:26:49 bergson11630 [<ffffffff8187ec8c>] nf_iterate+0x6c/0xb0
Feb 12 16:26:49 bergson11630 [<ffffffff8192f240>] ? br_forward_finish+0x0/0x60
Feb 12 16:26:49 bergson11630 [<ffffffff8187ed7e>] nf_hook_slow+0xae/0x110
Feb 12 16:26:49 bergson11630 [<ffffffff8192f240>] ? br_forward_finish+0x0/0x60
Feb 12 16:26:49 bergson11630 [<ffffffff8192f31e>] __br_forward+0x7e/0xd0
Feb 12 16:26:49 bergson11630 [<ffffffff8192f3d5>] br_forward+0x65/0x70
Feb 12 16:26:49 bergson11630 [<ffffffff81930128>] br_handle_frame_finish+0x128/0x260
Feb 12 16:26:49 bergson11630 [<ffffffff81935d78>] br_nf_pre_routing_finish_bridge+0x78/0xd0
Feb 12 16:26:49 bergson11630 [<ffffffff819360b0>] br_nf_pre_routing_finish+0x2e0/0x370
Feb 12 16:26:49 bergson11630 [<ffffffff8187ed7e>] ? nf_hook_slow+0xae/0x110
Feb 12 16:26:49 bergson11630 [<ffffffff81935dd0>] ? br_nf_pre_routing_finish+0x0/0x370
Feb 12 16:26:49 bergson11630 [<ffffffff81936848>] br_nf_pre_routing+0x708/0x8d0
Feb 12 16:26:49 bergson11630 [<ffffffff8187ec8c>] nf_iterate+0x6c/0xb0
Feb 12 16:26:49 bergson11630 [<ffffffff81930000>] ? br_handle_frame_finish+0x0/0x260
Feb 12 16:26:49 bergson11630 [<ffffffff8187ed7e>] nf_hook_slow+0xae/0x110
Feb 12 16:26:49 bergson11630 [<ffffffff81930000>] ? br_handle_frame_finish+0x0/0x260
Feb 12 16:26:49 bergson11630 [<ffffffff819303fd>] br_handle_frame+0x19d/0x240
Feb 12 16:26:49 bergson11630 [<ffffffff81930260>] ? br_handle_frame+0x0/0x240
Feb 12 16:26:49 bergson11630 [<ffffffff8183c287>] __netif_receive_skb+0x167/0x5d0
Feb 12 16:26:49 bergson11630 [<ffffffff816a6a4f>] ? bnx2_poll+0xcf/0x230
Feb 12 16:26:49 bergson11630 [<ffffffff8183c7a2>] process_backlog+0xb2/0x1a0
Feb 12 16:26:49 bergson11630 [<ffffffff8183ddb5>] net_rx_action+0xb5/0x220
Feb 12 16:26:49 bergson11630 [<ffffffff8105b555>] __do_softirq+0xb5/0x230
Feb 12 16:26:49 bergson11630 [<ffffffff81003edc>] call_softirq+0x1c/0x30
Feb 12 16:26:49 bergson11630 <EOI>  [<ffffffff81005f95>] ? do_softirq+0x65/0xa0
Feb 12 16:26:49 bergson11630 [<ffffffff8183e168>] netif_rx_ni+0x28/0x30
Feb 12 16:26:49 bergson11630 [<ffffffff81720c38>] tun_chr_aio_write+0x358/0x4d0
Feb 12 16:26:49 bergson11630 [<ffffffff819b4e05>] ? _raw_spin_lock_irq+0x15/0x20
Feb 12 16:26:49 bergson11630 [<ffffffff817208e0>] ? tun_chr_aio_write+0x0/0x4d0
Feb 12 16:26:49 bergson11630 [<ffffffff81124993>] do_sync_readv_writev+0xd3/0x110
Feb 12 16:26:49 bergson11630 [<ffffffffa0063986>] ? vmx_vcpu_put+0x16/0x40 [kvm_intel]
Feb 12 16:26:49 bergson11630 [<ffffffffa0015f58>] ? kvm_arch_vcpu_put+0x28/0x40 [kvm]
Feb 12 16:26:49 bergson11630 [<ffffffff8112479a>] ? rw_copy_check_uvector+0x7a/0x140
Feb 12 16:26:49 bergson11630 [<ffffffff811250a4>] do_readv_writev+0xd4/0x1f0
Feb 12 16:26:49 bergson11630 [<ffffffff81134a97>] ? do_vfs_ioctl+0xa7/0x560
Feb 12 16:26:49 bergson11630 [<ffffffff81125208>] vfs_writev+0x48/0x60
Feb 12 16:26:49 bergson11630 [<ffffffff81125331>] sys_writev+0x51/0xb0
Feb 12 16:26:49 bergson11630 [<ffffffff81003042>] system_call_fastpath+0x16/0x1b
Feb 12 16:26:49 bergson11630 FIX kmalloc-1024: Restoring 0xffff8802f8588446-0xffff8802f8588447=0x5a
Feb 12 16:26:49 bergson11630 
Feb 12 16:26:50 bergson11630 =============================================================================
Feb 12 16:26:50 bergson11630 BUG kmalloc-512: Object padding overwritten
Feb 12 16:26:50 bergson11630 ----------------------------------------------------------------------------- 




Pid: 0, comm: kworker/0:1 Tainted: G      D     2.6.37 #4
Call Trace:
 <IRQ>  [<ffffffff81113b13>] print_trailer+0x103/0x160
 [<ffffffff81114222>] check_bytes_and_report+0x102/0x130
 [<ffffffff81045e5a>] ? dequeue_task_fair+0x3a/0x90
 [<ffffffff811142fc>] check_object+0xac/0x260
 [<ffffffff81093ef5>] ? free_css_set_rcu+0x15/0x20
 [<ffffffff81114ed3>] free_debug_processing+0x133/0x260
 [<ffffffff81115154>] __slab_free+0x154/0x170
 [<ffffffff8111772e>] kfree+0xce/0x130
 [<ffffffff81093ef5>] free_css_set_rcu+0x15/0x20
 [<ffffffff810a313a>] __rcu_process_callbacks+0x11a/0x340
 [<ffffffff810a338d>] rcu_process_callbacks+0x2d/0xa0
 [<ffffffff8105b555>] __do_softirq+0xb5/0x230
 [<ffffffff81082f86>] ? tick_program_event+0x26/0x30
 [<ffffffff81003edc>] call_softirq+0x1c/0x30
 [<ffffffff81005f95>] do_softirq+0x65/0xa0
 [<ffffffff8105b415>] irq_exit+0x85/0x90
 [<ffffffff819b5930>] smp_apic_timer_interrupt+0x70/0x9b
 [<ffffffff81003993>] apic_timer_interrupt+0x13/0x20
 <EOI>  [<ffffffff813d4b56>] ? acpi_idle_enter_simple+0x13a/0x170
 [<ffffffff813d4b4f>] ? acpi_idle_enter_simple+0x133/0x170
 [<ffffffff813d4809>] acpi_idle_enter_bm+0xee/0x301
 [<ffffffff8180d591>] cpuidle_idle_call+0x91/0x140
 [<ffffffff810020d3>] cpu_idle+0x73/0x100
 [<ffffffff819ae5d8>] start_secondary+0x1b9/0x1be
FIX kmalloc-1024: Restoring 0xffff8807e7fff7de-0xffff8807e7fff7df=0x5a

=============================================================================
BUG kmalloc-2048: Object padding overwritten
-----------------------------------------------------------------------------

INFO: 0xffff8802f981b9f6-0xffff8802f981b9f7. First byte 0x0 instead of 0x5a
INFO: Allocated in sched_create_group+0x319/0x7c0 age=263926 cpu=2 pid=20454
INFO: Freed in skb_release_data+0xd0/0xe0 age=263927 cpu=0 pid=10313
INFO: Slab 0xffffea000a694540 objects=15 used=10 fp=0xffff8802f981eba8 flags=0x200000000004081
INFO: Object 0xffff8802f981b1b0 @offset=12720 fp=0x          (null)

Bytes b4 0xffff8802f981b1a0:  bf d1 04 00 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ��......ZZZZZZZZ
  Object 0xffff8802f981b1b0:  00 00 00 00 00 00 00 00 00 00 00 00 10 00 00 00 ................
  Object 0xffff8802f981b1c0:  c0 b1 81 f9 02 88 ff ff c0 b1 81 f9 02 88 ff ff �.�..���.�..��
  Object 0xffff8802f981b1d0:  d0 b1 81 f9 02 88 ff ff d0 b1 81 f9 02 88 ff ff б.�..��б.�..��
  Object 0xffff8802f981b1e0:  e0 b1 81 f9 02 88 ff ff e0 b1 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b1f0:  f0 b1 81 f9 02 88 ff ff f0 b1 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b200:  00 b2 81 f9 02 88 ff ff 00 b2 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b210:  10 b2 81 f9 02 88 ff ff 10 b2 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b220:  20 b2 81 f9 02 88 ff ff 20 b2 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b230:  30 b2 81 f9 02 88 ff ff 30 b2 81 f9 02 88 ff ff 0�.�..��0�.�..��
  Object 0xffff8802f981b240:  40 b2 81 f9 02 88 ff ff 40 b2 81 f9 02 88 ff ff @�.�..��@�.�..��
  Object 0xffff8802f981b250:  50 b2 81 f9 02 88 ff ff 50 b2 81 f9 02 88 ff ff P�.�..��P�.�..��
  Object 0xffff8802f981b260:  60 b2 81 f9 02 88 ff ff 60 b2 81 f9 02 88 ff ff `�.�..��`�.�..��
  Object 0xffff8802f981b270:  70 b2 81 f9 02 88 ff ff 70 b2 81 f9 02 88 ff ff p�.�..��p�.�..��
  Object 0xffff8802f981b280:  80 b2 81 f9 02 88 ff ff 80 b2 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b290:  90 b2 81 f9 02 88 ff ff 90 b2 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b2a0:  a0 b2 81 f9 02 88 ff ff a0 b2 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b2b0:  b0 b2 81 f9 02 88 ff ff b0 b2 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b2c0:  c0 b2 81 f9 02 88 ff ff c0 b2 81 f9 02 88 ff ff �.�..���.�..��
  Object 0xffff8802f981b2d0:  d0 b2 81 f9 02 88 ff ff d0 b2 81 f9 02 88 ff ff в.�..��в.�..��
  Object 0xffff8802f981b2e0:  e0 b2 81 f9 02 88 ff ff e0 b2 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b2f0:  f0 b2 81 f9 02 88 ff ff f0 b2 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b300:  00 b3 81 f9 02 88 ff ff 00 b3 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b310:  10 b3 81 f9 02 88 ff ff 10 b3 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b320:  20 b3 81 f9 02 88 ff ff 20 b3 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b330:  30 b3 81 f9 02 88 ff ff 30 b3 81 f9 02 88 ff ff 0�.�..��0�.�..��
  Object 0xffff8802f981b340:  40 b3 81 f9 02 88 ff ff 40 b3 81 f9 02 88 ff ff @�.�..��@�.�..��
  Object 0xffff8802f981b350:  50 b3 81 f9 02 88 ff ff 50 b3 81 f9 02 88 ff ff P�.�..��P�.�..��
  Object 0xffff8802f981b360:  60 b3 81 f9 02 88 ff ff 60 b3 81 f9 02 88 ff ff `�.�..��`�.�..��
  Object 0xffff8802f981b370:  70 b3 81 f9 02 88 ff ff 70 b3 81 f9 02 88 ff ff p�.�..��p�.�..��
  Object 0xffff8802f981b380:  80 b3 81 f9 02 88 ff ff 80 b3 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b390:  90 b3 81 f9 02 88 ff ff 90 b3 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b3a0:  a0 b3 81 f9 02 88 ff ff a0 b3 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b3b0:  b0 b3 81 f9 02 88 ff ff b0 b3 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b3c0:  c0 b3 81 f9 02 88 ff ff c0 b3 81 f9 02 88 ff ff �.�..���.�..��
  Object 0xffff8802f981b3d0:  d0 b3 81 f9 02 88 ff ff d0 b3 81 f9 02 88 ff ff г.�..��г.�..��
  Object 0xffff8802f981b3e0:  e0 b3 81 f9 02 88 ff ff e0 b3 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b3f0:  f0 b3 81 f9 02 88 ff ff f0 b3 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b400:  00 b4 81 f9 02 88 ff ff 00 b4 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b410:  10 b4 81 f9 02 88 ff ff 10 b4 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b420:  20 b4 81 f9 02 88 ff ff 20 b4 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b430:  30 b4 81 f9 02 88 ff ff 30 b4 81 f9 02 88 ff ff 0�.�..��0�.�..��
  Object 0xffff8802f981b440:  40 b4 81 f9 02 88 ff ff 40 b4 81 f9 02 88 ff ff @�.�..��@�.�..��
  Object 0xffff8802f981b450:  50 b4 81 f9 02 88 ff ff 50 b4 81 f9 02 88 ff ff P�.�..��P�.�..��
  Object 0xffff8802f981b460:  60 b4 81 f9 02 88 ff ff 60 b4 81 f9 02 88 ff ff `�.�..��`�.�..��
  Object 0xffff8802f981b470:  70 b4 81 f9 02 88 ff ff 70 b4 81 f9 02 88 ff ff p�.�..��p�.�..��
  Object 0xffff8802f981b480:  80 b4 81 f9 02 88 ff ff 80 b4 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b490:  90 b4 81 f9 02 88 ff ff 90 b4 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b4a0:  a0 b4 81 f9 02 88 ff ff a0 b4 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b4b0:  b0 b4 81 f9 02 88 ff ff b0 b4 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b4c0:  c0 b4 81 f9 02 88 ff ff c0 b4 81 f9 02 88 ff ff �.�..���.�..��
  Object 0xffff8802f981b4d0:  d0 b4 81 f9 02 88 ff ff d0 b4 81 f9 02 88 ff ff д.�..��д.�..��
  Object 0xffff8802f981b4e0:  e0 b4 81 f9 02 88 ff ff e0 b4 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b4f0:  f0 b4 81 f9 02 88 ff ff f0 b4 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b500:  00 b5 81 f9 02 88 ff ff 00 b5 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b510:  10 b5 81 f9 02 88 ff ff 10 b5 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b520:  20 b5 81 f9 02 88 ff ff 20 b5 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b530:  30 b5 81 f9 02 88 ff ff 30 b5 81 f9 02 88 ff ff 0�.�..��0�.�..��
  Object 0xffff8802f981b540:  40 b5 81 f9 02 88 ff ff 40 b5 81 f9 02 88 ff ff @�.�..��@�.�..��
  Object 0xffff8802f981b550:  50 b5 81 f9 02 88 ff ff 50 b5 81 f9 02 88 ff ff P�.�..��P�.�..��
  Object 0xffff8802f981b560:  60 b5 81 f9 02 88 ff ff 60 b5 81 f9 02 88 ff ff `�.�..��`�.�..��
  Object 0xffff8802f981b570:  70 b5 81 f9 02 88 ff ff 70 b5 81 f9 02 88 ff ff p�.�..��p�.�..��
  Object 0xffff8802f981b580:  80 b5 81 f9 02 88 ff ff 80 b5 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b590:  90 b5 81 f9 02 88 ff ff 90 b5 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b5a0:  a0 b5 81 f9 02 88 ff ff a0 b5 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b5b0:  b0 b5 81 f9 02 88 ff ff b0 b5 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b5c0:  c0 b5 81 f9 02 88 ff ff c0 b5 81 f9 02 88 ff ff �.�..���.�..��
  Object 0xffff8802f981b5d0:  d0 b5 81 f9 02 88 ff ff d0 b5 81 f9 02 88 ff ff е.�..��е.�..��
  Object 0xffff8802f981b5e0:  e0 b5 81 f9 02 88 ff ff e0 b5 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b5f0:  f0 b5 81 f9 02 88 ff ff f0 b5 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b600:  00 b6 81 f9 02 88 ff ff 00 b6 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b610:  10 b6 81 f9 02 88 ff ff 10 b6 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b620:  20 b6 81 f9 02 88 ff ff 20 b6 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b630:  30 b6 81 f9 02 88 ff ff 30 b6 81 f9 02 88 ff ff 0�.�..��0�.�..��
  Object 0xffff8802f981b640:  40 b6 81 f9 02 88 ff ff 40 b6 81 f9 02 88 ff ff @�.�..��@�.�..��
  Object 0xffff8802f981b650:  50 b6 81 f9 02 88 ff ff 50 b6 81 f9 02 88 ff ff P�.�..��P�.�..��
  Object 0xffff8802f981b660:  60 b6 81 f9 02 88 ff ff 60 b6 81 f9 02 88 ff ff `�.�..��`�.�..��
  Object 0xffff8802f981b670:  70 b6 81 f9 02 88 ff ff 70 b6 81 f9 02 88 ff ff p�.�..��p�.�..��
  Object 0xffff8802f981b680:  80 b6 81 f9 02 88 ff ff 80 b6 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b690:  90 b6 81 f9 02 88 ff ff 90 b6 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b6a0:  a0 b6 81 f9 02 88 ff ff a0 b6 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b6b0:  b0 b6 81 f9 02 88 ff ff b0 b6 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b6c0:  c0 b6 81 f9 02 88 ff ff c0 b6 81 f9 02 88 ff ff �.�..���.�..��
  Object 0xffff8802f981b6d0:  d0 b6 81 f9 02 88 ff ff d0 b6 81 f9 02 88 ff ff ж.�..��ж.�..��
  Object 0xffff8802f981b6e0:  e0 b6 81 f9 02 88 ff ff e0 b6 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b6f0:  f0 b6 81 f9 02 88 ff ff f0 b6 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b700:  00 b7 81 f9 02 88 ff ff 00 b7 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b710:  10 b7 81 f9 02 88 ff ff 10 b7 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b720:  20 b7 81 f9 02 88 ff ff 20 b7 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b730:  30 b7 81 f9 02 88 ff ff 30 b7 81 f9 02 88 ff ff 0�.�..��0�.�..��
  Object 0xffff8802f981b740:  40 b7 81 f9 02 88 ff ff 40 b7 81 f9 02 88 ff ff @�.�..��@�.�..��
  Object 0xffff8802f981b750:  50 b7 81 f9 02 88 ff ff 50 b7 81 f9 02 88 ff ff P�.�..��P�.�..��
  Object 0xffff8802f981b760:  60 b7 81 f9 02 88 ff ff 60 b7 81 f9 02 88 ff ff `�.�..��`�.�..��
  Object 0xffff8802f981b770:  70 b7 81 f9 02 88 ff ff 70 b7 81 f9 02 88 ff ff p�.�..��p�.�..��
  Object 0xffff8802f981b780:  80 b7 81 f9 02 88 ff ff 80 b7 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b790:  90 b7 81 f9 02 88 ff ff 90 b7 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b7a0:  a0 b7 81 f9 02 88 ff ff a0 b7 81 f9 02 88 ff ff .�.�..��.�.�..��
  Object 0xffff8802f981b7b0:  b0 b7 81 f9 02 88 ff ff b0 b7 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b7c0:  c0 b7 81 f9 02 88 ff ff c0 b7 81 f9 02 88 ff ff �.�..���.�..��
  Object 0xffff8802f981b7d0:  d0 b7 81 f9 02 88 ff ff d0 b7 81 f9 02 88 ff ff з.�..��з.�..��
  Object 0xffff8802f981b7e0:  e0 b7 81 f9 02 88 ff ff e0 b7 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b7f0:  f0 b7 81 f9 02 88 ff ff f0 b7 81 f9 02 88 ff ff ��.�..����.�..��
  Object 0xffff8802f981b800:  00 00 00 00 00 00 00 00 64 00 00 00 64 00 00 00 ........d...d...
  Object 0xffff8802f981b810:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b820:  00 00 00 00 00 00 00 00 28 b8 81 f9 02 88 ff ff ........(�.�..��
  Object 0xffff8802f981b830:  28 b8 81 f9 02 88 ff ff 38 b8 81 f9 02 88 ff ff (�.�..��8�.�..��
  Object 0xffff8802f981b840:  38 b8 81 f9 02 88 ff ff 00 00 00 00 00 00 00 00 8�.�..��........
  Object 0xffff8802f981b850:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b860:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b870:  00 1c c9 bf 00 88 ff ff 08 c9 ae fe 07 88 ff ff ..ɿ..��.ɮ�..��
  Object 0xffff8802f981b880:  00 02 20 00 00 00 ad de 20 e5 49 fe 07 88 ff ff ......��.�I�..��
  Object 0xffff8802f981b890:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b8a0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b8b0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b8c0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b8d0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b8e0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b8f0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b900:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b910:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b920:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b930:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b940:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b950:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b960:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b970:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b980:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b990:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
  Object 0xffff8802f981b9a0:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
 Redzone 0xffff8802f981b9b0:  cc cc cc cc cc cc cc cc                         ��������        
 Padding 0xffff8802f981b9f0:  5a 5a 5a 5a 5a 5a 00 00                         ZZZZZZ..        
Pid: 0, comm: swapper Tainted: G      D     2.6.37 #4
Call Trace:
 <IRQ>  [<ffffffff81113b13>] print_trailer+0x103/0x160
 [<ffffffff81114222>] check_bytes_and_report+0x102/0x130
 [<ffffffff811142fc>] check_object+0xac/0x260
 [<ffffffff8104aa43>] ? free_sched_group+0xd3/0x120
 [<ffffffff81114ed3>] free_debug_processing+0x133/0x260
 [<ffffffff81115154>] __slab_free+0x154/0x170
 [<ffffffff81078c4e>] ? hrtimer_try_to_cancel+0x7e/0xc0
 [<ffffffff8111772e>] kfree+0xce/0x130
 [<ffffffff8104aa43>] free_sched_group+0xd3/0x120
 [<ffffffff8104aaa5>] free_sched_group_rcu+0x15/0x20
 [<ffffffff810a313a>] __rcu_process_callbacks+0x11a/0x340
 [<ffffffff810a338d>] rcu_process_callbacks+0x2d/0xa0
 [<ffffffff8105b555>] __do_softirq+0xb5/0x230
 [<ffffffff81082f86>] ? tick_program_event+0x26/0x30
 [<ffffffff81003edc>] call_softirq+0x1c/0x30
 [<ffffffff81005f95>] do_softirq+0x65/0xa0
 [<ffffffff8105b415>] irq_exit+0x85/0x90
 [<ffffffff819b5930>] smp_apic_timer_interrupt+0x70/0x9b
 [<ffffffff81003993>] apic_timer_interrupt+0x13/0x20
 <EOI>  [<ffffffff813d49e4>] ? acpi_idle_enter_bm+0x2c9/0x301
 [<ffffffff813d49dd>] ? acpi_idle_enter_bm+0x2c2/0x301
 [<ffffffff8180d591>] cpuidle_idle_call+0x91/0x140
 [<ffffffff810020d3>] cpu_idle+0x73/0x100
 [<ffffffff8196a182>] rest_init+0x72/0x80
 [<ffffffff81f01dab>] start_kernel+0x397/0x3a2
 [<ffffffff81f0132a>] x86_64_start_reservations+0x131/0x135
 [<ffffffff81f01434>] x86_64_start_kernel+0x106/0x115
FIX kmalloc-2048: Restoring 0xffff8802f981b9f6-0xffff8802f981b9f7=0x5a
Comment 31 Avi Kivity 2011-02-13 16:11:53 UTC 
What happens if you disable (rmmod) all netfilter modules?
Comment 32 prochazka 2011-02-13 16:38:03 UTC 
Can you explain your idea ?
In fact, we are using DNAT and SNAT rules to redirect and access VM Guest ( VNC and RDP ) . And to reproduce bug, connection seems to be important : 

1 - Run VMs and connect to them on only one server : cannot reproduce bug
2 - Two server , one which redirect VNC and RDP connection and execute VM
                 second, only execute VM .
=> bug can reproduce when connect to a VM on second server.

So it's difficult to me to desactivate netfilter modules to reproduce.

Nicolas
Comment 33 Marcelo Tosatti 2011-02-13 19:39:09 UTC 
So something else appears to be corrupting memory (i verified and its unlikely to be KVM). Perhaps go back to 2.6.36.3 in an attempt to isolate the issue?
Comment 34 prochazka 2011-02-13 20:13:53 UTC 
When I said  it worked with 2.6.36 , in fact, I not used brigde and netfilter with this test , 
so now as avi says, perhaps it's more brctl or netfilter bug and not kvm.

If you think it's not kvm bug, close. And now my way seems to be very long 

Regards, 
Nicolas
Comment 35 Avi Kivity 2011-02-14 14:52:24 UTC 
Even if it isn't a kvm bug, it needs to be fixed.
Comment 36 prochazka 2011-02-14 15:50:02 UTC 
I agree with you but i must resend this problem to kernel module bridge or netfilter, I must isolate the issue .

Regards, 
Nicolas Prochazka.
Comment 37 prochazka 2011-02-14 15:51:14 UTC 
Sorry I do not read your post to netfilter devel list.
Nicolas
Comment 38 Avi Kivity 2011-02-14 16:02:06 UTC 
Please revert ca44ac386181ba7 and try again (with netfilter enabled).
Comment 39 Taisuke Yamada 2011-04-21 11:45:45 UTC 
Is this issue resolved?
I'm seeing almost the same issue with 2.6.38 + qemu-kvm-0.14.0 (on NEC Express5800/110Ge server), except that my backtrace is calling paging64_sync_page instead of paging32_sync_page.
Comment 40 Avi Kivity 2011-04-21 12:13:33 UTC 
(In reply to comment #39)
> Is this issue resolved?
> I'm seeing almost the same issue with 2.6.38 + qemu-kvm-0.14.0 (on NEC
> Express5800/110Ge server), except that my backtrace is calling
> paging64_sync_page instead of paging32_sync_page.


Do you also have netfilter enabled on the host?

What happens if you enable debug options as per comment #29?
Comment 41 prochazka 2011-04-21 12:39:11 UTC 
Hi, 
I never test #38 from avi kivity suggest, 
and now i suppres all bridge ( brctl) and netfilter configuration from my server, 
i'm using openvswitch without this problem.

Regards, 
Nicolas Prochazka.