Bug 42980

   0:	89 d0                	mov    %edx,%eax
   2:	8d 4c ff 0c          	lea    0xc(%rdi,%rdi,8),%ecx
   6:	4d 89 e0             	mov    %r12,%r8
   9:	48 d3 e8             	shr    %cl,%rax
   c:	4c 03 45 a8          	add    -0x58(%rbp),%r8
  10:	25 ff 01 00 00       	and    $0x1ff,%eax
  15:	41 39 f6             	cmp    %esi,%r14d
  18:	89 45 bc             	mov    %eax,-0x44(%rbp)
  1b:	89 c0                	mov    %eax,%eax
  1d:	49 8d 04 c0          	lea    (%r8,%rax,8),%rax
  21:	48 89 45 b0          	mov    %rax,-0x50(%rbp)
  25:	0f 84 e1 00 00 00    	je     0x10c
  2b:	4c 8b 00             	mov    (%rax),%r8
  2e:	41 f6 c0 01          	test   $0x1,%r8b
  32:	74 40                	je     0x74
  34:	4c 8b 0d 89 80 01 00 	mov    0x18089(%rip),%r9        # 0x180c4
  3b:	4d 89 c2             	mov    %r8,%r10

Appears to be __direct_map()'s

		if (!is_shadow_present_pte(*iterator.sptep)) {
			u64 base_addr = iterator.addr;

%rax is 0xffff87ffffffffff. That is one less than the base of the direct map of all physical memory.  So it looks like the code


static bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator)
{
	if (iterator->level < PT_PAGE_TABLE_LEVEL)
		return false;

	iterator->index = SHADOW_PT_INDEX(iterator->addr, iterator->level);
	iterator->sptep	= ((u64 *)__va(iterator->shadow_addr)) + iterator->index;
	return true;
}

saw iterator->shadow_addr == -1ULL.

That might be INVALID_PAGE assigned to pae_root (but that is masked out in shadow_walk_init()) or a stray -1 due to a completely unrelated bug.

Anything interesting about how this was triggered?

IIRC, it was pretty out of the blue. I might have had one or both of two KVMs running in the background at the time:
- 64-bit Gentoo with a Radeon 5850 passthrough'd (VT-d)
- 32-bit Ubuntu with a nested 32-bit KVM

You're a brave one.

It wasn't the nested one (at least, it wasn't running in the guest's guest at the moment of the crash), but it might be related.

I suppose I should mention I'd been running both of these stable for at least a month now (and the GPU passthrough for nearly a full year). One factor that might (or might not) be related - the GPU fan recently died. When this crash took me down, I removed the GPU, so I won't be able to do any further testing with that setup (unless I find another similar GPU at a good price).

vcpu_enter_guest()
  kvm_mmu_reload() // now root_hpa is valid
  inject_pending_event()
    vmx_interrupt_allowed()
      nested_vmx_vmexit()
        load_vmcs12_host_state()
          kvm_mmu_reset_context() // root_hpa now invalid
  kvm_guest_enter()
  ... page fault because root_hpa is invalid, oops

Created attachment 73244 [details]
Fix

Please test the attached patch.

Is there anything I can do to reproduce the problem condition for the test? It seems to only occur about once every 6 months normally.

Try running 

  while :; do :; done

in the nested (L2) guest, and ping -f the L1 guest from the host.

The while/ping thing doesn't reproduce it even before the patch. :(

For what it's worth, no crashes in over a month. But it wasn't common enough that it can't be coincidence either...

A patch referencing this bug report has been merged in Linux v3.5-rc1:

commit d8368af8b46b904def42a0f341d2f4f29001fa77
Author: Avi Kivity <avi@redhat.com>
Date:   Mon May 14 18:07:56 2012 +0300

    KVM: Fix mmu_reload() clash with nested vmx event injection

Sorry I didn't report it sooner, but I have had the same crash since June, with this patch. :(

Which kernel ?

I'm not sure if it was 3.4.0, 3.4.3, or 3.4.4. Since May 17, I have been building all my kernels (including those) with this patch applied.

3.4.0: http://luke.dashjr.org/tmp/code/20120624_002.jpg

Thanks

I just hit this.

Host:  Intel DQ67SW, Core i7 2600, 24GB RAM
BIOS:  SWQ6710H.86A.0065.2012.0917.1519

Host OS:  Fedora 17
          kernel-3.6.6-1.fc17.x86_64
          qemu-kvm-1.2.0-20.fc17.x86_64

L1 Guest OS:  RHEL 6.3
              kernel-2.6.32-279.14.1.el6.x86_64
              qemu-kvm-rhev-0.12.1.2-2.295.el6_3.5.x86_64

L2 Guest OS:  RHEL 6.3
              kernel-2.6.32-279.14.1.el6.x86_64

I was running a Pulp sync between a couple of L2 guests when this occurred, which presumably generated quite a bit of traffic across the virtual bridges.  I am using Open vSwitch for all of the bridges on the host OS.  The virtualized RHEV hypervisors use standard Linux bridges.

Please let me know if I can provide any additional information to help track this down.

(In reply to comment #11)
> A patch referencing this bug report has been merged in Linux v3.5-rc1:
> 
> commit d8368af8b46b904def42a0f341d2f4f29001fa77
> Author: Avi Kivity <avi@redhat.com>
> Date:   Mon May 14 18:07:56 2012 +0300
> 
>     KVM: Fix mmu_reload() clash with nested vmx event injection

Silly question.  Is this patch applicable to the physical host, the L1 guest (virtualized hypervisor), or both?

On 11/18/2012 12:10 AM, bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=42980
> 
> 
> 
> 
> 
> --- Comment #18 from Ian Pilcher <arequipeno@gmail.com>  2012-11-17 22:10:45
> ---
> (In reply to comment #11)
>> A patch referencing this bug report has been merged in Linux v3.5-rc1:
>> 
>> commit d8368af8b46b904def42a0f341d2f4f29001fa77
>> Author: Avi Kivity <avi@redhat.com>
>> Date:   Mon May 14 18:07:56 2012 +0300
>> 
>>     KVM: Fix mmu_reload() clash with nested vmx event injection
> 
> Silly question.  Is this patch applicable to the physical host, the L1 guest
> (virtualized hypervisor), or both?
> 

The physical host.  If you want to run a hypervisor in L2, you need to
apply it to L1 as well.

(In reply to comment #19)
> The physical host.  If you want to run a hypervisor in L2, you need to
> apply it to L1 as well.

OK.  If I'm parsing that correctly, it sounds like backporting the patch to the RHEL 6 kernel, so I could run it in the L1 hypervisors, wouldn't help anything.

Bummer.

Any ideas on how I can make this environment stable?

I see that Luke-Jr is also on a DQ67SW, and he's doing PCI passthrough.  I do have VT-d enabled, although I'm not actually doing any PCI-passthrough.  I that something that could be related to this?

I just hit this again (I think).  Pretty much out of the blue, with a bunch of VMs running, including at least 2 nested guests.

I have been trying to get a kdump of this, and I believe that I was at least somewhat successful.  The system didn't dump automatically, but I was able to get it to do so by hitting alt-sysrq-c.  The vmcore file is 3.7G, so suggestions as to a place to post it publicly would be appreciated.

Please tell against a newer kernel. This bug seems obsolete to me as of
kernel versions released in 2014 time frame.
Cheers Nick