The below commit can fix it.

commit 370777daab3f024f1645177039955088e2e9ae73
Author: Radim Krčmář <rkrcmar@redhat.com>
Date:   Fri Jul 3 15:49:28 2015 +0200

    KVM: VMX: fix vmwrite to invalid VMCS
    
    fpu_activate is called outside of vcpu_load(), which means it should not
    touch VMCS, but fpu_activate needs to.  Avoid the call by moving it to a
    point where we know that the guest needs eager FPU and VMCS is loaded.
    
    This will get rid of the following trace
    
     vmwrite error: reg 6800 value 0 (err 1)
      [<ffffffff8162035b>] dump_stack+0x19/0x1b
      [<ffffffffa046c701>] vmwrite_error+0x2c/0x2e [kvm_intel]
      [<ffffffffa045f26f>] vmcs_writel+0x1f/0x30 [kvm_intel]
      [<ffffffffa04617e5>] vmx_fpu_activate.part.61+0x45/0xb0 [kvm_intel]
      [<ffffffffa0461865>] vmx_fpu_activate+0x15/0x20 [kvm_intel]
      [<ffffffffa0560b91>] kvm_arch_vcpu_create+0x51/0x70 [kvm]
      [<ffffffffa0548011>] kvm_vm_ioctl+0x1c1/0x760 [kvm]
      [<ffffffff8118b55a>] ? handle_mm_fault+0x49a/0xec0
      [<ffffffff811e47d5>] do_vfs_ioctl+0x2e5/0x4c0
      [<ffffffff8127abbe>] ? file_has_perm+0xae/0xc0
      [<ffffffff811e4a51>] SyS_ioctl+0xa1/0xc0
      [<ffffffff81630949>] system_call_fastpath+0x16/0x1b
    
    (Note: we also unconditionally activate FPU in vmx_vcpu_reset(), so the
     removed code added nothing.)
    
    Fixes: c447e76b4cab ("kvm/fpu: Enable eager restore kvm FPU for MPX")
    Cc: <stable@vger.kernel.org>
    Reported-by: Vlastimil Holer <vlastimil.holer@gmail.com>
    Signed-off-by: Radim Krčmář <rkrcmar@redhat.com>
    Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Created attachment 185681 [details]
Test program 2 (C99)

You mean "can" as in "I think it does" or "it did for me"?

And anyway, it seems to only fix the most proximate cause of the crash. My biggest worry is that KVM_SET_USER_MEMORY_REGION ioctls with guest_phys_addr around the 0xfff00000 to 0xffff0000 range seem not to "register"; starting the VM looks like as if the region wasn't placed there.

I attach test program 2. Running that on my system with 0x44000 as an argument outputs "halted" (as expected), but 0x45000 and larger multiples of 0x1000 give "internal error, subcode 1".

Created attachment 185691 [details]
Test program 2 (C99) [non-oopsing version]

We hit the same issue with kernel 3.18.19.

After some debugging, I see that the first test program that felix attached, causes kvm_x86_ops->vcpu_create to return -EEXIST instead of a valid vcpu pointer. As a result, the call to kvm_x86_ops->fpu_activate tries to access an invalid pointer, and causes a NULL pointer dereference.

The suggested fix was delivered in kernel 4.2. Although it was tagged as "stable", I don't see that it was backported to earlier kernels. I believe that the fix addresses a different issue, in which the vcpu pointer is valid, but further VMCS write has a problem (this is my understanding). But, of course, this fix will address also the issue that felix reported. Although for the latter, a simpler fix would suffice:

--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -7012,20 +7012,24 @@ struct kvm_vcpu *kvm_arch_vcpu_create(struct kvm *kvm,
                                                unsigned int id)
 {
        struct kvm_vcpu *vcpu;

        if (check_tsc_unstable() && atomic_read(&kvm->online_vcpus) != 0)
                printk_once(KERN_WARNING
                "kvm: SMP vm created on host with unstable TSC; "
                "guest TSC will not be reliable\n");

        vcpu = kvm_x86_ops->vcpu_create(kvm, id);
+       if (IS_ERR(vcpu)) {
+               pr_err("kvm_x86_ops->vcpu_create id=%u err=%ld\n", id, PTR_ERR(vcpu));
+               return vcpu;
+       }

        /*
         * Activate fpu unconditionally in case the guest needs eager FPU.  It will be
         * deactivated soon if it doesn't.
         */
        kvm_x86_ops->fpu_activate(vcpu);
        return vcpu;
 }