+1

.mailmap

reviewed

··· 336 336 Oleksij Rempel <linux@rempel-privat.de> <fixed-term.Oleksij.Rempel@de.bosch.com> 337 337 Oleksij Rempel <linux@rempel-privat.de> <o.rempel@pengutronix.de> 338 338 Oleksij Rempel <linux@rempel-privat.de> <ore@pengutronix.de> 339 339 + Oliver Upton <oliver.upton@linux.dev> <oupton@google.com> 339 340 Pali Rohár <pali@kernel.org> <pali.rohar@gmail.com> 340 341 Paolo 'Blaisorblade' Giarrusso <blaisorblade@yahoo.it> 341 342 Patrick Mochel <mochel@digitalimplant.org>

+5

Documentation/admin-guide/cgroup-v2.rst

reviewed

··· 1355 1355 pagetables 1356 1356 Amount of memory allocated for page tables. 1357 1357 1358 1358 + sec_pagetables 1359 1359 + Amount of memory allocated for secondary page tables, 1360 1360 + this currently includes KVM mmu allocations on x86 1361 1361 + and arm64. 1362 1362 + 1358 1363 percpu (npn) 1359 1364 Amount of memory used for storing per-cpu kernel 1360 1365 data structures.

+4

Documentation/filesystems/proc.rst

reviewed

··· 982 982 SUnreclaim: 142336 kB 983 983 KernelStack: 11168 kB 984 984 PageTables: 20540 kB 985 985 + SecPageTables: 0 kB 985 986 NFS_Unstable: 0 kB 986 987 Bounce: 0 kB 987 988 WritebackTmp: 0 kB ··· 1091 1090 Memory consumed by the kernel stacks of all tasks 1092 1091 PageTables 1093 1092 Memory consumed by userspace page tables 1093 1093 + SecPageTables 1094 1094 + Memory consumed by secondary page tables, this currently 1095 1095 + currently includes KVM mmu allocations on x86 and arm64. 1094 1096 NFS_Unstable 1095 1097 Always zero. Previous counted pages which had been written to 1096 1098 the server, but has not been committed to stable storage.

+6 -107

Documentation/virt/kvm/api.rst

reviewed

··· 4074 4074 4.97 KVM_X86_SET_MSR_FILTER 4075 4075 ---------------------------- 4076 4076 4077 4077 - :Capability: KVM_X86_SET_MSR_FILTER 4077 4077 + :Capability: KVM_CAP_X86_MSR_FILTER 4078 4078 :Architectures: x86 4079 4079 :Type: vm ioctl 4080 4080 :Parameters: struct kvm_msr_filter ··· 4173 4173 allows user space to deflect and potentially handle various MSR accesses 4174 4174 into user space. 4175 4175 4176 4176 - If a vCPU is in running state while this ioctl is invoked, the vCPU may 4177 4177 - experience inconsistent filtering behavior on MSR accesses. 4176 4176 + Note, invoking this ioctl while a vCPU is running is inherently racy. However, 4177 4177 + KVM does guarantee that vCPUs will see either the previous filter or the new 4178 4178 + filter, e.g. MSRs with identical settings in both the old and new filter will 4179 4179 + have deterministic behavior. 4178 4180 4179 4181 4.98 KVM_CREATE_SPAPR_TCE_64 4180 4182 ---------------------------- ··· 5289 5287 authentication tag all of which are needed to decrypt the dump at a 5290 5288 later time. 5291 5289 5292 5292 - 5293 5293 - 4.126 KVM_X86_SET_MSR_FILTER 5294 5294 - ---------------------------- 5295 5295 - 5296 5296 - :Capability: KVM_CAP_X86_MSR_FILTER 5297 5297 - :Architectures: x86 5298 5298 - :Type: vm ioctl 5299 5299 - :Parameters: struct kvm_msr_filter 5300 5300 - :Returns: 0 on success, < 0 on error 5301 5301 - 5302 5302 - :: 5303 5303 - 5304 5304 - struct kvm_msr_filter_range { 5305 5305 - #define KVM_MSR_FILTER_READ (1 << 0) 5306 5306 - #define KVM_MSR_FILTER_WRITE (1 << 1) 5307 5307 - __u32 flags; 5308 5308 - __u32 nmsrs; /* number of msrs in bitmap */ 5309 5309 - __u32 base; /* MSR index the bitmap starts at */ 5310 5310 - __u8 *bitmap; /* a 1 bit allows the operations in flags, 0 denies */ 5311 5311 - }; 5312 5312 - 5313 5313 - #define KVM_MSR_FILTER_MAX_RANGES 16 5314 5314 - struct kvm_msr_filter { 5315 5315 - #define KVM_MSR_FILTER_DEFAULT_ALLOW (0 << 0) 5316 5316 - #define KVM_MSR_FILTER_DEFAULT_DENY (1 << 0) 5317 5317 - __u32 flags; 5318 5318 - struct kvm_msr_filter_range ranges[KVM_MSR_FILTER_MAX_RANGES]; 5319 5319 - }; 5320 5320 - 5321 5321 - flags values for ``struct kvm_msr_filter_range``: 5322 5322 - 5323 5323 - ``KVM_MSR_FILTER_READ`` 5324 5324 - 5325 5325 - Filter read accesses to MSRs using the given bitmap. A 0 in the bitmap 5326 5326 - indicates that a read should immediately fail, while a 1 indicates that 5327 5327 - a read for a particular MSR should be handled regardless of the default 5328 5328 - filter action. 5329 5329 - 5330 5330 - ``KVM_MSR_FILTER_WRITE`` 5331 5331 - 5332 5332 - Filter write accesses to MSRs using the given bitmap. A 0 in the bitmap 5333 5333 - indicates that a write should immediately fail, while a 1 indicates that 5334 5334 - a write for a particular MSR should be handled regardless of the default 5335 5335 - filter action. 5336 5336 - 5337 5337 - ``KVM_MSR_FILTER_READ | KVM_MSR_FILTER_WRITE`` 5338 5338 - 5339 5339 - Filter both read and write accesses to MSRs using the given bitmap. A 0 5340 5340 - in the bitmap indicates that both reads and writes should immediately fail, 5341 5341 - while a 1 indicates that reads and writes for a particular MSR are not 5342 5342 - filtered by this range. 5343 5343 - 5344 5344 - flags values for ``struct kvm_msr_filter``: 5345 5345 - 5346 5346 - ``KVM_MSR_FILTER_DEFAULT_ALLOW`` 5347 5347 - 5348 5348 - If no filter range matches an MSR index that is getting accessed, KVM will 5349 5349 - fall back to allowing access to the MSR. 5350 5350 - 5351 5351 - ``KVM_MSR_FILTER_DEFAULT_DENY`` 5352 5352 - 5353 5353 - If no filter range matches an MSR index that is getting accessed, KVM will 5354 5354 - fall back to rejecting access to the MSR. In this mode, all MSRs that should 5355 5355 - be processed by KVM need to explicitly be marked as allowed in the bitmaps. 5356 5356 - 5357 5357 - This ioctl allows user space to define up to 16 bitmaps of MSR ranges to 5358 5358 - specify whether a certain MSR access should be explicitly filtered for or not. 5359 5359 - 5360 5360 - If this ioctl has never been invoked, MSR accesses are not guarded and the 5361 5361 - default KVM in-kernel emulation behavior is fully preserved. 5362 5362 - 5363 5363 - Calling this ioctl with an empty set of ranges (all nmsrs == 0) disables MSR 5364 5364 - filtering. In that mode, ``KVM_MSR_FILTER_DEFAULT_DENY`` is invalid and causes 5365 5365 - an error. 5366 5366 - 5367 5367 - As soon as the filtering is in place, every MSR access is processed through 5368 5368 - the filtering except for accesses to the x2APIC MSRs (from 0x800 to 0x8ff); 5369 5369 - x2APIC MSRs are always allowed, independent of the ``default_allow`` setting, 5370 5370 - and their behavior depends on the ``X2APIC_ENABLE`` bit of the APIC base 5371 5371 - register. 5372 5372 - 5373 5373 - If a bit is within one of the defined ranges, read and write accesses are 5374 5374 - guarded by the bitmap's value for the MSR index if the kind of access 5375 5375 - is included in the ``struct kvm_msr_filter_range`` flags. If no range 5376 5376 - cover this particular access, the behavior is determined by the flags 5377 5377 - field in the kvm_msr_filter struct: ``KVM_MSR_FILTER_DEFAULT_ALLOW`` 5378 5378 - and ``KVM_MSR_FILTER_DEFAULT_DENY``. 5379 5379 - 5380 5380 - Each bitmap range specifies a range of MSRs to potentially allow access on. 5381 5381 - The range goes from MSR index [base .. base+nmsrs]. The flags field 5382 5382 - indicates whether reads, writes or both reads and writes are filtered 5383 5383 - by setting a 1 bit in the bitmap for the corresponding MSR index. 5384 5384 - 5385 5385 - If an MSR access is not permitted through the filtering, it generates a 5386 5386 - #GP inside the guest. When combined with KVM_CAP_X86_USER_SPACE_MSR, that 5387 5387 - allows user space to deflect and potentially handle various MSR accesses 5388 5388 - into user space. 5389 5389 - 5390 5390 - Note, invoking this ioctl with a vCPU is running is inherently racy. However, 5391 5391 - KVM does guarantee that vCPUs will see either the previous filter or the new 5392 5392 - filter, e.g. MSRs with identical settings in both the old and new filter will 5393 5393 - have deterministic behavior. 5394 5394 - 5395 5395 - 4.127 KVM_XEN_HVM_SET_ATTR 5290 5290 + 4.126 KVM_XEN_HVM_SET_ATTR 5396 5291 -------------------------- 5397 5292 5398 5293 :Capability: KVM_CAP_XEN_HVM / KVM_XEN_HVM_CONFIG_SHARED_INFO

+1 -27

Documentation/virt/kvm/vcpu-requests.rst

reviewed

··· 97 97 This means general bitops, like those documented in [atomic-ops]_ could 98 98 also be used, e.g. :: 99 99 100 100 - clear_bit(KVM_REQ_UNHALT & KVM_REQUEST_MASK, &vcpu->requests); 100 100 + clear_bit(KVM_REQ_UNBLOCK & KVM_REQUEST_MASK, &vcpu->requests); 101 101 102 102 However, VCPU request users should refrain from doing so, as it would 103 103 break the abstraction. The first 8 bits are reserved for architecture ··· 125 125 example from timer handlers that run on the host on behalf of a vCPU, 126 126 or in order to update the interrupt routing and ensure that assigned 127 127 devices will wake up the vCPU. 128 128 - 129 129 - KVM_REQ_UNHALT 130 130 - 131 131 - This request may be made from the KVM common function kvm_vcpu_block(), 132 132 - which is used to emulate an instruction that causes a CPU to halt until 133 133 - one of an architectural specific set of events and/or interrupts is 134 134 - received (determined by checking kvm_arch_vcpu_runnable()). When that 135 135 - event or interrupt arrives kvm_vcpu_block() makes the request. This is 136 136 - in contrast to when kvm_vcpu_block() returns due to any other reason, 137 137 - such as a pending signal, which does not indicate the VCPU's halt 138 138 - emulation should stop, and therefore does not make the request. 139 128 140 129 KVM_REQ_OUTSIDE_GUEST_MODE 141 130 ··· 285 296 architecture dependent. kvm_vcpu_block() calls kvm_arch_vcpu_runnable() 286 297 to check if it should awaken. One reason to do so is to provide 287 298 architectures a function where requests may be checked if necessary. 288 288 - 289 289 - Clearing Requests 290 290 - ----------------- 291 291 - 292 292 - Generally it only makes sense for the receiving VCPU thread to clear a 293 293 - request. However, in some circumstances, such as when the requesting 294 294 - thread and the receiving VCPU thread are executed serially, such as when 295 295 - they are the same thread, or when they are using some form of concurrency 296 296 - control to temporarily execute synchronously, then it's possible to know 297 297 - that the request may be cleared immediately, rather than waiting for the 298 298 - receiving VCPU thread to handle the request in VCPU RUN. The only current 299 299 - examples of this are kvm_vcpu_block() calls made by VCPUs to block 300 300 - themselves. A possible side-effect of that call is to make the 301 301 - KVM_REQ_UNHALT request, which may then be cleared immediately when the 302 302 - VCPU returns from the call. 303 299 304 300 References 305 301 ==========

-1

arch/arm64/kvm/arm.c

reviewed

··· 666 666 667 667 kvm_vcpu_halt(vcpu); 668 668 vcpu_clear_flag(vcpu, IN_WFIT); 669 669 - kvm_clear_request(KVM_REQ_UNHALT, vcpu); 670 669 671 670 preempt_disable(); 672 671 vgic_v4_load(vcpu);

+32 -4

arch/arm64/kvm/mmu.c

reviewed

··· 92 92 static void *stage2_memcache_zalloc_page(void *arg) 93 93 { 94 94 struct kvm_mmu_memory_cache *mc = arg; 95 95 + void *virt; 95 96 96 97 /* Allocated with __GFP_ZERO, so no need to zero */ 97 97 - return kvm_mmu_memory_cache_alloc(mc); 98 98 + virt = kvm_mmu_memory_cache_alloc(mc); 99 99 + if (virt) 100 100 + kvm_account_pgtable_pages(virt, 1); 101 101 + return virt; 98 102 } 99 103 100 104 static void *kvm_host_zalloc_pages_exact(size_t size) 101 105 { 102 106 return alloc_pages_exact(size, GFP_KERNEL_ACCOUNT | __GFP_ZERO); 107 107 + } 108 108 + 109 109 + static void *kvm_s2_zalloc_pages_exact(size_t size) 110 110 + { 111 111 + void *virt = kvm_host_zalloc_pages_exact(size); 112 112 + 113 113 + if (virt) 114 114 + kvm_account_pgtable_pages(virt, (size >> PAGE_SHIFT)); 115 115 + return virt; 116 116 + } 117 117 + 118 118 + static void kvm_s2_free_pages_exact(void *virt, size_t size) 119 119 + { 120 120 + kvm_account_pgtable_pages(virt, -(size >> PAGE_SHIFT)); 121 121 + free_pages_exact(virt, size); 103 122 } 104 123 105 124 static void kvm_host_get_page(void *addr) ··· 129 110 static void kvm_host_put_page(void *addr) 130 111 { 131 112 put_page(virt_to_page(addr)); 113 113 + } 114 114 + 115 115 + static void kvm_s2_put_page(void *addr) 116 116 + { 117 117 + struct page *p = virt_to_page(addr); 118 118 + /* Dropping last refcount, the page will be freed */ 119 119 + if (page_count(p) == 1) 120 120 + kvm_account_pgtable_pages(addr, -1); 121 121 + put_page(p); 132 122 } 133 123 134 124 static int kvm_host_page_count(void *addr) ··· 653 625 654 626 static struct kvm_pgtable_mm_ops kvm_s2_mm_ops = { 655 627 .zalloc_page = stage2_memcache_zalloc_page, 656 656 - .zalloc_pages_exact = kvm_host_zalloc_pages_exact, 657 657 - .free_pages_exact = free_pages_exact, 628 628 + .zalloc_pages_exact = kvm_s2_zalloc_pages_exact, 629 629 + .free_pages_exact = kvm_s2_free_pages_exact, 658 630 .get_page = kvm_host_get_page, 659 659 - .put_page = kvm_host_put_page, 631 631 + .put_page = kvm_s2_put_page, 660 632 .page_count = kvm_host_page_count, 661 633 .phys_to_virt = kvm_host_va, 662 634 .virt_to_phys = kvm_host_pa,

+2 -4

arch/mips/kvm/emulate.c

reviewed

··· 955 955 kvm_vcpu_halt(vcpu); 956 956 957 957 /* 958 958 - * We we are runnable, then definitely go off to user space to 958 958 + * We are runnable, then definitely go off to user space to 959 959 * check if any I/O interrupts are pending. 960 960 */ 961 961 - if (kvm_check_request(KVM_REQ_UNHALT, vcpu)) { 962 962 - kvm_clear_request(KVM_REQ_UNHALT, vcpu); 961 961 + if (kvm_arch_vcpu_runnable(vcpu)) 963 962 vcpu->run->exit_reason = KVM_EXIT_IRQ_WINDOW_OPEN; 964 964 - } 965 963 } 966 964 967 965 return EMULATE_DONE;

-1

arch/powerpc/kvm/book3s_pr.c

reviewed

··· 499 499 if (msr & MSR_POW) { 500 500 if (!vcpu->arch.pending_exceptions) { 501 501 kvm_vcpu_halt(vcpu); 502 502 - kvm_clear_request(KVM_REQ_UNHALT, vcpu); 503 502 vcpu->stat.generic.halt_wakeup++; 504 503 505 504 /* Unset POW bit after we woke up */

-1

arch/powerpc/kvm/book3s_pr_papr.c

reviewed

··· 393 393 case H_CEDE: 394 394 kvmppc_set_msr_fast(vcpu, kvmppc_get_msr(vcpu) | MSR_EE); 395 395 kvm_vcpu_halt(vcpu); 396 396 - kvm_clear_request(KVM_REQ_UNHALT, vcpu); 397 396 vcpu->stat.generic.halt_wakeup++; 398 397 return EMULATE_DONE; 399 398 case H_LOGICAL_CI_LOAD:

-1

arch/powerpc/kvm/booke.c

reviewed

··· 719 719 if (vcpu->arch.shared->msr & MSR_WE) { 720 720 local_irq_enable(); 721 721 kvm_vcpu_halt(vcpu); 722 722 - kvm_clear_request(KVM_REQ_UNHALT, vcpu); 723 722 hard_irq_disable(); 724 723 725 724 kvmppc_set_exit_type(vcpu, EMULATED_MTMSRWE_EXITS);

-1

arch/powerpc/kvm/powerpc.c

reviewed

··· 239 239 case EV_HCALL_TOKEN(EV_IDLE): 240 240 r = EV_SUCCESS; 241 241 kvm_vcpu_halt(vcpu); 242 242 - kvm_clear_request(KVM_REQ_UNHALT, vcpu); 243 242 break; 244 243 default: 245 244 r = EV_UNIMPLEMENTED;

-1

arch/riscv/kvm/vcpu_insn.c

reviewed

··· 191 191 kvm_vcpu_srcu_read_unlock(vcpu); 192 192 kvm_vcpu_halt(vcpu); 193 193 kvm_vcpu_srcu_read_lock(vcpu); 194 194 - kvm_clear_request(KVM_REQ_UNHALT, vcpu); 195 194 } 196 195 } 197 196

-2

arch/s390/kvm/kvm-s390.c

reviewed

··· 4343 4343 goto retry; 4344 4344 } 4345 4345 4346 4346 - /* nothing to do, just clear the request */ 4347 4347 - kvm_clear_request(KVM_REQ_UNHALT, vcpu); 4348 4346 /* we left the vsie handler, nothing to do, just clear the request */ 4349 4347 kvm_clear_request(KVM_REQ_VSIE_RESTART, vcpu); 4350 4348

+18 -4

arch/x86/include/asm/hyperv-tlfs.h

reviewed

··· 138 138 #define HV_X64_NESTED_GUEST_MAPPING_FLUSH BIT(18) 139 139 #define HV_X64_NESTED_MSR_BITMAP BIT(19) 140 140 141 141 + /* Nested features #2. These are HYPERV_CPUID_NESTED_FEATURES.EBX bits. */ 142 142 + #define HV_X64_NESTED_EVMCS1_PERF_GLOBAL_CTRL BIT(0) 143 143 + 141 144 /* 142 145 * This is specific to AMD and specifies that enlightened TLB flush is 143 146 * supported. If guest opts in to this feature, ASID invalidations only ··· 549 546 u64 guest_rip; 550 547 551 548 u32 hv_clean_fields; 552 552 - u32 hv_padding_32; 549 549 + u32 padding32_1; 553 550 u32 hv_synthetic_controls; 554 551 struct { 555 552 u32 nested_flush_hypercall:1; ··· 557 554 u32 reserved:30; 558 555 } __packed hv_enlightenments_control; 559 556 u32 hv_vp_id; 560 560 - 557 557 + u32 padding32_2; 561 558 u64 hv_vm_id; 562 559 u64 partition_assist_page; 563 560 u64 padding64_4[4]; 564 561 u64 guest_bndcfgs; 565 565 - u64 padding64_5[7]; 562 562 + u64 guest_ia32_perf_global_ctrl; 563 563 + u64 guest_ia32_s_cet; 564 564 + u64 guest_ssp; 565 565 + u64 guest_ia32_int_ssp_table_addr; 566 566 + u64 guest_ia32_lbr_ctl; 567 567 + u64 padding64_5[2]; 566 568 u64 xss_exit_bitmap; 567 567 - u64 padding64_6[7]; 569 569 + u64 encls_exiting_bitmap; 570 570 + u64 host_ia32_perf_global_ctrl; 571 571 + u64 tsc_multiplier; 572 572 + u64 host_ia32_s_cet; 573 573 + u64 host_ssp; 574 574 + u64 host_ia32_int_ssp_table_addr; 575 575 + u64 padding64_6; 568 576 } __packed; 569 577 570 578 #define HV_VMX_ENLIGHTENED_CLEAN_FIELD_NONE 0

+1 -1

arch/x86/include/asm/kvm-x86-ops.h

reviewed

··· 67 67 KVM_X86_OP(patch_hypercall) 68 68 KVM_X86_OP(inject_irq) 69 69 KVM_X86_OP(inject_nmi) 70 70 - KVM_X86_OP(queue_exception) 70 70 + KVM_X86_OP(inject_exception) 71 71 KVM_X86_OP(cancel_injection) 72 72 KVM_X86_OP(interrupt_allowed) 73 73 KVM_X86_OP(nmi_allowed)

+23 -16

arch/x86/include/asm/kvm_host.h

reviewed

··· 615 615 u32 enlightenments_eax; /* HYPERV_CPUID_ENLIGHTMENT_INFO.EAX */ 616 616 u32 enlightenments_ebx; /* HYPERV_CPUID_ENLIGHTMENT_INFO.EBX */ 617 617 u32 syndbg_cap_eax; /* HYPERV_CPUID_SYNDBG_PLATFORM_CAPABILITIES.EAX */ 618 618 + u32 nested_eax; /* HYPERV_CPUID_NESTED_FEATURES.EAX */ 619 619 + u32 nested_ebx; /* HYPERV_CPUID_NESTED_FEATURES.EBX */ 618 620 } cpuid_cache; 619 621 }; 620 622 ··· 639 637 struct hrtimer timer; 640 638 int poll_evtchn; 641 639 struct timer_list poll_timer; 640 640 + }; 641 641 + 642 642 + struct kvm_queued_exception { 643 643 + bool pending; 644 644 + bool injected; 645 645 + bool has_error_code; 646 646 + u8 vector; 647 647 + u32 error_code; 648 648 + unsigned long payload; 649 649 + bool has_payload; 642 650 }; 643 651 644 652 struct kvm_vcpu_arch { ··· 750 738 751 739 u8 event_exit_inst_len; 752 740 753 753 - struct kvm_queued_exception { 754 754 - bool pending; 755 755 - bool injected; 756 756 - bool has_error_code; 757 757 - u8 nr; 758 758 - u32 error_code; 759 759 - unsigned long payload; 760 760 - bool has_payload; 761 761 - u8 nested_apf; 762 762 - } exception; 741 741 + bool exception_from_userspace; 742 742 + 743 743 + /* Exceptions to be injected to the guest. */ 744 744 + struct kvm_queued_exception exception; 745 745 + /* Exception VM-Exits to be synthesized to L1. */ 746 746 + struct kvm_queued_exception exception_vmexit; 763 747 764 748 struct kvm_queued_interrupt { 765 749 bool injected; ··· 866 858 u32 id; 867 859 bool send_user_only; 868 860 u32 host_apf_flags; 869 869 - unsigned long nested_apf_token; 870 861 bool delivery_as_pf_vmexit; 871 862 bool pageready_pending; 872 863 } apf; ··· 1531 1524 unsigned char *hypercall_addr); 1532 1525 void (*inject_irq)(struct kvm_vcpu *vcpu, bool reinjected); 1533 1526 void (*inject_nmi)(struct kvm_vcpu *vcpu); 1534 1534 - void (*queue_exception)(struct kvm_vcpu *vcpu); 1527 1527 + void (*inject_exception)(struct kvm_vcpu *vcpu); 1535 1528 void (*cancel_injection)(struct kvm_vcpu *vcpu); 1536 1529 int (*interrupt_allowed)(struct kvm_vcpu *vcpu, bool for_injection); 1537 1530 int (*nmi_allowed)(struct kvm_vcpu *vcpu, bool for_injection); ··· 1641 1634 1642 1635 struct kvm_x86_nested_ops { 1643 1636 void (*leave_nested)(struct kvm_vcpu *vcpu); 1637 1637 + bool (*is_exception_vmexit)(struct kvm_vcpu *vcpu, u8 vector, 1638 1638 + u32 error_code); 1644 1639 int (*check_events)(struct kvm_vcpu *vcpu); 1645 1645 - bool (*handle_page_fault_workaround)(struct kvm_vcpu *vcpu, 1646 1646 - struct x86_exception *fault); 1647 1647 - bool (*hv_timer_pending)(struct kvm_vcpu *vcpu); 1640 1640 + bool (*has_events)(struct kvm_vcpu *vcpu); 1648 1641 void (*triple_fault)(struct kvm_vcpu *vcpu); 1649 1642 int (*get_state)(struct kvm_vcpu *vcpu, 1650 1643 struct kvm_nested_state __user *user_kvm_nested_state, ··· 1870 1863 void kvm_requeue_exception(struct kvm_vcpu *vcpu, unsigned nr); 1871 1864 void kvm_requeue_exception_e(struct kvm_vcpu *vcpu, unsigned nr, u32 error_code); 1872 1865 void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault); 1873 1873 - bool kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, 1866 1866 + void kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, 1874 1867 struct x86_exception *fault); 1875 1868 bool kvm_require_cpl(struct kvm_vcpu *vcpu, int required_cpl); 1876 1869 bool kvm_require_dr(struct kvm_vcpu *vcpu, int dr);

+17 -1

arch/x86/kvm/cpuid.c

reviewed

··· 311 311 } 312 312 EXPORT_SYMBOL_GPL(kvm_update_cpuid_runtime); 313 313 314 314 + static bool kvm_cpuid_has_hyperv(struct kvm_cpuid_entry2 *entries, int nent) 315 315 + { 316 316 + struct kvm_cpuid_entry2 *entry; 317 317 + 318 318 + entry = cpuid_entry2_find(entries, nent, HYPERV_CPUID_INTERFACE, 319 319 + KVM_CPUID_INDEX_NOT_SIGNIFICANT); 320 320 + return entry && entry->eax == HYPERV_CPUID_SIGNATURE_EAX; 321 321 + } 322 322 + 314 323 static void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) 315 324 { 316 325 struct kvm_lapic *apic = vcpu->arch.apic; ··· 355 346 vcpu->arch.cr4_guest_rsvd_bits = 356 347 __cr4_reserved_bits(guest_cpuid_has, vcpu); 357 348 358 358 - kvm_hv_set_cpuid(vcpu); 349 349 + kvm_hv_set_cpuid(vcpu, kvm_cpuid_has_hyperv(vcpu->arch.cpuid_entries, 350 350 + vcpu->arch.cpuid_nent)); 359 351 360 352 /* Invoke the vendor callback only after the above state is updated. */ 361 353 static_call(kvm_x86_vcpu_after_set_cpuid)(vcpu); ··· 417 407 418 408 kvfree(e2); 419 409 return 0; 410 410 + } 411 411 + 412 412 + if (kvm_cpuid_has_hyperv(e2, nent)) { 413 413 + r = kvm_hv_vcpu_init(vcpu); 414 414 + if (r) 415 415 + return r; 420 416 } 421 417 422 418 r = kvm_check_cpuid(vcpu, e2, nent);

+14 -17

arch/x86/kvm/emulate.c

reviewed

··· 1137 1137 static void decode_register_operand(struct x86_emulate_ctxt *ctxt, 1138 1138 struct operand *op) 1139 1139 { 1140 1140 - unsigned reg = ctxt->modrm_reg; 1140 1140 + unsigned int reg; 1141 1141 1142 1142 - if (!(ctxt->d & ModRM)) 1142 1142 + if (ctxt->d & ModRM) 1143 1143 + reg = ctxt->modrm_reg; 1144 1144 + else 1143 1145 reg = (ctxt->b & 7) | ((ctxt->rex_prefix & 1) << 3); 1144 1146 1145 1147 if (ctxt->d & Sse) { ··· 1955 1953 if (rc != X86EMUL_CONTINUE) 1956 1954 return rc; 1957 1955 1958 1958 - if (ctxt->modrm_reg == VCPU_SREG_SS) 1956 1956 + if (seg == VCPU_SREG_SS) 1959 1957 ctxt->interruptibility = KVM_X86_SHADOW_INT_MOV_SS; 1960 1958 if (ctxt->op_bytes > 2) 1961 1959 rsp_increment(ctxt, ctxt->op_bytes - 2); ··· 3647 3645 | ((u64)reg_read(ctxt, VCPU_REGS_RDX) << 32); 3648 3646 r = ctxt->ops->set_msr_with_filter(ctxt, msr_index, msr_data); 3649 3647 3650 3650 - if (r == X86EMUL_IO_NEEDED) 3651 3651 - return r; 3652 3652 - 3653 3653 - if (r > 0) 3648 3648 + if (r == X86EMUL_PROPAGATE_FAULT) 3654 3649 return emulate_gp(ctxt, 0); 3655 3650 3656 3656 - return r < 0 ? X86EMUL_UNHANDLEABLE : X86EMUL_CONTINUE; 3651 3651 + return r; 3657 3652 } 3658 3653 3659 3654 static int em_rdmsr(struct x86_emulate_ctxt *ctxt) ··· 3661 3662 3662 3663 r = ctxt->ops->get_msr_with_filter(ctxt, msr_index, &msr_data); 3663 3664 3664 3664 - if (r == X86EMUL_IO_NEEDED) 3665 3665 - return r; 3666 3666 - 3667 3667 - if (r) 3665 3665 + if (r == X86EMUL_PROPAGATE_FAULT) 3668 3666 return emulate_gp(ctxt, 0); 3669 3667 3670 3670 - *reg_write(ctxt, VCPU_REGS_RAX) = (u32)msr_data; 3671 3671 - *reg_write(ctxt, VCPU_REGS_RDX) = msr_data >> 32; 3672 3672 - return X86EMUL_CONTINUE; 3668 3668 + if (r == X86EMUL_CONTINUE) { 3669 3669 + *reg_write(ctxt, VCPU_REGS_RAX) = (u32)msr_data; 3670 3670 + *reg_write(ctxt, VCPU_REGS_RDX) = msr_data >> 32; 3671 3671 + } 3672 3672 + return r; 3673 3673 } 3674 3674 3675 3675 static int em_store_sreg(struct x86_emulate_ctxt *ctxt, int segment) ··· 4169 4171 4170 4172 ctxt->ops->get_dr(ctxt, 7, &dr7); 4171 4173 4172 4172 - /* Check if DR7.Global_Enable is set */ 4173 4173 - return dr7 & (1 << 13); 4174 4174 + return dr7 & DR7_GD; 4174 4175 } 4175 4176 4176 4177 static int check_dr_read(struct x86_emulate_ctxt *ctxt)

+32 -38

arch/x86/kvm/hyperv.c

reviewed

··· 38 38 #include "irq.h" 39 39 #include "fpu.h" 40 40 41 41 - /* "Hv#1" signature */ 42 42 - #define HYPERV_CPUID_SIGNATURE_EAX 0x31237648 43 43 - 44 41 #define KVM_HV_MAX_SPARSE_VCPU_SET_BITS DIV_ROUND_UP(KVM_MAX_VCPUS, 64) 45 42 46 43 static void stimer_mark_pending(struct kvm_vcpu_hv_stimer *stimer, ··· 931 934 stimer_prepare_msg(stimer); 932 935 } 933 936 934 934 - static int kvm_hv_vcpu_init(struct kvm_vcpu *vcpu) 937 937 + int kvm_hv_vcpu_init(struct kvm_vcpu *vcpu) 935 938 { 936 936 - struct kvm_vcpu_hv *hv_vcpu; 939 939 + struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); 937 940 int i; 941 941 + 942 942 + if (hv_vcpu) 943 943 + return 0; 938 944 939 945 hv_vcpu = kzalloc(sizeof(struct kvm_vcpu_hv), GFP_KERNEL_ACCOUNT); 940 946 if (!hv_vcpu) ··· 962 962 struct kvm_vcpu_hv_synic *synic; 963 963 int r; 964 964 965 965 - if (!to_hv_vcpu(vcpu)) { 966 966 - r = kvm_hv_vcpu_init(vcpu); 967 967 - if (r) 968 968 - return r; 969 969 - } 965 965 + r = kvm_hv_vcpu_init(vcpu); 966 966 + if (r) 967 967 + return r; 970 968 971 969 synic = to_hv_synic(vcpu); 972 970 ··· 1658 1660 if (!host && !vcpu->arch.hyperv_enabled) 1659 1661 return 1; 1660 1662 1661 1661 - if (!to_hv_vcpu(vcpu)) { 1662 1662 - if (kvm_hv_vcpu_init(vcpu)) 1663 1663 - return 1; 1664 1664 - } 1663 1663 + if (kvm_hv_vcpu_init(vcpu)) 1664 1664 + return 1; 1665 1665 1666 1666 if (kvm_hv_msr_partition_wide(msr)) { 1667 1667 int r; ··· 1679 1683 if (!host && !vcpu->arch.hyperv_enabled) 1680 1684 return 1; 1681 1685 1682 1682 - if (!to_hv_vcpu(vcpu)) { 1683 1683 - if (kvm_hv_vcpu_init(vcpu)) 1684 1684 - return 1; 1685 1685 - } 1686 1686 + if (kvm_hv_vcpu_init(vcpu)) 1687 1687 + return 1; 1686 1688 1687 1689 if (kvm_hv_msr_partition_wide(msr)) { 1688 1690 int r; ··· 1981 1987 return HV_STATUS_SUCCESS; 1982 1988 } 1983 1989 1984 1984 - void kvm_hv_set_cpuid(struct kvm_vcpu *vcpu) 1990 1990 + void kvm_hv_set_cpuid(struct kvm_vcpu *vcpu, bool hyperv_enabled) 1985 1991 { 1992 1992 + struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); 1986 1993 struct kvm_cpuid_entry2 *entry; 1987 1987 - struct kvm_vcpu_hv *hv_vcpu; 1988 1994 1989 1989 - entry = kvm_find_cpuid_entry(vcpu, HYPERV_CPUID_INTERFACE); 1990 1990 - if (entry && entry->eax == HYPERV_CPUID_SIGNATURE_EAX) { 1991 1991 - vcpu->arch.hyperv_enabled = true; 1992 1992 - } else { 1993 1993 - vcpu->arch.hyperv_enabled = false; 1995 1995 + vcpu->arch.hyperv_enabled = hyperv_enabled; 1996 1996 + 1997 1997 + if (!hv_vcpu) { 1998 1998 + /* 1999 1999 + * KVM should have already allocated kvm_vcpu_hv if Hyper-V is 2000 2000 + * enabled in CPUID. 2001 2001 + */ 2002 2002 + WARN_ON_ONCE(vcpu->arch.hyperv_enabled); 1994 2003 return; 1995 2004 } 1996 2005 1997 1997 - if (!to_hv_vcpu(vcpu) && kvm_hv_vcpu_init(vcpu)) 1998 1998 - return; 2006 2006 + memset(&hv_vcpu->cpuid_cache, 0, sizeof(hv_vcpu->cpuid_cache)); 1999 2007 2000 2000 - hv_vcpu = to_hv_vcpu(vcpu); 2008 2008 + if (!vcpu->arch.hyperv_enabled) 2009 2009 + return; 2001 2010 2002 2011 entry = kvm_find_cpuid_entry(vcpu, HYPERV_CPUID_FEATURES); 2003 2012 if (entry) { 2004 2013 hv_vcpu->cpuid_cache.features_eax = entry->eax; 2005 2014 hv_vcpu->cpuid_cache.features_ebx = entry->ebx; 2006 2015 hv_vcpu->cpuid_cache.features_edx = entry->edx; 2007 2007 - } else { 2008 2008 - hv_vcpu->cpuid_cache.features_eax = 0; 2009 2009 - hv_vcpu->cpuid_cache.features_ebx = 0; 2010 2010 - hv_vcpu->cpuid_cache.features_edx = 0; 2011 2016 } 2012 2017 2013 2018 entry = kvm_find_cpuid_entry(vcpu, HYPERV_CPUID_ENLIGHTMENT_INFO); 2014 2019 if (entry) { 2015 2020 hv_vcpu->cpuid_cache.enlightenments_eax = entry->eax; 2016 2021 hv_vcpu->cpuid_cache.enlightenments_ebx = entry->ebx; 2017 2017 - } else { 2018 2018 - hv_vcpu->cpuid_cache.enlightenments_eax = 0; 2019 2019 - hv_vcpu->cpuid_cache.enlightenments_ebx = 0; 2020 2022 } 2021 2023 2022 2024 entry = kvm_find_cpuid_entry(vcpu, HYPERV_CPUID_SYNDBG_PLATFORM_CAPABILITIES); 2023 2025 if (entry) 2024 2026 hv_vcpu->cpuid_cache.syndbg_cap_eax = entry->eax; 2025 2025 - else 2026 2026 - hv_vcpu->cpuid_cache.syndbg_cap_eax = 0; 2027 2027 + 2028 2028 + entry = kvm_find_cpuid_entry(vcpu, HYPERV_CPUID_NESTED_FEATURES); 2029 2029 + if (entry) { 2030 2030 + hv_vcpu->cpuid_cache.nested_eax = entry->eax; 2031 2031 + hv_vcpu->cpuid_cache.nested_ebx = entry->ebx; 2032 2032 + } 2027 2033 } 2028 2034 2029 2035 int kvm_hv_set_enforce_cpuid(struct kvm_vcpu *vcpu, bool enforce) ··· 2546 2552 case HYPERV_CPUID_NESTED_FEATURES: 2547 2553 ent->eax = evmcs_ver; 2548 2554 ent->eax |= HV_X64_NESTED_MSR_BITMAP; 2549 2549 - 2555 2555 + ent->ebx |= HV_X64_NESTED_EVMCS1_PERF_GLOBAL_CTRL; 2550 2556 break; 2551 2557 2552 2558 case HYPERV_CPUID_SYNDBG_VENDOR_AND_MAX_FUNCTIONS:

+5 -1

arch/x86/kvm/hyperv.h

reviewed

··· 23 23 24 24 #include <linux/kvm_host.h> 25 25 26 26 + /* "Hv#1" signature */ 27 27 + #define HYPERV_CPUID_SIGNATURE_EAX 0x31237648 28 28 + 26 29 /* 27 30 * The #defines related to the synthetic debugger are required by KDNet, but 28 31 * they are not documented in the Hyper-V TLFS because the synthetic debugger ··· 144 141 145 142 void kvm_hv_init_vm(struct kvm *kvm); 146 143 void kvm_hv_destroy_vm(struct kvm *kvm); 147 147 - void kvm_hv_set_cpuid(struct kvm_vcpu *vcpu); 144 144 + int kvm_hv_vcpu_init(struct kvm_vcpu *vcpu); 145 145 + void kvm_hv_set_cpuid(struct kvm_vcpu *vcpu, bool hyperv_enabled); 148 146 int kvm_hv_set_enforce_cpuid(struct kvm_vcpu *vcpu, bool enforce); 149 147 int kvm_vm_ioctl_hv_eventfd(struct kvm *kvm, struct kvm_hyperv_eventfd *args); 150 148 int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid,

+11 -27

arch/x86/kvm/lapic.c

reviewed

··· 3025 3025 struct kvm_lapic *apic = vcpu->arch.apic; 3026 3026 u8 sipi_vector; 3027 3027 int r; 3028 3028 - unsigned long pe; 3029 3028 3030 3030 - if (!lapic_in_kernel(vcpu)) 3031 3031 - return 0; 3032 3032 - 3033 3033 - /* 3034 3034 - * Read pending events before calling the check_events 3035 3035 - * callback. 3036 3036 - */ 3037 3037 - pe = smp_load_acquire(&apic->pending_events); 3038 3038 - if (!pe) 3029 3029 + if (!kvm_apic_has_pending_init_or_sipi(vcpu)) 3039 3030 return 0; 3040 3031 3041 3032 if (is_guest_mode(vcpu)) { ··· 3034 3043 if (r < 0) 3035 3044 return r == -EBUSY ? 0 : r; 3036 3045 /* 3037 3037 - * If an event has happened and caused a vmexit, 3038 3038 - * we know INITs are latched and therefore 3039 3039 - * we will not incorrectly deliver an APIC 3040 3040 - * event instead of a vmexit. 3046 3046 + * Continue processing INIT/SIPI even if a nested VM-Exit 3047 3047 + * occurred, e.g. pending SIPIs should be dropped if INIT+SIPI 3048 3048 + * are blocked as a result of transitioning to VMX root mode. 3041 3049 */ 3042 3050 } 3043 3051 3044 3052 /* 3045 3045 - * INITs are latched while CPU is in specific states 3046 3046 - * (SMM, VMX root mode, SVM with GIF=0). 3047 3047 - * Because a CPU cannot be in these states immediately 3048 3048 - * after it has processed an INIT signal (and thus in 3049 3049 - * KVM_MP_STATE_INIT_RECEIVED state), just eat SIPIs 3050 3050 - * and leave the INIT pending. 3053 3053 + * INITs are blocked while CPU is in specific states (SMM, VMX root 3054 3054 + * mode, SVM with GIF=0), while SIPIs are dropped if the CPU isn't in 3055 3055 + * wait-for-SIPI (WFS). 3051 3056 */ 3052 3052 - if (kvm_vcpu_latch_init(vcpu)) { 3057 3057 + if (!kvm_apic_init_sipi_allowed(vcpu)) { 3053 3058 WARN_ON_ONCE(vcpu->arch.mp_state == KVM_MP_STATE_INIT_RECEIVED); 3054 3054 - if (test_bit(KVM_APIC_SIPI, &pe)) 3055 3055 - clear_bit(KVM_APIC_SIPI, &apic->pending_events); 3059 3059 + clear_bit(KVM_APIC_SIPI, &apic->pending_events); 3056 3060 return 0; 3057 3061 } 3058 3062 3059 3059 - if (test_bit(KVM_APIC_INIT, &pe)) { 3060 3060 - clear_bit(KVM_APIC_INIT, &apic->pending_events); 3063 3063 + if (test_and_clear_bit(KVM_APIC_INIT, &apic->pending_events)) { 3061 3064 kvm_vcpu_reset(vcpu, true); 3062 3065 if (kvm_vcpu_is_bsp(apic->vcpu)) 3063 3066 vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE; 3064 3067 else 3065 3068 vcpu->arch.mp_state = KVM_MP_STATE_INIT_RECEIVED; 3066 3069 } 3067 3067 - if (test_bit(KVM_APIC_SIPI, &pe)) { 3068 3068 - clear_bit(KVM_APIC_SIPI, &apic->pending_events); 3070 3070 + if (test_and_clear_bit(KVM_APIC_SIPI, &apic->pending_events)) { 3069 3071 if (vcpu->arch.mp_state == KVM_MP_STATE_INIT_RECEIVED) { 3070 3072 /* evaluate pending_events before reading the vector */ 3071 3073 smp_rmb();

+8 -1

arch/x86/kvm/lapic.h

reviewed

··· 7 7 #include <linux/kvm_host.h> 8 8 9 9 #include "hyperv.h" 10 10 + #include "kvm_cache_regs.h" 10 11 11 12 #define KVM_APIC_INIT 0 12 13 #define KVM_APIC_SIPI 1 ··· 224 223 return lapic_in_kernel(vcpu) && vcpu->arch.apic->apicv_active; 225 224 } 226 225 227 227 - static inline bool kvm_apic_has_events(struct kvm_vcpu *vcpu) 226 226 + static inline bool kvm_apic_has_pending_init_or_sipi(struct kvm_vcpu *vcpu) 228 227 { 229 228 return lapic_in_kernel(vcpu) && vcpu->arch.apic->pending_events; 229 229 + } 230 230 + 231 231 + static inline bool kvm_apic_init_sipi_allowed(struct kvm_vcpu *vcpu) 232 232 + { 233 233 + return !is_smm(vcpu) && 234 234 + !static_call(kvm_x86_apic_init_signal_blocked)(vcpu); 230 235 } 231 236 232 237 static inline bool kvm_lowest_prio_delivery(struct kvm_lapic_irq *irq)

+18 -4

arch/x86/kvm/mmu/mmu.c

reviewed

··· 1667 1667 percpu_counter_add(&kvm_total_used_mmu_pages, nr); 1668 1668 } 1669 1669 1670 1670 + static void kvm_account_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp) 1671 1671 + { 1672 1672 + kvm_mod_used_mmu_pages(kvm, +1); 1673 1673 + kvm_account_pgtable_pages((void *)sp->spt, +1); 1674 1674 + } 1675 1675 + 1676 1676 + static void kvm_unaccount_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp) 1677 1677 + { 1678 1678 + kvm_mod_used_mmu_pages(kvm, -1); 1679 1679 + kvm_account_pgtable_pages((void *)sp->spt, -1); 1680 1680 + } 1681 1681 + 1670 1682 static void kvm_mmu_free_shadow_page(struct kvm_mmu_page *sp) 1671 1683 { 1672 1684 MMU_WARN_ON(!is_empty_shadow_page(sp->spt)); ··· 2136 2124 */ 2137 2125 sp->mmu_valid_gen = kvm->arch.mmu_valid_gen; 2138 2126 list_add(&sp->link, &kvm->arch.active_mmu_pages); 2139 2139 - kvm_mod_used_mmu_pages(kvm, +1); 2127 2127 + kvm_account_mmu_page(kvm, sp); 2140 2128 2141 2129 sp->gfn = gfn; 2142 2130 sp->role = role; ··· 2470 2458 list_add(&sp->link, invalid_list); 2471 2459 else 2472 2460 list_move(&sp->link, invalid_list); 2473 2473 - kvm_mod_used_mmu_pages(kvm, -1); 2461 2461 + kvm_unaccount_mmu_page(kvm, sp); 2474 2462 } else { 2475 2463 /* 2476 2464 * Remove the active root from the active page list, the root ··· 4304 4292 4305 4293 vcpu->arch.l1tf_flush_l1d = true; 4306 4294 if (!flags) { 4307 4307 - trace_kvm_page_fault(fault_address, error_code); 4295 4295 + trace_kvm_page_fault(vcpu, fault_address, error_code); 4308 4296 4309 4297 if (kvm_event_needs_reinjection(vcpu)) 4310 4298 kvm_mmu_unprotect_page_virt(vcpu, fault_address); ··· 6716 6704 6717 6705 ret = register_shrinker(&mmu_shrinker, "x86-mmu"); 6718 6706 if (ret) 6719 6719 - goto out; 6707 6707 + goto out_shrinker; 6720 6708 6721 6709 return 0; 6722 6710 6711 6711 + out_shrinker: 6712 6712 + percpu_counter_destroy(&kvm_total_used_mmu_pages); 6723 6713 out: 6724 6714 mmu_destroy_caches(); 6725 6715 return ret;

+1 -1

arch/x86/kvm/mmu/paging_tmpl.h

reviewed

··· 472 472 473 473 #if PTTYPE == PTTYPE_EPT 474 474 /* 475 475 - * Use PFERR_RSVD_MASK in error_code to to tell if EPT 475 475 + * Use PFERR_RSVD_MASK in error_code to tell if EPT 476 476 * misconfiguration requires to be injected. The detection is 477 477 * done by is_rsvd_bits_set() above. 478 478 *

+12

arch/x86/kvm/mmu/tdp_mmu.c

reviewed

··· 372 372 } 373 373 } 374 374 375 375 + static void tdp_account_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp) 376 376 + { 377 377 + kvm_account_pgtable_pages((void *)sp->spt, +1); 378 378 + } 379 379 + 380 380 + static void tdp_unaccount_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp) 381 381 + { 382 382 + kvm_account_pgtable_pages((void *)sp->spt, -1); 383 383 + } 384 384 + 375 385 /** 376 386 * tdp_mmu_unlink_sp() - Remove a shadow page from the list of used pages 377 387 * ··· 394 384 static void tdp_mmu_unlink_sp(struct kvm *kvm, struct kvm_mmu_page *sp, 395 385 bool shared) 396 386 { 387 387 + tdp_unaccount_mmu_page(kvm, sp); 397 388 if (shared) 398 389 spin_lock(&kvm->arch.tdp_mmu_pages_lock); 399 390 else ··· 1143 1132 if (account_nx) 1144 1133 account_huge_nx_page(kvm, sp); 1145 1134 spin_unlock(&kvm->arch.tdp_mmu_pages_lock); 1135 1135 + tdp_account_mmu_page(kvm, sp); 1146 1136 1147 1137 return 0; 1148 1138 }

+59 -65

arch/x86/kvm/svm/nested.c

reviewed

··· 55 55 nested_svm_vmexit(svm); 56 56 } 57 57 58 58 - static bool nested_svm_handle_page_fault_workaround(struct kvm_vcpu *vcpu, 59 59 - struct x86_exception *fault) 60 60 - { 61 61 - struct vcpu_svm *svm = to_svm(vcpu); 62 62 - struct vmcb *vmcb = svm->vmcb; 63 63 - 64 64 - WARN_ON(!is_guest_mode(vcpu)); 65 65 - 66 66 - if (vmcb12_is_intercept(&svm->nested.ctl, 67 67 - INTERCEPT_EXCEPTION_OFFSET + PF_VECTOR) && 68 68 - !WARN_ON_ONCE(svm->nested.nested_run_pending)) { 69 69 - vmcb->control.exit_code = SVM_EXIT_EXCP_BASE + PF_VECTOR; 70 70 - vmcb->control.exit_code_hi = 0; 71 71 - vmcb->control.exit_info_1 = fault->error_code; 72 72 - vmcb->control.exit_info_2 = fault->address; 73 73 - nested_svm_vmexit(svm); 74 74 - return true; 75 75 - } 76 76 - 77 77 - return false; 78 78 - } 79 79 - 80 58 static u64 nested_svm_get_tdp_pdptr(struct kvm_vcpu *vcpu, int index) 81 59 { 82 60 struct vcpu_svm *svm = to_svm(vcpu); ··· 446 468 unsigned int nr; 447 469 448 470 if (vcpu->arch.exception.injected) { 449 449 - nr = vcpu->arch.exception.nr; 471 471 + nr = vcpu->arch.exception.vector; 450 472 exit_int_info = nr | SVM_EVTINJ_VALID | SVM_EVTINJ_TYPE_EXEPT; 451 473 452 474 if (vcpu->arch.exception.has_error_code) { ··· 759 781 struct vcpu_svm *svm = to_svm(vcpu); 760 782 int ret; 761 783 762 762 - trace_kvm_nested_vmrun(svm->vmcb->save.rip, vmcb12_gpa, 763 763 - vmcb12->save.rip, 764 764 - vmcb12->control.int_ctl, 765 765 - vmcb12->control.event_inj, 766 766 - vmcb12->control.nested_ctl); 784 784 + trace_kvm_nested_vmenter(svm->vmcb->save.rip, 785 785 + vmcb12_gpa, 786 786 + vmcb12->save.rip, 787 787 + vmcb12->control.int_ctl, 788 788 + vmcb12->control.event_inj, 789 789 + vmcb12->control.nested_ctl, 790 790 + vmcb12->control.nested_cr3, 791 791 + vmcb12->save.cr3, 792 792 + KVM_ISA_SVM); 767 793 768 794 trace_kvm_nested_intercepts(vmcb12->control.intercepts[INTERCEPT_CR] & 0xffff, 769 795 vmcb12->control.intercepts[INTERCEPT_CR] >> 16, ··· 1286 1304 return 0; 1287 1305 } 1288 1306 1289 1289 - static bool nested_exit_on_exception(struct vcpu_svm *svm) 1307 1307 + static bool nested_svm_is_exception_vmexit(struct kvm_vcpu *vcpu, u8 vector, 1308 1308 + u32 error_code) 1290 1309 { 1291 1291 - unsigned int nr = svm->vcpu.arch.exception.nr; 1310 1310 + struct vcpu_svm *svm = to_svm(vcpu); 1292 1311 1293 1293 - return (svm->nested.ctl.intercepts[INTERCEPT_EXCEPTION] & BIT(nr)); 1312 1312 + return (svm->nested.ctl.intercepts[INTERCEPT_EXCEPTION] & BIT(vector)); 1294 1313 } 1295 1314 1296 1296 - static void nested_svm_inject_exception_vmexit(struct vcpu_svm *svm) 1315 1315 + static void nested_svm_inject_exception_vmexit(struct kvm_vcpu *vcpu) 1297 1316 { 1298 1298 - unsigned int nr = svm->vcpu.arch.exception.nr; 1317 1317 + struct kvm_queued_exception *ex = &vcpu->arch.exception_vmexit; 1318 1318 + struct vcpu_svm *svm = to_svm(vcpu); 1299 1319 struct vmcb *vmcb = svm->vmcb; 1300 1320 1301 1301 - vmcb->control.exit_code = SVM_EXIT_EXCP_BASE + nr; 1321 1321 + vmcb->control.exit_code = SVM_EXIT_EXCP_BASE + ex->vector; 1302 1322 vmcb->control.exit_code_hi = 0; 1303 1323 1304 1304 - if (svm->vcpu.arch.exception.has_error_code) 1305 1305 - vmcb->control.exit_info_1 = svm->vcpu.arch.exception.error_code; 1324 1324 + if (ex->has_error_code) 1325 1325 + vmcb->control.exit_info_1 = ex->error_code; 1306 1326 1307 1327 /* 1308 1328 * EXITINFO2 is undefined for all exception intercepts other 1309 1329 * than #PF. 1310 1330 */ 1311 1311 - if (nr == PF_VECTOR) { 1312 1312 - if (svm->vcpu.arch.exception.nested_apf) 1313 1313 - vmcb->control.exit_info_2 = svm->vcpu.arch.apf.nested_apf_token; 1314 1314 - else if (svm->vcpu.arch.exception.has_payload) 1315 1315 - vmcb->control.exit_info_2 = svm->vcpu.arch.exception.payload; 1331 1331 + if (ex->vector == PF_VECTOR) { 1332 1332 + if (ex->has_payload) 1333 1333 + vmcb->control.exit_info_2 = ex->payload; 1316 1334 else 1317 1317 - vmcb->control.exit_info_2 = svm->vcpu.arch.cr2; 1318 1318 - } else if (nr == DB_VECTOR) { 1319 1319 - /* See inject_pending_event. */ 1320 1320 - kvm_deliver_exception_payload(&svm->vcpu); 1321 1321 - if (svm->vcpu.arch.dr7 & DR7_GD) { 1322 1322 - svm->vcpu.arch.dr7 &= ~DR7_GD; 1323 1323 - kvm_update_dr7(&svm->vcpu); 1335 1335 + vmcb->control.exit_info_2 = vcpu->arch.cr2; 1336 1336 + } else if (ex->vector == DB_VECTOR) { 1337 1337 + /* See kvm_check_and_inject_events(). */ 1338 1338 + kvm_deliver_exception_payload(vcpu, ex); 1339 1339 + 1340 1340 + if (vcpu->arch.dr7 & DR7_GD) { 1341 1341 + vcpu->arch.dr7 &= ~DR7_GD; 1342 1342 + kvm_update_dr7(vcpu); 1324 1343 } 1325 1325 - } else 1326 1326 - WARN_ON(svm->vcpu.arch.exception.has_payload); 1344 1344 + } else { 1345 1345 + WARN_ON(ex->has_payload); 1346 1346 + } 1327 1347 1328 1348 nested_svm_vmexit(svm); 1329 1349 } ··· 1337 1353 1338 1354 static int svm_check_nested_events(struct kvm_vcpu *vcpu) 1339 1355 { 1340 1340 - struct vcpu_svm *svm = to_svm(vcpu); 1341 1341 - bool block_nested_events = 1342 1342 - kvm_event_needs_reinjection(vcpu) || svm->nested.nested_run_pending; 1343 1356 struct kvm_lapic *apic = vcpu->arch.apic; 1357 1357 + struct vcpu_svm *svm = to_svm(vcpu); 1358 1358 + /* 1359 1359 + * Only a pending nested run blocks a pending exception. If there is a 1360 1360 + * previously injected event, the pending exception occurred while said 1361 1361 + * event was being delivered and thus needs to be handled. 1362 1362 + */ 1363 1363 + bool block_nested_exceptions = svm->nested.nested_run_pending; 1364 1364 + /* 1365 1365 + * New events (not exceptions) are only recognized at instruction 1366 1366 + * boundaries. If an event needs reinjection, then KVM is handling a 1367 1367 + * VM-Exit that occurred _during_ instruction execution; new events are 1368 1368 + * blocked until the instruction completes. 1369 1369 + */ 1370 1370 + bool block_nested_events = block_nested_exceptions || 1371 1371 + kvm_event_needs_reinjection(vcpu); 1344 1372 1345 1373 if (lapic_in_kernel(vcpu) && 1346 1374 test_bit(KVM_APIC_INIT, &apic->pending_events)) { ··· 1364 1368 return 0; 1365 1369 } 1366 1370 1367 1367 - if (vcpu->arch.exception.pending) { 1368 1368 - /* 1369 1369 - * Only a pending nested run can block a pending exception. 1370 1370 - * Otherwise an injected NMI/interrupt should either be 1371 1371 - * lost or delivered to the nested hypervisor in the EXITINTINFO 1372 1372 - * vmcb field, while delivering the pending exception. 1373 1373 - */ 1374 1374 - if (svm->nested.nested_run_pending) 1371 1371 + if (vcpu->arch.exception_vmexit.pending) { 1372 1372 + if (block_nested_exceptions) 1375 1373 return -EBUSY; 1376 1376 - if (!nested_exit_on_exception(svm)) 1377 1377 - return 0; 1378 1378 - nested_svm_inject_exception_vmexit(svm); 1374 1374 + nested_svm_inject_exception_vmexit(vcpu); 1375 1375 + return 0; 1376 1376 + } 1377 1377 + 1378 1378 + if (vcpu->arch.exception.pending) { 1379 1379 + if (block_nested_exceptions) 1380 1380 + return -EBUSY; 1379 1381 return 0; 1380 1382 } 1381 1383 ··· 1714 1720 1715 1721 struct kvm_x86_nested_ops svm_nested_ops = { 1716 1722 .leave_nested = svm_leave_nested, 1723 1723 + .is_exception_vmexit = nested_svm_is_exception_vmexit, 1717 1724 .check_events = svm_check_nested_events, 1718 1718 - .handle_page_fault_workaround = nested_svm_handle_page_fault_workaround, 1719 1725 .triple_fault = nested_svm_triple_fault, 1720 1726 .get_nested_state_pages = svm_get_nested_state_pages, 1721 1727 .get_state = svm_get_nested_state,

+13 -22

arch/x86/kvm/svm/svm.c

reviewed

··· 461 461 return 0; 462 462 } 463 463 464 464 - static void svm_queue_exception(struct kvm_vcpu *vcpu) 464 464 + static void svm_inject_exception(struct kvm_vcpu *vcpu) 465 465 { 466 466 + struct kvm_queued_exception *ex = &vcpu->arch.exception; 466 467 struct vcpu_svm *svm = to_svm(vcpu); 467 467 - unsigned nr = vcpu->arch.exception.nr; 468 468 - bool has_error_code = vcpu->arch.exception.has_error_code; 469 469 - u32 error_code = vcpu->arch.exception.error_code; 470 468 471 471 - kvm_deliver_exception_payload(vcpu); 469 469 + kvm_deliver_exception_payload(vcpu, ex); 472 470 473 473 - if (kvm_exception_is_soft(nr) && 471 471 + if (kvm_exception_is_soft(ex->vector) && 474 472 svm_update_soft_interrupt_rip(vcpu)) 475 473 return; 476 474 477 477 - svm->vmcb->control.event_inj = nr 475 475 + svm->vmcb->control.event_inj = ex->vector 478 476 | SVM_EVTINJ_VALID 479 479 - | (has_error_code ? SVM_EVTINJ_VALID_ERR : 0) 477 477 + | (ex->has_error_code ? SVM_EVTINJ_VALID_ERR : 0) 480 478 | SVM_EVTINJ_TYPE_EXEPT; 481 481 - svm->vmcb->control.event_inj_err = error_code; 479 479 + svm->vmcb->control.event_inj_err = ex->error_code; 482 480 } 483 481 484 482 static void svm_init_erratum_383(void) ··· 1973 1975 u64 fault_address = svm->vmcb->control.exit_info_2; 1974 1976 u64 error_code = svm->vmcb->control.exit_info_1; 1975 1977 1976 1976 - trace_kvm_page_fault(fault_address, error_code); 1978 1978 + trace_kvm_page_fault(vcpu, fault_address, error_code); 1977 1979 return kvm_mmu_page_fault(vcpu, fault_address, error_code, 1978 1980 static_cpu_has(X86_FEATURE_DECODEASSISTS) ? 1979 1981 svm->vmcb->control.insn_bytes : NULL, ··· 2339 2341 enable_gif(svm); 2340 2342 if (svm->vcpu.arch.smi_pending || 2341 2343 svm->vcpu.arch.nmi_pending || 2342 2342 - kvm_cpu_has_injectable_intr(&svm->vcpu)) 2344 2344 + kvm_cpu_has_injectable_intr(&svm->vcpu) || 2345 2345 + kvm_apic_has_pending_init_or_sipi(&svm->vcpu)) 2343 2346 kvm_make_request(KVM_REQ_EVENT, &svm->vcpu); 2344 2347 } else { 2345 2348 disable_gif(svm); ··· 3521 3522 3522 3523 /* Note, this is called iff the local APIC is in-kernel. */ 3523 3524 if (!READ_ONCE(vcpu->arch.apic->apicv_active)) { 3524 3524 - /* Process the interrupt via inject_pending_event */ 3525 3525 + /* Process the interrupt via kvm_check_and_inject_events(). */ 3525 3526 kvm_make_request(KVM_REQ_EVENT, vcpu); 3526 3527 kvm_vcpu_kick(vcpu); 3527 3528 return; ··· 4696 4697 { 4697 4698 struct vcpu_svm *svm = to_svm(vcpu); 4698 4699 4699 4699 - /* 4700 4700 - * TODO: Last condition latch INIT signals on vCPU when 4701 4701 - * vCPU is in guest-mode and vmcb12 defines intercept on INIT. 4702 4702 - * To properly emulate the INIT intercept, 4703 4703 - * svm_check_nested_events() should call nested_svm_vmexit() 4704 4704 - * if an INIT signal is pending. 4705 4705 - */ 4706 4706 - return !gif_set(svm) || 4707 4707 - (vmcb_is_intercept(&svm->vmcb->control, INTERCEPT_INIT)); 4700 4700 + return !gif_set(svm); 4708 4701 } 4709 4702 4710 4703 static void svm_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector) ··· 4789 4798 .patch_hypercall = svm_patch_hypercall, 4790 4799 .inject_irq = svm_inject_irq, 4791 4800 .inject_nmi = svm_inject_nmi, 4792 4792 - .queue_exception = svm_queue_exception, 4801 4801 + .inject_exception = svm_inject_exception, 4793 4802 .cancel_injection = svm_cancel_injection, 4794 4803 .interrupt_allowed = svm_interrupt_allowed, 4795 4804 .nmi_allowed = svm_nmi_allowed,

+34 -15

arch/x86/kvm/trace.h

reviewed

··· 394 394 * Tracepoint for page fault. 395 395 */ 396 396 TRACE_EVENT(kvm_page_fault, 397 397 - TP_PROTO(unsigned long fault_address, unsigned int error_code), 398 398 - TP_ARGS(fault_address, error_code), 397 397 + TP_PROTO(struct kvm_vcpu *vcpu, u64 fault_address, u64 error_code), 398 398 + TP_ARGS(vcpu, fault_address, error_code), 399 399 400 400 TP_STRUCT__entry( 401 401 - __field( unsigned long, fault_address ) 402 402 - __field( unsigned int, error_code ) 401 401 + __field( unsigned int, vcpu_id ) 402 402 + __field( unsigned long, guest_rip ) 403 403 + __field( u64, fault_address ) 404 404 + __field( u64, error_code ) 403 405 ), 404 406 405 407 TP_fast_assign( 408 408 + __entry->vcpu_id = vcpu->vcpu_id; 409 409 + __entry->guest_rip = kvm_rip_read(vcpu); 406 410 __entry->fault_address = fault_address; 407 411 __entry->error_code = error_code; 408 412 ), 409 413 410 410 - TP_printk("address %lx error_code %x", 414 414 + TP_printk("vcpu %u rip 0x%lx address 0x%016llx error_code 0x%llx", 415 415 + __entry->vcpu_id, __entry->guest_rip, 411 416 __entry->fault_address, __entry->error_code) 412 417 ); 413 418 ··· 594 589 /* 595 590 * Tracepoint for nested VMRUN 596 591 */ 597 597 - TRACE_EVENT(kvm_nested_vmrun, 592 592 + TRACE_EVENT(kvm_nested_vmenter, 598 593 TP_PROTO(__u64 rip, __u64 vmcb, __u64 nested_rip, __u32 int_ctl, 599 599 - __u32 event_inj, bool npt), 600 600 - TP_ARGS(rip, vmcb, nested_rip, int_ctl, event_inj, npt), 594 594 + __u32 event_inj, bool tdp_enabled, __u64 guest_tdp_pgd, 595 595 + __u64 guest_cr3, __u32 isa), 596 596 + TP_ARGS(rip, vmcb, nested_rip, int_ctl, event_inj, tdp_enabled, 597 597 + guest_tdp_pgd, guest_cr3, isa), 601 598 602 599 TP_STRUCT__entry( 603 600 __field( __u64, rip ) ··· 607 600 __field( __u64, nested_rip ) 608 601 __field( __u32, int_ctl ) 609 602 __field( __u32, event_inj ) 610 610 - __field( bool, npt ) 603 603 + __field( bool, tdp_enabled ) 604 604 + __field( __u64, guest_pgd ) 605 605 + __field( __u32, isa ) 611 606 ), 612 607 613 608 TP_fast_assign( ··· 618 609 __entry->nested_rip = nested_rip; 619 610 __entry->int_ctl = int_ctl; 620 611 __entry->event_inj = event_inj; 621 621 - __entry->npt = npt; 612 612 + __entry->tdp_enabled = tdp_enabled; 613 613 + __entry->guest_pgd = tdp_enabled ? guest_tdp_pgd : guest_cr3; 614 614 + __entry->isa = isa; 622 615 ), 623 616 624 624 - TP_printk("rip: 0x%016llx vmcb: 0x%016llx nrip: 0x%016llx int_ctl: 0x%08x " 625 625 - "event_inj: 0x%08x npt: %s", 626 626 - __entry->rip, __entry->vmcb, __entry->nested_rip, 627 627 - __entry->int_ctl, __entry->event_inj, 628 628 - __entry->npt ? "on" : "off") 617 617 + TP_printk("rip: 0x%016llx %s: 0x%016llx nested_rip: 0x%016llx " 618 618 + "int_ctl: 0x%08x event_inj: 0x%08x nested_%s=%s %s: 0x%016llx", 619 619 + __entry->rip, 620 620 + __entry->isa == KVM_ISA_VMX ? "vmcs" : "vmcb", 621 621 + __entry->vmcb, 622 622 + __entry->nested_rip, 623 623 + __entry->int_ctl, 624 624 + __entry->event_inj, 625 625 + __entry->isa == KVM_ISA_VMX ? "ept" : "npt", 626 626 + __entry->tdp_enabled ? "y" : "n", 627 627 + !__entry->tdp_enabled ? "guest_cr3" : 628 628 + __entry->isa == KVM_ISA_VMX ? "nested_eptp" : "nested_cr3", 629 629 + __entry->guest_pgd) 629 630 ); 630 631 631 632 TRACE_EVENT(kvm_nested_intercepts,

+5 -9

arch/x86/kvm/vmx/capabilities.h

reviewed

··· 65 65 u64 cpu_based_3rd_exec_ctrl; 66 66 u32 vmexit_ctrl; 67 67 u32 vmentry_ctrl; 68 68 + u64 misc; 68 69 struct nested_vmx_msrs nested; 69 70 }; 70 71 extern struct vmcs_config vmcs_config; ··· 83 82 84 83 static inline bool cpu_has_virtual_nmis(void) 85 84 { 86 86 - return vmcs_config.pin_based_exec_ctrl & PIN_BASED_VIRTUAL_NMIS; 85 85 + return vmcs_config.pin_based_exec_ctrl & PIN_BASED_VIRTUAL_NMIS && 86 86 + vmcs_config.cpu_based_exec_ctrl & CPU_BASED_NMI_WINDOW_EXITING; 87 87 } 88 88 89 89 static inline bool cpu_has_vmx_preemption_timer(void) ··· 226 224 227 225 static inline bool cpu_has_vmx_shadow_vmcs(void) 228 226 { 229 229 - u64 vmx_msr; 230 230 - 231 227 /* check if the cpu supports writing r/o exit information fields */ 232 232 - rdmsrl(MSR_IA32_VMX_MISC, vmx_msr); 233 233 - if (!(vmx_msr & MSR_IA32_VMX_MISC_VMWRITE_SHADOW_RO_FIELDS)) 228 228 + if (!(vmcs_config.misc & MSR_IA32_VMX_MISC_VMWRITE_SHADOW_RO_FIELDS)) 234 229 return false; 235 230 236 231 return vmcs_config.cpu_based_2nd_exec_ctrl & ··· 369 370 370 371 static inline bool cpu_has_vmx_intel_pt(void) 371 372 { 372 372 - u64 vmx_msr; 373 373 - 374 374 - rdmsrl(MSR_IA32_VMX_MISC, vmx_msr); 375 375 - return (vmx_msr & MSR_IA32_VMX_MISC_INTEL_PT) && 373 373 + return (vmcs_config.misc & MSR_IA32_VMX_MISC_INTEL_PT) && 376 374 (vmcs_config.cpu_based_2nd_exec_ctrl & SECONDARY_EXEC_PT_USE_GPA) && 377 375 (vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_RTIT_CTL); 378 376 }

+129 -61

arch/x86/kvm/vmx/evmcs.c

reviewed

··· 10 10 #include "vmx.h" 11 11 #include "trace.h" 12 12 13 13 + #define CC KVM_NESTED_VMENTER_CONSISTENCY_CHECK 14 14 + 13 15 DEFINE_STATIC_KEY_FALSE(enable_evmcs); 14 16 15 17 #define EVMCS1_OFFSET(x) offsetof(struct hv_enlightened_vmcs, x) ··· 29 27 EVMCS1_FIELD(HOST_IA32_PAT, host_ia32_pat, 30 28 HV_VMX_ENLIGHTENED_CLEAN_FIELD_HOST_GRP1), 31 29 EVMCS1_FIELD(HOST_IA32_EFER, host_ia32_efer, 30 30 + HV_VMX_ENLIGHTENED_CLEAN_FIELD_HOST_GRP1), 31 31 + EVMCS1_FIELD(HOST_IA32_PERF_GLOBAL_CTRL, host_ia32_perf_global_ctrl, 32 32 HV_VMX_ENLIGHTENED_CLEAN_FIELD_HOST_GRP1), 33 33 EVMCS1_FIELD(HOST_CR0, host_cr0, 34 34 HV_VMX_ENLIGHTENED_CLEAN_FIELD_HOST_GRP1), ··· 82 78 HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP1), 83 79 EVMCS1_FIELD(GUEST_IA32_EFER, guest_ia32_efer, 84 80 HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP1), 81 81 + EVMCS1_FIELD(GUEST_IA32_PERF_GLOBAL_CTRL, guest_ia32_perf_global_ctrl, 82 82 + HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP1), 85 83 EVMCS1_FIELD(GUEST_PDPTR0, guest_pdptr0, 86 84 HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP1), 87 85 EVMCS1_FIELD(GUEST_PDPTR1, guest_pdptr1, ··· 132 126 HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP1), 133 127 EVMCS1_FIELD(XSS_EXIT_BITMAP, xss_exit_bitmap, 134 128 HV_VMX_ENLIGHTENED_CLEAN_FIELD_CONTROL_GRP2), 129 129 + EVMCS1_FIELD(ENCLS_EXITING_BITMAP, encls_exiting_bitmap, 130 130 + HV_VMX_ENLIGHTENED_CLEAN_FIELD_CONTROL_GRP2), 131 131 + EVMCS1_FIELD(TSC_MULTIPLIER, tsc_multiplier, 132 132 + HV_VMX_ENLIGHTENED_CLEAN_FIELD_CONTROL_GRP2), 133 133 + /* 134 134 + * Not used by KVM: 135 135 + * 136 136 + * EVMCS1_FIELD(0x00006828, guest_ia32_s_cet, 137 137 + * HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP1), 138 138 + * EVMCS1_FIELD(0x0000682A, guest_ssp, 139 139 + * HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_BASIC), 140 140 + * EVMCS1_FIELD(0x0000682C, guest_ia32_int_ssp_table_addr, 141 141 + * HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP1), 142 142 + * EVMCS1_FIELD(0x00002816, guest_ia32_lbr_ctl, 143 143 + * HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP1), 144 144 + * EVMCS1_FIELD(0x00006C18, host_ia32_s_cet, 145 145 + * HV_VMX_ENLIGHTENED_CLEAN_FIELD_HOST_GRP1), 146 146 + * EVMCS1_FIELD(0x00006C1A, host_ssp, 147 147 + * HV_VMX_ENLIGHTENED_CLEAN_FIELD_HOST_GRP1), 148 148 + * EVMCS1_FIELD(0x00006C1C, host_ia32_int_ssp_table_addr, 149 149 + * HV_VMX_ENLIGHTENED_CLEAN_FIELD_HOST_GRP1), 150 150 + */ 135 151 136 152 /* 64 bit read only */ 137 153 EVMCS1_FIELD(GUEST_PHYSICAL_ADDRESS, guest_physical_address, ··· 322 294 }; 323 295 const unsigned int nr_evmcs_1_fields = ARRAY_SIZE(vmcs_field_to_evmcs_1); 324 296 325 325 - #if IS_ENABLED(CONFIG_HYPERV) 326 326 - __init void evmcs_sanitize_exec_ctrls(struct vmcs_config *vmcs_conf) 327 327 - { 328 328 - vmcs_conf->cpu_based_exec_ctrl &= ~EVMCS1_UNSUPPORTED_EXEC_CTRL; 329 329 - vmcs_conf->pin_based_exec_ctrl &= ~EVMCS1_UNSUPPORTED_PINCTRL; 330 330 - vmcs_conf->cpu_based_2nd_exec_ctrl &= ~EVMCS1_UNSUPPORTED_2NDEXEC; 331 331 - vmcs_conf->cpu_based_3rd_exec_ctrl = 0; 332 332 - 333 333 - vmcs_conf->vmexit_ctrl &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL; 334 334 - vmcs_conf->vmentry_ctrl &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL; 335 335 - } 336 336 - #endif 337 337 - 338 297 bool nested_enlightened_vmentry(struct kvm_vcpu *vcpu, u64 *evmcs_gpa) 339 298 { 340 299 struct hv_vp_assist_page assist_page; ··· 349 334 * versions: lower 8 bits is the minimal version, higher 8 bits is the 350 335 * maximum supported version. KVM supports versions from 1 to 351 336 * KVM_EVMCS_VERSION. 337 337 + * 338 338 + * Note, do not check the Hyper-V is fully enabled in guest CPUID, this 339 339 + * helper is used to _get_ the vCPU's supported CPUID. 352 340 */ 353 341 if (kvm_cpu_cap_get(X86_FEATURE_VMX) && 354 342 (!vcpu || to_vmx(vcpu)->nested.enlightened_vmcs_enabled)) ··· 360 342 return 0; 361 343 } 362 344 363 363 - void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata) 345 345 + enum evmcs_revision { 346 346 + EVMCSv1_LEGACY, 347 347 + NR_EVMCS_REVISIONS, 348 348 + }; 349 349 + 350 350 + enum evmcs_ctrl_type { 351 351 + EVMCS_EXIT_CTRLS, 352 352 + EVMCS_ENTRY_CTRLS, 353 353 + EVMCS_2NDEXEC, 354 354 + EVMCS_PINCTRL, 355 355 + EVMCS_VMFUNC, 356 356 + NR_EVMCS_CTRLS, 357 357 + }; 358 358 + 359 359 + static const u32 evmcs_unsupported_ctrls[NR_EVMCS_CTRLS][NR_EVMCS_REVISIONS] = { 360 360 + [EVMCS_EXIT_CTRLS] = { 361 361 + [EVMCSv1_LEGACY] = EVMCS1_UNSUPPORTED_VMEXIT_CTRL, 362 362 + }, 363 363 + [EVMCS_ENTRY_CTRLS] = { 364 364 + [EVMCSv1_LEGACY] = EVMCS1_UNSUPPORTED_VMENTRY_CTRL, 365 365 + }, 366 366 + [EVMCS_2NDEXEC] = { 367 367 + [EVMCSv1_LEGACY] = EVMCS1_UNSUPPORTED_2NDEXEC, 368 368 + }, 369 369 + [EVMCS_PINCTRL] = { 370 370 + [EVMCSv1_LEGACY] = EVMCS1_UNSUPPORTED_PINCTRL, 371 371 + }, 372 372 + [EVMCS_VMFUNC] = { 373 373 + [EVMCSv1_LEGACY] = EVMCS1_UNSUPPORTED_VMFUNC, 374 374 + }, 375 375 + }; 376 376 + 377 377 + static u32 evmcs_get_unsupported_ctls(enum evmcs_ctrl_type ctrl_type) 378 378 + { 379 379 + enum evmcs_revision evmcs_rev = EVMCSv1_LEGACY; 380 380 + 381 381 + return evmcs_unsupported_ctrls[ctrl_type][evmcs_rev]; 382 382 + } 383 383 + 384 384 + static bool evmcs_has_perf_global_ctrl(struct kvm_vcpu *vcpu) 385 385 + { 386 386 + struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); 387 387 + 388 388 + /* 389 389 + * PERF_GLOBAL_CTRL has a quirk where some Windows guests may fail to 390 390 + * boot if a PV CPUID feature flag is not also set. Treat the fields 391 391 + * as unsupported if the flag is not set in guest CPUID. This should 392 392 + * be called only for guest accesses, and all guest accesses should be 393 393 + * gated on Hyper-V being enabled and initialized. 394 394 + */ 395 395 + if (WARN_ON_ONCE(!hv_vcpu)) 396 396 + return false; 397 397 + 398 398 + return hv_vcpu->cpuid_cache.nested_ebx & HV_X64_NESTED_EVMCS1_PERF_GLOBAL_CTRL; 399 399 + } 400 400 + 401 401 + void nested_evmcs_filter_control_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 *pdata) 364 402 { 365 403 u32 ctl_low = (u32)*pdata; 366 404 u32 ctl_high = (u32)(*pdata >> 32); 405 405 + u32 unsupported_ctrls; 367 406 368 407 /* 369 408 * Hyper-V 2016 and 2019 try using these features even when eVMCS ··· 429 354 switch (msr_index) { 430 355 case MSR_IA32_VMX_EXIT_CTLS: 431 356 case MSR_IA32_VMX_TRUE_EXIT_CTLS: 432 432 - ctl_high &= ~EVMCS1_UNSUPPORTED_VMEXIT_CTRL; 357 357 + unsupported_ctrls = evmcs_get_unsupported_ctls(EVMCS_EXIT_CTRLS); 358 358 + if (!evmcs_has_perf_global_ctrl(vcpu)) 359 359 + unsupported_ctrls |= VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL; 360 360 + ctl_high &= ~unsupported_ctrls; 433 361 break; 434 362 case MSR_IA32_VMX_ENTRY_CTLS: 435 363 case MSR_IA32_VMX_TRUE_ENTRY_CTLS: 436 436 - ctl_high &= ~EVMCS1_UNSUPPORTED_VMENTRY_CTRL; 364 364 + unsupported_ctrls = evmcs_get_unsupported_ctls(EVMCS_ENTRY_CTRLS); 365 365 + if (!evmcs_has_perf_global_ctrl(vcpu)) 366 366 + unsupported_ctrls |= VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL; 367 367 + ctl_high &= ~unsupported_ctrls; 437 368 break; 438 369 case MSR_IA32_VMX_PROCBASED_CTLS2: 439 439 - ctl_high &= ~EVMCS1_UNSUPPORTED_2NDEXEC; 370 370 + ctl_high &= ~evmcs_get_unsupported_ctls(EVMCS_2NDEXEC); 440 371 break; 441 372 case MSR_IA32_VMX_TRUE_PINBASED_CTLS: 442 373 case MSR_IA32_VMX_PINBASED_CTLS: 443 443 - ctl_high &= ~EVMCS1_UNSUPPORTED_PINCTRL; 374 374 + ctl_high &= ~evmcs_get_unsupported_ctls(EVMCS_PINCTRL); 444 375 break; 445 376 case MSR_IA32_VMX_VMFUNC: 446 446 - ctl_low &= ~EVMCS1_UNSUPPORTED_VMFUNC; 377 377 + ctl_low &= ~evmcs_get_unsupported_ctls(EVMCS_VMFUNC); 447 378 break; 448 379 } 449 380 450 381 *pdata = ctl_low | ((u64)ctl_high << 32); 451 382 } 452 383 384 384 + static bool nested_evmcs_is_valid_controls(enum evmcs_ctrl_type ctrl_type, 385 385 + u32 val) 386 386 + { 387 387 + return !(val & evmcs_get_unsupported_ctls(ctrl_type)); 388 388 + } 389 389 + 453 390 int nested_evmcs_check_controls(struct vmcs12 *vmcs12) 454 391 { 455 455 - int ret = 0; 456 456 - u32 unsupp_ctl; 392 392 + if (CC(!nested_evmcs_is_valid_controls(EVMCS_PINCTRL, 393 393 + vmcs12->pin_based_vm_exec_control))) 394 394 + return -EINVAL; 457 395 458 458 - unsupp_ctl = vmcs12->pin_based_vm_exec_control & 459 459 - EVMCS1_UNSUPPORTED_PINCTRL; 460 460 - if (unsupp_ctl) { 461 461 - trace_kvm_nested_vmenter_failed( 462 462 - "eVMCS: unsupported pin-based VM-execution controls", 463 463 - unsupp_ctl); 464 464 - ret = -EINVAL; 465 465 - } 396 396 + if (CC(!nested_evmcs_is_valid_controls(EVMCS_2NDEXEC, 397 397 + vmcs12->secondary_vm_exec_control))) 398 398 + return -EINVAL; 466 399 467 467 - unsupp_ctl = vmcs12->secondary_vm_exec_control & 468 468 - EVMCS1_UNSUPPORTED_2NDEXEC; 469 469 - if (unsupp_ctl) { 470 470 - trace_kvm_nested_vmenter_failed( 471 471 - "eVMCS: unsupported secondary VM-execution controls", 472 472 - unsupp_ctl); 473 473 - ret = -EINVAL; 474 474 - } 400 400 + if (CC(!nested_evmcs_is_valid_controls(EVMCS_EXIT_CTRLS, 401 401 + vmcs12->vm_exit_controls))) 402 402 + return -EINVAL; 475 403 476 476 - unsupp_ctl = vmcs12->vm_exit_controls & 477 477 - EVMCS1_UNSUPPORTED_VMEXIT_CTRL; 478 478 - if (unsupp_ctl) { 479 479 - trace_kvm_nested_vmenter_failed( 480 480 - "eVMCS: unsupported VM-exit controls", 481 481 - unsupp_ctl); 482 482 - ret = -EINVAL; 483 483 - } 404 404 + if (CC(!nested_evmcs_is_valid_controls(EVMCS_ENTRY_CTRLS, 405 405 + vmcs12->vm_entry_controls))) 406 406 + return -EINVAL; 484 407 485 485 - unsupp_ctl = vmcs12->vm_entry_controls & 486 486 - EVMCS1_UNSUPPORTED_VMENTRY_CTRL; 487 487 - if (unsupp_ctl) { 488 488 - trace_kvm_nested_vmenter_failed( 489 489 - "eVMCS: unsupported VM-entry controls", 490 490 - unsupp_ctl); 491 491 - ret = -EINVAL; 492 492 - } 408 408 + /* 409 409 + * VM-Func controls are 64-bit, but KVM currently doesn't support any 410 410 + * controls in bits 63:32, i.e. dropping those bits on the consistency 411 411 + * check is intentional. 412 412 + */ 413 413 + if (WARN_ON_ONCE(vmcs12->vm_function_control >> 32)) 414 414 + return -EINVAL; 493 415 494 494 - unsupp_ctl = vmcs12->vm_function_control & EVMCS1_UNSUPPORTED_VMFUNC; 495 495 - if (unsupp_ctl) { 496 496 - trace_kvm_nested_vmenter_failed( 497 497 - "eVMCS: unsupported VM-function controls", 498 498 - unsupp_ctl); 499 499 - ret = -EINVAL; 500 500 - } 416 416 + if (CC(!nested_evmcs_is_valid_controls(EVMCS_VMFUNC, 417 417 + vmcs12->vm_function_control))) 418 418 + return -EINVAL; 501 419 502 502 - return ret; 420 420 + return 0; 503 421 } 504 422 505 423 int nested_enable_evmcs(struct kvm_vcpu *vcpu,

+3 -7

arch/x86/kvm/vmx/evmcs.h

reviewed

··· 42 42 * PLE_GAP = 0x00004020, 43 43 * PLE_WINDOW = 0x00004022, 44 44 * VMX_PREEMPTION_TIMER_VALUE = 0x0000482E, 45 45 - * GUEST_IA32_PERF_GLOBAL_CTRL = 0x00002808, 46 46 - * HOST_IA32_PERF_GLOBAL_CTRL = 0x00002c04, 47 45 * 48 46 * Currently unsupported in KVM: 49 47 * GUEST_IA32_RTIT_CTL = 0x00002814, ··· 59 61 SECONDARY_EXEC_TSC_SCALING | \ 60 62 SECONDARY_EXEC_PAUSE_LOOP_EXITING) 61 63 #define EVMCS1_UNSUPPORTED_VMEXIT_CTRL \ 62 62 - (VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL | \ 63 63 - VM_EXIT_SAVE_VMX_PREEMPTION_TIMER) 64 64 - #define EVMCS1_UNSUPPORTED_VMENTRY_CTRL (VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL) 64 64 + (VM_EXIT_SAVE_VMX_PREEMPTION_TIMER) 65 65 + #define EVMCS1_UNSUPPORTED_VMENTRY_CTRL (0) 65 66 #define EVMCS1_UNSUPPORTED_VMFUNC (VMX_VMFUNC_EPTP_SWITCHING) 66 67 67 68 struct evmcs_field { ··· 209 212 vp_ap->enlighten_vmentry = 1; 210 213 } 211 214 212 212 - __init void evmcs_sanitize_exec_ctrls(struct vmcs_config *vmcs_conf); 213 215 #else /* !IS_ENABLED(CONFIG_HYPERV) */ 214 216 static __always_inline void evmcs_write64(unsigned long field, u64 value) {} 215 217 static inline void evmcs_write32(unsigned long field, u32 value) {} ··· 239 243 uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu); 240 244 int nested_enable_evmcs(struct kvm_vcpu *vcpu, 241 245 uint16_t *vmcs_version); 242 242 - void nested_evmcs_filter_control_msr(u32 msr_index, u64 *pdata); 246 246 + void nested_evmcs_filter_control_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 *pdata); 243 247 int nested_evmcs_check_controls(struct vmcs12 *vmcs12); 244 248 245 249 #endif /* __KVM_X86_VMX_EVMCS_H */

+313 -155

arch/x86/kvm/vmx/nested.c

reviewed

··· 439 439 return inequality ^ bit; 440 440 } 441 441 442 442 - 443 443 - /* 444 444 - * KVM wants to inject page-faults which it got to the guest. This function 445 445 - * checks whether in a nested guest, we need to inject them to L1 or L2. 446 446 - */ 447 447 - static int nested_vmx_check_exception(struct kvm_vcpu *vcpu, unsigned long *exit_qual) 448 448 - { 449 449 - struct vmcs12 *vmcs12 = get_vmcs12(vcpu); 450 450 - unsigned int nr = vcpu->arch.exception.nr; 451 451 - bool has_payload = vcpu->arch.exception.has_payload; 452 452 - unsigned long payload = vcpu->arch.exception.payload; 453 453 - 454 454 - if (nr == PF_VECTOR) { 455 455 - if (vcpu->arch.exception.nested_apf) { 456 456 - *exit_qual = vcpu->arch.apf.nested_apf_token; 457 457 - return 1; 458 458 - } 459 459 - if (nested_vmx_is_page_fault_vmexit(vmcs12, 460 460 - vcpu->arch.exception.error_code)) { 461 461 - *exit_qual = has_payload ? payload : vcpu->arch.cr2; 462 462 - return 1; 463 463 - } 464 464 - } else if (vmcs12->exception_bitmap & (1u << nr)) { 465 465 - if (nr == DB_VECTOR) { 466 466 - if (!has_payload) { 467 467 - payload = vcpu->arch.dr6; 468 468 - payload &= ~DR6_BT; 469 469 - payload ^= DR6_ACTIVE_LOW; 470 470 - } 471 471 - *exit_qual = payload; 472 472 - } else 473 473 - *exit_qual = 0; 474 474 - return 1; 475 475 - } 476 476 - 477 477 - return 0; 478 478 - } 479 479 - 480 480 - static bool nested_vmx_handle_page_fault_workaround(struct kvm_vcpu *vcpu, 481 481 - struct x86_exception *fault) 442 442 + static bool nested_vmx_is_exception_vmexit(struct kvm_vcpu *vcpu, u8 vector, 443 443 + u32 error_code) 482 444 { 483 445 struct vmcs12 *vmcs12 = get_vmcs12(vcpu); 484 446 485 485 - WARN_ON(!is_guest_mode(vcpu)); 447 447 + /* 448 448 + * Drop bits 31:16 of the error code when performing the #PF mask+match 449 449 + * check. All VMCS fields involved are 32 bits, but Intel CPUs never 450 450 + * set bits 31:16 and VMX disallows setting bits 31:16 in the injected 451 451 + * error code. Including the to-be-dropped bits in the check might 452 452 + * result in an "impossible" or missed exit from L1's perspective. 453 453 + */ 454 454 + if (vector == PF_VECTOR) 455 455 + return nested_vmx_is_page_fault_vmexit(vmcs12, (u16)error_code); 486 456 487 487 - if (nested_vmx_is_page_fault_vmexit(vmcs12, fault->error_code) && 488 488 - !WARN_ON_ONCE(to_vmx(vcpu)->nested.nested_run_pending)) { 489 489 - vmcs12->vm_exit_intr_error_code = fault->error_code; 490 490 - nested_vmx_vmexit(vcpu, EXIT_REASON_EXCEPTION_NMI, 491 491 - PF_VECTOR | INTR_TYPE_HARD_EXCEPTION | 492 492 - INTR_INFO_DELIVER_CODE_MASK | INTR_INFO_VALID_MASK, 493 493 - fault->address); 494 494 - return true; 495 495 - } 496 496 - return false; 457 457 + return (vmcs12->exception_bitmap & (1u << vector)); 497 458 } 498 459 499 460 static int nested_vmx_check_io_bitmap_controls(struct kvm_vcpu *vcpu, ··· 1568 1607 vmcs12->guest_rflags = evmcs->guest_rflags; 1569 1608 vmcs12->guest_interruptibility_info = 1570 1609 evmcs->guest_interruptibility_info; 1610 1610 + /* 1611 1611 + * Not present in struct vmcs12: 1612 1612 + * vmcs12->guest_ssp = evmcs->guest_ssp; 1613 1613 + */ 1571 1614 } 1572 1615 1573 1616 if (unlikely(!(hv_clean_fields & ··· 1618 1653 vmcs12->host_fs_selector = evmcs->host_fs_selector; 1619 1654 vmcs12->host_gs_selector = evmcs->host_gs_selector; 1620 1655 vmcs12->host_tr_selector = evmcs->host_tr_selector; 1656 1656 + vmcs12->host_ia32_perf_global_ctrl = evmcs->host_ia32_perf_global_ctrl; 1657 1657 + /* 1658 1658 + * Not present in struct vmcs12: 1659 1659 + * vmcs12->host_ia32_s_cet = evmcs->host_ia32_s_cet; 1660 1660 + * vmcs12->host_ssp = evmcs->host_ssp; 1661 1661 + * vmcs12->host_ia32_int_ssp_table_addr = evmcs->host_ia32_int_ssp_table_addr; 1662 1662 + */ 1621 1663 } 1622 1664 1623 1665 if (unlikely(!(hv_clean_fields & ··· 1692 1720 vmcs12->tsc_offset = evmcs->tsc_offset; 1693 1721 vmcs12->virtual_apic_page_addr = evmcs->virtual_apic_page_addr; 1694 1722 vmcs12->xss_exit_bitmap = evmcs->xss_exit_bitmap; 1723 1723 + vmcs12->encls_exiting_bitmap = evmcs->encls_exiting_bitmap; 1724 1724 + vmcs12->tsc_multiplier = evmcs->tsc_multiplier; 1695 1725 } 1696 1726 1697 1727 if (unlikely(!(hv_clean_fields & ··· 1741 1767 vmcs12->guest_bndcfgs = evmcs->guest_bndcfgs; 1742 1768 vmcs12->guest_activity_state = evmcs->guest_activity_state; 1743 1769 vmcs12->guest_sysenter_cs = evmcs->guest_sysenter_cs; 1770 1770 + vmcs12->guest_ia32_perf_global_ctrl = evmcs->guest_ia32_perf_global_ctrl; 1771 1771 + /* 1772 1772 + * Not present in struct vmcs12: 1773 1773 + * vmcs12->guest_ia32_s_cet = evmcs->guest_ia32_s_cet; 1774 1774 + * vmcs12->guest_ia32_lbr_ctl = evmcs->guest_ia32_lbr_ctl; 1775 1775 + * vmcs12->guest_ia32_int_ssp_table_addr = evmcs->guest_ia32_int_ssp_table_addr; 1776 1776 + */ 1744 1777 } 1745 1778 1746 1779 /* ··· 1850 1869 * evmcs->vm_exit_msr_store_count = vmcs12->vm_exit_msr_store_count; 1851 1870 * evmcs->vm_exit_msr_load_count = vmcs12->vm_exit_msr_load_count; 1852 1871 * evmcs->vm_entry_msr_load_count = vmcs12->vm_entry_msr_load_count; 1872 1872 + * evmcs->guest_ia32_perf_global_ctrl = vmcs12->guest_ia32_perf_global_ctrl; 1873 1873 + * evmcs->host_ia32_perf_global_ctrl = vmcs12->host_ia32_perf_global_ctrl; 1874 1874 + * evmcs->encls_exiting_bitmap = vmcs12->encls_exiting_bitmap; 1875 1875 + * evmcs->tsc_multiplier = vmcs12->tsc_multiplier; 1853 1876 * 1854 1877 * Not present in struct vmcs12: 1855 1878 * evmcs->exit_io_instruction_ecx = vmcs12->exit_io_instruction_ecx; 1856 1879 * evmcs->exit_io_instruction_esi = vmcs12->exit_io_instruction_esi; 1857 1880 * evmcs->exit_io_instruction_edi = vmcs12->exit_io_instruction_edi; 1858 1881 * evmcs->exit_io_instruction_eip = vmcs12->exit_io_instruction_eip; 1882 1882 + * evmcs->host_ia32_s_cet = vmcs12->host_ia32_s_cet; 1883 1883 + * evmcs->host_ssp = vmcs12->host_ssp; 1884 1884 + * evmcs->host_ia32_int_ssp_table_addr = vmcs12->host_ia32_int_ssp_table_addr; 1885 1885 + * evmcs->guest_ia32_s_cet = vmcs12->guest_ia32_s_cet; 1886 1886 + * evmcs->guest_ia32_lbr_ctl = vmcs12->guest_ia32_lbr_ctl; 1887 1887 + * evmcs->guest_ia32_int_ssp_table_addr = vmcs12->guest_ia32_int_ssp_table_addr; 1888 1888 + * evmcs->guest_ssp = vmcs12->guest_ssp; 1859 1889 */ 1860 1890 1861 1891 evmcs->guest_es_selector = vmcs12->guest_es_selector; ··· 1974 1982 bool evmcs_gpa_changed = false; 1975 1983 u64 evmcs_gpa; 1976 1984 1977 1977 - if (likely(!vmx->nested.enlightened_vmcs_enabled)) 1985 1985 + if (likely(!guest_cpuid_has_evmcs(vcpu))) 1978 1986 return EVMPTRLD_DISABLED; 1979 1987 1980 1988 if (!nested_enlightened_vmentry(vcpu, &evmcs_gpa)) { ··· 2320 2328 * are emulated by vmx_set_efer() in prepare_vmcs02(), but speculate 2321 2329 * on the related bits (if supported by the CPU) in the hope that 2322 2330 * we can avoid VMWrites during vmx_set_efer(). 2331 2331 + * 2332 2332 + * Similarly, take vmcs01's PERF_GLOBAL_CTRL in the hope that if KVM is 2333 2333 + * loading PERF_GLOBAL_CTRL via the VMCS for L1, then KVM will want to 2334 2334 + * do the same for L2. 2323 2335 */ 2324 2336 exec_control = __vm_entry_controls_get(vmcs01); 2325 2325 - exec_control |= vmcs12->vm_entry_controls; 2337 2337 + exec_control |= (vmcs12->vm_entry_controls & 2338 2338 + ~VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL); 2326 2339 exec_control &= ~(VM_ENTRY_IA32E_MODE | VM_ENTRY_LOAD_IA32_EFER); 2327 2340 if (cpu_has_load_ia32_efer()) { 2328 2341 if (guest_efer & EFER_LMA) ··· 2860 2863 nested_check_vm_entry_controls(vcpu, vmcs12)) 2861 2864 return -EINVAL; 2862 2865 2863 2863 - if (to_vmx(vcpu)->nested.enlightened_vmcs_enabled) 2866 2866 + if (guest_cpuid_has_evmcs(vcpu)) 2864 2867 return nested_evmcs_check_controls(vmcs12); 2865 2868 2866 2869 return 0; ··· 3142 3145 * L2 was running), map it here to make sure vmcs12 changes are 3143 3146 * properly reflected. 3144 3147 */ 3145 3145 - if (vmx->nested.enlightened_vmcs_enabled && 3148 3148 + if (guest_cpuid_has_evmcs(vcpu) && 3146 3149 vmx->nested.hv_evmcs_vmptr == EVMPTR_MAP_PENDING) { 3147 3150 enum nested_evmptrld_status evmptrld_status = 3148 3151 nested_vmx_handle_enlightened_vmptrld(vcpu, false); ··· 3361 3364 }; 3362 3365 u32 failed_index; 3363 3366 3367 3367 + trace_kvm_nested_vmenter(kvm_rip_read(vcpu), 3368 3368 + vmx->nested.current_vmptr, 3369 3369 + vmcs12->guest_rip, 3370 3370 + vmcs12->guest_intr_status, 3371 3371 + vmcs12->vm_entry_intr_info_field, 3372 3372 + vmcs12->secondary_vm_exec_control & SECONDARY_EXEC_ENABLE_EPT, 3373 3373 + vmcs12->ept_pointer, 3374 3374 + vmcs12->guest_cr3, 3375 3375 + KVM_ISA_VMX); 3376 3376 + 3364 3377 kvm_service_local_tlb_flush_requests(vcpu); 3365 3378 3366 3379 evaluate_pending_interrupts = exec_controls_get(vmx) & 3367 3380 (CPU_BASED_INTR_WINDOW_EXITING | CPU_BASED_NMI_WINDOW_EXITING); 3368 3381 if (likely(!evaluate_pending_interrupts) && kvm_vcpu_apicv_active(vcpu)) 3369 3382 evaluate_pending_interrupts |= vmx_has_apicv_interrupt(vcpu); 3383 3383 + if (!evaluate_pending_interrupts) 3384 3384 + evaluate_pending_interrupts |= kvm_apic_has_pending_init_or_sipi(vcpu); 3370 3385 3371 3386 if (!vmx->nested.nested_run_pending || 3372 3387 !(vmcs12->vm_entry_controls & VM_ENTRY_LOAD_DEBUG_CONTROLS)) ··· 3459 3450 } 3460 3451 3461 3452 /* 3462 3462 - * If L1 had a pending IRQ/NMI until it executed 3463 3463 - * VMLAUNCH/VMRESUME which wasn't delivered because it was 3464 3464 - * disallowed (e.g. interrupts disabled), L0 needs to 3465 3465 - * evaluate if this pending event should cause an exit from L2 3466 3466 - * to L1 or delivered directly to L2 (e.g. In case L1 don't 3467 3467 - * intercept EXTERNAL_INTERRUPT). 3468 3468 - * 3469 3469 - * Usually this would be handled by the processor noticing an 3470 3470 - * IRQ/NMI window request, or checking RVI during evaluation of 3471 3471 - * pending virtual interrupts. However, this setting was done 3472 3472 - * on VMCS01 and now VMCS02 is active instead. Thus, we force L0 3473 3473 - * to perform pending event evaluation by requesting a KVM_REQ_EVENT. 3453 3453 + * Re-evaluate pending events if L1 had a pending IRQ/NMI/INIT/SIPI 3454 3454 + * when it executed VMLAUNCH/VMRESUME, as entering non-root mode can 3455 3455 + * effectively unblock various events, e.g. INIT/SIPI cause VM-Exit 3456 3456 + * unconditionally. 3474 3457 */ 3475 3458 if (unlikely(evaluate_pending_interrupts)) 3476 3459 kvm_make_request(KVM_REQ_EVENT, vcpu); ··· 3719 3718 is_double_fault(exit_intr_info))) { 3720 3719 vmcs12->idt_vectoring_info_field = 0; 3721 3720 } else if (vcpu->arch.exception.injected) { 3722 3722 - nr = vcpu->arch.exception.nr; 3721 3721 + nr = vcpu->arch.exception.vector; 3723 3722 idt_vectoring = nr | VECTORING_INFO_VALID_MASK; 3724 3723 3725 3724 if (kvm_exception_is_soft(nr)) { ··· 3820 3819 return -ENXIO; 3821 3820 } 3822 3821 3823 3823 - static void nested_vmx_inject_exception_vmexit(struct kvm_vcpu *vcpu, 3824 3824 - unsigned long exit_qual) 3822 3822 + static void nested_vmx_inject_exception_vmexit(struct kvm_vcpu *vcpu) 3825 3823 { 3824 3824 + struct kvm_queued_exception *ex = &vcpu->arch.exception_vmexit; 3825 3825 + u32 intr_info = ex->vector | INTR_INFO_VALID_MASK; 3826 3826 struct vmcs12 *vmcs12 = get_vmcs12(vcpu); 3827 3827 - unsigned int nr = vcpu->arch.exception.nr; 3828 3828 - u32 intr_info = nr | INTR_INFO_VALID_MASK; 3827 3827 + unsigned long exit_qual; 3829 3828 3830 3830 - if (vcpu->arch.exception.has_error_code) { 3831 3831 - vmcs12->vm_exit_intr_error_code = vcpu->arch.exception.error_code; 3829 3829 + if (ex->has_payload) { 3830 3830 + exit_qual = ex->payload; 3831 3831 + } else if (ex->vector == PF_VECTOR) { 3832 3832 + exit_qual = vcpu->arch.cr2; 3833 3833 + } else if (ex->vector == DB_VECTOR) { 3834 3834 + exit_qual = vcpu->arch.dr6; 3835 3835 + exit_qual &= ~DR6_BT; 3836 3836 + exit_qual ^= DR6_ACTIVE_LOW; 3837 3837 + } else { 3838 3838 + exit_qual = 0; 3839 3839 + } 3840 3840 + 3841 3841 + if (ex->has_error_code) { 3842 3842 + /* 3843 3843 + * Intel CPUs do not generate error codes with bits 31:16 set, 3844 3844 + * and more importantly VMX disallows setting bits 31:16 in the 3845 3845 + * injected error code for VM-Entry. Drop the bits to mimic 3846 3846 + * hardware and avoid inducing failure on nested VM-Entry if L1 3847 3847 + * chooses to inject the exception back to L2. AMD CPUs _do_ 3848 3848 + * generate "full" 32-bit error codes, so KVM allows userspace 3849 3849 + * to inject exception error codes with bits 31:16 set. 3850 3850 + */ 3851 3851 + vmcs12->vm_exit_intr_error_code = (u16)ex->error_code; 3832 3852 intr_info |= INTR_INFO_DELIVER_CODE_MASK; 3833 3853 } 3834 3854 3835 3835 - if (kvm_exception_is_soft(nr)) 3855 3855 + if (kvm_exception_is_soft(ex->vector)) 3836 3856 intr_info |= INTR_TYPE_SOFT_EXCEPTION; 3837 3857 else 3838 3858 intr_info |= INTR_TYPE_HARD_EXCEPTION; ··· 3866 3844 } 3867 3845 3868 3846 /* 3869 3869 - * Returns true if a debug trap is pending delivery. 3847 3847 + * Returns true if a debug trap is (likely) pending delivery. Infer the class 3848 3848 + * of a #DB (trap-like vs. fault-like) from the exception payload (to-be-DR6). 3849 3849 + * Using the payload is flawed because code breakpoints (fault-like) and data 3850 3850 + * breakpoints (trap-like) set the same bits in DR6 (breakpoint detected), i.e. 3851 3851 + * this will return false positives if a to-be-injected code breakpoint #DB is 3852 3852 + * pending (from KVM's perspective, but not "pending" across an instruction 3853 3853 + * boundary). ICEBP, a.k.a. INT1, is also not reflected here even though it 3854 3854 + * too is trap-like. 3870 3855 * 3871 3871 - * In KVM, debug traps bear an exception payload. As such, the class of a #DB 3872 3872 - * exception may be inferred from the presence of an exception payload. 3856 3856 + * KVM "works" despite these flaws as ICEBP isn't currently supported by the 3857 3857 + * emulator, Monitor Trap Flag is not marked pending on intercepted #DBs (the 3858 3858 + * #DB has already happened), and MTF isn't marked pending on code breakpoints 3859 3859 + * from the emulator (because such #DBs are fault-like and thus don't trigger 3860 3860 + * actions that fire on instruction retire). 3873 3861 */ 3874 3874 - static inline bool vmx_pending_dbg_trap(struct kvm_vcpu *vcpu) 3862 3862 + static unsigned long vmx_get_pending_dbg_trap(struct kvm_queued_exception *ex) 3875 3863 { 3876 3876 - return vcpu->arch.exception.pending && 3877 3877 - vcpu->arch.exception.nr == DB_VECTOR && 3878 3878 - vcpu->arch.exception.payload; 3864 3864 + if (!ex->pending || ex->vector != DB_VECTOR) 3865 3865 + return 0; 3866 3866 + 3867 3867 + /* General Detect #DBs are always fault-like. */ 3868 3868 + return ex->payload & ~DR6_BD; 3869 3869 + } 3870 3870 + 3871 3871 + /* 3872 3872 + * Returns true if there's a pending #DB exception that is lower priority than 3873 3873 + * a pending Monitor Trap Flag VM-Exit. TSS T-flag #DBs are not emulated by 3874 3874 + * KVM, but could theoretically be injected by userspace. Note, this code is 3875 3875 + * imperfect, see above. 3876 3876 + */ 3877 3877 + static bool vmx_is_low_priority_db_trap(struct kvm_queued_exception *ex) 3878 3878 + { 3879 3879 + return vmx_get_pending_dbg_trap(ex) & ~DR6_BT; 3879 3880 } 3880 3881 3881 3882 /* ··· 3910 3865 */ 3911 3866 static void nested_vmx_update_pending_dbg(struct kvm_vcpu *vcpu) 3912 3867 { 3913 3913 - if (vmx_pending_dbg_trap(vcpu)) 3914 3914 - vmcs_writel(GUEST_PENDING_DBG_EXCEPTIONS, 3915 3915 - vcpu->arch.exception.payload); 3868 3868 + unsigned long pending_dbg; 3869 3869 + 3870 3870 + pending_dbg = vmx_get_pending_dbg_trap(&vcpu->arch.exception); 3871 3871 + if (pending_dbg) 3872 3872 + vmcs_writel(GUEST_PENDING_DBG_EXCEPTIONS, pending_dbg); 3916 3873 } 3917 3874 3918 3875 static bool nested_vmx_preemption_timer_pending(struct kvm_vcpu *vcpu) ··· 3923 3876 to_vmx(vcpu)->nested.preemption_timer_expired; 3924 3877 } 3925 3878 3879 3879 + static bool vmx_has_nested_events(struct kvm_vcpu *vcpu) 3880 3880 + { 3881 3881 + return nested_vmx_preemption_timer_pending(vcpu) || 3882 3882 + to_vmx(vcpu)->nested.mtf_pending; 3883 3883 + } 3884 3884 + 3885 3885 + /* 3886 3886 + * Per the Intel SDM's table "Priority Among Concurrent Events", with minor 3887 3887 + * edits to fill in missing examples, e.g. #DB due to split-lock accesses, 3888 3888 + * and less minor edits to splice in the priority of VMX Non-Root specific 3889 3889 + * events, e.g. MTF and NMI/INTR-window exiting. 3890 3890 + * 3891 3891 + * 1 Hardware Reset and Machine Checks 3892 3892 + * - RESET 3893 3893 + * - Machine Check 3894 3894 + * 3895 3895 + * 2 Trap on Task Switch 3896 3896 + * - T flag in TSS is set (on task switch) 3897 3897 + * 3898 3898 + * 3 External Hardware Interventions 3899 3899 + * - FLUSH 3900 3900 + * - STOPCLK 3901 3901 + * - SMI 3902 3902 + * - INIT 3903 3903 + * 3904 3904 + * 3.5 Monitor Trap Flag (MTF) VM-exit[1] 3905 3905 + * 3906 3906 + * 4 Traps on Previous Instruction 3907 3907 + * - Breakpoints 3908 3908 + * - Trap-class Debug Exceptions (#DB due to TF flag set, data/I-O 3909 3909 + * breakpoint, or #DB due to a split-lock access) 3910 3910 + * 3911 3911 + * 4.3 VMX-preemption timer expired VM-exit 3912 3912 + * 3913 3913 + * 4.6 NMI-window exiting VM-exit[2] 3914 3914 + * 3915 3915 + * 5 Nonmaskable Interrupts (NMI) 3916 3916 + * 3917 3917 + * 5.5 Interrupt-window exiting VM-exit and Virtual-interrupt delivery 3918 3918 + * 3919 3919 + * 6 Maskable Hardware Interrupts 3920 3920 + * 3921 3921 + * 7 Code Breakpoint Fault 3922 3922 + * 3923 3923 + * 8 Faults from Fetching Next Instruction 3924 3924 + * - Code-Segment Limit Violation 3925 3925 + * - Code Page Fault 3926 3926 + * - Control protection exception (missing ENDBRANCH at target of indirect 3927 3927 + * call or jump) 3928 3928 + * 3929 3929 + * 9 Faults from Decoding Next Instruction 3930 3930 + * - Instruction length > 15 bytes 3931 3931 + * - Invalid Opcode 3932 3932 + * - Coprocessor Not Available 3933 3933 + * 3934 3934 + *10 Faults on Executing Instruction 3935 3935 + * - Overflow 3936 3936 + * - Bound error 3937 3937 + * - Invalid TSS 3938 3938 + * - Segment Not Present 3939 3939 + * - Stack fault 3940 3940 + * - General Protection 3941 3941 + * - Data Page Fault 3942 3942 + * - Alignment Check 3943 3943 + * - x86 FPU Floating-point exception 3944 3944 + * - SIMD floating-point exception 3945 3945 + * - Virtualization exception 3946 3946 + * - Control protection exception 3947 3947 + * 3948 3948 + * [1] Per the "Monitor Trap Flag" section: System-management interrupts (SMIs), 3949 3949 + * INIT signals, and higher priority events take priority over MTF VM exits. 3950 3950 + * MTF VM exits take priority over debug-trap exceptions and lower priority 3951 3951 + * events. 3952 3952 + * 3953 3953 + * [2] Debug-trap exceptions and higher priority events take priority over VM exits 3954 3954 + * caused by the VMX-preemption timer. VM exits caused by the VMX-preemption 3955 3955 + * timer take priority over VM exits caused by the "NMI-window exiting" 3956 3956 + * VM-execution control and lower priority events. 3957 3957 + * 3958 3958 + * [3] Debug-trap exceptions and higher priority events take priority over VM exits 3959 3959 + * caused by "NMI-window exiting". VM exits caused by this control take 3960 3960 + * priority over non-maskable interrupts (NMIs) and lower priority events. 3961 3961 + * 3962 3962 + * [4] Virtual-interrupt delivery has the same priority as that of VM exits due to 3963 3963 + * the 1-setting of the "interrupt-window exiting" VM-execution control. Thus, 3964 3964 + * non-maskable interrupts (NMIs) and higher priority events take priority over 3965 3965 + * delivery of a virtual interrupt; delivery of a virtual interrupt takes 3966 3966 + * priority over external interrupts and lower priority events. 3967 3967 + */ 3926 3968 static int vmx_check_nested_events(struct kvm_vcpu *vcpu) 3927 3969 { 3928 3928 - struct vcpu_vmx *vmx = to_vmx(vcpu); 3929 3929 - unsigned long exit_qual; 3930 3930 - bool block_nested_events = 3931 3931 - vmx->nested.nested_run_pending || kvm_event_needs_reinjection(vcpu); 3932 3932 - bool mtf_pending = vmx->nested.mtf_pending; 3933 3970 struct kvm_lapic *apic = vcpu->arch.apic; 3934 3934 - 3971 3971 + struct vcpu_vmx *vmx = to_vmx(vcpu); 3935 3972 /* 3936 3936 - * Clear the MTF state. If a higher priority VM-exit is delivered first, 3937 3937 - * this state is discarded. 3973 3973 + * Only a pending nested run blocks a pending exception. If there is a 3974 3974 + * previously injected event, the pending exception occurred while said 3975 3975 + * event was being delivered and thus needs to be handled. 3938 3976 */ 3939 3939 - if (!block_nested_events) 3940 3940 - vmx->nested.mtf_pending = false; 3977 3977 + bool block_nested_exceptions = vmx->nested.nested_run_pending; 3978 3978 + /* 3979 3979 + * New events (not exceptions) are only recognized at instruction 3980 3980 + * boundaries. If an event needs reinjection, then KVM is handling a 3981 3981 + * VM-Exit that occurred _during_ instruction execution; new events are 3982 3982 + * blocked until the instruction completes. 3983 3983 + */ 3984 3984 + bool block_nested_events = block_nested_exceptions || 3985 3985 + kvm_event_needs_reinjection(vcpu); 3941 3986 3942 3987 if (lapic_in_kernel(vcpu) && 3943 3988 test_bit(KVM_APIC_INIT, &apic->pending_events)) { ··· 4039 3900 clear_bit(KVM_APIC_INIT, &apic->pending_events); 4040 3901 if (vcpu->arch.mp_state != KVM_MP_STATE_INIT_RECEIVED) 4041 3902 nested_vmx_vmexit(vcpu, EXIT_REASON_INIT_SIGNAL, 0, 0); 3903 3903 + 3904 3904 + /* MTF is discarded if the vCPU is in WFS. */ 3905 3905 + vmx->nested.mtf_pending = false; 4042 3906 return 0; 4043 3907 } 4044 3908 ··· 4051 3909 return -EBUSY; 4052 3910 4053 3911 clear_bit(KVM_APIC_SIPI, &apic->pending_events); 4054 4054 - if (vcpu->arch.mp_state == KVM_MP_STATE_INIT_RECEIVED) 3912 3912 + if (vcpu->arch.mp_state == KVM_MP_STATE_INIT_RECEIVED) { 4055 3913 nested_vmx_vmexit(vcpu, EXIT_REASON_SIPI_SIGNAL, 0, 4056 3914 apic->sipi_vector & 0xFFUL); 4057 4057 - return 0; 3915 3915 + return 0; 3916 3916 + } 3917 3917 + /* Fallthrough, the SIPI is completely ignored. */ 4058 3918 } 4059 3919 4060 3920 /* 4061 4061 - * Process any exceptions that are not debug traps before MTF. 3921 3921 + * Process exceptions that are higher priority than Monitor Trap Flag: 3922 3922 + * fault-like exceptions, TSS T flag #DB (not emulated by KVM, but 3923 3923 + * could theoretically come in from userspace), and ICEBP (INT1). 4062 3924 * 4063 4063 - * Note that only a pending nested run can block a pending exception. 4064 4064 - * Otherwise an injected NMI/interrupt should either be 4065 4065 - * lost or delivered to the nested hypervisor in the IDT_VECTORING_INFO, 4066 4066 - * while delivering the pending exception. 3925 3925 + * TODO: SMIs have higher priority than MTF and trap-like #DBs (except 3926 3926 + * for TSS T flag #DBs). KVM also doesn't save/restore pending MTF 3927 3927 + * across SMI/RSM as it should; that needs to be addressed in order to 3928 3928 + * prioritize SMI over MTF and trap-like #DBs. 4067 3929 */ 4068 4068 - 4069 4069 - if (vcpu->arch.exception.pending && !vmx_pending_dbg_trap(vcpu)) { 4070 4070 - if (vmx->nested.nested_run_pending) 3930 3930 + if (vcpu->arch.exception_vmexit.pending && 3931 3931 + !vmx_is_low_priority_db_trap(&vcpu->arch.exception_vmexit)) { 3932 3932 + if (block_nested_exceptions) 4071 3933 return -EBUSY; 4072 4072 - if (!nested_vmx_check_exception(vcpu, &exit_qual)) 4073 4073 - goto no_vmexit; 4074 4074 - nested_vmx_inject_exception_vmexit(vcpu, exit_qual); 3934 3934 + 3935 3935 + nested_vmx_inject_exception_vmexit(vcpu); 4075 3936 return 0; 4076 3937 } 4077 3938 4078 4078 - if (mtf_pending) { 3939 3939 + if (vcpu->arch.exception.pending && 3940 3940 + !vmx_is_low_priority_db_trap(&vcpu->arch.exception)) { 3941 3941 + if (block_nested_exceptions) 3942 3942 + return -EBUSY; 3943 3943 + goto no_vmexit; 3944 3944 + } 3945 3945 + 3946 3946 + if (vmx->nested.mtf_pending) { 4079 3947 if (block_nested_events) 4080 3948 return -EBUSY; 4081 3949 nested_vmx_update_pending_dbg(vcpu); ··· 4093 3941 return 0; 4094 3942 } 4095 3943 4096 4096 - if (vcpu->arch.exception.pending) { 4097 4097 - if (vmx->nested.nested_run_pending) 3944 3944 + if (vcpu->arch.exception_vmexit.pending) { 3945 3945 + if (block_nested_exceptions) 4098 3946 return -EBUSY; 4099 4099 - if (!nested_vmx_check_exception(vcpu, &exit_qual)) 4100 4100 - goto no_vmexit; 4101 4101 - nested_vmx_inject_exception_vmexit(vcpu, exit_qual); 3947 3947 + 3948 3948 + nested_vmx_inject_exception_vmexit(vcpu); 4102 3949 return 0; 3950 3950 + } 3951 3951 + 3952 3952 + if (vcpu->arch.exception.pending) { 3953 3953 + if (block_nested_exceptions) 3954 3954 + return -EBUSY; 3955 3955 + goto no_vmexit; 4103 3956 } 4104 3957 4105 3958 if (nested_vmx_preemption_timer_pending(vcpu)) { ··· 4412 4255 nested_vmx_abort(vcpu, 4413 4256 VMX_ABORT_SAVE_GUEST_MSR_FAIL); 4414 4257 } 4415 4415 - 4416 4416 - /* 4417 4417 - * Drop what we picked up for L2 via vmx_complete_interrupts. It is 4418 4418 - * preserved above and would only end up incorrectly in L1. 4419 4419 - */ 4420 4420 - vcpu->arch.nmi_injected = false; 4421 4421 - kvm_clear_exception_queue(vcpu); 4422 4422 - kvm_clear_interrupt_queue(vcpu); 4423 4258 } 4424 4259 4425 4260 /* ··· 4687 4538 struct vcpu_vmx *vmx = to_vmx(vcpu); 4688 4539 struct vmcs12 *vmcs12 = get_vmcs12(vcpu); 4689 4540 4541 4541 + /* Pending MTF traps are discarded on VM-Exit. */ 4542 4542 + vmx->nested.mtf_pending = false; 4543 4543 + 4690 4544 /* trying to cancel vmlaunch/vmresume is a bug */ 4691 4545 WARN_ON_ONCE(vmx->nested.nested_run_pending); 4692 4546 ··· 4753 4601 VMXERR_ENTRY_INVALID_CONTROL_FIELD); 4754 4602 WARN_ON_ONCE(nested_early_check); 4755 4603 } 4604 4604 + 4605 4605 + /* 4606 4606 + * Drop events/exceptions that were queued for re-injection to L2 4607 4607 + * (picked up via vmx_complete_interrupts()), as well as exceptions 4608 4608 + * that were pending for L2. Note, this must NOT be hoisted above 4609 4609 + * prepare_vmcs12(), events/exceptions queued for re-injection need to 4610 4610 + * be captured in vmcs12 (see vmcs12_save_pending_event()). 4611 4611 + */ 4612 4612 + vcpu->arch.nmi_injected = false; 4613 4613 + kvm_clear_exception_queue(vcpu); 4614 4614 + kvm_clear_interrupt_queue(vcpu); 4756 4615 4757 4616 vmx_switch_vmcs(vcpu, &vmx->vmcs01); 4758 4617 ··· 5193 5030 5194 5031 free_nested(vcpu); 5195 5032 5196 5196 - /* Process a latched INIT during time CPU was in VMX operation */ 5197 5197 - kvm_make_request(KVM_REQ_EVENT, vcpu); 5033 5033 + if (kvm_apic_has_pending_init_or_sipi(vcpu)) 5034 5034 + kvm_make_request(KVM_REQ_EVENT, vcpu); 5198 5035 5199 5036 return nested_vmx_succeed(vcpu); 5200 5037 } ··· 5230 5067 * state. It is possible that the area will stay mapped as 5231 5068 * vmx->nested.hv_evmcs but this shouldn't be a problem. 5232 5069 */ 5233 5233 - if (likely(!vmx->nested.enlightened_vmcs_enabled || 5070 5070 + if (likely(!guest_cpuid_has_evmcs(vcpu) || 5234 5071 !nested_enlightened_vmentry(vcpu, &evmcs_gpa))) { 5235 5072 if (vmptr == vmx->nested.current_vmptr) 5236 5073 nested_release_vmcs12(vcpu); ··· 6626 6463 if (ret) 6627 6464 goto error_guest_mode; 6628 6465 6466 6466 + if (vmx->nested.mtf_pending) 6467 6467 + kvm_make_request(KVM_REQ_EVENT, vcpu); 6468 6468 + 6629 6469 return 0; 6630 6470 6631 6471 error_guest_mode: ··· 6688 6522 * bit in the high half is on if the corresponding bit in the control field 6689 6523 * may be on. See also vmx_control_verify(). 6690 6524 */ 6691 6691 - void nested_vmx_setup_ctls_msrs(struct nested_vmx_msrs *msrs, u32 ept_caps) 6525 6525 + void nested_vmx_setup_ctls_msrs(struct vmcs_config *vmcs_conf, u32 ept_caps) 6692 6526 { 6527 6527 + struct nested_vmx_msrs *msrs = &vmcs_conf->nested; 6528 6528 + 6693 6529 /* 6694 6530 * Note that as a general rule, the high half of the MSRs (bits in 6695 6531 * the control fields which may be 1) should be initialized by the ··· 6708 6540 */ 6709 6541 6710 6542 /* pin-based controls */ 6711 6711 - rdmsr(MSR_IA32_VMX_PINBASED_CTLS, 6712 6712 - msrs->pinbased_ctls_low, 6713 6713 - msrs->pinbased_ctls_high); 6714 6714 - msrs->pinbased_ctls_low |= 6543 6543 + msrs->pinbased_ctls_low = 6715 6544 PIN_BASED_ALWAYSON_WITHOUT_TRUE_MSR; 6545 6545 + 6546 6546 + msrs->pinbased_ctls_high = vmcs_conf->pin_based_exec_ctrl; 6716 6547 msrs->pinbased_ctls_high &= 6717 6548 PIN_BASED_EXT_INTR_MASK | 6718 6549 PIN_BASED_NMI_EXITING | ··· 6722 6555 PIN_BASED_VMX_PREEMPTION_TIMER; 6723 6556 6724 6557 /* exit controls */ 6725 6725 - rdmsr(MSR_IA32_VMX_EXIT_CTLS, 6726 6726 - msrs->exit_ctls_low, 6727 6727 - msrs->exit_ctls_high); 6728 6558 msrs->exit_ctls_low = 6729 6559 VM_EXIT_ALWAYSON_WITHOUT_TRUE_MSR; 6730 6560 6561 6561 + msrs->exit_ctls_high = vmcs_conf->vmexit_ctrl; 6731 6562 msrs->exit_ctls_high &= 6732 6563 #ifdef CONFIG_X86_64 6733 6564 VM_EXIT_HOST_ADDR_SPACE_SIZE | 6734 6565 #endif 6735 6566 VM_EXIT_LOAD_IA32_PAT | VM_EXIT_SAVE_IA32_PAT | 6736 6736 - VM_EXIT_CLEAR_BNDCFGS | VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL; 6567 6567 + VM_EXIT_CLEAR_BNDCFGS; 6737 6568 msrs->exit_ctls_high |= 6738 6569 VM_EXIT_ALWAYSON_WITHOUT_TRUE_MSR | 6739 6570 VM_EXIT_LOAD_IA32_EFER | VM_EXIT_SAVE_IA32_EFER | 6740 6740 - VM_EXIT_SAVE_VMX_PREEMPTION_TIMER | VM_EXIT_ACK_INTR_ON_EXIT; 6571 6571 + VM_EXIT_SAVE_VMX_PREEMPTION_TIMER | VM_EXIT_ACK_INTR_ON_EXIT | 6572 6572 + VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL; 6741 6573 6742 6574 /* We support free control of debug control saving. */ 6743 6575 msrs->exit_ctls_low &= ~VM_EXIT_SAVE_DEBUG_CONTROLS; 6744 6576 6745 6577 /* entry controls */ 6746 6746 - rdmsr(MSR_IA32_VMX_ENTRY_CTLS, 6747 6747 - msrs->entry_ctls_low, 6748 6748 - msrs->entry_ctls_high); 6749 6578 msrs->entry_ctls_low = 6750 6579 VM_ENTRY_ALWAYSON_WITHOUT_TRUE_MSR; 6580 6580 + 6581 6581 + msrs->entry_ctls_high = vmcs_conf->vmentry_ctrl; 6751 6582 msrs->entry_ctls_high &= 6752 6583 #ifdef CONFIG_X86_64 6753 6584 VM_ENTRY_IA32E_MODE | 6754 6585 #endif 6755 6755 - VM_ENTRY_LOAD_IA32_PAT | VM_ENTRY_LOAD_BNDCFGS | 6756 6756 - VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL; 6586 6586 + VM_ENTRY_LOAD_IA32_PAT | VM_ENTRY_LOAD_BNDCFGS; 6757 6587 msrs->entry_ctls_high |= 6758 6758 - (VM_ENTRY_ALWAYSON_WITHOUT_TRUE_MSR | VM_ENTRY_LOAD_IA32_EFER); 6588 6588 + (VM_ENTRY_ALWAYSON_WITHOUT_TRUE_MSR | VM_ENTRY_LOAD_IA32_EFER | 6589 6589 + VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL); 6759 6590 6760 6591 /* We support free control of debug control loading. */ 6761 6592 msrs->entry_ctls_low &= ~VM_ENTRY_LOAD_DEBUG_CONTROLS; 6762 6593 6763 6594 /* cpu-based controls */ 6764 6764 - rdmsr(MSR_IA32_VMX_PROCBASED_CTLS, 6765 6765 - msrs->procbased_ctls_low, 6766 6766 - msrs->procbased_ctls_high); 6767 6595 msrs->procbased_ctls_low = 6768 6596 CPU_BASED_ALWAYSON_WITHOUT_TRUE_MSR; 6597 6597 + 6598 6598 + msrs->procbased_ctls_high = vmcs_conf->cpu_based_exec_ctrl; 6769 6599 msrs->procbased_ctls_high &= 6770 6600 CPU_BASED_INTR_WINDOW_EXITING | 6771 6601 CPU_BASED_NMI_WINDOW_EXITING | CPU_BASED_USE_TSC_OFFSETTING | ··· 6796 6632 * depend on CPUID bits, they are added later by 6797 6633 * vmx_vcpu_after_set_cpuid. 6798 6634 */ 6799 6799 - if (msrs->procbased_ctls_high & CPU_BASED_ACTIVATE_SECONDARY_CONTROLS) 6800 6800 - rdmsr(MSR_IA32_VMX_PROCBASED_CTLS2, 6801 6801 - msrs->secondary_ctls_low, 6802 6802 - msrs->secondary_ctls_high); 6803 6803 - 6804 6635 msrs->secondary_ctls_low = 0; 6636 6636 + 6637 6637 + msrs->secondary_ctls_high = vmcs_conf->cpu_based_2nd_exec_ctrl; 6805 6638 msrs->secondary_ctls_high &= 6806 6639 SECONDARY_EXEC_DESC | 6807 6640 SECONDARY_EXEC_ENABLE_RDTSCP | ··· 6878 6717 msrs->secondary_ctls_high |= SECONDARY_EXEC_ENCLS_EXITING; 6879 6718 6880 6719 /* miscellaneous data */ 6881 6881 - rdmsr(MSR_IA32_VMX_MISC, 6882 6882 - msrs->misc_low, 6883 6883 - msrs->misc_high); 6884 6884 - msrs->misc_low &= VMX_MISC_SAVE_EFER_LMA; 6720 6720 + msrs->misc_low = (u32)vmcs_conf->misc & VMX_MISC_SAVE_EFER_LMA; 6885 6721 msrs->misc_low |= 6886 6722 MSR_IA32_VMX_MISC_VMWRITE_SHADOW_RO_FIELDS | 6887 6723 VMX_MISC_EMULATED_PREEMPTION_TIMER_RATE | ··· 6972 6814 6973 6815 struct kvm_x86_nested_ops vmx_nested_ops = { 6974 6816 .leave_nested = vmx_leave_nested, 6817 6817 + .is_exception_vmexit = nested_vmx_is_exception_vmexit, 6975 6818 .check_events = vmx_check_nested_events, 6976 6976 - .handle_page_fault_workaround = nested_vmx_handle_page_fault_workaround, 6977 6977 - .hv_timer_pending = nested_vmx_preemption_timer_pending, 6819 6819 + .has_events = vmx_has_nested_events, 6978 6820 .triple_fault = nested_vmx_triple_fault, 6979 6821 .get_state = vmx_get_nested_state, 6980 6822 .set_state = vmx_set_nested_state,

+1 -1

arch/x86/kvm/vmx/nested.h

reviewed

··· 17 17 }; 18 18 19 19 void vmx_leave_nested(struct kvm_vcpu *vcpu); 20 20 - void nested_vmx_setup_ctls_msrs(struct nested_vmx_msrs *msrs, u32 ept_caps); 20 20 + void nested_vmx_setup_ctls_msrs(struct vmcs_config *vmcs_conf, u32 ept_caps); 21 21 void nested_vmx_hardware_unsetup(void); 22 22 __init int nested_vmx_hardware_setup(int (*exit_handlers[])(struct kvm_vcpu *)); 23 23 void nested_vmx_set_vmcs_shadowing_bitmap(void);

+1 -1

arch/x86/kvm/vmx/sgx.c

reviewed

··· 129 129 ex.address = gva; 130 130 ex.error_code_valid = true; 131 131 ex.nested_page_fault = false; 132 132 - kvm_inject_page_fault(vcpu, &ex); 132 132 + kvm_inject_emulated_page_fault(vcpu, &ex); 133 133 } else { 134 134 kvm_inject_gp(vcpu, 0); 135 135 }

+7 -17

arch/x86/kvm/vmx/vmenter.S

reviewed

··· 189 189 xor %ebx, %ebx 190 190 191 191 .Lclear_regs: 192 192 + /* Discard @regs. The register is irrelevant, it just can't be RBX. */ 193 193 + pop %_ASM_AX 194 194 + 192 195 /* 193 196 * Clear all general purpose registers except RSP and RBX to prevent 194 197 * speculative use of the guest's values, even those that are reloaded 195 198 * via the stack. In theory, an L1 cache miss when restoring registers 196 199 * could lead to speculative execution with the guest's values. 197 200 * Zeroing XORs are dirt cheap, i.e. the extra paranoia is essentially 198 198 - * free. RSP and RAX are exempt as RSP is restored by hardware during 201 201 + * free. RSP and RBX are exempt as RSP is restored by hardware during 199 202 * VM-Exit and RBX is explicitly loaded with 0 or 1 to hold the return 200 203 * value. 201 204 */ ··· 219 216 xor %r15d, %r15d 220 217 #endif 221 218 222 222 - /* "POP" @regs. */ 223 223 - add $WORD_SIZE, %_ASM_SP 224 224 - 225 219 /* 226 220 * IMPORTANT: RSB filling and SPEC_CTRL handling must be done before 227 221 * the first unbalanced RET after vmexit! ··· 233 233 234 234 FILL_RETURN_BUFFER %_ASM_CX, RSB_CLEAR_LOOPS, X86_FEATURE_RSB_VMEXIT,\ 235 235 X86_FEATURE_RSB_VMEXIT_LITE 236 236 - 237 236 238 237 pop %_ASM_ARG2 /* @flags */ 239 238 pop %_ASM_ARG1 /* @vmx */ ··· 292 293 push %r10 293 294 push %r11 294 295 #endif 295 295 - #ifdef CONFIG_X86_64 296 296 + 296 297 /* Load @field and @fault to arg1 and arg2 respectively. */ 297 297 - mov 3*WORD_SIZE(%rbp), %_ASM_ARG2 298 298 - mov 2*WORD_SIZE(%rbp), %_ASM_ARG1 299 299 - #else 300 300 - /* Parameters are passed on the stack for 32-bit (see asmlinkage). */ 301 301 - push 3*WORD_SIZE(%ebp) 302 302 - push 2*WORD_SIZE(%ebp) 303 303 - #endif 298 298 + mov 3*WORD_SIZE(%_ASM_BP), %_ASM_ARG2 299 299 + mov 2*WORD_SIZE(%_ASM_BP), %_ASM_ARG1 304 300 305 301 call vmread_error 306 306 - 307 307 - #ifndef CONFIG_X86_64 308 308 - add $8, %esp 309 309 - #endif 310 302 311 303 /* Zero out @fault, which will be popped into the result register. */ 312 304 _ASM_MOV $0, 3*WORD_SIZE(%_ASM_BP)

+156 -163

arch/x86/kvm/vmx/vmx.c

reviewed

··· 439 439 pr_warn_ratelimited(fmt); \ 440 440 } while (0) 441 441 442 442 - asmlinkage void vmread_error(unsigned long field, bool fault) 442 442 + void vmread_error(unsigned long field, bool fault) 443 443 { 444 444 if (fault) 445 445 kvm_spurious_fault(); ··· 864 864 return flags; 865 865 } 866 866 867 867 - static void clear_atomic_switch_msr_special(struct vcpu_vmx *vmx, 867 867 + static __always_inline void clear_atomic_switch_msr_special(struct vcpu_vmx *vmx, 868 868 unsigned long entry, unsigned long exit) 869 869 { 870 870 vm_entry_controls_clearbit(vmx, entry); ··· 922 922 vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, m->host.nr); 923 923 } 924 924 925 925 - static void add_atomic_switch_msr_special(struct vcpu_vmx *vmx, 925 925 + static __always_inline void add_atomic_switch_msr_special(struct vcpu_vmx *vmx, 926 926 unsigned long entry, unsigned long exit, 927 927 unsigned long guest_val_vmcs, unsigned long host_val_vmcs, 928 928 u64 guest_val, u64 host_val) ··· 1652 1652 1653 1653 /* 1654 1654 * Per the SDM, MTF takes priority over debug-trap exceptions besides 1655 1655 - * T-bit traps. As instruction emulation is completed (i.e. at the 1656 1656 - * instruction boundary), any #DB exception pending delivery must be a 1657 1657 - * debug-trap. Record the pending MTF state to be delivered in 1655 1655 + * TSS T-bit traps and ICEBP (INT1). KVM doesn't emulate T-bit traps 1656 1656 + * or ICEBP (in the emulator proper), and skipping of ICEBP after an 1657 1657 + * intercepted #DB deliberately avoids single-step #DB and MTF updates 1658 1658 + * as ICEBP is higher priority than both. As instruction emulation is 1659 1659 + * completed at this point (i.e. KVM is at the instruction boundary), 1660 1660 + * any #DB exception pending delivery must be a debug-trap of lower 1661 1661 + * priority than MTF. Record the pending MTF state to be delivered in 1658 1662 * vmx_check_nested_events(). 1659 1663 */ 1660 1664 if (nested_cpu_has_mtf(vmcs12) && 1661 1665 (!vcpu->arch.exception.pending || 1662 1662 - vcpu->arch.exception.nr == DB_VECTOR)) 1666 1666 + vcpu->arch.exception.vector == DB_VECTOR) && 1667 1667 + (!vcpu->arch.exception_vmexit.pending || 1668 1668 + vcpu->arch.exception_vmexit.vector == DB_VECTOR)) { 1663 1669 vmx->nested.mtf_pending = true; 1664 1664 - else 1670 1670 + kvm_make_request(KVM_REQ_EVENT, vcpu); 1671 1671 + } else { 1665 1672 vmx->nested.mtf_pending = false; 1673 1673 + } 1666 1674 } 1667 1675 1668 1676 static int vmx_skip_emulated_instruction(struct kvm_vcpu *vcpu) ··· 1692 1684 vmcs_write32(GUEST_ACTIVITY_STATE, GUEST_ACTIVITY_ACTIVE); 1693 1685 } 1694 1686 1695 1695 - static void vmx_queue_exception(struct kvm_vcpu *vcpu) 1687 1687 + static void vmx_inject_exception(struct kvm_vcpu *vcpu) 1696 1688 { 1689 1689 + struct kvm_queued_exception *ex = &vcpu->arch.exception; 1690 1690 + u32 intr_info = ex->vector | INTR_INFO_VALID_MASK; 1697 1691 struct vcpu_vmx *vmx = to_vmx(vcpu); 1698 1698 - unsigned nr = vcpu->arch.exception.nr; 1699 1699 - bool has_error_code = vcpu->arch.exception.has_error_code; 1700 1700 - u32 error_code = vcpu->arch.exception.error_code; 1701 1701 - u32 intr_info = nr | INTR_INFO_VALID_MASK; 1702 1692 1703 1703 - kvm_deliver_exception_payload(vcpu); 1693 1693 + kvm_deliver_exception_payload(vcpu, ex); 1704 1694 1705 1705 - if (has_error_code) { 1706 1706 - vmcs_write32(VM_ENTRY_EXCEPTION_ERROR_CODE, error_code); 1695 1695 + if (ex->has_error_code) { 1696 1696 + /* 1697 1697 + * Despite the error code being architecturally defined as 32 1698 1698 + * bits, and the VMCS field being 32 bits, Intel CPUs and thus 1699 1699 + * VMX don't actually supporting setting bits 31:16. Hardware 1700 1700 + * will (should) never provide a bogus error code, but AMD CPUs 1701 1701 + * do generate error codes with bits 31:16 set, and so KVM's 1702 1702 + * ABI lets userspace shove in arbitrary 32-bit values. Drop 1703 1703 + * the upper bits to avoid VM-Fail, losing information that 1704 1704 + * does't really exist is preferable to killing the VM. 1705 1705 + */ 1706 1706 + vmcs_write32(VM_ENTRY_EXCEPTION_ERROR_CODE, (u16)ex->error_code); 1707 1707 intr_info |= INTR_INFO_DELIVER_CODE_MASK; 1708 1708 } 1709 1709 1710 1710 if (vmx->rmode.vm86_active) { 1711 1711 int inc_eip = 0; 1712 1712 - if (kvm_exception_is_soft(nr)) 1712 1712 + if (kvm_exception_is_soft(ex->vector)) 1713 1713 inc_eip = vcpu->arch.event_exit_inst_len; 1714 1714 - kvm_inject_realmode_interrupt(vcpu, nr, inc_eip); 1714 1714 + kvm_inject_realmode_interrupt(vcpu, ex->vector, inc_eip); 1715 1715 return; 1716 1716 } 1717 1717 1718 1718 WARN_ON_ONCE(vmx->emulation_required); 1719 1719 1720 1720 - if (kvm_exception_is_soft(nr)) { 1720 1720 + if (kvm_exception_is_soft(ex->vector)) { 1721 1721 vmcs_write32(VM_ENTRY_INSTRUCTION_LEN, 1722 1722 vmx->vcpu.arch.event_exit_inst_len); 1723 1723 intr_info |= INTR_TYPE_SOFT_EXCEPTION; ··· 1946 1930 * sanity checking and refuse to boot. Filter all unsupported 1947 1931 * features out. 1948 1932 */ 1949 1949 - if (!msr_info->host_initiated && 1950 1950 - vmx->nested.enlightened_vmcs_enabled) 1951 1951 - nested_evmcs_filter_control_msr(msr_info->index, 1933 1933 + if (!msr_info->host_initiated && guest_cpuid_has_evmcs(vcpu)) 1934 1934 + nested_evmcs_filter_control_msr(vcpu, msr_info->index, 1952 1935 &msr_info->data); 1953 1936 break; 1954 1937 case MSR_IA32_RTIT_CTL: ··· 2509 2494 return cpuid_eax(0) >= 0x12 && (cpuid_eax(0x12) & BIT(0)); 2510 2495 } 2511 2496 2497 2497 + /* 2498 2498 + * Some cpus support VM_{ENTRY,EXIT}_IA32_PERF_GLOBAL_CTRL but they 2499 2499 + * can't be used due to errata where VM Exit may incorrectly clear 2500 2500 + * IA32_PERF_GLOBAL_CTRL[34:32]. Work around the errata by using the 2501 2501 + * MSR load mechanism to switch IA32_PERF_GLOBAL_CTRL. 2502 2502 + */ 2503 2503 + static bool cpu_has_perf_global_ctrl_bug(void) 2504 2504 + { 2505 2505 + if (boot_cpu_data.x86 == 0x6) { 2506 2506 + switch (boot_cpu_data.x86_model) { 2507 2507 + case INTEL_FAM6_NEHALEM_EP: /* AAK155 */ 2508 2508 + case INTEL_FAM6_NEHALEM: /* AAP115 */ 2509 2509 + case INTEL_FAM6_WESTMERE: /* AAT100 */ 2510 2510 + case INTEL_FAM6_WESTMERE_EP: /* BC86,AAY89,BD102 */ 2511 2511 + case INTEL_FAM6_NEHALEM_EX: /* BA97 */ 2512 2512 + return true; 2513 2513 + default: 2514 2514 + break; 2515 2515 + } 2516 2516 + } 2517 2517 + 2518 2518 + return false; 2519 2519 + } 2520 2520 + 2512 2521 static __init int adjust_vmx_controls(u32 ctl_min, u32 ctl_opt, 2513 2522 u32 msr, u32 *result) 2514 2523 { ··· 2565 2526 struct vmx_capability *vmx_cap) 2566 2527 { 2567 2528 u32 vmx_msr_low, vmx_msr_high; 2568 2568 - u32 min, opt, min2, opt2; 2569 2529 u32 _pin_based_exec_control = 0; 2570 2530 u32 _cpu_based_exec_control = 0; 2571 2531 u32 _cpu_based_2nd_exec_control = 0; 2572 2532 u64 _cpu_based_3rd_exec_control = 0; 2573 2533 u32 _vmexit_control = 0; 2574 2534 u32 _vmentry_control = 0; 2535 2535 + u64 misc_msr; 2575 2536 int i; 2576 2537 2577 2538 /* ··· 2591 2552 }; 2592 2553 2593 2554 memset(vmcs_conf, 0, sizeof(*vmcs_conf)); 2594 2594 - min = CPU_BASED_HLT_EXITING | 2595 2595 - #ifdef CONFIG_X86_64 2596 2596 - CPU_BASED_CR8_LOAD_EXITING | 2597 2597 - CPU_BASED_CR8_STORE_EXITING | 2598 2598 - #endif 2599 2599 - CPU_BASED_CR3_LOAD_EXITING | 2600 2600 - CPU_BASED_CR3_STORE_EXITING | 2601 2601 - CPU_BASED_UNCOND_IO_EXITING | 2602 2602 - CPU_BASED_MOV_DR_EXITING | 2603 2603 - CPU_BASED_USE_TSC_OFFSETTING | 2604 2604 - CPU_BASED_MWAIT_EXITING | 2605 2605 - CPU_BASED_MONITOR_EXITING | 2606 2606 - CPU_BASED_INVLPG_EXITING | 2607 2607 - CPU_BASED_RDPMC_EXITING; 2608 2555 2609 2609 - opt = CPU_BASED_TPR_SHADOW | 2610 2610 - CPU_BASED_USE_MSR_BITMAPS | 2611 2611 - CPU_BASED_ACTIVATE_SECONDARY_CONTROLS | 2612 2612 - CPU_BASED_ACTIVATE_TERTIARY_CONTROLS; 2613 2613 - if (adjust_vmx_controls(min, opt, MSR_IA32_VMX_PROCBASED_CTLS, 2614 2614 - &_cpu_based_exec_control) < 0) 2556 2556 + if (adjust_vmx_controls(KVM_REQUIRED_VMX_CPU_BASED_VM_EXEC_CONTROL, 2557 2557 + KVM_OPTIONAL_VMX_CPU_BASED_VM_EXEC_CONTROL, 2558 2558 + MSR_IA32_VMX_PROCBASED_CTLS, 2559 2559 + &_cpu_based_exec_control)) 2615 2560 return -EIO; 2616 2616 - #ifdef CONFIG_X86_64 2617 2617 - if (_cpu_based_exec_control & CPU_BASED_TPR_SHADOW) 2618 2618 - _cpu_based_exec_control &= ~CPU_BASED_CR8_LOAD_EXITING & 2619 2619 - ~CPU_BASED_CR8_STORE_EXITING; 2620 2620 - #endif 2621 2561 if (_cpu_based_exec_control & CPU_BASED_ACTIVATE_SECONDARY_CONTROLS) { 2622 2622 - min2 = 0; 2623 2623 - opt2 = SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES | 2624 2624 - SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE | 2625 2625 - SECONDARY_EXEC_WBINVD_EXITING | 2626 2626 - SECONDARY_EXEC_ENABLE_VPID | 2627 2627 - SECONDARY_EXEC_ENABLE_EPT | 2628 2628 - SECONDARY_EXEC_UNRESTRICTED_GUEST | 2629 2629 - SECONDARY_EXEC_PAUSE_LOOP_EXITING | 2630 2630 - SECONDARY_EXEC_DESC | 2631 2631 - SECONDARY_EXEC_ENABLE_RDTSCP | 2632 2632 - SECONDARY_EXEC_ENABLE_INVPCID | 2633 2633 - SECONDARY_EXEC_APIC_REGISTER_VIRT | 2634 2634 - SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY | 2635 2635 - SECONDARY_EXEC_SHADOW_VMCS | 2636 2636 - SECONDARY_EXEC_XSAVES | 2637 2637 - SECONDARY_EXEC_RDSEED_EXITING | 2638 2638 - SECONDARY_EXEC_RDRAND_EXITING | 2639 2639 - SECONDARY_EXEC_ENABLE_PML | 2640 2640 - SECONDARY_EXEC_TSC_SCALING | 2641 2641 - SECONDARY_EXEC_ENABLE_USR_WAIT_PAUSE | 2642 2642 - SECONDARY_EXEC_PT_USE_GPA | 2643 2643 - SECONDARY_EXEC_PT_CONCEAL_VMX | 2644 2644 - SECONDARY_EXEC_ENABLE_VMFUNC | 2645 2645 - SECONDARY_EXEC_BUS_LOCK_DETECTION | 2646 2646 - SECONDARY_EXEC_NOTIFY_VM_EXITING; 2647 2647 - if (cpu_has_sgx()) 2648 2648 - opt2 |= SECONDARY_EXEC_ENCLS_EXITING; 2649 2649 - if (adjust_vmx_controls(min2, opt2, 2562 2562 + if (adjust_vmx_controls(KVM_REQUIRED_VMX_SECONDARY_VM_EXEC_CONTROL, 2563 2563 + KVM_OPTIONAL_VMX_SECONDARY_VM_EXEC_CONTROL, 2650 2564 MSR_IA32_VMX_PROCBASED_CTLS2, 2651 2651 - &_cpu_based_2nd_exec_control) < 0) 2565 2565 + &_cpu_based_2nd_exec_control)) 2652 2566 return -EIO; 2653 2567 } 2654 2568 #ifndef CONFIG_X86_64 ··· 2619 2627 rdmsr_safe(MSR_IA32_VMX_EPT_VPID_CAP, 2620 2628 &vmx_cap->ept, &vmx_cap->vpid); 2621 2629 2622 2622 - if (_cpu_based_2nd_exec_control & SECONDARY_EXEC_ENABLE_EPT) { 2623 2623 - /* CR3 accesses and invlpg don't need to cause VM Exits when EPT 2624 2624 - enabled */ 2625 2625 - _cpu_based_exec_control &= ~(CPU_BASED_CR3_LOAD_EXITING | 2626 2626 - CPU_BASED_CR3_STORE_EXITING | 2627 2627 - CPU_BASED_INVLPG_EXITING); 2628 2628 - } else if (vmx_cap->ept) { 2630 2630 + if (!(_cpu_based_2nd_exec_control & SECONDARY_EXEC_ENABLE_EPT) && 2631 2631 + vmx_cap->ept) { 2629 2632 pr_warn_once("EPT CAP should not exist if not support " 2630 2633 "1-setting enable EPT VM-execution control\n"); 2631 2634 ··· 2640 2653 vmx_cap->vpid = 0; 2641 2654 } 2642 2655 2643 2643 - if (_cpu_based_exec_control & CPU_BASED_ACTIVATE_TERTIARY_CONTROLS) { 2644 2644 - u64 opt3 = TERTIARY_EXEC_IPI_VIRT; 2656 2656 + if (!cpu_has_sgx()) 2657 2657 + _cpu_based_2nd_exec_control &= ~SECONDARY_EXEC_ENCLS_EXITING; 2645 2658 2646 2646 - _cpu_based_3rd_exec_control = adjust_vmx_controls64(opt3, 2659 2659 + if (_cpu_based_exec_control & CPU_BASED_ACTIVATE_TERTIARY_CONTROLS) 2660 2660 + _cpu_based_3rd_exec_control = 2661 2661 + adjust_vmx_controls64(KVM_OPTIONAL_VMX_TERTIARY_VM_EXEC_CONTROL, 2647 2662 MSR_IA32_VMX_PROCBASED_CTLS3); 2648 2648 - } 2649 2663 2650 2650 - min = VM_EXIT_SAVE_DEBUG_CONTROLS | VM_EXIT_ACK_INTR_ON_EXIT; 2651 2651 - #ifdef CONFIG_X86_64 2652 2652 - min |= VM_EXIT_HOST_ADDR_SPACE_SIZE; 2653 2653 - #endif 2654 2654 - opt = VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL | 2655 2655 - VM_EXIT_LOAD_IA32_PAT | 2656 2656 - VM_EXIT_LOAD_IA32_EFER | 2657 2657 - VM_EXIT_CLEAR_BNDCFGS | 2658 2658 - VM_EXIT_PT_CONCEAL_PIP | 2659 2659 - VM_EXIT_CLEAR_IA32_RTIT_CTL; 2660 2660 - if (adjust_vmx_controls(min, opt, MSR_IA32_VMX_EXIT_CTLS, 2661 2661 - &_vmexit_control) < 0) 2664 2664 + if (adjust_vmx_controls(KVM_REQUIRED_VMX_VM_EXIT_CONTROLS, 2665 2665 + KVM_OPTIONAL_VMX_VM_EXIT_CONTROLS, 2666 2666 + MSR_IA32_VMX_EXIT_CTLS, 2667 2667 + &_vmexit_control)) 2662 2668 return -EIO; 2663 2669 2664 2664 - min = PIN_BASED_EXT_INTR_MASK | PIN_BASED_NMI_EXITING; 2665 2665 - opt = PIN_BASED_VIRTUAL_NMIS | PIN_BASED_POSTED_INTR | 2666 2666 - PIN_BASED_VMX_PREEMPTION_TIMER; 2667 2667 - if (adjust_vmx_controls(min, opt, MSR_IA32_VMX_PINBASED_CTLS, 2668 2668 - &_pin_based_exec_control) < 0) 2670 2670 + if (adjust_vmx_controls(KVM_REQUIRED_VMX_PIN_BASED_VM_EXEC_CONTROL, 2671 2671 + KVM_OPTIONAL_VMX_PIN_BASED_VM_EXEC_CONTROL, 2672 2672 + MSR_IA32_VMX_PINBASED_CTLS, 2673 2673 + &_pin_based_exec_control)) 2669 2674 return -EIO; 2670 2675 2671 2676 if (cpu_has_broken_vmx_preemption_timer()) ··· 2666 2687 SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY)) 2667 2688 _pin_based_exec_control &= ~PIN_BASED_POSTED_INTR; 2668 2689 2669 2669 - min = VM_ENTRY_LOAD_DEBUG_CONTROLS; 2670 2670 - opt = VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL | 2671 2671 - VM_ENTRY_LOAD_IA32_PAT | 2672 2672 - VM_ENTRY_LOAD_IA32_EFER | 2673 2673 - VM_ENTRY_LOAD_BNDCFGS | 2674 2674 - VM_ENTRY_PT_CONCEAL_PIP | 2675 2675 - VM_ENTRY_LOAD_IA32_RTIT_CTL; 2676 2676 - if (adjust_vmx_controls(min, opt, MSR_IA32_VMX_ENTRY_CTLS, 2677 2677 - &_vmentry_control) < 0) 2690 2690 + if (adjust_vmx_controls(KVM_REQUIRED_VMX_VM_ENTRY_CONTROLS, 2691 2691 + KVM_OPTIONAL_VMX_VM_ENTRY_CONTROLS, 2692 2692 + MSR_IA32_VMX_ENTRY_CTLS, 2693 2693 + &_vmentry_control)) 2678 2694 return -EIO; 2679 2695 2680 2696 for (i = 0; i < ARRAY_SIZE(vmcs_entry_exit_pairs); i++) { ··· 2689 2715 _vmexit_control &= ~x_ctrl; 2690 2716 } 2691 2717 2692 2692 - /* 2693 2693 - * Some cpus support VM_{ENTRY,EXIT}_IA32_PERF_GLOBAL_CTRL but they 2694 2694 - * can't be used due to an errata where VM Exit may incorrectly clear 2695 2695 - * IA32_PERF_GLOBAL_CTRL[34:32]. Workaround the errata by using the 2696 2696 - * MSR load mechanism to switch IA32_PERF_GLOBAL_CTRL. 2697 2697 - */ 2698 2698 - if (boot_cpu_data.x86 == 0x6) { 2699 2699 - switch (boot_cpu_data.x86_model) { 2700 2700 - case 26: /* AAK155 */ 2701 2701 - case 30: /* AAP115 */ 2702 2702 - case 37: /* AAT100 */ 2703 2703 - case 44: /* BC86,AAY89,BD102 */ 2704 2704 - case 46: /* BA97 */ 2705 2705 - _vmentry_control &= ~VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL; 2706 2706 - _vmexit_control &= ~VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL; 2707 2707 - pr_warn_once("kvm: VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL " 2708 2708 - "does not work properly. Using workaround\n"); 2709 2709 - break; 2710 2710 - default: 2711 2711 - break; 2712 2712 - } 2713 2713 - } 2714 2714 - 2715 2715 - 2716 2718 rdmsr(MSR_IA32_VMX_BASIC, vmx_msr_low, vmx_msr_high); 2717 2719 2718 2720 /* IA-32 SDM Vol 3B: VMCS size is never greater than 4kB. */ ··· 2705 2755 if (((vmx_msr_high >> 18) & 15) != 6) 2706 2756 return -EIO; 2707 2757 2758 2758 + rdmsrl(MSR_IA32_VMX_MISC, misc_msr); 2759 2759 + 2708 2760 vmcs_conf->size = vmx_msr_high & 0x1fff; 2709 2761 vmcs_conf->basic_cap = vmx_msr_high & ~0x1fff; 2710 2762 ··· 2718 2766 vmcs_conf->cpu_based_3rd_exec_ctrl = _cpu_based_3rd_exec_control; 2719 2767 vmcs_conf->vmexit_ctrl = _vmexit_control; 2720 2768 vmcs_conf->vmentry_ctrl = _vmentry_control; 2721 2721 - 2722 2722 - #if IS_ENABLED(CONFIG_HYPERV) 2723 2723 - if (enlightened_vmcs) 2724 2724 - evmcs_sanitize_exec_ctrls(vmcs_conf); 2725 2725 - #endif 2769 2769 + vmcs_conf->misc = misc_msr; 2726 2770 2727 2771 return 0; 2728 2772 } ··· 2985 3037 return 0; 2986 3038 2987 3039 vcpu->arch.efer = efer; 3040 3040 + #ifdef CONFIG_X86_64 2988 3041 if (efer & EFER_LMA) 2989 3042 vm_entry_controls_setbit(vmx, VM_ENTRY_IA32E_MODE); 2990 3043 else 2991 3044 vm_entry_controls_clearbit(vmx, VM_ENTRY_IA32E_MODE); 3045 3045 + #else 3046 3046 + if (KVM_BUG_ON(efer & EFER_LMA, vcpu->kvm)) 3047 3047 + return 1; 3048 3048 + #endif 2992 3049 2993 3050 vmx_setup_uret_msrs(vmx); 2994 3051 return 0; ··· 4280 4327 if (vmx_pt_mode_is_system()) 4281 4328 vmentry_ctrl &= ~(VM_ENTRY_PT_CONCEAL_PIP | 4282 4329 VM_ENTRY_LOAD_IA32_RTIT_CTL); 4283 4283 - /* Loading of EFER and PERF_GLOBAL_CTRL are toggled dynamically */ 4284 4284 - return vmentry_ctrl & 4285 4285 - ~(VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL | VM_ENTRY_LOAD_IA32_EFER); 4330 4330 + /* 4331 4331 + * IA32e mode, and loading of EFER and PERF_GLOBAL_CTRL are toggled dynamically. 4332 4332 + */ 4333 4333 + vmentry_ctrl &= ~(VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL | 4334 4334 + VM_ENTRY_LOAD_IA32_EFER | 4335 4335 + VM_ENTRY_IA32E_MODE); 4336 4336 + 4337 4337 + if (cpu_has_perf_global_ctrl_bug()) 4338 4338 + vmentry_ctrl &= ~VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL; 4339 4339 + 4340 4340 + return vmentry_ctrl; 4286 4341 } 4287 4342 4288 4343 static u32 vmx_vmexit_ctrl(void) 4289 4344 { 4290 4345 u32 vmexit_ctrl = vmcs_config.vmexit_ctrl; 4291 4346 4347 4347 + /* 4348 4348 + * Not used by KVM and never set in vmcs01 or vmcs02, but emulated for 4349 4349 + * nested virtualization and thus allowed to be set in vmcs12. 4350 4350 + */ 4351 4351 + vmexit_ctrl &= ~(VM_EXIT_SAVE_IA32_PAT | VM_EXIT_SAVE_IA32_EFER | 4352 4352 + VM_EXIT_SAVE_VMX_PREEMPTION_TIMER); 4353 4353 + 4292 4354 if (vmx_pt_mode_is_system()) 4293 4355 vmexit_ctrl &= ~(VM_EXIT_PT_CONCEAL_PIP | 4294 4356 VM_EXIT_CLEAR_IA32_RTIT_CTL); 4357 4357 + 4358 4358 + if (cpu_has_perf_global_ctrl_bug()) 4359 4359 + vmexit_ctrl &= ~VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL; 4360 4360 + 4295 4361 /* Loading of EFER and PERF_GLOBAL_CTRL are toggled dynamically */ 4296 4362 return vmexit_ctrl & 4297 4363 ~(VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL | VM_EXIT_LOAD_IA32_EFER); ··· 4348 4376 { 4349 4377 u32 exec_control = vmcs_config.cpu_based_exec_ctrl; 4350 4378 4379 4379 + /* 4380 4380 + * Not used by KVM, but fully supported for nesting, i.e. are allowed in 4381 4381 + * vmcs12 and propagated to vmcs02 when set in vmcs12. 4382 4382 + */ 4383 4383 + exec_control &= ~(CPU_BASED_RDTSC_EXITING | 4384 4384 + CPU_BASED_USE_IO_BITMAPS | 4385 4385 + CPU_BASED_MONITOR_TRAP_FLAG | 4386 4386 + CPU_BASED_PAUSE_EXITING); 4387 4387 + 4388 4388 + /* INTR_WINDOW_EXITING and NMI_WINDOW_EXITING are toggled dynamically */ 4389 4389 + exec_control &= ~(CPU_BASED_INTR_WINDOW_EXITING | 4390 4390 + CPU_BASED_NMI_WINDOW_EXITING); 4391 4391 + 4351 4392 if (vmx->vcpu.arch.switch_db_regs & KVM_DEBUGREG_WONT_EXIT) 4352 4393 exec_control &= ~CPU_BASED_MOV_DR_EXITING; 4353 4394 4354 4354 - if (!cpu_need_tpr_shadow(&vmx->vcpu)) { 4395 4395 + if (!cpu_need_tpr_shadow(&vmx->vcpu)) 4355 4396 exec_control &= ~CPU_BASED_TPR_SHADOW; 4397 4397 + 4356 4398 #ifdef CONFIG_X86_64 4399 4399 + if (exec_control & CPU_BASED_TPR_SHADOW) 4400 4400 + exec_control &= ~(CPU_BASED_CR8_LOAD_EXITING | 4401 4401 + CPU_BASED_CR8_STORE_EXITING); 4402 4402 + else 4357 4403 exec_control |= CPU_BASED_CR8_STORE_EXITING | 4358 4404 CPU_BASED_CR8_LOAD_EXITING; 4359 4405 #endif 4360 4360 - } 4361 4361 - if (!enable_ept) 4362 4362 - exec_control |= CPU_BASED_CR3_STORE_EXITING | 4363 4363 - CPU_BASED_CR3_LOAD_EXITING | 4364 4364 - CPU_BASED_INVLPG_EXITING; 4406 4406 + /* No need to intercept CR3 access or INVPLG when using EPT. */ 4407 4407 + if (enable_ept) 4408 4408 + exec_control &= ~(CPU_BASED_CR3_LOAD_EXITING | 4409 4409 + CPU_BASED_CR3_STORE_EXITING | 4410 4410 + CPU_BASED_INVLPG_EXITING); 4365 4411 if (kvm_mwait_in_guest(vmx->vcpu.kvm)) 4366 4412 exec_control &= ~(CPU_BASED_MWAIT_EXITING | 4367 4413 CPU_BASED_MONITOR_EXITING); ··· 5145 5155 * instruction. ICEBP generates a trap-like #DB, but 5146 5156 * despite its interception control being tied to #DB, 5147 5157 * is an instruction intercept, i.e. the VM-Exit occurs 5148 5148 - * on the ICEBP itself. Note, skipping ICEBP also 5149 5149 - * clears STI and MOVSS blocking. 5158 5158 + * on the ICEBP itself. Use the inner "skip" helper to 5159 5159 + * avoid single-step #DB and MTF updates, as ICEBP is 5160 5160 + * higher priority. Note, skipping ICEBP still clears 5161 5161 + * STI and MOVSS blocking. 5150 5162 * 5151 5163 * For all other #DBs, set vmcs.PENDING_DBG_EXCEPTIONS.BS 5152 5164 * if single-step is enabled in RFLAGS and STI or MOVSS ··· 5630 5638 vmcs_set_bits(GUEST_INTERRUPTIBILITY_INFO, GUEST_INTR_STATE_NMI); 5631 5639 5632 5640 gpa = vmcs_read64(GUEST_PHYSICAL_ADDRESS); 5633 5633 - trace_kvm_page_fault(gpa, exit_qualification); 5641 5641 + trace_kvm_page_fault(vcpu, gpa, exit_qualification); 5634 5642 5635 5643 /* Is it a read fault? */ 5636 5644 error_code = (exit_qualification & EPT_VIOLATION_ACC_READ) ··· 5702 5710 struct vcpu_vmx *vmx = to_vmx(vcpu); 5703 5711 5704 5712 return vmx->emulation_required && !vmx->rmode.vm86_active && 5705 5705 - (vcpu->arch.exception.pending || vcpu->arch.exception.injected); 5713 5713 + (kvm_is_exception_pending(vcpu) || vcpu->arch.exception.injected); 5706 5714 } 5707 5715 5708 5716 static int handle_invalid_guest_state(struct kvm_vcpu *vcpu) ··· 7422 7430 if (setup_vmcs_config(&vmcs_conf, &vmx_cap) < 0) 7423 7431 return -EIO; 7424 7432 if (nested) 7425 7425 - nested_vmx_setup_ctls_msrs(&vmcs_conf.nested, vmx_cap.ept); 7433 7433 + nested_vmx_setup_ctls_msrs(&vmcs_conf, vmx_cap.ept); 7426 7434 if (memcmp(&vmcs_config, &vmcs_conf, sizeof(struct vmcs_config)) != 0) { 7427 7435 printk(KERN_ERR "kvm: CPU %d feature inconsistency!\n", 7428 7436 smp_processor_id()); ··· 8062 8070 .patch_hypercall = vmx_patch_hypercall, 8063 8071 .inject_irq = vmx_inject_irq, 8064 8072 .inject_nmi = vmx_inject_nmi, 8065 8065 - .queue_exception = vmx_queue_exception, 8073 8073 + .inject_exception = vmx_inject_exception, 8066 8074 .cancel_injection = vmx_cancel_injection, 8067 8075 .interrupt_allowed = vmx_interrupt_allowed, 8068 8076 .nmi_allowed = vmx_nmi_allowed, ··· 8219 8227 if (setup_vmcs_config(&vmcs_config, &vmx_capability) < 0) 8220 8228 return -EIO; 8221 8229 8230 8230 + if (cpu_has_perf_global_ctrl_bug()) 8231 8231 + pr_warn_once("kvm: VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL " 8232 8232 + "does not work properly. Using workaround\n"); 8233 8233 + 8222 8234 if (boot_cpu_has(X86_FEATURE_NX)) 8223 8235 kvm_enable_efer_bits(EFER_NX); 8224 8236 ··· 8337 8341 8338 8342 if (enable_preemption_timer) { 8339 8343 u64 use_timer_freq = 5000ULL * 1000 * 1000; 8340 8340 - u64 vmx_msr; 8341 8344 8342 8342 - rdmsrl(MSR_IA32_VMX_MISC, vmx_msr); 8343 8345 cpu_preemption_timer_multi = 8344 8344 - vmx_msr & VMX_MISC_PREEMPTION_TIMER_RATE_MASK; 8346 8346 + vmcs_config.misc & VMX_MISC_PREEMPTION_TIMER_RATE_MASK; 8345 8347 8346 8348 if (tsc_khz) 8347 8349 use_timer_freq = (u64)tsc_khz * 1000; ··· 8375 8381 setup_default_sgx_lepubkeyhash(); 8376 8382 8377 8383 if (nested) { 8378 8378 - nested_vmx_setup_ctls_msrs(&vmcs_config.nested, 8379 8379 - vmx_capability.ept); 8384 8384 + nested_vmx_setup_ctls_msrs(&vmcs_config, vmx_capability.ept); 8380 8385 8381 8386 r = nested_vmx_hardware_setup(kvm_vmx_exit_handlers); 8382 8387 if (r)

+149 -23

arch/x86/kvm/vmx/vmx.h

reviewed

··· 477 477 return vmcs_read16(GUEST_INTR_STATUS) & 0xff; 478 478 } 479 479 480 480 - #define BUILD_CONTROLS_SHADOW(lname, uname, bits) \ 481 481 - static inline void lname##_controls_set(struct vcpu_vmx *vmx, u##bits val) \ 482 482 - { \ 483 483 - if (vmx->loaded_vmcs->controls_shadow.lname != val) { \ 484 484 - vmcs_write##bits(uname, val); \ 485 485 - vmx->loaded_vmcs->controls_shadow.lname = val; \ 486 486 - } \ 487 487 - } \ 488 488 - static inline u##bits __##lname##_controls_get(struct loaded_vmcs *vmcs) \ 489 489 - { \ 490 490 - return vmcs->controls_shadow.lname; \ 491 491 - } \ 492 492 - static inline u##bits lname##_controls_get(struct vcpu_vmx *vmx) \ 493 493 - { \ 494 494 - return __##lname##_controls_get(vmx->loaded_vmcs); \ 495 495 - } \ 496 496 - static inline void lname##_controls_setbit(struct vcpu_vmx *vmx, u##bits val) \ 497 497 - { \ 498 498 - lname##_controls_set(vmx, lname##_controls_get(vmx) | val); \ 499 499 - } \ 500 500 - static inline void lname##_controls_clearbit(struct vcpu_vmx *vmx, u##bits val) \ 501 501 - { \ 502 502 - lname##_controls_set(vmx, lname##_controls_get(vmx) & ~val); \ 480 480 + #define __KVM_REQUIRED_VMX_VM_ENTRY_CONTROLS \ 481 481 + (VM_ENTRY_LOAD_DEBUG_CONTROLS) 482 482 + #ifdef CONFIG_X86_64 483 483 + #define KVM_REQUIRED_VMX_VM_ENTRY_CONTROLS \ 484 484 + (__KVM_REQUIRED_VMX_VM_ENTRY_CONTROLS | \ 485 485 + VM_ENTRY_IA32E_MODE) 486 486 + #else 487 487 + #define KVM_REQUIRED_VMX_VM_ENTRY_CONTROLS \ 488 488 + __KVM_REQUIRED_VMX_VM_ENTRY_CONTROLS 489 489 + #endif 490 490 + #define KVM_OPTIONAL_VMX_VM_ENTRY_CONTROLS \ 491 491 + (VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL | \ 492 492 + VM_ENTRY_LOAD_IA32_PAT | \ 493 493 + VM_ENTRY_LOAD_IA32_EFER | \ 494 494 + VM_ENTRY_LOAD_BNDCFGS | \ 495 495 + VM_ENTRY_PT_CONCEAL_PIP | \ 496 496 + VM_ENTRY_LOAD_IA32_RTIT_CTL) 497 497 + 498 498 + #define __KVM_REQUIRED_VMX_VM_EXIT_CONTROLS \ 499 499 + (VM_EXIT_SAVE_DEBUG_CONTROLS | \ 500 500 + VM_EXIT_ACK_INTR_ON_EXIT) 501 501 + #ifdef CONFIG_X86_64 502 502 + #define KVM_REQUIRED_VMX_VM_EXIT_CONTROLS \ 503 503 + (__KVM_REQUIRED_VMX_VM_EXIT_CONTROLS | \ 504 504 + VM_EXIT_HOST_ADDR_SPACE_SIZE) 505 505 + #else 506 506 + #define KVM_REQUIRED_VMX_VM_EXIT_CONTROLS \ 507 507 + __KVM_REQUIRED_VMX_VM_EXIT_CONTROLS 508 508 + #endif 509 509 + #define KVM_OPTIONAL_VMX_VM_EXIT_CONTROLS \ 510 510 + (VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL | \ 511 511 + VM_EXIT_SAVE_IA32_PAT | \ 512 512 + VM_EXIT_LOAD_IA32_PAT | \ 513 513 + VM_EXIT_SAVE_IA32_EFER | \ 514 514 + VM_EXIT_SAVE_VMX_PREEMPTION_TIMER | \ 515 515 + VM_EXIT_LOAD_IA32_EFER | \ 516 516 + VM_EXIT_CLEAR_BNDCFGS | \ 517 517 + VM_EXIT_PT_CONCEAL_PIP | \ 518 518 + VM_EXIT_CLEAR_IA32_RTIT_CTL) 519 519 + 520 520 + #define KVM_REQUIRED_VMX_PIN_BASED_VM_EXEC_CONTROL \ 521 521 + (PIN_BASED_EXT_INTR_MASK | \ 522 522 + PIN_BASED_NMI_EXITING) 523 523 + #define KVM_OPTIONAL_VMX_PIN_BASED_VM_EXEC_CONTROL \ 524 524 + (PIN_BASED_VIRTUAL_NMIS | \ 525 525 + PIN_BASED_POSTED_INTR | \ 526 526 + PIN_BASED_VMX_PREEMPTION_TIMER) 527 527 + 528 528 + #define __KVM_REQUIRED_VMX_CPU_BASED_VM_EXEC_CONTROL \ 529 529 + (CPU_BASED_HLT_EXITING | \ 530 530 + CPU_BASED_CR3_LOAD_EXITING | \ 531 531 + CPU_BASED_CR3_STORE_EXITING | \ 532 532 + CPU_BASED_UNCOND_IO_EXITING | \ 533 533 + CPU_BASED_MOV_DR_EXITING | \ 534 534 + CPU_BASED_USE_TSC_OFFSETTING | \ 535 535 + CPU_BASED_MWAIT_EXITING | \ 536 536 + CPU_BASED_MONITOR_EXITING | \ 537 537 + CPU_BASED_INVLPG_EXITING | \ 538 538 + CPU_BASED_RDPMC_EXITING | \ 539 539 + CPU_BASED_INTR_WINDOW_EXITING) 540 540 + 541 541 + #ifdef CONFIG_X86_64 542 542 + #define KVM_REQUIRED_VMX_CPU_BASED_VM_EXEC_CONTROL \ 543 543 + (__KVM_REQUIRED_VMX_CPU_BASED_VM_EXEC_CONTROL | \ 544 544 + CPU_BASED_CR8_LOAD_EXITING | \ 545 545 + CPU_BASED_CR8_STORE_EXITING) 546 546 + #else 547 547 + #define KVM_REQUIRED_VMX_CPU_BASED_VM_EXEC_CONTROL \ 548 548 + __KVM_REQUIRED_VMX_CPU_BASED_VM_EXEC_CONTROL 549 549 + #endif 550 550 + 551 551 + #define KVM_OPTIONAL_VMX_CPU_BASED_VM_EXEC_CONTROL \ 552 552 + (CPU_BASED_RDTSC_EXITING | \ 553 553 + CPU_BASED_TPR_SHADOW | \ 554 554 + CPU_BASED_USE_IO_BITMAPS | \ 555 555 + CPU_BASED_MONITOR_TRAP_FLAG | \ 556 556 + CPU_BASED_USE_MSR_BITMAPS | \ 557 557 + CPU_BASED_NMI_WINDOW_EXITING | \ 558 558 + CPU_BASED_PAUSE_EXITING | \ 559 559 + CPU_BASED_ACTIVATE_SECONDARY_CONTROLS | \ 560 560 + CPU_BASED_ACTIVATE_TERTIARY_CONTROLS) 561 561 + 562 562 + #define KVM_REQUIRED_VMX_SECONDARY_VM_EXEC_CONTROL 0 563 563 + #define KVM_OPTIONAL_VMX_SECONDARY_VM_EXEC_CONTROL \ 564 564 + (SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES | \ 565 565 + SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE | \ 566 566 + SECONDARY_EXEC_WBINVD_EXITING | \ 567 567 + SECONDARY_EXEC_ENABLE_VPID | \ 568 568 + SECONDARY_EXEC_ENABLE_EPT | \ 569 569 + SECONDARY_EXEC_UNRESTRICTED_GUEST | \ 570 570 + SECONDARY_EXEC_PAUSE_LOOP_EXITING | \ 571 571 + SECONDARY_EXEC_DESC | \ 572 572 + SECONDARY_EXEC_ENABLE_RDTSCP | \ 573 573 + SECONDARY_EXEC_ENABLE_INVPCID | \ 574 574 + SECONDARY_EXEC_APIC_REGISTER_VIRT | \ 575 575 + SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY | \ 576 576 + SECONDARY_EXEC_SHADOW_VMCS | \ 577 577 + SECONDARY_EXEC_XSAVES | \ 578 578 + SECONDARY_EXEC_RDSEED_EXITING | \ 579 579 + SECONDARY_EXEC_RDRAND_EXITING | \ 580 580 + SECONDARY_EXEC_ENABLE_PML | \ 581 581 + SECONDARY_EXEC_TSC_SCALING | \ 582 582 + SECONDARY_EXEC_ENABLE_USR_WAIT_PAUSE | \ 583 583 + SECONDARY_EXEC_PT_USE_GPA | \ 584 584 + SECONDARY_EXEC_PT_CONCEAL_VMX | \ 585 585 + SECONDARY_EXEC_ENABLE_VMFUNC | \ 586 586 + SECONDARY_EXEC_BUS_LOCK_DETECTION | \ 587 587 + SECONDARY_EXEC_NOTIFY_VM_EXITING | \ 588 588 + SECONDARY_EXEC_ENCLS_EXITING) 589 589 + 590 590 + #define KVM_REQUIRED_VMX_TERTIARY_VM_EXEC_CONTROL 0 591 591 + #define KVM_OPTIONAL_VMX_TERTIARY_VM_EXEC_CONTROL \ 592 592 + (TERTIARY_EXEC_IPI_VIRT) 593 593 + 594 594 + #define BUILD_CONTROLS_SHADOW(lname, uname, bits) \ 595 595 + static inline void lname##_controls_set(struct vcpu_vmx *vmx, u##bits val) \ 596 596 + { \ 597 597 + if (vmx->loaded_vmcs->controls_shadow.lname != val) { \ 598 598 + vmcs_write##bits(uname, val); \ 599 599 + vmx->loaded_vmcs->controls_shadow.lname = val; \ 600 600 + } \ 601 601 + } \ 602 602 + static inline u##bits __##lname##_controls_get(struct loaded_vmcs *vmcs) \ 603 603 + { \ 604 604 + return vmcs->controls_shadow.lname; \ 605 605 + } \ 606 606 + static inline u##bits lname##_controls_get(struct vcpu_vmx *vmx) \ 607 607 + { \ 608 608 + return __##lname##_controls_get(vmx->loaded_vmcs); \ 609 609 + } \ 610 610 + static __always_inline void lname##_controls_setbit(struct vcpu_vmx *vmx, u##bits val) \ 611 611 + { \ 612 612 + BUILD_BUG_ON(!(val & (KVM_REQUIRED_VMX_##uname | KVM_OPTIONAL_VMX_##uname))); \ 613 613 + lname##_controls_set(vmx, lname##_controls_get(vmx) | val); \ 614 614 + } \ 615 615 + static __always_inline void lname##_controls_clearbit(struct vcpu_vmx *vmx, u##bits val) \ 616 616 + { \ 617 617 + BUILD_BUG_ON(!(val & (KVM_REQUIRED_VMX_##uname | KVM_OPTIONAL_VMX_##uname))); \ 618 618 + lname##_controls_set(vmx, lname##_controls_get(vmx) & ~val); \ 503 619 } 504 620 BUILD_CONTROLS_SHADOW(vm_entry, VM_ENTRY_CONTROLS, 32) 505 621 BUILD_CONTROLS_SHADOW(vm_exit, VM_EXIT_CONTROLS, 32) ··· 740 624 static inline bool vmx_can_use_ipiv(struct kvm_vcpu *vcpu) 741 625 { 742 626 return lapic_in_kernel(vcpu) && enable_ipiv; 627 627 + } 628 628 + 629 629 + static inline bool guest_cpuid_has_evmcs(struct kvm_vcpu *vcpu) 630 630 + { 631 631 + /* 632 632 + * eVMCS is exposed to the guest if Hyper-V is enabled in CPUID and 633 633 + * eVMCS has been explicitly enabled by userspace. 634 634 + */ 635 635 + return vcpu->arch.hyperv_enabled && 636 636 + to_vmx(vcpu)->nested.enlightened_vmcs_enabled; 743 637 } 744 638 745 639 #endif /* __KVM_X86_VMX_H */

+1 -1

arch/x86/kvm/vmx/vmx_ops.h

reviewed

··· 10 10 #include "vmcs.h" 11 11 #include "../x86.h" 12 12 13 13 - asmlinkage void vmread_error(unsigned long field, bool fault); 13 13 + void vmread_error(unsigned long field, bool fault); 14 14 __attribute__((regparm(0))) void vmread_error_trampoline(unsigned long field, 15 15 bool fault); 16 16 void vmwrite_error(unsigned long field, unsigned long value);

+396 -182

arch/x86/kvm/x86.c

reviewed

··· 173 173 module_param(enable_vmware_backdoor, bool, S_IRUGO); 174 174 EXPORT_SYMBOL_GPL(enable_vmware_backdoor); 175 175 176 176 - static bool __read_mostly force_emulation_prefix = false; 177 177 - module_param(force_emulation_prefix, bool, S_IRUGO); 176 176 + /* 177 177 + * Flags to manipulate forced emulation behavior (any non-zero value will 178 178 + * enable forced emulation). 179 179 + */ 180 180 + #define KVM_FEP_CLEAR_RFLAGS_RF BIT(1) 181 181 + static int __read_mostly force_emulation_prefix; 182 182 + module_param(force_emulation_prefix, int, 0644); 178 183 179 184 int __read_mostly pi_inject_timer = -1; 180 185 module_param(pi_inject_timer, bint, S_IRUGO | S_IWUSR); ··· 533 528 #define EXCPT_TRAP 1 534 529 #define EXCPT_ABORT 2 535 530 #define EXCPT_INTERRUPT 3 531 531 + #define EXCPT_DB 4 536 532 537 533 static int exception_type(int vector) 538 534 { ··· 544 538 545 539 mask = 1 << vector; 546 540 547 547 - /* #DB is trap, as instruction watchpoints are handled elsewhere */ 548 548 - if (mask & ((1 << DB_VECTOR) | (1 << BP_VECTOR) | (1 << OF_VECTOR))) 541 541 + /* 542 542 + * #DBs can be trap-like or fault-like, the caller must check other CPU 543 543 + * state, e.g. DR6, to determine whether a #DB is a trap or fault. 544 544 + */ 545 545 + if (mask & (1 << DB_VECTOR)) 546 546 + return EXCPT_DB; 547 547 + 548 548 + if (mask & ((1 << BP_VECTOR) | (1 << OF_VECTOR))) 549 549 return EXCPT_TRAP; 550 550 551 551 if (mask & ((1 << DF_VECTOR) | (1 << MC_VECTOR))) ··· 561 549 return EXCPT_FAULT; 562 550 } 563 551 564 564 - void kvm_deliver_exception_payload(struct kvm_vcpu *vcpu) 552 552 + void kvm_deliver_exception_payload(struct kvm_vcpu *vcpu, 553 553 + struct kvm_queued_exception *ex) 565 554 { 566 566 - unsigned nr = vcpu->arch.exception.nr; 567 567 - bool has_payload = vcpu->arch.exception.has_payload; 568 568 - unsigned long payload = vcpu->arch.exception.payload; 569 569 - 570 570 - if (!has_payload) 555 555 + if (!ex->has_payload) 571 556 return; 572 557 573 573 - switch (nr) { 558 558 + switch (ex->vector) { 574 559 case DB_VECTOR: 575 560 /* 576 561 * "Certain debug exceptions may clear bit 0-3. The ··· 592 583 * So they need to be flipped for DR6. 593 584 */ 594 585 vcpu->arch.dr6 |= DR6_ACTIVE_LOW; 595 595 - vcpu->arch.dr6 |= payload; 596 596 - vcpu->arch.dr6 ^= payload & DR6_ACTIVE_LOW; 586 586 + vcpu->arch.dr6 |= ex->payload; 587 587 + vcpu->arch.dr6 ^= ex->payload & DR6_ACTIVE_LOW; 597 588 598 589 /* 599 590 * The #DB payload is defined as compatible with the 'pending ··· 604 595 vcpu->arch.dr6 &= ~BIT(12); 605 596 break; 606 597 case PF_VECTOR: 607 607 - vcpu->arch.cr2 = payload; 598 598 + vcpu->arch.cr2 = ex->payload; 608 599 break; 609 600 } 610 601 611 611 - vcpu->arch.exception.has_payload = false; 612 612 - vcpu->arch.exception.payload = 0; 602 602 + ex->has_payload = false; 603 603 + ex->payload = 0; 613 604 } 614 605 EXPORT_SYMBOL_GPL(kvm_deliver_exception_payload); 606 606 + 607 607 + static void kvm_queue_exception_vmexit(struct kvm_vcpu *vcpu, unsigned int vector, 608 608 + bool has_error_code, u32 error_code, 609 609 + bool has_payload, unsigned long payload) 610 610 + { 611 611 + struct kvm_queued_exception *ex = &vcpu->arch.exception_vmexit; 612 612 + 613 613 + ex->vector = vector; 614 614 + ex->injected = false; 615 615 + ex->pending = true; 616 616 + ex->has_error_code = has_error_code; 617 617 + ex->error_code = error_code; 618 618 + ex->has_payload = has_payload; 619 619 + ex->payload = payload; 620 620 + } 615 621 616 622 static void kvm_multiple_exception(struct kvm_vcpu *vcpu, 617 623 unsigned nr, bool has_error, u32 error_code, ··· 637 613 638 614 kvm_make_request(KVM_REQ_EVENT, vcpu); 639 615 616 616 + /* 617 617 + * If the exception is destined for L2 and isn't being reinjected, 618 618 + * morph it to a VM-Exit if L1 wants to intercept the exception. A 619 619 + * previously injected exception is not checked because it was checked 620 620 + * when it was original queued, and re-checking is incorrect if _L1_ 621 621 + * injected the exception, in which case it's exempt from interception. 622 622 + */ 623 623 + if (!reinject && is_guest_mode(vcpu) && 624 624 + kvm_x86_ops.nested_ops->is_exception_vmexit(vcpu, nr, error_code)) { 625 625 + kvm_queue_exception_vmexit(vcpu, nr, has_error, error_code, 626 626 + has_payload, payload); 627 627 + return; 628 628 + } 629 629 + 640 630 if (!vcpu->arch.exception.pending && !vcpu->arch.exception.injected) { 641 631 queue: 642 632 if (reinject) { 643 633 /* 644 644 - * On vmentry, vcpu->arch.exception.pending is only 645 645 - * true if an event injection was blocked by 646 646 - * nested_run_pending. In that case, however, 647 647 - * vcpu_enter_guest requests an immediate exit, 648 648 - * and the guest shouldn't proceed far enough to 649 649 - * need reinjection. 634 634 + * On VM-Entry, an exception can be pending if and only 635 635 + * if event injection was blocked by nested_run_pending. 636 636 + * In that case, however, vcpu_enter_guest() requests an 637 637 + * immediate exit, and the guest shouldn't proceed far 638 638 + * enough to need reinjection. 650 639 */ 651 651 - WARN_ON_ONCE(vcpu->arch.exception.pending); 640 640 + WARN_ON_ONCE(kvm_is_exception_pending(vcpu)); 652 641 vcpu->arch.exception.injected = true; 653 642 if (WARN_ON_ONCE(has_payload)) { 654 643 /* ··· 676 639 vcpu->arch.exception.injected = false; 677 640 } 678 641 vcpu->arch.exception.has_error_code = has_error; 679 679 - vcpu->arch.exception.nr = nr; 642 642 + vcpu->arch.exception.vector = nr; 680 643 vcpu->arch.exception.error_code = error_code; 681 644 vcpu->arch.exception.has_payload = has_payload; 682 645 vcpu->arch.exception.payload = payload; 683 646 if (!is_guest_mode(vcpu)) 684 684 - kvm_deliver_exception_payload(vcpu); 647 647 + kvm_deliver_exception_payload(vcpu, 648 648 + &vcpu->arch.exception); 685 649 return; 686 650 } 687 651 688 652 /* to check exception */ 689 689 - prev_nr = vcpu->arch.exception.nr; 653 653 + prev_nr = vcpu->arch.exception.vector; 690 654 if (prev_nr == DF_VECTOR) { 691 655 /* triple fault -> shutdown */ 692 656 kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu); ··· 695 657 } 696 658 class1 = exception_class(prev_nr); 697 659 class2 = exception_class(nr); 698 698 - if ((class1 == EXCPT_CONTRIBUTORY && class2 == EXCPT_CONTRIBUTORY) 699 699 - || (class1 == EXCPT_PF && class2 != EXCPT_BENIGN)) { 660 660 + if ((class1 == EXCPT_CONTRIBUTORY && class2 == EXCPT_CONTRIBUTORY) || 661 661 + (class1 == EXCPT_PF && class2 != EXCPT_BENIGN)) { 700 662 /* 701 701 - * Generate double fault per SDM Table 5-5. Set 702 702 - * exception.pending = true so that the double fault 703 703 - * can trigger a nested vmexit. 663 663 + * Synthesize #DF. Clear the previously injected or pending 664 664 + * exception so as not to incorrectly trigger shutdown. 704 665 */ 705 705 - vcpu->arch.exception.pending = true; 706 666 vcpu->arch.exception.injected = false; 707 707 - vcpu->arch.exception.has_error_code = true; 708 708 - vcpu->arch.exception.nr = DF_VECTOR; 709 709 - vcpu->arch.exception.error_code = 0; 710 710 - vcpu->arch.exception.has_payload = false; 711 711 - vcpu->arch.exception.payload = 0; 712 712 - } else 667 667 + vcpu->arch.exception.pending = false; 668 668 + 669 669 + kvm_queue_exception_e(vcpu, DF_VECTOR, 0); 670 670 + } else { 713 671 /* replace previous exception with a new one in a hope 714 672 that instruction re-execution will regenerate lost 715 673 exception */ 716 674 goto queue; 675 675 + } 717 676 } 718 677 719 678 void kvm_queue_exception(struct kvm_vcpu *vcpu, unsigned nr) ··· 764 729 void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault) 765 730 { 766 731 ++vcpu->stat.pf_guest; 767 767 - vcpu->arch.exception.nested_apf = 768 768 - is_guest_mode(vcpu) && fault->async_page_fault; 769 769 - if (vcpu->arch.exception.nested_apf) { 770 770 - vcpu->arch.apf.nested_apf_token = fault->address; 771 771 - kvm_queue_exception_e(vcpu, PF_VECTOR, fault->error_code); 772 772 - } else { 732 732 + 733 733 + /* 734 734 + * Async #PF in L2 is always forwarded to L1 as a VM-Exit regardless of 735 735 + * whether or not L1 wants to intercept "regular" #PF. 736 736 + */ 737 737 + if (is_guest_mode(vcpu) && fault->async_page_fault) 738 738 + kvm_queue_exception_vmexit(vcpu, PF_VECTOR, 739 739 + true, fault->error_code, 740 740 + true, fault->address); 741 741 + else 773 742 kvm_queue_exception_e_p(vcpu, PF_VECTOR, fault->error_code, 774 743 fault->address); 775 775 - } 776 744 } 777 745 EXPORT_SYMBOL_GPL(kvm_inject_page_fault); 778 746 779 779 - /* Returns true if the page fault was immediately morphed into a VM-Exit. */ 780 780 - bool kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, 747 747 + void kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, 781 748 struct x86_exception *fault) 782 749 { 783 750 struct kvm_mmu *fault_mmu; ··· 797 760 kvm_mmu_invalidate_gva(vcpu, fault_mmu, fault->address, 798 761 fault_mmu->root.hpa); 799 762 800 800 - /* 801 801 - * A workaround for KVM's bad exception handling. If KVM injected an 802 802 - * exception into L2, and L2 encountered a #PF while vectoring the 803 803 - * injected exception, manually check to see if L1 wants to intercept 804 804 - * #PF, otherwise queuing the #PF will lead to #DF or a lost exception. 805 805 - * In all other cases, defer the check to nested_ops->check_events(), 806 806 - * which will correctly handle priority (this does not). Note, other 807 807 - * exceptions, e.g. #GP, are theoretically affected, #PF is simply the 808 808 - * most problematic, e.g. when L0 and L1 are both intercepting #PF for 809 809 - * shadow paging. 810 810 - * 811 811 - * TODO: Rewrite exception handling to track injected and pending 812 812 - * (VM-Exit) exceptions separately. 813 813 - */ 814 814 - if (unlikely(vcpu->arch.exception.injected && is_guest_mode(vcpu)) && 815 815 - kvm_x86_ops.nested_ops->handle_page_fault_workaround(vcpu, fault)) 816 816 - return true; 817 817 - 818 763 fault_mmu->inject_page_fault(vcpu, fault); 819 819 - return false; 820 764 } 821 765 EXPORT_SYMBOL_GPL(kvm_inject_emulated_page_fault); 822 766 ··· 4859 4841 return (kvm_arch_interrupt_allowed(vcpu) && 4860 4842 kvm_cpu_accept_dm_intr(vcpu) && 4861 4843 !kvm_event_needs_reinjection(vcpu) && 4862 4862 - !vcpu->arch.exception.pending); 4844 4844 + !kvm_is_exception_pending(vcpu)); 4863 4845 } 4864 4846 4865 4847 static int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu, ··· 5034 5016 static void kvm_vcpu_ioctl_x86_get_vcpu_events(struct kvm_vcpu *vcpu, 5035 5017 struct kvm_vcpu_events *events) 5036 5018 { 5019 5019 + struct kvm_queued_exception *ex; 5020 5020 + 5037 5021 process_nmi(vcpu); 5038 5022 5039 5023 if (kvm_check_request(KVM_REQ_SMI, vcpu)) 5040 5024 process_smi(vcpu); 5041 5025 5042 5026 /* 5043 5043 - * In guest mode, payload delivery should be deferred, 5044 5044 - * so that the L1 hypervisor can intercept #PF before 5045 5045 - * CR2 is modified (or intercept #DB before DR6 is 5046 5046 - * modified under nVMX). Unless the per-VM capability, 5047 5047 - * KVM_CAP_EXCEPTION_PAYLOAD, is set, we may not defer the delivery of 5048 5048 - * an exception payload and handle after a KVM_GET_VCPU_EVENTS. Since we 5049 5049 - * opportunistically defer the exception payload, deliver it if the 5050 5050 - * capability hasn't been requested before processing a 5051 5051 - * KVM_GET_VCPU_EVENTS. 5027 5027 + * KVM's ABI only allows for one exception to be migrated. Luckily, 5028 5028 + * the only time there can be two queued exceptions is if there's a 5029 5029 + * non-exiting _injected_ exception, and a pending exiting exception. 5030 5030 + * In that case, ignore the VM-Exiting exception as it's an extension 5031 5031 + * of the injected exception. 5032 5032 + */ 5033 5033 + if (vcpu->arch.exception_vmexit.pending && 5034 5034 + !vcpu->arch.exception.pending && 5035 5035 + !vcpu->arch.exception.injected) 5036 5036 + ex = &vcpu->arch.exception_vmexit; 5037 5037 + else 5038 5038 + ex = &vcpu->arch.exception; 5039 5039 + 5040 5040 + /* 5041 5041 + * In guest mode, payload delivery should be deferred if the exception 5042 5042 + * will be intercepted by L1, e.g. KVM should not modifying CR2 if L1 5043 5043 + * intercepts #PF, ditto for DR6 and #DBs. If the per-VM capability, 5044 5044 + * KVM_CAP_EXCEPTION_PAYLOAD, is not set, userspace may or may not 5045 5045 + * propagate the payload and so it cannot be safely deferred. Deliver 5046 5046 + * the payload if the capability hasn't been requested. 5052 5047 */ 5053 5048 if (!vcpu->kvm->arch.exception_payload_enabled && 5054 5054 - vcpu->arch.exception.pending && vcpu->arch.exception.has_payload) 5055 5055 - kvm_deliver_exception_payload(vcpu); 5049 5049 + ex->pending && ex->has_payload) 5050 5050 + kvm_deliver_exception_payload(vcpu, ex); 5056 5051 5057 5052 /* 5058 5053 * The API doesn't provide the instruction length for software ··· 5073 5042 * isn't advanced, we should expect to encounter the exception 5074 5043 * again. 5075 5044 */ 5076 5076 - if (kvm_exception_is_soft(vcpu->arch.exception.nr)) { 5045 5045 + if (kvm_exception_is_soft(ex->vector)) { 5077 5046 events->exception.injected = 0; 5078 5047 events->exception.pending = 0; 5079 5048 } else { 5080 5080 - events->exception.injected = vcpu->arch.exception.injected; 5081 5081 - events->exception.pending = vcpu->arch.exception.pending; 5049 5049 + events->exception.injected = ex->injected; 5050 5050 + events->exception.pending = ex->pending; 5082 5051 /* 5083 5052 * For ABI compatibility, deliberately conflate 5084 5053 * pending and injected exceptions when 5085 5054 * KVM_CAP_EXCEPTION_PAYLOAD isn't enabled. 5086 5055 */ 5087 5056 if (!vcpu->kvm->arch.exception_payload_enabled) 5088 5088 - events->exception.injected |= 5089 5089 - vcpu->arch.exception.pending; 5057 5057 + events->exception.injected |= ex->pending; 5090 5058 } 5091 5091 - events->exception.nr = vcpu->arch.exception.nr; 5092 5092 - events->exception.has_error_code = vcpu->arch.exception.has_error_code; 5093 5093 - events->exception.error_code = vcpu->arch.exception.error_code; 5094 5094 - events->exception_has_payload = vcpu->arch.exception.has_payload; 5095 5095 - events->exception_payload = vcpu->arch.exception.payload; 5059 5059 + events->exception.nr = ex->vector; 5060 5060 + events->exception.has_error_code = ex->has_error_code; 5061 5061 + events->exception.error_code = ex->error_code; 5062 5062 + events->exception_has_payload = ex->has_payload; 5063 5063 + events->exception_payload = ex->payload; 5096 5064 5097 5065 events->interrupt.injected = 5098 5066 vcpu->arch.interrupt.injected && !vcpu->arch.interrupt.soft; ··· 5161 5131 return -EINVAL; 5162 5132 5163 5133 process_nmi(vcpu); 5134 5134 + 5135 5135 + /* 5136 5136 + * Flag that userspace is stuffing an exception, the next KVM_RUN will 5137 5137 + * morph the exception to a VM-Exit if appropriate. Do this only for 5138 5138 + * pending exceptions, already-injected exceptions are not subject to 5139 5139 + * intercpetion. Note, userspace that conflates pending and injected 5140 5140 + * is hosed, and will incorrectly convert an injected exception into a 5141 5141 + * pending exception, which in turn may cause a spurious VM-Exit. 5142 5142 + */ 5143 5143 + vcpu->arch.exception_from_userspace = events->exception.pending; 5144 5144 + 5145 5145 + vcpu->arch.exception_vmexit.pending = false; 5146 5146 + 5164 5147 vcpu->arch.exception.injected = events->exception.injected; 5165 5148 vcpu->arch.exception.pending = events->exception.pending; 5166 5166 - vcpu->arch.exception.nr = events->exception.nr; 5149 5149 + vcpu->arch.exception.vector = events->exception.nr; 5167 5150 vcpu->arch.exception.has_error_code = events->exception.has_error_code; 5168 5151 vcpu->arch.exception.error_code = events->exception.error_code; 5169 5152 vcpu->arch.exception.has_payload = events->exception_has_payload; ··· 7300 7257 int handle_ud(struct kvm_vcpu *vcpu) 7301 7258 { 7302 7259 static const char kvm_emulate_prefix[] = { __KVM_EMULATE_PREFIX }; 7260 7260 + int fep_flags = READ_ONCE(force_emulation_prefix); 7303 7261 int emul_type = EMULTYPE_TRAP_UD; 7304 7262 char sig[5]; /* ud2; .ascii "kvm" */ 7305 7263 struct x86_exception e; ··· 7308 7264 if (unlikely(!kvm_can_emulate_insn(vcpu, emul_type, NULL, 0))) 7309 7265 return 1; 7310 7266 7311 7311 - if (force_emulation_prefix && 7267 7267 + if (fep_flags && 7312 7268 kvm_read_guest_virt(vcpu, kvm_get_linear_rip(vcpu), 7313 7269 sig, sizeof(sig), &e) == 0 && 7314 7270 memcmp(sig, kvm_emulate_prefix, sizeof(sig)) == 0) { 7271 7271 + if (fep_flags & KVM_FEP_CLEAR_RFLAGS_RF) 7272 7272 + kvm_set_rflags(vcpu, kvm_get_rflags(vcpu) & ~X86_EFLAGS_RF); 7315 7273 kvm_rip_write(vcpu, kvm_rip_read(vcpu) + sizeof(sig)); 7316 7274 emul_type = EMULTYPE_TRAP_UD_FORCED; 7317 7275 } ··· 7979 7933 int r; 7980 7934 7981 7935 r = kvm_get_msr_with_filter(vcpu, msr_index, pdata); 7936 7936 + if (r < 0) 7937 7937 + return X86EMUL_UNHANDLEABLE; 7982 7938 7983 7983 - if (r && kvm_msr_user_space(vcpu, msr_index, KVM_EXIT_X86_RDMSR, 0, 7984 7984 - complete_emulated_rdmsr, r)) { 7985 7985 - /* Bounce to user space */ 7986 7986 - return X86EMUL_IO_NEEDED; 7939 7939 + if (r) { 7940 7940 + if (kvm_msr_user_space(vcpu, msr_index, KVM_EXIT_X86_RDMSR, 0, 7941 7941 + complete_emulated_rdmsr, r)) 7942 7942 + return X86EMUL_IO_NEEDED; 7943 7943 + 7944 7944 + trace_kvm_msr_read_ex(msr_index); 7945 7945 + return X86EMUL_PROPAGATE_FAULT; 7987 7946 } 7988 7947 7989 7989 - return r; 7948 7948 + trace_kvm_msr_read(msr_index, *pdata); 7949 7949 + return X86EMUL_CONTINUE; 7990 7950 } 7991 7951 7992 7952 static int emulator_set_msr_with_filter(struct x86_emulate_ctxt *ctxt, ··· 8002 7950 int r; 8003 7951 8004 7952 r = kvm_set_msr_with_filter(vcpu, msr_index, data); 7953 7953 + if (r < 0) 7954 7954 + return X86EMUL_UNHANDLEABLE; 8005 7955 8006 8006 - if (r && kvm_msr_user_space(vcpu, msr_index, KVM_EXIT_X86_WRMSR, data, 8007 8007 - complete_emulated_msr_access, r)) { 8008 8008 - /* Bounce to user space */ 8009 8009 - return X86EMUL_IO_NEEDED; 7956 7956 + if (r) { 7957 7957 + if (kvm_msr_user_space(vcpu, msr_index, KVM_EXIT_X86_WRMSR, data, 7958 7958 + complete_emulated_msr_access, r)) 7959 7959 + return X86EMUL_IO_NEEDED; 7960 7960 + 7961 7961 + trace_kvm_msr_write_ex(msr_index, data); 7962 7962 + return X86EMUL_PROPAGATE_FAULT; 8010 7963 } 8011 7964 8012 8012 - return r; 7965 7965 + trace_kvm_msr_write(msr_index, data); 7966 7966 + return X86EMUL_CONTINUE; 8013 7967 } 8014 7968 8015 7969 static int emulator_get_msr(struct x86_emulate_ctxt *ctxt, ··· 8219 8161 } 8220 8162 } 8221 8163 8222 8222 - static bool inject_emulated_exception(struct kvm_vcpu *vcpu) 8164 8164 + static void inject_emulated_exception(struct kvm_vcpu *vcpu) 8223 8165 { 8224 8166 struct x86_emulate_ctxt *ctxt = vcpu->arch.emulate_ctxt; 8225 8225 - if (ctxt->exception.vector == PF_VECTOR) 8226 8226 - return kvm_inject_emulated_page_fault(vcpu, &ctxt->exception); 8227 8167 8228 8228 - if (ctxt->exception.error_code_valid) 8168 8168 + if (ctxt->exception.vector == PF_VECTOR) 8169 8169 + kvm_inject_emulated_page_fault(vcpu, &ctxt->exception); 8170 8170 + else if (ctxt->exception.error_code_valid) 8229 8171 kvm_queue_exception_e(vcpu, ctxt->exception.vector, 8230 8172 ctxt->exception.error_code); 8231 8173 else 8232 8174 kvm_queue_exception(vcpu, ctxt->exception.vector); 8233 8233 - return false; 8234 8175 } 8235 8176 8236 8177 static struct x86_emulate_ctxt *alloc_emulate_ctxt(struct kvm_vcpu *vcpu) ··· 8605 8548 } 8606 8549 EXPORT_SYMBOL_GPL(kvm_skip_emulated_instruction); 8607 8550 8608 8608 - static bool kvm_vcpu_check_code_breakpoint(struct kvm_vcpu *vcpu, int *r) 8551 8551 + static bool kvm_is_code_breakpoint_inhibited(struct kvm_vcpu *vcpu) 8609 8552 { 8553 8553 + u32 shadow; 8554 8554 + 8555 8555 + if (kvm_get_rflags(vcpu) & X86_EFLAGS_RF) 8556 8556 + return true; 8557 8557 + 8558 8558 + /* 8559 8559 + * Intel CPUs inhibit code #DBs when MOV/POP SS blocking is active, 8560 8560 + * but AMD CPUs do not. MOV/POP SS blocking is rare, check that first 8561 8561 + * to avoid the relatively expensive CPUID lookup. 8562 8562 + */ 8563 8563 + shadow = static_call(kvm_x86_get_interrupt_shadow)(vcpu); 8564 8564 + return (shadow & KVM_X86_SHADOW_INT_MOV_SS) && 8565 8565 + guest_cpuid_is_intel(vcpu); 8566 8566 + } 8567 8567 + 8568 8568 + static bool kvm_vcpu_check_code_breakpoint(struct kvm_vcpu *vcpu, 8569 8569 + int emulation_type, int *r) 8570 8570 + { 8571 8571 + WARN_ON_ONCE(emulation_type & EMULTYPE_NO_DECODE); 8572 8572 + 8573 8573 + /* 8574 8574 + * Do not check for code breakpoints if hardware has already done the 8575 8575 + * checks, as inferred from the emulation type. On NO_DECODE and SKIP, 8576 8576 + * the instruction has passed all exception checks, and all intercepted 8577 8577 + * exceptions that trigger emulation have lower priority than code 8578 8578 + * breakpoints, i.e. the fact that the intercepted exception occurred 8579 8579 + * means any code breakpoints have already been serviced. 8580 8580 + * 8581 8581 + * Note, KVM needs to check for code #DBs on EMULTYPE_TRAP_UD_FORCED as 8582 8582 + * hardware has checked the RIP of the magic prefix, but not the RIP of 8583 8583 + * the instruction being emulated. The intent of forced emulation is 8584 8584 + * to behave as if KVM intercepted the instruction without an exception 8585 8585 + * and without a prefix. 8586 8586 + */ 8587 8587 + if (emulation_type & (EMULTYPE_NO_DECODE | EMULTYPE_SKIP | 8588 8588 + EMULTYPE_TRAP_UD | EMULTYPE_VMWARE_GP | EMULTYPE_PF)) 8589 8589 + return false; 8590 8590 + 8610 8591 if (unlikely(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) && 8611 8592 (vcpu->arch.guest_debug_dr7 & DR7_BP_EN_MASK)) { 8612 8593 struct kvm_run *kvm_run = vcpu->run; ··· 8664 8569 } 8665 8570 8666 8571 if (unlikely(vcpu->arch.dr7 & DR7_BP_EN_MASK) && 8667 8667 - !(kvm_get_rflags(vcpu) & X86_EFLAGS_RF)) { 8572 8572 + !kvm_is_code_breakpoint_inhibited(vcpu)) { 8668 8573 unsigned long eip = kvm_get_linear_rip(vcpu); 8669 8574 u32 dr6 = kvm_vcpu_check_hw_bp(eip, 0, 8670 8575 vcpu->arch.dr7, ··· 8766 8671 * are fault-like and are higher priority than any faults on 8767 8672 * the code fetch itself. 8768 8673 */ 8769 8769 - if (!(emulation_type & EMULTYPE_SKIP) && 8770 8770 - kvm_vcpu_check_code_breakpoint(vcpu, &r)) 8674 8674 + if (kvm_vcpu_check_code_breakpoint(vcpu, emulation_type, &r)) 8771 8675 return r; 8772 8676 8773 8677 r = x86_decode_emulated_instruction(vcpu, emulation_type, ··· 8864 8770 8865 8771 if (ctxt->have_exception) { 8866 8772 r = 1; 8867 8867 - if (inject_emulated_exception(vcpu)) 8868 8868 - return r; 8773 8773 + inject_emulated_exception(vcpu); 8869 8774 } else if (vcpu->arch.pio.count) { 8870 8775 if (!vcpu->arch.pio.in) { 8871 8776 /* FIXME: return into emulator if single-stepping. */ ··· 8894 8801 unsigned long rflags = static_call(kvm_x86_get_rflags)(vcpu); 8895 8802 toggle_interruptibility(vcpu, ctxt->interruptibility); 8896 8803 vcpu->arch.emulate_regs_need_sync_to_vcpu = false; 8804 8804 + 8805 8805 + /* 8806 8806 + * Note, EXCPT_DB is assumed to be fault-like as the emulator 8807 8807 + * only supports code breakpoints and general detect #DB, both 8808 8808 + * of which are fault-like. 8809 8809 + */ 8897 8810 if (!ctxt->have_exception || 8898 8811 exception_type(ctxt->exception.vector) == EXCPT_TRAP) { 8899 8812 kvm_pmu_trigger_event(vcpu, PERF_COUNT_HW_INSTRUCTIONS); ··· 9761 9662 9762 9663 static void kvm_inject_exception(struct kvm_vcpu *vcpu) 9763 9664 { 9764 9764 - trace_kvm_inj_exception(vcpu->arch.exception.nr, 9665 9665 + trace_kvm_inj_exception(vcpu->arch.exception.vector, 9765 9666 vcpu->arch.exception.has_error_code, 9766 9667 vcpu->arch.exception.error_code, 9767 9668 vcpu->arch.exception.injected); 9768 9669 9769 9670 if (vcpu->arch.exception.error_code && !is_protmode(vcpu)) 9770 9671 vcpu->arch.exception.error_code = false; 9771 9771 - static_call(kvm_x86_queue_exception)(vcpu); 9672 9672 + static_call(kvm_x86_inject_exception)(vcpu); 9772 9673 } 9773 9674 9774 9774 - static int inject_pending_event(struct kvm_vcpu *vcpu, bool *req_immediate_exit) 9675 9675 + /* 9676 9676 + * Check for any event (interrupt or exception) that is ready to be injected, 9677 9677 + * and if there is at least one event, inject the event with the highest 9678 9678 + * priority. This handles both "pending" events, i.e. events that have never 9679 9679 + * been injected into the guest, and "injected" events, i.e. events that were 9680 9680 + * injected as part of a previous VM-Enter, but weren't successfully delivered 9681 9681 + * and need to be re-injected. 9682 9682 + * 9683 9683 + * Note, this is not guaranteed to be invoked on a guest instruction boundary, 9684 9684 + * i.e. doesn't guarantee that there's an event window in the guest. KVM must 9685 9685 + * be able to inject exceptions in the "middle" of an instruction, and so must 9686 9686 + * also be able to re-inject NMIs and IRQs in the middle of an instruction. 9687 9687 + * I.e. for exceptions and re-injected events, NOT invoking this on instruction 9688 9688 + * boundaries is necessary and correct. 9689 9689 + * 9690 9690 + * For simplicity, KVM uses a single path to inject all events (except events 9691 9691 + * that are injected directly from L1 to L2) and doesn't explicitly track 9692 9692 + * instruction boundaries for asynchronous events. However, because VM-Exits 9693 9693 + * that can occur during instruction execution typically result in KVM skipping 9694 9694 + * the instruction or injecting an exception, e.g. instruction and exception 9695 9695 + * intercepts, and because pending exceptions have higher priority than pending 9696 9696 + * interrupts, KVM still honors instruction boundaries in most scenarios. 9697 9697 + * 9698 9698 + * But, if a VM-Exit occurs during instruction execution, and KVM does NOT skip 9699 9699 + * the instruction or inject an exception, then KVM can incorrecty inject a new 9700 9700 + * asynchrounous event if the event became pending after the CPU fetched the 9701 9701 + * instruction (in the guest). E.g. if a page fault (#PF, #NPF, EPT violation) 9702 9702 + * occurs and is resolved by KVM, a coincident NMI, SMI, IRQ, etc... can be 9703 9703 + * injected on the restarted instruction instead of being deferred until the 9704 9704 + * instruction completes. 9705 9705 + * 9706 9706 + * In practice, this virtualization hole is unlikely to be observed by the 9707 9707 + * guest, and even less likely to cause functional problems. To detect the 9708 9708 + * hole, the guest would have to trigger an event on a side effect of an early 9709 9709 + * phase of instruction execution, e.g. on the instruction fetch from memory. 9710 9710 + * And for it to be a functional problem, the guest would need to depend on the 9711 9711 + * ordering between that side effect, the instruction completing, _and_ the 9712 9712 + * delivery of the asynchronous event. 9713 9713 + */ 9714 9714 + static int kvm_check_and_inject_events(struct kvm_vcpu *vcpu, 9715 9715 + bool *req_immediate_exit) 9775 9716 { 9717 9717 + bool can_inject; 9776 9718 int r; 9777 9777 - bool can_inject = true; 9778 9719 9779 9779 - /* try to reinject previous events if any */ 9780 9780 - 9781 9781 - if (vcpu->arch.exception.injected) { 9782 9782 - kvm_inject_exception(vcpu); 9783 9783 - can_inject = false; 9784 9784 - } 9785 9720 /* 9786 9786 - * Do not inject an NMI or interrupt if there is a pending 9787 9787 - * exception. Exceptions and interrupts are recognized at 9788 9788 - * instruction boundaries, i.e. the start of an instruction. 9789 9789 - * Trap-like exceptions, e.g. #DB, have higher priority than 9790 9790 - * NMIs and interrupts, i.e. traps are recognized before an 9791 9791 - * NMI/interrupt that's pending on the same instruction. 9792 9792 - * Fault-like exceptions, e.g. #GP and #PF, are the lowest 9793 9793 - * priority, but are only generated (pended) during instruction 9794 9794 - * execution, i.e. a pending fault-like exception means the 9795 9795 - * fault occurred on the *previous* instruction and must be 9796 9796 - * serviced prior to recognizing any new events in order to 9797 9797 - * fully complete the previous instruction. 9721 9721 + * Process nested events first, as nested VM-Exit supercedes event 9722 9722 + * re-injection. If there's an event queued for re-injection, it will 9723 9723 + * be saved into the appropriate vmc{b,s}12 fields on nested VM-Exit. 9798 9724 */ 9799 9799 - else if (!vcpu->arch.exception.pending) { 9800 9800 - if (vcpu->arch.nmi_injected) { 9801 9801 - static_call(kvm_x86_inject_nmi)(vcpu); 9802 9802 - can_inject = false; 9803 9803 - } else if (vcpu->arch.interrupt.injected) { 9804 9804 - static_call(kvm_x86_inject_irq)(vcpu, true); 9805 9805 - can_inject = false; 9806 9806 - } 9807 9807 - } 9725 9725 + if (is_guest_mode(vcpu)) 9726 9726 + r = kvm_check_nested_events(vcpu); 9727 9727 + else 9728 9728 + r = 0; 9808 9729 9730 9730 + /* 9731 9731 + * Re-inject exceptions and events *especially* if immediate entry+exit 9732 9732 + * to/from L2 is needed, as any event that has already been injected 9733 9733 + * into L2 needs to complete its lifecycle before injecting a new event. 9734 9734 + * 9735 9735 + * Don't re-inject an NMI or interrupt if there is a pending exception. 9736 9736 + * This collision arises if an exception occurred while vectoring the 9737 9737 + * injected event, KVM intercepted said exception, and KVM ultimately 9738 9738 + * determined the fault belongs to the guest and queues the exception 9739 9739 + * for injection back into the guest. 9740 9740 + * 9741 9741 + * "Injected" interrupts can also collide with pending exceptions if 9742 9742 + * userspace ignores the "ready for injection" flag and blindly queues 9743 9743 + * an interrupt. In that case, prioritizing the exception is correct, 9744 9744 + * as the exception "occurred" before the exit to userspace. Trap-like 9745 9745 + * exceptions, e.g. most #DBs, have higher priority than interrupts. 9746 9746 + * And while fault-like exceptions, e.g. #GP and #PF, are the lowest 9747 9747 + * priority, they're only generated (pended) during instruction 9748 9748 + * execution, and interrupts are recognized at instruction boundaries. 9749 9749 + * Thus a pending fault-like exception means the fault occurred on the 9750 9750 + * *previous* instruction and must be serviced prior to recognizing any 9751 9751 + * new events in order to fully complete the previous instruction. 9752 9752 + */ 9753 9753 + if (vcpu->arch.exception.injected) 9754 9754 + kvm_inject_exception(vcpu); 9755 9755 + else if (kvm_is_exception_pending(vcpu)) 9756 9756 + ; /* see above */ 9757 9757 + else if (vcpu->arch.nmi_injected) 9758 9758 + static_call(kvm_x86_inject_nmi)(vcpu); 9759 9759 + else if (vcpu->arch.interrupt.injected) 9760 9760 + static_call(kvm_x86_inject_irq)(vcpu, true); 9761 9761 + 9762 9762 + /* 9763 9763 + * Exceptions that morph to VM-Exits are handled above, and pending 9764 9764 + * exceptions on top of injected exceptions that do not VM-Exit should 9765 9765 + * either morph to #DF or, sadly, override the injected exception. 9766 9766 + */ 9809 9767 WARN_ON_ONCE(vcpu->arch.exception.injected && 9810 9768 vcpu->arch.exception.pending); 9811 9769 9812 9770 /* 9813 9813 - * Call check_nested_events() even if we reinjected a previous event 9814 9814 - * in order for caller to determine if it should require immediate-exit 9815 9815 - * from L2 to L1 due to pending L1 events which require exit 9816 9816 - * from L2 to L1. 9771 9771 + * Bail if immediate entry+exit to/from the guest is needed to complete 9772 9772 + * nested VM-Enter or event re-injection so that a different pending 9773 9773 + * event can be serviced (or if KVM needs to exit to userspace). 9774 9774 + * 9775 9775 + * Otherwise, continue processing events even if VM-Exit occurred. The 9776 9776 + * VM-Exit will have cleared exceptions that were meant for L2, but 9777 9777 + * there may now be events that can be injected into L1. 9817 9778 */ 9818 9818 - if (is_guest_mode(vcpu)) { 9819 9819 - r = kvm_check_nested_events(vcpu); 9820 9820 - if (r < 0) 9821 9821 - goto out; 9822 9822 - } 9779 9779 + if (r < 0) 9780 9780 + goto out; 9823 9781 9824 9824 - /* try to inject new event if pending */ 9782 9782 + /* 9783 9783 + * A pending exception VM-Exit should either result in nested VM-Exit 9784 9784 + * or force an immediate re-entry and exit to/from L2, and exception 9785 9785 + * VM-Exits cannot be injected (flag should _never_ be set). 9786 9786 + */ 9787 9787 + WARN_ON_ONCE(vcpu->arch.exception_vmexit.injected || 9788 9788 + vcpu->arch.exception_vmexit.pending); 9789 9789 + 9790 9790 + /* 9791 9791 + * New events, other than exceptions, cannot be injected if KVM needs 9792 9792 + * to re-inject a previous event. See above comments on re-injecting 9793 9793 + * for why pending exceptions get priority. 9794 9794 + */ 9795 9795 + can_inject = !kvm_event_needs_reinjection(vcpu); 9796 9796 + 9825 9797 if (vcpu->arch.exception.pending) { 9826 9826 - if (exception_type(vcpu->arch.exception.nr) == EXCPT_FAULT) 9798 9798 + /* 9799 9799 + * Fault-class exceptions, except #DBs, set RF=1 in the RFLAGS 9800 9800 + * value pushed on the stack. Trap-like exception and all #DBs 9801 9801 + * leave RF as-is (KVM follows Intel's behavior in this regard; 9802 9802 + * AMD states that code breakpoint #DBs excplitly clear RF=0). 9803 9803 + * 9804 9804 + * Note, most versions of Intel's SDM and AMD's APM incorrectly 9805 9805 + * describe the behavior of General Detect #DBs, which are 9806 9806 + * fault-like. They do _not_ set RF, a la code breakpoints. 9807 9807 + */ 9808 9808 + if (exception_type(vcpu->arch.exception.vector) == EXCPT_FAULT) 9827 9809 __kvm_set_rflags(vcpu, kvm_get_rflags(vcpu) | 9828 9810 X86_EFLAGS_RF); 9829 9811 9830 9830 - if (vcpu->arch.exception.nr == DB_VECTOR) { 9831 9831 - kvm_deliver_exception_payload(vcpu); 9812 9812 + if (vcpu->arch.exception.vector == DB_VECTOR) { 9813 9813 + kvm_deliver_exception_payload(vcpu, &vcpu->arch.exception); 9832 9814 if (vcpu->arch.dr7 & DR7_GD) { 9833 9815 vcpu->arch.dr7 &= ~DR7_GD; 9834 9816 kvm_update_dr7(vcpu); ··· 9981 9801 } 9982 9802 9983 9803 if (is_guest_mode(vcpu) && 9984 9984 - kvm_x86_ops.nested_ops->hv_timer_pending && 9985 9985 - kvm_x86_ops.nested_ops->hv_timer_pending(vcpu)) 9804 9804 + kvm_x86_ops.nested_ops->has_events && 9805 9805 + kvm_x86_ops.nested_ops->has_events(vcpu)) 9986 9806 *req_immediate_exit = true; 9987 9807 9988 9988 - WARN_ON(vcpu->arch.exception.pending); 9808 9808 + WARN_ON(kvm_is_exception_pending(vcpu)); 9989 9809 return 0; 9990 9810 9991 9811 out: ··· 10290 10110 * When APICv gets disabled, we may still have injected interrupts 10291 10111 * pending. At the same time, KVM_REQ_EVENT may not be set as APICv was 10292 10112 * still active when the interrupt got accepted. Make sure 10293 10293 - * inject_pending_event() is called to check for that. 10113 10113 + * kvm_check_and_inject_events() is called to check for that. 10294 10114 */ 10295 10115 if (!apic->apicv_active) 10296 10116 kvm_make_request(KVM_REQ_EVENT, vcpu); ··· 10587 10407 goto out; 10588 10408 } 10589 10409 10590 10590 - r = inject_pending_event(vcpu, &req_immediate_exit); 10410 10410 + r = kvm_check_and_inject_events(vcpu, &req_immediate_exit); 10591 10411 if (r < 0) { 10592 10412 r = 0; 10593 10413 goto out; ··· 10826 10646 if (hv_timer) 10827 10647 kvm_lapic_switch_to_hv_timer(vcpu); 10828 10648 10829 10829 - if (!kvm_check_request(KVM_REQ_UNHALT, vcpu)) 10649 10649 + /* 10650 10650 + * If the vCPU is not runnable, a signal or another host event 10651 10651 + * of some kind is pending; service it without changing the 10652 10652 + * vCPU's activity state. 10653 10653 + */ 10654 10654 + if (!kvm_arch_vcpu_runnable(vcpu)) 10830 10655 return 1; 10656 10656 + } 10657 10657 + 10658 10658 + /* 10659 10659 + * Evaluate nested events before exiting the halted state. This allows 10660 10660 + * the halt state to be recorded properly in the VMCS12's activity 10661 10661 + * state field (AMD does not have a similar field and a VM-Exit always 10662 10662 + * causes a spurious wakeup from HLT). 10663 10663 + */ 10664 10664 + if (is_guest_mode(vcpu)) { 10665 10665 + if (kvm_check_nested_events(vcpu) < 0) 10666 10666 + return 0; 10831 10667 } 10832 10668 10833 10669 if (kvm_apic_accept_events(vcpu) < 0) ··· 10869 10673 10870 10674 static inline bool kvm_vcpu_running(struct kvm_vcpu *vcpu) 10871 10675 { 10872 10872 - if (is_guest_mode(vcpu)) 10873 10873 - kvm_check_nested_events(vcpu); 10874 10874 - 10875 10676 return (vcpu->arch.mp_state == KVM_MP_STATE_RUNNABLE && 10876 10677 !vcpu->arch.apf.halted); 10877 10678 } ··· 11017 10824 11018 10825 int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) 11019 10826 { 10827 10827 + struct kvm_queued_exception *ex = &vcpu->arch.exception; 11020 10828 struct kvm_run *kvm_run = vcpu->run; 11021 10829 int r; 11022 10830 ··· 11046 10852 r = 0; 11047 10853 goto out; 11048 10854 } 11049 11049 - kvm_clear_request(KVM_REQ_UNHALT, vcpu); 11050 10855 r = -EAGAIN; 11051 10856 if (signal_pending(current)) { 11052 10857 r = -EINTR; ··· 11074 10881 goto out; 11075 10882 } 11076 10883 } 10884 10884 + 10885 10885 + /* 10886 10886 + * If userspace set a pending exception and L2 is active, convert it to 10887 10887 + * a pending VM-Exit if L1 wants to intercept the exception. 10888 10888 + */ 10889 10889 + if (vcpu->arch.exception_from_userspace && is_guest_mode(vcpu) && 10890 10890 + kvm_x86_ops.nested_ops->is_exception_vmexit(vcpu, ex->vector, 10891 10891 + ex->error_code)) { 10892 10892 + kvm_queue_exception_vmexit(vcpu, ex->vector, 10893 10893 + ex->has_error_code, ex->error_code, 10894 10894 + ex->has_payload, ex->payload); 10895 10895 + ex->injected = false; 10896 10896 + ex->pending = false; 10897 10897 + } 10898 10898 + vcpu->arch.exception_from_userspace = false; 11077 10899 11078 10900 if (unlikely(vcpu->arch.complete_userspace_io)) { 11079 10901 int (*cui)(struct kvm_vcpu *) = vcpu->arch.complete_userspace_io; ··· 11196 10988 kvm_set_rflags(vcpu, regs->rflags | X86_EFLAGS_FIXED); 11197 10989 11198 10990 vcpu->arch.exception.pending = false; 10991 10991 + vcpu->arch.exception_vmexit.pending = false; 11199 10992 11200 10993 kvm_make_request(KVM_REQ_EVENT, vcpu); 11201 10994 } ··· 11334 11125 } 11335 11126 11336 11127 /* 11337 11337 - * KVM_MP_STATE_INIT_RECEIVED means the processor is in 11338 11338 - * INIT state; latched init should be reported using 11339 11339 - * KVM_SET_VCPU_EVENTS, so reject it here. 11128 11128 + * Pending INITs are reported using KVM_SET_VCPU_EVENTS, disallow 11129 11129 + * forcing the guest into INIT/SIPI if those events are supposed to be 11130 11130 + * blocked. KVM prioritizes SMI over INIT, so reject INIT/SIPI state 11131 11131 + * if an SMI is pending as well. 11340 11132 */ 11341 11341 - if ((kvm_vcpu_latch_init(vcpu) || vcpu->arch.smi_pending) && 11133 11133 + if ((!kvm_apic_init_sipi_allowed(vcpu) || vcpu->arch.smi_pending) && 11342 11134 (mp_state->mp_state == KVM_MP_STATE_SIPI_RECEIVED || 11343 11135 mp_state->mp_state == KVM_MP_STATE_INIT_RECEIVED)) 11344 11136 goto out; ··· 11578 11368 11579 11369 if (dbg->control & (KVM_GUESTDBG_INJECT_DB | KVM_GUESTDBG_INJECT_BP)) { 11580 11370 r = -EBUSY; 11581 11581 - if (vcpu->arch.exception.pending) 11371 11371 + if (kvm_is_exception_pending(vcpu)) 11582 11372 goto out; 11583 11373 if (dbg->control & KVM_GUESTDBG_INJECT_DB) 11584 11374 kvm_queue_exception(vcpu, DB_VECTOR); ··· 11960 11750 struct fpstate *fpstate = vcpu->arch.guest_fpu.fpstate; 11961 11751 11962 11752 /* 11963 11963 - * To avoid have the INIT path from kvm_apic_has_events() that be 11964 11964 - * called with loaded FPU and does not let userspace fix the state. 11753 11753 + * All paths that lead to INIT are required to load the guest's 11754 11754 + * FPU state (because most paths are buried in KVM_RUN). 11965 11755 */ 11966 11756 if (init_event) 11967 11757 kvm_put_guest_fpu(vcpu); ··· 12290 12080 if (ret) 12291 12081 goto out_page_track; 12292 12082 12083 12083 + ret = static_call(kvm_x86_vm_init)(kvm); 12084 12084 + if (ret) 12085 12085 + goto out_uninit_mmu; 12086 12086 + 12293 12087 INIT_HLIST_HEAD(&kvm->arch.mask_notifier_list); 12294 12088 INIT_LIST_HEAD(&kvm->arch.assigned_dev_head); 12295 12089 atomic_set(&kvm->arch.noncoherent_dma_count, 0); ··· 12329 12115 kvm_hv_init_vm(kvm); 12330 12116 kvm_xen_init_vm(kvm); 12331 12117 12332 12332 - return static_call(kvm_x86_vm_init)(kvm); 12118 12118 + return 0; 12333 12119 12120 12120 + out_uninit_mmu: 12121 12121 + kvm_mmu_uninit_vm(kvm); 12334 12122 out_page_track: 12335 12123 kvm_page_track_cleanup(kvm); 12336 12124 out: ··· 12805 12589 if (!list_empty_careful(&vcpu->async_pf.done)) 12806 12590 return true; 12807 12591 12808 12808 - if (kvm_apic_has_events(vcpu)) 12592 12592 + if (kvm_apic_has_pending_init_or_sipi(vcpu) && 12593 12593 + kvm_apic_init_sipi_allowed(vcpu)) 12809 12594 return true; 12810 12595 12811 12596 if (vcpu->arch.pv.pv_unhalted) 12812 12597 return true; 12813 12598 12814 12814 - if (vcpu->arch.exception.pending) 12599 12599 + if (kvm_is_exception_pending(vcpu)) 12815 12600 return true; 12816 12601 12817 12602 if (kvm_test_request(KVM_REQ_NMI, vcpu) || ··· 12834 12617 return true; 12835 12618 12836 12619 if (is_guest_mode(vcpu) && 12837 12837 - kvm_x86_ops.nested_ops->hv_timer_pending && 12838 12838 - kvm_x86_ops.nested_ops->hv_timer_pending(vcpu)) 12620 12620 + kvm_x86_ops.nested_ops->has_events && 12621 12621 + kvm_x86_ops.nested_ops->has_events(vcpu)) 12839 12622 return true; 12840 12623 12841 12624 if (kvm_xen_has_pending_events(vcpu)) 12842 12842 - return true; 12843 12843 - 12844 12844 - if (kvm_test_request(KVM_REQ_TRIPLE_FAULT, vcpu)) 12845 12625 return true; 12846 12626 12847 12627 return false; ··· 13064 12850 { 13065 12851 if (unlikely(!lapic_in_kernel(vcpu) || 13066 12852 kvm_event_needs_reinjection(vcpu) || 13067 13067 - vcpu->arch.exception.pending)) 12853 12853 + kvm_is_exception_pending(vcpu))) 13068 12854 return false; 13069 12855 13070 12856 if (kvm_hlt_in_guest(vcpu->kvm) && !kvm_can_deliver_async_pf(vcpu)) ··· 13615 13401 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_page_fault); 13616 13402 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_msr); 13617 13403 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_cr); 13618 13618 - EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_vmrun); 13404 13404 + EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_vmenter); 13619 13405 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_vmexit); 13620 13406 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_vmexit_inject); 13621 13407 EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_intr_vmexit);

+10 -6

arch/x86/kvm/x86.h

reviewed

··· 82 82 void kvm_service_local_tlb_flush_requests(struct kvm_vcpu *vcpu); 83 83 int kvm_check_nested_events(struct kvm_vcpu *vcpu); 84 84 85 85 + static inline bool kvm_is_exception_pending(struct kvm_vcpu *vcpu) 86 86 + { 87 87 + return vcpu->arch.exception.pending || 88 88 + vcpu->arch.exception_vmexit.pending || 89 89 + kvm_test_request(KVM_REQ_TRIPLE_FAULT, vcpu); 90 90 + } 91 91 + 85 92 static inline void kvm_clear_exception_queue(struct kvm_vcpu *vcpu) 86 93 { 87 94 vcpu->arch.exception.pending = false; 88 95 vcpu->arch.exception.injected = false; 96 96 + vcpu->arch.exception_vmexit.pending = false; 89 97 } 90 98 91 99 static inline void kvm_queue_interrupt(struct kvm_vcpu *vcpu, u8 vector, ··· 275 267 return !(kvm->arch.disabled_quirks & quirk); 276 268 } 277 269 278 278 - static inline bool kvm_vcpu_latch_init(struct kvm_vcpu *vcpu) 279 279 - { 280 280 - return is_smm(vcpu) || static_call(kvm_x86_apic_init_signal_blocked)(vcpu); 281 281 - } 282 282 - 283 270 void kvm_inject_realmode_interrupt(struct kvm_vcpu *vcpu, int irq, int inc_eip); 284 271 285 272 u64 get_kvmclock_ns(struct kvm *kvm); ··· 289 286 290 287 int handle_ud(struct kvm_vcpu *vcpu); 291 288 292 292 - void kvm_deliver_exception_payload(struct kvm_vcpu *vcpu); 289 289 + void kvm_deliver_exception_payload(struct kvm_vcpu *vcpu, 290 290 + struct kvm_queued_exception *ex); 293 291 294 292 void kvm_vcpu_mtrr_init(struct kvm_vcpu *vcpu); 295 293 u8 kvm_mtrr_get_guest_memory_type(struct kvm_vcpu *vcpu, gfn_t gfn);

-1

arch/x86/kvm/xen.c

reviewed

··· 1065 1065 del_timer(&vcpu->arch.xen.poll_timer); 1066 1066 1067 1067 vcpu->arch.mp_state = KVM_MP_STATE_RUNNABLE; 1068 1068 - kvm_clear_request(KVM_REQ_UNHALT, vcpu); 1069 1068 } 1070 1069 1071 1070 vcpu->arch.xen.poll_evtchn = 0;

+2

drivers/base/node.c

reviewed

··· 433 433 "Node %d ShadowCallStack:%8lu kB\n" 434 434 #endif 435 435 "Node %d PageTables: %8lu kB\n" 436 436 + "Node %d SecPageTables: %8lu kB\n" 436 437 "Node %d NFS_Unstable: %8lu kB\n" 437 438 "Node %d Bounce: %8lu kB\n" 438 439 "Node %d WritebackTmp: %8lu kB\n" ··· 460 459 nid, node_page_state(pgdat, NR_KERNEL_SCS_KB), 461 460 #endif 462 461 nid, K(node_page_state(pgdat, NR_PAGETABLE)), 462 462 + nid, K(node_page_state(pgdat, NR_SECONDARY_PAGETABLE)), 463 463 nid, 0UL, 464 464 nid, K(sum_zone_node_page_state(nid, NR_BOUNCE)), 465 465 nid, K(node_page_state(pgdat, NR_WRITEBACK_TEMP)),

+2

fs/proc/meminfo.c

reviewed

··· 115 115 #endif 116 116 show_val_kb(m, "PageTables: ", 117 117 global_node_page_state(NR_PAGETABLE)); 118 118 + show_val_kb(m, "SecPageTables: ", 119 119 + global_node_page_state(NR_SECONDARY_PAGETABLE)); 118 120 119 121 show_val_kb(m, "NFS_Unstable: ", 0); 120 122 show_val_kb(m, "Bounce: ",

+14 -2

include/linux/kvm_host.h

reviewed

··· 151 151 #define KVM_REQUEST_NO_ACTION BIT(10) 152 152 /* 153 153 * Architecture-independent vcpu->requests bit members 154 154 - * Bits 4-7 are reserved for more arch-independent bits. 154 154 + * Bits 3-7 are reserved for more arch-independent bits. 155 155 */ 156 156 #define KVM_REQ_TLB_FLUSH (0 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) 157 157 #define KVM_REQ_VM_DEAD (1 | KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) 158 158 #define KVM_REQ_UNBLOCK 2 159 159 - #define KVM_REQ_UNHALT 3 160 159 #define KVM_REQUEST_ARCH_BASE 8 161 160 162 161 /* ··· 2245 2246 vcpu->stat.signal_exits++; 2246 2247 } 2247 2248 #endif /* CONFIG_KVM_XFER_TO_GUEST_WORK */ 2249 2249 + 2250 2250 + /* 2251 2251 + * If more than one page is being (un)accounted, @virt must be the address of 2252 2252 + * the first page of a block of pages what were allocated together (i.e 2253 2253 + * accounted together). 2254 2254 + * 2255 2255 + * kvm_account_pgtable_pages() is thread-safe because mod_lruvec_page_state() 2256 2256 + * is thread-safe. 2257 2257 + */ 2258 2258 + static inline void kvm_account_pgtable_pages(void *virt, int nr) 2259 2259 + { 2260 2260 + mod_lruvec_page_state(virt_to_page(virt), NR_SECONDARY_PAGETABLE, nr); 2261 2261 + } 2248 2262 2249 2263 /* 2250 2264 * This defines how many reserved entries we want to keep before we

+1

include/linux/mmzone.h

reviewed

··· 216 216 NR_KERNEL_SCS_KB, /* measured in KiB */ 217 217 #endif 218 218 NR_PAGETABLE, /* used for pagetables */ 219 219 + NR_SECONDARY_PAGETABLE, /* secondary pagetables, e.g. KVM pagetables */ 219 220 #ifdef CONFIG_SWAP 220 221 NR_SWAPCACHE, 221 222 #endif

+1

mm/memcontrol.c

reviewed

··· 1401 1401 { "kernel", MEMCG_KMEM }, 1402 1402 { "kernel_stack", NR_KERNEL_STACK_KB }, 1403 1403 { "pagetables", NR_PAGETABLE }, 1404 1404 + { "sec_pagetables", NR_SECONDARY_PAGETABLE }, 1404 1405 { "percpu", MEMCG_PERCPU_B }, 1405 1406 { "sock", MEMCG_SOCK }, 1406 1407 { "vmalloc", MEMCG_VMALLOC },

+5 -1

mm/page_alloc.c

reviewed

··· 6085 6085 " active_file:%lu inactive_file:%lu isolated_file:%lu\n" 6086 6086 " unevictable:%lu dirty:%lu writeback:%lu\n" 6087 6087 " slab_reclaimable:%lu slab_unreclaimable:%lu\n" 6088 6088 - " mapped:%lu shmem:%lu pagetables:%lu bounce:%lu\n" 6088 6088 + " mapped:%lu shmem:%lu pagetables:%lu\n" 6089 6089 + " sec_pagetables:%lu bounce:%lu\n" 6089 6090 " kernel_misc_reclaimable:%lu\n" 6090 6091 " free:%lu free_pcp:%lu free_cma:%lu\n", 6091 6092 global_node_page_state(NR_ACTIVE_ANON), ··· 6103 6102 global_node_page_state(NR_FILE_MAPPED), 6104 6103 global_node_page_state(NR_SHMEM), 6105 6104 global_node_page_state(NR_PAGETABLE), 6105 6105 + global_node_page_state(NR_SECONDARY_PAGETABLE), 6106 6106 global_zone_page_state(NR_BOUNCE), 6107 6107 global_node_page_state(NR_KERNEL_MISC_RECLAIMABLE), 6108 6108 global_zone_page_state(NR_FREE_PAGES), ··· 6137 6135 " shadow_call_stack:%lukB" 6138 6136 #endif 6139 6137 " pagetables:%lukB" 6138 6138 + " sec_pagetables:%lukB" 6140 6139 " all_unreclaimable? %s" 6141 6140 "\n", 6142 6141 pgdat->node_id, ··· 6163 6160 node_page_state(pgdat, NR_KERNEL_SCS_KB), 6164 6161 #endif 6165 6162 K(node_page_state(pgdat, NR_PAGETABLE)), 6163 6163 + K(node_page_state(pgdat, NR_SECONDARY_PAGETABLE)), 6166 6164 pgdat->kswapd_failures >= MAX_RECLAIM_RETRIES ? 6167 6165 "yes" : "no"); 6168 6166 }

+1

mm/vmstat.c

reviewed

··· 1247 1247 "nr_shadow_call_stack", 1248 1248 #endif 1249 1249 "nr_page_table_pages", 1250 1250 + "nr_sec_page_table_pages", 1250 1251 #ifdef CONFIG_SWAP 1251 1252 "nr_swapcached", 1252 1253 #endif

+1

tools/testing/selftests/kvm/.gitignore

reviewed

··· 28 28 /x86_64/max_vcpuid_cap_test 29 29 /x86_64/mmio_warning_test 30 30 /x86_64/monitor_mwait_test 31 31 + /x86_64/nested_exceptions_test 31 32 /x86_64/nx_huge_pages_test 32 33 /x86_64/platform_info_test 33 34 /x86_64/pmu_event_filter_test

+1

tools/testing/selftests/kvm/Makefile

reviewed

··· 91 91 TEST_GEN_PROGS_x86_64 += x86_64/kvm_pv_test 92 92 TEST_GEN_PROGS_x86_64 += x86_64/mmio_warning_test 93 93 TEST_GEN_PROGS_x86_64 += x86_64/monitor_mwait_test 94 94 + TEST_GEN_PROGS_x86_64 += x86_64/nested_exceptions_test 94 95 TEST_GEN_PROGS_x86_64 += x86_64/platform_info_test 95 96 TEST_GEN_PROGS_x86_64 += x86_64/pmu_event_filter_test 96 97 TEST_GEN_PROGS_x86_64 += x86_64/set_boot_cpu_id

+42 -3

tools/testing/selftests/kvm/include/x86_64/evmcs.h

reviewed

··· 203 203 u32 reserved:30; 204 204 } hv_enlightenments_control; 205 205 u32 hv_vp_id; 206 206 - 206 206 + u32 padding32_2; 207 207 u64 hv_vm_id; 208 208 u64 partition_assist_page; 209 209 u64 padding64_4[4]; 210 210 u64 guest_bndcfgs; 211 211 - u64 padding64_5[7]; 211 211 + u64 guest_ia32_perf_global_ctrl; 212 212 + u64 guest_ia32_s_cet; 213 213 + u64 guest_ssp; 214 214 + u64 guest_ia32_int_ssp_table_addr; 215 215 + u64 guest_ia32_lbr_ctl; 216 216 + u64 padding64_5[2]; 212 217 u64 xss_exit_bitmap; 213 213 - u64 padding64_6[7]; 218 218 + u64 encls_exiting_bitmap; 219 219 + u64 host_ia32_perf_global_ctrl; 220 220 + u64 tsc_multiplier; 221 221 + u64 host_ia32_s_cet; 222 222 + u64 host_ssp; 223 223 + u64 host_ia32_int_ssp_table_addr; 224 224 + u64 padding64_6; 214 225 }; 215 226 216 227 #define HV_VMX_ENLIGHTENED_CLEAN_FIELD_NONE 0 ··· 666 655 break; 667 656 case VIRTUAL_PROCESSOR_ID: 668 657 *value = current_evmcs->virtual_processor_id; 658 658 + break; 659 659 + case HOST_IA32_PERF_GLOBAL_CTRL: 660 660 + *value = current_evmcs->host_ia32_perf_global_ctrl; 661 661 + break; 662 662 + case GUEST_IA32_PERF_GLOBAL_CTRL: 663 663 + *value = current_evmcs->guest_ia32_perf_global_ctrl; 664 664 + break; 665 665 + case ENCLS_EXITING_BITMAP: 666 666 + *value = current_evmcs->encls_exiting_bitmap; 667 667 + break; 668 668 + case TSC_MULTIPLIER: 669 669 + *value = current_evmcs->tsc_multiplier; 669 670 break; 670 671 default: return 1; 671 672 } ··· 1191 1168 case VIRTUAL_PROCESSOR_ID: 1192 1169 current_evmcs->virtual_processor_id = value; 1193 1170 current_evmcs->hv_clean_fields &= ~HV_VMX_ENLIGHTENED_CLEAN_FIELD_CONTROL_XLAT; 1171 1171 + break; 1172 1172 + case HOST_IA32_PERF_GLOBAL_CTRL: 1173 1173 + current_evmcs->host_ia32_perf_global_ctrl = value; 1174 1174 + current_evmcs->hv_clean_fields &= ~HV_VMX_ENLIGHTENED_CLEAN_FIELD_HOST_GRP1; 1175 1175 + break; 1176 1176 + case GUEST_IA32_PERF_GLOBAL_CTRL: 1177 1177 + current_evmcs->guest_ia32_perf_global_ctrl = value; 1178 1178 + current_evmcs->hv_clean_fields &= ~HV_VMX_ENLIGHTENED_CLEAN_FIELD_GUEST_GRP1; 1179 1179 + break; 1180 1180 + case ENCLS_EXITING_BITMAP: 1181 1181 + current_evmcs->encls_exiting_bitmap = value; 1182 1182 + current_evmcs->hv_clean_fields &= ~HV_VMX_ENLIGHTENED_CLEAN_FIELD_CONTROL_GRP2; 1183 1183 + break; 1184 1184 + case TSC_MULTIPLIER: 1185 1185 + current_evmcs->tsc_multiplier = value; 1186 1186 + current_evmcs->hv_clean_fields &= ~HV_VMX_ENLIGHTENED_CLEAN_FIELD_CONTROL_GRP2; 1194 1187 break; 1195 1188 default: return 1; 1196 1189 }

+2 -5

tools/testing/selftests/kvm/include/x86_64/svm_util.h

reviewed

··· 9 9 #ifndef SELFTEST_KVM_SVM_UTILS_H 10 10 #define SELFTEST_KVM_SVM_UTILS_H 11 11 12 12 + #include <asm/svm.h> 13 13 + 12 14 #include <stdint.h> 13 15 #include "svm.h" 14 16 #include "processor.h" 15 15 - 16 16 - #define SVM_EXIT_EXCP_BASE 0x040 17 17 - #define SVM_EXIT_HLT 0x078 18 18 - #define SVM_EXIT_MSR 0x07c 19 19 - #define SVM_EXIT_VMMCALL 0x081 20 17 21 18 struct svm_test_data { 22 19 /* VMCB */

+4 -49

tools/testing/selftests/kvm/include/x86_64/vmx.h

reviewed

··· 8 8 #ifndef SELFTEST_KVM_VMX_H 9 9 #define SELFTEST_KVM_VMX_H 10 10 11 11 + #include <asm/vmx.h> 12 12 + 11 13 #include <stdint.h> 12 14 #include "processor.h" 13 15 #include "apic.h" ··· 102 100 #define VMX_EPT_VPID_CAP_AD_BITS 0x00200000 103 101 104 102 #define EXIT_REASON_FAILED_VMENTRY 0x80000000 105 105 - #define EXIT_REASON_EXCEPTION_NMI 0 106 106 - #define EXIT_REASON_EXTERNAL_INTERRUPT 1 107 107 - #define EXIT_REASON_TRIPLE_FAULT 2 108 108 - #define EXIT_REASON_INTERRUPT_WINDOW 7 109 109 - #define EXIT_REASON_NMI_WINDOW 8 110 110 - #define EXIT_REASON_TASK_SWITCH 9 111 111 - #define EXIT_REASON_CPUID 10 112 112 - #define EXIT_REASON_HLT 12 113 113 - #define EXIT_REASON_INVD 13 114 114 - #define EXIT_REASON_INVLPG 14 115 115 - #define EXIT_REASON_RDPMC 15 116 116 - #define EXIT_REASON_RDTSC 16 117 117 - #define EXIT_REASON_VMCALL 18 118 118 - #define EXIT_REASON_VMCLEAR 19 119 119 - #define EXIT_REASON_VMLAUNCH 20 120 120 - #define EXIT_REASON_VMPTRLD 21 121 121 - #define EXIT_REASON_VMPTRST 22 122 122 - #define EXIT_REASON_VMREAD 23 123 123 - #define EXIT_REASON_VMRESUME 24 124 124 - #define EXIT_REASON_VMWRITE 25 125 125 - #define EXIT_REASON_VMOFF 26 126 126 - #define EXIT_REASON_VMON 27 127 127 - #define EXIT_REASON_CR_ACCESS 28 128 128 - #define EXIT_REASON_DR_ACCESS 29 129 129 - #define EXIT_REASON_IO_INSTRUCTION 30 130 130 - #define EXIT_REASON_MSR_READ 31 131 131 - #define EXIT_REASON_MSR_WRITE 32 132 132 - #define EXIT_REASON_INVALID_STATE 33 133 133 - #define EXIT_REASON_MWAIT_INSTRUCTION 36 134 134 - #define EXIT_REASON_MONITOR_INSTRUCTION 39 135 135 - #define EXIT_REASON_PAUSE_INSTRUCTION 40 136 136 - #define EXIT_REASON_MCE_DURING_VMENTRY 41 137 137 - #define EXIT_REASON_TPR_BELOW_THRESHOLD 43 138 138 - #define EXIT_REASON_APIC_ACCESS 44 139 139 - #define EXIT_REASON_EOI_INDUCED 45 140 140 - #define EXIT_REASON_EPT_VIOLATION 48 141 141 - #define EXIT_REASON_EPT_MISCONFIG 49 142 142 - #define EXIT_REASON_INVEPT 50 143 143 - #define EXIT_REASON_RDTSCP 51 144 144 - #define EXIT_REASON_PREEMPTION_TIMER 52 145 145 - #define EXIT_REASON_INVVPID 53 146 146 - #define EXIT_REASON_WBINVD 54 147 147 - #define EXIT_REASON_XSETBV 55 148 148 - #define EXIT_REASON_APIC_WRITE 56 149 149 - #define EXIT_REASON_INVPCID 58 150 150 - #define EXIT_REASON_PML_FULL 62 151 151 - #define EXIT_REASON_XSAVES 63 152 152 - #define EXIT_REASON_XRSTORS 64 153 153 - #define LAST_EXIT_REASON 64 154 103 155 104 enum vmcs_field { 156 105 VIRTUAL_PROCESSOR_ID = 0x00000000, ··· 161 208 VMWRITE_BITMAP_HIGH = 0x00002029, 162 209 XSS_EXIT_BITMAP = 0x0000202C, 163 210 XSS_EXIT_BITMAP_HIGH = 0x0000202D, 211 211 + ENCLS_EXITING_BITMAP = 0x0000202E, 212 212 + ENCLS_EXITING_BITMAP_HIGH = 0x0000202F, 164 213 TSC_MULTIPLIER = 0x00002032, 165 214 TSC_MULTIPLIER_HIGH = 0x00002033, 166 215 GUEST_PHYSICAL_ADDRESS = 0x00002400,

+295

tools/testing/selftests/kvm/x86_64/nested_exceptions_test.c

reviewed

··· 1 1 + // SPDX-License-Identifier: GPL-2.0-only 2 2 + #define _GNU_SOURCE /* for program_invocation_short_name */ 3 3 + 4 4 + #include "test_util.h" 5 5 + #include "kvm_util.h" 6 6 + #include "processor.h" 7 7 + #include "vmx.h" 8 8 + #include "svm_util.h" 9 9 + 10 10 + #define L2_GUEST_STACK_SIZE 256 11 11 + 12 12 + /* 13 13 + * Arbitrary, never shoved into KVM/hardware, just need to avoid conflict with 14 14 + * the "real" exceptions used, #SS/#GP/#DF (12/13/8). 15 15 + */ 16 16 + #define FAKE_TRIPLE_FAULT_VECTOR 0xaa 17 17 + 18 18 + /* Arbitrary 32-bit error code injected by this test. */ 19 19 + #define SS_ERROR_CODE 0xdeadbeef 20 20 + 21 21 + /* 22 22 + * Bit '0' is set on Intel if the exception occurs while delivering a previous 23 23 + * event/exception. AMD's wording is ambiguous, but presumably the bit is set 24 24 + * if the exception occurs while delivering an external event, e.g. NMI or INTR, 25 25 + * but not for exceptions that occur when delivering other exceptions or 26 26 + * software interrupts. 27 27 + * 28 28 + * Note, Intel's name for it, "External event", is misleading and much more 29 29 + * aligned with AMD's behavior, but the SDM is quite clear on its behavior. 30 30 + */ 31 31 + #define ERROR_CODE_EXT_FLAG BIT(0) 32 32 + 33 33 + /* 34 34 + * Bit '1' is set if the fault occurred when looking up a descriptor in the 35 35 + * IDT, which is the case here as the IDT is empty/NULL. 36 36 + */ 37 37 + #define ERROR_CODE_IDT_FLAG BIT(1) 38 38 + 39 39 + /* 40 40 + * The #GP that occurs when vectoring #SS should show the index into the IDT 41 41 + * for #SS, plus have the "IDT flag" set. 42 42 + */ 43 43 + #define GP_ERROR_CODE_AMD ((SS_VECTOR * 8) | ERROR_CODE_IDT_FLAG) 44 44 + #define GP_ERROR_CODE_INTEL ((SS_VECTOR * 8) | ERROR_CODE_IDT_FLAG | ERROR_CODE_EXT_FLAG) 45 45 + 46 46 + /* 47 47 + * Intel and AMD both shove '0' into the error code on #DF, regardless of what 48 48 + * led to the double fault. 49 49 + */ 50 50 + #define DF_ERROR_CODE 0 51 51 + 52 52 + #define INTERCEPT_SS (BIT_ULL(SS_VECTOR)) 53 53 + #define INTERCEPT_SS_DF (INTERCEPT_SS | BIT_ULL(DF_VECTOR)) 54 54 + #define INTERCEPT_SS_GP_DF (INTERCEPT_SS_DF | BIT_ULL(GP_VECTOR)) 55 55 + 56 56 + static void l2_ss_pending_test(void) 57 57 + { 58 58 + GUEST_SYNC(SS_VECTOR); 59 59 + } 60 60 + 61 61 + static void l2_ss_injected_gp_test(void) 62 62 + { 63 63 + GUEST_SYNC(GP_VECTOR); 64 64 + } 65 65 + 66 66 + static void l2_ss_injected_df_test(void) 67 67 + { 68 68 + GUEST_SYNC(DF_VECTOR); 69 69 + } 70 70 + 71 71 + static void l2_ss_injected_tf_test(void) 72 72 + { 73 73 + GUEST_SYNC(FAKE_TRIPLE_FAULT_VECTOR); 74 74 + } 75 75 + 76 76 + static void svm_run_l2(struct svm_test_data *svm, void *l2_code, int vector, 77 77 + uint32_t error_code) 78 78 + { 79 79 + struct vmcb *vmcb = svm->vmcb; 80 80 + struct vmcb_control_area *ctrl = &vmcb->control; 81 81 + 82 82 + vmcb->save.rip = (u64)l2_code; 83 83 + run_guest(vmcb, svm->vmcb_gpa); 84 84 + 85 85 + if (vector == FAKE_TRIPLE_FAULT_VECTOR) 86 86 + return; 87 87 + 88 88 + GUEST_ASSERT_EQ(ctrl->exit_code, (SVM_EXIT_EXCP_BASE + vector)); 89 89 + GUEST_ASSERT_EQ(ctrl->exit_info_1, error_code); 90 90 + } 91 91 + 92 92 + static void l1_svm_code(struct svm_test_data *svm) 93 93 + { 94 94 + struct vmcb_control_area *ctrl = &svm->vmcb->control; 95 95 + unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; 96 96 + 97 97 + generic_svm_setup(svm, NULL, &l2_guest_stack[L2_GUEST_STACK_SIZE]); 98 98 + svm->vmcb->save.idtr.limit = 0; 99 99 + ctrl->intercept |= BIT_ULL(INTERCEPT_SHUTDOWN); 100 100 + 101 101 + ctrl->intercept_exceptions = INTERCEPT_SS_GP_DF; 102 102 + svm_run_l2(svm, l2_ss_pending_test, SS_VECTOR, SS_ERROR_CODE); 103 103 + svm_run_l2(svm, l2_ss_injected_gp_test, GP_VECTOR, GP_ERROR_CODE_AMD); 104 104 + 105 105 + ctrl->intercept_exceptions = INTERCEPT_SS_DF; 106 106 + svm_run_l2(svm, l2_ss_injected_df_test, DF_VECTOR, DF_ERROR_CODE); 107 107 + 108 108 + ctrl->intercept_exceptions = INTERCEPT_SS; 109 109 + svm_run_l2(svm, l2_ss_injected_tf_test, FAKE_TRIPLE_FAULT_VECTOR, 0); 110 110 + GUEST_ASSERT_EQ(ctrl->exit_code, SVM_EXIT_SHUTDOWN); 111 111 + 112 112 + GUEST_DONE(); 113 113 + } 114 114 + 115 115 + static void vmx_run_l2(void *l2_code, int vector, uint32_t error_code) 116 116 + { 117 117 + GUEST_ASSERT(!vmwrite(GUEST_RIP, (u64)l2_code)); 118 118 + 119 119 + GUEST_ASSERT_EQ(vector == SS_VECTOR ? vmlaunch() : vmresume(), 0); 120 120 + 121 121 + if (vector == FAKE_TRIPLE_FAULT_VECTOR) 122 122 + return; 123 123 + 124 124 + GUEST_ASSERT_EQ(vmreadz(VM_EXIT_REASON), EXIT_REASON_EXCEPTION_NMI); 125 125 + GUEST_ASSERT_EQ((vmreadz(VM_EXIT_INTR_INFO) & 0xff), vector); 126 126 + GUEST_ASSERT_EQ(vmreadz(VM_EXIT_INTR_ERROR_CODE), error_code); 127 127 + } 128 128 + 129 129 + static void l1_vmx_code(struct vmx_pages *vmx) 130 130 + { 131 131 + unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; 132 132 + 133 133 + GUEST_ASSERT_EQ(prepare_for_vmx_operation(vmx), true); 134 134 + 135 135 + GUEST_ASSERT_EQ(load_vmcs(vmx), true); 136 136 + 137 137 + prepare_vmcs(vmx, NULL, &l2_guest_stack[L2_GUEST_STACK_SIZE]); 138 138 + GUEST_ASSERT_EQ(vmwrite(GUEST_IDTR_LIMIT, 0), 0); 139 139 + 140 140 + /* 141 141 + * VMX disallows injecting an exception with error_code[31:16] != 0, 142 142 + * and hardware will never generate a VM-Exit with bits 31:16 set. 143 143 + * KVM should likewise truncate the "bad" userspace value. 144 144 + */ 145 145 + GUEST_ASSERT_EQ(vmwrite(EXCEPTION_BITMAP, INTERCEPT_SS_GP_DF), 0); 146 146 + vmx_run_l2(l2_ss_pending_test, SS_VECTOR, (u16)SS_ERROR_CODE); 147 147 + vmx_run_l2(l2_ss_injected_gp_test, GP_VECTOR, GP_ERROR_CODE_INTEL); 148 148 + 149 149 + GUEST_ASSERT_EQ(vmwrite(EXCEPTION_BITMAP, INTERCEPT_SS_DF), 0); 150 150 + vmx_run_l2(l2_ss_injected_df_test, DF_VECTOR, DF_ERROR_CODE); 151 151 + 152 152 + GUEST_ASSERT_EQ(vmwrite(EXCEPTION_BITMAP, INTERCEPT_SS), 0); 153 153 + vmx_run_l2(l2_ss_injected_tf_test, FAKE_TRIPLE_FAULT_VECTOR, 0); 154 154 + GUEST_ASSERT_EQ(vmreadz(VM_EXIT_REASON), EXIT_REASON_TRIPLE_FAULT); 155 155 + 156 156 + GUEST_DONE(); 157 157 + } 158 158 + 159 159 + static void __attribute__((__flatten__)) l1_guest_code(void *test_data) 160 160 + { 161 161 + if (this_cpu_has(X86_FEATURE_SVM)) 162 162 + l1_svm_code(test_data); 163 163 + else 164 164 + l1_vmx_code(test_data); 165 165 + } 166 166 + 167 167 + static void assert_ucall_vector(struct kvm_vcpu *vcpu, int vector) 168 168 + { 169 169 + struct kvm_run *run = vcpu->run; 170 170 + struct ucall uc; 171 171 + 172 172 + TEST_ASSERT(run->exit_reason == KVM_EXIT_IO, 173 173 + "Unexpected exit reason: %u (%s),\n", 174 174 + run->exit_reason, exit_reason_str(run->exit_reason)); 175 175 + 176 176 + switch (get_ucall(vcpu, &uc)) { 177 177 + case UCALL_SYNC: 178 178 + TEST_ASSERT(vector == uc.args[1], 179 179 + "Expected L2 to ask for %d, got %ld", vector, uc.args[1]); 180 180 + break; 181 181 + case UCALL_DONE: 182 182 + TEST_ASSERT(vector == -1, 183 183 + "Expected L2 to ask for %d, L2 says it's done", vector); 184 184 + break; 185 185 + case UCALL_ABORT: 186 186 + TEST_FAIL("%s at %s:%ld (0x%lx != 0x%lx)", 187 187 + (const char *)uc.args[0], __FILE__, uc.args[1], 188 188 + uc.args[2], uc.args[3]); 189 189 + break; 190 190 + default: 191 191 + TEST_FAIL("Expected L2 to ask for %d, got unexpected ucall %lu", vector, uc.cmd); 192 192 + } 193 193 + } 194 194 + 195 195 + static void queue_ss_exception(struct kvm_vcpu *vcpu, bool inject) 196 196 + { 197 197 + struct kvm_vcpu_events events; 198 198 + 199 199 + vcpu_events_get(vcpu, &events); 200 200 + 201 201 + TEST_ASSERT(!events.exception.pending, 202 202 + "Vector %d unexpectedlt pending", events.exception.nr); 203 203 + TEST_ASSERT(!events.exception.injected, 204 204 + "Vector %d unexpectedly injected", events.exception.nr); 205 205 + 206 206 + events.flags = KVM_VCPUEVENT_VALID_PAYLOAD; 207 207 + events.exception.pending = !inject; 208 208 + events.exception.injected = inject; 209 209 + events.exception.nr = SS_VECTOR; 210 210 + events.exception.has_error_code = true; 211 211 + events.exception.error_code = SS_ERROR_CODE; 212 212 + vcpu_events_set(vcpu, &events); 213 213 + } 214 214 + 215 215 + /* 216 216 + * Verify KVM_{G,S}ET_EVENTS play nice with pending vs. injected exceptions 217 217 + * when an exception is being queued for L2. Specifically, verify that KVM 218 218 + * honors L1 exception intercept controls when a #SS is pending/injected, 219 219 + * triggers a #GP on vectoring the #SS, morphs to #DF if #GP isn't intercepted 220 220 + * by L1, and finally causes (nested) SHUTDOWN if #DF isn't intercepted by L1. 221 221 + */ 222 222 + int main(int argc, char *argv[]) 223 223 + { 224 224 + vm_vaddr_t nested_test_data_gva; 225 225 + struct kvm_vcpu_events events; 226 226 + struct kvm_vcpu *vcpu; 227 227 + struct kvm_vm *vm; 228 228 + 229 229 + TEST_REQUIRE(kvm_has_cap(KVM_CAP_EXCEPTION_PAYLOAD)); 230 230 + TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SVM) || kvm_cpu_has(X86_FEATURE_VMX)); 231 231 + 232 232 + vm = vm_create_with_one_vcpu(&vcpu, l1_guest_code); 233 233 + vm_enable_cap(vm, KVM_CAP_EXCEPTION_PAYLOAD, -2ul); 234 234 + 235 235 + if (kvm_cpu_has(X86_FEATURE_SVM)) 236 236 + vcpu_alloc_svm(vm, &nested_test_data_gva); 237 237 + else 238 238 + vcpu_alloc_vmx(vm, &nested_test_data_gva); 239 239 + 240 240 + vcpu_args_set(vcpu, 1, nested_test_data_gva); 241 241 + 242 242 + /* Run L1 => L2. L2 should sync and request #SS. */ 243 243 + vcpu_run(vcpu); 244 244 + assert_ucall_vector(vcpu, SS_VECTOR); 245 245 + 246 246 + /* Pend #SS and request immediate exit. #SS should still be pending. */ 247 247 + queue_ss_exception(vcpu, false); 248 248 + vcpu->run->immediate_exit = true; 249 249 + vcpu_run_complete_io(vcpu); 250 250 + 251 251 + /* Verify the pending events comes back out the same as it went in. */ 252 252 + vcpu_events_get(vcpu, &events); 253 253 + ASSERT_EQ(events.flags & KVM_VCPUEVENT_VALID_PAYLOAD, 254 254 + KVM_VCPUEVENT_VALID_PAYLOAD); 255 255 + ASSERT_EQ(events.exception.pending, true); 256 256 + ASSERT_EQ(events.exception.nr, SS_VECTOR); 257 257 + ASSERT_EQ(events.exception.has_error_code, true); 258 258 + ASSERT_EQ(events.exception.error_code, SS_ERROR_CODE); 259 259 + 260 260 + /* 261 261 + * Run for real with the pending #SS, L1 should get a VM-Exit due to 262 262 + * #SS interception and re-enter L2 to request #GP (via injected #SS). 263 263 + */ 264 264 + vcpu->run->immediate_exit = false; 265 265 + vcpu_run(vcpu); 266 266 + assert_ucall_vector(vcpu, GP_VECTOR); 267 267 + 268 268 + /* 269 269 + * Inject #SS, the #SS should bypass interception and cause #GP, which 270 270 + * L1 should intercept before KVM morphs it to #DF. L1 should then 271 271 + * disable #GP interception and run L2 to request #DF (via #SS => #GP). 272 272 + */ 273 273 + queue_ss_exception(vcpu, true); 274 274 + vcpu_run(vcpu); 275 275 + assert_ucall_vector(vcpu, DF_VECTOR); 276 276 + 277 277 + /* 278 278 + * Inject #SS, the #SS should bypass interception and cause #GP, which 279 279 + * L1 is no longer interception, and so should see a #DF VM-Exit. L1 280 280 + * should then signal that is done. 281 281 + */ 282 282 + queue_ss_exception(vcpu, true); 283 283 + vcpu_run(vcpu); 284 284 + assert_ucall_vector(vcpu, FAKE_TRIPLE_FAULT_VECTOR); 285 285 + 286 286 + /* 287 287 + * Inject #SS yet again. L1 is not intercepting #GP or #DF, and so 288 288 + * should see nested TRIPLE_FAULT / SHUTDOWN. 289 289 + */ 290 290 + queue_ss_exception(vcpu, true); 291 291 + vcpu_run(vcpu); 292 292 + assert_ucall_vector(vcpu, -1); 293 293 + 294 294 + kvm_vm_free(vm); 295 295 + }

+6 -18

tools/testing/selftests/kvm/x86_64/nx_huge_pages_test.c

reviewed

··· 118 118 vm = vm_create(1); 119 119 120 120 if (disable_nx_huge_pages) { 121 121 - /* 122 122 - * Cannot run the test without NX huge pages if the kernel 123 123 - * does not support it. 124 124 - */ 125 125 - if (!kvm_check_cap(KVM_CAP_VM_DISABLE_NX_HUGE_PAGES)) 126 126 - return; 127 127 - 128 121 r = __vm_disable_nx_huge_pages(vm); 129 122 if (reboot_permissions) { 130 123 TEST_ASSERT(!r, "Disabling NX huge pages should succeed if process has reboot permissions"); ··· 241 248 } 242 249 } 243 250 244 244 - if (token != MAGIC_TOKEN) { 245 245 - print_skip("This test must be run with the magic token %d.\n" 246 246 - "This is done by nx_huge_pages_test.sh, which\n" 247 247 - "also handles environment setup for the test.", 248 248 - MAGIC_TOKEN); 249 249 - exit(KSFT_SKIP); 250 250 - } 251 251 + TEST_REQUIRE(kvm_has_cap(KVM_CAP_VM_DISABLE_NX_HUGE_PAGES)); 252 252 + TEST_REQUIRE(reclaim_period_ms > 0); 251 253 252 252 - if (!reclaim_period_ms) { 253 253 - print_skip("The NX reclaim period must be specified and non-zero"); 254 254 - exit(KSFT_SKIP); 255 255 - } 254 254 + __TEST_REQUIRE(token == MAGIC_TOKEN, 255 255 + "This test must be run with the magic token %d.\n" 256 256 + "This is done by nx_huge_pages_test.sh, which\n" 257 257 + "also handles environment setup for the test."); 256 258 257 259 run_test(reclaim_period_ms, false, reboot_permissions); 258 260 run_test(reclaim_period_ms, true, reboot_permissions);

+3 -6

virt/kvm/kvm_main.c

reviewed

··· 3409 3409 int ret = -EINTR; 3410 3410 int idx = srcu_read_lock(&vcpu->kvm->srcu); 3411 3411 3412 3412 - if (kvm_arch_vcpu_runnable(vcpu)) { 3413 3413 - kvm_make_request(KVM_REQ_UNHALT, vcpu); 3412 3412 + if (kvm_arch_vcpu_runnable(vcpu)) 3414 3413 goto out; 3415 3415 - } 3416 3414 if (kvm_cpu_has_pending_timer(vcpu)) 3417 3415 goto out; 3418 3416 if (signal_pending(current)) ··· 5879 5881 5880 5882 r = kvm_async_pf_init(); 5881 5883 if (r) 5882 5882 - goto out_free_5; 5884 5884 + goto out_free_4; 5883 5885 5884 5886 kvm_chardev_ops.owner = module; 5885 5887 ··· 5903 5905 5904 5906 out_unreg: 5905 5907 kvm_async_pf_deinit(); 5906 5906 - out_free_5: 5908 5908 + out_free_4: 5907 5909 for_each_possible_cpu(cpu) 5908 5910 free_cpumask_var(per_cpu(cpu_kick_mask, cpu)); 5909 5909 - out_free_4: 5910 5911 kmem_cache_destroy(kvm_vcpu_cache); 5911 5912 out_free_3: 5912 5913 unregister_reboot_notifier(&kvm_reboot_notifier);