A simple linux driver for vmlaunch

 The idea behind the driver is to demonstrate a real example of how to initialize the Virtual Machine Control Structure(VMCS) and to use Intel VT instructions to launch a virtual machine. The driver launches a guest (virtual machine) with vmlaunch, executes one instruction(that causes a vmexit) and then returns to the host. For the vmlaunch instruction to execute successfully, a lot of cpu state (host and guest state) needs to be initialized all of which is done by this driver. The driver also takes a simple approach in setting up the guest state by making it mirror the host state. This makes the design much simpler - for instance the guest does not need its own CR3, it shares it with the host. Inline assembly is used generously throughout the driver.

The driver source code (64bit)  is located here:

https://github.com/vishmohan/vmlaunch


The sequence leading to the launch of a virtual machine is as follows:

1. Check to make sure the cpu supportsVMX.
2. Check to see if the bios has enabled vmxon in the FEATURE_CONTROL_MSR (msr 0x3A).
3. vmxon.
4. vmptrld
5. Initialize guest vmcs
6. vmlaunch
7. Guest code executes, causes a vmexit
8. Back to the host.

Below is some discussion of the code - I have provided some code snippets for clarity.

The starting point is the function vmxon_init( ) :

1. First execute cpuid (leaf 1). Bit 5 of value returned in ecx indicates support for vmx. If cpuid indicates support for vmx then the code continues with normal execution. If vmx support is not indicated, the code exits.

asm volatile("cpuid\n\t"
       :"=c"(cpuid_ecx)
       :"a"(cpuid_leaf)
       :"%rbx","%rdx");

2. Read the feature_control_msr and look for lock bit (bit0) and vmxon bit(bit2). Both bits must be on - If not exit.

asm volatile("rdmsr\n"
                :"=a"(msr3a_value)
                :"c"(feature_control_msr_addr)
                :"%rdx"
               );


3. Call allocate_vmxon_region(). This allocates a 4k region for vmxon.

static void allocate_vmxon_region(void) {
   vmxon_region = kmalloc(MYPAGE_SIZE,GFP_KERNEL);
}


4. Set up the revision id in the vmxon region. The call to vmxon_setup_revid() sets up the revision-id. This revision id is then copied into the vmxon region. A rdmsr of VMX_BASIC_MSR(msr 0x480) returns the revision id.

vmxon_setup_revid();
memcpy(vmxon_region, &vmx_rev_id, 4); //copy revision id to vmxon region


static void vmxon_setup_revid(void){
   asm volatile ("mov %0, %%rcx\n"
       :
       : "m"(vmx_msr_addr)
       : "memory");
   asm volatile("rdmsr\n");
   asm volatile ("mov %%rax, %0\n"
       :
       :"m"(vmx_rev_id)
       :"memory");
}


5. Turn on the VMXE bit in CR4 (bit13). This enables the virtual machine extensions. Execution of vmxon will #UD without cr4.vmxe on. The function turn_on_vmxe() accomplishes this.

static void turn_on_vmxe(void) {
   asm volatile("movq %cr4, %rax\n"
           "bts $13, %rax\n"
           "movq %rax, %cr4\n"
          );
  }

6. Now execute vmxon by calling the function do_vmxon(). If vmxon fails for any reason, the CF or ZF in rflags will be set - The code checks for this case and restores the flags for debug (using pushfq and popfq below).

static void do_vmxon(void) {
   asm volatile (MY_VMX_VMXON_RAX
                 : : "a"(&vmxon_phy_region), "m"(vmxon_phy_region)
                 : "memory", "cc");
   asm volatile("jbe vmxon_fail\n");
     vmxon_success = 1;
     asm volatile("jmp vmxon_finish\n"
             "vmxon_fail:\n"
              "pushfq\n"
            );
     asm volatile ("popq %0\n"
                   :
                   :"m"(rflags_value)
                   :"memory"
                   );
     vmxon_success = 0;
      asm volatile("vmxon_finish:\n");
}


If vmxon executes successfully, the cpu is now in vmx root operation. INIT# and A20M# are blocked, CR4.VMXE cannot be cleared  and CR0.PG/PE are fixed to 1 in vmx root operation. note: cr4.vmxe can be cleared after the execution of vmxoff which takes the machine out of vmx root operation.


7. Next allocate the vmcs region for the guest and other data structures that are used in vmx non-root operation (iobitmaps, msrbitmaps etc). This is done by allocate_vmcs_region().

static void allocate_vmcs_region(void) {
   vmcs_guest_region  =  kmalloc(MYPAGE_SIZE,GFP_KERNEL);
   io_bitmap_a_region =  kmalloc(MYPAGE_SIZE,GFP_KERNEL);
   io_bitmap_b_region =  kmalloc(MYPAGE_SIZE,GFP_KERNEL);
   msr_bitmap_region  =  kmalloc(MYPAGE_SIZE,GFP_KERNEL);
   virtual_apic_page  =  kmalloc(MYPAGE_SIZE,GFP_KERNEL);

   //Initialize data structures
   memset(vmcs_guest_region, 0, MYPAGE_SIZE);
   memset(io_bitmap_a_region, 0, MYPAGE_SIZE);
   memset(io_bitmap_b_region, 0, MYPAGE_SIZE);
   memset(msr_bitmap_region, 0, MYPAGE_SIZE);
   memset(virtual_apic_page, 0, MYPAGE_SIZE);

}


8. Populate the guest vmcs region with the same revision id as the one used for vmxon region.

memcpy(vmcs_guest_region, &vmx_rev_id, 4); //copy revision id to vmcs region


9. Execute vmptrld. Checks rflags(ZF and CF) to make sure vmptrld executes successfully.

static void do_vmptrld(void) {
    asm volatile (MY_VMX_VMPTRLD_RAX
                : : "a"(&vmcs_phy_region), "m"(vmcs_phy_region)
                        : "cc", "memory");
     asm volatile("jbe vmptrld_fail\n");
     vmptrld_success = 1;
     asm volatile("jmp vmptrld_finish\n"
             "vmptrld_fail:\n"
              "pushfq\n"
            );
     asm volatile ("popq %0\n"
                   :
                   :"m"(rflags_value)
                   :"memory"
                   );
     vmptrld_success = 0;
     asm volatile("vmptrld_finish:\n");
}


10. Now its time to initialize the guest vmcs. This is accomplished by initialize_guest_vmcs(). It is advisable to keep the initialization of  the guest vmcs  consistent with the field encodings in Appendix B, Vol 3c. This will avoid any important fields from skiiping initialization and will save a great deal of headache trying to debug a vmlaunch fail due to invalid guest state.

static void initialize_guest_vmcs(void){
    initialize_16bit_host_guest_state();
    initialize_64bit_control();
    initialize_64bit_host_guest_state();
    initialize_32bit_control();
    initialize_naturalwidth_control();
    initialize_32bit_host_guest_state();
    initialize_naturalwidth_host_guest_state();
}

Note:
  1. The latest Intel manuals have newer fields defined - This code initializes the fields that are supported by the earliest processors to support VT. So for example VPID will not be in the initialization section as all processors do not support VPID.

2. All fields are expanded and written individually rather than iterating through a loop [for ease of debug]. Intel's vmentry checks are detailed and any issue with the initialization here that causes a vmentry fail will be less painful to debug with this code.

initialize_16bit_host_guest_state( ):
This function takes care of initializing the 16 bit guest and host states.

A sample initialization of the host and guest ES selector is given below:

  field = VMX_HOST_ES_SEL;
  field1 = VMX_GUEST_ES_SEL;
  asm ("movw %%es, %%ax\n"
                 :"=a"(value)
        );
   do_vmwrite16(field,value);
   do_vmwrite16(field1,value);

A sample initialization of the host and guest TR selector is given below:

   field = VMX_HOST_TR_SEL;
   field1 = VMX_GUEST_TR_SEL;
   asm("str %%ax\n" : "=a"(value));
   do_vmwrite16(field,value);
   do_vmwrite16(field1,value);

initialize_64bit_control( ):   
This function takes care of initializing the 64 bit controls.

A sample initialization of the IO bitmaps is given below:

   field = VMX_IO_BITMAP_A_FULL;
   io_bitmap_a_phy_region = __pa(io_bitmap_a_region);
   value = io_bitmap_a_phy_region;
   do_vmwrite64(field,value);

   field = VMX_IO_BITMAP_B_FULL;
   io_bitmap_b_phy_region = __pa(io_bitmap_b_region);
   value = io_bitmap_b_phy_region;
   do_vmwrite64(field,value);

 initialize_64bit_host_guest_state( ):
This function takes care of initializing the 64 bit host/guest state.

 field = VMX_VMS_LINK_PTR_FULL;
 value = 0xffffffffffffffffull;
 do_vmwrite64(field,value);
 field = VMX_GUEST_IA32_DEBUGCTL_FULL;
 value = 0;
 do_vmwrite64(field,value);

initialize_32bit_control( ):
32 bit controls are initialized here:

   field = VMX_PIN_VM_EXEC_CONTROLS;
   value = 0x1f ;
   do_vmwrite32(field,value);

Ideally this code should read the pin_based_ctl msr (msr 0x481) to find the allowed 0's and allowed 1's and then initialize this field. The earliest versions of cpu that supported vmx supported external interrupt exiting(bit0) and nmi exiting(bit 3) . The other bits in this field that are set to 1 are the must be 1 bits.  Hence the author sets the value to be 0x1f.


initialize_naturalwidth_control( ):

The CR0 and CR4 guest host mask are initialized below:

   field = VMX_CR0_MASK;
   value = 0;
   do_vmwrite64(field,value);
   field = VMX_CR4_MASK;
   value = 0;
   do_vmwrite64(field,value);

initialize_32bit_host_guest_state( ):

It initializes  32 bit guest/host state and a few of the natural width fields. Here are a few examples:

Initializing the AR bytes is a 2-step process - First find the access rights of the segment using the lar instruction. Then arrange the format of the access rights to match the guest access rights format described in the chapter Virtual Machine Control Structures, vol 3c(Chapter 24, Table 24.2 in the June 2013 manual).

  asm ("movw %%cs, %%ax\n"
         : "=a"(sel_value));
   asm("lar %%eax,%%eax\n" :"=a"(usable_ar) :"a"(sel_value));
   usable_ar = usable_ar>>8;
   usable_ar &= 0xf0ff; //clear bits 11:8

   field = VMX_GUEST_CS_ATTR;
   do_vmwrite32(field,usable_ar);
   value = do_vmread(field);





This code also initializes the GDT base(a natural width field) along with its limit(32 bit field) for convenience. The same process is repeated for IDTR and TR.

  asm("sgdt %0\n" : :"m"(gdtb));
   value = gdtb&0x0ffff;
   gdtb = gdtb>>16; //base

   if((gdtb>>47&0x1)){
     gdtb |= 0xffff000000000000ull;
   }
   field = VMX_GUEST_GDTR_LIMIT;
   do_vmwrite32(field,value);
   field = VMX_GUEST_GDTR_BASE;
   do_vmwrite64(field,gdtb);
   field = VMX_HOST_GDTR_BASE;
   do_vmwrite64(field,gdtb);






initialize_naturalwidth_host_guest_state( ):

Initializes the natural width guest and host states.

As is clear the host and guest cr0,cr4,cr3 are identical.

   field =  VMX_HOST_CR0;
   field1 = VMX_GUEST_CR0;
   asm ("movq %%cr0, %%rax\n"
                 :"=a"(value)
        );
   do_vmwrite64(field,value);
   do_vmwrite64(field1,value);

   field =  VMX_HOST_CR3;
   field1 = VMX_GUEST_CR3;
   asm ("movq %%cr3, %%rax\n"
                 :"=a"(value)
        );
   do_vmwrite64(field,value);
   do_vmwrite64(field1,value);

   field =  VMX_HOST_CR4;
   field1 = VMX_GUEST_CR4;
   asm ("movq %%cr4, %%rax\n"
                 :"=a"(value)
        );
   do_vmwrite64(field,value);
   do_vmwrite64(field1,value);


11. The last piece of initialization is the guest and host rip:

    After a vmexit the cpu transfers control to the host at the label 'vmexit_handler'.
   //host rip
   asm ("movq $0x6c16, %rdx");
   asm ("movq $vmexit_handler, %rax");
   asm ("vmwrite %rax, %rdx");



   After a vmentry the cpu transfers control to the guest at the label 'guest_entry_point'.
   //guest rip
   asm ("movq $0x681e, %rdx");
   asm ("movq $guest_entry_point, %rax");
   asm ("vmwrite %rax, %rdx");

12. Finally the vmlaunch:

   asm volatile (MY_VMX_VMLAUNCH);
   asm volatile("jbe vmexit_handler\n");
   asm volatile("nop\n"); //will never get here

   asm volatile("guest_entry_point:");  ---> after vmlaunch the code gets here:
   asm volatile(MY_VMX_VMCALL); ---> vmcall causes a vmexit with exit reason=0x12
   asm volatile("ud2\n"); //will never get here
   asm volatile("vmexit_handler:\n");   ---> after vmexit code starts executing here:


13. If the launch completes successfully, then the guest executes vmcall and vmexits.

14. After the vmexit, the vmexit_handler takes over. Reads the exit_reason and prints a message.

   asm volatile("vmexit_handler:\n");
   field_1 = VMX_EXIT_REASON;
   value_1 = do_vmread(field_1);
   asm volatile("sti\n");
  


15. When the driver module is removed (rmmod .ko) , vmxon_exit( ) function is called. It does the following: (a) do vmxoff (b) turn off CR4.VMXE (c) deallocate vmcs region and other data structures. (d) deallocate the vmxon region.


static void vmxon_exit(void) {
   if(vmxon_success==1) {
         do_vmxoff();
     vmxon_success = 0;
   }
   save_registers();
   turn_off_vmxe();
   restore_registers();
   deallocate_vmcs_region();
   deallocate_vmxon_region();
}


The dealloc just frees all the allocated memory regions:



static void deallocate_vmxon_region(void) {
   if(vmxon_region){
       kfree(vmxon_region);
   }
}

VMX and SMM – Dual monitor mode

The Dual monitor mode involves two monitors:  Executive monitor and the SMM monitor. The Executive monitor is analogous to the vmx-root hypervisor that exists outside of SMM. The SMM monitor is a special hypervisor that operates only in SMM. Under dual monitor treatment SMI’s cause vmexits and this information is recorded in a separate vmcs called SMM transfer vmcs. This enables the SMM monitor to assume control of vmexits caused by SMI# assertion.

Normal  vmx transitions (without dual monitor):

1.        vmxon executed by the executive monitor.

2.       vmptrld is executed.  The guest vmcs is then initialized via vmwrites.

3.       vmlaunch is executed. The guest virtual machine is launched.

4.       A vmexit from the guest traps back to the executive monitor. The executive monitor reads the exit_reason, exit_qual and a bunch of vmcs fields to extract more details on the vmexit.  After handling the vmexit , the executive monitor resumes the guest by executing vmresume.

5.       If there is a SMI# in the guest, vmx is turned off. The processor enters SMM. Upon a RSM, the processor takes us back to vmx guest.

Dual Monitor vmx transitions:

0.       Enable Dual Monitor. [see section on enabling dual  monitor below].

1.       A. vmxon executed by the executive monitor.  

B. Dual monitor treatment is activated [see section on activating dual  monitor below].

2.       vmptrld is executed.  The guest vmcs is then initialized via vmwrites.

3.       vmlaunch is executed. The guest virtual machine is launched.

4.       A vmexit from the guest traps back to the executive monitor. The executive monitor reads the exit_reason, exit_qual and a bunch of vmcs fields to extract more details on the vmexit.  After handling the vmexit , the executive monitor resumes the guest by executing vmresume.

5.       If there is a SMI# in the guest, then a SMM VMexit occurs. Control is transferred to the SMM monitor (instead of executive monitor).  The SMM monitor now handles the SMM vmexit by reading relevant fields in the SMM transfer vmcs.  After handling the SMM vmexit, it resumes the guest by executing a vmresume.

Steps 1A, 2,3 and 4 are identical for normal and dual-monitor vmx transitions.

The only difference between Normal vmx transitions and the Dual monitor vmx transitions is in the handling of SMI. In the dual monitor case, there is a new vmexit (SMM VMexit) that traps to the SMM monitor. All other vmexits continue to trap to the executive monitor.

When the machine is in the SMM monitor, it is considered to be in SMM. A  SMM VMexit is one that begins outside of SMM and ends in SMM. This means that SMM VMexits are also accompanied by the SMI_ACK special cycle. Similarly, a vmresume from a SMM monitor that resumes the guest is also accompanied by a SMI_ACK special cycle(since this vmresume takes the machine from SMM to outside of SMM).

The next two sections cover step 0 and 1B of Dual Monitor vmx transitions.

Enabling Dual Monitor Treatment:

Intel provides a new msr (msr 0x9b – SMM_MONITOR_CTL msr) for this.  Bit 0 of this msr is the valid bit. Bits 31:12 is the physical address (4K aligned) of the monitor segment (also called MSEG) that initializes the SMM transfer vmcs. This msr can be written only in SMM mode.  Here is a sample code:

mov ecx, 0x9B

mov eax, 0x00009001

xor edx, edx ; bits 63:32 are reserved. Clear edx

wrmsr

rsm  ; get out of SMM

The valid bit is set to 1.  Bits 31:12 = 0x9 – This implies that the physical address of the MSEG segment is 0x9000. Note that the above code snippet must run in SMM (SMI handlers that are dual-monitor aware may add the above code to initialize the MSEG). 

A sampleMSEG header looks like the one shown below.  In our example, the header is at physical address 0x9000 (what we wrote in msr 0x9b).

revision_identifier              dd 0

smm_monitor_features      dd 0

gdtr_limit                           dd 

gdtr_baseoffset                 dd

cs_sel                               dd

eip_offset                         dd

esp_offset                        dd

cr3_offset                        dd

The format of the MSEG_HEADER above matches the one described in Table 26.10(vol 3b, System Management Mode, chapter 26).  Note: Depending on the version of the Intel manual, table numbers may vary – but it will be found in the SMM chapter regardless of the manual version.

Activating Dual Monitor Treatment:

After enabling the dual-monitor treatment, software can activate it by executing vmcall instruction.  This execution of vmcall is in VMX_ROOT mode.  (Execution of vmcall in vmx_non_root mode always causes a vmexit. Vmcall execution in vmx_root mode thus has a special meaning – to activate dual monitor treatment).  Here is a sample code that accomplishes this:

; enable cr4.vmxe

mov eax, 0x00002010

mov cr4, eax

; do vmxon

VMXON [vmxon_ptr]

jbe fail

;load smm transfer vmcs pointer

vmclear [vmcs_smm_ptr]

jbe fail

vmptrld [vmcs_smm_ptr]

jbe fail

; now do vmcall

vmcall

When the processor executes vmcall instruction in vmx_root mode, internally does the following:

vmcall_flow:

if (vmx_root) {

       If(dual_monitor_active) {

          Perform SMM_VMEXIT;

       } else if (SMM_MONITOR_CTL_VALID){

         Activate_Dual_Monitor_SMM_VMexit;

      }

}

In the above code snippet,   SMM_MONITOR_CTL_VALID  comes directly from bit 0 of the SMM_MONITOR_CTL_MSR (msr 0x9b).  If these conditions are not met, vmcall fails. Also note that there are additional checks vmcall performs on the SMM transfer vmcs which are not discussed here. For those details reading the manual vol 3b is recommended.  [Also looking at the vmcall pseudo-code provided in Vol 2b (under vmx instructions) is recommended].

In the process of ‘Activate_Dual_Monitor_SMM_VMexit’, the processor does the following:

a.      

En  Enters SMM (issues a SMI_ACK bus cycle)

b.      Reads the MSEG revision identifier (offset 0). If it does not match the revision identifier supported by the processor then VMCALL fails. [The MSEG revision id supported by the processor is obtained by a rdmsr of IA32_VMX_MISC_MSR (msr 0x485 – bits 63:32).

c.       Reads the MSEG features field and performs checks on that field.

d.      After all checks pass, the processor starts executing instructions from the RIP indicated in the eip_offset field of the MSEG.

Sample MSEG code:

mov eax, 0x11ff

mov ebx, VMX_ENTRY_CONTROLS

vmwrite ebx, eax

mov eax, 0x008B

mov ebx, VMX_GUEST_TR_ATTR

vmwrite ebx, eax

 

mov ebx, VMX_EXIT_INSTR_LEN

vmread eax, ebx

mov ebx, VMX_GUEST_RIP

vmread ebx, ebx

add eax, ebx

mov ebx, VMX_GUEST_RIP

vmwrite ebx, eax

vmlaunch

The code above does only the bare minimum stuff (In reality, it will initialize the entire SMM_VMCS) – It initializes the entry_controls, updates the guest_rip and does a vmlaunch.  Where does this vmlaunch take the machine?   The answer to VMX_ROOT.  This is a special type of VMentry that takes the machine back to VMX_ROOT – Intel calls this VMentry as a ‘VMentry that returns from SMM’.   Remember that the machine performed a SMM_Vmexit when VMCALL was executed in VMX_ROOT mode – So this VMLAUNCH in the MSEG code takes us back to VMX_ROOT.  At this point, we are in the executive-monitor. This completes step 1B in the dual monitor flow.

VMX and System Management Mode - Part 1

There are two different modes of operation of VMX within SMM:
1.Normal Mode
2.Dual monitor mode

Normal Mode:

Under Normal mode, a SMI# assertion causes the processor to turn-off vmx and enter into SMM. Upon a RSM, the processor automatically enables VMX if it was either in VMX-ROOT or VMX-GUEST prior to the SMI#. Since the processor turns off VMX, it means that CR4.VMXE is treated as reserved bit and must be 0 during RSM.

Algorithmically,

if(smi){
if(vmx_root or vmx_guest){
save cr4.vmxe internally;
if(vmx_root) internal_state = vmx_root;
if(vmx_guest) internal_state = vmx_guest;
turn_off_vmx;
}
save cr4 to smm_ram;
}

during rsm:

if(rsm){
read cr4_val from smm_ram;
if(cr4_val.vmxe==1) jump_to_shutdown;
retrieve internal cr4.vmxe;
cr4 <- cr4_val | (cr4.vmxe<<13);
read internal_state;
if(internal_state==vmx_root) put_cpu_in_vmx_root;
if(internal_state==vmx_guest) put_cpu_in_vmx_guest;
}

Notice the jump_to_shutdown during RSM. Since the processor saves CR4.VMXE internally during SMM, the value saved in SMRAM for CR4.VMXE is always 0. During RSM, the CR4 value is first loaded from SMRAM and bit 13 is checked . It must be 0 – If not the cpu will jump to shutdown. The processor then retrieves the value of VMXE from an internal register and updates CR4 with this value. The state of the processor (whether it was in vmx-root or vmx-guest or normal ia32 operation) is also retrieved and the cpu is put in that state after the completion of RSM.

This process is the default treatment of SMIs with VMX.

Notes on System Management Mode [SMM]

SMM:
SMM [System Management Mode] is an operating mode entered through the assertion of the SMI# pin. The processor upon detecting a SMI# saves the processor state in SMRAM [The base address of the SMRAM is obtained form an internal SMBASE register. The reset value of SMBASE register is 0x30000]. The processor saves several architectural values into the SMRAM (like the values of CR0, CR3, CR4 etc) when it enters SMM. To exit out of SMM , software executes a RSM(resume) instruction. During the RSM instruction, the processor reloads the architectural state from SMRAM and gets back to the state it was prior to the SMI#.
Here is a loosely defined algorithm for entering and exiting SMM:
1.Processor is executing a task (say T).
2.SMI# is detected by the processor.
3.Processor saves all information pertaining to task T in the SMRAM. It issues SMI_ENTER_ACK bus cycle and enters SMM.
4.Processor executes code from the SMM space[starting at address 0x38000]
5.When it executes the RSM instruction, the processor reloads the prior architectural state from SMRAM and then issues SMI_EXIT_ACK bus cycle and exits SMM.
6.Processor resumes executing the task T.

During Step 5, while the processor loads architectural state, it performs few checks on the state being loaded:
1.It checks the reserved bits of CR4.
2.It checks CR0 register for illegal combinations. For eg: CR0.PG=1 and CR0.PE=0 or CR0.CD=0 and and NW=1 .
If the checks above fail, then the processor enters shutdown.
[Note: there may be additional checks performed. CR0 and CR4 values in SMRAM should be left untouched by the SMM handler. These checks exist to make sure that the handler does not modify values to put the processor in an incompatible state after the execution of RSM].

Software injection into V86 guest with interrupt redirection - What must be the IDT VECTOR INFO?

The following observation is made while launching a V86 guest on Intel Merom. As part of vmlaunch or vmresume, a software interrupt is injected into the V86 guest(The entry interruption info field reads 0x800004vv where vv is the vector number). The V86 virtual machine has:

a.  GUEST_RFLAGS.VM = 1 (indicating the guest is in V86 mode).
b. CR4.VME=1 (enables interrupt redirection provided the redirection bitmap says so in TSS).
c. The exception_bitmap in the guest is configured to vmexit on a #PF.

At the end of vmlaunch, the software interrupt is injected. The guest is in V86 mode and has CR4.VME=1. The cpu consults the TSS to read the interrupt redirection bitmap. The TSS page is not present and the cpu takes a #PF. The guest is configured to vmexit on #PF. After the vmexit, use vmread to read the following vmcs fields:
a. Exit reason (reads 0)
b. Exit Interruption Info (0x80000B0E - indicates a #PF)
c. IDT Vector  Info (reads 0)
d. Exit Qualification (0x - address that caused #PF).

Something interesting in the above results is the value of idt-vector-info. The idt-vector-info must have read 0x800004vv(vv=vector), since the vmexit was encountered in the process of injecting an event. This behavior appears to violate what is stated in  vol3b.

Injecting software interrupt into a V86 guest

To inject an interrupt or exception into a guest a hypervisor uses the ENTRY_INTERRUPTION_INFO field in the vmcs. For eg: if there is a need to inject a #GP exception into the guest as part of vmentry, the entry_interruption_info field would look like this: 0x80000B0D.

1 . Bits 7:0 of this field represent the vector (0x0D - which is vector 13)
2. Bits 10:8 indicate the type(in this case type = 0x3 which is a hardware-exception).
3. Bit   11 is the error-code valid bit which is true in the example above.
4. Bit 31 is the valid bit for  ENTRY_INTERRUPTION_INFO field.

To inject a software interrupt (say vector 0x8) hypervisor would program entry_interruption_info field as given under: 0x80000408 (type=0x4 and vector=0x8). If the guest is in V86 mode (GUEST_RFLAGS[VM]=1) , the processor behaves according to Table 15.2, Intel SDM, vol 3A .


Given below is a summary of the processor behavior during normal software-interrupt execution in V86 and during an event injection into a V86 guest:

1. EFLAGS.VM = 1 , CR4.VME=1, EFLAGS.IOPL=3
=> In this case the bit in the redirection bitmap of the TSS is consulted.
=> if bit in the redirection bitmap=0, the software interrupt is redirected to x86 style handler.
=> if bit in the redirection bitmap=1, the software interrupt is redirected to protected-mode handler.

2.  EFLAGS.VM = 1 , CR4.VME=1, EFLAGS.IOPL<3
=> In this case the bit in the redirection bitmap of the TSS is consulted.

=> if bit in the redirection bitmap=0, the software interrupt is redirected to x86 style handler. Notice that this is the same behavior as with EFLAGS.IOPL=3. The difference is in the value of eflags pushed on the stack. Here the IOPL of the eflags image is forced to 3 and the value of VIF is copied to IF.

Normal behavior:  if bit in the redirection bitmap=1, the interrupt is directed to a #GP handler.
During VMX event injection:  if bit in the redirection bitmap = 1, the processor will *NOT*  #GP due to IOPL < CPL.

3. EFLAGS.VM = 1 , CR4.VME=0, EFLAGS.IOPL=3
=> Normal behavior: Interrupt directed to a protected mode handler (No #GP).
=> During event injection: Same as above.

4. EFLAGS.VM = 1 , CR4.VME=0, EFLAGS.IOPL<3
=> Normal behavior: Interrupt directed to a #GP handler .
=> During Event Injection:  No #GP can occur due to IOPL< CPL. The behavior will be the same as with IOPL=3.

Summary:
From the above discussion is there will be no #GP due to IOPL < CPL during the injection of a software interrupt into a V86 guest. If the hypervisor wants this #GP to occur, it needs to inject a #GP directly into the guest instead of a software-interrupt.This can be achieved by programming the entry_interruption_info field to 0x80000B0D.

VMEXIT on INVLPG

A boundary case observed on Intel Merom:

(a) The virtual-machine is configured to vmexit on INVLPG(bit 9 of the PROCESSOR_EXECUTION_CONTROLS is 1).

(b) The virtual-machine has GS BASE = 0xFFFF8000_00000000

(c) Virtual machine executes: invlpg [gs:0-1]

(d) Execution of invlpg causes vmexit.

(e) The address of invlpg is recorded in exit-qualification. Upon a vmread of EXIT_QUALIFICATION the value obtained is:
=> FFFF7FFF_FFFFFFFF


Notice that the value recorded is a non-canonical address ie; address[63:48] != address[47]. This is the only case i have encountered where a non-canonical address shows up on the exit-qualification.

The only explanation I can come up with for this behavior is that : INVLPG unlike other instructions does not fault in 64-bit mode with a non-canonical operand. According to the instruction spec, INVLPG morphs into a NOP for such cases.

When a vmexit handler for INVLPG is written, this case must be taken into consideration(ie; a non-canonical address might show up in the exit-qualification field).

A full blown initialization of VMCS - Assembly code

The code below will outline the general steps prior to executing a VMLAUNCH or VMRESUME.
Prior to looking at the assembly code, here is a step-by-step description of what is being done:

The reader must know that:
A)this code will run only in ring0.
B)that paging is already enabled in CR0(bit 31).

(1) First Enable VMXE (bit 13) in CR4. Make sure that processor supports VMX by executing CPUID(leaf 1, ecx[5]).

(2) Intialize revision-id(msr 0x480,31:0) in the vmxon region and in the guest-vmcs region.

(3) Execute VMXON with the pointer to vmxon region. In some cases, if BIOS has not enabled bits 0, 2 of FEATURE_CONTROL_MSR (msr 0x3a) this will fail.

(4) Execute VMCLEAR with the pointer to the guest-vmcs region.

(5) Execute VMPTRLD with the pointer to the guest-vmcs region.

(6) Now initialize the guest-vmcs:
(a) First initialize the vmx controls. These include the following controls:
1. PIN_BASED
2. PROC_BASED
3. ENTRY_CONTROLS
4. EXIT_CONTROLS

(b) Next initialize the host-state and guest-state.

(c) Now do vmlaunch. If VMLAUNCH is successful, then the processor will start executing code
from the GUEST_CS:GUEST_RIP value specified in the VMCS.

Here comes the code:
////////////////////////////////////////////////////
mov eax, cr4
bts eax, 13
mov cr4, eax

mov ecx, 0x480
rdmsr
mov edx, [vmxon-ptr]
mov [edx], eax
mov edx, [guest-ptr]
mov [edx], eax

VMXON [vmxon-ptr]
jbe fail

vmclear [guest-ptr]
jbe fail

vmptrld [guest-ptr]
jbe fail

call initialize_vmx_controls
call initialize_vmx_host_guest_state
call do_vmlaunch

;ideally a hypervisor would read the VMX-MSRS
; to determine what values to write.
initialize_vmx_controls:
mov ebx, ENTRY_CONTROLS ;0x4012
mov eax, 0x11ff
vmwrite ebx, eax
mov ebx, PIN_CONTROLS; 0x4000
mov eax, 0x1f
vmwrite ebx, eax
mov ebx, PROC_CONTROLS ; 0x4002
mov eax, 0x0401E9F2
vmwrite ebx, eax
mov ebx, EXIT_CONTROLS ; 0x400C
mov eax, 0x36dff
vmwrite ebx, eax
ret

initialize_vmx_host_guest_state:
mov eax, cr3
mov ebx, HOST_CR3 ;0x6C02
mov edx, GUEST_CR3 ;0x6802
VMWRITE EBX,EAX
mov eax, pdebase_guest
VMWRITE EDX,EAX

mov ebx, HOST_RSP ;0x6c14
mov eax, tos ;top-of-stack
vmwrite ebx, eax

mov ebx, HOST_CR0 ; 0x6C00
mov eax, cr0
vmwrite ebx, eax
mov ebx, GUEST_CR0 ;0x6800
vmwrite ebx, eax
mov ebx, HOST_CR4 ; 0x6C04
mov eax, cr4
vmwrite ebx, eax
mov ebx, GUEST_CR4; 0x6804
vmwrite ebx, eax
mov ebx, HOST_CS_SEL ; 0x0c02
mov eax, cs
vmwrite ebx, eax
mov ebx,HOST_DS_SEL ; 0x0c06
mov eax, ds
vmwrite ebx, eax
mov ebx, HOST_SS_SEL ; 0x00000c04
mov eax, 0x18
vmwrite ebx, eax
mov ebx, HOST_TR_SEL; 0x00000c0c
mov eax, 0x18
vmwrite ebx, eax
mov ebx, GUEST_TR_SEL ;0x0000080e
mov eax, 0x18
vmwrite ebx, eax
mov ebx, GUEST_TR_ATTR ;0x00004822
mov eax, 0x8b
vmwrite ebx, eax
mov ebx, GUEST_TR_LIMIT ;0x0000480e
mov eax, 0xff
vmwrite ebx, eax
mov ebx, GUEST_LDTR_ATTR ;0x00004820
mov eax, 0x00010000
vmwrite ebx, eax
mov ebx, GUEST_SS_ATTR ;0x00004818
mov eax, 0xc093
vmwrite ebx, eax
mov ebx, GUEST_DS_ATTR ;0x0000481a
mov eax, 0xc093
vmwrite ebx, eax
mov ebx, GUEST_ES_ATTR ;0x00004814
mov eax, 0xc093
vmwrite ebx, eax
mov ebx, GUEST_FS_ATTR ;0x0000481c
mov eax, 0xc093
vmwrite ebx, eax
mov ebx, GUEST_GS_ATTR ;0x0000481e
mov eax, 0xc093
vmwrite ebx, eax
mov ebx, GUEST_SS_LIMIT ;0x00004804
mov eax, 0xffffffff
vmwrite ebx, eax
mov ebx, GUEST_DS_LIMIT ;0x00004806
vmwrite ebx, eax
mov ebx, GUEST_ES_LIMIT ;0x00004800
vmwrite ebx, eax
mov ebx, GUEST_FS_LIMIT ;0x00004808
vmwrite ebx, eax
mov ebx, GUEST_GS_LIMIT ;0x0000480a
vmwrite ebx, eax
mov ebx, LINK_PTR_FULL ;0x00002800
vmwrite ebx, eax
mov ebx, VMS_LINK_PTR_HIGH ;0x00002801
vmwrite ebx, eax
mov ebx, GUEST_GDTR_BASE ;0x00006816
mov eax, gdt32t
vmwrite ebx, eax
mov ebx, HOST_GDTR_BASE ;0x00006c0c
vmwrite ebx, eax
ov ebx, GUEST_CS_LIMIT ;0x00004802
mov eax, 0xffffffff
vmwrite ebx, eax
mov ebx, GUEST_CS_ATTR ;0x00004816
mov eax, 0xc09b
vmwrite ebx, eax
mov ebx, GUEST_RSP ;0x0000681c
mov eax, tos
vmwrite ebx, eax
mov ebx, GUEST_IDTR_BASE ;0x00006818
mov eax, idt32t
vmwrite ebx, eax
mov ebx, HOST_IDTR_BASE ;0x00006c0e
vmwrite ebx, eax
mov ebx, GUEST_CS_SEL ;0x00000802
mov eax, guest_sel
vmwrite ebx, eax
mov ebx, GUEST_CS_BASE ;0x00006808
mov eax, guest_base
vmwrite ebx, eax
mov ebx, GUEST_RIP ;0x0000681e
mov eax, 0
vmwrite ebx, eax
mov ebx, HOST_RIP ;0x00006c16
mov eax, after_vmexit
vmwrite ebx, eax
mov ebx, GUEST_RFLAGS ;0x00006820
mov eax, 2
vmwrite ebx, eax
mov ebx, EXCEPTION_BITMAP ;0x4004
mov eax,0xdeadfeef
vmwrite ebx, eax
ret

do_vmlaunch:
VMLAUNCH

after_vmexit:
;read EXIT_REASON and figure out what caused the vmexit.

///////////////////////////////////////////////////////////////

The HOST_RIP is where control is transferred after a vmexit. The hypervisor can determine the appropriate course of action by reading the vmexit fields from the vmcs.

VMCS Guest State Area

The VMCS has an area for guest register state and guest non-register state. The guest register state contains fields for control registers(CR0,CR3,CR4), segment registers(cs,ds etc) , segment selectors, segment attributes(also called ARBytes),segment-limits, debug-register(dr7) etc.

The guest non-register state has the following fields:

(a) VMCS GUEST ACTIVITY STATE: This is a 32 bit field that describes the state of the guest. Intel manual defines only the first four bits in this field. The others are marked as reserved. The four bits are as given under:

(i) Bit 0 - If this bit is 1 , the guest is in Active State .

(ii) Bit 1 - If this bit is 1 , the guest is in HLT state.

(iii) Bit 2 - If this bit is 1 , the guest is in Shutdown state.

(iv) Bit 3 - If this bit is 1 , the guest is in Wait-for-SIPI state.

Here is a scenario how the vmcs might indicate a guest in halt state:

The guest executes HLT and the execution of HLT does not cause a vmexit(assume proc_based_ctl[hlt] is 0). At this point the guest is in HLT state. Now an interrupt arrives and the processing of the interrupt causes a vmexit(assume pin_based_ctl for intr is 1). The vmexit causes the processor to update the GUEST ACTIVITY STATE in the VMCS as HLT. A subsequent VMRESUME will read this field from the vmcs and launch the guest in HLT state.

(b) VMCS GUEST INTR INFO : The INTR INFO describes the interruptibility state of the guest. Four bits are defined in this field.

(i) Bit 0 - Guest has STI blocking active

(ii) Bit 1 - Guest has MOV-SS/POP-SS active

(iii) Bit 2 - Blocking by SMI

(iv) Bit 3 - Blocking by NMI

Here is a scenario where the vmcs might indicate STI blocking:

Guest executes STI. Following STI, guest executes HLT which vmexits (assume proc_based_ctl[hlt] is 1). The vmexit due to hlt will cause the processor to update the interruptibility-info with STI blocking. A subsequent VMRESUME will put the guest back in STI blocking state. In this case, during a vmresume, the processor also checks the EFLAGS.IF = 1 and will fail VMEntry if it detects the following condition:

if(sti_blocking==TRUE && EFLAGS.IF==0) {

fail_vmentry();

}

(c)Pending Debug Exceptions Field: This field in the VMCS indicates if there are any debug exceptions that are pending in the guest. The meaningful bits in this field are:

Bit 12 -> Enabled Breakpoint

Bit 14 -> Single Step

Bits 3-0 -> Correspond to B0,B1,B2,B3 (meaningful only if bit 12 is 1).

Here is a scenario how the guest may end up with a single-step exception pending during a vmexit:

pushfd ; push flags

or dword [esp], 0x100 ; set eflags.TF

popfd ; pop flags now eflags.TF=1

mov ss, ax ; will delay recognition of single step until the end of the next instruction

vmcall ; cause vmexit

In the above code, at the time of vmexit , single-step is pending (it is delayed because of mov-ss blocking). This vmexit will cause the processor to record a single-step pending in the debug-exceptions field of the vmcs. Incidentally, the above vmexit will also cause the processor to record mov-ss blocking in the interruptibility field.

VMX EXECUTION_CONTROLS

Two types of execution controls are defined:
a) PIN_BASED execution controls
b) PROC_BASED execution controls


PIN_BASED Controls:
The vmcs encoding for this field is 0x4000. There are 2 bits in this 32-bit field that are interesting:
Bit 0 – External Interrupt Exiting
Bit 3 – NMI Exiting
After launching a vmx-guest, when an external interrupt is received in the guest and Bit0 is 1 then there is a vmexit due to external interrupt.
Bit3 setting controls the behavior of the processor in response to a NMI while running as vmx-guest. If bit3==1 and a nmi is received in the guest a vmexit occurs.
The other bits are reserved. The settings of the reserved bits(0 or 1) are obtained by reading msr 0x481. To initialize this field in the vmcs:

xor eax,eax
xor edx, edx
mov ecx, 0x481
rdmsr
or eax, edx ; it has the valid vector to be written into the vmcs.
bts eax, 0 ; set bit0 to vmexit due to interrupts
bts eax, 3; nmi exiting bit = 1
mov ebx, 0x4000 ; encoding for entry controls
vmwrite ebx, eax


PROC_BASED Controls:
The vmcs encoding for this field is 0x4002. It is a 32 bit field that determines the behavior of the processor when certain instructions are executed in the vmx-guest.
For eg:
Bit7 of this vector controls the processor behavior upon execution of the HLT instruction in vmx-guest. If 1 , execution of HLT will cause a vmexit. If 0, the instruction will be executed normally without any vmexit.

Similarly bit9 controls the behavior of the processor on INVLPG, bit19 controls the behavior on mov-to-cr8 and bit20 controls the behavior on mov-from-cr8 etc.

The bit positions are described below:

INTRWINDOW 2
TSCOFFSET 3
HLT 7
INVLPG 9
MWAIT 10
RDPMC 11
RDTSC 12
CR8LOAD 19
CR8STORE 20
TPRSHADOW 21
MOVDR 23
IOUNCOND 24
IOBITMAP 25
MSRBITMAP 28
MONITOR 29
PAUSE 30


MSR 0x482 indicates the allowed-0 and allowed-1 settings of these controls.

Note: Newer processors have additional bits defined for these controls. For more details see PRM Vol 3b.

VMX Exit Control fields

EXIT_CONTROLS:
This field is used by the processor during a vmexit. This is a 32-bit field (just like the entry controls) but only 2 bits are defined:
Bit 9 – Host address space – This value is loaded to EFER.LME and CS.L on a vmexit.
Bit 15 – Acknowledge Interrupt on Exit – If there is a vmexit due to interrupt this bit determines whether the interrupt is acknowledged or not. The interrupt vector is recorded in the vmcs.
All other bits are reserved. They are either 0s or 1s as determined by the EXIT_CTLS_MSR (msr 0x483).

EXIT_CONTROL FOR MSR:
This is exactly similar to ENTRY_CONTROL FOR MSR. The only difference is in the vmcs encodings . They are tabulated below:

EXIT_MSR_STORE_ADDR EQU 0x2006
EXIT_MSR_STORE_COUNT EQU 0x400E
The Guest MSRS are saved in the MSR store area during a vmexit. On a subsequent VMEntry, these MSRS will be loaded from the same area.

EXIT_MSR_LOAD_ADDR EQU 0x2008
EXIT_MSR_LOAD_COUNT EQU 0x4010
The Host MSRs are loaded from the physical address specified in EXIT_MSR_LOAD_ADDR.
The format of the msr-load/msr-store areas is exactly similar to the msr-load area that is used for vmentry.

VMX Entry Control fields

VMX Control fields

Control fields are of 3 types:
a) Entry Control fields
b) Exit Control fields
c) Execution Control fields.

Entry Control fields:
Used during VMEntry (Vmentry is the process by which CPU transitions from HOST state to the Guest state).

VMENTRY_CONTROLS:
This is a 32-bit field that sets up some critical information that is used by the processor during vmentry. Most of the fields in this 32-bit field is reserved.
Among the bits that are defined, the following 3 are interesting:
bit 9 - Guest is in long mode
bit 10 - Guest is in SMM
bit 11 - Deactivate Dual monitor treatment
For normal vmentries, bit 10 and bit 11 are always 0. Bit 9 can be 0 or 1 depending on whether the guest is in long-mode or protected mode.

Note:

(A) If a guest will be in compatibility-mode , bit 9 must be set to 1. When the processor loads state during Vmentry, if GUEST_CS.L bit is 0 and bit 9 of entry_control is 1 , then the guest will be in compatibility-mode after vmentry.

(B) During Vmentry the value of bit 9 is copied into EFER.LME. Since CR0.PG is fixed to 1, the value also propagates to EFER.LMA.

Sample code to set up entry controls:

To set up this field, software should consult msr 0x484 and extract the allowed-0 and allowed-1 settings of this field.
xor eax,eax
xor edx, edx
mov ecx, 0x484
rdmsr
or eax, edx ; it has the valid vector to be written into the vmcs.
btr eax, 10 ; clear the SMM bit
btr eax, 11 ; clear the deactivate dual monitor bit
mov rbx, 0x4012 ; encoding for entry controls
vmwrite rbx, rax


VMENTRY_CONTROL_MSR:
This field is used when msrs are to be loaded as part of vmentry. This is sometimes required for the hypervisor to present the guest with a msr value different than the host-value.

Sample code:
%define MSR_LOAD_ADDR EQU 0x200a
%define MSR_LOAD_COUNT EQU 0x4014
mov rax,
mov rbx, MSR_LOAD_ADDR
vmwrite rbx, rax
mov rax, 1
mov rbx, MSR_LOAD_COUNT
vmwrite rbx, rax
my_msr_address:
dd
dd 0
dd msr_data_lo
dd msr_data_hi
Note:
my_msr_address is the Physical Address of the msr-load area in memory.
The layout of my_msr_address must match the layout described above. my_msr_address must be 16B aligned.


VMENTRY_CONTROL_EVENT_INJECTION:
This field is used when delivering an event/exception to the guest during vmentry. For eg: If the hypervisor wants the control to be transferred to the guest_GP handler, it would do the following:

mov rax, 0x4016; vmcs encoding
mov rbx, 0x80000B0D ; bits 10:8 = 3 -> HW exception, bits 7:0 = 0x0d (vector 13)
vmwrite rax, rbx


Vol 3b has more details on this vmcs field. The hypervisor might use this technique to handle a vmexit from the guest due to an exception.

Initializing the VMCS

Software initializes the vmcs by using the vmwrite instruction. It can read the value from the vmcs using the vmread instruction. The VMCS is divided into four areas:

(a) Host Area
(b) Guest Area
(c) VMX Control fields
(d) VMX Exit Information fields

Each VMCS field is identified by an encoding which is used by the processor to write into the appropriate place in the vmcs.

Host Area:

Host selector fields:
------------------------
Host ES selector 0xC00
Host CS selector 0xC02
Host SS selector 0xC04
Host DS selector 0xC06
Host FS selector 0xC08
Host GS selector 0xC0A
Host TR selector 0xC0C
As an example, say the hypervisor wants to initialize the Task register selector with a value of 0x18:
mov rax, 0x0C0C
mov rbx, 0x18
vmwrite rbx, rax

To read a value from the vmcs, vmread is used:
mov rax, 0x0C0C
vmread rcx, rax ; Read from Host TR selector

Other Host state fields:
Host CR0 0x6C00
Host CR3 0x6C02
Host CR4 0x6C04
Host FS base 0x6C06
Host GS base 0x6C08
Host TR base 0x6C0A
Host GDTR base 0x6C0C
Host IDTR base 0x6C0E
Host IA32_SYSENTER_ESP 0x6C10
Host IA32_SYSENTER_EIP 0x6C12
Host RSP 0x6C14
Host RIP 0x6C16

As an example to write to host_cr0 in the vmcs, the following code snippet may be used:
mov rbx, cr0
mov rax, 0x6c00 ; encoding for host CR0
vmwrite rax,rbx


Similarly the other host state fields are to be intialized. For a complete list of the vmcs fields see Intel PRM Vol 3b .


Guest Area
The technique to intialize guest state area is the same as the host-state area. Hypervisors use vmwrite instruction to initialize the guest-state area. The encodings used as operands to the vmwrite instruction reflect the guest-state encodings. Here are few examples:

Guest CR0 0x6800
Guest CR3 0x6802
Guest CR4 0x6804
Guest ES base 0x6806
Guest CS base 0x6808

Follow the same approach as before to write to these vmcs fields. For eg: to write a guest CR4 value that has PAE=1, PGE=1, OSFXSR=1,OSXMMEXCPT=1 do the following:

mov rbx, 0x6A0 ; required value in cr4
mov rax, 0x6804 ; GUEST_CR4 encoding
vmwrite rax, rbx

A similar approach is adopted for intializing other GUEST_STATE fields.

VMPTRLD - Load VMCS pointer

vmptrld will load the vmcs pointer for the virtual-machine to be launched. The vmcs stands for Virtual Machine Control Structure. The vmcs is a region in memory which holds all the data for the virtual-machine to be launched. The instruction usage is similar to vmxon:

vmptrld [vmcs_ptr]
vmcs_ptr dq vmcs_region

vmcs_region:
rev_id dd 0

As with vmxon, the revision id of the vmcs_region should be updated with the revision-id supported by the processor (contained in msr 0x480) prior to executing vmptrld. As with vmxon, the vmcs_region must be located on a 4K boundary.


The only other thing worth mentioning is if you try to load the vmxon_ptr as an operand to vmptrld, then execution of vmptrld will fail. Meaning, a code sequence like the one shown below is guaranteed to fail vmptrld:
vmxon [vmxon_ptr]
jbe vmxon_failed
vmptrld [vmxon_ptr]
jbe vmptrld_failed

When the processor executes vmptrld, it realizes that vmptrld's pointer points to the same region as vmxon. This will cuase vmptrld to fail.

It may also be a good practice to execute vmclear before executing vmptrld to load the vmcs-pointer. So the hypervisor may want to do this:
vmclear [vmcs_ptr]
jbe vmclear_failed
vmptrld [vmcs_ptr]
jbe vmptrld_failed

At this point we have executed vmxon, entered VMX_ROOT mode, initialized the virtual-machine-vmcs with vmclear and loaded the virtual-machine-vmcs pointer into the processor by executing vmptrld. The next step is to initialize the vmcs with the virtual-machine's (hence forth referred to as guest) data and then launch the guest.

More on VMXON

vmxon takes as its operand a pointer to the vmxon region.

The code may look like this:

vmxon [vmxon_ptr]

vmxon_ptr dq vmxon_region_begin
vmxon_region_begin: vmxon_rev_id dd 0

Key things to note in the above snippet:

a. The operand to vmxon is vmxon_ptr. vmxon_ptr is a pointer to the vmxon_region. Note: vmxon_region is in physical memory.

b. The vmxon_region contains a 4-byte field called 'rev_id'. The hypervisor is expected to set up the revision-id in the vmxon-region provided by the processor.

How does the hypervisor determine the revision-id?

On Intel processors, the revision-id is contained in VMX_BASIC_MSR (0x480). Bits 31:0 of this MSR contains the revision-id of the processor.

So, for the example above, the hypervisor may want to do:

///////////////////////////////////////////////////

xor eax, eax

xor edx, edx

mov ecx, 0x480

rdmsr ; after rdmsr eax has the revision id

mov dword [vmxon_rev_id], eax ; write the rev-id into vmxon region

vmxon [vmxon_ptr]

////////////////////////////////////////////////

Note: Intel PRM also specifies that the vmxon_region must be aligned on a 4K boundary. If it is not 4K aligned , VMXON is guaranteed to fail.

It is worth repeating that the operand to vmxon is a pointer to the vmxon_region which is in physical memory. Hence vmxon regions should reside in unpaged memory.

Successful completion of vmxon will cause the processor to enter the VMX_ROOT operation.

Hypervisors must also check to see if the execution of vmxon was successful. That can be done by checking the state of eflags.ZF and eflags.CF. If eflags.ZF =0 and eflags.CF=0 then vmxon was successful.

Continuing our previous code:

///////////////////////////////////////////////////
xor eax, eax
xor edx, edx
mov ecx, 0x480
rdmsr
mov dword [vmxon_rev_id], eax
vmxon [vmxon_ptr]

jbe vmxon_failed

vmxon_pass:

if i am here then eflags.ZF=0 and eflags.CF=0. So vmxon was successful.

vmxon_failed:

either eflags.ZF=1 or eflags.CF=1

handle failed code here
//////////////////////////////////////////////////

First look at VMXON

Hypervisors should first begin with the execution of the vmxon instruction. VMXON enables vmx operation. Execution of VMXON puts the processor in VMX_ROOT mode. There are a few things that the hypervisor must ascertain before executing vmxon:

1. Hypervisor must turn on the CR4.VMXE bit. The VMX enable (VMXE) bit is bit 13 of CR4. A typical code sequence would be:

mov eax, cr4
or eax, 0x2000
mov cr4, eax

Executing VMXON without CR4.VMXE=1 will cause the processor to generate a #UD(undefined opcode) exception.

2. Hypervisor must set the fixed bits of CR0. CR0.NE,PG and PE are all fixed bits in vmx operation and they should always be 1 as long as the processor is in VMX_ROOT operation.

Any attempt to clear the fixed bits of CR0 after executing vmxon will cause the processor to generate a #GP exception.

3.A20M: A20M# must be off prior to the execution of vmxon. (violating this will result in a #GP).

4. The hypervisor must ensure that prior to execution of vmxon , the processor is not in V86 mode(eflags.vm must be 0) or in compatibility mode(efer.lma && !cs.l must be false).

The above 4 conditions must be satisfied for vmxon to work. (For it to be successful few other things need to be done).

Note that:
Assertions of INIT# will not be recoginzed by the cpu after the execution of vmxon.I think INIT# just stays pending until it gets unblocked.

VMX instructions in x86

Note: Intel PRM Vol 3b has a lot of details on VMX. If you want a quick snapshot read this blog and then Vol 3b will seem tractable.


To enable virtual machine architecture, Intel provides new instructions as part of their Virtual Machine Extensions(abbreviated VMX) instruction set. This instruction set is different from the one that AMD provides for SVM.

Here is a quick look at the instructions:
(a) VMXON - enter vmx operation
(b) VMXOFF - leave vmx operation
(c) VMREAD - read from the vmcs (vmcs will be discussed later)
(d) VMWRITE - write to the vmcs
(e) VMPTRLD - load vmcs pointer
(f) VMPTRST - store vmcs pointer
(g) VMLAUNCH/VMRESUME - launch or resume virtual machine
(h) VMCALL - call to the hypervisor

Processor/Firmware settings for VMX:

1. To make sure your processor supports VMX, execute CPUID with eax=1 (leaf 1) and check for bit 5 of ecx. If the bit is set the CPU supports VMX else it is not supported.

2. In addition to the above the BIOS must enable VMX by a write to the FEATURE_CONTROL_MSR (address 0x3a). If the msr value is initialized to 0x5 (bit0=1 and bit2=1), then vmx is enabled.
Bit 0 of the msr is the lock bit. If set, the msr is protected. This means the processor will throw a #GP exception when a wrmsr is attempted with the lock bit = 1. Bit 2 is the VMXON_ENABLE bit. Executing VMXON without bit2 set will cause the processor to generate a #GP exception.