In the previous Article 3r33282. (link) I talked about the basic concept of a hypervisor based on Intel hardware virtualization technology. Now, I propose expanding the capabilities of the hypervisor by adding support for multiprocessor architecture (SMP), and also consider an example of how the hypervisor can make changes to the operation of the guest OS.
 
 
All further actions will be carried out on a PC with the following configuration: 3r33283.  
CPU: Intel Core i???K
 
Motherboard: Asus X99-PRO
 
Ram: 16GB
 
Guest OS: Windows 7 x32 with PAE
disabled.  

 
I will begin with a description of the location of the components of the hypervisor on the hard disk (all values ​​are given in sectors).
 

 
The process of loading the hypervisor differs from the previous version only in the presence of a new module 3r-3377. hypervisor.ap [/i] , whose goal is basic processor AP initialization.
 
 
The process of loading modules into memory:
 
 
 
Support SMP
 
I implemented a hypervisor on the principle of symmetric multiprocessing, which means that the same copy of VMX will run on all logical processors present. In addition, IDT and GDT tables as well as tables for paging memory will be common to all logical processors. I did this because the hypervisor will immediately initialize memory for the guest OS address space and there is no need to dynamically reassign the physical addresses of individual pages. Also, with this approach, you can not monitor on the hypervisor side the compliance of TLB processor caches.
 
The initialization process for BSP and AP will be different. All the basic structures involved in the work of the hypervisor will be created during the initialization of the BSP. In addition, the Activity state for vmx non root AP mode processors will be set to HLT state. Thus, the guest OS environment will be emulated in accordance with what it would be without the use of virtualization.
 
 
BSP Initialization:
 
 
3r33939. Initialization of the spinlock
 
3r33939. Initialization and loading of GDT and IDT tables
 
3r33939. Initializing paging addressing tables 3–3–393.  
3r33939. Initialization of VMCS structures and creation of a common EPT table
 
3r33939. AP processor activation. To this end, the INIT - SIPI interrupt sequence is transmitted to each AP. The interrupt SIPI vector is 0x2? which corresponds to the transfer of control to the AP at address 0x20000 (module hypervisor.ap)
 
3r33939. Run the guest OS at 0x7C00 (module win7.mbr)
 
3r395.
 
 
Initialization AP:
 
 
3r33939. After activating the AP, the processor is in real mode. In the hypervisor.ap module, initialization of the memory and paging addressing tables is performed in order to switch to long mode
 
3r33939. Loading IDT, GDT, and also the catalog of tables of page addressing, created at the stage of initialization of BSP
 
3r33939. Initialization of VMCS structures, and loading of the EPT table created at the initialization stage of BSP
 
3r33939. Switch to vmx non-root mode with an active HLT state
 
3r395.
 
 
We can say that the implementation of SMP support in the hypervisor is quite simple, but there are a few points that I would like to draw attention to.
 
 
1.USB Legacy Support
 
The new motherboard models may not have PS /2 connectors, so USB Legacy Support is used for backward compatibility. This means that you can work with the usb keyboard or mouse using the same methods (input /output ports) as it was with the PS /2 standard. The implementation of USB Legacy Support depends not only on the motherboard model, but can also lure in various versions of the firmware. On my Asus X99-PRO motherboard, USB Legacy Support is implemented via SMI interrupts, in the handler of which PS /2 emulation occurs. I am writing about it in such detail, because in my case (firmware version 3801), USB Legacy Support is not compatible with long mode and when I return from SMM, the processor goes into shutdown state.
 
The simplest solution in this situation is to turn off USB Legacy Support before switching to long mode. However, in Windows, at the stage of choosing boot options, the PS /2 keyboard polling method is used, so USB Legacy Support must be activated again before the guest OS starts downloading.
 
 
2. Hardware Task Switch
 
In modern OS, switching between tasks is implemented, as a rule, by program methods. However, in Windows? for interrupts 2 - NMI and 8 - Double Fault, selectors are assigned that point to TSS, which means such interrupts will lead to a hardware context switch. Intel VMX does not support the hardware Task Switch, and attempting to execute it results in VM Exit. For such cases, I wrote my Task Switch handler (the GuestTaskSwitch function). Double Fault interrupts only in the event of a serious system conflict caused by improper handling of other interrupts. In the process of debugging, I did not come across it. But NMI appears on AP processors at the time of restarting Windows. I still doubt this because it is not clear whether these NMIs are the result of a regular reboot process or is it the incorrect operation of the hypervisor at some of the previous stages. If you have any information on this subject, please comment in the comments or write to me in a personal message.
 
 
Changes in the guest OS
 
Honestly, for a long time I couldn’t decide what changes in the operation of the guest OS should be made by the hypervisor. The fact is that, on the one hand, I wanted to show something interesting, such as the implementation of my handlers in the basic network protocols, but on the other hand, it would all rest on a large amount of code, and there is little related to the hypervisor. In addition, I did not want to bind the hypervisor to any particular set of hardware.
 
As a result, the following compromise was found: in this version of the hypervisor, control over system calls from user mode is implemented, in other words, it will be possible to control the operation of application programs running in the guest OS. This type of control is quite simple to implement, and besides it allows to get a visual result of the work.
 
Control over the work of application programs will be executed at the level of system calls. And the main goal will be to change the result of the function NtQuerySystemInformation so that when called with argument 3r32r. SystemProcessInformation [/i] (0x05) it was possible to intercept information about the processes.
 
In Windows ? the application program calls the sysenter assembly command to call the system function, after which control is transferred to the kernel at the r0 level to the handler 3r3277. KiFastCallEntry [/i] . To return to the application level r? use the sysexit command.
 
To access the results of the function NtQuerySystemInformation you need to save the number of the function being called every time you run sysenter. Then when doing 3r37777. sysexit [/i] compare the stored value with the number of the function being intercepted and, in case of coincidence, make changes to the data returned by the function.
 
Intel VMX does not provide direct means of monitoring the performance of 3r3277. sysenter /sysexit [/i] , however, if recorded in Guest MSR IA32_SYSENTER_CS a value of ? then in this case the command sysenter /sysexit will cause a gp exception that can be used to call the VM Exit handler. In order for the GP exception to cause VM Exit, you need to set 13 bits in the field. Exception Bitmap from VMCS.
 
The structure below is used when emulating a sysenter /sysexit pair.
 
typedef struct {
QWORD ServiceNumber;
QWORD Guest_Sys_CS;
QWORD Guest_Sys_EIP;
QWORD Guest_Sys_ESP;
} SysEnter_T;

 
Field ServiceNumber contains the number of the function being called and is updated each time sysenter is called.
 
Fields Guest_Sys_CS, Guest_Sys_EIP, Guest_Sys_ESP updated when the guest OS attempts to write to the corresponding MSR register. For this, write masks are set in MSR-Bitmap Address .
 
//174H 372 IA32_SYSENTER_CS SYSENTER_CS write mask
ptrMSR_BMP[0x100 + (0x174 6)]| = (1UL (0x174 & 0x3F));
//175H 373 IA32_SYSENTER_ESP SYSENTER_ESP write mask
ptrMSR_BMP[0x100 + (0x175 6)]| = (1UL (0x175 & 0x3F));
//176H 374 IA32_SYSENTER_EIP SYSENTER_EIP write mask
ptrMSR_BMP[0x100 + (0x176 6)]| = (1UL (0x176 & 0x3F));

 
The guest OS should not see changes made by the hypervisor in the work of calls to system functions. By setting the mask to read MSR IA32_SYSENTER_CS when reading, you can return the guest OS to the original register value.
 
//174H 372 IA32_SYSENTER_CS SYSENTER_CS read mask
ptrMSR_BMP[0x174 6]| = (1UL (0x174 & 0x3F));

 
Below is the command emulation scheme sysenter /sysexit .
 
3r3204.
 
At the emulation stage, 3r3r777. sysexit [/i] the stored number of the called function is compared with the number NtQuerySystemInformation (0x105). In the case of a match, it is checked that NtQuerySystemInformation is called with the System Process Information argument and if so then the function 3r-3377. ChangeProcessNames (DWORD SPI_GVA, DWORD SPI_size) [/i] makes changes to the structures containing information about the processes.
 
SPI_GVA - this is the guest virtual address of the structure. SYSTEM_PROCESS_INFORMATION
 
SPI_size - the total size of the structures in bytes.
 
The structure itself is SYSTEM_PROCESS_INFORMATION It looks like this:
 
typedef struct _SYSTEM_PROCESS_INFORMATION {3r-3291. ULONG NextEntryOffset;
ULONG NumberOfThreads;
BYTE Reserved1[48];
UNICODE_STRING ImageName;
KPRIORITY BasePriority;
HANDLE UniqueProcessId;
PVOID Reserved2;
ULONG HandleCount;
ULONG SessionId;
PVOID Reserved3;
SIZE_T PeakVirtualSize;
SIZE_T VirtualSize;
ULONG Reserved4;
SIZE_T PeakWorkingSetSize;
SIZE_T WorkingSetSize;
PVOID Reserved5;
SIZE_T QuotaPagedPoolUsage;
PVOID Reserved6;
SIZE_T QuotaNonPagedPoolUsage;
SIZE_T PagefileUsage;
SIZE_T PeakPagefileUsage;
SIZE_T PrivatePageCount;
LARGE_INTEGER Reserved7[6];
} SYSTEM_PROCESS_INFORMATION;

 
There is nothing complicated in its parsing, the main thing is not to forget to transfer the guest virtual address to the physical one, for this the function is used. GuestLinAddrToPhysAddr () .
 
For clarity of the result, I replaced the first two characters in the names of all processes with the sign ‘3r-3273. :) [/b] ’The result of such a replacement is visible in the screenshot.
 
 
 
Results
 
In general, the tasks set at the beginning of the article were completed. The hypervisor ensures stable operation of the guest OS, and also monitors the calling of system functions from the application layer. I note that the main disadvantage of using command emulation 3r3277. sysenter /sysexit [/i] This is a significant increase in VM Exit calls, which affects performance and this is especially noticeable when the guest OS is running in single-processor mode. This disadvantage can be eliminated by performing call control only in the context of selected processes.
 
And that's all for now. The source for the article can be taken
here 3r38282.
 
Thanks for attention.