Information Guide

Software and Hardware Techniques for x86 Virtualization

VMware ESX
Intheearlydaysofx86virtualization,uniformityruled:allCPUsimplementedessentiallythesame32bit architectureandthevirtualmachinemonitor(VMM)alwaysusedsoftwaretechniquestorunguestoperating systems.Thisuniformitynolongerexists.CPUstodaycomein32and64bitvariants.SomeCPUshave hardwaresupportforvirtualization;othersdonot.Moreover,thishardwaresupportcomesinmultipleforms forvirtualizingdifferentaspectsofthex86architecture. Thisdocumentdescribesthex86architecturefromavirtualizationpointofview,relatingcriticalarchitectural featurestothemajorreleasesofVMwareESX.Thegoalistoprovide,foreachversionofVMwareESX,an understandingof:

WhichCPUfeaturesarerequired WhichCPUfeaturescanbeutilized(butarenotrequired) WhichCPUfeaturescanbevirtualizedthatis,madeavailabletosoftwarerunninginthevirtual machine

WithabetterunderstandingofhowCPUfeaturesarerequired,used,andvirtualizedbyVMwareESX,you canreasonmorepreciselyaboutwhatcanbevirtualized,whatperformancelevelsmayresultforagiven combinationofCPU,guestoperatingsystem,andversionofVMwareESX,andhowworkloadsmayrespond toadjustingconfigurationparametersbothforsoftwarerunninginthevirtualmachineandattheVMware ESXlevel. MuchoftheinformationinthisguidealsoappliestoVMwarehostedproducts,suchasVMwareWorkstation forLinuxandWindows.ForsimplicitytheguidefocusesonESX,butyoucanapplythefollowingruleof thumb:anyWorkstationreleasegenerallybehavessimilarlytotheimmediatelysucceedingreleaseofESX. Thisguidecoversthefollowingtopics:

BriefHistoryofthex86Architectureonpage 2 BriefHistoryofVMwareESXonpage 2 Virtualizing32and64bitCPUsonpage 2 ExecutionModesonpage 3 MonitorModesonpage 6 Conclusionsonpage 8 Referencesonpage 9

VMware, Inc.

Software and Hardware Techniques for x86 Virtualization

Brief History of the x86 Architecture

Thex86architecturehasrootsthatreachbackto8bitprocessorsbuiltbyIntelinthelate1970s.As manufacturingcapabilitiesimprovedandsoftwaredemandsincreased,Intelextendedthe8bitarchitectureto 16bitswiththe8086processor.Laterstill,withthearrivalofthe80386CPUin1985,Intelextendedthe architectureto32bits.IntelcallsthisarchitectureIA32,butthevendorneutraltermx86isalsocommon. Overthefollowingtwodecades,thebasic32bitarchitectureremainedthesame,althoughsuccessive generationsofCPUsaddedmanynewfeatures,includinganonchipfloatingpointunit,supportforlarge physicalmemoriesthroughphysicaladdressextension(PAE),andvectorinstructions. In2003,AMDintroduceda64bitextensiontothex86architecture,initiallydubbedAMD64,andbegan shipping64bitOpteronCPUsin2004.Laterin2004,Intelannounceditsown64bitarchitecturalextensionof IA32,callingitIA32eandlateralsoEM64T. TheAMDandIntel64bitextensionsareextremelysimilar,althoughtheydifferinsomeminorways,oneof whichiscrucialforvirtualization(seethediscussionofsegmentlimitsinSoftwareTechnique:Binary Translationonpage 3).Thisguideusesthevendorneutraltermx64torefertothe64bitextensionstothex86 architecture.

Brief History of VMware ESX

In1999,VMwarereleasedthefirstversionofVMwareWorkstation.Itranon,andvirtualized,32bitx86CPUs. Soonafter,VMwareshippedtheESXServerproduct.IncontrasttoWorkstation,whichdependedoneithera LinuxorWindowshost,ESXServerdeployedacustombuiltkernelinstead.ThisVMkernelisdesignedto scalablyandefficientlyrunaworkloadthatconsistsprimarilyofvirtualmachineswhileprovidingstrong informationandperformanceisolationamongthevirtualmachines. Table 1. Physical CPU Requirements and Virtual CPU Options
ESX 1.0-2.5 VMkernel VirtualCPU 32bit 32bit ESX 3.0 32bit 32and64bit ESX 3.5 32bit 32and64bit ESX 4.0 64bit 32and64bit

TheVMkernelrowinTable 1showsthearchitecturalrequirementsforrunningtheVMkernelitselfindifferent versionsofESX.AllversionsofESXbefore4.0canrunonthe32bitx86architecture.Theyalsocanrunonx64 CPUsbutdonottakeadvantageofthe64bitarchitecturalextensions.BeginningwithVMwareESX4.0,a 64bitCPUisrequiredtoruntheVMkernel.Althoughthisrequirementcausesaslightlossofhardware compatibility,by2009themajorityofserverCPUsimplementthex64architecture,makingitdesirabletouse thelarge64bitaddressspaceandotherarchitecturaladvancestoimproveperformanceandscalability.

Virtualizing 32- and 64-bit CPUs

TheVMkerneldoesnotrunvirtualmachinesdirectly.Instead,itrunsaVMMthatinturnisresponsiblefor executionofthevirtualmachine.EachVMMisdevotedtoonevirtualmachine.Torunmultiplevirtual machines,theVMkernelstartsmultipleVMMinstances. BecausetheVMMdecouplesthevirtualmachinefromtheVMkernel,itispossibletorun64bitguest operatingsystemsona32bitVMkernel(andviceversa)aslongastheunderlyingphysicalCPUshaveallthe requiredfeatures.Thelogictohandlethesubtletiesof32and64bitguestoperatingsystemsisencapsulated withintheVMM,whichcantakeadvantageofa64bitphysicalCPUtoruna64bitguestoperatingsystem efficiently,eveniftheunderlyingVMkernelrunsin32bitmode.TheVirtualCPUrowinTable 1showswhich versionsofESXcanrunjust32bitvirtualmachinesandwhichcanrunboth32and64bitvirtualmachines.

VMware, Inc.

Software and Hardware Techniques for x86 Virtualization

Execution Modes
TheVMMimplementsthevirtualhardwareonwhichthevirtualmachineruns.Thishardwareincludesa virtualCPU,virtualI/Odevices,timers,andotherdevices.ThevirtualCPUhasthreefeaturesofinterest:the virtualinstructionset,thevirtualmemorymanagementunit(MMU),andthevirtualinterruptcontroller(PIC orAPIC).TheVMMcanimplementeachoftheseaspectsusingeithersoftwaretechniquesorhardware techniques. Thecombinationoftechniquesusedtovirtualizetheinstructionsetandmemorydeterminesanexecutionmode. Fordiagramsillustratingthetechniquesdescribedinthissection,seePerformanceAspectsofx86 Virtualization(seeReferencesonpage 9foralink).

Instruction Set Virtualization

Inordertorunoneormorevirtualmachinessafelyonasinglehost,ESXmustisolatethevirtualmachinesso thattheycannotinterferewitheachotherorwiththeVMkernel.Inparticular,itmustpreventthevirtual machinesfromdirectlyexecutingprivilegedinstructionsthatcouldaffectthestateofthephysicalmachineas awhole.Instead,itmustinterceptsuchinstructionsandemulatethemsotheireffectisappliedtothevirtual machineshardware,notthephysicalmachineshardware.Forexample,issuingtherebootcommandina virtualmachineshouldrebootjustthatvirtualmachine,nottheentirehost. Thissectioncomparesbinarytranslation,asoftwaretechniqueforrunningthevirtualmachinesinstruction stream,withthehardwaresupportforvirtualizationfoundinrecentCPUsfromIntel(theVTxfeature)and AMD(theAMDVfeature).

Software Technique: Binary Translation

Theoriginalapproachtovirtualizingthe(32bit)x86instructionsetisjustintimebinarytranslation(BT).This approachisimplementedinallversionsofVMwareESX,anditistheonlyapproachusedinVMwareESX1.x and2.x.Forspecificity,becausethistechniquevirtualizesthe32bitarchitecture,wecallitBT32. Whenrunningavirtualmachinesinstructionstreamusingbinarytranslation,thevirtualmachineinstructions mustbetranslatedbeforetheycanbeexecuted.Moreconcretely,whenavirtualmachineisabouttoexecutea blockofcodeforthefirsttime,ESXsendsthiscodethroughajustintimebinarytranslator,muchlikeaJava virtualmachinetranslatesJavabytecodeontheflyintonativeinstructions.ThetranslatorintheVMMdoes notperformamappingfromonearchitecturetoanother,butinsteadtranslatesfromthefullunrestrictedx86 instructionsettoasubsetthatissafetoexecute.Inparticular,thebinarytranslatorreplacesprivileged instructionswithsequencesofinstructionsthatperformtheprivilegedoperationsinthevirtualmachine ratherthanonthephysicalmachine.Thistranslationenforcesencapsulationofthevirtualmachinewhile preservingthex86semanticsasseenfromtheperspectiveofthevirtualmachine. Tokeeptranslationoverheadslow,theVMMtranslatesvirtualmachineinstructionsthefirsttimetheyare abouttoexecute,placingtheresultingtranslatedcodeinatranslationcache.Ifthesamevirtualmachinecode executesagaininthefuture,theVMMcanreusethetranslatedcode,therebyamortizingthetranslationcosts overallfutureexecutions.Tofurtherreducetranslationcostandtominimizememoryusagebythetranslation cache,theVMMcombinesbinarytranslationofkernelcoderunninginthevirtualmachinewithdirect executionofusermodecoderunninginthevirtualmachine.Thisissafebecauseusermodecodecannot executeprivilegedinstructions. ABTbasedVMMmustenforceastrictboundarybetweenthepartoftheaddressspacethatisusedbythe virtualmachineandthepartthatisusedbytheVMM.TheVMwareVMMenforcesthisboundaryusing segmentation. Segmentationisahardwarefeatureofthex86CPUthatdatesbacktoits16bitancestors.Asegmentisa consecutiverangeofmemory,identifiedbyabase(thestartingaddress)andalimit(thelengthofthesegment). Wheneveranx86instructionaccessesmemory,itdoessowithrespecttoaparticularsegment.The segmentationhardwarechecksthememoryaddressagainstthesegmentlimit.Ifitiswithinthelimit,thebase addressisaddedandtheaccessispermittedtoproceed.Iftheaddressexceedsthelimit,thememoryaccessis abortedandtheprocessorraisesaprotectionfault.

VMware, Inc.

Software and Hardware Techniques for x86 Virtualization

Becausemostmodernoperatingsystems,includingWindows,Linux,andSolaris,makeonlylightuseof segmentation,itispossiblefortheVMMtousesegmentationtoenforcetheboundarybetweenvirtual machineandVMM.InrarecaseswhentheusesofsegmentationbythevirtualmachineandtheVMMconflict, theVMMcanresorttosoftwaresegmentationchecks,albeitataslightlossofperformance. In2003,whenAMDextendedthex86architecturefrom32to64bits,iteliminatedsegmentlimitchecksfor 64bitcode(32bitcodestillretainedsegmentlimitchecksforbackwardscompatibility).Thischangemeant thataBTbasedVMMcouldnotusesegmentationtoprotecttheVMMfroma64bitvirtualmachine.Inother words,BT32couldvirtualizethe32bitx86architectureefficiently,butBT64couldnotvirtualizethe64bit architectureefficiently. Whenthisarchitecturaldeficiencybecameapparent,AMDaddedsegmentlimitsbackinto64bitcode.This additionmissedtheinitialOpteronRevCprocessor(whichshippedinlimitedquantitiesonly)butwas presentbeginningwiththenextrevision,RevD,andhasremainedpresentinAMD64bitCPUseversince. Thus,all64bitAMDCPUs,withtheexceptionoftheoriginalOpteronRevC,canrunvirtualmachineswith BT64. TheIntel64bitextensionstothex86architecturealsoomittedsupportforsegmentlimitchecksfor64bitcode. UnlikeAMD,however,Intelhasnotaddedsupportforsegmentlimitchecksinsubsequentprocessors.This limitationmakesitinefficienttorun64bitvirtualmachinesusingBT64onIntelCPUs. Table 2summarizestheformsofbinarytranslationthatdifferentversionsofESXcanuseonIntelandAMD CPUs. Table 2. Supported Uses of Binary Translation
ESX 1.0-2.5 AMD Intel BT32 BT32 ESX 3.0 BT32,BT64 BT32 ESX 3.5 BT32,BT64 BT32 ESX 4.0 BT32,BT64 BT32

Hardware Technique: VT-x and AMD-V

Aboutthesametimetheyweremakingthetransitionfrom32bitto64bithardware,bothIntelandAMD recognizedtheimportanceofvirtualization.Bothcompaniesbegandesigninghardwarethatmadeiteasierfor aVMMtorunvirtualmachines.Thefirsthardwaredesignsfocusedonthesubproblemofhowtovirtualize the32and64bitx86instructionset.TheInteldesign,calledVTx,tookonparticularimportancebecauseit providedawaytovirtualize64bitvirtualmachinesefficiently.(BT64isnotefficientbecauseofthelackof segmentlimitchecksin64bitmodeonIntelCPUs.)AMDsubsequentlyintroducedAMDVtoprovide hardwaresupportforinstructionsetvirtualizationbutvirtualizationof64bitvirtualmachinesusingBT64 waspossiblealreadyforAMDCPUs. VTxandAMDVaresimilarinaimbutdifferentindetail.BothdesignsallowaVMMtodoawaywithbinary translationwhilestillbeingabletofullycontroltheexecutionofavirtualmachinebyrestrictingwhichkinds of(privileged)instructionsthevirtualmachinecanexecutewithoutinterventionbytheVMM. Inspirit,thetwohardwaremechanismsreachbacktotheclassicdaysofvirtualizationonIBM360 mainframes.VTxandAMDVbothallowaVMMtogivetheCPUtoavirtualmachinefordirectexecution (anactioncalledaVMentry)upuntilthepointwhenthevirtualmachinetriestoexecuteaprivileged instruction.Atthatpoint,thevirtualmachineexecutionissuspendedandtheCPUisgivenbacktotheVMM (anactioncalledaVMexit).TheVMMthenfollowstheclassicapproachfrommainframedays,inspectingthe virtualmachineinstructionthatcausedtheexitaswellasotherinformationprovidedbythehardwarein responsetotheexit.Withtherelevantinformationcollected,theVMMemulatesthevirtualmachine instructionagainstthevirtualmachinestateandthenresumesexecutionofthevirtualmachinewithanother VMentry.

VMware, Inc.

Software and Hardware Techniques for x86 Virtualization

IntelCPUsacquiredVTxsupportinlaterPentium4CPUs(startingwithPrescott).AMDintroducedAMDV inAMDOpteronRevFCPUs. Table 3. Supported Uses of Hardware Instruction Set Virtualization

ESX 1.0-2.5 AMD Intel VTx64 ESX 3.0 ESX 3.5 AMDV32,AMDV64 VTx64 ESX 4.0 AMDV32,AMDV64 VTx32,VTx64

Table 3showswhichversionsofESXcanuseVTxorAMDVtorun32and64bitvirtualmachines.Thetable assumesanAMDBarcelonaCPUornewer,becausetheinitialAMDVimplementationinOpteronRevFis affectedbyanissuethatVMwarehaschosennottoworkaround.InESX3.5,AMDVcanbeusedonlyin combinationwithrapidvirtualizationindexing(RVI)(seeMemoryandMMUVirtualizationonpage 5), whereasinESX4.0,AMDVcanbeusedwithorwithoutRVI. AsshowninTable 2andTable 3,thenumberofsupportedwaystovirtualizethex86instructionsethas increasedovertime,asnewversionsofESXandnewCPUshavebecomeavailable.

Memory and MMU Virtualization

Allmodernx86CPUsimplementvirtualmemory,whichisatechniqueforflexiblymappingmultiplevirtual addressspaces(typicallyoneperprocess)intoapossiblysmalleramountofphysicalmemory.Thedetailsof howanoperatingsystemmanagesthismappingisbeyondthescopeofthisguide.However,forthex86 architecture,themappingisspecifiedusingasetofmemoryresidenthierarchical4KBpagetables.Atreeof suchpagetables,identifiedbyarootpagetable,specifiestheentiremappingofavirtualaddressspaceinto physicalmemory. Thex86MMUcontainstwomainstructures:apagetablewalkerandacontentaddressablememorycalleda translationlookasidebuffer(TLB)toaccelerateaddresstranslationlookups.Whenaninstructionaccessesa virtualaddress VA ,segmentationhardwareconvertsthevirtualaddresstoalinearaddress LA byadding thesegmentbase.ThenthepagetablewalkerreceivestheLAandtraversesthepagetabletreetoproducethe correspondingphysicaladdress PA .Whenthepagetablewalkcompletes,the LA ,PA pairisinsertedinto theTLBtoacceleratefutureaccessestothesameaddress. Accordingly,thetaskoftheVMMisnotonlytovirtualizememorybuttovirtualizevirtualmemorysothatthe guestoperatingsystemcanusevirtualmemory.Toaccomplishthistask,theVMMmustvirtualizethex86 MMU.ItdoessobyhavingtheVMMremapaddressesasecondtime,belowthevirtualmachine,fromPAto machineaddress MA ,toconfinethevirtualmachinetothemachinememorythattheVMMandVMkernel haveallowedittouse. Thefollowingsectionscompareshadowpagetables,asoftwaretechniqueformemoryvirtualization,withthe hardwaresupportfoundinrecentCPUsfromAMD(RVI)andIntel(extendedpagetables,orEPT).

Software Technique: Shadow Page Tables

Tovirtualizememorywithoutspecialhardwaresupport,theVMMcreatesashadowpagetableforeach primarypagetablethatthevirtualmachineisusing.TheVMMpopulatestheshadowpagetablewiththe compositionoftwomappings:

The LA PA mappingspecifiedbytheguestoperatingsystem,obtainedfromtheprimarypagetables The PA MA mappingdefinedbytheVMMandVMkernel

Bybuildingshadowpagetablesthatcapturethiscomposite LA MA mapping,theVMMcanpointthe hardwareMMUdirectlyattheshadows,allowingthevirtualmachinesmemoryaccessestorunatnative speedwhilebeingassuredthatthevirtualmachinecannotaccessmachinememorythatdoesnotbelongtoit. However,shadowpagetablesincuroverheadsinothersituations.

Whenthevirtualmachineupdatesaprimarypagetable,theVMMmusttraptheupdateandpropagate thechangeintothecorrespondingshadowpagetableortables.Thisslowsdownmemorymapping operationsaswellascreationofnewprocessesinvirtualmachines.

VMware, Inc.

Software and Hardware Techniques for x86 Virtualization

Whenthevirtualmachinetouchesmemoryforthefirsttime,theshadowpagetableentrymappingthis memorymustbecreatedondemand,slowingdownthefirstaccesstomemory.(Thenativeequivalentis aTLBmiss.) Whenthevirtualmachineswitchescontextfromoneprocesstoanother,theVMMmustinterveneto switchthephysicalMMUtothenewprocessshadowpagetableroot. Shadowpagetablesconsumeadditionalmemory.

Hardware Technique: RVI and EPT

Toaddresstheoverheadsinherentinshadowpagetables,bothAMDandIntelnowbuildspecialpurpose hardwaretosupportMMUvirtualization.AMDintroducedsupportforMMUvirtualization,calledRVI,in thequadcoreOpteron(Barcelona)CPU.Intelintroducedsimilarfunctionality,calledEPT,initsNehalem generationofCPUs. JustasAMDVandVTxaresimilarinspiritbutdifferentindetail,soareRVIandEPT.Bothdesignspermit thetwolevelsofaddressmappingtobeperformedinhardwarebypointingthephysicalMMUattwodistinct setsofpagetables.Thefirstisdefinedbythevirtualmachine: LA PA .Thesecond,invisibletothevirtual machine,iscontrolledbytheVMM: PA MA .Giventhesetwomappings,thephysicalCPUspagewalker canwalkthetwosetsofpagetablestoproduce (LA,MA) pairsthatarecachedintheTLB. Thisarrangementdoesawaywithshadowpagetablesatthecostofasinglesetofnestedorextendedpage tablesthatmapfromPAtoMA.Becausethenestedorextendedpagetablesarelargelystaticandneedno updatewheneverthevirtualmachinecreatesormodifiespagetables,theVMMneednotintervenewhen virtualmachinepagetablesareupdated.Moreover,theVMMdoesnotneedtobeinvolvedinvirtualmachine contextswitches.Thevirtualmachinecanchangethe LA PA pagetablerootonitsown. Table 4. Support for RVI and EPT in ESX
ESX 1.0-2.5 AMDRVI IntelEPT ESX 3.0 ESX 3.5 yes ESX 4.0 yes yes

AlthoughRVIandEPThavecompellingadvantages,thereisonepotentialdownside:aTLBmissisnowmore expensivebecauseitmustbeservicedbyatwolevelpagewalker. Formostworkloads,RVIorEPTprovidesanoverallperformancewinovershadowpagetables,butthereare exceptionstothisrule:workloadsthatsufferfrequentTLBmissesorthatperformfewcontextswitchesorpage tableupdates.Formoreinformation,seethepapersPerformanceEvaluationofAMDRVIHardwareAssist andPerformanceEvaluationofIntelEPTHardwareAssist(seeReferencesonpage 9forlinks). Table 4showswhichversionsofESXcanuseRVIorEPT.Whensuchsupportisnotpresent,eitherinthe versionofESXorinthephysicalCPU,thefallbackisshadowpagetables.

Monitor Modes
Thisguidedescribesatwowaychoicebetweensoftwareandhardwaretechniquesforinstructionset virtualization(BTononehandandAMDVorVTxontheotherhand).Similarly,itdescribesatwowaychoice betweensoftwareandhardwaretechniquesformemoryvirtualization(shadowpagetablesononehandand RVIorEPTontheotherhand). Unfortunately,thetwoformsofhardwaresupportarenotorthogonalfeatures.RVIisinseparablefrom AMDVandEPTisinseparablefromVTx.Thisleavesonlythreevalidcombinations:

BTswMMUbinarytranslationandshadowpagetables HVswMMUAMDVorVTxandshadowpagetables HVhwMMUAMDVwithRVIorVTxwithEPT

VMware, Inc.

Software and Hardware Techniques for x86 Virtualization

Inthelistabove,HVstandsforhardwaresupportforinstructionvirtualization.Itisaconvenientcommon termforeitherAMDVorVTx.Werefertotheabovethreeoptionsasmonitormodesbecausetheydescribethe waytheVMMrunsaparticularvirtualmachineonagivenphysicalCPU.

Choice of Monitor Mode

Whenavirtualmachineispoweringon,theVMMinspectsthephysicalCPUsfeaturesandtheguestoperating systemtypetodeterminethesetofpossibleexecutionmodes.OnESX3.0andearlier,thesetofcandidate executionmodeshasexactlyonechoice,asyoucandeterminefromprecedingtables. However,beginningwithESX3.5,andespeciallywithESX4.0,therearecasesinwhichmorethanone executionmodeispossible.Insuchacase,howdoestheVMMmakeachoice?Itfirstfindsthesetofmodes allowed.Thenitrestrictstheallowedmodesbyconfigurationfilesettings.Finally,amongtheremaining candidates,itchoosesthepreferredmode.Thefollowingexamplesillustratetheprocess:

ESX3.5onanAMDShanghaiCPUanda64bitvirtualmachineTheallowedmodesareBTswMMUand HVhwMMU(AMDVwithRVI).Thepreferredoptionfora64bitvirtualmachineisHVhwMMU,sothe VMMchoosesthismodeatpowerontime. ESX3.5onanIntelNehalemCPUanda64bitvirtualmachineRunwithHVswMMU(becauseESX3.5 doesnotsupportEPT). ESX4.0onanAMDShanghaiCPUanda64bitvirtualmachineThechoiceisamongBTswMMU, HVswMMU,andHVhwMMU.HVhwMMUwins. ESX4.0onanolderOpteronRevFCPUanda64bitvirtualmachineOnlyoneoptionisavailable: BTswMMU(becauseESXcannotuseAMDVonthisCPU). ESX4.0onanIntelNehalemCPUanda64bitvirtualmachineTheallowedmodesareHVswMMUand HVhwMMU(BTisnotallowedfor64bitvirtualmachinesonIntelCPUsbecausesegmentlimitchecks aremissing).TheVMMchoosesHVhwMMU.

Certainfeaturesmayrestricttheavailablemodes.Forexample,VMwareFaultTolerancecannotuseRVIor EPTbecauseoftheirlackofdeterminism,anditavoidsBT,narrowingthechoicetoHVswMMU. Whenmultiplechoicesremain,aprioritizationalgorithmrunstochoosethebestmode:

ForESX3.5,theonlycaseinwhichthereisachoiceisonAMDCPUsonwhichBTswMMUand HVhwMMUmightbothbeavailable.Thedefaultchoicefor32bitvirtualmachinesisBTswMMU.For 64bitvirtualmachines,itisHWhwMMU,althoughOS/2,UnixWare,andOpenServerdefaultto HWhwMMU.(Thechoicesfor32and64bitvirtualmachinesdifferbecausemany32bitguestoperating systemssufferhighoverheadsonAPICvirtualizationwhenrunningwithhardwarevirtualization becausetheymakefrequentaccesstoamemorymappedTPRregister.) ForESX4.0,manymoresituationscanresultinmultipleallowableexecutionmodes.Thegeneralpriority orderforCPUsthathavehardwaresupportforAPICvirtualization(mostnewerIntelCPUsfrom Penrynonwards)is:HVhwMMU,followedbyHVswMMU,followedbyBTswMMU.ForCPUs withouthardwaresupportforAPICvirtualization,theorderfor32bitWindowsguestoperatingsystems is:HVhwMMU,followedbyBTswMMU,followedbyHVswMMU.Forcertainunusualguestoperating systems,includingOS/2,UnixWare,andOpenServer,BTisdiscouragedordisallowed.

Specifying the Preferred Monitor Mode

Insomecases,anexplicitspecificationofmonitormodepreferencemaybedesirable.Althoughthissituation israre,thecomplexityofworkloadsandvirtualmachineconfigurationsmakesamanualoverridedesirablein casesinwhichthedefaultchoiceleadstolessthanoptimalperformance. Invirtualmachineconfigurationfiles,youcanrestrictthesetofmodesbysettingoneorbothofthefollowing options: monitor.virtual_mmu = software | hardware | automatic monitor.virtual_exec = software | hardware | automatic

VMware, Inc.

Software and Hardware Techniques for x86 Virtualization

Chooseoneofsoftware,hardware,orautomaticforeachvariable.BothESX3.5andESX4.0recognizethe monitor.virtual_mmusetting.OnlyESX4.0recognizesmonitor.virtual_exec.Youcanexpressall possibleESX3.5modechoiceswiththemonitor.virtual_mmuoptionalone. Ifasettingisnotspecified,theeffectisthesameasautomatic.Ifitissettohardware,itforcestheuseofthe givenformofhardwaresupportifthefeatureisavailableandsupported.Likewise,ifthesettingissoftware, theVMMattemptstorunthevirtualmachinewithoutthegivenformofhardwaresupport,ifallowed. Althoughtheconfigurationfilesettingsareflexibleenoughtoexpressallofthe22possiblecombinations, onlythreeofthefourcombinationsarevalid.Validcombinationsareusedtoselectoneofthethreeexecution modes.IftheCPUdoesnotsupporttherequestedexecutionmode,thesettingsareignored.Inaddition,the settingsareignorediftheCPUimplementstheexecutionmodebuttheversionofESXdoesnotsupportit. ThevSphereClientuserinterfacepresentsathreewaychoice.ThechoiceyoumakeinthevSphereClientis mappedtotheabove(monitor.virtual_mmuandmonitor.virtual_exec)settingsbeforethevirtual machinepowerson. Thesettingsarealwaystreatedashints.IfyouuseVMotiontomoveavirtualmachinefromonehostwhere thesettingshadaneffecttoanotherhostwheretheyareatoddswiththephysicalCPUortheversionofESX, thesettingsareignored.IfyouuseVMotiontomovethevirtualmachinebacktothefirsthost,thesettingstake effectagain.

Determining Active Monitor Mode

Giventhecomplexityofthemodedeterminationlogic,thedependencyontheversionofESX,andthe diversityofCPUs,itisnotfeasibletoprovideanexhaustivelistofallpossibleoutcomes.Instead,thisguide describeshowtodeterminefromalogfilewhichmonitormodeisused. ForESX3.5onanIntelCPU,ifthevirtualmachineis32bit,themodeisBTswMMU.Ifthevirtualmachineis 64bit,themodeisHVswMMU(thatis,usingVTxwithoutEPT).ForESX3.5onanAMDCPU,lookforthe stringhv-svmorhv-noneinvmware.log.Ifthestringhv-svmispresent,themonitormodeisHVhwMMU, otherwiseitisBTswMMU. ForESX4.0,lookforlinesthatcontainMONITOR MODEinvmware.log.Theselineslisttheallowedmodes,the filteringsteps,andthefinalchosenmode.Forexample:
greenland 1034> grep MONITOR vmware.log MONITOR MODE: allowed modes : BT HV HWMMU MONITOR MODE: user requested modes : BT MONITOR MODE: guestOS preferred modes: BT HWMMU HV MONITOR MODE: filtered list : BT

OnIntelCPUsyoumayseeBT32listedinsteadofBT.Thisreflectsthefactthattheabilitytousesoftware virtualizationontheseCPUsislimitedto32bitvirtualmachines.WhenusingBT32,themonitorrunsthe virtualmachinewithsoftwarevirtualizationforaslongaspossible,butitdynamicallyswitchestohardware virtualizationifthevirtualmachineenters64bitmode. VMwareexpectstheESX4.0approachtomodedeterminationtoremainvalidinfuturereleasesofESX, althoughtheexactformatoftheconfigurationfilelinesmightchangeasESXsupportsnewhardwarefeatures infutureCPUsorasVMwarefindswaystoimprovetheVMMitself.

Conclusions
Virtualizationusedtobesimple:allCPUswere32bit,andtheVMMalwaysusedbinarytranslationand shadowpagetablestorunthevirtualmachine.Thelandscapetodayisirregularandchanging.Althoughthis mightseemlikeanunwelcomedifficulty,VMwarepreferstoviewitasanopportunity:havinghardware vendorscompetetooptimizetheirhardwareforvirtualizationmightcreatediversity,butitalsodelivers efficiencygains.Atthesametime,thecomplexitycanbedifficultforusersofvirtualizationsoftware,anditis notviableforeveryonetospendthetimetobecomeexperts. Thisguide,althoughitisricherindetailthanthetypicalfactsheet,makesanefforttostrikeausefulbalance, allowingyoutounderstandthestateofvirtualizationwithcurrenthardwareandsoftwareandalsoto understandwherevirtualizationmightbegoing.
VMware, Inc. 8

Software and Hardware Techniques for x86 Virtualization

References

K.AdamsandO.Agesen,Acomparisonofsoftwareandhardwaretechniquesforx86virtualizationin ASPLOSXII:Proceedingsofthe12thinternationalconferenceonArchitecturalsupportforprogramming languagesandoperatingsystems(SanJose,California,USA,2006),ACM,pp.213. http://www.vmware.com/pdf/asplos235_adams.pdf R.Bhargava,B.Serebrin,F.Spadini,andS.Manne,Acceleratingtwodimensionalpagewalksfor virtualizedsystemsinASPLOSXIII:Proceedingsofthe13thinternationalconferenceonArchitectural supportforprogramminglanguagesandoperatingsystems(Seattle,Washington,USA,2008),ACM,pp. 2635. http://www.amd64.org/fileadmin/user_upload/pub/p26bhargava.pdf T.Cramer,R.Friedman,T.Miller,D.Seberger,R.Wilson,andM.Wolczko,CompilingJavajustintime. IEEEMicro17,3(1997),3643. OleAgesen,PerformanceAspectsofx86Virtualization http://www.vmworld.com/docs/DOC2476 NikhilBhatia,PerformanceEvaluationofAMDRVIHardwareAssist http://www.vmware.com/resources/techresources/1079 NikhilBhatia,PerformanceEvaluationofIntelEPTHardwareAssist http://www.vmware.com/resources/techresources/10006

About the Author

OleAgesenhasworkedintheVMwaremonitorgroupsince1999.

Acknowledgments
Theauthorthanksthosewhocontributedtothisguide.HenrikMnsterfirstpointedouttheneedforsucha guide.JimMattsonandJeffreySheldonprovidedmanycommentsthatimprovedthetechnicalaccuracy.Sean Bormanassistedtogreatlyimprovetheclarityandfocus.

If you have comments about this documentation, submit your feedback to: docfeedback@vmware.com VMware, Inc. 3401 Hillview Ave., Palo Alto, CA 94304 www.vmware.com Copyright 2009 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware, the VMware boxes logo and design, Virtual SMP, and VMotion are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies. Revision: 20090622 Item: EN-000240-00