RSSServer.docx

RSSServer.docx

《RSSServer.docx》由会员分享，可在线阅读，更多相关《RSSServer.docx（24页珍藏版）》请在冰豆网上搜索。

RSSServer

Receive-SideScalingEnhancementsinWindowsServer 2008

November5,2008

Abstract

Thispaperprovidesinformationaboutreceive-sidescaling（RSS）,atechnologythatenablespacketreceive-processingtoscalewiththenumberofavailablecomputerprocessors.ThispaperprovidesanoverviewofRSSforNDISdriverdevelopersanddiscussestheimplicationsofRSSfordriverandhardwaredevelopment.ThepaperalsodescribesthenewenhancementsintroducedinWindowsServer®2008thatimplementNDISv6.1forRSScoexistencewithPCIv3.0messagesignaledinterrupts（MSI-X）.

Originalequipmentmanufacturers（OEMs）andITadministratorsarealsoencouragedtoreadthispapertogainbetterunderstandingofRSSandhowitisimplementedinWindowsServer2008.

Theinformationinthispaperappliestothefollowingoperatingsystems:

WindowsVistaSP1

WindowsServer2008

Referencesandresourcesdiscussedherearelistedattheendofthispaper.

Forthelatestinformation,see:

Disclaimer:

Thisisapreliminarydocumentandmaybechangedsubstantiallypriortofinalcommercialreleaseofthesoftwaredescribedherein.

TheinformationcontainedinthisdocumentrepresentsthecurrentviewofMicrosoftCorporationontheissuesdiscussedasofthedateofpublication.BecauseMicrosoftmustrespondtochangingmarketconditions,itshouldnotbeinterpretedtobeacommitmentonthepartofMicrosoft,andMicrosoftcannotguaranteetheaccuracyofanyinformationpresentedafterthedateofpublication.

ThisWhitePaperisforinformationalpurposesonly.MICROSOFTMAKESNOWARRANTIES,EXPRESS,IMPLIEDORSTATUTORY,ASTOTHEINFORMATIONINTHISDOCUMENT.

Complyingwithallapplicablecopyrightlawsistheresponsibilityoftheuser.Withoutlimitingtherightsundercopyright,nopartofthisdocumentmaybereproduced,storedinorintroducedintoaretrievalsystem,ortransmittedinanyformorbyanymeans（electronic,mechanical,photocopying,recording,orotherwise）,orforanypurpose,withouttheexpresswrittenpermissionofMicrosoftCorporation.

Microsoftmayhavepatents,patentapplications,trademarks,copyrights,orotherintellectualpropertyrightscoveringsubjectmatterinthisdocument.ExceptasexpresslyprovidedinanywrittenlicenseagreementfromMicrosoft,thefurnishingofthisdocumentdoesnotgiveyouanylicensetothesepatents,trademarks,copyrights,orotherintellectualproperty.

Unlessotherwisenoted,theexamplecompanies,organizations,products,domainnames,e-mailaddresses,logos,people,placesandeventsdepictedhereinarefictitious,andnoassociationwithanyrealcompany,organization,product,domainname,emailaddress,logo,person,placeoreventisintendedorshouldbeinferred.

©2008MicrosoftCorporation.Allrightsreserved.

Microsoft,Windows,WindowsServer,andWindowsVistaareeitherregisteredtrademarksortrademarksofMicrosoftCorporationintheUnitedStatesand/orothercountries.

Thenamesofactualcompaniesandproductsmentionedhereinmaybethetrademarksoftheirrespectiveowners.

DocumentHistory

Date

Change

November5,2008

Firstpublication

Contents

Introduction3

PacketReceive-ProcessingLimitationswithoutRSS3

PacketReceive-ProcessingwithRSS4

RSSAlgorithm4

RSSVersusnon-RSSReceiveProcessing4

RSSSetup6

RSSCapabilitiesAdvertisement6

ConfiguringtheRSSParameters7

SelectionofCPUsEligibleforRSS8

SelectionoftheDefaultRSSHashFunction9

ToeplitzHashFunctionSpecification9

MappingPacketstoProcessors11

PacketReceiveProcessingwithRSS12

RSSImplementation12

RSSandMSI-X13

ExampleRSSNICwithMSI-XCapability13

RSSLoad-BalancingImplementationinWindowsServer200816

RSSConfigurationParameters16

RSSLimitations17

Resources18

Introduction

Today’ssystemshaveanincreasingnumberofCPUs.TheabilityofthenetworkingprotocolstackoftheWindows®operatingsystemtoscalewellonamulti-CPUsystemisrestricted.ThisrestrictioniscausedbythearchitectureoftheNetworkDriverInterfaceSpecification（NDIS）inWindowsServer®2003andearlierversions,whichlimitsreceiveprotocolprocessingtoasingleCPUatanyonetime.Receive-sidescaling（RSS）resolvesthisissuebyallowingthenetworkloadfromanetworkadaptertobedistributedacrossmultipleCPUs.

ThispaperisforthetechnicalcommunitythatwantstogaindeeperinsightintohowRSSoperatesinWindowsServer2008.Itprovidesspecificinsightsintoimplementationissuesforindependenthardwarevendors（IHVs）,andfororiginalequipmentmanufacturers（OEMs）andsystemadministratorswhowanttounderstandhowthetechnologyworks.

WindowsServer2003SP1andearlierversionsallowsonlyasingledeferredprocedurecall（DPC）foreachnetworkadaptertoexecuteatanyonetime.WindowsServer2003SP2andnewerversionsofWindowsServerthatuseRSSenablemultipleDPCsondifferentCPUsforeachinstanceofanetworkadapterminiportdriver,whiletheypreservein-orderdeliveryofmessagesonaper-streambasis.RSSalsosupportsdynamicloadbalancing,asecurehashingmechanism,parallelinterrupts,andparallelDPCs.

TheinformationinthispaperappliestoWindowsServer2008andWindowsVista®SP1unlessotherwisenoted.

PacketReceive-ProcessingLimitationswithoutRSS

WindowsServer2003SP1andearlierversionsdonotallowmultipleprocessorstoconcurrentlyprocessreceiveindicationsfromasingle-networkadapter.NDISversion 5.xwasincludedwithWindowsServer2003SP1andearlierversions.Inthisversion,apacketthatisreceivedfromthenetworkonaspecificnetworkadaptermanifestsitselfasaninterrupttothehostprocessorfromthenetworkadapterandeventuallycausesaDPCtobequeuedononeofthesystemprocessors.TheDPCrunstocompletion,typicallyontheprocessorthathostedtheinterrupt,andadditionalinterruptsfromthenetworkadapteraredisableduntiltheDPCcompletesitscycle.

Manyscenarios,suchaslargefiletransmissions,requirethehostprotocolstacktoperformsignificantworkinthecontextofreceiveDPCprocessing（forexample,sendingoutnewdataorperformingmemorycopy）.Inthesescenarios,thelackofparallelisminNDISv5.xpacketreceiveprocessingresultsinanoveralllackofscaling.

Inaddition,somecontemporaryCPUsandchipsetsrouteallinterruptsfromasinglenetworkadaptertoonespecificprocessor,whichresultsinasimilarlackofparallelism.Therefore,scalingissuesonlyincreasebecauseoneCPUhandlesalldeviceinterrupts.

PacketReceive-ProcessingwithRSS

Thesingle-CPUprocessingissuesareresolvedbyimplementingRSS.Thistechnologyenablesreceiveprocessingtobebalancedacrossmultipleprocessorsinthesystemwhilein-orderdeliveryofthedataismaintained.RSSenablesparallelDPCs.Inaddition,inWindowsServer2008andlaterversionsifthecomputerandnetworkadaptersupportit,RSSenablesparallelinterrupts.

RSSprovidesthefollowingbenefits:

∙Parallelreceiveprocessing.

ReceivepacketsfromasinglenetworkadaptercanbeindicatedbygeneratinginterruptsandDPCsconcurrentlyonmultipleCPUs.

∙Preservingin-orderpacketdelivery.

ReceivedpacketsforaspecificstreamfromasinglenetworkadapteraredeliveredinordertotheTCP/IPprotocoldriver.

∙Dynamicloadbalancing.

Assystemloadonthehostvaries,RSSrebalancesthenetworkprocessingloadamongtheprocessors.

∙Cachelocality.

Becausepacketsfromasingleconnectionaremappedtoaspecificprocessor,stateforaparticularconnectionstaysresidentinthecacheoftheprocessor.Thiseliminatescachethrashingandalsoimprovesperformance.

∙Send-sidescaling.

TheTCP/IPprotocolpassestheRSShashvaluetotheNICineachpacketontheegresspath,whichallowsthesendcompletionstobeindicatedonthesameCPU.Thisenablesbetterscalingonthesendside.

∙Toeplitzhashing.

ThedefaultgeneratedRSSsignatureisstatisticallysecure.Thismakesitmoredifficultformaliciousremotehoststoforcethesystemintoanunbalancedstate.

RSSAlgorithm

ThissectiondefinestheRSSalgorithmandcontrastsitwiththenon-RSSpacketprocessingalgorithm.Generally,RSSenablespacketsfromasinglenetworkadaptertobeprocessedinparallelonmultipleCPUswhileitpreservesin-orderdeliverytoTCPconnections.

RSSVersusnon-RSSReceiveProcessing

TheNDIS5.1architectureforprocessingincomingpacketsisimplantedinWindowsServer2003andearlierversions.Anetworkadaptervendortypicallyimplementsthisarchitecturebytakingadvantageofareceivedescriptorqueuebetweenthenetworkadapterandtheminiportadaptertopassper-packetinformation.Thepacketsareprocessedinthefollowingsequence:

1.Aspacketsarriveoffthewireatthenetworkadapter,thepacketcontentsaretransferredintohostmemorybyusingdirectmemoryaccess（DMA）,andareceivedescriptoristransferredintothereceivedescriptorqueue（againthroughDMA）.Aninterruptiseventuallypostedtothehosttoindicatethatnewdataispresent.Exactlywhentheinterruptfiresdependsontheinterruptmoderationscheme.

2.Dependingonthesystem’schipsetandCPUs,eithertheinterruptisdistributedtooneofthehostprocessorsoritisalwaysroutedtothesameprocessor.

3.Ifadditionalpacketsarriveatthenetworkadapter,thendataanddescriptorsaretransferredtohostmemorybyusingDMA.Aninterruptisnotfired.

4.Theinterruptserviceroutine（ISR）runsonthehostprocessortowhichtheinterruptwasrouted,whichdisablesfurtherinterruptsfromthenetworkadapter.TheISRthenschedulestheminiportadapter’sDPCtorunonaspecificprocessor—usuallythes