SRIOV在intel 82599上的应用.docx
《SRIOV在intel 82599上的应用.docx》由会员分享,可在线阅读,更多相关《SRIOV在intel 82599上的应用.docx(12页珍藏版)》请在冰豆网上搜索。
SRIOV在intel82599上的应用
SR-IOV的配置步骤:
1)BIOS中开启SRIOV功能
因为BIOS需要重启服务器,所以没有截图。
大致的步骤是按F2进入BIOS,进入Advanced
菜单:
PCIconfiguration-INTELVT-D,将这项enable了
PCIconfiguration-SR-IOV,将这项enable了
2)以 root 用户登录服务器,并查看系统内 Intel 82599 网卡的信息
登录后,使用如下命令查看 82599 网卡信息:
[root@server-68.0.dev.ustack.in~]$lspci|grep-iintel|grep-i'ethernet\|network'
05:
00.0Ethernetcontroller:
IntelCorporationEthernet10G2PX520Adapter(rev01)
05:
00.1Ethernetcontroller:
IntelCorporationEthernet10G2PX520Adapter(rev01)
3)修改内核启动选项
VM平台上创建VF网卡,需要内核支持I/O内存管理(IOMMU)功能,但是Fedora17 中默
认不开启此选项,需要用户修改/boot/grub2/grub.cfg文件,实验时使用的linux系统为CentOS系统,默认这个选项不开。
有时一台服务器上安装了多个linux的内核,最好将每个内核启动的参数都加上“intel_iommu=on”
[root@server-68.0.dev.ustack.in~]$cat/etc/grub.conf
#grub.confgeneratedbyanaconda
#
#Notethatyoudonothavetorerungrubaftermakingchangestothisfile
#NOTICE:
Youhavea/bootpartition.Thismeansthat
#allkernelandinitrdpathsarerelativeto/boot/,eg.
#root(hd0,1)
#kernel/vmlinuz-versionroroot=/dev/mapper/ustack_pv-rootvol
#initrd/initrd-[generic-]version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,1)/grub/splash.xpm.gz
hiddenmenu
password--encrypted$1$IiRfupPN$cI9IPTIJ4a88GYyGnbUVM0
titleCentOS(3.12.21-1.el6.x86_64)
root(hd0,1)
kernel/vmlinuz-3.12.21-1.el6.x86_64roroot=/dev/mapper/ustack_pv-rootvolrd_NO_LUKSLANG=en_US.UTF-8nofbquietsplash=quietbiosdevname=0nodmraidrd_LVM_LV=ustack_pv/rootvolnodmraidrd_MD_UUID=c9f93a65:
e38b29c4:
efa4c96d:
a66f546fSYSFONT=latarcyrheb-sun16crashkernel=autoKEYBOARDTYPE=pcKEYTABLE=usrd_NO_DMrd_MD_UUID=2604a5ff:
94fb70e8:
26fc9ac8:
05ef6991rhgbquietintel_iommu=on
initrd/initramfs-3.12.21-1.el6.x86_64.img
titleCentOS(3.14.1-1.el6.x86_64)
root(hd0,1)
kernel/vmlinuz-3.14.1-1.el6.x86_64roroot=/dev/mapper/ustack_pv-rootvolrd_NO_LUKSLANG=en_US.UTF-8nofbquietsplash=quietbiosdevname=0nodmraidrd_LVM_LV=ustack_pv/rootvolnodmraidrd_MD_UUID=c9f93a65:
e38b29c4:
efa4c96d:
a66f546fSYSFONT=latarcyrheb-sun16crashkernel=autoKEYBOARDTYPE=pcKEYTABLE=usrd_NO_DMrd_MD_UUID=2604a5ff:
94fb70e8:
26fc9ac8:
05ef6991rhgbquietintel_iommu=on
initrd/initramfs-3.14.1-1.el6.x86_64.img
4)在/etc/modprobe.d目录下创建ixgbe.conf文件
,用于在加载Intel 82599 网卡时携带max_vfs参数。
intel新版本的 82599 网卡驱动中,例如 3.15.1 中,支持“max_vfs=W,X,Y,Z……”形式的参数,在系统内对应的物理端口上生成对应数量的VF网卡,对应的顺序与使用命令“lspci | grep –i intel |grep -i 'ethernet\|network'”看到的顺序相同
/etc/modprobe.d/ixgbe.conf 文件
[root@server-68.0.dev.ustack.in~]$cat/etc/modprobe.d/ixgbe.conf
optionsixgbemax_vfs=8
5)阻止 Fedora17 为虚拟出的 VF 网卡加载驱动
编辑/etc/modprobe.d/blacklist.conf文件,把ixgbevf驱动加入黑名单
[root@server-68.0.dev.ustack.in~]$cat/etc/modprobe.d/blacklist.conf
#
#Listingamoduleherepreventsthehotplugscriptsfromloadingit.
#Usuallythat'dbesothatsomeotherdriverwillbinditinstead,
#nomatterwhichdriverhappenstogetprobedfirst.Sometimesuser
#modetoolscanalsocontroldriverbinding.
#
#Syntax:
seemodprobe.conf(5).
#
#watchdogdrivers
blacklisti8xx_tco
#framebufferdrivers
blacklistaty128fb
blacklistatyfb
blacklistradeonfb
blacklisti810fb
blacklistcirrusfb
blacklistintelfb
blacklistkyrofb
blacklisti2c-matroxfb
blacklisthgafb
blacklistnvidiafb
blacklistrivafb
blacklistsavagefb
blacklistsstfb
blacklistneofb
blacklisttridentfb
blacklisttdfxfb
blacklistvirgefb
blacklistvga16fb
blacklistviafb
blacklistixgbevf
#ISDN-seebugs154799,159068
blacklisthisax
blacklisthisax_fcpcipnp
#sounddrivers
blacklistsnd-pcsp
#I/Odynamicconfigurationsupportfors390x(bz#563228)
blacklistchsc_sch
6)为虚拟出的 VF 网卡设定 MAC 地址
需要为虚拟出的 VF 网卡设置 MAC 地址,在保证 MAC 地址不冲突的前提下,VF 网卡才能正常使
用。
需要为每一个 VF 网卡指定一个单独的 MAC 地址,此 MAC 地址的内容并没有严格的限定,用
户可以根据自己的实际组网自行定义。
把设定 MAC 地址的命令保存到 Fedora17 启动执行的脚本
中,确保系统每次启动为 VF 分配的 MAC 地址保持恒定。
创建/etc/rc.d/rc.local文件,
[root@server-68.0.dev.ustack.in~]$cat/etc/rc.d/rc.local
ntpdate10.0.0.251
ntpdate10.0.0.251
puppetagent-vt
iplinkseteth3vf0macb8:
88:
77:
66:
55:
11
iplinkseteth3vf1maca0:
88:
77:
66:
55:
22
iplinkseteth3vf2maca0:
88:
77:
66:
55:
33
iplinkseteth3vf3maca0:
88:
77:
66:
55:
44
iplinkseteth3vf4mace3:
88:
77:
66:
55:
55
iplinkseteth3vf5mace3:
88:
77:
66:
55:
66
iplinkseteth3vf6mace3:
88:
77:
66:
55:
77
iplinkseteth3vf7mace3:
88:
77:
66:
55:
88
iplinkseteth2vf0mac66:
88:
77:
66:
55:
11
iplinkseteth2vf1mac66:
88:
77:
66:
55:
22
iplinkseteth2vf2mac66:
88:
77:
66:
55:
33
iplinkseteth2vf3mac66:
88:
77:
66:
55:
44
iplinkseteth2vf4mac66:
88:
77:
66:
55:
55
iplinkseteth2vf5mac66:
88:
77:
66:
55:
66
iplinkseteth2vf6mac66:
88:
77:
66:
55:
77
iplinkseteth2vf7mac66:
88:
77:
66:
55:
88
exit0
注意:
之所以为eth2和eth3这2个接口设置VF,是因为lspci–v|grep-iethernet,看到的PCI设备编号为0000:
05:
00,
使用ethtool–ieth2,ethtool-ieth3,看到的PCI设备编号与上面的intel82599网卡对应,此网卡有两个万兆接口。
[root@server-68.0.dev.ustack.in~]$iplinkshoweth2
100:
eth2:
mtu1500qdiscmqstateUPmodeDEFAULTgroupdefaultqlen1000
link/ethera0:
36:
9f:
2f:
dc:
60brdff:
ff:
ff:
ff:
ff:
ff
vf0MAC00:
00:
00:
00:
00:
00,spoofcheckingon,link-stateauto
vf1MACe6:
ae:
a7:
d9:
9c:
0c,spoofcheckingon,link-stateauto
vf2MAC00:
00:
00:
00:
00:
00,spoofcheckingon,link-stateauto
vf3MACb6:
90:
d1:
d0:
69:
eb,spoofcheckingon,link-stateauto
vf4MAC00:
00:
00:
00:
00:
00,spoofcheckingon,link-stateauto
vf5MAC00:
00:
00:
00:
00:
00,spoofcheckingon,link-stateauto
vf6MAC00:
00:
00:
00:
00:
00,spoofcheckingon,link-stateauto
vf7MAC6e:
8f:
2c:
eb:
a5:
8c,spoofcheckingon,link-stateauto
[root@server-68.0.dev.ustack.in~]$iplinkshoweth3
99:
eth3:
mtu2700qdiscmqmasterovs-systemstateUPmodeDEFAULTgroupdefaultqlen1000
link/ethera0:
36:
9f:
2f:
dc:
62brdff:
ff:
ff:
ff:
ff:
ff
vf0MAC00:
00:
00:
00:
00:
00,spoofcheckingon,link-stateauto
vf1MAC5e:
96:
61:
89:
9c:
56,spoofcheckingon,link-stateauto
vf2MAC00:
00:
00:
00:
00:
00,spoofcheckingon,link-stateauto
vf3MAC00:
00:
00:
00:
00:
00,spoofcheckingon,link-stateauto
vf4MAC00:
00:
00:
00:
00:
00,spoofcheckingon,link-stateauto
vf5MACae:
90:
44:
8b:
cf:
c2,spoofcheckingon,link-stateauto
vf6MAC00:
00:
00:
00:
00:
00,spoofcheckingon,link-stateauto
vf7MAC00:
00:
00:
00:
00:
00,spoofcheckingon,link-stateauto
7)设置kvm模块的interruptremap功能,
新建文件/etc/modprobe.d/kvm_iommu_map_guest.conf
[root@server-68.0.dev.ustack.in~]$cat/etc/modprobe.d/kvm_iommu_map_guest.conf
optionskvmallow_unsafe_assigned_interrupts=1
查看kvm内核模块的属性设置:
[root@server-68.0.dev.ustack.in~]$modinfokvm
filename:
/lib/modules/3.12.21-1.el6.x86_64/kernel/arch/x86/kvm/kvm.ko
license:
GPL
author:
Qumranet
srcversion:
4C720F23795EEBD0795CABF
depends:
intree:
Y
vermagic:
3.12.21-1.el6.x86_64SMPmod_unloadmodversions
parm:
ignore_msrs:
bool
parm:
min_timer_period_us:
uint
parm:
tsc_tolerance_ppm:
uint
parm:
allow_unsafe_assigned_interrupts:
Enabledeviceassignmentonplatformswithoutinterruptremappingsupport.(bool)
8)重启服务器,重启完成后,使用lspci–v|grep–iethernet
[root@server-68.0.dev.ustack.in~]$lspci|grep-iintel|grep-i'ethernet\|network'
05:
00.0Ethernetcontroller:
IntelCorporationEthernet10G2PX520Adapter(rev01)
05:
00.1Ethernetcontroller:
IntelCorporationEthernet10G2PX520Adapter(rev01)
05:
10.0Ethernetcontroller:
IntelCorporation82599EthernetControllerVirtualFunction(rev01)
05:
10.1Ethernetcontroller:
IntelCorporation82599EthernetControllerVirtualFunction(rev01)
05:
10.2Ethernetcontroller:
IntelCorporation82599EthernetControllerVirtualFunction(rev01)
05:
10.3Ethernetcontroller:
IntelCorporation82599EthernetControllerVirtualFunction(rev01)
05:
10.4Ethernetcontroller:
IntelCorporation82599EthernetControllerVirtualFunction(rev01)
05:
10.5Ethernetcontroller:
IntelCorporation82599EthernetControllerVirtualFunction(rev01)
05:
10.6Ethernetcontroller:
IntelCorporation82599EthernetControllerVirtualFunction(rev01)
05:
10.7Ethernetcontroller:
IntelCorporation82599EthernetControllerVirtualFunction(rev01)
05:
11.0Ethernetcontroller:
IntelCorporation82599EthernetControllerVirtualFunction(rev01)
05:
11.1Ethernetcontroller:
IntelCorporation82599EthernetControllerVirtualFunction(rev01)
05:
11.2Ethernetcontroller:
IntelCorporation82599EthernetControllerVirtualFunction(rev01)
05:
11.3Ethernetcontroller:
IntelCorporation82599EthernetControllerVirtualFunction(rev01)
05:
11.4Ethernetcontroller:
IntelCorporation82599EthernetControllerVirtualFunction(rev01)
05:
11.5Ethernetcontroller:
IntelCorporation82599EthernetControllerVirtualFunction(rev01)
05:
11.6Ethernetcontroller:
IntelCorporation82599EthernetControllerVirtualFunction(rev01)
05:
11.7Ethernetcontroller:
IntelCorporation82599EthernetControllerVirtualFunction(rev01)
07:
00.0Ethernetcontroller:
IntelCorporation82576GigabitNetworkConnection(rev01)
07:
00.1Ethernetcontroller:
IntelCorporation82576GigabitNetworkConnection(rev01
上图中阴影部分的pci设备就是虚拟出来的VF(virtualfuncion),pci设备编号是05:
10和05:
11,
Function编号从0---7,一共8个VF,与 /etc/modprobe.d/ixgbe.conf 文件中设置的VF的最大数量一致。
9)使用virsh工具查看VF设备,并将VF设备与宿主机deattach,以供虚拟机使用。
1.识别设备
#virshnodedev-list--tree|greppci
2.获取设备xml,此处PCI设备号与上面lspci查看到的必须一致
[root@server-68.0.dev.ustack.in~]$virshnodedev-dumpxmlpci_0000_05_10_0
pci_0000_05_10_0
/sys/devices/pci0000:
00/0000:
00:
09.0/0000:
05:
10.0
pci_0000_00_09_0
pci-stub
0
5
16
0
82599EthernetControllerVirtualFunction
IntelCorporation