Infiniband configuration and testing
Add Infiniband modules to /etc/modules
Either vi or echo/append them to /etc/modules
modprobe modules to dynamically load them without restart
### Set the IB modules they may wish to use (these are just some of the available modules, but should get them started):
IBMOD="ib_umad mlx4_core mlx4_ib mlx4_en ib_ipoib ib_ucm ib_uverbs ib_umad ib_cm ib_sa ib_mad ib_core ib_addr"
Load the modules (now and during next boot).
for i in $IBMOD ; do echo $i >> /etc/modules; modprobe $i; done
Test IP performance
iperf
tuning
http://pve.proxmox.com/wiki/Infiniband
sudo apt-get install iperf
Server
iperf -s
Client
iperf -c <address of server>
Result
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.10 GBytes 944 Mbits/sec
Either vi or echo/append them to /etc/modules
modprobe modules to dynamically load them without restart
### Set the IB modules they may wish to use (these are just some of the available modules, but should get them started):
IBMOD="ib_umad mlx4_core mlx4_ib mlx4_en ib_ipoib ib_ucm ib_uverbs ib_umad ib_cm ib_sa ib_mad ib_core ib_addr"
Load the modules (now and during next boot).
for i in $IBMOD ; do echo $i >> /etc/modules; modprobe $i; done
Install opensm
sudo apt-get install opensm
Start opensm
service opensm start
Check it's running
ps aux | grep opensm
Determine the IB card model:
lspci -d 15b3:
Query IB interface status
ibstat
ibstatus
ibhost
ibswitches
iblinkinfo
Install other software
ibverbs-utils
libibcm1
librdmacm1
libaio1
Add/configure interface
vi /etc/network/interfaces and add
auto ib0
iface ib0 inet static
address
netmask
network
broadcast
mtu 65520
pre-up echo connected > /sys/class/net/ib0/mode
Up the interface
ifup ib0
Test link
ibping is icmp ping
Server
ibping -S
Client
ibping -G <server port guid>
Test RDMA latency
Server
ib_rdma_lat
Client
ib_rdma_bw <hostname-of-server>
above from http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-4.html#ss4.2
Test IP performance
iperf
tuning
http://pve.proxmox.com/wiki/Infiniband
sudo apt-get install iperf
Server
iperf -s
Client
iperf -c <address of server>
Result
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.10 GBytes 944 Mbits/sec
Comments