Infiniband configuration and testing

Add Infiniband modules to /etc/modules

Either vi or echo/append them to /etc/modules
modprobe modules to dynamically load them without restart

### Set the IB modules they may wish to use (these are just some of the available modules, but should get them started):
IBMOD="ib_umad mlx4_core mlx4_ib mlx4_en ib_ipoib ib_ucm ib_uverbs ib_umad ib_cm ib_sa ib_mad ib_core ib_addr"

Load the modules (now and during next boot).
for i in $IBMOD ; do echo $i >> /etc/modules; modprobe $i; done

Install opensm
sudo apt-get install opensm

Start opensm
service opensm start

Check it's running
ps aux | grep opensm

Determine the IB card model:
lspci -d 15b3:

Query IB interface status
ibstat
ibstatus
ibhost
ibswitches
iblinkinfo







Install other software





ibverbs-utils



libibcm1



librdmacm1



libaio1









Add/configure interface 



vi /etc/network/interfaces and add









auto ib0



iface ib0 inet static



         address 



         netmask 



         network 



         broadcast 



         mtu 65520



         pre-up echo connected > /sys/class/net/ib0/mode









Up the interface



ifup ib0







Test link



ibping is icmp ping



Server



ibping -S



Client




ibping -G <server port guid>




Test RDMA latency


Server


ib_rdma_lat


Client


ib_rdma_bw <hostname-of-server>




above from http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-4.html#ss4.2





Test IP performance

iperf

tuning

http://pve.proxmox.com/wiki/Infiniband



sudo apt-get install iperf



Server

iperf -s

Client

iperf -c <address of server>


Result

[ ID] Interval       Transfer     Bandwidth

[  3]  0.0-10.0 sec  1.10 GBytes   944 Mbits/sec

































Comments








Post a Comment













Popular posts from this blog









Sites, Newsletters, and Blogs





















BLOGS  Java.net Weblogs  Martin Fowler's blog  Paul Graham  Ruminations of a Programmer  Form Follows Function   Newsletters  Core Java Tech Tips  Enterprise Java Tech Tips  JavaSpecialists.EU  TheServerSide.com   Sites  Java.net Today  Hot Java Sites  CSS Zen Garden   Evernote  Hacker News  Pragmatic Programmer  








Read more










Oracle JDBC ReadTimeout QueryTimeout





















If a query exceeds the oracle.jdbc.ReadTimeout without receiving any data, an exception is thrown and the connection is terminated by the Oracle driver on the client. Unfortunately the session will still be queued on the database and continue to wait for locks, hold any current locks, and complete any DML/PL*SQL procedures that are pending on the server-side of the orphaned connection.   If an application, on a another connection, due to ReadTimeout exception, retries DML/PL*SQL  which requires locks, those queries will queue behind the initial DML/PL*SQL. This will increase the workload exacerbating the situation. On applications with retries, this can be observed by querying the v$session table  or gv$session on RAC and noting new sessions started periodically based on the ReadTimeout interval. The sessions may often have the same SQL_ID and/or SQL_HASH_VALUE.   If stmt.setQueryTimeout(Seconds) is issued and the statement exceeds the timeout, it will atte...








Read more










Locks held on Oracle for hours after sessions abnormally terminated by node failure





















If a session holds db locks and is abnormally terminated (no fin/ack), the locks will persist until the db session is closed, typically around 2 hours and 12 minutes with default network tcp_keepalive settings. Abnormally terminated does not include CTRL C or kill -9. The ojdbc6 driver apparently has a shutdown hook thread that closes the connections in graceful shutdowns. The OS apparently closes the connections when processes are killed. Abnormally terminated might include power failure, firewall failures, Out Of Memory, OS/kernel crash, network connection failure, JBoss or other server node failure, etc.     This was easily reproduced by creating a  process that connected to the db and updated a record but did not commit. While the first process was waiting, a second process with a contending update was started which blocked on the first update. The first clients network cable was then disconnected. The second clients transaction waited for approximately 2 hours until the databa...








Read more