Notes on setting up RoCE (RDMA over Converged Ethernet) on aarch64 running Ubuntu Server.
The instructions below are based off the guide found here https://www.rdmamojo.com/2014/11/08/working-rdma-ubuntu/. However, some package dependencies are not available on arm64. I’ve updated the list to work with the Ubuntu packages available on arm64.
I am going to be using the Utah CloudLab cluster, which contains a bunch of HP Moonshot nodes with the following hardware:
- HP Moonshot m400
- Eight 64-bit ARMv8 (Atlas/A57) cores at 2.4 GHz (APM X-GENE)
- 64GB ECC Memory (8x 8 GB DDR3-1600 SO-DIMMs)
- 120 GB of flash (SATA3 / M.2, Micron M500)
- Dual-port Mellanox ConnectX-3 10 GB NIC (PCIe v3.0, 8 lanes)
The nodes are running the following software:
root@node:~$ uname -a
Linux node.nwatkins-qv1183.emulab-net.utah.cloudlab.us 3.13.0-40-generic #69-Ubuntu SMP Thu Nov 13 19:05:44 UTC 2014 aarch64 aarch64 aarch64 GNU/Linux
root@node:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.1 LTS
Release: 14.04
Codename: trusty
Install the following packages:
root@node:~$ apt-get install libibverbs1 ibverbs-utils librdmacm1 rdmacm-utils libcxgb3-1 libmlx4-1 libmthca1 libnes1 libibcommon1
Next we need to install some kernel modules, as Ubuntu currently doesn’t autoload these.
modprobe rdma_cm
modprobe ib_uverbs
modprobe rdma_ucm
modprobe ib_ucm
modprobe ib_umad
modprobe ib_ipoib
modprobe mlx4_ib
modprobe mlx4_en
modprobe iw_cxgb3
modprobe iw_cxgb4
modprobe iw_nes
modprobe iw_c2
Now we can test it using the guide https://community.mellanox.com/docs/DOC-1465. First, start the server:
nwatkins@node-1:~$ rdma_server
rdma_server: start
rdma_server: end 0
Then run a client:
nwatkins@node-1:~$ rdma_client -s 10.10.1.2
rdma_client: start
rdma_client: end 0
nwatkins@node-1:~$
And… end 0 means success.
