↓Skip to main content

Setting up RoCE on aarch64 Ubuntu

·2 mins

These are notes on setting up RoCE (RDMA over Converged Ethernet) on aarch64 running Ubuntu Server. The instructions below are based off the guide found here https://www.rdmamojo.com/2014/11/08/working-rdma-ubuntu/. However, some package dependencies are not available on arm64. I’ve updated the list to work with the Ubuntu packages available on arm64.

I am going to be using the Utah CloudLab cluster, which contains a bunch of HP Moonshot nodes with the following hardware:

  • HP Moonshot m400
  • Eight 64-bit ARMv8 (Atlas/A57) cores at 2.4 GHz (APM X-GENE)
  • 64GB ECC Memory (8x 8 GB DDR3-1600 SO-DIMMs)
  • 120 GB of flash (SATA3 / M.2, Micron M500)
  • Dual-port Mellanox ConnectX-3 10 GB NIC (PCIe v3.0, 8 lanes)

The nodes are running the following software:

root@node:~$ uname -a
Linux node.nwatkins-qv1183.emulab-net.utah.cloudlab.us 3.13.0-40-generic #69-Ubuntu SMP Thu Nov 13 19:05:44 UTC 2014 aarch64 aarch64 aarch64 GNU/Linux
root@node:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.1 LTS
Release:    14.04
Codename:   trusty

Install the following packages:

root@node:~$ apt-get install libibverbs1 ibverbs-utils librdmacm1 rdmacm-utils libcxgb3-1 libmlx4-1 libmthca1 libnes1 libibcommon1

Next we need to install some kernel modules, as Ubuntu currently doesn’t autoload these.

modprobe rdma_cm
modprobe ib_uverbs
modprobe rdma_ucm
modprobe ib_ucm
modprobe ib_umad
modprobe ib_ipoib
modprobe mlx4_ib
modprobe mlx4_en
modprobe iw_cxgb3
modprobe iw_cxgb4
modprobe iw_nes
modprobe iw_c2

Now we can test it using the guide https://community.mellanox.com/docs/DOC-1465. First, start the server:

nwatkins@node-1:~$ rdma_server
rdma_server: start
rdma_server: end 0

Then run a client:

nwatkins@node-1:~$ rdma_client -s 10.10.1.2
rdma_client: start
rdma_client: end 0
nwatkins@node-1:~$

And… end 0 means success.