MetaRoCE is an open RDMA transport protocol designed for large-scale AI workloads on Ethernet networks, with its specification, reference implementation and compliance suite being contributed to the Open Compute Project.
Meta has introduced MetaRoCE, an RDMA transport protocol designed for large-scale AI workloads running over Ethernet. MetaRoCE aims to provide high throughput, low tail latency and resilient communication between accelerators as AI clusters scale. Meta plans to contribute the MetaRoCE specification, reference software implementation and compliance test suite to the Open Compute Project.
Network Interface Cards using MetaRoCE split connections into several logical paths, with each path tracking information such as round-trip time, ECN state and utilisation. This allows MetaRoCE-enabled NICs to make decisions based on the condition of each path independently. MetaRoCE also supports multipathing and out-of-order packet delivery, allowing traffic to use available paths instead of remaining on congested or faulty routes.
MetaRoCE is designed to work over lossy Ethernet networks without requiring Priority Flow Control (PFC) or pause frames. It maintains sequence information for each path to identify missing data and retransmit the required packets. MetaRoCE also combines an ECN-controlled sending mechanism with receiver-driven rate control to manage congestion.
Meta and AMD validated MetaRoCE using an AMD Pensando programmable NIC on a 64-node AMD GPU cluster. During the evaluation, MetaRoCE was compared with RoCEv2 using RCCL collective operations across all-reduce and all-to-all workloads. The evaluation reported higher throughput and shorter flow-completion times compared with the existing solution. MetaRoCE achieved around 86 per cent throughput at a 1 per cent packet-loss rate and continued to deliver bandwidth at a 10 per cent packet-loss rate.
The open approach adopted for MetaRoCE is intended to facilitate integration by third-party companies rather than limiting the technology to Meta’s proprietary architecture. Meta plans to release the MetaRoCE specification and production compliance framework in collaboration with OCP. The company also plans to release libsoftmetaroce, a software implementation developed with DPDK that uses generic Linux UDP sockets and does not require specialised networking hardware.
