RDMA_CM(7)

NAME
rdma_cm - RDMA communication manager.

SYNOPSIS
#include <rdma/rdma_cma.h>

DESCRIPTION
Used to establish communication over RDMA transports.

NOTES
The RDMA CM is a communication manager used to setup reliable, connected and unreliable datagram data transfers. It provides an RDMA transport neutral interface for establishing connections. The API is based on sockets, but adapted for queue pair (QP) based semantics: communication must be over a specific RDMA device, and data transfers are message based.

The RDMA CM only provides the communication management (connection setup / teardown) portion of an RDMA API. It works in conjunction with the verbs API defined by the libibverbs library. The libibverbs library provides the interfaces needed to send and receive data.

CLIENT OPERATION
This section provides a general overview of the basic operation for the active, or client, side of communication. A general connection flow would be:

rdma_create_event_channel
    create channel to receive events

rdma_create_id
    allocate an rdma_cm_id, this is conceptually similar to a socket

rdma_resolve_addr
    obtain a local RDMA device to reach the remote address

rdma_get_cm_event
    wait for RDMA_CM_EVENT_ADDR_RESOLVED event

rdma_ack_cm_event
    ack event

rdma_create_qp
    allocate a QP for the communication

rdma_resolve_route
    determine the route to the remote address

rdma_get_cm_event
    wait for RDMA_CM_EVENT_ROUTE_RESOLVED event

rdma_ack_cm_event
    ack event

rdma_connect
    connect to the remote server

rdma_get_cm_event
    wait for RDMA_CM_EVENT_ESTABLISHED event

rdma_ack_cm_event
    ack event

Perform data transfers over connection

rdma_disconnect
    tear-down connection

rdma_get_cm_event
    wait for RDMA_CM_EVENT_DISCONNECTED event

rdma_ack_cm_event
    ack event

rdma_destroy_qp
    destroy the QP

rdma_destroy_id
    release the rdma_cm_id

rdma_destroy_event_channel
    release the event channel

An almost identical process is used to setup unreliable datagram (UD) communication between nodes. No actual connection is formed between QPs however, so disconnection is not needed.

Although this example shows the client initiating the disconnect, either side of a connection may initiate the disconnect.

SERVER OPERATION
This section provides a general overview of the basic operation for the passive, or server, side of communication. A general connection flow would be:

rdma_create_event_channel
    create channel to receive events

rdma_create_id
    allocate an rdma_cm_id, this is conceptually similar to a socket

rdma_bind_addr
    set the local port number to listen on

rdma_listen
    begin listening for connection requests

rdma_get_cm_event
    wait for RDMA_CM_EVENT_CONNECT_REQUEST event with a new rdma_cm_id

rdma_create_qp
    allocate a QP for the communication on the new rdma_cm_id

rdma_accept
    accept the connection request

rdma_ack_cm_event
    ack event

rdma_get_cm_event
    wait for RDMA_CM_EVENT_ESTABLISHED event

rdma_ack_cm_event
    ack event

Perform data transfers over connection

rdma_get_cm_event
    wait for RDMA_CM_EVENT_DISCONNECTED event

rdma_ack_cm_event
    ack event

rdma_disconnect
    tear-down connection

rdma_destroy_qp
    destroy the QP

rdma_destroy_id
    release the connected rdma_cm_id

rdma_destroy_id
    release the listening rdma_cm_id

rdma_destroy_event_channel
    release the event channel

RETURN CODES

=  0
    success

= -1
    error - see errno for more details

Librdmacm functions return 0 to indicate success, and a -1 return value to indicate failure. If a function operates asynchronously, a return value of 0 means that the operation was successfully started. The operation could still complete in error; users should check the status of the related event. If the return value is -1, then errno will contain additional information regarding the reason for the failure.

Prior versions of the library would return -errno and not set errno for some cases related to ENOMEM, ENODEV, ENODATA, EINVAL, and EADDRNOTAVAIL codes. Applications that want to check these codes and have compatability with prior library versions must manually set errno to the negative of the return code if it is < -1.

SEE ALSO
rdma_create_event_channel(3), rdma_get_cm_event(3), rdma_create_id(3), rdma_resolve_addr(3), rdma_bind_addr(3), rdma_create_qp(3), rdma_resolve_route(3), rdma_connect(3), rdma_listen(3), rdma_accept(3), rdma_reject(3), rdma_join_multicast(3), rdma_leave_multicast(3), rdma_notify(3), rdma_ack_cm_event(3), rdma_disconnect(3), rdma_destroy_qp(3), rdma_destroy_id(3), rdma_destroy_event_channel(3), rdma_get_devices(3), rdma_free_devices(3), rdma_get_peer_addr(3), rdma_get_local_addr(3), rdma_get_dst_port(3), rdma_get_src_port(3), rdma_set_option(3) ucmatose(1), udaddy(1), mckey(1), rping(1)