Remote Procedure Call over QUIC Version 1

RPC-over-QUIC Framework RPC is first and foremost a message-passing protocol. This section covers the implementaion details of exchanging RPC messages over QUIC. Readers should already be familiar with the fundamentals of ONC RPC . RPC-over-QUIC relies on QUIC version 1 as the underlying transport . The use of other QUIC transport versions with RPC MAY be defined by future specifications.

Establishing a Connection When a network host wishes to send RPC requests to a remote service via QUICv1, it must first find an established QUICv1 connection, or establish a new one. For the purpose of explanation, this document refers to the peer that initiates QUICv1 connection establishment as an "RPC client" peer. This document refers to the peer that passively accepts an incoming connection request as an "RPC server" peer. QUICv1 connections are not completely defined by the classic 5-tuple (IP proto, source address, source port, destination address, and destination port). Each connection is also defined by its QUIC connection ID. For instance, if the IP address of either peer should change, or a NAT/PAT binding and the source UDP port changes, the receiver can still recognize an ingress QUICv1 packet as belonging to an established connection. As a result, due to network conditions or administrative actions, an RPC-over-QUIC connection can be replaced (a reconnect event) or migrated (a failover event) without interrupting the operation of an upper layer protocol such as RPC-over-QUIC. A more complete discussion can be found in .

Connection Transport Parameters When establishing a connection, peers exchange transport parameters, as described in .

Initial Flow Control Limits These limits control the amount of data that each peer may send on a newly-created stream. The limits are used for flow control and cap the amount of memory needed by both peers to keep data flowing on the connection. The value of these limits are typically based on the bandwidth-delay of the physical link between the peers, and are not exposed to RPC applications.

Number of Streams Per Connection Each QUICv1 peer may limit the number of streams per connection; see . Given the definition of RPC message framing in , it is possible for an RPC client to create a stream, send one RPC Call, receive one RPC Reply, then destroy the stream. That usage might be common with simple RPC-based protocols like rpcbind. For protocols that carry a more intensive workload, this style of stream allocation generates needless overhead. Moreover, stream identifiers cannot be re-used on a single QUICv1 connection, so eventually a QUICv1 connection can no longer create a new stream for each RPC XID. Finally, a connection peer may advertise a max_streams value that is significantly lower than 2 ^ 60. Instead, RPC clients may create enough streams to maximize workload parallism, and should avoid sending only a few RPCs on each stream before creating a new one. For example, an RPC client could allocate a handful of streams per CPU core to reduce contention for the streams and their associated data structures. Or, an RPC client could create a set of streams whose count is the same as the number of slots in an NFSv4.1 session. Even so, to provide a framework that makes implementing RPC-over-QUIC as fast and simple as possible, this specification needs to focus on enabling the use of as few as a single stream per connection. Servers that implement RPC-over-QUIC must be mindful that each additional stream amounts to incremental overhead. RPC servers MAY deny the creation of new streams if an RPC client already has many active streams. RPC clients need to be prepared for that behavior.

Maximum Frame Size This size is the largest QUIC frame that can appear in any stream on this connection. The QUIC framing protocol is not visible to the RPC application. The RPC client and server can therefore negotiate a frame size that enables efficient transit of RPC traffic with minimal internal memory fragmentation.

RPC Service Discovery For RPC, the destination port is special. RPC services may use a standardized destination port that is bound to an RPC program number. Such ports are assigned in the IANA Service Name and Transport Protocol Port Number registry . For example, the rpcbind program, which is RPC program 100000, listens on port 111. This is done so that RPC clients can always contact the rpcbind service and discover the other RPC services that are operating on that network peer. In other cases, an RPC service might use any available port. The RPC server registers its port number with the local rpcbind service so that RPC clients can contact that service. This mechanism is no different for RPC-over-QUIC than it is for RPC on other network transports. rpcbind clients specify an RPC program number and either the "quic" or "quic6" netid when requesting information about a QUIC-based RPC service. More detail is available in .

Transport Layer Security During connection establishment, the client peer indicates RPC-over-QUIC support by presenting the ALPN token "sunrpc" in the TLS handshake. Support for other application-layer protocols MAY be offered in the same handshake. As part of establishing a QUICv1 connection, the two connecting peers authenticate to each other and choose encryption parameters to establish a confidential channel of communication. All traffic on one QUICv1 connection is thus bound to the authenticated identities that were presented during connection establishment. These peer identities apply to the streams and RPC messages carried by that connection. RPC-over-QUIC provides peer authentication and encryption services using a framework based on Transport Layer Security (TLS). Ergo, RPC-over-QUIC inherently fulfills many of the requirements of . The details of QUIC's use of TLS are specified in . In particular:

With QUICv1, security at the transport layer is always enabled. Therefore, the discussion in about the opportunistic use of TLS does not apply to RPC-over-QUIC, and the STARTTLS mechanism described in MUST NOT be used on RPC-over-QUIC connections.
The peer authentication requirements in apply to RPC-over-QUIC.
The PKIX Extended Key Usage values defined in are valid for use with RPC-over-QUIC.
The ALPN defined in is also used for RPC-over-QUIC.

QUIC Streams RPC-over-QUIC connections are mediated entirely by each peer's RPC layer and, aside from authentication and connection transport parameters, are not otherwise visible to RPC applications. An RPC client establishes an RPC-over-QUIC connection whenever there are application RPC transactions to be executed. QUICv1 provides a "stream" abstraction, described in . A QUICv1 connection carries one or more streams. Once a QUICv1 connection has been established, either connection peer may create a stream. Typically, the RPC client peer creates the first stream on a connection. Unless explicitly specified, when RPC upper layer protocol specifications refer to a "connection", for RPC-over-QUIC, this is a QUIC stream. As an example, an NFSv4.1 BIND_CONN_TO_SESSION operation binds to a QUICv1 stream. As another example, to signify the loss of an RPC request, an NFS server closes the QUICv1 stream that received that request, but it does close not the encompassing QUICv1 connection. In terms of TI-RPC semantic labels, a QUICv1 stream behaves as a "tpi_cots_ord" transport: connection-oriented and in order.

RPC Message Framing RPC-over-QUIC uses only bidirectional streams. When a connection peer creates a QUICv1 stream, that peer's stream endpoint is referred to as a "Requester", and MUST emit only RPC Call messages on that stream. The other endpoint is referred to as a "Responder", and MUST emit only RPC Reply messages on that stream. Receivers MUST silently discard RPC messages whose direction field does not match its Requester/Responder role. Requesters and Responders match RPC Calls to RPC Replies using the XID carried in each RPC message. Responders MUST send RPC Replies on the same stream on which they received the matching RPC Call. Each QUICv1 stream provides reliable in-order delivery of bytes. However, each stream makes no guarantees about delivery order with regard to bytes on other streams on the same connection. The stream data containing RPC records is carried by QUIC STREAM frames, but this framing is invisible to the RPC layer. The transport layer buffers and orders received stream data, exposing only a reliable byte stream to the RPC layer. Although QUIC permits out-of-order delivery within a stream, RPC-over-QUIC does not make use of this feature. Because each QUICv1 stream is an ordered-byte stream, an RPC-with-QUIC stream carries only a sequence of complete RPC messages. Although data from multiple streams can be interleaved on a single QUICv1 connection, RPC messages MUST NOT be interleaved on one stream. Just as with RPC on a TCP socket, each RPC message is an ordered sequence of one or more records on a single stream. Such RPC records bear no relationship to QUIC stream frames; in fact, stream frames as defined in are not visible to RPC endpoints. Each RPC record begins with a four-octet record marker. A record marker contains the count of octets in the record in its lower 31 bits, and a flag that indicates whether the record is the last record in the RPC message in the highest order bit. See for a comparison with TCP record markers.

Receiver Data Placement Assistance One recurring weakness with RPC on TCP is that large payloads (for instance, in NFS WRITEs) can land at arbitrary offsets in receive buffers, limiting the ability for receivers to handle the payloads with zero-touch tactics such as direct I/O. It remains an open question whether RPC-over-QUIC should implement RDMA-like features or features that simply provide help with data placement on receivers. Possibilities include:

A single additional integer giving the offset of a payload, serving only as a hint;
Include references to separate streams in the same connection that contain opaque payloads, similar to RDMA chunks; this would presume that it is valid for some streams on a QUIC connection to carry traffic that is not in the form of an RPC message sequence.

Long-term there could be interest in supporting RDMA over QUIC. Direct data placement over TCP can already be accomplished today using MPA/DDP protocols (formerly known as iWARP; see ). Using a software iWARP implementation means no special hardware is required. If the MPA/DDP protocols themselves can be made to operate directly on QUIC transports, much of the need for a separate RPC-over-QUIC becomes moot. It would bring transport layer security to other RDMA-enabled protocols, such as RPC-over-RDMA .

QUIC Load Balancing Large-scale RPC deployments often distribute incoming connections across multiple backend servers using load balancers. The QUIC Load Balancing specification defines standardized methods for encoding routing information in QUIC connection IDs, enabling stateless or low-state load balancing even when clients migrate to new network addresses. QUIC-LB provides several advantages for RPC server pools:

Load balancers can route all packets for a given RPC-over-QUIC connection to the same backend server by extracting the server ID from the connection ID, even as the client’s network address changes due to NAT rebinding or deliberate migration.
Because routing decisions are encoded directly in connection IDs, load balancers can operate with minimal or no per-connection state, improving scalability and resilience to load balancer failures or restarts.
Since RPC-over-QUIC may use multiple streams within a single QUIC connection (see ), QUIC-LB ensures that all streams within a connection are consistently routed to the same server, preserving the connection-level semantics that upper-layer RPC protocols may depend upon.
The connection ID length self-encoding feature of QUIC-LB, when enabled, assists hardware cryptographic offload devices that need to efficiently look up connection-specific keys, improving performance in high-throughput RPC deployments.

RPC-over-QUIC implementations MAY use QUIC-LB to facilitate load balancing in RPC server pool deployments. A full specification of this facility is beyond the scope of the current document. QUIC-LB is transparent to QUIC clients. They do not need to know whether servers are using QUIC-LB encoding. Clients simply:

Use server-provided connection IDs as-is
Respond to NEW_CONNECTION_ID frames normally
Perform address migration as permitted by server transport parameters