]> TCP RST Diagnostic Payload Orange
mohamed.boucadair@orange.com
Nokia
India kondtir@gmail.com
Tencent
kerneljasonxing@gmail.com
TCP Maintenance and Minor Extensions Service diagnostic This document specifies a diagnostic payload format returned in TCP RST segments. Such payloads are used to share with an endpoint the reasons for which a TCP connection has been reset. Sharing this information is meant to ease diagnostic and troubleshooting. Discussion Venues Discussion of this document takes place on the TCP Maintenance and Minor Extensions mailing list (tcpm@ietf.org), which is archived at . Source for this draft and an issue tracker can be found at .
Introduction A TCP connection can be reset by a peer for various reasons, e.g., received data does not correspond to an active connection. Also, a TCP connection can be reset by an on-path service function (e.g., Carrier Grade NAT (CGN) , NAT64 , or firewall) for several reasons. Typically, a Network Address Translator (NAT) function can generate an RST segment to notify an endpoint upon the expiry of the lifetime of the corresponding mapping entry or because an RST segment was received from a peer (). A TCP connection can also be closed by a user or an application at any time. However, the peer that receives an RST segment does not have any hint about the reason that led to terminating the connection. Likewise, the application that relies upon such a TCP connection may not easily identify the reason for the connection closure. Troubleshooting such events at the remote side of the connection that receives the RST segment may not be trivial. This document fills this void by specifying a format of the diagnostic payload that is returned in an RST segment. Returning such data is consistent with the provision in for RST segments, especially:
"TCP implementations SHOULD allow a received RST segment to include data (SHLD-2)."
This document does not change the conditions under which an RST segment is generated (). The generic procedure for processing an RST segment is specified in . Only the deviations from that procedure to insert and validate a diagnostic payload is provided in . provides a set of examples to illustrate the use of TCP RST diagnostic payloads. This document specifies the format and the overall approach to ease maintaining the list of codes while allowing for adding new codes as needed in the future and accommodating any existing vendor-specific codes. An initial version of error codes is available in Table 2. However, the authoritative source to retrieve the full list of error codes is the IANA-maintained registry ().
  • Design note: Other alternate encoding designs may be considered (TLV, Plain text, etc.); each has their own pros and cons, mainly: amplification impact, need or not of a kernel library and availability of such library (if needed), impact of conversion on CPU, integration with traffic visualisation tools. The encoding will be updated to reflect the WG consensus.
Investigation based on some major CGN vendors revealed that RSTs with data are not discarded and are translated according to any matching mapping entry. Moreover, implementation and experimental validation in Linux are detailed in .
Conventions and Definitions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document makes use of the terms defined in .
RST Diagnostic Payload The RST diagnostic payload MUST be encoded using Concise Binary Object Representation (CBOR) Sequence . The Concise Data Definition Language (CDDL) for the diagnostic payload is shown in .
Structure of the RST Diagnostic Payload
The RST diagnostic payload comprises a magic cookie that is used to unambiguously identify an RST payload that follows this specification. It MUST be set to the RFC number to be assigned to this document.
  • Note to the RFC Editor: Please replace "12345" with the RFC number assigned to this document.
All parameters in the reason component of an RST diagnostic payload are mapped to their CBOR key values as specified in . The description of these parameters is as follows:
reason-code:
This parameter takes a value from an available registry such as the "TCP Failure Causes" registry ().
pen:
Includes a Private Enterprise Number . This parameter MAY be included when the reason code is not taken from the IANA-maintained registry (), but from a vendor-specific registry.
reason-description:
Includes a brief description of the reset reason encoded as UTF-8 . This parameter MUST NOT be included if a reason code is supplied. This parameter is useful only for reset reasons that are not yet registered or for application- specific reset reasons.
At least one of "reason-code" and "reason-description" parameters MUST be included in an RST diagnostic payload. The "pen" parameter MUST be omitted if a reason code from the IANA-maintained registry () fits the reset case. Malformed RST diagnostic payload messages that include the magic cookie MUST be silently ignored by the receiver. A peer that receives a valid diagnostic payload may pass the reset reason information to the local application in addition to the information (MUST-12) described in . That information may also be logged locally, unless a local policy specifies otherwise. How the information is passed to an application and how it is stored locally is implementation-specific. Per , one or more RST segments can be sent to reset a connection. Whether a TCP endpoint elects to send more than one RST with only a subset of them that include the diagnostic payload is implementation-specific.
Some Examples To ease readability, the CBOR diagnostic notation () with the parameter names rather than their CBOR key values in is used in Figures 3, 4, 5, and 6. depicts an example of an RST diagnostic payload that is generated to inform the peer that the TCP connection is reset because an ACK was received from that peer while the connection is still in the LISTEN state ().
Example of an RST Diagnostic Payload with Reason Code (CBOR Encoding)
depicts the same RST diagnostic payload as the one shown in but following the CBOR diagnostic notation.
Example of an RST Diagnostic Payload with Reason Code (Diagnostic Notation)
shows an example of an RST diagnostic payload that includes a free description to report a case that is not covered by an appropriate code from the IANA-maintained registry ().
Example of an RST Diagnostic Payload with Reason Description (Diagnostic Notation)
An RST diagnostic payload may also be sent by an on-path service function. For example, the following diagnostic payload is returned by a NAT function upon expiry of the mapping entry to which the TCP connection is bound ().
Example of an RST Diagnostic Payload to Report Connection Timeout (Diagnostic Notation)
illustrates an RST diagnostic payload that is returned by a peer that resets a TCP connection for a reason code 1234 defined by a vendor with the private enterprise number 32473.
Example of an RST Diagnostic Payload to Report Vendor-Specific Reason Code (Diagnostic Notation)
uses the Enterprise Number 32473 defined for documentation use .
IANA Considerations
RST Diagnostic Payload CBOR Key Values IANA is requested to create a new registry titled "RST Diagnostic Payload CBOR Key Values" under the "Transmission Control Protocol (TCP) Parameters" registry group . The key value MUST be an integer in the 1-255 range. The assignment policy for this registry is "IETF Review" (). The structure of this subregistry and the initial values are provided in . Initial CBOR Keys
Parameter Name CBOR Key CBOR Major Type & Information Reference
reason-code 1 0 unsigned [ThisDocument]
pen 2 0 unsigned [ThisDocument]
reason-description 3 3 text string [ThisDocument]
New Registry for TCP Failure Causes This document requests IANA to create a new registry entitled "TCP Failure Causes" under the "Transmission Control Protocol (TCP) Parameters" registry group . Values are taken from the 1-65535 range. The assignment policy for this registry is "Expert Review" (). The designated experts may approve registration once they checked that the new requested code is not covered by an existing code and if the provided reasoning to register the new code is acceptable. A registration request may supply a pointer to a specification where that code is defined. However, a registration may be accepted even if no permanent and readily available public specification is available. The registry is initially populated with the values listed in . Initial TCP Failure Causes
Value Description Specification (if available)
1 Illegal Option
2 Desynchronized state
3 New data is received after CLOSE is called Sections 3.6.1 and 3.10.7.1 of
4 ABORT Process
5 Unexpected ACK received by non-synchronized state connection
6 Unexpected SYN in the window
7 Unexpected security compartment
8 Malformed Message [ThisDocument]
9 Not Authorized [ThisDocument]
10 Resource Exceeded [ThisDocument]
11 Network Failure [ThisDocument]
12 Reset received from he peer [ThisDocument]
13 Destination Unreachable [ThisDocument]
14 Connection Timeout [ThisDocument]
15 Too much outstanding data
16 Unacceptable performance
17 Middlebox interference
Note that codes in the 8-14 range can be used by service functions (Carrier Grade NAT (CGN), firewall, proxy, etc.).
Security Considerations discusses TCP-related security considerations. In particular, RST-specific attacks and their mitigations are discussed in . In addition to these considerations, it is RECOMMENDED to control the size of acceptable diagnostic payload and keep it as brief as possible. The RECOMMENDED acceptable maximum size of the RST diagnostic payload is 255 octets. Also, it is RECOMMENDED to avoid leaking privacy-related information as part of the diagnostic payload (e.g., including a description such as "user X resets explicitly the connection" is not recommended). The "reason-description" string, when present, MUST NOT include any private information that an observer would not otherwise have access to. The presence of vendor-specific reason codes (Section 3) may be used to fingerprint hosts. Such a concern does not apply if the reason codes are taken from the IANA-maintained registry. Implementers are, thus, encouraged to register new codes within IANA instead of maintaining specific registries. The reason description, when present, MUST NOT be displayed to end users but is intended to be consumed by applications. Such a description may carry a malicious message to mislead the end-user.
References Normative References Transmission Control Protocol (TCP) This document specifies the Transmission Control Protocol (TCP). TCP is an important transport-layer protocol in the Internet protocol stack, and it has continuously evolved over decades of use and growth of the Internet. Over this time, a number of changes have been made to TCP as it was specified in RFC 793, though these have only been documented in a piecemeal fashion. This document collects and brings those changes together with the protocol specification from RFC 793. This document obsoletes RFC 793, as well as RFCs 879, 2873, 6093, 6429, 6528, and 6691 that updated parts of RFC 793. It updates RFCs 1011 and 1122, and it should be considered as a replacement for the portions of those documents dealing with TCP requirements. It also updates RFC 5961 by adding a small clarification in reset handling while in the SYN-RECEIVED state. The TCP header control bits from RFC 793 have also been updated based on RFC 3168. Key words for use in RFCs to Indicate Requirement Levels In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings. Concise Binary Object Representation (CBOR) Sequences This document describes the Concise Binary Object Representation (CBOR) Sequence format and associated media type "application/cbor-seq". A CBOR Sequence consists of any number of encoded CBOR data items, simply concatenated in sequence. Structured syntax suffixes for media types allow other media types to build on them and make it explicit that they are built on an existing media type as their foundation. This specification defines and registers "+cbor-seq" as a structured syntax suffix for CBOR Sequences. Concise Data Definition Language (CDDL): A Notational Convention to Express Concise Binary Object Representation (CBOR) and JSON Data Structures This document proposes a notational convention to express Concise Binary Object Representation (CBOR) data structures (RFC 7049). Its main goal is to provide an easy and unambiguous way to express structures for protocol messages and data formats that use CBOR or JSON. UTF-8, a transformation format of ISO 10646 ISO/IEC 10646-1 defines a large character set called the Universal Character Set (UCS) which encompasses most of the world's writing systems. The originally proposed encodings of the UCS, however, were not compatible with many current applications and protocols, and this has led to the development of UTF-8, the object of this memo. UTF-8 has the characteristic of preserving the full US-ASCII range, providing compatibility with file systems, parsers and other software that rely on US-ASCII values but are transparent to other values. This memo obsoletes and replaces RFC 2279. Concise Binary Object Representation (CBOR) The Concise Binary Object Representation (CBOR) is a data format whose design goals include the possibility of extremely small code size, fairly small message size, and extensibility without the need for version negotiation. These design goals make it different from earlier binary serializations such as ASN.1 and MessagePack. This document obsoletes RFC 7049, providing editorial improvements, new details, and errata fixes while keeping full compatibility with the interchange format of RFC 7049. It does not create a new version of the format. Guidelines for Writing an IANA Considerations Section in RFCs Many protocols make use of points of extensibility that use constants to identify various protocol parameters. To ensure that the values in these fields do not have conflicting uses and to promote interoperability, their allocations are often coordinated by a central record keeper. For IETF protocols, that role is filled by the Internet Assigned Numbers Authority (IANA). To make assignments in a given registry prudently, guidance describing the conditions under which new values should be assigned, as well as when and how modifications to existing values can be made, is needed. This document defines a framework for the documentation of these guidelines by specification authors, in order to assure that the provided guidance for the IANA Considerations is clear and addresses the various issues that are likely in the operation of a registry. This is the third edition of this document; it obsoletes RFC 5226. TCP Extensions for Multipath Operation with Multiple Addresses TCP/IP communication is currently restricted to a single path per connection, yet multiple paths often exist between peers. The simultaneous use of these multiple paths for a TCP/IP session would improve resource usage within the network and thus improve user experience through higher throughput and improved resilience to network failure. Multipath TCP provides the ability to simultaneously use multiple paths between peers. This document presents a set of extensions to traditional TCP to support multipath operation. The protocol offers the same type of service to applications as TCP (i.e., a reliable bytestream), and it provides the components necessary to establish and use multiple TCP flows across potentially disjoint paths. This document specifies v1 of Multipath TCP, obsoleting v0 as specified in RFC 6824, through clarifications and modifications primarily driven by deployment experience. Informative References Transmission Control Protocol (TCP) Parameters Private Enterprise Numbers Common Requirements for Carrier-Grade NATs (CGNs) This document defines common requirements for Carrier-Grade NATs (CGNs). It updates RFC 4787. Stateful NAT64: Network Address and Protocol Translation from IPv6 Clients to IPv4 Servers This document describes stateful NAT64 translation, which allows IPv6-only clients to contact IPv4 servers using unicast UDP, TCP, or ICMP. One or more public IPv4 addresses assigned to a NAT64 translator are shared among several IPv6-only clients. When stateful NAT64 is used in conjunction with DNS64, no changes are usually required in the IPv6 client or the IPv4 server. Updates to Network Address Translation (NAT) Behavioral Requirements This document clarifies and updates several requirements of RFCs 4787, 5382, and 5508 based on operational and development experience. The focus of this document is Network Address Translation from IPv4 to IPv4 (NAT44). This document updates RFCs 4787, 5382, and 5508. Enterprise Number for Documentation Use This document describes an Enterprise Number (also known as SMI Network Management Private Enterprise Code) for use in documentation. This memo provides information for the Internet community. The Constrained Application Protocol (CoAP) The Constrained Application Protocol (CoAP) is a specialized web transfer protocol for use with constrained nodes and constrained (e.g., low-power, lossy) networks. The nodes often have 8-bit microcontrollers with small amounts of ROM and RAM, while constrained networks such as IPv6 over Low-Power Wireless Personal Area Networks (6LoWPANs) often have high packet error rates and a typical throughput of 10s of kbit/s. The protocol is designed for machine- to-machine (M2M) applications such as smart energy and building automation. CoAP provides a request/response interaction model between application endpoints, supports built-in discovery of services and resources, and includes key concepts of the Web such as URIs and Internet media types. CoAP is designed to easily interface with HTTP for integration with the Web while meeting specialized requirements such as multicast support, very low overhead, and simplicity for constrained environments.
Implementation and Experimental Validation in Linux Questions and concerns have been raised regarding whether RST with payload affects the normal termination of flows across different software platforms, operating systems, middleboxes, etc. Even though explicitly allows this behavior, a full implementation is needed to widely verify if unexpected cases can happen in the real world. The overall design in Linux is to pre-allocate a large enough zeroed buffer, put a reset reason code in the first byte and sent it out to verify whether the RST with payload can be possibly declined by any equipment in between two sides and the other side successfully parses the RST with payload.
Implementation The following implementation is accomplished on top of Linux 6.16:
Payload Attachment:
Allocate a 1000-byte data payload attached to all generated RST packets.
Reason Code Encoding:
The first byte of the payload is used to store a predefined reset reason code that is listed in include/net/rstreason.h file, while the remainder of the payload is zero-padded. The reason code is generated by the existing mechanism called TCP reset reasons.
Handling of Reset Types:
The implementation distinguishes between the two primary reset scenarios in tcp_send_active_reset() and tcp_v4_send_reset() respectively:
  • For an Active Reset, initiated proactively by the local system, the payload is placed in the linear area of the socket buffer (sk_buff).
  • For a Passive Reset, sent in response to an unexpected or invalid incoming packet, the payload is stored in the non-linear (paged) area of the sk_buff.
Complete patch is shown in .
Complete Patch sk_mark = 0; ctl_sk->sk_priority = 0; } + memcpy(buffer, (char *)&rep, arg.iov[0].iov_len); + /* put rst reason into the first byte in payload */ + buffer[arg.iov[0].iov_len] = reason; + arg.iov[0].iov_len += PAYLOAD_LEN; ip_send_unicast_reply(ctl_sk, sk, skb, &TCP_SKB_CB(skb)->header.h4.opt, ip_hdr(skb)->saddr, ip_hdr(skb)->daddr, diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index b616776e3354..c07dd009a0de 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -3628,12 +3628,14 @@ void tcp_send_fin(struct sock *sk) void tcp_send_active_reset(struct sock *sk, gfp_t priority, enum sk_rst_reason reason) { + u32 len = MAX_TCP_HEADER + PAYLOAD_LEN; + char payload[PAYLOAD_LEN]; struct sk_buff *skb; TCP_INC_STATS(sock_net(sk), TCP_MIB_OUTRSTS); /* NOTE: No TCP options attached and we never retransmit this. */ - skb = alloc_skb(MAX_TCP_HEADER, priority); + skb = alloc_skb(len, priority); if (!skb) { NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTFAILED); return; @@ -3641,8 +3643,13 @@ void tcp_send_active_reset(struct sock *sk, gfp_t priority, /* Reserve space for headers and prepare control bits. */ skb_reserve(skb, MAX_TCP_HEADER); + skb_put(skb, PAYLOAD_LEN); tcp_init_nondata_skb(skb, tcp_acceptable_seq(sk), TCPHDR_ACK | TCPHDR_RST); + memset(payload, 0, PAYLOAD_LEN); + payload[0] = reason; + skb_store_bits(skb, 0, payload, PAYLOAD_LEN); + TCP_SKB_CB(skb)->end_seq += PAYLOAD_LEN; tcp_mstamp_refresh(tcp_sk(sk)); /* Send it off. */ if (tcp_transmit_skb(sk, skb, 0, priority)) ]]>
Experimental Validation To ensure a thorough evaluation, a multi-layered experimental methodology was designed, progressing from basic functional checks to complex, real-world compatibility and stability tests. The whole implementation has been deployed in Tencent's production environment for almost six months.
Functional Verification The basic functionality test is using iperf or iperf3 to construct a normal termination senario. The tcpdump tool with -X option effectively helps to show the [RST+] flag and the 1000-byte payload, confirming that the kernel correctly generated and transmitted the augmented RST packets. Two servers, designated as Client A and Server B. The test is conducted as following:
  1. Start the iperf3 server on Server B (iperf3 -s).
  2. Initiate a connection from Client A to Server B (iperf3 -c [IP_of_B]).
  3. After the connection is established, one of the iperf3 processes is terminated using the kill command, triggering the kernel to send an RST packet.
  4. Simultaneously, tcpdump is run on either host to capture the reset packet using the filter: 'tcp[tcpflags] & tcp-rst != 0' -X -nn -vv -S.
Compatibility Verification
Hardwares and Kernels:
Tests were conducted on various Linux distributions (e.g., Ubuntu, CentOS) with different kernel versions. The physical hosts were equipped with a range of network interface cards (NICs), including Intel i40e, ixgbe, and Mellanox mlx5.
Virtualization:
The mechanism was tested in a virtualized environment where the VM used a virtio_net driver and the host employed DPDK to redirect packets in the host.
Middleboxes:
Tests were performed with Layer 4 (L4) and Layer 7 (L7) gateways placed between the client and server to verify correct packet parsing and forwarding.
Wide Area Network (WAN):
The setup was tested over long-haul international links to simulate complex conditions, including China-to-Singapore (RTT > 30ms) and China-to-Germany (RTT > 200ms).
In conclusion, across all complex environment tests, the RST packets with payloads were successfully received by the peer. No instances of packets being dropped or mishandled by intermediate middleboxes, gateways, or diverse hardware and software configurations were observed.
Acknowledgments The "diagnostic payload" name is inspired by that was cited by Carsten Bormann in the tcpm mailing list. Thanks to Jon Shallow for the comments. Thanks also to Li Jinghui for the discussion.