In-situ OAM raw data export with IPFIX

In-situ Operations, Administration, and Maintenance (IOAM) records operational and telemetry information in the packet while the packet traverses a path between two points in the network. IOAM data fields are defined in . This document discusses how In-situ Operations, Administration, and Maintenance (IOAM) information can be exported in raw format, i.e. uninterpreted format, from network devices to systems, such as monitoring or analytics systems using IPFIX . "Raw export of IOAM data" refers to a mode of operation where a node exports the IOAM data as it is received in the packet. The exporting node neither interprets, aggregates nor reformats the IOAM data before it is exported. Raw export of IOAM data is to support an operational model where the processing and interpretation of IOAM data is decoupled from the operation of encapsulating/updating/decapsulating IOAM data, which is also referred to as IOAM data-plane operation. The figure below shows the separation of concerns for IOAM export: Exporting IOAM data is performed by the "IOAM node" which performs IOAM data-plane operation, whereas the interpretation of IOAM data is performed by the IOAM data processing system. The separation of concerns is to off-load interpretation, aggregation and formatting of IOAM data from the node which performs data-plane operations. In other words, a node which is focused on data-plane operations, i.e. forwarding of packets and handling IOAM data will not be tasked to also interpret the IOAM data, but can leave this task to another system. Note that for scalability reasons, a single IOAM node could choose to export IOAM data to several IOAM data processing systems.

Raw export of IOAM data is typically generated by network devices at the edges of the network. Deployment and use-case dependent, such as in case of direct export or in cases where the operator is interested in dropped packets, raw export of IOAM data may be generated by IOAM transit nodes.

Requirements for raw export of IOAM data: Export all IOAM information contained in a packet. Export a specific IOAM data type - Incremental Trace type, Preallocated Trace type, Proof of Transit type, Edge to Edge type, Direct Export type. Export IOAM trace data associated with a packet, even if that data was never included in a transmitted or received packet in the network, for example in case of direct export. Support coalescing of the IOAM data from multiple packets into a single raw export packet. Support export of additional parts of the packet, other than the IOAM data as part of the raw export. This could be parts of the packet header and/or parts of the packet payload. This additional information provides context to the IOAM data (e.g. to be used for flow identification) and is to enable the IOAM data processing system to perform further analysis on the received data. Report the reason why IOAM data was exported. The "reason for export" is to complement the IOAM data retrieved from the packet. For example, if a packet was dropped by a node due to congestion, it could be helpful to export the IOAM data of this dropped packet along with an indication that the packet that the IOAM data belongs to was dropped due to congestion.

This document discusses raw export of IOAM data using IPFIX. The following is considered out of scope for this document: Protocols other than IPFIX for raw export of IOAM data. Interpretation or aggregation of IOAM data prior to exporting. Configuration of network devices so that they can determine when to generate IOAM reports, and what information to include in those reports. Events that trigger generation of IOAM reports. Selection of particular destinations within distributed telemetry monitoring systems, to which IOAM reports will be sent. Export format for flow statistics or processed/interpreted/aggregated IOAM data.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in . Abbreviations used in this document: Edge to Edge In-situ Operations, Administration, and Maintenance Maximum Transmit Unit Operations, Administration, and Maintenance Proof of Transit

IPFIX, being a generic export protocol, can export any Information Elements as long as they are described in the information model. The IPFIX protocol is well suited for and is defined as the protocol for exporting packet samples in . IPFIX/PSAMP , already define many of the information elements needed for exporting sections of packets needed for deriving context and raw IOAM data export. This document specifies extensions of the IPFIX information model for meeting the requirements in .

The existing IPFIX Information Elements that are required for IOAM raw data export are listed here. Their details are available in IANA's IPFIX registry . The existing IPFIX Information Elements used to carry the sections of the packets including IOAM data within it are as follows: - ipHeaderPacketSection - dataLinkFrameSection The following Information Elements will be used to provide context to the ipHeaderPacketSection and dataLinkFrameSection as described in : - dataLinkFrameType - sectionOffset - sectionExportedOctets The following Information Element will be used to provide forwarding status of the flow and any attached reasons. - forwardingStatus

IOAM data raw export using IPFIX requires a set of new information elements which are described in this section.

Description: This Information Element describes properties associated with an IOAM report. The ioamReportFlags data type is an 8-bit field. The following bits are defined here: Dropped Association - Dropped packet of interest. Congested Queue Association - Indicates the presence of congestion on a monitored queue. Tracked Flow Association – Matched a flow of interest. Reserved IANA is requested to create a new subregistry for IOAM Report Flags and fill it with the initial list from the description. New assignments for IOAM Encapsulation Types are administered by IANA through Expert Review i.e., review by one of a group of experts designated by an IETF Area Director. Abstract Data Type: unsigned8 Data Type Semantics: flags ElementId: TBD1 Status: current

Description: This Information Element specifies the type of encapsulation to interpret ioamPreallocatedTraceData, ioamIncrementalTraceData, ioamE2EData, ioamPOTData, ioamDirectExportData. The following ioamEncapsulationType values are defined here: None : IOAM data follows format defined in GRE : IOAM data follows format defined in IPv6 : IOAM data follows format defined in VXLAN-GPE : IOAM data follows format defined in GENEVE Option: IOAM data follows format defined in GENEVE Next Protocol: IOAM data follows format defined in NSH : IOAM data follows format defined in IANA is requested to create a new subregistry for IOAM Encapsulation Types and fill it with the initial list from the description. New assignments for IOAM Encapsulation Types are administered by IANA through Expert Review i.e., review by one of a group of experts designated by an IETF Area Director. Abstract Data Type: unsigned8 Data Type Semantics: identifier ElementId: TBD2 Status: current

Description: This Information Element carries n octets of IOAM Preallocated Trace data defined in . The format of the data is determined by the ioamEncapsulationType information element, if present. When the ioamEncapsulationType information element is present and has a value other than "None", and with sufficient length, this element may also report octets from subsequent headers and payload. If no ioamEncapsulationType information element is present, then the encapsulation type shall be assumed to be "None" and this information element only contains octets from the IOAM Preallocated Trace Option. Abstract Data Type: octetArray ElementId: TBD3 Status: current

Description: This Information Element carries n octets of IOAM Incremental Trace data defined in . The format of the data is determined by the ioamEncapsulationType information element, if present. When the ioamEncapsulationType information element is present and has a value other than "None", and with sufficient length, this element may also report octets from subsequent headers and payload. If no ioamEncapsulationType information element is present, then the encapsulation type shall be assumed to be "None" and this information element only contains octets from the IOAM Incremental Trace Option. Abstract Data Type: octetArray ElementId: TBD4 Status: current

Description: This Information Element carries n octets of IOAM E2E data defined in . The format of the data is determined by the ioamEncapsulationType information element, if present. When the ioamEncapsulationType information element is present and has a value other than "None", and with sufficient length, this element may also report octets from subsequent headers and payload. If no ioamEncapsulationType information element is present, then the encapsulation type shall be assumed to be "None" and this information element only contains octets from the IOAM Edge-to-Edge Option. Abstract Data Type: octetArray ElementId: TBD5 Status: current

Description: This Information Element carries n octets of IOAM POT data defined in . The format of the data is determined by the ioamEncapsulationType information element, if present. When the ioamEncapsulationType information element is present and has a value other than "None", and with sufficient length, this element may also report octets from subsequent headers and payload. If no ioamEncapsulationType information element is present, then the encapsulation type shall be assumed to be "None" and this information element only contains octets from the IOAM Proof of Transit Option. Abstract Data Type: octetArray ElementId: TBD6 Status: current

Description: This Information Element carries n octets of IOAM Direct Export data defined in . In addition to the fields from the IOAM Direct Export Option header in the packet, this information element includes all of the trace data from the exporting node, based on the IOAM-Trace-Type value. This data is appended inside ioamDirectExportData following the bit order of the IOAM-Trace-Type field, similar to the way that IOAM encapsulating nodes append trace data in Incremental Trace Option headers. The format of the data is determined by the ioamEncapsulationType information element, if present. When the ioamEncapsulationType information element is present and has a value other than "None", and with sufficient length, this element may also report octets from subsequent headers and payload. If no ioamEncapsulationType information element is present, then the encapsulation type shall be assumed to be "None" and this information element only contains octets from the IOAM Direct Export Option plus the corresponding trace data. Abstract Data Type: octetArray ElementId: TBD7 Status: current

Description: This Information Element carries a series of n octets from the IP header of a sampled packet, starting sectionOffset octets into the IP header. However, if no sectionOffset field corresponding to this Information Element is present, then a sectionOffset of zero applies, and the octets MUST be from the start of the IP header. With sufficient length, this element also reports octets from the IP payload. However, full packet capture of arbitrary packet streams is explicitly out of scope per the Security Considerations sections of and . When this Information Element has a fixed length, this MAY include padding octets that are used to fill out that fixed length. When this information element has a variable length, the variable length MAY include up to 3 octets of padding, used to preserve 4-octet alignment of subequent Information Elements or subsequent records within the same set. In either case of fixed or variable length, the amount of populated octets MAY be specified in the sectionExportedOctets field corresponding to this Information Element, in which case the remainder (if any) MUST be padding. If there is no sectionExportedOctets field corresponding to this Information Element, then all octets MUST be populated unless the total length of the IP packet is less than the fixed length of this Information Element, in which case the remainder MUST be padding. Abstract Data Type: octetArray ElementId: TBD8 Status: current

Description: This Information Element carries a series of n octets from the IEEE 802.3 Ethernet frame of a sampled packet, starting after the preamble and start frame delimiter (SFD), plus sectionOffset octets into the frame if there is a sectionOffset field corresponding to this Information Element. With sufficient length, this element also reports octets from the Ethernet payload. However, full packet capture of arbitrary packet streams is explicitly out of scope per the Security Considerations sections of and . When this Information Element has a fixed length, this MAY include padding octets that are used to fill out that fixed length. When this information element has a variable length, the variable length MAY include up to 3 octets of padding, used to preserve 4-octet alignment of subequent Information Elements or subsequent records within the same set. In either case of fixed or variable length, the amount of populated octets MAY be specified in the sectionExportedOctets field corresponding to this Information Element, in which case the remainder (if any) MUST be padding. If there is no sectionExportedOctets field corresponding to this Information Element, then all octets MUST be populated unless the total length of the Ethernet frame is less than the fixed length of this Information Element, in which case the remainder MUST be padding. Abstract Data Type: octetArray ElementId: TBD9 Status: current

This section shows a set of examples of how IOAM information along with other parts of the packet can be carried using IPFIX.

This example shows a fixed length IP packet. IOAM data is part of the ipHeaderPacketSection.

This examples shows a variable length IP packet, with length < 255 bytes. IOAM data is part of the ipHeaderPacketSectionWithPadding.

This examples shows a variable length IP packet, with length > 255 bytes. IOAM data is part of the ipHeaderPacketSectionWithPadding.

This examples shows a variable length Ethernet packet, with length < 255 bytes. IOAM data is part of the ethernetFrameSection.

This examples shows a variable length IP packet with length < 255 bytes and fixed length ioamIncrementalTraceData carried separately.

This examples shows a variable length IP packet with length < 255 bytes and variable length ioamIncrementalTraceData with length < 255 bytes carried separately.

IANA is requested to allocate code points for the following Information Elements in : ioamReportFlags ioamEncapsulationType ioamPreallocatedTraceData ioamIncrementalTraceData ioamE2EData ioamPOTData ioamDirectExportData ipHeaderPacketSectionWithPadding ethernetFrameSection See for further details. IANA is requested to create subregistries for ioamReportFlags defined in and ioamEncapsulationType defined in .

Manageability considerations will be addressed ín a later version of this document.

Security considerations will be addressed ín a later version of this document.

The authors would like to thank Barak Gafni, Tal Mizrahi, Jennifer Lemon, and Aviv Kfir for their thoughts and comments on raw IOAM data export.