ZTE Corporation

China yan.jinjie@zte.com.cn

China Unicom

China hanzx21@chinaunicom.cn

ZTE Corporation

China zhu.xiangyang@zte.com.cn

Routing DetNet This document describes the specific requirements of the Operations, Administration, and Maintenance (OAM) in Enhanced DetNet, and analyzes the gaps with existing OAM methods. The document also presents considerations for potential OAM solutions to address these requirements.

Introduction The framework of OAM for DetNet has been specified in . As per , DetNet functionality is divided into forwarding sub-layer and service sub-layer. lists general functional requirements for DetNet OAM as well as functional requirements in each of the DetNet sub-layers of a DetNet domain. The IP and MPLS DetNet data plane have been defined in and , respectively. The DetNet data plane enhancement requirements are described in  , which calls for enhancements based on the existing bounded latency mechanisms and the corresponding data plane. The data plane enhancement solutions such as ECQF[IEEE 802.1Qdv], Multi-CQF, TCQF, CSQF, TQF, C-SCORE, EDF, gLBFhave been proposed and discussed, with the criteria for classifying them provided in. These queuing mechanisms demand high precision to achieve deterministic latency especially as link speeds increase from 100Mbps to 1Gbps, 10Gbps, 100Gbps, or even higher.  For DetNet OAM, it is essential to provide DetNet services with high precision in a large-scale network, ensuring OAM performance requirements are strictly met. Existing OAM methods including proactive and reactive techniques are insufficient to meet the monitoring and measurement requirements for Enhanced DetNet. Based on these considerations, this document specifies the requirements of the OAM for Enhanced DetNet, analyzes the gaps with the existing OAM methods, and presents considerations for potential OAM solutions.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in .

Terminology The terminology is defined as.

Requirements and Gap Analysis This section presents the enhanced requirements for Enhanced DetNet OAM and analyzes the technical gaps when applying OAM technologies as per in large-scale networks.

Support Microsecond-level Measurement Precision As per , deterministic networks can utilize higher-speed links. With the increasing data rates, the network scheduling cycle can be shortened, provided that the amount of application data transmitted at any given time remains unchanged. This requires more precise time control, in the order of microseconds or sub-microseconds. As per, the service protection provided by the DetNet Service sub-layer aims to mitigate network failures more rapidly than the expected response time of the DetNet Controller Plane. Therefore, the time accuracy of the DetNet OAM mechanism must meet or exceed the accuracy requirements specified in SLAs. For instance, if service delay measurements require microsecond-level precision, the OAM mechanism should support sub-microsecond precision to assess whether service delay and jitter performance comply with SLA requirements. The precision of time measurements depends on both the time synchronization protocol utilized in the network and the specific location where timestamps are captured during the forwarding process. According to , , Enhanced DetNet, which targets large-scale networks, should accommodate time asynchrony, making the efficient one-way delay measurement critical. Furthermore, based on the assumption of time asynchrony, synchronization technologies leveraging network effects, especially in metropolitan area networks with spine-leaf topologies, provide solutions capable of achieving synchronization accuracy down to the microsecond level.

Support Per-packet Performance Monitoring According to , large-scale networks are becoming increasingly complex to meet the growing demands for higher bandwidth, lower latency and jitter, customized traffic prioritization, and SLA-grade network resilience. A more complex network infrastructure requires deeper visibility into Enhanced DetNet. The OAM of Enhanced DetNet should be capable of analyzing network behavior at the packet level. Legacy network monitoring technologies are insufficient since they do not provide real-time and granular visibility required for such detailed analysis. DetNet requires high reliability to meet the demands of service flows. However, detecting SLA violations on a per-packet basis presents significant challenges in large-scale networks with thousands of service flows. The feasibility of implementing per-packet monitoring with the OAM mechanism depends on device capabilities, including counter resources, CPU resources, and port speeds. For instance, if the processing time for forwarding packets at line speed is 10 nanoseconds per packet in in-situ OAM mode, the network processor chip must parse the OAM data fields for each packet within this time frame. Any delay exceeding 10 nanoseconds per packet would hinder the device's capability to forward packets at line speed. In large-scale networks with thousands of services, mainstream equipment currently operates at capacities typically in the range of a few thousand, making it challenging to implement per-packet OAM effectively.

Support High-speed Detection and Response In DetNet, due to the limited network scale, the impact of detection and response time on OAM performance is less noticeable. However, in large-scale networks, long-distance transmission increases latency and the presence of a large number of devices leads to a higher likelihood of network link failures. To achieve real-time monitoring and awareness of the network’s operating state, high-speed monitoring and rapid response are critical for ensuring that the services are not be interrupted for extended periods or that SLA violations do not occur. The traditional detection methods relying on switching and notification are ineffective in detecting millisecond-level latency variations and low-probability packet loss issues, making it unsuitable for Enhanced DetNet, as follows:

1. Active OAM protocols, such as Bidirectional Forwarding Detection (BFD) , typically operate at millisecond-level at most.
2. Currently, in-band detection methods include the Alternate-Marking Method which operates with a detection duration of up to ten-second level, and IOAM Direct Export (DEX), achieving near real-time monitoring but requires packet encapsulated with a sequence number field. Out-of-band reporting methods such as extensions ofgenerally offer millisecond-level delay.
3. Telemetry can provide monitoring at minute or hour levels at most. A key limitation of telemetry is its reliance on uniform packet upload and processing. Moreover, fixed round-trip time overhead across node-to-controller paths, node CPU performance, uplink bandwidth, and controller processing capability all serve as bottlenecks for Telemetry.

Consideration for Enhanced DetNet OAM Solutions OAM methods can be classified into three types according to :

1. Active OAM mode (e.g., TWAMP ).defines DetNet OAM with the MPLS Data Plane, while [I-D.ietf-detnet-ip-oam] discusses DetNet OAM with the IP Data Plane. These drafts focusing on the active OAM mode, indicate that the fate sharing between test packets and service packets is theoretically achievable. However, in practical implementation, it encounters significant challenges. While path sharing is relatively simple, achieving fate sharing between test packets and service packets without causing congestion remains a difficult task for network operators.
2. Passive OAM mode. In a strict sense, passive methods do not modify packet encapsulation, making it difficult to calculate packet loss when packets do not carry sequence numbers.
3. Hybrid OAM mode. At present, the two most popular in-band OAM technologies are the Alternate-Marking (coloring) Method and In-situ Network Telemetry (INT), , which are both hybrid OAM methods and offer solutions for achieving fate-sharing among test packets and service packets on the forwarding plane. Based on forwarding plane detection technologies utilizing coloring and INT, there are two main OAM solutions:
   • The head-end or tail-end node performs local OAM data computations, such as packet loss rates and delay. Out-of-band OAM technologies (such as, STWAP, and their extensions) May optionally be used for information exchange between head-end and tail-end nodes.
   • Telemetry methods (like iFIT ). The head-end and tail-end nodes transmit their local data to a controller or cloud platform via southbound interfaces. Here, third-party nodes then perform OAM-related computations and notify the nodes to generate responses, also via southbound interfaces.

Based on the above analysis, there are two technical approaches for Enhanced DetNet OAM to meet the high-reliability requirements.

Based on Traditional OAM Protection Mechanism Traditional protection switching networks employ the following three types of protection methods, each with different response speeds:

1. End-to-end 1+1 protection: There is a working link serving as the primary path and a protection link as the backup path, both pre-established. At the head-end node, service traffic is replicated and transmitted over both the working link and the protection link. If the working link fails, the tail-end node switches to receiving traffic from the protection link, a process known as single-end switching. While this method switches relatively quickly, it has low bandwidth utilization as both links are allocated for a single service.
2. End-to-end 1:1 protection: Service traffic is transmitted only over the working link, leaving the protection link idle or available for low-priority services. If the working link fails, the tail-end node notifies the head-end node to switch protected service flows from the working link to the protection link. This switching requires actions at both ends, resulting in slower switching but achieving higher bandwidth utilization.
3. Fast Reroute (FRR): Enhanced DetNet requires explicit paths, making rerouting based on distributed protocols unsuitable.

For Enhanced DetNet, where OAM protection is best-effort and primarily focused on deploying protection switching mechanisms, traditional forwarding sub-layer OAM protection methods can be utilized:

1. The INT mechanism which carryies the sequence number and collects data in Direct Export mode offers near real-time detection under limited packet disordering, while the response time depends on the protection method used. This approach achieves optimal detection speed but requires encapsulating/decapsulating in-situ OAM message headers as required and supporting Direct Export Type.
2. The in-situ flow detection mechanism with traffic coloring offers a detection granularity ranging from 1 second to 300 seconds (with an approximate average of 10 seconds), aligning performance monitoring duration within at least the second range.
3. Active OAM mechanisms based on fate sharing compromise the requirements in Chapters 3.1 and 3.2. The CC-CV method relies on protocols such as BFD.

Observability technologies introduce extra latency and are thus unsuitable for real-time OAM detection of online services.

Supporting PREF Protection Function in DetNet PREF aims to guarantee forwarding sub-layer performance through service sub-layer functionality, employing multi-path packet duplication to mitigate the path-switching delay. According to and, ensuring high reliability for time-sensitive services is challenging without the deployment of PREF. When the network supports PREF configuration, it inherently meets reliability requirements including constraints on packet loss rate and latency violation, thus eliminating the need for enhanced detection speed. Another key factor contributing to high reliability is the ability to provide protection at node level and link level rather than solely at the path level. The PREF mechanism requires each packet to be encapsulated with a unique flow-id and a sequence number to facilitate selective receiving. While the network supports PREF can guarantee a packet loss rate as low as 0.0001%, it remains essential to proactively monitor the performance of each individual path to promptly detect path defects. This proactive approach helps prevent SLA violations such as link degradation to ensure stable transmission of service traffic under PREF protection.

Security Considerations TBA

IANA Considerations TBA

Acknowledgements TBA

References Normative References China Mobile Huawei Futurewei Technologies USA ZTE Corporation ETRI New H3C Technologies Huawei Futurewei China Mobile China Telecom LG U+ SK Telecom Futurewei Technologies USA Huawei Technologies University of Surrey ICS Malis Consulting ETRI China Mobile Huawei Technologies Huawei Technologies Huawei Technologies This memo specifies a forwarding method for bounded latency and bounded jitter for Deterministic Networks and is a variant of the IEEE TSN Cyclic Queuing and Forwarding (CQF) method. Tagged CQF (TCQF) supports more than 2 cycles and indicates the cycle number via an existing or new packet header field called the tag to replace the cycle mapping in CQF which is based purely on synchronized reception clock. This memo standardizes TCQF as a mechanism independent of the tagging method used. It also specifies tagging via the (1) the existing MPLS packet Traffic Class (TC) field for MPLS packets, (2) the IP/IPv6 DSCP field for IP/IPv6 packets, and (3) a new TCQF Option header for IPv6 packets. Target benefits of TCQF include low end-to-end jitter, ease of high- speed hardware implementation, optional ability to support large number of flow in large networks via DiffServ style aggregation by applying TCQF to the DetNet aggregate instead of each DetNet flow individually, and support of wide-area DetNet networks with arbitrary link latencies and latency variations as well as low accuracy clock synchronization. Sangmyung University Huawei ZTE Corporation Futurewei Technologies This draft is to facilitate the understanding of the data plane enhancement solutions, which are suggested currently or can be suggested in the future, for deterministic networking. This draft provides criteria for classifying data plane solutions. Examples of each category are listed, along with reasons where necessary. Strengths and limitations of the categories are described. Suitability of the solutions for various services of deterministic networking are also briefly mentioned. Huawei Huawei This document presents a queuing mechanism with multiple cyclic buffers. Huawei Huawei China Mobile Sangmyung University ETRI One of the goals of DetNet is to provide bounded end-to-end latency for critical flows. This document defines how to leverage Segment Routing (SR) to implement bounded latency. Specifically, the SR Identifier (SID) is used to specify transmission time (cycles) of a packet. When forwarding devices along the path follow the instructions carried in the packet, the bounded latency is achieved. This is called Cycle Specified Queuing and Forwarding (CSQF) in this document. Since SR is a source routing technology, no per-flow state is maintained at intermediate and egress nodes, SR-based CSQF naturally supports flow aggregation that is deemed to be a key capability to allow DetNet to scale to large networks. ZTE China Mobile Oxford Brookes University ZTE Beijing Jiaotong University Beijing University of Posts and Telecommunications IP/MPLS networks use packet switching (with the feature store-and- forward) and are based on statistical multiplexing. Statistical multiplexing is essentially a variant of time division multiplexing, which refers to the asynchronous and dynamic allocation of link timeslot resources. In this case, the service flow does not occupy a fixed timeslot, and the length of the timeslot is not fixed, but depends on the size of the packet. Statistical multiplexing has certain challenges and complexity in meeting deterministic QoS, and its delay performance is dependent on the used queueing mechanism. This document further describes a generic time division multiplexing scheme in IP/MPLS networks, which we call timeslot queueing and forwarding (TQF) mechanism. TQF is an enhancement based on TSN TAS, which defines a cyclic period consisting of multiple timeslots, and a flow is assigned to be transmited within one or more dedicated timeslots. ZTE Corporation China Mobile Oxford Brookes University New H3C Technologies Beijing Jiaotong University China Unicom This document describes a deterministic forwarding mechanism to IP/ MPLS network, as well as corresponding resource reservation, admission check, policing, etc, to provide guaranteed latency. Especially, latency compensation with core stateless is discussed to replace reshaping to be suitable for Diff-Serv architecture [RFC2475]. Sangmyung University ETRI ETRI Huawei China Mobile This document specifies the framework and the operational procedure for deterministic networking with work conserving packet schedulers that guarantee end-to-end latency bounds to flows. Schedulers in core nodes of the network do not need to maintain flow states. Instead, the entrance node of a flow marks an ideal service completion time, called Finish Time (FT), of a packet in the packet header. The subsequent core nodes update FT by adding a delay factor, which is a function of the flow and upstream nodes. The packets in the queue are served in the ascending order of the FT. This technique is called Fair Queuing (FQ). The result is that flows are isolated from each other almost perfectly. The latency bound of a flow depends only on the flow's intrinsic parameters, except the link capacities and the maximum packet lengths among other flows sharing each output link with the flow. Futurewei Technologies USA Futurewei Technologies USA Independent ZF Friedrichshafen AG This memo proposes a mechanism called "guaranteed Latency Based Forwarding" (gLBF) as part of DetNet for hop-by-hop packet forwarding with per-hop deterministically bounded latency and minimal jitter. gLBF is intended to be useful across a wide range of networks and applications with need for high-precision deterministic networking services, including in-car networks or networks used for industrial automation across on factory floors, all the way to ++100Gbps country-wide networks. Contrary to other mechanisms, gLBF does not require network wide clock synchronization, nor does it need to maintain per-flow state at network nodes, avoiding drawbacks of other known methods while leveraging their advantages. Specifically, gLBF uses the queuing model and calculus of Urgency Based Scheduling (UBS, [UBS]), which is used by TSN Asynchronous Traffic Shaping [TSN-ATS]. gLBF is intended to be a plug-in replacement for TSN-ATN or as a parallel mechanism beside TSN-ATS because it allows to keeping the same controller-plane design which is selecting paths for TSN-ATS, sizing TSN-ATS queues, calculating latencies and admitting flows to calculated paths for calculated latencies. In addition to reducing the jitter compared to TSN-ATS by additional buffering (dampening) in the network, gLBF also eliminates the need for per-flow, per-hop state maintenance required by TSN-ATS. This avoids the need to signal per-flow state to every hop from the controller-plane and associated scaling problems. It also reduces implementation cost for high-speed networking hardware due to the avoidance of additional high-speed speed read/write memory access to retrieve, process and update per-flow state variables for a large number of flows.