Fast Recovery for EVPN Designated Forwarder Election

Fast Recovery for EVPN Designated Forwarder Election Cisco

pbrisset@cisco.com

Cisco

sajassi@cisco.com

Cisco

lburdet@cisco.com

Juniper

jdrake@juniper.net

Nokia

jorge.rabadan@nokia.com

General BESS Working Group EVPN Designated Forwarder Convergence Recovery The Ethernet Virtual Private Network (EVPN) solution provides Designated Forwarder (DF) election procedures for multihomed Ethernet Segments. These procedures have been enhanced further by applying Highest Random Weight (HRW) algorithm for Designated Forwarder election in order to avoid unnecessary DF status changes upon a failure. This document improves these procedures by providing a fast Designated Forwarder election upon recovery of the failed link or node associated with the multihomed Ethernet Segment. This document updates by optionally introducing delays between some of the events therein. The solution is independent of the number of EVPN Instances (EVIs) associated with that Ethernet Segment and it is performed via a simple signaling between the recovered node and each of the other nodes in the multihoming group.

The Ethernet Virtual Private Network (EVPN) solution is becoming pervasive in data center (DC) applications for Network Virtualization Overlay (NVO) and DC inte)rconnect (DCI) services, and in service provider (SP) applications for next generation virtual private LAN services. describes Designated Frowarder (DF) election procedures for multihomed Ethernet Segments. These procedures are enhanced further in by applying the Highest Random Weight (HRW) algorithm for DF election in order to avoid unnecessary DF status changes upon a link or node failure associated with the multihomed Ethernet Segment. This document makes further improvements to the DF election procedures in by providing an option for a fast DF election upon recovery of the failed link or node associated with the multihomed Ethernet Segment. This DF election is achieved independent of the number of EVPN Instances (EVIs) associated with that Ethernet Segment and it is performed via simple signaling between the recovered node and each of the other nodes in the multihomed group.
This document updates the state machine described in , by optionally introducing delays between some events, as further detailed in . The solution is based on a simple one-way signaling mechanism.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

PE:: Provider Edge device.
Designated Forwarder (DF):: A PE that is currently forwarding (encapsulating/decapsulating) traffic for a given VLAN in and out of a site.
EVI:: An EVPN instance spanning the Provider Edge (PE) devices participating in that EVPN.

In EVPN technology, multiple Provider Edge (PE) devices have the ability to encap and decap data belonging to the same VLAN. In certain situations, this may cause L2 duplicates and even loops if there is a momentary overlap of forwarding roles between two or more PE devices, leading to broadcast storms. EVPN currently uses timer based synchronization among PE devices in a redundancy group that can result in duplications (and even loops) because of multiple DFs if the timer is too short or packets being dropped if the timer is too long. Using split-horizon filtering () can prevent loops (but not duplicates). However, if there are overlapping DFs in two different sites at the same time for the same VLAN, the site identifier will be different upon the packet re-entering the Ethernet Segment and hence the split-horizon check will fail, leading to L2 loops. The updated DF procedures in use the well known Highest Random Weight (HRW) algorithm to avoid reshuffling of VLANs among PE devices in the redundancy group upon failure/&wj;recovery. This reduces the impact to VLANs not assigned to the failed/&wj;recovered ports and eliminates loops or duplicates at failure/&wj;recovery events. However, upon PE insertion or a port being newly added to a multihomed Ethernet Segment, HRW also cannot help as a transfer of DF role to the new port must occur while the old DF is still active.

In , when PE2 is inserted in the Ethernet Segment or its CE1-facing interface recovered, PE1 will transfer the DF role of some VLANs to PE2 to achieve load balancing. However, because there is no handshake mechanism between PE1 and PE2, duplication of DF roles for a given VLAN is possible. Duplication of DF roles may eventually lead to duplication of traffic as well as L2 loops. Current EVPN specifications and rely on a timer-based approach for transferring the DF role to the newly inserted device. This can cause the following issues:

Loops/Duplicates if the timer value is too short
Prolonged Traffic Blackholing if the timer value is too long

The clock-synchronization solution presented in this document follows several design principles and presents multiples advantages, namely:

Complicated handshake signamling mechanisms and state machines are avoided in favor of a simple uni-directional signaling approach.
The solution is backwards-compatible (see ), by PEs simply discarding the unrecognized new BGP Extended Community.
Existing DF Election algorithms are supported.
The solution is independent of any BGP delays in propagation of Ethernet Segment routes (Route Type 4)
The solution is agnostic of the actual time synchronization mechanism used.

The solution relies on the concept of common clock alignment between partner PEs participating in a common Ethernet Segment i.e. PE1 and PE2 in . The main idea is to have all peering PEs of that Ethernet Segment perform DF election, and apply the result at the same pre-announced time. The DF Election procedure, as described in and as optionally signalled in , is applied. All PEs attached to a given Ethernet Segment are clock-synchronized using a networking protocol for clock synchronization (e.g., NTP, PTP). When a new PE is inserted in an Ethernet Segment or a failed PE device of the Ethernet Segment recovers, that PE communicates to peering partners the current time plus the value of the timer for partner discovery from step 2 in . This constitutes an "end time" or "absolute time" as seen from the local PE. That absolute time is called the "Service Carving Time" (SCT). A new BGP Extended Community, the Service Carving Timestamp is advertised along with the Ethernet Segment route (RT-4) to communicate the Service Carving Time to other partners. Upon receipt of that new BGP Extended Community, partner PEs can determine the service carving time of the newly insterted PE. The notion of skew is introduced to eliminate any potential duplicate traffic or loops. The receiving partner PEs add a skew (default = -10ms) to the Service Carving Time to enforce this. The previously inserted PE(s) must carve first, followed shortly (skew) by the newly insterted PE. To summarize, all peering PEs carve almost simultaneously at the time announced by the newly added/recovered PE. The newly inserted PE initiates the SCT, and carves immediately on its local timer expiry. The previously inserted PE(s) receiving Ethernet Segment route (RT-4) with a SCT BGP extended community, carve shortly before Service Carving Time.

A new BGP extended community is defined to communicate the Service Carving Timestamp for each Ethernet Segment. A new transitive extended community where the Type field is 0x06, and the Sub-Type is 0x0F is advertised along with the Ethernet Segment route. The expected Service Carving Time is encoded as an 8-octet value as follows:

The timestamp exchanged uses the NTP epoch of January 1, 1900 . As the current NTP era value is not exchanged, a local clock which is "synchronized" but to the wrong era is outside of the scope of this document.
The 64-bit timestamp of NTP consists of a 32-bit part for seconds and a 32-bit part for fractional second:

Timestamp Seconds: 32-bit NTP seconds are encoded in this field.
Timestamp Fractional Seconds: the high order 16 bits of the NTP fractional seconds are encoded in this field. The use of a 16-bit fractional seconds yields adequate precision of 15 microseconds (2^-16 s).

This document introduces a new flag called "T" (for Time Synchronization) to the bitmap field of the DF Election Extended Community defined in .

Bit 3: Time Synchronization (corresponds to Bit 27 of the DF Election Extended Community). When set to 1, it indicates the desire to use Time Synchronization capability with the rest of the PEs in the Ethernet Segment.

This capability is used in conjunction with the agreed upon DF Type (DF Election Type). For example if all the PEs in the Ethernet Segment indicate having Time Synchronization capability and are requesting the DF type to be HRW, then the HRW algorithm is used in conjunction with this capability.

This document introduces an additional delay to the events and transitions defined for the default DF election algorithm FSM in without changing the FSM states or events itself. The peering PE's FSM in DF_DONE which receives a RECV_ES transitions to DF_CALC. Because of the SCT carried in the Ethernet-Segment update, the output of the DF_CALC and transition back into DF_DONE are delayed, as are accompanying forwarding updates to DF/NDF state. The corresponding actions when transitions are performed or states are entered/exited is modified as follows:

DF_CALC on CALCULATED: Mark the election result for the VLAN or Bundle.
1. Where SCT timestamp is present on the RECV_ES event of Action 11, wait until the time indicated by the SCT before continuing to 9.2.
2. Assume a DF/NDF for the local PE for the VLAN or VLAN Bundle, and transition to DF_DONE.

Let's take as an example where initially PE2 had failed and PE1 had taken over. This example shows the problem with the DF&nbhy;Election mechanism in , using the value of the timer configured for all PEs on the Ethernet Segment. Based on and using the default 3 second timer in step 2:

Initial state: PE1 is in steady-state, PE2 is recovering
PE2 recovers at (absolute) time t=99
PE2 advertises RT-4 (sent at t=100) to partner PE1
PE2 starts a 3 second timer to allow the reception of RT-4 from other PE nodes
PE1 carves immediately on RT-4 reception, i.e. t=100 + minimal BGP propagation delay
PE2 carves at time t=103

aims of favouring traffic being dropped over duplicate traffic. With the above procedure, traffic drops will occur as part of each PE recovery sequence since PE1 has transitioned some VLANs to Non-Designated-Forwarder (NDF) immediately upon reception.
The timer value (default = 3 seconds) has a direct effect on the duration of the packets being dropped. A shorter (especially zero) timer may, however, result in duplicate traffic or traffic loops. Based on the Service Carving Time (SCT) approach:

Initial state: PE1 is in steady-state, PE2 is recovering
PE2 recovers at (absolute) time t=99
PE2 advertises RT-4 (sent at t=100) with target SCT value t=103 to partner PE1
PE2 starts a 3 second timer to allow the reception of RT-4 from other PE nodes
PE1 starts service carving timer, with remaining time until t=103
Both PE1 and PE2 carve at (absolute) time t=103

In fact, PE1 should carve slightly before PE2 (skew) to maintain the preference of minimal loss over duplicate traffic. The previously inserted PE2 that is recovering performs both transitions DF to NDF and NDF to DF per VLANs at the timer's expiry. Since the goal is to prevent duplicates, the original PE1, which received the SCT will apply:

DF to NDF transition at t=SCT minus skew, where both PEs are NDF for 'skew' amount of time
NDF to DF transition at t=SCT

It is this split-behaviour which ensures a good transition of DF role with contained amount of loss. Using SCT approach, the negative effect of the timer to allow the reception of RT-4 from other PE nodes is mitigated. Furthermore, the BGP Ethernet Segment route (RT-4) transmission delay (from PE2 to PE1) becomes a non-issue. The use of SCT approach remedies the problem associated with this timer: the 3 second timer window is shortened to the order of milliseconds.

In the eventuality 2 or more PEs in a peering Ethernet Segment group are recovering concurrently or roughly the same time, each will advertise a Service Carving Timestamp. This SCT value would correspond to what each recovering PE considers the "end time" for DF Election. A similar situation arises in staggered recovering PEs, when a second PE recovers at rougly a first PE's advertised SCT expiry, and with its own new SCT-2 outside of the initial SCT window. In the case of multiple outstanding DF elections, one requested by each of the recovering PEs, the SCTs must simply be time-ordered and all PEs execute only a single DF Election at the service carving time corresponding to the largest received timestamp value. The DF Election will involve all the active PEs in a single DF Election update. Example:

Initial state: PE1 is in steady-state, all services elected at PE1.
PE2 recovers at time t=100, advertises RT-4 with target SCT value t=103 to partners (PE1)
PE2 starts a 3 second timer to allow the reception of RT-4 from other PE nodes
PE1 starts service carving timer, with remaining time until t=103
PE3 recovers at time t=102, advertises RT-4 with target SCT value t=105 to partners (PE1, PE2)
PE3 starts a 3 second timer to allow the reception of RT-4 from other PE nodes
PE2 cancels the running timer, starts service carving timer with remaining time until t=105
PE1 updates service carving timer, with remaining time until t=105
PE1, PE2 and PE3 carve at (absolute) time t=105

In the eventuality a PE in a Ethernet Segment group recovers during the discovery window specified in , and does not support or advertise the T-bit, then all PEs in the current peering sequence SHALL immediately revert to the default behavior.

Per redundancy group, for the DF election procedures to be globally convergent and unanimous, it is necessary that all the participating PEs agree on the DF Election algorithm to be used. It is, however, possible that some PEs continue to use the existing modulo-based DF election and do not rely on the new SCT BGP extended community. PEs running a baseline DF election mechanism will simply discard the new SCT BGP extended community as unrecognized. A PE can indicate its willingness to support clock-synched carving by signaling the new 'T' DF Election Capability as well as including the new Service Carving Time BGP extended community along with the Ethernet Segment Route (Type-4). In the case where one or more PEs attached to the Ethernet Segment do not signal T=1, all PEs in the Ethernet Segment SHALL revert back to the timer approach. This is especially important in the context of the VLAN shuffling with more than 2 PEs.

The mechanisms in this document use EVPN control plane as defined in . Security considerations described in are equally applicable. For the new SCT Extended Community, attack vectors may be setting the value to zero, to a value in the past or to large times in the future. The procedures in this document address implicitly what occurs with a carving time in the past, as this would be a naturally occurring event with a large BGP propagation delay: the receiving PE SHALL treat the DF Election at the peer as having occurred already, and proceed without starting any carving delay timer. For timestamp values in the future, a rogue PE may be advertising a value inconsistent with its local behaviour. This is no different than a rogue PE setting all its DF Election results inconstently to its peers using (or ignoring adherence to) the procedures from , and the result would similarly be duplicate or dropped traffic. This document uses MPLS and IP-based tunnel technologies to support data plane transport. Security considerations described in and in are equally applicable.

IANA maintains the "EVPN Extended Community Sub-Types" registry set up by . IANA is requested to confirm the First Come First Served assignment as follows:

IANA should replace the field TBD with the date of publicaton of this document as an RFC. IANA maintains the "DF Election Capabilities" registry set up by . IANA is requested to make the following assignment from this registry:

IANA should replace the field TBD with the date of publicaton of this document as an RFC.

In addition to the authors listed on the front page, the following co-authors have also contributed substantially to this document: Gaurav Badoni
Cisco Email: gbadoni@cisco.com Dhananjaya Rao
Cisco Email: dhrao@cisco.com

Authors would like to acknowledge helpful comments and contributions of Satya Mohanty and Bharath Vasudevan. Also thank you to Anoop Ghanwani for his thorough review with valuable comments and corrections.