Convergence of Congestion
Control from Retained StateThales Alenia Spacenicolas.kuhn.ietf@gmail.comOrangeemile.stephan@orange.comUniversity of AberdeenDepartment of EngineeringFraser Noble BuildingAberdeenAB24 3UEUKgorry@erg.abdn.ac.ukUniversity of AberdeenDepartment of EngineeringFraser Noble BuildingAberdeenAB24 3UEUKr.secchi@erg.abdn.ac.ukPrivate Octopus Inc.huitema@huitema.net
Transport
Internet Engineering Task ForceTransport, Congestion Control, QUICThis document specifies a cautious method for IETF transports that
enables fast startup of congestion control for a wide
range of connections.
It reuses a set of computed congestion control parameters that
are based on previously observed path characteristics between
the same pair of transport endpoints. These parameters
are stored, allowing them to be later used to modify the congestion control behavior
of a subsequent connection.It describes assumptions
and defines requirements for how a
sender utilizes these parameters to provide opportunities for a
connection to more rapidly get up to speed and rapidly utilize available
capacity. It discusses how use of the method impacts the capacity at a
shared network bottleneck and the safe response that is needed after any
indication that the new rate is inappropriate.All Internet transports are required to either
use a Congestion Control (CC) method, or
to constrain their rate of transmission . In 2010,
a survey of alternative CC methods , noted that there
are challenges when a CC method operates across an Internet path with a high and/or
varying Bandwidth-Delay Product (BDP). This mechanism targets a
solution for these challenges.A CC method typically takes time to ramp-up the sending rate,
called the "slow-start phase", informally known as the time to "Get up
to speed". This slow-start phase defines a time in which a sender
intentionally uses less capacity than might be available, with the
intention to avoid or limit overshooting the available capacity for the path.
The slow-start design can increase queuing (latency or jitter) and/or
congestion packet loss for the flow. Any overshoot can have a
detrimental effect on other flows sharing a common bottleneck.
A sender can use a method to observe the rate of acknowledged data,
and seek to avoid overshooting the bottleneck capacity (e.g., Hystart++
).
In the extreme case, an overshoot can result in persistent congestion
with unwanted starvation of
other flows (i.e., preventing other flows
from successfully sharing the capacity at a common bottleneck).This document proposes a CC method that is expected to
reduce the time to complete a transfer
when the transfer sends significantly more data than allowed by the
Initial congestion Window (IW), and
where the BDP of the path is also significantly
more than the IW.
It introduces an alternative method to select initial CC parameters,
that seek to more rapidly and safely grow the sending rate controlled by
the congestion window (CWND). CC methods that are rate-based can make
similar adjustments to their target sending rate.This method is based on temporal sharing (sometimes known as
caching) of a saved set of CC parameters that relate to previous observations
of the same path. The parameters include:
the saved_cwnd for the path and the minimum Round Trip Time (RTT). These
parameters are stored and used to modify the CC
behavior of a subsequent connection between the
same endpoints.When used with the QUIC transport, this provides transport services that resemble
those currently available in TCP, using methods such as TCP Control Block (TCB)
caching.CC parameters are used by Careful Resume for three functions:
Information about the utilised path capacity (saved_cwnd) to
determine an appropriate set of CC parameters for re-using the path.Information to characterize the saved path to
confirm whether the current path
is consistent with a saved path.Information to check the validity of the saved CC parameters, including
the time for which the parameters remain valid."Generally, implementations are advised
to be cautious when using saved CC parameters on a new path",
as stated in .
While this statement has been proposed in the context of QUIC standardization, this advice is appropriate for any IETF transport protocol.
Care is therefore needed
to assure safe use and to be robust to changes
in traffic patterns, network routing, and link/node conditions.
There are cases where using the saved parameters of a previous
connection is not appropriate (e.g., ).Whilst a sender could take optimization decisions without considering
the receiver's preference, there are cases where a receiver
could have information that
is not available at the sender, or might benefit from
understanding that Careful Resume might be used.
In these cases, a receiver
could explicitly ask to enable or inhibit tuning of the CC
when an application initiates a new session or resume an
existing one.
A receiver could also tune policies for using the connection
(e.g., managing the
receiver window or flow credit). Examples where a receiver might request not to use Careful Resume include:
a receiver that can predict the pattern of traffic
(e.g., insight into the volume of data to be sent,
the expected length of a session, or the required maximum transfer rate);a receiver with a local indication that a path/local
interface has changed since the CC parameters were stored;knowledge of the current hardware limitations at a receiver;a receiver that can predict additional capacity will be needed
for other concurrent or later flows
(i.e., prefers to activate Careful Resume for a different connection).A related document proposes an extension for QUIC that allows
sender-generated CC parameters to be stored at the receiver
.
Transferring the information to a receiver releases the need for a
sender to retain transport state for each
receiver, and allows a receiver to express a preference for whether
to use the method. This section provides a set of examples where Careful Resume is expected to improve performance.Either endpoint can assume the role of a
sender or a receiver. Careful Resume also supports a
bidirectional data transfer,
where both endpoints simultaneously send data
(e.g., remote execution of an application, or a
bidirectional video conference call).In one example, an application uses a series of connections over a path
(i.e., resumes a connection to the same endpoint).
Without a new method,
each connection would need to individually
discover appropriate CC parameters, whereas Careful Resume allows the flow
to use a rate that is based on the previously observed CC parameters.In another example, an application connects after a disruption
had temporarily reduced the path
capacity. When the endpoint
returns to use a path with the original characteristics, using
a rate that is based on the previously observed CC parameters.
There is particular benefit for
any path with an RTT that is much larger than typical
Internet paths.
In a specific example, an application connected via a satellite access network
could require 9 seconds to complete a 5.3 MB transfer
using standard CC, whereas a sender using Careful Resume
could reduce this transfer time to 4 seconds. The time to complete a 1 MB transfer could
similarly be reduced by 62 % . This benefit is also
expected for other sizes of transfer and for different path
characteristics when a path has a large BDP.This subsection provides a brief summary of key terms and the
requirements language.The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14
when, and only when, they appear in all
capitals, as shown here. The document uses language drawn
from a range of IETF RFCs. It defines current, and saved values for a set of CC
parameters:
CC parameters: A set of saved congestion control parameters from
a previously observed connection (see ).Careful Resume (CR): The method specified in this document to select
initial CC parameters, that seeks to more rapidly and safely
increase the initial sending rate.current_endpoint_token: The Endpoint Token of the current receiver;current_rtt: A sample measurement of the current RTT;endpoint_token: An Endpoint Token identifying a path to a receiver;flight_size: The currently unacknowledged volume of data sent by the CC method;jump_cwnd: The resumed CWND, used in the Unvalidated Phase.LifeTime: The time for which the saved CC parameters can be safely re-used.max_jump : The maximum configured jump_cwnd;PipeSize: A measure of the validated available capacity based on the acknowledged data;saved_cwnd: A value of CWND derived from observation of a
previous connection, which reflects capacity
that was utilised by the observed connection;saved_endpoint_token: The Endpoint Token of a previous connection to a
receiver;saved_rtt: The preserved minimum RTT, e.g., corresponding to the minimum
of a set RTT of measurements over the last 5 minutes of a connection.Unvalidated Packet: A packet sent when the CWND has been increased
beyond the size normally permitted by the congestion control algorithm;
if such a packet is acknowledged, it contributes to the PipeSize, but
if congestion is detected, it triggers entry to the Safe Retreat Phase.The Endpoint Token is described in .
This section defines a series of phases that the
CC algorithm moves through as a connection
uses Careful Resume, as shown in Figure 1. Connect -> Reconnaissance --------------------> Normal
(Normal) | ^
v |
Unvalidated --------------------------+
| | |
| +--> Validating --------------+
| | |
| | |
+---------------+--> Safe Retreat ---+
]]> Figure 1: Phases when a connection uses the Careful Resume. The Observe Phase
is performed by an established connection as an action within the Normal Phase.
Examples of the transitions between phases are provided in .During a previous established connection, the CC parameters for the specific path
to an endpoint are saved. This characterizes the path and
determines the saved_cwnd.
The saved_cwnd is a measure of the currently utilised capacity for the connection,
measured as the number of bytes sent over a RTT. This could be computed
by measuring the volume of data acknowledged in one RTT.
The CC parameters also include the minimum RTT
(saved_rtt) and the receiver
Endpoint Token (saved_endpoint_token).An implementation
can store the CC parameters at the server (or could exchange this information
with a receiver ).Observe Phase: The sender updates the stored CC parameters
and/or sends the updated CC parameter
information after each observation.
If the measured CWND is less than four times the Initial Window (IW)
(i.e., CWND less than IW*4),
a sender can choose to not store and/or send CC parameters, because the additional actions associated with
performing Careful Resume for a small CWND would not justify the use of the method.Observe Phase (Sending CC Parameters): When sending the CC parameters to a receiver,
these ought to be updated
if there are significant changes in the saved CC parameters;
The frequency of update SHOULD be less than
one update for several RTTs .Implementation notes are provided in . A sender enters the Reconnaissance Phase after connection setup.
In this phase, the CWND is initialised to the IW, and the sender transmits initial data.
The CWND MAY be increased using normal CC as each acknowledgment
confirms delivery of a packet (i.e., the CC is unchanged).The phase seeks to determine if the path is consistent with
a previously observed path (saved in the CC parameters).
There are a set of conditions that need to be confirmed before the sender is
permitted to enter the Unvalidated Phase:
Reconnaissance Phase (Endpoint change):
If the current remote endpoint is not the same as a saved endpoint,
the sender MUST enter the Normal Phase. If the Endpoint Token differs
(i.e., the saved_endpoint_token is different from the
current_endpoint_token), it is assumed to represent a different network path.
The sender also enters the Normal Phase when there are no corresponding saved CC parameters.Reconnaissance Phase (Lifetime of saved CC parameters): The CC parameters are temporal.
If the LifeTime of the observed CC parameters is exceeded ,
the CC parameters are no longer used
and sender enters the Normal Phase.Reconnaissance Phase (Confirming the RTT): During this phase,
a sender MUST record the minimum RTT for the current connection.Reconnaissance Phase (Avoiding using Careful Resume): A receiver can use a method (e.g.,
) to request
that the sender instead enters the Normal Phase.Reconnaissance Phase (Detected congestion): If the sender detects
congestion (e.g., packet loss or ECN-CE marking), the sender does not use the Careful
Resume method and MUST enter the Normal Phase to respond to the detected congestion.
Reconnaissance Phase (Using saved_cwnd):
Only one connection can use a specific set of saved CC parameters.
If another connection has already started to use the saved_cwnd, the sender
MUST enter the Normal Phase.Reconnaissance Phase (Rate-limited sender):
If the sender is rate-limited ,
it might send insufficient data
to be able to validate transmission at the higher rate.
Careful Resume is allowed to remain in the Reconnaissance Phase and to not
transition to the Unvalidated Phase until the sender
has more data ready to send in the transmission buffer
than is permitted by the current CWND.
In some implementations, the decision to enter the
Unvalidated Phase could require coordination with the management
of buffers in the interface to the higher layers.When a sender confirms the path and it
receives an acknowledgement for the initial data without reported congestion,
it MAY then enter the Unvalidated Phase.
This transition occurs when a sender has more data than permitted
by the current CWND.When a path is not confirmed, Careful Resume is not used
and the sender enters the Normal Phase.Implementation requirements are provided in .The Unvalidated Phase is designed to enable the CWND
to more rapidly get up to speed.
The sender only enters this phase when the saved CC parameters are
confirmed:Unvalidated Phase (Confirming the path on entry): The calculation
of a sending rate from a saved_cwnd
is directly impacted by the RTT, therefore a significant change in the RTT
is a strong indication that the previously observed CC
parameters may not be valid for the current path.
If the RTT measurement is
not confirmed, i.e.,
the current_rtt is greater than or equal to
(saved_rtt / 2) or the current_rtt is less than or equal to (saved_rtt x 10)
(see ),
the sender MUST enter the Normal Phase (see trigger rtt_not_validated in ).When the RTT is confirmed:Unvalidated Phase (Initialising PipeSize): The variable PipeSize is initialised to CWND on
entry to the Unvalidated Phase. This records the
value before the jump is applied.Unvalidated Phase (Setting the jump_cwnd):
To avoid starving other flows that could have either started
or increased their used capacity after the Observation Phase,
the jump_cwnd MUST be no more than half of the saved_cwnd.
Hence, jump_cwnd is less than or equal to the (saved_cwnd/2). CWND = jump_cwnd. Unvalidated Phase (Pacing trandmission): Transmission using an unvalidated
CWND MUST use pacing.Unvalidated Phase (Confirming the path during transmission)
If a sender determines that the previous CC parameters
are not valid (due to a detected path change),
the Safe Retreat Phase is entered.
(The sender has not yet received feedback for the jump in CWND, because
less than an RTT has passed before the Unvalidated Phase was entered.
Therefore, any detected congestion must have resulted from packets sent
before the Unvalidated Phase.)Unvalidated Phase (Receiving acknowledgements for reconnaissance packets):
The variable PipeSize is increased by the amount of data that is acknowledged
by each acknowledgment (in bytes). This indicates a previously unacknowledged packet has been
succesfully sent over the path.Unvalidated Phase (Receiving acknowledgements for an unvalidated packet):
The sender enters the Validating Phase when an acknowledgement is
received for the first packet number (or higher) that was sent in the Unvalidated Phase
(see first_unvalidated_packet_acknowledged in ).Implementation notes are provided in . The Validating Phase checks that the packets
sent in the Unvalidated Phase were received without inducing congestion.
The CWND remains unvalidated and the sender typically remains in this phase for one RTT.
Validating Phase (Check flight_size on entry):
On entry to the Validating Phase, if the flight_size is less
equal to the PipeSize (the validated window),
the CWND is reset to the PipeSize and
the Normal Phase is then immediately entered.
There is no need to validate the current data in flight.Validating Phase (Limiting CWND on entry):
On entry to the Validating Phase, the CWND is set to the flight_size.
This corresponds to the capacity being validated
(see trigger rate_limited in ).Validating Phase (Updating CWND): The CWND is updated using the
normal rules for the current congestion controller, this typically
allows CWND to be increased for each ACK received that
indicates a packet has been successfully sent across the path.Validating Phase (Receiving acknowledgements for unvalidated packets):
The variable PipeSize is increased upon each
acknowledgment that indicates a packet has been
successfully sent over the path. This records
the validated PipeSize in bytes.Validating Phase (Congestion indication):
If a sender determines that congestion was experienced
(e.g., packet loss or ECN-CE marking), Careful Resume
enters the Safe Retreat Phase
(see trigger packet_loss and ECN_CE in ).Validating Phase (Receiving acknowledgement for all unvalidated packets):
The sender enters the Normal Phase when an acknowledgement is
received for the last packet number (or higher)
that was sent in the Unvalidated Phase
(see last_unvalidated_packet_acknowledged in ).This phase is entered when the first loss/ECN-CE marking is
detected for unvalidated packets. On entry to the Safe Retreat Phase, the sender MUST
drain the path of any other unvalidated packets before entering the Normal Phase.
(This trigger is the same as used by a QUIC sender to transition
from Slow Start to Recovery .)Safe Retreat Phase (Removing saved information): The set of saved CC parameters for
the path are deleted, to prevent these
from being used again by other flows.
To avoid persistent overshoot, on entering the Normal Phase,
the congestion control algorithm needs to itself determine the path capacity
(e.g., using SS or bandwidth probing), which MUST NOT consider capacity information
deduced from packets sent in the Unvalidated Phase.
Safe Retreat Phase (Re-initializing CC): On entry,
the CWND MUST be reduced to
no more than the (PipeSize/2).
This avoids persistent starvation by allowing capacity for other flows to regain
their share of the total capacity.Note: The minimum CWND in QUIC is 2 packets (see: section 4.8).Safe Retreat Phase (QUIC recovery): When the CWND is reduced,
a QUIC sender can immediately send a single packet prior to the reduction
.
(This speeds up loss
recovery if the data in the lost packet is retransmitted and is
similar to TCP as described in Section 5 of .)Safe Retreat Phase (Increasing CWND):
The CWND MUST NOT be increased in the Safe Retreat Phase.Safe Retreat Phase (Tracking PipeSize):
The sender continues to update
the PipeSize after processing each ACK.
This value is used to
reset the ssthresh when leaving this phase, it does
not modify CWND.Safe Retreat Phase (Receiving acknowledgement for all unvalidated packets):
The sender enters Normal Phase
when the last packet (or later) sent during the
Unvalidated Phase has been acknowledged. The sender
MUST set the ssthresh to no more than the PipeSize
(see exit_recovery in ).Implementation requirements are provided in
.Unacknowledged packets that were sent in the Unvalidated Phase
can be lost when there is congestion.
Loss recovery commences using the reduced CWND
that was set on entry to the Safe Retreat Phase.
NOTE: A TCP or SCTP sender is required to retransmit
all lost data.
For QUIC and DCCP, the need for loss recovery
depends on the sender policy for retransmission.NOTE: During loss recovery, a receiver can cumulatively acknowledge
data that was previously sent in the Unvalidated Phase in addition
to acknowledging successful retransmission of data.
describes how to appropriately
account for such acknowledgments.NOTE: On entry to the Safe Retreat Phase, the CWND can be
significantly reduced, when there was multiple loss,
recovery of all lost data could require multiple RTTs to complete.The sender leaves the Safe Retreat Phase when
the last packet number (or higher) sent in the
Unvalidated Phase is acknowledged.
If the last packet number is not cumulatively acknowledged, then
additional packets might need to be retransmitted.In the Normal Phase, the sender transitions to using
the normal CC method (e.g., in congestion avoidance if CWND is more than ssthresh).
(Note that when the sender did not use the entire jump_cwnd the CWND was reduced
on entering the Validating Phase.Implementation requirements are provided in .A sender that experiences a Retransmission Time Out (RTO) expiry
ceases to use Careful Resume.
The sender continues enters the Normal Phase.
NOTE: As in loss recovery, data sent in the
Unvalidated Phase could be later acknowledged after an RTO event
(see ).This section provides guidance for implementation and use.There are various approaches to measuring the capacity used by a connection.
Congestion controllers, such as Reno or CUBIC , can estimate the
capacity by utilizing the
CWND or flight_size. A different approach could
estimate the same parameters for a rate-based congestion
controller, such as BBR ,
or by observing the rate at which data is acknowledged by the remote endpoint.Implementations are expected to include a
LifeTime parameter in the CC parameters that can be used to remove old CC parameters
when no longer needed, or the CC parameters are out of date.Observe Phase: There are cases where the current CWND
does not reflect the path capacity. At the end of slow
start, the CWND can be significantly larger than
needed to fully utilize the path (i.e., a CWND
overshoot). It is inappropriate to use an
overshoot in the CWND as a basis for estimating the
capacity. In most cases, the CWND will converge to a stable
value after several more RTTs.
One mitigation could be to set the
saved_cwnd based on
the flight_size, or an averaged CWND.Observe Phase (Rate-limited sender): When the sender
is rate-limited, or in an RTT following a burst of
transmission, a sender typically transmits
much less data than allowed. Such observations could be discounted when
estimating the saved_cwnd (e.g., when a previous
observation recorded a higher value.) The CC is not modified during the Reconnaissance Phase.
A sender therefore needs to limit the initial data,
sent in the first RTT of transmitted data,
to not more than the IW .
This transmission using the IW is
assumed to be a safe starting point for any path to avoid
adding excessive load to a potentially congested path.
(When used in a controlled
network, additional information about local path characteristics
could be known, which might be used to configure a non-standard
IW.)The method does not permit multiple concurrent reuse of
the saved CC parameters. When multiple new concurrent connections
are made to a server, each can have a valid endpoint_token,
but the saved_cwnd can only be used for one new connection.
This is designed to prevent a sender from performing multiple jumps in the cwnd,
each individually based on the same saved_cwnd, and hence creating an
excessive aggregate load at the bottleneck.The method used to prevent re-use of the saved CC parameters
will depend on the design of the server that is being used
(e.g., if all connections from a given client IP arrive at the
same server process, then the server process could use a hash table).
A distributed system might be required when using some types of load
balancing, to ensure this invariant when the load balancing hashes
connections by 4-tuple and hence multiple connections from the same
client device are served by different server processes.In the Reconnaissance Phase a sender initiates a connection
and starts sending initial data.
This measures the current minimum RTT.
If a decision is made to use Careful Resume, this is used to confirm the path.Path characteristics can change over time for many reasons,
resulting in the previously observed CC parameters
becoming irrelevant. The sender therefore compares the
saved_RTT with each of a
series of measured RTT samples.If the current RTT sample is less than a half of the
saved_RTT, this is regarded as too small, and is an indicator of a path change.
(This factor of two arises, because the rate should not exceed the observed rate when
the saved_cwnd was measured, because the jump_cwnd is calculated as half the
measured saved_cwnd.)A current RTT larger than that at the time the saved_cwnd was measured results
in a proportionally lower resumed rate, because the transmission using the CR method
is paced based on the current RTT (i.e., the larger RTT samples reduces the paced
sending rate in the Unvalidated Phase - and hence is still safe).Also note that when the RTT is incorrectly measured as larger
than the actual path RTT, the sender will receive an acknowledgment for an
unvalidated packet before it might have completed the Unvalidated Phase, this
acknowledgment resets the CWND to reflect the flight_size, and the sender enters
the Validating Phase.an RTT sample more than ten times the
saved_RTT is indicative of a path change.
(The value of ten was chosen to accommodate both increases in latency from buffering
on a path, and any variation between RTT samples). A sender in Reconnaissance Phase reverts to the Normal Phase if congestion is detected.
Some transport protocols implement methods that infer potential congestion
from an increase in the RTT.
In the Reconnaissance Phase, this indication occurs earlier than congestion
which is reported by
loss or by ECN marking. Designs need to consider if this is
a suitable trigger to revert to the Normal Phase. This section defines the safety requirements
for using saved CC parameters to tentatively update the CWND.
These safety requirements mitigate the risk of
adding excessive congestion to an already congested path.Unvalidated Phase (Jump): A connection must not directly use the previously
saved_cwnd to directly initialize a new flow causing it to resume sending at the same
rate. The jump_cwnd must be no more than half the previously saved_cwnd.The long-term use of the previously observed parameters is not appropriate,
a lifetime therefore needs to be specified during
which the saved CC parameters can be safely re-used. provides guidance on the implementation of
TCP Control Block Interdependence, but does not specify how long a saved
parameter can safely be reused. specifies a method for managing an
unvalidated CWND. This states:
"After a fixed period of time (the non-validated period (NVP)), the sender
adjusts the cwnd (Section 4.4.3). The NVP SHOULD NOT exceed five minutes."
Section 5 of discusses the rationale for
choosing that period.
However, RFC 7661 targets rate-limited connections using normal
CC. The method described in the present specification includes additional
mechanisms to avoid and mitigate the effects of overshoot, and therefore
this can be used to justify a longer lifetime of the saved_cwnd using
the Careful Resume method. The sender must avoid sending a burst of packets greater than IW as a result of a
step-increase in the CWND. (This is consistent with ,
).
Pacing sent packets as a function of
the current RTT, rather than the saved_RTT,
provides an additional safety during the
Unvalidated Phase.
Other sender mitigations have also been suggested to
avoid line-rate bursts (e.g., ).Pacing places a limitation on the minimum acceptable current_RTT
to avoid sending at a rate higher than was previously observed.The following example provides a relevant pacing rhythm using the RTT and the
saved_cwnd. The Inter-packet Transmission Time (ITT) is determined
by using the current Maximum Message Size (MMS),
the saved_cwnd and the RTT. A safety
margin can avoid sending more than a recommended maximum
(max_jump):
jump_cwnd = min(max_jump,saved_cwnd/2)ITT = (current_RTT x MMS)/jump_cwndThis follows the idea presented in ,
and
.Careful Resume has been designed to be robust to
changes in network conditions
due to variations in the forwarding path, such as reconfiguration of
equipment, or changes in the link conditions. This is mitigated
by path confirmation.Careful Resume has been designed to be robust to changes
in network traffic, including the
arrival of new flows that compete for capacity at a shared bottleneck.
This is mitigated by jumping to no more than a half of
the saved_cwnd and by using pacing.Careful Resume has been designed to prevent unduly suppressing flows
that used the capacity since the available capacity was measured. This is further
mitigated by bounding the duration of the Unvalidated Phase (and the following
Validating Phase).When a sender completes the Unvalidated Phase, either by sending a jump_cwnd of data
or after one RTT, it ceases to use the unvalidated CWND. That is, CWND is reset
to the flight_size, and the sender awaits reception of the acknowledgments to validate the
use of this capacity. New packets are sent when previously
sent data is newly acknowledged. The purpose is to trigger a
entry to the Safe Retreat Phase when the capacity is not validated.Note that the CWND is increased during the Validating Phase,
based on received ACKs. This allows new data to be sent,
but this does not have any final impact on the CWND
if congestion is detected.This section defines the safety requirements
after congestion has been detected during the Unvalidated Phase.The Safe Retreat Phase sets a safe CWND value to drain unvalidated packets
from the path when a packet loss is detected or acknowledgments indicate sent
packets were ECN CE-marked.The Safe Retreat reaction differs from a traditional
reaction to detected congestion, because
the jump_cwnd can result in a significantly higher rate than would be allowed by the
slow-start mechanism. This could aggressively feed a congested bottleneck,
resulting in overshoot where a disproportionate number of packets
from existing flows are displaced from
the buffer at the congested bottleneck.
For this reason, a sender needs to react to detected congestion by
reducing CWND significantly
below the saved_cwnd.Note: Proportional Rate Reduction (PRR)
assumes that it is safe to reduce
the rate gradually when in congestion avoidance.
PRR is therefore not appropriate
when there might be significant overshoot in the use of the capacity, which can
be the case when the Safe Retreat Phase is entered.Acknowledgements
for unvalidated packets are tracked to
measure the maximum capacity, called the PipeSize
(The first unvalidated packet can be determined by
recording the sequence number
of the first packet sent in this phase.)
This PipeSize is not a safe measure of the currently
available share of the capacity whenever
there was also a significant overshoot at the bottleneck,
but may be used to reset ssthresh.After using Careful Resume, the CC controller returns to the Normal Phase.
The implementation details for different transports depend on the
design of the transport.In the Normal Phase, a sender can enter the Observation Phase to perform
observation of the path.{XXX-Editor note: A future revision should discuss updating
the saved parameters, whether used or not, after reaching normal operation for use
the next time even if that update is to just refresh the expiration time.}A sender is limited by any rate-limitation of the transport
protocol that is used.
For QUIC this includes flow control mechanisms or preventing amplification
attacks. In particular, a QUIC receiver might need to issue proactive
MAX_DATA frames to increase the flow control limits of a connection
that is started when using Careful Resume to gain the expected benefit.A TCP sender is limited by the receiver window (rwnd).
Unless configured at a receiver, the rwnd constrains the rate
of increase for a connection and reduces the benefit of Careful Resume.This section provides definitions that enable the
Careful Resume method
to generate qlog events when using QUIC.
It introduces an event to report the current phase of a sender,
and the associated description.The event and data structure definitions in this section are
expressed in the Concise Data Definition Language (CDDL)
and its
extensions described in .
The current convention is to use long names for variables.
For example, "cwnd" is expanded as "congestion_window"
and "saved_cwnd" is expanded as "saved_congestion_window" below.Importance: ExtraWhen the CC algorithm changes the Careful Resume Phase
described in of this specification.Definition:The authors would like to thank John Border, Gabriel Montenegro, Patrick McManus,
Ian Swett, Igor Lubashev, Robin Marx, Roland Bless, Franklin Simo, Kazuho Oku, Tong,
Ana Custura, Neal Cardwell, and Joerg Deutschmann for
their fruitful comments on earlier versions of this document.The authors would like to particularly thank Tom Jones for co-authoring
several previous versions of this document. Ana Custura and Robin Marx developed the qlog support.No current parameters are required to be registered by IANA.This document does not exhibit specific security considerations.
Security considerations for the
interactions with the receiver are discussed in .Issues in TCP Slow-Start Restart After IdleISIISIISIFeedback from using QUIC's 0-RTT-BDP extension over SATCOM
public accessGoogle QUIC performance over a public SATCOM accessHalfback: Running Short Flows Quickly and Safely The table below is provided to illustrate the operation of Careful Resume.
This table is informative, please refer to the body of the document
for the normative specification. The description is based on a Normal
CC that uses Reno or CUBIC. The PipeSize tracks the validated CWND.PS) |CWND=(PS/2) |
|entry:| | |CWND |{CWND=FS} | |
| | | |=jump_cwnd |else | |
| | | | |{CWND=PS; | |
| | | | |enter |
| | | | |Normal} | |
+------+---------+---------+------------+-----------+------------+
|CWND: |When in |CWND |CWND is not |CWND can |CWND is not |
| |observe, |increases|increased |increase |increased |
| |measure |using | |using | |
| |sav_cwnd |normal CC| |normal CC | |
| | |methods | |methods | |
+------+---------+---------+------------+-----------+------------+
|PS: | - | - | PS+=ACked |
+------+---------+---------+------------+-----------+------------+
|RTT: |Measure |Measure | - | - | - |
| |saved_rtt|current | | | |
| | |_rtt | | | |
+------+---------+---------+------------+-----------+------------+
|If |Normal |Normal | Enter | - |
|loss |CC method|CC method| Safe | |
|or | |CR is not| Retreat | |
|ECNCE:| |allowed | | |
+------+---------+---------+------------+-----------+------------+
|Next |Observe |If ( |If (FS=CWND |If (ACK |If (ACK |
|Phase:|(as |FS=CWND |or >1 RTT |>= last |>= last |
| |needed) |and CR |has passed |unvalidated|unvalidated |
| | |confirmed|or ACK |packet), |packet), |
| | |), enter |>= last |enter |{ssthresh=PS|
| | |Unvalidat|unvalidated |Normal |and enter |
| | |-ing else|packet), | |Normal} |
| | |enter |enter | | |
| | |Normal |Validating | | |
+------+---------+---------+------------+-----------+------------+
]]>The following abbreviations are used
SS = slow start FS= flight_size; PS = PipeSize. The PipeSize is set to the CWND
on entry to the Unvalidated Phase. This tracks the validated portion of the CWND and
is updated as each additional packet is acknowledged.Note:
For an implementation that keeps track of transmitted data in terms of packets:
In the Unvalidated Phase, the first unvalidated packet corresponds
to the highest sent packet recorded on entry to this phase.
In the Validating and Safe Retreat Phases, corresponds to the last unvalidated packet
is also the highest sent packet recorded on entry to this phase.
The remaining subsections provide informative examples of use.
Although the QLOG variables are expressed in bytes,
to simplify the description, these examples
are described in term of packet numbers.
In the first example of using the Careful Resume method,
the sender starts by sending IW (10) packets in the Reconnaissance Phase,
and then continues in a subsequent RTT to send more packets until the sender
becomes CWND-limited (i.e., flight_size = CWND).The sender then
confirms the RTT and other conditions for using Careful Resume.
In this example, this is confirmed when the sender has 29 packets in flight.
The sender enters the Unvalidated Phase.
This confirmation could have happened earlier if data was available to send. The
sender initialises the PipeSize to the CWND (the same as the flight_size, i.e., 29 packets)
and sets the CWND to 150 packets (based upon a previously
observed saved_cwnd of 300 packets).It now sends 121 unvalidated packets. This is the unused portion of the CWND.
Each time a packet is sent, the sender checks whether 1 RTT has passed since entering the
Unvalidated Phase (otherwise, the Validating Phase is entered). This check triggers only
for cases where the sender is rate-limited, see the following example.The PipeSize will increase after each ACK is received.When the first unvalidated packet is acknowledged (packet number 30) the sender
enters the Validating Phase. This transition would also occur if the flight_size increases to equal CWND.
During this phase, the CWND can be increased for each ACK for an unvalidated packet, because
this indicates that the packet was indeed validated.When an ACK is received for the last packet sent in the Unvalidated Phase,
the sender completes using Careful Resume. It enters the Normal Phase. If CWND is less than ssthresh,
a Reno or CUBIC sender in the Normal Phase is permitted to use slow start to grow the CWND towards
the ssthresh, and will then enter congestion avoidance.A rate-limited sender will not fully utilize the available CWND when using Careful Resume,
and CWND is therefore reset on entry to the Validating Phase, as described below.The sender in this example starts by sending IW packets (10) in the Reconnaissance Phase,
and sends as described in the first example, transitioning to the Unvalidated Phase.
This sets the CWND to 150 packets, and the PipeSize is set to the flight_size (i.e., 29 packets).The sender then becomes rate-limited because it only sends 50 unvalidated packets. After ~1 RTT (detected by using local timestamps or by receiving an ACK for the first
unvalidated packet), the sender will still not have fully used the CWND. It then enters
the Validating Phase and resets the CWND to the current flight_size, (i.e., 50 packets).
During this phase, the CWND can be increased for each ACK that validates reception of a packet.
The PipeSize also increases with each ACK received, to reflect the discovered capacity.When an ACK is received for the last packet sent in the Unvalidated Phase,
the sender has completed using Careful Resume. It enters the Normal Phase. If CWND is less than ssthresh,
a Reno or CUBIC sender in the Normal Phase is permitted to use slow start to grow the CWND towards
the ssthresh, and will then enter congestion avoidance.When a packet is lost in the Reconnaissance Phase, the sender enters the Normal Phase
and recovers this using normal method. There is no change to the CC method, because
the sender has discovered a potential capacity limit and is not
allowed to use Careful Resume.As in the first Example the sender enters the Unvalidated Phase it sets the CWND to 150 packets
with the PipeSize initialized to the flight_size (i.e., 29 packets).The sender now sends 121 unvalidated packets (the remaining unused CWND).
This example considers the case when one of the unvalidated packet is lost, which
we choose to be packet 64 (the 35th packet in the Unvalidated Phase).Acknowledgements confirm the first 34 unvalidated packets are received without loss.
The PipeSize at this point is equal to 29 + 34 = 63 packets.The loss is then detected (by receiving three ACKs that do not cover packet number 35),
the sender then enters the Safe Retreat Phase because the window was not validated.
The PipeSize at this point is equal to 29 + 34 = 66 packets. Assuming IW=10.
The CWND is reset to max(10,ps/2) = max(10,66/2) = 33 packets.
This CWND is used during the Safe Retreat Phase, because congestion was detected and the
sender still does not know if the remaining unvalidated packets will be
successfully acknowledged. A conservative CWND calculation ensures the sender drains
the path after this potentially severe congestion event.
There is no further increase in CWND in this phase.The sender continues to receives ACKs for the remaining 86 (121-35) unvalidated packets
(recall that the 35th unvalidated packet was lost and had packet number 29+35=64).
The PipeSize continues to further increase as each ACK acknowledges new data, because this tracks the
size of the pipe discovered by the unvalidated packets.
Although this cannot be used to safety initialise CWND (because was measured when the sender
aggressively created overload), the estimated PipeSize
(which, in this case, is 121-1 = 120 packets) can be used to set the ssthresh on exit
from Safe Retreat, since it indicates an upper limit to the current capacity.At the point where all packets sent in the Unvalidated Phase have been either acknowledged
or declared lost, the sender enters the Normal Phase, but first updates ssthresh.
Because CWND will now be less than ssthresh, a sender in the Normal Phase is permitted to use
slow start to grow the CWND towards the ssthresh, after which it will enter congestion avoidance.
This annex proposes an Endpoint Token to allow a sender to identify its own
view of the network path that it is using. In
this Endpoint Token could be shared and used as an
opaque path identifier to other parties and the sender can verify if
this is one of its current paths.
When computing the Endpoint Token, the sender includes information to identify
the path on which it sends, for example, this:
needs to include a unique identifier for itself (e.g., a globally
assigned address/prefix; or randomly chosen value such as a nonce);
needs to include an identifier for the destination (e.g., a
destination IP address or name);
needs to include an interface identifier (e.g., an index value or a MAC address to associate the
endpoint with the interface on which the path starts);
could include other information such as the DSCP, ports, flow
label, etc (recognising that this additional information might improve the path
differentiation, but that this can reduce the re-usability of the
token);
this could include any other information the sender chooses to
include, and potentially including PvD information or
information relating to its public-facing IP address;
When creating an Endpoint Token, the sender has to ensure the following:
To reduce the likelihood of misuse of the Endpoint Token, the value
ought to be encoded in a way that hides the component information
from the recipient and any eavesdropper on the path (this could already protected by methods
such as TLS).The sender can recalculate the Endpoint Token to validate a
previously issued token; and can be
included in the computed integrity check for any path
information it provides. The Endpoint Token is designed so that if shared, it prevents another party from deriving
private data from the token, or to use the token to perform
unwanted likability with other information. Therefore,
the Endpoint Token MUST necessarily be different when used to identify
paths using different interfaces.Previous individual submissions were discussed in TSVWG and QUIC.
WG -00 included clarifications and restructuring to form the 1st WG draft.WG -01 included review comments and suggestions from John Border,
and follows the setting of the TSVWG milestone
with an intended status of "Proposed Standard".WG -02 includes steps to complete the spec. In particular, consideration of rate-limited
senders; selection of reasoned parameters; specification of the Safe Retreat Phase; and
improvements to the consistency throughout. Added the Validating Phase.WG -03, explain entry to Validating Phase, editorial tidy.WG -04, update based on review comments from Kazuho Oku.WG-05, update based on review comments from Neal Cardwell. WG feedback from IETF-118.
Reviewed the requirements v. guidelines; clarified that CC is not changed in recon., but
the recon. info is used to steer the next phase; clarified saved_cwnd can be computed
from ACK rate; use jump once; that real server platforms are complex.
Clarified lifetime for saved CC params. Incorporates comments from Tong.WG-06, SR updated following Hackathon comments from Kazuho Oku, and rework of use of PipeSize.
Added an informative summary of actions, on suggestion by Tong. Added examples based on text by Ana Custura. WG-07, Use "rate-limited" uniformaly instead of application and data limited.Updated to exit early when unvalidated CWND not utilised, detected in tests by Q Misell. Change
pipe_size to be PipeSize.WG-08, Updated CDDL (AC), and made constraints in the Observe Phase into guidance, they say what
makes sense - but do not need to be followed for conformance. Updated table in annexe to align
with text. Formatted CDDL and added cross-refs to triggers in the text (from a comments by JD and AC).
Fixed typos and consistency (JD, RS, GF).