Automating QUIC Interoperability Testing

ABSTRACT

We present QuicInteropRunner (QIR) [1,2], a test framework for automated and on-demand interoperability testing between implementations of the QUIC protocol [3]. QIR is a framework in which QUIC clients and servers interact with each other over a network that simulates various network conditions using ns-3 [4]. QIR automates QUIC interoperability testing by running a suite of test cases between containerized QUIC implementations. We describe the key constraints and insights that defined our work, recent innovations that made the framework possible, a high-level overview of our design, and a few exemplary tests. QIR is now supported and used by ten QUIC implementations as part of their development process, confirming our thesis that there is a need for automating interoperability testing and making it available on demand.

1 INTRODUCTION

The Internet is a multi-vendor system defined by open standards. Interoperability testing has long been a cornerstone of the development of these open standards, for two reasons. First, different implementations of an open protocol interact with each other on the Internet and therefore need to be tested for those interactions. Second, it exposes gaps and ambiguities in the specification, as different implementations can make conflicting assumptions in such cases. For both of these reasons, the IETF, the primary standards body for Internet protocols, requires interoperability testing as a part of the development process [5].

Interoperability testing however has historically been a manual process. For example, SCTP [6] interoperability meetings entailed bringing computers into a room, wiring them to the same network, manually running various tests, and examining the outcomes locally. More recent efforts have relied on Internet infrastructure for testing where possible, removing the need for co-locating implementations. For instance, HTTP/2 and QUIC implementers would set up servers running their implementations on publicly accessible Internet endpoints, against which others could run tests. Nevertheless, the testing itself has remained manual [7].

Manual testing suffers from three significant scaling limitations. First, it limits the number of implementations that can be tested, since there are a quadratic number of combinations to test. Second, it limits the number of features that can be tested, a problem that is made worse by the complexity of protocols built for the modern Internet. Finally, it limits the range of network conditions under which the protocol is tested. Ad-hoc testing over a local network or over the public Internet does not test the implementations' performance in the variety of network conditions under which the protocol is expected to perform well. As a result, various parts of the protocol, such as those designed to handle adverse network conditions, likely remain untested.

In the IETF's QUIC working group [8], these limitations meant that comprehensive interoperability testing was only performed roughly once every month. This interoperability did not include repeatable and precise tests of the implementations, and the outcomes of the tests were determined by manual inspection of logs [7].

The QuicInteropRunner (QIR) is our attempt at overcoming these limitations in building performant and robust QUIC implementations [1].

Figure 1: Console output of a local run of the QUIC interop runner. Column headers refer to servers and row headers refer to clients. Tests are indicated by their letter symbols in each cell. Test outcomes are Success, Unsupported, or Failure, as shown in the top, middle, and bottom rows within each cell. Endpoint implementations and test cases can be specified via command line parameters, allowing implementers to focus their testing on specific pairs and interactions. Test cases shown here are H: Handshake, S: Retry, M: Multiplexing, B: Blackhole, Z: 0-RTT, 3: HTTP/3.

QIR automates QUIC interoperability testing by running a suite of test cases between containerized QUIC implementations. QIR also makes performance measurements under different network conditions possible. Importantly, QIR can be run locally, making both on-demand and continuous interoperability testing possible. To our knowledge, QIR is the first automated interoperability testing framework for a network protocol.

QIR includes several major QUIC implementations. As of this writing, ten QUIC implementations (two of which implement only client functions) are included in QIR. Any implementer can include their implementation in QIR by building a compatible container image, making it publicly available, and adding it to the list of implementations [9].

2 QIR DESIGN

QIR’s design came out of our experience with the limitations of manual QUIC interoperability testing. We first go through the design constraints that shaped QIR’s design, followed by detailed descriptions of QIR’s components.

Since most QUIC implementations are in user space, we decided to focus our efforts on supporting user-space implementations. Since these implementations could all be built on Linux, we also chose to limit ourselves to that one platform instead of trying to build for multiple platforms. We acknowledge that this restricts us from testing existing kernel implementations or user-space ones that cannot be built on Linux.

2.1 Design constraints

QIR’s design constraints were gleaned from our experience with manual interoperability testing with QUIC, and were as follows:

• No source code: QUIC implementations are written in different programming languages, under a variety of licenses, and with vastly different build environments and requirements. Building all implementations from their source for regular testing could require a significant amount of time and other resources. Importantly, we could not assume that the source code for all QUIC implementations would be available. As a result, our test framework could not expect the source code to be available, and would preferably not build implementations from source.
• Maintenance delegation: Each implementer would maintain and update their own endpoint, and make it publicly available on their own schedule.
• On-demand and continuous testing: To enable interoperability testing as part of a typical development workflow, the test framework would need to allow for on-demand testing of any set of implementations. This could then be extended to continuous testing of any set of implementations.
• Repeatable performance testing: Performance testing of the different implementations would need to use carefully constructed network scenarios and would need to be repeatable for debugging purposes.

To meet these constraints, our key insight was to use containers as QIR’s basic building block. Containers give implementers control over their binary images, enabling them to bundle all build- and runtime-dependencies into their own, independent environments, and allowing them to publish updated images on their own schedule. Since they are distributed as binaries, containers also allow closed-source implementations to participate in the framework.

2.2 QIR components

As shown in figure 3, QIR is a test harness that uses three Docker containers [10]: a client container, a server container, and a network container. Docker Compose [11] is used to orchestrate the three containers. QUIC implementers publish endpoint containers running their implementations on DockerHub [12], and each container can be instantiated as a server or as a client depending on the implementation’s role in a test (the role is provided as an environment variable).

QUIC servers are expected to receive packets on UDP port 443 on a pre-specified IP address, configured as the address of the server container’s virtual network interface. QUIC clients are expected to send requests to this pre-configured address.

Figure 3: Network setup used in QIR tests. Boxes represent Docker containers [10] running a QUIC client, the network simulator, and a QUIC server. IP addresses and routes are configured such that packets between the client and the server have to pass through the network container, where ns-3 [4] is used to simulate different network conditions.

The network interfaces of the server and client containers are on different IP subnets, to prevent the host operating system from forwarding packets directly between the two endpoint containers, and to force the packets to be forwarded through the network container instead. The network container has two network interfaces connecting to the server and client containers. All traffic between the endpoint containers passes through the network container, where various network conditions can be simulated.

2.3 Using ns-3 for network simulation

Within the network container, QIR uses the ns-3 network simulator [4], running in real-time simulation mode, to read and write packets from and to the two network interfaces, and to simulate a network topology between them. We chose ns-3 for the ease with which we could introduce new behaviors in the network simulation for various tests (see Section 3 for examples). We were also aware of its rich set of channel propagation and mobility models for different wireless and wired links, which we wanted to explore. Importantly, ns-3 allowed packets from the real world to be introduced into the simulated world and vice-versa.

After much testing, we chose 10 Mbit/s as the bandwidth of the bottleneck link of the simulated network to ensure that a commodity laptop could run the QIR setup without using up all its compute power.

In the simplest configuration, QIR uses ns-3 to simulate a fixed-bandwidth link with a finite queue size. This setup exercises a fair bit of QUIC’s machinery. Using a fixed-bandwidth link requires QUIC congestion controllers to determine the available bandwidth, typically by filling the queue at the bottleneck and reacting to any resulting packet loss. Other scenarios include inducing packet loss to test QUIC’s loss recovery, both during the handshake and later in the connection; inducing packet corruption to test QUIC’s ability to discard invalid packets; and temporary black-holing of the connection, to test QUIC’s recovery from temporary outages.

3 QIR TESTS

A test case in QIR creates a scenario and observes the behavior of the QUIC endpoints, where a scenario is a specific network topology and behavior. For example, a simple test case could require a client to download a specific object from the server.

3.1 Scenarios

In QIR, a scenario represents a network topology and behavior, that we implement in C++ as a part of ns-3. QIR currently includes the following four scenarios, which are used in the tests it implements:

• simple-p2p: A simple point-to-point link with a specified bandwidth, delay, and queue size.
• drop-rate: A simple point-to-point link with a specified bandwidth, delay, queue size, and a configurable packet drop rate.
• corrupt-rate: A simple point-to-point link with a specified bandwidth, delay, queue size, and a configurable rate at which packets are corrupted (a random byte in the first fifty bytes of the QUIC packet is corrupted).
• blackhole: A simple point-to-point link with a specified bandwidth, delay, queue size, and configurable periods of time during which the link is either forwarding or dropping packets.

3.2 QIR Workflow

QIR first generates objects to be transferred for the test. These objects are of random sizes and content, and they are made available in the server container via a mounted directory. The client is expected to download these objects and store them into a separate mounted directory. At the end of each test, this setup allows QIR to access and validate the number and content of the downloaded objects.

In addition to these two directories, QIR sets up log directories to be used by the client and the server for recording their logs. Endpoints can record log files in their preferred logging format within the endpoint containers, and these are exported and made available by QIR after the test has completed. To facilitate analysis and debugging, the network container maintains various logs and detailed packet captures at both network interfaces.

QIR then starts the containers up and waits until the test completes (or times out). QIR provides necessary configuration information, such as the name of the test and the names of the objects to download, to the endpoint containers using environment variables that are available within the containers. Test completion is indicated by at least one container shutting down.

3.3 Handshake Test

In this simple test, a client is expected to:

1. Establish a QUIC connection with the server at the statically configured IP and port;
2. Request a single (small) file, the URL for which is specified in the REQUEST environment variable; for example, https://server/xqsdfiuywerf;
3. Record the received object in a file with the same name, in the /downloads directory in the client container.

The server is expected to accept incoming connections and respond to requests using objects located in the /www directory in the server container.

3.4 Retry Test

QUIC’s Retry mechanism is designed for a server to validate the client’s IP address prior to committing any state to the connection. This test extends the Handshake test to exercise the Retry mechanism.

3.5 Multiplexing Test

The Multiplexing test extends the Handshake test to exercise QUIC’s stream multiplexing features.

3.6 Performance Tests

The Throughput test is the first of QIR’s performance tests. This test is the same as the Handshake test with the modifications of the object transferred being large in size and repeated a number of times to show statistical confidence in the results.

4 RELATED WORK

While there has been a lot of work in network simulation and emulation, there is very little work on interoperability frameworks. The closest related work to ours is QUIC Tracker [18]. This project uses a custom-built QUIC client to run tests against public QUIC servers, to test their compliance to the QUIC specification.

5 CONCLUSIONS AND FUTURE WORK

While interoperability testing has been one of the hallmarks of open standards and protocol development, the process remains inadequate and limited. Using simple container orchestration and network simulation, QIR makes it possible to meet the constraints of implementers while automating this process for on-demand, continuous, and repeatable interoperability and performance testing.

We are planning to expand the framework to include more tests. Test cases of interest include:

• A high bandwidth test, testing QUIC flows at bandwidths up to 100 Mbit/s.
• A handshake latency test, measuring the time to handshake completion at various packet loss rates, averaged over multiple runs.

REFERENCES

[1] QUIC Interop Runner. https://github.com/marten-seemann/quic-interop-runner. [2] QUIC Interop Runner Web Interface. https://interop.seemann.io. [3] J. Iyengar and M. Thompson. QUIC: A UDP-Based Multiplexed and Secure Transport. February 2020. https://tools.ietf.org/html/draft-ietf-quic-transport-27. [4] The ns-3 Network Simulator. https://www.nsnam.org/.