Solving connections, not requests

· 5 min

For a server-side rendered page, the usual bottleneck is the network requests made during render. If the response can’t be sent until they resolve, the slowest one sets the floor for time to first byte (TTFB), and every later paint metric inherits the delay. With the requests running in parallel, render time is at least max(...requests). If they run sequentially, it would be the sum.

That makes it the number to measure, per region. A global percentile hides the customers furthest from where the services run.

Investing time into max() across the matrix of deployed infrastructure regional selection across customer edge locations can yield some useful data points.

The problem

Not every service runs in every region. On a serverless stack that’s about dependencies, not cost. A service that depends on an external SaaS product that isn’t multi-region should be co-located in the same region as that product for reliability. If the SaaS is hosted in Europe, the service should run in a European or neighbouring region, chosen by its proximity to the other data points such as customers rather than calling it cross-region. An AWS Lambda in ap-southeast-2 calling a European region pays at least one ~265ms round trip per call, and three on a cold connection. Timeouts and retry behaviour are set by the vendor, so every millisecond spent in transit comes out of a fixed budget. A short hop leaves more of that budget for the dependency itself.

This is problematic where the data is both dynamic and specific to a user.

One round trip isn’t the problem

Sydney to Europe and back is roughly 265ms. That’s one round trip (RTT).

A single request on an existing connection: 1 RTT ≈ 265ms.

One round trip describes a request on a connection that already exists. A cold connection has to be set up first:

  1. TCP handshake, 1 RTT. SYN out, SYN-ACK back.
  2. TLS 1.3 handshake, 1 RTT. ClientHello out, ServerHello and certificate back. TLS 1.2 needs two.
  3. The request itself, 1 RTT.

That’s 3 × 265ms, about 800ms before the first useful byte.

A cold connection: TCP handshake, TLS handshake, then the request. 3 RTT ≈ 800ms.

Most of that is setup, before anything has been asked for. It’s about solving connections, not requests.

The connection in question isn’t the caller’s. A request enters AWS CloudFront at a nearby edge location, and that hop is short. The expensive one is CloudFront’s own connection from that edge back to the origin in eu-west-1 as an example region.

Keeping connections warm

CloudFront pools its connections to the origin and reuses one if it’s open. A warm connection skips TCP and TLS, so the request costs 1 RTT, not 3.

Cold versus warm, drawn to the same scale: 3 RTT ≈ 800ms against 1 RTT ≈ 265ms.

The obvious lever is the origin keep-alive timeout, which is how long CloudFront keeps an idle origin connection open. The default is 5 seconds, and the origin’s own idle timeout caps it.

Will raising it solve the problem? It will help, but it won’t produce a big drop. Why?

Because there isn’t one connection pool. CloudFront consists of many points of presence (PoPs, the edge locations), each holding its own connections to the origin. A connection only stays warm if another request arrives on it before the idle timer runs out, so what matters is how often requests reach each pool. Keep-alive doesn’t change that.

Take a route that sends one request every 2 seconds. Through a single pool with a 5 second timeout, the connection never goes cold. Spread across ten pools, each sees a request every 20 seconds, and most arrive cold. Australian traffic to a European origin is the second case.

So it’s still a roll of the dice. Is there a way to fix the distribution itself? Yes, but it’s an unusual one.

AWS CloudFront Origin Shield

Origin Shield adds one more CloudFront cache layer, in a single chosen region, between every edge location and the origin:

without:  caller → edge location → origin
with:     caller → edge location → Origin Shield → origin

It has two primary features:

Neither helps a dynamic request. The response is different for every customer, so there’s nothing to cache or collapse, and AWS’s guidance is that Origin Shield may not suit dynamic content that’s just proxied to the origin.

The hypothesis

Both features depend on every origin-bound request, from every edge location, passing through the same place. If the requests converge, so do the connections. The edge locations stop holding their own connections to the origin. Origin Shield holds them, in one region.

So the hypothesis: if origin connections are pooled centrally and all of the traffic flows through them, they should stay warm. A request from Australia wouldn’t need a TCP or TLS handshake with the origin at all.

Without Origin Shield: many cold pools and low reuse. With Origin Shield: fewer hot pools and high reuse.

It isn’t literally one pool. Origin Shield is a fleet across at least three availability zones, so fewer, hotter pools is the accurate way to put it. And it sits in the same region as the origin, so when it does need a new connection, each round trip of the handshake takes a millisecond or two.

Using a warm connection would go from a sporadic gamble to the default.

Origin Shield was enabled for the origin, in eu-west-1, and released on 29 June.

Results

Both timers tracked for the request went from roughly 650ms to roughly 350ms at the moment of the release and have been flat since. The variance dropped too. Before, the line is noisy, which is what a mix of warm and cold connections looks like.

That took around 300ms off homepage TTFB for Australia, and showed up on other pages that wait on the same request. These are p75 values, scaled and not exact:

Page p75 TTFB before p75 TTFB after Change
Homepage 1.95s 1.65s -0.30s
Product details 2.48s 2.10s -0.38s

It isn’t free. Origin Shield bills per request for dynamic content, which includes any GET with caching disabled. Here that came to around $7 a day relative to traffic volume.

None of this used the cache. Origin Shield earned its place purely as somewhere for connections to converge, and AWS Solutions Architects were surprised, pleasantly of course, by the result. One AWS post on dynamic workloads does note in passing that Origin Shield “can also increase the connection reuse rate”, but the main guidance points the other way.

What’s next

The chart shows the outcome, not connections being reused. CloudFront’s Server-Timing response header includes cdn-upstream-connect. Zero means an existing origin connection was reused, and anything else is the time spent on TCP and TLS setup. When measured, reuse increased by 81%.

References