A WebSocket upgrade walked straight past our Lambda@Edge

An attacker sent plain HTTP/1.1 requests dressed up as WebSocket upgrades to our CloudFront distribution, and they reached the origin without our viewer-request Lambda@Edge function seeing them. Any site that keeps its security logic in a viewer-request function may have the same gap, and AWS WAF can close it.

George Oastler
George Oastler

Technical 9 min read

We recently had an attack that taught us something uncomfortable about CloudFront and Lambda@Edge. If your security logic lives in a viewer-request function, there may be a way around it, using a request type that is easy to forget about: the WebSocket upgrade.

Short version: an attacker sent HTTP/1.1 requests with WebSocket upgrade headers to our CloudFront distribution, and they reached our origin without our viewer-request Lambda@Edge function stopping them. Edge functions can't see the Upgrade header, and CloudFront always accepts HTTP/1.1, so block WebSocket upgrades in AWS WAF on paths that don't need them, and make your origin refuse anything the edge function hasn't signed.

A typical CloudFront and Lambda@Edge setup

This is a common way to protect a site on AWS:

  • CloudFront sits in front of everything and accepts HTTP/1.1, HTTP/2 and HTTP/3.
  • A Lambda@Edge viewer-request function runs your security logic, such as bot checks, authentication, rate limiting and geo-blocking.
  • An origin behind it assumes the edge function has already vetted every request.

AWS's documentation on Lambda@Edge triggers says the viewer-request function:

executes when CloudFront receives a request from a viewer, before it checks to see whether the requested object is in the CloudFront cache.

The same page lists only two exceptions: fetching a custom error page, and CloudFront's automatic HTTP-to-HTTPS redirect. It's easy to read that as "every request goes through my function", and that was our assumption too.

The attacker sent WebSocket upgrades

Instead of normal page or API requests, the attacker sent HTTP/1.1 requests that asked to switch to WebSocket:

GET /api/login HTTP/1.1
Host: www.example.com
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13

Those requests reached our origin and our viewer-request function didn't block them, so our bot checks never got the chance to challenge the client.

Normal requests from the same client were still caught as usual, and only the upgrade requests got through.

CloudFront treats upgrades differently (as far as we can tell)

AWS doesn't document this behaviour either way, so we want to be careful here and keep what we know separate from what we believe.

What AWS documents:

  • CloudFront supports WebSockets over HTTP/1.1 only, and the feature is switched on for every distribution automatically, with no setting in the distribution to turn it off.
  • Connection and Upgrade aren't exposed to edge functions. Your function can't see them, so it can't tell an upgrade request from a normal one, and it can't block on them.
  • CloudFront's HTTP version setting is the highest version viewers may use. Clients that don't speak HTTP/2 or HTTP/3 fall back to HTTP/1.1, and we found no way to refuse HTTP/1.1 altogether.

What we believe: CloudFront handles a WebSocket upgrade differently from a normal request. A WebSocket connection can't be cached, so CloudFront treats every upgrade as unique and passes it through to the origin, and in our case the viewer-request function didn't stand in the way.

We haven't had AWS confirm the mechanism, and the documentation is silent on it, so neither our word nor AWS's silence is proof either way. Test your own distribution: send the request above with curl --http1.1 to a path your edge function should block, and check whether it reaches your origin.

Four ways to close the gap

1. Avoid HTTP/1.1 where you can

WebSockets on CloudFront need HTTP/1.1, so the cleanest fix would be not to accept it, but CloudFront doesn't let you switch it off. Treat HTTP/1.1 as something an attacker can always choose, and plan the rest of your defences around that.

2. Block WebSocket upgrades in AWS WAF

If your site doesn't use WebSockets, nobody has a legitimate reason to send an upgrade request. AWS WAF can inspect the Upgrade header that your edge function can't see, so block it there:

{
  "Name": "block-websocket-upgrade",
  "Priority": 0,
  "Action": { "Block": {} },
  "Statement": {
    "ByteMatchStatement": {
      "FieldToMatch": { "SingleHeader": { "Name": "upgrade" } },
      "PositionalConstraint": "CONTAINS",
      "SearchString": "websocket",
      "TextTransformations": [{ "Priority": 0, "Type": "LOWERCASE" }]
    }
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "block-websocket-upgrade"
  }
}

If some paths really do serve WebSockets, put the match inside an AndStatement with a NotStatement that exempts them (for example a ByteMatchStatement on UriPath starting with /ws/), and make sure those paths do their own authentication at the origin.

Start the rule in Count mode, check the WAF logs for anything legitimate it would have blocked, and then switch it to Block.

3. Be very careful with WebSocket paths

If you need WebSockets, give them their own cache behavior on their own path, and don't let that behavior match anything else, because your edge function may not protect anything an upgrade request can reach. Authenticate the WebSocket connection at the origin with a short-lived token, rather than with a cookie the edge function was meant to check.

4. Make the origin check the edge's work

This fix would have saved us. Have your viewer-request function add a header that proves it ran, such as an HMAC of the request using a secret only the edge and the origin know, and have the origin reject any request without a valid one.

Then if a request ever reaches the origin without passing through your function, for whatever reason, it fails closed instead of open. Edge functions can add their own custom headers, because the restriction only covers a short list such as Upgrade and Connection.

Edge security assumes every request passes through it

Edge security is only as strong as the assumption that every request passes through it. CloudFront has a handful of request types that behave differently: redirects, error pages and, it turns out, WebSocket upgrades. Each one is a place where hand-written security logic can quietly stop applying.

That's a lot to keep on top of when bot protection isn't your day job, and it's why we build Prosopo Protect: site-wide bot protection that runs at the edge, with Access Rules, behavioural checks and a team whose job is finding gaps like this one before attackers do. If you'd rather not maintain your own Lambda@Edge security logic, get in touch.

TL;DR

  1. HTTP/1.1 WebSocket upgrade requests got past our Lambda@Edge viewer-request function and reached the origin.
  2. AWS doesn't document whether viewer-request functions run on upgrades, so test your own distribution.
  3. Edge functions can't see the Upgrade header, and CloudFront always accepts HTTP/1.1.
  4. Block Upgrade: websocket in AWS WAF on every path that doesn't need it.
  5. Keep WebSocket paths separate, and authenticate them at the origin.
  6. Have the origin reject any request your edge function hasn't signed, so a bypass fails closed.

Questions

Something else? Ask an engineer.

Can a WebSocket upgrade request bypass a Lambda@Edge viewer-request function?

In our incident, HTTP/1.1 requests carrying WebSocket upgrade headers reached the origin without our viewer-request function blocking them. AWS's documentation doesn't say whether viewer-request functions run on WebSocket upgrades, so test your own distribution rather than assuming they do.

Can I turn off HTTP/1.1 on CloudFront?

Not as far as we can tell. CloudFront's HTTP version setting is the highest version viewers may use, and clients that don't support HTTP/2 or HTTP/3 fall back to HTTP/1.1. CloudFront supports WebSockets only over HTTP/1.1.

Can a Lambda@Edge function see the Upgrade header?

No. AWS lists Connection and Upgrade among the headers that aren't exposed to edge functions, so a function can't check for them or block on them. Use AWS WAF, or a separate cache behavior, to deal with upgrade requests instead.

How do I block WebSocket upgrades on CloudFront?

Add an AWS WAF rule to the distribution that blocks requests whose Upgrade header contains websocket, with an exception for any paths that really do serve WebSockets.

Ready to ditch Google reCAPTCHA?

Procaptcha is free for 10K requests a month. No credit card, no cookies.