On the "HollowByte" denial-of-service report

Over the past week a denial-of-service (DoS) report against OpenSSL, named “HollowByte” by the Okta Red Team who reported it, has received a good deal of press attention. A number of the articles ask reasonable questions about how we assessed the report and why we handled the fix the way we did. This post sets out our analysis and the reasoning behind our decisions.

We are grateful to the Okta Red Team for the report and for the detail they put into it. The behaviour they describe is real, and we have changed OpenSSL in response to it. But the report combines two quite different things under a single headline, and separating them is the key to understanding our response.

What was reported

During a TLS handshake, each handshake message begins with a four-byte header: a one-byte message type followed by a three-byte length. In older versions of OpenSSL, once that header had been read the state machine grew its handshake receive buffer to the full length declared in the header, before the body of the message had arrived. An attacker could therefore send a small record whose header declares a large message, and OpenSSL would allocate a correspondingly large buffer while waiting for a body that the attacker never intended to send.

The report demonstrated this with an 11-byte record declaring a message of roughly 128 KB, and went on to describe server outages in Apache, nginx, Node.js and others, together with elevated resident memory (RSS) that persisted after the offending connections had closed.

Two separate issues

The report treats the memory allocation and the server outages as one vulnerability, with the first causing the second. On analysis they are largely independent.

1. The allocation itself. OpenSSL did allocate the declared amount up front. That allocation was, however, bounded and per-connection. OpenSSL enforces a maximum handshake message size, around 128 KiB, and rejects anything larger, so a single connection could never consume an arbitrary amount of memory. OpenSSL frees the buffer when the connection ends; whether the platform then returns that memory to the OS is a separate, allocator-level question we cover below. Resident memory can stay elevated after close, but in our testing that elevation is bounded — it reaches a plateau and is reused rather than growing without limit. We changed the allocation itself because reducing what we request up front is cleaner and lowers the peak on every platform, whatever the allocator does afterward.

2. The outages. The service failures the report describes are not caused by the size of the allocation. They arise when connections complete a TCP handshake, begin a TLS handshake, and then stop sending — while the server waits on a blocking socket with no timeout and no cap on how long a single connection may stall. This is the long-understood slowloris class of attack, and it does not depend on a large declared length at all: a connection that declares a message only slightly larger than what it actually sends, and then falls silent, ties up the same worker or file descriptor. Where a server becomes unresponsive after a number of such connections, the cause is usually connection or worker exhaustion rather than memory: on a normally-resourced box we observed worker exhaustion, not out-of-memory, with service recovering as soon as the stalled connections were released. The out-of-memory outcome in the report was on a memory-constrained host. The amplification didn’t create that failure mode, but it could accelerate it — another reason we chose to bound the allocation.

The distinction matters because the two issues have different owners. The first is something OpenSSL can improve, and has. The second is addressed where the connection lifecycle is actually managed: in the application and the platform, using handshake and idle timeouts, connection and rate limiting, and non-blocking I/O — the same controls that any internet-facing service needs against slow-read attacks regardless of which TLS library is in use.

The persistent-memory observation

Several articles highlight the report’s most striking claim: that after the attacking connections close, resident memory stays elevated rather than being returned to the operating system.

Whether freed memory is returned to the operating system, and how quickly, is decided by the platform’s memory allocator, not by OpenSSL. On glibc/Linux, the platform in the report, we reproduced the behaviour: after the connections close, resident memory does stay elevated rather than returning immediately, because the allocator retains freed memory for reuse. But in our testing that resident memory plateaus at a high-water mark set by peak concurrent handshakes and is then reused; it does not grow without bound across repeated rounds of the attack, including under the randomised message sizes intended to defeat reuse, and this held on both the unpatched and patched builds. The residual cost is a bounded, one-time increase set by peak concurrency, not an accumulating leak — which is why capping concurrent connections does bound it. What OpenSSL controls is how much it requests and when it frees it, and our change reduces the request itself.

Why we treated this as a hardening fix

Because some of the coverage has drawn a direct comparison, it is worth addressing head on: in the same set of releases we assigned CVE-2026-34183 (Moderate) to a QUIC PATH_CHALLENGE flooding issue, but we did not assign a CVE to HollowByte. Why the difference, when both are described as memory-exhaustion denial of service?

The PATH_CHALLENGE issue was a flaw in OpenSSL’s own protocol handling: OpenSSL placed no limit on how many PATH_CHALLENGE frames it would accept and act on, and no application configuration could prevent it. The fix had to be in the library, and an attacker could reach the condition against any deployment. That is what a CVE is for.

HollowByte is different, and the deciding factor for us was bounded versus unbounded impact. The PATH_CHALLENGE issue grew without limit and no application configuration could neutralise it — only a library change could, the defining case for a CVE. HollowByte’s allocation is bounded per connection (in our testing it plateaus and is reused, not accumulating), and the residual risk is the ordinary slow-connection problem that standard deployment controls already handle. On that basis we classified it as a hardening fix. Classification is a judgement, and if new information changes the analysis we’ll revisit it.

We should also address the report’s description of the fix as having been “silently included”, with “silent backports” to the stable releases. It was developed entirely in public: the changes were made in ordinary pull requests on GitHub (#30792, #30793 and #30794) with public commits and review, exactly as any non-security bug fix is. What we did not do is issue a security advisory or a CVE, because we did not assess the issue as warranting one — that is a deliberate classification, not concealment. We do, though, accept a narrower point: a fix developed in public is not the same as one a defender can readily find, and anyone tracking advisories or changelogs rather than commits had no clear signal to prioritise this. To address that we have retrofitted a CHANGES.md entry recording the change and the releases it appears in (pull requests #32039, #32040, #32041, #32042 and #32043).

What we changed

Although we did not consider this a vulnerability, we agreed that allocating the full declared message size before any of it has arrived is needlessly generous, and we have changed it. OpenSSL now grows the handshake buffer incrementally as data actually arrives, in chunks of at most one TLS record (16 KB), rather than trusting the declared length up front. A peer that declares a large message and then does not send it now causes no more than a single record’s worth of allocation. The relevant code carries the reasoning directly:

/*
 * We incrementally allocate the buffer to guard against the peer
 * claiming a very large message size and then not sending it.
 */

This change is present in OpenSSL 4.0.1, 3.6.3, 3.5.7, 3.4.6 and 3.0.21.

In our nginx testing, with every connection declaring a large (~65 KB) message — the largest our test harness produced — worst-case peak resident memory per worker fell from roughly 102 MB to 53 MB; under a realistic mix of message sizes, from roughly 70 MB to 55 MB. The effect scales with the declared size. A portion of the remainder, on the order of 30 KB per connection, is ordinary per-connection state present regardless of the declared length, not attacker-controlled.

OpenSSL 1.1.1 and 1.0.2 have reached end of public support and are maintained only under extended-support contracts. Customers on those contracts receive this change through their supported releases.

The same treatment is harder to apply to DTLS, where a single handshake message may be delivered as fragments that arrive out of order, and the natural implementation is to size the buffer once and place fragments into it as they come. Handling that incrementally is possible but adds real complexity, and given that the underlying concern is bounded we have not judged that complexity worthwhile for DTLS at this time. The per-message maximum still bounds the DTLS allocation, so DTLS isn’t left worse off than the pre-fix TLS path; what it doesn’t yet get is the tighter incremental behaviour. We treat that as outstanding work, not declined, and will revisit it.

Guidance for operators

Our recommendation is twofold, and the second half is the more important:

We thank the Okta Red Team again for the report, which prompted a hardening improvement we’re glad to have made. We’d encourage operators to upgrade and apply the connection-management controls above.