A newly disclosed vulnerability reminds us how deeply our digital infrastructure relies on foundational libraries.
The Okta Red Team recently discovered “HollowByte,” a Denial of Service (DoS) flaw in OpenSSL that allows a remote, unauthenticated attacker to force a server to allocate disproportionate memory chunks before any security handshake even begins, using a payload just 11 bytes long.
The TLS handshake begins with a ClientHello message wrapped in a record. Each handshake message carries a 4-byte header that indicates the size of the incoming message body.
OpenSSL DoS Vulnerability
Older versions of OpenSSL allocate a receive buffer based on that attacker-declared length before any data has actually arrived. When the malicious 11-byte payload lands, the TLS state machine reads the header and triggers an unvalidated pre-allocation based on its 3-byte length declaration:
Read Header → grow_init_buf() → OPENSSL_clear_realloc() → malloc(attacker_size)
Since there’s no payload validation at this stage, malloc() allocates up to 131 KB based solely on the untrusted packet’s claim. The worker thread then blocks indefinitely, waiting for data that never arrives.
Holding connections open to exhaust threads is a classic trick, similar to Slowloris. HollowByte compounds this with a nastier side effect rooted in how glibc handles memory.
When a connection drops, OpenSSL frees the buffer, but glibc doesn’t immediately return small-to-medium allocations to the OS; it retains them for potential reuse.
By launching waves of connections with randomized claimed sizes, an attacker prevents the allocator from reusing freed chunks. The heap fragments heavily, and the server’s Resident Set Size (RSS) continues to climb.
Even after the attacker disconnects, the server remains bloated; the only fix is to kill the process. Testing unpatched and patched OpenSSL instances under NGINX revealed the scale of the threat. In a 1 GB RAM environment, the unpatched server was OOM-killed at 547 MB of frozen, fragmented memory.
In a 16 GB RAM environment, the attack locked up 25% of total system memory while staying safely under typical connection ceilings, meaning standard connection-limiting defenses won’t stop it.
Because OpenSSL is so widely embedded, this flaw affects web servers (Apache, NGINX), language runtimes (Node.js, Python, Ruby, PHP), and databases (MySQL, PostgreSQL).
OpenSSL resolved the issue by switching to incremental buffer growth, merged via PRs #30792, #30793, and #30794. The fix shipped silently in OpenSSL v4.0.1, with backports to 3.6.3, 3.5.7, 3.4.6, and 3.0.21.
Rather than trusting the declared header size outright, OpenSSL now grows buffers only as bytes actually arrive on the wire, so an empty claim costs the server nothing. Notably, OpenSSL classified this as a hardening fix rather than issuing a formal CVE advisory.
Regardless, security teams should prioritize upgrading their OpenSSL packages immediately, particularly on internet-facing servers where connection-limiting alone offers no protection against this attack.
Strengthen Your SOC by Accelerating Threat Detection & Rapid Investigations. -> Integrate ANY.RUN With Your SOC Now.
