An 11-byte message can knock OpenSSL servers offline, researchers warn

Key points

• Okta's Red Team disclosed a denial-of-service flaw in OpenSSL, nicknamed HollowByte, in an advisory published this week.
• An unauthenticated attacker can trigger the bug by sending just 11 bytes over a TLS connection.
• OpenSSL quietly patched the issue in version 4.0.1 and backported the fix to 3.6.3, 3.5.7, 3.4.6 and 3.0.21.
• In tests against NGINX, higher-spec servers lost up to 25% of their memory while attack traffic stayed below normal alerting levels.
• OpenSSL is bundled into most Linux systems, so administrators should update packages now.

A tiny message, just 11 bytes long, is enough to make some of the internet's most important servers grind to a halt.

That's the finding from Okta's Red Team, who this week published details of a flaw they call HollowByte. It sits inside OpenSSL, the free software library that handles the padlock icon in your browser and encrypts most traffic between web servers and the services behind them.

The bug is a denial-of-service issue, meaning it doesn't steal data or run code on a victim's machine. It simply makes the server stop working properly. First reported by BleepingComputer, the flaw was fixed quietly by the OpenSSL project without a formal security identifier.

How does an 11-byte message break a server?

By tricking the server into reserving memory for data that never arrives.

Here's the plain version. When two computers set up an encrypted connection, they exchange short greeting messages in a process called a TLS handshake. Each TLS handshake message begins with a 4-byte header declaring how large the next message will be.

Vulnerable versions of OpenSSL believed that promise without checking. The server would set aside a block of memory, then sit waiting for the payload. An attacker sends the 11-byte header, promises a huge payload, then sends nothing. Memory stays reserved.

Repeat that across many connections with different claimed sizes each time, and the server's memory fills with empty rooms it can never rent out again.

Okta's researchers noted a second twist. The GNU C Library, known as glibc, which manages memory on most Linux systems, doesn't always return freed memory to the operating system immediately. It holds onto small blocks for potential reuse. That means even after the attacker disconnects, the server stays bloated. Restarting it is the only clean fix.

Why this matters beyond one library

OpenSSL is almost everywhere. It ships inside NGINX and Apache web servers, inside language runtimes like Node.js and Python, and inside databases such as MySQL and PostgreSQL. Most Linux distributions include it by default. We covered a related NGINX vulnerability on 16 July in F5 Fixes Serious Security Flaws in NGINX and BIG-IP, which is worth keeping in mind when triaging the patch queue.

In Okta's tests against NGINX, a small server could be drained of memory quickly. A larger server lost about a quarter of its memory while the attacker's traffic stayed low enough to slip past typical alerting rules. A normal DDoS attack is loud and easy to spot. This one whispers.

Denial-of-service bugs are usually treated as less serious than flaws that leak data or let attackers take control. Fair enough. But a web server that has to be restarted every few hours is a real problem for the business relying on it and for the customers trying to use it.

What should administrators do?

Update. HollowByte is fixed in OpenSSL 4.0.1, with backports to 3.6.3, 3.5.7, 3.4.6 and 3.0.21. The patched versions only grow the memory buffer when data actually arrives, ignoring the header's promises.

Okta labelled its own advice bluntly: upgrade your distribution's OpenSSL packages immediately, even though the project itself calls this a "hardening fix" rather than a security vulnerability.

One more thing, honestly: multi-factor authentication wouldn't have helped here. HollowByte doesn't care who you are and doesn't need to log in. The fix has to happen on the server side, before the attacker ever gets to say hello.