An 11-byte message can knock OpenSSL servers offline, researchers warn

Key points

• Okta's Red Team disclosed a denial-of-service flaw in OpenSSL, nicknamed HollowByte, in an advisory published this week.
• An unauthenticated attacker can trigger the bug by sending just 11 bytes over a TLS connection.
• OpenSSL quietly patched the issue in version 4.0.1 and backported the fix to 3.6.3, 3.5.7, 3.4.6 and 3.0.21.
• In tests against NGINX, higher-spec servers lost up to 25% of their memory while the attack traffic stayed below normal alerting levels.
• OpenSSL is bundled into most Linux systems, so administrators should update packages now.

A tiny message, just 11 bytes long, is enough to make some of the internet's most important servers grind to a halt.

That is the finding from Okta's Red Team, who this week published details of a flaw they call HollowByte. It sits inside OpenSSL, the free software library that handles the padlock icon in your browser and encrypts most traffic between web servers, apps and databases.

The bug is a denial-of-service issue, meaning it does not steal data or run code on a victim's machine. It simply makes the server stop working properly. First reported by BleepingComputer, the flaw was fixed quietly by the OpenSSL project without a formal security identifier.

How does an 11-byte message break a server?

By tricking the server into reserving memory for data that never arrives.

Here is the plain version. When two computers set up an encrypted connection, they exchange short greeting messages in a process called a TLS handshake. Each greeting starts with a small header that says, in effect, "a message of this size is coming next."

Vulnerable versions of OpenSSL believed that promise without checking. The server would set aside a block of memory to hold the incoming message, then sit and wait for it. An attacker sends the 11-byte header, promises a huge payload, and then sends nothing. The server keeps waiting. The memory stays reserved.

Repeat that across many connections, with different claimed sizes each time, and the server's memory fills up with empty rooms it can never rent out again.

Okta's researchers noted a second twist. The GNU C Library, known as glibc, which handles memory on most Linux systems, does not always hand freed memory back to the operating system straight away. It holds onto small blocks in case they are needed again. That means even after the attacker disconnects, the server stays bloated. The only clean fix is to restart it.

Why this matters beyond one library

OpenSSL is almost everywhere. It ships inside the NGINX and Apache web servers, inside programming languages like Node.js, Python, Ruby and PHP, and inside databases such as MySQL and PostgreSQL. It comes pre-installed on most Linux distributions.

In Okta's tests against NGINX, a small server could be drained of memory quickly. A larger server lost about a quarter of its memory while the attacker's traffic stayed low enough to slip past typical alerting rules. That is the uncomfortable part: a normal DDoS attack is loud and easy to spot, but this one whispers.

Denial-of-service bugs are usually treated as less serious than flaws that leak data or let attackers take control. Fair enough. But a web server that has to be restarted every few hours is still a real problem for the business relying on it, and for the customers trying to log in, book an appointment or make a payment.

What should administrators do?

Update. HollowByte is fixed in OpenSSL 4.0.1, with backports to 3.6.3, 3.5.7, 3.4.6 and 3.0.21. The patched versions only grow the memory buffer when data actually arrives, ignoring the promises in the header.

Okta labelled its own advice bluntly: upgrade your distribution's OpenSSL packages immediately, even though the project itself calls this a "hardening fix" rather than a security vulnerability.

And this one is worth noting honestly: multi-factor authentication would not have helped. HollowByte does not care who you are. It does not need to log in. That is precisely why the fix has to happen on the server side, before the attacker ever gets to say hello.