2.47 Million Simulated Attacks Suggest We're Measuring the Wrong Things in Phishing Training
New research points out a gap between what most companies track (who clicked a fake link) and what actually matters (whether stolen passwords are being spotted and staff are reporting suspicious emails).
Key points
- A study analysing 2.47 million simulated phishing attacks found that click rates alone are a poor measure of whether staff can actually resist real attacks.
- Researchers say organisations should also track credential theft, meaning how often fake login pages successfully capture staff passwords, and how many suspicious emails employees report.
- Most companies currently treat a drop in click rates as proof their training is working, but the research suggests that view is too narrow.
- Security awareness testing, the regular practice of sending fake scam emails to employees to see who falls for them, needs a broader scorecard.
Most security teams running phishing simulations, those regular tests where a company sends its own staff a fake scam email to see who gets tricked, have one number they care about: the click rate. How many people clicked the dodgy link? That figure goes into a dashboard, gets reported to the board, and shapes next quarter's training budget.
New research, covered by SecurityWeek, suggests that one number is not enough.
An analysis of 2.47 million simulated attacks found two problems with a click-only view. First, it ignores credential harvesting, which is what happens when a fake login page quietly records the username and password a victim types in. Clicking a link is bad. Handing over your actual password is significantly worse, yet plenty of organisations never measure whether their staff are doing exactly that.
Second, it ignores the other side of the equation entirely: reporting. A phishing email that gets spotted and flagged by an employee before anyone clicks is a win. In practice, most simulations record that win as a neutral result, because nobody clicked. That is a flawed accounting.
The failure mode here is familiar to anyone who has written a postmortem. You measure what is easy to instrument, not what actually matters. Click rates are easy. They produce a clean percentage. They trend down over time, which keeps everyone comfortable.
But a real phishing attack, the kind that empties a bank account or hands criminals the keys to a corporate email system, does not care about your click rate trend. It cares whether one person on a Monday morning types their password into a convincing fake Microsoft 365 login page, a cloud-based office software platform used by millions of businesses worldwide.
What should companies do differently?
Three changes follow directly from the research. Organisations running phishing simulations should add fake login pages to some of their tests and record how many staff actually type in credentials. They should build a clear reporting channel so staff can flag suspicious emails with one click, and then measure how often that happens. Finally, both numbers, credential submission rates and reporting rates, should sit alongside click rates in any security dashboard.
For employees themselves, the practical takeaway is simple: if an email asks you to log in somewhere, go directly to the website by typing it yourself rather than clicking any link in the message. If something looks odd, tell your IT or security team, even if you are not certain.
One thing the post-mortem will say, after the real breach, is that several people saw the email and thought it looked suspicious but said nothing, because nobody told them saying something was part of the job.
Measure what causes harm, not what fits neatly in a report.
