Security Metrics: Abuse Metrics vs. Security Metrics
Measuring security effectively involves understanding not only the type of risks but also the appropriate metrics that reflect real-world threats. Two crucial categories of metrics—abuse metrics and security metrics—serve distinct purposes, each critical in their own way. Here’s why I think distinguishing them matters.
Abuse Metrics: Measuring Real-World Impact
Abuse metrics quantify actual instances of misuse or harmful behavior occurring in the real world. Examples include spam rates, account takeovers, fraudulent transactions, and ToS violations. These metrics directly measure negative experiences users face daily.
The key characteristic of abuse metrics is that they deal with frequent, measurable events. For instance, reducing spam emails or phishing attempts by 90% directly translates to an improved user experience. Even if some abuse remains, such a substantial reduction represents significant real-world progress.
Security Metrics: Targeting Zero Exploits
Security metrics, on the other hand, focus primarily on vulnerabilities that could potentially be exploited to achieve great user harms. Unlike abuse, the objective in security isn’t merely reduction—it’s elimination. Why? Because even a single vulnerability exploitation can have catastrophic consequences.
Consider an RCE vulnerability: just one successful exploit can grant an attacker full control over critical systems, causing extensive harm and reputational damage. Thus, security metrics aim not at incremental reduction but at the absolute target of zero vulnerabilities.
Why Security Exploits Can’t be Measured the Same Way as Abuse
Security exploits differ fundamentally from abuse incidents in two critical aspects:
1. Rarity:
Security exploits, especially severe ones, are rare, targeted, and potentially invisible until catastrophic events occur. Because they’re infrequent, it’s impractical and unreliable to measure their occurrence directly, as one would with abuse metrics.
2. Non-Stochastic Attack Patterns:
Attackers who exploit security vulnerabilities don’t operate purely based on random chance or predictable probabilities. They’re strategic, continuously seeking and targeting the weakest points in a system. Their behavior can’t be effectively modeled through probabilistic frameworks alone. Simply put, probabilities don’t capture the intentionality and adaptability of human attackers.
A Better Approach: Measuring Security Health
Given these challenges, a more effective approach isn’t measuring security risks purely as probabilities or incident counts. Instead, it’s about assessing security health—evaluating how inherently secure an ecosystem is by design and how rigorously it adheres to secure development practices and principles.
Security health metrics can include:
- Adoption of secure by design frameworks
- Implementation of overlapping hardening mitigations to reduce the risk of compromise
- Adoption of other security best practices (e.g. keeping 3P libraries updated, vulnerability scanning, rigorous code reviews, etc)
- Robust threat modeling and penetration testing procedures
- Effective incident response readiness
By focusing on security health, metrics can guide proactive and systematic reduction in risk, significantly decreasing the likelihood of severe security breaches.
Conclusion
Security risk measurement requires a nuanced understanding: abuse metrics capture and quantify actual measurable harm, offering clear reduction targets, while security metrics must aim higher, toward zero tolerance. By shifting security metrics’ focus to security health and secure design principles, metrics can help guide security organizations to positive outcomes.