{"id":4523,"date":"2014-01-23T05:00:03","date_gmt":"2014-01-23T13:00:03","guid":{"rendered":"https:\/\/www.paloaltonetworks.com\/blog\/?p=4523"},"modified":"2020-04-21T14:39:56","modified_gmt":"2020-04-21T21:39:56","slug":"cybersecurity-canon-security-metrics","status":"publish","type":"post","link":"https:\/\/origin-researchcenter.paloaltonetworks.com\/blog\/2014\/01\/cybersecurity-canon-security-metrics\/","title":{"rendered":"The Cybersecurity Canon: Security Metrics"},"content":{"rendered":"

\"cybersec<\/span><\/div><\/a><\/p>\n

For the past decade, I have had this notion that there must be a <\/i>Cybersecurity Canon:<\/i><\/a> a list of must-read books where the content is timeless, genuinely represents an aspect of the community that is true and precise and that, if not read, leaves a hole in cybersecurity professional\u2019s education. I\u2019ll be presenting on this topic at RSA 2014, and between now and then, <\/i>I\u2019d like to discuss a few of my early candidates for inclusion<\/i><\/a>. I love a good argument, so feel free to let me know what you think.<\/i><\/p>\n

Security Metrics: Replacing Fear, Uncertainty and Doubt (2007) <\/b>by Andrew Jaquith<\/p>\n

I have been interested in cybersecurity metrics and how to visualize them since before we were connecting the Internet with strings and soup cans. In 2011, I had been looking for somebody to put some rigor to the idea when I stumbled upon a strong, positive review of Andrew Jaquith\u2019s book<\/a> on Amazon. A little more digging told me this was a book I really should check out.<\/p>\n

From the beginning, Jaquith attacks the security community\u2019s sacred cow of applying annualized loss expectancy (ALE) to convince management that the security program it is paying for is working. I have to say that I loved this attack. I remember first learning about ALE when I was studying for the Certified Information Systems Security Professional (CISSP) exam back in the day. I thought then that ALE sounded well and good when you said it fast, but in reality, you were just making up the numbers to plug into a formula that sounded scientific.<\/p>\n

According to Jaquith, and most every CISSP preparatory exam book on the planet,\u00a0\u201cALE is the monetary loss that can be expected for an asset due to a risk over a 1-year period and is calculated by multiplying the single loss expectancy (SLE) by the annualized rate of occurrence (ARO).\u201d<\/p>\n

Doesn\u2019t that sound precise and mathematical? Indeed it does. But it turns out that there are lots of problems with this formula. The biggest problem is that we don\u2019t know what the probabilities are. How can we possibly know what the probability is that an advanced-persistent-threat-style attack will compromise the computer that your chief of counsel\u2019s secretary uses? This is not the insurance industry; we do not have actuary tables derived from decades of data collection that can tell us precisely what these adversaries will do, how often they will do it and how much it will cost us when they do it.<\/p>\n

So what, Jaquith and others have asked, do ALE practitioners do in the absence of hard data? They guess. They estimate. They fudge. And when they do this, they undermine the veracity of the very process that they are trying to convince management is so exacting. What good is a scientific formula if all you do is fill it with garbage data?<\/p>\n

Jaquith\u2019s thesis is that, instead of using imprecise models like ALE, security professionals should use metrics instead. He says that\u00a0\u201c[this change in thinking] requires practitioners to think about security in the same way that other disciplines do \u2013 as activities that can be named, and whose efficiencies can be measured with key indicators.\u201d<\/p>\n

Coincidentally, the first time I read Jaquith\u2019s book, I just happened to listen to the Patrick Gray Risky Business podcast<\/a> from April 2011 where he interviewed Brian Snow. Snow is a former NSA information assurance technical director, and he had a lot to say then about the folly of using probabilistic risk assessments, like ALE, to improve the cost-effectiveness of securing nuclear facilitates and government information assurance programs.<\/p>\n

Snow made the point that these models are fine for standard risks that routinely occur\u2014like what is the mean time to failure of the hard drive in your laptop\u2014but that they fail miserably when trying to predict cases that have high impact to an organization but are not likely to occur. These cases that Snow referred to are called \u201cblack swan events.\u201d<\/p>\n

Black Swan Events<\/b><\/p>\n

The \u201cblack swan event\u201d term was made famous by Nassim Nicholas Taleb in his 2007 book \u201cThe Black Swan: The Impact of the Highly Improbable.\u201d For some organizations, computer breaches are black swan events that Taleb describes as \u201coutliers that carry extreme impact.\u201d They are outliers because the chances of something like that happening to your network are pretty small, but when it does, the cost to your organization is extreme.<\/p>\n

Jaquith\u2019s solution is to\u00a0\u201c\u2026 quantify, classify, and measure information security operations in a modern enterprise environment\u201d and to provide \u201c\u2026 a set of key indicators that tell customers how healthy their security operations are.\u201d<\/p>\n

He spends a good portion of his book, two entire chapters actually, explaining what some of these metrics might be. Your organization might not have a use for all of them, but you will appreciate the thoroughness that Jaquith uses to explain why they should be considered.<\/p>\n

As a bonus, he spends a chapter reviewing the fundamentals of statistics. If you are like me and slept through your probability and statistics course in college, you will welcome this refresher. Jaquith\u2019s simple explanation alone about what a standard deviation is and what correlation really means is worth the price of admission.<\/p>\n

As an extra bonus, he spends a chapter on visualization. I am a fan of Dr. Edward Tufte<\/a>, who is in my opinion the world\u2019s leading expert on how to visually display complex data. Tufte devotees will learn nothing new here but will appreciate how Jaquith reduces Tufte\u2019s four seminal books on the subject to six rules:<\/p>\n