21 March 2015

Metrics matter (updated)

An article by Mintz Levin about the 2013 privacy breach/information security incident at US retailer Target stated that the company has disclosed gross costs of $252 million, with some $90m recovered from its insurer leading to a net cost of $162m, up to the end of 2014 anyway (the incident is not over yet!).

Given that the breach apparently involved personal information on about 40 million people, it's trivial to work out that the incident apparently cost Target roughly $6 per compromised record ($4/record net of insurance payouts) ... but before anyone runs amock with those headline numbers, let's delve a bit deeper.

First off, what confidence do we have in the numbers themselves? The article cites its sources as 8-K filings, in other words Target's official reports concerning the incident to the Securities and Exchange Commission. Personally, I'm quite happy with that: the dollar amounts are not mere speculation but (I believe, not being a specialist in US laws and regulations) have been carefully drawn up, audited and formally approved by management - specifically Target's CFO. We could pore over the filed 8-K reports to verify them since they are published on the SEC site, or we could simply accept Mintz Levin's word: they are a law firm so there's a degree of trust and confidence. 

I took the 40 million compromised record count from a convenient web page somewhere - not as easy to verify but there are many such pages reporting similar numbers, so let's assume the figure is based on a count and disclosure by Target.  And let's assume it's correct (yes, another assumption).

Now dig further. Having tried to track and calculate the financial costs from relatively small information security incidents myself, I appreciate just how tough that can be in practice. The costs fall across two main categories, direct and indirect or consequential. The direct costs are a bit of a nightmare to monitor when everyone is running around frantically dealing with the incident at its height, but they can be estimated retrospectively and tracked fairly accurately once things calm down: it's a matter of cost accounting. Simply stated, someone assigns the direct expenses associated with the incident to an accounting code for the incident, and the financial system tots up and spews out the numbers. There are several opportunities for substantial error in there (for instance, signficant costs wrongly coded or neglected, and investments in information security/privacy improvements that would have been made anyway, regardless of the incident, being charged against it in order to secure the budgets and inflate the insurance claims), but these errors pale into insignificance against the indirect or consequential costs ...

A serious information security incident that becomes public knowledge seems likely to have an adverse impact on the organization's image and hence its brand values, but how much of an effect, in dollar terms? It's almost impossible to say with any certainty. In the case of a major incident, the company's marketing and financial people could evaluate and estimate the effects using metrics such as customer footfall, turnover, profitability, market surveys and so forth ... but potentially there is a conflict of interest there since those self-same people are charged with maintaining or boosting the company's brands and value, hence they may be understandably reluctant to report bad news to management. Furthermore, there are no easy, generally-accepted, accurate or independently-verifiable ways to convert changes in most of these metrics (such as "brand recognition") into dollars without a great deal of argument and doubt.

On top of that, there is some truth to the saying that "There's no such thing as bad news". Publicity of incidents is also publicity for the organizations and individuals involved. Publicity equates to media exposure and brand recognition hence, paradoxically, bad incidents might actually benefit those involved.

That leads us to consider stock price as another possible measure of the gross effects of an incident, one that conveniently enough is already in dollars and is widely reported, with historical data just a few click away (e.g. see the 3-year Target share price graph to the left here, courtesy of those nice people at MarketWatch.com). Given the number of shares issued (requiring a few more clicks), it's not too hard to convert the share price at any point into a market capitalization value for the company, and thus to calculate the effect the incident had on that value, but now it gets really interesting. After the incident was initially disclosed and widely reported in 2013, Target's share price declined markedly and then recovered in 2014, and is now well above the 2013 peak. What relation does that have to the incident? Again, it's almost impossible to say because there are just so many factors involved: stockbrokers, dealers and investors take a professional interest in identifying, evaluating and predicting those factors, and some of them are very successful so you might try asking them about the incident, but don't be surprised if they confuse you with statistics while keeping their trade secrets to themselves!

The same issue cropped up in the Sony hack at the end of last year. Sony's share price (plotted on the right over the past 6 months) has moved quite consistently upwards. There was a noticeable dip around the year end but it pretty much recovered its original trajectory by the end of January. I'm quite sure I could fit a straight line trend to the data with little statistical variance. 

OK, is that all there is to it? Well, no, we're not finished yet, not by a long chalk. 

So far we've only considered Target's costs: what about those whose personal information was disclosed, and the banks and other companies who have lost out to identity fraud? How much has the incident as a whole cost? How on Earth can we measure or calculate that? Once again, the short answer is that we can only estimate at best. 

What price would YOU put on the personal aggravation and grief caused by discovering that YOUR privacy has been breached and you may be the victim of identity theft? Go ahead, think about it and name your price! If enough of us did so, we might generate some sort of mean value but it's obviously highly subjective and doubtless extremely sensitive to the context and the precise questions we pose - plus of course there's the issue of our sampling strategy and sample size, since we can't ask everyone. Unfortunately, even a small error in our per-victim cost estimate will be massively amplified if we multiply that by the 40 million, so we really ought to take more care over this if the numbers matter - which they surely do as we'll come on to in a moment.

First, though, consider that the relationship between the total cost of a privacy breach/incident and the number of records disclosed is generally implied, but that is another unproven and potentially highly misleading assumption. We don't actually know the nature of the relationship, and it is likely to vary according to a number of factors aside from just the number of records. Identities belonging to the rich and famous are probably worth much more to identity thieves than those belonging to the the poor, for example, so a breach involving data from high-worth individuals, organizations or celebrities seems likely to result in greater losses than one involving the same number of records for "ordinary" people. Different items or types of information vary markedly in their inherent value (e.g. contrast the value of someone's email address or phone number to their credit card number - and then consider the additional value to fraudsters of obtaining multiple items in linked records). One might argue on basic arithmetic that the per-record costs decrease exponentially as the number of records increases, or that the relationship is non-linear due to the additional impact of news headlines with nice round figures ("more than 40 million" is worse than "almost 40 million", and far worse than "40 thousand"!). 

In privacy breaches, the black-market price of credit card numbers etc. is sometimes used to estimate the overall costs (e.g. if 'the average' record is worth, say, $2 to criminals, then $40m records are worth $80m). That simplistic approach begs various questions about how we determine the black-market price (which, by its very nature, is not openly available information), and at what point we measure it (since the value of stolen credit card numbers declines quite rapidly as word about the incident spreads and victims, banks and credit card companies progressively identify and cancel the cards). Furthermore, the costs accruing to the victims (i.e. Target and its owners/stakeholders, the data subjects, the banks and other institutions involved, oh and the FBI, police etc.) as a result of the incident may be related to but almost certainly exceed the profits accruing to the identity thieves, fraudsters and assorted middle-men exploiting it. Society as a whole picks up the discrepancies in a diffuse fashion.

That brings us to our final issue. Who cares how much infosec incidents such as this actually cost anyway? It matters because the information gets used in all sorts of ways, for example to justify investment in information security and privacy controls, incident management, insurance premiums, identity theft cover, contingency sums and more. It gets used for budgeting and benchmarking, for policy- and law-making. It feeds into our general appreciation of the information risks associated with personal information, and information risks as a whole. 

Stepping back a pace or two, this whole issue could be considered the elephant in the room for information risk and security professionals. We put enormous effort into promoting and justifying investments in information security controls to reduce the probability of, and damage caused by, incidents, trying our level best to persuade management to take heed of our concerns, support our business cases and invest adequately in security, especially proactive measures, systematic approaches and good practices such as ISO27k ... but if we look coldly and dispassionately at the situation including the assumptions and arguments laid out above, it could be said that incidents are not nearly as bad as we tend to make out, in other words we are crying wolf.  

Oh oh!  I guess we ought to firm up some of those estimates and assumptions, pronto, before we all lose our jobs! Metrics do matter, in fact.

PS The 2015 Verizon Data Breach Investigation Report attempts to define the mathematical relationship between 'Payout' and 'Records Lost' in so-called data breach incidents (see figure 21 and associated text), but acknowledges that although they have improved their model, they still don't have a firm grasp of all the relevant factors. Perhaps this blog piece will prompt them to re-evaluate their assumptions and presumptions, maybe even to do the research given the data and other resources available to them. Don't hold your breath though. I fully expect the mythical linkage between incident costs and records compromised to persist for many years yet, despite my best efforts. Its the infosec equivalent of the search for the holy grail - the Monty Python version. 

03 March 2015

Comparative security metrics

In situations where it is infeasible or impracticable to quantify something in the form of a discrete count or a value in specific units, comparative or relative measures are a useful alternative. They are better than not measuring at all, and in some cases easier to comprehend and more useful in a practical sense. In this respect, we disagree with those in the field who fervently insist that all metrics must be expressed as numbers of units (e.g. "20 centimetres"). It seems to us "A bit longer than a pencil", while obviously imprecise, might be a perfectly legitimate and helpful measure of something (regardless of what that thing might be - a cut on your arm for instance).

Cardinal numbers and units of measure have their place, of course, but so do ordinals, comparatives and even highly subjective measures - all the way down to sheer guesswork (and, yes, 'down to' itself implies a comparative value). Douglas Hubbard's "How To Measure Anything" is an excellent, throught-provoking treatise on this very subject.

In information security, comparisons or relations can provide answers to entirely valid and worthwhile questions such as:

  • Are we more or less secure than our peers?
  • Are we getting more or less secure over time?
  • If we both sustain our present rate of change, how long will it be before we'll surpass our competitors' level of information security?
  • Are our information risks increasing or decreasing?
  • Which are our strongest and weakest areas or aspects of security?
  • Of all the myriad changes currently occuring in information security, what are the most worrying trends?
  • Does information risk X fall within or exceed our risk appetite or tolerance?
  • Which business unit, function, department or site is the most/least vulnerable?
  • Are we spending too little, about the right amount, or too much on information security
As part of an information security awareness case study on 'the Sony hack', a management discussion paper describes three types of comparative security metrics with several examples of each.