Security metric #57: % of information assets classified

Security Metric of the Week #57: Proportion of information assets correctly classified

Patently, this metric relates to the classification of information, an important form of control.  

The assumption underlying classification is that the majority of an organization's information is neither critical nor sensitive.  It is therefore wasteful to secure all the information to the extent that is appropriate for the small amount that is highly critical or sensitive.  Likewise, the basic or baseline controls that are appropriate for most information are unlikely to be sufficient for the more critical or sensitive stuff.

The classification process can be as simple or as complicated as you like, according to the number of classes.  Taken to extremes:

• A single classification level such as "Corporate Classified" could be defined in which case everything would end up being protected to the same extent.   
• More likely, certain important items of information would be deemed "Corporate Classified" with the remainder being "Corporate Unclassified", meaning a two-level classification scheme (OK, three if you count the information assets that have yet to be classified!).
• At the opposite end of the scale, the classification could be so granular in detail that many classes contain just a single information asset with a unique set of security controls for that specific asset.
• Classification is essentially a pointless exercise at both extremes.  It's value increases in the middle ground where 'a reasonable number' of classes are defined, each containing 'a reasonable number' of information assets.  It's up to you to determine what's reasonable!

The driver for classification is also a variable.  Although we mentioned 'criticality' and 'sensitivity', those are not the only parameters.  For example, picture a 3x3x3 Rubik's cube with low-medium-high categories for confidentiality, integrity and availability, or a classification scheme that depends on the value of the information, howsoever defined.  

Military and government classification schemes appear quite simple in that they are largely or exclusively concerned with confidentiality (e.g. Secret, Top Secret, Ultra), but there are numerous wrinkles in practice such as subtly different definitions of the classes by different countries, and subsidiary markings identifying who is authorized to access the information. 

Corporate classification schemes commonly distinguish personal information, trade secrets, other internal-use information and public information, but again there are numerous variations.

Classifying information involves two key steps: 

1. The information is assessed to determine the appropriate class using defined classification criteria.  
2. Information security controls deemed appropriate for the particular classification level are applied.  

This week's example metric concerns step 1, and is only indicative of step 2 if we assume that a sound process is being followed religiously.   Step 2 could be measured independently using a suitable compliance metric.

The illustrative graphic above shows an hypothetical organization systematically assessing and classifying its information assets, measuring and reporting the metric month-by-month.  The graph plots "Proportion of information assets correctly classified" by month.  The simple Red-Amber-Green color-coding makes it obvious that things have improved substantially since the start of the initiative, with two step-changes in the levels presumably representing discrete projects or stages that made significant progress.

Actually measuring this metric could be something of a mission if you insist on doing so accurately (more on that point below).  First, since you are reporting a proportion, you need to determine the size of the whole, in other words how many information assets are there to be classified, in total?  Answering that further requires clarity over what constitutes an information asset.  Leaving aside the question of whether the term includes ICT hardware and storage media, or just the information/data content, the unit of analysis is also unclear.  For instance, does a customer database containing 1,000 customer records each with 100 fields count as one information asset, or 100, or 1,000, or 100,000, or some other number?   The answer is not immediately obvious.

In the same vein, the metric explicitly refers to assets being 'correctly' classified implying that, strictly speaking, someone should check the veracity of the classifications - potentially a huge amount of work and additional cost just for the sake of the metric.  

On the other hand, clarity over 'information asset' and 'correctly classified' may have value to the organization's information security beyond the metric.

Anyway, let's pick up on that point about the accuracy requirement for this metric.  Since we are reporting a proportion, the absolute numbers are less important than their relative quantities.  Rather than accuracy, consistency of the measurement approach is the primary concern.  With that in mind, it doesn't particularly matter how we define 'information asset' or 'correctly classified' just so long as the definition remains the same from month to month.  For various other reasons, it may occasionally be necessary to alter the definitions, in which case we should probably re-base prior values in order to maintain consistency of the metric.

Another big advantage of reporting a proportion is that it is possible to select and measure a representative sample of the population - 'representative' being the crucial term.  We're not going to discuss sampling methods today, though.  If you need more, there are brief notes about sampling in PRAGMATIC Security Metrics, while any decent statistics text covers it in laborious detail.

The excellent PRAGMATIC ratings indicate this metric is a hit for Acme Enterprises Inc:

P	R	A	G	M	A	T	I	C	Score
75	75	97	85	90	80	80	80	80	82%

In discussing various candidate metrics, Acme's managers were particularly impressed with this one's Actionability and clarity of Meaning (notwithstanding the notes above - presumably they already had a clear picture in the areas mentioned).   Driving up the proportion of information assets correctly classified was seen as a valid and viable goal to improve information security - not so much a goal in itself but a means of achieving a general security improvement for Acme as a whole, on the reasonable assumption that, following classification, security resources would be applied more rationally to implement more appropriate security controls.

Security metric #56: embarrassment factor

Security Metric of the Week #56: embarrassment factor

This naive metric involves counting the privacy breaches and other information security incidents that become public knowledge and so embarrasses management and/or the organization.  The time period corresponds to the reporting frequency - for example it might be calculated and reported as a rolling count every 3-12 months, depending on the normal rate of embarrassing incidents.  

In bureaucratic or highly formalized organizations, it would be a challenge even to define what constitutes 'embarrassing', although most of us can figure it out for ourselves without getting too anal about it.

The metric's purpose, of course, is to reduce the number of embarrassing breaches/incidents that occur, which may involve reducing the rate of breaches/incidents and/or reducing the extent to which they are embarrassing.    With that end in mind, the precise definition of 'embarrassing' doesn't actually matter much, just so long as the audience appreciates that the metric fairly indicates the underlying trend.  Annotating the graph to remind viewers about specific incidents should have the desired effect.

In PRAGMATIC terms, ACME management rated this metric at 54%, in other words it would be unlikely to make the cut in their Information Security Measurement System or Executive Security Dashboard.  However, this is such a simple, easy and cheap metric to generate that the CISO might like to keep an informal tally of embarrassing incidents for his/her own purposes.  So long as the trend remains positive, the metric has little impact.  On the other hand, if ACME experiences a rash of embarrassing incidents, mentioning the metric's adverse trend could be an opportunity for the CISO to raise the matter with senior management.  

Sometimes, getting things on the agenda is half the battle.

Security metric #55: policy coverage

Information Security Metric of the Week #55: information security policy coverage of frameworks such as ISO/IEC 27000

In much the same way that two-dimensional maps of three-dimensional landscapes are useful for hill-walkers, various frameworks, standards and methods such as ISO27k, SP800-53, COBIT and the Standard of Good Practice are useful guides for navigating the field of information security.  Just as cartographers must transform the literal land into graphic representations on the map, the standards bodies and assorted authors take somewhat arbitrary decisions about which elements of information security to cover and in what sequence.

For example, section 7 of ISO/IEC 27002:2005 covers two distinct but related issues in asset management: 7.1 Responsibilities for protecting information assets, and 7.2 Classification of information.  Those two aspects could have been scoped and titled differently and might have been placed in separate sections or incorporated into other sections of the standard but the ISO/IEC committee, in its wisdom, chose to cover them both together in section 7.  

Security responsibilities and information classification are relevant to various information security risks and control objectives, hence they (along with most other controls) could have been discussed from different perspectives in several parts of the standard.  However this would have created duplication and confusion.  Instead, the controls are each discussed once and, where necessary, cross-referenced elsewhere.

ISO/IEC 27002 provides a convenient map that is widely understood.  Aside from the structure - more importantly in fact - the standard lays out a reasonably comprehensive suite of information security controls that could be considered a basic or minimal set: with some exceptions, most organizations that take information security seriously are using most of the controls listed in the standards.  Therefore, comparing an organization's information security controls against those recommended in the standard to identify any gaps is one way to measure the comprehensiveness of its controls.

That said, ISO27k is imperfect.  Aside from issues with the wording and meaning of the standard when it was published, there is a further dynamic aspect.  ISO/IEC 27002:2005 has become outdated in various respects, for example it does not explicitly and comprehensively cover cloud computing since cloud computing was barely even conceived when the standard was drafted.  With some artistic license, several recommended controls in the standard can be interpreted in the cloud computing context, but other necessary controls are either completely missing from the standard or are of limited value as currently worded.  To fill-in the gaps, we could wait for the standard to be updated and released (later this year, hopefully), or we could use various other security standards and frameworks in the meantime, supplementing them with advice from information security, risk, compliance, governance and related professionals, tailored to our specific circumstances.

Against that background, let's look at the value of a metric that measures the extent to which the organization's security policies cover the entire security landscape.

When they assessed this metric using the PRAGMATIC method, ACME management had in mind using their own information security coverage map which had been drawn up by the CISO to reflect the common ground across several security standards.  They envisaged the CISO systematically checking for discrepancies between the suite of policies and ACME's map and drawing up a simple color-coded coverage diagram similar to that shown above - red meaning "Inadequately covered", amber being "Partially covered" and green for "Fully covered".

P	R	A	G	M	A	T	I	C	Score
70	75	90	69	85	76	72	65	85	76%

  

The managers recognized a potential bias in that the person assessing and measuring the policies also owns them.  The CISO might honestly believe that one or more given ACME policies entirely cover part of the coverage map, whereas another security professional might feel that the policies don't go far enough to address the associated risks.  They could get around this limitation by commissioning an independent consultant or auditor to assess and measure the policies, and perhaps by separately measuring the correspondence between their information security map and applicable standards.  They might even go as far as to adopt the excellent Unified Compliance Framework, a rigorous synthesis of information security-related recommendations and obligations drawn from practically all the standards and laws in this area.  On the other hand, all that extra work would markedly delay the production of the metric and increase the costs.  A more pragmatic approach might be to have someone from Internal Audit or Risk Management cast a cynical eye over the scoring and challenge the CISO to justify her decisions - a process known as normalization in the world of metrics.  The CISO would also be asked to make notes during the measurement which would be useful for planning updates both to the policies and to the coverage map ... and here we're already talking about using the metric to inform decisions, implying that it definitely has potential.  In summary, the metric's 76% PRAGMATIC score feels right.

This is just one of a few similar metrics discussed in the book, and it would not be hard to think up many more along these lines, including variants of the ones we have discussed, similar metrics proposed elsewhere, and novel metrics invented for this purpose.  The PRAGMATIC method enables us to analyze and compare the metrics in a rational and systematic way, forcing us to think through the pros and cons of each one before selecting "a few good information security metrics".  We don't mean to trivialize the effort required to complete the metrics design, specify any mathematical analysis and presentation, implement them and of course use them, but PRAGMATIC gets us over by far the biggest obstacle: selecting the right metrics.

Securing the security metrics

At the risk of appearing security-obsessed, I'd like explore the information security risks and control requirements that should be taken into account when designing an information security measurement system, particularly if (as is surely the aim) the metrics are going to materially affect the organization's information security arrangements. 

I'm talking here about the measurement system as a whole, not just the elements and metrics within it.   Information security is undoubtedly a concern for the executive suite's information security dashboard, the metrics database maintained by the CISO, and the monthly metrics report, but I'm taking a broader perspective.

It is not appropriate for me to propose specific information security controls for your information security measurement system since I can barely guess at your circumstances - the threats, vulnerabilities and impacts in relation to your security metrics, your business situation.  However, the rhetorical questions that follow will hopefully prompt you to explore the security and related requirements for your metrics, and think about what matters to your organization:

• How are the source data for metrics - the base measurements - obtained?  Are the sources themselves (both automated and manual) sufficiently trustworthy?  Where are the biases, and how severe are they?  Are sufficient data points available to generate valid and useful statistics?  How much variability/variation is there, and how much should there be?  Does measurement variance itself qualify as a worthwhile metric?!?
• Who is gathering, storing and processing measurements?  Are they sufficiently competent, diligent and trustworthy?  Are they well trained?  Do they follow documented procedures?  Are the criteria and processes for taking measurements sufficiently well defined so as to avoid ambiguity and to reduce the potential for abuse or fraud (e.g. selective use of ‘beneficial’ or positive data and disregard of negative values)?
• What about the IT systems and programs supporting the measurement processes: has anyone actually verified that analytic tools, spreadsheets and databases are correctly, accurately and completely processing measurement data?  Are changes to the systems that generate, analyze and present security metrics properly managed, for instance are code or design changes adequately specified and tested before release?  If the measurement processes or systems change, are prior data properly re-based or normalized for trends analysis?
• Is there a rational and systematic process for proposing, considering and selecting security metrics?  Does it cope with changes to the information requirements or emphasis/priorities, new opportunities, newly-identified information gaps or constraints, novel metrics suggestions etc.?  Is there a rational mechanism for specifying, testing, implementing, using, managing, maintaining and eventually retiring metrics?
• Do metrics reporting processes accurately present ‘the truth, the whole truth, and nothing but the truth’?  How can we ensure sufficient objectivity and accuracy of reported data, and what do we mean by 'sufficient' anyway?  Is there potential at any part of the process for malicious actors to meddle with things?  Where are the weakest points?  Where might the threats originate?  What's in it for them?
• Are good decisions made sensibly and rationally on the basis of the metrics?  Is the information used in the best interests of the organization?  Or are people intentionally or unknowingly playing games with them?  Does anyone monitor this kind of thing, or indeed the other issues raised here, and act accordingly?
• How reliable are the metrics?  How reliable do they need to be?  Are some metrics absolutely crucial, supporting business decisions that could prove disastrous if incorrect?  Are there corroborating sources for such metrics, or ways to cross-check the metrics and/or correct the decisions?  Are any of the metrics of limited/marginal value, making them candidates for retirement, reducing the amount of distracting noise as well as cutting costs?
• How serious would it be if the metrics turned up late?  Would important meetings or decisions be delayed?  Would this cause compliance issues?  What if the metrics were completely missing - for whatever reason they could not longer be provided?  Would people be forced to limp along without them?  Might there be alternative sources of information - and if so, would they be as good?  Are there situations where rough estimates, provided much sooner, would be at least as good if not better than more accurate and factual metrics provided later?
• Given that they concern the organization's information security, are the metrics commercially confidential?  Are any of them particularly sensitive?  Would anyone else be interested in them, outside the intended audience?  Could they infer decisions and actions on security, incident levels and costs, vulnerabilities etc. from the metrics?  Conversely, are any of the metrics suitable for wider publication, consideration and use, for example in awareness or marketing?  Would any of them be beneficial and valuable for employees in general, business partners, sales prospects, authorities/regulators, auditors, owners or other stakeholders?  Are any of them dangerous in the governance sense, being undeniable evidence that management has been made aware of certain issues and consequently can be held to account for their decisions?

To close, I'll just mention that these generic considerations apply in much the same way to virtually ANY measurement system in ANY context: financial metrics, HR metrics, strategic metrics, risk metrics, product metrics, health and safety metrics, societal and political metrics, scientific metrics ... you name it.  Maybe it's worth talking to your colleagues about their metrics too.

Security metric #54: documentation of important operations

Security Metric of the Week #54: number of important operations with documented and tested procedures

At first glance, this week's example metric doesn't sound very promising.  The wording is ambiguous, its value unclear and its  purpose uncertain.

If you were a senior executive sitting on "mahogany row", trying to select some information security metrics to use as part of your governance role, you might well be tempted to reject this metric without further ado, given obvious concerns such as:

1. It implies a straightforward count, giving no indication of how many "important operations" remain to be procedurally documented and tested. 
2. What are "important operations" anyway?   Is it referring to business processes, information security processes, IT processes, activities, tasks or something else?  Who decides which ones qualify as "important", and on what basis?
3. "Documented and tested" sticks two distinct issues into the one metric.  Does it mean that the documentation is tested, or the "important operations" are tested, or both?

On the other hand, imagine there was a corporate policy that the organization's business-critical processes should be documented and the documentation quality tested, this could be a worthwhile compliance metric, useful to drive through the documentation of key business operations.  The graph above shows how the metric might indicate the number of such processes that are both documented and tested (upper line) and documented but not yet tested (lower line), addressing point 3.  Furthermore, if the policy explicitly referred to "the top fifty business-critical processes" or the top ten or whatever, then concern 1 would also be addressed.  

It is clear from their analysis that ACME's management took a real shine to this metric, giving it an overall PRAGMATIC score of 84%.  The phrase "important operations" evidently Means something specific in ACME's corporate lingo, and since they also rated the metric high on Predictability and Relevance, they must believe that the documentation and testing of those "important operations" is key to ACME's information security.

This is a classic illustration of the drawbacks of those generic lists or databases of 'recommended' or 'best practice' or 'top N' information security metrics.  The organizations and individuals behind them undoubtedly mean well but, as Anton Aylward quite rightly keeps reminding us, context is everything.  In your situation, this metric may be as good as ACME's managers believe, maybe even better.  For many organizations, however, it is mediocre at best and probably outshone by others.  The PRAGMATIC method gives us the means not just to say metric X is better than metric Y, but to explain why, and to develop and discuss our reasoning in some depth.

There may be particular reasons why this metric scores so well right now for ACME.  Perhaps there is a corporate initiative to improve the documentation of ACME's business-critical processes as part of a drive towards ISO/IEC 27001 certification.  A year or so into the future, when most if not all of the processes are documented and tested, the metric will probably have outlived its usefulness.  ACME managers will find it straightforward to reconsider this year's PRAGMATIC ratings and the associated notes to remind themselves what made them favor the metric before, updating their thinking and the PRAGMATIC score in their regular metrics review.  Retiring this metric will be no big deal.

Compare the enlightened, rational and consensual PRAGMATIC approach to those dark and dismal days when we used to sit around endlessly complaining about metrics and sniping at each other.  What started out with someone insisting that we needed to "sort out our security metrics" soon turned into a bun-fight, each of us becoming ever more entrenched in defending our pet metrics while dismissively criticizing our colleagues'.   The horribly divisive and unsatisfying process meant that, once the dust had settled, there was very little appetite to review the metrics ever again, except perhaps for those battle-scarred veterans who relished every opportunity to re-play the same old arguments, more stridently each  time.   Without regular reviews, the metrics gradually decayed until eventually the whole vicious cycle kicked off again with someone insisting that we "sort out our security metrics" ...

We've been there, done that, soiled the bandages, but does this ring true to you, or are we barking up the wrong tree?  Is it all sweetness and light in your organization, or does your C-suite resemble Flanders whenever metrics are discussed?  Do let us know ...

Security Metric #53: entropy

Information Security Metric of the Week #53: entropy of encrypted content

Randomness is a crucial concept in cryptography.  Aside from steganography, strongly encrypted information  appears totally random with no discernible patterns or indicators that would give cryptanalysts clues to recover the original plaintext.  

"Entropy" is a convenient term we're using here to describe a measure of randomness or uncertainty - we're being deliberately vague in order to avoid getting embroiled in the details of measuring or calculating this metric.  And, to be frank, because Shannon goes way over our heads.

We envisage ACME using this metric (howsoever defined) to compare encryption systems or algorithms on a common basis, for instance when assessing new encryption products for use in protecting an extremely confidential database of pre-patent information.  Faced with a shortlist of products, management seeks reassurance as to their suitability beyond the vendors' marketing hyperbole.  The assessment process involves encrypting one or more specific data files with each of the systems or algorithms, then determining the randomness of the resulting ciphertexts using an appropriate mathematical calculation, or indeed several.  For completeness, the calculations might be repeated using a variety of encryption keys in case any of the systems/algorithms has limitations in that respect.  The ones that produce the most random ciphertext are the strongest encryption systems/algorithms.  QED.

The PRAGMATIC ratings for this metrics are mostly high, apart from a glaring exception: Meaningfulness rated a pitiful 3% when the metric was assessed by ACME's management, since it appears Shannon went way over their heads too!  The overall PRAGMATIC score of 59% would no doubt have been much higher if management understood the concept.  In any case, the metric is of interest to ACME's IT and information security professionals involved directly in the product selection process, in other words this could be a worthwhile operational as opposed to management metric, even if the teccies need to  explain the end result to their bosses, patiently, in monosyllabic terms!

PRAGMATIC Security Metric of the Year, 2013

Having just discussed our fifty-second Security Metric of the Week here on the blog, it's time now to announce our top-rated example security metrics from the past year.  

<Cue drum roll>

The PRAGMATIC Security Metric of the Year, 2013, is ... "Security metametrics"

<Fanfare, riotous applause>

Here are the PRAGMATIC ratings for the winner and seven runners-up, all eight example metrics having scored greater than 80%:

Example metric	P	R	A	G	M	A	T	I	C	Score
Security metametrics	96	91	99	92	88	94	89	79	95	91%
Access alert message rate	87	88	94	93	93	94	97	89	79	90%
Business continuity maturity	90	95	70	80	90	85	90	87	90	86%
Asset management maturity	90	95	70	80	90	85	90	85	90	86%
Infosec compliance maturity	90	95	70	80	90	85	90	85	90	86%
Physical security maturity	90	95	70	80	90	85	90	85	90	86%
HR security maturity	90	95	70	80	90	85	90	85	90	86%
Security traceability	85	89	88	90	91	87	65	84	85	85%

Before you rush off to implement these eight metrics back at the ranch, please note that the PRAGMATIC scores were calculated in the context of an imaginary organization, ACME Enterprises Inc.  They reflect ACME's situation, and ACME management's perspectives, understanding, prejudices and measurement objectives.  They are merely worked examples, demonstrating how to apply the PRAGMATIC method in practice.  You may well already have better security metrics in place, and we know there are many other excellent security metrics - not least because there are other high-scoring examples in the book!  In short ...

Y M M V 

Your Metrics May Vary

You have no doubt noticed that five of the top eight are "maturity metrics", and if we include "security metametrics", fully six of the top eight are our own invention ... which probably reveals a bias in the way we scored and ranked the metrics.  These six are our babies and, naturally, we love them to bits, warts and all.  We are blind to their imperfections.  On the other hand, using the PRAGMATIC approach, we have elaborated in some detail on why we believe they are such strong candidates for ACME's information security measurement system.  We've shown our workings, and actively encourage you to review and reconsider these and other candidate metrics in your own contexts.  

It might be nice if we could develop and agree on a comprehensive suite of universally-applicable information security metrics, particularly as we now have a more rational approach than "Trust us, these are great security metrics!"   However, that may be just a pipe-dream since we are all so different.  Is it realistic to presume that the half-dozen information security metrics that have been chosen by, say, a small charity would also feature among the two dozen selected by a large bank, or the four dozen imposed on a government department by some regulatory authority?  We suspect not, but  having said that we would be delighted to reach a consensus on a handful of PRAGMATIC security metrics that have proven themselves invaluable to almost everyone.

OK, that completes the first year of our cook's tour of information security metrics.  In the months ahead, we plan to continue discussing and scoring other example metrics from the book, along with various others that pop into our consciousness from time-to-time.   If you'd like us to consider and score your favorite information security metric, then why not join the security metametrics discussion forum and tell us all about it?  Does yours score above 80%?  What makes it shine?