Security awareness metrics

Some say that information security awareness is hard to measure, and yet a moment's thought reveals several obvious, straightforward and commonplace metrics in this area, such as:

• Attendance numbers, trends, rates or proportions at awareness and training events;
• Feedback scores and comments from attendees at/participants in said events, or concerning other awareness activities, promotions, media, messages etc.; 
• General, broad-brush, state-of-the-nation security awareness surveys of various populations or constituencies conducted on paper or using electronic forms or polls;
• More specific information recall and comprehension tests relating to awareness topics or sessions, conducted on paper or online (maybe through the Learning Management System);
• Awareness program metrics concerning activities planned and completed, topics covered (breadth and depth of coverage), budget and expenditure ($ and man-days), comparisons against other forms of security control and against other awareness programs (in other fields and/or other organizations). 

With a little more thinking time, it's quite easy (for me, anyway) to come up with a broader selection of awareness metrics also worth considering: 

• More elaborate versions of the above, perhaps combining metrics for more meaningful analysis - for instance using attendance records and feedback to compare the popularity and effectiveness of different types of awareness and training events, different topics, different timings, different presenters, different media etc.;
• Page hit rates, stickiness and various other webserver metrics concerning the popularity of/interest in the information security intranet site, including various elements within it, such as the security policies and specific topic areas;
• Metrics gleaned from personnel records (e.g. proportions of the workforce with basic, intermediate or advanced qualifications, or with skills and competencies relating to information security, privacy, governance, risk etc., and currency of their skills, knowledge, competencies and qualifications);
• Targeted surveys/polls comparing and contrasting awareness levels between various groups (e.g. different business units, departments, teams, levels, specialisms, ages, sexes, cultures/nationalities etc.) or times (e.g. before, during and after specific awareness/training events, awareness focus periods, business periods etc.) or topics (e.g. phishing vs. other forms of social engineering, malware, fraud etc.);
• Workforce security awareness/culture surveys and studies conducted in person by trained and competent survey/research teams (a more expensive method that can generate better quality, richer, more valuable information);
• Maturity metrics using audits, reviews, surveys and self-assessments to determine the maturity and quality of the organization's overall approach to security awareness and training relative to the state of the art in awareness (as documented in various standards, books and websites);
• Benchmarking - comparing information security awareness levels, activities, spending etc. against other fields (such as health and safety or legal compliance) or organizations, industries etc.;
• Risk-based awareness metrics, perhaps assessing the relevance of employee awareness, understanding, knowledge, competence, responsiveness etc. to various information risks, issues or challenges facing the organization, giving a natural priority to the planned awareness and training topics and a basis for budgeting (including resourcing for the security awareness and training program);
• Risk-based information security metrics looking at myriad sources to identify current information risks, trends, predictions, technology directions, emerging threats etc. (useful for strategic planning in information security, of course, with an obvious link through to the corresponding awareness and training needs);
• Change metrics concerning change management and changes affecting the organization, especially those relevant to information risk, security, privacy etc., as well as measuring and driving changes within the awareness program itself;
• Process metrics concerning various information risk, security, privacy, governance and compliance-related processes (again including those concerning awareness and training) and various parameters thereof (e.g. cost and effort, efficiency, effectiveness, consistency, complexity, compliance, creativity, risk ...); 
• Quality metrics concerning the awareness content/materials including policies, procedures and guidelines: there are many possible parameters here e.g. the style of writing and graphics, professionalism, review and authorization status, breadth and depth of coverage, currency/topicality and relevance, readability (e.g. Flesch scores), interest/engagement levels, consistency;
• Awareness surveys conducted by information security presenters, trainers and other professionals: people attending training courses, conferences, workshops and so forth are generally accustomed to completing survey/feedback forms concerning the events e.g. the quality and competence of the presenter/trainer/facilitator, the materials, the venue, the catering etc. and, fair enough, that's quite useful information for the planners of such events. Why not also get the people who present/train/facilitate/lead the events to rate their audiences as well, on parameters such as interest in the topic, engagement, knowledge levels, receptiveness etc.?  Your Information Security Management, Security Admin, Help Desk, PC Support, Risk and Compliance people will have a pretty good idea about awareness and competence levels around the organization. Management, as a whole, knows this stuff too, and so do the auditors ... so ask them!;
• Customer contact metrics for the information security team including the security awareness people, measuring the nature and extent of their interactions with people both within and without the business (e.g. their attendance at professional meetings, conferences, webinars, courses etc.);
• Various awareness metrics gleaned from Help Desk/incident records relating to events and incidents reported (e.g. mean time to report, as well as mean time to resolve, incidents), help requests (number and complexity, perhaps split out by business unit or department), issues known or believed to have been caused by ignorance/carelessness etc., as well as general security metrics concerning incident rates for various types of information security incident - another driver to prioritize the planning and coverage of your awareness activities.

I could continue but even my eyes are glazing over at this point, so instead I want to end with some quick comments about how to make sense of all those and other options, and how you might go about selecting 'a few good security awareness metrics' that might be worth actually using.

Two specific approaches I recommend are PRAGMATIC and GQM.  

GQM starts with some exploration and analysis of your organization's goals or strategic objectives for information risk, security, privacy, governance, compliance and all that jazz (especially how these aspects support or enable core business), leading to some fairly obvious high-level questions (e.g. "Are we sufficiently compliant with our legal obligations towards privacy?") and thence to the kinds of metrics that would generate the data that might address or answer those questions (privacy compliance metrics in that case).   At a lower level of detail, the same approach can be used to determine the goals, questions and kinds of metrics for security awareness.  [Sorry, I'm not going to do that for you - it's your homework for today!]  [For more on GQM, read Lance Hayden's book IT Security Metrics].

PRAGMATIC is a rational basis for choosing between a bunch of possible metrics and assorted variants, or to guide the creative development of new metrics, or to drive improvement by weeding out ineffective metrics and getting more value out of those that remain, using nine key criteria or parameters for metrics: Predictiveness, Relevance, Actionability, Genuineness, Meaninfulness, Accuracy, Timeliness, Integrity/Independence and Cost-effectiveness.  [For more on PRAGMATIC, read our book PRAGMATIC Security Metrics, browse this website or blog, or ask me!]