Interesting metrics in the Novopay report

Here in New Zealand, a major government project to introduce Novopay, a new payroll system for schools, has made the news headlines repeatedly over many months, culminating in a media storm, the resignation of the accountable minister and independent investigations into the fiasco.  

In my experience, the issues reported are far from unique to Novopay.  I recognize the characteristics of many large software implementation or business change projects I have audited in the past, such as poor project governance, technical complexity arising from complex and probably inadequately specified user requirements, extensive customization of the core IT system, the introduction of new business and IT processes, and rapidly deteriorating commercial relations between the customer and supplier as the implementation went badly, to say the least.  I'll leave the question of why these kinds of issue crop up over and over again for another rainy day.  For now, suffice to say that the project has so far narrowly avoided becoming a train wreck.

On the up-side, the government has opened up about the issues, since the minister resigned at least, publishing quite detailed reports for instance and stating a number of recommendations to sort the mess out.  The independent investigations took just three months, no mean feat in such a complicated and politically-charged situation.  The new minister in charge, Steven Joyce, acknowledged the severity of the situation on prime-time TV just the other evening, and responded positively to robust questioning by a well-briefed journalist.  An impressive, confident performance.

Anyway, I digress.  The technical report is a good example of how metrics can be used to substantiate as well as illustrate high-level management reports.  

Plucking out a single example, page 31 of Deloitte's Novopay Technical Review report says "As at 07 March 2013, 500 defects have been open for an average of 116 days (and this is relatively consistent across different severity levels)." This sentence includes the number of defects and the average number of days that defects take to close, two key metrics.  The final parenthetical clause refers to additional data concerning the breakdown of defects by severity: one might expect that relatively serious defects would be fixed more quickly than trivia, but in practice serious defects tend to be inherently difficult to fix, especially as it appears as if the team has learnt its lesson and is now paying more attention to testing fixes before they are implemented.  

Recommendation 2 of the Novopay Technical Review report concerns metrics. "Establish robust performance management processes and checkpoints to monitor remediation progress.  To support achievement of the objectives, clear monitoring and performance measures need to be established and managed through a strong performance management process.  This should include milestones for key activities and definition of stability from an operational metrics perspective (e.g. the number of no-pays, percentage of errors)." [page 41].  FWIW I recommend that the education ministry and Talent2, the contractors concerned, are PRAGMATIC about it.  I'm waiting by the phone ...


PS  The separate ministerial inquiry report also makes fascinating reading.  There have clearly been serious issues in several areas, so it is no surprise that  a number of senior people burnt out (I wonder if "burn out rate" qualifies as a useful metric on these huge politically-charged projects!).  

One little paragraph that really caught my eye said "Work commenced on the service requirements in October 2008.  This process was lengthy, and was never actually completed.  Even after Go Live, new requirements were being discovered."  That's a classic example of "Ready, Fire, Aim" if ever I heard one.