Whether your data is quantitative (based on numbers) or qualitative (based on perceptions), it’s integrity depends on 5 widely recognised qualities.
Make sure the data you have selected is directly appropriate to the purpose of the KPI you selected it for. Be careful of data that seems interesting: it doesn’t mean it is relevant. Trying to gather more data than you really need, especially in surveys, can negatively impact on the other dimensions of data integrity (below).
—> Be ruthless and collect only the data you have a use for in monitoring and diagnosing performance.
Collect enough data and collect it carefully to ensure that it is precise enough (especially if it is an estimate based on a sample) and continues to be precise enough as you collect it over time. Would you rely on one day’s rainfall to draw conclusions about annual rainfall? What about five days’ rainfall? How many days rainfall would you need to get a precise enough estimate of annual rainfall? And what would this depend on?
—> Design your sample sizes to give the reliability you need. Don’t guess.
It is important that the data you collect are observable events or characteristics that describe the full scope of what your Key Performance Indicator is supposed to be measuring. This means that it is unbiased, or accurate enough. The last thing you need is for your data to tell you only what the “squeaky wheels” have to say, drowning out the valid and important and balancing views of the “well oiled wheels”. Squeaky wheels, volunteer surveys and easiest-ones-to-measure are examples of data sources unlikely to give you accurate enough data.
—> Define your population carefully, and select random samples to avoid bias.
Unless the data you collect is clearly defined, legibly presented, easy to organise for analysis, makes sense to its users and can be easily interpreted and understood by them, it won’t matter how relevant, representative or reliable it is. It just won’t be usable. The numbers need to be in a format you can use.
—> Design your data collection forms and questionnaires carefully to give you the data in the format your analysis needs.
Trade off the degree to which your data is relevant, representative, reliable and readable with the level of resources you will need to invest to make it so. Make sure the value you get from using your data is greater than the effort you invested in getting it. Beware of the temptation to invest in sophisticated automatic data capture systems (such as bar-coding and voice recognition software) – if you haven’t got a simple manual system working well first, then these systems are likely to cost you much, much more than the savings they appear to promise.
—> Pilot test your data collection processes to be sure they will deliver cost-effective data.
If you have a performance metrics or KPI that triggers more debate about data quality than it does about performance levels, then use the 5 Rs of data integrity to work out where the data collection process can be improved.
ABOUT THE AUTHOR
Stacey Barr is the Performance Measure Specialist, helping strategic planners, business analysts and performance measurement officers confidently facilitate their organisation to create and use meaningful performance measures with lots of buy-in. Sign up for Stacey’s free email tips at www.staceybarr.com/202tipsKPI.html and receive a complimentary copy of her renowned e-book “202 Tips for Performance Measurement”.