menu SIGN IN
Stacey Barr
Stacey Barr
Performance Measure Specialist

Stacey Barr Pty Ltd
Websites:

3 Essential Signals to Look for in Your KPIs

Posted over 5 years ago

You won’t find true signals of changes in performance by looking at month-to-month comparisons, or trendlines, or moving averages. The signals you really need to know about, the only signals that you ought to respond to, are revealed through one particular graph only…

The typical analysis methods we use for our performance measures are based on assumptions that don’t make much sense.

Month-to-month comparisons assume that there is no routine variation over time and wrongly interpret any difference as a signal.

Trendlines assume that all change is linear and gradual, and that if Excel can calculate a trend line then there must be a trend.

Moving averages assume that seasonal patterns exist, and also that change is smooth and gradual.

The only analysis technique I have ever come across that clearly filters the noise and highlights the signals in our performance measures is the XmR chart.

http://www.staceybarr.com/images/xmrchartcentralline.jpg

The XmR chart filters the noisy routine variation in our measures by showing us how much of this routine variation there is, by way of the Natural Process Limits. And coupled with the Central Line, these Natural Process Limits give us a meaningful baseline to quickly assess when performance has changed; when there is something else going on that’s not part of the routine variation.

There are three very specific signals to look for.

SIGNAL 1: Outlier or special cause

When a measure value falls outside the Natural Process Limits, it means that more than just the routine variation is at play. It’s a signal that something else has happened.

http://www.staceybarr.com/images/xmrchartonepoint.jpg

If Employee Attendance plummeted below the bottom Natural Process Limit, a likely cause could be a flu epidemic or local natural disaster that kept many more people away from work in that period.

Even though this is a signal that something out of the norm has happened, because it’s just a one-time event, we don’t react to it. We find out what caused it, but we don’t run around madly trying to fix it. That would be a massive waste of time and money because we’d essentially by trying to fix something that won’t happen again, or that is completely outside our control.

SIGNAL 2: Long run

To be convinced that a change in the level of performance has happened, we need to see seven (yes, seven) points in a row on the same side of the Central Line. The probability that a pattern like that is part of routine variation is close to zero (0.78%, to be precise). Seven points, not three or five or one.

http://www.staceybarr.com/images/xmrchartsevenpoints.jpg

If a measure of Invoice Accuracy showed a long run above the Central Line, it might be evidence that an initiative to simplify the pricing strategy had successfully reduced the errors in invoices.

When we see a long run signal in our measure, we certainly need to find the cause for it. Sometimes it will be a signal of improvement, and we want to confirm what caused the improvement. Other times it will be a signal that performance has deteriorated and the cause of that is very important to identify!

SIGNAL 3: Short run

You’re no doubt thinking to yourself ‘I can’t wait for seven months before I can know if I should take action!’ You can either measure more frequently to pick up signals sooner (as long as it makes sense to), or plan for bigger signals.

A bigger signal appears as a short run, of three out of four consecutive measure values closer to a Natural Process Limit than they are to the Central Line. The probability of this pattern happening also has a very close to zero probability.

http://www.staceybarr.com/images/xmrchartfourpoints.jpg

A short run above the Central Line for On-time Deliveries for a trucking company would likely be due to an initiative that had a substantially large impact. It could be something like doubling the fleet size. But it also could be a new competitor in the market that poached a large percentage of their customers.

Again, with a signal like the short run, it’s really important to find the cause before responding.

XmR charts take only a little effort to create, but their usefulness is so powerful they are absolutely worth trying.

TAKE ACTION:

If you’re one of the first 5 readers to submit 20 to 30 values of your performance measure in time series, I will put your data into an XmR chart and report back on the Measure Up blog on signals the measure might contain. Yes, I’ll keep it anonymous unless you say it’s okay to identify your organisation!

Comments (3)

Helen Lindsay
Helen Lindsay
Bowen Craggs

I found your post very interesting. I’m analysing online surveys on corporate websites. We ask Did you achieve the goal of your visit? (Yes, Partly, No). I have an example with data from January 2012 (about 400 responses) per month. Would this be suitable as an example? If so, how do I send you an Excel file.

Posted over 5 years ago | permalink
Stacey Barr
Stacey Barr
Performance Measure Specialist

Helen, you can email to info@staceybarr.com and please place in the subject line “KPI Library XmR Chart Example” so we can respond to you quickly.

Smiles, Stacey.

Posted over 5 years ago | permalink
Jonathan Strong
Jonathan Strong
Strong Edge LLC

Having spent time in financial services, market data analytics, business intelligence and also audio signal processing, this is a familiar topic indeed — just in a different setting. Certainly critical to filter out the noise so management decisions can, in fact, be driven by signal. I know I’m responding a couple of years after your post — but the concept is every bit as crucial today as it was back then. Thanks! – Jon

Posted about 3 years ago | permalink

Log in to post comments.