Quality Tools – Run Charts – Part 2: Variation and Trends

This post is part 2 in our series on run charts. In this one, we discuss how variation is present in all processes and the two types of variation we are looking for in run chart interpretation – common cause and special cause variation. The use of run chart rules to differentiate between the two types of variation is introduced with an explanation of the run chart rule for trends. Subsequent posts will cover other run chart rules. (Duration 11 min 27 sec.)

Notes and Resources

  • An excellent summary article on use of run charts and application of run chart rules in healthcare – Perla RJ, Provost LP, Murray SK: The run chart: a simple analytical tool for learning from variation in healthcare processes. British Medical Journal – Quality and Safety. 2011;20:46. e51. doi:10.1136/bmjqs.2009.037895. https://goo.gl/MaEcCL
  • Posted March 2018

Transcript

In the prior vlog on run charts in this series, the virtues of looking at multiple data points over a period of time in the form of a run chart were described. Looking at just one data point is like looking at a snapshot. Looking at a run chart is like watching a movie – a lot more information! We also talked about how often to add new data points to your run chart. Adding points more often let’s you detect a change sooner. So now that we are looking at run charts, we need to interpret them correctly. Of course, you can just look at it, see all the variation, and get a general impression of how things are doing – generally going up, generally going down, or staying about the same. But sometimes it’s not so clear. There’s a BIG pitfall here you should be aware of. If you’re going to make a decision based on the run chart, you should be a little more certain about the interpretation than just casually eyeballing it. A significant change in the behavior of a process isn’t always obvious. So it’s extremely helpful to understand a bit about the nature of process variation to help you know when the process has actually changed, and when it hasn’t!

You see, all processes have variation. The variation may be big, it may be small, but all processes have it. Let’s look at an example. Many STEMI systems of care track the time interval between EMS making contact with a patient and the time that the first 12 lead ECG was acquired if the patient was over 35 years old and presented with non-traumatic chest pain. Those cases are usually pretty clear-cut for needing a 12 lead ECG, so as soon as the patient complains of chest pain without associated trauma, the process of getting the 12 lead should start right away. But that time interval will not always be exactly the same, even when the crews are all consistently following the same policies, processes and protocols.

Here’s a run chart of data for EMS contact time to 1st 12 lead ECG acquisition time intervals on consecutive cases with patients over 35 years old that presented to EMS with non-traumatic chest pain. There were no new initiatives, no new protocols, no new procedures introduced while this data was being collected. So the scattering of results represents the amount of natural variation in this process. The random differences in the time intervals may be from things like the chest pain being mentioned by some patients right way or in other cases, the crew only finds out about the chest pain after they were told an elaborate story about the patient calling 9-1-1 after they spoke with their sister in Nebraska, who’s a retired nurse, with two really cute cats, named Bang and Boom, that her sister found as strays after they were frightened by fireworks this past 4th of July. Or maybe some of the patients did not speak English well. Or maybe some of the patients were slow to agree to have a 12 lead ECG. In any case, there can be all sorts of reasons for the process variations, even when everything in the patient assessment process is going just as it is supposed to.

With an understanding that there will always be variation, we can now appreciate that we need to be able to tell the difference between the variation that commonly takes place in situations like I just described, which is referred to as ‘common cause variation’ and the variation that shows up when some uncommon or special circumstances are taking place, which is referred to as ‘special cause variation’. As an example of that, consider a scenario where a software upgrade with a bug is installed in all of the ECG monitors. That bug is now causing delays in all of the monitors used for capturing the 12 lead ECGs and now the whole 12 lead ECG acquisition process is taking longer for the whole organization. This is the kind of issue that would show up in the run chart data as special cause variation.

Now sometimes when you’re monitoring a run chart, you’ll be wanting reassurance that things are performing up to expectations and staying on track. If you see special cause variation showing up, it constitutes a red flag of sorts that can let you know that some investigation is needed to see what’s going on and why. But sometimes, special cause variation showing up is exactly what you want to happen. If you have recently launched an improvement project, you’re intentionally -trying- to create special cause variation. The change at the core of your improvement project represents a change to the design of the process. The hope, your hypothesis, is that the change you made will result in a change in performance from where it is now, to something better. If you do not see special cause variation start after you made the process change and after you’ve given adequate time for something to happen, you may need to conclude that the change did not make a meaningful impact and it might be time to try something else. That’s important to know – and the run chart can help you with that. Similarly, maybe the change made things worse. Also good to know – and it may lead to a management decision to stop the change and roll it back to where things were before or again go back to the drawing board and come up with another idea to try.

Now that we know what we are looking for in different scenarios and why, we need to know if we are seeing common or special cause variation on our run chart. When you look at the run chart and see the variation, you should be looking for signs in the data that strongly suggest something unusual is taking place. These are also called statistical signals. These signs, or signals, show that something has changed in the process to the point that these signals start showing up in the run chart. They’re revealing special cause variation. When these signals are not showing up in the run chart, it strongly suggests that the only variation we are seeing is from ‘common’ causes. Fortunately, there are tools we can use with a run chart to help us tell the difference between these common and special cause variations. They’re called run chart rules.

So let’s start looking at some of these run chart rules. Each of these rules reveal a statistical signal in the data strongly suggesting that the process has changed. The first run chart rule we’ll cover is for trends. In the context of run chart data interpretation, we say that there is a trend when there are at least five data points in row that are all rising or all falling. For adjacent data points with the same value, skip one point and count the rest. A set of data points of the same value does not make or break a trend. The first data point does not count. Now, this sort of  pattern does not usually occur randomly. Something ‘special’ going on is usually required to cause a true trend to show up on your run chart.

Here is what a rising trend looks like. In this case we have 5 points in a row that are rising, again not counting the first one and counting the two points at the same value by skipping the first one and counting the rest, which is one. It can also go down, as seen here.

Let’s look again at the run chart of the EMS contact to first 12 lead ECG time intervals to see if there are any trends. Nowhere do we see at least five points in a row all going up or down. No trends there. But, here’s where many people mis-interpret the chart. Too often, as the data flows in, people see a few data points in a row climbing up or down and they call it a trend. There isn’t a statistically valid signal here that the process has changed. Calling this a trend is an error in interpretation and can lead to bad decisions.

To see what I mean, let’s rewind the ‘movie’ shown in this run chart. Let’s pretend that they only started collecting data on the EMS contact to first 12 lead time interval at the same time they changed to the new super duper rapid patch ECG electrodes, where I have the arrow. These new electrodes are supposed to make the process of getting a 12 lead in the field go much faster than with conventional electrodes. So, putting these in place is being conducted as a QI project to reduce the EMS contact to first 12 lead time interval. After collecting these first four data points, the medical director sees that the EMS contact to 1st 12 lead time interval is getting longer, not shorter. He is ready to call the QI project off in defeat and insist that the electrodes get pulled from all of the units. But let’s let the run chart data movie continue to play with more data flowing in. We can see that in reality, those first several data points did not meet the definition of a trend. It wasn’t a valid statistical signal suggesting a decline in performance. It was all just normal process variation. Prematurely pulling all of the new electrodes may have not just wasted resources, but might also miss a great improvement opportunity if it turns out that with further analysis over a longer period of time, the new electrodes actually made things go much faster, not slower. After all, time is muscle and we want to do whatever we can to save time, to save more muscle, to improve more lives, to save more lives.

So, correct interpretation of a run chart requires us to be able to differentiate between common and special cause variation. Mis-interpretation can lead to bad decisions. Run charts rules are used to help us make that differentiation between common and special cause variation. We just looked at one of the run chart rules – for trends. In the next video, we’ll look at some more of the run chart rules.