When Is a Change an Improvement?

One of my Black Belt students recently submitted an interim project report. The goal of her project is to reduce the time it takes to complete a certain transaction. The project Y metric is success rate, where a success is defined as completing the transaction in 24 hours or less. The goal after improvement is 99% success. The baseline performance of the process before improvement is 69% success.  The student turned in a report where she stated that improvement had been made because the success rate increased from 69% to 73%.

The fact that the output of a process has changed does not necessarily mean that improvement has been achieved. For a project to be considered successful, the results must be of practical significance and of statistical significance.

From a practical standpoint, the requirement is that 99% of the transactions be completed in 24 hours or less. This level was not achieved.

From a statistical standpoint, we must conduct the appropriate two-sample test to see if the difference is statistically significant. The null hypothesis is no improvement, i.e. the data after improvement comes from the same distribution of data as before improvement. The alternate hypothesis is that there was improvement, i.e. the data come from different distributions.

In her project report, the student included the results of a two-sample percent defective test that showed that there was NO statistically significant improvement. She included the following comment:

“The manager wants the inventory specialist to complete (the transactions) within 24 hours 99% of the time. He does not expect improvement over night but wanted to see if there was a difference or improvement in success rate. The next slide shows a hypothesis test to compare two samples. 1st group is for all the receipts created from January to August 2018. 2nd group is after the solutions in the project were implemented. The graph states there is not a significant difference, but there is a difference and it lets us know we are on the right track to keep solutions in place and enforce the control plan.”

My reply to the student was simply this – there was no improvement achieved. The change from 69% to 73% success was not significant from a practical standpoint – it did not come anywhere near the goal of 99% success. Nor was the change significant from a statistical standpoint – the data could not be proven to come from different distributions. There is no evidence to suggest that the difference between 69% and 73% was due to anything other than normal expected variation in the process. There is no basis to conclude that the project is on the “…. right track……”.

Your comments or questions about this article are welcome, as are suggestions for future articles. Feel free to contact me by email at roger@keyperformance.com.

About the author: Mr. Roger C. Ellis is an industrial engineer by training and profession. He is a Six Sigma Master Black Belt with over 50 years of business experience in a wide range of fields. Mr. Ellis develops and instructs Six Sigma professional certification courses for Key Performance LLC. For a more detailed biography, please refer to www.keyperformance.com.