One of my Black Belt students recently submitted his final project report. He used the two proportions test in Minitab to compare the proportion of on time performance for completion of employee appraisals, before and after improvement. I sent him the following feedback regarding this test.
“Hi. Your null and alternate hypotheses are correctly stated. The null is no difference in on time rates before and after improvement; the alternate is that there is a difference.
You failed to give the basis for your conclusion that there was a statistically significant improvement. Your conclusion must be reached by comparing the alpha decision value for your desired level of confidence with the calculated p value. If the calculated p value is lower than alpha, reject the null hypothesis.
In this case, the calculated p value is .058. For 95% confidence (which is what you used), the alpha decision level is .05. This case is right on the borderline between significant and not significant. The confidence interval calculated in Minitab does include zero, so I am OK with your decision to reject the null hypothesis, but please recognize that this is a borderline case.
At a confidence level of 90%, i.e. alpha of .10, the calculated p value is lower than alpha and we reject the null hypothesis. At a 99% level of confidence, i.e. alpha of .01, the calculated p value is higher than alpha and we fail to reject the null hypothesis.
Mark Twain often used this quote, which he attributed to British Prime Minister Benjamin Disraeli. “There are three types of lies: lies, damned lies and statistics.”
If I wanted to take advantage of a group of people who did not understand hypothesis testing very well, I could slant this case either way. At a 90% confidence level, I could be emphatic that improvement was achieved. At a 99% confidence level, I could be quite definitive that improvement was not achieved. And I could offer proof of both with statistics!”
Your comments or questions about this article are welcome, as are suggestions for future articles. Feel free to contact me by email at firstname.lastname@example.org.
About the author: Mr. Roger C. Ellis is an industrial engineer by training and profession. He is a Six Sigma Master Black Belt with over 48 years of business experience in a wide range of fields. Mr. Ellis develops and instructs Six Sigma professional certification courses for Key Performance LLC. For a more detailed biography, please refer to www.keyperformance.com.