|
Schuele and Justice introduce two misinterpretations of statistical significance in their article on effect sizes (Aug. 15, 2006). One is in a quoted statement that statistical significance means that "the probability of the observed difference arising by chance was sufficiently small." Statistical significance does not assess the probability that chance is the reason for the observed difference. Under the null hypothesis, assumed true a priori, any observed difference is a chance occurrence with a probability of 1.0. Inferential statistical analysis indicates how rare an observed difference would be if the null hypothesis were true. If, for example, a researcher finds that the observed difference would exist 5% or fewer times in a (null) distribution, the researcher may judge that to be rare enough to reject the null hypothesis. This does not mean, however, that the observed difference arose by chance.
The second misinterpretation is in the statement that "Statistical analysis indicates whether a non-zero difference...is likely to be found again and again if the study is repeated." Statistical significance is not replication, and cannot be used in place of replication. Attanasio (Inferential Statistics and Treatment Efficacy Studies in Communication Disorders, Journal of Speech and Hearing Research, Vol. 37, 755-759, August 1994) indicates that the test of statistical significance does not assess the probability that a mean difference can be replicated. Replication of the experiment invalidates the inference model. To know if the same results can be obtained again, actual replication must be done.
Joseph S. Attanasio
Montclair, NJ
attanasioj@mail.montclair.edu
Authors' Response
We thank Joseph Attanasio for bringing to light certain clarifications relevant to Schuele and Justice (Aug. 15, 2006). We encourage readers to attend carefully to the two points he raised. Additionally, readers are encouraged to pursue additional clarification on statistical significance and effect sizes in Attanasio (1994, JSHR), Meline and Schmidt (1997, AJSLP), and Turner and Bernard (2006, CICSD), as well as the references provided in Schuele and Justice. A list of these references can be found below.
C. Melanie Schuele
Nashville TN
melanie.schuele@vanderbilt.edu
Laura M. Justice
Charlottesville, VA
ljustice@virginia.edu
References
Attanasio, J. S. (1994) Inferential statistics and treatment efficacy studies in communication disorders. Journal of Speech, Language, and Hearing Research, 37, 755-759.
Meline, T., & Schmitt, J. F. (1997). Case studies for evaluating statistical significance in group designs. American Journal of Speech-Language Pathology, 6, 33-41.
Turner, H. M., & Bernard, R. M. (2006, Spring). Calculating and synthesizing effect sizes. Contemporary Issues in Communication Sciences and Disorders, 33, 42-55.
|