September 21, 2004 Features

Evidence-Based Practice in Schools

Integrating Craft and Theory with Science and Data

Decision making is hard work, especially if you happen to be a school-based speech-language pathologist.

The SLP's application of evidence-based practice (EBP) to making assessment and intervention decisions in schools requires the deliberate integration of clinical expertise, substantive theoretical knowledge, child and family preferences, and best available evidence from systematic empirical research. By this definition, it should be evident that EBP does not refer to a singular focus on using research to make decisions about children. Rather, EBP emphasizes the systematic and deliberate integration of science and craft, or, alternatively, data and theory.

Although both craft and theory, achieved through training, practice, and experience, are viable forms of knowledge to use in clinical decision-making, most consumers expect (and often assume) that clinicians' choices when implementing services are derived from the realms of science. Like many other allied disciplines that are transforming themselves into evidence-based professions, such as nursing, clinical psychology, clinical medicine, and special education, school-based speech-language pathology is gearing itself to shift from a craft-based and "intuition-driven" practice to one in which consideration of the preponderance of evidence from scientific investigations is systematically integrated with clinical expertise and contextual factors.

Rationale for EBP in Schools

Consider the following purely hypothetical example. A clinician has just attended a workshop given by a colleague in a neighboring school district on a new procedure claimed to have great potential for children with severe oral and written language disorders. Specifically, it involves the application of warm oil to the child's scalp twice daily for a period of two months.

The clinician is a bit incredulous, but the presenter of this workshop has some interesting data on the procedure. Ten children who fell more than two standard deviations below the mean on multiple language tests received the treatment for the 2-month treatment period. On average, the children gained 3.5 points on a new set of standardized tests known to be correlated to those in the original battery. And all of this change occurred in only two months!

Our clinician finds several reasons to criticize this treatment, the presenter, and colleagues who might be tempted rush home to try it. Particularly, our clinician notes that the evidence presented is weak for a number of reasons. First, the study is plagued by threats to internal validity, or to the certainty of the findings. For example, the pretest-posttest research design included no control group to study maturation or other possible non-treatment influences; the testers were the same as the clinicians who delivered the scalp oil treatment and may have introduced bias of many sorts into the children's scores; children's gains could reflect a statistical artifact rather than real change (i.e., regression to the mean); and only standardized assessments were used to demonstrate outcome rather than measures of communicative performance in authentic contexts.

Second, the evidence presented is also weak because of its lack of independence and peer review. A hallmark of science is convergence of data across independent researchers and laboratories as well as the importance of objective review of evidence by peer scientists. Neither of these criteria is met for the scalp oil treatment.

Third, the evidence presented is weak in its theoretical bases. Our clinician is dubious because there is no reasonable theory to even suggest such a treatment for children with language disorders, much less to presume that it could be effective. Our clinician considers her training and her experience with numerous children with language disorders and can come up with no conceptual or theoretical reason why the scalp oil treatment might work.

The point of this silly exercise should be evident: Our clinician's decision not to use this approach is a clear application of EBP. Despite the presenter's testimonial and presentation of data, our clinician concluded that the only research supporting the approach was extremely weak in its design and quality. The lack of independent findings supporting effectiveness of the approach across laboratories, the lack of peer review, and the lack of a theoretical basis all work to make the treatment ethically unsupportable.

This exercise illustrates the twofold outcomes of EBP. First, the use of EBP allows professionals to discard treatments that are clinically unacceptable even when their colleagues and clients endorse use of those treatments. Second, the use of EBP allows professionals to select treatments that are clinically acceptable. Nonetheless, concerning the second outcome, we need to ask individually as clinicians and researchers and collectively as a profession: What would happen if we applied the same criteria used by our clinician in this example to the approaches we actually use in schools on a daily basis?

Hopefully, we could at least be able to defend our approaches on the basis of some theoretical grounds. But, what's the level and quality of evidence supporting our approaches? Would other approaches be more efficient or effective? EBP requires clinicians to seek the answers to these and similar questions, and it requires researchers to do more to help them to find the answers.  As shown in the illustration on p. 30, the transition of school-based speech-language pathology to an evidence-based profession thus requires a dual focus on ensuring the use of proven practices in the field, and building the knowledge base of proven practices.

Contextual Influences

The pressure for EBP in school-based speech-language pathology and many other disciplines embodies the 21st century's twofold emphasis on reform based on accountability, and scaling up of research. Together, these concepts contextualize discussions of EBP as it applies to delivering speech-language services in schools. 

Reform based on accountability

Readers are likely well aware of the current climate of accountability, particularly in our nation's schools. This climate emphasizes personal and institutional responsibility for actions and decisions concerning our nation's schoolchildren, and stems in part from the spiraling costs of special education services. The No Child Left Behind Act of 2001 exemplifies these new directions in public education.

Accountability emphasizes the need for school-based professionals to deliver instruction and interventions that have demonstrated efficiency (the time taken to reach a desired outcome) and effectiveness (the likelihood that the desired outcome will be achieved). The proliferation of commercial products and training workshops can make it very challenging for consumers to differentiate between viable and unviable practices, or "good and bad information" (Stanovich & Stanovich, 2003, p. 5). By contrast, the nature of peer-reviewed scientific literature makes it the best (and often only) resource for objective information on what works and doesn't work.

The emphasis on EBP in school-based speech-language pathology does not place a singular focus on the use of evidence and research when making clinical decisions; rather, it emphasizes the need to consult the best available research to ensure clinical objectivity and currency. In this way, practitioners show greater accountability for their decisions.

Scaling up

Two hallmarks of science are the accumulation of evidence and the preponderance of evidence. The former concept refers to a discipline's building of an evidentiary and knowledge base using many different methodological tools and research designs to resolve important questions. The latter concept refers to a discipline's established evidentiary and knowledge base that is derived through scrutiny of the outcomes of accumulated research. The preponderance of evidence provides a discipline with new knowledge, and in school-based speech-language pathology, new knowledge provides the means to improve clinical services and solve pressing problems. Federal agencies, such as the National Institutes of Health and the U.S. Department of Education, increasingly emphasize the need for disciplines to rigorously accumulate evidence that is programmatically focused on solving pressing problems-particularly concerning the achievement of at-risk children and youth-and to translate this accumulated evidence into improved clinical and educational practices.

Together, these concepts are referred to as scaling up, a term used often to describe efforts to eradicate the research-to-practice gap, or the length of time between the conduct of research and its application to practices in the field. Discussions of EBP often over-emphasize the clinician's role as consumer of science relative to the research community's role as producer of research. Nonetheless, application of the scaling-up concept to evidence-based practices in school-based speech-language pathology requires two cooperative processes. First comes the scaling up of research, or endorsing researchers' conduct and publication of research that targets real clinical problems and, ultimately, is implemented in non-laboratory, clinical settings. Second comes the scaling up of practices, or removing barriers that affect the translation of research findings to clinical practices in schools. EBP requires a synergy between the research community, which is charged with accumulating evidence, and the clinical community, which is charged with examining the preponderance of evidence, to make decisions about the best ways to evaluate and treat children.

Three Essential Organization Needs

Putting EBP in Schools

EBP is much more than an effort to get clinicians to study research articles. Rather, EBP is a clinical decision-making paradigm that mirrors contemporary perspectives concerning the importance of accountability and the need to differentiate between bad and good information. EBP also mirrors our discipline's vitality, or our ability to achieve a knowledge base that addresses pressing real-life problems, and to translate this knowledge for the betterment of our nation's children and youth.

To achieve the highest standards in EBP, however, the roles and responsibilities of both clinicians and researchers require adjustment. Clinicians must seek the best designed, highest quality evidence available to inform their clinical decisions and must become skilled in integrating this evidence with their theoretical and craft-based knowledge. Possibly more important, the research community must increase the quantity and quality of research that is relevant to such decisions, and it must work diligently to ensure that reports of research findings are presented in forms that are readily accessible to and assimilable by busy practitioners with little time to separate the wheat from the chaff.

It is our opinion that our discipline must undergo dramatic change to achieve the level of EBP that seems both necessary and sufficient for maximizing the SLP's role in supporting the needs of our nation's schoolchildren. Systemic changes are needed first at the organizational level, encompassing the discipline and its professional associations, and secondly, at the individual level. Although discussion of EBP often emphasizes the level of the individual-or the clinician's responsibility to understand scientific reasoning, to study the scientific literature, and to integrate scientific findings into everyday practices-here we broaden this discussion to emphasize three essential needs at the level of the organization as we make the transition to an evidence-based profession.

Comprehensive meta-analysis

Although several meta-analyses have been conducted to examine the body of evidence supporting various approaches to speech-language treatment (e.g., Law et al., 1998, 2004; Nye, Foster, & Seaman, 1987), school-based speech-language pathology requires access to a comprehensive cardinal document that provides a description of the available evidence concerning assessment procedures, intervention approaches, and treatment components used in schools. Such a document is critical to scaling up school-based speech-language pathology, to providing contemporary analyses of how well knowledge is being accumulated concerning specific practices and, importantly, to describing what types of practices are supported through a preponderance of evidence. As an example of such a cardinal document, this one in the area of literacy, the U.S. government convened the National Reading Panel (NRP, 2000) to summarize the available evidence on interventions and instruction supporting reading in five areas: vocabulary, fluency, phonemic awareness, comprehension, and phonics. This panel of experts attempted to vet the good from bad research (an act that remains highly controversial) and then to document the support available for different approaches in these five areas of reading.

The cardinal document resulting from the Panel's work, available at no cost to the public (http://www.nationalreadingpanel.org/), is an important resource for reading teachers, elementary educators, special educators, SLPs, and others to use when selecting treatments to support reading in schoolchildren. Currently, school-based SLPs and researchers who study school-based practices do not have a cardinal document of this sort focused on language and speech specifically, but their efforts to implement EBP would be facilitated greatly if they had access to this information.

Evidentiary guidelines and systematic reviews

Guidelines must be developed for judging the preponderance of evidence concerning a particular diagnostic or treatment option or approach. How are clinicians who turn to the research for guidance on clinical decisions to know how much and what type and quality of evidence is adequate for supporting their decision?

To illustrate how an organization might approach this need, the American Psychological Association (APA) organized a task force to develop guidelines for determining when a particular childhood intervention had an acceptable level of evidence constituting "probable efficacy" and/or "demonstrated effectiveness" (Lonigan, Elber, & Johnson, 1998). For instance, the designation of "demonstrated effectiveness" is provided to those interventions for which the accumulated evidence includes two or more well-conducted group-design studies conducted by different research teams showing the treatment to be better than an alternative treatment or equivalent to an already established treatment. School-based SLPs who turn to the accumulated evidence on a particular treatment currently have no guidance on determining the adequacy of the evidence in quality and quantity.

A plan for developing and disseminating systematic reviews is also greatly needed by school-based SLPs. Systematic reviews or practice guidelines make recommendations on use of a procedure or intervention based on the quality of evidence available in the research literature, in conjunction with an evaluation of the burden of suffering from the condition and the characteristics of the treatment, such as its intensity and invasiveness (United States Preventive Services Task Force, 1989).  Although many suggestions have been presented concerning how to improve clinicians' access to research literature and understanding of statistics and research design principles, the reality is that these suggestions are not realistic nor do they reflect how other disciplines are going about the transition to EBP. Rather, other disciplines, such as the medical fields, use media outlets geared specifically to providing widely accessible and easily digestible practice guidelines based on the newest literature to busy practitioners.

The Cochrane Collaboration (www.cochrane.org) and the Campbell Collaboration (www.campbellcollaboration.org) provide two exemplary examples of organizations focused specifically on turning bodies of research into usable guidelines for public policy and clinical practice. The focus of these sister organizations, with Cochrane focused on health care interventions and Campbell focused on social, behavioral, and educational interventions, is systematic review of the cumulative evidence on specific interventions. This includes identifying topics for systematic review to meet the needs of policy and practice communities, providing guidelines on conduct of systematic review, and disseminating the findings from these reviews in usable formats. A newer initiative, the What Works Clearinghouse of the U.S. Department of Education's Institute of Education Sciences (IES) (www.w-w-c.org), follows a similar design to provide systematic reports on educational interventions, including intervention reports. The U. S. Department of Health and Human Services' Agency for Healthcare Research and Quality (http://www.ahrq.gov/) is an additional useful resource for evidence-based practice guidelines and regularly updated summaries of clinical-outcomes research related to healthcare. This organization's U.S. Preventive Services Task Force (USPSTF) oversees systematic reviews of effectiveness evidence and publishes guidelines which may be useful to both speech-language pathologists and audiologists working with children and youth. For instance, the USPSTF recently summarized and evaluated the evidence concerning the effectiveness of newborn hearing screening, available at http://www.ahrq.gov/clinic/uspstf/uspsnbhr.htm.

The systematic reviews from the Cochrane and Campbell collaborations and the IES that are so useful to other professionals, including psychologists, social workers, nurses, physicians, teachers, and the like, can provide useful models for developing a collaboration focused specifically on producing reviews for school-based SLPs. Within the communication disorders disciplines, the evidence-based practice guidelines generated by the Academy of Neurologic Communication Disorders and Sciences (2001) provide a useful model for the development of systematic reviews for school-based speech-language pathology.

Research infrastructure

It may seem superfluous to emphasize here the need to promote the numbers of PhD-level researchers in speech-language pathology, given that this has been a resounding and recurrent theme in our profession for the last decade. Nonetheless, a critical value in the EBP equation is the role of the researcher in accumulating knowledge. It cannot be understated here how great the need is for persons with research training who can comfortably and competently engage in asking questions that explicitly relate to school-based services, and who can objectively and proficiently evaluate their peers' research during the peer-review process.

Research conducted in real-life schools with SLPs, teachers, and students has different challenges than research conducted in laboratories that is focused on either examining basic processes or establishing the causal relationships between real or potential treatment variables and speech and language outcomes. When research is taken to scale, researchers require a set of tools and techniques that allow them to study processes and outcomes in real-life environments that may be different than those tools of the trade used in laboratories. As we consider the research infrastructure as it relates to accumulating knowledge for school-based speech-language pathology, it is readily apparent that we have too few persons who are able to conduct research along the continuum that moves from the laboratory into real-life settings. As our discipline continues to explore ways to increase the number of doctoral-level trainees, we must focus specific efforts on ensuring that adequate numbers of these trainees focus on applied research that is directly relevant to the issues and challenges faced by clinicians within our schools.

In this article, we have provided an overview of EBP that is potentially useful for shaping discussions about improving speech-language services in schools and in scaling up the school-based discipline within the broader speech-language pathology organization. Some early steps toward each of these objectives have been taken by the Evidence-Based Practice in Child Language Disorders Working Group. Materials on this group and a report on their early efforts can be found at the Web site for the Bamford-Lahey Children's Foundation, www.bamford-lahey.org/ebp.html. We see this as the beginning of what will be a long discussion focused on improving speech-language services for our nation's children and youth.

Laura M. Justice, is assistant professor in the University of Virginia's Curry School of Education, where she teaches in the McGuffey Reading Center and directs the Preschool Language and Literacy Lab. Contact her at ljustic@virginia.edu.

Marc E. Fey, is professor in the Department of Hearing and Speech at the University of Kansas Medical Center, which is part of the KU Intercampus Program in Communicative Disorders. Contact him at mfey@kumc.edu.

cite as: Justice, L. M.  & Fey, M. E. (2004, September 21). Evidence-Based Practice in Schools : Integrating Craft and Theory with Science and Data. The ASHA Leader.

Examples of Outcomes From Systematic Review

U. S. Preventive Services Task Force (1989)

Grade Recommendation Criterion
A Good evidence for inclusion There is good peer-review evidence supporting consideration of use for the intervention
B Fair evidence for inclusion There is fair peer-review evidence supporting consideration of use for the intervention
C Insufficient evidence There is insufficient peer-review evidence for inclusion supporting consideration of use for the intervention, although recommendations for use are possible on other grounds
D Fair evidence for exclusion There is fair peer-review evidence supporting that the intervention should be excluded from consideration of use
E Good evidence for exclusion There is good peer-review evidence supporting that the intervention should be excluded from consideration


Examples of Evidentiary Guidelines for Evaluating Preponderance of Evidence

Quality  Designation Criteria 
  U. S. Preventive Services Task Force (1989) 
Level I Evidence from one well-conducted randomized clinical trial
Level II-1 Evidence from one well-conducted study with controls but without randomization
Level II-2 Evidence from one well-designed cohort or case-control study preferably from independent researchers
Level II-3 Evidence from multiple time-series single-subject investigations or dramatic results from non-controlled experiments
Level III Opinions of authorities, descriptive studies, case studies, reports of expert committees
  Section 1 Task Force of the Division of Clinical Psychology of the American Psychological Association Task Force (1998)
Demonstrated Effectiveness Two or more well-conducted group-design studies conducted by different research teams showing the treatment to be better than an alternative treatment or equivalent to an already established treatment or a large series of well-conducted single-subject design studies with n > 9 that compare the intervention to another treatment
Probable Efficacy Two or more well-conducted group- and/or single-subject design studies (n < 3) showing the treatment superior to a no-treatment, baseline, or alternative treatment condition or two well-conducted group-design studies meeting criteria for demonstrated effectiveness but both studies conducted by the same research team

 



References

Academy of Neurologic Communication Disorders and Sciences. (2001).  Evidence-based practice guidelines for the management of communication disorders in neurologically-impaired individuals: Project introduction. [Electronic version]. Retrieved June 5, 2004, from http://www.ancds.org/.

Coalition for Evidence-Based Policy. (2002). Bringing evidence-driven progress to education: A recommended strategy for the U. S. Department of Education. Washington, DC: Author.

Law, J., Boyle, J., Harris, F., Harkness, A., & Nye, C. (1998). Executive Summary: Screening for speech and language delay: A systematic review of the literature. Health Technology Assessment, 2(9). Retrieved June 5, 2004, from http://www.ncchta.org/execsumm/summ209.shtml.

Law, J., Garrett, Z., & Nye, C. (2004). Speech and language therapy interventions for children with primary speech and language delay or disorder. The Cochrane Library, Issue 2.

Lonigan, C., Elber, J., & Johnson, S. (1998). Empirically supported interventions for children: An overview. Journal of Clinical Child Psychology, 27, 138−145.

National Reading Panel. (2000). Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. Washington, DC: U.S. National Institute for Literacy.

Nye, C., Foster, S. H., & Seaman, D. (1987). Effectiveness of language intervention with the language/learning disabled. Journal of Speech and Hearing Research, 52, 342−357.

Rosswurm, M. A., & Larrabee, J. H. (1999). A model for change to evidence-based practice. The Journal of Nursing Scholarship, 31(4), 317−322.

Sackett, D. L., Rosenberg, W. M., Gray, J. A., Haynes, R. B., & Richardson, W. S. (1996). Evidence based medicine: What it is and what it isn’t. British Medical Journal, 312, 71−72.

Stanovich, P. J., & Stanovich, K. E. (2003). Using research and reason in education: How teachers can use scientifically based research to make curricular instructional decisions. Washington, DC: The Partnership for Reading.

United States Preventive Services Task Force. (1989). Appendix A Task Force Ratings [Electronic version]. Retrieved March 3, 2004, from http://www.ncbi.nlm.nih.gov/books/bv.fcgi?rid=hstat.

Vaughn, S., & Dammann, J. E. (2001). Science and sanity in special education. Behavioral Disorders, 27, 21−29.



  

Advertise With UsAdvertisement