March 6, 2007 Features

The State of the Evidence

ASHA Develops Levels of Evidence for Communication Sciences and Disorders

For the first time, clinicians and researchers in speech-language pathology and audiology will have a common framework for characterizing the scientific evidence underlying clinical practice.

This framework will offer a look at the state of the scientific evidence in different areas of communication sciences and disorders—and demonstrate areas in which more research is needed—in an era when practicing clinicians, researchers, government agencies, and third-party payers are evaluating the evidence to improve the quality and effectiveness of service delivery.

"SLPs regularly make decisions about treatment plans while under constraints of government, education, and financial agencies. Information from this LOE system, which is a trusted source for evidence, will aid clinicians making those decisions," said Janet Patterson, a member content expert who appraised research studies using ASHA's LOE system in an evidence-based systematic review. "The entire discipline will be able to speak with one voice about the extent of scientific evidence underpinning clinical practice."

To describe the evidence in communication sciences and disorders, ASHA's Advisory Committee on Evidence-Based Practice (ACEBP) and National Center for Evidence-Based Practice in Communication Disorders (N-CEP) developed a system of levels of evidence (LOE) specific to communication sciences and disorders. LOE is an approach to evaluating evidence based on its quality, using criteria such as protection from bias and confounding, effect size, and precision (visit ASHA's evidence-based practice section for a glossary of terminology).

LOE is useful as a way to communicate and compare the relative strength or weakness of scientific support for a specific intervention or diagnostic technique. Speech-language pathologists and audiologists often look to levels of evidence as a factor (along with clinical expertise and client values) in making clinical decisions. The LOE system also builds member awareness of the principles involved in evaluating research and provides practicing clinicians with a means for evaluating relevant research when making clinical decisions.

Externally, Medicare and third-party payers are increasingly demanding information about the level of evidence to make reimbursement decisions. Insurance companies have characterized the evidence by using various LOE systems and have developed reviews of evidence to justify reimbursement decisions. In addition, government agencies seek input on research needs in setting the agenda for the direction of research and establishing funding priorities. Levels of evidence also can be used to identify gaps in the research for prioritization of future research needs.

In Search of a System

Identification of a system of LOE for use throughout ASHA was part of the initial charge to ACEBP when it was established in 2005. The original intent was to identify the existing LOE that best fit the particular needs of speech-language pathology and audiology. But no best fit could be found because of two factors that complicated the search.

One issue was study quality. ACEBP and N-CEP felt very strongly that poorly done and well-done studies that used similar study designs should not be given equal weight. While this caveat seems obvious, many LOE systems were eliminated because they did not take study quality into consideration. Still others were eliminated because they did not define high and low quality. While these systems would give more weight, for example, to a high-quality cohort study than to a low-quality cohort study, no objective criteria were provided to determine high versus low quality.

The second complicating factor was study design. Most LOE systems were developed by and/or for physicians and were based on a medical model. As a result, very few of these systems include single-subject design studies as even a potential source of scientific evidence. This design, however, constitutes a very important part of the research in communication sciences and disorders, and often offers the best ways to address some of the research questions of interest to ASHA members. Because of their focus on an individual subject and in-depth examination of behavior, this type of experimental design may be suited to developing and testing specific treatments. This design is especially valuable in relatively low-prevalence areas such as augmentative and alternative communication or fluency.

Only a small number of LOE systems took sufficient account of study quality, and even fewer included single-subject designs. As no single system met both criteria, ACEBP and N-CEP undertook development of an LOE system specific to speech-language pathology and audiology in 2005.

ASHA's Levels of Evidence

ASHA's LOE system is a four-step process that involves assessing the extent to which a study meets quality indicators, deciding the stage of research to which a study applies, "scoring" the quality of the study in the context of the stage of research, and finally, synthesizing this information for all studies to give an overall picture of the level of evidence available on a particular clinical issue.

Step 1: Appraise the quality of individual studies related to the topic under review. Eight factors are considered in the appraisal process (see Figure 1).

Step 2: Characterize the research stage into which a particular study falls. In communication sciences and disorders, a single-subject design study could, in some circumstances, constitute the best evidence on a particular question. The ASHA LOE is unique in that strong consideration is given to the appropriateness of a study design in the particular context of the clinical question, rather than placing it on a rigid hierarchy of study designs regardless of context. Drawing heavily from Randall Robey's 2004 article on stages of clinical research, a set of research stages was adopted to provide that context. See Figure 2 [PDF] for the ASHA LOE four stages of research.

Step 3: Assess the quality of a study in the context of the research stage to which it applies. For example, the exploratory stage of research includes studies that lead to an eventual decision on the appropriateness of conducting a very expensive and time-consuming randomized controlled trial to test a hypothesis. It would not be appropriate to assess these studies on a criterion such as "intention to treat," which is a criterion specifically related to randomized The stage of research, once determined, in turn determines the highest possible level of quality for each Step 1 quality criterion for the study designs that fall into that research stage. These levels constitute the standard against which the individual studies are judged. Studies that meet the highest relevant level within each of the seven quality criteria are rated as highest quality. There are seven, rather than eight, quality criteria because "intention to treat" is used only when the study design is a controlled trial, and study design is used as a factor only when the study is not a controlled trial. Studies scoring a five or six are rated as good quality; studies scoring below five are rated as poor quality.

Step 4: Synthesize this information into a single table as a snapshot to illustrate the state of the evidence. Combining the information as in the hypothetical table (Figure 3) allows the body of evidence in a given area to be described in terms of research stage and quality.

The hypothetical table shows that a number of high-quality efficacy studies have been conducted. One could conclude that research priorities should include an emphasis on well-designed effectiveness research, as there might be little to be gained from additional exploratory or efficacy research related to the same question.

Refining the System

The LOE system in its current form has been piloted in an evidence-based systematic review on the frequency and intensity of treatment of stroke patients with aphasia. This topic was selected because of the many questions about the most effective way to deliver aphasia treatment, and the lack of a high-quality synthesis of the evidence related to frequency and intensity.

ASHA began the review by identifying thousands of abstracts for aphasia research. The ASHA N-CEP staff reviewed the abstracts, narrowing them down to fewer than two dozen pertinent articles, and then obtained the full-text journal articles. These articles were then reviewed using the LOE system by member volunteers Janet Patterson, Stacie Raymer, and Leora Cherney of Division 2, Neurophysiology and Neurogenic Speech and Language Disorders.

As the system is applied to a greater number of topics and used by more individuals and groups, refinements will be made to make it as relevant and useful as possible. ACEBP and N-CEP hope that this tool can and will be improved with additional input from ASHA members. All members are invited and encouraged to visit the Web discussion forum on evidence-based practice; a thread has been started on which members may voice opinions and/or post questions about the system.

Rob Mullen, is director of ASHA's National Center for Evidence-Based Practice in Communication Disorders. Contact him by e-mail at rmullen@asha.org.

cite as: Mullen, R. (2007, March 06). The State of the Evidence : ASHA Develops Levels of Evidence for Communication Sciences and Disorders. The ASHA Leader.

Figure 3: The State of the Evidence on a Fictitious Topic

The hypothetical LOE table summarizes the quality of evidence and the types of research questions that have been addressed. In a systematic review, a table such as this will include every individual study contained in that review. This particular example indicates that a number of high-quality efficacy studies have been conducted. One might then conclude that research priorities should include an emphasis on well-designed effectiveness research, as there might be little to be gained from additional exploratory or efficacy research related to the same question.

 

 

Exploratory

Efficacy

Effectiveness

Policy

High Quality

 

Washington 1994

Eisenhower 1998

Grant 1999

Truman 1999

 

 

Good Quality

 

Lincoln 1992

 

Jefferson 1997

Nixon 2000

Kennedy 1999

 

Lower Quality

 

Roosevelt 1994

 

Adams 2001



  

Advertise With UsAdvertisement