Guidelines for Developing and Evaluating Plans for Assessing Student Achievement
The Council on Academic Accreditation (CAA) has developed the following guidelines and questions for programs to consider as they continue the development of student achievement assessment plans. This document is in follow-up to the packet of information forwarded to all programs in October 2001, Guidelines for Developing Formative Assessment Plans for Implementation of New Standards for the Certificate of Clinical Competence.
This information is provided to assist CAA-accredited programs in maintaining compliance with accreditation Standards 1.2, 1.6, and 3.1, in relation to the scheduled implementation of the new Standards and Implementation for the Certificate of Clinical Competence in Speech-Language Pathology in 2005 and the new Standards and Implementation for the Certificate of Clinical Competence in Audiology in 2007.
At a minimum, assessment plans must include the six components listed below. Specific questions are posed to facilitate the development of your plan.
1. Examination of Academic and Clinical Curriculum
Determine where/if the academic and/or clinical curriculum includes each identified knowledge and skill in the certification standards.
- Is there a place in the academic and/or clinical curriculum designed to specifically elicit the knowledge or skill?
2. Identification of Learning Goals
Develop behaviorally defined indicators of achievement or performance related to each knowledge and skill identified in the certification standards (Standards III & IV in Speech-Language Pathology, Standard IV in Audiology)
- Define the level of success for student demonstration of each knowledge and skill.
- Are the learning goals linked to your program goals and objectives?
3. Assessment Mechanism
Describe the mechanisms and time schedule used to assess students' progress in reaching defined indicators of achievement for each learning goal.
- What types of assessment will you use? (e.g., exams [written, practical, oral] papers, presentations, demonstrations)
- Why did you choose these instruments?
- What evidence will you look for? (See Evidence of Student Learning.)
- When/how frequently will you assess the student on a knowledge or skill to determine retention?
- Will there be multiple modes of demonstration?
- Who will be involved in the assessment? Are there multiple evaluators for clinical skills? Are conditions consistent across evaluators and settings?
- How does the program integrate the assessments of academic faculty, clinical faculty, and off-campus supervisors
4. Record Keeping
Develop tracking/record keeping systems to document students' progress toward each indicator of achievement for each knowledge and skill.
- What documentation will you maintain on each student? For how long?
- How will you provide feedback to students, faculty, and clinical supervisors?
Provide feedback and remediation. Inform students, faculty, staff, and off-campus supervisors of the indicators of achievement for each knowledge and skill and assessment procedures. Provide frequent feedback to students about their progress.
- How will you provide specific feedback to students, faculty, and supervisors regarding students' level of achievement related to each knowledge and skill?
- What are your plans for remediation if a student does not achieve the knowledge, skill, or outcomes?
The Council For Clinical Certification (CFCC) is developing an example matrix that programs may choose to use in tracking student progress in achieving the specific knowledge and skills identified in the certification standards. These materials will be forwarded to all programs in Summer 2002.
Note: In addition to verifying achievement of the knowledge and skills, programs will still need to maintain for each student a record of the total number of hours in clinical practicum and semester credit hours in the graduate program of study in order for students to be eligible for ASHA certification and other credentials.
5. Validation of Indicators
Specify the rationale for selecting the indicators of achievement regarding each knowledge and skill.
- What criteria did you use? Based on what data?
- To what degree do the assessment mechanisms and indicators of achievement provide evidence that adequately reflects the nature of the underlying knowledge, skill, or ability being assessed?
6. Evaluation and Continuous Program Improvement
Develop plan and mechanisms to evaluate the effectiveness of the program related to student learning outcomes and the ability to prepare students to enter professional practice; describe how the data collected are used to enhance program effectiveness
- What activities have you engaged in to evaluate your program?
- From where/whom have you sought input (e.g., students, alumni, employers, others)?
- What data have you collected?
- Have you identified patterns of strengths and weaknesses in student achievement, your assessment plan, or program effectiveness?
- Is the design of your assessment plan capable of credibly demonstrating that established learning goals are being attained?
- Have you identified corrective measures and/or revised your assessment plan or the academic and clinical curriculum, as needed, to strengthen or improve the program?
Evidence of Student Learning
In developing your program's assessment plan, consider the following principles and characteristics of good evidence of student learning. (Ewell, Peter T. (2001) Accreditation and Student Learning Outcomes: A Proposed Point of Departure, Council for Higher Education Accreditation Occasional Paper, Washington, DC.) These principles address the design of a program's assessment plan, focusing specifically on the extent to which the design is capable of credibly demonstrating that established learning goals are being attained. The essence of these principles and characteristics is reflected in the questions posed above for each component of the assessment plan.
Principles of Evidence
- Comprehensiveness - the degree to which the assessment system is capable of providing evidence about the full range of student learning goals established by the program
- Multiple Judgements - the extent to which multiple sources of evidence are used in a mutually reinforcing way to examine outcomes
- Multiple Dimensions - the degree to which different facets of student performance with respect to established learning goals can be investigated so that patterns of strength and weakness can be identified and addressed
- Directness - the extent to which the approach relies upon direct measures of student attainment instead of self-reports about learning or other "proxy" indicators of attainment such as graduation rates or job placement
Characteristics of Good Evidence
The following characteristics of evidence are intended to provide general guidance and should be applied as appropriate. Because there are outcomes that are not necessarily related to student learning, programs should consider, in developing their student achievement assessment plans, the differences between student learning, other outcomes or outputs of the program, and quality assurance processes. Evidence of student learning should be:
- Relevant - the extent to which the evidence is capable of representing the underlying learning goal, with a clear rationale for why it is related.
- Verifiable - the process of assembling evidence that is documentable and replicable; sufficient information is available to enable a reviewer to independently corroborate what was found.
- Representative - the extent to which the evidence is typical of an underlying situation or condition, not an isolated case, particularly when data are provided as trends over time.
- Cumulative - the use of multiple sources, methods, and approaches that provide independent corroboration for issues of importance to the program; triangulation of information from several data points.
- Actionable - focusing on evidence that is reflectively analyzed and interpreted so that it will reveal specific implications for the program and provide the program with guidance for action and improvement; disaggregate the evidence to reveal underlying patterns of strength and weakness, or to uncover specific opportunities for intervention and improvement.
Accreditation Standards Related To Program Objectives, Assessment of Student Achievement, Program Improvement, and Review of Curriculum Regarding Certification Standards
The program's mission, goals, and objectives are consistent with ASHA-recognized national standards for entry into professional practice and with the mission of the institution.
The application should include the mission statements of the institution and college as well as of the program. The program faculty and professional staff should regularly evaluate the consistency of program and institutional goals and objectives and the extent to which they are achieved.
The program conducts ongoing and systematic assessment of academic and clinical education and performance of its students and graduates. Students have ongoing opportunity to assess their academic and clinical education program. Results of the assessments are used to plan and implement program improvements that promote high-quality educational experiences for students.
The program should detail the procedures followed in evaluating the quality, currency, and effectiveness of its graduate program, the academic and clinical preparation of its students, the professional performance of its graduates, and the process by which it engages in systematic self-analysis. The plan should indicate the mechanisms used to evaluate each component and the schedule on which the evaluations are conducted. Results of such evaluations should be reported, as well as specific modifications to the program that result from the evaluations. Student performance on the Praxis series examinations in speech-language pathology and/or audiology is an example of an expected outcome measure.
The curriculum (academic and clinical education) is consistent with the mission and goals of the program and is sufficient to permit students to meet ASHA-recognized national standards for entry into professional practice.
The program should describe the curriculum leading to a master's or doctoral degree with a major emphasis in speech-language pathology and/or audiology. The program must offer appropriate courses and clinical experiences on a regular basis so that students enrolled in the program may satisfy the requirement for a graduate degree(s) in speech-language pathology and/or audiology for entry into professional practice.
The intent of this standard is to ensure that program graduates are able to meet ASHA certification standards. The program may demonstrate how ASHA's recognized standards are met by describing outcome evaluations of students' knowledge and skills needed for entry into professional practice.