Assessment to support program quality and development: An ACAPT Initiative to Foster Educational Excellence through Benchmarking
Purpose
The purpose of this session is to illustrate the VALUE of integrating a validated national tool, The Physical Therapist Educational Program Quality Survey(PTEPQS) into comprehensive DPT program assessment plans. Speakers will share results of their research and validation efforts in tool design and offer recommendations on how to navigate data from multiple stakeholders for internal and external benchmarking to meet accreditation standards while focusing on aspirations of individual and collective DPT program excellence. Use of data triangulation to support evidence based education decisions and program future planning will be discussed. The Engagement Theory of Program Quality used as a framework for describing and sustaining DPT program excellence will be explored.
Methods and/or Description of Project
A primary goal for ACAPT was to develop mechanisms to assist academic physical therapist programs in achieving excellence. The 2011 Education Leadership Conference (ELC) focused on identifying the key elements of educational excellence. The ACAPT Benchmarking Excellence (BenEx) Task Force used the data and feedback to 1) operationalize the definition of excellence; 2) develop a tool to measure the construct; and 3) recommend a benchmarking process for programs to work toward achieving excellence.
The BenEx Task Force members conducted a literature search focusing on excellence in higher education. This search and subsequent deliberations led to the adoption of the Engagement Theory of Program Quality as a framework for defining excellence as a first step in determining potential benchmarks. The BenEx Task Force matched the theoretical framework to the components of excellence in physical therapy education leading to the development of the initial Physical Therapist Educational Program Quality survey (PTEPQS). Over the course of the past several years, the survey was pilot tested, reevaluated, and made available to membership. Most recently, in the fall of 2016, a portal was developed to allow programmatic access to the data.
Results/Outcomes
Instrument testing and validation
Survey measurement characteristics met expected, acceptable levels of internal consistency. The analyses revealed some areas that lacked consistency in responses. For example, the element related to culminating/capstone experiences showed poor internal consistency. In spite of the fact that elements did not fit into a unifying attribute, they may provide useful information for program evaluation
Program Assessment
Surveys are administered to faculty and students in the graduating class for that year. The data is compiled and made available through an interactive web-based portal. Program administrators are able to compare student and faculty responses across clustered categories and subcategories created by internal consistency analysis. In addition, programs are able to compare outcomes within these categories to other programs with similar characteristics. The interactive portal allows programs to select a comparison group by self-selecting distinguishing characteristics such as class size or Carnegie Classification.
Benchmarking
Through a benchmarking process, physical therapist educational programs may match their own performance on identified criteria for excellence against all or, a relevant sample of, other programs, and use that information to determine areas of strength and needed improvement. The benchmarking process includes self-assessment from diverse perspectives, allowing for triangulation. Data gathered from the PTEPQS will allow programs to self-assess performance and identify strengths. Initial benchmarks for excellence based on past survey findings will be presented and discussed.
Conclusions/Relevance to the conference theme: Through the Looking Glass: Transforming Physical Therapy Education
This session explicitly addresses the 2017 conference theme of using evidence and benchmarks in curricula planning as the presenters intend to share PTEPQS data for interpretation of comprehensive program assessment both internally by assessing faculty and student stakeholders as well as externally by assessing overall program performance against institutions of similar characteristics. The findings will be linked to curricular planning and outcomes, programmatic performance interpretation, and accreditation standards. Presenters will share pragmatic use of the data for curricular and programmatic assessment. Participants will have an opportunity to interact with the ACAPT data portal from their own institution or using sample data to allow attendees hands on appreciation of data gathering and interpretation use within their own personal settings.
References
1. Busche G. The Appreciative Inquiry Model. In: Kessler EH, ed. Encyclopedia of Management Theory. Thousand Oaks, CA: Sage Publications; 2013.
2. Hobson R, Rolland S, Rotgans J, et al. Quality assurance, benchmarking, assessment and mutual international recognition of qualifications. Eur J Dent Educ 2008;12(s1):92-100.
3. Higgs J, McMeeken J. Achieving Quality in Physiotherapy Programmes through Benchmarking. NZ J Physiother. 1997;25(3):19-22.
4. Haworth JG, Conrad C. Emblems of quality in higher education: developing and sustaining high-quality programs. Needham, MA: Allyn and Bacon; 1997.
5. Warden S. Testing the Engagement Theory of Program Quality in CACREP-Accredited Counselor Education Programs. Counselor Education and Supervision. 2012;51(2):127-140.
6. Wang Hm. What matters in graduate school? Exploring patterns of student engagement, academic and personal development. Association for Institutional Research; 2003; Tampa, FL.
7. Newswander LK, Borrego M. Engagement in two interdisciplinary graduate programs. Higher Education. 2009;58(4):551-562.
8. Grignon TP, Henley E, Lee KM, Abentroth MJ, Jette DU. Expected graduate outcomes in US physical therapist education programs: a qualitative study. J Phys Ther Educ. 2014;28(1):48.
9. Standards and Required Elements for Accreditation of Physical Therapist Education Programs, (2016).
Course Objectives
Describe the value of leading change through program assessment
Explore the Engagement Theory of Program Quality to guide best practices that really matter to our students.
Describe what PTEPQS can and can’t do for your program
Review national data being used to establish benchmarks.
Learn efficient ways to interface with the ACAPT web-based portal to interpret data that can guide program assessment and aspirations for achieving excellence in DPT programs.
Instructional Methods
Presentation
Group Discussion
Internet activities
Panel Discussion
Question and Answers
Tentative Outline/Schedule
Excellence requires leadership and leadership requires vision and data driven decision making (20 min)
What if? The ACAPT journey of defining and measuring program excellence (20 min)
Benchmarking, what it can and can’t do for your program (15 min)
Practice makes perfect: Interfacing with the ACAPT Portal (30 min)
Panel Discussion-Sharing experiences of PTEQS, its VALUE and ideas to overcome potential barriers (15 min)
Reactions, Ideas and Questions (10 min)