Student and Clinical Instructor Perceptions Comparing the Ciet and Cpi
Purpose
Recent concerns have emerged with the PT CPI: Version 2006, including a lack of concordance amongst CIs with the interpretation of Entry Level Performance (ELP)1 and the excessive amount of time it takes to complete the assessment. A call for reform was put out in a recent study that warranted further investigation into alternate clinical assessment tools.2 Challenges related to workload demands of clinical faculty, reimbursement limitations, and site competition 3,4,5 deserves a serious look into the efficiency and effectiveness of student clinical assessment. The ability to effectively mentor and educate a DPT student while meeting productivity standards can create an unrealistic workload for the physical therapist. A clinical learning assessment tool that may reduce the time requirement to rate the studentsÕ performance effectively can help bridge the time-intensive task of completing current evaluations.6 The purpose of this platform session is to present findings from our investigation regarding the operationalization of two different clinical performance assessment tools for DPT students and engage the participants in a discussion about the opportunities and risks of implementing different assessment tools.
Methods/Description:
Study design/type: This study was a mixed methods quasi-experimental design. The quantitative portion included data from the participants completion of both the CPI and CIET of students (SPTs) in their terminal clinical experience in Spring 2020.The study used two surveys: one delivered prior to or early in the clinical experience and a second survey delivered following the student performance assessment of the terminal clinical experience for DPT students. The qualitative portion consisted of analysis of comments written in the free text boxes that were embedded in the surveys. Procedures: Following recruitment the investigators educated both the CI and SPT pair how to access and use the CIET using the CIET training video and additional instructions provided by the researchers. Following the training video, both CI and SPT participants completed a post-training assessment on their knowledge of the CIET. Both already had training and experience in completing the CPI. CIÕs and SPTs were paired for the clinical experience using the studentÕs institutional procedures; this process was not altered to conform to this study. Prior to or early in the clinical experience, both the CI and SPT completed the first survey. Then, CIs and SPTs followed the same clinical experience guidelines they would have regardless of study participation. A formal CI/SPT assessment was required at Ômid-termÕ and ÔfinalÕ. At this juncture, the CI completed both the CPI and the CIET, recording their perception of the studentÕs performance. All student participants performed a self-assessment and recorded their ratings on each tool. The student used the same access strategies as the CI. The clinical experience continued as per each institutions policy. At the conclusion of the experience both the CI and the SPT completed both assessment tools and engaged in their final feedback session regarding the entire experience. The second survey was an electronic survey sent to the CI and the student via an email invitation shortly after the conclusion of the clinical experience. Both surveys were developed in Checkbox, a secure, web-based software program and delivered via an email that contains a link to the survey. Inclusion and exclusion criteria: Inclusion criteria for CIs was that they must have had at least 2 DPT students prior to the clinical experience where the data was being collected and have had used the CPI. Additionally, both CI and SPT were required to have access to a computer to electronically record the assessments. Students were in their terminal clinical education experience prior to gradation and had to have used the CPI in previous experiences. Recruitment plan for participants: Participants were recruited using purposeful sampling. CIs who were paired with a student during the designated time frame of a terminal clinical experience each were invited to participate by one of the investigators. Each of the investigators worked in clinical education in a different DPT program in Florida. Five DPT Programs from four public and private universities in the Florida Consortium of Clinical Educators had a representative investigator or significant contributor. Each investigator planned to recruit up to 10 CI/SPT pairs from their institution for a total of 50 CI/SPT pairs (100 study participants). The invitation was delivered via email. Both CIs and SPTs had to agree to participate. If one chose to not participate, another pair was invited. Each CI and SPT were awarded a $50 gift card for their time completing the second assessment tool at each formal assessment.
Results/Outcomes:
Pre-Study Survey results: Due to the onset of COVID-19 virus and cancellation of many clinical experiences, only thirty-two CIs and thirty-one students from three of the five Florida DPT Programs completed the Pre-Study Survey, CPI and CIET midterm student assessment and the Post-Study Survey. Seven CI/student pairs completed that as well as the final CPI and CIET student assessment. Participating CIÕs ages ranged from 28 to 67 years, with a mean age of 36 years. CI gender was 65% female and 35% male. Out of 33 CIs, four (13%) were SCCEs. Participating CIs were employed in multiple inpatient and outpatient practice settings. Their highest PT degree ranged from BS (6%) to DPT (73%). The years of PT experience of participating CIs ranged from two and a half years to 43 years, with an average of 11 years. Experience as a CI ranged from one year to 30 years with an average of eight years. Most of the CIs were APTA Credentialed CIs (79%) and from those three (12%) had completed the level 2 Credentialed CI Program. 18% of CIs had supervised students in a collaborative (1:2 or 1:3) model before, however not as part of this study. CIs had an average of 7 years of experience in completing the CPI for student assessment. CPI experience ranged from no prior experience (two CIs) to 25 years of CPI experience. CIs with CPI experience on average reported spending 91 minutes to complete a CPI student assessment. The reported typical time it took to complete the CPI student assessment ranged from 25 to 240 minutes. Thirty-one students in their terminal full-time clinical experience participated in the Pre-Study Survey. They had completed an average 25 weeks of full-time clinical experiences, ranging from 24 weeks to 27 weeks prior to this final terminal clinical experience. Their average age was 26 years old, with ages ranging from 23 to 31 years. Students reported spending on average 100 minutes on completing the midterm or final CPI self-assessment in their prior clinical experiences. The typical time to complete the CPI ranged from 40 minutes to 240 minutes. Post-Study Survey Results: After completing the CIET and CPI student assessments, CIs and students were asked to complete the Post-Study Survey. A total of 45 participants, 22 CIs and 23 students, completed this survey. Efficiency: Efficiency was evaluated related to how many minutes it took to complete the midterm and final CPI and CIET: Both CIs and students spent less time to complete the CIET compared to the CPI. CI mean time to complete the midterm CPI was 76.14 and mean time to complete the CIET was 34.55 minutes. A Paired t-Test analysis identified that there was a statistically significant difference in efficiency between CPI and CIET midterm (t=6.744, N=22, p=0.00) and final assessment tool completion time (t=3.405, N=22, p=0.003). Student time to complete the midterm mean was 88.91 for the CPI and 34.57 minutes for the CIET. A Paired t-Test analysis identified that there was a statistically significant difference in efficiency between CPI and CIET midterm (t=5.170, N=23, p=0.00) and final assessment tool completion time (t=2.362, N=23, p=0.027). Comprehensiveness assessment: After completing the midterm (and final student assessment for seven CI/student teams) CPI and CIET, CIs and students were asked to rate their agreement with the following statements on a Likert scale from Strongly Disagree (1), Disagree (2), Neutral (3), Agree (4), to Strongly Agree (5). The items CIs scored high for the use of CIET with the most difference compared to CPI were: ¥ÊThe assessment tool was easy to use (Mean CIET of 4.14 versus 1.91 for CPI, t=-6.738, N=22, p=0.00) ¥ÊThe items were asked without redundancy (Mean CIET of 4.14 versus 1.91 for CPI, t= -8.481, N=22, p=0.00) ¥ÊThe training prepared me well for using the assessment tool (Mean CIET 3.91 versus 3.64 for CPI, t=-1.368, N=22, p=.186) ¥ÊThe assessment tool rating scale accurately reflected the studentÕs level of performance (Mean CIET 4.09 versus 3.73 for CPI, t=-1.702, N=22, p=0.104) ¥ÊThe assessment tool provided appropriate categories for me to sufficiently evaluate a student (Mean CIET 4.09 versus 3.77 for CPI, t=-2.084, N=22, p=0.050) There was a statistically significant difference between CPI and CIET for the items Òtool was easy to useÓ and Òitems were asked without redundancyÓ. Items related to comprehensive assessment of the tool were not statistically different, indicating that CIs and studentsÕ perception of CPI and CIET was that both were comprehensive student assessment tools, however the CIET was regarded less time consuming and more efficient. Three items were scored the same by CIs for the use of CPI and CIET. These were related to the comprehensiveness of the assessment tools, indicating that CIs perceive both tools as comprehensive assessment tools. CIs scored only one item lower for CIET than the CPI on the Likert scale: The assessment tool provided a thorough assessment of the student performance (Mean CIET 3.95 versus 4.14 for the CPI). However, the same item was scored statistically significantly higher for the CIET than the CPI by students (Mean CIET 4.35 versus 3.83 for the CPI, t=-3.425, N=22, p=0.002). Students: Students scored CIET higher on the Likert scale with more items showing a difference and with larger differences between CIET and CPI. The items students scored high for the use of CIET with the most difference compared to CPI were: ¥ÊThe items were asked without redundancy (Mean CIET of 4.3 versus 1.78 for CPI, t=-11.642, p=0.00) ¥ÊThe assessment tool was easy to use (Mean CIET of 4.48 versus 2.87 for CPI, t=-8.199, N=23, p=0.00) ¥ÊThe training prepared me well for using the assessment tool (Mean CIET of 4.17 versus 3.3 for CPI, t=-4.114, N=23, p=0.00) ¥ÊThe assessment tool allowed me to self-assess accurately in my practice setting (Mean CIET of 4.17 versus 3.30 for CPI, t=-4.800, N=23, p=0.00) ¥ÊThe assessment tool provided appropriate categories for me to sufficiently self-assess (Mean CIET 4.3 versus 3.39 for CPI, t=-4.041, N=23, p=0.01) ¥ÊThe assessment tool rating scale accurately reflected my level of performance (Mean CIET 4.13 versus 3.39 for CPI, t=-4.101, N=23, p=0.00) ¥ÊThe assessment tool provided an objective measurement of my performance (Mean CIET 4.26 versus 3.57 for CPI, t=-4.058, N=23, p=0.01) There was a statistically significant difference between CPI and CIET for 2 items related to efficiency and 8 items related to training and comprehensive self-assessment. The participants were asked to elaborate on what they considered positive attributes to the CPI and CIET and on challenges experienced (if any) when competing the CPI and CIET. CI themes: Positive attributes to the CPI: ¥ÊThorough categories for assessment of student performance ¥ÊSample behaviors assist with evaluation ¥ÊAbility to elaborate in comment sections ¥ÊRed flag items Positive attributes of the CIET: ¥ÊUser friendly/easy to use ¥ÊThorough/comprehensive without redundancy ¥ÊBrief and to the point ¥ÊLess time consuming Challenges completing the CPI: ¥ÊRedundancy time consuming ¥ÊRating scale Ð confusing, based on % of caseload, not very clear Challenges completing the CIET: ¥ÊRating scale Ð perceived as somewhat negative, challenging descriptors ¥ÊInitial training is confusing and time consuming Student themes: Positive attributes to CPI: ¥ÊIt is very thorough ¥ÊProvides sample behaviors which are very helpful ¥ÊSpace for comments for each category to elaborate ¥ÊMultiple categories for assessment Ð relevant to PT practice ¥ÊAssists with communication between CI and student ¥ÊMidterm/final comparison Positive attributes of CIET: ¥ÊEfficient/concise/shorter ¥ÊComprehensive without being redundant ¥ÊGrading scale is more useful easier to understand ¥ÊMore objective measure of studentÕs abilities ¥ÊLooks at broader areas of PT practice Challenges completing the CPI: ¥ÊRedundant ¥ÊTakes too long to complete ¥ÊInconsistencies in grading noted between CIÕs ¥ÊRating scale is too specific and confusing ¥ÊSome categories are not applicable to all settings Challenges completing the CIET: ¥ÊNo challenges ¥ÊTraining video is confusing/not great ¥ÊTerminology not intuitive, lack of clarity ¥ÊLess options for rating self ¥ÊMore practice needed using the tool/rating scale Finally, CIs and students were asked what tool they preferred: CPI or CIET. 72% of CIs preferred the CIET, 14% preferred the CPI and 14 % did not have a preference. 83% of the students preferred the CIET and 17% had no preference. Limitation: The anticipated sample size of 100 participants was not met because of the COVID pandemic effect on clinical education during the data collection time. Only one program with students who were in their terminal clinical experience early enough was able to have 6 students complete the full clinical experience. Two programs had students cancelled after the midterm, and two programs did not have students in the clinical experience long enough to complete the midterm. This resulted in less participants to complete the midterm assessments and post study survey, reducing sample size to 45. Conclusions/Relevance to the conference theme: In the current clinical environment with increased productivity demands, many clinicians are more hesitant to supervise students due to the perceived increased workload of student instruction and assessment. Academic programs are charged to enhance the team well-being of their clinical partners through innovative physical therapy education, and therefore encouraged to investigate the need for a thorough, but time efficient student assessment tool for clinical education for clinicians and students. This research study supports that students and CIs perceive both CPI and CIET student assessment tools in clinical education allow for thorough and effective assessment of the studentÕs performance in all areas of clinical practice. However, the CIET was perceived as easier to use, less time consuming and less redundant.