Rater, Test-Retest Reliability, Internal Consistency and Response Stability of Evidence-Based Practice Assessments in PT Students
Purpose/Hypothesis: Evidence-based practice (EBP) refers to integrating best available research with clinical expertise and patients’ preferences during patient-provider encounters. EBP knowledge and performance assessments typically developed for clinicians, need to be psychometrically examined in Physical Therapy (PT) students for generalizability. Therefore, this study investigated the rater reliability, test-retest reliability, internal consistency and response stability of EBP knowledge, skills, attitudes and confidence assessments in entry-level PT students.
Number of Subjects: Twenty
Materials and Methods: Twenty entry-level PT students (6F/14M) within their first year of education were included. Students completed online surveys to measure their knowledge of EBP (modified Fresno test), attitude towards EBP (EBP attitude and utilization survey), and self-perceived confidence in utilizing EBP (EBP Confidence scale) on two separate occasions a week apart. The rater reliability, test-retest reliability, internal consistency and response stability of EBP assessments were examined using Intraclass Correlation Coefficient (ICC), Cronbach’s alpha, Standard Error of Measurement (SEM) and Minimal Detectable Difference at 95% confidence level (MDD95).
Results: Moderate to excellent intra-rater reliability (ICC=0.52 - 0.83), and inter-rater reliability (ICC=0.58 - 0.91) for modified Fresno test were determined across 3 raters. Moderate to excellent test-retest reliability (ICC=0.52 - 0.83) for EBP attitude and confidence surveys, moderate internal consistency for EBP attitude survey and modified Fresno test (Cronbach alpha = 0.58 and 0.65), while excellent internal consistency for EBP confidence scale (alpha = 0.96) were found. The SEM and MDD95 values were calculated.
Conclusions: The EBP assessments demonstrated moderate-to-excellent rater, test-retest and internal consistency reliability. The SEM and MDD95 values describe the limits for change that can be considered above the threshold of measurement error suggesting response stability. These findings support the utility of EBP assessments in PT students, however, further evaluation of these instruments is still warranted.
Clinical Relevance: Due to the significance of EBP and its application in clinical practice, the PT academic programs should consider these evaluative tools to assess and demonstrate changes in EBP knowledge, skills, attitudes and confidence, when designing EBP educational activities.