The agreement of scores obtained from different groups of assessors in the 360-degree assessment of the professional competency of pediatric residents

Abstract

Background: The 360-degree assessment of the professional competency of pediatric residents at the Department of Pediatrics, Faculty of Medicine, Siriraj Hospital, is in accordance with the guidelines established by the Royal College of Pediatricians of Thailand. This evaluation involves diverse groups of assessors who collaborate with pediatric residents within patient care teams. The results derived from this comprehensive evaluation play a pivotal role in determining the progression and completion of the training program. The findings of the study will clarify the agreement of scores among the mentioned assessor groups, thus contributing to fair and prudent decision-making processes.
Objective: To evaluate inter-rater reliability of 360-degree assessment of pediatric resident’s professional competency
Methods: Data was collected by gathering the competency assessment forms of first-year pediatric residents. These forms cover six domains of competency, with each domain being rated on a nine-point scale. The 360-degree assessment results were obtained from medical professionals who collaborated with the residents within in-patient units over a one-month period. In order to assess inter-rater reliability, the rubric scores were analyzed using the intra-class correlation coefficient (ICC). Additionally, Pearson’s correlation coefficient (r) was utilized to examine the relationship between the rating scores given by different assessors.
Results: The data on resident physicians in this study consists of 53 individuals (46 females, 7 males). Analyzing the results of the 360-degree assessment of professional competency for first-year pediatric residents across six domains revealed consistently negative Intraclass Correlation Coefficient (ICC) values. This indicates a lack of agreement among assessors within each group regarding their assessments across all domains. Furthermore, upon analyzing the correlation between assessment scores in each domain, it was consistently observed that the correlation coefficient (r) remained below 0.2. This highlights a distinct absence of a meaningful correlation in the same direction among assessors, which includes attending physicians, senior residents, nurses, and medical students.
Conclusion: The 360-degree assessment format used to evaluate the professional competency of first-year pediatric residents shows inconsistent assessment scores and a lack of correlation among the scores provided by the different group of assessors.