Interrater reliability and agreement of the NEUMOBACT checklist about infection-prevention performance of intensive care nurses in simulation-based scenarios

Abstract

Objective To analyse the interrater reliability of the NEUMOBACT checklist and verify whether consistent results are reproducible.

Methods A validation study with a cross-sectional design, compliant with the GRRAS checklist, among ICU nurses attending a SIMULAZERO course with an Objective Structured Clinical Evaluation simulation format, to verify transfer from theory to clinical practice of knowledge and skills in ventilator-associated pneumonia (VAP) and catheter-related bacteraemia (CRB) prevention. A minimum sample size of 111 pairs of nurse raters was calculated. Interrater agreement was analysed using Gwet’s AC1 for each item and as a total for each of the three checklists in the NEUMOBACT instrument.

Results A total of 95 pairs of valid NEUMOBACT checklists were completed by 190 raters with a median age of 29 [25–35] years, 93.7% were female. At the Central Venous Catheter insertion station, Gwet’s AC1 was 0.934 (95% CI [0.919–0.949]). Only 2 of the 17 items scored below 0.9. At the Endotracheal Suctioning station, Gwet’s AC1 was 0.869 (95% CI [0.851–0.886]). Of the 26 items that made up this station, 16 had an agreement percentage above 0.9, a further 9 were between 0.821 and 0.884, and item 13 had a value of 0.789. At the Patient Care station, Gwet’s AC1 was 0.911 (95% CI [0.896–0.927]). Of the 21 items, 17 showed an agreement percentage above 0.9 and 4 were between 0.810 and 0.894.

Conclusions The interrater reliability of the NEUMOBACT checklist shows substantial agreement between pairs of raters and is therefore validated in this large sample of ICU nurses.

Relevance to clinical practice The NEUMOBACT checklist can be useful for assessing skills before and after training in VAP and CRB prevention measures and during debriefing (post-simulation feedback), to reinforce the scientific evidence behind actions and decisions for items that have been performed incorrectly, thus consolidating training already received.