Mycoses. 2024;67:e13694.   | 1 of 5 https://doi.org/10.1111/myc.13694 wileyonlinelibrary.com/journal/myc 1  |  BACKGROUND Onychomycosis (ONM) is a fungal infection affecting the nail unit and the surrounding tissue. It represents the most prevalent nail- related pathology, with a prevalence ranging from 4% to 13%. The responsible pathogens can include dermatophyte fungi, yeasts or non-dermatophyte moulds.1–5 The most characteristic clinical signs include nail thicken- ing, subungual hyperkeratosis, nail plate discolouration and onycholysis.1–4,6,7 Traditionally, ONM is typically classified based on these clin- ical signs through visual assessment. However, it is worth noting that there are numerical classifications available, though not all of them are validated. Examples include the classifications proposed Received: 5 November 2023  | Accepted: 26 December 2023 DOI: 10.1111/myc.13694 O R I G I N A L A R T I C L E Inter-observer reliability of the Onychomycosis Severity Index depending on clinical experience: A review of 50 cases David Navarro-Pérez1  | Sara García-Oreja2  | Aroa Tardáguila-García1  | Diego León-Herce2  | Francisco Javier Álvaro-Afonso1 | José Luis Lázaro-Martínez1 This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made. © 2024 The Authors. Mycoses published by Wiley-VCH GmbH. 1Diabetic Foot Unit, Clínica Universitaria de Podología, Facultad de Enfermería, Fisioterapia y Podología, Universidad Complutense de Madrid, Instituto de Investigación Sanitaria del Hospital Clínico San Carlos (IDISSC), Madrid, Spain 2Clínica Universitaria de Podología, Facultad de Enfermería, Fisioterapia y Podología, Universidad Complutense de Madrid, Instituto de Investigación Sanitaria del Hospital Clínico San Carlos (IDISSC), Madrid, Spain Correspondence Sara García-Oreja, Clínica Universitaria de Podología, Facultad de Enfermería, Fisioterapia y Podología, Universidad Complutense de Madrid. Instituto de Investigación Sanitaria del Hospital Clínico San Carlos (IDISSC), Pabellón 1, Avda, Complutense s/n, Madrid 28040, Spain. Email: sagarc14@ucm.es Abstract Background: Onychomycosis (ONM) is the most prevalent nail unit pathology, and its severity and monitoring are often based on the visual judgement of clinicians. Objective: The objective of this study is to assess the reliability of the Onychomycosis Severity Index (OSI) classification when utilized by three clinicians with varying lev- els of clinical experience: an experienced podiatrist (with 5 years of experience), a moderately experienced podiatrist (with 2 years of experience) and an inexperienced podiatrist (a recent graduate familiar with the OSI classification but lacking clinical experience). Additionally, we compared the severity assessments made through visual inspection with those determined using the OSI by different clinicians. Methods: We evaluated reliability using the intraclass correlation index (ICC), analys- ing 50 images of ONM. Results: The OSI demonstrated a very high level of reliability (ICC: 0.889) across cli- nicians, irrespective of their experience levels. Conversely, a statistically significant increase in severity was observed when comparing visual assessments with the OSI (p < .001) for ONM severity evaluation. Conclusion: The OSI proves to be a reproducible classification system, regardless of the clinical experience of the practitioner employing it. K E Y W O R D S classification, nail pathology, onychomycosis, severity index https://doi.org/10.1111/myc.13694 www.wileyonlinelibrary.com/journal/myc https://orcid.org/0000-0003-3588-0557 mailto: https://orcid.org/0000-0002-2937-7122 https://orcid.org/0000-0001-5650-7351 https://orcid.org/0009-0009-7973-9652 https://orcid.org/0000-0001-6110-0265 http://creativecommons.org/licenses/by-nc-nd/4.0/ mailto:sagarc14@ucm.es http://crossmark.crossref.org/dialog/?doi=10.1111%2Fmyc.13694&domain=pdf&date_stamp=2024-01-04 2 of 5  |     NAVARRO-­PÉREZ et al. by Baran et al.8 and Sergeev et al.9 for evaluating the severity and treatment response of ONM. The Onychomycosis Severity Index (OSI), introduced and vali- dated by Carney et al.3 stands as the only validated score classifica- tion system available for effectively assessing the severity of ONM. This index serves as a valuable tool for guiding treatment decisions and predicting therapeutic outcomes.3 The OSI classification is based on specific clinical characteristics, in- cluding the affected area, the proximity of the lesion, and the presence of dermatophytoma or hyperkeratosis larger than 2 mm, which is used to categorize ONM as mild, moderate or severe (Table 1; Figure 1). The presence of a dermatophytoma is characterized by the accu- mulation of fungal hyphae, manifesting as a yellow, white or orange longitudinal stripe or a white or yellow rounded patch near the nail fold (Figure 2). It is worth noting that antifungal treatments may have limited effectiveness against dermatophytomas.1,3 Subungual hyperkeratosis is considered a prognostic factor asso- ciated with a poorer prognosis, as an increase of more than 2 mm in hyperkeratosis can reduce the penetration of antifungal drugs. Given the subjective nature of visually assessing these clinical signs, our objective was to examine the concordance in utilizing the OSI classification, a validated scale, among three podiatrists with varying levels of clinical experience: an experienced podiatrist (with 5 years of experience), a moderately experienced podiatrist (with 2 years of experience) and an inexperienced podiatrist (a re- cent graduate familiar with the OSI classification but lacking clinical experience). 2  |  MATERIAL S AND METHODS 2.1  |  Participants and samples This study was conducted in accordance with the Declaration of Helsinki and the current national legislation governing research in- volving the use of patients.10 A cross-sectional study was conducted involving 50 images of ONM from patients attending a specialized clinic dedicated to ONM management. Previously, a primary clinical investigator reviewed the medi- cal records of patients diagnosed with ONM between January and December 2022. He was tasked with gathering and blinding the photographs for evaluation. Furthermore, the clinical investigator was responsible for documenting the results in the data collection notebook and subsequently conducting statistical analysis using the collected data. The inclusion criteria encompassed individuals aged 18 years and older who had received a diagnosis of ONM through a microbiolog- ical culture combined with PCR, as detailed in the study conducted by Navarro-Pérez et al.11 Exclusion criteria applied to patients for whom there was no available image of ONM or in cases where the image was taken after the nail plate had been debrided. The assessment was carried out using the OSI scale3 by three dif- ferent research podiatrists with varying levels of clinical experience: an experienced podiatrist (with 5 years of experience), a moderately experienced podiatrist (with 2 years of experience) and an inexperi- enced podiatrist (a recent graduate familiar with the OSI classifica- tion but lacking clinical experience). All images were captured as part of routine clinical practice, and they were taken before any nail plate debridement procedures were conducted as part of the ONM treatment in the clinic. Before applying the OSI scale, the investigators independently assessed each photograph and visually classified the ONM based on their criteria, categorizing the infections as mild, moderate or severe. Subsequently, they compared these classifications with the results obtained using the OSI scale. 2.2  |  Statistical analysis ONM has a prevalence ranging from 4% to 13% and to determine the required sample size for this study, we utilized the Granmo Sample Size Calculator, version 7.12, developed by the Institut Municipal d'Investigació Médica in Barcelona, Spain. We aimed to establish a confidence level of 95% with a precision of ±10 percentage units, resulting in a sample size of 50 individuals. Additionally, a 10% re- placement rate was accounted for. Statistical analysis was conducted using SPSS for MacOS, ver- sion 25.0, developed by SPSS, Inc. in Chicago, IL. The analysis was based on the intraclass correlation index (ICC) with a 95% confi- dence level. Concordance was considered high at 0.6 and very high at 0.81. The chi-square test was employed to assess the association between visual classification and classification using the OSI scale. Qualitative variables were presented based on their frequency and percentage, while quantitative variables were described using their mean and standard deviation. TA B L E 1 The Onychomycosis Severity Index (OSI) scale3 is calculated by multiplying the value of ‘Area of Involvement’ by the value of ‘Proximity of infection to the matrix’. To this result, 10 points would be added in case of hyperkeratosis or dermatophytoma. Figure 1 shows an example of how to calculate the severity of the infected nail using the OSI scale. OSI Area of involvement (%) Proximity of infection to the matrix Presence of dermatophytoma or hyperkeratosis >2 mm 1–10 1 <1/4 1 No 0 11–25 2 1/4–1/2 2 Yes 10 26–50 3 >1–2 to 3/4 3 51–75 4 >3/4 4 76–100 5 Matrix involvement 5 14390507, 2024, 1, D ow nloaded from https://onlinelibrary.w iley.com /doi/10.1111/m yc.13694 by Spanish C ochrane N ational Provision (M inisterio de Sanidad), W iley O nline L ibrary on [22/04/2024]. See the T erm s and C onditions (https://onlinelibrary.w iley.com /term s-and-conditions) on W iley O nline L ibrary for rules of use; O A articles are governed by the applicable C reative C om m ons L icense     |  3 of 5NAVARRO-­PÉREZ et al. 3  |  RESULTS High concordance was achieved using the OSI scale when aggregat- ing the results of the three clinical researchers (ICC = 0.889). When comparing two by two, a higher concordance was observed be- tween the most experienced and moderately experienced clinician (ICC = 0.873), whereas the concordance between the moderately experienced and inexperienced clinician was lower (ICC = 0.781), al- though it still demonstrated a strong level of concordance. The mean OSI scores varied among the different clinicians, with an average of 23.23 ± 10.72 for the most experienced clini- cian, 18.24 ± 11.09 for the moderately experienced clinician and 17.4 ± 10.87 for the inexperienced clinician. When comparing the results between the visual scale and the OSI scale, statistically significant differences were observed. In the case of the expert clinician, the p-value was .01, and for the interme- diate expert and inexpert clinicians, it was less than 0.001. The OSI scale resulted in a higher percentage of severe ONM cases compared to the visual assessment. Specifically, the most experienced clinician increased from 38% to 80% in severe ONM cases, the moderately experienced clinician from 42% to 68% and the inexperienced clini- cian from 28% to 52% (Figures 3 and 4). 4  |  DISCUSSION In the current study, we observed a significant level of concordance (ICC = 0.889) on the OSI scale among various podiatrists, even those with varying levels of clinical experience. It is recommended that these assessments be conducted by professionals who are either ex- perts or possess moderate expertise. While a reasonably high level of concordance was also observed between the expert and non- expert assessors, it was slightly lower (ICC = 0.781). Sergeev et al.9 introduced the Clinical Index of Onychomycosis Score (SCIO) as a composite score to evaluate the severity of ONM. This score takes into account various factors, including the clinical form, extent of nail involvement, thickness of subungual hyperkera- tosis, the location of the condition and the patient's age. In contrast, Baran et al.8 introduced Baran-Hay's Severity Index in 2008, which assesses ONM severity based on 10 different parameters. The OSI scale, introduced by Carney et  al.3 in 2011 and later consolidated by Gupta et al.12 in their 2018 update on the diagnosis and treatment of ONM, offers distinct advantages over prior clas- sification systems. Unlike previous methods, this scale evaluates only three key parameters, minimizing subjectivity by focusing on the presence or absence of dermatophytoma and hyperkeratosis ex- ceeding 2 mm. Additionally, its formula is easy to calculate and can be done promptly. Various studies, including those conducted by Watjer et  al.4 Elewski et  al.7 and Gupta et  al.12 have conducted pre- and post- treatment assessments using the OSI scale. However, none of these studies specify the number of professionals who administered the OSI scale. It is crucial to determine whether the results may vary based on the assessor's level of expertise. On the other hand, we observed statistically significant dispar- ities when categorizing severity through visual classification com- pared to utilizing the OSI scale. This discrepancy may be due from the omission of the affected nail quarter in some cases, a variable that can amplify the OSI scale's final value by up to fivefold. To the best of our knowledge, prior studies have not explored this aspect. Consequently, it implies that an ONM case with a more favour- able visual appearance but greater proximal involvement could be F I G U R E 1 Example of assessment using the OSI scale. Area of involvement, 51%–75% (4 points); proximity of infection to the matrix, >1/2 to 3/4 (3 points); dermatophytoma, no (0 points) and hyperkeratosis, no (0 points). Calculation: 4 × 3 + 0 = 12 points. Moderate onychomycosis. OSI, Onychomycosis Severity Index. F I G U R E 2 Example of dermatophytoma. 14390507, 2024, 1, D ow nloaded from https://onlinelibrary.w iley.com /doi/10.1111/m yc.13694 by Spanish C ochrane N ational Provision (M inisterio de Sanidad), W iley O nline L ibrary on [22/04/2024]. See the T erm s and C onditions (https://onlinelibrary.w iley.com /term s-and-conditions) on W iley O nline L ibrary for rules of use; O A articles are governed by the applicable C reative C om m ons L icense 4 of 5  |     NAVARRO-­PÉREZ et al. classified as more severe than a lesion with a poorer visual presenta- tion but more distal involvement. In 2022, Dubljanin et al.6 conducted a study involving 417 pa- tients, where they assessed the SCIO scale. They opted against using the OSI scale due to concerns about potential inter-observer vari- ability. Nonetheless, in our study, we have achieved a sufficiently high level of concordance, making it a viable option for our purposes. The primary limitation of this study stemmed from its single- centre design, where in only podiatrists affiliated with the same re- search group participated. Conversely, a key strength of our study lies in its novelty, as it is, to the best of our knowledge, the first to assess OSI by distinguishing levels of clinical experience among podiatrists to gauge concordance. It is worth noting that the 2011 study on OSI was conducted exclusively among dermatologists. Based on the findings of this study, the OSI appears to be a re- liable scale, demonstrating concordance among various clinicians with varying levels of clinical experience. Consequently, it would be interesting to consider conducting future studies that establish treatment guidelines based on the OSI score. In this study, we can conclude that the OSI classification exhibits strong concordance agreement irrespective of the clinicians' level of experience, demonstrating reproducibility among profession- als with differing levels of expertise without significant variations. Additionally, the utilization of the OSI scale has revealed a higher incidence of severe ONM when compared to visual classification, regardless of the clinician's level of experience. AUTHOR CONTRIBUTIONS David Navarro-Pérez: Conceptualization; methodology; valida- tion; investigation; data curation; writing – original draft. Sara García-Oreja: Conceptualization; methodology; software; valida- tion; formal analysis; investigation; writing – original draft. Aroa Tardáguila-García: Data curation; writing – review and editing. Diego León-Herce: Conceptualization; methodology; software; in- vestigation; writing – original draft; visualization. Francisco Javier Álvaro-Afonso: Visualization. José Luis Lázaro-Martínez: Validation; resources; writing – review and editing; supervision; funding acquisi- tion; project administration. ACKNO​WLE​DG E​MENTS No specific grant was received from funding agencies in the public, commercial or not-for-profit sectors. F I G U R E 3 Researchers' visual assessment of the photographs, classifying them as mild, moderate or severe. The most experienced clinician classified 38% (19/50) as severe, 24% (12/50) as moderate and 38% (19/50) as mild. The moderately experienced clinician classified 42% (21/50) as severe, 28% (14/50) as moderate and 30% (15/50) as mild. The inexperienced clinician classified 28% (14/50), 44% (22/50) as moderate and 28% (14/50) as mild. ONM, onychomycosis. F I G U R E 4 The investigators' OSI assessment of the photographs, classifying them as mild, moderate or severe. The most experienced clinician classified 80% (40/50) as severe, 8% (4/50) as moderate and 10% (5/50) as mild. The moderately experienced clinician classified 68% (34/50) as severe, 4% (2/50) as moderate and 26% (13/50) as mild. The inexperienced clinician classified 52% (26/50), 36% (18/50) as moderate and 12% (6/50) as mild. ONM, onychomycosis; OSI, Onychomycosis Severity Index. 14390507, 2024, 1, D ow nloaded from https://onlinelibrary.w iley.com /doi/10.1111/m yc.13694 by Spanish C ochrane N ational Provision (M inisterio de Sanidad), W iley O nline L ibrary on [22/04/2024]. See the T erm s and C onditions (https://onlinelibrary.w iley.com /term s-and-conditions) on W iley O nline L ibrary for rules of use; O A articles are governed by the applicable C reative C om m ons L icense     |  5 of 5NAVARRO-­PÉREZ et al. FUNDING INFORMATION This research received no external funding. CONFLIC T OF INTERE S T S TATEMENT The authors have no conflict of interest to declare. DATA AVAIL ABILIT Y S TATEMENT The data that support the findings of this study are available from the corresponding author upon reasonable request. ORCID David Navarro-Pérez  https://orcid.org/0000-0003-3588-0557 Sara García-Oreja  https://orcid.org/0000-0002-2937-7122 Aroa Tardáguila-García  https://orcid.org/0000-0001-5650-7351 Diego León-Herce  https://orcid.org/0009-0009-7973-9652 José Luis Lázaro-Martínez  https://orcid. org/0000-0001-6110-0265 R E FE R E N C E S 1. Scher RK, Baran R. Onychomycosis in clinical practice: factors con- tributing to recurrence. Br J Dermatol. 2003;149(Suppl 65):5-9. 2. Lipner SR, Scher RK. Onychomycosis: clinical overview and diagno- sis. J Am Acad Dermatol. 2019;80(4):835-851. 3. Carney C, Tosti A, Daniel R, et al. A new classification system for grading the severity of onychomycosis: onychomycosis severity index. Arch Dermatol. 2011;147(11):1277-1282. 4. Watjer RM, Bonten TN, Arkesteijn MA, et al. The accuracy of clin- ical diagnosis of onychomycosis in Dutch general practice: a diag- nostic accuracy study. BJGP Open. 2023;7(3):BJGPO.2022.0186. 5. Navarro-Pérez D, Tardáguila-García A, García-Oreja S, López- Moral M, García-Madrid M, Lázaro-Martínez JL. Onychomycosis associated with diabetic foot syndrome: a systematic review. Mycoses. 2023;66(6):459-466. 6. Dubljanin E, Dzamic A, Vujcic I, et al. Correlation of clinical charac- teristics, by calculation of SCIO index, with the laboratory diagnosis of onychomycosis. Braz J Microbiol. 2022;53(1):221-229. 7. Elewski BE, Gupta AK, Rosen T, et al. Onychomycosis: does cure equate to treatment success? J Drugs Dermatol. 2016;15(5):626-632. 8. Baran R, Hay RJ, Garduno JI. Review of antifungal therapy and the severity index for assessing onychomycosis: part I. J Dermatolog Treat. 2008;19(2):72-81. 9. Sergeev AY, Gupta AK, Sergeev YV. The scoring clinical index for onychomycosis (SCIO index). Skin Therapy Lett. 2002;7(Suppl 1):6-7. 10. World Medical Association. World medical association declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA. 2013;310(20):2191-2194. 11. Navarro-Pérez D, García-Oreja S, Tardáguila-García A, León-Herce D, Álvaro-Afonso FJ, Lázaro-Martínez JL. Microbiological culture combined with PCR for the diagnosis of onychomycosis: descriptive analysis of 121 patients. Mycoses. 2023;66(12):1045-1049. 12. Gupta AK, Versteeg SG, Shear NH. A practical application of ony- chomycosis cure - combining patient, physician and regulatory body perspectives. J Eur Acad Dermatol Venereol. 2019;33(2):281-287. How to cite this article: Navarro-Pérez D, García-Oreja S, Tardáguila-García A, León-Herce D, Álvaro-Afonso FJ, Lázaro-Martínez JL. Inter-observer reliability of the Onychomycosis Severity Index depending on clinical experience: A review of 50 cases. Mycoses. 2024;67:e13694. doi:10.1111/myc.13694 14390507, 2024, 1, D ow nloaded from https://onlinelibrary.w iley.com /doi/10.1111/m yc.13694 by Spanish C ochrane N ational Provision (M inisterio de Sanidad), W iley O nline L ibrary on [22/04/2024]. See the T erm s and C onditions (https://onlinelibrary.w iley.com /term s-and-conditions) on W iley O nline L ibrary for rules of use; O A articles are governed by the applicable C reative C om m ons L icense https://orcid.org/0000-0003-3588-0557 https://orcid.org/0000-0003-3588-0557 https://orcid.org/0000-0002-2937-7122 https://orcid.org/0000-0002-2937-7122 https://orcid.org/0000-0001-5650-7351 https://orcid.org/0000-0001-5650-7351 https://orcid.org/0009-0009-7973-9652 https://orcid.org/0009-0009-7973-9652 https://orcid.org/0000-0001-6110-0265 https://orcid.org/0000-0001-6110-0265 https://orcid.org/0000-0001-6110-0265 https://doi.org/10.1111/myc.13694 Inter-­observer reliability of the Onychomycosis Severity Index depending on clinical experience: A review of 50 cases Abstract 1|BACKGROUND 2|MATERIALS AND METHODS 2.1|Participants and samples 2.2|Statistical analysis 3|RESULTS 4|DISCUSSION AUTHOR CONTRIBUTIONS ACKNO​WLE​DGE​MENTS FUNDING INFORMATION CONFLICT OF INTEREST STATEMENT DATA AVAILABILITY STATEMENT REFERENCES