ICC4IRR is an R/Shiny application to estimate interrater reliability (IRR) from quantitative planned incomplete data, resulting from observation studies in which raters (partly) vary across subjects.
👉 Use the app here: https://tasospsy.shinyapps.io/icc4irr_app/
📖 Reference: Psychogyiopoulos, Koopman & Ten Hove (2025)
- Estimates interrater reliability via hierarchical linear models and maximum likelihood estimation
- Provides six types of intraclass correlation coefficients with Monte Carlo confidence intervals
- Computes variance components (subjects, raters, residual)
- Supports planned-incomplete designs with non-overlapping raters
- Includes tools to compute or estimate design factors (
$k$ ,$\hat{k}$ ,$Q$ ) for your study - Flowchart support to help decide which ICC to interpret
👉 See an example in Ten Hove et al. (2025, Multivariate Behavioral Research).
Software citation
Psychogyiopoulos, A., Koopman, L., & Ten Hove, D. (2025). ICC4IRR: A Shiny application to estimate interrater reliability using intraclass correlation coefficients. https://tasospsy.shinyapps.io/icc4irr_app/
Example in-text citation
We investigated the interrater consistency [or agreement] using intraclass correlation coefficients (ICCs) [that accounted for partially non-overlapping raters across subjects] using the R/Shiny application ICC4IRR (Psychogyiopoulos, Koopman, & Ten Hove, 2025).
- Tasos Psychogyiopoulous
- Letty Koopman
- Debby ten Hove
📧 Contact: d.ten.hove@vu.nl
-
Ten Hove, D., Jorgensen, T. D., & van der Ark, L. A. (2024). Updated guidelines on selecting an intraclass correlation coefficient for interrater reliability, with applications to incomplete observational designs. Psychological Methods, 29(5), 967–979. https://doi.org/10.1037/met0000516
-
Ten Hove, D., Jorgensen, T. D., & van der Ark, L. A. (2025). How to estimate intraclass correlation coefficients for interrater reliability from planned incomplete data. Multivariate Behavioral Research. https://doi.org/10.1080/00273171.2025.2507745