Single Case Analysis and Review Framework
Overview
To download an Excel file that can be used for scoring via SCARF, click below:
To download an Excel file that can be used to automatically generate inter-assessor agreement (IAA), click below. See the bottom of the page for citations that should be used for the SCARF and the accompanying IAA tool.
The SCARF is intended as a tool to assess the quality and outcomes of single case design studies. For the purposes of the tool, “study” refers to any single design, which may include a single participant (e.g., A-B-A-B designs) or multiple participants (e.g., multiple baseline design across participants). Each design should be evaluated separately, even if multiple designs are present in a single source.
Note: This tool is designed for the assessment of groups of articles for the purposes of answering the question: To what extent are studies sufficient and to what extent are outcomes consistent and replicated for Intervention X for changing Behavior Y for Participants with Z inclusion characteristics? Studies should be included or excluded based on your research questions.
Coding Study Characteristics and Primary Outcomes
Internal Validity: The first scale is for assessing the extent to which a study has adequate internal validity. There are 13 items, for a “best” internal validity score of 13, and a “worst” of 0. The higher the validity score, the more confident you can generally be in conclusions drawn from the study, although all items may not be equally important.
External Validity: The second scale is for assessing the extent to which study outcomes may be important outside of study contexts (e.g., the extent to which studies include generalization data, evidence of social validity, maintenance data, endogenous implementers, typical contexts). Some of these indicators may not be relevant in a given review.
Reporting: The third scale is for assessing the extent to which study authors reported necessary information for replication.
Outcomes: Primary outcomes are coded based on the consistency of outcomes across potential demonstrations. A score of 4 is given for consistent positive effects, a score of 0 is given for counter-therapeutic effects. Scores of 1, 2, and 3 indicate null, inconsistent, or weak positive effects. Scores of 3 and 4 are consistent with functional relation identification.
Graphical Representations
Internal Validity: This is a raw score denoting the number of internal validity features are present. Scores closer to the right side of the graph represent high-quality studies, which scores closer to the left axis represent studies with fewer internal validity features. These studies (to the left) may provide less convincing evidence, although if all evidence across internal validity scores (e.g., low and high quality studies) is similar, it may suggest these studies may include valid results.
Primary Outcomes: (range 0-4) = Score coded by reviewer, based on visual analysis
Generalization Measurement: Reviewers code the nature of generalization measurement, with more consistent data collection throughout the study resulting in more confidence in outcomes.
Generalization Outcomes: Similar to primary outcomes, reviewers score generalized outcomes for strong, weak, inconsistent, null, or counter-therapeutic effects.
Maintenance Measurement: We evaluate the extent to which maintenance measurement is separated in time from intervention including no measurement, immediate measurement (i.e., maintenance data are collected right after intervention is withdrawn), or data collected following specific periods of time after intervention is withdrawn.
Maintenance Outcomes: Similar to primary outcomes, reviewers score maintained outcomes for strong, weak, inconsistent, null, or counter-therapeutic effects.
Published Manuscripts using the SCARF
Barton, E. E., Murray, R., O’Flaherty, C., Sweeney, E. M., & Gossett, S. (2020). Teaching Object Play to Young Children With Disabilities: A Systematic Review of Methods and Rigor. American Journal on Intellectual and Developmental Disabilities, 125(1), 14-36.
Chazin, K. T., Ledford, J. R., & Pak, N. (2020) Does it work, for whom, and under what conditions: A systematic review of moderators in augmented input interventions. Available at: https://osf.io/b3urx/
Chazin, K. T., Velez, M. S., & Ledford. J. R. (2020) Interventions without escape extinction to treat escape-maintained behavior: A systematic review and meta-analysis. Available at: https://osf.io/4nxvc/
Dubin, A. H., & Lieberman-Betz, R. G. (2020). Naturalistic Interventions to Improve Prelinguistic Communication for Children with Autism Spectrum Disorder: A Systematic Review. Review Journal of Autism and Developmental Disorders, 1-17.
Gibbs, A. R., & Tullis, C. A. (2020). The Emergence of Untrained Relations in Individuals with Autism and Other Intellectual and Developmental Disabilities: a Systematic Review of the Recent Literature. Review Journal of Autism and Developmental Disorders, 1-26.
Hardy, J. K., McLeod, R. H., Sweigart, C. A., & Landrum, T. (2022). Comparing and Contrasting Quality Frameworks Using Research on High-Probability Requests With Young Children. Infants & Young Children, 35(4), 267-284.
Herrod, J. L., Snyder, S. K., Hart, J. B., Frantz, S. J., & Ayres, K. M. (2023). Applications of the Premack Principle: A review of the literature. Behavior Modification, 47(1), 219-246.
Ledford, J. R., & Pustejovsky, J. E. (2020). Stay-play-talk Meta-analysis. Available at: https://osf.io/u7cph
Ledford, J. R., Trump, C., Chazin, K. T., Windsor, S. A., Eyler, P. B., & Wunderlich, K. (2023). Systematic review of interruption and redirection procedures for autistic individuals. Behavioral Interventions, 38(1), 198-218.
Ledford, J. R., & Windsor, S. A. (2022). Systematic review of interventions designed to teach imitation to young children with disabilities. Topics in Early Childhood Special Education, 42(2), 202-214. Available at: https://osf.io/wsjn4
Trump, C. E., Ayres, K. M., Quinland, K. K., & Zabala, K. A. (2019). Differential reinforcement without extinction: A review of the literature. Behavior Analysis: Research and Practice.
Zimmerman, K. N., & Ledford, J. R. (2017). Beyond ASD: Evidence for the effectiveness of social narratives. Journal of Early Intervention, 39(3), 199-217.
Zimmerman, K. N., Ledford, J. R., Severini, K. E., Pustejovsky, J. E., Barton, E. E., & Lloyd, B. P. (2018). Single-case synthesis tools I: Comparing tools to evaluate SCD quality and rigor. Research in Developmental Disabilities, 79, 19-32.
To cite the SCARF (v 3.1):
Ledford, J. R., Chazin, K. T., Lane, J. D., Zimmerman, K. N., Bennett, P. B., & Ayres, K. A. (2023, May). Single case analysis and review framework (SCARF). Retrieved from: http://ebip.vkcsites.org/scarfv2
To cite the SCARF IAA tool:
Todt, M. J. (2023, May). Single case analysis and review framework (SCARF) inter-assessor agreement tool. Retrieved from: http://ebip.vkcsites.org/scarfv2