Toward more valid and reliable classroom assessment of digital multimodal composition performances: developing a scoring rubric to assess upper elementary students' digital multimodal book reviews
Date
2018
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
University of Delaware
Abstract
Digital multimodal composition has drawn attention from literacy scholars and educators as a new type of writing activity that leverages digital technologies and their affordances. However, there is limited research on what and how to assess elementary students’ task-specific digital multimodal composition performances. This mixed-methods study aims to explore the following four research questions in four phases: 1) what are the key domains and criteria that represent the construct of digital multimodal composition? 2) how might these domain and criteria be structured in a task- and grade-specific rubric to evaluate upper-elementary-grade students’ digital multimodal book reviews? 3) to what extent does the use of the proposed rubric display evidence of inter-rater reliability, construct validity and usability, as indicated by raters’ scores and interview feedback? and 4) how does the evidence of inter-rater reliability, construct validity, and raters’ feedback inform the process of rubric revision? In the first phase, a systematic literature review was conducted and a total of five domains and 19 distinguishable criteria were identified from 111 criteria in pre-existing 17 studies. In the second phase, a rubric was created in six steps. In the third phase, rubric validation was conducted by training two raters and collecting both quantitative and qualitative evidence on inter-rater reliability, construct validity, and usability of the developed rubric. The analyses revealed that there were five criteria (i.e., C3-conventions of oral language, C4-conventions of written language, C6-relationship between oral language and written language, C10-quality of summary, and C11-quality of opinion) that seemed to be problematic. In the final phase, the rubric was revised by closely examining the problematic criteria and discussing possible solutions for those through a focus group interview. The findings could contribute to our understanding of digital multimodal composition, rubric development procedures, and the role of the rubric for formative assessments in classrooms. ☐ Keywords: digital multimodal composition, digital multimodal book review, scoring rubric, inter-rater reliability, construct validity, formative assessment, upperelementary grades