Using automated questions to improve reading comprehension
I first read about Project LISTEN back in 2001 or 2000, when I was working on a similar project, which nevery turned out anything. The project director is Jack Mostow - Research Professor, Robotics, Language Technologies, Human Computer Interaction, Automated Learning and Discovery, Program in Interdisciplinary Educational Research.
Project LISTEN (Literacy Innovation that Speech Technology ENables) is an inter-disciplinary research project at Carnegie Mellon University to develop a novel tool to improve literacy — an automated Reading Tutor that displays stories on a computer screen, and listens to children read aloud. … The Reading Tutor adapts Carnegie Mellon’s Sphinx-II speech recognizer to analyze the student’s oral reading. The Reading Tutor intervenes when the reader makes mistakes, gets stuck, clicks for help, or is likely to encounter difficulty. The Reading Tutor responds with assistance modelled in part after expert reading teachers, but adapted to the capabilities and limitations of the technology.
This particular paper is interesting, although the abstract sounds very cryptic.
Mostow, J., Beck, J., Bey, J., Cuneo, A., Sison, J., Tobin, B., & Valeri, J. (2004). Using automated questions to assess reading comprehension, vocabulary, and effects of tutorial interventions. Technology, Instruction, Cognition and Learning, 2, 97-134. Click here to download .pdf file.
Abstract: We describe the automated generation and use of 69,326 comprehension cloze questions and 5,668 vocabulary matching questions in the 2001-2002 version of Project LISTEN’s Reading Tutor used by 364 students in grades 1-9 at seven schools. To validate our methods, we used students’ performance on these multiple-choice questions to predict their scores on the Woodcock Reading Mastery Test. A model based on students’ cloze performance predicted their Passage Comprehension scores with correlation R=.85. The percentage of vocabulary words that students matched correctly to their definitions predicted their Word Comprehension scores with correlation R=.61.
We used both types of questions in a within-subject automated experiment to compare four ways to preview new vocabulary before a story - defining the word, giving a synonym, asking about the word, and doing nothing. Outcomes included comprehension as measured by performance on multiple-choice cloze questions during the story, and vocabulary as measured by matching words to their definitions in a posttest after the story. A synonym or short definition significantly improved posttest performance compared to just encountering the word in the story - but only for words students didn’t already know, and only if they had a grade 4 or better vocabulary. Such a preview significantly improved performance during the story on cloze questions involving the previewed word - but only for students with a grade 1-3 vocabulary.