How Effective Are Intelligent Tutoring Systems?

Intelligent tutoring systems are educational applications of artificial intelligence and machine learning technologies. Intelligent tutoring systems are designed to interact directly with students and perform many of the instructional functions usually reserved for teachers or tutors. The systems have been used to teach students in such diverse domains as language, law, mathematics, medicine, physics, and reading comprehension.

Having emerged as a scholarly discipline more than 40 years ago, intelligent tutoring systems research has continued to expand the variety of instructional functions, subject domains, and student responses that the systems can handle. Central to every intelligent tutoring system is the ability to capture data about student responses, use it to model each student’s knowledge, metacognition, motivation or emotion, and adapt instruction to individual needs.

Intelligent tutoring systems often present interfaces with which students interact throughout the learning activity. By tracking student moves at each step, an intelligent tutoring system can build a more detailed student model and also provide hints and feedback at the step-level, not only upon completion of an activity.

In a recently published article, Ma, Adesope, Nesbit, and Liu (2014) (PDF, 146KB) reviewed original research that quantitatively compared outcomes of learners using an intelligent tutoring system to outcomes of learners using other instructional methods. From an exhaustive search of major research databases, the researchers obtained 107 results that met their criteria. The reviewed research contained data from a total of 14,321 learners.

A meta-analysis was applied to calculate the average effect of using intelligent tutoring systems across all the studies and for defined categories of studies. The average effect sizes were calculated as Hedges g, a weighted mean difference between groups expressed as a standard deviation in such a way as to adjust for small sample sizes.

The effect size obtained from an evaluative study of an intelligent tutoring system depends on what instructional treatment the intelligent tutoring system is compared to. Studies that compared a group of learners who used an intelligent tutoring system with a group of learners who received no instruction had a large mean effect size of g = 1.23. For the 107 results in which the comparison group did receive instruction, the mean effect size was g = .41.

Dividing the comparison treatments into categories, intelligent tutoring systems showed statistically significant benefits compared to large-group, teacher-led instruction (g = .44); individual, non-intelligent tutoring system computer-based instruction (g = .57), and individual study with a textbook or workbook (g = .36). When intelligent tutoring systems were compared with small-group instruction (g = .05) and one-to-one human tutoring (g = -.11) there were no statistically significant differences.

Learning from intelligent tutoring systems was associated with higher outcome scores regardless of the level of schooling (elementary to postsecondary), whether the research was conducted in a laboratory setting or a classroom, whether the posttest assessed retention or transfer, whether the posttest was a standardized instrument or an instrument developed by the researcher, whether students learned procedural or declarative knowledge, whether or not the intelligent tutoring system provided feedback, and whether or not it modeled misconceptions.

In fact, over the 21 moderator variables examined, there were only a few categories in which intelligent tutoring systems was not statistically associated with higher posttest scores, and these tended to be categories represented by an exceptionally small number of studies.

The review found that over a wide array of conditions, learning from intelligent tutoring systems was associated with higher outcome scores.

Can it be inferred that intelligent tutoring systems should replace large-group instruction, textbooks, and non-intelligent tutoring system forms of computer-based instruction? While the results demonstrate that intelligent tutoring systems hold great promise, they are biased by researchers’ reasonable practice of evaluating instructional systems under the particular conditions for which they were designed.

Another source of implicit bias is that the development of an intelligent tutoring system often involves a much more thorough analysis of learner needs and instructional tasks than typical instruction. Presuming they accurately represented typical instructional practices, the instruction received by the comparison groups is less likely to have benefited from such systematic design practices.

Citation:
Ma, W., Adesope, O. O., Nesbit, J. C., & Liu, Q. (2014). Intelligent tutoring systems and learning outcomes: A meta-analysis. Journal of Educational Psychology, 106(4), 901–918. http://dx.doi.org/10.1037/a0037123

Return to APA Journals – Article Spotlight homepage

Note: This article is in the Educational Psychology topic area. View more articles in the Educational Psychology topic area.