Using Item Analysis to Improve Tests

First principles: Why testing is important to the educational enterprise

  1. Learning and memory are strengthened when students use effortful recall to answer questions in a well designed test.
  2. Persistent misconceptions can be identified when students answer questions in a well designed test.
  3. Learning reinforced within tests strengthens long term recall more effectively than do other instructional activities, including lecture and reading of a textbook.
  4. The most effective test questions stimulate student problem solving and review of prior learning. Simply recalling specific information tends not to improve learning or long term memory.
  5. A careful review of test results can help to identify issues in course design and materials as well as problems with the questions in the test.

The testing cycle: Steps to improving instruction systematically through careful review and revision of test design and test questions

  1. Design a new test or review an existing test in relation to important course objectives.
  2. For each course objective create questions that ask students to solve problems and make decisions or predictions while reviewing prior learning. If you are asking students to write short answers or essays, define and publish for student review the rubrics that will be used to evaluate these responses.
  3. Administer the test. If you are using computer-based testing and your class is large, find out about the pros and cons of question sampling.
  4. Review the item analysis to determine the effectiveness of each question.
  5. Change instructional design and/or test questions as indicated by the item analysis.
  6. Return to step 1 in the next instructional period.

Steps in a review of an item analysis report

  1. Review the difficulty and discrimination of each question.
  2. For each question having low values of discrimination review the distribution of responses along with the question text to determine what might be causing a response pattern that suggests student confusion.
  3. If the text of the question is confusing, change the text or remove the question from the course database. If the question text is not confusing or faulty, then try to identify the instructional component that may be leading to student confusion.
  4. Carefully examine the questions that discriminate well between high and low scoring students to fully understand the role that instructional design played in leading to these results. Ask yourself what aspects of the instructional process appear to be most effective.