Creating tests by question sampling

Creating tests or quizzes be sampling from a test bank (or question pool) prevents students from being able to share information about specific questions with each other because each student gets a different set of questions.

When to sample

If the test-taking period is longer than a single day, question sampling may be an important strategy to minimize students ability to share test information with each other. In addition, multiple choice questions that ask for recall of facts are relatively easy for students to remember. It is possible to use an item analysis to help you decide if sampling is necessary. If the questions become easier over time or students take less time to answer them, you have evidence of the need to use question sampling.

When not to sample

If you are giving the test all at one time, or over the course of a single day, question sampling is not necessary. In this case, simply changing the order of the questions and/or responses will be enough to deter students from being able to share information with others.
In addition, it may not be necessary to use sampling if the question types are complex enough to reduce students’ ability to remember and share them. Read more about complex question types that are difficult to share.

Sampling for low stakes quizzes

Sampling of questions is commonly used for low stakes quizzing that count for a small proportion of the grade and it is relatively easy to implement in many computer-based testing systems, including ANGEL [link to ANGEL question pool instructions] and TestPilot.

Sampling for tests

If the test you are giving counts for a substantial portion of the grade then extra care must be taken to ensure that each student gets a test that assesses the same information and is of similar difficulty level. If care is not taken to see that the questions sampled from each group are parallel (similar in difficulty level), then some students will get easier tests than others. In this case, student scores will not accurately reflect their own ability.

Creating fair tests with sampled questions

1. Create content areas (sampling groups) within which your questions will fit.
2. Write several questions for each sampling group. These questions should be parallel; that is, they address the same concept and have the same difficulty level.
3. Include the questions on a test.
4. Assess the questions using an item analysis.
5. Adjust questions as needed.