Test conditions where question sampling is not recommended
Test students all at one time
Testing all students at the same time in a room or rooms that have proctors on duty effectively removes any opportunity for students to engage in unauthorized sharing of information. This is clearly the best way to eliminate cheating through sharing information about test questions. If questions are presented one at a time on the computer screen, there is even little need to scramble questions and responses. This approach supports tests where the order of questions is important. This strategy assumes that proctors clearly establish the identity of each student at the time they present themselves to take the test and that proctors further ensure that cell phones, cameras and other electronic devices are not brought into the testing room. It also assumes that proctors actively monitor the behavior of students during the test. Under this set of circumstances it is virtually impossible for students to effectively gain unearned advantage during the testing period. Beyond the ability to control student behavior to a greater extent, this method has other advantages including
- the need to create just one version of the questions,
- the ability to avoid complex measurement issues in the scoring of the test and
- the ability to make simple adjustments to the scoring of the test, if needed.
The standard item analysis for tests with just one set of questions can be easily interpreted and used to explore problems on particular questions as well as provide information that can be used to help make changes in how particular topics are taught. It is an ideal procedure.
Scramble questions and responses
If students in a class complete a test over several hours up to one day then no extraordinary measures are likely needed. Simply scrambling the questions and the responses of each question makes sharing specific usable information about a test much more difficult. For example, students would not be able to simply memorize the location of the correct answer, i.e. A, B, etc., but rather would have to memorize most or all of the text of the correct answer in order to be able to give information specific enough to be useful to other test takers. It is also true, however, that some questions can be relatively easy to memorize either because there is very little text in the stem of the question or each response is short. Of course both of these characteristics may be true of some questions in which case the sharing of information is easier. Since the order of questions is different for each test taker, students wanting to share information will have to resort to remembering key words rather than the whole text of the complete question stem. Once again the risk of disclosure increases if the text in the question is short. It is standard practice to write multiple choice questions so that one distracter is similar to the correct answer but clearly incorrect and others are less like the correct response. In effect students may only have to memorize one key word in order to give away a correct answer. This method assumes that proctoring is in place during the time when students are completing the test. Since this method involves each student answering the exact same set of questions, the measurement advantages mentioned above are also advantages here. This strategy may be all that is needed when the testing period spans several hours up to one day.