stylesheet
MCQ Help - Pros and cons of question types
Back to Help Index
MCQ Help - Pros and cons of question types
The five question types are compared in various different ways. Please
note that the phrase "ignorant student" is used below as short hand for
"student who is ignorant of the subject matter of the MCQ paper".
User interface
| Pure Multiple Choice |
One radio button is presented for each statement, initially whith
none selected. If, after selecting a statement the student selects
another, the first statement will automatically be unselected so it
it isn't possible to select more than one statement.
|
| Multiple Response (multiple true) |
One check box is presented for each statement, initially whith
none selected. Any combination of check boxes can be selected.
|
| Multiple True/False |
Three radio button are presented for each statement labelled true, false
and undecided,initially with the undecided button selected. Only one of the
three buttons can be selected - selecting one automatically unselects the other.
However, selections are independent between statements. I.e. the student
can select any combination of true/false/undecided.
If the student has been informed about the marking scheme they will never
leave "undecided" selected. This button is provided purely as a
convenience for the student to use as a 'memo to self' that they should
review the statement before submitting their responses.
|
| Strict Multiple True/False |
The presentation and user interface as the same as for normal
multiple true/false questions.
|
| Multiple True/False/Don't know |
Three check boxes are presented for each statement, initially whith
none selected. The student can select true, false or don't know buttons
for each statement in any combination. The don't know option is provided
because there is a penalty for incorrectly selecting true or false.
|
Marking Scheme
| Pure Multiple Choice |
Simple to understand. One mark is gained for selecting the
correct true statement and no marks are awarded for any other
response.
|
| Multiple True |
One mark is gained for each true
statement selected, one mark is lost for each false statement
selected but if the result of this is a negative number the
marks are rounded up to zero so that poor performance on one
question won't jeopardise marks gained on others.
|
| Multiple True/False |
One mark is gained for each false statement identified and
one mark is gained for each true statement identified. No marks
are lost for incorectly identifying statements and so a "don't
know" option is pointless.
|
| Strict Multiple True/False |
Very simple but strict marking scheme. Five marks are gained
if all five statements are correctly identified as true and false
but in all other circumstances no marks are awarded.
|
| Multiple True/False/Don't know |
One mark is gained for each false statement identified and
one mark is gained for each true statement identified. One mark
is lost for incorrectly identifying a true statement and one mark
is lost for incorrectly identifying a false statement. Don't know
selections score zero. If the final mark is a negative number the
marks are rounded up to zero so that poor performance on one
question won't jeopardise marks gained on others.
|
Effect of guessing part 1 - Well written questions
| Pure Multiple Choice |
An ignorant student may randomly select
one option on each question and would expect to score about 20%
on the paper. If there are a small number of questions a much higher
score will be acheived by some students.
|
| Multiple True (MRQ) |
An ignorant student will select one statement in each question
and would expect to score about 20%. Even with small numbers of
questions it will be rare to score much higher.
|
| Multiple True/False |
Guess work is encouraged by this question type and students
who are motivated to score marks will never leave a statement
unanswered. Ignorant students will expect to score 50% by randomly
selecting either true or false on each question and
many students will score much higher if there are few questions
in the paper. The situation will be much worse if the
questions are badly written.
|
| Strict Multiple True/False |
Ignorant students may choose to randomly select true or false on each
statement but there is a 31 out of 32 chance that they will score 0
on each question. They will expect to score 3% on the paper even
with small numbers of questions.
|
| Multiple True/False/Don't know |
An ignorant student would randomly select one statement and
randomly select true or false. They would expect to score about 10%
on the paper. The situation will be much worse if the
questions are badly written.
|
Effect of guessing part 2 - badly written questions
Multiple true/false and true/false/don't know question should
never contain statements which make it obvious that the majority
are false or the majority are true.
For example, a question which is more obviously a pure multiple
choice question will fall into this category. The ignorant student knows
that there is one true answer and can gain marks on the false ones.
| Multiple True/False |
An ignorant student will select false on every statement and
expect to gain 80% on the paper as a whole.
|
| Strict Multiple True/False |
Ignorant students may choose to randomly select one statement
as true and select the others as false. They will expect to score 20%
on the paper as a whole.
|
| Multiple True/False/Don't know |
An ignorant student will select false on every statement and
expect to gain 60% on the paper as a whole.
|
Effect of guessing - Tabulated
Please refer to the two sections above for an explanation of these figures.
| Type |
Good Questions |
Bad Questions |
| Single True (MCQ) |
20% |
N/A |
| Multiple True (MRQ) |
20% (narrow spread) |
N/A |
| Multiple True/False |
50% |
80% |
| Strict Multiple True/False |
3% |
20% |
| Multiple True/False/Don't know |
10% |
60% |
Effect of Naivety
Assessment are supposed to rank students according to ability
at the subject matter of the assessment. Able students can apply
guesswork just as well as ignorant ones and so you might assume
that easy to guess questions will simply shift the marks higher
for the whole class. However, this assumes that all students
are aware of the best strategy for guessing. In reality the
class will be composed of students with a range of ability at
MCQ strategy. To highlight this compare the difference in
expected marks for a naive+ignorant student and a devious+ignorant
student. (It is assumed that even a naive student will spot
the most obvious opportunities for guessing safely.)
Students who don't know the subject
| Type |
Naive |
Devious |
Advantage |
| Single True (MCQ) |
20% |
20% |
No advantage |
| Multiple True (MRQ) |
0% |
20% |
20% |
| Multiple True/False |
0% |
50% |
50% |
| Strict Multiple True/False |
0% |
3% |
3% |
| Multiple True/False/Don't know |
0% |
10% |
10% |
Students who half know the subject
The following are more real world figures and apply to two students who are
able to correctly and confidently answer half of the questions on a paper
and may or may not apply devious guessing strategies to the others.
| Type |
Naive |
Devious |
Advantage |
| Single True (MCQ) |
60% |
60% |
No advantage |
| Multiple True (MRQ) |
50% |
60% |
10% |
| Multiple True/False |
50% |
75% |
25% |
| Strict Multiple True/False |
50% |
51.5% |
1.5% |
| Multiple True/False/Don't know |
50% |
55% |
5% |
Recommendations
| Pure Multiple Choice |
Best for use with mutually exclusive options
Suitable for self assessment and for summative assessment
This question type is more difficult to compose. However, if the
question you want to ask obviously has only one right answer you
should use this question type. It is difficult sometimes to avoid
'silly' false statements and this can lead students to waste time
applying a process of elimination which involves time consuming
comparison of statements.
|
| Multiple Response (multiple true) |
Use for compatibility with paper based examinations that use this scheme
Question of this type are easier to compose than pure MCQ questions
It persuades most students not to apply
processes of elimination and rely more on knowledge. However, students
are not rewarded for correctly identifying false statements and
questions have variable available marks.
|
| Multiple True/False |
OK for gentle self-assessment
Big rewards for guesswork strategies
Severely penalises students for naivety
Very high mean mark
Less useful statistical analysis
This question type is easily understood and is very suitable
for self assessment and diagnostic tests. However, its use in formal
assessment (or practice for formal assessment) is problematic.
Naivety versus deviousness may have a much bigger influence on the
ranking of students in the class than real ability. You can gaurd
against that problem by informing all students of the best way to
guess. A greater problem is the danger of accidently writing
statements that are more correctly single true types. Great care
must be taken over question design. An obvious problem that is
sometimes overlooked is that pass marks and grade boundaries
ought to be reviewed if this type of question is used to replace
other types.
|
| Strict Multiple True/False |
Best all-round question for formal assessment & practice formal assessment
May dishearten students if used for self-assessment or diagnostic assessment
Almost no reward for guesswork
No penalty for guesswork either
Ranking closely matched to real ability
Obvious that knowing the answer is the only strategy
This is an ideal question type for most purposes. Guesswork
is hardly rewarded at all, class ranking should be closely matched
to real ability and questions are quite easy to write.
If by accident statements are designed that are more appropriate
for a single true type of question then guesswork will be more
richly rewarded - but no more than if a single true type had been
selected anyway.
Some students may feel that the question type is very severe and
strict but this can be counteracted by designing questions and
statements that are not too difficult. In particular, questions
in which each of the five statements asks the student to repeat
the same intellectual skill or process of deductive reasoning.
If you run practive assessments it will be easy to persuade
students not to guess at all and that way the analysis of class and
individual performance will be a very accurate representation
of misunderstandings.
|
| Multiple True/False/Don't know |
Compromise between strict multiple true/false and normal multiple true/false
This is a compromise question type - it rewards gueswork less
than the normal multiple true/false but has a less severe marking
scheme than the strict multiple true/false type. However, some students
may waste time working on guesswork strategy.
|