Advice on using this document. If you want to work through the instructions, carrying them out as you read, then a printed copy may be useful, but you could also switch back and forth between this window and the main window as you progress. Printing varies depending on your operating system and the web browser software you have chosen to use. If you choose to print, adjust the size of the text appropriately according to your preference and use a preview feature if your web browser has this capability. You might also prefer to change the colours to black on white using the options link in the @Bodington@ navigation bar before you print.
AUTHORS Jon Maber with contributions and revisions from Andrew Booth, Lawrence Hamburg and Terry Wassall of the University of Leeds
DATE February 2004
This document assumes that that the reader is familiar with the process of setting up a suite of rooms in this web site, creating rooms andresources in a suite, and setting access rights to these rooms and resources. The instructions in this document assume you are working in a room or suite of rooms you havealready created.
In is almost always essential that MCQ papers are invigilated if they are used for formal, summative assessment. Therefore MCQ papers delivered through the @Bodington@ software are almost always used for self-assessment, formative assessment and for assessing the performance of your teaching. There are several reasons for this;
If you intend students to use carry out a time consuming activity before recording responses, such as doing library research or performing calculations, please supply them with a suitably formatted web page or printed copy of the questions so they can log out of their computer and complete the work before returning to the computer to enter responses.
You are advised to devise more detailed guidelines, with your fellow teachers, for the appropriate use of MCQ papers in your discipline.
The marking scheme used on MCQ papers in @Bodington@ is quite inflexible because it is based on many years of experience in the Biochemistry department of the University of Leeds, refining the best possible marking scheme. Also the tool was conceived mainly for students to practice for formal assessment and it was considered desirable to avoid confusing students with a large number of different marking schemes. In summary;
For genuine MCQ questions 1 mark is available per question. The student can only select one statement. If it is the true statement they gain the mark but if it isn't they score zero. There is no negative marking of any kind for genuine MCQ questions.
For MRQ statements the number of available marks is equal to the number of true statements in the question. The student can score between zero and full marks but cannot score less than zero. Each true statement selected gains a mark but each false statement selected will cancel a mark out if any are left to cancel. This subtle and minimal form of negative marking is essential because without it the students could select all the statements and score 100% on the test.
The author of this document has analysed a very large amount of data from MCQ questions and is of the opinion that adjusting the number of marks deducted (e.g. deducting 0.5 marks instead of 1 mark) in a negative marking scheme makes very little difference to the ranking of the students and therefore has little affect on the power of the MCQ paper to discriminate effectively between students of different ability. However, the no-negative-questions scheme used by @Bodington@ avoids penalising students on the basis of personality extremes, i.e. timid students are not put off attempting any given question because they no they cannot get a negative mark which will make them lose marks they gained on other questions, overly bold students are not excessively penalised for having attempted questions which they are not fully confident about. When this scheme was adopted many students who received very low marks on MCQ compared to other forms of assessment, started to get marks more in line with their other types of assessment. Another advantage of the scheme is that the lowest and highest possible marks on the test are 0% and 100% whereas for a simple negative marking system there is a variable bottom mark which is likely to be far below 0%, distorting the normal, bell shaped mark distribution that you would expect for assessments. There are plans to include new question formats in @Bodington@ in the future and these will require development of further marking schemes.
In the final mark that the software reports the marks gained are divided by the marks that are available and this is shown as a percentage with one decimal place.
An MCQ paper resembles other @Bodington@ interactive documents in because it has a location, title, description, introduction, access rights etc. What makes it different is that it has a few special properties, contains a list of questions and also contains records of users' attempts at the paper. An empty MCQ paper is created by completing a form in the same way as for other resources and then questions are added and then edited. After the questions have been edited students can be allowed access to the test and you can monitor its usage and analyse student performance.
Three frames are displayed when you enter an MCQ paper. The top frame, like all @Bodington@ tools, contains the title and navigation icons, the left hand frame contains commands and the right hand frame initially shows the introductory text but will change to reflect the output of selected commands. When the MCQ paper is actually attempted an entirely new window appears containing the questions. Commands in the left-hand frame vary depending on the access rights of the user. The creator of the MCQ paper will see a nearly full list of commands and can make this a full list of commands by changing his/her own access rights.
Users of audible browsers should be aware that when they select commands in frame 2, pages often appear in frame 3. Some audible browsers and screen readers will alert you to this and take you to the appropriate frame. It is the developers' intention to redesign this aspect of the user interface to make it easier to use.
Upload
a file here.
Create a new resource
here.
Bring
rooms or resources here.
Manage this location.
All of the text and options in this form can be changed later if you want.
The command area of the MCQ paper will look similar to this;
You can set access rights to an MCQ paper in the same way as any other @Bodington@ resource. There are also MCQ specific options that affect what users can do with the MCQ paper. Here is a full list of rights and options;
| see | Users with this access right will see the MCQ paper listed when they enter the room where it is located. |
| view | Users with this access right will be able to enter the MCQ paper and run it subject to the deadline and one attempt only options. |
| see | Users with this access right will see the MCQ paper listed when they enter the room where it is located. |
| review | Users with this access right will be able to access a record of their own personal record of attempts at the test paper. This option is not subject to any other options. Please note that even as the owner of the MCQ paper you may need to grant yourself this access right. |
| edit | Users with this access right will be able to edit questions. Note that a question cannot be deleted if one or more users have attempted the test because its text is required for those users' debrief pages. |
| upload | Users with this access right will be able to upload files to the MCQ paper. These could be multimedia files used to illustrate the questions or used in debrief text. |
| manage | Users with this access right will be able to change access rights and view class performance. |
| one attempt only | A user who has recorded responses already will not be permitted to do so again if this option is selected. It is not possible to prevent a student viewing the questions and then coming back to record responses at a later date. |
| mark withheld | If this option is selected the mark will not appear on the page that confirms that a users responses have been recorded. Please note that if the users have been given review access they can see their mark that way. If the withhold mark option is used it is common practice to only grant review access after the date that you expect students to complete the paper. This option is sometimes used with students who lack maturity and will collaborate on practice test papers instead of making a genuine attempt. With most students it is recommended that you allow immediate feedback on performance so the students can reflect on this while the reasons they selected specific responses are fresh in their minds. |
| deadline | If a date is supplied here users are not allowed to record responses after 00:00 (not 24:00) on the day specified. This is sometimes used to motivate students to complete a practice assessment and therefore study course materials and lecture notes, soon after teaching material has been delivered. |
Question 1 (100) |
||
| Question text here. | ||
| A | TRUE | Option text here. |
| B | TRUE | Option text here. |
| C | TRUE | Option text here. |
| D | TRUE | Option text here. |
| E | TRUE | Option text here. |
| Explanatory text here. | ||
The different elements of the template question are displayed in tabular form. The number in brackets (100) will be used by the database to sort questions in order and can be edited. The text Question 1 is generated automatically whenever the test paper is displayed. The text Question text here is the question stem and will appear on the question paper above the five statements. There are always exactly five options for each question in the current version of @Bodington@. Obviously the words TRUE and FALSE will not appear next to the statements when students attempt the test and they are displayed here for the benefit of the editor. The text that appears after the statements; Explanatory text here is the debrief text. It will not appear when the student attempts the test but it will be printed on the debrief page - this is a useful way of providing feedback to students for each individual question.
The best way to find out how to use the Nathan Bodington MCQ paper facility is to experiment with the different options available in an experimental MCQ paper.
Once your students have taken the test (get your colleagues to try it out if you want to experiment with a trial paper) you can use the View Class Results option. The page of options for viewing class results includes an option for detailed statistical analysis of the questions. Further information about editing questions and the statistical analysis features are outlined in the rest of this guide.
Students answers cannot be deleted from a @Bodington@ MCQ. In any case, it is useful to archive completed MCQs for teaching quality assurance and other purposes. However, the questions in an MCQ can be exported and then imported into a new MCQ. The questions are exported to a file formatted to the IMS QTI standard. QTI stands for Questions and Test Interoperability. At the bottom of the command menu in the left hand frame of the questionnaire you will see the commands that enable this process. To export the questions to a file click on Export questions to IMS QTI file. After a few seconds you will see a confirmation that the file has been created and instructions onhow to download it to your hard disk. Right click on the download link, choose the option to save the target file and select a location on your PC. The file will be called qti.xml. You may wish to rename the file to something more helpful for identifying its contents if you download and store more than one qti file.
To import the questions into a new MCQ simply create a new MCQ in the NBB location required. Then, instead of adding and editing questions as detailed above in this document, click on the Import questions to IMS QTI file. A browse box will open for you to select the file. Once you have selected the qti file click on the Send Now button. The questions will be added to the end of the MCQ paper so you can combine questions from multiple sources. You can now edit them if you wish, delete and add new questions, reorder them by editing the ordinals, etc.
Technically these boxes require HTML (HyperText Markup Language). But in effect that means plain text unless you want formatted output. Some advice on putting formatted text and graphics into questions follows later but for these first questions stick to plain text. There are some things to note;
MCQ authoring may be the most demanding form of creative writing there is. Poorly thought out questions can confuse and frustrate students and have a negative impact on their performance on your course. Some rough guidelines are provided here but this is not the place for an exhaustive review of the art of MCQ composition.
It is absolutely vital that you make a list of the knowledge and competencies that you wish to assess. (In the case of MCQ papers in @Bodington@, remember that theseare self-assessment or practice papers and the assessment objectives may be different to those you would list for your formal assessment.) Contrary to popular belief MCQ papers can test more than factual recall but there are limitations to the assessment objectives they can cover. Here are some examples of what you can and cannot test:
There are many different forms of test in which the student is offered multiple answers to questions. In the tests available via @Bodington@, each question always has five alternative answers. You may decide that you will design questions which always have one correct answer and four wrong ones. These are true Multiple Choice Question(MCQ) tests. Alternatively, you may decide to design questions in which more than one of the five alternative answers are correct. Although we tend to call these MCQ as well, more correctly they are Multiple Response (MRQ) tests. As we will see, MRQs offer some advantages over MCQs. In the Nathan @Bodington@ Building, you can design both MCQ and MRQ questions. You can design tests in which some questions are MCQ and others are MRQ. For good educational (and programming) reasons, you cannot design questions in which none of the answers are correct - Irrelevant Choice Question (IRQ) tests are not available via @Bodington@.
Any MCQ or MRQ test will contain questions together with alternative answers. At least one answer must be correct. The other, wrong, alternatives are referred to as distracters. It is generally much more difficult for academics to create distracters than to write the correct answers. As an expert in your field of study, your familiarity with the subject will probably make it difficult to predict the misconceptions that confuse students. A good distracter must not be obviously wrong, yet it must be unequivocally incorrect. It is not easy to devise a single distracter, and it is much harder to devise the four of them required for a true Multiple Choice Question. Since distracters are so hard to write, MRQ tests are becoming increasingly popular, since they contain more correct answers and hence fewer distracters. If you tell your students thatfor each question, "at least one alternative answer will be correct, but any or all alternatives may be correct", you will not make yourself popular, but you will at least discourage the random guessing that true MCQ tests can produce. Since a student is allowed to select any number up to five responses to each MRQ it is essential that some element of negative marking is applied - otherwise a student can select all options on all questions and score 100% on the paper.
Some tips:
Who was the Prime Minister of Jamaica in 1997?
The simpler the question, the harder it is to write good distracters. (It was P.J. Patterson. Sir Florizel Glasspole was Governor General of Jamaica. Probably abad distracter, but a wonderful name.)
The opening question, the five statements and the explanation provided in a debrief are all entered using HTML. That means that pretty much anything that you've seen on a web page could be put into a question. That includes formatted text, graphics, audio clips etc. The sky's the limit but the use of media is more difficult to achieve than formatted text.
A complete treatise on HTML is beyond the scope of this guide but you only need to know enough to achieve the types of questions you want to write and so you may get by with minimal knowledge of the subject. It was mentioned earlier that the < and > symbols are special characters in HTML. That is because they are used to introduce tags to the text. The angle brackets enclose special codes that switch on and off formatting features. Here are some useful tags;
<em> Switch on emphasis. </em> Switch off emphasis.
<strong> Switch on strong emphasis. </strong> Off
Switch on <sup> superscript. </sup> Off
Switch on <sub> subscript. </sub> Off
<P> Start paragraph. </P> End paragraph.
Your can refer to any good guide on HTML to find out more. There are some consequences of your HTML appearing in an MCQ question. In effect you are composing just one fragment of an HTML page. So;
Graphics and are other media are included in HTML documents using a radically different approach than word processors. The media files are not stored within the text but are stored as distinct files and are simply referenced in the text. Web browsers recognise references to import media and load all the appropriate files when the text is loaded. The tag for including an in-line graphic is;
<img src="image.gif" />
There is no end of image tag and that is the purpose of the space and slash within the img tag. The image file will be automatically loaded and displayed within the page. Please note that this display should be previewed by running the test since there are possible technical problems You can upload a graphic into your @Bodington@ MCQ page by selecting the Manage and then Manage Uploaded Files links. If the graphic belongs to someone else and is on an open access web server you don't need to upload it to your MCQ paper - just use a full web address to the graphic on the web server it originates from. It is best to use this method only if you are already familiar with HTML tags and their use for including graphics in web pages.
Web browsers can be equipped to understand types of media other than just graphics and there is a general purpose tag <embed> that can be used to include other media types within a page. Alternatively you can provide link directly to media file and these will be displayed by the web browser in a separate window. So you could use the following URL;
Listen to this piece of <a href="clack.wav">music</a> and identify the composer. You will need to upload the music file into the question by selecting the Manage and then Manage Uploaded Files links where you can then select your file (in this example, clack.wav) and click on the Send now button. When the student clicks on the link in the question your browser will open a control panel to play, pause, rewind etc. the sound clip. One should be awarethat unusual media types will only work if the students are using a web browser with installed support for the file type. Web browser software may provide a user interface for listing supported media types.
You can obtain a table containing class results and you can also see a detailed record of individual attempts at the test. This enables you to identify students whose performance suggests that personal feedback would be beneficial and also helps you to decide on feedback to deliver to the class as a whole. In most circumstances this functionality is the WHOLE reason for creating the MCQ paper in the first place but despite this the author observes that few teachers bother to put it into practice.
There may be many hundreds of attempts at your test and it is often best to filter out some attempts. This part of the form includes a number of different options for filtering. Students Only filters out non-student attempts based on non-membership of the @Bodington@ 'allstudents' group. Your fellow teachers may have tried out the test but you don't want these to be included in your analysis of class performance. Ignore Blank Attempts will filter out the results where the student selected none of the statements on any questions. Sometimes a student will "cheat" the test by completing a blank test just to gain access to the explanatory text. It is best to filter these out if you are going to apply a statistical analysis. First Attempt Only will take only the first attempt for each user and filter out the rest. You should be aware that if you use this in combination with the previous option this will return the first non-blank attempt and so some students may have obtained this mark with thebenefit of seeing the explanatory text.
Match Only These Student Ids allows you to specify specific user(s). There may be times when you wish to pick out the results on a test of a specific student or list of students. For example, you may have granted open access to all students on your practice MCQ test but you want to see results only for students enrolled on your module. On the class results options form you can use the "Match only these student IDs" box to specify which students to search for. The same box can be used to provide a set of marks for the students that you wish to correlate with marks on this MCQ paper. Please note that currently in the @Bodington@ software 'student IDs' means '@Bodington@ system ID numbers of the students' and not the student identifiers that you assign to students. This is because @Bodington@ can contain students of multiple colleges whose college assigned IDs may clash with each other. You can obtain @Bodington@ ID numbers from local college/university IDs using the User Directory tool. The format for text in this box is designed to help you to use it in conjunction with your own spreadsheet or database. If you haven't selected "Custom Correlation" you simply provide a list of student IDs with each one on a line of its own - this can be done by selecting a column of IDs in a spreadsheet and pasting in. If you have selected "Custom Correlation" then each student ID should be followed by a space or TAB and then a mark on a scale from 0 to 100. This can be done by selecting two columns in a spreadsheet and pasting in.
After is the lower cut off date for filtering. Before is the higher cut off date for filtering. If you leave these boxes blank no filtering by date will be done.
This section affects the format of the output of individual results. You can have more ot less detail in the output and you can choose to view on the web page or get the results in a text box for export to a spreadsheet or other software. If you select a format for export you won't get to see hypertext links to individual records for each student.
This provides options for statistical analysis of the class results. You should be aware that only records that pass through the filtering process are analysed and all other records are ignored. There are several levels of detail available for the analysis; Basic Statistics gives you the mean and standard deviation of the overall mark for the paper. Question Analysis will check for a link between performance on each question and performance on the test overall. Question and Item Analysis adds individual analysis of the statements within a question to the analysis of the question as a whole.
Custom correlation is a facility for correlating performance on this MCQ test against some other assessment. You could, for example, provide a list of final course module marks and check for a relationship between these and performance on your practice MCQ test. This facility should be used with care - there's little point checking for correlation between two assessment methods that test different competencies. You should also bear in mind that individual performances on a practice test may bear little relationship to the ability of the student anyway because sometimes students only attempt a few questions in the paper and sometimes they repeat the test until they have learned the answers by rote.
Class Performance Query Name Student ID Mark W R Hinds Not student 11.1% C P Walker Not student 16.7% G Watts Not student 22.2% W D Patterson Not student 38.9% C C Taylor Not student 66.7% C C Taylor Not student 22.2% R Francks Not student 11.1%
Warning - there is no shortcut to studying all of this section if you are going to responsibly analyse the performance of your questions and indirectly your teaching.
Two statistical methods are employed to help you judge the effectiveness of your questions and the performance of your class; Correlation and Analysis of Variance (AnoVa). First let's just review these statistical methods, then discuss how they apply to analysis of test results.
Correlation is used to test the relationship between two variables. The method assumes Normal variation in both of the variables. The statistical value calculated is the Product Moment Correlation Coefficient r. This number can range from -1.0 up to 1.0 (or -100% to 100%). A value of 100% indicates a strong relationship between the two variables and if they were graphed against each other you would see a perfect straight line sloping upward. A value of -100% also indicates a strong relationship between the two variables but an inverse one - if they were graphed against each other you would see a perfect straight line sloping downward. A value of 0% indicates no relationship between the variables - if graphed you would see a nebulous area of data points with no obvious way to fit a line through. Intermediate values occur when a line can be fitted to the data but there is random variation either side of that line.
Particularly with small data sets, it is possible to get a high value for r due to chance variation in the data. So a statistical test is made (a type of t test) to find out if r is significantly non-zero. If the significance is greater than 95% one can practically assume that there is a genuine relationship between the two variables. When using @Bodington@ MCQ tests you do not need to perform any calculations and you don't need to know the formulae used - you only need to look at the bottom line level of significance and whether a significant relationship is positive or negative.
AnoVa is used to compare two or more groups of figures with each other to see if there is significant variation between group means. There are number of varieties of AnoVa but for MCQ analysis only the simplest scenario applies - students are sorted into two groups and the mean mark for one group is tested against theother. The obvious statistic is the difference in the two means for the two groups. It is possible that the difference between the two means is due only to random variation inthe data (especially with small data sets) so AnoVa determines the significance of the difference. If the significance is greater than 95% one can reasonably assume that there is a genuine difference between the two groups.
When you request question analysis statistical values are calculated based on the class performance on the question compared to the class performance on the test paper as a whole. The selection of the analytical method depends on the type of question. For questions that have only one true statement an AnoVa is performed but for questions with multiple true statements and therefore a sliding scale of performance, a correlation is performed. Before you attempt to interpret the statistical values you must be very clear what performance on the test as a whole measures. Was it a random collection of general knowledge questions? Was it testing factual recall within a segment of your syllabus? Was it testing problem solving ability? If it was a self assessment test, does the final mark boil down to an inverse measure of apathy? Did it just test the ability of students to decipher waffly, ambiguous and misleading questions?
Question analysis looks at the ability of the question to discriminate between students on the same criteria as the test as a whole. A bad question in a bad test paper may score well!
When a question awards marks on a scale because there are multiple true statements a correlation is performed. For each student X is taken to be their mark on the question and Y is their mark on the whole test. One must take care in interpreting the results of the correlation and pay attention to the significance level. Particularly one should ignore the correlation coefficient if it is not significant. This leaves you three scenarios;
Correlation between the mark on the question against the mark on the test paper is only able to highlight possible problem questions. The statistics can't tell you why the result came out the way it did and so it is probably is better not to provide a practice MCQ paper unless you are prepared to do some investigation in response to the analysis. Talk to the author of the question, the person who taught the topic area and some of the students and try to find out what went wrong.
A correlation doesn't make sense for questions where the student either gets it wrong or right because there isn't a sliding scale of performance on the question. So for these questions the students are divided into two groups; Those that got the question right (scored 1) and those that got it wrong (scored 0). Getting the question wrong could be due to; not attempting it, selecting the wrong statement or selecting more than one statement (if allowed). In the analysis you get to see how many students got the mark and how many didn't. You also see the mean performance of each group on the test as a whole. You would probably expect that the students who got this question right generally did well on the paper as a whole. If the question is a good discriminator the students who got it right will have scored better on the paper than the students who got it wrong. To measure this the analysis presents the difference between the two group means - this runs on a scale from -100% to 100%. A value near zero suggests that the question doesn't discriminate well because equally able students got this question right and wrong. A non-zero value for this difference in means can occur due to random variation in the data so a significance test (AnoVa) is done. If the significance exceeds 95% one can reasonably assume that the difference between the two means is due to a real difference in ability.
As with the correlation test above questions will fall into one of three categories;
The explanations for these results are the same as before.
Item analysis looks at the individual statements within questions. These are all true/false responses so AnoVa is used in the analysis in exactly the same way as for questions as described above. You have to be more careful about the interpretation of the results though. The marking scheme implemented on these tests means that students select statements they think are true but they may leave statements unselected not because they think they are false but because they are uncertain about them and don't want to risk marks. So item analysis of false statements is much less meaningful than item analysis of true statements. The best advice is to treat true statements in the same way as for the question as a whole but for false statements you need not be as concerned when there is no significant difference in test performance. Significant negative relationships on false statements are a cause for concern because this indicates a false statement that is attractive to the most able students.
If you work to a particular significance threshold you will sometimes identify significance wrongly. If you want to institute a particular question review policy based on the statistical values you ought to choose threshold values that take into account balance between type I and type II errors. (Type 1 = taking action which was not needed, Type 2 = not taking action when you should have.)
To help you with the appropriate interpretation of the results a message is generated for each question/item analysed. It is possible to base all your conclusions and follow up investigation on these and ignore the numerical information. Here is a full list of the messages;
Not enough data to reach conclusions.
Either too few students attempted the test or too few selected a particular
box, or too few didn't select a particular box.
The more able students performed very significantly WORSE on this question!!!
This wording is used if if the level of significance exceeds 99%
The more able students performed very significantly better on this
question.
This wording is used if if the level of significance exceeds 99%
The more able students performed significantly WORSE on this question!!!
This wording is used if if the level of significance exceeds 95% but not 99%
The more able students performed significantly better on this question.
This wording is used if if the level of significance exceeds 95% but not 99%
The more able students may have performed worse on this question.
This wording is used if if the level of significance exceeds 90% but not 95%
The more able students may have performed better on this question.
This wording is used if if the level of significance exceeds 90% but not 95%
There is no evidence that performance on this question relates to ability.
This wording is used if if the level of significance is lower than 90%