Questionmark Perception

Questionmark Perception

Oct 30 2020 |
Logged in as : anon
Change font size


The purpose of this example test is to provide you with an overview of the different types of objective style questions supported by an online system. Each example question includes a description of the question, some considerations about question design and feedback options. 

Question types included:  

Multiple Choice
Multiple Response
Extended matching
Assertion reason question
Language testing (questions with multimedia content)
Confidence-based testing


Roberta Perli, e-assessment support
Education Support Unit, University of Bristol

e-assessment workshop


1  of 17

Question type: Multiple Choice (MCQ)
Description: Multiple choice questions require the participant to select one or more options from a list of possible answers. They are believed to offer a reliable way of testing student knowledge.
Example: The example below shows a fairly common type of MCQ, with an image and six options. We may observe that in this example what is being tested is whether the student can recognise the name of the instrument from a  list, rather than whether s/he can recall it unprompted.
* Easily assesses knowledge of content
* Skilled test writers can design to test higher order thinking
* Rapid scoring so quick feedback
* Can help diagnose problem areas
Generally not as good at testing deeper thinking
* Don’t allow demonstration of knowledge beyond the options offered
* Don’t determine if student can create/formulate acceptable answer on own
* Guessing is encouraged; can be highly rewarded if distracters are poorly written. 
* MCQs need to be treated with great care; it must be clear and unambiguous what answer is required and the range of possible answers must be very limited. For this reason this question is well suited to use with images, for names, definitions etc.

Question: What is the name of this surgical instrument?


2  of 17

Question type: True/False
Description: True/False questions are quite popular because they are generally easy to write; one does not have to think of lots of plausible but incorrect answers as with an MCQ.
* Better than multiple choice because you can cover more material more quickly.  (Students can do three T-F in same time it takes to do two M-C)
Disadvantages (versus multiple choice)
* Students have 50% chance of guessing right
* Hard to write unambiguous statements that are definitively true or false
* Students may recognize a false statement w/out understanding concept*
* However one has to be a little careful with them: a true/false question should contain only a single statement and be one that is either clearly true or false.
It is also arguable that they are unsuitable for summative assessment because the student would score 50% simply by answering at random.

Question: Hypoadrenocorticalism is also known as Cushing's disease


3  of 17

Question type: Multiple Response (MRQ)
Description: A standard type of Multiple Response, or Multiple Answer question looks like an MCQ except that the student can choose more than one answer.  Multiple Answer questions are often used to elicit from the student all the items that match a particular definition or that conform to a particular condition. Multiple response questions can be answered in different ways and are more difficult to author compared to MCQs.
Considerations: When writing MRQs it is important to consider the following: 
- Including the number of correct responses in the instructions. 
- Deciding on the scoring system that best reflects students' performance. MRQ can be partially correct/incorrect, or all correct/incorrect, can also be negatively marked, scored by giving +/- (n) of marks for correct/incorrect answer or by giving a total (n) of marks for a number of correct answers. 
Example: In this example the student is asked to identify the two countries that joined the EU in 2007. The scoring has been set to give +1 mark for each correct answer and -1 for each incorrect answer. The total score is +2.

Question: Which countries joined the EU in January 2007?


4  of 17

Question type: True, False, Don't know (Matrix)
Description: A matrix question looks rather like a Multiple Answer question, but has a crucial difference. With a Multiple Answer question the student selects the items s/he thinks are correct.  However if an item is left blank, it is ambiguous whether this means the student thinks it is wrong, or if s/he is not sure. 
In a TFD question (or a Matrix) each item requires a definite response, avoiding this ambiguity. This also enables TFD questions to be used in conjunction with negative marking, where if the student selects True or False s/he receives 1 mark if correct and -1 if incorrect. If they respond Don’t Know they receive 0 marks. A different scoring system could allocate +2 to the correct answer, -1 to the incorrect answer and 0 marks for Don't know.
Considerations: Another way of looking at this type of question is that it is a series of T/F questions. By using negative marking one eliminates the problem of students being able to get 50% by guessing the answer. Questions of this type are commonly used in Medicine and related subjects.

Question: When blood passes through systemic capillaries

the pH increases
bicarbonate ions pass from the red cells to the plasma
the concentration of chloride ions in the red cells falls
its oxygen dissociation curve shifts to the right
the velocity of blood flow is less then the aorta


5  of 17

Question type: Extended Matching 
: Extended Matching questions are consist of four key elements: a theme, a lead-in, a set of options, and a series of stems (or scenario/ vignettes) that the students have to match to the given options. 
* a theme (the topic addressed by the set EMQs)
*an option list (between 8 and 20) 
* a lead-in statement which contains instructions
* a series of stems ( for example 3 to 6 scenarios vignettes e.g. patient's symptoms, the results of lab tests) 
Transform questions into small problems that students have to resolve applying their knowledge (e.g assess, draw conclusions) rather than recall isolated pieces of information like in MCQs. 
*This can prevent students answering by elimination
*Stems can be listed in order of difficulty to challenge the best students with the last stems.
writing EMQ can take up time
* students who are used to being tested with MCQs may need time to familiarise themselves with EMQs
Advice on Writing Matching Questions
* Lead-in statements indicate the relationship between the stems and the options and can be used to provide students with instructions on how to answer the questions.
*Write clear statements
*Avoid to omit lead-in statement as this could result in an ambiguous task.
*Every option in answer group should be a plausible answer. Options may be a single word or a short phrase and should be listed in alphabetical order.
*Allow each option to be used more than once or not at all.

Theme: Early preganncy

Lead -in For each of the following patients select the most likely diagnosis


6  of 17

Question type: Matching question (Labelling)
: A matching question essentially presents two lists of items and requires the student to match one item from the first list to an item from the second. Matching questions are very versatile and can be used to present a broad range of exercises. They can also use three different ways of interacting with the computer. The example below (drag and drop)  requires the diagrams (images) to be matched to the box corrsponnding to the appropriate stage of rotation, by dragging and dropping the images in the boxes. 
Matching Items - Another form of multiple choice
* Good means to assess content knowledge; can move to application by asking students to match concepts to 
   examples of them, causes with likely effects, etc
* Easier to write than multiple choice since distracters are unnecessary
* More efficient than multiple choice because students need to read only one set of options to answer several items
* May be difficult to come up with plausible, differentiated options
* Absence of distracters makes it harder to know where students are confused
* More potential for correct guesses than in multiple choice
Advice on Writing Matching Questions
* Every option in answer group should be a plausible answer to every item or question
* Limit the number of matching items to 10-15
* Allow each option to be used more than once or not at all.  Perfect matches aid test savvy students and hurt those who misunderstand one item
* Give clear directions.  Explain how the two columns are related, and that some options may be used more than once or not at all.
* Make it easy for students who understand to find the right answer by limiting answers to single words or short phrases.  Limit longer statements to the “questions.”

Question: The figure below is an illustration of a cross-section section through an ampulla in the semi-circular canal.  

This vestibular structure is responsible for detecting angular (rotational) acceleration.  Within the ampulla is a gelatinous mass, ‘the cupula’, which contains hair-like processes.   This is attached at the base, but ‘floats’ in endolymph fluid, which is said to have ‘inertia’ (ie. it resists changes in motion).  Spinning will generate angular/rotational acceleration, which will result in the apparent direction of endolymph flow going in the opposite direction, displacing the cupula and causing action potentials being fired.

Note: It takes approximately 15-30 seconds for the endolymph fluid to ‘catch-up’ with the semi-circular canal’s rate of movement.  


Given this information, imagine that a subject spins around in the same direction for 15-30 seconds before coming to an abrupt halt.  Look at the four diagrams below.  Each one represents the state of the cupula relative to the ampulla at a different stage during this rotation.  The red arrow indicates the direction in which the semi-circular canal is rotating; the blue arrow indicates the 'relative' direction of the endolymph fluid.

Click on each of these diagrams and ‘drag and drop’ them into the box corresponding to the appropriate stage of rotation. 


7  of 17

Qyestion type: Matching question (matrix)
One potential problem with matching questions is that they can be solved by a process of elimination, where you only need to know a couple of the answers to be able to narrow down the remaining options.
One way of preventing this is to set up the question so that the number of items on each side of the question is asymmetrical. 

Question: Match the country with its capital city



8  of 17

Question type: Interactive (Hotspot)
A hotspot question is a question where the student has to select the correct area of an image. This can be by dragging a pointer to the correct place, or by clicking the mouse over the correct area. Some hotspot questions give no indication of where the hotspot might be. In this sense it falls into the free text category because it requires a completely free response rather than selecting from a list of options

Question: Drag and drop the cross (bottom of the page) on top of that part of the nephron where the primary filtration occurs.


9  of 17

Question type: Multiple choice with audio for listening comprehension
: Language objective testing is widely used by language units to assess specific language competence such as grammar or vocabulary but also reading and listening comprehension.
The advantage of this type of exercise is that students can practice specific listening tasks outside class in an independent, interactive and flexible way. Depending on the length of the passage, or what is being tested, the audio/video can be embedded in a single question, at the beginning of the test or open in a separate window if the link takes the students to an external URL.  
Consideration: If the question type requires the students to select from a list of different options, like a multiple choice or a matching, some careful consideration should be given to the wording of the options, especially if they are presented in the target language. The key and the distracters should be clear and easy to understand otherwise students may spend too much time reading the question rather than concentrating on the passage, turning the exercise into a reading rather than a listening practice.  

Question: Ecoutez le passage audio et choisissez la phrase qui convient.


10  of 17

Question type: Select a blank (Cloze test)
A cloze test involves a piece of text, from which a number of words have been removed. The student is required to insert the missing words. In some cases the missing words are supplied as a single list, from which the student has to select. In this case one is effectively matching a word with the space where it belongs – thus it is a matching question.
Considerations: A problem with such questions is that they are hard to design in a way that does not give away the answers. More common, and a lot more difficult, is a multiple choice cloze test, where each gap has its own separate list of words to choose from, as in this example

Question: Definition of preterm delivery.

The following sentence might be from any standard textbook. It gives the definition of preterm delivery.  However some words were lost!  Fill in the missing words.

Preterm labour and delivery occurs before weeks gestation. It occurs afterweeks gestation, and prior to this, such a delivery is equivalent to a. Some % of all deliveries are preterm. Prematurity accounts for% of perinatal mortality. Much of the mortality from prematurity is due to .


11  of 17

Question type: Numeric
: A numerical entry question is similar to a free text question except that it requires a number rather than a word, and does not suffer from the same risks of interpretation and misspellings associated with the text entry question. A further advantage is that some systems allow a proportion of marks to be given for any response entered within a set numerical range

Question: The resting heart rate of a Daphnia is recorded as 242 beats per minute.  When placed in a solution of 2mg/ml solution of ‘drug X’, the observer counts 54 beats in 10 seconds.  What is the new heart rate as a percentage of the resting heart rate?  Give your answer to the nearest whole number.

(Do NOT type in the % symbol.  Type in the NUMBER ONLY)


12  of 17

Does CAA only assess lower level of learning?

Where MCQs are used in CAA they are commonly perceived as only testing surface learning.
This is not necessarily the case. Careful design of questions can enable the testing of higher order skills.
The following questions are examples of how objective testing is used to test  more than knowledge or comprehension or specific practicle skills as in language testing

Question type: Matrix (question style that can be used to address higher levels) 

Description: The following example has been authored by Dr Phil Langton (Physiology). Dr Langton believes this is a question which tests students’ knowledge at a high level and is worth 2 essay questions:
* the columns of numbers are never presented in this way, so the question cannot be answered by memory alone
* the question requires students to bring together knowledge from two parts of the course, the cardiovascular and respiratory systems, and show understanding of the link between these two systems
* this particular question does rely on students having some memory of (or ability to work out) the ‘normal’ values for the quantities in the table. In each of the non-normal conditions one would expect to see changes from the normal state. None of them can be answered by looking for only one change – they all result in multiple changes.
* the question requires the student to bring together their knowledge of the normal values and their understanding of the cardiovascular and respiratory system to interpret the data. They must use this to work out what is happening on in the systems and then match that with the external conditions that would bring this about.


Question: Select the set of values (A,B,C, etc) which corresponds best to a 70 Kg man:


13  of 17

Question type: Matching (Pull down).
The following example has been authored by A.Zhok, Italian Department. The question is used in a Y4 grammar test and is specifically designed to test language interference ('false friends') in the use of some adverbs because they look similar in the two languages but differ in meaning.
Students are asked to match the Italian adverbs (in bold) to their equivalent in English listed in the the drop down menu. In order to answer this question students need to recall the meaning of the adverb, determine how it is being used in context and then match them to the correct translation in English.
Considerations: In 'asynchronous matching questions' the number of choices and options are not the same, this is to prevent students answering the question by a process of elimination. This question type offers several scoring options, including negative marking, e.g. 1 point can be given for each ocrrect answer for a number of correct answers.

Question: Scegli tra le opzioni disponibili la traduzione più adatta per gli avverbi italiani in grassetto.


14  of 17

Question type: Assertion Reason (ARQ)
Description: Like traditional MCQs, ARQs present students with a number of possible solutions. In contrast to traditional MCQs, however, ARQs also include a true/false element (CAA Centre, 2000).
Specifically, each item consists of two statements, an assertion and a reason, that are linked by the word ‘because’. The student then selects from a multiple-choice legend after proceeding through a number of steps. First, he or she must determine whether the ‘assertion’ is true or false, and then whether the ‘reason’ is true or false. If one, or both, of the statements is deemed false, then the answer will be either (c), (d), or (e) accordingly. If, on the other hand, both statements are deemed true, a third step is required whereby the respondent must determine whether the second statement provides an accurate explanation for the first.
According to Williams (2006) “traditional MCQs usually test only one issue/concept. ARQs, on the other hand, test two per question (the assertion and the reason statements) plus the validity of the ‘because’ statement in the event assertion and reason are both correct statements. On the basis that judging the correctness of two statements must be harder than judging the correctness of one, it would follow that ARQs present more of an intellectual challenge than traditional MCQs.
Considerations: Bull and McKenna (2004) make several recommendations for writing ARQs: “Assertion-reason tests can be used to explore cause and effect and identify relationships. When writing assertion-reason questions, keep in mind the following points: 

  • The reason should be a free standing sentence so that it can be considered separately from the assertion.
  • Avoid using minor reasons. These can result in an ambiguous question.
  • Repeat options A-E in full for each question.
  • Use all five options as key equally


Questions: Below is the Hjulstrom (1935) curve showing the relationship between particle diameter and critical entrainment velocities for water.

Each statement below consists of an assertion and a reason. Indicate your answer from the alternatives listed below by choosing the appropriate letter.

  Assertion Reason Link
A True True Reason is correct explanation
B True True Reason is NOT a correct explanation
C True False -
D False True -
E False False -


  Assertion   Reason
Pair 1 Sand has a lower critical entrainment velocity than clay BECAUSE Sand particles are lighter than clay particles
Pair 2 Clay has a higher critical entrainment velocity than sand BECAUSE Clay particles are held together by cohesive forces
Pair 3 Sediment transport velocities are higher than the sediment entrainment velocities BECAUSE It takes less energy to entrain particles than to maintain them in transport



15  of 17

Question type: Confidence-based (marking)
 Confidence-based questions are used in assessments in which a student's rating of his/her confidence in an answer is taking into account in the marking of the answer. (Tony Gardner-Medwin). The marking scheme applied in the following example is: 3 points for correct answer and 0,-2,-6 for wrong answers according to confidence level.
Consideration: Confidence-based marking is not meant to reward or to discourage self-confidence but to encourage reflection, self awareness and the expression of appropriate level of confidence. Because the scoring system is a real crucial feature in confidence-based assessment it's very important that students receive detailed information about the marking system and understand that guessing answers or answers based on misconception can be heavily penalised but that admitting to have low confidence can actually be rewarded. "One of the major learning issues in use of confidence-based marking is the realisation that you can be rewarded for aknowledging and communicating low confidence. Correct and honest expression of confidence is a valued communication skill in any area. (Formative and summative confidence-based assessment, Tony Gardner-Medwin, 2003).

Question: Choose the correct item from the first menu, and indicate how certain you are of your answer using the 2nd menu.


16  of 17

Question type: Gap fill
: Completion/fill-in-the-blank - Students are required to enter one word (Usually but could also be two words) in the text box provided. To be true objective question, must have only one right answer (i.e., anyone with an answer key can score correctly).  Short answer questions usually are more subjective.
* Can assess understanding of key content
* Requires recall rather than simply recognition
* May work better than multiple choice when correct answer would be obvious
* Provide little information on areas of confusion
* Tend to be very specific to the point of pickiness
* Can be more difficult to score 
* all possible answer need to be anticipated and entered into the system

: Marco, avendo bisogno di alcuni libri, ha pensato di chieder
a Paolo, un suo amico. Paolo ha prestati volentieri, ma qualche settimana dopo i due amici si sono visti a casa di Paolo:
-“Ciao, Marco, allora
hai riportato i libri che avevo prestato?”
ho riportati, ecco.”
hai riportati tutti?”
ho portati solo due.”
-“Ma io
avevo prestati tre!”
-“È vero, ma l’altro
ho perso!”


17  of 17

Question type: Interactive (Hotspot)
A hotspot question is a question where the student has to select the correct area of an image. This can be by dragging a pointer to the correct place, or by clicking the mouse over the correct area. Some hotspot questions give no indication of where the hotspot might be. In this sense it falls into the free text category because it requires a completely free response rather than selecting from a list of options

Question: If the gradient of a line on a graph is given by dy/dx, which of the diagrams labelled A to D would provide the best estimate of conduction velocity (CV; the velocity of the action potential in the ulnar nerve)? 

[INSTRUCTIONS: Using the mouse, click and hold on one of the yellow pointers labelled 'Cv=gradient' or 'Cv=1/gradient', and drag the pointer over the graph (A-D) that you wish to choose and unclick]