Assessment writing can be a very difficult thing. Many times we have certain expectations and presumptions about our assessments. Moreover, there seems to be a multitude of ways to create and administer an assessment. This resource is aimed to help unpack some of the pedagogical practices that revolve around two of the main question types found on the most common assessment – tests and examinations. It will also aim to discuss why and how we should use multiple choice and essay-type questions on our midterms and examinations to better capture student understand.
Strategies and Tips Regarding Writing Multiple Choice Questions
What are multiple choice questions?
Multiple choice questions are questions usually used on a test or assessment where a question is presented with usually 4 or 5 options. Test-takers are instructed to choose the “best” choice among the answers.
Why are some advantages for multiple choice questions?
Multiple choice exams have been used for many decades in both the academic and non-academic worlds, with uses ranging from standardized testing to personality questionnaires. When thinking about whether you should be using multiple choice questions as part of your assessment arsenal, many questions will arise. One main question is the advantages of multiple choice questions, which include:
- Versatility: Multiple choice exams can be designed to test a variety of levels of understanding. Furthermore, MC exams can be extremely practical, allowing for many questions to be asked on a single assessment tool, as well as reducing the amount of resources needed for grading and statistical analyses.
- Reliability: Removes scorer reliability, and allows for the possibility of multiple questions to be asked on a given topic
- Validity: There has been a lot of evidence showing the validity of MC questions. The fact that it generally takes less time to answer a MC question than a written one allows for more questions to be asked, increasing the validity of the assessment tool.
What are some disadvantages of multiple choice questions
- Students have the ability to guess. A student who never studies will be able to perform at chance due to the nature of providing options and answers.
- Constructing effective stems and lures can be difficult, which may lead students to pick up subtle differences in the answers aiding them in their ability to guess.
- Exposing students to false information (lures) can distract them and cause interference, which may not have happened if you just asked them the questions and provided them space to answer
- Multiple choice questions do not allow for individual interpretation, as well as constricts the content to the context of the question
Possible misconceptions regarding multiple choice questions
Multiple choice questions only test rote memory: While rote memory is something that multiple choice questions excel at assessing, these types of questions also have the ability to test higher order thinking. It has been demonstrated that constructing effective questions and answers, you are able to test critical thinking, problem solving, and even reduce the ability of students to guess
Multiple choice questions are easier than written answers: There is anecdotal evidence (many times from students) that multiple choice questions are easier. Some of the reasoning behind it is that instead of having to think critically about the material, as long as you can recognize the correct key terms, you will be able to select the correct answer. However, this is a misunderstanding of multiple choice questions, and is generally reflected by poor multiple choice writing practice. These types of questions can be written to make them more difficult and to test the same amount of knowledge as if they were essay questions.
Strategies on how to construct effective multiple choice questions
Start with your own learning outcomes: When we construct any type of assessment, we always have to think about what our learning objectives are. Different assessment tools are better at getting at different types of learning. By understanding what you want your students to demonstrate, you are better able to pick the most appropriate assessment tool for the job.
Think about how you want to construct the question (the stem): Stems are essentially the questions that students are given. Stems can be written in a variety of different ways, and these differences can produce intentional or sometimes unintentional outcomes. Some questions that might help with creating a statistically reliable and valid stem include:
- What knowledge/skill do I want my students to demonstrate by answering this question?
- Does my stem make sense? Will the language confuse students?
- What ‘level of processing’ do I want to address? Some questions may be geared to measure rote memory, while other questions might be aimed at measuring the students’ ability to synthesize or deconstruct an idea/argument.
By thinking about the purpose of each stem, you will be able to be more mindful of how you construct your questions. Some other strategies that might be helpful as you think about your stem are:
- Is the stem meaningful and does it propose a problem
- Is there irrelevant information? By removing as much irrelevant information, you reduce the amount of unnecessary cognitive load, which will increase the validity of the question.
- Does the language produce a barrier for English Language/Multi Language learners? Are there words that are not important that might be a stumbling block to non-native English speakers?
- Are you using negative statements? Negative statements can create unnecessary cognitive load (especially for English Language Learners).
- Negative statements should be avoided UNLESS the outcome requires it (for example: identification of hazardous materials might benefit from negative phrasing)
Next think about your alternative answers (lures): Writing good stems is only half the battle when constructing effective multiple choice questions. The other half is reliant on writing effective alternative answers (or lures). One concern that many people have regarding multiple choice assessments are that there is a factor of guessing. By constructing effective lures, you minimize any advantage some students might have by just being “good test-takers”. Some strategies that might help in constructing effective lures are:
- Alternatives should all be plausible. Increasing the plausibility of the lures decreases the chance of getting the correct answer based on inferring information through context.
- Alternatives should be as homogenous in content as possible. This is very similar to the point above, which is that creating homogenous content reduces the ability to infer any correct answers based on question-content/structure and not course content.
- Alternatives should be stated clearly and in similar grammatical and syntactical styling. Having similar grammar reduces unnecessary cognitive load and reduces any advantages that students might get from inferring correct answers based on grammar and style.
- Alternatives should be mutually exclusive. Ensuring that alternatives are mutually exclusive reduces any confusion regarding which might be the correct answer.
- Avoid unnecessary complexity
Ask yourself what level of thinking you want to assess: As mentioned above, multiple choice questions have the capabilities of measuring not only rote memory, but also application and critical thinking. Rote memory multiple choice questions are usually the easiest to write, but sometimes thinking about how we can test higher level thinking with multiple choice can be difficult. Some strategies that might be helpful in constructing more higher level thinking questions are:
- Is the information provided in their notes? If so, this might be a more rote memory type of question
- Can the question be asked in a new way or in a different context? This might help with measuring students’ ability to apply their understanding of the material
- Does the question require the synthesis or comparison of multiple pieces of information? If so, this usually allows for the assessment of critical thinking or problem solving skills.
What about “all of the above” and “none of the above”
Researchers over the years have investigated the role of answers like “all of the above” and “none of the above”. Most agree that “none of the above” questions increase cognitive load (Tollefson, 1987), and don’t actually assess what the students know, but only test that they know the options are NOT correct. Recently, this research has also extended to “all of the above”, showing that only under very limited circumstances do answers like “all of the above” produce strong assessment validity (Paneerselvam & Callendar, 2016).
How do we test higher order thinking with multiple choice questions
One of the main criticisms of multiple choice questions it that they tend to be perceived as shallower compared to short/long answers or even other forms of assessments (like essays or labs). However, researchers over the years have suggested that, through thoughtful construction, multiple choice exams can test higher order cognitive functions (Considine, Botti & Thomas, 2005; Morison & Walsh, 2004). Below are some ways of constructing multiple choice questions that test these higher levels of thinking:
- Creating questions that require more than one piece of knowledge
- Creating questions that require the integration across facts/ideas/lectures
- Creating questions that require application of an idea to a new or novel context
- Creating answers/lures that are not taken directly from text
- Creating answers/lures that require a high level of discrimination
Strategies and Tips Regarding Essay Questions on an Examination
What is an essay question?
When we think of an essay, we tend to think of a multiple-page written response to some question, prompt or scenario. However, essay-type questions can be brought into exam situations and can be quite beneficial in assessing student understanding. Regarding the definition of an essay question, one that has been helpful in deconstructing some of the core components of this type of questioning comes from John Stalnaker (1951). In his book, he defines an essay question to be
“A test item which requires a response composed by the examinee, usually in the form of one or more sentences, of a nature that no single response or pattern of responses can be listed as correct, and the accuracy and quality of which can be judged subjectively only by one skilled or informed in the subject.”
Based on this definition, there are a few criteria that inform us on the characteristics of an essay question.
- Something in which the examinee (or student) has to compose or create, rather than select from a list of options
- It inherently includes some form of writing, usually in the form of multiple sentences
- Answers are not bound by a single response or pattern of responses
- It can only be evaluated or judged by someone informed or skilled in the subject
Maybe input an example of what an essay question might look like under these different criteria.
Why are some advantages for essay type questions?
Essay questions bring a lot to the table when thinking about the purpose of your assessment tools. There are many different advantages to why we should use essay type questions on a midterm or a final exam. However, keep in mind that these advantages are context specific, and are presented to help you discern whether these will aid you in the assessment of your learning outcomes
- Essay questions can help assess higher-order thinking and critical thinking skills
- Essay questions can also assess in-the-moment writing skills
- Essay questions can evaluate student problem solving and reasoning skills
By understanding the advantages of an essay type question on a midterm, test or final exam, educators are then able to better assess whether these types of questions are useful for their purposes and learning objectives.
What are some disadvantages towards essay type questions
- Can assess limited sample of the range of content.
- Can be difficult to grade and be administratively more costly to implement
- Can lead to bad writing habits, due to the lack of feedback and editing processes
Possible misconceptions regarding essay type questions
Just because you are providing students space to write more does not automatically mean that higher order thinking is being tested and evaluated. Poorly written essay questions can test rote memory, and can lend itself to a more factually-based answer. At the same time, well-crafted multiple choice questions can test higher order thinking (as described above). Therefore, it is important to be cautious when assuming essay questions automatically test higher order thinking.
There is a large concern when constructing question types that provide answers to students (such as multiple choice, true/false, or matching) where students can come to the correct answer based on luck or guessing. There is also an assumption that when students are writing an essay question, guessing has been eliminated, which is a misnomer. Essay questions can lend itself to a new form of guessing, or what researchers call “Bluffing” (Thelin & Scott, 1928). Bluffing is when students provide vague or generalized content in order to pad or add credibility to their writing. Many times, when grading large amounts of essay questions and when time is a factor in grading, it can seem that students who write a lot have a deeper understanding of content. However, succinct writing can be just as, if not more, effective in answer an essay prompt. Therefore, instructors need to be aware not only when constructing essay questions, but also during the grading of the potential of bluffing.
Strategies on how to construct effective essay questions
As stated before, it is important to always think about the intentions and learning objectives of your assessment tool. Are you hoping to assess a baseline of knowledge, or is this a tool to be used to assess critical thinking and problem solving skills. Writing out explicit, clear and concise learning objectives will aid in how you create your essay questions. Understanding what you want your students to express will help you work backwards in creating your essay question.
Many times when students are given an essay question in an examination, it can feel extremely overwhelming. If we provide too vague of an essay question, students will feel the desire to write everything they know about the topic, many times to the detriment of actually answering the question. Therefore, providing clear boundaries and limitations (in your question and scenario) will help students focus their answer. Instead of having a need to write everything down, they will be able to formulate a clear response to your question if your question is clearly stated.
Not all verbs are the same! Be mindful of what you want students to do. If you want students to provide an explanation, verbs like describe or explain might be useful. If you are hoping for students to provide their own interpretations with supportive evidence, verbs like analyze or evaluate.
This is similar to the previous idea of writing out your learning objective but takes it one step further. Since you have already written out your learning objective, use the same language for the essay question. If you are hoping for students to demonstrate their own opinions on a certain topic, feel free to use that language “demonstrating personal opinion” so students know what your intended learning outcomes are.
Essay questions are not essays. Essays go through a process of drafting, editing and revision. Essay questions on a test very rarely get edited, and generally do not start from a place of an outline or skeleton. Understand that your students are under considerable time pressure and many of them will start writing immediately even before drafting an outline. Therefore, be mindful of the time pressures of the writing and adjust your grading accordingly. Be clear as to your expectations on the quality of writing, the use of bullet points and short form, and other ways students can save time on the actual writing. Also be mindful of how many essay questions you can effectively put on an examination. Even if each question could be answered in 30 minutes, the more you add, the slower students will be to answer subsequent questions due to mental and physical fatigue.
One possible shortcoming of essay type questions is that they can promote bad writing habits. This is mainly due to the fact that students, very rarely, concern themselves with strong writing skills when completing an essay question on an examination. Therefore, it is up to the instructor to ensure that writing quality does not degrade during essay-type questions on a test. Integrating writing support during the course is not enough, for writing an essay or an assignment might be quite different than writing an essay question in a short period of time. Some strategies that might be helpful would be to encourage students to read the feedback on their essay questions. Furthermore, options can be provided for students to re-submit their essay questions (without knowing the answer key) after getting their marks back for a small, almost insignificant bonus in order to promote the editing and reflective process that is more evident in essay writing. Moreover, providing students with practice on how to effectively write essay questions, and even allocating grades to writing style and effectiveness on the actual essay question itself will help encourage students to not neglect one large aspect of these types of questions. One of the best ways to promote strong writing is to provide quality writing and have students deconstruct the piece of work. Giving the opportunities to be reflective and to try to take something as subjective as writing and create some objective guiding principles will aid in student writing development.
One of the biggest fears of students is misunderstanding the question, which leads to providing an answer that does not represent their true understanding of the content. This becomes exaggerated with English language/Multi-language learners. When constructing your question, do your best to remove any unnecessary fillers and keep the prompt as succinct as possible. Very similarly to writing a good stem for multiple choice questions, any jargon or unnecessary information should be removed to reduce the amount of extraneous cognitive load and possible points of confusion.
Students should be aware of the criteria in which they are being graded. Now, unlike essays, many times students are not given the opportunity to review the grading rubric or criteria ahead of time. Therefore, it might be beneficial for students to know ahead of time how they might be graded (on general themes such as writing style, clarity, having a strong thesis statement, etc). Furthermore, on the actual essay question itself, it may be wise to provide a breakdown of what is expected and how the grades will be allocated. If you do provide a short breakdown of the grading scheme, keep it short and succinct as many students will just glance over it due to the time pressures of the test.
What about providing the questions ahead of time
Many educators believe that one way of dealing with the time crunch is to provide students with essay questions/topics ahead of time prior to the actual examination. Furthermore, some instructors will allow students to bring in “cheat sheets”, or work that has been written prior to aid them in the creation of their written answers. However, there are many pedagogical and administrative issues that come with providing questions ahead of time.
Administratively, instructors have to ask themselves, if they are providing questions ahead of time, why they have chosen to use these essay questions on an assessment tool like a midterm or test, rather than an essay assignment. Having more time, and the opportunities to edit, re-evaluate, and even resubmit a piece of writing will be more beneficial to students’ writing abilities than a one-off essay question on a test. Furthermore, providing students with more time to write the piece of work will generally lend itself to a higher caliber of work that reflects the students’ full capabilities. Therefore, educators need to ask how their learning objectives are better accomplished with this version of an essay question on a test rather than an essay assignment.
Pedagogically, it may seem beneficial to provide students with the essay questions ahead of time. Moreover, many instructors will provide a larger sample of essay questions, and only chose a subsection of them on the midterm. This may seem useful in helping students from just memorizing a script, while still assessing their mastery of the content and their critical thinking skills. However, the research has shown that by providing students with choices (specifically in the context of essay-type questions), assessments can become less valid and less robust in measuring the overall ability of your class. Some of these reasons include the fact that some of your essay questions might be easier than others, therefore students who chose to write a “harder” question may be at a slight disadvantage. Furthermore, students may waste time choosing a topic to write on (even if the questions are given ahead of time), which provides additional cognitive strain on the student that affects the sensitivity of your assessment tool. Lastly, if students are given the option on the actual test, the assessment may suffer in its reliability to capture the performance of your class. If students answer different questions, then it becomes difficult to assess if all students are equally knowledgeable about the topics covered on the test.
Final Thought
One thing that that always arises when creating and administering assessments is to always re-evaluate the data. Many times we may follow these guidelines and create an assessment that still does not work the way we hope, and that is OKAY! The reality of education is that many times it is a learning process, and assessments that are tweaked and reflected upon both before and after administration are the ones that develop the most. Therefore, after administrating the assessment, it is always effective to get feedback. Ask the students how they felt about the assessment; ask them their perceptions PRIOR to receiving any marks, which may lead to emotional biases. Moreover, spend time reflecting on the assessment after it has been administered. Many times we gain clarity only after we go through the event. Here are a few questions to ask yourself to help in this reflective process
- What was the overall impression of the assessment?
- What were the most common questions asked during the administration of the assessment
- Did students finish on time?
- What was the general mood of students as they left the assessment?
By reflecting on your feedback, you are able to refine and adapt your assessment over time to create a better version of it. Just remember, assessment design is a process. Many times it takes a few iterations before we start feeling confident in our assessment design.
References and Resources
Brame, C., (2013) Writing good multiple choice test questions.
Butler, A. C., Karpicke, J. D., & Roediger III, H. L. (2007). The effect of type and timing of feedback on learning from multiple-choice tests. Journal of Experimental Psychology: Applied, 13(4), 273.
Conklin, J., Anderson, L. W., Krathwohl, D., Airasian, P., Cruikshank, K. A., Mayer, R. E., … & Wittrock, M. C. (2005). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives Complete Edition.
Considine, J., Botti, M., & Thomas, S. (2005). Design, format, validity and reliability of multiple choice questions for use in nursing research and education. Collegian, 12(1), 19-24.
Gay, L. R. (1980). The comparative effects of multiple‐choice versus short‐answer tests on retention. Journal of Educational Measurement, 17(1), 45-50.
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied measurement in education, 15(3), 309-333.
Marsh, E. J., Roediger, H. L., Bjork, R. A., & Bjork, E. L. (2007). The memorial consequences of multiple-choice testing. Psychonomic Bulletin & Review, 14(2), 194-199.
McCoubrie, P. (2004). Improving the fairness of multiple-choice questions: a literature review. Medical teacher, 26(8), 709-712.
Morrison, S., & Free, K. W. (2001). Writing multiple-choice test items that promote and measure critical thinking. Journal of Nursing Education, 40(1), 17-24.
Paneerselvam, B. & Callender, A. The benefits of All-of-the-Above question on multiple choice test. Poster presented at the 57th annual meeting of the Psychonomics Society (2016).
Reiner, C. M., Bothell, T. W., Sudweeks, R. R., & Wood, B. (2002). Preparing Effective Essay Questions.
Roediger III, H. L., & Marsh, E. J. (2005). The positive and negative consequences of multiple-choice testing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31(5), 1155.
Scouller, K. M., & Prosser, M. (1994). Students’ experiences in studying for multiple choice question examinations. Studies in Higher Education, 19(3), 267-279.
Stalnaker, J. M. (1951). The essay type of examination. Educational measurement, 1, 495-530.
Thelin, E., & Scott, P. C. (1928). An investigation of bluffing. The American Journal of Psychology, 613-619.
Tollefson, N. (1987). A comparison of the item difficulty and item discrimination of multiple-choice items using the “none of the above” and one correct response options. Educational and psychological measurement, 47(2), 377-383.
Zeidner, M. (1987). Essay versus multiple-choice type classroom exams: The student’s perspective. The Journal of Educational Research, 80(6), 352-358.