Royal Army College

Lesson Overview

Lesson 03 surveyed the methods of assessment, written, oral, and practical, and the rule that the method must truly measure the outcome. This lesson and the next take two of those methods in depth. Here we deal with the assessment of knowledge, what a candidate understands and can recall and reason with, through written and oral tests; the next lesson deals with the assessment of practical skills. Knowledge is assessed differently from a skill because it lives in the head, and the assessor's task is to draw it out and judge it fairly through questions, written or spoken. Done well, a knowledge test measures real understanding against the standard; done badly, it measures the wrong thing, the candidate's guessing, their handwriting, the assessor's mood, and the qualification behind it is hollowed out.

The whole of this lesson serves the four principles of Lesson 02. A knowledge test must be valid, measuring the understanding the outcome names and not something else; reliable, giving the same result whoever marks it and whenever it is taken; fair, giving every candidate the same clear chance; and transparent, testing against a known standard. Most of the faults in written and oral testing are failures of one of these: a question that measures reading speed rather than knowledge fails validity; a multiple-choice question whose answer is given away by being the longest fails validity too; an essay marked by feel fails reliability. So the craft taught here is the craft of writing and running knowledge tests that hold those four principles, which is the craft of testing what the candidate actually knows, cleanly.

This is the knowledge layer. It teaches you the kinds of written and oral test, how to write good questions of each kind, and how to run them fairly, so that you can build a knowledge assessment that measures the right thing. The judgement of marking a borderline answer, of probing an oral candidate without leading them, is built by assessing real candidates under a qualified assessor and signed off in person. Read this to know how knowledge is tested; learn to test it by testing.

By the end you will be able to choose the right kind of written or oral test for an outcome, write good multiple-choice and open written questions, run an oral assessment fairly and consistently, and recognise and avoid the common faults that make a knowledge test measure the wrong thing.

Key Terms

Knowledge assessment: the judging of what a candidate understands, recalls, and can reason with, as opposed to what they can physically do (a skill, assessed in Lesson 07).
Written test: an assessment in which the candidate answers in writing, by selecting or constructing answers; includes objective and open forms.
Objective test: a written test whose questions have a single predetermined correct answer, such as multiple-choice, so marking is consistent whoever marks it.
Multiple-choice question (MCQ): an objective question giving a stem and several options, of which one is correct and the rest are plausible wrong answers (distractors).
Distractor: a wrong option in a multiple-choice question, which must be plausible to a candidate who does not know the answer, drawn from a real misconception.
Open (constructed-response) question: a written question the candidate answers in their own words, from a short answer to an extended essay, marked against a scheme.
Oral assessment: an assessment conducted by spoken question and answer, allowing the assessor to probe and follow up, judged against a standard.
Marking scheme: the prepared statement of what a correct or creditworthy answer contains, used so that answers are marked consistently and against the standard (Lesson 08).
Validity: the principle that an assessment measures the outcome it claims to, and not something else, such as reading speed or guessing.
Leading question: a question that gives away or steers toward its own answer, which destroys the validity of an oral assessment.

Choosing the kind of test

Knowledge can be tested in writing or by mouth, and within writing by objective questions or open ones, and the first task is to choose the kind that truly measures the outcome. The choice follows the same validity rule as everything in assessment: pick the form that measures the understanding the outcome names, not the form that is easiest to mark or quickest to set.

Objective written tests (chiefly multiple-choice) are best for testing breadth of knowledge and understanding across many points, quickly and with perfectly consistent marking. Because each question has one predetermined correct answer, an objective test is marked identically by anyone, even by a key, which makes it highly reliable and removes marker bias entirely. Its limits are real: it tests recognition and understanding well but cannot test a candidate's ability to construct, explain, or argue in their own words, and a poorly written objective question can be guessed or gamed. Use objective tests to cover a wide field of knowledge reliably.

Open written tests (short answer to extended essay) are best for testing whether a candidate can explain, reason, or construct an answer in their own words. They reach understanding an objective test cannot, the candidate's ability to put a thing together and express it, but they are slower to mark and harder to mark reliably, because judgement enters, which is why they need a careful marking scheme (Lesson 08). Use open questions when the outcome is about explaining or reasoning, not mere recall.

Oral tests are best when the assessor needs to probe and follow up, to ask "why?" and "what if?", to test depth and check that an answer is understood rather than parroted. They are powerful for judging real understanding but the weakest on reliability and record, because they vary with the assessor and leave no written trace unless one is made. Use oral assessment where probing understanding matters, with the safeguards below.

   CHOOSING A KNOWLEDGE TEST  (match to the outcome)

   OBJECTIVE / MCQ    breadth of knowledge, marked perfectly
                      consistently; can't test "in own words";
                      guessable if badly written
   OPEN WRITTEN       explaining, reasoning, constructing in own
                      words; slower and harder to mark reliably
                      (needs a marking scheme)
   ORAL               probing depth, follow-up, "why?/what if?";
                      powerful but weakest on reliability and record

   The rule (validity): choose the form that measures the OUTCOME,
   not the one easiest to set or mark.

Writing good multiple-choice questions

The multiple-choice question is the most useful objective form and the easiest to write badly, so it repays care. A good MCQ has a clear stem (the question or problem) and several options, of which exactly one is correct and the rest are distractors, plausible wrong answers. The whole art is that a candidate who knows the material can find the right answer and a candidate who does not cannot, except by genuine guessing at the stated odds. Several rules secure that, and they are the rules this College uses for its own course quizzes and exams.

One unambiguously correct answer. Exactly one option is right, and it is clearly, defensibly right; no "most correct" arguments, no second option a knowledgeable candidate could justify. If two options can be defended, the question is broken.
Plausible distractors, from real misconceptions. The wrong options must tempt a candidate who does not know, which means drawing them from the actual mistakes and misunderstandings students have, not from filler or nonsense. A distractor no one would ever choose tests nothing and narrows the real guess.
Options of similar length, detail, and form. This is the rule candidates exploit most: if the correct answer is consistently the longest, most detailed, or most carefully qualified option, a test-wise candidate picks it without knowing the material. Keep all options of similar length and grammatical form, so the answer is found by knowing, never by spotting the odd one out.
Test understanding, not trivia or trickery. A good MCQ tests whether the candidate understands and can apply the material, not whether they remember an obscure detail or can untangle a deliberately confusing sentence. Avoid trick wording, double negatives, and "all of the above" or "none of the above", which test puzzle-solving rather than knowledge.
Vary the correct option's position. Across a test, spread the correct answer evenly across the positions, so a candidate cannot pass by always choosing the same letter, and so no unconscious pattern (the answer is usually third) creeps in.

These rules are not pedantry; each closes a way for a candidate to get the right answer without the knowledge, or the wrong answer despite it, and so each protects the validity the whole assessment depends on. An MCQ that breaks them measures test-wiseness, not knowledge, and a test built of such questions certifies the wrong thing.

   A SOUND MULTIPLE-CHOICE QUESTION

   ONE correct answer           clearly, defensibly right; no "most
                                correct" second option
   PLAUSIBLE distractors        drawn from REAL misconceptions, so they
                                tempt the candidate who doesn't know
   OPTIONS of similar length    no giving the answer away by making it
     and form                   the longest / most detailed / most qualified
   TEST understanding           not trivia, not trick wording; avoid
                                double negatives, all/none-of-the-above
   VARY the correct position    so the answer-letter pattern gives nothing

   Each rule closes a way to be right WITHOUT the knowledge, or wrong
   WITH it. Together they protect validity.

Writing good open and oral questions

Open written questions are answered in the candidate's own words, and their quality lies in being clear, single, and markable. A good open question asks for one thing clearly, so the candidate knows exactly what is wanted and is not left guessing the assessor's intent, and it is pitched at the level the outcome states, asking for recall, explanation, or argument as the outcome requires and no more. Crucially, every open question is written with its marking scheme at the same time (Lesson 08): the assessor states, before any candidate answers, what a creditworthy answer contains and how marks are awarded, because a question whose marking is worked out afterward, by feel, cannot be marked reliably or fairly. If you cannot write the marking scheme, the question is not yet clear enough to ask.

Oral questions are spoken, and their great power, probing and follow-up, comes with a great danger: the leading question, which gives away or steers toward its own answer and so measures nothing. "So the principle is validity, isn't it?" tests nothing, because it hands the candidate the answer; "what principle is at work here, and why?" tests understanding. The oral assessor asks open, neutral questions, lets the candidate answer without prompting the words, and follows up to probe depth ("why?", "what if this changed?") rather than to rescue. To keep an oral assessment fair and reliable despite its spoken, unrecorded nature, it is structured: the assessor plans the core questions in advance so every candidate faces the same ground, judges each against the same standard, and makes a written record of the questions and the judgement, so the oral leaves a defensible trace like any other assessment.

Running a knowledge test fairly

Writing good questions is half the task; running the test fairly is the other half, and it applies the fairness of Lesson 04 to knowledge testing specifically. A few points are particular to written and oral knowledge tests.

Same conditions for all. Every candidate gets the same test, the same time, the same clear instructions, and the same conditions, so that the result reflects their knowledge and not who got longer or a quieter room. Where a candidate has a genuine need, such as a reading difficulty, fair adjustment is made to the conditions without lowering the standard, exactly the distinction the course holds throughout.

Brief clearly, and remove avoidable obstacles. The candidate is told plainly how the test works, how long they have, and how it is marked, so that no one fails through misunderstanding the test rather than the material. A knowledge test should test the knowledge, not the candidate's ability to decode the test, which is why instructions are plain and questions are clearly worded.

Guard the integrity of the test. A knowledge test is only worth anything if its result is the candidate's own, so the assessor guards against copying and against questions leaking in advance, and keeps objective-test keys secure. This matters most for an asynchronous, self-marked quiz, where integrity rests partly on the honesty the College builds in its members and partly on keeping answer keys separate from the questions so a candidate sits the test before seeing the answers. The integrity of the test is the integrity of the qualification.

In Practice: Two Versions of One Test

An assessor of the Royal Army College must build a knowledge test for a course outcome that has two parts: a wide body of facts and principles the candidate must know, and the ability to explain, in their own words, why one principle matters. A weak assessor would reach for whatever is easiest to mark and write a single kind of question for both. The College's assessor matches the form to each part of the outcome.

For the wide body of knowledge, she writes an objective section of multiple-choice questions, because it tests breadth reliably and marks identically whoever marks it. She writes each MCQ by the rules: one defensibly correct answer, distractors drawn from the real mistakes students make, all options of similar length so the answer is never the obvious longest one, no trick wording or "all of the above", and the correct answers spread evenly across the positions. For the part of the outcome about explaining, she writes an open question asking the candidate to explain the principle in their own words, and she writes its marking scheme at the same time, stating exactly what a creditworthy explanation contains, so it can be marked reliably and fairly. She considers adding a short oral to probe one candidate whose written explanation is borderline, and plans neutral, non-leading questions for it, with a written record.

She runs it under the same conditions for all, briefs the candidates plainly on how it works and is marked, and keeps the answer key secure and separate. The result measures what the outcome actually named, the knowledge by the objective section, the understanding by the open question, each by the form that truly tests it, and it marks consistently because the objective section has one key and the open question has its scheme. The qualification behind it is sound, because the test measured the right things in the right ways, which is the whole of this lesson.

Check Your Understanding

Explain how knowledge assessment differs from skill assessment, and set out the three kinds of knowledge test (objective, open written, oral), what each is best for, and the validity rule that governs the choice between them.
State the rules for writing a good multiple-choice question, and explain for two of them exactly which way of being "right without the knowledge" or "wrong with it" the rule closes. Why does keeping all options the same length matter so much?
Explain what a leading question is and why it destroys an oral assessment, and how an oral assessment is made fair and reliable despite being spoken and unrecorded. Then explain why an open written question must be written together with its marking scheme.

Reflection (write a short paragraph): Think of a written or oral test you have sat that felt unfair or seemed to measure the wrong thing, a question with two defensible answers, an essay marked by mood, an obvious longest-answer giveaway, a confusingly worded question. Using this lesson, name exactly which rule was broken and which of the four principles (valid, reliable, fair, transparent) it failed. Then, picturing yourself writing a knowledge test for something you know well, describe the one kind of question you would find hardest to write well, and the rule you would have to watch yourself most carefully to keep.

Summary

Knowledge is assessed by written and oral tests that draw out and judge what the candidate understands. Every such test must hold the four principles of Lesson 02: valid, reliable, fair, and transparent. Most testing faults are failures of one of these.
Choose the form by validity: objective/MCQ tests for breadth of knowledge with perfectly consistent marking; open written questions for explaining and reasoning in the candidate's own words; oral for probing depth and following up. Match the form to the outcome, not to ease of marking.
A sound multiple-choice question has one unambiguously correct answer, plausible distractors from real misconceptions, options of similar length and form (so the answer is never given away by being longest), tests understanding not trivia or trickery (no double negatives or all/none-of-the-above), and varies the correct option's position. Each rule protects validity.
Open written questions ask one thing clearly, at the outcome's level, and are written with their marking scheme at the same time; if you cannot write the scheme, the question is not clear enough. Oral questions must be open and non-leading, structured so every candidate faces the same ground, judged to the same standard, and recorded.
Run a knowledge test fairly: same conditions and instructions for all, fair adjustment of conditions without lowering the standard, clear briefing so the test measures the knowledge and not the decoding of the test, and integrity guarded (keys kept secure and separate, especially for asynchronous self-marked quizzes).
This is the knowledge layer; marking borderline answers and probing an oral candidate are mastered by assessing real candidates under a qualified assessor and signed off in person. This lesson deepens the methods of Lesson 03, serves the principles of Lesson 02, leads into marking and moderation in Lesson 08, and pairs with the assessment of practical skills in Lesson 07.

Assessing Knowledge: Written and Oral Tests