The art of setting questions

CERP’s Assessment Expertise Training Developer Victoria Spalding considers the challenges of question writing

Question paperI’ve booked this great holiday next week. Can you tell me the best way to get there?

If I asked you this question your first response would be: where precisely are you going? Similarly, question writers need to know exactly what they want to test. This process is not as simple as it may first appear.

What is the best way to get there?

If my holiday was in Spain, you might suggest I fly, but if Bognor Regis was my destination then this would not be appropriate. Likewise, there is no point trying to assess physical skill in dance via a written exam. The mode of assessment must be the most valid way of testing the knowledge, understanding or skill the item writer is targeting. (An item is the whole question, instruction, stimulus and mark scheme). Even within a written exam there are different types of items. Multiple choice questions are good at testing recall over a wide topic range. Short answer questions are good at testing recall, application and understanding. Essay questions can test higher order skills, such as analysis.

Getting directions

You will need to know the exact destination in order to give good directions. Item writers need to be precise in order to enable the candidate to demonstrate the skills they seek to assess. If the examiner does not know exactly what they want to test they cannot write a valid assessment. Sounds obvious, but it is much harder than you might think.

Starting points

Item writers need to know how to control the demand of their questions to set them at an appropriate level. Giving GCSE foundation students A-level difficulty items would not be a good way to assess what they have learned over the course of their study.

How we can lose our way

Consider this question:

Geography exam question

And now the mark scheme:

Mark scheme for geography exam question

On the face of it, the question appears well written. The response includes four key points: ‘gravity’, ‘makes the glacier move slowly’, and ‘over time’ it ‘smooths the surface’.

The question asks for an explanation yet the answer, given full marks, does not explain at all. The mark scheme is not rewarding the skill the question intends to assess – the candidates’ ability to explain a process. A candidate who writes all the words they know on the topic will do better than a candidate who wrote ‘it scrapes the land like sandpaper’ (2 marks) – despite demonstrating a greater understanding of the mechanism of glacial abrasion. The mark scheme is rewarding the wrong skill and is invalidating the item. Had this question asked for a description then the mark scheme would have been appropriate. However, as it is asking for an explanation the item writer should have used a mark scheme like the one below:

Revised mark scheme for geography exam question

The importance of importance

Keeping items valid is crucial for good assessment. Assessment researchers Alastair Pollitt and Ayesha Ahmed came up with the following model to ensure that validity drives the question-setting process.

'What's important in this subject' diagram

Item writers need to be creative but also methodical in their approach: question setting is both an art and a science.

Victoria Spalding

Share this page