When is a mark not a mark?

You might think setting exam questions and mark schemes is straightforward – but watch out, there are pitfalls about…

Hand holding a pen and circling a cross on a piece of paperMark schemes are used by examiners to guide them as to how to award students marks for their answers in exams. How difficult can it be to set up the mark scheme for a question, you might ask - surely anyone with subject knowledge should be able to do it; particularly if the possible responses to the question aren’t subjective? 

If we wanted to explore students’ knowledge of the area of a circle, for example, the question on the exam paper might be:

Work out the area of a circle of radius 6m.

……………………………………………………………………………………

……………………………………………………………………………………

……………………………………………………………………………………

……………………………………………………………………………………

Answer ………………………………………….m                    (2 marks)

The mark scheme would award one mark for the correct method and one mark for the correct answer; all very straightforward.

It might come as a surprise then to find that when this question was posed to GCSE Mathematics students in summer 2012, less than 5 per cent were awarded a mark of one.

Graph showing the distribution of a two mark question

Should we have been worried that the mark of one was used so infrequently? Surely it is sufficient to know whether or not a student has the skills needed to calculate the area of a circle.  Maybe in a two mark question this isn’t a problem, but what happens when the maximum number of marks available for the question is greater?

If the mark distribution for a ten mark question looked like this:

Graph showing the distribution of a ten mark question

what would it tell us about the students? A quarter know nothing of the subject area, just over a half know everything required, and the rest know something. 

A question and mark scheme should be designed to differentiate between students. It should allow for marks to be awarded across the whole mark range so that clear distinctions can be made between the quality of a student’s performance at each mark. With the majority of marks being awarded at the minimum and maximum in this example, it seems unlikely that this ten mark question is allowing adequate differentiation.

Again, we might ask whether we should be worried about some marks in a mark scheme being underused. After all, we can still distinguish between a poor, mediocre and good performance in the subject area. But the implications of failing to differentiate between students on a single question permeate throughout the whole paper.

If this ten mark question was designed to contribute to a 40 mark paper, for example, then its weight in the assessment would have been 25 per cent. However, if the distribution of marks achieved by students sitting the exam was as described above, the question would have been effectively marked out of three (reflecting ‘poor’, ‘mediocre’ and ‘good’ performances). This alters the intended weight. The achieved weight of the question would be better described as 3 out of 33 or, 9 per cent.

In the abstract, we might be able to accept idiosyncrasies in a given question and its mark scheme but, when these idiosyncrasies are viewed alongside how a qualification and its content and assessment objectives have been designed, they become more difficult to defend. The GCSE subject criteria published by Ofqual state that:

“The assessment arrangements for GCSEs must show the proportion of marks allocated to each assessment objective (or group of assessment objectives) and to each assessment component.”

So, while it may be easy to write a mark scheme for a question, it is not so easy to write an effective mark scheme that fulfils these assessment requirements. Nowadays, principal examiners make sure they focus as heavily on the mark scheme as they do on the exam questions when they construct an assessment. By first identifying the domain to be tested, then writing a mark scheme detailing where marks should be awarded, and only after that devising the questions, we can avoid nasty surprises and ensure that our exams really are doing what we want them to do.

Anne Pinot de Moira

Share this page