Writing OSCE checklists that actually discriminate

Why OSCE checklists often fail to discriminate

Consider a final-year cardiology OSCE. The station is acute asthma in a 19-year-old. The checklist has 18 items. In the last cohort of 60 students, 55 students scored above 80%. The two students who failed had genuinely missed the diagnosis. But so had another three students who passed. The checklist did not separate the competent from the weak; it separated the verbally fluent from the hesitant.

This is the most common failure of OSCE checklists. They measure what is easy to observe, rather than what matters clinically. A student who says "I would give salbutamol" scores the same as a student who states the dose, the route, the timing of reassessment and the criteria for escalation. A checklist that rewards an action once, regardless of quality, cannot discriminate.

There are three recurring causes. The first is vague item wording. Items such as "assesses the patient appropriately" or "manages the patient correctly" require the examiner to make an unstated judgement. Different examiners will apply different standards, so the item adds measurement noise rather than signal. The second is all-or-nothing items. When an item is scored one point for any attempt, a candidate who performs the action badly receives the same credit as one who performs it expertly. The third is over-length. A 40-item checklist forces examiners to spend their cognitive energy keeping score rather than listening and observing. They are more likely to miss omissions and to make errors, especially as stations become more clinically complex.

A checklist that does not discriminate also creates an ethical problem. In high-stakes settings, borderline students are the ones who most need distinction between a competent standard and an unsafe one. If a checklist cannot make that distinction, the examination is not fulfilling its purpose.

The anatomy of a discriminating checklist

A discriminating checklist has a specific internal structure. It distinguishes between essential actions, supporting actions and quality markers. The essential actions are those that, if omitted, would compromise patient safety. The supporting actions are those that competent clinicians would usually perform but whose absence alone would not necessarily fail the station. The quality markers are the specific behaviours that distinguish an expert performance from a barely adequate one.

In the acute asthma example, an essential action is "administers an inhaled short-acting beta-agonist within the first three minutes of the encounter." A supporting action is "asks about previous intensive care admissions." A quality marker is "states that the patient should be reassessed after 10 to 15 minutes to determine the response." These three types of item serve different purposes in the scoring and, crucially, in the interpretation of the score.

An item must also be observable. If an examiner cannot see or hear the behaviour, it should not be on a checklist. This means avoiding cognitive items such as "considers differential diagnosis" or "recognises the severity." Instead, the cognitive work must be externalised into observable language. For example, "names at least two differential diagnoses" is observable, and "considers differential diagnoses" is not.

The length of a checklist is a core design decision. A station of eight to ten minutes should typically have between 12 and 20 items, depending on the complexity. If a checklist exceeds 25 items for a ten-minute station, the examiner is likely to be reading the checklist rather than observing the candidate. The checklist should operate as an aid to observation, not as a script that drives the encounter.

Writing behaviourally anchored items

The single most effective way to improve item quality is to remove descriptive adjectives and replace them with concrete behaviour. Consider these examples from a respiratory station.

Weak item	Discriminating item
Assesses the patient's severity	States that the patient has signs of life-threatening asthma, and names at least one of: altered consciousness, exhaustion, cyanosis, or silent chest
Asks relevant history	Asks about the number of previous hospital admissions for asthma
Manages the patient appropriately	Prescribes a back-to-back nebulised salbutamol, 5 mg, before the end of the encounter
Explains the plan	Explains to the patient that if symptoms do not improve after the nebuliser, they will be admitted to hospital

The key move is writing the item as a specific action with a specified threshold. The threshold can be a time, a dose, a number, or a named criterion. This does not make the item excessively rigid; it makes it testable. A candidate who nebulises the patient after 15 minutes has not performed the action at the same clinically appropriate speed as one who does it immediately. The checklist should reflect that difference.

Another approach is to use two-tier items. Instead of giving one mark for a behaviour, the item awards two marks for a full performance and one mark for a partial performance. This is especially useful for complex clinical reasoning. For example, in a diabetic ketoacidosis station:

• Two points: States that the patient needs an intravenous insulin infusion and a potassium replacement, and explains that if potassium is low, insulin should not be started.
• One point: States that the patient needs insulin and potassium.

This scoring approach preserves the objectivity of the checklist while allowing it to reward depth. It avoids the awkwardness of a single tick that implies equivalence between a safe and an unsafe decision.

Weighting and scoring strategies

Not all items have the same clinical importance. A global weighting system can improve the discrimination of a checklist, but it must be used carefully. The most straightforward approach is to assign weighted points to each item based on its clinical significance. In a ten-station examination with a total score of 100, a station might carry 10 marks. Within that station, an essential safety action might carry 2 points, a supporting action 1 point, and a quality marker 0.5 points.

However, weighted checklists can be difficult for examiners to apply consistently. An alternative is to keep the items unweighted and rely on a global rating score alongside the checklist. The global rating is a separate judgement, usually on a five-point scale, that captures dimensions such as communication effectiveness, clinical judgement and overall competence. Research in assessment design suggests that global ratings are at least as effective as checklists at discriminating between performance levels. They also allow the examiner to compensate for the limitations of the checklist without sacrificing its structure.

The important principle is that the scoring strategy must align with the decision being made. In a borderline competence decision, some items must be non-compensatory. For example, if a candidate fails an essential safety item, the station result should be a clear fail, regardless of the total points. This means the checklist should have a fail zone, a set of items that, if missed, automatically triggers a failure. This is distinct from simply having a percentage threshold.

A common risk in weighting is overcomplicating the marking process. If an examiner needs to mentally calculate weighted sums while also managing a patient and an actor, the reliability of the scores will drop. The solution is to design the checklist so that the weights are intuitive. In practice, this means using either a small number of weighted critical items or a single global rating, and not both unless the weighting is minimal.

Piloting and revising your checklist

No checklist should be used in a high-stakes examination without piloting. The purpose of a pilot is not to check whether the content is accurate; it is to check whether the items behave well in practice. A well-performing item is one that experienced examiners score consistently and that a wide range of candidates answer in different ways.

During a pilot, collect three types of data: item difficulty, item discrimination and examiner feedback. Item difficulty is the proportion of candidates who scored the item. If 100% of candidates score an item, it provides no information about the ability of borderline students. It may still be worth keeping as an essential safety item, but it will not contribute to discrimination. Item discrimination can be measured by comparing the scores of the top-scoring candidates with the bottom-scoring candidates. An item that is scored at the same rate by both groups is not adding value.

Examiner feedback is often more useful than any statistical analysis. Ask examiners to identify items they found difficult to judge, items they felt were ambiguous, and items they would like to see added. A classic example from a suturing station illustrates the point. The item "uses a clean technique" was confusing because examiners could not agree on whether this referred to hand hygiene, sterile draping or the handling of the needle driver. Rewriting the item as "wears a new pair of non-sterile gloves before starting" resolved the ambiguity.

After the pilot, revise the checklist with attention to items that are either too easy or too hard. Very easy items should be moved to the fail zone or removed if they are not safety critical. Very hard items, such as an obscure dose that almost no candidate recalls, should be deleted because they test recall rather than clinical judgement. The goal is a checklist where the total score spreads across the full range, with the borderline region containing enough information to make a defensible pass or fail decision.

Finally, remember that a checklist is a living document. It should be reviewed after every examination cycle, particularly when an examiner panel flags a recurring issue. A checklist that worked five years ago against one clinical guideline may no longer be appropriate now that practice has changed. Regular review, informed by examiner feedback and item analysis, keeps the assessment defensible and its results meaningful.

Content provided through Oskela is intended for education and training. It is not a substitute for professional clinical judgement, institutional protocols, current formularies, prescribing information, or specialist consultation.