The Gap Between What Is Measured and What Matters

Ask what makes somebody good at a role and you get one list. Look at what the assessment measures and you get another. The two rarely match.

The principles in “The Gap Between What Is Measured and What Matters” become easier to maintain when ownership and time spent are visible. Teams evaluating the practical overview can use it to coordinate the administrative work around assessment, identify stages that repeatedly expand and plan capacity, while keeping the hiring decision tied to job-relevant evidence instead of raw activity totals.

For an independent benchmark, compare the process with U.S. OPM assessment resources; the useful test is whether the local method remains job-related, proportionate and explainable.

What usually gets measured

Credentials, because they are on the application and require no work to check.

Years of experience, for the same reason.

Interview performance, which measures the ability to perform in interviews.

Technical knowledge that can be tested quickly, whether or not it is the constraint in the role.

And availability to attend four rounds, which nobody intends to select on and everybody does.

What usually matters

Whether the person can do the core task, which is frequently not tested directly.

Judgement in ambiguous situations, which is hard to assess and is what most roles actually consist of.

Working with the specific people and systems in place.

And learning speed, which matters more in fast-changing roles than any current knowledge.

How the gap opens

Convenience. What is easy to measure gets measured, and then it gets treated as what matters.

Inheritance. The process was designed for a different role five years ago and nobody revisited it.

Defensibility. Credentials feel safe to point at if a hire fails, which makes them attractive independently of prediction.

And the person designing the assessment frequently has not done the job, which is the structural cause and has a straightforward fix.

Closing it

Start from the job analysis: what does somebody actually do, and what distinguishes people who do it well.

Then ask, of each stage: what does this predict, and is that on the list?

Remove anything that fails that question. Most processes lose two stages and improve.

And add at least one assessment that resembles the work itself.

The experience trap

Worth naming because it is the commonest proxy.

Years in a role measures exposure, not competence, and the relationship between them is weak beyond the first couple of years.

A five-year requirement excludes capable people and admits people who repeated one year five times.

Where experience is genuinely necessary, say what it is needed for — and then consider whether you could test for that directly instead.

The credential question

Sometimes required by regulation or by a client, which is a real constraint.

Frequently inherited, which is not.

Ask what the credential guarantees, and whether anything in the role depends on it. Organisations that do this exercise usually drop several requirements and widen their candidate pool substantially.

What to check

List what your process measures and what the job requires, side by side.

How many of the first list appear on the second?

Is there any stage predicting something not on the job list?

And when did anybody who does the job last review the assessment?

The point

List what your process measures and what the job requires, side by side.

The overlap is usually smaller than anybody expects.

Underlying all of this

Almost everything in this collection reduces to one discipline: write down what the job requires, assess that thing directly, record the evidence, and look at your own outcomes afterwards. None of it requires buying anything, and organisations that do those four things consistently outperform ones running longer processes built from instruments chosen before the requirements were known.