Receipts or Results – Part 2: The Wrong Metric for the Right Question

A gardener doesn’t judge the health of a garden by how much they’ve watered it. They look at whether things are actually growing. Watering is a necessity, but it’s not a measure of success – plant growth is.

Security culture measurement has the same problem, and it shows up in two very specific ways. The first is measuring how people feel about training and calling it evidence of culture change. The second is measuring behaviour immediately after an intervention and believing this tells you about its effectiveness. Both mistakes are common and often expensive. Neither tells you anything meaningful about whether your programme is actually managing behaviour over time.

The survey score trap

Many security training programmes includes a satisfaction survey:

• Did you find this useful?
• How would you rate the content?

Satisfaction surveys are relevant metrics in so far as desired change is unlikely to happen if people dislike your training. But a five-star rating tells you how people feel about the experience; it doesn’t tell you if the training changed how the participants think or act.

The issue isn’t with satisfaction surveys themselves; it’s that their outputs get applied to questions they were never designed to answer. Satisfaction surveys measure the training experience, not behaviour change. Enjoyable training does not automatically make it effective.

The novelty trap

Even when organizations move past just satisfaction surveys and start measuring actual behaviour, there’s a second mistake waiting. Measuring too soon while not measuring the impact over time.

Immediately after a training intervention, behaviour may look better. People are primed, aware, and paying attention. What you’re seeing here is novelty, not change.

The data you want is what remains weeks later. When nobody is thinking about the training anymore, their inbox is full, and there’s a deadline to meet. That’s when you find out whether knowledge has been adopted into behaviour, or whether you just measured the echo of a good session.

This is why culture is the hardest thing to evidence in behaviour management. Culture exists in the decisions people make when nothing is prompting them, and no one is watching. Measuring it requires patience to wait for the right data rather than reach for the convenient kind.

Timing and what to look for

In Part 1 of this series, we talked about measuring at 7, 30, and 90 days after an intervention. For culture, the timelines are longer, and the patience required is greater. Phishing simulation results can change in weeks. Changing culture is different.

You’re looking for signals that accumulate gradually:

• unprompted behaviours appearing more frequently,
• the security team being consulted earlier in decisions,
• changes in how risk is discussed across the organisation.

An organization where employees are engaging in security often see the patterns below:

• People raising security concerns in project meetings before anyone asked them to;
• Employees coming to the security team with questions before acting rather than after something went wrong
• Managers flagging unusual behaviour in their teams without being told to look for it

These behaviours are the closest thing you have to evidence that culture is actually impacted: people making security conscious decisions when nothing is requiring it of them.

Measuring culture is not a comfortable endeavour. There is no single number that tells you it’s working. What you’re looking for is direction and degree of change over time: are the signals moving the right way and are those changes meaningful or marginal.

You shouldn’t throw away static activity metrics, but they’re not truly valuable without behaviour context to go with them. Completion rates, simulation scores, policy sign-offs still have a place, but that place is telling you what happened at a point in time. The real information is in the behavioural indicators giving them context: the unprompted behaviours, the quality of reporting, and accumulating cultural signals.

Activity metrics are your receipts. Time based behavioural indicators are your results. Culture is what you get when you read them together.