When measures don’t converge, are they still valid?

One of the most influential concepts in psychological measurement is convergent validity.

The idea seems straightforward. If several different measures are supposed to assess the same phenomenon, we would expect them to agree. If they do, we often become more confident that we’ve measured something “real.”

That sounds sensible.

Until you stop and ask a simple question:

What exactly have we observed?

Imagine someone with a snake phobia.

As they approach the snake, their heart rate increases. Their hands begin to sweat. At the same time, they continue walking toward the snake because their therapist is encouraging them and they are committed to treatment. If you ask them what they’re thinking, they tell you they are imagining themselves somewhere else entirely.

Three different response systems.

Three different patterns of behavior.

Which one tells us how afraid they “really” are?

From a behavioral perspective, that question is actually the wrong place to begin.

Each of those responses is behavior occurring under somewhat different environmental influences. The physiological response may reflect years of learning in which snakes became associated with danger. Walking toward the snake may be controlled by social reinforcement. The person’s verbal report may indicate rule-governed behavior, or not.

Why would we expect these behaviors to match perfectly?

What happens if, instead of asking “which one tells us how ‘afraid’ they really are?”, we ask “which one is going to be most tied to the change we’d like to drive?” We need to stop assuming “afraid” is some ‘thing’ we can accurately detect. 

Staying close to what we actually observed

This is one of the places where I think psychology sometimes moves too quickly away from its observations. Suppose someone completes a questionnaire designed to measure anxiety.

What did we actually observe?

We observed that a person endorsed a series of verbal statements in response to a particular set of prompts, under a particular set of circumstances.

That’s the observation.

Everything beyond that requires another inference.

If that questionnaire correlates highly with another questionnaire asking similar questions, what have we learned?

We have learned that people tend to respond similarly to similar verbal prompts.

That is an interesting and important finding.

But it is not, by itself, proof that an underlying psychological entity called “anxiety” or “fear” has been discovered.

The data demonstrate consistency in responding. Whether that consistency reflects an internal construct, a learned repertoire, recurring environmental contingencies, stable patterns of relational responding, characteristics of the measurement situation, or some combination of these remains a separate scientific question.

Those are different claims.

And they require different evidence.

Behavior analysis takes a more conservative position.

Behavior is treated as behavior.

Not as a sign pointing toward some hidden thing that must exist behind it.

That doesn’t mean internal events are ignored. It means we begin with what we can actually observe and then ask what variables are influencing those observations. When different measurements disagree, the first question isn’t, “Which one is right?”

Instead, it’s, “What environmental variables might be influencing each of these different behaviors?”

Sometimes disagreement between measures isn’t ‘measurement error’. Sometimes it is exactly what we should expect.

A person’s physiology, overt behavior, and verbal behavior are all influenced by overlapping—but not identical—histories and current contingencies.

Understanding why they diverge may teach us far more than forcing them into agreement.