I knew five minutes in (the line that survived every measurement against it)

I knew five minutes in

Nine forty on a Tuesday morning, in the meeting room of a four-hundred-person company in Vicente López. Fourth interview for the Operations manager role; the calendar says forty-five minutes. It runs seventy.

There are three people inside: the industrial director, the head of Human Resources and the candidate. They talk about a boss they have in common, and about the traffic. At minute fifty the first two questions about the plant appear. Nobody takes notes. On the table, three coffees and a CV with a single handwritten mark: a circle around the name of one company.

The director comes out first and waits in the corridor to speak:

"This is the one. I knew five minutes in."

Nobody argues. The head of Human Resources had prepared a grid of questions and used none of them: the conversation went somewhere else and cutting it short felt rude. The candidate starts the following month and leaves seven months later; on the way out, the word "fit" appears.

None of that is a failure of the process: it is the process. The same company has a quality manual and requires two signatures for a salary advance; to choose the person who will run the plant, seventy minutes of conversation and a circle drawn by hand.

(The scene is a composite reconstruction. From here on, everything with a number in it has been measured and has a source.)

The line has a reasonable defence: twenty years of interviewing must have taught you something. But they only teach if somebody keeps score of how each one turned out, and nobody keeps it: deciding without measuring the outcome does not produce expertise, it produces confidence without accuracy.

A warning: none of the four measurements that follow was carried out on an Argentine owner choosing the manager for his plant. What travels is not the result but the mechanism.

In 2013, three researchers built the experiment that puts the line to the test. It is fairly cruel.

The answers picked at random

The task was concrete and verifiable: predict the grade point average another student would earn the following semester. The ones doing the predicting were students too, on the side of the table that concerns you: the interviewer's. They had that person's previous average —hard data from the file— and twenty minutes of conversation.

There were three versions: one with yes-or-no questions, another with open questions, and a third identical to the first, yes or no, except for something the interviewer never knew: the person across the table had been given a secret instruction.

The interview had a break at the ten-minute mark. From there on, in the third version, the interviewee stopped listening to the question and answered yes or no according to a mechanical rule about the letters of the last two words he had heard: a coin toss in disguise, executed in a second and leaving no pattern. (The literal rule, in reference 1.) You ask whether he sat a hard final exam, the other says yes, and you build meaning out of that: building meaning is what an interviewer does.

Does it show? The authors recorded sixteen of those interviews and showed them to sixty-four people from outside. Of the thirty-two times somebody watched a rigged one, nineteen judged it genuine: only thirteen caught the trick. Tossing a coin would have caught half; with full attention on it, fewer.

And confidence did not move: asked on a 1-to-4 scale how much they could infer about the person, the three groups came out equally convinced.

The number that carries the argument is still missing. First, ten seconds of translation, because this number comes back five times: when a selection method is measured, the result is a correlation — how closely what it predicted resembles what the person went on to do. It runs from 0, tossing a coin, to 1, predicting perfectly. In real jobs, no selection method reaches 0.60; in this laboratory experiment —predicting a grade when you already have the previous grade— accuracy starts higher, so the numbers that follow will clear that bar without contradicting it: they are playing an easier game.

Each participant predicted the average of several people: some he interviewed, others he assessed with the file alone. With the file alone, his accuracy was 0.61; with the file plus the twenty minutes of conversation, 0.31. Same data, same people, half the accuracy. And that 0.31 pools the three versions, sincere conversations included: nothing needs to be rigged.

One defence remains: that a person lost to a formula. The 0.61 against the 0.31 dismantles it — the same people, with and without the conversation. The 0.65 is only the bar: with the previous average used exactly as it stands, nobody in the room, accuracy would have been that.

Read it slowly. The ones who talked had the previous average on the same sheet, and a conversation on top of it. They ended up aiming worse than the sheet alone. This is called the dilution effect: the interview does not add noise alongside the good information, it eats its place.

And then the punchline. Another group of a hundred and sixty-nine people had the whole experiment described to them and was asked to rank the options: 57% put the rigged interview above having none at all.

The people who hire for a living

Here somebody raises a hand: they were students, and what they were predicting was a grade. A manager with twenty years on the job does not interview like that.

It is the right objection, and there is a measurement that answers it. Three researchers gathered 132 people with real responsibility for hiring decisions, 39 years old on average, who rated themselves 4.6 out of 6 on experience. That self-rating is not a credential, and that is precisely why it works: it is the same certificate you hold about your own eye. (US decision-makers choosing between two files, not Argentine owners.)

The task was ten pairs of real applicants to an airline: pick which one would perform better. Tossing a coin gets you 50%. With the test scores in the file, they got 69% right. Adding a free-form interview —no fixed questions, no rubric, the one we all run—, 62%.

Confidence moved the other way: that is the finding. On the same scale as the hits: with the tests alone they said they were sure about 74 out of every 100 choices and got 69 right; adding the interview they said 78 and got 62 right. The gap between how sure and how right almost tripled: from 0.06 to 0.17 in the paper's finer decimals. The authors: unstructured interviews "not only fail to help selection decisions: they can hurt them".

Stop on that 69 against 62: those are hits out of a hundred, the only place where the numbers speak the language you decide in. Seven choices in a hundred flipping over sounds like little; put it on your own scale: with three manager searches a year, it is one extra bad hire every five years, and each one costs —between severance and the replacement's ramp-up— around six months of salary. The free-form conversation is not free: it has a price, and that is the calculation that turns this piece into money. Run it with your own numbers.

The table everyone cites went out of date in 2022

Why is free judgement still the default method? Scott Highhouse put it down in 2008 to comparing yourself against 100% rather than against the coin —seven points of accuracy feel like nothing— and to almost nobody keeping the list of what actually predicts up to date.

That list goes around on LinkedIn, in every HR course and in almost every assessment proposal. It ranks methods by how well each one anticipates later performance on the job, on the correlation scale from the section above. It is Schmidt and Hunter, 1998, and what stuck is that general intelligence rules, at 0.51 —the one everybody remembers, not the highest: that was work samples, at 0.54.

What they corrected is an arithmetic problem with a technical name, and it is worth telling first without the name. You only ever see how the people you hired perform, and the ones who got in resemble each other, so from the inside any method looks as if it predicts less than it does. To compensate, researchers add a correction. It is called the range restriction correction, and Sackett, Zhang, Berry and Lievens showed it had been applied too generously. They redid the sums: of the ten methods in the figure, nine came down.

In the redone list, the structured interview ended up on top.

Structured means three things, none of them expensive: the same questions for everyone, a rubric written before meeting anybody, and each interviewer scoring on their own. The end of this piece is how to build it.

Here the easy headline lies, so here is the full subtraction. The structured interview, which in 1998 tied with cognitive ability at 0.51, came down to 0.42. It is not that it improved: almost all of them came down, cognitive ability included, which collapsed to 0.31. It started high and it took the correction. (The tests that beat the free-form conversation in the Kausel study belong to this family: against the free-form one they win, against the structured one they lose.)

At the other end fell the unstructured interview, the free-form conversation from Vicente López: 0.19, an average across very different contexts. In the least favourable 10% —the edge of the range the authors report— it is −0.01: there it predicts nothing, or predicts backwards.

And there is something else in that table, which orders everything that follows. "Show me how you do the work" beats "tell me what you are like": the four strongest predictors —structured interview, job knowledge, biodata (a scored record of past behaviour) and work samples— measure something specific to the job, not a general trait. Wernimont and Campbell called it, back in 1968, sign versus sample.

Go back to Vicente López. The circle around a company's name is a sign: it says where the person comes from. Nobody asked him for a sample —how he gets a plant moving on a Monday with two absences and a late order— which is exactly what the new table puts at the top.

So far, what predicts better. The other half is still missing: what happens when the person deciding has the forecast on the table and chooses something else.

When the manager beats the system

The manager walks in with two files. "The system says this one is best, but I'm going with the other." The owner trusts him. Nobody keeps count of how often it happened or how it ended. Somebody did.

Three economists followed 15 service companies that rolled out a test —green, yellow or red for each applicant— site by site, not all at once: while some sites were already using it, others in the same company were not yet — and that difference is what makes it possible to isolate the effect of the test. That is 400,000 applicants and 91,000 hires in low-skill jobs, and quality is measured by how long the person stays, not by their performance.

When the test came in, tenure rose by 25%.

Then they measured the two-files scene: every time a manager hired a yellow when a green was available, or a red over a yellow. They gave that act —skipping the order the test proposes— a name and a count: an exception. With it they built two laboratory companies, one made of the managers who almost never skip the test and another of the ones who always do: in the first, people stay around 15% longer.

The classic defence is still missing: yes, they stay less, but while they are there they perform better. The authors went looking for it in daily output per hour and did not find it —absence of evidence in favour of the exception, not evidence that the person performs worse—. Gut feel had its chance and did not show up: the figure that follows is a measured cost beside an absent benefit.

In the mid-sized Argentine company, skipping the order is not exceptional: it is the procedure, because the owner does the interviewing and nobody reviews his verdict. The exception is not the enemy; the problem is that here it leaves no trace: in the study it could be measured because every one of them was logged, in your company it is not.

The minimum viable structure

None of this recommends you stop interviewing: the structured one came out first. What does not work is running it free-form. At 32sur we call it The Grid —six moves; the timings and thresholds are on the card at the end.

One: define. Before the job ad, the three or four things the person has to know how to do, in verbs, written by whoever will suffer if it goes wrong.

Two: write the questions. The same ones for everybody: if you asked one about his worst week and did not ask the other, you do not have two measurements, you have two conversations.

Three: ask for a sample. A real problem from the company, the same one for every finalist: "show me how you do the work" turned into a step.

Four: the rubric. Three levels with a written example, drafted before interviewing anybody. It is what turns a conversation into a measurement.

Five: score separately. Everyone hands in their sheet before the candidate is discussed, and the person who decides hands in theirs first: if they speak first, what comes after is not information, it is echoes. It is what was missing in Vicente López: the head of Human Resources had the questions written down and did not use them so as not to interrupt the director's conversation.

Six: deliberate at the end. The big differences get discussed: a 3 against a 1 on the same question is the conversation you need to have.

And if you interview alone. It is the most common case, and the one no measurement covers. Write your score before you leave the room: what it protects there is your own opinion from ten minutes ago. And add an interviewer who does not report to you. The ceiling comes from an internal Google analysis —four interviews, 86% confidence; a single company, no peer review: a rule of thumb—; the floor is ours, with no measurement behind it: two.

And that 0.42 is an average: across contexts it varies with a standard deviation of almost 0.20 —spread, not accuracy—. Which is why no method replaces measuring your own: question 6 in the closing list.

Against the dilution effect from the opening there is only one antidote: the interview adds to the file, it does not replace it. If the conversation erased what you knew before you walked in, the problem was not the candidate.

In other countries, not doing this is explained by a lack of data, of scale or of budget. Here it is not: the same questions for everybody, a sample of the real work, a one-page rubric and a scoring sheet do not cost money. They cost one meeting.

The four lines that always come up

They will come up the moment you propose any of the six moves. All four are true up to a point, and that point is what keeps them alive.

It is the half that is true that keeps the line alive.

Questions for Monday

The first five need nothing opened. If getting started requires setting up a system, nobody gets started.

The Operations manager lasted seven months. "Fit" is the word used when nobody wrote down in advance what the person had to know how to do. The grid the head of Human Resources prepared that morning is still in a folder: writing the questions before walking in was, without anybody knowing it, the second move of The Grid — orphaned, because the first one, defining what had to be known, never happened. The most valuable thing on that table, worth more than the three coffees and a good deal more than the circle.

Nobody ever told the industrial director that he had got it wrong, and strictly speaking he had not: he liked the man, and he was right, he did like him. Nobody looked at the file of the man who left again. He opened the search for the replacement himself, a month later, and the first interview ran seventy minutes again. Nobody wrote anything down — and writing things down, not the coffee or the circle, was the only thing that was needed.

The line in the corridor has an honest and far more useful translation: five minutes in, I knew I liked him. That is real information, and confusing it with something else is paid for in months of salary.

The one thing it is not, is a prediction.