AI hiring assessment for expert mentors

23behaviours that separate top from bottom experts, mined from ~5,000 real call transcripts

~1,200real calls the rubric was validated on, joined to CRM outcomes

71% vs 25%next-stage rate for calls scoring 8+ vs under 2 on the rubric

~$2cost per candidate assessed · $1.50 voice call, $0.30 scoring, $0.001 CV screen

01 — The problemInterviews measured charm. The job needs something else.

An expert is credible because they have built a company, and the call works when they do specific things: ask what prompted the lead to book, recap the story back, narrow to one customer segment, assign a time-boxed piece of homework, say plainly that they have no financial stake. None of that shows up in a conversational interview, where every founder with an exit sounds good.

The team was hiring on gut, finding out three months later from the conversion numbers, and starting again. I wanted the on-the-job measurement before the hire.

02 — What I builtA pipeline from CV to recommendation, with the rubric built from evidence.

ResearchRubric miningPulled about 5,000 call transcripts, tied each to its CRM outcome, took the top and bottom experts by conversion, and had an LLM contrast what they did. Kept the 23 behaviours that distinguished them, then validated against roughly 1,200 further calls.

Stage 0CV screenA cheap model reads the CV against hard requirements: built and scaled a company to seven figures or an exit. Disqualifiers are explicit. About a tenth of a cent per CV.

Stage 1The mock callThe candidate talks to an AI founder for 30 minutes in the browser. The founder has a hidden fact sheet, a bias to build before validating, sceptical parents and a part-time job, and it withholds detail until asked properly. Same persona for every candidate, so scores are comparable.

Stage 2Three-rater scoringThe transcript is scored three times, under strict, lenient and adversarial framings, and the consensus is reported. Every evidence quote the model cites is substring-checked against the transcript, so it cannot invent what the candidate said.

OutputRecommendation emailStrong hire, hire, hire with 30-day review, or no hire, with the scored dimensions, quotes and auto-fail flags. Goes to the hiring team. Candidates never see it.

ReuseReviewer toolA paste-a-transcript version of the same scorer for the hiring team to check human-run mocks against the same bar.

03 — How it worksDecisions that keep the score honest.

Behaviours, not vibes. Each rubric item is a concrete, observable action with a quote from a real top performer attached, so the model and the humans score the same thing.
Auto-fail tripwires for the things that lose deals outright: a monologue, a cold read of the lead, a fabricated outcome, dishonest pricing. Standard financing and accreditation facts are explicitly allowed, because a false flag on them cost a good candidate once.
Thresholds set from the rubric's own definitions. A 2 on every dimension is "competent", so straight 2s is a hire. The original cut-off demanded better than competent and rejected people the rubric itself said to hire. It was lowered and the reason written down.
Two implementations, one source of truth. The Python reviewer tool and the TypeScript candidate pipeline share framings, schema and prompt. A change to one is a change to both, or the deployed pipeline silently drifts.
Determinism where possible. Temperature zero, fixed model. Runs still differ by about a point, which is reported as the measurement error it is.

04 — What brokeWhat the data later said about the rubric itself.

The bar was a strong-hire barCompetent candidates were scoring no-hire because the cut-off needed an above-competent average.

Thresholds re-derived from the rubricHire at 10 of 15, strong hire at 13, review band at 8. Written up so nobody quietly tightens it again.

A false compliance flagThe scorer flagged a mention of federal financial aid as a compliance risk. It is a true, allowed fact about the programme.

An allow-list in the rubricApproved claims are listed and never flagged. Tripwires are reserved for fabricated outcomes, tax promises and accreditation workarounds.

The composite score over-promisedLater research on the live cockpit showed the rubric's composite separates the bottom of the scale clearly and the middle hardly at all. Expert tiers built on it inverted once.

Use it as a floor detectorThe hiring recommendation leans on auto-fails and the bottom band, where the signal is real, and treats differences between good candidates as noise to be resolved by a human debrief.

Robotic voiceThe first AI founder sounded like a narrator, and candidates played to it.

Tuned for conversationConversational voice model, lower stability, higher temperature, sparse audio cues. Judged in the browser, because phone-line audio flattens everything.

05 — What I'd do differentlyTrack hires through to their first 50 live calls.

The assessment predicts the behaviours that top experts show. The loop that would prove it predicts conversion is to follow each hire's live calls through the scoring cockpit and compare their assessment band with their real outcomes. The data exists on both sides now. I would have joined them from the first cohort, and I would publish the hit rate to the hiring team, including the misses.