A six-question test
Your AI feels shallow. Here's why — and how to prove it.
Most internal call-intelligence tools work top-down: you ask a question, and the tool speed-reads every transcript hoping the answer floats to the top. Run these six questions against your own tool. Count the passes.
Ask-then-organize vs. organize-then-ask
The whole gap comes down to when the work happens.
Top-down · theirs
Work happens at query time
Every question re-reads the pile: embed the question, pull the top-k chunks from all transcripts, stuff them in a prompt, hope the answer surfaced. No memory between questions, and a hard context budget on every one.
Bottom-up · Telephos
Work happens at ingestion time
Every mention is extracted, entity-resolved, labeled, and embedded once, as calls arrive. A question becomes a filter over the full population, then a rank — a lookup, not a search-and-pray.
Top-down is re-watching the whole day's security footage every time someone asks “how many red cars drove by.” Bottom-up logged every car's color as it passed — now it's a one-line query, and you can answer questions you never anticipated.
The cataloguing is the product.
To be fair:a homebuilt tool is genuinely fine for point lookups — “find me the call where they mentioned pricing.” The line is everything that needs to see the whole populationat once. That's what the next six questions test.
The bake-off
Ask your own tool each question. Swap the examples for your own vocabulary. Mark whether the answer would hold up — then read your score below.
How many distinct customers raised the same complaint in the last 90 days?
e.g. "How many accounts flagged slow checkout since April?"
List every account that asked about a specific integration or feature.
e.g. "Every account that asked for a Klaviyo integration."
Show every mention of a vendor that goes by several names.
e.g. a tool the team calls by its product name, its company name, and two misspellings.
Which calls in a segment never discussed a given topic?
e.g. "Which enterprise deals had no security conversation?"
Is a complaint growing or shrinking, quarter over quarter?
e.g. "Is churn-risk language about pricing rising?"
Who said a specific thing, on which call, and when?
e.g. "Who asked for SSO, on what call, what date?"
Roughly how many tokens — or how long — did each answer take?
To answer anything, a top-down tool hauls a big slice of your transcript pile into the prompt — on every single question. Bottom-up does the extraction once, at ingestion, so each query is a small filter-and-rank. The bill isn't a one-time cost; top-down pays it again every time someone asks.
Your score
0 of 6 answered — mark the rest above for your full score.
You're speed-reading a pile.
Your tool retrieves — it doesn't organize. Every population question is a coin flip, and the answers change depending on which chunks happened to surface.
For the skeptic on your team
Each question maps to a failure mode that's structural, not a tuning problem. Forward this to whoever built the internal tool.
Counting
Retrieval isn't enumeration. A top-down tool pulls the handful of passages that look most relevant and summarizes them — it can't count the customers it never retrieved. Ask the same question twice and the number moves.
Long-tail recall
Top-k retrieval has a hard ceiling: it returns the k most-similar chunks and stops. The loud, frequent mentions crowd out the one-off asks — so the long tail silently vanishes.
Entity resolution
Naive semantic search treats each spelling and sub-brand as a different thing, so mentions scatter across unlinked fragments. Nothing collapses them into one entity.
Absence
Absence is unanswerable by retrieval — there's no passage to fetch for a thing that was never said. A top-down tool can find where something happened, never where it didn't.
Trend over time
Trend needs every mention stamped and counted over time. Without time-labeled structure, a top-down tool gives you 'it comes up in several calls' — no direction, no trajectory.
Verifiable attribution
Without structured provenance the model reconstructs attribution from context — which means it guesses. Plausible, often wrong, never checkable.
Cost that scales the wrong way
Top-down cost grows with question volume × corpus hauled per question — the bill climbs every time the team asks something. Bottom-up pays the extraction cost once at ingestion; marginal cost per answer approaches zero.
Run the same six live — on your data.
We'll take your questions, against your own calls, and show you what organize-then-ask looks like when the answer has to be complete, countable, and cited.