A six-question test
Your AI feels shallow.
Here's why — and how to prove it.
Most internal call-intelligence tools work top-down: you ask a question, and the tool speed-reads every transcript hoping the answer floats to the top. Run these six questions against your own tool. Count the passes.
01 · The bake-off01 · The bake-off
Six questions. Ask your own setup.
Swap the examples for your own vocabulary. Mark whether the answer would hold up — your score keeps itself as you go.
To be fair: a homebuilt tool is genuinely fine for point lookups — “find me the call where they mentioned pricing.” The line is everything that needs to see the whole population at once. That's what these six questions test.
1. Counting
How many distinct customers raised the same complaint in the last 90 days?
e.g. "How many accounts flagged slow checkout since April?"
2. Long-tail recall
List every account that asked about a specific integration or feature.
e.g. "Every account that asked for a Klaviyo integration."
3. Entity resolution
Show every mention of a vendor that goes by several names.
e.g. a tool the team calls by its product name, its company name, and two misspellings.
4. Absence
Which calls in a segment never discussed a given topic?
e.g. "Which enterprise deals had no security conversation?"
5. Trend over time
Is a complaint growing or shrinking, quarter over quarter?
e.g. "Is churn-risk language about pricing rising?"
6. Verifiable attribution
Who said a specific thing, on which call, and when?
e.g. "Who asked for SSO, on what call, what date?"
Bonus · what did that answer cost?
Roughly how many tokens — or how long — did each answer take?
To answer anything, a top-down tool hauls a big slice of your transcript pile into the prompt — on every single question. Bottom-up does the extraction once, at ingestion, so each query is a small filter-and-rank. The bill isn't a one-time cost; top-down pays it again every time someone asks.
02 · Your score
0/ 6
You're speed-reading a pile.
Your tool retrieves — it doesn't organize. Every population question is a coin flip, and the answers change depending on which chunks happened to surface.
↑ Back to the questions03 · The difference
Ask-then-organize vs. organize-then-ask
Top-down is re-watching the whole day's security footage every time someone asks “how many red cars drove by.” Bottom-up logged every car's color as it passed — now it's a one-line query, and you can answer questions you never anticipated.
The whole gap comes down to when the work happens.
All day’s traffic · 64 cars
Ask the question.
Top-down · theirs
Work happens at query time
Every question re-reads the pile: embed the question, pull the top-k chunks from all transcripts, stuff them in a prompt, hope the answer surfaced. No memory between questions, and a hard context budget on every one.
Illustrative demo data
The cataloguing is the product.
04 · For the skeptic on your team
Structural, not a tuning problem.
Each question maps to a failure mode that no prompt tuning moves. Forward this to whoever built the internal tool.
Counting
Why it breaks top-down
Retrieval isn't enumeration. A top-down tool pulls the handful of passages that look most relevant and summarizes them — it can't count the customers it never retrieved. Ask the same question twice and the number moves.
A pass looks like
One stable, correct count — with the list of accounts behind it — that holds when you re-ask.
Run it for real
Run the same six live — on your data.
We'll take your questions, against your own calls, and show you what organize-then-ask looks like when the answer has to be complete, countable, and cited.