Pre-Sales Research: the visibility check before the first call

01What the prospect sees

Before the call: one sentence, the rest blurred behind a lock. After the call: the full page. Nothing on it is written by a model. The sentences are fixed building blocks with numbers filled in.

Eleven of these pages are live as real rooms in the Superapp (Dominik has the links), and the full data of all 50 companies is in the repo under eval/visibility-check/2026-09-02/. They show what a good page looks like, what a page with nothing to say looks like, and what a stopped page looks like.

02How the page is made

One run per booking, started by the room engine. The stages below run in order. Deterministic steps are grey, the three model gates are gold.

03The three gates, and what they may not do

The model never writes customer text and never changes a number. It corrects inputs, rejects candidates and judges. Every action is applied by code, validated against evidence and logged. Text from other websites and from AI answers counts as untrusted data in every prompt.

Prompts live in tool/reports/check_review.py. Every change of a prompt is a new PROMPT_VERSION, so every number in a report can be tied to the prompt that produced it.

04Rules the eval bought

Each rule exists because a run on a real company produced something a prospect would have caught. They are deterministic; the gates judge on top of them. The full list with the story behind each is in docs/PROSPECT-CHECK.md. The ones you will meet first:

• The search hook needs at least four queries people actually search (volume 10 or more). Otherwise the AI hook takes over; if that does not fire either, there is no hook and the page is not sent.
• A regional firm without a town never buys data. Without a town the queries go national, where only portals rank.
• The prospect's own second domains count as the prospect: gate 2 marks them, and the brand word in the domain finds them too (wiest-autohaeuser.de and wiest-group.de).
• A page without a competitor, or where the shown competitors take under one percent of the searches, is blocked: the page would name the wrong click-takers.
• A manufacturer in the same trade is a competitor; a chain competes with a national or multi-site prospect, not with the local plumber.
• One row per question. "knx smart home kosten" once, not twice.
• A reviewer verdict of fail without a single blocking finding becomes a note. The owner cannot act on a no without a reason.

05The numbers today

The proof set: 50 won customers of Glow Careers from Close, 40 for tuning, 10 held back. Each page was judged by hand against the live data (correct or not, send or not, every shown competitor, every counted AI mention). The gates are then measured against that judgement.

The one false pass is SBS Technik, a crane-service firm from Andernach whose profile came out as national; the national page names crane-rental chains as rivals. The welcome form question "Wo sind Ihre Kunden?" fixes that case: the prospect's answer locks the reach.

Of the 50 pages, 27 pass automatically. Eleven of the rest are correct pages with nothing to show: the firm is found everywhere and named everywhere, so there is no hook. That is a feature. The test set is made of established companies; Omni1's prospects arrive because they have a visibility problem.

The model challenge

Seven models ran on identical prompts, identical cached data and identical gold, each twice. The two models without a false pass reject almost everything (Qwen fails 20 of 39 correct pages, GLM flips every third verdict). Kimi K3 and Claude Sonnet 5 are equally stable; Kimi rejects fewer correct pages and returns cleaner JSON, Sonnet is faster. Kimi K3 stays, Sonnet 5 is the fallback. The full table is in eval/run-1/models.md on the VPS.

05bWhy 27 of 50 pass, company by company

27 of 50 is not "27 good, 23 bad". Most of the 23 are pages that must not go out, and the gates were right to hold them.

Of the 25 tuning pages a person would send, 21 pass automatically. Of the 15 that must not go out, 2 slip through. Start with the four false fails: one code, one prompt sentence, measurable on the cached data in ten minutes.

The same table, with every file behind it, is in the repo under eval/visibility-check/2026-09-02/ (README first). Every company has its profile, raw data, gate records, review rounds and the human judgement there.

06How to tune it

Change one thing, run the eval, read the report, keep or revert. Never tune on a single company.

1. Pick the failure. The report lists every disagreement between the gates and the gold: false passes, false fails, competitor leaks, wrong profiles. Start with the class that has the most cases, not the loudest single case.
2. Decide where it belongs. A term that measures the wrong market is a research problem (code in prospect_check.py). A wrong judgement about an entity or a mention is a prompt problem (check_review.py). A wrong number is a lint problem. Rules catch what is deterministic; prompts judge the grey.
3. Change it, bump the version. Prompt edits change PROMPT_VERSION. Rule edits get a line in docs/PROSPECT-CHECK.md with the company that caused them.
4. Run the eval on the cached data. Nothing is bought again unless the research step changed. Recorded model answers are replayed where the prompt is unchanged, so a lint change costs nothing and a prompt change costs about $3 for all 50.
5. Read the report, then the pages. Metrics first, then rejudge.py for the per-company sheet: kept and dropped terms, the hook, every row, every competitor with its evidence, the verdict.
6. Update the gold when the page changed. If a fix changes what a page shows, the human judgement for that page has to be redone. Gold that describes an old page produces false metrics.
7. Post the before and after in #pre-sales-call-research. One message: what changed, which companies moved, the metric line. Dominik decides deploys.

# on the VPS, in /root/ck-test (venv of the app)
python -m tool.reports.check_eval --subjects subjects.json --out eval/run-1 --stage derive --record --replay-gates --concurrency 8
python -m tool.reports.check_eval --subjects subjects.json --out eval/run-1 --stage review --record --replay-gates --concurrency 8
python -m tool.reports.check_eval --subjects subjects.json --out eval/run-1 --stage report
python rejudge.py eval/run-1 sbs-technik          # one company, the sheet a reviewer reads
python -m tool.test_check_review                   # 100 unit checks, no network, no database

Stages: collect buys data (only after a research change, with --reuse-serp), derive runs gate 2 and the numbers, review runs the lint and gate 3, report writes report.<model>.md and models.md. --model runs any OpenRouter model on the same prompts; --repeat 2 measures how often a verdict flips.

What a good change looks like

• It fixes a class of cases, and the report shows the class shrinking without another class growing.
• It is deterministic where it can be. A rule beats a prompt sentence when the check is arithmetic or string-level.
• It strikes on contradictions, not on missing confirmation. Websites rarely say what a firm does in the words you expect.
• It is written down with the company that caused it.

07Where things live

Deploys go through the two doors described in Two-Door Deploys: frontend without database access, backend with dump and rollback. The eval never touches the production database; tests force SQLite when a Postgres DSN is in the environment.

08Open ends

• Holdout. The ten held-back companies have not been judged yet. They run once with the chosen model, and zero false passes is the bar.
• Profile stability. The profile model changes the core term between runs on identical text (IABG, GIFAS, ATZ). Planned: compute the profile once per site version and let reruns inherit it.
• Reach. The form question fixes it when answered. Without an answer the model's guess stands; do not try to replace it with a text rule, that was measured.
• Duration. p95 is seven minutes against a target of four. Most of it is the SERP purchase and the two review rounds; the customer card promises five to eight minutes.
• Auto-release. Off until Dominik turns it on (OMNI1_CHECK_AUTO_RELEASE). Until then every page goes through the cockpit.