"Not measured" vs 0: why a missing AI engine reading should never look like a real answer

Every score on a dashboard eventually gets treated as ground truth by whoever's looking at it — a founder deciding whether to invest in content, an agency reporting to a client. That only works if the number actually means what it appears to mean. Today we shipped a fix to close a gap in our own product where it didn't: an engine we never successfully probed and an engine we probed 25 times and got zero mentions from were rendering as the exact same thing. This post is the honest account of the gap, the fix, and the one part of it we haven't closed yet.

The gap

Every scan puts 25 buyer questions to four engines — ChatGPT, Claude, Gemini, Perplexity — and scores each one based on whether it named you. That's the mechanism we've written about in detail. But scoring an engine requires actually reaching it: a valid API key, a live endpoint, answers that come back instead of timing out. When one of those four things fails — a key expires, a provider has an outage, a call errors out — the engine that never answered a single question was scored exactly the same way as an engine that answered 25 questions and never once mentioned you: a flat 0.

Those are not the same fact. One means "we have real evidence you're invisible to this engine." The other means "we have no evidence at all." A reader looking at a per-engine grid showing "ChatGPT 61, Claude 58, Gemini 0, Perplexity 55" would reasonably conclude Gemini has a real problem with the brand — when the true story might just be that our Gemini key was dead that day. Rendering "we don't know" as if it were "we checked and the answer is no" is the kind of small dishonesty that quietly poisons trust in every other number on the page, including the ones that are completely real.

What we shipped

We added a real status field to every per-engine score: live, mocked, or failed — computed from whether that engine actually had a valid key and returned surviving results for the scan, not inferred after the fact. It's threaded through the full path a score travels: the scoring engine, the stored history record, and every surface that renders a per-engine number — the dashboard's brand card, the weekly progress view, the report page's per-engine breakdown, and the score-methodology panel that explains the formula. Wherever an engine's status isn't "live," the surface now renders "Not measured" instead of a number. A 0 only ever appears when an engine was genuinely reached, genuinely asked, and genuinely never mentioned you.

It also fixes a second, related lie: comparing scores over time. If last week's scan had a working key for an engine and this week's didn't, the naive comparison would show a dramatic score drop — "you lost 40 points on Claude!" — when nothing about your actual visibility changed; a key just broke. Deltas now skip any comparison where either side of the pair is an unmeasured reading, so a broken key produces an honest gap in the trend, not a fabricated crash.

Old scan history — everything recorded before today — didn't carry this field at all, so we made it optional and treat any record without it as "live." That's the honest assumption for old data: those scans predate the possibility of silent mock/failure disclosure, and prod itself has never quietly shipped a mocked run, so defaulting old rows to "live" doesn't invent anything that wasn't true.

The part we haven't fixed yet

Here's the honest part, because burying it would defeat the point of writing this post at all. Your overall visibility score is still the plain average of all four engine scores — as documented — and that average still folds in a 0 for any unmeasured engine. So even though every individual engine reading is now honestly labeled, a single dead key can still drag your headline number down, exactly the way a genuine zero-mentions engine would. We've fixed the disclosure — you can now see plainly which engines were actually measured — but we haven't yet fixed the arithmetic that produces the one number people look at first.

We're not shipping that fix today because it's a bigger, riskier change than it sounds: excluding unmeasured engines from the average would make every historical score comparison ambiguous (was last month's 61 an average of four engines or three?), and it touches copy we've already published elsewhere describing the formula as a flat four-way average. That's a real product decision, not a quick patch, and we'd rather tell you it's still open than quietly ship something half-considered. It's on the list for a proper look, not swept under the rug.

Why this matters more than one component

We built PingMyBrand because AI answers are a place buyers make decisions with no visible paper trail — you can't see the leak the way you can watch organic traffic dip in Search Console. If the tool that's supposed to make that invisible thing visible starts quietly inventing numbers of its own, it's not fixing the trust problem — it's adding a second one. So when you look at your own report and see "Not measured" on an engine instead of a number, that's not missing data being hidden from you. It's the opposite: it's us telling you exactly where the evidence stops, instead of guessing on your behalf.

See it for yourself: run the free scan and look at the per-engine breakdown on your own report. Every number you see there was actually measured — and now, so is every gap.