Five Answer Rates, One Month, All Correct

Five measurements of the same month returned five different answer rates. All five were correct — and no two of them could be compared.

An operational number is not a fact. It is the output of three choices — the population you drew from, the formula you ran, and the window you fixed — and changing any one of them changes the answer without making the previous answer wrong. Between June and August 2026 we measured inbound phone traffic at a commercial property-and-casualty brokerage while building a baseline that a pass/fail gate could read. June alone produced five defensible populations, ranging from 1,160 records to 2,241, and answer rates from 24.3% on one channel to 59% on another. Every one of those numbers survives an audit. Any comparison across them does not.

The discipline that keeps them apart has three parts, and we call it Population, Formula, Window: name the population, pin the formula in code rather than prose, and fix the window before you look at the result. The corollary matters more than the steps. When the formula changes, the old numbers do not become wrong. They become a different measurement, and both stay on the record.

The five populations, all June 2026

Population	In scope	Answered live
Analytics, queue level — the labelled client anchor	1,160 calls	680 (59%)
Call log, main line, business hours, formula v1	1,385 calls	660 (47.7%)
Call log, main line, business hours, formula v2	1,385 calls	49.1%
Account-wide inbound records, every channel, every hour	2,241 records across 20,851 legs	not computed at this scope
Split by channel — main line vs. agent direct lines	1,489 vs. 752	45.7% vs. 24.3%

The queue-level population also records 480 calls (41%) not answered live, 126 (11%) reaching voicemail, and roughly 354 (about 30%) that reached nobody and left nothing behind. The main-line call-log population for the same month, under formula v1, records 195 voicemails (14.1%) and 530 calls (38.3%) with no answer and no voicemail.

Two of those rows reconcile exactly, which is the only reason the rest can be trusted. The main-line channel count of 1,489 for June 2026 decomposes into 1,385 in-scope business-hours calls, 68 after-hours calls and 36 weekend calls. The arithmetic closes with nothing left over. A population that cannot be reconciled to a wider one is a number somebody typed.

Step 1 — Name the population, then live with what it says

The pinned formula v1 is a set of filters, not a description: destination is the main line, Monday to Friday, 09:00 to 18:00 Central, one call counted per session identifier, internal test numbers subtracted. Applied to June 2026, it returns a 47.7% live-answer rate. The independently audited queue-level baseline for the same month returns 59%.

That is roughly eleven points of difference between two correct numbers, and the more rigorous instrument is the one that reads worse.

Most agencies resolve that by quietly adopting whichever number flatters. Here the harder number is the one wired to the gate. The call-log main-line population is what the pass/fail check reads, in both directions — baseline and post-change window — and the softer queue-level figure stays as the client-facing June anchor because it is the audited baseline of record for a different population, and because it is labelled as one every time it appears. The reason is written into the record rather than left as a habit:

The audited June baseline of record is an Analytics queue-level population and stays the client-facing June anchor, labeled June. The gate compares this formula's post-change window against this formula's baseline; the two populations are never mixed in one comparison.

That last clause is the whole guard. Neither number is being hidden. Neither is being promoted. They are being kept in separate columns, and the one that decides whether work ships is the unflattering one.

Step 2 — Pin the formula in code, not prose, and expect to rewrite it

Formula v1 was wrong, and the way it was wrong is the most useful thing in this post.

The formula had to decide, for each call, whether a human answered it or the automated assistant did. It tested the assistant lane first, and that test won unconditionally — if the assistant appeared anywhere in the call's legs, the call was scored as assistant-answered, regardless of what happened next. On a specific record from 2026-08-07, an agent answered the call and stayed on it for 251 seconds. Formula v1 scored that call as answered by the automated assistant.

Nothing about that failure is visible in the output. The rate it produced was plausible, stable month over month, and consistent with the rest of the record. It was also systematically undercounting the exact event the whole measurement exists to count. The 251-second call turns up again in our Instruments That Lie series, where it represents the family in which a number is quietly a different number.

Formula v2 inverts the test. A human answer is now marked by a "call connected" leg on an extension that is neither the assistant nor any of the sixteen queue identifiers — a positive assertion about a human endpoint rather than an absence of the assistant. Recomputed under v2 on the same population and the same windows, June 2026 reads 49.1% and July 2026 reads 56.0%, against 47.7% and 54.3% under v1. Formula v1 is retired for gate reads. Formula v2 is pinned. Both sets of numbers stay on the record, labelled by formula, because deleting v1's readings would destroy the ability to audit anything that was decided while v1 was live.

A single day reads 70.0% under v2 on 2026-08-07. That is one day, in one population, under one formula. It is in the record and it is not a result.

Step 3 — Fix the window before you look, and re-derive the bar when the formula moves

This is where the rewrite nearly did damage instead of good.

Under formula v1, a proposed reordering of the call flow was projected at 54.0% against a pinned bar of 54.3%. It failed by three tenths of a point, and the change did not ship. That is a gate doing its job — until you notice that the formula producing both the projection and the bar was mislabelling successful human answers as assistant answers.

Correcting the labelling moves the reading up by roughly ten points. Leave the bar pinned at its v1-derived value and every subsequent read clears it automatically, forever, with no operational change required at all. The line we wrote down at the time: the gate does not get harder; it stops being a gate.

So the bar moves with the formula. A threshold derived under v1 cannot adjudicate a reading produced under v2, for the same reason a June queue-level percentage cannot adjudicate a July main-line one. Both halves of a comparison have to come out of the same instrument or the comparison is decorative.

The monitor that was watching nothing

The same change retired a monitoring task rather than repointing it. Left alone, it would have kept exiting cleanly with a "window complete" status against a window definition that no longer described anything the gate used — a healthy monitor watching nothing, reporting green indefinitely.

A monitor that cannot fail is worse than no monitor, because somebody eventually cites its green as evidence. Retiring it was cheaper than repairing it and more honest than leaving it up.

The channel the gate cannot see

Roughly 30% of inbound calls never touch the main line at all. In June 2026, 752 records arrived on agent direct dial-in numbers against 1,489 on the main line; in July 2026, 586 sessions arrived on direct lines, about 29% of account-wide inbound. Those direct lines answer far worse than the main line — 24.3% against 45.7% in June, 23.9% against 51.2% in July.

The pinned formula does not see any of it, by design. Folding direct-line traffic into the gate's population would change the denominator mid-engagement and make every prior reading incomparable, which is precisely the failure this framework exists to prevent. So the direct-line answer rate is tracked as its own population, on its own record, and is explicitly barred from the pinned formula. It is a real problem and it is a separate measurement.

The other structural finding sits inside the pinned population. The 12:00 hour is simultaneously the highest-volume and the worst-performing hour of the day, losing 74–75% of its calls. In the business-hours main-line population, the lunch slice recorded 216 in-scope calls in June 2026 and 168 in July 2026, with zero answered live in either month. Two consecutive months of a literal zero, in the busiest hour, is the single largest lever in the data — and it was invisible in every blended figure above.

What this post does not claim

Phone work was underway across this window. This post makes no claim about whether it worked, in either direction. The month-over-month movement in the main-line population under formula v2 — 49.1% in June 2026, 56.0% in July 2026 — is a change in a number, measured the same way both times. Attributing it to any specific change requires a fixed window, a pinned formula and a bar derived under that formula, and that read is a separate document with its own date. Nothing here is that read.

What would prove this wrong

• An independent recomputation of June 2026 from the same call-log export, under the pinned v2 filters, that does not return 49.1%. Then the formula is not deterministic and pinning it in code bought nothing.
• A routing record showing the automated assistant did handle the 251-second call on 2026-08-07. Then v1 was labelling correctly, v2 introduced the error, and the retirement was backwards.
• The main-line channel count for June 2026 failing to decompose into 1,385 + 68 + 36. The reconciliation is the only external check on the in-scope filter; if it does not close, the population is unverified.
• Direct dial-in lines turning out to be a marginal share of inbound rather than roughly 30%. Then the invisible channel is a footnote, not a structural gap, and excluding it costs nothing.
• Two of these five populations reconciling to the same denominator. If they do, the guard is unnecessary and this framework is overhead.

Three things stapled to every number

Every operational number needs three things stapled to it before anyone is allowed to compare it to anything: the population, the formula, and the window. If a report in your account shows a rate improving month over month, ask whether the denominator was the same both times — and whether the formula that produced it changed in between. That question takes ten minutes and occasionally invalidates a quarter.