Methodology

The Log That Doesn't Log

Vendor observability surfaces carry scope boundaries nobody documents. Five logs that answered confidently about a world they could not see.

A change log is not a record of what changed — it is a product, with a scope. The scope is rarely written down, and the change that matters is often sitting just outside it.

In July 2026 the paid side of an HVAC contractor's account came apart. Search fell from 47.1% of spend to 4.3% while YouTube rose from 5.9% to 92.7% — pre-frame 2026-07-01 to 07-18, post-frame 2026-07-23 to 08-10, read from the Google Ads API (GAQL, v25) on 2026-08-11, with 07-19 to 07-22 excluded because delivery was dark. One configuration change set that in motion, and it does not appear in Google's change-event log. Not because anybody failed to record it: conversion-action changes are not covered by that surface at all. An analyst who queries the log and finds nothing concludes — correctly according to the instrument, wrongly according to the world — that nobody touched the account.

Vendor observability surfaces are products. They have boundaries, sort orders, pagination semantics, and documentation written against a version that may no longer be live — none of it disclosed at the point of use. So before a surface settles a question, it gets interviewed. Four questions. We call it The Scope Interview.

The five instances

#InstrumentWhat it saidWhat was true
1Google Ads change-event logno change was madeconversion-action changes are not covered at all
9wrangler deployments listthe worker is "3 commits behind"byte-for-byte identical — the command prints oldest-first
13A CRM's paged aggregationSigned and Installed = 0unstable default sort, 797 records sharing one import timestamp
14A CRM's published API docsa webhook topic namethe live catalog uses a different name; the documented one is rejected
15A pagination parameter stepped by 1925 rows61 unique — it is an item index, not a page index
The change that sat outside the log's scope An ellipse marks what the change-event log covers — budget edits, schedule edits, status changes. Outside it, a highlighted dot marks the conversion-action edit that collapsed the account, a change the log cannot record at all. The closing callout: silence in a log is not evidence of stillness — a log that returns rows is telling you what it covers, not what happened. THE SCOPE BOUNDARY NOBODY DOCUMENTSwhat the change log coversbudget editsschedule editsstatus changesthe conversion-action editthat collapsed the account —outside the log's scope entirely“Silence in a log is not evidence of stillness.” A log that returns rows is telling youwhat it covers, not what happened. Interview the scope before trusting the surface.
Instances 1, 9, 13, 14 and 15 of the register: the change log, a deployments list that prints oldest-first, an unstable paged aggregation, docs naming a webhook the live catalog rejects, and an item-index pagination parameter.

Question 1 — What class of change does this surface not record?

The action at the centre of the collapse was a phone-tap conversion carrying a flat five-dollar default value, promoted into the bidding signal against a target cost per acquisition high enough to impose no binding constraint on it at all. The Five-Dollar Counter That Google's Change Log Never Recorded carries that arithmetic and the placements it bought.

Everything turns on when the action was promoted, and the change log has no answer. The date came out of the conversion data instead: read as two daily series — counted conversions against all conversions, Google Ads API, 2026-08-11 — the transition from counted-at-zero to counted-equals-all is unambiguous, and it puts the promotion on 2026-07-20.

The log is not broken. On the same account, a change-event query covering 2026-07-19 through 2026-08-18 on the Local Services campaign returned exactly two touchpoints, on 2026-07-20 and on 2026-08-11 — reliable enough to rule out a configuration cause for a separate collapse in that channel. A surface can be authoritative for what it covers and silent about what it does not, and never say which is which.

Nothing here claims the remediation worked. The verification read on that account is scheduled for 2026-08-24.

Question 2 — What order do rows come back in, and does the order survive paging?

Two instances, one mechanism.

A deployment-list command on Cloudflare's Wrangler CLI reported a live worker as "3 commits behind," which reads as a failed deploy. What settled it was a byte-for-byte comparison of the deployed bundle against the build from current source: identical. The command prints oldest-first, so the row at the top of the output — where every other tool in the stack puts the newest item — was the oldest deployment on record. The instrument was accurate; the ordering assumption came from somewhere else.

The second is worse, because the ordering is not merely surprising, it is non-deterministic. A paged aggregation over a storm-restoration CRM's pipeline returned zero at Signed and zero at Installed, for a business that demonstrably had rows in both. The cause was a tie: 797 records shared one bulk-import timestamp, and that is the field the default sort uses. When the sort key ties across a page boundary, ordering between requests is not guaranteed, so page two is not the continuation of page one. Some records come back twice, others never, and an aggregation built by walking pages inherits the gap as a zero — indistinguishable from a stage that is genuinely empty. The repair is dull: append a unique tiebreaker so the order is determined before the first request.

Question 3 — Is the documentation the same artifact as the live catalog?

The same CRM publishes API documentation naming a webhook topic. Subscribing to that exact topic fails; the live catalog carries a different name, differing only in how one word boundary is punctuated. The documentation is not describing a system that never existed. It is describing a system that used to, with no marker to say which.

Two findings from the same window generalize it past one vendor. On the Google Ads API (v25, read 2026-08-11), the field controlling whether a conversion action counts into the reported conversions metric is immutable — a direct mutate returns an immutable-field error, because control moved to the conversion-goal model. Any playbook instructing you to flip that field is not executable as written. And on 2026-08-11 that vendor blocked an older API version as a partial rollout: of three identically shaped calls, one succeeded and two failed, which presents as flakiness rather than deprecation.

Documentation is a hypothesis. The live catalog is the authority, and the check costs one request.

Question 4 — Does the pagination parameter count pages, or items?

A parameter named for pages was stepped by one. The loop returned 925 rows. The distinct count was 61.

The parameter was an item index, so each step advanced the window by one record and consecutive requests overlapped almost entirely. Every failure signal you would look for was absent: no errors, no empty pages, no throttling, a row count rising steadily throughout.

The check that catches this catches most of the family. Run the pagination against a collection whose size you already know from another route, then compare the distinct count — not the row count — against it. If 61 records yield 925 rows, the loop is re-reading; if they yield 40, it is skipping. Ordering and limits belong in the same check: a reprocessor with an oldest-first sort and a fixed row limit re-read the same oldest page every run, and live arithmetic on 2026-08-17 put 35 of 135 rows — rows in an upload queue, not a count of jobs — outside its reach. The Match Key Is the Product follows that one to its end.

What would prove this wrong

  • A Google Ads change-event query that returns a conversion-action edit. Then the surface covers that class, the promotion was recoverable there all along, and this post's claim collapses.
  • A counted-versus-all series dating the promotion to any day other than 2026-07-20. The date is derived, not observed.
  • A deployment-list output whose top row is demonstrably the newest deployment. Then ordering was not the mechanism, and the byte-for-byte match needs another explanation.
  • The CRM aggregation returning correct stage counts on repeated runs with the tie still in place. If the unstable sort does not reproduce, ties are not the cause.
  • The documented webhook topic being accepted by the live API. Then the documentation was current and the rejection has another cause.

Interview the surface before it settles the question

Ask every observability surface what it does not record, what order it returns rows in, whether its documentation matches its live catalog, and what its pagination parameter counts. The runnable version, this week: page through a collection whose size you already know from a second source, and compare distinct records rather than rows. If they disagree, every report built on that loop is provisional — and you found out before your client did.

The next post takes numbers that are not missing, not empty and not wrong — they are answering a different question than the one asked.

  • #blog
  • #measurement
  • #instrumentation
  • #observability
  • #instruments-that-lie
Share X LinkedIn
Build it yourself?

Get the kit, not just the theory.

We'll send the build checklist behind this post — and the next pillar when it ships. One email, no drip sequence. Unsubscribe in one click.

Want this built for you?

Book a discovery call. We'll walk your numbers.

20 minutes. Tell us what's broken, hear what we'd ship in the next 90 days. No pitch deck.