Measuring Adoption You Cannot See

The measurement gap

Classic developer marketing measurement assumes a visible journey: a person visits pages, analytics scripts fire, campaigns get credit, and the funnel reconciles. AI-mediated adoption breaks every link in that chain:

• AI crawlers and assistant fetchers do not execute your JavaScript beacons, so that traffic is invisible to script-based analytics.
• Agentic browsers do run JavaScript, but their sessions land in your analytics indistinguishable from humans, inflating the "human" bucket instead.
• Answers are consumed inside chat interfaces; the developer may never load your site before signing up.
• When they do arrive, they arrive directly at a signup or install command, stripped of referrer context.
• The influence often happened weeks earlier, inside a model's training data, where no attribution system reaches.

The result: teams see flat or declining docs traffic while API signups hold or grow, then misread it as a content problem. The traffic did not disappear; it moved to readers your analytics cannot see.

The blocking decision is per class of machine traffic, not one switch:

• Training crawlers feed future model weights. Blocking them is a slow-compounding opt-out of the funnel's first stage, affordable only if your category's training data already leans your way.
• Retrieval and assistant fetchers are answering a live developer question right now. Blocking them turns real evaluations away at the door and is almost never worth the saved bandwidth.
• Agentic browsers ride full browser sessions. You cannot block them without blocking humans, so plan for measurement contamination instead.

What you can measure today

None of these are precise. Together they form a usable picture:

1. AI traffic in server logs. Script-based analytics miss AI crawlers and fetchers, but your server logs do not. Count requests by known AI user agents (model-provider crawlers, assistant fetchers, agent frameworks) and track the trend, not the absolute number. Watch which pages they fetch most; that is your machine-readership ranking.
2. Machine-surface fetches. Requests for /llms.txt, /llms-full.txt, per-page Markdown endpoints, and your OpenAPI spec are almost purely non-human signals, which makes them unusually clean indicators.
3. Dark-funnel signups. Accounts created with no prior web session from that identity. This bucket always existed; its growth rate is your best single proxy for chat-mediated adoption. Agentic-browser evaluations can create a prior session, so treat it as a floor, not an exact count.
4. Self-reported attribution. A free-text "how did you hear about us" on signup, actually read by a human monthly. Developers say "asked Claude", "ChatGPT suggested it", "my coding agent used it" with surprising frequency, and the phrasing tells you which surface did the work.
5. Support and community tells. Questions that quote model-generated code, or reference instructions you never wrote, indicate models are onboarding people onto your product, correctly or not.

Leading indicators worth tracking

The lagging metrics above confirm what already happened. The leading ones tell you where answers are forming:

1. Presence in model answers. Monthly, run a fixed set of category prompts ("best way to do X", "compare tools for Y") across the major assistants, with search enabled and disabled. Record whether you appear, how accurately, and what gets cited. Keep the prompt set stable so the trend means something.
2. Accuracy of model answers about your product. Ask the assistants your top ten real integration questions and grade the answers. Wrong answers are a docs bug with a distribution channel; fix the source text they retrieve, then re-test.
3. MCP server installs and tool-call volume, if you ship one, per MCP servers as the new SDK.
4. Agent-framework and registry integrations. Being packaged into agent toolkits and client registries is distribution that compounds ahead of any traffic you can see.

Reporting honestly

The credibility rules from classic DevRel measurement apply double here, because every number is soft:

• Report direction and mix, not false precision. "Machine fetches up 40 percent quarter over quarter, dark-funnel signups now a third of new accounts" is honest; a decimal-pointed AI-attribution percentage is fiction.
• Present it as triangulation: several weak signals agreeing beat one fabricated strong one.
• Keep definitions stable and written down, the same discipline as any goal-setting framework.
• Say what you cannot know. Executives handle "attribution here is directional" better than they handle discovering it later.

A minimal instrumentation plan

Shippable in a week, without new vendors:

1. Add a log-based count of AI user agents and machine-surface fetches to whatever dashboard you already run.
2. Add the free-text attribution field to signup and calendar a monthly reading of it.
3. Define the dark-funnel signup segment in your existing analytics.
4. Write the fixed prompt set for presence testing and run it for the first time.
5. Put a quarterly reminder on reviewing your crawler policy, so blocking decisions stay deliberate.

Then leave the definitions alone for two quarters. Trend lines need time to become evidence, and this corner of measurement changes fast enough without self-inflicted definition drift.

Related

• Agents are your new developers for the funnel these instruments observe
• Writing docs LLMs can use for improving what the machines retrieve
• Goals planning with OKRs for wrapping these metrics in accountable goals