llms.txt Implementation: The Part Nobody Measures

My robots.txt has a line that says AI search and citation crawlers are “explicitly welcomed.” I wrote that in July, four months after I’d already shipped an llms.txt file to this exact site. In that time I have not been able to confirm that a single one of those crawlers has ever opened it.

That is not the story you get from most llms.txt guides. Search “llms.txt implementation” and you get a dozen “Complete 2026 Guide” posts explaining the syntax, telling you to keep it to 20-50 curated links instead of dumping your sitemap, and listing which companies now publish one. All correct. All missing the part that actually matters once you have shipped it: how do you know it did anything?

What I actually shipped, and when

I added llms.txt to veheria.tech back in March, before I had more than two blog posts to point at (PR #63). It sat there as a curated summary: who I am, what I do, how to reach me. In July, once the AI-native engineering wedge posts started shipping, I updated it to list them explicitly (PR #84), each with a one-line description of what the post actually covers, not just a URL:

## Writing

- [Claude Code for iOS Development: The Workflow, Not the Setup](https://veheria.tech/blog/claude-code-ios-development/): where an AI agent is trusted in a real iOS team's loop, and where a human keeps the call.
- [A Claude Code Review Workflow That Saved 30 Minutes Per Engineer a Day](https://veheria.tech/blog/claude-code-code-review-workflow/): what to automate, what to keep human, and the one number measured after months of daily use.

Three posts, not twenty sitemap URLs. That was a deliberate call, not laziness. It is the same rule I run my code-review agents on: every finding has to cite a file, a line, and a concrete failure scenario, or it gets dropped. A link with no reason for a model to follow it is noise, whether it comes from an agent’s review comment or from a sitemap dump into a file that is supposed to be curated. The entire pitch of llms.txt over a sitemap is that a human decided what matters. Paste the sitemap in and you have thrown that advantage away to save five minutes.

I left more out than I put in. The full site has language variants in English, Spanish, and Ukrainian; only the English canonical URLs made the file, because an assistant summarizing my work does not need three copies of the same claim. It has a contact page, a work page, and an about page; only /work earned a line, because that is the one page where the outcome of reading this file (someone reaching out) actually happens, and a model deciding whether to surface me is more useful with one clear next step than three competing ones. Every entry I kept had to answer “what does a model do with this that it could not do with the URL alone,” the same test I apply to a code comment before I let it survive review. Most links fail that test. That is exactly why most llms.txt files, mine included at first draft, start too long and get cut, not the other way around.

The measurement problem nobody in the guides admits

Here is where it gets uncomfortable. I already run SEO on this site the way I run everything else here, headless, through scripts that pull Google Search Console and GA4 on a schedule. That pipeline can tell me, to the click, when a page gets indexed and what query it shows up for. It cannot tell me whether GPTBot, ClaudeBot, or PerplexityBot ever fetched /llms.txt.

The reasons are structural, not a gap in my scripts. GA4 fires on JavaScript execution in a browser; a crawler doing a raw HTTP fetch of a static Markdown file never runs a line of JS, so it never reports. Cloudflare Pages, on the free tier this site runs on, does not expose raw request logs, only aggregate analytics that do not break out user agents at the path level. The companies the guides cite as proof llms.txt “works”, Anthropic, Stripe, Vercel, Cloudflare, all run their own infrastructure with real access logs. A solo site on a hosting tier built for cheap static delivery does not get that visibility for free.

So the honest state of my llms.txt, four months in, is: it costs nothing, it might be read, and I have no instrument that would tell me either way. Before writing this I checked the other properties I run content or SEO for, not just this one. Most of them already ship an llms.txt. None of them, this site included, has any way to confirm a crawler opened one. Every guide I found treats adoption as the finish line. Adoption is the easy 90% of the work. The other 10%, knowing if it did anything, is the part that would actually justify spending more time on it, and nowhere in what I run had anyone, including me until an hour ago, gotten past the first 90%.

The mistake, and what I’d do differently

I have made this exact mistake before, on a bigger stack. When I put Claude Code agents into a multinational iOS team’s review loop, I deployed first and picked the metric of record after, which meant the first stretch of tuning was against a feeling instead of a number. llms.txt is the same failure in miniature: I shipped the artifact and treated “it’s live” as the win condition, without deciding up front how I would ever know if it earned its keep.

What I’d do differently is treat instrumentation as part of the ship, not a follow-up. Search Console will not help here; it reports on Googlebot’s own crawl, not on GPTBot, ClaudeBot, or PerplexityBot fetching a static file. The version I actually plan to try next: a Cloudflare Pages Function in front of the /llms.txt route that logs the User-Agent header on every request before serving the file. Cheap to build, and it is the difference between “I hope a crawler reads this” and a line in a log that says one did. I did not build that when I shipped the file in March, or when I updated it in July. That is the mistake, not the file itself.

Should you still ship it

Yes, and the guides are right about that much. Adoption cost is minutes, the downside is zero, and being the entry that shows up when an assistant is asked to summarize your site is worth having even if you cannot yet prove the mechanism. What I’d change about how everyone, myself included, talks about it: stop writing as if shipping the file were the result. It is the setup. The result is whatever you can actually verify happened after, and right now, for anyone without an enterprise logging stack, that verification does not exist by default. Say that part out loud instead of implying the file is doing invisible work you cannot see.

This is the same judgment call that runs through most of what I write about putting AI agents into a real engineering workflow: the interesting question is never whether the tool is worth adopting, it almost always is. The interesting question is whether you built a way to tell if it is working, before you started telling other people to adopt it too.

If you are deciding where to spend engineering time on AI-visibility work versus everything else competing for the sprint, that kind of “does this actually move something, or is it cargo cult” call is exactly what I help teams work through in a strategy session.

If you have shipped llms.txt on your own site, have you found any way to confirm a crawler actually read it? I would like to know I am not the only one flying blind on this.