Quick Answer: AI crawler analytics tracks which AI bots request your pages, how often they return, which responses they receive, and whether those visits connect to AI search referrals. Browser scripts miss most of this because crawlers don't run your tracking code. Server and CDN logs capture it directly, and a self-hosted tool like GEOInflow can combine those logs with your content inventory.
Most teams working on generative search visibility can't tell you which AI bots touched their site last week. Standard browser analytics won't show them. Crawlers request HTML, JSON, and sitemaps without running your JavaScript, so your dashboard stays quiet while your server handles a steady stream of bot traffic. AI crawler analytics fills that gap by treating server logs as the primary evidence and browser data as one input among several.
Why browser analytics miss most AI crawler activity
Your tracking snippet fires only when a real browser loads the page and runs your script. A crawler that fetches a URL, reads the HTML, and moves on never triggers it. That's why a site can show healthy pageviews while a training crawler has already visited hundreds of its URLs.
This creates a blind spot that's easy to misread. Low browser traffic doesn't mean low AI interest. It may mean the interest arrives as machine requests your analytics was never built to see.
The fix is to add the evidence that browser tools skip. Server logs and CDN logs record every request, including those from bots. Content inventories, such as a sitemap or a CSV of URLs, tell you which pages should be visible in the first place. When you compare the two, you can see which important pages get no observed crawl activity at all.
GEOInflow, an MIT-licensed project published by 猎看科技 and 北京日报社智源GEO, is built around this split. It treats browser tracking, crawl logs, and content inventories as separate inputs. Each one can be connected on its own, and the analysis states its limitations when a source is missing.
Identity, purpose, and verification are three different questions
Most people treat a User-Agent string as proof of who's visiting. That's the first mistake. Any client can send a header that says it's GPTBot or a Google crawler, so a declared name is only a claim.
Serious AI crawler analytics stores three separate signals:
- Declared identity: the User-Agent string the request claims.
- Network match: whether the source IP falls inside the provider's published ranges.
- Verification evidence: DNS lookups or trusted edge signals that tie the request to the provider.
Only the combination supports a confident conclusion. A request with a matching User-Agent but an IP outside published ranges deserves suspicion, not a label. Historical records without verification evidence should stay marked as unknown, not quietly reclassified.
Purpose matters too. The GEOInflow catalog distinguishes training, AI search, and user-triggered requests. A single provider can run several products, so the same company may appear in more than one category. Treating all of its traffic as one thing leads to bad decisions. Blocking a training crawler and blocking a search crawler have different consequences for your visibility.
What a crawl log can and cannot prove
Here's the counter-intuitive part that most guides skip: a crawl log proves access, not outcomes. When a bot requests your pricing page, you know it reached the page. You don't know it indexed the page, trained a model on it, cited it in an answer, or recommended your product.
GEOInflow's documentation makes this explicit: crawl logs establish observed access, and indexing, model training, answer citations, recommendation rankings, and customer acquisition each require their own evidence. Ignoring that boundary is how teams end up reporting "AI visibility" numbers that a client or executive can't defend.
The practical rule is simple. Report crawl volume as crawl volume. Report referral sessions as referral sessions. Report conversions as conversions. Keep the three separate, and state how they relate only when you can show it.
Here's a useful way to classify your evidence:
| Signal | What it shows | What it does not show | Where to look |
|---|---|---|---|
| Declared User-Agent | Which identity the request claims | Whether the claim is true | Server and CDN logs |
| IP range match | The request came from a published provider range | That the bot's purpose is what you assume | Log enrichment against provider ranges |
| DNS or edge verification | Strong support for the claimed identity | Whether the page was indexed | Verification fields in log ingestion |
| Repeated page requests | Which URLs a crawler returns to | Whether those URLs appear in answers | Page rankings and repeat-request views |
| AI referral session | A visitor arrived from a supported AI search referrer | That the visit came from a crawl | Browser tracking with landing pages |
Finding the important pages AI crawlers never reach
Crawl volume alone won't tell you whether your site is working. The more useful question is whether the pages that matter get requested at all.
Import your content inventory, whether that's a sitemap, a CSV, or a JSON export, and compare it against observed logs. The gap list is where the work starts. A product page with zero observed crawl activity over a long window is a signal worth investigating. Check whether it's linked from anywhere a crawler can reach, whether robots rules block it, and whether it returns a clean response.
Here's a short process that works for most teams:
- Export the URLs you want AI systems to read, such as product, documentation, and pricing pages.
- Compare that list against the paths that appear in your crawl logs over the same date range.
- Sort the missing pages by business value, not by count.
- Check internal links, sitemap inclusion, and robots directives for the top gaps.
- Recheck after a defined window, using the same data scope so the comparison is fair.
Keep the data scope consistent. Comparing a 30-day window against a 7-day window will make a healthy site look broken or a broken site look fine. Date-range discipline matters more than any single chart.
Diagnosing failed requests and blocked access
Crawler failures are often invisible until a coverage drop shows up weeks later. Status codes tell you what the server returned. A cluster of 403 or 5xx responses to a specific bot can point to a firewall rule, a rate limit, or an origin error that only affects one user agent.
Separate the causes before you act. A 403 might be a deliberate block you configured in robots rules or an Nginx policy. It might also be a CDN challenge that catches bots you wanted to allow. Those need different fixes.
GEOInflow exposes HTTP status, GET and HEAD methods, caching behavior, latency, and error patterns in separate views. That separation matters. Teams can record handling status, notes, and deferrals, then review the relevant time window with the evidence in front of them.
Access policy changes carry their own risk. The tool supports exporting robots and Nginx configuration. Cloudflare controls, according to the project documentation, require separate Zone credentials, a preview, explicit confirmation, background execution, and a read-back of the remote state. That's the right design. Pushing a blanket block to a CDN without a preview can remove your site from the very systems you're trying to influence.
Self-hosted or hosted: the trade-offs
A hosted analytics product is quicker to start. A self-hosted one keeps your raw logs under your control, which matters if those logs include IP addresses, paths with user identifiers, or internal URLs you'd rather not send to a third party.
Self-hosting has real costs, and most comparisons understate them. You'll run PostgreSQL and Redis, manage backups, maintain a reverse proxy, and handle upgrades. The GEOInflow deployment script generates secrets, builds the application image, runs migrations, and starts the application, log worker, analytics worker, PostgreSQL, and Redis. It needs Docker Engine and Docker Compose V2, and production access needs a domain, HTTPS, and a trusted reverse proxy.
Here's the trade-off in plain terms. If you have a single marketing site and no one to run infrastructure, a hosted tool is likely the better choice. If you manage several sites, need audit-level control over raw logs, or handle data that your compliance team won't let leave your network, self-hosting becomes the practical option.
One more nuance: the project states that core website analytics and crawler identification can run without a model API key. AI analysis is disabled by default, and when you enable it, you choose the model connection and the website materials sent to it. That default matters if your legal team reviews data flows.
Connecting crawl activity to AI referral traffic
Crawler data and referral data answer different questions, and they should stay separate until you have evidence linking them. A crawl tells you a bot read a page. A referral session tells you a person arrived from an AI search product. Both can be useful, but the sequence doesn't prove cause.
Browser tracking can recognize supported AI search referrals and tie them to landing pages, events, goals, and conversions. Over time, you can ask whether pages that gained crawl activity later attracted more AI referral sessions. That's a hypothesis worth testing, not a conclusion to publish.
Keep the time windows aligned, note the sample size, and flag any change you made to content or access rules in the same period. Most of the apparent wins in this space come from changes nobody logged.
Frequently Asked Questions
What is AI crawler analytics?
AI crawler analytics measures which AI bots request your website, which pages they visit, how often they return, and what HTTP responses they receive. It relies mainly on server and CDN logs, because crawlers usually don't run browser tracking scripts. It answers access questions, not ranking or citation questions.
How do I see which AI crawlers visit my site?
Start with your server or CDN access logs and look at the User-Agent field, then verify the claim. Compare the source IP against the provider's published ranges and use DNS or edge verification where available. A User-Agent string alone isn't enough, because any client can claim to be a known crawler.
Why does my analytics show no AI traffic if bots are visiting?
Most browser analytics tools only record visits where the tracking script runs in a real browser. Crawlers request pages without executing that script, so they never appear in the dashboard. You need server-side log collection to see them.
When should I block AI crawlers in robots.txt?
Block a crawler when you have a clear reason, such as a licensing position on training use or a server-load problem you've documented. Before you block, identify its purpose, since training and search crawlers have different effects on your visibility. Preview any change and recheck the logs afterward to confirm it took effect.
Does a crawl mean my content will appear in AI answers?
No. A crawl proves the bot requested the page, not that it indexed the content, used it in training, or cited it in an answer. Treat crawl data as evidence of access and measure answer appearances and referrals separately.
Crawl logs are the most direct evidence you have about AI bot behavior, and most sites are underusing them. Start this week by exporting your ten most important URLs and checking which ones show no crawl activity in your last 30 days of logs. Pass this to whoever owns your server logs, and ask them to confirm the verification fields are being captured before you draw any conclusions.