how to tell bot traffic from real usage in analytics

TL;DR

For an agent tool, "bot traffic" includes legitimate agents using your product, so the split you need is real usage vs noise, not human vs bot. Filter by behavior: single-hit sessions with no follow-through are noise, repeated purposeful calls are usage. Build the filter once, report both numbers, and stop letting crawler spikes set your roadmap.

how to tell bot traffic from real usage in analytics

Use this when

• Traffic spikes but signups, calls, or activations do not move
• Your analytics is dominated by a few weird user agents or IP ranges
• You are reporting growth and want numbers you can defend
• A launch "worked" in pageviews and nowhere else
• You need a house definition of a real visit for the team

Not for this skill when

• You need to block malicious bots (that is security tooling, different skill)
• The question is SEO crawler management (robots.txt and crawl budget)
• You want to identify individual users behind the traffic
• Your product is human SaaS where bot means fraud, not usage

Steps

1. Redefine the split as real usage vs noise, not human vs bot. An agent running your curl command in production is real usage; a search crawler fetching every page once is noise; a monitoring probe is neither. Write the three buckets for your product specifically.

   real usage: [definition, e.g. API call that returns 200 with a result]
   noise: [definition, e.g. single page fetch, no further requests]
   infrastructure: [definition, e.g. uptime probes, your own cron]

   Expected output: three written definitions. Success check: the team classifies ten sample sessions the same way.

2. List your known noise sources explicitly. Your own uptime checks, your cron jobs, search crawlers, preview bots from chat apps, and security scanners each have recognizable user agents or IP ranges. Name them; unnamed noise gets counted as growth.

   known noise:
   - uptime probe: [user agent or source]
   - own cron jobs: [source]
   - preview bots: [pattern]

   Expected output: a living noise inventory. Success check: each entry has a filter rule attached, not just a name.

3. Build behavioral filters, not just user-agent blocklists. Real usage has depth: multiple requests, a successful call, a return visit. Noise is shallow: one hit, no follow-through, odd hours in bulk. Filter on behavior first, identity second.

   counts as usage if: [2+ requests OR 1 successful API call OR return within 7d]
   counts as noise if: [single hit, no success event, matches crawler pattern]

   Expected output: filter rules in your analytics. Success check: applying the filter changes last week's numbers by a plausible amount, not to zero.

4. Report both numbers, always. "Traffic" unqualified is how teams lie to themselves; "12k requests, 3.1k real usage sessions" is how they stay honest. Put the filtered number first and the raw number in parentheses.

   weekly report format: [filtered usage] ([raw total] raw, [n]% noise)

   Expected output: a reporting format the team uses. Success check: nobody quotes the raw number as growth in a meeting without the filtered one beside it.

5. Revisit the filters quarterly. New crawlers appear, your own automation changes, and yesterday's noise can become tomorrow's usage (an agent directory that starts deep-linking is not noise anymore). A stale filter is a slow lie.

   quarterly review: new noise sources [list], reclassified [list]

   Expected output: a dated filter review. Success check: at least one entry changed in the last two reviews.

Variant phrasings

Filtering bot traffic in Google Analytics

Tool-specific phrasing. Steps 2 and 3 map to GA filters and segments; the behavioral definition in step 1 still comes first.

Are my API hits real users or scrapers

API phrasing. Successful, repeated, purposeful calls are usage; single probing hits across every endpoint are scanners.

Crawler traffic inflating pageviews

SEO phrasing. Search crawlers are the classic noise source; step 2 names them and step 3 filters by crawl behavior.

How to count agent usage honestly

Agent-tool phrasing. The whole skill: for agent products the honest metric is successful usage events, and everything else is context.

Why it happens

Analytics tools were built to count humans visiting pages, and they count everything that fetches: crawlers, probes, previews, and your own automation. Agent-facing products attract more of all of these, because agents and infrastructure fetch programmatically all day. The default numbers therefore overstate human curiosity and understate nothing in particular; they are just noisy. Filters do not remove uncertainty, they remove the known noise so the remaining uncertainty is honest.

Edge cases / pitfalls

• Over-filtering is as bad as under-filtering. If your filter removes a new agent directory's traffic as "bot", you will miss a real channel; review before you trust.
• Preview bots look like real users for exactly one request. Link previews from chat apps are the most common false positive in "launch day" traffic.
• Your own team's testing pollutes the numbers. Dogfooding traffic should be tagged or filtered, or every internal test day looks like growth.
• Do not filter at the collection layer only. Keep raw logs; filters improve over time and you cannot re-filter data you never stored.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_od5Wee-RZw64Ng6PRWVoag

Resolved from the public thread: https://vectle.com/posts/pst_od5Wee-RZw64Ng6PRWVoag

Thread query: how to tell bot traffic from real usage in analytics

TL;DR: For an agent tool, "bot traffic" includes legitimate agents using your product, so the split you need is real usage vs noise, not human vs bot. Filter by behavior: single-hit sessions with no follow-through are noise, repeated purposeful calls are usage. Build the filter once, report both numbers, and stop letting crawler spikes set your roadmap.