Bots in e-commerce — from price scrapers to AI crawlers Cloudflare can't stop

Over the past few years the nature of bots I encounter on e-commerce sites has shifted significantly. The first case was straightforward: a price bot hammered a shop so hard it took the hosting down. These days I increasingly run into traffic that Cloudflare passes without hesitation — either because the bot openly declares itself an AI crawler, or the opposite: it says nothing, has a perfectly normal browser fingerprint, and arrives through a residential proxy.

Several cases from practice, and how to spot them.

---

Case 1: the price bot that took down a shop

A custom CMS shop with tens of thousands of SKUs. One day the site starts falling apart — hosting barely responds, admin panel unresponsive, 503s everywhere. I open the access logs.

One user agent, one IP range from a large cloud provider’s ASN, a perfectly steady rhythm — around 100 requests per second. The bot was hitting product pages and category pages with no rate limiting, no backoff. It either didn’t know or didn’t care that PHP, a database, and a cache sat on the other side and couldn’t keep up with that volume.

Traffic characteristics

• User agent explicitly declaring itself a scraper or price aggregator.
• Constant, metronome-like request rhythm — nothing human about it.
• All IPs from a single cloud ASN.
• No cookies, no JavaScript execution.
• Only GET requests to products and categories — never touches any form.

What worked

ASN-level rate limiting and a user agent block in Cloudflare. Going forward: server-side caching, so that even if a bot gets through, it receives a static response instead of hitting PHP and the database.

---

A recurring problem: AI crawlers and bots Cloudflare doesn’t catch

I’ve seen this pattern several times — on different shops, at different points in time, with varying intensity. Each time it looked slightly different on the surface, but the mechanism was the same. And every time, standard WAF rules didn’t react.

Variant A: bots that openly admit they’re AI

Some of them are transparent about it — GPTBot, ClaudeBot, Amazonbot, meta-externalagent, Bytespider, and dozens of others declare in their user agent that they’re collecting training data. The problem is that even these “polite” bots frequently ignore robots.txt, don’t respect Crawl-delay, and can generate enough volume to stress a server — especially on shops with large product catalogs.

Identification is straightforward (user agent), and so is blocking. A Cloudflare WAF rule or robots.txt entries for those that respect it. But this is an increasingly small part of the problem.

Variant B: bots impersonating real browsers

This is where it gets difficult. Over time I’ve run into this pattern on several different shops, each time at a different intensity — once it was a clear, concentrated burst over a few hours; another time a persistent, low-level hum over weeks that I only noticed while digging through logs for something else entirely.

The common fingerprint looked like this:

• Normal TLS fingerprint — indistinguishable from Chrome.
• Executes JavaScript, handles cookies, passes Cloudflare challenges without issue.
• IPs from residential proxies — geolocation shows Poland or Germany, a local ISP, nothing suspicious.
• Low frequency per IP (a few to a dozen requests per minute), but backed by a pool of hundreds of rotating addresses.
• Total volume significant, distributed in a way that per-IP rules don’t fire at all.

One of these cases was particularly persistent: a specific IP was hitting so systematically that I had to block it manually — Cloudflare with the interactive challenge turned on did nothing. The traffic had the fingerprint of a normal browser, but the timing and navigation pattern were clearly machine-generated. Within minutes of the manual block, new IPs appeared from the same ASN — the classic sign of an operator with a rotating address pool.

In another case the intensity was much lower — a few dozen requests per hour from dozens of different addresses — but the pattern was identical: always the same endpoints, always without accompanying resources, no trace of human navigation.

How to spot it when the WAF stays silent

Signals that give useful information even with residential proxies and headless browsers:

• Navigation pattern — only direct URLs to products or API endpoints, none of the paths typical of a human session (home → category → product). Humans wander; bots go straight to the target.
• Timing — regular, predictable intervals between requests within a session. Human rhythm is irregular; bots have a scheduler.
• No accompanying resources — a real user fetches HTML, CSS, JS, fonts, images. A bot often takes only HTML or just an API endpoint.
• Accept-Language and related headers — headless browsers in their default configuration have characteristic values; modern bots spoof them, but not always perfectly.
• Honeypot links — a link hidden via CSS that’s invisible to users. If anything requests it, it’s not a human.
• Resource ratio — a text/html to CSS/JS/image request ratio well below normal is a signal.
• IP rotation with identical behavior — same user agent, different IPs, identical request pattern. Aggregating by user agent rather than IP makes this immediately visible.

Why Cloudflare doesn’t stop it

Cloudflare’s heuristics rely on signals that advanced bots are increasingly good at faking: TLS fingerprint, JavaScript execution, cookie handling, human-ish timing. Residential proxies compound the problem — the IP looks like a real user in Poland, not a datacenter in Amsterdam. The bot score can be entirely useless against a well-configured bot operator.

---

How I approach it

There’s no single rule that stops everything. Several layers together make a significant difference:

1. Access logs as the foundation — user agent, timing, ratio of HTML to accompanying resources, ASN range. Cloudflare is not a substitute for logs.
2. Block known AI crawlers by user agent — the list is long and growing; I maintain a WAF rule with an updated list.
3. Rate limiting by ASN, not just by IP — with a rotating IP pool, per-IP rate limiting does nothing; per-ASN or per-subnet already responds.
4. Server-side cache — if a bot gets a response anyway, let it come from cache rather than PHP and the database.
5. Honeypot links — cheap, effective, zero false positives for real users.
6. Manual block and post-incident monitoring — with an aggressive operator, sometimes no automation responds fast enough. I block the ASN and watch where the next IPs from that range come from.