Context
Part of a long engagement on tintadlaplastykow.pl (custom CMS, 36k+ SKU, since 2022) — this thread: cache, performance and bots. A catalog of over 36,000 items (including archival rows still in the DB) stressed TTFB, SQL and RAM. Price scrapers and AI crawlers could hit ~100 req/s and push hosting into 503s — it looked like a store outage.
Scope
- Fragment cache — view-level cache (Symfony Cache) with per-user blocks and invalidation; bots get a static response instead of PHP/DB.
- Cache stampede protection — worker queue for expiring keys; no mass DB queries under traffic.
- Custom product indexer — sorting, pagination and filters for 36k+ SKU without loading the main product table.
- Bot limits — ASN rate limiting, Cloudflare WAF for AI crawlers, honeypots; access-log diagnostics.
- WebP and background jobs — async image conversion; Symfony Console/Messenger for indexes and queues.
- Chunking and idempotency — data batches when syncing 36k SKU; admin and workers safe on duplicate requests.
Result
- The store responds faster with lower CPU/RAM; imports of thousands of SKU do not take the site down, and at worst a bot gets cache.
- Custom CMS at 36k+ SKU — Symfony fragment cache, stampede protection, indexer, WebP, ASN/WAF bots.