Projects

Open Source

Wiki Scraper Newsletter

Go
Nuxt
Scraping
Docker
Open Source

Public full-stack application example — Wikipedia scraper (chromedp), Nuxt 4 panel, page filters, Basic/Premium subscriptions, example payment, and e-mail notifications.

browser Nuxt 4 /dashboard/ Nginx :8080 Go API Gin · JWT · Redis Go scraper chromedp · wikipedia.pl Go notify cmd/notify · email PostgreSQL shared by API · scraper · notify Redis RabbitMQ MailHog (API + notify) filters: Basic 3 · Premium 10 POST /order/ · example PSP page_notification → email Docker Compose · each worker reads/writes PostgreSQL Wiki Scraper Newsletter Nuxt 4 · Go · PostgreSQL · RabbitMQ

About the project

A public example of a Wikipedia scraping application with a user panel, page filters, and a subscription model. Built as an adaptation of the author’s private repository (newsletter-scraper) for the Wikipedia Page domain — the architecture was preserved (Docker + Go API + scraper + Nuxt + panel), while models and UI were rewritten for Wikipedia pages.

Repository: github.com/kopolot/example-wikipedia-scraper

Architecture

Browser (Nuxt 4) → Nginx :8080 → Go API
                                 → Go scraper (chromedp)
                                 → Go notify (cmd/notify)
         PostgreSQL · Redis · RabbitMQ · MailHog

In development, everything sits behind a single reverse proxy (localhost:8080): dashboard, API, MailHog, Adminer, RabbitMQ management.

Stack

Layer Technologies
Backend Go 1.25, Gin, GORM, golang-migrate, JWT, idempotency (Redis)
Scraper chromedp, browser pool, proxy per-site, retry on 403/rate-limit
Frontend Nuxt 4, Vue 3, i18n PL/EN
Infra Docker Compose, Nginx, PostgreSQL 17, RabbitMQ, MailHog

Key elements

Scraper

The worker saves new pages with notified = false. JS scrapers in app/resource/scraper/js/, configuration sites_to_scrape (default wikipedia.pl).

User panel

Registration → e-mail verification (MailHog) → JWT login. Filter CRUD on /panel/wanted_filters — requires an active subscription.

Plan Filter limit
Basic 3
Premium 10

Subscriptions and payments

Flow: panel → plan selection → POST /order/ → redirect to example payment (stub PSP) → RabbitMQ extends subscription_level and subscription_expiration.

Notifier (cmd/notify)

Matches filters of users with an active subscription against new pages, page_notification queue, HTML e-mails (PageNotificationEmail).

Quality and origin

Migrations 000001–000009 (users, pages, filters, orders, Basic/Premium seed). The project was built iteratively with Cursor — diff between repositories, DTO/migration adaptation, deliberate omission of elements outside the domain (Grafana, production PSP).