Soak-testing API + scraper + notifier — validating a pipeline short CI never sees

I build systems where value does not stop at a synchronous HTTP response. A typical layout has three layers: API (the user-facing contract), scraper (continuous fetch and normalization from external sources, writes to the database, signals to observability), and notifier (matching new records to preferences and delivering notifications).

Each service has its own lifecycle, dependencies and failure modes. Business meaning only appears in the chain: new data → match → notification, while the API stays available. Unit, integration and build checks in CI protect against code regressions — but they do not answer whether that chain holds for hours under load and against live external sources.

Below I describe how I design and run endurance / soak for this architecture — without diving into product-domain details.

The architectural problem

In an API + background worker(s) + async notifications setup, risks show up that second-scale tests never see:

1. “Alive container” vs “alive business process” — the scraper health check can be green while the scrape loop is not feeding the pipeline.
2. Notifier asynchrony — an API success (creating a filter) does not guarantee an email; it depends on fresh scraper data and queues.
3. State shared across runs — the same database and queues as the developer environment distort SLOs (false write errors, leftover messages, “spent” records already marked processed).
4. Parallel user flows — many virtual users on one API at once expose cross-scenario dependencies (e.g. account → subscription → privileged action).
5. External degradation vs internal failure — an unavailable external source must not be the same signal as a write error to our own database or a growing DLQ.

The harness I build aims for a clear verdict on full-stack stability, with priority on the data pipeline — and deliberately not as a hard CI gate (time, CPU, outbound traffic, flaky live nature).

Test architecture (not just “a longer E2E”)

The Go harness runs two parallel tracks. Runtime (scraper → database → notifier → mail, plus API) and the soak overlay that observes it and drives E2E:

Pipeline observer (primary SLO)

Periodic poll (tens of seconds):

• scraper health and per-source status (including circuit breaker),
• events from logs/metrics: heartbeat, rate limit, external unavailability, batch-save errors,
• database backlog (records waiting for the notifier),
• broker queue depth (retry, DLQ).

This is the verdict priority. If the pipeline fails, the soak is a fail — regardless of whether the API still returns 200.

API E2E workers (secondary SLO)

N virtual users run weighted HTTP scenarios (no UI):

• account onboarding,
• subscription activation (test payment),
• preference CRUD,
• resource browsing,
• pipeline_notify — preference → wait for email notification.

The last scenario is a three-service integration test, not an API-only test. That is why it has separate rules (warmup, longer time horizon) and must not dominate short smoke runs.

Dual SLOs (pipeline / API) let you read the result like a production incident: where did it break — ingest, queues, or the user path?

Problems: causes, consequences, countermeasures

1. Isolating the soak environment from local dev

Cause: running soak on the same Compose project as local development shares volumes, networks and often how the scraper starts (manual process vs dedicated container).

Consequences: the verdict mixes “code quality” with “developer machine history”.

Countermeasure: a separate Compose project, its own volumes, subnet, host port mapping, and a default volume reset before the run. Migrations provide schema and reference data; user state is created during E2E.

2. Preflight as part of the test contract

Cause: docker compose up does not mean the scraper is delivering data or the notifier is consuming queues.

Consequences: soak starts “on hope” — the first minutes are noise or a false fail before the stack is actually working.

Countermeasure: a dedicated step before soak — migrations, forced recreate/restart of workers, fail-fast on health, clearing queues and the test mailbox, a scraper activity gate (proof of heartbeat or new records). Soak does not start without proof the pipeline is alive.

3. State model in E2E tests

Cause: a virtual user lives across many random scenarios. The API architecture requires a privilege sequence (account → subscription → premium actions). Harness-side state cache easily drifts from server truth.

Consequences: flaky assertions, “premium” scenarios on accounts without a subscription, hard-to-diagnose fails that look like product bugs.

Countermeasure: I treat VU state like production state — sync with the API before flows that need a subscription; reset the session when creating a new account. This is a test design problem, not just “a bad assertion”.

4. The notify scenario as a full-chain test

Cause: the API can be green at zero notification success rate if the scraper is not delivering fresh data or the notifier cannot keep up. A short E2E timeout confuses that with a product outage.

Consequences: either constant false positives on notify, or artificially lowered thresholds that hide real pipeline problems.

Countermeasure: I split SLOs (core API vs notify), add warmup, and before waiting for mail I check database preconditions (backlog to process). I measure what the architecture actually promises: preference + new data → message.

5. Classifying external failures

Cause: the scraper depends on live sources and Chromium. An empty source response ≠ a write error to our own infrastructure.

Consequences: soak fails on partial internet degradation the same way as on our own database failure — the result becomes operationally useless.

Countermeasure: in the observer I separate external signals (warning / pass with warnings) from internal ones (fail primary SLO: heartbeat, batch save, DLQ). Soak should stay useful under partial degradation, not only “all or nothing”.

How I run it

./bin/run-endurance --duration 20m --vus 3
./bin/run-endurance --no-reset --duration 20m --vus 3   # deliberately on existing state
./bin/run-endurance --down-volumes                        # cleanup

Soak stack: API + scraper + notifier + reverse proxy + Postgres + Redis + RabbitMQ + test mailbox + Loki/Grafana — prod-like, isolated from day-to-day dev.

Result: JSON report + exit code (pass / pipeline fail / pass with external warnings / API E2E fail).

Results

On an isolated, clean stack, soak shows what I expected from the architecture:

• scraper (primary) — stable heartbeat, no internal SLO breaches caused by environment “dirt”,
• API E2E (secondary) — core user paths at ~99%+,
• pipeline_notify — works end-to-end; as the slowest path and the most dependent on external timing, it needs longer runs and careful thresholds,
• CI — soak deliberately outside the hard gate (manual / does not block merge); the gate stays on lint, unit, integration and build.

This is not “more assertions in the pipeline”. It is a separate path for validating distributed-system behaviour over time.

What I assume and how I approach it

When designing tests for systems with background workers, I assume that:

• a health contract ≠ a business contract — you must separately prove pipeline activity,
• async notifications are tested as a pipeline, not as a single endpoint,
• isolation and state reset are part of test quality, same as a good assert,
• SLOs are worth splitting by layer (ingest / queues / API) so the result reads like a diagnosis,
• heavy soaks do not belong in hard CI — their value is in deliberate runs before release or after worker changes, not in blocking every MR.

Technically I base this on Compose with an isolated project, a Go harness, observability (logs/events), a message broker and a headless browser on the scraper side — the same set production uses, only in a controlled, repeatable ritual.

Summary

Endurance soak for API + scraper + notifier fills the gap between green CI and confidence that the pipeline works over time. The key is not the number of HTTP scenarios, but the test architecture: isolation from local dev, preflight with proof of scraper work, dual SLOs, and a notify scenario treated as a full-chain test. That is how I validate systems where value emerges asynchronously — not only in a synchronous API response.