---
title: "How to Send 1,000,000 Notifications Without Falling Over"
description: "A vendor-neutral system-design walkthrough of the message-queue architecture that sends a million notifications without taking your API down: fan-out workers, back-pressure, retries, and dead-letter queues."
author: Aleksei Aleinikov
date: 2026-07-13
lang: en
tags: [message-queue, event-driven-architecture, fan-out, notification-system, back-pressure, dead-letter-queue]
canonical: https://www.alekseialeinikov.com/en/blog/topics/architecture/send-one-million-notifications-without-falling-over-2026
source: alekseialeinikov.com
---

# How to Send 1,000,000 Notifications Without Falling Over

Sending one notification is a function call. Sending a million is a distributed-systems problem wearing a function call's clothes.

The naive version looks innocent: a user triggers an event, you loop over a million recipients, and for each one you call the push, email, or SMS provider. It works in the demo. It works for a thousand users. Then a product launch fans a single event out to your whole user base, every send waits on a slow third-party API, and the request thread that was supposed to return in 50 ms is now holding a million open calls. The API falls over — not because the code is wrong, but because the *shape* is wrong.

This is a vendor-neutral walkthrough of the shape that holds: a **message-queue architecture** with fan-out workers, back-pressure, and the boring reliability parts that keep a "send" from turning into an outage. The examples are cloud-agnostic — the same pattern runs on RabbitMQ, Kafka, SQS, or Pub/Sub.

## The Mistake: Doing the Work Inside the Request

The first thing to fix is *where* the work happens. When you send inside the request handler, the thread that accepted the HTTP call is the same thread waiting on a slow external provider — a million times over.

![Sending inside the request blocks the thread and floods on a spike; enqueue-and-return keeps the API fast while workers drain the queue.](https://www.alekseialeinikov.com/blog/million-notifications-problem-2026.webp)

Two things go wrong at once. First, **latency stacks**: your API's response time is now the sum of every provider call, so one slow vendor drags the whole request down. Second, **concurrency explodes**: a spike doesn't queue anywhere, it opens a million concurrent sends against providers that will happily rate-limit or drop you, while your own connection pools and memory run dry.

The rule of thumb is simple:

> The request should *accept* the job, not *do* the job.

Accept the event, write it to a durable queue, and return `202 Accepted` in milliseconds. The heavy lifting moves to background workers that you can scale, throttle, and retry independently of the request path. The spike is still there — but now it lands in a queue, where a backlog is an inconvenience, not an outage.

One subtlety hides in "write it to a durable queue." If you write to your database *and* publish to the queue as two separate steps, a crash between them loses the event or sends a phantom one. The fix is the **transactional outbox**: commit the event to an `outbox` table in the *same* database transaction as your business write, then a separate relay ships those rows to the queue. One atomic write, no lost events, no double sends.

## The Core: A Fan-Out Pipeline

Once the work is off the request path, the architecture becomes a pipeline. One event enters; per-channel notifications leave. The piece that makes it scale is **fan-out**: a single ingress queue, split into one isolated lane per channel.

![One producer publishes to an ingress topic; a fan-out step splits work into isolated push, email, and SMS queues, each with its own workers and provider.](https://www.alekseialeinikov.com/blog/million-notifications-fanout-2026.webp)

Walk the path:

1. **Producer → ingress queue.** The API (or an upstream event) publishes one message per logical event to a single durable topic. This is the only thing the request path touches, and it's fast and append-only.
2. **Fan-out.** A lightweight step reads the ingress event, decides which channels apply (this user wants push + email, not SMS), and publishes a message to each channel's queue. Fan-out is where "one event" becomes "three sends."
3. **Per-channel queues + workers.** Each channel — push, email, SMS — has its **own** queue and its **own** worker pool. This isolation is the whole point: a slow SMS gateway backs up the SMS queue only. Push and email keep flowing.
4. **Providers.** Each worker pool talks to its provider (APNs/FCM, SMTP/SES, an SMS gateway) at a rate that provider can absorb.

The isolation buys you three things you can't get from a single shared worker pool: independent **scaling** (spin up more email workers without touching SMS), independent **failure** (one dead provider doesn't stall the others), and independent **tuning** (email tolerates minutes of delay; push often can't).

Two levers turn this from a diagram into a production system. **Batch to the provider**: most push, email, and SMS APIs accept hundreds of recipients per call, so a worker that groups its queue into batches sends far more per second than one that goes one-at-a-time. And **split by priority, not just by channel**: a transactional OTP and a marketing blast should never share a queue — give latency-critical traffic its own lane so a million-message campaign can't delay a login code. Where per-user ordering matters, key those messages to the same partition so a single worker handles them in sequence.

### Scale on Queue Depth, Not Request Rate

The workers are where a million events actually get sent, and the trick is *what you scale on*. Don't autoscale on CPU or request rate — scale on **queue depth** and **message age**. When the backlog grows past a threshold, add worker replicas; when it drains, scale back down.

This is what decouples arrival speed from send speed. A million events can arrive in a minute; your workers might take an hour to send them, throttled to what the providers allow. That's fine — the queue absorbs the difference. The system's job is not to send everything instantly. It's to **never lose an event and never fall over**, while draining as fast as the downstream providers permit.

Those worker pools usually run on an autoscaling container platform, and if that platform is Kubernetes, the base it runs on matters as much as the pipeline itself — see [Secure-by-Default GKE: A Reference Architecture for 2026](https://www.alekseialeinikov.com/en/blog/topics/architecture/secure-by-default-gke-reference-architecture-2026).

> This is the same event-driven backbone behind any decoupled system. If you're building it on GCP specifically, the trade-offs between the transport and the router are worth reading next: [Pub/Sub or Eventarc? Event-Driven GCP Without the Spaghetti](https://www.alekseialeinikov.com/en/blog/topics/architecture/pubsub-vs-eventarc-2026-event-driven-gcp-without-spaghetti).

## Staying Up Under Load: The Boring Parts That Matter

A queue and some workers will get you through a demo. What keeps a million-message run alive at 3 a.m. is the reliability layer — the parts everyone skips until production teaches them. There are four, and they only work *because* you have a queue.

![Four guards keep it up: retry with backoff, a dead-letter queue, idempotency keys, and per-provider back-pressure.](https://www.alekseialeinikov.com/blog/million-notifications-reliability-2026.webp)

**Retry with backoff.** Providers fail transiently — a timeout, a 503, a brief rate-limit. Retry, but with **exponential backoff and jitter**, never a tight loop. A tight retry loop across thousands of workers is indistinguishable from a denial-of-service attack, and the target is your own provider. Backoff spreads the retries out so a blip recovers instead of amplifying.

**Dead-letter queue (DLQ).** Some messages never succeed: a malformed payload, a permanently-invalid address, a bug. After N attempts, move the message to a **dead-letter queue** instead of retrying forever. The bad message is parked for inspection — never lost, never blocking the line — and the healthy traffic behind it keeps moving. Without a DLQ, one poison message retries until it starves the queue.

**Idempotency.** Every durable queue delivers **at-least-once**. A redelivery, a slow ack, a worker that crashes after sending but before acking — any of these means the same message gets processed twice. If "twice" means a second email or a double charge, that's a bug you wrote. Dedupe on a **stable message ID**: record "message `abc123` already sent" and treat a re-run as a no-op. Keep that record in a fast store with a TTL that covers your retry window — you only need to remember recent IDs, not all of history. Redelivery becomes harmless.

**Back-pressure.** This is the guard that keeps the *whole system* from tipping over. Each worker respects a **per-provider concurrency limit and rate limit**. When a provider slows down, workers slow with it — the queue grows, but nothing crashes. Back-pressure turns "the provider is struggling" into "the backlog is deeper today" instead of "we hammered them until they blocked us."

Put together, these four are why the design survives its own success. The queue makes them possible; skip them and your retry logic becomes a self-inflicted DDoS on the exact providers you depend on.

## Putting It Together

Trace one launch through the finished shape. A campaign fires one event. The API writes it to the ingress queue and returns in milliseconds — the user-facing path never even knows a million sends are coming. Fan-out splits the event into push, email, and SMS work on three isolated queues. Each worker pool scales on its own backlog, sends at its provider's pace, retries transient failures with backoff, dead-letters the hopeless ones, and dedupes redeliveries on a message ID. The spike becomes three draining queues. Nothing falls over.

None of this is exotic. It's a queue, a fan-out step, worker pools that scale on depth, and four reliability guards. The mistake was never the tools — it was doing a distributed-systems job inside a single request thread. Move the work off that thread, isolate the channels, and let back-pressure absorb the spike, and "send a million notifications" stops being a scary number and becomes a Tuesday.

### The Field Rule

Accept the job, don't do it. Fan out to one queue per channel so failures stay local. Scale workers on queue depth, not request rate. And treat retries, dead-letter queues, idempotency, and back-pressure as load-bearing, not optional — they're the difference between a backlog and an outage.