Databases
Message Queues
A message queue sits between producers, which create work or events, and consumers, which process them, holding messages until a consumer is ready to handle them. Publish/subscribe (pub/sub) is a related pattern where a message is broadcast to every interested subscriber rather than handed to exactly one consumer. Both exist to decouple the timing and load of different parts of a system from each other.
Why it matters
- It stops a slow step from blocking the request that triggered it
- A web request can enqueue a job — sending an email, resizing an image — and return immediately instead of waiting on that work to finish before responding to the user.
- It smooths out traffic spikes
- Consumers can work through a backlog at their own sustainable pace instead of every producer needing enough instant capacity to handle a sudden peak.
- It decouples services from needing to know about each other directly
- A producer doesn't need to know which service, or how many instances of it, will eventually process a given message.
- Retries and dead-letter queues give a structured way to handle failure
- Instead of a failed operation silently losing work, a message that keeps failing can be retried automatically and eventually routed somewhere a human can investigate it.
Queue vs pub/sub
A point-to-point queue typically delivers each message to exactly one consumer — a classic job queue, where it doesn't matter which worker picks up a given task, only that exactly one of them does. Publish/subscribe broadcasts a message to every subscriber interested in it, which fits an event ('an order was placed') that several independent parts of a system all need to react to separately.
{
"id": "msg_8a2f",
"type": "email.send",
"payload": { "to": "user@example.com", "template": "welcome" },
"attempts": 0,
"enqueuedAt": "2026-09-16T10:00:00Z"
}Delivery guarantees and why idempotency matters
Most real systems only guarantee at-least-once delivery, not exactly-once, which means a consumer can receive and process the same message more than once — a truly exactly-once guarantee is possible in some systems but expensive enough that it's rarely the default. Because of this, a consumer needs to be idempotent: processing the same message twice should produce the same end result as processing it once, typically by tracking which message IDs have already been handled.
When a queue is worth the complexity
A queue earns its place when a task can genuinely happen asynchronously — background jobs like emails or notifications, work that needs to be distributed across many workers, or events multiple independent services need to react to. For something that's really just a simple, fast, synchronous call, adding a queue adds latency and an operational moving part for no real benefit.
Mistakes people make here
- Assuming exactly-once delivery
- Most real message queues guarantee at-least-once delivery, not exactly-once — a consumer that isn't written to safely handle processing the same message twice will eventually do something wrong (like sending a duplicate email) when a message is redelivered.
- Using a queue as a permanent database
- Messages are meant to be consumed and removed, not stored indefinitely and queried later — a system that needs to look back at historical events belongs in a proper data store, not left sitting in a queue.
- Not setting up retry limits or a dead-letter queue
- Without one, a 'poison message' that always fails processing can retry forever, consuming resources and blocking the messages queued behind it, rather than being set aside for a human to investigate.
- Reaching for a queue when a simple synchronous call would do
- Adding a queue for something that's actually fast and doesn't need to be decoupled adds latency and a new operational component to monitor, for a benefit that doesn't apply to that particular case.
Strengths and trade-offs
Where it is strong
- Decouples producers and consumers so each can scale, fail, or deploy independently of the other.
- Smooths traffic spikes by letting consumers work through a backlog rather than requiring instant capacity for the peak.
- Built-in retry and dead-letter handling give a structured way to deal with failures instead of silently dropping work.
- A natural fit for background jobs — emails, image processing, notifications — that don't need to block the request that triggered them.
The trade-offs
- Adds an operational moving part to monitor, and a new place for things to go wrong, such as a stuck consumer or a growing backlog nobody notices.
- Most real-world queues only guarantee at-least-once delivery, so every consumer has to be written to tolerate duplicate messages.
- Ordering isn't always guaranteed either, depending on the system and how it's partitioned across consumers.
- Debugging an asynchronous flow spread across a queue is harder than reading a single synchronous call stack top to bottom.
Who needs this
Relevant to anyone building background jobs, event-driven services, or systems where a slow step shouldn't block the request that triggered it.
Questions about message queues
- What's the difference between a queue and pub/sub?
- A queue typically delivers each message to exactly one consumer, which fits distributing discrete tasks across workers; pub/sub broadcasts a message to every subscriber, which fits an event that several independent parts of a system all need to react to.
- Do I need a dedicated broker like Kafka or RabbitMQ for a small app?
- Often not — Redis, or even a simple database-backed job table, covers a lot of real workloads before the operational overhead of a dedicated message broker is actually justified.
- Can messages be lost?
- Yes, unless both the broker and the consumer are configured for durability and proper acknowledgment, which is a deliberate trade-off against raw speed and simplicity.
- What does 'idempotent consumer' mean?
- A consumer that produces the same end result whether it processes a given message once or several times — necessary because most brokers can and do redeliver messages under normal, expected operation.