Queue vs Topic vs Log: Picking the Broker

Most broker decisions are made in a ten-minute conversation that ends with "we'll just use Kafka," or from the other side, "SQS is fine, it's cheap." Both sentences treat the broker as a throughput-and-price decision. It isn't. What you are actually choosing is a contract: who owns a message after it is delivered, who tracks how far each consumer has read, and whether yesterday's events still exist tomorrow.

Get the contract wrong and no tuning fixes it. A team that picks a queue and later needs a new service to bootstrap from the last 30 days of order events has nothing to replay. A team that picks Kafka for background jobs ends up owning partition planning, rebalances, and head-of-line blocking to get what a queue provides out of the box.

Three primitives, three contracts

QUEUE (competing consumers) TOPIC (fan-out) LOG (retained stream) producer ──▶ [ m3 m2 m1 ] ──┬▶ A producer ──▶ topic ──┬▶ [sub-A] ▶ A producer ──▶ [m1 m2 m3 m4 m5 …] └▶ B ├▶ [sub-B] ▶ B ▲ ▲ each message goes to ONE each subscription gets consumer X consumer Y consumer, then is deleted its OWN copy (offset 2) (offset 5) 

The last row is where teams get surprised. In a queue, message 7 failing does not stop message 8. In a log, the consumer's offset means "everything before this is done" — a message that keeps failing stalls its partition unless you build retry topics yourself.

Mapping to AWS and Azure

Three mapping details decide real designs:

SNS and Service Bus topics are not equivalent. On Azure, a topic subscription is a queue with its own lock, delivery count, and dead-letter sub-queue. On AWS, SNS pushes to a subscriber — the standard answer is SNS → SQS per consumer, which recreates what Service Bus gives natively.

EventBridge and Event Grid are routers, not backbones. Right for cloud/SaaS event reactions. Wrong for a high-volume internal event backbone where you need ordering and replay at a known cost.

Partition count is a design decision, not a tuning knob. Kafka partitions can be increased but not reduced. Event Hubs Basic/Standard partition count is fixed at creation. Size it for how far back a recovering consumer might need to go.

The pattern that covers most systems: topic fan-out into per-consumer queues

When several services need the same event, publish once to a topic and give every consumer its own queue. Each consumer gets isolation, its own DLQ, and independent scaling.

AWS: SNS → SQS with filtering

resource "aws_sqs_queue" "billing" { name = "billing-order-events" visibility_timeout_seconds = 120 # must exceed worst-case handler time redrive_policy = jsonencode({ deadLetterTargetArn = aws_sqs_queue.billing_dlq.arn maxReceiveCount = 5 }) } resource "aws_sns_topic_subscription" "billing" { topic_arn = aws_sns_topic.orders.arn protocol = "sqs" endpoint = aws_sqs_queue.billing.arn raw_message_delivery = true # otherwise the body is an SNS JSON envelope filter_policy_scope = "MessageBody" filter_policy = jsonencode({ eventType = ["OrderPlaced", "OrderCancelled"] }) } 

The non-obvious decisions: raw_message_delivery so consumers read your payload and not an SNS envelope, filter_policy_scope = "MessageBody" so routing works on the payload, and a visibility timeout longer than your slowest handler.

Azure: Service Bus topic with a filtered subscription

resource billing 'Microsoft.ServiceBus/namespaces/topics/subscriptions@2021-11-01' = { parent: topic name: 'billing' properties: { lockDuration: 'PT2M' maxDeliveryCount: 5 deadLetteringOnMessageExpiration: true } } resource billingFilter 'Microsoft.ServiceBus/namespaces/topics/subscriptions/rules@2021-11-01' = { parent: billing name: 'order-lifecycle' properties: { filterType: 'SqlFilter' sqlFilter: { sqlExpression: 'eventType IN (\'OrderPlaced\', \'OrderCancelled\')' } } } 

Service Bus SQL filters evaluate message properties, not the body — the publisher must set eventType as an application property. A subscription without an explicit rule gets a catch-all $Default rule; confirm it's not still attached after deployment or the subscription receives everything.

When you need a log

Reach for a log when retention is the requirement: new consumers must read history, projections must be rebuildable, or stream processing needs ordered per-key state.

kafka-topics.sh --bootstrap-server "$BOOTSTRAP" --create \ --topic order-events \ --partitions 12 \ --replication-factor 3 \ --config min.insync.replicas=2 \ --config retention.ms=604800000 # 7 days 

--partitions 12 is your parallelism ceiling per consumer group: a thirteenth consumer sits idle. replication-factor 3 with min.insync.replicas=2 and acks=all tolerates one broker loss without losing acknowledged writes.

Four questions, in order

Answers can combine. A common production shape is a log as the system-of-record backbone with queues downstream for consumers that need per-message retry and a DLQ — so one poison message cannot stall a partition.

Read the Full Article

This summary covers the three primitives, the AWS/Azure service mapping, and the decision framework. The full article includes:

• Complete Terraform for the SNS → SQS fan-out pattern including the IAM queue policy scoped to the topic ARN
• Full Bicep for the Service Bus topic with duplicate detection and filtered subscription
• The "exactly-once" section — what SQS FIFO deduplication, Service Bus duplicate detection, and Kafka transactions actually cover (and don't cover)
• The archival consumer pattern: how to attach cheap replay insurance from day one when you choose a queue or topic
• What the wrong choice looks like in production — queue chosen when a log was needed, log chosen when a queue was needed, topic without a durable buffer

👉 Queue vs Topic vs Log: Picking the Broker — Full Article