Your API creates an order and commits it to PostgreSQL.

The next step is publishing an OrderCreated event to Kafka so that inventory, notifications, and analytics services can react.

You write something like this:

func (s *OrderService) CreateOrder( ctx context.Context, order Order, ) error { if err := s.repo.Create(ctx, order); err != nil { return err } return s.publisher.Publish(ctx, OrderCreated{ OrderID: order.ID, }) } 

It looks straightforward.

But there is a failure window.

What happens if PostgreSQL commits the order successfully and Kafka is unavailable?

The order exists, but no event is published.

What if Kafka accepts the event, but your application crashes before recording that it was published?

Your application might publish the same event again after restarting.

These are not unusual edge cases. They are consequences of coordinating independent systems without a shared transaction.

The transactional outbox pattern addresses this problem by storing the business change and the event to be published in the same database transaction.

Instead of trying to atomically commit to PostgreSQL and Kafka, we make the database transaction the source of truth and publish events asynchronously.

In this article, we'll build a PostgreSQL-backed transactional outbox in Go, implement concurrent workers, handle retries and leases, and explore the delivery guarantees that matter in production.

1. The Dual-Write Problem

A dual write happens when an application needs to update two independent systems as part of one logical operation.

For example:

HTTP Request | v Create Order in PostgreSQL | v Publish OrderCreated to Kafka 

There are two possible execution orders.

Database first

1. PostgreSQL commit succeeds 2. Kafka publish fails 3. Application returns an error 

The order exists, but downstream services never receive the event.

Retrying the entire HTTP request might create another order unless the endpoint is idempotent.

Message broker first

1. Kafka publish succeeds 2. PostgreSQL commit fails 3. Application returns an error 

Now downstream services have received an event for an order that does not exist.

Neither order solves the problem.

A database transaction cannot automatically include a Kafka publish. They are independent systems with independent failure and commit semantics.

Distributed transactions and two-phase commit are possible in some environments, but they introduce operational and architectural complexity.

The transactional outbox pattern takes a different approach.

2. How the Transactional Outbox Works

The central idea is simple:

Store the business change and the event in the same database transaction.

Instead of publishing directly to the broker, insert an event into an outbox_events table.

 HTTP Request | v PostgreSQL TX | +-------+-------+ | | v v Create Order Insert Event | | +-------+-------+ | COMMIT | v Outbox Worker | v Message Broker | v Downstream Consumers 

If the transaction commits, both the order and the outbox event exist.

If the transaction rolls back, neither exists.

This eliminates the inconsistency between the business write and the durable record of the event that must be published.

It does not make publishing exactly once. We'll discuss that distinction later.

3. Designing the Outbox Table

Let's start with a PostgreSQL schema.

CREATE TABLE outbox_events ( id BIGSERIAL PRIMARY KEY, event_id UUID NOT NULL UNIQUE, aggregate_type TEXT NOT NULL, aggregate_id TEXT NOT NULL, event_type TEXT NOT NULL, payload JSONB NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), published_at TIMESTAMPTZ, attempts INTEGER NOT NULL DEFAULT 0, next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), claim_token UUID, lease_until TIMESTAMPTZ, last_error TEXT ); CREATE INDEX idx_outbox_pending ON outbox_events (next_attempt_at, id) WHERE published_at IS NULL; 

Let's look at the important fields.

Field Purpose
event_id Stable identifier used for deduplication
aggregate_type Business entity type
aggregate_id Entity identifier
event_type Event name
payload Serialized event data
published_at Publication completion timestamp
attempts Number of publication attempts
next_attempt_at Earliest time the event can be retried
claim_token Identifies the worker currently holding the lease
lease_until When another worker can reclaim the event
last_error Most recent publication error

Why do we need both claim_token and lease_until?

A worker can crash after claiming an event. Without a lease, the event might remain stuck forever.

A lease makes the event available again after a bounded period.

The claim token prevents an old worker from updating a row after another worker has reclaimed it.

This is a form of ownership fencing for database updates. It does not prevent an expired worker from publishing to the broker, so consumers still need deduplication.

4. Writing the Business Change and Event Atomically

The outbox only works if the business write and event insertion share the same transaction.

Let's define an event:

package outbox import ( "context" "encoding/json" "fmt" "time" "github.com/google/uuid" ) type Event struct { ID uuid.UUID AggregateType string AggregateID string Type string Payload json.RawMessage CreatedAt time.Time } 

A repository method can insert the event using the same sql.Tx as the business operation.

func Insert( ctx context.Context, tx *sql.Tx, event Event, ) error { const query = ` INSERT INTO outbox_events ( event_id, aggregate_type, aggregate_id, event_type, payload, created_at ) VALUES ($1, $2, $3, $4, $5, $6) ` _, err := tx.ExecContext( ctx, query, event.ID, event.AggregateType, event.AggregateID, event.Type, event.Payload, event.CreatedAt, ) if err != nil { return fmt.Errorf("insert outbox event: %w", err) } return nil } 

Now let's use it in an order service.

func (s *OrderService) Create( ctx context.Context, req CreateOrderRequest, ) error { tx, err := s.db.BeginTx(ctx, nil) if err != nil { return fmt.Errorf("begin transaction: %w", err) } defer tx.Rollback() order := Order{ ID: uuid.NewString(), Customer: req.CustomerID, Amount: req.Amount, CreatedAt: time.Now().UTC(), } if err := s.orders.Insert(ctx, tx, order); err != nil { return fmt.Errorf("insert order: %w", err) } payload, err := json.Marshal(OrderCreated{ OrderID: order.ID, Customer: order.Customer, Amount: order.Amount, CreatedAt: order.CreatedAt, }) if err != nil { return fmt.Errorf("marshal event: %w", err) } event := outbox.Event{ ID: uuid.New(), AggregateType: "order", AggregateID: order.ID, Type: "order.created", Payload: payload, CreatedAt: time.Now().UTC(), } if err := outbox.Insert(ctx, tx, event); err != nil { return fmt.Errorf("insert event: %w", err) } if err := tx.Commit(); err != nil { return fmt.Errorf("commit order transaction: %w", err) } return nil } 

There is no Kafka call inside this transaction.

That is intentional.

The transaction only guarantees that the order and its event record are persisted together.

The actual event publication happens asynchronously.

What if the commit response is lost?

Suppose PostgreSQL commits successfully, but the connection drops before the application receives the commit response.

The application might not know whether the transaction succeeded.

This is another reason the HTTP operation should be idempotent.

A retry should not blindly create another order and another event. Use the idempotency mechanism from the previous article, or a durable business uniqueness constraint, to recognize the logical operation.

The outbox guarantees atomic persistence of the event record with the business write. It does not replace request-level idempotency.

5. Claiming Events With Multiple Workers

A single worker can process pending events, but production systems often need multiple workers.

Imagine three worker instances:

Worker A Worker B Worker C 

If all three execute the same query:

SELECT * FROM outbox_events WHERE published_at IS NULL; 

they can select the same events.

That creates duplicate work immediately.

PostgreSQL provides a useful mechanism for this problem:

FOR UPDATE SKIP LOCKED.

It allows workers to claim different rows without waiting for rows already locked by other workers.

However, we should not hold a database transaction open while publishing to Kafka.

Instead, we use a short transaction to claim a batch, commit it, and publish outside the transaction.

Claiming a batch

WITH candidates AS ( SELECT id FROM outbox_events WHERE published_at IS NULL AND next_attempt_at <= NOW() AND ( lease_until IS NULL OR lease_until < NOW() ) ORDER BY id FOR UPDATE SKIP LOCKED LIMIT $1 ) UPDATE outbox_events AS events SET claim_token = $2, lease_until = NOW() + ($3 * INTERVAL '1 second'), attempts = attempts + 1 FROM candidates WHERE events.id = candidates.id RETURNING events.id, events.event_id, events.aggregate_type, events.aggregate_id, events.event_type, events.payload, events.attempts; 

Parameters:

  • $1: batch size.
  • $2: unique claim token for this batch.
  • $3: lease duration in seconds.

The FOR UPDATE SKIP LOCKED clause prevents workers from claiming the same locked rows.

The UPDATE records ownership and establishes a lease.

Once the transaction commits, the worker can publish the returned events.

A simplified Go representation:

type ClaimedEvent struct { ID int64 EventID uuid.UUID AggregateType string AggregateID string Type string Payload []byte Attempts int } type Claim struct { Token uuid.UUID Events []ClaimedEvent } 

The repository method should execute the claim query in a short transaction and return the rows after committing.

The key design rule:

Never keep database row locks open while waiting for a message broker.

A slow broker should not hold database transactions and locks indefinitely.

6. Publishing Events Outside the Transaction

Define a small publisher interface:

type Publisher interface { Publish( ctx context.Context, event ClaimedEvent, ) error } 

The implementation can use Kafka, RabbitMQ, NATS, or another messaging system.

The outbox worker should not depend on the broker's concrete client.

Now let's define the worker:

type Worker struct { repo Repository publisher Publisher batchSize int lease time.Duration } 

A useful repository interface:

type Repository interface { Claim( ctx context.Context, limit int, lease time.Duration, ) (Claim, error) MarkPublished( ctx context.Context, id int64, token uuid.UUID, ) error ScheduleRetry( ctx context.Context, id int64, token uuid.UUID, delay time.Duration, cause error, ) error } 

The worker processes one claimed batch:

func (w *Worker) ProcessBatch( ctx context.Context, ) error { claim, err := w.repo.Claim( ctx, w.batchSize, w.lease, ) if err != nil { return fmt.Errorf("claim outbox events: %w", err) } for _, event := range claim.Events { if err := ctx.Err(); err != nil { return err } publishCtx, cancel := context.WithTimeout( ctx, 10*time.Second, ) err := w.publisher.Publish(publishCtx, event) cancel() if err != nil { delay := retryDelay(event.Attempts) scheduleErr := w.repo.ScheduleRetry( ctx, event.ID, claim.Token, delay, err, ) if scheduleErr != nil { return fmt.Errorf( "schedule retry for event %s: %w", event.EventID, scheduleErr, ) } continue } if err := w.repo.MarkPublished( ctx, event.ID, claim.Token, ); err != nil { return fmt.Errorf( "mark event %s as published: %w", event.EventID, err, ) } } return nil } 

This is a sequential batch processor. You can add bounded parallelism later, but the ownership and completion rules should remain the same.

Notice that the worker does not delete an event immediately after publishing it.

It marks the event as published.

This makes it possible to inspect publication history, measure delivery delays, and troubleshoot failures.

For large systems, published rows should eventually be archived or removed according to a retention policy.

7. Marking Events as Published Safely

The claim token must be checked when updating an event.

UPDATE outbox_events SET published_at = NOW(), claim_token = NULL, lease_until = NULL, last_error = NULL WHERE id = $1 AND claim_token = $2 AND published_at IS NULL; 

Why is this important?

Consider this sequence:

  1. Worker A claims an event.
  2. Worker A pauses for a long time.
  3. Its lease expires.
  4. Worker B reclaims the event.
  5. Worker A resumes.

Without an ownership check, Worker A could update the event even though Worker B now owns it.

The claim token prevents that stale update.

Your repository should also check RowsAffected().

If no row was updated, the claim is no longer owned by that worker, or the event has already been completed.

However, there is an important limitation.

The claim token protects database state. It cannot stop Worker A from publishing to Kafka after its lease expires.

A lease is not a distributed lock over the external broker.

That is why consumer-side deduplication remains necessary.

8. Retry Strategy and Backoff

Message brokers can be temporarily unavailable.

A worker should not retry a failed event continuously.

That creates unnecessary load and can make an existing outage worse.

Use exponential backoff with jitter.

func retryDelay(attempt int) time.Duration { const ( base = time.Second max = 5 * time.Minute ) if attempt < 1 { attempt = 1 } exponent := min(attempt-1, 8) delay := base * time.Duration(1<<exponent) if delay > max { delay = max } jitter := time.Duration( rand.Int64N(int64(delay/2) + 1), ) return delay/2 + jitter } 

This implementation assumes a Go version with math/rand/v2.

The delay grows with each failed attempt but is capped to prevent excessively long retry intervals.

The retry update can look like this:

UPDATE outbox_events SET next_attempt_at = NOW() + ($1 * INTERVAL '1 second'), claim_token = NULL, lease_until = NULL, last_error = $2 WHERE id = $3 AND claim_token = $4 AND published_at IS NULL; 

The repository should pass a bounded, sanitized error message rather than storing arbitrary error output indefinitely.

What about permanently failing events?

Some events will never succeed without intervention.

Examples:

  • Invalid payload.
  • Unsupported schema version.
  • Message exceeds broker limits.
  • A downstream system permanently rejects the event.

A production design needs a poison-event policy.

Possible approaches:

  • Maximum attempt count.
  • Dead-letter table or queue.
  • Manual replay.
  • Alerting and operational investigation.
  • Event quarantine without losing the original payload.

Do not silently delete failed events.

A dead-letter policy should preserve enough information to investigate and replay the operation safely.

Also remember that a maximum retry count is a business and operational decision. Some important events may require indefinite retry with alerting rather than permanent abandonment.

9. The Most Important Failure Window: Publish Succeeds, Database Update Fails

Let's examine the most important scenario.

1. Worker claims event 2. Worker publishes event to Kafka 3. Kafka acknowledges the message 4. Worker crashes 5. published_at is never updated 6. Lease expires 7. Another worker claims and publishes the event again 

The event is delivered twice.

This is expected behavior.

The outbox pattern provides durable, at-least-once publication when configured with retries and recovery. It does not provide exactly-once delivery across PostgreSQL and an external broker.

The consumer must be idempotent.

This is not an optional optimization. It is part of the design.

10. Consumer-Side Deduplication

Every event should have a stable identifier.

For example:

{ "event_id": "b89b5f43-0d7c-45d7-8a9b-9dbcf9fcb1f8", "event_type": "order.created", "aggregate_id": "order-123", "payload": { "order_id": "order-123" } } 

The consumer stores processed event IDs.

CREATE TABLE processed_events ( consumer_name TEXT NOT NULL, event_id UUID NOT NULL, processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), PRIMARY KEY (consumer_name, event_id) ); 

The consumer should insert the deduplication record and apply its business change in the same transaction.

func (c *Consumer) Handle( ctx context.Context, event Event, ) error { tx, err := c.db.BeginTx(ctx, nil) if err != nil { return fmt.Errorf("begin transaction: %w", err) } defer tx.Rollback() result, err := Source:https://dev.to/serifcolakel/transactional-outbox-in-go-reliable-event-publishing-without-distributed-transactions-1dol

Related posts