Transactional Outbox and Inbox in Spring Boot

A polling-based transactional outbox and inbox for Spring Boot, taken apart: how events leave the database, why the polling delay adapts, why each event type gets its own settings, and the three clocks that decide how long a duplicate is remembered.

The write that has two destinations

I wrote a transactional outbox for two of my own projects. The existing options did not fit, and log-based CDC with Debezium felt too heavy for a simple requirement: save an event together with the data, and send it later. oncebox is what that outbox grew into.

The requirement exists because a service that changes its database and then publishes a message is writing to two systems, and a database transaction covers only one of them. If the process dies between the commit and the publish, the data has changed and nobody was told. Reverse the order and a message goes out for a change that never happened.

The outbox pattern closes the gap by making the event part of the transaction. Instead of publishing straight to the broker, the service writes the event into a table next to its business data, and a separate step delivers it later.

In the outbox variant the event is a row committed with the business data. Publishing becomes a separate, retryable step that cannot lose the event, because the event is already durable.*

The outbox: one transaction, two rows

Publishing is one of two calls. The manual one takes an event type and a payload, or a list of payloads for a batch. The declarative one is an annotation on the method that already holds the transaction:

@Transactional @OutboxPublish(eventType = "create-example-event") public ExampleDto save(ExampleCreateDto dto) { ExampleEntity entity = repository.save(mapper.toEntity(dto)); return mapper.toDto(entity); } 

Both write to outbox_events through methods annotated @Transactional(propagation = MANDATORY). The library never opens the transaction itself. It joins yours, and that is the whole guarantee: the event row commits or rolls back together with the business row. With the annotation, the payload is the method result by default (#result) and can be pointed at a parameter with a SpEL expression.

Every event also gets a UUIDv7 id. Time-ordered ids make inserts land at the tail of the primary key index instead of scattering across it, which matters for a table that takes a write for every event. The same id travels in the oncebox_event_id header and later becomes the deduplication key on the consumer side.

Once the row is in the table, it moves through a small set of states. Four background jobs drive it forward, and the diagram shows who does what.

The poller moves events forward, stuck-event recovery rescues a batch whose publisher died, the DLQ transfer moves exhausted events across in a single transaction (lock, insert into target, delete from source, commit), and clean-up removes processed rows.*

The IN_PROCESS state is the reason a crashed publisher does not lose events. The poller marks a batch the moment it locks it. If the instance dies before the broker confirms, those rows stay IN_PROCESS. A recovery worker finds rows that have sat there longer than max-batch-processing-time and returns them to PENDING without counting a failed attempt, since the event never had a chance to fail.

Delivery on this side is at-least-once. The poller waits for the broker acknowledgement before marking an event processed, so a lost acknowledgement means a second send, never a missing one. Removing that duplicate is the inbox's job, which comes later in this article.

Polling: how events leave the table

Each event type is polled on its own schedule. A library-managed thread pool runs one poller per type next to a few background jobs: stuck-event recovery and clean-up, plus three more when the DLQ is on. The sizing rule is 2 + n threads without the DLQ and 5 + n with it, where n is the number of event types.

Several application instances can poll the same table at once. The query that takes a batch uses FOR UPDATE SKIP LOCKED (on PostgreSQL inside a CTE that also moves the rows to IN_PROCESS in the same statement), so each instance locks a different slice of the pending rows instead of waiting on the instance in front of it.

Instances do not elect a leader or talk to each other. The database row locks are the only coordination, and they hand out disjoint batches. Events are ordered by next_retry_at inside a type, which is the creation time until a first failure.*

MySQL note. Run the outbox datasource at READ COMMITTED. Under InnoDB's default REPEATABLE READ, the polling query takes next-key (gap) locks, so several instances polling the same table deadlock instead of each taking a disjoint batch. PostgreSQL and Oracle are unaffected.

That leaves one question, and it is the one that decides how the library behaves in production: how often should each poller ask the database whether there is anything to send?

Adaptive polling

A fixed delay forces a trade. A short one gets events out quickly and queries an empty table around the clock. A long one is cheap and makes every event wait. Most outboxes spend most of their time with nothing to send, so the short delay is paid for mostly in empty answers.

The adaptive strategy removes the trade by letting the delay follow the workload. After each poll, the poller sets the delay for the next one with a single rule:

The poll returned a full batch (as many events as batch-size): more are probably waiting, so the delay resets to min-fixed-delay.

The poll returned less than a full batch, or nothing: the backlog is drained, so the delay is multiplied by multiplier, up to max-fixed-delay.

With the defaults (250ms minimum, multiplier 1.5, maximum 1m), consecutive empty polls stretch the delay like this:

It takes 14 empty polls, a little over three minutes of idleness, to reach the one-minute ceiling. Computed from the scheduling rule in AdaptiveOnceboxScheduleStrategy.*

To see what that is worth, count the polls one event type makes against an idle table in an hour, on one instance, using the same rule:

A fixed 250 ms delay asks the database 14,400 times; adaptive with default settings asks about 70 times, the ramp included. Same configured minimum, roughly 200 times fewer empty answers.*

The reset rule is what keeps this safe under load. A burst fills batches completely, every full batch resets the delay to the minimum, and the poller drains the backlog at the fastest cadence the type is configured for. Once a poll comes back short, it starts backing off again.

Eight empty polls, a burst of five full batches, one short batch, then idle again. The scenario is illustrative; the delays follow the library's rule exactly.*

The delay ceiling doubles as a latency budget. An event that lands just after a poll waits at most one current delay, so max-fixed-delay is the longest an idle type can make an event wait for pickup. That number is a setting, and every event type has its own.

Per-event-type settings

Events in one service rarely share the same urgency. A payment confirmation should leave within a fraction of a second. A nightly report can wait minutes. One global polling setting forces both onto the same cadence, so a service has to choose between a database that is queried constantly and critical events that wait.

oncebox keeps one defaults block and lets every event type override any part of it: polling strategy and delays, batch size, retry count and backoff. Unset values are inherited, so an event type states only what makes it different.

oncebox: outbox: defaults: batch-size: 200 polling: type: adaptive min-fixed-delay: 250ms max-fixed-delay: 1m multiplier: 1.5 max-retries: 3 backoff: enabled: true delay: 10s multiplier: 3.0 events: high-priority: topic: events polling: min-fixed-delay: 250ms # inherits everything else low-priority: topic: events polling: min-fixed-delay: 10s max-fixed-delay: 5m multiplier: 2.0 backoff: enabled: false 

The two profiles behave very differently on the same idle table:

The high-priority type is back at the one-minute ceiling after about three and a half minutes. The low-priority type climbs to five minutes. Computed from the two configurations above.*

Retries are per type as well. The delay before a retry is delay × multiplier^n, so the defaults give 10 s, 30 s, 90 s. A type that is cheap to repeat can retry quickly; a type that hits a fragile downstream can back off hard or stop after two attempts. Once retries run out, an event of any type moves to the same DLQ.

Because every type has its own poller, a slow type does not hold up a fast one. A burst on create-order shortens only that type's delay; the other pollers keep their own pace.

The inbox

Everything so far was about getting an event out reliably. Reliable delivery has a price on the receiving side: at-least-once means the same event can reach a consumer twice, because of a lost broker acknowledgement, a stuck-event recovery, or a consumer group restart. The outbox cannot prevent that, so the consumer has to tolerate it.

oncebox's inbox is a table of processed event ids. It does not store messages and does not process anything asynchronously. An event whose id is already recorded is skipped. Everything is keyed on the UUIDv7 that the outbox assigned when the event was created, which the consumer reads from the oncebox_event_id header.

public ExampleKafkaListener(@TransactionalConsumer IdempotentConsumer idempotentConsumer) { ... } @KafkaListener(topics = "example-events", groupId = "example-group", containerFactory = "inboxKafkaListenerContainerFactory") public void listen(Message<ExampleDto> message) { idempotentConsumer.consume( message, OnceboxHeadersUtils::extractId, m -> handlers.get(OnceboxHeadersUtils.extractEventType(m)).accept(m.getPayload()) ); } 

There are two implementations, and the choice is a statement about what a duplicate costs you:

Use the transactional one when the effect is a write to the same database: the id and the effect commit together or not at all, so a redelivered event never changes data twice. Use the non-transactional one for effects that cannot be undone, such as sending an email, where a repeated side effect is worse than a missed one. The README calls this effectively-once rather than exactly-once, because the library cannot deduplicate inside the broker itself.

The cache sits in front of the database check. It is written only after the wrapped consumer returns, so it never claims an event that was not processed.*

Batches use the same inbox. When a listener receives a list, ids that were already consumed are filtered out and the operation receives only the new ones, so a partly redelivered batch does the right thing without any code in the handler.

One rule keeps the bookkeeping honest: within a service, one event is handled by one consumer. The inbox records an event by its id alone, whichever consumer recorded it. Two listeners in the same service that both want the same event would split it, so the second one skips it. Give each effect its own event type, or its own service.

The cache: what it does for the inbox

Without a cache, every delivery checks the database: a lookup of the id, and for the transactional consumer an insert inside the business transaction. For a stream where redeliveries are rare, that lookup is paid on every message just to catch the occasional duplicate.

The optional cache moves repeated lookups out of the database. It plugs into Spring's CacheManager, so Redis or any other cache you already run works. On a hit, the consumer returns before it touches the database. On a miss it falls through to the inbox as usual and, after the consumer succeeds, puts the id into the cache.

oncebox: inbox: cache: enabled: true cache-name: "orders-service:oncebox:inbox" 

Three details decide whether it behaves:

• The cache name is per service. The key is the event id alone. Two services that consume the same event and share one Redis under one name would mark each other's events as seen. Prefix the name with the service name, or give each service its own key prefix.
• You own the expiry. The library hands the cache an id and nothing else. How long it lives is a property of the CacheManager bean you configure, which brings the cache into the retention story below.
• The cache is an accelerator. A cache that loses an entry costs one database lookup; the inbox table is still the source of truth. Single-event consume calls go through the cache, and batch calls go straight to the inbox table.

Three clocks

An inbox only deduplicates what it still remembers. Ids are kept until clean-up removes them (oncebox.inbox.clean-up.ttl), so a redelivery that arrives later than that is treated as a new event. How long that window needs to be depends on how long a duplicate can still be on its way, and three separate systems each run a clock that answers a part of that question.

1. The broker: how long a message can come back

oncebox does not configure the broker. Kafka keeps messages for its retention period, whatever consumers have read; a consumer group reset or a replay can deliver any message still inside that period. Kafka's own default is seven days. RabbitMQ holds an unacknowledged message until it is acknowledged or expires through a queue or message TTL you set, and an unacknowledged message is requeued when the listener throws.

Acknowledgement itself is on your side of the fence, and the defaults are right for the inbox. Spring Kafka commits offsets after the listener returns, and RabbitMQ's AUTO mode acknowledges after the listener returns. enable.auto.commit=true breaks that, and the starter logs a warning when it sees it.

2. The inbox table: how long an id is remembered

The inbox clean-up job deletes ids older than its ttl, 24 hours by default, in batches, on one instance at a time under a distributed lock. Its polling is adaptive like everything else, so rows live slightly longer than the ttl: the job removes them on its next pass, not at the exact second of expiry.

3. The cache: how long the shortcut lasts

The cache entry expires on your CacheManager's schedule. The README's rule is that inbox clean-up happens after the cache entry has gone. In other words, the cache lifetime must not exceed the inbox ttl. The cache then only ever shortens the path to an answer the table would give, and the inbox ttl stays the single number that defines the deduplication window.

The inbox ttl has to reach as far as a duplicate can still arrive. The cache sits inside it. Bars for oncebox settings show the library defaults; the broker bar shows Kafka's own default retention and moves with your cluster; the cache bar is an example length, and any value up to the inbox ttl is valid.*

Two rules follow from this:

cache TTL ≤ inbox ttl, so a cached id never outlives the row that backs it.

inbox ttl ≥ the longest time a duplicate can still arrive: the time the outbox can republish an event (retry backoff, plus the max-batch-processing-time of 300 s after which a stuck event is re-sent) and, when you replay or reset consumer offsets, the broker's retention.

The defaults make this concrete. A 24-hour inbox ttl covers every republish the outbox itself can produce, which happens within minutes. It also covers ordinary redelivery after a consumer restart. A replay from a Kafka topic that keeps seven days of history reaches further back than 24 hours, so a service that replays old offsets sets the inbox ttl to match the retention it replays from.

The outbox has a clean-up ttl of its own, also 24 hours by default, but it belongs to a different table. It removes PROCESSED rows from outbox_events and has no effect on deduplication. Neither does the DLQ's RESOLVED clean-up.

The whole thing in one file

A service with an outbox and an inbox needs the starter, exactly one database dialect module and one broker module. Everything else has a default, so a working configuration stays short:

oncebox: thread-pool-size: 5 outbox: sender: type: kafka defaults: batch-size: 200 polling: type: adaptive min-fixed-delay: 250ms max-fixed-delay: 1m multiplier: 1.5 events: create-order: topic: orders daily-report: topic: reports polling: min-fixed-delay: 10s max-fixed-delay: 5m dlq: enabled: true inbox: source: type: kafka clean-up: ttl: 24h cache: enabled: true cache-name: "orders-service:oncebox:inbox" 

Beyond the parts covered here, the library also ships a REST API for working through the dead letter queue (inspect, retry, resolve, delete), Micrometer metrics with cached gauges, and a fully containerized example project with a traffic generator.

Repository: github.com/dmitriy-iliyov/oncebox
Maven Central: io.github.dmitriy-iliyov:oncebox-starter
Example project: oncebox-example

Charts in this post are computed from the library's scheduling rule and its documented defaults. They do not show measurements from a running system.