A live map has one constraint that changes the design: after a disconnect, the client must distinguish an event it never received from an event it received and then lost from memory. A faster transport cannot answer that question. Only an ordered, replayable delivery contract can.
Short answer: Assign every device-status event a stable identity and a monotonic position within its stream, acknowledge batch members individually, and reconnect from the last contiguous position rather than the largest position observed.
That choice treats a batch as a transport envelope, not a transaction. If positions 410 through 419 arrive and 414 fails validation, the dashboard may render later positions provisionally, but its durable checkpoint remains 413. The server can then backfill from 414 without guessing, while deduplication makes the repeated 415 through 419 harmless. This is the same conservative instinct used for a ledger: progress means a contiguous, auditable prefix, not a collection of optimistic successes.
How should realtime event delivery recover from batch partial failures?
Start by separating three facts that are often collapsed into one boolean: whether the producer accepted an update, whether the delivery service durably recorded it, and whether a particular dashboard applied it. An accepted: true response can establish the second fact only if the response identifies the exact member; it cannot establish the third. Likewise, a socket write completing says little about what survived a browser refresh. Don't promote transport completion into business completion.
For a medical-device delivery tracking map, a useful stream key is the tracked shipment or device journey, not the entire tenant. Each event needs an immutable event ID, a stream ID, a stream position, an occurrence time, and the status payload. The position orders changes within that one journey; it does not pretend to establish a universal order among unrelated deliveries. A correction is a new event that refers to the affected state rather than an edit that erases history. That preserves the evidence needed for reconciliation and any retention or access controls required by the applicable compliance regime. The exact retention period is a policy decision, not something the realtime protocol should invent.
The batch result must have the same granularity as the input. One response shape can classify each member as accepted, already accepted, or rejected, with a stable reason for rejection. A whole-batch success flag loses information. A whole-batch retry is acceptable only when event IDs make it idempotent; otherwise a single malformed member turns retry into duplicate state transitions.
Keep the checkpoint rule strict: advance it while next_position == checkpoint + 1, then stop at the first gap.
The batch contract is an accounting boundary
The ingest path should validate identity and sequence metadata before publishing anything downstream. It may process members independently, yet it should return one result per input event so the caller can reconcile cardinality as well as status. If the caller submits ten events and receives nine results, the response itself is incomplete and none of its apparent success should silently close the audit record.
A compact state table makes the retry rule explicit:
| Member result | Producer action | Consumer effect |
|---|---|---|
| Accepted | Record the event ID and position | Apply once when its turn arrives |
| Already accepted | Treat as successful retry | Ignore the duplicate payload |
| Rejected, permanent | Quarantine with the reason | Do not advance across its position |
| Outcome unknown | Retry with the same event ID | Deduplicate if the first attempt committed |
The permanent-rejection row deserves care. If position 414 is structurally invalid, retrying the same bytes forever cannot repair it, but silently skipping it would make the displayed route look complete when it is not. The operator needs an audited resolution: replace the invalid update with a valid, newly identified correction under a documented policy, or mark that position as deliberately resolved. Which option is lawful for device data depends on the organization's recordkeeping rules. I'm not sure a generic transport library can make that decision correctly; the evidence needed is the data classification and the approved correction policy.
Exactly-once delivery is the wrong external promise. Networks can lose acknowledgements, so a producer may have to send the same event again. The attainable contract is an idempotent write plus an exactly-once state transition at the consumer: insert the event ID under a uniqueness constraint, apply the transition in the same local transaction, and return the prior result when the ID already exists. Short version: retry facts, not guesses.
A Go consumer that refuses to checkpoint across a gap
The following state machine is deliberately smaller than a full websocket server. It shows the correctness boundary: duplicate IDs do nothing, out-of-order events wait, and the durable checkpoint moves only through a contiguous run. Persistence implementations should commit seen, pending events, derived status, and checkpoint atomically; the in-memory maps here express the contract rather than a storage recommendation.
package delivery import ( "errors" "sync" ) type Event struct { ID string StreamID string Position uint64 Status string } type Cursor struct { mu sync.Mutex checkpoint uint64 seen map[string]struct{} pending map[uint64]Event status string } func NewCursor(checkpoint uint64) *Cursor { return &Cursor{ checkpoint: checkpoint, seen: make(map[string]struct{}), pending: make(map[uint64]Event), } } func (c *Cursor) Apply(e Event) (uint64, error) { c.mu.Lock() defer c.mu.Unlock() if e.ID == "" || e.StreamID == "" || e.Position == 0 { return c.checkpoint, errors.New("missing event identity or position") } if _, duplicate := c.seen[e.ID]; duplicate { return c.checkpoint, nil } if e.Position <= c.checkpoint { c.seen[e.ID] = struct{}{} return c.checkpoint, nil } c.seen[e.ID] = struct{}{} c.pending[e.Position] = e for { next := c.checkpoint + 1 event, ready := c.pending[next] if !ready { break } c.status = event.Status delete(c.pending, next) c.checkpoint = next } return c.checkpoint, nil } There is a subtle collision the sample leaves to the persistence layer: two different event IDs must not be allowed to claim the same stream position. Enforce uniqueness on both (stream_id, position) and event_id, then retain the original payload or its integrity digest alongside the decision. Otherwise a retry with a mutated payload can masquerade as ordinary deduplication. Auditability requires being able to explain which bytes won and why.
The dashboard may still show a later location marker while a gap is open, but it should label that view as provisional and keep the durable resume token at the contiguous checkpoint. For some clinical workflows, even provisional rendering may be inappropriate; in that case hold later events in the reorder buffer and show that the feed is catching up. This is a product-policy choice with safety implications, not a transport optimization.
Reconnect is a bounded backfill protocol
On reconnect, the client presents its stream ID and last contiguous checkpoint. The server returns events after that position in ascending order, in bounded pages, before switching the connection to live delivery. The handoff needs a defined boundary so an event created during backfill is neither omitted nor applied without ordering: capture a high-water position, replay through it, and then subscribe beyond it. If live events arrive early, buffer them by position until backfill reaches the boundary.
Bound the buffer.
A device that reconnects after a long offline interval may be millions of positions behind, and letting its pending map grow without limit converts one stale tab into a memory-pressure incident. Set limits for page size, pending event count, replay duration, and maximum supported history based on load tests and retention policy. When a client exceeds a limit, stop incremental replay and fetch a versioned snapshot plus events after the snapshot position. The catch is that snapshots trade fine-grained reconstruction for bounded recovery time; they are not suitable when an auditor must reproduce every intermediate dashboard state from the client artifact alone. Stick with event replay when that reconstruction is mandatory and the retained log can meet the recovery objective.
Backpressure belongs in this protocol too. A client should advertise or imply how much work it can accept, and the server should not keep extending an unacknowledged window. Acknowledgements should report the highest contiguous applied position, not a bag of recently seen positions. That single scalar is conservative, easy to persist, and sufficient to restart replay. Selective gap information can accelerate repair, but it must not replace the contiguous cursor.
WebRTC data channels are one possible peer-to-peer transport, and the W3C recommendation defines their association and data-channel behavior. They do not remove the application-level need for event identity, durable replay, idempotent application, or an audit trail. The transport choice comes after the recovery contract.
Test the fractures, then roll out by cohort
A happy-path load test proves almost nothing here. Exercise partial acceptance at every member of a batch, a lost acknowledgement after durable acceptance, duplicate delivery with the same ID, conflicting IDs at one position, disconnects before and after checkpoint persistence, events arriving beyond a gap, snapshot handoff, and reconnect storms. Assert both the rendered device status and the audit record after every test; a green map with an unexplained cursor is still a failed test.
Operational metrics should expose gap age, pending-buffer size, duplicate rate, permanent rejection count, replay page latency, reconnect attempts, and the distance between the stream high-water position and each active client's checkpoint. Alert on sustained age and distance rather than raw reconnect count alone. Reconnects are expected. A gap that never closes is the incident.
Rollout can stay compact: first shadow-compute the new contiguous checkpoint without changing display state, then compare it with the existing resume behavior; next enable replay for an internal cohort with capped buffers; finally expand by cohort while preserving the ability to reset a client from a versioned snapshot. Record protocol version, rollout cohort, cursor transitions, and operator resolutions in the audit trail. Roll back the reader, not the event history.
The resulting design does not promise that every batch succeeds. It promises something more useful: every member has an attributable outcome, retries are idempotent, gaps remain visible, and a reconnect has one defensible place from which to resume.


