Imagine this.
Your payment service is working perfectly. Your order service is healthy. Your inventory database is responding normally. Every unit test passes, and your monitoring dashboard is green.
A customer places an order.
The payment provider processes the payment. Your application receives a webhook, updates the order, reduces inventory, and sends a confirmation email.
Everything looks good.
Except the payment provider never receives your application's acknowledgement. It retries the webhook. Your application processes the same event again.
Now the inventory is reduced twice. The customer receives duplicate notifications. Your finance team notices a payment reconciliation mismatch.
Which service failed?
Probably none of them.
The problem happened between them.
This is one of the most overlooked challenges in software engineering. Teams spend considerable time making individual services reliable, but the real complexity often appears when those services communicate.
A service can be healthy while the business workflow depending on it is broken.
Let's examine why this happens and how to design integrations that survive real production conditions.
1. A Successful API Response Doesn't Guarantee a Successful Workflow
Developers often think about integrations in terms of endpoints.
An application sends a request, another service processes it, and a response comes back. That sounds straightforward.
Production environments introduce conditions that are difficult to reproduce during local development.
Consider these situations:
- A request reaches the receiving service, but the response gets lost.
- A service completes an operation after the caller has already timed out.
- A third-party API becomes temporarily unavailable.
- Two events arrive in an unexpected order.
- A background worker processes the same message more than once.
- One service completes its operation while another service fails halfway through.
These are not necessarily coding mistakes. They are consequences of distributed systems.
A timeout does not prove that an operation failed. A successful HTTP response does not prove that every downstream operation completed.
The application must account for uncertainty.
One useful design principle is to assume that communication can fail at any point, even when individual services are reliable.
2. Five Integration Problems That Cause Production Incidents
Problem 1: Retrying Requests Without Idempotency
Retries are essential to building reliable applications.
When an API request fails because of a temporary network issue, retrying can recover the operation without requiring manual intervention.
But retries introduce another problem.
What happens when the original request was successfully processed, but the response never reached the calling application?
The caller assumes the operation failed and sends the same request again.
Without idempotency, the receiving application may perform the operation twice.
Consider a payment request:
POST /payments Idempotency-Key: 8e03978e-40d5-43e8-bc93-6894a57f9324 Content-Type: application/json The server should associate this key with the original logical operation.
When it receives the same request again, it should return the previously recorded result instead of creating another payment.
There are three important implementation details:
- The same logical operation must reuse the same idempotency key across retries.
- The server should persist the key and its processing result.
- Concurrent requests using the same key must be handled atomically.
It is also worth storing a request fingerprint. Reusing the same key with a different payment amount or payload should be rejected rather than silently returning an unrelated result.
For event consumers, use a stable event identifier and enforce uniqueness in persistent storage.
The important distinction: Idempotency is not simply checking whether a request was received before. It is ensuring that repeating an operation does not repeat its business effect.
Problem 2: Timeouts That Leave Operations in an Unknown State
Developers often treat a timeout as a failed request.
But a timeout only tells you that the caller did not receive a response within the configured period.
The receiving service might have:
- Never received the request.
- Received it but failed during processing.
- Completed the operation but failed to return a response.
- Started a background operation that is still running.
Blindly retrying every timeout is dangerous, particularly for financial transactions and operations that modify business records.
A better approach is to design important operations around explicit states.
For example:
{ "operation_id": "op_84721", "status": "processing" } Possible states include:
- Pending
- Processing
- Completed
- Failed
- Requires review
The caller can use the operation ID to check the actual result rather than blindly creating a second operation.
For transient failures, apply bounded retries with exponential backoff and jitter. Do not retry permanent errors such as invalid input or authorization failures.
A circuit breaker can also stop repeated calls to an unhealthy dependency and give it time to recover.
For financial operations, reconciliation is particularly important. When the outcome is uncertain, check the original transaction's status before attempting another operation.
Problem 3: API Contracts Change Without Warning
Your team maintains an order service. Another team maintains a shipping service.
The shipping service expects:
{ "customer_id": 1001, "order_id": 84721 } A developer updates the order service and renames customer_id to user_id.
The application still compiles. Unit tests pass. The change looks harmless.
But the shipping service starts rejecting requests.
This is a contract problem.
An API is not just an implementation. It is an agreement between the producer and its consumers.
Changes to that agreement can break applications even when the updated service is functioning correctly.
How to prevent it
Use API specifications such as OpenAPI to document request and response structures.
Introduce consumer-driven contract tests where appropriate. These tests verify that changes made by a service provider do not violate the expectations of its consumers.
Before releasing a change, consider:
- Are fields being renamed or removed?
- Are optional fields becoming mandatory?
- Have enum values changed?
- Are response types still compatible?
- Will older application versions continue working?
Prefer additive changes where possible. When breaking changes are unavoidable, introduce a migration strategy with clear versioning and a transition period.
Independent deployment is useful only when services can tolerate differences between their release schedules.
Problem 4: Events Arrive in an Unexpected Order
Event-driven architecture allows services to publish events without waiting for every downstream operation to finish.
But asynchronous processing introduces ordering challenges.
Imagine an account management system publishing these events:
AccountCreated AccountUpdated AccountSuspended If messages are consumed concurrently, the receiving application might process them in a different order.
For example:
AccountCreated AccountSuspended AccountUpdated An older update could overwrite information after the account has already been suspended.
This is not a universal problem with every messaging system. Ordering behavior depends on the broker, partitioning strategy, message key and consumer configuration.
How to prevent it
Use an ordering strategy based on the business entity.
For example:
- Partition messages using an account or order ID.
- Include a monotonically increasing sequence number or entity version.
- Reject stale updates when business rules require ordering.
- Avoid processing dependent events concurrently without coordination.
- Make consumers resilient to duplicates and delayed messages.
Do not rely solely on timestamps to determine which update is newer. Clock differences and event-processing delays can make that unreliable.
An explicit version number is usually a better choice when strict entity-level ordering matters.
Problem 5: Integration Permissions Become a Security Weakness
Imagine an analytics service that only needs read access to customer records.
But the credentials it receives allow it to modify and delete those records.
The integration works. Nothing appears wrong.
Until a compromised service, configuration mistake or unexpected bug modifies data it should never have been able to touch.
Every integration creates a security boundary.
How to prevent it
Apply least-privilege access.
Give services only the permissions they need. Use separate identities, secure credential storage, short-lived tokens where supported, and regular secret rotation.
For webhook integrations, validate signatures and timestamps where supported. This helps prevent forged requests and replay attacks.
Also consider:
- Encrypting sensitive information in transit.
- Avoiding unnecessary personal data in event payloads.
- Maintaining audit logs for important actions.
- Applying rate limits to external integrations.
- Validating authorization at the receiving service.
Authentication identifies who is making a request. Authorization determines what that identity is allowed to do.
Both matter.
3. Build Integration Reliability Into the Architecture
Fixing individual failures is useful. Repeatedly fixing the same category of failure usually indicates an architectural problem.
Several patterns help address these issues.
Use an Adapter Layer
Avoid spreading third-party API logic throughout your application.
Instead, introduce an adapter that handles:
- External request and response mapping.
- Authentication.
- Error translation.
- Retry policies.
- Rate limits.
- Provider-specific behavior.
Your application should communicate through a consistent internal interface rather than depend directly on the implementation details of every external provider.
For example:
Business Application | v Internal Interface | v Adapter Layer / | \ / | \ CRM API ERP API Payment API This approach is particularly useful when integrating older enterprise applications with modern services.
It also makes it easier to replace an external provider without changing the application's business logic.
Use the Transactional Outbox Pattern
Consider an order service that saves an order in its database and then publishes an OrderCreated event.
Two independent operations are taking place:
- Save the order.
- Publish the event.
What happens if the database transaction succeeds but the message broker is unavailable?
The order exists, but downstream services never receive the event.
What if the application publishes the event first and the database transaction subsequently fails?
A downstream service might process an order that does not exist.
The transactional outbox pattern addresses this dual-write problem.
Instead of updating the database and publishing an event independently, the application saves both the business change and an outbox record in the same database transaction.
A simplified structure might look like this:
CREATE TABLE outbox_events ( id VARCHAR(100) PRIMARY KEY, aggregate_id VARCHAR(100) NOT NULL, event_type VARCHAR(100) NOT NULL, payload JSON NOT NULL, created_at TIMESTAMP NOT NULL, published_at TIMESTAMP NULL ); The order creation process becomes:
- Start a database transaction.
- Insert the order.
- Insert an
OrderCreatedrecord into the outbox. - Commit the transaction.
A background publisher reads unpublished records and sends them to the message broker.
The business change and the event record now succeed or fail together.
However, there is another important failure scenario.
The publisher sends an event successfully but crashes before marking it as published. When it restarts, it might send the same event again.
That means the outbox does not automatically provide exactly-once processing.
This is why the consumer also needs deduplication.
Use an Inbox or Processed-Event Mechanism
The receiving service can maintain a table of processed event identifiers.
For example:
CREATE TABLE processed_messages ( consumer_name VARCHAR(100) NOT NULL, message_id VARCHAR(100) NOT NULL, processed_at TIMESTAMP NOT NULL, PRIMARY KEY (consumer_name, message_id) ); The consumer then processes the event inside a database transaction:
await db.transaction(async (tx) => { const result = await tx.query( `INSERT INTO processed_messages (consumer_name, message_id, processed_at) VALUES (?, ?, CURRENT_TIMESTAMP) ON CONFLICT DO NOTHING RETURNING message_id`, [consumerName, event.id] ); if (result.rowCount === 0) { return; // Already processed } await applyBusinessChange(tx, event); }); This example assumes a database that supports ON CONFLICT DO NOTHING and RETURNING, such as PostgreSQL.
The processed-message record and the business change must commit in the same transaction. Otherwise, a crash between them can still create inconsistent results.
The practical design is usually at-least-once message delivery combined with idempotent processing.
That is much more realistic than assuming that duplicate delivery can never happen.
4. When One Business Transaction Spans Multiple Services, Consider Sagas
The transactional outbox solves a local database-to-message problem.
But it does not automatically make an entire workflow across multiple services atomic.
Consider an online order involving three systems:
Order Service | v Inventory Service | v Payment Service | v Shipping Service The inventory service reserves stock. The payment service authorizes payment. The shipping service creates a shipment.
What happens when payment succeeds but shipping creation fails?
The customer has an authorized payment, inventory is reserved, and no shipment exists.
This is where the Saga Pattern becomes useful.
A saga coordinates a sequence of local transactions across multiple services. When a step fails, compensating operations help restore a valid business state.
For example:
| Step | Operation | Failure recovery |
|---|---|---|
| 1 | Create pending order | Mark order failed |
| 2 | Reserve inventory | Release reserved stock |
| 3 | Authorize payment | Void authorization |
| 4 | Create shipment | Retry or initiate compensation |
| 5 | Confirm order | Complete workflow |
A compensation is not necessarily a literal rollback.
For example, a completed payment capture may require a refund rather than an operation that erases the original transaction.
There are two common ways to coordinate a saga.
Choreography: Each service publishes events that trigger the next service. This works well for relatively simple workflows, but complex flows can become difficult to follow.
Orchestration: A central workflow coordinator controls the sequence, tracks execution state and decides what happens when a step fails. This gives better visibility for complex business processes but introduces another component to maintain.
The important part is that every step needs a defined success condition, failure behavior and recovery strategy.
For a deeper explanation, see Microsoft's Saga distributed transactions pattern.
5. Integration Reliability Gets More Important With AI Agents
AI applications are moving beyond generating responses.
They can retrieve business information, interact with APIs, update records and trigger workflows.
That introduces a different level of responsibility.
A chatbot returning an incorrect answer is a problem.
An AI agent making an incorrect change to a customer record or triggering the wrong business operation can have a much bigger impact.
The underlying integration still needs:
- Scoped permissions.
- Reliable API contracts.
- Idempotent operations.
- Human approval for sensitive actions.
- Auditable execution history.
- Clear failure and recovery behavior.
The AI component does not remove the need for traditional software engineering. It makes reliable integration even more important.
For developers building these systems, two related reads on VoidSyntax are:
- AI Agents Explained: How They Work and Real Examples
- MCP Server Best Practices: Building Secure and Reliable AI Agent Integrations
Both explore how AI applications connect with external tools and business systems, including the permissions and reliability considerations involved.
6. Legacy Applications Make Integration More Complicated
Modern applications usually have several integration options, including REST APIs, webhooks, message brokers and standardized authentication.
Legacy applications are not always that flexible.
Some expose SOAP interfaces. Others rely on scheduled file transfers, direct database access or proprietary communication protocols. Some have very limited integration capabilities.
Replacing every legacy system is rarely practical.
A more realistic approach is to introduce an integration layer that translates between older systems and newer applications.
For example:
Legacy ERP | v Integration Layer | +---------+----------+ | | | Validation Mapping Security | | | +---------+----------+ | v Modern APIs | v Business Workflows This keeps legacy-specific behavior isolated instead of allowing it to spread throughout the application.
It also creates a path for gradual modernization.
You can improve the integration layer, replace individual components and introduce newer APIs without rebuilding the entire business system at once.
For organizations connecting AI services with existing enterprise systems, VoidSyntax AI Integration Services covers the role of API, database, CRM, ERP and legacy-system connectivity.
7. How to Investigate an Integration Failure in Production
Let's return to the duplicate payment and inventory problem from the introduction.
Instead of immediately restarting services or replaying every failed message, investigate the business operation from beginning to end.
Here is a practical sequence.
Step 1: Find the original operation
Start with the order ID, payment ID or event ID.
Find the original request and identify every downstream operation associated with it.
Step 2: Follow the request across services
Use distributed tracing and trace context propagation to understand how the request moved through the system.
A trace ID helps connect spans belonging to a distributed request. A business correlation ID can help connect related operations across a longer-running workflow.
They are related, but they serve different purposes.
Step 3: Check the actual business state
Do not assume that a timeout means a payment failed.
Check the payment provider, local database and order status. Determine whether the operation was completed, partially completed or never processed.
Step 4: Inspect retries and message delivery
Check:
- Number of delivery attempts.
- Duplicate event identifiers.
- Queue depth and message age.
- Dead-letter queue records.
- Consumer processing failures.
Step 5: Compare database changes with published events
Check whether the business record and corresponding outbox event exist.
Look for events that remain unpublished or records marked as processed without the expected business change.
Step 6: Recover safely
Once the actual state is known, retry only operations that are safe to retry.
Use idempotency keys, processed-event records and reconciliation procedures to prevent additional side effects.
A useful incident investigation should establish not only where the error appeared, but also which business operations actually completed.
8. Production Readiness Checklist
Before releasing an integration, ask your engineering team these questions.
Reliability
- Can the operation handle duplicate requests safely?
- What happens when the receiving service becomes unavailable?
- Are retries bounded and configured appropriately?
- Can failed events be recovered or replayed?
Data consistency
- Are API contracts documented?
- Are schema changes backward compatible?
- Can events arrive out of order?
- Is there a reconciliation process for incomplete operations?
- Are distributed workflows designed with appropriate compensation?
Security
- Does each service have only the permissions it needs?
- Are credentials stored securely?
- Are webhook signatures validated?
- Is sensitive information excluded from unnecessary logs?
Monitoring
- Can we follow a request across services?
- Do we monitor queue depth and retry frequency?
- Can we identify operations stuck in processing?
- Do we receive alerts before users report problems?
Recovery
- Can failed operations be retried safely?
- Is there a dead-letter queue or equivalent failure-handling mechanism?
- Can we identify partially completed workflows?
- Is there a documented recovery procedure?
If your team cannot answer these questions, the integration probably needs more engineering work before it is ready for production.
Final Thoughts
Good software architecture is not just about building services that work independently.
It is about making sure they continue working together when conditions are less than ideal.
A service might pass every test, respond within milliseconds and maintain excellent uptime. But if its communication with another service is unreliable, the business process built on top of it will eventually fail.
The difference between an application that works in development and one that survives years in production often comes down to how well it handles these boundaries.
So the next time a production issue appears, do not investigate only the service that reported the error.
Look at what happened before the request reached it, what happened after it responded, and what assumptions were made between those steps.
Your next production bug might not be inside your code. It might be hiding in the conversation between your services.
Further Reading
For developers who want to explore the underlying patterns in more depth:
