System Integration: The Failure Modes Nobody Budgets For

Integrating two systems is straightforward while everything works. The estimate is usually built on that assumption, and the estimate is usually wrong, not because the happy path is hard, but because the majority of integration code exists to handle everything that is not the happy path.

These are the failure modes worth designing for on day one, because retrofitting any of them means reworking the whole flow.

Everything happens more than once

This is the foundational one. Networks time out after the request succeeded but before the response arrived. Providers retry webhooks they are unsure you received. Someone double-clicks. A queue redelivers on restart.

So the same message will arrive twice, and if processing it twice does damage, whether that is a second payment, a duplicate order or two emails, you have a defect that appears intermittently in production and is miserable to reproduce.

The fix is idempotency, and it is structural rather than clever. Every inbound message needs a stable identifier from the sender. Record processed identifiers. On arrival, check first: if it is already recorded, acknowledge and do nothing.

The important detail is that recording the identifier and performing the work must be atomic. If you do the work, then crash before recording it, the retry does it again, which is precisely the case you were defending against. Same database transaction, or the guarantee is not real.

The other end will be down

Not might. Third-party APIs have maintenance windows, deploys and outages. Assume any external call can hang or fail.

• Set explicit timeouts. Default HTTP client timeouts are frequently infinite. One hung call then occupies a thread indefinitely, and enough of them take down a service that is itself perfectly healthy.
• Retry with exponential backoff and jitter. Immediate uniform retries from every client turn a brief blip into a stampede at the moment of recovery.
• Only retry what is safe to retry. Which returns you to idempotency, because retrying a non-idempotent operation is how one payment becomes three.
• Fail fast when it is clearly down. A circuit breaker that stops calling a dead service for thirty seconds serves users a quick, honest error instead of a queue of thirty-second timeouts.

Partial failure is the hard case

Step one succeeds, step two fails. You now have a half-completed operation spanning systems that share no transaction.

Distributed transactions are almost never the answer at this scale. What works:

• Order operations so the reversible part goes first. Record intent locally, then call the external system. If the call fails, you have a local record to retry from, rather than an external side effect with nothing tracking it.
• Make the local record the source of truth for what still needs doing. A pending row with a status is more reliable than an in-memory retry loop that a restart discards.
• Write compensating actions where reversal is possible. If the second step fails permanently, something must undo the first, and that logic has to exist before you need it.

Choose polling or push deliberately

Webhooks are efficient and near-real-time. They also fail silently: if your endpoint is down when the provider calls, the event may be retried a few times and then dropped forever. You do not know what you did not receive.

Polling is wasteful and slower, but self-healing, because a missed poll is corrected by the next one.

For anything that matters financially, use both: webhooks for latency, and a periodic reconciliation job that asks the provider what happened over the last interval and repairs anything missed. This is the pattern that separates integrations that quietly lose records from ones that do not.

Never trust the payload

Two distinct concerns get conflated here.

Authenticity. A webhook endpoint is a public URL that anyone can post to. If it acts on unverified input, anyone can trigger it. Verify the provider's signature over the raw request body, and note that it must be the raw bytes; parsing and re-serialising JSON changes the signature and the check silently stops meaning anything.

Shape. Validate against what you expect rather than assuming the documented schema. Providers add fields, change types and occasionally send nulls where the documentation promises a value. Fail loudly on unexpected input instead of propagating a null three layers into your domain.

Make it observable before you need to

Integration bugs are reported as "the order did not come through". Without a record of what was sent and received, diagnosis is guesswork across two systems, one of which you do not control.

Log every inbound and outbound message with its identifier, timestamp, and outcome. Give each flow a correlation ID that travels through both systems so a single request can be traced end to end. Alert on the absence of expected traffic, not just on errors. An integration that has silently stopped produces no errors at all, which is exactly why it goes unnoticed for a week.

Redact credentials and personal data in those logs. They are the most likely thing to be over-shared during debugging.

The non-technical failure modes

Integration projects stall for reasons that are not code.

The API is not what the documentation says. Undocumented rate limits, fields that behave differently in production, sandbox environments that diverge from live. Build a throwaway proof of concept against the real API early, before the design depends on assumptions you have not tested.

Nobody owns the other end. If the vendor's integration support is a general enquiries address, timelines are outside your control. Establish a named contact before committing to a date.

The data does not line up. Two systems each holding "customers" rarely agree on what identifies one. Deciding the matching key, and what happens when it is missing or ambiguous, is usually a business decision, not a technical one, and it needs to be made by someone with the authority to make it.

A rule of thumb

If an integration estimate is mostly the happy path, roughly double it. The remainder is idempotency, retries, reconciliation, observability and the data mismatches you have not found yet. That work is not optional, only deferrable.

Deferred, it arrives later as intermittent production bugs, which is the most expensive way to buy it.

We do this work: connecting payroll, accounting, payment and operational systems, including the ones you already run. If you have an integration that keeps losing records, get in touch.