Why Your ERP Integration Fails Silently Under Load

A hard failure — a crashed service, a revoked API key, a network partition — is visible within minutes. Monitoring catches it, someone pages the on-call engineer, and the fix is usually mechanical: restart the process, rotate the credential, restore the route.

Silent failures present differently. The integration continues to process most transactions successfully, but a fraction of the data — the fraction that arrived during a rate-limiting window, or that hit a timeout mid-batch — either disappears into an unmonitored dead letter queue or gets silently dropped by a retry mechanism that exhausted its attempts.

The typical sequence involves a spike in order volume that exceeds the ERP’s API ingest rate. The middleware queues the surplus. If the retry mechanism uses fixed intervals instead of exponential backoff with jitter, the retries cluster together and hit the same rate limit again — a pattern called a retry storm.

According to AWS Prescriptive Guidance, exponential backoff with added randomness reduces contention and prevents synchronized retries from amplifying load spikes (retry with backoff pattern — AWS). The messages that survive this cycle may eventually expire or land in a queue that nobody monitors. Finance discovers the gap when the ERP’s revenue total doesn’t match the storefront’s transaction log.

The root cause is almost never a single bug. It traces back to architectural decisions — sometimes deliberate, often accidental — made when the integration was first built for lower volume. The four decisions below determine whether the integration degrades gracefully or silently.

The most common ownership model assigns entire entities to a single system: “the ERP owns orders, the storefront owns products.” This breaks down the moment both systems write to the same entity. A customer updates their shipping address on the storefront after the order has been sent to the ERP. The ERP retains the original address. A nightly sync overwrites the storefront’s update with the ERP’s version, and the package ships to the wrong location. Neither system logged a conflict because neither system knew it was in one.

Field-level ownership resolves this by asking a more precise question: which system owns payment status? Which owns shipping address? Which owns inventory count? When these assignments are explicit, documented, and enforced by the integration layer as write permissions, bidirectional sync conflicts become deterministic. The system that owns the field is the only system permitted to write to it; every other system reads. Conflicts don’t disappear — they become visible and resolvable instead of silent and cumulative.

Diagnosing undefined field ownership

Pull the last 30 days of sync logs and search for cases where both systems wrote to the same field within the same sync cycle. If those cases exist and no conflict resolution rule was applied, field-level ownership is undefined. The fix does not require re-architecture — it requires a mapping document listing every shared field, its owning system, and the behavior the integration layer enforces when a non-owning system attempts a write. Most teams can produce this document in a working session and implement the enforcement in the middleware within a sprint.

Inventory, orders, customers, and prices have fundamentally different latency tolerances, consistency requirements, and failure modes. A single sync pattern applied uniformly across all four is a common root cause of silent failures — the pattern that works for customer records (hourly batch) will cause overselling when applied to inventory during a traffic spike.

Inventory requires near-real-time sync because the cost of overselling is immediate: cancelled orders, refund processing, and customer churn. Polling the ERP every 15 minutes works at low volume but creates a blind window during flash sales. Event-driven sync — where the WMS pushes a stock change the moment it occurs — closes this gap, though it adds operational complexity in the form of message broker infrastructure, monitoring, and replay capability.

Orders tolerate slightly more latency, but each one must arrive exactly once. Customer records are the most tolerant of eventual consistency, though bidirectional sync still requires conflict resolution rules. Prices become a special case during promotions: the storefront must be authoritative for the promo window, or the ERP’s master catalog will overwrite the discount before it ends.

When on-prem ERP adds sync complexity

When ERP infrastructure is on-premises while the storefront runs in the cloud, the sync model must also account for network reliability between environments. VPN tunnel drops, firewall timeouts, and bandwidth constraints introduce failure modes that don’t exist in cloud-to-cloud setups. Bluepes has documented these architectural considerations in their hybrid integration architecture for cloud and on-prem guide, covering message buffering during connectivity gaps and the trade-offs of gateway-mediated versus direct API integration.

ecommerce-erp-sync-models-by-data-type

Different e-commerce data types need different sync models: inventory needs near-real-time updates, orders need idempotent transactional delivery, customers can tolerate eventual consistency, and prices require clear source-of-truth rules.

If your integration is already showing reconciliation drift or inventory mismatches after traffic spikes, a focused technical review will clarify whether the issue is a sync model mismatch, a missing idempotency layer, or an observability gap. Bluepes provides dedicated development teams for ongoing integration monitoring and remediation. Start with a diagnostic call.

Making one side of the integration retry-safe is not sufficient. Both ERP → storefront and storefront → ERP must handle duplicate messages without creating duplicate records. If the storefront resends an order because it didn’t receive an acknowledgment and the ERP processes it as a new entry, the result is a duplicate fulfillment. If the ERP resends an inventory adjustment during a retry and the storefront applies it twice, the displayed stock count diverges from actual stock.

Idempotency in distributed systems relies on a unique identifier — a client token or idempotency key — attached to each operation. The AWS Builders’ Library documents this pattern: when an API receives a request with a client token it has already processed, it returns the original result without executing the operation again (making retries safe with idempotent APIs — AWS). This is distinct from application-layer deduplication, which checks after the write. Idempotency prevents the side effect in the first place.

The practical implementation requires two components per endpoint: a deterministic identifier generated at the source (a composite of order ID and timestamp, or a UUID generated once per operation), and a lookup on the receiving side that checks whether that identifier has already been processed before executing the write. For database operations, a conditional write works — INSERT ... ON CONFLICT DO NOTHING in PostgreSQL, or attribute_not_exists in a DynamoDB PutItem call. For third-party APIs, the identifier must be passed as a request header the receiving API recognizes.

On the Kortreistved marketplace project, Bluepes introduced idempotent keys in the supplier → order → inventory flow to prevent duplicate stock updates. This ensured that if webhook retries overlapped with scheduled batch synchronizations, the same operation could not be applied twice. The inventory adjustment endpoint simply rejected any request that contained an operation ID it had already processed, no matter which sync channel the request came from.

Diagnosing missing idempotency in a live system

Query your order table for duplicate entries with the same external order ID but different internal record IDs. If duplicates exist, the receiving endpoint is not checking for prior processing. For inventory, compare the count of adjustment events sent by the source system against adjustments applied at the destination — if the applied count exceeds the sent count, adjustments are being double-applied during retries. Both checks can run as SQL queries against production data without disrupting traffic.

The difference between a two-hour post-incident diagnosis and a two-week manual investigation comes down to whether the right data was being logged before the failure occurred. Most e-commerce integrations log success responses and hard errors. Almost none capture the information needed to reconstruct a silent failure: partial successes, retry attempts, queue depth trends, and the difference between what was sent and what was acknowledged.

The observability minimum for ERP integrations

Six data points, logged consistently, make the difference between guessing and diagnosing:

• Correlation ID per transaction — a single identifier tracing a record from storefront event through middleware into the ERP and back. Without this, debugging requires manual timestamp matching across systems.
• Queue depth at regular intervals — every 60 seconds under normal load, every 10 seconds during high-traffic windows. A queue depth that rises and doesn’t recover within 5 minutes signals a bottleneck.
• Retry state per message — attempt count, response code on each attempt, and backoff interval between retries.
• Dead letter queue depth with alerting at any value above zero — a message in the DLQ means a transaction was abandoned. According to Microsoft’s Azure Service Bus documentation, DLQs hold messages that couldn’t be delivered or processed, allowing inspection before data is permanently lost (Service Bus dead-letter queues — Microsoft Learn).
• Payload snapshots — the actual request body sent and the response received, retained for at least 72 hours. When a discrepancy surfaces in reconciliation, the payload is the only artifact showing whether the data left the source correctly.
• Sync lag as a tracked metric — the time delta between when a record changed at the source and when the change was confirmed at the destination. A sync lag that gradually increases over days indicates a capacity problem heading toward data loss.

Bluepes has published a detailed guide on what to instrument in their integration observability with Boomi article, covering correlation ID propagation, alerting thresholds, and dashboard design for integration health monitoring.

Strict field-level source-of-truth ownership works well in single-region setups. In multi-region e-commerce deployments with regional overrides — a European storefront adjusting VAT-inclusive pricing independently from the US storefront, for example — field ownership becomes conditional on region. The mapping document must include region as a dimension, and the integration layer must evaluate it at runtime. This adds complexity that a small engineering team may not be able to maintain.

Event-driven sync for inventory closes the latency gap but introduces operational overhead: a message broker (RabbitMQ, Amazon SQS, Azure Service Bus) requires its own monitoring, scaling, and failure handling. For a team of three to five engineers managing the entire e-commerce stack, maintaining a message broker may cost more than the occasional oversell that batch sync produces. The decision should be based on the actual revenue impact of overselling at current volume.

Implementing idempotency across every endpoint is the recommendation, but it requires cooperation from the receiving system. If the ERP’s API doesn’t support client tokens or conditional writes, idempotency must be enforced at the middleware layer — which means maintaining a deduplication store, typically a database table or cache with TTL. This is achievable, but it adds a stateful component to what might otherwise be a stateless integration service. For teams evaluating whether to absorb this complexity or adopt an iPaaS, the Boomi e-commerce integration guide covers the platform selection criteria in detail.

• Silent failures cost more than outages because they surface weeks later as reconciliation gaps, when diagnosis and correction are an order of magnitude more expensive.
• Define source of truth at the field level — entity-level ownership produces unpredictable overwrites the moment both systems write to the same record.
• Match each data type to its own sync model: inventory needs near-real-time events, orders need idempotent transactional delivery, customer records tolerate eventual consistency.
• Implement idempotency on both sides of the integration using deterministic operation IDs and conditional writes.
• Log correlation IDs, retry states, queue depth, DLQ depth, and payload snapshots before the first incident.

The four decisions covered here — field-level ownership, data-type-specific sync models, bidirectional idempotency, and pre-incident observability — share a common trait: each one is diagnosable in a running integration without taking it offline. The diagnostic steps in each section (sync log audits, duplicate record queries, queue depth monitoring, DLQ alerting) are designed to produce a clear picture of which gaps exist before any code changes begin.

Starting with an architecture change before completing the audit is how teams introduce new failure modes while trying to fix old ones. The reconciliation gap only widens as order volume increases, and the hidden costs of poor integration compound with every month of undetected data loss.

Bluepes engineers have diagnosed and resolved these failure modes across multiple e-commerce development projects, and can assess your current integration architecture in a focused technical review. Request an integration architecture assessment.