Keeping Shopify and the Back Office in Agreement

Connecting Shopify to the systems around it gets scoped as plumbing. Pick the app, map the fields, watch the first order land in the other system, mark it done.

It usually holds for a while. What arrives later is quieter: the store sells three of something that has one left, a customer is told an order shipped before it did, and finance is reconciling by hand every month because two reports have never quite matched. None of these announce themselves as an integration matter. They present as service issues.

The nightly export stops being enough sooner than expected

A batch that runs overnight means the store spends the day working from yesterday’s numbers.

At low volume that is invisible, because the numbers barely move. The moment a product sells faster than the batch runs, the store is quoting stock it no longer has, and the gap is widest exactly when it costs the most: a launch, a promotion, the week you actually wanted to go well.

Nobody agrees what stock means

Before deciding how systems talk, it is worth noticing that they are often not even measuring the same thing. Take one SKU on an ordinary afternoon:

Every one of these is correct within its own system. The store is not wrong about what it holds, it simply has no view of what has been promised. It will happily sell all forty, and twelve of those customers will hear about it later.

Shopify models this properly, which is worth knowing before you build around it. Inventory is held per location, and a quantity has a state: on hand, committed, available, incoming. Available is a derived figure rather than a number somebody types, and it is the only one a storefront should be selling against. An integration that writes to the available figure directly, rather than recording the commitment that produced it, is overwriting the platform’s arithmetic with its own and will disagree with itself within a week.

Multi-location turns the question from how many into how many, where, and reachable by whom. A single national figure is a fiction the moment two warehouses can both fulfill, and a store selling against the sum of all locations will promise stock that cannot reach the customer in the window it quoted.

One owner per field, not one owner per system

The useful decision is not which system is the source of truth. It is which system owns each field.

Stock levels are usually owned by whatever physically holds the goods. Contract pricing is owned by the system where the contract lives. Customer consent is owned by wherever the customer gave it. Order status is owned by whatever is actually moving the box. Product copy and imagery are usually owned by whoever merchandises, which is often neither of the systems being connected.

Trouble concentrates wherever two systems both believe they own the same field, because then the last write wins and the last write is a matter of timing. A price corrected in the ERP gets overwritten by a stale storefront value, and the correction has to be made twice before anyone notices why.

Writing this down as a list of fields and owners takes an afternoon and prevents most of what follows. It is also the point at which you find the fields nobody owns, which are the ones already drifting.

Where a field genuinely has to be writable in two places, and occasionally one does, the mechanism is compare and set rather than write and hope. Read the current value with a version, send the change conditional on that version still being current, and handle the rejection by rereading rather than by retrying blindly. That turns a silent overwrite into a visible conflict, which is the outcome you want.

The same message, twice, out of order

Shopify delivers webhooks at least once, which means sometimes more than once, and not reliably in the order events happened. Three separate pieces of discipline follow from that, and they are usually collapsed into one and done badly.

Verify it before you parse it

A webhook endpoint is a public write path into your systems. It has to check the signature Shopify sends against the raw request body, using a constant time comparison, before the payload is deserialized or trusted for anything.

The detail that catches people is the word raw. Frameworks that helpfully parse JSON before your handler runs will have changed the bytes, and a signature computed over reserialized JSON will not match. The verification has to see the body exactly as it arrived.

Make it safe to repeat

Every write should land the same way whether it happens once or five times. In practice that means keying the work on the event rather than on the moment it arrived, and keying the record on an identifier the other system also knows:

// The event id is the idempotency key. Retained longer than the provider's
// retry window, so a delivery two days late is still recognized as a repeat.
if (await seen(event.id)) return ok();

// Keyed on the external id, never on an auto-increment of our own, so a
// repeat updates the row it created the first time instead of adding another.
await orders.upsert({ where: { shopifyId: event.orderId }, data: fields(event) });

await remember(event.id, { ttl: days(7) });
return ok();

Two failure modes worth calling out. Marking the event seen before the work completes turns a crash into a permanently skipped record. Marking it seen after the work completes, without the upsert being keyed externally, turns a retry into a duplicate. The pair has to be considered together.

Decide whether it is stale

Out of order delivery is the piece most integrations ignore, and it is why a record sometimes reverts to a value somebody corrected an hour ago.

Every event should carry something monotonic, a version or an updated timestamp, and the handler should compare it against what is already held and discard anything older. Last write wins is a reasonable policy, but only once you can tell which write was actually last, and arrival order does not tell you that.

When one of them is down

Something in the chain will be unavailable. A warehouse system takes maintenance, an ERP is upgraded on a weekend, a network path fails for six minutes.

Accept the message quickly, put it on a durable queue, and do the delivery from there. An endpoint that tries to reach three downstream systems before it answers will start timing out, and a provider that times out retries, which means the slowest part of your chain is what decides how much duplicate traffic you receive.

Retries want exponential backoff with jitter. The backoff is obvious. The jitter is the part that gets left out, and it is what stops every queued message arriving at the same instant the moment a system comes back, which is a second outage caused by the recovery from the first.

Separate what is worth retrying from what is not. A timeout or a server error is transient and should be tried again. A validation rejection will be rejected identically forever, and a queue retrying it every minute is a stall that looks like activity. Those belong in a dead letter queue with the response attached, and somebody has to be looking at that queue, because a dead letter queue nobody reads is a folder of lost orders.

What should never happen is the message quietly disappearing, which is the default in a surprising number of setups and the reason some orders exist in one system and not the other with no record of the difference. An outage should slow the flow down. It should not lose anything.

Reconciliation, because drift arrives anyway

Even a careful integration drifts. A manual adjustment in one system, a canceled order handled in the wrong place, a product edited directly.

So compare the two sides on a schedule and state the difference plainly. Three comparisons cover most of it: a count of records on each side, a sum of a meaningful figure such as units or order value, and a set difference on identifiers to name what exists in one place and not the other. Counts and sums are cheap and catch the common faults. The set difference is what tells you which record to go and look at.

Compare a window that has stopped moving. A job that reconciles up to the current instant will report differences that are simply messages in flight, the report will cry wolf, and within a month nobody will open it. Hold a watermark and reconcile up to it.

Read in bulk rather than record by record. Shopify’s rate limiting is cost based rather than a flat number of requests, so a job that pages naively through a catalog asking for every field will exhaust its budget and slow to a crawl, while the same job requesting only the fields it compares will finish comfortably. For a genuinely large reconciliation, the bulk operation route exists precisely so a full extract does not have to be paged.

Resist correcting automatically, at least at first. A reconciliation job with write access and a subtly wrong idea of who owns a field can propagate a fault faster than any human could. Report, watch it for a while, and automate only the corrections you have watched be right.

When a simple sync is the right answer

Often. One warehouse, moderate volume, no ERP, a catalog that does not change much: a well-configured app will do this properly and the money is better spent elsewhere. We will say so.

The case for building it changes when the count matters minute to minute, when more than two systems are involved, when contract pricing or multi-region stock enters the picture, or when the cost of a single oversell is high enough that being right occasionally is not good enough.

Where to start

List the fields that appear in more than one system, and write one owner beside each. Where you cannot decide, that is where the work is, and where the numbers are drifting today.