Work / Case study mock content, anonymised
Rebuilding order orchestration for a multi-brand retailer
Four storefront brands shared one order service that had grown for nine years. Every team deployed to it, nobody owned it, and a failed payment callback could leave an order in a state no screen could show. The brief was not “microservices”; it was “let brands release independently and let support see what happened to an order.”
- Role
- Technical Architect; design lead for two squads of five
- Duration
- 14 months, delivered in four increments
- Industry
- Retail and e-commerce
- Stack
- .NET 8, Kafka, PostgreSQL, Redis, Kubernetes on a private cloud
Outcome at a glance
- 4 → 1deploy approvals needed for a brand-level change
- 2read-model rebuilds in production, both uneventful
Business context
The group acquired its third and fourth brands within two years. Each came with its own storefront and its own promises to customers: next-day delivery, click and collect, split shipments. All of those promises ended up implemented as branches inside a single order service and a single shared database.
Releases were weekly, coordinated by a release manager, and increasingly cancelled. The business case was framed around time to market for a new brand and the cost of support tickets about “orders stuck in processing.”
What the business asked for
- Brands release on their own schedule
- Support can see an order’s full history
- Onboard a fifth brand in under a quarter
Architectural problem
The order lifecycle was spread across synchronous calls: storefront to order service, order service to payment provider, payment provider back to a webhook, webhook to warehouse API. Any link failing left the order in whatever state the last successful call had written. There was no record of attempts, only of final states.
Underneath that, the real problem was ownership. Pricing, stock, fulfilment and notification rules all lived in the same codebase and the same tables, so no team could change a rule without a cross-team review.
Constraints that shaped the design
- No freeze on the daily release train
- Legacy warehouse system stays, SOAP and all
- Two squads, not ten
- Private cloud only; no managed streaming
System map after the rebuild
Solid boxes are services the team owns. Dashed boxes are external or legacy. Arrows show the direction of commands and events, not every network call.
Key decisions and their trade‑offs
Decision 1
An event log as the only integration contract
Downstream domains subscribe to order events instead of reading order tables. Shared database access was removed by rotating credentials, not by asking nicely.
Cost: stock is eventually consistent; a nightly reconciliation job compares warehouse and order views and raises tickets for drift.
Decision 2
Orchestration, not choreography, for the order saga
A single orchestrator owns the order lifecycle so support can answer “where is this order” from one place. Choreography was tested and rejected because the failure paths were impossible to explain to the support team.
Cost: the orchestrator is a coordination point; it is stateless, horizontally scaled, and treated as the most reviewed code in the system.
Decision 3
Outbox pattern from the first service
Order writes and event publication share a transaction. A relay publishes to Kafka. No dual writes anywhere, even where it felt like overkill.
Cost: a little extra latency and one more table per service. Worth it the first time a broker was unavailable for 40 minutes.
Decision 4
Strangle the monolith by brand, not by feature
The smallest brand moved first, end to end. The monolith kept serving the other three until each moved. Feature-by-feature migration would have meant two code paths for every brand at once.
Cost: the largest brand ran on the old system for nine of the fourteen months.
Operational concerns
- 01Observability. Every event carries the order id as a trace attribute; the support console links straight to the trace.
- 02Replay. Read models can be rebuilt from the log; this was used twice in the first quarter to fix projection bugs.
- 03Dead letters. A small UI lists dead-lettered sagas with a one-click retry; on-call stopped needing database access.
- 04Capacity. Seasonal sale load tested by replaying the previous year’s log at 3× speed.
Outcomes and lessons
4 → 1
deploy approvals needed for a brand-level change
2
read-model rebuilds in production, both uneventful
What I would do differently: start the support console earlier. It became the thing that made the architecture legible to non-engineers, and it arrived in month nine. And I would push harder on removing the third payment provider instead of wrapping it.
Working on something similar?
If your order flow, release process or event backbone looks like this, I am glad to talk it through.