Resilient systems, the endgame. Because failure is inevitable

This article is part of the XPRT. Magazine #21

Heads-up: I love cooking, so this article is generously flavoured with food references. Grab something tasty to make the read even better!

Imagine this: You invited some friends for dinner. You spent a week planning it: a delicious menu, fresh ingredients, fancy drinks, and a great playlist. You started the moussaka early in the afternoon. The meat sauce slow-cooked to perfection, rich and deep. The béchamel sauce came out silky and creamy. The table was set. The guests were already texting their ETAs. Everything was going according to plan.

Right when it's time to put the moussaka back into the oven, you realize it has stopped working. And just like that, the evening collapses. Not because the food was bad, not because you did anything wrong, but because resilience was never part of the plan. No backup dish, no alternative cooking method, not even a passing thought about what would happen if the one critical piece of equipment you depended on decided to quit. A single point of failure, and the whole system went down.

This is not a story about bad luck. It is a story about a plan that was optimized exclusively for the happy path.

We engineers spend a lot of time building systems that work brilliantly under normal conditions: smooth network traffic, healthy dependencies, stable APIs, databases that respond on time, you name it. The happy path is fast, elegant, and thoroughly tested. But then a traffic spike hits. A downstream service starts timing out. A misconfigured connection pool rolls out with a deployment. And suddenly this carefully constructed, well-monitored system falls over in a way that nobody anticipated, because the team designed it to work but not necessarily to fail.

The reality is simple: failure is not an edge case. It is inevitable. Every system will experience failures at some point. The question is whether you designed it to absorb that failure, adapt, and keep delivering value.

I hope that by the end of this article, you will have a clear understanding of what makes a system resilient, why resilience should be considered from day one, and how the five core principles can guide you towards building robust solutions.

What Is a Resilient System?

Before we go deeper into resilient systems, let's start by having a clear understanding about what resilience actually means and how it differs from related concepts.

Resilience is the ability of a system to withstand disruptions, gracefully degrade when necessary, recover quickly, and continue delivering essential value. Notice that this is not the same as "a system that never fails". The goal is not perfection, it is continuity.

Imagine being halfway through a recipe when you suddenly realize you are out of a key ingredient: a resilient home cook substitutes, adapts the dish, and still serves dinner. A cook who cannot deviate from the recipe stops entirely. Again, resilience is not about preventing failure. It is about designing for the moment failure happens.

Reliability and availability are often used interchangeably with resilience, but each addresses a fundamentally different problem.

A system can be highly reliable or highly available and still not be resilient. Reliability focuses on performing correctly under expected conditions, and availability focuses on being up and reachable. Both assume a well-behaved environment.

Resilience, however, is about what happens when something unexpected occurs. A system can pass every reliability test and maintain excellent uptime, yet still collapse when faced with a sudden disruption such as a regional outage, a configuration error, or a surge in traffic. It may operate flawlessly under normal circumstances but fail catastrophically the moment something unexpected occurs. Resilience fills this gap by ensuring the system keeps running even when everything around it starts falling apart.

Why Is It Important?

No serious restaurant bets the night's service on a single oven, a single chef, or a single supplier. That would be a fragile plan dressed up as confidence. Yet engineering teams make exactly that bet every time they ship a system with no fallbacks, no redundancy, and no plan for the moment things stop cooperating.

Understanding why resilience matters means looking at the same problem from three different angles: what it costs when things break, how it impacts user trust, and how it affects the team's velocity.

Cost

Every minute of downtime represents direct lost revenue, but the hidden costs are often larger: the on-call engineer pulling an all-nighter, the war room that eats a full sprint's worth of senior time, and the post-incident cleanup that never quite ends. Cascading failures make this exponential: one service going down shouldn't bring down two others.

However, the goal isn't maximum resilience everywhere. Investing heavily in resilience for a nightly batch job looks very different from doing the same for a payment gateway. Over-engineering a low-criticality service wastes resources; under-engineering a high-criticality one accumulates silent risk until something forces the conversation. Complexity added in the right places shrinks the blast radius of failure; complexity added in the wrong places just adds more things that can fail.

Consider a restaurant that suffers a major equipment failure mid-Saturday-service. The lost revenue from that evening is the smallest part of the bill. There are health inspections, press coverage, and a reputation that takes months to rebuild. The cost of that single incident consistently outweighs what a proper maintenance and contingency plan would have cost over years.

Trust

Trust is harder to quantify but easier to lose. Users who experience platforms that stay up, degrade gracefully, and recover quickly develop a form of loyalty that is difficult for competitors to displace. Conversely, users who experience hard failures develop a very specific kind of distrust, one that is difficult to recover from. Resilience, experienced repeatedly by users over time, becomes a brand property.

Imagine a restaurant that, on a busy evening, runs out of half the menu, serves cold dishes, and keeps customers waiting for forty-five minutes. The food might be excellent on every other night. But the reviews written that evening in frustration will be what new customers read for the next two years. One hard failure recovers slowly. Users remember outages the same way diners remember bad experiences.

Engineering velocity

Engineering velocity is the compounding dividend of investing in resilience early. Teams that invest in resilience early spend dramatically less time in reactive incident response as systems mature. That time reinvests into features, performance, and further resilience work. The upfront investment pays compound returns. Teams that do not make that investment find their velocity increasingly constrained by the operational debt of a fragile system.

Picture a kitchen where nothing is labelled, prep work happens ad hoc during service, and the team spends half their shift searching for ingredients. Orders take twice as long. The chef is too busy firefighting to refine the menu. The kitchen's capacity to improve is entirely consumed by its own disorder. Engineering teams in fragile systems work the same way: so much time goes to incident response and workarounds that there is none left for building what matters.

Ultimately, it should be presented to leadership for what it truly represents: resilience is risk management. Every shortcut that trades robustness for speed is a risk the organisation is quietly carrying, unpriced and growing. That bill eventually comes due, whether through an incident, a lost customer, or a competitor who simply built more carefully. Resilience is what makes a system, and the team behind it, sustainable for the long run.

5 Principles of Resilient Systems

Knowing that resilience matters is one thing. Knowing where to start is another. Resilience is not a single property you bolt on at the end; it is a set of design concerns, each targeting a different way a system can bend or break under stress. Here are the 5 principles I keep returning to when reviewing a system design.

Fault Tolerance

Imagine you are halfway through making a sauce and the blender gives up. If your kitchen is equipped with a hand blender in the drawer, dinner continues without drama. The dish gets to the table on time, and no one will know anything went wrong. The kitchen was set up so that no single utensil is the single point of failure for completing the meal.

In software, fault tolerance means building systems where components can fail without taking everything else with them. Here are some well-established patterns:

Hard question: "If this component fails right now, what exactly happens to the rest of the system?"

Graceful Degradation

You are back in the kitchen ready to finish the crème brûlée with a blowtorch. But the gas canister is empty, and you don't have a refill. Rather than cancelling dessert entirely, you pivot and sprinkle some cinnamon on top instead. It is not what you planned, but it is still delicious. A reduced experience is still a valuable one, and far better than no experience at all.

In software, graceful degradation means identifying the parts of your system that are essential versus enhancing and designing separate response strategies for each. These are some ways you can achieve this:

Hard question: "What is the minimum viable experience we can still provide when this dependency goes down?"

Automatic Recovery

Consider a well-organised kitchen during a busy service. There is a timer on every burner, a temperature probe in the oven, a smoke alarm that triggers the extractor fan the moment something starts to burn. The kitchen was designed to catch and contain small failures automatically, so cooking can continue without crisis. You are not expected to stand at the stove staring at every pot indefinitely.

This is critical in distributed systems at scale, because at a certain number of services and instances, human operators simply cannot respond to every failure fast enough to prevent user impact. In these scenarios, automatic recovery can be enabled by:

Hard question: "How long does it take to recover, and is human intervention required?"

Isolation

Imagine cooking four dishes simultaneously. The pasta water boils over, creating a mess. But because each burner and pot is independent, it does not ruin the roasting vegetables, the bread in the oven, or the salad being assembled on the counter. The mess stays contained. In a well-organised kitchen, one failing element does not cascade into a ruined meal.

When something goes wrong, the damage should stay bounded. A failure in one part of the system should not be able to cascade freely into everything else. Isolation can be implemented through the following patterns:

Hard question: "If this fails, what else fails with it, and why?"

Observability

An experienced cook uses all their senses while cooking. They smell the caramel before it burns. They listen to the sizzle to judge whether the pan is at the right temperature. They taste as they go. They do not wait until the dish is plated to discover something went wrong forty minutes ago. Without that constant, multi-channel monitoring, problems silently accumulate until it is too late to recover properly.

In software systems, observability is achieved via the following key pillars:

Hard question: "Could we diagnose and resolve this incident purely from the signals our system produces?"

Resilience From Day One

No serious cook waits until mid-service to check whether the oven works. Kitchens are tested, stocked, and set up long before the first guest walks through the door. The time to discover that the heat probe is unreliable is not while a rack of lamb is in the oven and eight people are waiting at the table. Resilience is preparation work, done in the calm before the dinner rush begins. Once you are in service, your options narrow dramatically.

The cost of retrofitting resilience

Retrofitting resilience into a live system is expensive in ways that are not immediately obvious:

A resilient system reflects a resilient mindset. Teams that openly acknowledge failure and treat it as a learning moment are the ones that truly get ahead.

Game days and chaos engineering are the engineering equivalent of a trial-run dinner: deliberately introducing problems, e.g., killing instances, saturating queues, injecting latency, to find out what breaks before your users do. Blameless postmortems are the honest debrief after something goes wrong: what was in the recipe, what was the technique, what should we do differently next time. The goal is learning, not blame. And defining SLOs (Service Level Objectives) before you build the first service is the equivalent of agreeing on what a great meal looks like before you go shopping. It sets the standard everyone is building towards.

Resilience-first architecture is not over-engineering. It is engineering done right. The systems that earn long-term trust are built by people who asked the uncomfortable question early: "What happens when this breaks?", and answered it with more than optimism.

Mise en Place

Mise en place is the French culinary practice of preparing and organising every ingredient, tool, and kitchen vessel before cooking begins. Professional kitchens treat it as non-negotiable: nothing starts until everything is in its place.

The engineering equivalent is resilience design as a prerequisite, not as an afterthought. Before the first service is deployed, define the failure modes. Design the fallbacks. Implement the health checks. Agree on the SLOs. Set up the dashboards.
Chefs who skip mise en place spend the whole service scrambling. Engineering teams that skip resilience design spend every incident doing the same.

What Resilience Looks Like in Practice

Theory is useful, but concrete examples are better. Let's walk through how the 5 principles manifest in a real (yet simplified) scenario.

Imagine an online platform where customers order home-cooked meal kits: curated ingredients and recipes delivered to the door so they can cook a restaurant-quality dinner at home. The platform has several services: order management, payment processing, recipe recommendations, delivery logistics, and notifications.

Here is how resilience thinking shapes the design decisions made before the first line of production code is written:

Notice what these decisions have in common: they were made at design time, i.e., in architecture reviews, whiteboard sessions, and early sprint planning, not in the chaos of an incident retrospective. Just like an experienced cook who prepares a backup dessert and confirms dietary requirements before guests arrive, the team designed for failure scenarios before the first user ever connected.

Resilience does not appear in systems by accident. It appears because someone asked the hard questions, discussed them openly with the team and decided to do something about it.

Conclusion

Let's come back to the dinner party. The evening did not have to end with cold moussaka and awkward apologies. A resilient host plans for the oven to fail, not because they expect it, but because they know any single point of failure is a risk. A backup dish in mind. An alternative cooking method. A pantry stocked for improvisation.

The same is true of the systems we build. Failure is inevitable. Resilience is not a destination, it is the endgame of every design review. The 5 principles we covered (fault tolerance, graceful degradation, automatic recovery, isolation, and observability) are not independent features to be added to a backlog one by one. They are a coherent philosophy of system design. A philosophy that says: My users are trusting me with something that matters to them. I owe them a system designed to hold up when reality stops cooperating. That is why taking the five hard questions seriously is not optional, it is the commitment behind every resilient system.

Here is where you start:

The goal is not to build a system that never fails. The goal is to build a system worthy of being trusted when it does.

An experienced cook who has planned well, prepared thoroughly, and built flexibility into the menu can host a dinner party with confidence, not because nothing will go wrong, but because they have already thought through what to do when it does.

Build your systems the same way :)