For decades, technology leaders have chased impossible goals: system components that never go down, software updates that never cause problems, and technology that works everywhere. But after 48 years of building and running complex systems, I’ve learned something you can count on: Some degree of failure is inevitable.

Networks fail. Hardware breaks. Air-conditioning units overheat. Power drops. Generators don’t come online as expected even after regular testing. Software behaves in ways no one predicted. Sometimes it’s a bug, sometimes a storm, sometimes an adversary. The specifics don’t matter, but your response does.

If a system isn’t secure and available, none of its wonderful features matter. Resilience doesn’t come from preventing failure. It comes from surviving it well.

Once you accept that failure is normal, your focus shifts. Instead of wondering if something will break, you try to manage how much breaks when it does. Software has to be written with the expectation that the underlying infrastructure will eventually let you down.

Resilient systems don’t rely on “strong” hardware or perfect networks. They assume the opposite. Services and traffic spread across multiple servers are more forgiving by design. If one component fails, traffic shifts. If one system degrades, the rest keep running. The system isn’t reliant on any single piece to remain online.

In practice, this changes how you deploy. Physical infrastructure gets distributed widely enough to survive natural disasters, so a single event doesn’t take everything down. After you finish alpha, beta, and gamma testing, you don’t push code everywhere at once and hope for the best. You start with one server and load it up. If it fails, you roll back and fix it. When you’re confident, you expand to one zone. Then another. Then you repeat, region by region—always with the option to fail and go back to the prior stage.

The most resilient systems I’ve seen intentionally isolate services, separate workloads, and assume that trust must be continuously earned.

Practices like multi-availability zones—running a system across physically separate data centers within the same region—and multi-region deployment aren’t about redundancy for its own sake. They’re about containment. Zero-trust principles reinforce the same idea from a security perspective: Never assume a component is healthy just because it exists inside the perimeter.

And then there’s defense in depth. Real resilience takes more than a single layer of protection. It stacks multiple, independent, and heterogeneous layers—network paths, firewalls, encryption mechanisms—so a flaw in one doesn’t compromise them all.

This matters even more in national security and critical infrastructure environments, where failure is operationally disastrous. Systems must continue to perform without failing completely and keep working long enough for humans to intervene.

Teams recover faster with systems that feature built-in monitoring to track performance, identify problems, and show what is happening in real time. They know what “normal” looks like, so anomalies stand out immediately. In most environments, a small number of root causes drive most outages. Resilient teams identify those causes, automate them away, and keep working until issues become less frequent and less severe.

That discipline doesn’t just live in incident calls—it shapes how we build systems in the first place. Booz Allen-developed platforms like Recreation.gov are distributed across regions, observable under stress, and designed to learn every time something goes wrong.

Now, AI and automation are supercharging both the process and the payoff. AI-assisted operations can accelerate log analysis, find patterns humans might miss, and speed diagnosis when seconds matter. Over time, with the right inputs, agentic systems could become self-healing—detecting and automatically correcting issues as they happen.

Resilience also requires knowing what matters most. Not every system deserves the same level of protection, and pretending otherwise wastes time and money.

Before an outage happens, leaders need to classify their applications, defining which are mission critical, which are business critical, and which can tolerate some downtime. Those decisions shouldn’t be made mid-crisis.

Every level of resilience carries tradeoffs in cost, performance, and complexity. Decide ahead of time what risks you’re willing to take. Resilience should be intentional, not reactive.

These lessons matter even more now.

Today’s systems don’t live in a single data center or cloud. They span massive cloud computing platforms, on-premises networks, computers that process data on site, and devices operating far from reliable connectivity. AI and autonomy raise the stakes even further.

In national security and defense missions, systems must operate even when the network is degraded, jammed, or gone entirely. Resilience can’t depend on constant connectivity, perfect data, or centralized control. It must be designed from the start—and strengthened over time. Organizations that thrive aren’t chasing perfection—they’re prepared for imperfection.

This piece is adapted from Booz Allen Chief Technology Officer Bill Vass’ column in the Summer 2026 issue of Booz Allen’s Velocity magazine.