Design for Failure- Amazon CTO, Werner Vogels said at the first AWS user conference, “Everything fails all of the time.” Despite this warning, too many smart people continue to design IT services based on the idea that everything stays up all the time. The service has to stay up, but you can bet the house that some of the systems are going to fail. Your architecture needs to be built with the mindset that the service will survive even when components fail. Have you done a fail-case analysis for your service? What happens when individual components fail; what is the user experience if server X dies; what about both X and Y? If the very idea sends a chill of dread down your spine then you have not designed your service for failure. When I was working for Ted Cahall, then CTO at AOL, he put it best: “I’m a pessimist. As such, I’m either prepared or I’m pleasantly surprised. The best an optimist can hope for is to simply break even.” A pessimist designs for failure. The basic tenant of a resilient architecture is to decompose your service into small, loosely coupled, stateless building blocks. These building blocks all talk to each other via API’s and can be managed through dynamic configuration parameters so you don’t have to restart code just to make a change. If you have to log into your server to change something, your automation is broken. You cannot run at web-scale without a serious commitment to automation and tooling. At web-scale, Black Swans – those incredibly rare events – are commonplace. A data center with 64,000 servers, each with 2 disks, will average 5 servers and 17 disk failures each day. Your odds of a system failure are 80 times higher than winning the $1,000 prize with the $1 lottery scratch ticket (1:244,800). Make your users happy and your nights peaceful – design for failure. Brian Sullivan Scale Transformations of Global IT Systems
Posted on: Tue, 24 Jun 2014 09:55:42 +0000
Trending Topics
Recently Viewed Topics
© 2015