{"id":1848,"date":"2010-09-24T08:51:57","date_gmt":"2010-09-24T15:51:57","guid":{"rendered":"http:\/\/www.talesfromthe.net\/jon\/?p=1848"},"modified":"2010-09-24T08:51:57","modified_gmt":"2010-09-24T15:51:57","slug":"facebook-outage-an-unfortunate-handling-of-an-error-condition","status":"publish","type":"post","link":"https:\/\/2024.thenexus.today\/index.php\/2010\/09\/24\/facebook-outage-an-unfortunate-handling-of-an-error-condition\/","title":{"rendered":"Facebook outage: &#8220;an unfortunate handling of an error condition&#8221;"},"content":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" class=\"alignright\" style=\"margin-left: 5px; margin-right: 5px; float: right;\" src=\"http:\/\/farm1.static.flickr.com\/225\/503165914_a680a56c77.jpg\" alt=\"\" width=\"200\" height=\"75\" \/>Facebook was down for a couple hours yesterday, <a href=\"http:\/\/www.cnn.com\/2010\/TECH\/social.media\/09\/23\/facebook.down\/\">the second day in a row<\/a> they&#8217;ve had problems.\u00c2\u00a0 In <a href=\"http:\/\/www.facebook.com\/notes\/facebook-engineering\/more-details-on-todays-outage\/431441338919\">More Details on Today&#8217;s Outage<\/a>, Director of Software Engineering Robert Johnson explains:<\/p>\n<blockquote><p>The key flaw that caused this outage to be so severe was an unfortunate  handling of an error condition. An automated system for verifying  configuration values ended up causing much more damage than it fixed.<\/p><\/blockquote>\n<p>Don&#8217;t you just hate it when that happens?<\/p>\n<p><!--more-->It&#8217;s a very clear post, detailing how things went downhill:<\/p>\n<blockquote><p>We had entered a feedback loop that didn\u00e2\u20ac\u2122t allow the databases to recover.<\/p>\n<p>The  way to stop the feedback cycle was quite painful &#8211; we had to stop all  traffic to this database cluster, which meant turning off the site. Once  the databases had recovered and the root cause had been fixed, we  slowly allowed more people back onto the site.<\/p><\/blockquote>\n<p>Reliability issues &#8230; emergence of <a href=\"http:\/\/www.talesfromthe.net\/jon\/?p=1819\">open source competitors<\/a> &#8230; business practices that push the envelope &#8230; <a href=\"http:\/\/dashes.com\/anil\/2010\/09\/the-facebook-reckoning-1.html\">arrogance<\/a> &#8230;\u00c2\u00a0 hmm &#8230;<\/p>\n<p>Reminds me a lot of Microsoft in the 1990s.<\/p>\n<p>jon<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Facebook was down for a couple hours yesterday, the second day in a row they&#8217;ve had problems.\u00c2\u00a0 In More Details on Today&#8217;s Outage, Director of Software Engineering Robert Johnson explains: The key flaw that caused this outage to be so severe was an unfortunate handling of an error condition. An automated system for verifying configuration [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1848","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/posts\/1848","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/comments?post=1848"}],"version-history":[{"count":0,"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/posts\/1848\/revisions"}],"wp:attachment":[{"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/media?parent=1848"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/categories?post=1848"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/2024.thenexus.today\/index.php\/wp-json\/wp\/v2\/tags?post=1848"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}