2021 Facebook outage

Last updated

Traffic volume for Facebook services on October 4, 2021 with a drop during the global outage Facebook-outage-traffic-dropoff (cropped).png
Traffic volume for Facebook services on October 4, 2021 with a drop during the global outage

On October 4, 2021, at 15:39 UTC, the social network Facebook and its subsidiaries, Messenger, Instagram, WhatsApp, Mapillary, and Oculus, became globally unavailable for a period of six to seven hours. [1] [2] [3] The outage also prevented anyone trying to use "Log in with Facebook" from accessing third-party sites. [4] It lasted for 7 hours and 11 minutes.

Contents

During the outage, many users flocked to Twitter, Discord, Signal, and Telegram, resulting in disruptions on these sites' servers. [9] The outage was caused by the loss of IP routes to the Facebook Domain Name System (DNS) servers, which were all self-hosted at the time. [10] [5] Border Gateway Protocol (BGP) routing was restored for the affected prefixes at about 21:50, and DNS services began to be available again at 22:05 UTC, with application-layer services gradually restored to Facebook, Instagram, and WhatsApp over the following hour, with service generally restored for users by 22:50. [11]

Causes

Major DNS resolvers returning "SERVFAIL" status for Facebook.com Facebook DNS Outage 2021.png
Major DNS resolvers returning "SERVFAIL" status for Facebook.com

Security experts identified the problem as a Border Gateway Protocol (BGP) withdrawal of the IP address prefixes in which Facebook's Domain Name System (DNS) servers were hosted, making it impossible for users to resolve Facebook and related domain names, and reach services. [3] [12] Effects were visible globally; for example, Swiss Internet service provider Init7 recorded a massive drop in internet traffic to the Facebook servers after the change in the Border Gateway Protocol. [13]

Cloudflare reported that at 15:39 UTC, Facebook made a significant number of BGP updates, including the withdrawal of routes to the IP prefixes, which included all of their authoritative nameservers. This made Facebook's DNS servers unreachable from the Internet. By 15:50 UTC, Facebook's domains had expired from the caches in all major public resolvers. A little before 21:00 UTC, Facebook resumed announcing BGP updates, with Facebook's domain name becoming resolvable again at 21:05 UTC. [14]

On October 5, Facebook's engineering team posted a blog post explaining the cause of the outage. During maintenance, a command was run to assess the global backbone capacity, and that command accidentally disconnected all of Facebook's data centers. While Facebook's DNS servers ran on a separate network, they were designed to withdraw their BGP routes if they could not connect to Facebook's data centers, making it impossible for the rest of the internet to connect to Facebook. [15]

Facebook gradually returned after a team got access to server computers at the Santa Clara, California, data center and reset them. [5] By about 22:45 UTC, Facebook and related services were generally available again. [16]

Impact

The outage cut off Facebook's internal communications, preventing employees from sending or receiving external emails, accessing the corporate directory, and authenticating to some Google Docs and Zoom services. [2] [3] The New York Times reported that employees were unable to access buildings and conference rooms with their security badges. [2] The site Downdetector, which monitors network outages, recorded over 10 million problem reports – the largest number for an incident to date. [17] [18] Steve Gibson, a security researcher, said a "Routine BGP update went wrong" locking out "people with remote access" to the servers to fix the mistake and people with physical access do not have authorization to fix the mistake. [19]

The Google Public DNS service also slowed down as a result of the outage, [20] while users of Gmail, TikTok, and Snapchat also experienced slowdowns. [21] CNBC reported that the outage was the worst experienced by Facebook since 2008. [22] During the day of the outage, shares in the company dropped by nearly 5% and Facebook CEO Mark Zuckerberg's wealth fell by more than $6 billion. [22] [23] [24] According to a report produced by Fortune and Snopes, Facebook lost at least $60 million in advertising revenue. [25]

The outage had a major impact on people in the developing world, who depend on Facebook's "Free Basics" program, affecting communication, business and humanitarian work. [26]

Response

Facebook's chief technology officer, Mike Schroepfer, wrote an apology after the downtime had extended to several hours, saying, "Teams are working as fast as possible to debug and restore as fast as possible." [27]

Alexandria Ocasio-Cortez, a U.S. representative from New York, tweeted about the outage, asking people to share "evidence-based" stories on Twitter, making fun of Facebook's reputation for spreading factually questionable content. [28] Twitter and Reddit also posted tweets on their official Twitter accounts commenting on the outage. [18]

Users on both Twitter and Telegram reported a slowdown in response times, believed to be caused by people normally on Facebook services switching to those services. [27]

Some media outlets highlighted the coincidence of Frances Haugen's testimony with that of the outage, [16] although those two events are unrelated to each other. [29]

See also

Related Research Articles

<span class="mw-page-title-main">Anycast</span> Network addressing and routing methodology

Anycast is a network addressing and routing methodology in which a single IP address is shared by devices in multiple locations. Routers direct packets addressed to this destination to the location nearest the sender, using their normal decision-making algorithms, typically the lowest number of BGP network hops. Anycast routing is widely used by content delivery networks such as web and name servers, to bring their content closer to end users.

BGP hijacking is the illegitimate takeover of groups of IP addresses by corrupting Internet routing tables maintained using the Border Gateway Protocol (BGP).

<span class="mw-page-title-main">Internet censorship in India</span> Overview of Internet censorship in India

Internet censorship in India is done by both central and state governments. DNS filtering and educating service users in suggested usages is an active strategy and government policy to regulate and block access to Internet content on a large scale. Measures for removing content at the request of content creators through court orders have also become more common in recent years. Initiating a mass surveillance government project like Golden Shield Project is an alternative that has been discussed over the years by government bodies.

<span class="mw-page-title-main">Facebook</span> Social networking service owned by Meta Platforms

Facebook is a social media and social networking service owned by American technology conglomerate Meta Platforms. Created in 2004 by Mark Zuckerberg with four other Harvard College students and roommates Eduardo Saverin, Andrew McCollum, Dustin Moskovitz, and Chris Hughes, its name derives from the face book directories often given to American university students. Membership was initially limited to Harvard students, gradually expanding to other North American universities. Since 2006, Facebook allows everyone to register from 13 years old, except in the case of a handful of nations, where the age limit is 14 years. As of December 2022, Facebook claimed 3 billion monthly active users. As of October 2023, Facebook ranked as the 3rd most visited website in the world, with 22.56% of its traffic coming from the United States. It was the most downloaded mobile app of the 2010s.

Facebook has been the subject of criticism and legal action since it was founded in 2004. Criticisms include the outsize influence Facebook has on the lives and health of its users and employees, as well as Facebook's influence on the way media, specifically news, is reported and distributed. Notable issues include Internet privacy, such as use of a widespread "like" button on third-party websites tracking users, possible indefinite records of user information, automatic facial recognition software, and its role in the workplace, including employer-employee account disclosure. The use of Facebook can have negative psychological and physiological effects that include feelings of sexual jealousy, stress, lack of attention, and social media addiction that in some cases is comparable to drug addiction.

<span class="mw-page-title-main">Roskomnadzor</span> Russian government agency

The Federal Service for Supervision of Communications, Information Technology and Mass Media, abbreviated as Roskomnadzor (RKN), is the Russian federal executive agency responsible for monitoring, controlling and censoring Russian mass media. Its areas of responsibility include electronic media, mass communications, information technology and telecommunications, supervising compliance with the law, protecting the confidentiality of personal data being processed, and organizing the work of the radio-frequency service.

<span class="mw-page-title-main">History of Facebook</span> History of the social networking service

Facebook is a social networking service originally launched as TheFacebook on February 4, 2004, before changing its name to simply Facebook in August 2005. It was founded by Mark Zuckerberg and his college roommates and fellow Harvard University students, in particular Eduardo Saverin, Andrew McCollum, Dustin Moskovitz, and Chris Hughes. The website's membership was initially limited by the founders to Harvard students, but was expanded to other colleges in the Boston area, the Ivy League, and gradually most universities in the United States and Canada, corporations, and by September 2006, to everyone with a valid email address along with an age requirement of being 13 or older.

<span class="mw-page-title-main">Internet outage</span> Loss of internet functionality over a small or large area

An Internet outage or Internet blackout or Internet shutdown is the complete or partial failure of the internet services. It can occur due to censorship, cyberattacks, disasters, police or security services actions or errors.

<span class="mw-page-title-main">Instagram</span> Social media platform owned by Meta Platforms

Instagram is a photo and video sharing social networking service owned by Meta Platforms. It allows users to upload media that can be edited with filters, be organized by hashtags, and be associated with a location via geographical tagging. Posts can be shared publicly or with preapproved followers. Users can browse other users' content by tags and locations, view trending content, like photos, and follow other users to add their content to a personal feed. A Meta-operated image-centric social media platform, it is available on iOS, Android, Windows 10, and the web. Users can take photos and edit them using built-in filters and other tools, then share them on other social media platforms like Facebook. It supports 32 languages including English, Spanish, French, Korean, and Japanese.

<span class="mw-page-title-main">WhatsApp</span> Messaging and VoIP service owned by Meta Platforms

WhatsApp is an instant messaging (IM) and voice-over-IP (VoIP) service owned by technology conglomerate Meta. It allows users to send text, voice messages and video messages, make voice and video calls, and share images, documents, user locations, and other content. WhatsApp's client application runs on mobile devices, and can be accessed from computers. The service requires a cellular mobile telephone number to sign up. In January 2018, WhatsApp released a standalone business app called WhatsApp Business which can communicate with the standard WhatsApp client.

Internet censorship circumvention, also referred to as going over the wall or scientific browsing in China, is the use of various methods and tools to bypass internet censorship.

<span class="mw-page-title-main">DDoS attacks on Dyn</span> 2016 cyberattack in Europe and North America

On October 21, 2016, three consecutive distributed denial-of-service attacks were launched against the Domain Name System (DNS) provider Dyn. The attack caused major Internet platforms and services to be unavailable to large swathes of users in Europe and North America. The groups Anonymous and New World Hackers claimed responsibility for the attack, but scant evidence was provided.

<span class="mw-page-title-main">Meta Platforms</span> American multinational technology conglomerate

Meta Platforms, Inc., doing business as Meta, and formerly named Facebook, Inc., and TheFacebook, Inc., is an American multinational technology conglomerate based in Menlo Park, California. The company owns and operates Facebook, Instagram, Threads, and WhatsApp, among other products and services. Meta ranks among the largest American information technology companies, alongside other Big Five corporations Alphabet (Google), Amazon, Apple, and Microsoft. The company was ranked #31 on the Forbes Global 2000 ranking in 2023.

<span class="mw-page-title-main">Google services outages</span> Outages of Googles online services

During eight episodes, one in 2013, one in 2014, one in 2018, three in 2020, and two in 2022, Google suffered from severe outages that disrupted a variety of their services. The first was a five-minute outage of every Google service in August 2013. The second was a 25-minute outage of Gmail, Google+, Google Calendar, and Google Docs in January 2014. The third was a YouTube outage in October 2018. The fourth was a Gmail/Google Drive outage in August 2020. The fifth, in November 2020, affected mainly YouTube, and the sixth, in December 2020, affected most of their services. The seventh, in August 2022, affected Google Search, Maps, Drive and YouTube. The eighth affected Google Maps and Google Street View. These outages seemed to be global.

Downdetector is an online platform that provides users with real-time information about the status of various websites and services.

Kentik is an American network observability, network monitoring and anomaly detection company headquartered in San Francisco, California.

Doug Madory is an American Internet routing infrastructure expert, who specializes in analyzing Internet Border Gateway Protocol (BGP) routing data to diagnose Internet routing disruptions, such as those caused by communications fiber cable cuts, routing equipment failures, and governmental censorship. His academic background is in computer engineering, and he was a signals specialist in the U.S. Air Force, before arriving at his present specialty, which has occupied his professional career.

<span class="mw-page-title-main">Threads (social network)</span> Social networking service owned by Meta Platforms

Threads is an online social media and social networking service operated by Meta Platforms. The app offers users the ability to post and share text, images, and videos, as well as interact with other users' posts through replies, reposts, and likes. Closely linked to Meta platform Instagram and additionally requiring users to both have an Instagram account and use Threads under the same Instagram handle, the functionality of Threads is similar to X. The application is available on iOS and Android devices; the web version offers limited functionality and requires a mobile app to be installed first. It is the fastest-growing consumer software application in history, gaining over 100 million users in its first five days, surpassing the record previously set by ChatGPT. Its early success was not sustained and the user base of the app plummeted more than 80% to 8 million daily active users by the end of July.

<span class="mw-page-title-main">2024 Facebook outage</span> Outage affecting all Facebook operated services

On March 5, 2024, the social network Facebook and its subsidiaries Messenger, Instagram and Threads became globally unavailable. The messaging app WhatsApp, owned by the same company, Meta Platforms, was unaffected.

References

  1. Madory, Doug (October 5, 2021). "Facebook's historic outage, explained". Kentik. Archived from the original on January 4, 2022. Retrieved October 7, 2021.
  2. 1 2 3 "Locked out and totally down: Facebook is scrambling to fix massive outage". The Verge. October 4, 2021. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  3. 1 2 3 Lawler, Richard (October 4, 2021). "Facebook is down, along with Instagram, WhatsApp, Messenger, and Oculus VR". The Verge . Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  4. Barrett, Brian. "Why Facebook, Instagram, and WhatsApp All Went Down Today". Wired. ISSN   1059-1028. Archived from the original on October 5, 2021. Retrieved October 5, 2021.
  5. 1 2 3 Isaac, Mike; Frenkel, Sheera (October 4, 2021). "Facebook and all of its apps go down simultaneously". The New York Times . Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  6. "Millions Flock to Signal and Telegram After Facebook Outage". Bloomberg.com. October 5, 2021. Archived from the original on October 16, 2021. Retrieved October 6, 2021.
  7. "Discord down? Current problems and outages". Downdetector . Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  8. Dewar, Caitlyn (October 4, 2021). "Is Twitter down?: Users report outage after Facebook, WhatsApp and Instagram go down". The Herald . Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  9. [5] [6] [7] [8]
  10. Madory, Doug (October 4, 2021). "Facebook suffers global outage". Kentik. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  11. Frenkel, Sheera. "Facebook coming back online after lengthy outage". The New York Times. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  12. Duffy, Clare; Lyngaas, Sean (October 4, 2021). "Facebook, Instagram, WhatsApp go down". CNN Business . Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  13. "Insta und WhatsApp down – so stark schrumpfte der Internet-Traffic". watson.ch (in German). Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  14. Strickx, Tom; Martinho, Celso (October 4, 2021). "Understanding How Facebook Disappeared from the Internet". Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  15. "More details about the October 4 outage". October 5, 2021. Archived from the original on October 5, 2021.
  16. 1 2 "Facebook, Whatsapp and Instagram back after outage". BBC News. October 4, 2021. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  17. Patnaik, Subrat; Mathews, Eva (October 4, 2021), Facebook, Instagram appear to partly reconnect after nearly six-hour outage, Reuters, archived from the original on October 5, 2021, retrieved October 5, 2021
  18. 1 2 "Facebook, Whatsapp and Instagram suffer outage". BBC News. October 4, 2021. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  19. Gibson, Steve [@SGgrc] (October 4, 2021). "Someone on the Facebook recovery effort has explained that a routine BGP update went wrong, which in turn locked out those with remote access who could reverse the mistake. Those who do have physical access do not have authorization on the servers. Catch-22" (Tweet). Archived from the original on October 5, 2021. Retrieved October 5, 2021 via Twitter.
  20. Schoon, Ben (October 4, 2021). "Google's 8.8.8.8 DNS service slows down amid massive Facebook outage". 9to5Google . Archived from the original on October 4, 2021. Retrieved October 5, 2021.
  21. Massie, Graeme (October 4, 2021). "Gmail, TikTok and Snapchat Users Complain Apps Slowing Down During Facebook Outage". The Independent . Archived from the original on October 4, 2021. Retrieved October 5, 2021.
  22. 1 2 "Facebook is suffering its worst outage since 2008". CNBC. October 4, 2021. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  23. Carpenter, Scott (October 4, 2021). "Zuckerberg Loses $6 Billion in Hours as Facebook Plunges". Bloomberg. Archived from the original on October 5, 2021. Retrieved October 5, 2021.
  24. "Facebook Outage Causes Share Drop, Zuckerberg Loses US$5.9B Net Worth – October 5, 2021". Daily News Brief. October 5, 2021. Archived from the original on October 5, 2021. Retrieved October 5, 2021.
  25. Wong, Queenie. "Facebook's massive outage costs the company an estimated $60 million in revenue". CNET . Archived from the original on October 5, 2021. Retrieved October 5, 2021.
  26. Schapiro, Avi Asher; Teixeira, Fabio (October 5, 2021). "Small businesses to aid: How Facebook outage upended lives". Thomson Reuters . Archived from the original on October 6, 2021. Retrieved October 6, 2021.
  27. 1 2 Massie, Graeme (October 4, 2021). "Twitter users suggest site is slowing down as it takes pressure from Facebook and Instagram outage". The Independent. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  28. O'Connell, Oliver (October 4, 2021). "AOC mocks Facebook outage by asking people to share positive 'evidence-based' stories on rival platform Twitter". The Independent. Archived from the original on October 4, 2021. Retrieved October 4, 2021.
  29. Siberling, Amanda (October 5, 2021). "Facebook whistleblower Frances Haugen testifies before the Senate". TechCrunch . Archived from the original on October 10, 2021. Retrieved October 10, 2021.