2023 Optus outage

Last updated

2023 Optus outage
Date8 November 2023 (2023-11-08)
Time04:05 (AEDT)
Duration04:05 – 16:00 (approx. 12–13 hours)
LocationAustralia (nationwide)
Type Network outage
CauseAutomated shutdown due to BGP event. [1]
Outcome
  • Disruption across government and private sectors
  • Landline-based Emergency 000 calls failing
  • A$2 billion stock loss to Optus
  • Investigations including Senate inquiry into the outage
Inquiries Senate inquiry

The 2023 Optus outage was an unplanned outage of all Optus internet, cellular and fixed-line services in Australia first detected on 8 November 2023 at 04:05 AEDT. [2] The outage caused disruption across government and corporate sectors including hospitals, banks, train services, EFTPOS payment systems, and calls to emergency services. [2] [3] Restoration was gradual with some services being restored around 13:00 AEDT. [4] The outage directly affected more than 10 million people and 400,000 businesses across Australia. [5] [6]

Contents

The cause was likely due to several routers disconnecting as an automated protective measure against routing update overload. [1] Optus stated that the outage was not due to a cyberattack. [2] [7]

Background

Optus is an Australian telecommunications company owned by Singtel. It is the second-largest wireless carrier in Australia, serving over 10 million customers in part with a 4G network that is reachable by 96.9% of the population. [8] Around 25% of Australians use Optus for their cellular service and 19% for broadband internet. [9]

Summary

Outage

The outage begun early morning on Wednesday, 8 November at around 04:05 AEDT following a planned software upgrade to a Singtel North American exchange. [1] [10] The customer network begun disconnecting first at around 04:05 and some domain name services stopped working.

Restoration

Restoration was a large-scale effort across more than 100 devices at 14 sites nationwide and took several hours. [11] [12] [1] The recovery was performed remotely and in some cases required physical access to sites. Services were restored gradually with 11% restored by 11:30am AEDT and 88% by 13:00 AEDT. [1] By 18:00 AEDT, Optus said that most of their services had been restored though connectivity was intermittent until 9 November 2023. [4] [1]

Impact

The outage impacted all internet, cellular and fixed-line services using the Optus network in Australia. [13] Public and private sectors, such as banks, hospitals and some train services, were impacted. Some people were unable to call emergency services. [14] Many businesses were not able to take digital payments via EFTPOS. [7] Dodo and Amaysim, who both rely on the Optus network as resellers, experienced outages across the country. [14] Aussie Broadband and Moose Mobile customers were also affected. [15]

A citywide Metro Trains Melbourne shutdown lasted from 05:00 to 06:00 due to drivers not being able to communicate with the control centre, causing hundreds of trains to be cancelled throughout the day on the Melbourne suburban network. [16]

Singtel lost over AU$2 billion in value over the day of the outage after its share price dropped 4.8% on the SGX. [4]

Response

On 9 November 2023, Optus CEO Kelly Bayer Rosmarin announced that customers on post-paid mobile plans would be able to access 200 GB of extra mobile data, stating that mobile data was "much more valuable" than refunding customers. The 200 GB of data must be activated between 13 November 2023 and the end of the calendar year. An Optus spokesperson later stated that eligible prepaid mobile users are being given unlimited data on weekends for the remainder of the year, beginning on 18 November. The spokesperson noted that if any businesses were "uniquely impacted" during the outage, they should contact Optus' Business Centre directly. [17]

Despite this, user sentiment was overwhelmingly negative, with many businesses being forced to shut down for the day and incur massive losses. [18] [19] [20] In addition to Optus suffering an A$2 billion stock loss (more than 5%), rival telecom companies such as Telstra and Vodafone saw an increase in customers due to the mass exodus, with some industry analysts, such as JPMorgan Chase, predicting that Telstra alone could increase their earnings projections by A$125 million over the next few years if even just 2.5% of customers sever their ties with Optus. [21] [22] On 20 November 2023, Bayer Rosmarin resigned as CEO. [23]

Reactions

The outage raised concerns about the move to a cashless society and Australia's over-reliance on internet access. [24]

South Australia's government is reconsidering using Optus as its service provider. [12]

The federal government has called for financial compensation to those affected by the outage. [25] Communications Minister Michelle Rowland announced a review into the matter. [26] The Australian Senate also passed a motion to hold a public Senate inquiry into the outage. [27]

Causes

A Border Gateway Protocol (BGP) routing problem played a role in the outage. [28] [29] [30] Public data from CloudFlare showed a spike in BGP route announcements from the Optus network around the time the outage occurred — over 940,000 announcements in an hour from a node that normally makes less than 3,000 announcements per hour — indicative of a BGP routing problem. [29] [31]

A submission tabled to the Australian Senate Standing Committee on Environment and Communications committee describes the outage as a gradual event triggered by loss of connectivity between neighbouring computer networks. [1] The report suggests that approximately 90 edge provider routers disconnected as an automated protective measure against routing update overload. The failures occurred following a software upgrade at a North American Singtel exchange that caused one of the routers to disconnect. This, in turn, triggered Optus's routers to rapidly update its own routing tables which triggered the shutdown due to the pre-configured default threshold limits set by Cisco Systems being exceeded. The tabled report and Singtel stressed that the software upgrade was not the cause of the fault. [1] [10]

See also

Related Research Articles

<span class="mw-page-title-main">Telecommunications in Singapore</span> Description of telecommunications in Singapore

The telecommunication infrastructure of Singapore spans the entire city-state. Its development level is high, with close accessibility to the infrastructure from nearly all inhabited parts of the island and for all of the population, with exceptions. Today, the country is considered an international telecommunications hub, an achievement that was driven by Singapore's view that high-quality telecommunications is one of the critical factors that support its economic growth.

<span class="mw-page-title-main">Telstra</span> Australian telecommunications company

Telstra Group Limited is an Australian telecommunications company that builds and operates telecommunications networks and markets related products and services. It is a member of the S&P/ASX 20 and Australia's largest telecommunications company by market share.

<span class="mw-page-title-main">Telecommunications in Australia</span> Overview of telecommunications in Australia

Telecommunications in Australia refers to communication in Australia through electronic means, using devices such as telephone, television, radio or computer, and services such as the telephony and broadband networks. Telecommunications have always been important in Australia given the "tyranny of distance" with a dispersed population. Governments have driven telecommunication development and have a key role in its regulation.

A virtual private network (VPN) is a mechanism for creating a secure connection between a computing device and a computer network, or between two networks, using an insecure communication medium such as the public Internet.

Singtel Optus Pty Limited is an Australian telecommunications company headquartered in Macquarie Park, a suburb in the Northern Sydney region of Sydney, New South Wales, Australia. It is a wholly owned subsidiary of Singaporean telecommunications company Singtel.

<span class="mw-page-title-main">Singtel</span> Telecommunications company in Singapore

Singapore Telecommunications Limited, commonly known as Singtel, is a Singaporean telecommunications conglomerate, the country's principal fixed-line operator and one of the four major mobile network operators operating in the country.

<span class="mw-page-title-main">Virgin Mobile</span> Wireless communications brand

Virgin Mobile is a wireless communications brand used by seven independent brand-licensees worldwide. Virgin Mobile branded wireless communications services are available in the United Kingdom, Ireland, Canada, Colombia, Chile, Kuwait, Saudi Arabia, United Arab Emirates, Poland and Mexico. Virgin Mobile branded services used to be offered in Australia, France, Singapore, India, Qatar, South Africa and the United States.

<span class="mw-page-title-main">Austar</span> Defunct Australian telecommunication company

Austar was an Australian telecommunications company founded in 1995 as Community Entertainment Television (CETV). Its main business activity was subscription television. It was also involved with internet access and mobile phones.

This is a list of the satellites operated by Optus, an Australian telecommunications company. The satellite communications facility is located at Belrose on Sydney's Northern Beaches. Optus' satellites are divided into 4 classes A, B, C and D. As of April 2014 it owns and operates Optus B3, Optus C1, Optus D1, Optus D2 and Optus D3. Optus A1, Optus A2, Optus A3 and Optus B1 satellites have been retired. Optus has the largest network of satellites in Australia and New Zealand.

BGP hijacking is the illegitimate takeover of groups of IP addresses by corrupting Internet routing tables maintained using the Border Gateway Protocol (BGP).

<span class="mw-page-title-main">Unwired</span>

Unwired Australia Pty Ltd was an Australian company dedicated to fixed wireless telecommunications network offering carrier grade Internet services. They provided coverage in Melbourne and Sydney. As of 8 March 2010 Unwired had 52,320 customers and 97 employees. The technology used by Unwired was provided by American manufacturer Navini Networks and Venturi Wireless Solutions.

<span class="mw-page-title-main">Bridge Alliance</span>

The Bridge Alliance is a business alliance of 34 major mobile telecommunications companies in Asia, Australia, Africa and the Middle East. It uses connectivity and a suite of integrated value-added services, including IoT/M2M, Optimization and Enterprise Mobility, for all alliance members' subscribers while roaming on each other's networks. The concept is similar to that of the FreeMove alliance in Europe, with whom the Bridge Alliance has a partnership.

<span class="mw-page-title-main">Primus Telecommunications (Australia)</span> Australian subsidiary of the telecommunications provider

Primus Telecommunications Pty Ltd. is an Australian telecommunications company and wholly owned subsidiary of Vocus Communications that primarily focuses on fixed, mobile, and broadband services.

<span class="mw-page-title-main">National Broadband Network</span> Telecommunications network in Australia

The National Broadband Network (NBN) is an Australian national wholesale open-access data network. It includes wired and radio communication components rolled out and operated by NBN Co, a government-owned corporation. Internet service providers, known under NBN as retail service providers or RSPs, contract with NBN to access the data network and sell fixed Internet access to end users.

Amaysim is an Australian provider of mobile phone plans. Amaysim operates as a Mobile Virtual Network Operator on the Optus mobile network, and specialises in offering a range of SIM-only mobile plans.

<span class="mw-page-title-main">Circles.Life</span> Telecommunications company in Singapore

Circles.Life is a Singaporean digital telecommunications company. The company was founded in 2016, initially operating exclusively in Singapore, leasing its network from M1.

<span class="mw-page-title-main">2021 Facebook outage</span> Outage affecting all Facebook operated services

On October 4, 2021, at 15:39 UTC, the social network Facebook and its subsidiaries, Messenger, Instagram, WhatsApp, Mapillary, and Oculus, became globally unavailable for a period of six to seven hours. The outage also prevented anyone trying to use "Log in with Facebook" from accessing third-party sites. It lasted for 7 hours and 11 minutes.

Boost Mobile is an American wireless service provider owned by Dish Wireless. It uses the Dish, AT&T and T-Mobile networks to deliver wireless services. As of Q3 2023, Boost Mobile, along with its sister brands Boost Infinite, Gen Mobile, and Ting Mobile had 7.50 million customers.

On July 8, 2022, Canadian telecom provider Rogers Communications experienced a major service outage affecting more than 12 million users of Rogers' cable internet and cellular networks, including those of subsidiary brands Rogers Wireless, Fido, Cityfone, and Chatr. This followed another major national outage a year prior in April 2021.

In September 2022, Optus, Australia's third-largest telecommunications company, suffered a data breach, affecting up to 10 million current and former customers, or over a third of Australia's population. Information illegally obtained included names, birthdates, home addresses, phone numbers, email contacts, and passport and driving licence numbers. Conflicting claims have been made about how the breach happened; Optus presented it as a complicated attack on their systems, while an Optus insider and the Australian government have claimed that a human error causing a vulnerability in the company's API occurred. A ransom notice was made, asking for A$1,500,000 to stop the data from being sold online. After a few hours, they deleted the ransom notice and apologised for their actions.

References

  1. 1 2 3 4 5 6 7 8 Optus Network Outage, Canberra: Senate Standing Committee on Environment and Communications (Parliament of Australia) (published November 2023), 17 November 2023, archived from the original on 17 November 2023, retrieved 18 November 2023
  2. 1 2 3 "Optus outage hits millions: Telco says services fully restored after nearly 14 hours". www.9news.com.au. 8 November 2023. Archived from the original on 9 November 2023. Retrieved 9 November 2023.
  3. "Optus outage affects millions of customers across Australia". amp.smh.com.au. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  4. 1 2 3 "Optus says outage fixed after nearly 14 hours". www.9news.com.au. 8 November 2023. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  5. Jose, Renju; Needham, Kirsty; Kaye, Ron (8 November 2023). "Optus network outage affects millions of Australians". Reuters. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  6. "Optus likely to explain outage 'disaster' at senate inquiry | Daily Telegraph". Archived from the original on 9 November 2023. Retrieved 11 November 2023.
  7. 1 2 Seeto, Tamika (8 November 2023). "Optus outage: Services 'gradually' resuming for millions of Aussies but 'may take hours'". Yahoo Finance. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  8. Reichert, Corinne (16 May 2018). "Optus doubles NBN customers as mobile subs hit 10 million". ZDNET. Archived from the original on 24 October 2022. Retrieved 8 November 2023.
  9. Grant-Geary, Belinda (8 November 2023). "Optus outage: Is outage 'final straw' for frustrated Aussies considering 'mass exodus'". yahoo! finance. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  10. 1 2 "Singtel refutes reports that its system upgrade caused Optus outage". ZDNET. Retrieved 18 November 2023.
  11. Maguire, Dannielle; Dervisevic, Hanan (7 November 2023). "Optus outage live: Company announces some services are being restored, but warns it may take a few hours". ABC News. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  12. 1 2 Wind, Emily; Cait, Kelly (8 November 2023). "Australia news live: Optus network outage; CEO says 'path to restoration' found after nine-hour blackout; SA government considers dumping Optus". The Guardian. ISSN   0261-3077. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  13. Varghese, Sam (8 November 2023). "Optus hit by national mobile, broadband outage since 4am". iTWire. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  14. 1 2 Smith, Paul; Wiggins, Jenny (7 November 2023). "Optus outage cuts off millions, emergency calls". Australian Financial Review. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  15. Turnbull, Tiffanie (8 November 2023). "Optus outage: Millions affected by Australian network failure". BBC. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  16. "About 500 train services cancelled across Melbourne after outage". www.9news.com.au. 8 November 2023. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  17. "'They're not listening to us': Business owner outraged by Optus data offer after outage". ABC News. 9 November 2023. Archived from the original on 11 November 2023. Retrieved 9 November 2023.
  18. "'There's no value to data': Optus customers outraged over 200GB compensation". www.9news.com.au. 10 November 2023. Archived from the original on 11 November 2023. Retrieved 10 November 2023.
  19. Businesses outraged by Optus data offer after outage | ABC News, archived from the original on 10 November 2023, retrieved 10 November 2023
  20. Optus customers slam compensation offer | 9 News Australia, archived from the original on 11 November 2023, retrieved 10 November 2023
  21. Lawler, Mitchell (10 November 2023). "What does the Optus fumble mean for ASX telco shares?". The Motley Fool Australia. Archived from the original on 10 November 2023. Retrieved 10 November 2023.
  22. Businesses outraged by Optus data offer after outage | ABC News, archived from the original on 10 November 2023, retrieved 10 November 2023
  23. Kelly Bayer Rosmarin resigns in the best intetest of Optus after natioanwide outage ABC News 20 November 2023
  24. Seeto, Tamika (8 November 2023). "Cashless Australia: Optus outage 'proof' Australia could be 'doomed'". yahoo! finance. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  25. Jeffrey, Daniel (9 November 2023). "Can you get compensation for the massive Optus outage?". 9 News. Archived from the original on 9 November 2023. Retrieved 9 November 2023.
  26. Vidler, Adam (9 November 2023). "Optus outage: Government pledges review into network collapse". 9 News. Archived from the original on 8 November 2023. Retrieved 9 November 2023.
  27. "Optus CEO to face grilling at Senate inquiry". ABC listen. 16 November 2023. Retrieved 18 November 2023.
  28. Swan, David; Grubb, Ben (8 November 2023). "What caused the Optus outage?". The Sydney Morning Herald. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  29. 1 2 Davidson, John (8 November 2023). "What early signs suggest may have gone wrong at Optus". Australian Financial Review. Archived from the original on 8 November 2023. Retrieved 8 November 2023.
  30. Taylor, Josh (8 November 2023). "Optus network outage may have been caused by same issue that brought Facebook down in 2021". The Guardian. ISSN   0261-3077. Archived from the original on 11 November 2023. Retrieved 9 November 2023.
  31. "Overview for AS4804". CloudFlare. Retrieved 8 November 2023.