2022 Rogers Communications outage

Last updated

2022 Rogers Communications outage
Cloudflare IP prefix withdrawals during 2022 Rogers Communications outage.svg
Chart of withdrawals from IP prefixes from Rogers Communications on July 8, 2022, between approximately 05:00 and 16:30 UTC, as measured by Cloudflare [1]
DateJuly 8, 2022 (2022-07-08)
Time4:30 a.m. (EDT) [2]
Duration15 hours to several days
Type Network outage
Outcome

On July 8, 2022, Canadian telecom provider Rogers Communications experienced a major service outage affecting more than 12 million users of Rogers' cable internet and cellular networks, [3] including those of subsidiary brands Rogers Wireless, Fido, Cityfone, and Chatr. This followed another major national outage a year prior in April 2021. [4]

Contents

The 2022 outage impacted internet service providers with wholesale access to the Rogers network, such as TekSavvy, [5] as well as various other information systems nationwide that relied on the Rogers network, including Interac (this resulted in businesses nationwide being unable to accept debit transactions), OLG, [6] and some federal government services. Multiple international web monitoring organizations observed the outage.

The event spurred new government policies requiring all telecommunications providers to provide mutual assistance to each other in the event of an outage, offer emergency roaming service for rivals' affected customers, and establish plans for how to communicate to the public about outages. [3]

Rebates for customers are estimated to cost Rogers $28 million [4] to $70 million. [7] Splitting their physical and wireless lines for $261 million, and giving out $150 million in customer credits became part of a $10 billion plan for Rogers over a three-year span involving increased management and additionally the use of artificial intelligence to supervise potential future errors. [8] The economic toll to the Canadian economy for the outage period was projected at $142 million. [9]

An executive summary was published by the Canadian Radio-television and Telecommunications Commission in July 2024. [10]

Impact

The outage had a significant impact on various information systems nationwide, much of which extended past residential and commercial Rogers customers, due to their dependencies on Rogers infrastructure:

Recovery

Rogers had begun to slowly restore service that evening, but CEO Tony Staffieri stated there was no estimated time for when services would become fully operational again. [20] The next day, Rogers stated that it had restored service to the "vast majority" of its customers; however, not all service had been restored across the country. [21] On July 11, a spike of service issues was reported to the web traffic monitoring company Ookla's Downdetector, with some users reporting that service had not recovered for them. [22] [23]

Causes

A report by Cloudflare suggested that the outage was due to internal, rather than external, causes. It identified spikes in BGP updates, as well as withdrawals of IP prefixes, noting that Rogers was not advertising its presence, causing other networks to not find the Rogers network. [1] As of the day after the outage, the cause remained unknown. Public Safety Canada stated that it was not a cyberattack. [24] The outage was later said to be caused by a maintenance upgrade that caused routers to malfunction. [25] A similar outage occurred in April 2021, which was attributed to a software update. [4]

Postmortem

In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network. The deletion occurred during the sixth phase in a seven-phase update to its core IP network. The failure occurred at 4:45 a.m. and resulted in a nationwide loss of access to the company's phone and internet services, as well as internal access to systems such as the company's VPN to its core network nodes, hampering the ability of the company's employees to mobilize a team and identify the issue. Some employees were able to connect through "emergency SIMs" on alternate networks, which they had due to a practice established through reciprocal agreements made with other carriers in 2015. Others traveled to centralized locations to establish network access.

At 6 a.m., the company's CTO Jorge Fernandes, who has since been replaced, contacted leadership at rival companies Bell and Telus, advising them of the issues they were facing, as well as warning "to watch out for possible cyber-attacks." Both competitors offered assistance to Rogers, though the company determined that it was not possible to switch its users over to alternate networks, citing the inability to access its user database, as well as the excess load that the sudden addition of more than 10 million users would put on the other networks. A Memorandum of Understanding on how the carriers can work together in the future was submitted in September to the Minister of Innovation, Science and Economic Development. [26] Rogers plans to physically separate its wireless and wireline core networks to help mitigate the fallout of future outages. [27]

Reactions

Critics have cited the outage as an example of the impact of Canada's telecom oligopoly, and an argument against Rogers' proposed acquisition of Shaw Communications (which would further consolidate the industry). [28] [20] NDP leader Jagmeet Singh accused the Liberal Party government of "protecting the profits of telecoms giants", [29] while Conservative MP Michelle Rempel Garner called for an investigation, arguing that "given the critical infrastructure that's affected, and that because the Canadian Radio-television and Telecommunications Commission (CRTC) itself is affected, the cause of the Rogers outage should be immediately explained." [29] A spokesperson for the CRTC promised that the organization would examine the outage and "put in place the necessary measures to prevent it from happening again." [11]

Critics also highlighted the importance of consumers and businesses diversifying their telecommunications services, such as businesses having backup service contracts with multiple providers and consumers reducing their dependence on bundled "all-in-one" cable, internet and phone service packages. [29]

Response

The Minister of Innovation, Science and Industry, Francois-Philippe Champagne, in 2022. Champagne met with the CEOs of Canada's major telecom companies on July 11, shortly after the outage began. Francois-Philippe Champagne AL7I1923.jpg
The Minister of Innovation, Science and Industry, François-Philippe Champagne, in 2022. Champagne met with the CEOs of Canada's major telecom companies on July 11, shortly after the outage began.

Rogers planned to credit customers for five days of lost service, which was to be credited August 1. [27] [30] On July 10, 2022, Tony Staffieri, president and CEO of Rogers Communications, sent an email to all customers to officially apologize for the incident and inform them that Rogers would credit all customers' accounts for Friday. [31] [32] The last time they credited consumers on such a scale was soon after their April 19, 2021 outage, when "wireless calls, SMS and data services were down across Canada for almost an entire day because of an issue with a software update". [33]

On July 8, the same day as the outage, an internal email was sent requesting that retail locations remove signage that contained the slogan "Canada's Most Reliable 5G Network." [34]

On July 11, the CEOs of Rogers, Bell, Telus, Shaw, Vidéotron, SaskTel, and EastLink were all invited to take part in a conference call with federal cabinet minister François-Philippe Champagne to discuss a new "mutual assistance" framework of cooperation between the companies in emergencies to prevent the recurrence of a similar incident in the future. [35] On the same day, Champagne announced new policies requiring all telecommunications providers to provide mutual assistance to each other in the event of an outage, offer emergency roaming service for rivals' affected customers, and establish plans for how to communicate to the public about said incidents. Champagne said that the companies would have 60 days to reach agreements with each other to that effect. [3]

The House of Commons of Canada were discussing requests to conduct a study of the outage involving members of parliament from the Liberal Party, Conservative Party, New Democratic Party, and the Bloc Québécois. [36]

A class action lawsuit was filed in the Superior Court of Quebec which requested all customers receive a $400 rebate. Reasons cited included negligence of the CRTC regulations to make 9-1-1 calls available at all times. [37] [34]

The CFIB requested Rogers to give a free month of service to compensate for losses to others. [7]

As a result of the outage, the city government of Toronto reviewed its legacy contract for sourcing with a variety of providers. Council meetings were planned to review the full cost to the economy and the ability to use local ConnectTO in the city's infrastructure instead in the future. [16]

On July 21, it was reported that the Chief Technology and Information Officer, Jorge Fernandes, was to be replaced by Ron McKenzie, who was the former president of Rogers for Business. The leadership team was updated the previous night on the Rogers Website to reflect this change. [38]

See also

Related Research Articles

Present-day telecommunications in Canada include telephone, radio, television, and internet usage. In the past, telecommunications included telegraphy available through Canadian Pacific and Canadian National.

<span class="mw-page-title-main">Bell Canada</span> Canadian telecommunications company

Bell Canada is a Canadian telecommunications company headquartered at 1 Carrefour Alexander-Graham-Bell in the borough of Verdun, Quebec, in Canada. It is an ILEC in the provinces of Ontario and Quebec; as such, it was a founding member of the Stentor Alliance. It is also a CLEC for enterprise customers in the western provinces.

Singtel Optus Pty Limited is an Australian telecommunications company headquartered in Macquarie Park, a suburb in the Northern Sydney region of Sydney, New South Wales, Australia. It is a wholly owned subsidiary of Singaporean telecommunications company Singtel.

<span class="mw-page-title-main">Rogers Communications</span> Canadian telecommunications company

Rogers Communications Inc. is a Canadian communications and media company operating primarily in the fields of wireless communications, cable television, telephony and Internet, with significant additional telecommunications and mass media assets. Rogers has its headquarters in Toronto, Ontario.

Vidéotron is a Canadian integrated telecommunications company active in cable television, interactive multimedia development, video on demand, cable telephony, wireless communication and Internet access services. Owned by Quebecor, it primarily serves Quebec and Ottawa, as well as the Francophone communities of New Brunswick and some parts of Eastern Ontario. Its principal competitors are Bell Canada and Telus Communications.

<span class="mw-page-title-main">BCE Inc.</span> Canadian telecommunications and media company

BCE Inc., an abbreviation of its former name Bell Canada Enterprises Inc., is a publicly traded Canadian holding company for Bell Canada, which includes telecommunications providers and various mass media assets under its subsidiary Bell Media Inc. Founded through a corporate reorganization in 1983, when Bell Canada, Northern Telecom, and other related companies all became subsidiaries of Bell Canada Enterprises Inc., it is one of Canada's largest corporations. The company is headquartered at 1 Carrefour Alexander-Graham-Bell in the Verdun borough of Montreal, Quebec, Canada.

<span class="mw-page-title-main">Spark New Zealand</span> Telecommunications company in New Zealand

Spark New Zealand Limited is a New Zealand telecommunications and digital services company providing fixed-line telephone services, mobile phone services, broadband, and digital technology services. Its customers range from consumers to small - medium business, government agencies and large enterprise clients. It was formerly known as Telecom New Zealand until it was rebranded to Spark on 8 August 2014. It has operated as a publicly traded company since 1990. Spark's mobile network reaches 98% of New Zealand, with over 2.7 million mobile connections and 687,000 broadband connections

Bell MTS Inc. is a subsidiary of BCE Inc. that operates telecommunications services in Manitoba.

Bell Aliant Inc. is a brand name used by Bell Canada for telecommunications services in Atlantic Canada.

<span class="mw-page-title-main">Shaw Communications</span> Former Canadian communications company

Shaw Communications Inc. was a Canadian telecommunications company which provided telephone, Internet, television, and mobile services. The company was founded in 1966 as Capital Cable Television Company, Ltd. by JR Shaw in Edmonton. The company was acquired by and amalgamated into Rogers Communications in 2023; most operations were rebranded to the Rogers brand beginning in July of that year, although they are still operational.

<span class="mw-page-title-main">Fido (wireless carrier)</span> Cellular Carrier

Fido Solutions Inc. is a Canadian mobile network operator owned by Rogers Communications. Since its acquisition by Rogers in 2004, it has operated as a Mobile virtual network operator (MVNO) using the Rogers Wireless network.

<span class="mw-page-title-main">Globe Telecom</span> Telecommunications company in the Philippines

Globe Telecom, Inc., commonly shortened as Globe, is a major provider of telecommunications services in the Philippines. The company operates the largest mobile network in the Philippines and one of the largest fixed-line and broadband networks. As of November 2023, Globe has 54.7 million subscribers, making it the second largest network in terms of subscriber base.

<span class="mw-page-title-main">Internet in Canada</span>

Canada ranks as 28th in the world for Internet usage with an estimated 35.34 million users or 93% of the population as of 2021. According to Harvard researchers, Canada has some of the lowest internet standards among OECD countries, as a result of high costs and slow internet speeds.

Lumen Technologies, Inc. is an American telecommunications company headquartered in Monroe, Louisiana, which offers communications, network services, security, cloud solutions, voice and managed services through its fiber optic and copper networks, as well as its data centers and cloud computing services. Its has been on the S&P 600 index since being removed from the S&P 500 in March 2023.

<span class="mw-page-title-main">Freedom Mobile</span> Canadian wireless service provider

Freedom Mobile, a Canadian wireless telecommunications provider owned by Québecor, holds a 6% market share of the Canadian wireless market, primarily concentrated in urban areas of Ontario, British Columbia, Alberta, and Manitoba. As the fourth-largest wireless carrier in Canada, Freedom Mobile serves 2,290,497 subscribers as of November 30, 2022. In addition to mobile phone plans, Freedom also offers home internet and TV services.

<span class="mw-page-title-main">Mobilicity</span>

Data and Audio-Visual Enterprises Wireless, d/b/a Mobilicity, was a Canadian mobile virtual network operator (MVNO) owned by Rogers Communications. Its name was a portmanteau of the words "mobility" and "simplicity". Mobilicity was one of several new mobile network operators, along with Public Mobile and Wind Mobile, which launched in Canada after a government initiative to encourage competition in the wireless sector. The carrier had over 250,000 Mobilicity subscriptions on May 16, 2013, the day in which Telus announced its failed attempt to acquire Mobilicity. The subscription count decreased to 157,000 by April 2015 according to court documents filed by Mobilicity's Chief Restructuring Officer in that month.

The Commission for Complaints for Telecom-television Services is Canada's national, independent and industry-funded organization created to resolve telecommunication and television service complaints from consumers and small business customers fairly and free of charge.

This is a timeline of Telus Corporation, a publicly traded Canadian multinational holding company offering a range of telecommunications, health, safety, and security products and services. The company operates Telus Communications Inc., which offers telephony, television, data and Internet services, Telus Mobility, a division that offers wireless services, Telus Health, which operates companies that provide health products and services, and Telus International which operates worldwide, providing multilingual customer service outsourcing and digital IT services.

The 2023 Optus outage was an unplanned outage of all Optus internet, cellular and fixed-line services in Australia first detected on 8 November 2023 at 04:05 AEDT. The outage caused disruption across government and corporate sectors including hospitals, banks, train services, EFTPOS payment systems, and calls to emergency services. Restoration was gradual with some services being restored around 13:00 AEDT. The outage directly affected more than 10 million people and 400,000 businesses across Australia.

References

  1. 1 2 Tomé, João; Strickx, Tom; Zhang, Mingwei (July 8, 2022). "Cloudflare's view of the Rogers Communications outage in Canada". The Cloudflare Blog. Archived from the original on July 9, 2022. Retrieved July 9, 2022.
  2. ↑ "Rogers starts restoring service after outage, CEO apologizes to customers". Cornwall Seaway News. July 9, 2022. Retrieved July 10, 2022.[ permanent dead link ]
  3. 1 2 3 4 "Ottawa announces it will require telecoms to provide backup for each other during outages following Rogers system failure". thestar.com. July 11, 2022. Archived from the original on July 12, 2022. Retrieved July 12, 2022.
  4. 1 2 3 Evans, Pete (July 8, 2022). "Rogers says services mostly restored after daylong outage left millions offline". CBC News. Archived from the original on July 11, 2022. Retrieved July 11, 2022.
  5. 1 2 3 4 da Silva, Michelle; Augustin, Mathilde (July 8, 2022). "The Rogers outage disrupted services across Canada. A list of what was affected". The Globe and Mail. Archived from the original on July 9, 2022. Retrieved July 9, 2022.
  6. ↑ "Rogers says it has restored service for 'vast majority' of customers". thestar.com. July 8, 2022. Archived from the original on September 4, 2022. Retrieved September 4, 2022.
  7. 1 2 Alini, Erica; Hannay, Chris (July 12, 2022). "Consumers and businesses expected to take Rogers to court over network outage". The Globe and Mail. Archived from the original on July 13, 2022. Retrieved July 13, 2022– via www.theglobeandmail.com.
  8. ↑ "Rogers to spend $261M to split networks, but can't quantify economic impact of outage". CTVNews. August 25, 2022. Retrieved November 3, 2022.
  9. ↑ Bloomberg, B. N. N. (July 12, 2022). "Rogers pledges five-day credits as Bay Street weighs outage impact - BNN Bloomberg". BNN.
  10. ↑ "Assessment of Rogers Networks for Resiliency and Reliability Following the 8 July 2022 Outage - Executive Summary". July 4, 2024.
  11. 1 2 "Rogers CEO apologizes for massive service outage, blames maintenance update". CBC News. July 9, 2022. Archived from the original on July 9, 2022. Retrieved July 9, 2022.
  12. ↑ "Netblocks tweet on Rogers outage". Twitter. July 8, 2022. Archived from the original on July 8, 2022. Retrieved July 10, 2022.
  13. ↑ Hristova, Bobby (July 12, 2022). "Hamilton man was unable to call 911 during Rogers outage as sister was dying". CBC News. Archived from the original on July 12, 2022. Retrieved July 12, 2022.
  14. ↑ Lopez-Martinez, Melissa; Lee, Michael (July 8, 2022). "Rogers wireless services 'starting to recover,' company says". CTVNews. Archived from the original on July 9, 2022. Retrieved July 9, 2022.
  15. ↑ "Rogers outage cost small businesses thousands of dollars: CFIB". July 11, 2022. Archived from the original on July 11, 2022. Retrieved July 13, 2022.
  16. 1 2 "City of Toronto eyeing ways to reduce reliance on Rogers after outage". Toronto Star . July 11, 2022. Archived from the original on July 14, 2022. Retrieved July 13, 2022.
  17. ↑ Rosas, Aura; Hristova, Bobby. "Businesses, doctors, DARTS service in Hamilton and Niagara among those hit by massive Rogers outage". CBC News. Archived from the original on July 9, 2022. Retrieved July 10, 2022.
  18. ↑ Morton, Michelle; Casaletto, Lucas (July 8, 2022). "Weeknd postpones concert in Toronto as Canadians face Rogers outage". CityNews . Archived from the original on July 9, 2022. Retrieved July 8, 2022.
  19. ↑ Live Nation Ontario, (@LiveNationON) (July 18, 2022). ".@theweeknd show at Rogers Centre has been rescheduled to Thursday, September 22nd. Please hold onto your tickets as they will be valid for the new date". Twitter . Archived from the original on July 18, 2022. Retrieved July 18, 2022.
  20. 1 2 Karadeglija, Anja (July 8, 2022). "Critics call for Parliament, CRTC to immediately investigate Rogers outage". National Post. Retrieved July 9, 2022.
  21. ↑ Roth, Emma (July 9, 2022). "Rogers restores service for 'vast majority' of customers after massive outage". The Verge. Archived from the original on July 9, 2022. Retrieved July 9, 2022.
  22. ↑ Phan Nay, Isaac (July 11, 2022). "It's day 4 of the Rogers outage. Here's 4 things you need to know". thestar.com. Archived from the original on July 11, 2022. Retrieved July 12, 2022.
  23. ↑ "Rogers down? Realtime status and problems overview". downdetector.ca. July 12, 2022. Archived from the original on July 12, 2022. Retrieved July 12, 2022.
  24. ↑ "Rogers network resuming after major outage hits millions of Canadians". Reuters. July 8, 2022. Archived from the original on July 9, 2022.
  25. ↑ Callan, Isaac (July 9, 2022). "Rogers CEO apologizes, says 'maintenance upgrade' behind major outage". Global News. Archived from the original on July 10, 2022. Retrieved July 10, 2022.
  26. ↑ "Memorandum of Understanding on Telecommunications Reliability". Innovation, Science and Economic Development Canada. Archived from the original on September 7, 2022. Retrieved December 1, 2022.
  27. 1 2 Shecter, Barbara (July 24, 2022). "Rogers shares explanation after 'unprecedented' outage". Financial Post. Archived from the original on July 25, 2022. Retrieved July 24, 2022.
  28. ↑ Deschamps, Tara (July 8, 2022). "Far-reaching implications of Rogers outage shows need for competition: Expert". CityNews Ottawa. Archived from the original on July 9, 2022. Retrieved July 9, 2022.
  29. 1 2 3 Logan, Nick (July 8, 2022). "Rogers outage shows need for Plan B when wireless, internet services fail, analysts say". CBC News. Archived from the original on July 9, 2022. Retrieved July 9, 2022.
  30. ↑ Paperny, Anna Mehler; Rajagopal, Divya (July 12, 2022). "Canada's Rogers to credit customers with 5 days service after massive outage". Reuters. Archived from the original on July 13, 2022. Retrieved July 13, 2022.
  31. ↑ Staffieri, Tony (July 10, 2022). "A Message from Rogers President and CEO". Rogers. Archived from the original on July 12, 2022. Retrieved July 12, 2022.
  32. ↑ Helena Hanson (April 21, 2021). "Here's Everything You Need To Know About Claiming Rogers Credit After The Outage". Narcity Media. Archived from the original on July 8, 2022. Retrieved July 10, 2022.
  33. ↑ Lisa Belmonte (July 9, 2022). "The Last Canada-Wide Rogers Outage Resulted In Credits For Customers & This Is How It Worked". Narcity Media. Archived from the original on July 12, 2022. Retrieved July 10, 2022.
  34. 1 2 Zukra, Joey (July 11, 2022). "Application to Authorize the Bringing of A Class Action and to Appoint the Status of Representative Plaintiff" (PDF). LPC Avocat Inc. Archived (PDF) from the original on July 11, 2022. Retrieved July 24, 2022.
  35. ↑ Alexandra Posadzki, "Rogers outage prompts meeting with telecom execs, Industry Minister to discuss 'mutual assistance' framework, sources say" Archived July 11, 2022, at the Wayback Machine . The Globe and Mail , July 11, 2022.
  36. ↑ "Rogers to credit customers with five days of service as it faces scrutiny from CRTC after major outage". The Globe and Mail. July 12, 2022. Archived from the original on July 13, 2022. Retrieved July 13, 2022– via www.theglobeandmail.com.
  37. ↑ "Rogers customers in Toronto could get $400 rebate thanks to class action suit over outage". www.blogto.com. Archived from the original on July 13, 2022. Retrieved July 13, 2022.
  38. ↑ Willis, Andrew (July 21, 2022). "Rogers replaces chief technology officer amid the fallout from nationwide outage". The Globe and Mail. Archived from the original on July 21, 2022. Retrieved July 21, 2022– via www.theglobeandmail.com.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.