Rogers executives on Monday walked MPs through the chaos that unfolded earlier this month when what they called a “truly unprecedented incident” took down the “brain” of the company’s network.
The company’s new chief technology officer, Ron McKenzie, said it was a coding error that triggered the widespread network failure that knocked out service for millions of Canadians for more than 15 hours and left many customers unable to reach 911.
“(It) essentially (affected) the core of the network, the brain of the network, that supports all wireless, wireline, internal communications and connectivity,” McKenzie said. “It was a complete loss of connection.”
Speaking during a hearing of the House of Commons industry committee, which is studying the outage, Rogers CEO Tony Staffieri said the company now plans to physically separate its wireless and internet networks to ensure more redundancy in the future. If one part of the network were to go down in the future, he said, the company could move traffic to the other one.
“This added layer of protection will be expensive,” Staffieri said, adding the company estimates it will cost at least $250 million. “But we know it’s the right thing to do.”
The Rogers executives told MPs the outage began shortly before 5 a.m. after the coding error was introduced in the sixth part of a seven-phase update process that spanned several weeks. McKenzie said there were no problems during the previous five phases.
In the initial hours, Staffieri said Rogers personnel themselves could not communicate, which delayed their response as well as communications with the public and government. He said the company was preoccupied with two main questions: determining whether it had been hit by a cyberattack and also when it would be able to get its network back up.
Earlier Monday, Innovation Minister François-Philippe Champagne told MPs that he was the one who first reached out to Staffieri during the outage to find out what was happening.
“It should not be (up) to the minister to chase the CEO of a major telco when something like that happens. It should be the other way around,” Champagne said.
“In retrospect I regret that and I’m disappointed we didn’t reach out sooner,” Staffieri said, when asked by one MP about when he notified the Quebec premier about the outage. “There were reasons. We were focused on the solution and getting our customers up and running, but nonetheless, those communications should have happened sooner for an important stakeholder such as the government.”
Champagne said Staffieri has since been in regular touch to provide updates, but he said the fact that he was the one who had to reach out is part of the reason why he has since demanded better communication from telecoms in the case of outages.
Three days after the outage, Champagne convened a meeting of Staffieri and the CEOs of other major telecoms and said he demanded they strike formal agreements within 60 days to support each other during network failures in the future.
“The responsibility is solely on Rogers but we all have to be in solution mode,” Champagne said Monday. “The clock is ticking and there is 45 days left to deliver on that commitment.”
Ian Scott, chair of the Canadian Radio-television and Telecommunications Commission, told MPs Monday afternoon that Rogers’ communications about the outage were “unacceptable.”
“Rogers has said they’ll do better, the CRTC will make sure that they do,” Scott said, noting that as a first step, the regulator has asked for a detailed account of what happened and what measures are in place to prevent similar events in the future.
The company provided answers to those questions on Friday and Scott said the CRTC is reviewing them and determining next steps. “I assure you that we’ll act quickly.”
In the its response to the CRTC, which was partially redacted in the public version published late Friday evening, Rogers also revealed more details about the outage, which left the carrier unable to route customers’ 911 calls or to deliver four emergency alerts issued that day in Saskatchewan.
As a result of the failure in the core network, Rogers could not shut down its radio network, which could have allowed customers to automatically connect to 911 on another carrier’s network.
At the hearing on Monday, Staffieri told MPs that if the company had shut down its radio network, it could have extended the entire outage by another day.
Scott noted that the 911 network itself — which is run by Bell, Telus and SaskTel — was still fully functional, but the Rogers core network failure was the reason its customers couldn’t get through to emergency responders.
The company also said it was unable to transfer traffic to rival carriers despite offers of assistance from Bell and Telus, as those networks would not have been able to handle the extra volume of more than 10 million customers.
On Sunday, Staffieri said in a letter to customers that the company is working on a formal agreement with competing wireless carriers to “switch 911 calls to each other’s networks automatically — even in the event of an outage on any carrier’s network.”
“I believe this is the only responsible way forward and I am personally committed to making it possible for all Canadians,” Staffieri said.
He said Rogers is doing a “full review” of its network and plans to invest $10 billion in the next three years as part of its continued “focus on reliability.”
“This includes more oversight, more testing and greater use of artificial intelligence to ensure we’re able to deliver the reliable service you deserve.”
McKenzie was named chief technology officer last week when the company said Jorge Fernandes would step down from the role in the wake of the network failure.
MPs also raised the point of whether there is adequate competition in Canada’s telecom industry, which has become a pivotal issue in Rogers’ proposed $26-billion acquisition of Shaw. The Competition Bureau is going to court to fight the transaction.
The federal government must also approve the takeover and when asked about the deal Monday, Champagne said the companies have not yet submitted a formal proposal to his department. He referenced previous comments he has made saying that he would not allow the “wholesale transfer” of spectrum licences from Shaw to Rogers.
Staffieri said he does believe Canadians have choice when it comes to telecom service and that the Shaw deal would give the combined company greater scale and ability to invest in network resiliency. He said the deal would allow Rogers to separate its wireless and internet networks in about half the time that it would otherwise take.
Pressed by NDP MP Brian Masse, who questioned the minister on what consequences Rogers or other telecoms could face when it comes to network outages, Champagne said, “When the minister speaks, they listen.”
“They didn’t even call you,” Masse replied, adding that he’s skeptical of an approach to network reliability and public safety that relies on influence and the goodwill of the telecom companies.
“You’ll be able to rely on a formal binding agreement,” Champagne said.
With files from The Canadian Press
JOIN THE CONVERSATION
Anyone can read Conversations, but to contribute, you should be registered Torstar account holder. If you do not yet have a Torstar account, you can create one now (it is free)
