Fatal triple-zero Optus outage blamed on poor culture

By Grace Crivellaro and Tess Ikonomou

Australia’s second-largest telco had appropriate processes in place to deal with a triple-zero outage, but a “culture of lack of care” led to it not being followed.

The September 18 outage, which lasted almost 14 hours, affected hundreds of triple-zero calls in four states and territories and has been linked to two deaths due to emergency calls failing.

A parliamentary inquiry is under way in Canberra with Kerry Schott, who completed an independent review into the deadly incident, giving evidence.

Offshore call centre staff repeatedly failed to escalate warnings from customers about the triple-zero outage, which was caused in part by errors in the telco’s outsourced network management team.

Hours later, a customer contacted Optus directly about the outage, which is when the company became aware of the severity of the failure.

Dr Schott, a corporate and public sector veteran, said call centre employees had a sheet of instructions to follow in the event of an emergency.

“The first thing they had to do was ask whether there was any immediate danger or problem and if the answer was positive, then they were able to be connected through to the emergency call person,” she said.

“But if they weren’t in immediate danger, then the call centre tried to work out what the problem was.

“The call centre had no knowledge of an outage, so they assumed that it was some sort of technical difficulty.”

When the outage was referred to the call centre hours later, it was after the knowledge that two fatalities occurred, Dr Schott said.

“At that time, they were extraordinarily distressed,” she said.

“Senior people at the call centre actually seemed to spend an hour or more just calming everyone down.

“The management of the call centre at that time was not adequate, and has since been changed, and was changed very quickly after this event.”

Dr Schott said it was astounding alerts were missed and blamed it on a “culture of lack of care” that they were written off as a firewall upgrade.

“When you get technical people, they sometimes can forget that what they are doing is part of providing an essential service,” she said.

“It’s a culture that I can understand how it crept up and happened, but it is unforgivable and it must be fixed.”

In her review, Dr Schott handed down 21 recommendations and described the failings as inexcusable.

The crisis management document was 40-pages long at the time, Dr Schott said, which was too dense for employees to easily follow.

She recommended the telco condense the document into a checklist, so employees had clear instructions in an emergency.

Optus chief executive Stephen Rue said the company was in the process of implementing the checklist.

He said several other changes had been made since the fatal blunder, including onboarding new onshore call centre roles, strengthening escalation processes and improving real-time visibility of triple-zero performance.

“There have also been substantial changes to executive leadership, including in network security, technology, legal compliance and risk,” Mr Rue told the inquiry.

He confirmed there would be 200 to 300 job cuts at Optus as it tried to reset following the outage.

Apple Australia and Google Australia will appear at the hearing, along with NSW Ambulance and the National Emergency Management Agency.

Optus extended its “deepest sympathies” to the families and friends of the people who died.

“It weighs heavily on me and our board and on the people of Optus. We recognise the seriousness of what happened and the responsibility we bear,” Mr Rue said.