Earlier this month, a suspected lightning strike on a critical component of the emergency services phone system prevented calls from reaching the 000 number. This disruption only lasted two hours, but seriously calls in to question how something like this could occur.
The component hit was a Telstra telecom pit located in rural NSW—but caused near-nation-wide outages of the service.
Backup systems were in place for the service, but they took over two hours to activate and allow calls to get through once more. Within these two hours, cases have been reported of people requiring emergency assistance, but falling on deaf ears.
This outage bears a very harsh reminder that can be taken by all of us: the importance of redundant systems.
The backup systems that were in place had been designed to automatically reroute all services should a line breakage occur, and this held true for most services. However, as reported by Mike Wright, Telstra’s group managing director of networks, one such service that did not reroute correctly was the 000 line.
The fault of this incident has been firmly planted on Telstra, and they are expected to launch a full investigation as to why these systems did not uphold their redundancy.
Having these systems in place was certainly the correct way to protect their environment, however as with any redundant system, regular checks to their reliability should be made to ensure they will fulfil their role to expectation.
Commonly this is done using some sort of failure simulation but can also be achieved by simply causing a failure in a controlled environment. These tests should be performed often enough to ensure confidence that full redundancy can be maintained.
Not all environments require full redundancy, but they should at the very least have a disaster recovery plan in place—not just for a critical event where systems need to be recovered from a total loss, but also from small losses to key components. How well could your business keep operating if you lost your phones or internet for a couple of hours? What about a couple of days?
If you’re not sure when the last time your current redundancy systems were tested, or worse —you have yet to implement any—please get in touch with the Altitude Innovations Team to discuss your options.