We would like to follow up with you regarding the recent network outage which we experienced across the entirety of our services and provide a post-mortem technical report for the causes of the outage.
Currently, we have two (2) physical servers that act as a firewall for the protection of our network. These servers act as a failover for each other. Should one of the units fail, the other will pick up the traffic to avoid any interruption to service.
Last night, one of the units did fail, and the other picked up the traffic as intended. The second unit, that had all of the traffic at this time, then also failed. There was no ability to pass the traffic back to the first unit in its failed state.
We worked to restore all traffic with in-house staff, having our Director of Operations present at our Toronto data center for troubleshooting. After initial tests could not determine the root cause of the server failures, a priority ticket was opened with Cisco, one of our hardware vendors.
After some trial and error, the cause was not found and an executive decision was made to force a restart of both physical units. This restart did restore network connectivity, but wiped out log files needed to determine the root cause of the failures. Cisco is attempting to recover any information to pinpoint the root cause, which we will be happy to update you with if it is located.