As you all know yesterday there was an issue with the main router, what happened was that it went into a reboot loop but it was staying up long enough to pick up all the traffic and then it reset itself over and over again.
In order to regain connectivity we moved the Pipe connection to the secondary router and services started coming online, however with just one provider, the network was congested and things were running slow.
We worked on the main router to stop the reboot loop and perform basic health checks, while everything appeared ok and the router stayed online for a few hours, the moment we tried to route traffic through it, the reboot loop issue reappeared. Firmware and software updates did not help with this issue and as a result we decided to leave the main router offline.
In order to cope with the traffic we added another provider to the secondary router (AAPT) and this helped with the congestion.
There were a few other issues as a result of the main router flapping, one of them resulted in some traffic being filtered by Pipe and we had to go through their support in order to get it cleared.
At this stage Traffic is flowing through Pipe and AAPT on the secondary router.
If you experience any routing issues, please open a support ticket and provide trace routes and MTRs so that we can investigate the problems. Most of the time these are related to ARP and DNS caching and a server reboot is the fastest way to fix the problem.
We are sourcing a replacement router as we speak and once this is delivered we will have both routers online and each one will be connected to different providers, this way if anything like this happens again traffic will continue to flow through the other router.
Once the new router is delivered we will schedule network maintenance in order to set up the network as described above.