Network Issues Reported
09:10 - We are currently experiencing network issues which are affecting some inbound and outbound calls, updates to follow.
11:14 - Emergency work has restored all services.
13:00 - We have identified a recurrence of the earlier issues, our engineers are working to resolve the problem.
14:30 - We have a monitored a sustained period of stability, an RFO will follow in due course.
All services are now working normally
Update - Further investigations have concluded that this issue was caused by a switch in our datacentre providers core network which had hit 100% CPU activity due to the device switching from hardware to software routing as a result of TCAM exhaustion. Similar issues had affected many other network providers that day, with the day being named 512K Tuesday by the industry,
http://arstechnica.com/security/2014/08/internet-routers-hitting-512k-limit-some-become-unreliable/
As a preventative measure available routing table resource has been increased by 20% and measures have been implemented to monitor routing table allocation.
