Internet Solutions has issued an explanation for a series of outages which affected three of its five international...
The company says it first experienced an outage on three of the circuits on 26 July. These three 45mbps circuits operate between New York City and Cape Town and Rosebank in South Africa. The other two fibre circuits are via the SAT2 fibre and SAT3 fibre cables, and these have not been affected. IS also operates a sixth circuit, via satellite, and is adding a sixth fibre circuit on 1 September, giving seven circuits in total, the company said.
The total duration of the initial outage, said IS, was just over three hours, and occurred just before 9am on 26 July. The outage was as a result of a cable fault on a segment of the cable system in Spain, where this portion of the cable system is managed by the Spanish telecom, Telefonica. The net result of this outage was that about half of IS's international capacity was unavailable.
On 29 July, at 3am, the same three circuits were taken out of commission for what was to be emergency repair work performed by Telefonica. It was believed that, after this work, the service would have returned to stability.
But on 11 August the same three international fibre circuits failed again, just before 9am and only returned to operation just before 1pm. It was confirmed on Thursday by Telkom that the fault causing this outage occurred in a telco exchange in Spain, when the backplane of a telco switch failed. IS says it is still waiting for the detailed report from Telkom and AT&T.
On 12 August, at 1pm, the same three international fibre circuits failed again.
IS is contracted with Telkom and AT&T respectively for the provisioning of these three circuits (an STM1). These circuits are mapped across different cable systems managed by different telcos.
From South Africa, the circuits operate on the SAT3 undersea fibre cable, and in the Canary Islands are connected with the PenCan6 undersea fibre cable and some terrestrial cable components to Conil on the Spanish mainland, where the last connection en route to the US, to the Columbus 3 undersea fibre cable (crossing the north Atlantic), is made.
The Spanish component, where all three outages over the last three weeks have occurred, is under the management of Telefonica. Telkom and AT&T have contracted with Telefonica for the connecting service between SAT3 and Columbus 3.
These Telefonica services have been in commission for just under a year, and, barring the last three catastrophes, have been exceptionally stable and robust, with these the first critical failures to have occurred on these services, the ISP notes.
The outages obviously had, and are still having, a massive impact on services to customers, IS says. Though IS still has three other large bandwidth services available, the nature of the outages is such that services to all customers are affected.
During, and after the outage on Thursday, IS had extensive meetings with executives and a senior technical planning team from Telkom and AT&T. Good progress was made in evaluating potentially weak points in the service and developing alternatives. The company says it is unfortunate a third outage occurred before it was able to action the changes agreed on.
IS is aggressively pursuing the stabilisation of this service, which involves the mapping of this service onto alternative underlying infrastructure, and notes that its focus now is on returning the service to operation as soon as possible, after which it will make changes to the underlying infrastructure to ensure the robust service that customers expect.
Written by Computing South Africa staff