At CI-Net we strongly believe in the provision of Secure Resilient Networks. During this outage a shortcoming in the delivery of the DSL services we provide to you was highlighted. I would like to take this opportunity to explain the extent of the issue and how we intend to permanently resolve the shortcoming.

 

During the afternoon of Saturday 4^th April a third party contractor using a large thrust borer damaged a BT tunnel 32 meters underneath the A14 destroying the fibre infrastructure carried within. This has severely affected emergency services, mobile phone operators, ISP’s, over 70,000 homes and hundreds of businesses across the UK.

 

Because of the severity of the damage BT was unable to reach the affected site until Tuesday with a target resolution date of Friday 10^th April. BT worked tirelessly to resolve this issue by routing via alternative exchanges and through different tunnels. Spare equipment was resourced from across the UK to speed resolution.

 

BT has declared an MBORC (matter beyond our reasonable control) on this issue, which has been approved by Ofcom. This effectively renders CI-Net and all other customers incapable of claiming compensation for the outage.

 

ADSL services are provided in two of our data centres in London, the DSL services are transported using Fibre Optic cable to our data centres using different routes, one from the Ealing area and one from the Ilford area. Each of these Fibre Optic cables is routed separately within the data centre and presented to the top and bottom of our racks. We keep each presentation as diverse as possible to avoid a problem in one geography affecting all our DSL services. Given the location of the incident in Ilford it was not clear why all of our DSL services were affected.

 

On detection of the failure the CI-Net Incident Response Team convened to discuss the appropriate course of action, reports of the true cause were varied at this stage and dictated that we should contact all managed customers to inform them of the extent of the outage and to agree to keep them informed throughout the incident. In parallel the systems and support teams contacted BT to glean further information as to why all our circuits could be affected.

 

During a conference call on the evening of Sunday 5^th BT accepted that the diverse routing of Fibres that they provide to us were both routed through the same tunnel in Ilford. Because of this routing error, our circuits were prioritised for resolution along with Blue Light Services. Not all other ISP’s have been so fortunate, with some services only being fully restored at 6am on Tuesday 14^th April.

 

As a matter of extreme urgency we are seeking written assurances from BT that the Fibre they provide to deliver services from their network is routed separately and that the diverse routes do not converge anywhere at street level.

 

Our new LLU services will be available by Monday 11^th May in over 1230 Exchanges across the UK. These links are delivered using separate equipment within the exchange and a different Fibre provider to our datacentres. Where we are able to, DSL services will be provided via our 2 available networks to provide additional resilient options. This will be carried out with the full knowledge of the customer but without additional cost.

 

Graham McLean

CI-Net Managing Director