Active Incident

Incident Status

Degraded Performance

Components

Global IP Transit, MPLS/VPLS/L2VPN

Locations

BYF



November 6, 2019 16:24 GMT
[Identified] Our third party supplier in Byfleet is experiencing a network issue within their core at present. All resilient circuits provided by Custodian DC have failed over as expected, with sub-second recovery. Whilst this is being investigated, please be aware of reduced resilience on circuits terminating at Byfleet. We are liaising with the supplier on this. Services should continue to run as expected however if you have any queries, please contact us with [CDC-ID: #232091] in the subject or call us on 01622 230 382 - option 1

November 7, 2019 09:35 GMT
[Identified] Our third-party supplier in Byfleet have been investigating the instability within their core and have identified the fault. They have isolated the faulty switch to bring back network stability. Due to this they are running at reduced resiliency until this evening when they are scheduling to replace a faulty part and bring the affected switch back in to service. During these works they will need to perform a reload of the device. We are continuing to liaise with the supplier and will provide a further update once we have more details for the works tonight. A separate notification will be sent out by us for the emergency maintenance window in Byfleet and will include further details once we have them.

November 8, 2019 09:54 GMT
[Identified] Unfortunately the 3rd party was unable to resolve this issue during their overnight works. They are continuing to engage with their TAC and work towards a resolution. Additional updates and maintenance windows will be posted here as we are informed. Services in BYF should be considered at-risk until this has been resolved.

November 11, 2019 12:55 GMT
[Identified] 4D Datacentres networking team are continuing to work with Cisco TAC to resolve the issues within the VSS cluster at their Byfleet facility. Connections and services provided from Custodian's core in Byfleet remain resilient at this point in time and are not impacted by this. Only customers with services directly provisioned on 4D equipment are seeing a failure of 1 of their 2 diverse feeds since the issue occurred. We will post further updates as soon as we receive them from the 4D team.

November 11, 2019 17:51 GMT
[Identified] Today (11/11) at 22:30 4D Datacentres have notified us they will be carrying out an emergency move that will cause a temporary period of disruption in our transport to Byfleet. Connections and services provided from Custodian's core in Byfleet remain resilient at this point in time but will lose resilience on one leg whilst these works are undertaken. All resilient circuits provided by Custodian DC have should fail over as expected, with sub-second recovery. 4D networking team are still working with Cisco TAC to resolve the issues within the VSS cluster at their Byfleet facility. This specific maintenance is being separately tracked under [CDC-ID: #232114]

November 19, 2019 08:52 GMT
[Identified] After an extended maintenance window overnight, the team at 4D Datacentres were unable to restore the VSS. This incident remains open as before - Only services provided directly by 4D in Byfleet are impacted.

Global IP Transit




Degraded Performance

MPLS/VPLS/L2VPN




Degraded Performance

Colocation




Fully Operational

DWDM/Wavelength




Fully Operational

Website / Domain Hosting




Fully Operational

Support Desk Operations




Fully Operational

Maidstone IP Core

100.00%




London IP Core

100.00%




Byfleet IP Core

100.00%




Maidstone IP Core

100.00%




London IP Core

100.00%




Byfleet IP Core

100.00%




Maidstone IP Core

100.00%




London IP Core

100.00%




Byfleet IP Core

100.00%