Incident Update

Status Update

Update Number: 9 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

Current Action Plan:

Parts are ready for collection in Slough.

Engineer details confirmed and will be collecting at 0700.

Access request to DC in progress.

Engineers ETA to Slough DC will be approx 0730

Once in site engineer will commence with the Card swapout.

Next Update:

07:30

Status Update

Update Number: 8 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

Current Action Plan:

Parts remain in transit to nearest Slough location, ETA still remains 03:45. Engineer attendance now requested and awaiting confirmation their ETA . Plan remains the same for the engineer to collect the parts, make his way to the Slough Datatcentre and commence Card swapout.

We will confirm ETA for engineers arrival at Slough DC as soon as possible

Next Update:

06:30

Status Update

Update Number: 7 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

Current Action Plan:

Awaiting transfer of parts to nearest Slough location, this is in progress and ETA of parts to this location is 03:45. Once arrived engineer will collet and make his way to the Slough Datatcentre and will commence Card swapout.

We should be able to estimate arrival time of engineer to Datacentre on the next update

Next Update:

03:30

Status Update

Update Number: 6(Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

Current Action Plan:

Issue identified with sourcing part, CityFibre NOC currently assessing options available

Next Update:

01:30

Status Update

Update Number: 5 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of car performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

Current Action Plan:

Escalation made to to Hardware Supplier as no current ETA can be given. Engineering attendance has been requested to CityFibre Slough Datacentre to carry out a reseat of the card. Once reseated they will remain on site for at least 30 mins to confirm success. If the issue manifests again then engineer will proceed with a Card Swap out.

Next Update:

23:30

Status Update

Update Number: 4 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of car performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

Current Action Plan:

Engineering attendance has been requested to CityFibre Slough Datacentre to carry out a reseat of the card. Once reseated they will remain on site for at least 30 mins to confirm success. If the issue manifests again then engineer will proceed with a Card Swap out.

Currently awaiting an ETA for engineers attendance to Slough

Next Update:

21:30

Status Update

Update Number: 3 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

Current Action Plan:

Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

It has been found that the origin point of the issue is on a line card situated within a core network device. During these investigations, two major actions have taken place in an attempt to restore service. The first action was to perform a soft clear on the line card itself which appeared to restore service for a short period before the issue resurfaced. And the second action was performing a full remote reboot of the affected line card.

During a period of monitoring, it has been seen that an issue has surfaced following the full reboot which is currently being investigated on a conference call between CityFibre NOC and our hardware supplier. Following on from the results of these investigations, the next course of action would be for engineers to be engaged to attend the site of the affected line card in order to perform a physical reseat of the line card itself.

Further to this, we are also considering a line card replacement should the issue resurface following a short monitoring period after the line card reseat.

Next Update:

19:30

Status Update

The issue that is affecting a number of client leased lines / EFM circuits is in a state of remediation with the ISP (Entanet / Cityfibre) who have identified a piece of equipment that they believe is causing the issues.

At this point in time our internal monitoring is showing lines are stable, however they may drop without notice if Entanet / Cityfibre alter anything within their network to resolve the issues fully. Due to the fact that drops may occur at anytime it may be prudent to ask if staff can work from home (where possible) for the remainder of the day

Local internet traffic from your offices may also be affected due to issues within the Entanet / Cityfibre network that cause pages to load either really slowly or not at all, these issues may be sporadic in nature as well, this is due to line / tunnel flaps within the Entanet / Cityfibre network.

We will continue to post update as we get them from Entanet / Cityfibre.

Explainer: A circuit / line / tunnel flap is when a connection goes between a state of up (working) and down (Not working) rapidly, giving the impression of a line that is up (working), but its non functional and data cannot be passed over it.

The issue can be compounded by the way the internet works where if it see this happening it tries to work around it, but if that new route does the same thing, then the issue gets exponentially worse until other systems see it as having issues and drop their connections automatically as a precaution to protect their networks, which then causes an issue called convergence where the networks need to work out new paths to each other to “heal” the issue, this of course takes time (few seconds to several minutes)  to carry out as there are a lot of moving parts involved to try to work around issues and keep data moving.

Status Update

 
Description: Major Incident – INC0107444 – Multiple circuit outage

Current Impact: Impacting a number of Leased Line, Broadband and Consumer circuits. Exact numbers unknown

Incident Start Time: 22/07/2022 11:57:59

Major Incident Raised: 22/07/2022 12:25:43

Major Incident Manager: Sam Abberley

Location/City: Nationwide

Update Number: 2

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

Current Action Plan:

CityFibre NOC have been investigating further and have determined that the issue is emanating from a specific location with the core network. As such, this device has been checked and the hardware supplier has been contacted directly for a Priority 1 case to be raised with them. During this time, CityFibre NOC have attempted to alleviate some issues across the core network by performing a workaround to reroute traffic to avoid the suspected affected device.

All of the requisite logs have been provided and CityFibre NOC are working alongside the hardware supplier to determine our next course of action. As of yet, it is still unclear as to when full service can be restored however we will endeavour to provide this as soon as we have it available.

We apologise for the disruption and hope to have a further update soon.

Next Update:

17:30

Status Update

From the reports we are receiving this was a major issue on one of the main comms providers in the UK which is affecting a significant part of the country.
 
We are working with our respective comms providers to fully understand the root cause, but all client connectivity issues should now be resolved.
 
There may be some further interruptions whilst the issue is fully investigated.