Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

Current Action Plan:

Parts remain in transit to nearest Slough location, ETA still remains 03:45. Engineer attendance now requested and awaiting confirmation their ETA . Plan remains the same for the engineer to collect the parts, make his way to the Slough Datatcentre and commence Card swapout.

We will confirm ETA for engineers arrival at Slough DC as soon as possible

Next Update:

06:30

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

Current Action Plan:

Awaiting transfer of parts to nearest Slough location, this is in progress and ETA of parts to this location is 03:45. Once arrived engineer will collet and make his way to the Slough Datatcentre and will commence Card swapout.

We should be able to estimate arrival time of engineer to Datacentre on the next update

Next Update:

03:30

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

Current Action Plan:

Issue identified with sourcing part, CityFibre NOC currently assessing options available

Next Update:

01:30

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of car performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

Current Action Plan:

Escalation made to to Hardware Supplier as no current ETA can be given. Engineering attendance has been requested to CityFibre Slough Datacentre to carry out a reseat of the card. Once reseated they will remain on site for at least 30 mins to confirm success. If the issue manifests again then engineer will proceed with a Card Swap out.

Next Update:

23:30

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of car performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

Current Action Plan:

Engineering attendance has been requested to CityFibre Slough Datacentre to carry out a reseat of the card. Once reseated they will remain on site for at least 30 mins to confirm success. If the issue manifests again then engineer will proceed with a Card Swap out.

Currently awaiting an ETA for engineers attendance to Slough

Next Update:

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

Current Action Plan:

Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

It has been found that the origin point of the issue is on a line card situated within a core network device. During these investigations, two major actions have taken place in an attempt to restore service. The first action was to perform a soft clear on the line card itself which appeared to restore service for a short period before the issue resurfaced. And the second action was performing a full remote reboot of the affected line card.

During a period of monitoring, it has been seen that an issue has surfaced following the full reboot which is currently being investigated on a conference call between CityFibre NOC and our hardware supplier. Following on from the results of these investigations, the next course of action would be for engineers to be engaged to attend the site of the affected line card in order to perform a physical reseat of the line card itself.

Further to this, we are also considering a line card replacement should the issue resurface following a short monitoring period after the line card reseat.

Next Update:

19:30

At this point in time our internal monitoring is showing lines are stable, however they may drop without notice if Entanet / Cityfibre alter anything within their network to resolve the issues fully. Due to the fact that drops may occur at anytime it may be prudent to ask if staff can work from home (where possible) for the remainder of the day

Local internet traffic from your offices may also be affected due to issues within the Entanet / Cityfibre network that cause pages to load either really slowly or not at all, these issues may be sporadic in nature as well, this is due to line / tunnel flaps within the Entanet / Cityfibre network.

We will continue to post update as we get them from Entanet / Cityfibre.

Explainer: A circuit / line / tunnel flap is when a connection goes between a state of up (working) and down (Not working) rapidly, giving the impression of a line that is up (working), but its non functional and data cannot be passed over it.

The issue can be compounded by the way the internet works where if it see this happening it tries to work around it, but if that new route does the same thing, then the issue gets exponentially worse until other systems see it as having issues and drop their connections automatically as a precaution to protect their networks, which then causes an issue called convergence where the networks need to work out new paths to each other to “heal” the issue, this of course takes time (few seconds to several minutes)  to carry out as there are a lot of moving parts involved to try to work around issues and keep data moving.