Update Number: 17 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

-Engineer details confirmed and will be collecting at 0700.

-Access request to DC in confirmed

-Issue with retrieving parts from location resolved

– Engineer attended Slough DC

-Engineer has completed card swap successfully

-Testing and checks completed

-BGP reenabled

-Network stability confirmed

-CityFibre NOC Engineers have advised they are seeing network instability issues and are currently investigating.

-Initial assessment appears to have identified that post the reintroduction of traffic following repairs stability was observed to deteriorate. This recovery step has been reverted

-Investigation continued with diagnostics being carried out on the Network Device. Network traffic remains rerouted and services have been observed as stable at this time.

– Card causing alarms remains out of service

-Resilient links remain carrying traffic mitigating any remaining impact to services previously utilising that card and to prevent any further service disruption by our planned restoration activity.

Current Action Plan:

Engineers have arrived on site and will be completing the card swap within the next 30 minutes

There remains no customer service impact that we are aware of, with services either taking alternative routes round the network or utilising their designed service resiliency at this location.

Next Update:

20:30

Update Number: 16 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

-Engineer details confirmed and will be collecting at 0700.

-Access request to DC in confirmed

-Issue with retrieving parts from location resolved

– Engineer attended Slough DC

-Engineer has completed card swap successfully

-Testing and checks completed

-BGP reenabled

-Network stability confirmed

-CityFibre NOC Engineers have advised they are seeing network instability issues and are currently investigating.

-Initial assessment appears to have identified that post the reintroduction of traffic following repairs stability was observed to deteriorate. This recovery step has been reverted

-Investigation continued with diagnostics being carried out on the Network Device. Network traffic remains rerouted and services have been observed as stable at this time.

Current Action Plan:

With errors continuing following the suspected faulty line card being replaced, focus has switched to another core device which connects directly to it. Joint investigations between CityFibre and the vendor of the other core device commenced a short while ago, and further alarms were quickly identified here.

The card causing the alarms was taken out of service in order for redundant links on another card to be forced into use. This will mitigate any remaining impact to services previously utilising that card and to prevent any further service disruption by our planned restoration activity.

The next stage is for engineers to attend the relevant location with replacement parts. The ETA for this activity is 18:30.

There is currently no customer service impact that we are aware of, with services either taking alternative routes round the network or utilising their designed service resiliency at this location.

Next Update:

18:30

Update Number: 15 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

-Engineer details confirmed and will be collecting at 0700.

-Access request to DC in confirmed

-Issue with retrieving parts from location resolved

– Engineer attended Slough DC

-Engineer has completed card swap successfully

-Testing and checks completed

-BGP reenabled

-Network stability confirmed

-CityFibre NOC Engineers have advised they are seeing network instability issues and are currently investigating.

-Initial assessment appears to have identified that post the reintroduction of traffic following repairs stability was observed to deteriorate. This recovery step has been reverted

Current Action Plan:

investigation continues with diagnostics being carried out on the Network Device. Network traffic remains rerouted and services have been observed as stable at this time

Next Update:

14:00

Update Number: 14 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

-Engineer details confirmed and will be collecting at 0700.

-Access request to DC in confirmed

-Issue with retrieving parts from location resolved

– Engineer attended Slough DC

-Engineer has completed card swap successfully

-Testing and checks completed

-BGP reenabled

-Network stability confirmed

Current Action Plan:

CityFibre NOC Engineers have advised they are seeing network instability issues and are currently investigating.

Initial assessment appears to have identified that post the reintroduction of traffic following repairs stability was observed to deteriorate.

This recovery step has been reverted and investigation continues with CityFibre Engineers and Network Equipment Supplier TAC

Next Update:

12:00

Update Number: 13 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

-Engineer details confirmed and will be collecting at 0700.

-Access request to DC in confirmed

-Issue with retrieving parts from location resolved

– Engineer attended Slough DC

-Engineer has completed card swap successfully

-Testing and checks completed

-BGP reenabled

-Network stability confirmed

Current Action Plan:

CityFibre NOC Engineers have advised they are seeing network instability issues and are currently investigating.

Next Update:

12:00

Update Number: 12 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

-Engineer details confirmed and will be collecting at 0700.

-Access request to DC in confirmed

-Issue with retrieving parts from location resolved

– Engineer attended Slough DC

-Engineer has completed card swap successfully

-Testing and checks completed

-BGP reenabled

-Network stability confirmed

Current Action Plan:

Continuous network monitoring for the remainder of today

Next Update:

12:30

Update Number: 11 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

-Engineer details confirmed and will be collecting at 0700.

-Access request to DC in confirmed

-Issue with retrieving parts from location resolved

– Engineer attended Slough DC

-Engineer has completed card swap successfully

Current Action Plan:

Testing and checks in progress before moving BGP traffic back to original routing

Next Update:

10:30

Update Number: 10 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

-Engineer details confirmed and will be collecting at 0700.

-Access request to DC in confirmed

Current Action Plan:

Issue with retrieving parts from location, CityFibre NOC working with shipping partner to resolve currently

Engineer is in attendance awaiting to take parts to the Slough DC

Engineers ETA to Slough DC will now be approx 0830

Once in site engineer will commence with the Card swapout.

Next Update:

09:30

Update Number: 9 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

Current Action Plan:

Parts are ready for collection in Slough.

Engineer details confirmed and will be collecting at 0700.

Access request to DC in progress.

Engineers ETA to Slough DC will be approx 0730

Once in site engineer will commence with the Card swapout.

Next Update:

07:30