Incident Update

Update Number: 12 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

-Engineer details confirmed and will be collecting at 0700.

-Access request to DC in confirmed

-Issue with retrieving parts from location resolved

– Engineer attended Slough DC

-Engineer has completed card swap successfully

-Testing and checks completed

-BGP reenabled

-Network stability confirmed

Current Action Plan:

Continuous network monitoring for the remainder of today

Next Update:

12:30

Update Number: 11 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

-Engineer details confirmed and will be collecting at 0700.

-Access request to DC in confirmed

-Issue with retrieving parts from location resolved

– Engineer attended Slough DC

-Engineer has completed card swap successfully

Current Action Plan:

Testing and checks in progress before moving BGP traffic back to original routing

Next Update:

10:30

Update Number: 10 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

-Engineer details confirmed and will be collecting at 0700.

-Access request to DC in confirmed

Current Action Plan:

Issue with retrieving parts from location, CityFibre NOC working with shipping partner to resolve currently

Engineer is in attendance awaiting to take parts to the Slough DC

Engineers ETA to Slough DC will now be approx 0830

Once in site engineer will commence with the Card swapout.

Next Update:

09:30

Update Number: 9 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

Current Action Plan:

Parts are ready for collection in Slough.

Engineer details confirmed and will be collecting at 0700.

Access request to DC in progress.

Engineers ETA to Slough DC will be approx 0730

Once in site engineer will commence with the Card swapout.

Next Update:

07:30

Update Number: 7 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

Current Action Plan:

Awaiting transfer of parts to nearest Slough location, this is in progress and ETA of parts to this location is 03:45. Once arrived engineer will collet and make his way to the Slough Datatcentre and will commence Card swapout.

We should be able to estimate arrival time of engineer to Datacentre on the next update

Next Update:

03:30

Update Number: 8 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

– Part sourcing resolved

Current Action Plan:

Parts remain in transit to nearest Slough location, ETA still remains 03:45. Engineer attendance now requested and awaiting confirmation their ETA . Plan remains the same for the engineer to collect the parts, make his way to the Slough Datatcentre and commence Card swapout.

We will confirm ETA for engineers arrival at Slough DC as soon as possible

Next Update:

06:30

Update Number: 6(Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

– Escalation made to Hardware Supplier confirm part availability and Engineer ETA

Current Action Plan:

Issue identified with sourcing part, CityFibre NOC currently assessing options available

Next Update:

01:30

Update Number: 5 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of car performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

Current Action Plan:

Escalation made to to Hardware Supplier as no current ETA can be given. Engineering attendance has been requested to CityFibre Slough Datacentre to carry out a reseat of the card. Once reseated they will remain on site for at least 30 mins to confirm success. If the issue manifests again then engineer will proceed with a Card Swap out.

Next Update:

23:30

Update Number: 4 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

– It has been found that the origin point of the issue is on a line card situated within a core network device.

– Soft clear of card performed without success

– Full remote reboot of car performed which was successful for a period of approx. 30 mins before the issue manifested again

– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps

Current Action Plan:

Engineering attendance has been requested to CityFibre Slough Datacentre to carry out a reseat of the card. Once reseated they will remain on site for at least 30 mins to confirm success. If the issue manifests again then engineer will proceed with a Card Swap out.

Currently awaiting an ETA for engineers attendance to Slough

Next Update:

21:30

Update Number: 3 (Entanet / Cityfibre)

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

– Conference call between CityFibre NOC and hardware supplier convened

Current Action Plan:

Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.

It has been found that the origin point of the issue is on a line card situated within a core network device. During these investigations, two major actions have taken place in an attempt to restore service. The first action was to perform a soft clear on the line card itself which appeared to restore service for a short period before the issue resurfaced. And the second action was performing a full remote reboot of the affected line card.

During a period of monitoring, it has been seen that an issue has surfaced following the full reboot which is currently being investigated on a conference call between CityFibre NOC and our hardware supplier. Following on from the results of these investigations, the next course of action would be for engineers to be engaged to attend the site of the affected line card in order to perform a physical reseat of the line card itself.

Further to this, we are also considering a line card replacement should the issue resurface following a short monitoring period after the line card reseat.

Next Update:

19:30