Update Number: 20 (Entanet / Cityfibre)
Completed Actions:
- Reports of circuit impact into the CityFibre TSC
- CityFibre TSC engaged CityFibre NOC for initial investigations
- CityFibre NOC confirmed an issue seen on active monitoring
- MI process engaged
- MI accepted
- Internal Bridge call scheduled
- NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC
- Further impact to Consumer circuits discovered and acknowledged
- NOC investigations determined an issue within the core network emanating from a specific location
- NOC contacted hardware supplier and raised a Priority 1 case
- All logs provided to hardware supplier for analysis
- Internal Bridge call convened
- Conference call between CityFibre NOC and hardware supplier convened
- Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.
- It has been found that the origin point of the issue is on a line card situated within a core network device.
- Soft clear of card performed without success
- Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again
- Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps
- Escalation made to Hardware Supplier confirm part availability and Engineer ETA
- Part sourcing resolved
- Engineer details confirmed and will be collecting at 0700.
- Access request to DC in confirmed
- Issue with retrieving parts from location resolved
- Engineer attended Slough DC
- Engineer has completed card swap successfully
- Testing and checks completed
- BGP reenabled
- Network stability confirmed
- CityFibre NOC Engineers have advised they are seeing network instability issues and are currently investigating.
- Initial assessment appears to have identified that post the reintroduction of traffic following repairs stability was observed to deteriorate. This recovery step has been reverted
- Investigation continued with diagnostics being carried out on the Network Device. Network traffic remains rerouted and services have been observed as stable at this time.
- Card causing alarms remains out of service
- Resilient links remain carrying traffic mitigating any remaining impact to services previously utilising that card and to prevent any further service disruption by our planned restoration activity.
- Engineers swapped the Card causing alarms successfully at 19:25
- Network Stability has been seen since Card replacement
Resolution Notes:
- Network Cards at two network locations replaced.
- The remaining two costed out links are internal to CityFibre and there is no impact to customers and will be forward managed separately
Resolved:
14:31 (Customer impact resolved post card Swap out at 19:25 on Saturday 23rd July)