Update Number: 18(Entanet / Cityfibre)
Completed Actions:
- Reports of circuit impact into the CityFibre TSC
- CityFibre TSC engaged CityFibre NOC for initial investigations
- CityFibre NOC confirmed an issue seen on active monitoring
- MI process engaged
- MI accepted
- Internal Bridge call scheduled
- NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC
- Further impact to Consumer circuits discovered and acknowledged
- NOC investigations determined an issue within the core network emanating from a specific location
- NOC contacted hardware supplier and raised a Priority 1 case
- All logs provided to hardware supplier for analysis
- Internal Bridge call convened
- Conference call between CityFibre NOC and hardware supplier convened
- Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.
- It has been found that the origin point of the issue is on a line card situated within a core network device.
- Soft clear of card performed without success
- Full remote reboot of car performed which was successful for a period of approx. 30 mins before the issue manifested again
- Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps
- Escalation made to Hardware Supplier confirm part availability and Engineer ETA
- Part sourcing resolved
- Engineer details confirmed and will be collecting at 0700.
- Access request to DC in confirmed
- Issue with retrieving parts from location resolved
- Engineer attended Slough DC
- Engineer has completed card swap successfully
- Testing and checks completed
- BGP reenabled
- Network stability confirmed
- CityFibre NOC Engineers have advised they are seeing network instability issues and are currently investigating.
- Initial assessment appears to have identified that post the reintroduction of traffic following repairs stability was observed to deteriorate. This recovery step has been reverted
- Investigation continued with diagnostics being carried out on the Network Device. Network traffic remains rerouted and services have been observed as stable at this time.
- Card causing alarms remains out of service
- Resilient links remain carrying traffic mitigating any remaining impact to services previously utilising that card and to prevent any further service disruption by our planned restoration activity.
- Engineers swapped the Card causing alarms successfully at 19:25
Current Action Plan:
- Current service impact – None, all resilient ports are back in service
- Network impact – None, restored to previous state. With the exception of 2 core links which are still costed out
- Monitoring will now commence for 24 hours, after this time period the 2 costed out links will be brought back into service sequentially under controlled engineer conditions
- Further update will be posted prior to commencement of work to bring the 2 links back into service
Next Update:
12:00 Sunday 24th July