– Reports of circuit impact into the CityFibre TSC
– CityFibre TSC engaged CityFibre NOC for initial investigations
– CityFibre NOC confirmed an issue seen on active monitoring
– MI process engaged
– MI accepted
– Internal Bridge call scheduled
– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC
– Further impact to Consumer circuits discovered and acknowledged
– NOC investigations determined an issue within the core network emanating from a specific location
– NOC contacted hardware supplier and raised a Priority 1 case
– All logs provided to hardware supplier for analysis
– Internal Bridge call convened
– Conference call between CityFibre NOC and hardware supplier convened
– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.
– It has been found that the origin point of the issue is on a line card situated within a core network device.
– Soft clear of card performed without success
– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again
– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps
– Escalation made to Hardware Supplier confirm part availability and Engineer ETA
– Part sourcing resolved
-Engineer details confirmed and will be collecting at 0700.
-Access request to DC in confirmed
-Issue with retrieving parts from location resolved
– Engineer attended Slough DC
-Engineer has completed card swap successfully
-Testing and checks completed
-Network stability confirmed
-CityFibre NOC Engineers have advised they are seeing network instability issues and are currently investigating.
-Initial assessment appears to have identified that post the reintroduction of traffic following repairs stability was observed to deteriorate. This recovery step has been reverted
Current Action Plan:
investigation continues with diagnostics being carried out on the Network Device. Network traffic remains rerouted and services have been observed as stable at this time