Completed Actions:
– Reports of circuit impact into the CityFibre TSC
– CityFibre TSC engaged CityFibre NOC for initial investigations
– CityFibre NOC confirmed an issue seen on active monitoring
– MI process engaged
– MI accepted
– Internal Bridge call scheduled
– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC
– Further impact to Consumer circuits discovered and acknowledged
– NOC investigations determined an issue within the core network emanating from a specific location
– NOC contacted hardware supplier and raised a Priority 1 case
– All logs provided to hardware supplier for analysis
– Internal Bridge call convened
– Conference call between CityFibre NOC and hardware supplier convened
– Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.
– It has been found that the origin point of the issue is on a line card situated within a core network device.
– Soft clear of card performed without success
– Full remote reboot of card performed which was successful for a period of approx. 30 mins before the issue manifested again
– Further internal call held with CityFibre NOC and Hardware Supplier to agree next steps
– Escalation made to Hardware Supplier confirm part availability and Engineer ETA
– Part sourcing resolved
-Engineer details confirmed and will be collecting at 0700.
-Access request to DC in confirmed
-Issue with retrieving parts from location resolved
– Engineer attended Slough DC
-Engineer has completed card swap successfully
Current Action Plan:
Testing and checks in progress before moving BGP traffic back to original routing
Next Update:
10:30