Completed Actions:
– Reports of circuit impact into the CityFibre TSC
– CityFibre TSC engaged CityFibre NOC for initial investigations
– CityFibre NOC confirmed an issue seen on active monitoring
– MI process engaged
– MI accepted
– Internal Bridge call scheduled
– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC
– Further impact to Consumer circuits discovered and acknowledged
– NOC investigations determined an issue within the core network emanating from a specific location
– NOC contacted hardware supplier and raised a Priority 1 case
– All logs provided to hardware supplier for analysis
– Internal Bridge call convened
– Conference call between CityFibre NOC and hardware supplier convened
Current Action Plan:
Following discussions between CityFibre NOC and our hardware supplier, there have been developments on this incident in regards to restoration.
It has been found that the origin point of the issue is on a line card situated within a core network device. During these investigations, two major actions have taken place in an attempt to restore service. The first action was to perform a soft clear on the line card itself which appeared to restore service for a short period before the issue resurfaced. And the second action was performing a full remote reboot of the affected line card.
During a period of monitoring, it has been seen that an issue has surfaced following the full reboot which is currently being investigated on a conference call between CityFibre NOC and our hardware supplier. Following on from the results of these investigations, the next course of action would be for engineers to be engaged to attend the site of the affected line card in order to perform a physical reseat of the line card itself.
Further to this, we are also considering a line card replacement should the issue resurface following a short monitoring period after the line card reseat.
Next Update:
19:30