Incident Update

Status Update

July 22, 2022 @ 4:04 pm
The issue that is affecting a number of client leased lines / EFM circuits is in a state of remediation with the ISP (Entanet / Cityfibre) who have identified a piece of equipment that they believe is causing the issues.

At this point in time our internal monitoring is showing lines are stable, however they may drop without notice if Entanet / Cityfibre alter anything within their network to resolve the issues fully. Due to the fact that drops may occur at anytime it may be prudent to ask if staff can work from home (where possible) for the remainder of the day

Local internet traffic from your offices may also be affected due to issues within the Entanet / Cityfibre network that cause pages to load either really slowly or not at all, these issues may be sporadic in nature as well, this is due to line / tunnel flaps within the Entanet / Cityfibre network.

We will continue to post update as we get them from Entanet / Cityfibre.

Explainer: A circuit / line / tunnel flap is when a connection goes between a state of up (working) and down (Not working) rapidly, giving the impression of a line that is up (working), but its non functional and data cannot be passed over it.

The issue can be compounded by the way the internet works where if it see this happening it tries to work around it, but if that new route does the same thing, then the issue gets exponentially worse until other systems see it as having issues and drop their connections automatically as a precaution to protect their networks, which then causes an issue called convergence where the networks need to work out new paths to each other to “heal” the issue, this of course takes time (few seconds to several minutes)  to carry out as there are a lot of moving parts involved to try to work around issues and keep data moving.

Status Update

July 22, 2022 @ 3:37 pm
 
Description: Major Incident – INC0107444 – Multiple circuit outage

Current Impact: Impacting a number of Leased Line, Broadband and Consumer circuits. Exact numbers unknown

Incident Start Time: 22/07/2022 11:57:59

Major Incident Raised: 22/07/2022 12:25:43

Major Incident Manager: Sam Abberley

Location/City: Nationwide

Update Number: 2

Completed Actions:

– Reports of circuit impact into the CityFibre TSC

– CityFibre TSC engaged CityFibre NOC for initial investigations

– CityFibre NOC confirmed an issue seen on active monitoring

– MI process engaged

– MI accepted

– Internal Bridge call scheduled

– NOC investigations ongoing with several examples of affected circuits provided from information gathering by TSC

– Further impact to Consumer circuits discovered and acknowledged

– NOC investigations determined an issue within the core network emanating from a specific location

– NOC contacted hardware supplier and raised a Priority 1 case

– All logs provided to hardware supplier for analysis

– Internal Bridge call convened

Current Action Plan:

CityFibre NOC have been investigating further and have determined that the issue is emanating from a specific location with the core network. As such, this device has been checked and the hardware supplier has been contacted directly for a Priority 1 case to be raised with them. During this time, CityFibre NOC have attempted to alleviate some issues across the core network by performing a workaround to reroute traffic to avoid the suspected affected device.

All of the requisite logs have been provided and CityFibre NOC are working alongside the hardware supplier to determine our next course of action. As of yet, it is still unclear as to when full service can be restored however we will endeavour to provide this as soon as we have it available.

We apologise for the disruption and hope to have a further update soon.

Next Update:

17:30

Status Update

March 23, 2021 @ 4:35 pm
From the reports we are receiving this was a major issue on one of the main comms providers in the UK which is affecting a significant part of the country.
 
We are working with our respective comms providers to fully understand the root cause, but all client connectivity issues should now be resolved.
 
There may be some further interruptions whilst the issue is fully investigated. 

Status Update

February 22, 2021 @ 2:34 pm

This should be resolved in the next 5 minutes.

Status Update

February 8, 2021 @ 8:27 am
Mimecast believe they have identified the issue within their network. We are starting to see sporadic email delivery to and from Mimecast.
 
Please note: Emails may be delayed as the service recovers.