Inbound and Outbound Calling

Major incident Telephony
2023-06-29 11:57 BST · 19 hours, 22 minutes

Updates

Resolved

We believe that the issue is now resolved and calls should be flowing as normal.

If you are experiencing any remaining issues - please email us [email protected] and we’ll investigate asap.

many thanks

June 30, 2023 · 07:19 BST
Update

We are releasing traffic back into the platform and monitoring mitigation closely. Customers will be seeing service restore.

June 29, 2023 · 14:34 BST
Update

We are releasing traffic back into the platform and monitoring mitigation closely. Customers will be seeing service restore.

June 29, 2023 · 14:26 BST
Update

Our teams continue to mitigate todays incident. A further update will be provided at 2:30pm if not before.

June 29, 2023 · 14:09 BST
Update

Nothing further to report at this stage. Mitigation remains ongoing. A further update will be provided in 15 minutes.

June 29, 2023 · 13:43 BST
Update

The traffic volumes remain largely mitigated but we are bringing online additional capacity to ensure we have un-usually high levels of overhead as further mitigation at our core. We add capacity routinely during normal volumes of traffic, but not when under the pressure of current volumes, hence restricting parts of our network whilst we do so. Customers will be seeing traffic flow momentarily as we do so but we don’t expect full resolution until this work is complete. A further update will be provided in 15 minutes.

June 29, 2023 · 13:30 BST
Update

We are proactively flushing stale traffic resulting from the traffic peaks to restore access to all. A further update will provided in 15 minutes.

June 29, 2023 · 13:07 BST
Update

Please note we are pro-actively limiting traffic to some accounts at present. We recognise this is disruptive and hope to restore full access to all accounts shortly. The issue remains as reported at the start of our incident.

June 29, 2023 · 12:50 BST
Update

We will continue to mitigate the traffic and monitor but all metrics have resumed to normal. A more detailed post mortem will be published later in the business day once monitoring has completed and our teams have fully debriefed on the incident.

June 29, 2023 · 12:38 BST
Update

Further mitigation on inbound has been implemented and we expect all inbound calls to resume without the need to re-register devices within the next minute.

June 29, 2023 · 12:38 BST
Monitoring

We are confident our mitigation is effective and our NOC team will now monitor. Traffic and packets per second peaked at close to 10x our normal volumes. Note that we disabled our proxy solutions for a period of 30 seconds or so during mitigation and as a result hardware may need to re-register by rebooting devices. This would impact inbound but not outbound calling.

June 29, 2023 · 12:37 BST
Investigating

We have now reduced traffic to levels that allow us to restore access to all. We continue to investigate but service has now been restored to all users. We continue to mitigate substantive volumes and are not closing the incident at this stage.

June 29, 2023 · 12:14 BST
Update

Our provider hasn’t fully isolated the root cause, but as of this update, they are seeing traffic typical of a substantive DDOS attack. These are something they manage routinely and daily, however In this case the traffic is inordinate in scale. They continue to investigate and will provide more substantive updates as their NOC teams continue their urgent work to mitigate and restore access to all accounts.

June 29, 2023 · 12:12 BST
Investigating

Our upstream provider will be providing updates shortly. Further updates to follow within 15 minutes.

June 29, 2023 · 12:00 BST
Issue

We are aware of an issue which is affecting outbound and inbound calls at the moment.

If you are trying to reach us and can’t get through, please email [email protected]

June 29, 2023 · 11:57 BST

← Back