Update – 09/30/2021 @ 5:36pm MDT
All services have continued to be fully operational throughout the day – no service impacts or events.
Update – 09/30/2021 @ 9:57am MDT
After the network issues last night, service has continued to be fully operational. In-depth inspection indicates that anti-DDoS mechanisms provided by Cloudflare have been inserted into the routing path to all locations around the country, dramatically reducing RTT & Jitter.
Service Issues – Update – 09/29/2021 @ 5:08PM MDT
One of our upstream carriers is experiencing sudden, nationwide impacts. Call processing is materially impacted at this time.
Update – 5:22pm – outbound calling has been rerouted to reduce post-dial delay, and outbound calls are flowing successfully. Inbound services materially impacted.
Update – 5:40pm – This does not appear to be a result of continued DDoS, but instead appears to be an Internet routing issue as carrier is entirely offline/routes are removed from the Internet.
Update – 5:44pm – Routes have been restored to the global Internet table, and connectivity has been restored; although call processing appears to be working, we are running a battery of tests.
Service Issues – Update – 09/29/2021 @ 1:00PM MDT
Service continues to be stable with no reported issues.
Service Issues – Update – 09/28/2021 @ 3:00PM MDT
Although we have seen a few cases of post-dial delay while rerouting calls, service has been very stable with little visible impact.
Note that audio may have a slight 1/16th-1/8th of a second delay as RTP is being routed through various scrubbers.
Please do report any service issues so that we can triangulate and isolate accordingly.
Update @ ~9:05pm MDT – Service has been fully stabilized with no call issues.
Service Issues – Update – 09/28/2021 @ 11:30am MDT
This morning, the DDoS ramped up again around 7:30am MDT, but mitigation efforts appear to be effective. We have seen small amounts of post-dial delay and the occassionally choppy audio, but most services are functional.
Please do report any service issues so that we can triangulate and isolate accordingly.
Service Issues – Update – 09/28/2021 @ 9:05am MDT
Last night at approximately 5:30pm MDT service appeared to stabilize materially.
This morning, we have seen a few statistics pop showing jitter/audio artifacts, but they evened out after a few minutes.
In the meantime, we have put an additional outbound call path into play through Denver, and will migrate outbound call load according to the call quality metrics that we see throughout the day. For now, it is just a backup in case of a call failure/no connect.
Off-net Call Service Issues – 09/27/2021
Early on the morning of September 27th, 2021 we became aware of reduced reachability to several of our upstream carriers IP addresses. Throughout the day, we have seen intermittent call completion, and after lots of tuning have stable outbound call flow while inbound calls continue to occassionaly have one-way audio or report a call cannot be completed error (Verizon mostly).
What is Happening?
Multiple Distributed Denial of Service (DDoS) attacks are being waged against multiple very large Voice backbone providers, with the aim of disrupting them, and in some cases obtaining a ransom[1]. These Voice providers are global in scope, and provide service to many very well known voip providers — the impact of these attacks is global and is estimated to be impacting ~40% of all VoIP traffic worldwide.
Option9 itself is up and running great, but connectivity to off-net numbers is impacted.
When did this start?
The DDoS attacks started back on September 16th against VoIP.ms [2], a Canadian backbone provider, and continues currently. On Saturday 9/25, a trial-DDoS was aimed at Bandwidth and VoIP Innovations, temporarily impacting service. On 09/27/21 around 9am EST, a full-scale attack was launched at these two providers, impacting tens of thousands of organizations.
What are we doing about this?
Option9 services utilize three separate providers to provide flexibility during outage situations. Two of these three providers are currently under attack, and we are actively using the third for outbound dialing wherever possible. We are aggressively lining up a migration strategy in case we need to port our many thousands of numbers to an alternate carrier.
We are also lining up temporary numbers for customers in case of mid-term solution is needed.
Why can’t you just fix it?
Telephone networks are not as flexible as the Internet – all inbound calls to your number must go through the backbone provider on which it lives. Moving numbers between backbones (porting) is at best a multi-day process. As such, inbound call completion will continue to be impacted while mitigations are put into place.
Please do reach out to us with your questions, concerns and comments – we’re happy to help however we can!
Sincerely,
Your Option9 Team