Internet Disruption Report: March 2020

Although Internet disruptions have always been problematic for affected users, March 2020 was arguably the month where they became of ever greater global concern as so many shifted to working and learning from home due to COVID-19 driven lockdowns and shelter-in-place orders. The Internet has become a lifeline, supporting communication with loved ones, enabling online education, and facilitating days full of videoconferencing in place of in-person meetings.

While this blog has never claimed to be exhaustive in its coverage of Internet disruptions, it has endeavored to catalog the various causes of those disruptions, and most months see quite a few documented causes. Interestingly, March only saw documented disruptions due to power outages and cable/fiber issues, with a couple of additional due to possible network issues. (There were, of course, a number of other observed disruptions, but root causes were unable to be identified or confirmed through research or social media outreach.)

Power Outages

Unfortunately, the instability of Venezuela’s electrical power infrastructure has been the cause of Internet disruptions within the country multiple times in the past. During March, NetBlocks Tweeted about seven distinct Internet disruptions due to power failure in Venezuela – these occurred on March 1st, 3rd, 5th, 8th, 12th, 20th, and 26th. The four graphs below illustrate CAIDA IODA Active Probing results across 15 Venezuelan states across the month (one graph per week). For the disruptions visible on the dates highlighted by Netblocks, Táchira was one of the states with the most significant impact, as was Merida. It is interesting to note that similar Internet disruptions are also visible on March 7th, 9th, 13th, and 28th, although it is not clear whether these were also due to power outages in the affected states – Táchira appears to be impacted by these as well.

On March 22, a power outage also disrupted Internet connectivity on the Solomon Islands. Our Telekom (Solomon Telekom Company Limited), a local telecommunications service provider, posted on its Facebook page that its main exchange had experienced an outage due to a power failure. This outage is clearly visible in both the country- and network-level graphs below, starting around 0620 GMT, and ending around 1400 GMT.

Cable/Fiber Issues

On March 6, Oracle Internet Intelligence Tweeted that a new fault in a segment of the ACE submarine cable was impacting connectivity in Mauritania. The disruption is clearly visible in both the Oracle Internet Intelligence and CAIDA graphs below, starting mid-afternoon GMT, with partial recovery early the following day.

Although unconfirmed, the disruption visible in both graphs several days prior may also be related to problems with the ACE cable. On March 2, the CAIDA graph shows that all three measured metrics dropped shortly before noon GMT, with a gradual recovery taking place through the rest of the day, presumably as traffic shifted to alternate paths. In the Oracle graph, the active probing and BGP graphs showed concurrent drops as expected, but the DNS Query Rate graph spiked, indicating that whatever outbound traffic that actually made it out of Mauritania likely included requests by recursive resolvers driven by constant retries (due to lack of response) from DNS clients.

On March 9, Madagascar suffered an Internet disruption reportedly caused by roadwork severing a fiber optic cable. According to a published report (via Google Translate),

“At the end of yesterday afternoon, Internet users had the unpleasant surprise of not being able to access the Internet. It is a total cut causing discomfort to users, especially those who work on the Internet and use the Telma network. It was only in the early evening that everything returned to normal, much to everyone’s relief. According to a source with Telma, the cut comes from road works carried out in the southern part of the country and having affected the optical fiber. It was thus necessary to carry out a back up, either a reconnection or failover, on Lion cable to satisfy the users. It is a common practice between operators in case one of the companies is in a delicate situation as was Telma yesterday. Contrary to the claims of some science fiction enthusiasts, it is indeed a technical failure and not another reason.”

The mid-afternoon disruption is visible at a country level in the figures below, impacting both the active probing and BGP metrics. As the quote above notes, TELMA (Telecom Malagasy) was also impacted – the graphs below illustrate that the active probing metrics declined in both Oracle’s and CAIDA’s measurements, while Oracle’s measurements also show a sharp increase in latency during the period of disruption.

On March 10, Gabon Telecom alerted its users via Twitter (see below) and Facebook that a cut to the SAT-3 submarine cable disrupted Internet traffic to its fixed network the day before. The post also references the ACE submarine cable cut in Mauritania discussed above, and appears to claim that the repairs to the ACE cable, which also connects to Gabon, will help [restore] Gabon Telecom’s traffic.

After posting a status update on March 11, Gabon Telecom confirmed the restoration of service on March 12. The figures below show that the associated Internet disruption lasted from March 9 -12 at both a country and network level. At a country level, the impact of the cable problems appears to be more severe in the CAIDA measurements than in Oracle’s, likely owing to the difference in the respective measurement vantage points. At a network level, the initial disruption is visible as a brief outage in the Oracle Internet Intelligence graph, although traffic from Gabon Telecom quickly began transiting a new upstream provider, but with increased latency. The graph shows that the latency returned to normal after the cable was repaired, although the upstream provider remained the same. On March 12, Gabon Telecom also notified customers that it would be granting up to 5 GB of additional mobile data to those impacted.

On March 12, Oracle Internet Intelligence Tweeted that a historic “dragon storm” was impacting Internet connectivity in Egypt. The graphs below show nominal disruptions to the active probing metrics, becoming most evident around mid-day GMT. Interestingly, the traffic-related metrics in both graphs also saw significant declines, with the diurnal patterns effectively flattening out. Telecom Egypt explained in a LinkedIn post on March 18 that “The service was interrupted in few segments located in Egypt as a result of terrestrial cable cuts resulting from the unusually powerful storm, recognized as the fiercest to hit the country in over 25 years, and accompanied by heavy rainfall, destructive winds, and dust storms, causing large-scale power disruptions in multiple areas in Egypt.”

A cable issue on March 14 impacted Internet connectivity across more than a dozen countries. Initially reported by Oracle Internet Intelligence as a submarine cable cut, a subsequent Tweet clarified that the subsea cable outage was caused by a cut in terrestrial fiber optic in Egypt. According to the Tweet below, the problem was with the EASSy cable system. (A submarine cable industry observer explains that Egypt is a critical transit cross road for international East-West cables, as well as terrestrial and submarine diversified backhaul links beween Suez Canal North (Alexandria / Port Saïd) and South (Suez) landing stations.)

The graphs below show the impact of the issue at a country level in Burundi, Djibouti, Tanzania, Seychelles, Sudan, and the United Arab Emirates. In the United Arab Emirates, the disruption is barely evident in the active probing and BGP metrics, but it is significantly more visible across the other countries.

The Oracle Internet Intelligence graphs below show the impact of the disruption at a network level for providers in Bangladesh (AS132602), India (AS55410), Kenya (AS33771), Mozambique (AS37342), Saudi Arabia (AS47794), and Thailand (AS38040). Measurements to these networks saw a brief period of increased latency when the disruption occurred, though as the graphs show, measurements to these networks generally shifted to go through alternative upstream providers when one was impacted.

Possible Network Issues

Early in the morning (GMT) of March 23, a brief but significant Internet disruption was observed in Algeria. The graphs below show that Oracle Internet Intelligence recorded a partial decline in both the active probing and BGP metrics, while CAIDA IODA saw a significant drop in the active probing metrics, but no change to the BGP metric.

The disruption was also clearly evident in measurements of Telecom Algeria, as seen in the graphs below. Oracle Internet Intelligence testing measured a drop in the number completed traceroutes to targets within the network across multiple upstream providers, while the CAIDA IODA measurements saw a similar sharp decline in successful active probing.

However, there was apparently some question about the cause of the disruption, as Telecom Algeria posted a “disclaimer” to its Facebook page on March 24 that claimed (via Google Translate) “Contrary to what has been announced and reported by some media and on social networks about of a supposed shutdown of the Internet at the level national. Algeria Telecom wants to deny and reassure his kind clientele that no break is programmed.” It isn’t clear if this disclaimer refers to the disruption observed on the previous day, as the translation appear to refer to a future planned Internet shutdown, despite being posted the day after the disruption. (Checking against other online tools also results in similar translations.)

Closing out March, a brief Internet disruption was observed in Grenada on the 31st. The graphs below show that the disruption was clearly visible at both a country and network level (Columbus Communications Grenada a/k/a FLOW Grenada).

In response to an inquiry over Facebook Messenger, FLOW Grenada explained “There was an unplanned outage that had affected our Grenada customers on March 31st. Our team had managed to have services working in less than 24 hours. We sincerely apologize for the inconvenience caused.” The provider also compensated customers with free data as an apology for the outage.

Conclusion

As traffic volumes grew and traffic patterns shifted during March due to increased usage, many worried that the Internet would “break” and that Internet disruptions would become more significant and more frequent. As we came out of the month, it was clear that, as predicted, Internet resilience remained firmly in place. Core Internet infrastructure providers reported that while they saw increased traffic on their networks, they could easily handle it. Internet Exchange Points (IXPs) and Content Delivery Networks (CDNs) also reported record traffic growth, but they helped keep that traffic local. Well-provisioned last mile broadband networks reported increased traffic and shifting peak times, but neither caused disruptions.

However, the Internet has strained to adapt in underserved and unserved areas, where insufficient investment has been made to make high-speed broadband available and affordable. Users on these networks perceive that Internet disruptions have occurred more frequently as their connections are unable to cope with the demands of videoconferencing, streaming media, and online learning, often all concurrently competing for bandwidth. The depth of the “digital divide” became much clearer around the world in March – policy changes and directed investment are needed to deliver an Internet that is resilient for all users.

2 Comments

Leave a comment