While this blog has never claimed to be exhaustive in its coverage of Internet disruptions, it has endeavored to catalog the various causes of those disruptions, and most months see quite a few documented causes. Interestingly, March only saw documented disruptions due to power outages and cable/fiber issues, with a couple of additional due to possible network issues. (There were, of course, a number of other observed disruptions, but root causes were unable to be identified or confirmed through research or social media outreach.)

Power Outages

Unfortunately, the instability of Venezuela’s electrical power infrastructure has been the cause of Internet disruptions within the country multiple times in the past. During March, NetBlocks Tweeted about seven distinct Internet disruptions due to power failure in Venezuela – these occurred on March 1st, 3rd, 5th, 8th, 12th, 20th, and 26th. The four graphs below illustrate CAIDA IODA Active Probing results across 15 Venezuelan states across the month (one graph per week). For the disruptions visible on the dates highlighted by Netblocks, Táchira was one of the states with the most significant impact, as was Merida. It is interesting to note that similar Internet disruptions are also visible on March 7th, 9th, 13th, and 28th, although it is not clear whether these were also due to power outages in the affected states – Táchira appears to be impacted by these as well.

On March 22, a power outage also disrupted Internet connectivity on the Solomon Islands. Our Telekom (Solomon Telekom Company Limited), a local telecommunications service provider, posted on its Facebook page that its main exchange had experienced an outage due to a power failure. This outage is clearly visible in both the country- and network-level graphs below, starting around 0620 GMT, and ending around 1400 GMT.

Cable/Fiber Issues

On March 6, Oracle Internet Intelligence Tweeted that a new fault in a segment of the ACE submarine cable was impacting connectivity in Mauritania. The disruption is clearly visible in both the Oracle Internet Intelligence and CAIDA graphs below, starting mid-afternoon GMT, with partial recovery early the following day.

Although unconfirmed, the disruption visible in both graphs several days prior may also be related to problems with the ACE cable. On March 2, the CAIDA graph shows that all three measured metrics dropped shortly before noon GMT, with a gradual recovery taking place through the rest of the day, presumably as traffic shifted to alternate paths. In the Oracle graph, the active probing and BGP graphs showed concurrent drops as expected, but the DNS Query Rate graph spiked, indicating that whatever outbound traffic that actually made it out of Mauritania likely included requests by recursive resolvers driven by constant retries (due to lack of response) from DNS clients.

On March 9, Madagascar suffered an Internet disruption reportedly caused by roadwork severing a fiber optic cable. According to a published report (via Google Translate),

“At the end of yesterday afternoon, Internet users had the unpleasant surprise of not being able to access the Internet. It is a total cut causing discomfort to users, especially those who work on the Internet and use the Telma network. It was only in the early evening that everything returned to normal, much to everyone’s relief. According to a source with Telma, the cut comes from road works carried out in the southern part of the country and having affected the optical fiber. It was thus necessary to carry out a back up, either a reconnection or failover, on Lion cable to satisfy the users. It is a common practice between operators in case one of the companies is in a delicate situation as was Telma yesterday. Contrary to the claims of some science fiction enthusiasts, it is indeed a technical failure and not another reason.”

The mid-afternoon disruption is visible at a country level in the figures below, impacting both the active probing and BGP metrics. As the quote above notes, TELMA (Telecom Malagasy) was also impacted – the graphs below illustrate that the active probing metrics declined in both Oracle’s and CAIDA’s measurements, while Oracle’s measurements also show a sharp increase in latency during the period of disruption.

On March 10, Gabon Telecom alerted its users via Twitter (see below) and Facebook that a cut to the SAT-3 submarine cable disrupted Internet traffic to its fixed network the day before. The post also references the ACE submarine cable cut in Mauritania discussed above, and appears to claim that the repairs to the ACE cable, which also connects to Gabon, will help [restore] Gabon Telecom’s traffic.

COMMUNIQUÉ pic.twitter.com/XAleUwmGfK

— Gabon Telecom (@gt_officiel) March 10, 2020

After posting a status update on March 11, Gabon Telecom confirmed the restoration of service on March 12. The figures below show that the associated Internet disruption lasted from March 9 -12 at both a country and network level. At a country level, the impact of the cable problems appears to be more severe in the CAIDA measurements than in Oracle’s, likely owing to the difference in the respective measurement vantage points. At a network level, the initial disruption is visible as a brief outage in the Oracle Internet Intelligence graph, although traffic from Gabon Telecom quickly began transiting a new upstream provider, but with increased latency. The graph shows that the latency returned to normal after the cable was repaired, although the upstream provider remained the same. On March 12, Gabon Telecom also notified customers that it would be granting up to 5 GB of additional mobile data to those impacted.

On March 12, Oracle Internet Intelligence Tweeted that a historic “dragon storm” was impacting Internet connectivity in Egypt. The graphs below show nominal disruptions to the active probing metrics, becoming most evident around mid-day GMT. Interestingly, the traffic-related metrics in both graphs also saw significant declines, with the diurnal patterns effectively flattening out. Telecom Egypt explained in a LinkedIn post on March 18 that “The service was interrupted in few segments located in Egypt as a result of terrestrial cable cuts resulting from the unusually powerful storm, recognized as the fiercest to hit the country in over 25 years, and accompanied by heavy rainfall, destructive winds, and dust storms, causing large-scale power disruptions in multiple areas in Egypt.”

A cable issue on March 14 impacted Internet connectivity across more than a dozen countries. Initially reported by Oracle Internet Intelligence as a submarine cable cut, a subsequent Tweet clarified that the subsea cable outage was caused by a cut in terrestrial fiber optic in Egypt. According to the Tweet below, the problem was with the EASSy cable system. (A submarine cable industry observer explains that Egypt is a critical transit cross road for international East-West cables, as well as terrestrial and submarine diversified backhaul links beween Suez Canal North (Alexandria / Port Saïd) and South (Suez) landing stations.)

EASSy cable system is down due to a terrestrial fault in Egypt between Abu Talat and Zaafarana on the Djibouti to Marseilles segment. We are waiting for confirmation as to where the SEACOM break occurred and whether this is in the same location.

— SA NREN Operational Updates & Alerts (@RENAlerts) March 14, 2020

The graphs below show the impact of the issue at a country level in Burundi, Djibouti, Tanzania, Seychelles, Sudan, and the United Arab Emirates. In the United Arab Emirates, the disruption is barely evident in the active probing and BGP metrics, but it is significantly more visible across the other countries.

The Oracle Internet Intelligence graphs below show the impact of the disruption at a network level for providers in Bangladesh (AS132602), India (AS55410), Kenya (AS33771), Mozambique (AS37342), Saudi Arabia (AS47794), and Thailand (AS38040). Measurements to these networks saw a brief period of increased latency when the disruption occurred, though as the graphs show, measurements to these networks generally shifted to go through alternative upstream providers when one was impacted.

Possible Network Issues

Early in the morning (GMT) of March 23, a brief but significant Internet disruption was observed in Algeria. The graphs below show that Oracle Internet Intelligence recorded a partial decline in both the active probing and BGP metrics, while CAIDA IODA saw a significant drop in the active probing metrics, but no change to the BGP metric.

The disruption was also clearly evident in measurements of Telecom Algeria, as seen in the graphs below. Oracle Internet Intelligence testing measured a drop in the number completed traceroutes to targets within the network across multiple upstream providers, while the CAIDA IODA measurements saw a similar sharp decline in successful active probing.

However, there was apparently some question about the cause of the disruption, as Telecom Algeria posted a “disclaimer” to its Facebook page on March 24 that claimed (via Google Translate) “Contrary to what has been announced and reported by some media and on social networks about of a supposed shutdown of the Internet at the level national. Algeria Telecom wants to deny and reassure his kind clientele that no break is programmed.” It isn’t clear if this disclaimer refers to the disruption observed on the previous day, as the translation appear to refer to a future planned Internet shutdown, despite being posted the day after the disruption. (Checking against other online tools also results in similar translations.)

Closing out March, a brief Internet disruption was observed in Grenada on the 31st. The graphs below show that the disruption was clearly visible at both a country and network level (Columbus Communications Grenada a/k/a FLOW Grenada).

In response to an inquiry over Facebook Messenger, FLOW Grenada explained “There was an unplanned outage that had affected our Grenada customers on March 31st. Our team had managed to have services working in less than 24 hours. We sincerely apologize for the inconvenience caused.” The provider also compensated customers with free data as an apology for the outage.

We know many of you were without service for a short period today, we are giving 1GB of data free to say thank you for your support and understanding. pic.twitter.com/AIkOSXfJfM

— Flow Grenada (@FLOWGrenada) March 31, 2020

Conclusion

As traffic volumes grew and traffic patterns shifted during March due to increased usage, many worried that the Internet would “break” and that Internet disruptions would become more significant and more frequent. As we came out of the month, it was clear that, as predicted, Internet resilience remained firmly in place. Core Internet infrastructure providers reported that while they saw increased traffic on their networks, they could easily handle it. Internet Exchange Points (IXPs) and Content Delivery Networks (CDNs) also reported record traffic growth, but they helped keep that traffic local. Well-provisioned last mile broadband networks reported increased traffic and shifting peak times, but neither caused disruptions.

However, the Internet has strained to adapt in underserved and unserved areas, where insufficient investment has been made to make high-speed broadband available and affordable. Users on these networks perceive that Internet disruptions have occurred more frequently as their connections are unable to cope with the demands of videoconferencing, streaming media, and online learning, often all concurrently competing for bandwidth. The depth of the “digital divide” became much clearer around the world in March – policy changes and directed investment are needed to deliver an Internet that is resilient for all users.

Cable Complications

Greenland receives international Internet connectivity via the Greenland Connect submarine cable, linking the country to Iceland and Canada through landing points in Nuuk and Qaqortaq. In December 2018, the submarine cable link between these two cities suffered damage, disrupting Internet connectivity for users in Greenland. TELE-POST, a subsidiary of national telecommunications company TELE Greenland, has been tracking cable repair efforts over the last several months — efforts that have been slowed due to weather and ice cover.

Some repairs took place in early May, stabilizing Internet connectivity for users in North Greenland. However, disruptions to Internet connectivity in Greenland were also observed during this time frame, as seen in the figures below. While it has not been definitively confirmed that these disruptions were, in fact, due to cable repair efforts, TELE-POST has noted on the status site that connectivity for some customers may be turned on/off while repair efforts are underway.

A similar disruption was also observed on May 22, as shown in the figures below. That day, TELE-POST noted that it would be postponing repair work until August due to ice conditions. It is likely that similar disruptions will continue to occur over the coming months, as the update states: “However, in the coming period, until the important repair work next to Qaqortoq has been completed, there will be an increased risk of operational disturbances, particularly of the Internet service.”

On May 4, Globe Telecom subscribers in several regions of the Philippines experienced Internet service disruptions due to damaged fiber optic cables. A submarine cable was damaged by a 5.5-magnitude earthquake, while another cable was damaged by a vehicle dragging the cable at a road crossing. The figure below shows the impact of the cable damage to connectivity for AS4775 (Globe Telecom) – a minor change was seen in BGP, while successful active probing was impacted for just over three hours; measured darknet traffic dropped briefly at the start of the event, but appeared to recover quickly.

Apparently a popular day for Internet disruptions, Haiti also experienced an issue during the afternoon of May 4, as shown in the figure below. According to a number of Twitter responses from Digicel Haiti (@DigicelHT) to user complaints, the brief disruption was related to issues with a submarine cable.

The figure below shows that connectivity to Digicel Haiti through Digicel Jamaica was disrupted. The Fibralink submarine cable connects the two countries, so it may have been the cable referenced (but not named) in the @DigicelHT Tweets.

Just over two weeks later on May 19, AS15964 (Cameroon Telecommunications Network) experienced a connectivity disruption. As shown in the figure below, the number of traceroutes to endpoints in the network going through Tata dropped significantly for several hours, with Telxius and Orange picking up the slack.

This disruption impacted downstream networks in different ways. The figures below illustrate the impact of the disruption to two of those networks – AS30992 (MTN Cameroon) and AS37620 (VIETTEL CAMEROUN SARL). The number of successful traceroutes to endpoints in MTN declined by approximately half during the disruption period, while the number of traceroutes to endpoints in VIETTEL going through Cameroon Telecommunications Network unexpectedly increased during that period, although that shift also brought a significant increase in latency.

In a Tweet from @MTNCameroon that was subsequently deleted, the company acknowledged the apparent connectivity issues. The deleted Tweet was in response to a user complaint that included what appeared to be a screenshot of a text message that reads (via Google Translate) “Dear customer, following an incident on the optical fiber provided to us, our network is momentarily disrupted. MTN sexcuse for possible inconvenience.” This same (original) text was also Tweeted by several other users (1, 2) that same day. However, it is also worth noting that the text appears in Tweets from February 2019 and October/September/June/April 2018, so it may just be a stock customer support response from MTN Cameroon, rather than a definitive root cause.

On May 21 & 22, Malawi held elections to name a President, new members of Parliament, and local councilors. After the polls closed, Internet connectivity was disrupted across multiple providers in the country, reportedly due to damage to a cable owned by Malawi Telecommunications Limited. The figure below shows the impact to Internet connectivity for AS36969 (Malawi Telecommunications Limited) – successful active probing trended lower for several hours, and darknet activity was reduced as well.

The disruption also impacted local media outlets, as seen in the Facebook post below from Mibawa TV.

Satellite Suspicions

During May 2019, several Internet disruptions were observed that appear to be related to a reliance on satellite connectivity by major telecommunications providers in the impacted countries. In each case, the observed country-level disruption is coincident with a drop in the number of completed traceroutes reaching endpoints on the providers’ networks through satellite connectivity providers. In the figures below, the disruptions in South Sudan, Solomon Islands, Kiribati, and Cook Islands appear to be related to issues with O3b, while in Eritrea, connectivity through Asia Broadcast Satellite was problematic.

Wayward Weather

Damage caused by severe weather events frequently wreaks havoc on electrical and communications infrastructure. In early May, a storm in the U.S. Virgin Islands impacted Internet connectivity for users across multiple networks within the territory. Reduced levels of successful traceroute completions and active probing can be seen late in the day (UTC) on May 1 in the figures below.

These disruptions were also visible in network-level graphs for local network providers, as seen in the figures below. A Facebook thread highlighted issues with Broadband VI, and a Facebook Messenger response from a representative of Virgin Islands Next Generation Network noted “Our IT Team has indicated that there was a storm affecting the island of St. Croix on Wednesday, May 1. We were able to confirm with Broadband VI that they experienced a lightning strike. This appears to correspond with what you observed.”

The highly localized impact of extreme weather was highlighted by CAIDA’s IODA team towards the end of the month. In the Tweet embedded below, the team shared graphs of disruptions in active probing to endpoints in two Ohio counties on May 28, with the disruption timeframes correlating with reports of a tornado in those areas.

Other Outages

In addition to the disruptions discussed above that had frequently seen (probable) underlying causes, several additional disruptions were observed in May that were significant enough to mention here as well.

On May 5, the @BGPStream Twitter feed posted 70 “outage” alerts for autonomous systems (ASNs) associated with the United States Department of Defense, all at 01:51 UTC. BGPStream maintainer Andre Toonk explained what happened in the Tweet embedded below, highlighting the risk of single-homing multiple networks behind a single provider.

Just over a week later, on May 13 a significant Internet disruption was observed in China, apparently due to an issue at China Telecom. This event was explored in more detail in a previous post: “Multi-Hour Disruption at China Telecom”.

On May 22, published reports (The Straits Times, The Independent) indicated that thousands of fiber broadband subscribers of Singapore provider M1 (MobileOne) experienced a multi-hour Internet disruption, but M1 did not say what caused the disruption. Interestingly, it occurred a day after hundreds of customers of three local providers – Singtel, StarHub and M1 – had Internet connectivity disrupted by a fiber cut at Netlink Trust, which owns Singapore’s fiber network and resells fiber connectivity to these telecommunications providers.

Conclusion

Physical damage to terrestrial and submarine cables has long been a cause of Internet disruptions, as has problems with satellite communications for those regions underserved with wired connectivity. The continued growth in both submarine cable buildout and satellite Internet services should help provide redundancy and resiliency, reducing the frequency and severity of large-scale Internet disruptions. However, these additional avenues of connectivity will not change the likelihood that government-directed Internet disruptions will continue to occur (probably becoming more targeted and frequent), nor will they mitigate transient disruptions impacting local areas or individual network providers.