Political Unrest

Unfortunately, during periods of political unrest, national governments have taken to implementing Internet shutdowns and social media filtering/blocking as a means of preventing the spread of (mis)information and limiting citizen ability to organize protests and similar activity.

On June 3, Internet connectivity in Sudan was disrupted in conjunction with attacks on protestors – the protests were related to the transition of power after the removal of Omar al-Bashir as Sudan’s president in April. The figure below shows drops in all three metrics measured by Oracle Internet Intelligence occurring around noon GMT.

Three weeks later, the Internet was restored… but only for a single lawyer that had filed a lawsuit against telecommunications provider Zain Sudan over the disruption ordered by Sudan’s military rulers. However, on July 9, local telecommunications providers were ordered to restore Internet service. This service restoration can be seen in the figure below, with increases in both the traceroute and BGP-based metrics.

An Internet disruption took place in Ethiopia on June 21 in the wake of an attempted coup, with the impacts visible in the figures below. Active inbound measurements by Oracle Internet Intelligence and CAIDA IODA saw success rates drop significantly as a result of the Internet shutdown, while passively measured DNS (Oracle), Darknet (CAIDA), and Web (Google) traffic dropped to near-zero levels. Connectivity was restored on June 27.

Following a contested presidential election in Mauritania, an Internet shutdown was put into place at approximately 15:30 GMT on June 25 as seen in the CAIDA IODA figure below – a significant drop in routed network prefixes occurred for several hours, while Active Probing and Darknet traffic sources dropped significantly.

Network diversity within the country is extremely limited. Looking at the figure for AS29544 (Mauritanian Telecommunication Company) below, it is clear that the routing instability in the country-level graph is related to this network’s routes essentially being removed from the Internet for that multi-hour period. Internet connectivity began to return across the country on July 3.

Cable Damage

Damage to terrestrial and undersea cables has long been the nemesis of a properly functioning Internet. The Internet disruptions due to cable damage observed during the first and last weeks of June tended to be brief and/or localized; similar issues seen in the past have caused significant country-wide Internet disruptions.

On June 4, Filipino telecommunications provider Globe Telecom posted a Public Advisory on its Facebook page that noted (in part): “Between 1-2PM today, the Globe Network experienced multiple fiber cuts, two of which were submarine cables.” The impact of these fiber cuts is clearly evident in the figures below for AS4775 and AS132199, both network identifiers associated with Globe Telecom in the Philippines, with multi-hour drops visible in both the Active Probing and BGP metrics. A slight perturbation to these metrics was barely visible in the country-level graph. A published report noted that connectivity was restored about four hours after the issue started, in line with the recovery time seen in the graphs.

On June 6, Internet monitoring and advocacy organization Netblocks Tweeted about an Internet outage in Venezuela that affected local network providers Digitel and Supercable. Netblocks noted that Digitel was completely disrupted while Supercable only experienced a partial outage. This is in line with the figures below, showing all metrics dropping to zero at Digitel starting around 15:00 GMT, and only a nominal decline in the Active Probing metric for Supercable around the same time. The Netblocks tweet stated that “sources report fiber cut”, but this root cause was not independently confirmed.

A brief Internet disruption observed in Pakistan on June 24 was due to multiple cable cuts, according to a press release issued by Pakistan Telecommunication Company Limited (PTCL). The figure below from CAIDA IODA shows a clear drop in the Active Probing metric at a country level, along with a slight dip in Darknet traffic sources.

The impact of the cable cuts can also be seen at a network level, visible in the two figures below for PTCL-owned networks.

The overall impact of the cable cuts appeared to be nominal, and the disruption short-lived, with metrics returning to normal levels within approximately four hours. @PTCLOfficial announced the full restoration of service just a few hours after the disruption began:

Iran is no stranger to politically-motivated Internet disruptions, and country leaders have spoken in the past about disconnecting the country from the global Internet. However, a brief disruption that occurred on June 26, shown in the figure below, may have instead been related to physical layer problems. A Tweet posted by Iranian ICT minister Azari Jahromi suggested that “a disruption of fiber connectivity” at upstream routes in Europe between Bulgaria & Frankfurt, which are used by several ISPs in Iran, caused the 1.5 hour disruption in connectivity.

National Exams

The governments of Syria, Ethiopia, and Iraq have all ordered Internet shutdowns in the past to prevent student cheating on national exams, and all three countries experienced such disruptions once again in June.

First up was Syria, where the first observed disruption, between 01:00 – 05:30 GMT on June 9, took place in conjunction with 712,000 university students starting their second term exams, along with 9th grade (O-Level) and 12th grade (A-Level) students starting public exams as well. Additional disruptions were also seen on June 11, 13, 17, and 19, all effectively appearing as complete country-wide outages in the Oracle Internet Intelligence, CAIDA IODA, and Google Transparency Report Traffic figures below.

It is interesting to note that within the Oracle Internet Intelligence Map graphs, the Traceroute Completion Ratio and BGP Routes metrics drop to zero during the periods of disruption, but the DNS Query Rate metric spikes. Doug Madory, Director of Internet Analysis at Oracle Internet Intelligence, explained why this happened:

On June 9, the first of four Internet disruptions apparently related to exams took place in Ethiopia, and lasted approximately six hours. A second disruption on June 11 lasted five hours, from 07:00 to 12:00 GMT. The third disruption, lasting approximately 12 hours, started in the evening of June 11, carrying over to the next day. The final disruption started the evening of June 12, and lasted 39 hours, until mid-day on June 14. The impact of these disruptions to connectivity in Ethiopia can be seen in the figures below — all caused a significant, but not complete, loss of connectivity at a country level. Published reports claimed that the outages were intended to prevent student cheating on national exams.

Ethio Telecom acknowledged, and apologized for, the disruptions in a Tweet, but neither the telecommunications company nor the government confirmed the reason they occurred.

Finally, on June 26, Iraq reportedly implemented regional Internet shutdowns intended to prevent student cheating on national Physics exams. The country-level CAIDA IODA figure for Iraq below shows a brief, and limited, drop in successful Active Probing late in the morning (GMT).

The CAIDA IODA figure for Sala ad-Din, a region of Iraq just north of Baghdad, shows a brief drop in the number of routed prefixes geolocated to the region. The drop occurs concurrent with the disruption seen at a national level.

The disruption is also visible at a network level, as the CAIDA IODA figure below for AS50710 (Earthlink) shows. Earthlink is one of the largest network service providers in Iraq, and during the period of disruption, a drop in the Active Probing metric can be seen.

Power Outage

On June 16, a massive power outage, called “unprecedented in its scope” by local officials, impacted all of mainland Argentina and Uruguay, affecting nearly 48 million people. The outage started at approximately 10:00 GMT, with near immediate impact on Internet connectivity to the affected countries and networks within them, as shown in the figures below. While local power utilities quickly began restoring power, Oracle Internet Intelligence and CAIDA IODA measurement metrics took nearly 12 hours to return to near-normal levels.

Electrical utilities in both countries provided status updates on Twitter throughout the event – the @UTE_ComCorp Tweet below indicated that 98.5% of power had been restored in Uruguay, approximately 11 hours after their first Tweet regarding the power failure, and the Tweet regarding restoration of service in Argentina from @OficialEdesur followed a similar timeline.

Route Leaks

Route leaks remain a common occurrence on the Internet. However, adoption of basic safeguards necessary to prevent propagation of routing leaks, such as the Internet Society’s Mutually Agreed Norms for Routing Security (MANRS) project, can mitigate the impact of such leaks. Unfortunately, neither China Telecom nor Verizon have apparently yet implemented such safeguards, and both carriers were implicated in massive route leaks that resulted in increased latency and/or service disruptions in June.

The first leak was observed on June 6, when Swiss data center colocation company Safe Host SA (AS21217) leaked over 70,000 routes to China Telecom (AS4134) in Frankfurt, Germany. An excellent and detailed blog post from Oracle’s Doug Madory explained the impact of the leak:

“China Telecom then announced these routes on to the global Internet redirecting large amounts of Internet traffic destined for some of the largest European mobile networks through China Telecom’s network. Impacts were seen by some of Europe’s largest networks in Switzerland, Holland, and France among other countries. … Often routing incidents like this only last for a few minutes, but in this case many of the leaked routes in this incident were in circulation for over two hours.”

The impact of the leak from a traceroute and latency perspective was illustrated in a Tweet from @InternetIntel:

The second notable route leak occurred on June 24, impacting approximately 20,000 prefixes from 2,400 network providers, according to BGPmon. Content delivery and security service provider Cloudflare was significantly impacted by the routing leak, and it too published an excellent and detailed blog post explaining what happened.

“An Internet Service Provider in Pennsylvania  (AS33154 – DQE Communications) was using a BGP optimizer in their network, which meant there were a lot of more specific routes in their network. Specific routes override more general routes… DQE announced these specific routes to their customer (AS396531 – Allegheny Technologies Inc). All of this routing information was then sent to their other transit provider (AS701 – Verizon), who proceeded to tell the entire Internet about these “better” routes.”

In addition to Cloudflare, this route leak impacted connectivity for many other major organizations. BGPmon’s BGPStream Web site publishes alerts about BGP hijacks, leaks, and outages, and the table below is based on data extracted from the BGPStream site for alerts associated with this leak. It is clear in looking at the list that there was a global impact, with networks in Europe and the Asia Pacific region among those affected.

Localized Disruptions

While these blog posts generally cover large-scale, country-level disruptions, easily identifiable within publicly available monitoring tools, more localized disruptions are often visible through these tools as well.

On June 7, Consolidated Communications, a network provider that services Vermont, reportedly experienced a “router issue” at its office in Burlington, disrupting connectivity to all 80,000 Internet customers in the state, including businesses and homes. The figures below show the impact of the disruption, with active measurement metrics declining for a brief period of time. Although the autonomous system label in the graphs is for FairPoint Communications, the related PeeringDB entry shows it is owned by Consolidated Communications. (FairPoint was acquired by Consolidated in 2016.)

The @outagedetect Twitter account, associated with Internet connection and connected device monitoring company Fing, posts numerous alerts each day about localized Internet outages. On June 19, it posted the Tweet below, highlighting the end of a “considerable” hour-long Internet outage detected in Poland for users connected to network service provider Orange Polska.

One of the figures below shows that this “considerable” outage impacted CAIDA’s Active Probing measurements for Poland at a country level, as well as driving a concurrent decline in Darknet sources. The other figure shows the impact at a network level, with Active Probing measurements and Darknet sources for Orange Polska dropping considerably during the hour-long disruption.

Conclusion

While the major Internet disruptions observed in June were largely due to a familiar set of underlying causes, many of these causes remain avoidable.

Of course, convincing governments not to implement Internet outages during times of political unrest is a non-trivial effort, but organizations including the Internet Society and Access Now continue to advocate for user rights and drive #KeepItOn efforts.

In contrast, convincing network providers, especially Tier 1 global providers, to improve their routing hygiene through through participation in efforts like MANRS should be comparatively trivial.

Cable Complications

Greenland receives international Internet connectivity via the Greenland Connect submarine cable, linking the country to Iceland and Canada through landing points in Nuuk and Qaqortaq. In December 2018, the submarine cable link between these two cities suffered damage, disrupting Internet connectivity for users in Greenland. TELE-POST, a subsidiary of national telecommunications company TELE Greenland, has been tracking cable repair efforts over the last several months — efforts that have been slowed due to weather and ice cover.

Some repairs took place in early May, stabilizing Internet connectivity for users in North Greenland. However, disruptions to Internet connectivity in Greenland were also observed during this time frame, as seen in the figures below. While it has not been definitively confirmed that these disruptions were, in fact, due to cable repair efforts, TELE-POST has noted on the status site that connectivity for some customers may be turned on/off while repair efforts are underway.

A similar disruption was also observed on May 22, as shown in the figures below. That day, TELE-POST noted that it would be postponing repair work until August due to ice conditions. It is likely that similar disruptions will continue to occur over the coming months, as the update states: “However, in the coming period, until the important repair work next to Qaqortoq has been completed, there will be an increased risk of operational disturbances, particularly of the Internet service.”

On May 4, Globe Telecom subscribers in several regions of the Philippines experienced Internet service disruptions due to damaged fiber optic cables. A submarine cable was damaged by a 5.5-magnitude earthquake, while another cable was damaged by a vehicle dragging the cable at a road crossing. The figure below shows the impact of the cable damage to connectivity for AS4775 (Globe Telecom) – a minor change was seen in BGP, while successful active probing was impacted for just over three hours; measured darknet traffic dropped briefly at the start of the event, but appeared to recover quickly.

Apparently a popular day for Internet disruptions, Haiti also experienced an issue during the afternoon of May 4, as shown in the figure below. According to a number of Twitter responses from Digicel Haiti (@DigicelHT) to user complaints, the brief disruption was related to issues with a submarine cable.

The figure below shows that connectivity to Digicel Haiti through Digicel Jamaica was disrupted. The Fibralink submarine cable connects the two countries, so it may have been the cable referenced (but not named) in the @DigicelHT Tweets.

Just over two weeks later on May 19, AS15964 (Cameroon Telecommunications Network) experienced a connectivity disruption. As shown in the figure below, the number of traceroutes to endpoints in the network going through Tata dropped significantly for several hours, with Telxius and Orange picking up the slack.

This disruption impacted downstream networks in different ways. The figures below illustrate the impact of the disruption to two of those networks – AS30992 (MTN Cameroon) and AS37620 (VIETTEL CAMEROUN SARL). The number of successful traceroutes to endpoints in MTN declined by approximately half during the disruption period, while the number of traceroutes to endpoints in VIETTEL going through Cameroon Telecommunications Network unexpectedly increased during that period, although that shift also brought a significant increase in latency.

In a Tweet from @MTNCameroon that was subsequently deleted, the company acknowledged the apparent connectivity issues. The deleted Tweet was in response to a user complaint that included what appeared to be a screenshot of a text message that reads (via Google Translate) “Dear customer, following an incident on the optical fiber provided to us, our network is momentarily disrupted. MTN sexcuse for possible inconvenience.” This same (original) text was also Tweeted by several other users (1, 2) that same day. However, it is also worth noting that the text appears in Tweets from February 2019 and October/September/June/April 2018, so it may just be a stock customer support response from MTN Cameroon, rather than a definitive root cause.

On May 21 & 22, Malawi held elections to name a President, new members of Parliament, and local councilors. After the polls closed, Internet connectivity was disrupted across multiple providers in the country, reportedly due to damage to a cable owned by Malawi Telecommunications Limited. The figure below shows the impact to Internet connectivity for AS36969 (Malawi Telecommunications Limited) – successful active probing trended lower for several hours, and darknet activity was reduced as well.

The disruption also impacted local media outlets, as seen in the Facebook post below from Mibawa TV.

Satellite Suspicions

During May 2019, several Internet disruptions were observed that appear to be related to a reliance on satellite connectivity by major telecommunications providers in the impacted countries. In each case, the observed country-level disruption is coincident with a drop in the number of completed traceroutes reaching endpoints on the providers’ networks through satellite connectivity providers. In the figures below, the disruptions in South Sudan, Solomon Islands, Kiribati, and Cook Islands appear to be related to issues with O3b, while in Eritrea, connectivity through Asia Broadcast Satellite was problematic.

Wayward Weather

Damage caused by severe weather events frequently wreaks havoc on electrical and communications infrastructure. In early May, a storm in the U.S. Virgin Islands impacted Internet connectivity for users across multiple networks within the territory. Reduced levels of successful traceroute completions and active probing can be seen late in the day (UTC) on May 1 in the figures below.

These disruptions were also visible in network-level graphs for local network providers, as seen in the figures below. A Facebook thread highlighted issues with Broadband VI, and a Facebook Messenger response from a representative of Virgin Islands Next Generation Network noted “Our IT Team has indicated that there was a storm affecting the island of St. Croix on Wednesday, May 1. We were able to confirm with Broadband VI that they experienced a lightning strike. This appears to correspond with what you observed.”

The highly localized impact of extreme weather was highlighted by CAIDA’s IODA team towards the end of the month. In the Tweet embedded below, the team shared graphs of disruptions in active probing to endpoints in two Ohio counties on May 28, with the disruption timeframes correlating with reports of a tornado in those areas.

Other Outages

In addition to the disruptions discussed above that had frequently seen (probable) underlying causes, several additional disruptions were observed in May that were significant enough to mention here as well.

On May 5, the @BGPStream Twitter feed posted 70 “outage” alerts for autonomous systems (ASNs) associated with the United States Department of Defense, all at 01:51 UTC. BGPStream maintainer Andre Toonk explained what happened in the Tweet embedded below, highlighting the risk of single-homing multiple networks behind a single provider.

Just over a week later, on May 13 a significant Internet disruption was observed in China, apparently due to an issue at China Telecom. This event was explored in more detail in a previous post: “Multi-Hour Disruption at China Telecom”.

On May 22, published reports (The Straits Times, The Independent) indicated that thousands of fiber broadband subscribers of Singapore provider M1 (MobileOne) experienced a multi-hour Internet disruption, but M1 did not say what caused the disruption. Interestingly, it occurred a day after hundreds of customers of three local providers – Singtel, StarHub and M1 – had Internet connectivity disrupted by a fiber cut at Netlink Trust, which owns Singapore’s fiber network and resells fiber connectivity to these telecommunications providers.

Conclusion

Physical damage to terrestrial and submarine cables has long been a cause of Internet disruptions, as has problems with satellite communications for those regions underserved with wired connectivity. The continued growth in both submarine cable buildout and satellite Internet services should help provide redundancy and resiliency, reducing the frequency and severity of large-scale Internet disruptions. However, these additional avenues of connectivity will not change the likelihood that government-directed Internet disruptions will continue to occur (probably becoming more targeted and frequent), nor will they mitigate transient disruptions impacting local areas or individual network providers.