The conclusion of last month’s post noted “There was a noticeable absence of government-directed Internet disruptions in April. That is not to say that there were none, but those that did occur were not significant enough to be observed through publicly available tools.” This trend, which had also been observed over the prior few months, continued into May. (Unfortunately, this is not the case for June, but that will be covered in next month’s post.)

In May, a power outage in Venezuela once again disrupted Internet connectivity across the country, and a cyclone impacted connectivity in Bhutan. Fiber and submarine cable cuts (and repairs) caused Internet disruptions across multiple countries, as did unspecified network issues.

Power Outage

On May 5, a Tweet from Internet monitoring firm Netblocks confirmed that a power outage in Venezuela had disrupted Internet connectivity across multiple states within the country. Unfortunately, such power outages are a relatively common occurrence in Venezuela, and frequently impact Internet connectivity across the country. The Oracle Internet Intelligence and CAIDA IODA country-level graphs below show that the primary impact was to the active probing and traffic metrics, but the BGP metric also saw a brief nominal drop. The aggregated state-level CAIDA graph below illustrates the impact on active probing measurements to endpoints within the listed Venezuelan states – a multi-hour Internet disruption was measured in each of them.

Weather

Internet connectivity was disrupted in Bhutan on May 20 & 21 because of damage to Bhutan Telecom infrastructure caused by Cyclone Amphan. As shown in both the Oracle and CAIDA graphs below, the disruption began around 12:00 GMT on May 20 and lasted for approximately 24 hours, though a brief restoration of connectivity occurred between 17:40-19:40 GMT. The infrastructure damage did not cause a complete country-wide outage, but the graphs show that all three measured metrics were impacted.

The impact was more significant, however, at a network level. The CAIDA graphs below for AS18024 (Bhutan Telecom) and AS17660 (DrukNet/Bhutan Telecom) show complete outages for all three measured metrics.

Fiber/Cable Cut

Fiber optic cables cut by vandals disrupted Internet connectivity for tens of thousands of users in France on May 5 & 6. The cables that were damaged impacted connectivity for both home and business subscribers, as well as cloud provider Scaleway. The Oracle graphs below show the impact on traceroute measurements to endpoints within three French network providers. In each case, traceroutes through the network’s primary upstream provider drop to near zero, but begin reaching the target network through an alternate upstream almost immediately. Nominal increases in latency are also evident, but for these network providers, redundant connectivity meant that the overall impact was nominal.

According to a Twitter thread from Venezuelan journalist Fran Monroy Moret, repairs to a submarine cable resulted in a multi-hour Internet disruption across multiple network providers in Venezuela on May 17. Multiple submarine cables land in Venezuela, but the thread does not mention which cable was being repaired. However, references to CenturyLink within the thread could indicate that the cable in question is either the Americas-II or the South American Crossing (SAC), as CenturyLink is listed among the owners of both.

The Twitter thread suggested that the disruption would last eight hours, from 02:00 – 10:00 local time (06:00 – 14:00 GMT). At a country level, the eight hour-disruption is most evident in the BGP metric across both the Oracle and CAIDA graphs, as the impact to the active probing and traffic-related metrics appeared to be briefer in duration. Local network provider Movistar (AS6306) also experienced a disruption for the full repair duration, as shown in the graphs below. However, local providers Inter (AS21826) and Net Uno (AS11562) appeared to experience briefer disruptions, occurring in the middle of the repair period and primarily evident in the active probing and BGP metrics.

A May 19 Tweet from Oracle Internet Intelligence highlighted Internet disruptions detected across multiple countries, and an associated Tweet pointed to the ACE submarine cable as the likely culprit. Impacted countries included Sierra Leone, Niger, Mauritania, Liberia, and the Gambia. The graphs below show that the disruptions began around 01:30 GMT, lasting approximately 90 minutes, although the impacts linger for several additional hours in Niger’s graphs.

On May 27, a brief Internet disruption was detected on French Polynesia starting just after 12:00 GMT and lasting for about an hour, as shown in the graphs below. The active probing, BGP, and traffic metrics were impacted across both Oracle and CAIDA measurements. The Honotua submarine cable is the island nation’s primary international Internet connection, connecting it to Hawaii, and it is likely that problems with this cable were the cause of the observed disruption. The network-level graphs below all show AS36149 (Hawaiian Telecom Services Company) as an upstream provider, and a drop was seen in the number of successful traceroutes across this network in all three graphs.

Network Issues

According a May 7 post to its Facebook page, Greenland network provider TELE-POST suffered an unspecified network issue that disrupted Internet connectivity for DSL users.

Visible in the graphs below, it appears that the disruption began around 06:30 GMT, and lasted for just over six hours. However, after the issue was addressed and connectivity restored, TELE-POST customers continued to have issues with Wi-Fi connectivity, leading to a series of additional Facebook posts (1, 2, 3) that provided situational updates and guidance on resetting modems.

The French territory of St. Pierre and Miquelon, located south of the Canadian island of Newfoundland, experienced a multi-hour Internet disruption on May 10. The graphs below show that the disruption began around 13:00 GMT, with connectivity restored about five hours later. Both the Oracle and CAIDA graphs show a significant decline in the active probing metrics, and the Oracle graph shows a similar decline in the BGP metric.

We're aware that customers in Harbour Breton and parts of Burin Peninsula are currently experiencing an interruption in service. We're investigating the cause and working to restore service as quickly as possible. We appreciate your patience and will update later this afternoon.

— Eastlink Support (@EastlinkSupport) May 10, 2020

The Tweet shown above from the support account of Canadian Internet provider Eastlink notes that the network was experiencing a service interruption. This interruption is evident in the Oracle graph below for AS11260 (Eastlink), and also in the graph below for AS3695 (SPM Telecom), which uses Eastlink as its primary upstream provider.

Thousands of TalkTalk users in the United Kingdom experienced an Internet disruption for a couple of hours during the morning of May 29. While the disruption was covered in articles published on the Web sites of the Independent and the Daily Mail, neither included any information on the root cause of the issue. TalkTalk acknowledged the problem in Tweeted replies to customer complaints, but did not provide any information on what caused the disruption.

The graphs below show that the disruption began approximately 09:00 GMT and lasted for approximately 90 minutes, which is consistent with the timeline in the published reports. The active probing metric in the CAIDA graphs shows an obvious decline, while the Darknet metric remained largely unaffected, as did the BGP metric.

Conclusion

Over the last several month, COVID-19 lockdowns and social distancing have driven nearly everyone onto the Internet for school, work, shopping, and entertainment. Given this increased reliance on the Internet for everyday life, it is imperative that network providers do a better job of communicating about Internet shutdowns, outages, and other disruptions that impact their networks. Ideally, this communication would be proactive and public, posted on status pages/sites (and archived for future reference), as well as to associated Twitter accounts and Facebook pages. (And by acknowledging issues in a timely fashion, and communicating about them openly, providers can control the narrative around such events, preventing rumors from spreading, and even potentially buying themselves a little customer goodwill.)

At the very least, providers need to respond in a timely fashion to inquiries about Internet disruptions observed on their networks made via social media and/or e-mail. In this day and age, given the importance of Internet connectivity, ignoring these inquiries is simply unacceptable.

While this blog has never claimed to be exhaustive in its coverage of Internet disruptions, it has endeavored to catalog the various causes of those disruptions, and most months see quite a few documented causes. Interestingly, March only saw documented disruptions due to power outages and cable/fiber issues, with a couple of additional due to possible network issues. (There were, of course, a number of other observed disruptions, but root causes were unable to be identified or confirmed through research or social media outreach.)

Power Outages

Unfortunately, the instability of Venezuela’s electrical power infrastructure has been the cause of Internet disruptions within the country multiple times in the past. During March, NetBlocks Tweeted about seven distinct Internet disruptions due to power failure in Venezuela – these occurred on March 1st, 3rd, 5th, 8th, 12th, 20th, and 26th. The four graphs below illustrate CAIDA IODA Active Probing results across 15 Venezuelan states across the month (one graph per week). For the disruptions visible on the dates highlighted by Netblocks, Táchira was one of the states with the most significant impact, as was Merida. It is interesting to note that similar Internet disruptions are also visible on March 7th, 9th, 13th, and 28th, although it is not clear whether these were also due to power outages in the affected states – Táchira appears to be impacted by these as well.

On March 22, a power outage also disrupted Internet connectivity on the Solomon Islands. Our Telekom (Solomon Telekom Company Limited), a local telecommunications service provider, posted on its Facebook page that its main exchange had experienced an outage due to a power failure. This outage is clearly visible in both the country- and network-level graphs below, starting around 0620 GMT, and ending around 1400 GMT.

Cable/Fiber Issues

On March 6, Oracle Internet Intelligence Tweeted that a new fault in a segment of the ACE submarine cable was impacting connectivity in Mauritania. The disruption is clearly visible in both the Oracle Internet Intelligence and CAIDA graphs below, starting mid-afternoon GMT, with partial recovery early the following day.

Although unconfirmed, the disruption visible in both graphs several days prior may also be related to problems with the ACE cable. On March 2, the CAIDA graph shows that all three measured metrics dropped shortly before noon GMT, with a gradual recovery taking place through the rest of the day, presumably as traffic shifted to alternate paths. In the Oracle graph, the active probing and BGP graphs showed concurrent drops as expected, but the DNS Query Rate graph spiked, indicating that whatever outbound traffic that actually made it out of Mauritania likely included requests by recursive resolvers driven by constant retries (due to lack of response) from DNS clients.

On March 9, Madagascar suffered an Internet disruption reportedly caused by roadwork severing a fiber optic cable. According to a published report (via Google Translate),

“At the end of yesterday afternoon, Internet users had the unpleasant surprise of not being able to access the Internet. It is a total cut causing discomfort to users, especially those who work on the Internet and use the Telma network. It was only in the early evening that everything returned to normal, much to everyone’s relief. According to a source with Telma, the cut comes from road works carried out in the southern part of the country and having affected the optical fiber. It was thus necessary to carry out a back up, either a reconnection or failover, on Lion cable to satisfy the users. It is a common practice between operators in case one of the companies is in a delicate situation as was Telma yesterday. Contrary to the claims of some science fiction enthusiasts, it is indeed a technical failure and not another reason.”

The mid-afternoon disruption is visible at a country level in the figures below, impacting both the active probing and BGP metrics. As the quote above notes, TELMA (Telecom Malagasy) was also impacted – the graphs below illustrate that the active probing metrics declined in both Oracle’s and CAIDA’s measurements, while Oracle’s measurements also show a sharp increase in latency during the period of disruption.

On March 10, Gabon Telecom alerted its users via Twitter (see below) and Facebook that a cut to the SAT-3 submarine cable disrupted Internet traffic to its fixed network the day before. The post also references the ACE submarine cable cut in Mauritania discussed above, and appears to claim that the repairs to the ACE cable, which also connects to Gabon, will help [restore] Gabon Telecom’s traffic.

COMMUNIQUÉ pic.twitter.com/XAleUwmGfK

— Gabon Telecom (@gt_officiel) March 10, 2020

After posting a status update on March 11, Gabon Telecom confirmed the restoration of service on March 12. The figures below show that the associated Internet disruption lasted from March 9 -12 at both a country and network level. At a country level, the impact of the cable problems appears to be more severe in the CAIDA measurements than in Oracle’s, likely owing to the difference in the respective measurement vantage points. At a network level, the initial disruption is visible as a brief outage in the Oracle Internet Intelligence graph, although traffic from Gabon Telecom quickly began transiting a new upstream provider, but with increased latency. The graph shows that the latency returned to normal after the cable was repaired, although the upstream provider remained the same. On March 12, Gabon Telecom also notified customers that it would be granting up to 5 GB of additional mobile data to those impacted.

On March 12, Oracle Internet Intelligence Tweeted that a historic “dragon storm” was impacting Internet connectivity in Egypt. The graphs below show nominal disruptions to the active probing metrics, becoming most evident around mid-day GMT. Interestingly, the traffic-related metrics in both graphs also saw significant declines, with the diurnal patterns effectively flattening out. Telecom Egypt explained in a LinkedIn post on March 18 that “The service was interrupted in few segments located in Egypt as a result of terrestrial cable cuts resulting from the unusually powerful storm, recognized as the fiercest to hit the country in over 25 years, and accompanied by heavy rainfall, destructive winds, and dust storms, causing large-scale power disruptions in multiple areas in Egypt.”

A cable issue on March 14 impacted Internet connectivity across more than a dozen countries. Initially reported by Oracle Internet Intelligence as a submarine cable cut, a subsequent Tweet clarified that the subsea cable outage was caused by a cut in terrestrial fiber optic in Egypt. According to the Tweet below, the problem was with the EASSy cable system. (A submarine cable industry observer explains that Egypt is a critical transit cross road for international East-West cables, as well as terrestrial and submarine diversified backhaul links beween Suez Canal North (Alexandria / Port Saïd) and South (Suez) landing stations.)

EASSy cable system is down due to a terrestrial fault in Egypt between Abu Talat and Zaafarana on the Djibouti to Marseilles segment. We are waiting for confirmation as to where the SEACOM break occurred and whether this is in the same location.

— SA NREN Operational Updates & Alerts (@RENAlerts) March 14, 2020

The graphs below show the impact of the issue at a country level in Burundi, Djibouti, Tanzania, Seychelles, Sudan, and the United Arab Emirates. In the United Arab Emirates, the disruption is barely evident in the active probing and BGP metrics, but it is significantly more visible across the other countries.

The Oracle Internet Intelligence graphs below show the impact of the disruption at a network level for providers in Bangladesh (AS132602), India (AS55410), Kenya (AS33771), Mozambique (AS37342), Saudi Arabia (AS47794), and Thailand (AS38040). Measurements to these networks saw a brief period of increased latency when the disruption occurred, though as the graphs show, measurements to these networks generally shifted to go through alternative upstream providers when one was impacted.

Possible Network Issues

Early in the morning (GMT) of March 23, a brief but significant Internet disruption was observed in Algeria. The graphs below show that Oracle Internet Intelligence recorded a partial decline in both the active probing and BGP metrics, while CAIDA IODA saw a significant drop in the active probing metrics, but no change to the BGP metric.

The disruption was also clearly evident in measurements of Telecom Algeria, as seen in the graphs below. Oracle Internet Intelligence testing measured a drop in the number completed traceroutes to targets within the network across multiple upstream providers, while the CAIDA IODA measurements saw a similar sharp decline in successful active probing.

However, there was apparently some question about the cause of the disruption, as Telecom Algeria posted a “disclaimer” to its Facebook page on March 24 that claimed (via Google Translate) “Contrary to what has been announced and reported by some media and on social networks about of a supposed shutdown of the Internet at the level national. Algeria Telecom wants to deny and reassure his kind clientele that no break is programmed.” It isn’t clear if this disclaimer refers to the disruption observed on the previous day, as the translation appear to refer to a future planned Internet shutdown, despite being posted the day after the disruption. (Checking against other online tools also results in similar translations.)

Closing out March, a brief Internet disruption was observed in Grenada on the 31st. The graphs below show that the disruption was clearly visible at both a country and network level (Columbus Communications Grenada a/k/a FLOW Grenada).

In response to an inquiry over Facebook Messenger, FLOW Grenada explained “There was an unplanned outage that had affected our Grenada customers on March 31st. Our team had managed to have services working in less than 24 hours. We sincerely apologize for the inconvenience caused.” The provider also compensated customers with free data as an apology for the outage.

We know many of you were without service for a short period today, we are giving 1GB of data free to say thank you for your support and understanding. pic.twitter.com/AIkOSXfJfM

— Flow Grenada (@FLOWGrenada) March 31, 2020

Conclusion

As traffic volumes grew and traffic patterns shifted during March due to increased usage, many worried that the Internet would “break” and that Internet disruptions would become more significant and more frequent. As we came out of the month, it was clear that, as predicted, Internet resilience remained firmly in place. Core Internet infrastructure providers reported that while they saw increased traffic on their networks, they could easily handle it. Internet Exchange Points (IXPs) and Content Delivery Networks (CDNs) also reported record traffic growth, but they helped keep that traffic local. Well-provisioned last mile broadband networks reported increased traffic and shifting peak times, but neither caused disruptions.

However, the Internet has strained to adapt in underserved and unserved areas, where insufficient investment has been made to make high-speed broadband available and affordable. Users on these networks perceive that Internet disruptions have occurred more frequently as their connections are unable to cope with the demands of videoconferencing, streaming media, and online learning, often all concurrently competing for bandwidth. The depth of the “digital divide” became much clearer around the world in March – policy changes and directed investment are needed to deliver an Internet that is resilient for all users.

Note: Previous posts have frequently differentiated between the “Traceroute Completion Ratio” metric used by Oracle Internet Intelligence and the “Active Probing” metric used by CAIDA IODA. As traceroute-based measurement is a form of active probing, posts will generally refer to both metrics as “active probing” going forward. However, references to specific metrics in a specific graph will continue to use the appropriate label.

Earthquake

On January 7, a 6.4 magnitude earthquake struck Puerto Rico, disrupting both fixed and wireless Internet connectivity in the territory. The impact of the earthquake at a country level can be seen in the Oracle Internet Intelligence, CAIDA IODA, and Google Transparency Report graphs below. While the BGP metric remains effectively unchanged, the active probing metrics show a nominal drop at the time the earthquake occurs, with a gradual recovery over the next several days. The traffic-related metrics exhibit a more significant drop, with a similar recovery pattern. These observations are consistent with what is frequently seen in these scenarios, as earthquakes often cause widespread power outages. Because key infrastructure, including the routers making BGP announcements, is likely to be in data centers with backup power sources, the BGP metrics will see little change while power failures take active probing targets and end-user systems offline.

The figures below illustrate the impact of the earthquake on three local fixed connectivity providers. The impact to active probing measurements appears to be similar across both the Oracle Internet Intelligence and CAIDA IODA graphs. However, it is interesting to note that the CAIDA Darknet metric shows a severe drop at the time the earthquake occurs, reflecting a significant loss of connectivity to those systems connecting to the darknet address space.

The figures below illustrate the impact of the earthquake on three local wireless connectivity providers. As would be expected, the impacts observed through these external measurements appear to be less notable than the impacts to the fixed providers. Although many mobile customers may have lost connectivity, external measurement into wireless networks is challenging due to the network architecture, leading to a more limited perspective.

Power Outages

On January 18, Internet monitoring firm Netblocks Tweeted about a significant power outage impacting multiple states in Venezuela. The CAIDA IODA figures below show an Internet disruption occurring just after midnight GMT – given the time difference, this is roughly in alignment with Netblocks’ observations. The figures also show additional disruptions occurring mid-day (GMT) on January 19, as well as another one early on January 20. It isn’t known whether these additional two issues are also related to power outages, although it is likely.

Just a few days later, on January 22, Netblocks Tweeted about another major power that knocked out Internet connectivity across multiple states in Venezuela. In the Oracle Internet Intelligence and CAIDA IODA graphs below, the disruption is visible as a brief drop in the active probing metric around 00:00 GMT on January 23 (again, due to the time difference).

The CAIDA IODA figures below show the impact observed for five of the affected states. While the disruption is visible across all five, it appears that connectivity dropped most significantly in Mérida, Táchira, and Zulia.

Network Maintenance

Mid-afternoon (GMT) on January 8, a brief disruption to Internet connectivity occurred on the South Pacific island nation of Vanuatu. As seen in the Oracle Internet Intelligence and CAIDA IODA graphs below, the disruption impacted both the active probing and BGP metrics, starting around 13:00 GMT, lasting around 45 minutes.

Digging in further, the problem appeared to be related to a disruption at Telecom Vanuatu. According to an e-mail from a contact at the provider, the disruption was related to a planned maintenance activity on their uplink interface.

Late on January 27 into early January 28 (GMT), a significant Internet disruption was detected in Samoa. As shown in the country-level figures below, Oracle’s active probing metric dropped to about half its previous level, while CAIDA’s dropped to near zero. The BGP metric across both Oracle Internet Intelligence and CAIDA IODA measurements saw multiple oscillations during the period of disruption.

As noted in the Facebook update from Samoa Bluesky (SamoaTel) above, they experienced an issue with their “gateway”, impacting service availability. The graphs below show the impact of this issue on on Samoa Bluesky’s network. The Oracle Internet Intelligence graphs show that latency to the network spiked during the disruption, indicating that the few measurements that did complete successfully did so over a much slower path.

Government Directed

Iran is no stranger to government-directed Internet shutdowns, and in mid-January, two more occurred in response to protests for those killed on Flight PS752, the Ukraine International Airlines plane that was reportedly shot down on January 8 by Iran’s Islamic Revolutionary Guard Corps.

A Tweet from Netblocks on January 13 highlighted an Internet disruption detected at Sharif University of Technology in Tehran, starting just before noon UTC, and noted that students were protesting at the university for colleagues and alumni killed on the flight. The disruption can be seen in the CAIDA IODA graph below, with both the Active Probing and Darknet metrics dropping significantly around noon. The Active Probing metric recovered after several hours, while the Darknet metric recovered to continue what was likely a diurnal decline.

A day later, another Internet disruption was observed in Iran by Netblocks and by CAIDA IODA, beginning just before 14:00 GMT. In the country-level graphs below, the disruption is most evident in the active probing metrics from both Oracle Internet Intelligence and CAIDA IODA, and is clear in CAIDA’s Darknet metric as well.

Similar impacts can also be seen in the three CAIDA network-level graphs below, with sharp but brief declines seen in the Active Probing and Darknet metrics across three leading Iranian telecommunications firms.

Cable Cuts

January saw several major cable cuts that had significant and wide ranging impacts.

The first was observed in Yemen on January 9. In the figures below, a significant disruption can be seen in Oracle’s and CAIDA’s active probing metrics and in Google Web search traffic for Yemen around 18:00 GMT, but were preceded by a gradual decline beginning around 09:00 GMT. Initial indication that the disruption was due to a cable cut reportedly came from an SMS message from Yemen’s national Internet service provider to its users.

Oracle Internet Intelligence Tweeted confirmation of the disruption, showing the impact to YemenNet, and the provider posted the update shown below to its Facebook page, noting that the disruption was caused by a problem with the FALCON submarine cable, owned by Global Cloud Xchange. (English translation of the post via Facebook.)

More than 80 % of international internet hours in Yemen have been released as a result of an international cable outage outside Yemen

Sana ‘ a January 2020, 9 (Spa)-General announced that more than 80 % of international internet hours in Yemen were out of service as a result of an international cable outage outside Yemen.

The responsible source of the company and company of the Yemeni news agency / spa explained that a large number of internet links were out of service due to the exposure of the sea cable falcon in Suez, causing more than 80 percent of internet hours International in Yemen.

The source explained that there are ongoing efforts to follow up on cable repair and return of service soon.

The figures below show the impact on two of YemenNet’s autonomous systems, with the disruption affecting connectivity to AS30873 significantly more than AS12486. The Oracle Internet Intelligence graphs show that when the cable cut occurred and upstream connectivity to AS15412 (Global Cloud Xchange) was lost, YemenNet was able to quickly fail over to backup/alternate upstream providers

The damage to the FALCON cable also impacted networks in a number of additional countries, including Comoros, Ethiopia, Kuwait, Tanzania, Saudi Arabia, and Sudan. The cable cut is evident in the CAIDA graphs below as a brief drop in the active probing metric. However, the impact is more pronounced in the Oracle Internet Intelligence graphs, where it resulted in increased latency concurrent with traffic shifting to alternate providers.

On January 12, Global Cloud Xchange posted an update confirming the damage to the FALCON cable, and noting that the FLAG Europe-Asia (FEA) cable had also been damaged. It stated that “Initial findings indicate that probable cause was an anchor drag by a large merchant vessel in the immediate area.” Just over a month later, Global Cloud Xchange posted another update, announcing the that FALCON submarine cable had been repaired in “record time”, with connectivity restored by February 13. It expected repairs to the FEA cable to be completed by February 22.

Venezuelan telecommunications provider CANTV Tweeted on January 13 that it was working on restoring telephone and Internet services in the Anzoátegui, Bolívar, Monagas, Sucre and Delta Amacuro states. The CAIDA IODA figures below show the observed impact of the disruption to Internet connectivity for those states, with problems most visible in Bolívar, Monagas, and Delta Amacuro. While the Active Probing metric showed a decline, as would be expected, it is interesting to note that the BGP metric was also impacted, possibly indicating that connectivity to key local routing infrastructure was interrupted.

#13Ene Equipo técnico de #Cantv Región Oriente y Guayana realizaron las maniobras necesarias, para restituir servicios de telefonía fija e Internet en los estados Anzoátegui, Bolívar, Monagas, Sucre y Delta Amacuro. pic.twitter.com/PZ51s0jgiO

— SaladePrensaCantv (@salaprensaCantv) January 13, 2020

On January 16, Oracle Internet Intelligence Tweeted about a submarine cable outage that happened earlier that morning which severely degraded Internet connectivity for multiple countries along the west coast of Africa. Published reports (including ITWeb, SubTel Forum, and TechCentral) indicated that both the SAT-3/WASC and the West African Cable System (WACS) experienced breaks. The SAT-3/WASC break was reported to be in the vicinity of Libreville, Gabon, while the WACS break was reported to be in the vicinity of Luanda, Angola.

The damage to these submarine cables reportedly disrupted Internet connectivity to a number of African countries, including Angola, Cameroon, Congo, Democratic Republic of Congo, Gabon, the Gambia, Ghana, Ivory Coast, Mali, Mauritania, Namibia, Niger, Nigeria, South Africa, Zambia, and Zimbabwe. The figures below illustrate the impact as measured by Oracle Internet Intelligence and CAIDA IODA, and as seen through Google Web Search traffic from the affected countries.

In reviewing the figures above, the disruption caused by the cable breaks was most evident in the graphs for Congo, the Democratic Republic of Congo, and Namibia, with lesser impact evident in Angola, Cameroon, Gabon, Ghana, and Ivory Coast.

As expected, the Internet disruption was heavily covered on news Web sites in the impacted countries, and affected telecom providers including Telecom Namibia, MTN Namibia, CAMTEL, Cool Ideas, and RSAWeb provided status updates to subscribers via social media or their corporate Web sites.

The WACS cable was reportedly damaged by “heavy sediment”, and connectivity was fully restored on both the WACS and SAT-3/WASC cable systems by February 19, according to a published report.

On January 23, Togo Telecom posted the Tweet below alerting customers that an Internet disruption that started the day before was due to a problem with the WACS cable.

Togocom vous présente ses excuses pour ce désagrément indépendant de sa volonté et vous remercie pour votre patience et votre confiance. pic.twitter.com/sfwOzNN7j8

— TOGO TELECOM (@TOGOTELECOM1) January 23, 2020

Dear clients,

Since Wednesday January 22, 2020. Togo, like all the sub-region, whose internet connection depends on the West African submarine cable (WACS- West African Cable System) faces a partial or total absence of internet service This follows to a double section on the submarine cable segments as well as on the terrestrial backup segment.

The Togocom teams as well as the international teams of the cable operator are already hard at work for the rapid resolution of this incident. All the means are implemented by all the stakeholders for an effective and rapid resolution of the problem. ,

Togocom will keep you informed in real time of the progress of the work to restore complete internet service.

Togocom apologizes for this inconvenience beyond its control and thanks you for your patience and your confidence.

(via Google Translate)

The figures below illustrate the impact of this problem on Internet connectivity at a country level in Togo, as well as to connectivity to Togo Telecom.

There was some distrust of Togo Telecom’s claim of a technical problem, given Togo’s history of Internet shutdowns and the fact that this issue occurred exactly a month before an election. However, the Oracle and CAIDA figures below show that an Internet disruption of extremely similar scope and duration was observed at the same time in Cape Verde, with a disruption in Cameroon also starting at the same time, though it was less significant than the ones in Togo and Cape Verde. Because Cape Verde and Cameroon are also connected to the WACS submarine cable, this would seem to support the claim that there was indeed a problem with the cable.

Closing out the month, a brief Internet disruption was observed in Yemen. lasting for several hours during the late morning (GMT) of January 30. The figures below show the impact at a country level, as well as at a network level for AS30873 (YemenNet). Both the Oracle Internet Intelligence and CAIDA IODA graphs show a brief drop in the BGP metric. However, it is interesting to note that while the Oracle graphs show a corresponding drop for active probing, that metric does not decline in the CAIDA measurements. Oddly, it actually appears to improve during the period of disruption, which is unexpected.

According to a published report, the disruption was reportedly caused by a Saudi-led coalition mercenary militia in Aden province storming the international link station and cutting off access to the international Aden-Djibouti cable.

Conclusion

#Kashmir has lost some Rs 1.78 lakh crore ($25 billion) in economic output in the five months since August 2019. This is 11% of the erstwhile state’s nominal gross state domestic product (GSDP) of Rs 1.59 lakh crore for 2018-19. @qzindia https://t.co/75kHirozkl

— InternetShutdowns.in (@NetShutdowns) February 21, 2020

The Internet disruptions covered in this month’s report were all generally short-lived, likely resulting in limited economic impact. However, the Internet shutdown in the Indian states of Jammu and Kashmir started in August 2019, and had an estimated impact of USD $25 billion in the five months after in began. However, there are positive signs, as in January, the Indian Supreme court said that an indefinite shutdown of the Internet in Kashmir was illegal. Limited Internet service was restored to the regions in the middle of the month, including broadband in hotels, travel establishments, and hospitals, but with access restricted to white-listed Web sites. 2G mobile connectivity also returned, but was limited to white-listed Web sites as well.

This final Internet Disruption Report post for 2019 is a long one, covering disruptions caused by a DDoS attack, power outages, cable/fiber/network issues, and government direction. Some countries make multiple appearances in this month’s report, and some have been featured in multiple reports throughout the year. In addition to observed disruptions, we also review Russia’s reported Internet disconnection test, as well as a few additional related observations.

DDoS Attack

Three months after a DDoS attack in September caused an Internet disruption at South African provider Cool Ideas, another DDoS attack targeting the provider again impacted connectivity. As shown in the Fing Internet Alerts and CAIDA IODA figures below, the attack appears to have occurred in several waves, with the first starting around 0815 GMT, lasting for approximately two hours, and the second starting around 1200 GMT, lasting for just over an hour.

Recognizing the problems that these attacks create for local providers, a published report highlighted how South Africa plans to fight similar DDoS attacks in the future:

South Africa’s Internet Service Providers’ Association (ISPA) has said that in an attempt to mitigate the severity of DDoS attacks against local ISPs, administrators at South Africa’s Internet exchanges are creating a “blackhole” that will funnel identified DDoS traffic through the exchanges into oblivion.

Power Outages

On December 4, NetBlocks posted a Tweet confirming that the third major power outage in Venezuela in just two days had disrupted Internet connectivity across multiple states. These disruptions are visible at a country level in the figures below, apparently occurring around 1830 GMT on December 3, 0645 GMT on December 4, and 1800 GMT on December 4.

A Tweet from @CorpoElecInfo (the Twitter account of Venezuela’s national electric company) that afternoon referenced system recovery efforts, presumably associated with the power outage.

NetBlocks listed fifteen Venezuelan state where Internet connectivity was disrupted by the power outage, and the figures below show the impact on three of those states. In Tachira and Zulia, the 0645 and 1800 GMT disruptions are clearly visible. However, in Aragua, there is a significant disruption evident starting around 0300 GMT that lasted over six hours – this could potentially be related to a more severe, localized power outage. The 1800 GMT disruption is visible for Aragua as well, although the impact is lower than was seen for Tachira and Zulia.

CAIDA IODA also observed Internet disruptions in Venezuelan states including Aragua, Cojedes, and Guárico on December 29. However, there was no publicly available information associating these disruptions with a power outage event. (But they are highlighted here because of the frequency of power outage-caused Internet disruptions in Venezuela.)

Just two days before Christmas, a nationwide power outage struck the island nation of Malta, reportedly caused by a fault on the interconnector cable between Malta and Sicily. As illustrated in the Oracle and CAIDA IODA figures below, the Internet disruption caused by this power outage started around 0600 GMT, with connectivity gradually returning to normal after noon GMT. The Google Transparency Report figure below shows the impact that the disruption had on traffic from Malta to Google’s GMail application, with a significant decline seen between 0600-0630 GMT, and a clear recovery visible six hours later.

Melita, a major telecommunications provider in Malta, provided status updates to its customers via its Facebook page. The figures below show the power outage’s impact to Melita’s network, with the Fing Internet Alerts graph showing a complete loss of communication with local probes. The Oracle and CAIDA IODA graphs show a significant decline in the active probing metrics, and the CAIDA IODA graph shows the relative volume of Darknet source IPs in that network dropping to near zero, in line with Fing’s observation of an end-user connectivity outage.

Cable/Fiber Cuts & Network Issues

Late in the day (GMT) on November 30, Dhiraagu, the largest network service provider in the Maldives, posted an update to its Facebook page alerting customers to “unexpected technical issues” on its fixed broadband network. The impact of these issues lasted into December 1, and were visible at both a country and network level. The Oracle and CAIDA IODA figures below show that active probing measurements to endpoints in the country declined just before midnight (GMT), and started to recover approximately eight hours later.

The figures below illustrate the impact on one of Dhiraagu’s autonomous systems. The Oracle graph shows that latency for traceroutes to endpoints within the network increased significantly during the period of disruption. Although Dhiraagu posted a second Facebook update a little more that five hours after the first one alerting subscribers that the issue had been fully resolved, it is clear that connectivity took several additional hours to return to “normal” levels.

On December 1, NetBlocks reported that planned maintenance to a submarine cable disrupted Internet connectivity for multiple network providers in Venezuela. Several of these providers had alerted subscribers via Twitter that this maintenance would be taking place, apologizing in advance for the inconvenience that the lack of Internet connectivity would cause.

El cable submarino privado principal que conecta nuestro país con el resto del mundo, tendrá un mantenimiento programado el día de mañana, 01 de diciembre, desde las 8:00 a.m. hasta las 3:00 p.m.

— InterCliente (@InterCliente) November 30, 2019

#ATENCIÓN
Mañana 1 de diciembre, de 8:00 am a 3:00 pm, nuestro proveedor de acceso a los servicios de internet estará realizando un mantenimiento programado para reparar una falla del cable submarino

— Movistar Venezuela (@MovistarVe) November 30, 2019

The Internet disruption caused by the submarine cable maintenance is clearly visible at a country level, as seen in the figures below. Both Oracle and CAIDA IODA observed notable declines in the Traceroute Completion/Active Probing and BGP metrics, but it is interesting to note that there was no apparent impact to the traffic-derived metrics, which may indicate that traffic took advantage of alternative paths to exit the country. And although the Tweets from the network providers indicated that the maintenance window was expected to last from 0800 to 1500 local time (1200 to 1900 GMT), it appears that it took longer than expected, with connectivity being restored closer to 1800 local time (2200 GMT).

The figures below show the impact of the cable maintenance at a network level, illustrating the Internet disruptions experienced by four Venezuelan Internet service providers: NetLink América, Inter/Corporación Telemic, Net Uno, and Movistar/Telefónica. The Oracle graphs show that during the maintenance window, traceroutes to targets in these networks via AS3549 (Level 3) failed to complete. In some cases (Inter, Net Uno), a smaller percentage of them transited AS52320 (GlobeNet), while for Movistar, other networks picked up the slack.

Although the cable that was being repaired wasn’t specifically named, clues in the CenturyLink memo in the Tweet shown below would seem to indicate that it was the South American Crossing (SAC) cable system. This particular cable is one of two that lands in Panama and St. Croix, as well as Venezuela, but is the only one that has a Panama to St. Croix segment, as referenced in the memo. In addition, CenturyLink is one of the owners of the SAC cable system.

Debido a reparaciones en el cable submarino Panama-St Croix, Proveedores de Servicio de Internet como @InterCliente y @MovistarVe han informado que mañana #1dic será interrumpido el servicio desde las 8:00am hasta las 3:00pm pic.twitter.com/TDIi8mu4ty

— RedesAyuda (@RedesAyuda) December 1, 2019

On December 8, concurrent Internet disruptions were observed in Oracle and CAIDA IODA graphs for Sierra Leone, Guinea, Liberia, and Benin from 1725 to 1825 GMT. In November, a similar event occurred, and we noted in that month’s post, “Because the Africa Coast to Europe (ACE) submarine cable lands in each of these countries, it was suspected that problems with the cable caused these observed disruptions.” That suspicion was confirmed by @acesubmarinec, but we received no response to outreach regarding this December disruption or a followup inquiry. However, given that all of the countries shown below are connected to the ACE cable, it is likely the culprit in this disruption as well.

On December 10, Fing Internet Alert Tweeted about a significant Internet disruption experienced in multiple cities by customers of U.S. network provider Suddenlink.

This is #suddenlink situation today.#suddenlinkdown #suddenlinkhelp pic.twitter.com/YslH9SHvKb

— Fing Internet Alert (@outagedetect) December 10, 2019

The figures below indicate that the issues started around 0630 GMT, with significant impact for approximately three hours. The Active Probing metric in the CAIDA IODA graph shows that the situation began to improve around 1000 GMT, but that the metric didn’t return to pre-disruption levels until around 1500 GMT. Although Suddenlink apologized to customers via Twitter, it did not respond to a request for more information about the cause of the disruption.

Making its second appearance in this month’s report, Malta also suffered an Internet disruption on December 12. However, this one was due to an outage on international connectivity for Go, the country’s leading telecommunications company. According to posts on the company’s Facebook page, the outage impacted both the company’s fixed and mobile Internet services.

The figures below show a country-level disruption starting around 1500 GMT across all metrics in both the Oracle and CAIDA IODA graphs, lasting until 1845 GMT. At a network level, the CAIDA IODA graph below shows all three metrics dropping to near zero during the period of disruption, while the Oracle graph appears to show that some traceroutes continued to successfully reach endpoints within the network. Malta is serviced by five international submarine cables, but Go did not specify which one suffered the outage. However, as Go is the owner of the GO-1 Mediterranean Cable System, and co-owner of the Italy-Malta cable, it is likely that the problem occurred on one of these.

On December 15, PTCL, Pakistan’s national telecommunications company, alerted customers of potential service degradation due to a fault in the Asia Africa Europe-1 (AAE-1) submarine cable.

Internet services are impacted & you may face some service degradation due to technical fault in the International Submarine Cable AAE-1 at Doha. PTCL in conjunction with the International Submarine Consortium, is working to fully restore internet services across the country. 1/2

— PTCL (@PTCLOfficial) December 15, 2019

The figures below show the impact of the cable issue on Oracle Internet Intelligence traceroutes to endpoints in two PTCL autonomous systems. In both instances, the number of successful traceroutes did not decline, but combined latency increased by 50-75ms. Based on the timing of a followup Tweet posted by @PTCLofficial, service was restored after approximately five-and-a-half hours.

After a government-directed nationwide Internet disruption in November, initial reactions to connectivity issues observed in Iran on December 19 reflected concerns that another such event was taking place. However, Sadjad Bonabi (@sadjadb), a member of the Board of Directors of Iranian infrastructure provider TIC, posted a series of Tweets [1, 2, 3] over several hours explaining that the disruption was due to fiber breaks near Bucharest and Munich, and that service providers such as Google were also impacted. An Iranian news site also noted that the disruption was “related to fiber optic lines from Europe.” A subsequent update from NetBlocks indicated that connectivity to Google services in Turkey and Bulgaria had been impacted at the same time.

The figures below highlight the impact of this network issue at a country level in Iran, Turkey, and Bulgaria. Both the Oracle and CAIDA IODA graphs show that the most significant impact was observed in Iran, with changes to the measured metrics for Turkey and Bulgaria much more nominal in the Oracle graphs, and nearly imperceptible in the CAIDA IODA graphs.

As was noted by Sadjad Bonabi and NetBlocks, the network disruption impacted Google services in the affected countries, and Google’s Cloud Status Dashboard reported an “issue with multiple simultaneous fiber cuts affecting traffic routed through Google’s network in Bulgaria.” The figures below illustrate how the fiber cuts impacted traffic to Google’s YouTube, Maps, and GMail services (left-to-right) from users in Iran, Turkey, and Bulgaria (top-to-bottom). A brief but obvious dip in traffic is evident in each graph at the time the disruption occurred.

A day later, mid-Atlantic region customers of U.S. Internet provider Comcast experienced a brief Internet disruption. The screenshot below from Fing Internet Alerts shows that the disruption impacted subscribers in Maryland, Virginia, the District of Columbia, Delaware, and West Virginia.

The Fing Internet Alerts graphs below show that the disruption began just before midnight local time (0500 GMT) on December 20, and lasted for approximately an hour. This timing is corroborated by the CAIDA IODA graph below, but given the localized impact of the disruption, the impact to the measured metrics is minimal.

A thread on the Outages mailing list explained that the disruption happened because “There was a maintenance in one part of a regional network, but there was an optical line failure in another part of that same network that happened during the maintenance.”

Government Directed

No stranger to government directed Internet shutdowns, India moved to shut down Internet services in the northern states of Assam, Tripura, and Meghalaya on December 12. The shutdown was ordered in response to protests stemming from the passage of the Citizenship Amendment Bill. According to a Tweet and subsequent report posted by NetBlocks, the actions impacted both fixed and mobile network providers. Within the Tweet and report, NetBlocks showed connectivity levels it measured for the autonomous systems of seven local network providers.

The figures below show how connectivity to these providers was seen by CAIDA IODA measurements during the multi-day disruption. While the start of the disruption at ~1600 GMT on December 12 is generally evident across the graphs, there are noticeable differences across them as well. These differences may be due to the types of network each autonomous system is associated with, as mobile networks are significantly harder to measure into from external vantage points. Several see restoration and disruption events that align with those seen in NetBlocks’ report, while others appear to show a complete loss of connectivity for over four days.

A court order published on December 19 ordering the restoration of services highlighted the detrimental affect that the loss of Internet connectivity has had on everyday life:

• “It has been impressed on the Court that even electricity supply has been snapped to certain houses of the lawyers because pre-paid meters have been installed, which can only be charged through the Internet.”
• “Under the circumstances, none of the business establishments is able to transact business causing serious disruption in normal living of the citizens in the area.”
• “To say the least, with the advancement of science and technology, mobile Internet services now plays a major role in the daily walks of life, so much so, shut-down of the mobile Internet service virtually amounts to bringing life to a grinding halt.”

Additional protests over the Citizenship Amendment Bill caused the Indian government to order the shutdown of mobile Internet connectivity in New Delhi, as well as the suspension of Internet services in Aligarh, Meerut, Malda, Murshidabad, Howrah, and parts of West Bengal during this timeframe as well. On December 26, similar steps were taken in Uttar Pradesh.

As reviewed in the November 2019 Internet Disruption Report, Iran suffered a week-long government directed Internet shutdown in response to protests over gasoline prices. On December 25, the Iranian government ordered restrictions on mobile Internet access in several provinces ahead of protests and ceremonies to remember those killed in the prior month’s protests.

A NetBlocks Tweet about the ordered mobile shutdown looked at the impact on connectivity for AS57218 (Rightel), one of three mobile providers in Iran.

Update: Mobile internet connectivity has fallen further in #Iran amid reports of security reinforcement; real-time network data show connectivity at 5% of ordinary levels on specified networks after four distinct cuts; incident ongoing #Internet4Iran

https://t.co/BNTPP9wEyt pic.twitter.com/Wh2uy5xIxp

— NetBlocks.org (@netblocks) December 26, 2019

The figure below shows the impact of the order shutdown on Rightel as observed by CAIDA IODA’s measurements. The changes to the BGP and Active Probing metrics show that connectivity was lost in several stages, with Active Probing reaching its lowest level just before 1200 GMT on December 26. Connectivity was apparently restored around 0500 GMT on December 28 as the Active Probing and BGP metrics rose rapidly, returning to pre-disruption levels.

Interestingly, CAIDA IODA graphs for the country’s other two national mobile providers (Iran Cell and MCI) did not show similar disruptions to connectivity.

Russian Internet Disconnection Test

On December 23, a post to tech news aggregator Slashdot claimed “Russia has temporarily shut off many of its citizens’ access to the global internet today in a test of its controversial RuNet program, according to an internal government document.” That same day, ZDNet declared “Russia successfully disconnected from the internet”, noting “RuNet disconnection tests were successful, according to the Russian government.” The BBC picked up the story two days later.

According to the BBC article, “Details of what the test involved were vague but, according to the Ministry of Communications, ordinary users did not notice any changes.” The ZDNet article noted “The tests were carried out over multiple days, starting last week, and involved Russian government agencies, local internet service providers, and local Russian internet companies. … Internet traffic was re-routed internally, effectively making Russia’s RuNet the world’s largest intranet.” With this explanation, these tests would presumably have been visible in some fashion from outside the country.

However, that did not seem to be the case. Graphs from Oracle’s Internet Intelligence Map and CAIDA IODA did not show any noticeable changes to the measured metrics, and routing statistics from RIPENCC confirmed BGP’s stability over the period in question. Google Transparency Report traffic graphs for Web search, YouTube, GMail, and Blogger services did not show any deviations from the expected diurnal patterns and contacts at two leading CDN providers also reported no noticeable changes to traffic patterns from Russia during the time the test reportedly took place. Additionally, traffic graphs from IXPs within Russia, as well as DECIX, did not show any evidence of changes in traffic volume that would be expected if an Internet shutdown had occurred.

With a lack of evidence across major Internet platforms and measurement tools, we might ask whether the disconnection test actually occurred, as was reported, and if so, what was tested. The translation of a Russian-language article may provide some insight. According to the article, the disconnection may have been an “exercise” conducted in a test environment, rather than on Russia’s production Internet. The article also notes that exercises held on December 16 & 17 involved the SS7 and Diameter signaling protocols – these are used in the back-end of telecom networks, and are not related specifically to the Internet. It isn’t clear whether the tests that reportedly took place on/around December 23 also only focused on telecoms, rather than native Internet, infrastructure. Several additional Russian-language articles (1, 2) also questioned whether the tests had actually taken place, as well as the reported results of the tests.

Additional Observations

Recognizing that its reliance on a lone submarine cable for international Internet access represents a single point of failure, the government of Tonga is reportedly taking steps to implement backup satellite-based Internet connectivity. The goal is to avoid a repeat of the nationwide Internet outage that occurred in January 2019.

A “digital decree” passed by Spain’s government in late November reportedly gives it the power to shut down Internet connectivity in specified areas without prior judicial order. The relevant text is an adaptation of Section 4 of Article 4 of the current General Telecommunications Law:

“The Government, on an exceptional and transitory basis, may agree on the assumption by the General State Administration of direct management or the intervention of electronic communications networks and services in certain exceptional cases that may affect public order, public safety and national security This exceptional power may affect any infrastructure, associated resource or element or level of the network or service that is necessary to preserve or restore public order, public safety and national security. “ (via Google Translate)

As expected, concerns were raised about the constitutionality of the new rules, and about potential human rights violations associated with Internet shutdowns enabled by the decree.

In December, the Internet shutdown in Rakhine, Myanmar entered its seventh month and the shutdown in the Kashmir region of India entered its fifth month. These shutdowns have had a significant impact on the local economies, affecting small local businesses, as well as limiting the delivery of e-government services. At the end of 2019, it was observed by CNN and Quartz that Internet shutdowns have effectively become the ‘new normal’ and are/will be an increasingly popular mechanism for government repression.

Conclusion

I launched the Internet Disruption Report in May 2019 as a continuation of work that I had done as founding editor of Akamai’s State of the Internet/Connectivity Report, as well as the monthly “Last Month In Internet Intelligence” blog posts I published as a member of the Oracle Internet Intelligence team. Response has been very positive, and I am extremely grateful to those organizations that not only monitor and measure the Internet, but also make their findings and analysis publicly available. I also applaud those network providers that make effective use of social media channels to provide notice of, and status updates regarding, both planned and unanticipated disruptions, as well as those providers that respond to requests for more information about such disruptions. (Hint, hint…)

Given the growing trend towards government directed Internet shutdowns, and the otherwise unavoidable Internet disruptions caused by power outages, cable and fiber cuts, routing issues, and DDoS attacks, I am likely (for better or worse) to have plenty of content for many future Internet Disruption Report posts. I look forward to sharing my insights on these events with you in 2020 and beyond.

Power Outages

On November 4, the country of Curaçao experienced a power outage that affected the entire island. According to a published report, the power went out around 0915 local time (1315 GMT), and started to come back on around nine hours later. This aligns with the periods of disruption seen in both the Oracle Internet Intelligence and CAIDA IODA graphs below. However, in looking at those graphs, we see that there was nominal impact to the BGP metric in each, and that the Traceroute Completion/Active Probing metrics only saw a slight decline – these two factors indicate that core network infrastructure on the island is likely protected with backup power sources, and also serves as a target for active measurement. Conversely, the Darknet metric at a country- and network-level declined to near zero during the power outage, as shown in the CAIDA IODA graphs below, likely indicating a greater impact on end-user systems.

On November 12, Internet monitoring firm NetBlocks noted that a series of blackouts knocked multiple Venezuelan states offline for several hours. Although the NetBlocks post indicates that the power outages and associated connectivity issues started during the morning, the CAIDA IODA graphs below show Internet disruptions occurring later in the day at both a country and regional level. The regional graphs show four Venezuelan states listed by NetBlocks as seeing significant connectivity impact due to the power issues. (Note that the Internet disruptions visible in the graphs may not be related to the power outages highlighted by NetBlocks, given the time differences, but Venezuela is no stranger to Internet disruptions caused by power outages, having experienced multiple such issues over the last few years.)

Fiber/Cable Issues

Yon kab fib optik ki te koupe nan mitan lannuit la te lakoz rezo a gen difikilte, e sa te afekte sèvis tankou entènèt, apèl entènasyonal, sms. Teknisyen nou yo travay san kanpe pou repare li prese prese. Nou eskize n pou tout enkonvenyan sa te poze pou kliyan nou yo

— Digicel Haïti (@DigicelHT) November 6, 2019

On the morning of November 6, Digicel Haiti Tweeted an explanation of an Internet disruption experienced overnight, noting (via Google Translate) “A fiber optic cable cut in the middle of the night caused network difficulties, affecting services such as Internet, international calls, sms. Our technicians work tirelessly to repair it quickly. We apologize for any inconvenience this posed to our customers.” The CAIDA IODA graphs below show an Internet disruption at both a country- and network-level starting around 0600 GMT (0100 local time), recovering approximately six hours later.

Another fiber cut impacted connectivity in Haiti a week later. On November 13, the Chairman and CEO of Digicel Haiti Tweeted that (via Google Translate) “Today, still a case of sabotage of a major optical fiber of @DigicelHT . This time affecting the 55 antennas covering Carrefour. Access was very complicated for engineers (a lot of shooting in the area). They are on site and repairs are in progress.”

Aujourd’hui, encore un cas de sabotage d’une fibre optique majeure de @DigicelHT. Cette fois ci affectant les 55 antennes couvrant Carrefour.

L’accès a été très compliqué pour les ingénieurs (beaucoup de tirs dans la zone).

Ils sont sur place et les réparations sont en cours.

— Maarten Boute (@mboute) November 13, 2019

The Oracle Internet Intelligence and CAIDA IODA graphs below show that the disruption began around 0300 GMT, impacting the Traceroute Completion/Active Probing and BGP metrics at a country level. It is interesting to note that the BGP metric was impacted – the fiber sabotage apparently impacted connectivity to key Digicel Haiti routing infrastructure. This is more evident in the network-level graphs below, with the CAIDA graph showing a complete loss of routed networks for AS27653 at the same time that both the Oracle and CAIDA graphs show traceroutes/active probing to Digicel Haiti effectively dropping to zero.

On November 17, concurrent Internet disruptions were observed in Sierra Leone, Liberia, and the Gambia, occurring around 1045 GMT, as illustrated in the figures below. Because the Africa Coast to Europe (ACE) submarine cable lands in each of these countries, it was suspected that problems with the cable caused these observed disruptions.

In response to an inquiry, the Twitter account associated with the ACE submarine cable system (@acesubmarinec) confirmed that the observed disruptions were, in fact, due to problems with the cable.

@InternetDsrptns
Yes there was a disruption of the traffic on ACE between Banjul and Dakar to fix shunt falt. Everything went well and the traffic is now on. Sorry for the inconvenient and thanks for reaching out.

— Africa Coast to Europe (@acesubmarinec) December 6, 2019

Making its third appearance in this month’s report, Venezuela experienced an Internet disruption on November 22 and 23 due to a fiber optic cable cut at state-run Internet service provider CANTV. In a Twitter thread, CANTV stated that (via Google Translate) “Cantv Informs that motivated to technical incidence presented at the La Pedrera plant in #Táchira , and a fiber optic cut originated in the Cabimas-El Venado axis, in #Zulia , telecommunications are affected in #Barinas #Mérida #Táchira #Trujillo #Falcón and #Zulia” The disruption is evident at a country level in the CAIDA IODA graph below, with nominal impact visible starting mid-day (GMT) on November 22 and lasting approximately six hours. The level of impact is similar at a network level, as the CAIDA IODA graph for CANTV below illustrates. The disruption visible on November 23 is likely related to a second CANTV issue that was reported by NetBlocks.

The CANTV Tweet listed six specific Venezuelan states where telecommunications were affected on November 22. The CAIDA IODA graphs below for these states highlight the impacts to Internet connectivity in each. It appears that the disruption was the least significant in Barinas, and most severe in Mérida, Táchira, and Zulia – the latter two were mentioned in the Tweet as locations where problems occurred.

Routing Issues

On November 5, Oracle Internet Intelligence (@InternetIntel) Tweeted about a route leak that briefly interrupted Internet connectivity in Indonesia. The screenshot below from BGPStream shows that a route leak from China Telecom just after 0330 GMT impacted a number of Indonesian networks.

The Oracle Internet Intelligence graph below shows that their measurements saw a brief blip in the BGP metric, with more noticeable (but still brief and nominal) impacts to the Traceroute Completion and DNS Query Rate metrics. In line with the latter impacts shown in the Oracle graph, the CAIDA IODA graph below shows significant impacts to their Active Probing and Darknet metrics, with no clear impact to the BGP metric.

Just a few days later, Oracle Internet Intelligence Tweeted about another route leak, this one affecting networks in Ecuador. They stated that the leak started just before 0900 GMT on November 9, and lasted almost five hours. According to the BGPStream screenshot below, global service provider CenturyLink and Ecuadorian service provider Telconet were involved.

The figures below show the impact of this route leak at a country level, based on Oracle Internet Intelligence and CAIDA IODA measurements. While Oracle saw a minimal impact on its Traceroute Completion metric, the impact on CAIDA’s equivalent Active Probing metric appears to be more significant. However, there was no visible impact to the BGP metric on either graph. The network level graphs below show the impact to Telconet. Both measurement platforms show a significant decrease in their active measurement metrics. Interestingly, there was no decrease in the CAIDA Darknet metric, which would indicate that although packets were having a harder time getting into the network, they were still able to get out.

Government Directed

NetBlocks was the first to sound the alarm on Twitter of a wide-scale Internet shutdown in Iraq starting at 0000 GMT on November 5. The shutdown occurred in the wake of reported anti-government protests that resulted in multiple deaths.

Alert: Internet shut down across most of #Iraq as of 00:00 5 November with loss of life reported as protests intensify; national connectivity down to 19% of ordinary levels; incident ongoing #iraq_protests #KeepItOn https://t.co/SeCyP2mj1T pic.twitter.com/WxYX54WXH9

— NetBlocks.org (@netblocks) November 4, 2019

As the figures below illustrate, the disruption lasted a week, from November 5 through November 12. Connectivity returned briefly on the 5th and 7th, but the restorations were short-lived. Longer, more consistent restoration of connectivity is evident during the last three days of the disruption, in a pattern NetBlocks called “curfew-like information controls”. Given the timing, it appears that Internet connectivity was restored roughly in line with the local work day, and then shut down again until the next morning. (Iraq and other countries have implemented a similar “curfew” model in the past.) And while Internet connectivity returned to “normal” on the 12th, access to social media platforms reportedly remained blocked, as it had been since early October.

On November 15, protests erupted across Iran in response to a steep increase (on the order of 50%) in officially set gasoline prices. The protests broadened to include additional grievances, including social repression and corruption, and a number of people were killed after police clashed with the protestors. In response to the protests, the Iranian government disrupted Internet connectivity and mobile data services in an effort to prevent citizens from organizing additional protests and sharing information.

The figures below show that the Internet disruption began on November 16, and lasted until November 23. However, as the Google Transparency Report traffic graph shows, while connectivity was initially restored on the 23rd, it took nearly another week for traffic levels to Google services to reach pre-disruption levels.

The difficulty of shutting down Internet connectivity in a country depends in part of the centralization of such connectivity. In countries with a centralized infrastructure, where international Internet traffic is routed via a state-run telecommunications provider, a shutdown can be implemented through the actions of that single provider, effectively creating a so-called “kill switch”. In countries with international Internet connectivity routed via multiple providers, a shutdown must be coordinated across those providers.

Although the Iranian government has focused on building out a centralized national “intranet” over the past decade, the CAIDA IODA figure below shows that this disruption was of the latter type described above, with connectivity dropping across multiple networks over the course of several hours, instead of all at once, as if a single “kill switch” had been thrown. The graph shows that, among the network providers shown in the graph, connectivity was first lost at 1430 GMT on AS197207 (Mobile Communication Company of Iran), while connectivity on AS58224 (Iran Telecommunication Company) and AS16322 (Pars Online) didn’t drop until 1730 GMT. Interestingly, while connectivity to AS44244 (Iran Cell Service and Communication Company) dropped around 1500 GMT, endpoints within the network were again reachable by active probing measurements starting a couple of hours later.

(The autonomous systems in the figures below are those that were included in a now-deleted Tweet from Cloudflare’s CTO (@jgrahamc) that showed traffic volumes to Cloudflare’s platform from these networks in response to the initial Internet disruption and subsequent restoration of connectivity.)

The figure below shows the results of CAIDA IODA Active Probing measurements to the set of autonomous systems during the disruption event period. As noted above, AS44244 began responding to active probing measurements again several hours after it initially lost connectivity. As seen in the graph below, a partial restoration of connectivity for AS197207 apparently occurred as well, as endpoints within that network began responding to measurement probes around 0600 GMT on November 17. It also appears that connectivity for AS39501 (NGSAS) was never completely disrupted, as it maintained some level of response to measurement probes across the duration of the disruption.

Restoration of connectivity was gradual, and nominally concurrent, as the Active Probing metric began a slow increase on November 21, leading to a more significant jump on the 23rd, reaching near-“normal” levels on the 24th.

In the figure above, the active probing metric approaches, but never fully reaches, zero across the surveyed networks, suggesting that the disruption was not a full and complete shutdown, at least for traffic to those networks. The figure below shows the CAIDA IODA Darknet metric across the surveyed autonomous systems during the disruption period. (The Darknet metric represents unique source IP addresses in those networks generating unsolicited traffic reaching the UCSD Network Telescope, which is monitoring an unutilized /8 address block.) While the beginning and end of the disruption are evident in the graph, it also shows that unsolicited traffic was detected originating from IP addresses in several surveyed networks, including AS16322 and AS49100 (Pishgaman Toseeh Ertebatat Company), among others. This suggests that some traffic was able to get out of these networks as well. (The gaps in the graph on November 19, 22, and 23 were due to problems with the underlying data processing infrastructure, and are not related to the Internet disruption discussed here.)

The observation that some small amount of traffic was apparently able to get into and out of some of the disrupted networks is in no way intended to minimize the severity of what occurred. Rather, it highlights the complexity of executing a complete Internet shutdown, and also suggests that there may have been some reason that some level of connectivity was maintained to selected networks — it is possible that customers include government agencies, financial institutions, or other influential organizations.

Additional coverage of the initial disruption and subsequent restoration of connectivity, including views from other metrics, can be found in posts from NetBlocks (1,2), Oracle Internet Intelligence (1,2), the Monash IP Observatory, OONI, and Filterwatch, as well as a Twitter thread from Internet security and digital rights reseaercher Amir Rashidi.

Conclusion

The two government-directed Internet disruptions reviewed in this month’s post are noteworthy and extremely concerning due to their severity and their duration. These actions disconnected citizens in both Iran and Iraq from the global Internet for week-long periods, including those uninvolved in the protests, and also prevented citizens from communicating with friends and loved ones within the country and across the rest of the world.

The United Nations has explicitly identified such government directed Internet shutdowns and censorship as a human rights violation, but that does not seem to deter these countries from taking such actions. However, these actions can also have unintended effects, as they also serve to limit a government’s ability to conduct online surveillance of citizens, as well as potentially fostering even more powerful protest movements among impacted citizens, even if they aren’t able to use Internet-based tools to organize. Governments also need to recognize that these disruptions have international technical impacts and long-term financial impacts, hurting their country now and into the future.

(Apologies for posting this month’s update a few weeks later than usual – sometimes life and work get in the way of blogging…)

Power Outages

We have observed Internet disruptions in Venezuela in the past that were caused by the country’s unstable electrical power infrastructure, and October was no different. The figure below shows the impact of power outages that occurred on both October 1 & 2, both just before 1800 UTC. On both days, the BGP metric remains stable (indicating that the systems announcing the routes were unaffected by the power problems), but the Active Probing and Darknet metrics saw declines due to loss of connectivity within the last mile networks that the probes target and that originate the Darknet traffic.

In Gibraltar, the country’s electricity authority took to Twitter to alert customers to an ongoing power outage, and to apprise them of the cause of the outage. The figures below show that the power outage likely started about a half hour before the Tweets were published, with a country-level Internet disruption in Gibraltar starting just after 1100 UTC. Connectivity returned to normal about three hours later.

In California, regional power company Pacific Gas & Electric (PG&E) once again implemented “Public Safety Power Shutoffs“, noting:

“For public safety, it may be necessary for us to turn off electricity when gusty winds and dry conditions, combined with a heightened fire risk, are forecasted. … While customers in high fire-threat areas are more likely to be affected, any of PG&E’s more than five million electric customers could have their power shut off.”

These power outages obviously have the ability to impact local Internet connectivity. As one such example (out of several over the course of October), network monitoring firm Fing observed a disruption at regional service provider Valley Internet (the d/b/a name of Personal Network Computing) lasting for more than half a day, as shown in the Tweet below from Fing’s @outagedetect account, as well as the figure below. The locations of the measured Internet service disruption aligned with the map of expected affected areas posted to the PG&E Outage Center on October 9, providing a high likelihood of correlation between the two events.

The localized network-level disruption was also observed through other network monitoring tools. The figures below show that both Oracle Internet Intelligence and CAIDA IODA recorded declines for the actively measured metrics during the course of the power outage and associated disruption.

Cable Cuts

On October 16, Sudan suffered a nearly three-hour Internet disruption. The figures below show a sharp decline in the three metrics measured by Oracle Internet Intelligence and CAIDA IODA.

A local contact reported that the disruption was caused by a “double cut” to a Sudatel fiber ring, and that MTN was also impacted by the fiber cut. The figures below illustrate the impact of the cut to both Sudatel and MTN Sudan.

And while we don’t often cover it here, these Internet disruptions do impact local users and businesses. As one example, CashQ, a mobile payment application, was unable to provide services to its users as a result of this Internet disruption in Sudan, as it explained on Twitter.

A week later, a problem with the SEACOM Subsea Cable System disrupted Internet connectivity in Kenya and Comoros on October 23. In a series of Tweets (1, 2, 3), @SEACOM stated that the outage began at 2200 UTC on October 22, impacting its subsea cable system between Mombasa [Kenya] and Zafarana [Egypt], and forcing it to route traffic over West coast transmission links. The figures below show that both Oracle Internet Intelligence and CAIDA IODA detected a minor perturbation in the BGP metric at a country level for Kenya around the time that the outage occurred. Oracle’s measurements also recorded a decline in successfully completed traceroutes to endpoints in the country, while a nominal impact was seen in the Darknet sources metric measured by CAIDA.

One Communication/Safaricom is a leading telecommunications provider in Kenya, and is clearly reliant on the SEACOM cable for international Internet connectivity. The Oracle Internet Intelligence graph in the figure below shows a significant drop in completed traceroutes to endpoints within the network, while the CAIDA IODA graph shows a near complete loss of visible networks associated with the autonomous system visible in the routing table. CAIDA also shows a concurrent drop in the Darknet Sources metric, indicating that systems within that network were no longer probing the address space used by CAIDA’s Network Telescope.

Impacted networks in Kenya shifted upstream providers during the cable outage, resulting in increased latency for traffic to these networks. The figures below illustrate the latency impact for two networks: Safaricom saw latency increase by approximately 50% as it shifted to Emirates Telecommunications as its primary upstream provider, while Kenyan Post/Telkom Kenya saw latency increase by approximately 10-20% as it moved to Emirates Telecommunications.

As noted above, Comoros was also impacted by this problem with the SEACOM submarine cable. The Oracle Internet Intelligence figure below shows a significant impact to the Traceroute Completion Ratio metric, and a very slight impact to the BGP metric (with the exception of a sharp but brief drop). The CAIDA IODA figure below shows a somewhat more significant impact to the BGP metric, with a similar sharp but brief drop towards the end of the disruption period.

Comores Telecom was also heavily affected, with both Oracle and CAIDA showing a near complete outage for the networks, as seen in the figures below.

Pakistan experienced an Internet disruption on October 29, due to a fault in an international submarine cable, as explained in a Tweet from @PTAofficialpk, a Twitter account associated with the Pakistan Telecommunication Authority. The figure below shows the impact on CAIDA IODA’s Active Probing to endpoints within the country. There was no obvious routing impact, and Darknet traffic also appeared to be unaffected.

According to a subsequent Tweet from the Pakistan Telecom Company (PTCL), the cable fault occurred on the SMW-4 cable. The figures below show the impact across two of PTCL’s autonomous systems, with Oracle Internet Intelligence showing increased latencies as more traffic flowed over Cogent as an upstream provider, while CAIDA IODA showed a decline in the Active Probing metric.

A published article highlighted a local impact of this Internet disruption, quoting a Tweet from Pakistan International Airlines (PIA) that explained that the the issues that the cable cut caused for PTCL ultimately resulted in an outage of PIA flight check-in systems across the country, forcing them to move to backup systems and manual processes and resulting in passenger delays.

Government Directed

The most significant Internet disruption that occurred in October took place in Iraq, with connectivity impaired for over a week. Published accounts (AlMonitor, CPJ, AccessNow) reported violent protests over unemployment, government corruption, and a lack of basic services in Baghdad and other Iraqi cities on October 1-2, resulting in over a dozen deaths and hundreds of injuries. In response to the protests, the Iraqi government imposed a near total Internet shutdown on October 2, apparently beginning at approximately 1600 UTC, as shown in the figures below. The figures also illustrate that the shutdown did not completely remove Iraq from the Internet. This is reportedly due to the Kurdistan Regional Government’s (KRG) Ministry of Transportation and Communications rejection of the call by the Iraqi federal government to cut off the Internet in the Kurdistan Region. The Kurdistan region’s connection to international communications infrastructure is separate from the one in southern and central Iraq.

As the figures below show, the Internet disruption lasted for over a week, with measured metrics returning to “normal” levels on October 11. After a few brief restorations of connectivity on October 3 and 6, Internet availability settled into a “curfew” model, as described by NetBlocks, with access returning during the workday, and then being cut again in the evening.

Unsurprisingly, trends at a network level are nearly identical to the activity seen at a country-level. The figures below show measurements across three major Internet service providers in Iraq, with BGP and Active Probing patterns that appear very similar to the ones seen in the country-level graph above.

One published report explained that some Iraqis initially attempted to use VPN technology to circumvent connectivity restrictions, while those that could afford to made use of satellite-based Internet connectivity. Others queued content posts and videos within the social media and chat applications on their mobile devices, with the expectation that they would be sent when these devices were finally able to connect to the Internet.

Based on measurements done by Geoff Huston at APNIC Labs, it also appears that some Iraqi users are configuring third-party public DNS resolver services as backups in the event that the resolvers at their providers (generally configured as the default) become unavailable. Based on Huston’s experiments, the figure below shows that prior to October’s Internet disruption, approximately 65% of DNS requests made by Iraqi users reached authoritative DNS servers through a resolver in the same autonomous system (“sameas”) – that is, the user’s ISP. Approximately 30% were made through Google’s 8.8.8.8 resolver (“googlepdns”), while less than 10% were made through CenturyLink/Level 3’s open public resolver (“level3”). However, for a multi-day period after the disruption started, the percentage of DNS requests from Iraqi users coming through Google surpassed those coming from resolvers at the user’s service provider. Usage of Level 3’s resolver also increased during this period. The graph shows that request volume through ISP resolvers returned to previous levels in line with the restoration of connectivity in Iraq. (It was pointed out that the conclusion reached here was incorrect: “You need to look at that in terms of absolute values. The relative values are biased, as the denominator changed. Those affected by the shutdown stopped reporting. In this case, The Kurdish ISPs were not shutdown. It may be the case that the distribution in those networks is different, and that was the reason for the shift, and not actual changes by the users.” The original text and figure remain for transparency, but should be considered deprecated.)

As noted in the introduction, this shutdown had a significant impact on the Iraqi economy, with an article in the International Business Times placing the estimated impact at nearly a billion dollars. The article highlights the financial pain experienced by members of the emerging digital economy in Iraq, including startups, e-commerce sites, online delivery companies and couriers, and travel companies.

Other

Similar to previous months, several additional less-well attributed Internet disruptions were observed in October. For some, we could observe the disruption but could find no root cause information, or did not receive any response to social media outreach to impacted providers. For those included below, acknowledgement of the issue was made by the providers through their social media channels, even if few details were given.

On October 14, SamoaTel/Bluesky Samoa informed users via Facebook and Twitter of an disruption to fixed Internet services. Unfortunately, no specific information about what caused this disruption was provided. The figures below show that while a nearly complete outage was seen at a network level, the impact was less significant at a country level.

Also on October 14, Telekomn Brunei (TelBru) apologized to users via Twitter for downtime for broadband services apparently caused by a third-party service provider. The figures below show that the disruption was partial at both a country and network level, and fairly brief, lasting just a couple of hours. To its credit, TelBru regularly posts updates to Twitter (@telbru_bn) alerting customers of upcoming downtime to to maintenance activities.

Internet monitoring firm Fing tracked a 15+ hour Internet disruption at provider Three UK that started the evening of October 16, lasting until mid-day on October 17.

The figure below shows the impact of disruption at a network level, with drops seen in the Active Probing and Darknet metrics. @ThreeUK acknowledged the network disruption in a Tweet, but failed to provide any additional information about what happened.

And near the end of the month, Maldives provider Dhiraagu alerted users via Twitter and Facebook to service degradation on mobile & fixed broadband services.

The figures below show that, similar to the TelBru issue above, the disruption was partial at both a country and network level, and fairly brief, again lasting just a couple of hours.

Conclusion

While both Internet outages and Internet shutdowns rob users of the ability to communicate with others, purchase goods and services, and consume media, there are significant differences between the two types of disruptions. In general, outages are unintentional or accidental disruptions to Internet connectivity – in the case they are intentional, it is generally due to infrastructure or network maintenance. Regardless, they are generally short lived, as impacted providers scramble to address the underlying issues as much as they have the ability to.

However, Internet shutdowns are the intentional disruption of Internet connectivity at the direction of a local, state, or national government, and often last for several days, if not weeks or months. These shutdowns have significant human rights impacts for citizens of the affected country, including limiting freedom of expression online, as well as significant economic impacts, with estimated costs running into multiple millions of (US) dollars each day the Internet is unavailable, as well as reducing the likelihood of future international investment within the country.

With the world increasingly watching, governments need to explore alternative ways of addressing the underlying issues that lead to Internet shutdowns, instead of leveraging these actions as a blunt-force policy tool. #KeepItOn

Power Outages

Internet connectivity in Venezuela was disrupted several times over the course of the month due to multiple widespread power outages. On September 12, NetBlocks reported that Caracas was experiencing its second power outage in two days, corroborated by a Tweet from @CORPOELECinfo, an account associated with Venezuela’s national electric corporation.

The impact of these two outages can be seen in the figures below for the Capital District and Miranda – Caracas straddles both states. A minor disruption can be seen mid-day on September 11, with a more significant disruption occurring mid-day on September 12.

Just over a week later, another significant power outage disrupted Internet connectivity in Venezuela for several hours, as shown in the figures below. Starting around 1500 UTC on September 20, the disruption lasted until approximately 0100 UTC on September 21.

The disruption was also very visible at a local level as the CAIDA IODA graphs below for the Venezuelan states of Lara, Tachira, and Zulia show, with marked drops in the Active Probing and Darknet metrics.

Brief power outages early in the day (GMT) on September 24 and mid-afternoon (GMT) on September 27 again caused Internet disruptions visible at a country level, as seen in the figures below.

National Exams

After seeing several Internet disruptions during the last week of August intended to prevent cheating on national exams, two more disruptions occurred in Iraq during the first few days of September. As the figures below show, partial disruptions were observed on September 1, 2, and 3, with diminishing severity across the three days.

Hurricanes

Several significant Atlantic Ocean hurricanes impacted Internet connectivity during September, starting with Hurricane Dorian. On September 1 & 2, Dorian made multiple landfalls in the Bahamas, causing significant damage. As the Oracle Internet Intelligence and CAIDA IODA figures below show, Internet connectivity in the Bahamas began to degrade shortly after midnight (GMT) on September 2, and then degraded significantly around 0730 GMT. The impact can also be observed at a city and network level, as seen in the additional CAIDA IODA figures below. However, in the main city of Freeport, the Internet disruption started earlier, with a significant drop in the Active Probing metric observed around 0200 GMT. While a slight decline was observed in the Active Probing metric for AS15146 (Cable Bahamas) around 0200 GMT, a more significant decline occurred around 0730 GMT. As the Google Transparency Report graph for the Bahamas below shows, connectivity took several weeks to return to “normal” levels.

A week later, after moving up the eastern seaboard of the United States, Hurricane Dorian landed in the Canadian province of Nova Scotia. There, the strong winds impacted Internet connectivity for subscribers of local network provider Eastlink in cities including Halifax, Dartmouth, and Truro, according to Fing Internet Alerts. The figure below from Fing shows that the disruption began around 1320 GMT, which corresponds to the time that the Active Probing and Darknet metrics began to decline in the CAIDA IODA figure below.

On September 19, Hurricane Humberto battered Bermuda, reportedly knocking out power for 80% of the island’s electricity customers. This widespread impact is best reflected in the lower peak in the DNS Query Rate metric seen in the Oracle figure below, the significant drop in the Darknet metric in the CAIDA IODA figure, and the visibly lower peak and deeper trough in the Google figure. Oracle’s Traceroute Completion Ratio metric and IODA’s Active Probing metric also saw declines, as would be expected, but the drops were less pronounced. This may be because a large portion of the targets selected for these active measurements are infrastructure components kept online with backup generators or other alternative power sources.

Network Issues

On the morning of September 1 (GMT), country-level disruptions are visible in the Oracle Internet Intelligence Country Statistics graph for Haiti, shown in the figure below. Three separate disruptions occurred – the first lasted approximately four hours, starting at 0300 GMT, followed by two separate hour-long disruptions, starting at approximately 0740 GMT and 0900 GMT.

The observed disruption was apparently due to a fiber cut experienced by local provider Digicel Haiti, as it explained in the Tweet below. (English translation, via Google Translate: “We want to apologize to all customers who encountered a difficulty yesterday or morning while using the network! Unfortunately one of our underwater fiber optic cables was cut off! Our technicians work tirelessly to repair it quickly, now all our services are working.”) The Oracle Internet Intelligence Traffic Shifts figure shown below illustrates the impact of the fiber cut in Digicel Haiti – a near complete failure of traceroutes through upstream providers to reach the network, with a significant spike in latency for the few that did get through.

On September 16, Internet users in St. Thomas on the U.S. Virgin Islands experienced an Internet disruption due to a fiber cut. In a post to its Facebook page, local service provider Viya noted that a public works contractor had severed a fiber optic cable, impacting connectivity. The CAIDA IODA figure below shows a visible drop in the Active Probing metric for St. Thomas starting just after 1200 GMT. (However, the apparent corresponding increase in the Darknet metric is unusual – it is not clear why it occurred.)

Singaporean provider StarHub experienced a multi-hour disruption on September 17 that impacted both fixed and mobile Internet users. To their credit, StarHub was quick to respond, and ultimately posted an update on its Web site, providing more information on the incident:

“An internet routing issue has been identified as the root cause of the incident, and StarHub has ruled out a cyber attack. StarHub takes this incident seriously and is implementing measures to prevent a recurrence.”

The CAIDA IODA figures below, ordered by the number of IPv4 prefixes originated, show the impact of the routing issue on five StarHub autonomous systems. The most significant impact appeared to be with AS55430, associated with StarHub’s Next Generation Nationwide Broadband Network. Connectivity for AS10091, associated with StarHub’s discontinued cable Internet service, also saw a major impact.

Fing Internet Alert Tweeted about an Internet disruption affecting New Zealand provider Trustpower on September 29. The CAIDA IODA figure below for the network shows that the disruption was brief but significant, with clear declines occurring in both the Active Probing and Darknet metrics. Trustpower posted several apology Tweets like the one shown below, explaining that it had experienced a brief outage in its fiber services.

Somewhat unsurprisingly, chronically problematic submarine cable SEA-ME-WE-3 suffered another failure on September 20, according to a published report. While the cable connects Australia to Asia and Europe, the Perth-to-Singapore segment suffered issues this time, with the problem apparently occurring between repeaters 345 and 346. The segment has experienced documented problems in the past, but traffic has been able to fail over to the Australia-Singapore Cable (ASC) since September 2018.

DDoS Attack

Cool Ideas, one of South Africa’s largest Internet Service Providers, was targeted by a DDoS attack on September 21 & 22. The attack reportedly used DNS Amplification to overwhelm Cool Ideas’ connections with traffic. Cool Ideas posted status updates to its Web site while the attack was underway, alerting users to potential connectivity loss and degraded performance.

Based on the CAIDA IODA figure below, it appears that the attack may have come in several waves, with multiple declines seen in the Active Probing metric during the two day period. The Oracle Internet Intelligence figure below shows how the attack impacted latency for traceroutes to targets within the Cool Ideas network, with increases observed during the waves of the attack. The figure also shows how Cool Ideas’ upstream connectivity shifted as a result of the attack, with traffic moving from AS6939 (Hurricane Electric) to AS3741 (Internet Solutions). A Cool Ideas representative noted that they had integrated attack mitigation mechanisms with their Internet Solutions connectivity, which is likely why they shifted to having more traffic flow through that upstream provider (so it could be scrubbed).

This upstream connectivity shift is illustrated in more detail in the figure below from IIJ Lab’s Internet Health Report. The graph shows the AS dependencies for Cool Ideas’ ASN during the duration of the attack. Before the attack began, Cool Idea had a significant dependency on AS6939, but this dropped over the course of the attack, with other network providers, including AS3741 and AS174 (Cogent), seeing increased AS hegemony.

Correction

In the August 2019 Internet Disruption Report, we classified the Internet shutdown in the Jammu and Kashmir region of India as a sub-national shutdown that was not visible on publicly available monitoring tools. A CAIDA IODA team member alerted us that the disruption was in fact visible in their tool. The figure below shows a decline in the Active Probing metric on August 4, when the shutdown began, along with a corresponding drop in the Darknet metric. We thank the CAIDA team for the update, and apologize for the error.

Conclusion

September also saw a number of Internet disruptions where the cause was unclear – research did not uncover any associated local event or information published by a service provider on their Web site or social media presence. That latter point continues to be a source of frustration, with multiple service providers again failing to respond to requests made via social media for additional information about Internet disruptions observed to have impacted their networks.

Having said that, the underlying reasons for government-directed Internet shutdowns ultimately become well-known, and in early September, the New York Times published an article that examined both the human and economic costs of such shutdowns, looking at several that have recently occurred in African nations.

“The shutdowns do more than stunt the democratic process. They can batter whole economies and individual businesses, as well as drastically disrupt the daily life of ordinary citizens, turning the search for mobile service into a game of cat and mouse with the police and driving people across borders just to send emails for work.”

And although this has been in planning for quite some time, Russia will apparently soon begin testing its ability to effectively isolate the Russian Internet (RuNet) from the global Internet. Driven by legislation signed earlier this year, according to a published report, the legislation mandates that Russian Internet service providers have the technical means to disconnect from the rest of the world and reroute Internet traffic through exchange points managed by Roskomnadzor, Russia’s telecommunications and media regulator. If they are successful in implementing such a system, tests and/or activation will likely result in Internet disruptions being visible in popular monitoring tools not only for Russia, but also neighboring countries that route traffic through Russia.

Today, Internet connectivity is significantly more ubiquitous, faster and less expensive (in most places), and generally reliable. With increased Internet availability and usage, however, disruptions become more noticeable, and impact a significantly larger population of users. In August, we observed Internet disruptions around the world due to power outages, national exams, and network issues. Several government-directed disruptions were widely reported as well, but were not easily observable in monitoring tools.

Power Outages

On Sunday, August 4, a significant power outage occurred in Indonesia at midday local time (05:00 GMT), lasting approximately 12 hours. The power outage impacted tens of millions of people in Indonesia’s capital Jakarta, and nearby provinces and cities, as well as affecting mobile phone networks.

The figures below illustrate the impact of the power outage at a country level. Both the Oracle Internet Intelligence and CAIDA IODA graphs show a decline in successful active probing measurements concurrent with the start of the power outage, recovering about 12 hours later. Fluctuations in DNS traffic (Oracle) and darknet sources (CAIDA) can also be seen in the respective graphs. BGPStream reported outages across several Indonesian autonomous systems around the time the blackout started, but these networks appeared to recover quickly, as the changes seen in the BGP graphs were minimal and short-lived.

At a more local level, the impact of the power outage is also visible in CAIDA IODA graphs for the provinces of Jakarta Raya and Jawa Barat, as shown in the figures below.

Early in the evening local time on August 22, a multi-hour Internet disruption was observed in Gibraltar, impacting both active and traffic measurements, as seen in the figures below.

The disruption was apparently caused by a widespread power outage, as the Gibraltar Electricity Authority apologized on their Facebook page later that day for the disruption in service and the inconvenience it caused.

National Exams

Continuing with the set of exam-related disruptions experienced at the end of July, Syria saw an another five disruptions during the first week of August. The figures below show that these were nearly complete outages, with active probing and traffic-related measurements dropping to nearly zero during the exam periods. Additionally, the spikes seen in DNS traffic in the Oracle Internet Intelligence graphs indicates that the outage was asymmetrical — DNS traffic could apparently get out of the country, but there were no routes available for traffic to return to those resolvers.

During the last week of the month, several exam-related disruptions were observed in Iraq. However, in contrast to Syria, these disruptions were more limited in severity, as illustrated in the figures below – none of the metrics experienced near-complete outages.

The Iraqi Ministry of Education is also apparently experimenting (successfully) with sending questions to exam centers electronically, in an effort to eliminate the opportunity for test questions to leak.

“The Information Office stated that the committee provided some exam centers with internet-enabled devices that withdraw and reproduce questions within one minute. In order to eliminate the leakage of questions, indicating that the experience was applied in the Directorate General of Education Baghdad Rusafa second and the General Directorate of Education Wasit and proved successful (100%).“

http://www.moedu.gov.iq/index.php?name=News&file=article&sid=5407 (via Google Translate)

Network Issues

On August 9, Fing Internet Alert’s @outagedetect Twitter account posted an alert about a major outage detected on TekSavvy, a network provider in Ontario, impacting a number of cities in the Canadian province. TekSavvy acknowledged the outage on the company’s network status Twitter account:

The impact of the outage was barely visible in country-level graphs, with just slight perturbations seen in the CAIDA IODA graph for Canada shown below.

As expected, however, the disruption is clearly evident in a CAIDA IODA graph for TekSavvy’s autonomous system, with the graph below showing a marked drop in the Active Probing and Darknet metrics.

Published reports indicated that Canadian network service provider Rogers also experienced an outage at the same time, and a Tweet from TekSavvy suggested that the two issues were related, noting that all “Third Party Internet Access” networks were impacted, including their connection to Rogers, their upstream provider.

Customers of network service provider du in Dubai experienced an hour-long disruption to their Internet service on August 20. Although the disruption only had a nominal impact at a country level, as shown in the United Arab Emirates figure below, it was significantly more visible in the graphs shown below for AS15082, du’s autonomous system.

A published report highlighted the multiple customer complaints sent to the company via Twitter (@dutweets), with responses noting that the problem was being addressed. Although the company did not provide a root cause for the outage in these responses, the published report quoted a company statement claiming that the problem occurred with its fixed line broadband service.

Three days later, on August 23, Fing Internet Alert’s @outagedetect Twitter account highlighted an Internet disruption taking place in Iowa:

Fing’s monitoring, shown in the figure below, provides a localized view of the disruption, showing that it impacted multiple cities in Iowa for approximately three hours. The disruption was also visible in the CAIDA IODA graph for AS209 (labeled as Qwest Communications, which merged with CenturyLink in 2011) shown below. While hard to see, the disruption caused a slight decline in successful active probing between approximately 15:00 to 18:00 GMT.

CenturyLink ultimately noted that the Internet disruption was due to a fiber cut:

And in a rare bit of good news, Greenland’s TELE-POST noted on August 22 that the repair saga for the Greenland Connect cable had finally came to a conclusion. As reported in May’s Internet Disruption Report, the cable initially suffered damage in December 2018. Some repairs were done in early May, but progress was slow because of unfavorable weather conditions. TELE-POST’s Technology and IT Director Jonas Hasselriis stated:

“The repair of the submarine cable is now over, and we have done everything we can to minimize the impact on our customers. We are now reopening the connections, and we are all reminded how important it is that we all take good care of the search cables. We thank the customers many times for their patience and regret that this was necessary.”
https://telepost.gl/da/nyheder/reparationerne-af-soekablet-er-nu-slut-tele-post-genaabner-forbindelserne-i-sydgroenland via Google Translate

Government Directed

While many government directed disruptions have a nation-wide impact on Internet connectivity, making them easily observable in publicly available monitoring tools, there can often be issues with detecting/observing sub-national disruptions with these tools. This may be due to a targeted shutdown of mobile services and/or use of Carrier-Grade Network Address Translation (known as CGN) by the impacted providers — which means there is limited IP address space being tested through active measurements, resulting in a shutdown being largely invisible to tools that employ such techniques, such as Oracle’s Internet Intelligence or CAIDA IODA. However, loss of signaling from within the impacted networks can indicate a disruption — Fing’s OutageDetect measurement tool uses this approach. However, their visibility is ultimately limited by the footprint of their Fingbox deployment. (Depending on the size of the user population impacted by the shutdown, the disruption may be visible in tools that show Web traffic levels, such as Google’s Transparency Report.) In August, three such government-directed Internet disruptions were in the news, but not evident in monitoring tools.

A post on the Russian-language Internet Protection Society Web site covered an Internet shutdown that occurred at the direction of the authorities in Moscow on August 3. The post notes that in order to prevent demonstrators at a rally taking place that day from coordinating actions, law enforcement agencies ordered mobile operators to disable mobile data services. While the shutdown was reportedly supposed to last ten hours, local measurements cited in the post indicate that the duration may have been just a few hours.

On August 4, the Indian government effectively shut down all telecommunications within the Jammu and Kashmir region, including cutting off all Internet connectivity, mobile service, and even land lines. This was done as the government moved to revoke Kashmir’s autonomy, which was granted under Article 370 of India’s Constitution. While this shutdown was obviously newsworthy, it was not the first time it has happened in the region – tracker internetshutdowns.in counts 54 previous shutdowns in the region in 2019 alone.

The Indonesian government was reported to have disrupted Internet services in the province of West Papua on August 22, following the deployment of additional military and police personnel to the region to help to quell ongoing protests and secure vital public facilities. Indonesia’s information ministry said that the ‘temporary’ shutdown was meant ‘to accelerate government effort to restore order’, and noted that only [mobile] data services had been disrupted, with texting and phone calls unaffected.

Conclusion

The August 2019 (U.S.) issue of Wired Magazine contained just a single Internet Service Provider advertisement (for Comcast Business), but most of the articles and advertisements involve the Internet in some way, showing just how ubiquitous it has become over the last quarter century. However, as we see on a regular basis, Internet access and access to content, applications, and services, are frequently disrupted by weather, power outages, infrastructure failure or damage, and government direction. Impacts of the former three can potentially be mitigated, at least in part, through redundant systems and infrastructure, but it is the government directed Internet disruptions that are arguably the greatest threat. More and more of these disruptions are occurring at the sub-national level, escaping the visibility of publicly available tools. While tools such as NetBlocks and Monash University’s IP Observatory have some visibility into these disruptions, they currently lack both a public interface and public disclosure of their measurement methodologies.

National Exams

After ending June with regional Internet disruptions reportedly intended to prevent student cheating on national Physics exams in Iraq, there were six additional nationwide disruptions observed over the first half of July, as seen in the figures below. Multi-hour disruptions took place on July 1, 4, 7, 10, 13, and 15. Published reports indicate that the Internet was again disrupted at the request of the education ministry in an effort to prevent student cheating. Iraq has been implementing such tactics since 2015.

After five exam-related Internet disruptions were observed in Syria during the first three weeks of June, another four were observed across the last days of July, as seen in the figures below. The impact is shown at a country level, as well as for AS29256 (Syrian Telecom), the state-affiliated telecommunications provider. Although no specific documentation of these disruptions being related to national exams could be found online (either in published reports or on the Syrian education ministry Web site), their frequency and duration closely resembles similar verified disruptions seen in the past. As such, they are being categorized as exam-related.

Power Outage

In its first major power outage since March, Venezuela suffered a massive blackout on July 22, plunging more than half of Venezuela’s 23 states into darkness. The figures below show that power failed around 20:00 GMT, ultimately disrupting Internet connectivity for more than a day. As illustrated in the Oracle Internet Intelligence and CAIDA IODA graphs, the active measurements (traceroute & ping) dropped immediately and took a significant amount of time to recover; the passive (user traffic) measurements (DNS, darknet, Google Web search) exhibited the same behavior. However, only minor churn is evident in Oracle’s and CAIDA’s BGP-based metrics, indicating that the routes to networks within the country remained available, even if endpoints on those networks were unreachable.

The @outagedetect Twitter account, associated with Internet connection and connected device monitoring company Fing, also observed the disruption within its monitoring of subscriber devices on CANTV, Venezuela’s state-run Internet service provider and posted the Tweet below:

Government Directed

Also on July 22, a fire broke out at the Kondengui Central Prison in Cameroon’s capital city of Yaounde after detainees held a protest in the overcrowded detention facility. A published report noted “The internet was shut down while a drone was flown over the prison.” This shutdown is likely the root cause of the Internet disruptions shown in the graphs below, for both Cameroon at a country level, as well as for AS30992 (MTN Cameroon), a major network provider within the country.

Severe Weather

While this blog generally looks at country-scale Internet disruptions, the network impact of severe weather events is much more localized, wreaking havoc and causing connectivity problems across a much smaller geographic area.

On July 23, Fing’s @outagedetect Twitter account highlighted a “considerable” outage impacting customers of network provider Optimum Online in New York & New Jersey.

Fing’s detail view showed that the disruption started at 20:45 (ET) on July 22, and lasted for nearly 10 hours. The disruption can also be seen in the CAIDA IODA graph below, with drops in the Active Probing and Darknet metrics seen for several hours, roughly aligned with the orange portion of the Fing graph. (Optimum Online is the high-speed Internet brand of Cablevision Systems.)

The disruption coincided with heavy rainstorms that hit the greater New York City area, and Optimum Online’s @OptimumHelp Twitter account confirmed that the observed outages were, in fact, storm-related.

On July 23, powerful storms hit several towns on Cape Cod in Massachusetts, spawning multiple tornadoes. At 12:12 (ET), a Boston-area weather forecaster Tweeted that the towns of South Brewster, East Harwich, and South Orleans were next up in the storm’s path. Just a few minutes later, Fing’s @outagedetect account Tweeted about an Internet outage impacting Comcast users in those towns. The associated detail view indicated that the disruption lasted for nearly 11 hours.

Fiber/Cable Issues

Late in the day (GMT) on July 7, the Gambia suffered a fairly significant multi-hour Internet disruption, as seen in the figure below. The BGP metric dropped to near-zero, indicating that a few of the country’s network prefixes were still being actively routed, which accounts for the continued traceroute and DNS activity.

The next morning, Gambia Telecommunications Company (Gamtel) noted that the observed issue was related to technical issues at the gateway to the ACE (Africa Coast to Europe) submarine cable. While the country has other terrestrial international Internet connections, the ACE cable is its sole submarine connection. The impact to Gamtel’s Internet connectivity is shown in the figure below — latency spiked as traceroutes to endpoints within the country failed over to a connection through Sonatel, as paths through the other carriers became unavailable.

Earlier that same day, Swaziland also suffered an Internet disruption, as seen in the figure below. Although the traceroute and BGP metrics only showed a slight decline, the DNS query rate dropped significantly.

A significant disruption was observed at the same time at AS327765 (Swazi MTN), a Swazi telecommunications company. The figure below shows that during the disruptions, traceroutes to endpoints within the network completely stop transiting AS19711 (Swaziland PTC). The next day, @MTNSwaziland Tweeted an apology for the prior day’s disruption, noting that it was due to a fiber optic “breakdown” at Eswatini Post & Telecom Corporation – the same network noted above. (Swaziland was renamed “the Kingdom of eSwatini” in 2018.)

Unattributed Causes

In the Internet disruptions reviewed above, the root causes were discoverable thanks to news coverage of an associated event or a social media post by an impacted network service provider. However, there were a number of other disruptions observed across multiple countries in July where the root causes were unknown – no associated social media posts from impacted providers or known local events that could have had a significant impact to Internet connectivity. In addition, for many of these disruptions, the affected service providers did not respond to outreach on social media, including Twitter and Facebook.

United Kingdom

The Gambia

Togo

Congo

Gabon

Saint Barthelemy

Conclusion

Information and data on the severity, duration, and extent of Internet disruptions, both at national and local scales, is becoming increasingly available through organizations like Oracle Internet Intelligence, CAIDA IODA, Fing Internet Alert, the Google Transparency Report, Netblocks, and the Monash University IP Observatory. As such it is becoming easier to learn about and publicize such disruptions when they occur, and to hold the providers accountable for these disruptions.

As seen above and in past posts, many providers are transparent, and use social media to explain (and sometimes even apologize for) what happened. However, many providers not only fail to proactively provide information to their subscribers and the public when such events occur, but they also fail to reply to inbound requests for additional information. (Compounding the problem, some have social media presences that have been inactive for months or years.)

We call on telecommunication and network service providers to be more transparent and more active on social media with respect to providing insights into the causes of observed disruptions, and to actively monitor their social media presence for inbound requests for information. (And of course, to respond to those requests in a timely fashion).

Political Unrest

Unfortunately, during periods of political unrest, national governments have taken to implementing Internet shutdowns and social media filtering/blocking as a means of preventing the spread of (mis)information and limiting citizen ability to organize protests and similar activity.

On June 3, Internet connectivity in Sudan was disrupted in conjunction with attacks on protestors – the protests were related to the transition of power after the removal of Omar al-Bashir as Sudan’s president in April. The figure below shows drops in all three metrics measured by Oracle Internet Intelligence occurring around noon GMT.

Three weeks later, the Internet was restored… but only for a single lawyer that had filed a lawsuit against telecommunications provider Zain Sudan over the disruption ordered by Sudan’s military rulers. However, on July 9, local telecommunications providers were ordered to restore Internet service. This service restoration can be seen in the figure below, with increases in both the traceroute and BGP-based metrics.

An Internet disruption took place in Ethiopia on June 21 in the wake of an attempted coup, with the impacts visible in the figures below. Active inbound measurements by Oracle Internet Intelligence and CAIDA IODA saw success rates drop significantly as a result of the Internet shutdown, while passively measured DNS (Oracle), Darknet (CAIDA), and Web (Google) traffic dropped to near-zero levels. Connectivity was restored on June 27.

Following a contested presidential election in Mauritania, an Internet shutdown was put into place at approximately 15:30 GMT on June 25 as seen in the CAIDA IODA figure below – a significant drop in routed network prefixes occurred for several hours, while Active Probing and Darknet traffic sources dropped significantly.

Network diversity within the country is extremely limited. Looking at the figure for AS29544 (Mauritanian Telecommunication Company) below, it is clear that the routing instability in the country-level graph is related to this network’s routes essentially being removed from the Internet for that multi-hour period. Internet connectivity began to return across the country on July 3.

Cable Damage

Damage to terrestrial and undersea cables has long been the nemesis of a properly functioning Internet. The Internet disruptions due to cable damage observed during the first and last weeks of June tended to be brief and/or localized; similar issues seen in the past have caused significant country-wide Internet disruptions.

On June 4, Filipino telecommunications provider Globe Telecom posted a Public Advisory on its Facebook page that noted (in part): “Between 1-2PM today, the Globe Network experienced multiple fiber cuts, two of which were submarine cables.” The impact of these fiber cuts is clearly evident in the figures below for AS4775 and AS132199, both network identifiers associated with Globe Telecom in the Philippines, with multi-hour drops visible in both the Active Probing and BGP metrics. A slight perturbation to these metrics was barely visible in the country-level graph. A published report noted that connectivity was restored about four hours after the issue started, in line with the recovery time seen in the graphs.

On June 6, Internet monitoring and advocacy organization Netblocks Tweeted about an Internet outage in Venezuela that affected local network providers Digitel and Supercable. Netblocks noted that Digitel was completely disrupted while Supercable only experienced a partial outage. This is in line with the figures below, showing all metrics dropping to zero at Digitel starting around 15:00 GMT, and only a nominal decline in the Active Probing metric for Supercable around the same time. The Netblocks tweet stated that “sources report fiber cut”, but this root cause was not independently confirmed.

A brief Internet disruption observed in Pakistan on June 24 was due to multiple cable cuts, according to a press release issued by Pakistan Telecommunication Company Limited (PTCL). The figure below from CAIDA IODA shows a clear drop in the Active Probing metric at a country level, along with a slight dip in Darknet traffic sources.

The impact of the cable cuts can also be seen at a network level, visible in the two figures below for PTCL-owned networks.

The overall impact of the cable cuts appeared to be nominal, and the disruption short-lived, with metrics returning to normal levels within approximately four hours. @PTCLOfficial announced the full restoration of service just a few hours after the disruption began:

Iran is no stranger to politically-motivated Internet disruptions, and country leaders have spoken in the past about disconnecting the country from the global Internet. However, a brief disruption that occurred on June 26, shown in the figure below, may have instead been related to physical layer problems. A Tweet posted by Iranian ICT minister Azari Jahromi suggested that “a disruption of fiber connectivity” at upstream routes in Europe between Bulgaria & Frankfurt, which are used by several ISPs in Iran, caused the 1.5 hour disruption in connectivity.

National Exams

The governments of Syria, Ethiopia, and Iraq have all ordered Internet shutdowns in the past to prevent student cheating on national exams, and all three countries experienced such disruptions once again in June.

First up was Syria, where the first observed disruption, between 01:00 – 05:30 GMT on June 9, took place in conjunction with 712,000 university students starting their second term exams, along with 9th grade (O-Level) and 12th grade (A-Level) students starting public exams as well. Additional disruptions were also seen on June 11, 13, 17, and 19, all effectively appearing as complete country-wide outages in the Oracle Internet Intelligence, CAIDA IODA, and Google Transparency Report Traffic figures below.

It is interesting to note that within the Oracle Internet Intelligence Map graphs, the Traceroute Completion Ratio and BGP Routes metrics drop to zero during the periods of disruption, but the DNS Query Rate metric spikes. Doug Madory, Director of Internet Analysis at Oracle Internet Intelligence, explained why this happened:

On June 9, the first of four Internet disruptions apparently related to exams took place in Ethiopia, and lasted approximately six hours. A second disruption on June 11 lasted five hours, from 07:00 to 12:00 GMT. The third disruption, lasting approximately 12 hours, started in the evening of June 11, carrying over to the next day. The final disruption started the evening of June 12, and lasted 39 hours, until mid-day on June 14. The impact of these disruptions to connectivity in Ethiopia can be seen in the figures below — all caused a significant, but not complete, loss of connectivity at a country level. Published reports claimed that the outages were intended to prevent student cheating on national exams.

Ethio Telecom acknowledged, and apologized for, the disruptions in a Tweet, but neither the telecommunications company nor the government confirmed the reason they occurred.

Finally, on June 26, Iraq reportedly implemented regional Internet shutdowns intended to prevent student cheating on national Physics exams. The country-level CAIDA IODA figure for Iraq below shows a brief, and limited, drop in successful Active Probing late in the morning (GMT).

The CAIDA IODA figure for Sala ad-Din, a region of Iraq just north of Baghdad, shows a brief drop in the number of routed prefixes geolocated to the region. The drop occurs concurrent with the disruption seen at a national level.

The disruption is also visible at a network level, as the CAIDA IODA figure below for AS50710 (Earthlink) shows. Earthlink is one of the largest network service providers in Iraq, and during the period of disruption, a drop in the Active Probing metric can be seen.

Power Outage

On June 16, a massive power outage, called “unprecedented in its scope” by local officials, impacted all of mainland Argentina and Uruguay, affecting nearly 48 million people. The outage started at approximately 10:00 GMT, with near immediate impact on Internet connectivity to the affected countries and networks within them, as shown in the figures below. While local power utilities quickly began restoring power, Oracle Internet Intelligence and CAIDA IODA measurement metrics took nearly 12 hours to return to near-normal levels.

Electrical utilities in both countries provided status updates on Twitter throughout the event – the @UTE_ComCorp Tweet below indicated that 98.5% of power had been restored in Uruguay, approximately 11 hours after their first Tweet regarding the power failure, and the Tweet regarding restoration of service in Argentina from @OficialEdesur followed a similar timeline.

Route Leaks

Route leaks remain a common occurrence on the Internet. However, adoption of basic safeguards necessary to prevent propagation of routing leaks, such as the Internet Society’s Mutually Agreed Norms for Routing Security (MANRS) project, can mitigate the impact of such leaks. Unfortunately, neither China Telecom nor Verizon have apparently yet implemented such safeguards, and both carriers were implicated in massive route leaks that resulted in increased latency and/or service disruptions in June.

The first leak was observed on June 6, when Swiss data center colocation company Safe Host SA (AS21217) leaked over 70,000 routes to China Telecom (AS4134) in Frankfurt, Germany. An excellent and detailed blog post from Oracle’s Doug Madory explained the impact of the leak:

“China Telecom then announced these routes on to the global Internet redirecting large amounts of Internet traffic destined for some of the largest European mobile networks through China Telecom’s network. Impacts were seen by some of Europe’s largest networks in Switzerland, Holland, and France among other countries. … Often routing incidents like this only last for a few minutes, but in this case many of the leaked routes in this incident were in circulation for over two hours.”

The impact of the leak from a traceroute and latency perspective was illustrated in a Tweet from @InternetIntel:

The second notable route leak occurred on June 24, impacting approximately 20,000 prefixes from 2,400 network providers, according to BGPmon. Content delivery and security service provider Cloudflare was significantly impacted by the routing leak, and it too published an excellent and detailed blog post explaining what happened.

“An Internet Service Provider in Pennsylvania  (AS33154 – DQE Communications) was using a BGP optimizer in their network, which meant there were a lot of more specific routes in their network. Specific routes override more general routes… DQE announced these specific routes to their customer (AS396531 – Allegheny Technologies Inc). All of this routing information was then sent to their other transit provider (AS701 – Verizon), who proceeded to tell the entire Internet about these “better” routes.”

In addition to Cloudflare, this route leak impacted connectivity for many other major organizations. BGPmon’s BGPStream Web site publishes alerts about BGP hijacks, leaks, and outages, and the table below is based on data extracted from the BGPStream site for alerts associated with this leak. It is clear in looking at the list that there was a global impact, with networks in Europe and the Asia Pacific region among those affected.

Localized Disruptions

While these blog posts generally cover large-scale, country-level disruptions, easily identifiable within publicly available monitoring tools, more localized disruptions are often visible through these tools as well.

On June 7, Consolidated Communications, a network provider that services Vermont, reportedly experienced a “router issue” at its office in Burlington, disrupting connectivity to all 80,000 Internet customers in the state, including businesses and homes. The figures below show the impact of the disruption, with active measurement metrics declining for a brief period of time. Although the autonomous system label in the graphs is for FairPoint Communications, the related PeeringDB entry shows it is owned by Consolidated Communications. (FairPoint was acquired by Consolidated in 2016.)

The @outagedetect Twitter account, associated with Internet connection and connected device monitoring company Fing, posts numerous alerts each day about localized Internet outages. On June 19, it posted the Tweet below, highlighting the end of a “considerable” hour-long Internet outage detected in Poland for users connected to network service provider Orange Polska.

One of the figures below shows that this “considerable” outage impacted CAIDA’s Active Probing measurements for Poland at a country level, as well as driving a concurrent decline in Darknet sources. The other figure shows the impact at a network level, with Active Probing measurements and Darknet sources for Orange Polska dropping considerably during the hour-long disruption.

Conclusion

While the major Internet disruptions observed in June were largely due to a familiar set of underlying causes, many of these causes remain avoidable.

Of course, convincing governments not to implement Internet outages during times of political unrest is a non-trivial effort, but organizations including the Internet Society and Access Now continue to advocate for user rights and drive #KeepItOn efforts.

In contrast, convincing network providers, especially Tier 1 global providers, to improve their routing hygiene through through participation in efforts like MANRS should be comparatively trivial.