(Apologies for posting this month’s update a few weeks later than usual – sometimes life and work get in the way of blogging…)

Power Outages

We have observed Internet disruptions in Venezuela in the past that were caused by the country’s unstable electrical power infrastructure, and October was no different. The figure below shows the impact of power outages that occurred on both October 1 & 2, both just before 1800 UTC. On both days, the BGP metric remains stable (indicating that the systems announcing the routes were unaffected by the power problems), but the Active Probing and Darknet metrics saw declines due to loss of connectivity within the last mile networks that the probes target and that originate the Darknet traffic.

In Gibraltar, the country’s electricity authority took to Twitter to alert customers to an ongoing power outage, and to apprise them of the cause of the outage. The figures below show that the power outage likely started about a half hour before the Tweets were published, with a country-level Internet disruption in Gibraltar starting just after 1100 UTC. Connectivity returned to normal about three hours later.

In California, regional power company Pacific Gas & Electric (PG&E) once again implemented “Public Safety Power Shutoffs“, noting:

“For public safety, it may be necessary for us to turn off electricity when gusty winds and dry conditions, combined with a heightened fire risk, are forecasted. … While customers in high fire-threat areas are more likely to be affected, any of PG&E’s more than five million electric customers could have their power shut off.”

These power outages obviously have the ability to impact local Internet connectivity. As one such example (out of several over the course of October), network monitoring firm Fing observed a disruption at regional service provider Valley Internet (the d/b/a name of Personal Network Computing) lasting for more than half a day, as shown in the Tweet below from Fing’s @outagedetect account, as well as the figure below. The locations of the measured Internet service disruption aligned with the map of expected affected areas posted to the PG&E Outage Center on October 9, providing a high likelihood of correlation between the two events.

The localized network-level disruption was also observed through other network monitoring tools. The figures below show that both Oracle Internet Intelligence and CAIDA IODA recorded declines for the actively measured metrics during the course of the power outage and associated disruption.

Cable Cuts

On October 16, Sudan suffered a nearly three-hour Internet disruption. The figures below show a sharp decline in the three metrics measured by Oracle Internet Intelligence and CAIDA IODA.

A local contact reported that the disruption was caused by a “double cut” to a Sudatel fiber ring, and that MTN was also impacted by the fiber cut. The figures below illustrate the impact of the cut to both Sudatel and MTN Sudan.

And while we don’t often cover it here, these Internet disruptions do impact local users and businesses. As one example, CashQ, a mobile payment application, was unable to provide services to its users as a result of this Internet disruption in Sudan, as it explained on Twitter.

A week later, a problem with the SEACOM Subsea Cable System disrupted Internet connectivity in Kenya and Comoros on October 23. In a series of Tweets (1, 2, 3), @SEACOM stated that the outage began at 2200 UTC on October 22, impacting its subsea cable system between Mombasa [Kenya] and Zafarana [Egypt], and forcing it to route traffic over West coast transmission links. The figures below show that both Oracle Internet Intelligence and CAIDA IODA detected a minor perturbation in the BGP metric at a country level for Kenya around the time that the outage occurred. Oracle’s measurements also recorded a decline in successfully completed traceroutes to endpoints in the country, while a nominal impact was seen in the Darknet sources metric measured by CAIDA.

One Communication/Safaricom is a leading telecommunications provider in Kenya, and is clearly reliant on the SEACOM cable for international Internet connectivity. The Oracle Internet Intelligence graph in the figure below shows a significant drop in completed traceroutes to endpoints within the network, while the CAIDA IODA graph shows a near complete loss of visible networks associated with the autonomous system visible in the routing table. CAIDA also shows a concurrent drop in the Darknet Sources metric, indicating that systems within that network were no longer probing the address space used by CAIDA’s Network Telescope.

Impacted networks in Kenya shifted upstream providers during the cable outage, resulting in increased latency for traffic to these networks. The figures below illustrate the latency impact for two networks: Safaricom saw latency increase by approximately 50% as it shifted to Emirates Telecommunications as its primary upstream provider, while Kenyan Post/Telkom Kenya saw latency increase by approximately 10-20% as it moved to Emirates Telecommunications.

As noted above, Comoros was also impacted by this problem with the SEACOM submarine cable. The Oracle Internet Intelligence figure below shows a significant impact to the Traceroute Completion Ratio metric, and a very slight impact to the BGP metric (with the exception of a sharp but brief drop). The CAIDA IODA figure below shows a somewhat more significant impact to the BGP metric, with a similar sharp but brief drop towards the end of the disruption period.

Comores Telecom was also heavily affected, with both Oracle and CAIDA showing a near complete outage for the networks, as seen in the figures below.

Pakistan experienced an Internet disruption on October 29, due to a fault in an international submarine cable, as explained in a Tweet from @PTAofficialpk, a Twitter account associated with the Pakistan Telecommunication Authority. The figure below shows the impact on CAIDA IODA’s Active Probing to endpoints within the country. There was no obvious routing impact, and Darknet traffic also appeared to be unaffected.

According to a subsequent Tweet from the Pakistan Telecom Company (PTCL), the cable fault occurred on the SMW-4 cable. The figures below show the impact across two of PTCL’s autonomous systems, with Oracle Internet Intelligence showing increased latencies as more traffic flowed over Cogent as an upstream provider, while CAIDA IODA showed a decline in the Active Probing metric.

A published article highlighted a local impact of this Internet disruption, quoting a Tweet from Pakistan International Airlines (PIA) that explained that the the issues that the cable cut caused for PTCL ultimately resulted in an outage of PIA flight check-in systems across the country, forcing them to move to backup systems and manual processes and resulting in passenger delays.

Government Directed

The most significant Internet disruption that occurred in October took place in Iraq, with connectivity impaired for over a week. Published accounts (AlMonitor, CPJ, AccessNow) reported violent protests over unemployment, government corruption, and a lack of basic services in Baghdad and other Iraqi cities on October 1-2, resulting in over a dozen deaths and hundreds of injuries. In response to the protests, the Iraqi government imposed a near total Internet shutdown on October 2, apparently beginning at approximately 1600 UTC, as shown in the figures below. The figures also illustrate that the shutdown did not completely remove Iraq from the Internet. This is reportedly due to the Kurdistan Regional Government’s (KRG) Ministry of Transportation and Communications rejection of the call by the Iraqi federal government to cut off the Internet in the Kurdistan Region. The Kurdistan region’s connection to international communications infrastructure is separate from the one in southern and central Iraq.

As the figures below show, the Internet disruption lasted for over a week, with measured metrics returning to “normal” levels on October 11. After a few brief restorations of connectivity on October 3 and 6, Internet availability settled into a “curfew” model, as described by NetBlocks, with access returning during the workday, and then being cut again in the evening.

Unsurprisingly, trends at a network level are nearly identical to the activity seen at a country-level. The figures below show measurements across three major Internet service providers in Iraq, with BGP and Active Probing patterns that appear very similar to the ones seen in the country-level graph above.

One published report explained that some Iraqis initially attempted to use VPN technology to circumvent connectivity restrictions, while those that could afford to made use of satellite-based Internet connectivity. Others queued content posts and videos within the social media and chat applications on their mobile devices, with the expectation that they would be sent when these devices were finally able to connect to the Internet.

Based on measurements done by Geoff Huston at APNIC Labs, it also appears that some Iraqi users are configuring third-party public DNS resolver services as backups in the event that the resolvers at their providers (generally configured as the default) become unavailable. Based on Huston’s experiments, the figure below shows that prior to October’s Internet disruption, approximately 65% of DNS requests made by Iraqi users reached authoritative DNS servers through a resolver in the same autonomous system (“sameas”) – that is, the user’s ISP. Approximately 30% were made through Google’s 8.8.8.8 resolver (“googlepdns”), while less than 10% were made through CenturyLink/Level 3’s open public resolver (“level3”). However, for a multi-day period after the disruption started, the percentage of DNS requests from Iraqi users coming through Google surpassed those coming from resolvers at the user’s service provider. Usage of Level 3’s resolver also increased during this period. The graph shows that request volume through ISP resolvers returned to previous levels in line with the restoration of connectivity in Iraq. (It was pointed out that the conclusion reached here was incorrect: “You need to look at that in terms of absolute values. The relative values are biased, as the denominator changed. Those affected by the shutdown stopped reporting. In this case, The Kurdish ISPs were not shutdown. It may be the case that the distribution in those networks is different, and that was the reason for the shift, and not actual changes by the users.” The original text and figure remain for transparency, but should be considered deprecated.)

As noted in the introduction, this shutdown had a significant impact on the Iraqi economy, with an article in the International Business Times placing the estimated impact at nearly a billion dollars. The article highlights the financial pain experienced by members of the emerging digital economy in Iraq, including startups, e-commerce sites, online delivery companies and couriers, and travel companies.

Other

Similar to previous months, several additional less-well attributed Internet disruptions were observed in October. For some, we could observe the disruption but could find no root cause information, or did not receive any response to social media outreach to impacted providers. For those included below, acknowledgement of the issue was made by the providers through their social media channels, even if few details were given.

On October 14, SamoaTel/Bluesky Samoa informed users via Facebook and Twitter of an disruption to fixed Internet services. Unfortunately, no specific information about what caused this disruption was provided. The figures below show that while a nearly complete outage was seen at a network level, the impact was less significant at a country level.

Also on October 14, Telekomn Brunei (TelBru) apologized to users via Twitter for downtime for broadband services apparently caused by a third-party service provider. The figures below show that the disruption was partial at both a country and network level, and fairly brief, lasting just a couple of hours. To its credit, TelBru regularly posts updates to Twitter (@telbru_bn) alerting customers of upcoming downtime to to maintenance activities.

Internet monitoring firm Fing tracked a 15+ hour Internet disruption at provider Three UK that started the evening of October 16, lasting until mid-day on October 17.

The figure below shows the impact of disruption at a network level, with drops seen in the Active Probing and Darknet metrics. @ThreeUK acknowledged the network disruption in a Tweet, but failed to provide any additional information about what happened.

And near the end of the month, Maldives provider Dhiraagu alerted users via Twitter and Facebook to service degradation on mobile & fixed broadband services.

The figures below show that, similar to the TelBru issue above, the disruption was partial at both a country and network level, and fairly brief, again lasting just a couple of hours.

Conclusion

While both Internet outages and Internet shutdowns rob users of the ability to communicate with others, purchase goods and services, and consume media, there are significant differences between the two types of disruptions. In general, outages are unintentional or accidental disruptions to Internet connectivity – in the case they are intentional, it is generally due to infrastructure or network maintenance. Regardless, they are generally short lived, as impacted providers scramble to address the underlying issues as much as they have the ability to.

However, Internet shutdowns are the intentional disruption of Internet connectivity at the direction of a local, state, or national government, and often last for several days, if not weeks or months. These shutdowns have significant human rights impacts for citizens of the affected country, including limiting freedom of expression online, as well as significant economic impacts, with estimated costs running into multiple millions of (US) dollars each day the Internet is unavailable, as well as reducing the likelihood of future international investment within the country.

With the world increasingly watching, governments need to explore alternative ways of addressing the underlying issues that lead to Internet shutdowns, instead of leveraging these actions as a blunt-force policy tool. #KeepItOn

Today, Internet connectivity is significantly more ubiquitous, faster and less expensive (in most places), and generally reliable. With increased Internet availability and usage, however, disruptions become more noticeable, and impact a significantly larger population of users. In August, we observed Internet disruptions around the world due to power outages, national exams, and network issues. Several government-directed disruptions were widely reported as well, but were not easily observable in monitoring tools.

Power Outages

On Sunday, August 4, a significant power outage occurred in Indonesia at midday local time (05:00 GMT), lasting approximately 12 hours. The power outage impacted tens of millions of people in Indonesia’s capital Jakarta, and nearby provinces and cities, as well as affecting mobile phone networks.

The figures below illustrate the impact of the power outage at a country level. Both the Oracle Internet Intelligence and CAIDA IODA graphs show a decline in successful active probing measurements concurrent with the start of the power outage, recovering about 12 hours later. Fluctuations in DNS traffic (Oracle) and darknet sources (CAIDA) can also be seen in the respective graphs. BGPStream reported outages across several Indonesian autonomous systems around the time the blackout started, but these networks appeared to recover quickly, as the changes seen in the BGP graphs were minimal and short-lived.

At a more local level, the impact of the power outage is also visible in CAIDA IODA graphs for the provinces of Jakarta Raya and Jawa Barat, as shown in the figures below.

Early in the evening local time on August 22, a multi-hour Internet disruption was observed in Gibraltar, impacting both active and traffic measurements, as seen in the figures below.

The disruption was apparently caused by a widespread power outage, as the Gibraltar Electricity Authority apologized on their Facebook page later that day for the disruption in service and the inconvenience it caused.

National Exams

Continuing with the set of exam-related disruptions experienced at the end of July, Syria saw an another five disruptions during the first week of August. The figures below show that these were nearly complete outages, with active probing and traffic-related measurements dropping to nearly zero during the exam periods. Additionally, the spikes seen in DNS traffic in the Oracle Internet Intelligence graphs indicates that the outage was asymmetrical — DNS traffic could apparently get out of the country, but there were no routes available for traffic to return to those resolvers.

During the last week of the month, several exam-related disruptions were observed in Iraq. However, in contrast to Syria, these disruptions were more limited in severity, as illustrated in the figures below – none of the metrics experienced near-complete outages.

The Iraqi Ministry of Education is also apparently experimenting (successfully) with sending questions to exam centers electronically, in an effort to eliminate the opportunity for test questions to leak.

“The Information Office stated that the committee provided some exam centers with internet-enabled devices that withdraw and reproduce questions within one minute. In order to eliminate the leakage of questions, indicating that the experience was applied in the Directorate General of Education Baghdad Rusafa second and the General Directorate of Education Wasit and proved successful (100%).“

http://www.moedu.gov.iq/index.php?name=News&file=article&sid=5407 (via Google Translate)

Network Issues

On August 9, Fing Internet Alert’s @outagedetect Twitter account posted an alert about a major outage detected on TekSavvy, a network provider in Ontario, impacting a number of cities in the Canadian province. TekSavvy acknowledged the outage on the company’s network status Twitter account:

The impact of the outage was barely visible in country-level graphs, with just slight perturbations seen in the CAIDA IODA graph for Canada shown below.

As expected, however, the disruption is clearly evident in a CAIDA IODA graph for TekSavvy’s autonomous system, with the graph below showing a marked drop in the Active Probing and Darknet metrics.

Published reports indicated that Canadian network service provider Rogers also experienced an outage at the same time, and a Tweet from TekSavvy suggested that the two issues were related, noting that all “Third Party Internet Access” networks were impacted, including their connection to Rogers, their upstream provider.

Customers of network service provider du in Dubai experienced an hour-long disruption to their Internet service on August 20. Although the disruption only had a nominal impact at a country level, as shown in the United Arab Emirates figure below, it was significantly more visible in the graphs shown below for AS15082, du’s autonomous system.

A published report highlighted the multiple customer complaints sent to the company via Twitter (@dutweets), with responses noting that the problem was being addressed. Although the company did not provide a root cause for the outage in these responses, the published report quoted a company statement claiming that the problem occurred with its fixed line broadband service.

Three days later, on August 23, Fing Internet Alert’s @outagedetect Twitter account highlighted an Internet disruption taking place in Iowa:

Fing’s monitoring, shown in the figure below, provides a localized view of the disruption, showing that it impacted multiple cities in Iowa for approximately three hours. The disruption was also visible in the CAIDA IODA graph for AS209 (labeled as Qwest Communications, which merged with CenturyLink in 2011) shown below. While hard to see, the disruption caused a slight decline in successful active probing between approximately 15:00 to 18:00 GMT.

CenturyLink ultimately noted that the Internet disruption was due to a fiber cut:

And in a rare bit of good news, Greenland’s TELE-POST noted on August 22 that the repair saga for the Greenland Connect cable had finally came to a conclusion. As reported in May’s Internet Disruption Report, the cable initially suffered damage in December 2018. Some repairs were done in early May, but progress was slow because of unfavorable weather conditions. TELE-POST’s Technology and IT Director Jonas Hasselriis stated:

“The repair of the submarine cable is now over, and we have done everything we can to minimize the impact on our customers. We are now reopening the connections, and we are all reminded how important it is that we all take good care of the search cables. We thank the customers many times for their patience and regret that this was necessary.”
https://telepost.gl/da/nyheder/reparationerne-af-soekablet-er-nu-slut-tele-post-genaabner-forbindelserne-i-sydgroenland via Google Translate

Government Directed

While many government directed disruptions have a nation-wide impact on Internet connectivity, making them easily observable in publicly available monitoring tools, there can often be issues with detecting/observing sub-national disruptions with these tools. This may be due to a targeted shutdown of mobile services and/or use of Carrier-Grade Network Address Translation (known as CGN) by the impacted providers — which means there is limited IP address space being tested through active measurements, resulting in a shutdown being largely invisible to tools that employ such techniques, such as Oracle’s Internet Intelligence or CAIDA IODA. However, loss of signaling from within the impacted networks can indicate a disruption — Fing’s OutageDetect measurement tool uses this approach. However, their visibility is ultimately limited by the footprint of their Fingbox deployment. (Depending on the size of the user population impacted by the shutdown, the disruption may be visible in tools that show Web traffic levels, such as Google’s Transparency Report.) In August, three such government-directed Internet disruptions were in the news, but not evident in monitoring tools.

A post on the Russian-language Internet Protection Society Web site covered an Internet shutdown that occurred at the direction of the authorities in Moscow on August 3. The post notes that in order to prevent demonstrators at a rally taking place that day from coordinating actions, law enforcement agencies ordered mobile operators to disable mobile data services. While the shutdown was reportedly supposed to last ten hours, local measurements cited in the post indicate that the duration may have been just a few hours.

On August 4, the Indian government effectively shut down all telecommunications within the Jammu and Kashmir region, including cutting off all Internet connectivity, mobile service, and even land lines. This was done as the government moved to revoke Kashmir’s autonomy, which was granted under Article 370 of India’s Constitution. While this shutdown was obviously newsworthy, it was not the first time it has happened in the region – tracker internetshutdowns.in counts 54 previous shutdowns in the region in 2019 alone.

The Indonesian government was reported to have disrupted Internet services in the province of West Papua on August 22, following the deployment of additional military and police personnel to the region to help to quell ongoing protests and secure vital public facilities. Indonesia’s information ministry said that the ‘temporary’ shutdown was meant ‘to accelerate government effort to restore order’, and noted that only [mobile] data services had been disrupted, with texting and phone calls unaffected.

Conclusion

The August 2019 (U.S.) issue of Wired Magazine contained just a single Internet Service Provider advertisement (for Comcast Business), but most of the articles and advertisements involve the Internet in some way, showing just how ubiquitous it has become over the last quarter century. However, as we see on a regular basis, Internet access and access to content, applications, and services, are frequently disrupted by weather, power outages, infrastructure failure or damage, and government direction. Impacts of the former three can potentially be mitigated, at least in part, through redundant systems and infrastructure, but it is the government directed Internet disruptions that are arguably the greatest threat. More and more of these disruptions are occurring at the sub-national level, escaping the visibility of publicly available tools. While tools such as NetBlocks and Monash University’s IP Observatory have some visibility into these disruptions, they currently lack both a public interface and public disclosure of their measurement methodologies.