Internet-related media coverage in December tends to be e-commerce related, with discussions about how retail sites performed (or failed to) on Black Friday and Cyber Monday, followed by complaints about problems downloading software updates or games, and registering new connected devices after the holiday presents are unwrapped. However, when Internet disruptions occur, that coverage shifts to highlight the problems caused by the disruptions.

This final Internet Disruption Report post for 2019 is a long one, covering disruptions caused by a DDoS attack, power outages, cable/fiber/network issues, and government direction. Some countries make multiple appearances in this month’s report, and some have been featured in multiple reports throughout the year. In addition to observed disruptions, we also review Russia’s reported Internet disconnection test, as well as a few additional related observations.

DDoS Attack

Three months after a DDoS attack in September caused an Internet disruption at South African provider Cool Ideas, another DDoS attack targeting the provider again impacted connectivity. As shown in the Fing Internet Alerts and CAIDA IODA figures below, the attack appears to have occurred in several waves, with the first starting around 0815 GMT, lasting for approximately two hours, and the second starting around 1200 GMT, lasting for just over an hour.

Recognizing the problems that these attacks create for local providers, a published report highlighted how South Africa plans to fight similar DDoS attacks in the future:

South Africa’s Internet Service Providers’ Association (ISPA) has said that in an attempt to mitigate the severity of DDoS attacks against local ISPs, administrators at South Africa’s Internet exchanges are creating a “blackhole” that will funnel identified DDoS traffic through the exchanges into oblivion.

Power Outages

On December 4, NetBlocks posted a Tweet confirming that the third major power outage in Venezuela in just two days had disrupted Internet connectivity across multiple states. These disruptions are visible at a country level in the figures below, apparently occurring around 1830 GMT on December 3, 0645 GMT on December 4, and 1800 GMT on December 4.

A Tweet from @CorpoElecInfo (the Twitter account of Venezuela’s national electric company) that afternoon referenced system recovery efforts, presumably associated with the power outage.

NetBlocks listed fifteen Venezuelan state where Internet connectivity was disrupted by the power outage, and the figures below show the impact on three of those states. In Tachira and Zulia, the 0645 and 1800 GMT disruptions are clearly visible. However, in Aragua, there is a significant disruption evident starting around 0300 GMT that lasted over six hours – this could potentially be related to a more severe, localized power outage. The 1800 GMT disruption is visible for Aragua as well, although the impact is lower than was seen for Tachira and Zulia.

• 

• 

• 

CAIDA IODA also observed Internet disruptions in Venezuelan states including Aragua, Cojedes, and Guárico on December 29. However, there was no publicly available information associating these disruptions with a power outage event. (But they are highlighted here because of the frequency of power outage-caused Internet disruptions in Venezuela.)

Just two days before Christmas, a nationwide power outage struck the island nation of Malta, reportedly caused by a fault on the interconnector cable between Malta and Sicily. As illustrated in the Oracle and CAIDA IODA figures below, the Internet disruption caused by this power outage started around 0600 GMT, with connectivity gradually returning to normal after noon GMT. The Google Transparency Report figure below shows the impact that the disruption had on traffic from Malta to Google’s GMail application, with a significant decline seen between 0600-0630 GMT, and a clear recovery visible six hours later.

• 

• 

Melita, a major telecommunications provider in Malta, provided status updates to its customers via its Facebook page. The figures below show the power outage’s impact to Melita’s network, with the Fing Internet Alerts graph showing a complete loss of communication with local probes. The Oracle and CAIDA IODA graphs show a significant decline in the active probing metrics, and the CAIDA IODA graph shows the relative volume of Darknet source IPs in that network dropping to near zero, in line with Fing’s observation of an end-user connectivity outage.

• 

• 

Cable/Fiber Cuts & Network Issues

Late in the day (GMT) on November 30, Dhiraagu, the largest network service provider in the Maldives, posted an update to its Facebook page alerting customers to “unexpected technical issues” on its fixed broadband network. The impact of these issues lasted into December 1, and were visible at both a country and network level. The Oracle and CAIDA IODA figures below show that active probing measurements to endpoints in the country declined just before midnight (GMT), and started to recover approximately eight hours later.

• 

• 

The figures below illustrate the impact on one of Dhiraagu’s autonomous systems. The Oracle graph shows that latency for traceroutes to endpoints within the network increased significantly during the period of disruption. Although Dhiraagu posted a second Facebook update a little more that five hours after the first one alerting subscribers that the issue had been fully resolved, it is clear that connectivity took several additional hours to return to “normal” levels.

• 

• 

On December 1, NetBlocks reported that planned maintenance to a submarine cable disrupted Internet connectivity for multiple network providers in Venezuela. Several of these providers had alerted subscribers via Twitter that this maintenance would be taking place, apologizing in advance for the inconvenience that the lack of Internet connectivity would cause.

El cable submarino privado principal que conecta nuestro país con el resto del mundo, tendrá un mantenimiento programado el día de mañana, 01 de diciembre, desde las 8:00 a.m. hasta las 3:00 p.m.

— InterCliente (@InterCliente) November 30, 2019

#ATENCIÓN
Mañana 1 de diciembre, de 8:00 am a 3:00 pm, nuestro proveedor de acceso a los servicios de internet estará realizando un mantenimiento programado para reparar una falla del cable submarino

— Movistar Venezuela (@MovistarVe) November 30, 2019

The Internet disruption caused by the submarine cable maintenance is clearly visible at a country level, as seen in the figures below. Both Oracle and CAIDA IODA observed notable declines in the Traceroute Completion/Active Probing and BGP metrics, but it is interesting to note that there was no apparent impact to the traffic-derived metrics, which may indicate that traffic took advantage of alternative paths to exit the country. And although the Tweets from the network providers indicated that the maintenance window was expected to last from 0800 to 1500 local time (1200 to 1900 GMT), it appears that it took longer than expected, with connectivity being restored closer to 1800 local time (2200 GMT).

The figures below show the impact of the cable maintenance at a network level, illustrating the Internet disruptions experienced by four Venezuelan Internet service providers: NetLink América, Inter/Corporación Telemic, Net Uno, and Movistar/Telefónica. The Oracle graphs show that during the maintenance window, traceroutes to targets in these networks via AS3549 (Level 3) failed to complete. In some cases (Inter, Net Uno), a smaller percentage of them transited AS52320 (GlobeNet), while for Movistar, other networks picked up the slack.

• 

• 

• 

• 

• 

• 

• 

• 

Although the cable that was being repaired wasn’t specifically named, clues in the CenturyLink memo in the Tweet shown below would seem to indicate that it was the South American Crossing (SAC) cable system. This particular cable is one of two that lands in Panama and St. Croix, as well as Venezuela, but is the only one that has a Panama to St. Croix segment, as referenced in the memo. In addition, CenturyLink is one of the owners of the SAC cable system.

Debido a reparaciones en el cable submarino Panama-St Croix, Proveedores de Servicio de Internet como @InterCliente y @MovistarVe han informado que mañana #1dic será interrumpido el servicio desde las 8:00am hasta las 3:00pm pic.twitter.com/TDIi8mu4ty

— RedesAyuda (@RedesAyuda) December 1, 2019

On December 8, concurrent Internet disruptions were observed in Oracle and CAIDA IODA graphs for Sierra Leone, Guinea, Liberia, and Benin from 1725 to 1825 GMT. In November, a similar event occurred, and we noted in that month’s post, “Because the Africa Coast to Europe (ACE) submarine cable lands in each of these countries, it was suspected that problems with the cable caused these observed disruptions.” That suspicion was confirmed by @acesubmarinec, but we received no response to outreach regarding this December disruption or a followup inquiry. However, given that all of the countries shown below are connected to the ACE cable, it is likely the culprit in this disruption as well.

• 

• 

• 

• 

• 

• 

• 

• 

On December 10, Fing Internet Alert Tweeted about a significant Internet disruption experienced in multiple cities by customers of U.S. network provider Suddenlink.

This is #suddenlink situation today.#suddenlinkdown #suddenlinkhelp pic.twitter.com/YslH9SHvKb

— Fing Internet Alert (@outagedetect) December 10, 2019

The figures below indicate that the issues started around 0630 GMT, with significant impact for approximately three hours. The Active Probing metric in the CAIDA IODA graph shows that the situation began to improve around 1000 GMT, but that the metric didn’t return to pre-disruption levels until around 1500 GMT. Although Suddenlink apologized to customers via Twitter, it did not respond to a request for more information about the cause of the disruption.

Making its second appearance in this month’s report, Malta also suffered an Internet disruption on December 12. However, this one was due to an outage on international connectivity for Go, the country’s leading telecommunications company. According to posts on the company’s Facebook page, the outage impacted both the company’s fixed and mobile Internet services.

The figures below show a country-level disruption starting around 1500 GMT across all metrics in both the Oracle and CAIDA IODA graphs, lasting until 1845 GMT. At a network level, the CAIDA IODA graph below shows all three metrics dropping to near zero during the period of disruption, while the Oracle graph appears to show that some traceroutes continued to successfully reach endpoints within the network. Malta is serviced by five international submarine cables, but Go did not specify which one suffered the outage. However, as Go is the owner of the GO-1 Mediterranean Cable System, and co-owner of the Italy-Malta cable, it is likely that the problem occurred on one of these.

• 

• 

• 

• 

On December 15, PTCL, Pakistan’s national telecommunications company, alerted customers of potential service degradation due to a fault in the Asia Africa Europe-1 (AAE-1) submarine cable.

Internet services are impacted & you may face some service degradation due to technical fault in the International Submarine Cable AAE-1 at Doha. PTCL in conjunction with the International Submarine Consortium, is working to fully restore internet services across the country. 1/2

— PTCL (@PTCLOfficial) December 15, 2019

The figures below show the impact of the cable issue on Oracle Internet Intelligence traceroutes to endpoints in two PTCL autonomous systems. In both instances, the number of successful traceroutes did not decline, but combined latency increased by 50-75ms. Based on the timing of a followup Tweet posted by @PTCLofficial, service was restored after approximately five-and-a-half hours.

After a government-directed nationwide Internet disruption in November, initial reactions to connectivity issues observed in Iran on December 19 reflected concerns that another such event was taking place. However, Sadjad Bonabi (@sadjadb), a member of the Board of Directors of Iranian infrastructure provider TIC, posted a series of Tweets [1, 2, 3] over several hours explaining that the disruption was due to fiber breaks near Bucharest and Munich, and that service providers such as Google were also impacted. An Iranian news site also noted that the disruption was “related to fiber optic lines from Europe.” A subsequent update from NetBlocks indicated that connectivity to Google services in Turkey and Bulgaria had been impacted at the same time.

The figures below highlight the impact of this network issue at a country level in Iran, Turkey, and Bulgaria. Both the Oracle and CAIDA IODA graphs show that the most significant impact was observed in Iran, with changes to the measured metrics for Turkey and Bulgaria much more nominal in the Oracle graphs, and nearly imperceptible in the CAIDA IODA graphs.

• 

• 

• 

• 

• 

• 

As was noted by Sadjad Bonabi and NetBlocks, the network disruption impacted Google services in the affected countries, and Google’s Cloud Status Dashboard reported an “issue with multiple simultaneous fiber cuts affecting traffic routed through Google’s network in Bulgaria.” The figures below illustrate how the fiber cuts impacted traffic to Google’s YouTube, Maps, and GMail services (left-to-right) from users in Iran, Turkey, and Bulgaria (top-to-bottom). A brief but obvious dip in traffic is evident in each graph at the time the disruption occurred.

• 

• 

• 

• 

• 

• 

• 

• 

• 

A day later, mid-Atlantic region customers of U.S. Internet provider Comcast experienced a brief Internet disruption. The screenshot below from Fing Internet Alerts shows that the disruption impacted subscribers in Maryland, Virginia, the District of Columbia, Delaware, and West Virginia.

The Fing Internet Alerts graphs below show that the disruption began just before midnight local time (0500 GMT) on December 20, and lasted for approximately an hour. This timing is corroborated by the CAIDA IODA graph below, but given the localized impact of the disruption, the impact to the measured metrics is minimal.

A thread on the Outages mailing list explained that the disruption happened because “There was a maintenance in one part of a regional network, but there was an optical line failure in another part of that same network that happened during the maintenance.”

• 

• 

• 

• 

• 

Government Directed

No stranger to government directed Internet shutdowns, India moved to shut down Internet services in the northern states of Assam, Tripura, and Meghalaya on December 12. The shutdown was ordered in response to protests stemming from the passage of the Citizenship Amendment Bill. According to a Tweet and subsequent report posted by NetBlocks, the actions impacted both fixed and mobile network providers. Within the Tweet and report, NetBlocks showed connectivity levels it measured for the autonomous systems of seven local network providers.

The figures below show how connectivity to these providers was seen by CAIDA IODA measurements during the multi-day disruption. While the start of the disruption at ~1600 GMT on December 12 is generally evident across the graphs, there are noticeable differences across them as well. These differences may be due to the types of network each autonomous system is associated with, as mobile networks are significantly harder to measure into from external vantage points. Several see restoration and disruption events that align with those seen in NetBlocks’ report, while others appear to show a complete loss of connectivity for over four days.

• 

• 

• 

• 

• 

• 

• 

A court order published on December 19 ordering the restoration of services highlighted the detrimental affect that the loss of Internet connectivity has had on everyday life:

• “It has been impressed on the Court that even electricity supply has been snapped to certain houses of the lawyers because pre-paid meters have been installed, which can only be charged through the Internet.”
• “Under the circumstances, none of the business establishments is able to transact business causing serious disruption in normal living of the citizens in the area.”
• “To say the least, with the advancement of science and technology, mobile Internet services now plays a major role in the daily walks of life, so much so, shut-down of the mobile Internet service virtually amounts to bringing life to a grinding halt.”

Additional protests over the Citizenship Amendment Bill caused the Indian government to order the shutdown of mobile Internet connectivity in New Delhi, as well as the suspension of Internet services in Aligarh, Meerut, Malda, Murshidabad, Howrah, and parts of West Bengal during this timeframe as well. On December 26, similar steps were taken in Uttar Pradesh.

As reviewed in the November 2019 Internet Disruption Report, Iran suffered a week-long government directed Internet shutdown in response to protests over gasoline prices. On December 25, the Iranian government ordered restrictions on mobile Internet access in several provinces ahead of protests and ceremonies to remember those killed in the prior month’s protests.

A NetBlocks Tweet about the ordered mobile shutdown looked at the impact on connectivity for AS57218 (Rightel), one of three mobile providers in Iran.

Update: Mobile internet connectivity has fallen further in #Iran amid reports of security reinforcement; real-time network data show connectivity at 5% of ordinary levels on specified networks after four distinct cuts; incident ongoing 📉#Internet4Iran

📰https://t.co/BNTPP9wEyt pic.twitter.com/Wh2uy5xIxp

— NetBlocks.org (@netblocks) December 26, 2019

The figure below shows the impact of the order shutdown on Rightel as observed by CAIDA IODA’s measurements. The changes to the BGP and Active Probing metrics show that connectivity was lost in several stages, with Active Probing reaching its lowest level just before 1200 GMT on December 26. Connectivity was apparently restored around 0500 GMT on December 28 as the Active Probing and BGP metrics rose rapidly, returning to pre-disruption levels.

Interestingly, CAIDA IODA graphs for the country’s other two national mobile providers (Iran Cell and MCI) did not show similar disruptions to connectivity.

Russian Internet Disconnection Test

On December 23, a post to tech news aggregator Slashdot claimed “Russia has temporarily shut off many of its citizens’ access to the global internet today in a test of its controversial RuNet program, according to an internal government document.” That same day, ZDNet declared “Russia successfully disconnected from the internet”, noting “RuNet disconnection tests were successful, according to the Russian government.” The BBC picked up the story two days later.

According to the BBC article, “Details of what the test involved were vague but, according to the Ministry of Communications, ordinary users did not notice any changes.” The ZDNet article noted “The tests were carried out over multiple days, starting last week, and involved Russian government agencies, local internet service providers, and local Russian internet companies. … Internet traffic was re-routed internally, effectively making Russia’s RuNet the world’s largest intranet.” With this explanation, these tests would presumably have been visible in some fashion from outside the country.

However, that did not seem to be the case. Graphs from Oracle’s Internet Intelligence Map and CAIDA IODA did not show any noticeable changes to the measured metrics, and routing statistics from RIPENCC confirmed BGP’s stability over the period in question. Google Transparency Report traffic graphs for Web search, YouTube, GMail, and Blogger services did not show any deviations from the expected diurnal patterns and contacts at two leading CDN providers also reported no noticeable changes to traffic patterns from Russia during the time the test reportedly took place. Additionally, traffic graphs from IXPs within Russia, as well as DECIX, did not show any evidence of changes in traffic volume that would be expected if an Internet shutdown had occurred.

With a lack of evidence across major Internet platforms and measurement tools, we might ask whether the disconnection test actually occurred, as was reported, and if so, what was tested. The translation of a Russian-language article may provide some insight. According to the article, the disconnection may have been an “exercise” conducted in a test environment, rather than on Russia’s production Internet. The article also notes that exercises held on December 16 & 17 involved the SS7 and Diameter signaling protocols – these are used in the back-end of telecom networks, and are not related specifically to the Internet. It isn’t clear whether the tests that reportedly took place on/around December 23 also only focused on telecoms, rather than native Internet, infrastructure. Several additional Russian-language articles (1, 2) also questioned whether the tests had actually taken place, as well as the reported results of the tests.

Additional Observations

Recognizing that its reliance on a lone submarine cable for international Internet access represents a single point of failure, the government of Tonga is reportedly taking steps to implement backup satellite-based Internet connectivity. The goal is to avoid a repeat of the nationwide Internet outage that occurred in January 2019.

A “digital decree” passed by Spain’s government in late November reportedly gives it the power to shut down Internet connectivity in specified areas without prior judicial order. The relevant text is an adaptation of Section 4 of Article 4 of the current General Telecommunications Law:

“The Government, on an exceptional and transitory basis, may agree on the assumption by the General State Administration of direct management or the intervention of electronic communications networks and services in certain exceptional cases that may affect public order, public safety and national security This exceptional power may affect any infrastructure, associated resource or element or level of the network or service that is necessary to preserve or restore public order, public safety and national security. “ (via Google Translate)

As expected, concerns were raised about the constitutionality of the new rules, and about potential human rights violations associated with Internet shutdowns enabled by the decree.

In December, the Internet shutdown in Rakhine, Myanmar entered its seventh month and the shutdown in the Kashmir region of India entered its fifth month. These shutdowns have had a significant impact on the local economies, affecting small local businesses, as well as limiting the delivery of e-government services. At the end of 2019, it was observed by CNN and Quartz that Internet shutdowns have effectively become the ‘new normal’ and are/will be an increasingly popular mechanism for government repression.

Conclusion

I launched the Internet Disruption Report in May 2019 as a continuation of work that I had done as founding editor of Akamai’s State of the Internet/Connectivity Report, as well as the monthly “Last Month In Internet Intelligence” blog posts I published as a member of the Oracle Internet Intelligence team. Response has been very positive, and I am extremely grateful to those organizations that not only monitor and measure the Internet, but also make their findings and analysis publicly available. I also applaud those network providers that make effective use of social media channels to provide notice of, and status updates regarding, both planned and unanticipated disruptions, as well as those providers that respond to requests for more information about such disruptions. (Hint, hint…)

Given the growing trend towards government directed Internet shutdowns, and the otherwise unavoidable Internet disruptions caused by power outages, cable and fiber cuts, routing issues, and DDoS attacks, I am likely (for better or worse) to have plenty of content for many future Internet Disruption Report posts. I look forward to sharing my insights on these events with you in 2020 and beyond.