Internet Disruption Report: June 2019

Simply put, June was a pretty rough month for the global Internet. In addition to all of the usual small/transient issues, there were quite a few significant disruptions that occurred over the course of the month. This month’s post covers government-mandated Internet shutdowns due to political unrest and national exams, disruptions due to cable damage, power outage-related Internet outages, two large route leaks, and a couple of localized disruptions (for good measure).

Political Unrest

Unfortunately, during periods of political unrest, national governments have taken to implementing Internet shutdowns and social media filtering/blocking as a means of preventing the spread of (mis)information and limiting citizen ability to organize protests and similar activity.

On June 3, Internet connectivity in Sudan was disrupted in conjunction with attacks on protestors – the protests were related to the transition of power after the removal of Omar al-Bashir as Sudan’s president in April. The figure below shows drops in all three metrics measured by Oracle Internet Intelligence occurring around noon GMT.

Oracle Internet Intelligence Map Country Statistics graph for Sudan

Three weeks later, the Internet was restored… but only for a single lawyer that had filed a lawsuit against telecommunications provider Zain Sudan over the disruption ordered by Sudan’s military rulers. However, on July 9, local telecommunications providers were ordered to restore Internet service. This service restoration can be seen in the figure below, with increases in both the traceroute and BGP-based metrics.

Oracle Internet Intelligence Map Country Statistics graph for Sudan

An Internet disruption took place in Ethiopia on June 21 in the wake of an attempted coup, with the impacts visible in the figures below. Active inbound measurements by Oracle Internet Intelligence and CAIDA IODA saw success rates drop significantly as a result of the Internet shutdown, while passively measured DNS (Oracle), Darknet (CAIDA), and Web (Google) traffic dropped to near-zero levels. Connectivity was restored on June 27.

Following a contested presidential election in Mauritania, an Internet shutdown was put into place at approximately 15:30 GMT on June 25 as seen in the CAIDA IODA figure below – a significant drop in routed network prefixes occurred for several hours, while Active Probing and Darknet traffic sources dropped significantly.

CAIDA IODA graph for Mauritania

Network diversity within the country is extremely limited. Looking at the figure for AS29544 (Mauritanian Telecommunication Company) below, it is clear that the routing instability in the country-level graph is related to this network’s routes essentially being removed from the Internet for that multi-hour period. Internet connectivity began to return across the country on July 3.

CAIDA IODA graph for AS29544 (Mauritanian Telecommunication Company)

Cable Damage

Damage to terrestrial and undersea cables has long been the nemesis of a properly functioning Internet. The Internet disruptions due to cable damage observed during the first and last weeks of June tended to be brief and/or localized; similar issues seen in the past have caused significant country-wide Internet disruptions.

On June 4, Filipino telecommunications provider Globe Telecom posted a Public Advisory on its Facebook page that noted (in part): “Between 1-2PM today, the Globe Network experienced multiple fiber cuts, two of which were submarine cables.” The impact of these fiber cuts is clearly evident in the figures below for AS4775 and AS132199, both network identifiers associated with Globe Telecom in the Philippines, with multi-hour drops visible in both the Active Probing and BGP metrics. A slight perturbation to these metrics was barely visible in the country-level graph. A published report noted that connectivity was restored about four hours after the issue started, in line with the recovery time seen in the graphs.

On June 6, Internet monitoring and advocacy organization Netblocks Tweeted about an Internet outage in Venezuela that affected local network providers Digitel and Supercable. Netblocks noted that Digitel was completely disrupted while Supercable only experienced a partial outage. This is in line with the figures below, showing all metrics dropping to zero at Digitel starting around 15:00 GMT, and only a nominal decline in the Active Probing metric for Supercable around the same time. The Netblocks tweet stated that “sources report fiber cut”, but this root cause was not independently confirmed.

A brief Internet disruption observed in Pakistan on June 24 was due to multiple cable cuts, according to a press release issued by Pakistan Telecommunication Company Limited (PTCL). The figure below from CAIDA IODA shows a clear drop in the Active Probing metric at a country level, along with a slight dip in Darknet traffic sources.

CAIDA IODA graph for Pakistan

The impact of the cable cuts can also be seen at a network level, visible in the two figures below for PTCL-owned networks.

The overall impact of the cable cuts appeared to be nominal, and the disruption short-lived, with metrics returning to normal levels within approximately four hours. @PTCLOfficial announced the full restoration of service just a few hours after the disruption began:

Iran is no stranger to politically-motivated Internet disruptions, and country leaders have spoken in the past about disconnecting the country from the global Internet. However, a brief disruption that occurred on June 26, shown in the figure below, may have instead been related to physical layer problems. A Tweet posted by Iranian ICT minister Azari Jahromi suggested that “a disruption of fiber connectivity” at upstream routes in Europe between Bulgaria & Frankfurt, which are used by several ISPs in Iran, caused the 1.5 hour disruption in connectivity.

CAIDA IODA graph for Iran

National Exams

The governments of Syria, Ethiopia, and Iraq have all ordered Internet shutdowns in the past to prevent student cheating on national exams, and all three countries experienced such disruptions once again in June.

First up was Syria, where the first observed disruption, between 01:00 – 05:30 GMT on June 9, took place in conjunction with 712,000 university students starting their second term exams, along with 9th grade (O-Level) and 12th grade (A-Level) students starting public exams as well. Additional disruptions were also seen on June 11, 13, 17, and 19, all effectively appearing as complete country-wide outages in the Oracle Internet Intelligence, CAIDA IODA, and Google Transparency Report Traffic figures below.

It is interesting to note that within the Oracle Internet Intelligence Map graphs, the Traceroute Completion Ratio and BGP Routes metrics drop to zero during the periods of disruption, but the DNS Query Rate metric spikes. Doug Madory, Director of Internet Analysis at Oracle Internet Intelligence, explained why this happened:

On June 9, the first of four Internet disruptions apparently related to exams took place in Ethiopia, and lasted approximately six hours. A second disruption on June 11 lasted five hours, from 07:00 to 12:00 GMT. The third disruption, lasting approximately 12 hours, started in the evening of June 11, carrying over to the next day. The final disruption started the evening of June 12, and lasted 39 hours, until mid-day on June 14. The impact of these disruptions to connectivity in Ethiopia can be seen in the figures below — all caused a significant, but not complete, loss of connectivity at a country level. Published reports claimed that the outages were intended to prevent student cheating on national exams.

Ethio Telecom acknowledged, and apologized for, the disruptions in a Tweet, but neither the telecommunications company nor the government confirmed the reason they occurred.

Finally, on June 26, Iraq reportedly implemented regional Internet shutdowns intended to prevent student cheating on national Physics exams. The country-level CAIDA IODA figure for Iraq below shows a brief, and limited, drop in successful Active Probing late in the morning (GMT).

CAIDA IODA graph for Iraq

The CAIDA IODA figure for Sala ad-Din, a region of Iraq just north of Baghdad, shows a brief drop in the number of routed prefixes geolocated to the region. The drop occurs concurrent with the disruption seen at a national level.

CAIDA IODA graph for Sala ad-Din, Iraq

The disruption is also visible at a network level, as the CAIDA IODA figure below for AS50710 (Earthlink) shows. Earthlink is one of the largest network service providers in Iraq, and during the period of disruption, a drop in the Active Probing metric can be seen.

CAIDA IODA graph for AS50710 (Earthlink)

Power Outage

On June 16, a massive power outage, called “unprecedented in its scope” by local officials, impacted all of mainland Argentina and Uruguay, affecting nearly 48 million people. The outage started at approximately 10:00 GMT, with near immediate impact on Internet connectivity to the affected countries and networks within them, as shown in the figures below. While local power utilities quickly began restoring power, Oracle Internet Intelligence and CAIDA IODA measurement metrics took nearly 12 hours to return to near-normal levels.

Electrical utilities in both countries provided status updates on Twitter throughout the event – the @UTE_ComCorp Tweet below indicated that 98.5% of power had been restored in Uruguay, approximately 11 hours after their first Tweet regarding the power failure, and the Tweet regarding restoration of service in Argentina from @OficialEdesur followed a similar timeline.

Route Leaks

Route leaks remain a common occurrence on the Internet. However, adoption of basic safeguards necessary to prevent propagation of routing leaks, such as the Internet Society’s Mutually Agreed Norms for Routing Security (MANRS) project, can mitigate the impact of such leaks. Unfortunately, neither China Telecom nor Verizon have apparently yet implemented such safeguards, and both carriers were implicated in massive route leaks that resulted in increased latency and/or service disruptions in June.

The first leak was observed on June 6, when Swiss data center colocation company Safe Host SA (AS21217) leaked over 70,000 routes to China Telecom (AS4134) in Frankfurt, Germany. An excellent and detailed blog post from Oracle’s Doug Madory explained the impact of the leak:

“China Telecom then announced these routes on to the global Internet redirecting large amounts of Internet traffic destined for some of the largest European mobile networks through China Telecom’s network. Impacts were seen by some of Europe’s largest networks in Switzerland, Holland, and France among other countries.Often routing incidents like this only last for a few minutes, but in this case many of the leaked routes in this incident were in circulation for over two hours.”

The impact of the leak from a traceroute and latency perspective was illustrated in a Tweet from @InternetIntel:

The second notable route leak occurred on June 24, impacting approximately 20,000 prefixes from 2,400 network providers, according to BGPmon. Content delivery and security service provider Cloudflare was significantly impacted by the routing leak, and it too published an excellent and detailed blog post explaining what happened.

“An Internet Service Provider in Pennsylvania  (AS33154 – DQE Communications) was using a BGP optimizer in their network, which meant there were a lot of more specific routes in their network. Specific routes override more general routes… DQE announced these specific routes to their customer (AS396531 – Allegheny Technologies Inc). All of this routing information was then sent to their other transit provider (AS701 – Verizon), who proceeded to tell the entire Internet about these “better” routes.”

In addition to Cloudflare, this route leak impacted connectivity for many other major organizations. BGPmon’s BGPStream Web site publishes alerts about BGP hijacks, leaks, and outages, and the table below is based on data extracted from the BGPStream site for alerts associated with this leak. It is clear in looking at the list that there was a global impact, with networks in Europe and the Asia Pacific region among those affected.

ORIGIN ASSTART TIME (UTC)DETAILS
WASHPOST – Washington Post, US (AS 30281)6/24/19 12:28More detail
JENNISON – Jennison Associates LLC, US (AS 30394)6/24/19 12:27More detail
ASN-TGBP-OM – Western Union Business Solutions., US (AS 55068)6/24/19 12:26More detail
24SHELLS – 24 SHELLS, US (AS 55081)6/24/19 12:20More detail
NJIT-AS – New Jersey Institute of Technology, US (AS 4246)6/24/19 12:19More detail
TELEFONICA TELXIUS, ES (AS 12956)6/24/19 12:19More detail
AS-CHOOPA – Choopa, LLC, US (AS 20473)6/24/19 12:19More detail
CLOUDFLARENET – Cloudflare, Inc., US (AS 13335)6/24/19 12:19More detail
BANDCON – Bandcon, US (AS 26769)6/24/19 12:19More detail
123NET – 123.Net, Inc., US (AS 12129)6/24/19 12:19More detail
LINODE-AP Linode, LLC, US (AS 63949)6/24/19 12:19More detail
CDM – CDM, US (AS 6428)6/24/19 12:19More detail
UNIFIEDLAYER-AS-1 – Unified Layer, US (AS 46606)6/24/19 12:19More detail
EONIX-COMMUNICATIONS-ASBLOCK-62904 – Eonix Corporation, US (AS 62904)6/24/19 12:19More detail
AMAZON-02 – Amazon.com, Inc., US (AS 16509)6/24/19 12:19More detail
FASTLY – Fastly, US (AS 54113)6/24/19 12:19More detail
CSAA-INSURANCE-EXCHANGE – CSAA Insurance Exchange, US (AS 31966)6/24/19 12:19More detail
VNPT-AS-VN VNPT Corp, VN (AS 45899)6/24/19 12:19More detail
FEYNMANGROUP – Feynman Group, Inc., US (AS 27550)6/24/19 12:19More detail
AKAMAI-ASN1, US (AS 20940)6/24/19 12:19More detail
MULTIDATA-ID-AP PT Multidata Rancana Prima, ID (AS 58552)6/24/19 12:19More detail
BLOOMBERG-NET – Bloomberg, LP, US (AS 10361)6/24/19 12:19More detail
WORKNET-1 – WORKNET INC., US (AS 22027)6/24/19 12:19More detail
ACE-US – ACE INA HOLDINGS INC., US (AS 63111)6/24/19 12:19More detail
LUS-FIBER-LCG – LUS Fiber, US (AS 25921)6/24/19 12:19More detail
COUPONS-COM-INCORPORATED – Quotient Technology Inc, US (AS 19950)6/24/19 12:19More detail
6750REI – Recreational Equipment Inc., US (AS 11307)6/24/19 12:19More detail
ASN-RISKMGMT-NJ – Risk Management Solutions, Inc., US (AS 40276)6/24/19 12:19More detail
TPGROUP – Teleperformance Group, Inc., US (AS 393290)6/24/19 12:19More detail
EPSEL-ASN – Evercore Partners Services East, LLC, US (AS 46667)6/24/19 12:14More detail
IDEALCLOUD – Ideal Integrations, Inc., US (AS 22804)6/24/19 12:12More detail
MACMILLAN-PUBLISHERS – Macmillan Publishers Inc, US (AS 395060)6/24/19 12:12More detail
HETZNER-AS, DE (AS 24940)6/24/19 12:10More detail
SERVEREL-AS – Serverel Inc., US (AS 15317)6/24/19 12:02More detail
LGI-UPC formerly known as UPC Broadband Holding B.V., AT (AS 6830)6/24/19 12:02More detail
PTC-OKC-ASN – Perimeter Technology Center, LLC, US (AS 27582)6/24/19 12:02More detail
VERSATEL, DE (AS 8881)6/24/19 12:02More detail
ARTMOTION-AS, RS (AS 33983)6/24/19 12:02More detail
SWISSCOM Swisscom (Switzerland) Ltd, CH (AS 3303)6/24/19 12:01More detail
T-MOBILE-AS21928 – T-Mobile USA, Inc., US (AS 21928)6/24/19 12:01More detail
LWLCOM, DE (AS 50629)6/24/19 12:01More detail
KNIPP-AMS-AS, DE (AS 48519)6/24/19 11:27More detail
LRC-104 – LRC GROUP LLC, US (AS 26465)6/24/19 11:26More detail
EWETEL Cloppenburger Strasse 310, DE (AS 9145)6/24/19 10:40More detail
OVH, FR (AS 162766/24/19 10:40More detail
VALVE-CORPORATION – Valve Corporation, US (AS 32590)6/24/19 10:39More detail
CYBERCON – CYBERCON, INC., US (AS 7393)6/24/19 10:39More detail
CHARTER-20115 – Charter Communications, US (AS 20115)6/24/19 10:39More detail
SHAW – Shaw Communications Inc., CA (AS 6327)6/24/19 10:39More detail
CHINANET-BACKBONE No.31,Jin-rong Street, CN (AS 4134)6/24/19 10:39More detail
UNWIRED – Unwired, US (AS 32354)6/24/19 10:39More detail
SPRINTLINK2 – Sprint Government Systems Division, US (AS 1790)6/24/19 10:39More detail
BBIL-AP BHARTI Airtel Ltd., IN (AS 9498)6/24/19 10:39More detail
FACEBOOK – Facebook, Inc., US (AS 32934)6/24/19 10:39More detail
WEBZILLA, NL (AS 35415)6/24/19 10:39More detail
NEWPIRVINE – Newport Corporation, US (AS 26487)6/24/19 10:39More detail
FASTWEB, IT (AS 12874)6/24/19 10:39More detail
ASN-DIS – Dallas Infrastructure Services, LLC, US (AS 393398)6/24/19 10:39More detail
WVTM-TV – HEARST CORPORATION, US (AS 54204)6/24/19 10:39More detail
ABRAMSCAPITALMANAGEMENTLLC – Abrams Capital Management, LLC, US (AS 29786)6/24/19 10:39More detail
COSLINK – Cherryland Services Inc, US (AS 13904)6/24/19 10:39More detail
REALNET – RealNetworks, Inc., US (AS 26353)6/24/19 10:38More detail
GRAVITAS-TECHNOLOGY-SERVICES-LLC – Gravitas Technology Services, LLC, US (AS 54566)6/24/19 10:38More detail
PSAM-ASN – P. Schoenfeld Asset Management, LLC, US (AS 26200)6/24/19 10:38More detail
TAMPNET-AS Tampnet International Carrier, NO (AS 200781)6/24/19 10:38More detail
DEBEVOISE – Debevoise and Plimpton LLP, US (AS 394122)6/24/19 10:37More detail
(AS 2147483647)6/24/19 10:37More detail
(AS 2147483647)6/24/19 10:37More detail
MINNESOTATWINSLLC – Minnesota Twins, LLC, US (AS 16915)6/24/19 10:37More detail
FMCTI-FMC-TECHNOLOGIES-INC-US – FMC Technologies, Inc., US (AS 394936)6/24/19 10:37More detail
ESWIKIKER, ES (AS 200845)6/24/19 10:36More detail
PITNEYBOWES-AS-1 – Pitney Bowes Incorporated, US (AS 11086)6/24/19 10:36More detail
ELYNX-CORP-NET – eLynx Ltd., US (AS 17259)6/24/19 10:36More detail
DESERTSCHOOLS-ASN-1 – Desert Schools Federal Credit Union, US (AS 14590)6/24/19 10:36More detail
BLACKROCK-FINANCIAL-MANAGEMENT-2 – BlackRock Financial Management, Inc., US (AS 31747)6/24/19 10:36More detail
FMA-ALLIANCE-LTD – FMA Alliance, Ltd., US (AS 393576)6/24/19 10:36More detail
AS-AFILIAS-REGISTRY-SERVICES – Afilias Canada, Corp., CA (AS 21775)6/24/19 10:36More detail
AMCON – AMCON Distributing Company, US (AS 26373)6/24/19 10:36More detail
USL-ASN – Upsher-Smith Laboratories Inc., US (AS 13715)6/24/19 10:36More detail
PMA-INTERNET – Preferred Managing Agency, LLC, US (AS 395869)6/24/19 10:36More detail
BREMERFINANCIAL – BREMER FINANCIAL SERVICES, US (AS 55153)6/24/19 10:36More detail
HNS-DIRECPC – Hughes Network Systems, US (AS 6621)6/24/19 10:36More detail
TELEPERFORMANCE-USA – TELEPERFORMANCE USA, US (AS 33680)6/24/19 10:36More detail
FARM-CREDIT – Farm Credit Financial Partners, Inc., US (AS 16910)6/24/19 10:36More detail
MHS – Meridian Health System, US (AS 1436)6/24/19 10:36More detail
SCHOLASTIC – Scholastic Inc., US (AS 10717)6/24/19 10:36More detail
BCIU-22 – Bucks County Intermediate Unit #22, US (AS 16958)6/24/19 10:36More detail
ROGERS-COMMUNICATIONS – Rogers Communications Canada Inc., CA (AS 812)6/24/19 10:36More detail
KANREN – Kansas Research and Education Network, US (AS 2495)6/24/19 10:35More detail
VITU-49 – Virtuoso, Ltd., US (AS 53890)6/24/19 10:35More detail
PROSKAUER-1 – Proskauer Rose LLP, US (AS 25955)6/24/19 10:35More detail
SLG – SL Green Realty Corp, US (AS 46152)6/24/19 10:35More detail
CAL-NET-INC – Cal.net, Inc., US (AS 21580)6/24/19 10:35More detail
SCATUI – SCATUI, US (AS 46294)6/24/19 10:35More detail
RACKNET1 – Rack Wizards Inc, US (AS 46897)6/24/19 10:34More detail
NETSPECTRUM – Netspectrum Wireless Internet Solutions, CA (AS 22060)6/24/19 10:34More detail
MOTOROLA-MOBILITY – Motorola, Inc., US (AS 22244)6/24/19 10:34More detail
SHOPZILLA – Connexity, Inc., US (AS 14332)6/24/19 10:34More detail
-Reserved AS-, ZZ (AS 21666)6/24/19 10:34More detail
SEATTLE – City of Seattle, Dept. of Admin. Services, US (AS 3401)6/24/19 10:34More detail
COG-SEOVEC-ISP – COG-SEOVEC, US (AS 30298)6/24/19 10:34More detail

Localized Disruptions

While these blog posts generally cover large-scale, country-level disruptions, easily identifiable within publicly available monitoring tools, more localized disruptions are often visible through these tools as well.

On June 7, Consolidated Communications, a network provider that services Vermont, reportedly experienced a “router issue” at its office in Burlington, disrupting connectivity to all 80,000 Internet customers in the state, including businesses and homes. The figures below show the impact of the disruption, with active measurement metrics declining for a brief period of time. Although the autonomous system label in the graphs is for FairPoint Communications, the related PeeringDB entry shows it is owned by Consolidated Communications. (FairPoint was acquired by Consolidated in 2016.)

The @outagedetect Twitter account, associated with Internet connection and connected device monitoring company Fing, posts numerous alerts each day about localized Internet outages. On June 19, it posted the Tweet below, highlighting the end of a “considerable” hour-long Internet outage detected in Poland for users connected to network service provider Orange Polska.

One of the figures below shows that this “considerable” outage impacted CAIDA’s Active Probing measurements for Poland at a country level, as well as driving a concurrent decline in Darknet sources. The other figure shows the impact at a network level, with Active Probing measurements and Darknet sources for Orange Polska dropping considerably during the hour-long disruption.

Conclusion

While the major Internet disruptions observed in June were largely due to a familiar set of underlying causes, many of these causes remain avoidable.

Of course, convincing governments not to implement Internet outages during times of political unrest is a non-trivial effort, but organizations including the Internet Society and Access Now continue to advocate for user rights and drive #KeepItOn efforts.

In contrast, convincing network providers, especially Tier 1 global providers, to improve their routing hygiene through through participation in efforts like MANRS should be comparatively trivial.

One Comment

Leave a comment