Simply put, June was a pretty rough month for the global Internet. In addition to all of the usual small/transient issues, there were quite a few significant disruptions that occurred over the course of the month. This month’s post covers government-mandated Internet shutdowns due to political unrest and national exams, disruptions due to cable damage, power outage-related Internet outages, two large route leaks, and a couple of localized disruptions (for good measure).
Unfortunately, during periods of political unrest, national governments have taken to implementing Internet shutdowns and social media filtering/blocking as a means of preventing the spread of (mis)information and limiting citizen ability to organize protests and similar activity.
On June 3, Internet connectivity in Sudan was disrupted in conjunction with attacks on protestors – the protests were related to the transition of power after the removal of Omar al-Bashir as Sudan’s president in April. The figure below shows drops in all three metrics measured by Oracle Internet Intelligence occurring around noon GMT.
Three weeks later, the Internet was restored… but only for a single lawyer that had filed a lawsuit against telecommunications provider Zain Sudan over the disruption ordered by Sudan’s military rulers. However, on July 9, local telecommunications providers were ordered to restore Internet service. This service restoration can be seen in the figure below, with increases in both the traceroute and BGP-based metrics.
An Internet disruption took place in Ethiopia on June 21 in the wake of an attempted coup, with the impacts visible in the figures below. Active inbound measurements by Oracle Internet Intelligence and CAIDA IODA saw success rates drop significantly as a result of the Internet shutdown, while passively measured DNS (Oracle), Darknet (CAIDA), and Web (Google) traffic dropped to near-zero levels. Connectivity was restored on June 27.
Following a contested presidential election in Mauritania, an Internet shutdown was put into place at approximately 15:30 GMT on June 25 as seen in the CAIDA IODA figure below – a significant drop in routed network prefixes occurred for several hours, while Active Probing and Darknet traffic sources dropped significantly.
Network diversity within the country is extremely limited. Looking at the figure for AS29544 (Mauritanian Telecommunication Company) below, it is clear that the routing instability in the country-level graph is related to this network’s routes essentially being removed from the Internet for that multi-hour period. Internet connectivity began to return across the country on July 3.
Damage to terrestrial and undersea cables has long been the nemesis of a properly functioning Internet. The Internet disruptions due to cable damage observed during the first and last weeks of June tended to be brief and/or localized; similar issues seen in the past have caused significant country-wide Internet disruptions.
On June 4, Filipino telecommunications provider Globe Telecom posted a Public Advisory on its Facebook page that noted (in part): “Between 1-2PM today, the Globe Network experienced multiple fiber cuts, two of which were submarine cables.” The impact of these fiber cuts is clearly evident in the figures below for AS4775 and AS132199, both network identifiers associated with Globe Telecom in the Philippines, with multi-hour drops visible in both the Active Probing and BGP metrics. A slight perturbation to these metrics was barely visible in the country-level graph. A published report noted that connectivity was restored about four hours after the issue started, in line with the recovery time seen in the graphs.
On June 6, Internet monitoring and advocacy organization Netblocks Tweeted about an Internet outage in Venezuela that affected local network providers Digitel and Supercable. Netblocks noted that Digitel was completely disrupted while Supercable only experienced a partial outage. This is in line with the figures below, showing all metrics dropping to zero at Digitel starting around 15:00 GMT, and only a nominal decline in the Active Probing metric for Supercable around the same time. The Netblocks tweet stated that “sources report fiber cut”, but this root cause was not independently confirmed.
A brief Internet disruption observed in Pakistan on June 24 was due to multiple cable cuts, according to a press release issued by Pakistan Telecommunication Company Limited (PTCL). The figure below from CAIDA IODA shows a clear drop in the Active Probing metric at a country level, along with a slight dip in Darknet traffic sources.
The impact of the cable cuts can also be seen at a network level, visible in the two figures below for PTCL-owned networks.
The overall impact of the cable cuts appeared to be nominal, and the disruption short-lived, with metrics returning to normal levels within approximately four hours. @PTCLOfficial announced the full restoration of service just a few hours after the disruption began:
Iran is no stranger to politically-motivated Internet disruptions, and country leaders have spoken in the past about disconnecting the country from the global Internet. However, a brief disruption that occurred on June 26, shown in the figure below, may have instead been related to physical layer problems. A Tweet posted by Iranian ICT minister Azari Jahromi suggested that “a disruption of fiber connectivity” at upstream routes in Europe between Bulgaria & Frankfurt, which are used by several ISPs in Iran, caused the 1.5 hour disruption in connectivity.
The governments of Syria, Ethiopia, and Iraq have all ordered Internet shutdowns in the past to prevent student cheating on national exams, and all three countries experienced such disruptions once again in June.
First up was Syria, where the first observed disruption, between 01:00 – 05:30 GMT on June 9, took place in conjunction with 712,000 university students starting their second term exams, along with 9th grade (O-Level) and 12th grade (A-Level) students starting public exams as well. Additional disruptions were also seen on June 11, 13, 17, and 19, all effectively appearing as complete country-wide outages in the Oracle Internet Intelligence, CAIDA IODA, and Google Transparency Report Traffic figures below.
It is interesting to note that within the Oracle Internet Intelligence Map graphs, the Traceroute Completion Ratio and BGP Routes metrics drop to zero during the periods of disruption, but the DNS Query Rate metric spikes. Doug Madory, Director of Internet Analysis at Oracle Internet Intelligence, explained why this happened:
On June 9, the first of four Internet disruptions apparently related to exams took place in Ethiopia, and lasted approximately six hours. A second disruption on June 11 lasted five hours, from 07:00 to 12:00 GMT. The third disruption, lasting approximately 12 hours, started in the evening of June 11, carrying over to the next day. The final disruption started the evening of June 12, and lasted 39 hours, until mid-day on June 14. The impact of these disruptions to connectivity in Ethiopia can be seen in the figures below — all caused a significant, but not complete, loss of connectivity at a country level. Published reports claimed that the outages were intended to prevent student cheating on national exams.
Ethio Telecom acknowledged, and apologized for, the disruptions in a Tweet, but neither the telecommunications company nor the government confirmed the reason they occurred.
Finally, on June 26, Iraq reportedly implemented regional Internet shutdowns intended to prevent student cheating on national Physics exams. The country-level CAIDA IODA figure for Iraq below shows a brief, and limited, drop in successful Active Probing late in the morning (GMT).
The CAIDA IODA figure for Sala ad-Din, a region of Iraq just north of Baghdad, shows a brief drop in the number of routed prefixes geolocated to the region. The drop occurs concurrent with the disruption seen at a national level.
The disruption is also visible at a network level, as the CAIDA IODA figure below for AS50710 (Earthlink) shows. Earthlink is one of the largest network service providers in Iraq, and during the period of disruption, a drop in the Active Probing metric can be seen.
On June 16, a massive power outage, called “unprecedented in its scope” by local officials, impacted all of mainland Argentina and Uruguay, affecting nearly 48 million people. The outage started at approximately 10:00 GMT, with near immediate impact on Internet connectivity to the affected countries and networks within them, as shown in the figures below. While local power utilities quickly began restoring power, Oracle Internet Intelligence and CAIDA IODA measurement metrics took nearly 12 hours to return to near-normal levels.
Electrical utilities in both countries provided status updates on Twitter throughout the event – the @UTE_ComCorp Tweet below indicated that 98.5% of power had been restored in Uruguay, approximately 11 hours after their first Tweet regarding the power failure, and the Tweet regarding restoration of service in Argentina from @OficialEdesur followed a similar timeline.
Route leaks remain a common occurrence on the Internet. However, adoption of basic safeguards necessary to prevent propagation of routing leaks, such as the Internet Society’s Mutually Agreed Norms for Routing Security (MANRS) project, can mitigate the impact of such leaks. Unfortunately, neither China Telecom nor Verizon have apparently yet implemented such safeguards, and both carriers were implicated in massive route leaks that resulted in increased latency and/or service disruptions in June.
The first leak was observed on June 6, when Swiss data center colocation company Safe Host SA (AS21217) leaked over 70,000 routes to China Telecom (AS4134) in Frankfurt, Germany. An excellent and detailed blog post from Oracle’s Doug Madory explained the impact of the leak:
“China Telecom then announced these routes on to the global Internet redirecting large amounts of Internet traffic destined for some of the largest European mobile networks through China Telecom’s network. Impacts were seen by some of Europe’s largest networks in Switzerland, Holland, and France among other countries. … Often routing incidents like this only last for a few minutes, but in this case many of the leaked routes in this incident were in circulation for over two hours.”
The impact of the leak from a traceroute and latency perspective was illustrated in a Tweet from @InternetIntel:
The second notable route leak occurred on June 24, impacting approximately 20,000 prefixes from 2,400 network providers, according to BGPmon. Content delivery and security service provider Cloudflare was significantly impacted by the routing leak, and it too published an excellent and detailed blog post explaining what happened.
“An Internet Service Provider in Pennsylvania (AS33154 – DQE Communications) was using a BGP optimizer in their network, which meant there were a lot of more specific routes in their network. Specific routes override more general routes… DQE announced these specific routes to their customer (AS396531 – Allegheny Technologies Inc). All of this routing information was then sent to their other transit provider (AS701 – Verizon), who proceeded to tell the entire Internet about these “better” routes.”
In addition to Cloudflare, this route leak impacted connectivity for many other major organizations. BGPmon’s BGPStream Web site publishes alerts about BGP hijacks, leaks, and outages, and the table below is based on data extracted from the BGPStream site for alerts associated with this leak. It is clear in looking at the list that there was a global impact, with networks in Europe and the Asia Pacific region among those affected.
|ORIGIN AS||START TIME (UTC)||DETAILS|
|WASHPOST – Washington Post, US (AS 30281)||6/24/19 12:28||More detail|
|JENNISON – Jennison Associates LLC, US (AS 30394)||6/24/19 12:27||More detail|
|ASN-TGBP-OM – Western Union Business Solutions., US (AS 55068)||6/24/19 12:26||More detail|
|24SHELLS – 24 SHELLS, US (AS 55081)||6/24/19 12:20||More detail|
|NJIT-AS – New Jersey Institute of Technology, US (AS 4246)||6/24/19 12:19||More detail|
|TELEFONICA TELXIUS, ES (AS 12956)||6/24/19 12:19||More detail|
|AS-CHOOPA – Choopa, LLC, US (AS 20473)||6/24/19 12:19||More detail|
|CLOUDFLARENET – Cloudflare, Inc., US (AS 13335)||6/24/19 12:19||More detail|
|BANDCON – Bandcon, US (AS 26769)||6/24/19 12:19||More detail|
|123NET – 123.Net, Inc., US (AS 12129)||6/24/19 12:19||More detail|
|LINODE-AP Linode, LLC, US (AS 63949)||6/24/19 12:19||More detail|
|CDM – CDM, US (AS 6428)||6/24/19 12:19||More detail|
|UNIFIEDLAYER-AS-1 – Unified Layer, US (AS 46606)||6/24/19 12:19||More detail|
|EONIX-COMMUNICATIONS-ASBLOCK-62904 – Eonix Corporation, US (AS 62904)||6/24/19 12:19||More detail|
|AMAZON-02 – Amazon.com, Inc., US (AS 16509)||6/24/19 12:19||More detail|
|FASTLY – Fastly, US (AS 54113)||6/24/19 12:19||More detail|
|CSAA-INSURANCE-EXCHANGE – CSAA Insurance Exchange, US (AS 31966)||6/24/19 12:19||More detail|
|VNPT-AS-VN VNPT Corp, VN (AS 45899)||6/24/19 12:19||More detail|
|FEYNMANGROUP – Feynman Group, Inc., US (AS 27550)||6/24/19 12:19||More detail|
|AKAMAI-ASN1, US (AS 20940)||6/24/19 12:19||More detail|
|MULTIDATA-ID-AP PT Multidata Rancana Prima, ID (AS 58552)||6/24/19 12:19||More detail|
|BLOOMBERG-NET – Bloomberg, LP, US (AS 10361)||6/24/19 12:19||More detail|
|WORKNET-1 – WORKNET INC., US (AS 22027)||6/24/19 12:19||More detail|
|ACE-US – ACE INA HOLDINGS INC., US (AS 63111)||6/24/19 12:19||More detail|
|LUS-FIBER-LCG – LUS Fiber, US (AS 25921)||6/24/19 12:19||More detail|
|COUPONS-COM-INCORPORATED – Quotient Technology Inc, US (AS 19950)||6/24/19 12:19||More detail|
|6750REI – Recreational Equipment Inc., US (AS 11307)||6/24/19 12:19||More detail|
|ASN-RISKMGMT-NJ – Risk Management Solutions, Inc., US (AS 40276)||6/24/19 12:19||More detail|
|TPGROUP – Teleperformance Group, Inc., US (AS 393290)||6/24/19 12:19||More detail|
|EPSEL-ASN – Evercore Partners Services East, LLC, US (AS 46667)||6/24/19 12:14||More detail|
|IDEALCLOUD – Ideal Integrations, Inc., US (AS 22804)||6/24/19 12:12||More detail|
|MACMILLAN-PUBLISHERS – Macmillan Publishers Inc, US (AS 395060)||6/24/19 12:12||More detail|
|HETZNER-AS, DE (AS 24940)||6/24/19 12:10||More detail|
|SERVEREL-AS – Serverel Inc., US (AS 15317)||6/24/19 12:02||More detail|
|LGI-UPC formerly known as UPC Broadband Holding B.V., AT (AS 6830)||6/24/19 12:02||More detail|
|PTC-OKC-ASN – Perimeter Technology Center, LLC, US (AS 27582)||6/24/19 12:02||More detail|
|VERSATEL, DE (AS 8881)||6/24/19 12:02||More detail|
|ARTMOTION-AS, RS (AS 33983)||6/24/19 12:02||More detail|
|SWISSCOM Swisscom (Switzerland) Ltd, CH (AS 3303)||6/24/19 12:01||More detail|
|T-MOBILE-AS21928 – T-Mobile USA, Inc., US (AS 21928)||6/24/19 12:01||More detail|
|LWLCOM, DE (AS 50629)||6/24/19 12:01||More detail|
|KNIPP-AMS-AS, DE (AS 48519)||6/24/19 11:27||More detail|
|LRC-104 – LRC GROUP LLC, US (AS 26465)||6/24/19 11:26||More detail|
|EWETEL Cloppenburger Strasse 310, DE (AS 9145)||6/24/19 10:40||More detail|
|OVH, FR (AS 16276||6/24/19 10:40||More detail|
|VALVE-CORPORATION – Valve Corporation, US (AS 32590)||6/24/19 10:39||More detail|
|CYBERCON – CYBERCON, INC., US (AS 7393)||6/24/19 10:39||More detail|
|CHARTER-20115 – Charter Communications, US (AS 20115)||6/24/19 10:39||More detail|
|SHAW – Shaw Communications Inc., CA (AS 6327)||6/24/19 10:39||More detail|
|CHINANET-BACKBONE No.31,Jin-rong Street, CN (AS 4134)||6/24/19 10:39||More detail|
|UNWIRED – Unwired, US (AS 32354)||6/24/19 10:39||More detail|
|SPRINTLINK2 – Sprint Government Systems Division, US (AS 1790)||6/24/19 10:39||More detail|
|BBIL-AP BHARTI Airtel Ltd., IN (AS 9498)||6/24/19 10:39||More detail|
|FACEBOOK – Facebook, Inc., US (AS 32934)||6/24/19 10:39||More detail|
|WEBZILLA, NL (AS 35415)||6/24/19 10:39||More detail|
|NEWPIRVINE – Newport Corporation, US (AS 26487)||6/24/19 10:39||More detail|
|FASTWEB, IT (AS 12874)||6/24/19 10:39||More detail|
|ASN-DIS – Dallas Infrastructure Services, LLC, US (AS 393398)||6/24/19 10:39||More detail|
|WVTM-TV – HEARST CORPORATION, US (AS 54204)||6/24/19 10:39||More detail|
|ABRAMSCAPITALMANAGEMENTLLC – Abrams Capital Management, LLC, US (AS 29786)||6/24/19 10:39||More detail|
|COSLINK – Cherryland Services Inc, US (AS 13904)||6/24/19 10:39||More detail|
|REALNET – RealNetworks, Inc., US (AS 26353)||6/24/19 10:38||More detail|
|GRAVITAS-TECHNOLOGY-SERVICES-LLC – Gravitas Technology Services, LLC, US (AS 54566)||6/24/19 10:38||More detail|
|PSAM-ASN – P. Schoenfeld Asset Management, LLC, US (AS 26200)||6/24/19 10:38||More detail|
|TAMPNET-AS Tampnet International Carrier, NO (AS 200781)||6/24/19 10:38||More detail|
|DEBEVOISE – Debevoise and Plimpton LLP, US (AS 394122)||6/24/19 10:37||More detail|
|(AS 2147483647)||6/24/19 10:37||More detail|
|(AS 2147483647)||6/24/19 10:37||More detail|
|MINNESOTATWINSLLC – Minnesota Twins, LLC, US (AS 16915)||6/24/19 10:37||More detail|
|FMCTI-FMC-TECHNOLOGIES-INC-US – FMC Technologies, Inc., US (AS 394936)||6/24/19 10:37||More detail|
|ESWIKIKER, ES (AS 200845)||6/24/19 10:36||More detail|
|PITNEYBOWES-AS-1 – Pitney Bowes Incorporated, US (AS 11086)||6/24/19 10:36||More detail|
|ELYNX-CORP-NET – eLynx Ltd., US (AS 17259)||6/24/19 10:36||More detail|
|DESERTSCHOOLS-ASN-1 – Desert Schools Federal Credit Union, US (AS 14590)||6/24/19 10:36||More detail|
|BLACKROCK-FINANCIAL-MANAGEMENT-2 – BlackRock Financial Management, Inc., US (AS 31747)||6/24/19 10:36||More detail|
|FMA-ALLIANCE-LTD – FMA Alliance, Ltd., US (AS 393576)||6/24/19 10:36||More detail|
|AS-AFILIAS-REGISTRY-SERVICES – Afilias Canada, Corp., CA (AS 21775)||6/24/19 10:36||More detail|
|AMCON – AMCON Distributing Company, US (AS 26373)||6/24/19 10:36||More detail|
|USL-ASN – Upsher-Smith Laboratories Inc., US (AS 13715)||6/24/19 10:36||More detail|
|PMA-INTERNET – Preferred Managing Agency, LLC, US (AS 395869)||6/24/19 10:36||More detail|
|BREMERFINANCIAL – BREMER FINANCIAL SERVICES, US (AS 55153)||6/24/19 10:36||More detail|
|HNS-DIRECPC – Hughes Network Systems, US (AS 6621)||6/24/19 10:36||More detail|
|TELEPERFORMANCE-USA – TELEPERFORMANCE USA, US (AS 33680)||6/24/19 10:36||More detail|
|FARM-CREDIT – Farm Credit Financial Partners, Inc., US (AS 16910)||6/24/19 10:36||More detail|
|MHS – Meridian Health System, US (AS 1436)||6/24/19 10:36||More detail|
|SCHOLASTIC – Scholastic Inc., US (AS 10717)||6/24/19 10:36||More detail|
|BCIU-22 – Bucks County Intermediate Unit #22, US (AS 16958)||6/24/19 10:36||More detail|
|ROGERS-COMMUNICATIONS – Rogers Communications Canada Inc., CA (AS 812)||6/24/19 10:36||More detail|
|KANREN – Kansas Research and Education Network, US (AS 2495)||6/24/19 10:35||More detail|
|VITU-49 – Virtuoso, Ltd., US (AS 53890)||6/24/19 10:35||More detail|
|PROSKAUER-1 – Proskauer Rose LLP, US (AS 25955)||6/24/19 10:35||More detail|
|SLG – SL Green Realty Corp, US (AS 46152)||6/24/19 10:35||More detail|
|CAL-NET-INC – Cal.net, Inc., US (AS 21580)||6/24/19 10:35||More detail|
|SCATUI – SCATUI, US (AS 46294)||6/24/19 10:35||More detail|
|RACKNET1 – Rack Wizards Inc, US (AS 46897)||6/24/19 10:34||More detail|
|NETSPECTRUM – Netspectrum Wireless Internet Solutions, CA (AS 22060)||6/24/19 10:34||More detail|
|MOTOROLA-MOBILITY – Motorola, Inc., US (AS 22244)||6/24/19 10:34||More detail|
|SHOPZILLA – Connexity, Inc., US (AS 14332)||6/24/19 10:34||More detail|
|-Reserved AS-, ZZ (AS 21666)||6/24/19 10:34||More detail|
|SEATTLE – City of Seattle, Dept. of Admin. Services, US (AS 3401)||6/24/19 10:34||More detail|
|COG-SEOVEC-ISP – COG-SEOVEC, US (AS 30298)||6/24/19 10:34||More detail|
While these blog posts generally cover large-scale, country-level disruptions, easily identifiable within publicly available monitoring tools, more localized disruptions are often visible through these tools as well.
On June 7, Consolidated Communications, a network provider that services Vermont, reportedly experienced a “router issue” at its office in Burlington, disrupting connectivity to all 80,000 Internet customers in the state, including businesses and homes. The figures below show the impact of the disruption, with active measurement metrics declining for a brief period of time. Although the autonomous system label in the graphs is for FairPoint Communications, the related PeeringDB entry shows it is owned by Consolidated Communications. (FairPoint was acquired by Consolidated in 2016.)
The @outagedetect Twitter account, associated with Internet connection and connected device monitoring company Fing, posts numerous alerts each day about localized Internet outages. On June 19, it posted the Tweet below, highlighting the end of a “considerable” hour-long Internet outage detected in Poland for users connected to network service provider Orange Polska.
One of the figures below shows that this “considerable” outage impacted CAIDA’s Active Probing measurements for Poland at a country level, as well as driving a concurrent decline in Darknet sources. The other figure shows the impact at a network level, with Active Probing measurements and Darknet sources for Orange Polska dropping considerably during the hour-long disruption.
While the major Internet disruptions observed in June were largely due to a familiar set of underlying causes, many of these causes remain avoidable.
Of course, convincing governments not to implement Internet outages during times of political unrest is a non-trivial effort, but organizations including the Internet Society and Access Now continue to advocate for user rights and drive #KeepItOn efforts.
In contrast, convincing network providers, especially Tier 1 global providers, to improve their routing hygiene through through participation in efforts like MANRS should be comparatively trivial.
[…] a terrible, horrible, no good, very bad month in June, the number and severity of significant Internet disruptions observed in July was markedly lower. […]