Google data centers

Last updated

Former Google data center in Eemshaven, Netherlands Google datacenter (2007) - panoramio - erwinboogert (2).jpg
Former Google data center in Eemshaven, Netherlands
External videos
YouTube Logo 2017.svg
YouTube
Nuvola apps kaboodle.svg Google Data Center 360° Tour

Google data centers are the large data center facilities Google uses to provide their services, which combine large drives, computer nodes organized in aisles of racks, internal and external networking, environmental controls (mainly cooling and humidification control), and operations software (especially as concerns load balancing and fault tolerance).

Contents

There is no official data on how many servers are in Google data centers, but Gartner estimated in a July 2016 report that Google at the time had 2.5 million servers. This number is changing as the company expands capacity and refreshes its hardware. [1]

Locations

The locations of Google's various data centers by continent are as follows: [2] [3]

ContinentLocationGeoProducts LocationCloud LocationTimelineDescription
North America Arcola (VA), USA 38°56′35.99″N77°31′27.61″W / 38.9433306°N 77.5243361°W / 38.9433306; -77.5243361 Loudoun County N. Virginia (us-east4)2017 - announced [4] [5]
North America Atlanta (GA), USA 33°44′59.04″N84°35′5.33″W / 33.7497333°N 84.5848139°W / 33.7497333; -84.5848139 Douglas County -2003 - launched350 employees
South America Cerrillos, Santiago, Chile 33°31′14″S70°43′18″W / 33.520515°S 70.721695°W / -33.520515; -70.721695 [6] -Santiago (southamerica-west1)2020 - announced [7]

2021 - launched [8]

Asia Changhua County, Taiwan 24°08′18.6″N120°25′32.6″E / 24.138500°N 120.425722°E / 24.138500; 120.425722 Changhua County Taiwan

(asia-east1)

2011 - announced

2013 - launched

60 employees
North America Clarksville (TN), USA 36°37′16″N87°15′47″W / 36.6211599°N 87.2630735°W / 36.6211599; -87.2630735 Montgomery County -2015 - announced
North America Columbus (OH), USA -Columbus (us-east5)2022 - launched [9]
North America Council Bluffs (IA), USA 41°13′17.7″N95°51′49.92″W / 41.221583°N 95.8638667°W / 41.221583; -95.8638667 Council Bluffs 2007 - announced

2009 - completed first phase completed

2012 and 2015 - expanded

130 employees
North America Council Bluffs (IA), USA 41°10′06″N95°47′46″W / 41.1682533°N 95.7961246°W / 41.1682533; -95.7961246 Iowa (us-central1)
Asia Delhi, India -Delhi (asia-south2)2020 - announced

2021 - launched [10]

Middle East Doha, Qatar -Doha (me-central1)2023 - launched [11]
Europe Dublin, Ireland 53°19′12.39″N6°26′31.43″W / 53.3201083°N 6.4420639°W / 53.3201083; -6.4420639 Dublin -2011 - announced

2012 - launched

150 employees [12]
Europe Eemshaven, Netherlands 53°25′31″N6°51′44″E / 53.4252171°N 6.8622574°E / 53.4252171; 6.8622574 Eemshaven 2014 - announced

2016 - launched

2018, 2019 - expansion

200 employees
Europe Frankfurt, Germany 50°07′21″N8°58′27″E / 50.1226299°N 8.9741678°E / 50.1226299; 8.9741678 [13] -Frankfurt (europe-west3)2022 - expanded [14]
Europe Fredericia, Denmark 55°33′29.5″N9°39′20.8″E / 55.558194°N 9.655778°E / 55.558194; 9.655778 Fredericia -2018 - announced [15]

2020 - launched

€600M building costs
Europe Ghlin, Hainaut, Belgium 50°28′09.6″N3°51′55.7″E / 50.469333°N 3.865472°E / 50.469333; 3.865472 Saint-Ghislain Belgium (europe-west1)2007 - announced

2010 - launched

12 employees
Europe Hamina, Finland 60°32′11.68″N27°7′1.21″E / 60.5365778°N 27.1170028°E / 60.5365778; 27.1170028 Hamina Finland

(europe-north1)

2009 - announced

2011 - first phase completed

2022 - expansion

6 buildings, 400 employees [16]
North America Henderson (NV), USA 36°03′20″N115°00′37″W / 36.0556248°N 115.0102258°W / 36.0556248; -115.0102258 Henderson Las Vegas (us-west4)2019 - announced [17]

2020 - launched

64-acres; $1.2B building costs [18] [19]
Asia Hong Kong, Hong Kong -Hong Kong (asia-east2)2017 - announced [20]

2018 - launched [21]

Asia Inzai, Japan 35°49′04″N140°07′57″E / 35.8177804°N 140.1323916°E / 35.8177804; 140.1323916 Inzai -2023 - launched
Asia Jakarta, Indonesia -Jakarta (asia-southeast2)2020 - launched [22]
Asia Koto-Ku, Tokyo, Japan -Tokyo

(asia-northeast1)

2016 - launched [23]
North America Leesburg (VA), USA 39°3′6.38″N77°32′20.38″W / 39.0517722°N 77.5389944°W / 39.0517722; -77.5389944 Loudoun County N. Virginia (us-east4)2017 - announced [4] [5]
North America Lenoir (NC), USA 35°53′54.78″N81°32′50.58″W / 35.8985500°N 81.5473833°W / 35.8985500; -81.5473833 Lenoir -2007 - announced

2009 - launched

over 110 employees
AsiaLok Yang Way, Pioneer, Singapore 1°19′26″N103°41′36″E / 1.3239859°N 103.693253°E / 1.3239859; 103.693253 [24] Singapore Singapore (asia-southeast1)2022 - launched
Europe London, UK -London

(europe-west2)

2017 - launched [25]
North America Los Angeles (CA), USA -Los Angeles (us-west2)
Europe Madrid, Spain 40°31′10″N3°20′27″W / 40.5195333°N 3.3409366°W / 40.5195333; -3.3409366 -Madrid (europe-southwest1)2022 - launched [26]
Pacific Melbourne, Australia -Melbourne

(australia-southeast2)

2021 - launched [27]
Europe Middenmeer, Noord-Holland, The Netherlands 52°47′24″N5°01′45″E / 52.7901053°N 5.0292193°E / 52.7901053; 5.0292193 [28] Middenmeer Netherlands (europe-west4)2019 - announced [29]
North America Midlothian (TX), USA 32°26′35″N97°03′44″W / 32.443170°N 97.062324°W / 32.443170; -97.062324 Midlothian Dallas (us-south1)2019 - announced

2022 - launched [30]

375-acres; $600M building costs [31]
Europe Milan, Italy -Milan (europe-west8)2022 - launched [32]
North America Moncks Corner (SC), USA 33°03′50.8″N80°02′36.1″W / 33.064111°N 80.043361°W / 33.064111; -80.043361 Berkeley County South Carolina (us-east1)2007 - launched

2013 - expanded

150 employees
North America Montreal, Quebec, Canada [33] -Montréal (northamerica-northeast1)2018 - launched [34] 62.4-hectares; $600M building costs [35]
Asia Mumbai, India -Mumbai (asia-south1)2017 - launched [36]
North America New Albany (OH), USA 40°03′41″N82°45′31″W / 40.0613352°N 82.7586278°W / 40.0613352; -82.7586278 New Albany -2019 - announced400-acres; $600M building costs [37] [38]
Asia Osaka, Japan -Osaka

(asia-northeast2)

2019 - launched [39]
South America Osasco, São Paulo, Brazil -São Paulo (southamerica-east1)2017 - launched [40]
North America Papillion (NE), USA 41°08′00″N96°08′39″W / 41.1332915°N 96.144178°W / 41.1332915; -96.144178 Papillion -2019 - announced275-acres; $600M building costs [41] [42]
Europe Paris, France -Paris (europe-west9)2022 - launched [43]
North America Pryor Creek (OK), USA 36°14′28.1″N95°19′48.22″W / 36.241139°N 95.3300611°W / 36.241139; -95.3300611 Mayes County -2007 - announced

2012 - expanded

over 400 employees, [44] land at MidAmerica Industrial Park
South America Quilicura, Santiago, Chile 33°21′30.5″S70°41′50.4″W / 33.358472°S 70.697333°W / -33.358472; -70.697333 Quilicura -2012 - announced

2015 - launched

up to 20 employees expected. A million dollar investment plan to increase capacity at Quilicura was announced in 2018. [45]
North America Reno (NV), USA 39°30′04″N119°25′46″W / 39.5011558°N 119.4295537°W / 39.5011558; -119.4295537 Storey County -2017 - 1,210 acres of land bought in the Tahoe Reno Industrial Center [46]

2018 - announced

2018 November - project approved by the state of Nevada [47] [48]

North America Salt Lake City (UT), USA -Salt Lake City (us-west3)2020 - launched [49]
Asia Seoul, South Korea -Seoul

(asia-northeast3)

2020 - launched [50]
Pacific Sydney, Australia -Sydney

(australia-southeast1)

2017 - launched [51]
Middle East Tel Aviv, Israel [52] -Tel Aviv (me-west1)2022 - launched [53]
North America The Dalles (OR), USA 45°37′57.04″N121°12′8.16″W / 45.6325111°N 121.2022667°W / 45.6325111; -121.2022667 The Dalles Oregon (us-west1)2006 - launched80 full-time employees
North America Toronto, Canada -Toronto (northamerica-northeast2)2021 - launched [54]
Europe Turin, Italy 45°08′48″N7°44′32″E / 45.146729°N 7.742147°E / 45.146729; 7.742147 -Turin (europe-west12)2023 - launched [55]
South America Vinhedo, São Paulo, Brazil São Paulo (southamerica-east1)
Europe Warsaw, Poland -Warsaw (europe-central2)2019 - announced

2021 - launched [56]

Asia Wenya, Jurong West, Singapore 1°21′04.8″N103°42′35.2″E / 1.351333°N 103.709778°E / 1.351333; 103.709778 Singapore Singapore (asia-southeast1)2011 - announced

2013 - launched

2015 - expanded

North America Widows Creek (Bridgeport) (AL), USA 34°54′48.4″N85°44′53.1″W / 34.913444°N 85.748083°W / 34.913444; -85.748083 [57] Jackson County -2018 - broke ground
Europe Zürich, Switzerland 47°26′45″N8°12′39″E / 47.4459257°N 8.2109085°E / 47.4459257; 8.2109085 [58] -Zurich (europe-west6)2018 - announced

2019 - launched [59]

Europe Austria 2022 - announced [60]
Europe Berlin, Germany [61] Berlin (europe-west10)2021 - announced [62] 2023 August - launched [63]
Middle East Dammam, Saudi Arabia 2021 - announced [64]
Europe Athens, Greece 2022 - announced [60]
North America Kansas City, Missouri 2019 - announced [65]
Middle East Kuwait 2023 - announced [66]
Asia Malaysia 2022 - announced [67]
Pacific Auckland, New Zealand 2022 - announced [67]
Europe Oslo, Norway 2022 - announced [60]
North America Querétaro, Mexico 2022 - announced [68]
Africa Johannesburg, South Africa Johannesburg (africa-south1)2022 - announced [60] 2024 - launched
Europe Sweden 2022 - announced [60]
Asia Tainan City, Taiwan -Taiwan

(asia-east1)

2019 September - announced [69] [70] [71]
Asia Thailand 2022 - announced [67]
Asia Yunlin County, Taiwan -Taiwan (asia-east1)2020 September - announced [72]
Europe Waltham Cross, Hertfordshire, UK 51°41′44″N0°02′55″W / 51.695666°N 0.048728°W / 51.695666; -0.048728 2024 January - announced [73]

Hardware

Original hardware

Google's first production server rack, circa 1998 Googles First Server.jpg
Google's first production server rack, circa 1998

The original hardware (circa 1998) that was used by Google when it was located at Stanford University included: [74]

Production hardware

As of 2014, Google has used a heavily customized version of Debian Linux. They migrated from a Red Hat-based system incrementally in 2013. [75]

The customization goal is to purchase CPU generations that offer the best performance per dollar, not absolute performance. How this is measured is unclear, but it is likely to incorporate running costs of the entire server, and CPU power consumption could be a significant factor. [76] Servers as of 2009–2010 consisted of custom-made open-top systems containing two processors (each with several cores [77] ), a considerable amount of RAM spread over 8 DIMM slots housing double-height DIMMs, and at least two SATA hard disk drives connected through a non-standard ATX-sized power supply unit. [78] The servers were open top so more servers could fit into a rack. According to CNET and a book by John Hennessy, each server had a novel 12-volt battery to reduce costs and improve power efficiency. [77] [79]

According to Google, their global data center operation electrical power ranges between 500 and 681 megawatts. [80] [81] The combined processing power of these servers might have reached from 20 to 100 petaflops in 2008. [82]

Network topology

Details of the Google worldwide private networks are not publicly available, but Google publications [83] [84] make references to the "Atlas Top 10" report that ranks Google as the third largest ISP behind Level 3.

In order to run such a large network, with direct connections to as many ISPs as possible at the lowest possible cost, Google has a very open peering policy. [85]

From this site, we can see that the Google network can be accessed from 67 public exchange points and 69 different locations across the world. As of May 2012, Google had 882 Gbit/s of public connectivity (not counting private peering agreements that Google has with the largest ISPs). This public network is used to distribute content to Google users as well as to crawl the internet to build its search indexes. The private side of the network is a secret, but a recent disclosure from Google [86] indicate that they use custom built high-radixswitch-routers (with a capacity of 128 × 10 Gigabit Ethernet port) for the wide area network. Running no less than two routers per datacenter (for redundancy) we can conclude that the Google network scales in the terabit per second range (with two fully loaded routers the bi-sectional bandwidth amount to 1,280 Gbit/s).

These custom switch-routers are connected to DWDM devices to interconnect data centers and point of presences (PoP) via dark fiber.

From a datacenter view, the network starts at the rack level, where 19-inch racks are custom-made and contain 40 to 80 servers (20 to 40 1U servers on either side, while new servers are 2U rackmount systems. [87] Each rack has an Ethernet switch). Servers are connected via a 1 Gbit/s Ethernet link to the top of rack switch (TOR). TOR switches are then connected to a gigabit cluster switch using multiple gigabit or ten gigabit uplinks. [88] The cluster switches themselves are interconnected and form the datacenter interconnect fabric (most likely using a dragonfly design rather than a classic butterfly or flattened butterfly layout [89] ).

From an operation standpoint, when a client computer attempts to connect to Google, several DNS servers resolve www.google.com into multiple IP addresses via Round Robin policy. Furthermore, this acts as the first level of load balancing and directs the client to different Google clusters. A Google cluster has thousands of servers, and once the client has connected to the server additional load balancing is done to send the queries to the least loaded web server. This makes Google one of the largest and most complex content delivery networks. [90]

Google has numerous data centers scattered around the world. At least 12 significant Google data center installations are located in the United States. The largest known centers are located in The Dalles, Oregon; Atlanta, Georgia; Reston, Virginia; Lenoir, North Carolina; and Moncks Corner, South Carolina. [91] In Europe, the largest known centers are in Eemshaven and Groningen in the Netherlands and Mons, Belgium. [91] Google's Oceania Data Center is located in Sydney, Australia. [92]

Data center network topology

To support fault tolerance, increase the scale of data centers and accommodate low-radix switches, Google has adopted various modified Clos topologies in the past. [93]

Project 02

Google data center in The Dalles, Oregon Google data center.jpg
Google data center in The Dalles, Oregon

One of the largest Google data centers is located in the town of The Dalles, Oregon, on the Columbia River, approximately 80 miles (129 km) from Portland. Codenamed "Project 02", the complex was built in 2006 and is approximately the size of two American football fields, with cooling towers four stories high. [94] [95] The site was chosen to take advantage of inexpensive hydroelectric power, and to tap into the region's large surplus of fiber optic cable, a remnant of the dot-com boom. A blueprint of the site appeared in 2008. [96]

Summa papermill

In February 2009, Stora Enso announced that they had sold the Summa paper mill in Hamina, Finland to Google for 40 million Euros. [97] [98] Google invested 200 million euros on the site to build a data center and announced additional 150 million euro investment in 2012. [99] [100] Google chose this location due to the availability and proximity of renewable energy sources. [101]

Floating data centers

In 2013, the press revealed the existence of Google's floating data centers along the coasts of the states of California (Treasure Island's Building 3) and Maine. The development project was maintained under tight secrecy. The data centers are 250 feet long, 72 feet wide, 16 feet deep. The patent for an in-ocean data center cooling technology was bought by Google in 2009 [102] [103] (along with a wave-powered ship-based data center patent in 2008 [104] [105] ). Shortly thereafter, Google declared that the two massive and secretly-built infrastructures were merely "interactive learning centers, [...] a space where people can learn about new technology." [106]

Google halted work on the barges in late 2013 and began selling off the barges in 2014. [107] [108]

Software

Most of the software stack that Google uses on their servers was developed in-house. [109] According to a well-known Google employee, C++, Java, Python and (more recently) Go are favored over other programming languages. [110] For example, the back end of Gmail is written in Java and the back end of Google Search is written in C++. [111] Google has acknowledged that Python has played an important role from the beginning, and that it continues to do so as the system grows and evolves. [112]

The software that runs the Google infrastructure includes: [113]

Google has developed several abstractions which it uses for storing most of its data: [121]

Software development practices

Most operations are read-only. When an update is required, queries are redirected to other servers, so as to simplify consistency issues. Queries are divided into sub-queries, where those sub-queries may be sent to different ducts in parallel, thus reducing the latency time. [87]

To lessen the effects of unavoidable hardware failure, software is designed to be fault tolerant. Thus, when a system goes down, data is still available on other servers, which increases reliability.

Search infrastructure

Google data center in The Dalles, Oregon Google Data Center, The Dalles.jpg
Google data center in The Dalles, Oregon

Index

Like most search engines, Google indexes documents by building a data structure known as inverted index. Such an index obtains a list of documents by a query word. The index is very large due to the number of documents stored in the servers. [90]

The index is partitioned by document IDs into many pieces called shards. Each shard is replicated onto multiple servers. Initially, the index was being served from hard disk drives, as is done in traditional information retrieval (IR) systems. Google dealt with the increasing query volume by increasing number of replicas of each shard and thus increasing number of servers. Soon they found that they had enough servers to keep a copy of the whole index in main memory (although with low replication or no replication at all), and in early 2001 Google switched to an in-memory index system. This switch "radically changed many design parameters" of their search system, and allowed for a significant increase in throughput and a large decrease in latency of queries. [126]

In June 2010, Google rolled out a next-generation indexing and serving system called "Caffeine" which can continuously crawl and update the search index. Previously, Google updated its search index in batches using a series of MapReduce jobs. The index was separated into several layers, some of which were updated faster than the others, and the main layer wouldn't be updated for as long as two weeks. With Caffeine, the entire index is updated incrementally on a continuous basis. Later Google revealed a distributed data processing system called "Percolator" [127] which is said to be the basis of Caffeine indexing system. [119] [128]

Server types

Google's server infrastructure is divided into several types, each assigned to a different purpose: [87] [90] [129] [130] [131]

Security

External videos
YouTube Logo 2017.svg
YouTube
Nuvola apps kaboodle.svg Google Data Center Security: 6 Layers Deep

In October 2013, The Washington Post reported that the U.S. National Security Agency intercepted communications between Google's data centers, as part of a program named MUSCULAR. [132] [133] This wiretapping was made possible because, at the time, Google did not encrypt data passed inside its own network. [134] This was rectified when Google began encrypting data sent between data centers in 2013. [135]

Environmental impact

Google data center in Mayes County, Oklahoma at MidAmerica Industrial Park Google Mayes County P0004991a.jpg
Google data center in Mayes County, Oklahoma at MidAmerica Industrial Park

Google's most efficient data center runs at 35 °C (95 °F) using only fresh air cooling, requiring no electrically powered air conditioning. [136]

In December 2016, Google announced that—starting in 2017—it would purchase enough renewable energy to match 100% of the energy usage of its data centers and offices. The commitment will make Google "the world's largest corporate buyer of renewable power, with commitments reaching 2.6 gigawatts (2,600 megawatts) of wind and solar energy". [137] [138] [139]

Related Research Articles

<span class="mw-page-title-main">Windows 2000</span> Fifth major release of Windows NT, released in 2000

Windows 2000 is a major release of the Windows NT operating system developed by Microsoft and designed for businesses. It was the direct successor to Windows NT 4.0, and was released to manufacturing on December 15, 1999, and was officially released to retail on February 17, 2000 and September 26, 2000 for Windows 2000 Datacenter Server. It was Microsoft's business operating system until the introduction of Windows XP Professional in 2001.

<span class="mw-page-title-main">Spotlight (Apple)</span> macOS search feature

Spotlight is a system-wide desktop search feature of Apple's macOS and iOS operating systems. Spotlight is a selection-based search system, which creates an index of all items and files on the system. It is designed to allow the user to quickly locate a wide variety of items on the computer, including documents, pictures, music, applications, and System Settings. In addition, specific words in documents and in web pages in a web browser's history or bookmarks can be searched. It also allows the user to narrow down searches with creation dates, modification dates, sizes, types and other attributes. Spotlight also offers quick access to definitions from the built-in New Oxford American Dictionary and to calculator functionality. There are also command-line tools to perform functions such as Spotlight searches.

<span class="mw-page-title-main">Desktop search</span>

Desktop search tools search within a user's own computer files as opposed to searching the Internet. These tools are designed to find information on the user's PC, including web browser history, e-mail archives, text documents, sound files, images, and video. A variety of desktop search programs are now available; see this list for examples. Most desktop search programs are standalone applications. Desktop search products are software alternatives to the search software included in the operating system, helping users sift through desktop files, emails, attachments, and more.

<span class="mw-page-title-main">Urs Hölzle</span> Swiss computer scientist

Urs Hölzle is a Swiss software engineer and technology executive. As Google's eighth employee and its first VP of Engineering, he has shaped much of Google's development processes and infrastructure, as well as its engineering culture. His most notable contributions include leading the development of fundamental cloud infrastructure such as energy-efficient data centers, distributed compute and storage systems, and software-defined networking. Until July 2023, he was the Senior Vice President of Technical Infrastructure and Google Fellow at Google. In July 2023, he transitioned to being a Google Fellow only.

Bigtable is a fully managed wide-column and key-value NoSQL database service for large analytical and operational workloads as part of the Google Cloud portfolio.

<span class="mw-page-title-main">Apache Solr</span> Open-source enterprise-search platform

Solr is an open-source enterprise-search platform, written in Java. Its major features include full-text search, hit highlighting, faceted search, real-time indexing, dynamic clustering, database integration, NoSQL features and rich document handling. Providing distributed search and index replication, Solr is designed for scalability and fault tolerance. Solr is widely used for enterprise search and analytics use cases and has an active development community and regular releases.

Microsoft SQL Server is a proprietary relational database management system developed by Microsoft. As a database server, it is a software product with the primary function of storing and retrieving data as requested by other software applications—which may run either on the same computer or on another computer across a network. Microsoft markets at least a dozen different editions of Microsoft SQL Server, aimed at different audiences and for workloads ranging from small single-machine applications to large Internet-facing applications with many concurrent users.

<span class="mw-page-title-main">Windows Search</span> Desktop search platform by Microsoft

Windows Search is a content index desktop search platform by Microsoft introduced in Windows Vista as a replacement for both the previous Indexing Service of Windows 2000 and the optional MSN Desktop Search for Windows XP and Windows Server 2003, designed to facilitate local and remote queries for files and non-file items in compatible applications including Windows Explorer. It was developed after the postponement of WinFS and introduced to Windows constituents originally touted as benefits of that platform.

<span class="mw-page-title-main">Apache Cassandra</span> Free and open-source database management system

Cassandra is a free and open-source, distributed, wide-column store, NoSQL database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure. Cassandra offers support for clusters spanning multiple datacenters, with asynchronous masterless replication allowing low latency operations for all clients. Cassandra was designed to implement a combination of Amazon's Dynamo distributed storage and replication techniques combined with Google's Bigtable data and storage engine model.

<span class="mw-page-title-main">Microsoft Azure</span> Cloud computing platform by Microsoft

Microsoft Azure, often referred to as Azure, is a cloud computing platform run by Microsoft. It offers access, management, and the development of applications and services through global data centers. It also provides a range of capabilities, including software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS). Microsoft Azure supports many programming languages, tools, and frameworks, including Microsoft-specific and third-party software and systems.

<span class="mw-page-title-main">Redis</span> Open-source in-memory key–value database

Redis is an open-source in-memory storage, used as a distributed, in-memory key–value database, cache and message broker, with optional durability. Because it holds all data in memory and because of its design, Redis offers low-latency reads and writes, making it particularly suitable for use cases that require a cache. Redis is the most popular NoSQL database, and one of the most popular databases overall. Redis is used in companies like Twitter, Airbnb, Tinder, Yahoo, Adobe, Hulu, Amazon and OpenAI.

<span class="mw-page-title-main">Couchbase Server</span> Open-source NoSQL database

Couchbase Server, originally known as Membase, is a source-available, distributed multi-model NoSQL document-oriented database software package optimized for interactive applications. These applications may serve many concurrent users by creating, storing, retrieving, aggregating, manipulating and presenting data. In support of these kinds of application needs, Couchbase Server is designed to provide easy-to-scale key-value, or JSON document access, with low latency and high sustainability throughput. It is designed to be clustered from a single machine to very large-scale deployments spanning many machines.

<span class="mw-page-title-main">OVHcloud</span> French web hosting and cloud computing company

OVH, legally OVH Groupe SA, is a French cloud computing company which offers VPS, dedicated servers and other web services. As of 2016 OVH owned the world's largest data center in surface area. As of 2019, it was the largest hosting provider in Europe, and the third largest in the world based on physical servers. The company was founded in 1999 by the Klaba family and is headquartered in Roubaix, France. OVH is incorporated as a simplified joint-stock company under French law. In 2019 OVH adopted OVHcloud as its public brand name.

<span class="mw-page-title-main">Elasticsearch</span> Search engine

Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java and is dual-licensed under the source-available Server Side Public License and the Elastic license, while other parts fall under the proprietary (source-available) Elastic License. Official clients are available in Java, .NET (C#), PHP, Python, Ruby and many other languages. According to the DB-Engines ranking, Elasticsearch is the most popular enterprise search engine.

Google Compute Engine (GCE) is the Infrastructure as a Service (IaaS) component of Google Cloud Platform which is built on the global infrastructure that runs Google's search engine, Gmail, YouTube and other services. Google Compute Engine enables users to launch virtual machines (VMs) on demand. VMs can be launched from the standard images or custom images created by users. GCE users must authenticate based on OAuth 2.0 before launching the VMs. Google Compute Engine can be accessed via the Developer Console, RESTful API or command-line interface (CLI).

<span class="mw-page-title-main">SAP HANA</span> Database management system by SAP

SAP HANA is an in-memory, column-oriented, relational database management system developed and marketed by SAP SE. Its primary function as the software running a database server is to store and retrieve data as requested by the applications. In addition, it performs advanced analytics and includes extract, transform, load (ETL) capabilities as well as an application server.

Google Cloud Platform (GCP), offered by Google, is a suite of cloud computing services that provides a series of modular cloud services including computing, data storage, data analytics, and machine learning, alongside a set of management tools. It runs on the same infrastructure that Google uses internally for its end-user products, such as Google Search, Gmail, and Google Docs, according to Verma, et.al. Registration requires a credit card or bank account details.

The history of Microsoft SQL Server begins with the first Microsoft SQL Server database product – SQL Server v1.0, a 16-bit relational database for the OS/2 operating system, released in 1989.

<span class="mw-page-title-main">Swisscows</span> Web search engine operated by Hulbee AG

Swisscows is a web search engine launched in 2014, a project of Hulbee AG, a company based in Egnach, Switzerland. It uses semantic data recognition that gives faster answers to queries and claims to not store users' data. Swisscows also deems itself family-friendly, with explicit results entirely omitted. The engine's servers are based in underground data centers under the Swiss Alps, and geographically outside of EU and US.

RavenDB is an open-source document-oriented database written in C#, developed by Hibernating Rhinos Ltd. It is cross-platform, supported on Windows, Linux, and Mac OS. RavenDB stores data as JSON documents and can be deployed in distributed clusters with master-master replication.

References

  1. ↑ "How Many Servers Does Google Have?". Data Center Knowledge. March 16, 2017. Retrieved September 20, 2018.
  2. ↑ "Google data centers, locations" . Retrieved July 21, 2014.
  3. ↑ "ISO/IEC 27001 - Compliance". Google Cloud. Retrieved July 11, 2023.
  4. 1 2 Report, Times-Mirror Staff (June 19, 2019). "Google 'caps off' $600M investment in Loudoun County". LoudounTimes.com. Retrieved February 12, 2021.
  5. 1 2 "Google Plans 2 Loudoun Data Centers". LoudounNow.com. November 29, 2017. Retrieved April 12, 2023.
  6. ↑ Arellano (CIPER), Alberto (May 25, 2020). "Las zonas oscuras de la evaluación ambiental que autorizó "a ciegas" el megaproyecto de Google en Cerrillos". CIPER Chile (in Spanish). Retrieved July 10, 2023.
  7. ↑ "Google instalará un nuevo data center en Chile". diarioeldia.cl (in Spanish). Archived from the original on September 22, 2021. Retrieved September 28, 2019.
  8. ↑ "Una nueva Región de Google Cloud en Santiago, todo el potencial de la nube ahora más cerca". Google. December 1, 2021. Retrieved July 10, 2023.
  9. ↑ "A Google Cloud region now available in Columbus, Ohio". Google Cloud Blog. Retrieved July 10, 2023.
  10. ↑ "Namaste, India. Our new cloud region in Delhi NCR is now live". cloudonair.withgoogle.com.
  11. ↑ "New Google Cloud region now open in Qatar". Google Cloud Blog. Retrieved July 10, 2023.
  12. ↑ "Dublin, Ireland – Data Centers – Google". www.google.com. Retrieved April 2, 2019.
  13. ↑ "PROJEKT IN HANAU: RECHENZENTRUM FÜR GOOGLE ENTWICKELT". NDC-GARBE. September 2, 2021.
  14. ↑ "Google Cloud investing in Germany with new infrastructure and sustainable energy". Google Cloud Blog. Retrieved July 10, 2023.
  15. ↑ "Breaking ground for Google's first data center in Denmark". Google. November 20, 2018. Retrieved July 10, 2023.
  16. ↑ "Google gets green light for its sixth data center in Hamina, Finland". September 11, 2020.
  17. ↑ "Investing in Google infrastructure, investing in Nevada". Google Cloud Blog. Retrieved July 10, 2023.
  18. ↑ Torres-Cortez, Ricardo (September 16, 2020). "Google to invest additional $600M at Henderson data center – Las Vegas Sun Newspaper". lasvegassun.com. Retrieved February 12, 2021. "With this latest announcement, Google will bring their total investment in the city of Henderson to $1.2 billion," said Mayor Debra March in the release
  19. ↑ Baxtel. "Google Henderson NV Data Center". baxtel.com. Retrieved February 12, 2021.
  20. ↑ "Coming soon: GCP's Hong Kong region". Google Cloud Blog. Retrieved July 10, 2023.
  21. ↑ "Growing our presence in Asia Pacific: New GCP regions in Hong Kong and Jakarta". Google Cloud Blog. Retrieved July 10, 2023.
  22. ↑ "The new Google Cloud region in Jakarta is now open". Google Cloud Blog. Retrieved September 15, 2021.
  23. ↑ "Google Cloud Platform Tokyo region now open for business". Google Cloud Blog. Retrieved July 10, 2023.
  24. ↑ "Google - Singapore". www.streetdirectory.com. Retrieved July 11, 2023.
  25. ↑ "Google Cloud Platform now open in London". Google Cloud Blog. Retrieved July 10, 2023.
  26. ↑ "New Google Cloud region in Madrid, Spain now open". Google Cloud Blog. Retrieved July 10, 2023.
  27. ↑ "The Google Cloud region in Melbourne is now open". Google Cloud Blog. Retrieved July 10, 2023.
  28. ↑ "Welcome". Agriport A7 (in Dutch). Retrieved July 10, 2023.
  29. ↑ "Google to Spend $1.1 Billion on New Data Centers in Netherlands". Data Center Knowledge. June 24, 2019.
  30. ↑ "A Google Cloud region now available in Dallas, Texas". Google Cloud Blog. Retrieved July 10, 2023.
  31. ↑ "Google's massive $600M data center takes shape in Ellis County as tech giant ups Texas presence". Dallas News. June 14, 2019. Retrieved February 12, 2021.
  32. ↑ "New Google Cloud region in Milan, Italy now open". Google Cloud Blog. Retrieved July 10, 2023.
  33. ↑ "Project to expand Google's activities in Quebec - Future computer data center in Beauharnois". www.quebec.ca (in French). Retrieved May 10, 2021.
  34. ↑ "GCP arrives in Canada with launch of Montréal region". Google Cloud Blog. Retrieved July 10, 2023.
  35. ↑ "Google's ongoing commitment in Quebec". Google Blog - Company News. Retrieved February 12, 2021.
  36. ↑ Stiver, Dave (November 1, 2017). "GCP arrives in India with launch of Mumbai region". Google Cloud Blog. Retrieved July 30, 2019.
  37. ↑ Williams, Mark. "Google joins New Albany high-tech crowd with $600 million data center". The Columbus Dispatch. Archived from the original on December 30, 2020. Retrieved February 12, 2021.
  38. ↑ "New Albany, Ohio – Data Centers – Google". Google Data Centers. Retrieved February 12, 2021.
  39. ↑ "Google Cloud launches new Osaka region to support growing customer base in Japan". Google Cloud Blog. Retrieved July 10, 2023.
  40. ↑ "GCP arrives in South America with launch of São Paulo region!". Google Cloud Blog. Retrieved July 10, 2023.
  41. ↑ "Google confirms it is behind $600m Papillion data center project". www.datacenterdynamics.com. Retrieved February 12, 2021.
  42. ↑ "Papillion, Nebraska – Data Centers – Google". Google Data Centers. Retrieved February 12, 2021.
  43. ↑ "Google Cloud region in Paris France now open". Google Cloud Blog. Retrieved July 10, 2023.
  44. ↑ Dawn-Hiscox, Tanwen (February 20, 2018). "Google to spend m on Pryor data center expansion". Data Centre Dynamics. Archived from the original on April 23, 2019. Retrieved April 23, 2019.
  45. ↑ "Google ha decido de invertir millones de dólares en su centro de datos en Chile". Newtechmag.net (in Spanish). September 28, 2018. Archived from the original on March 30, 2019. Retrieved December 8, 2018.
  46. ↑ Tanwen Dawn-Hiscox (April 18, 2017). "Google is planning a massive data center in Nevada". Datacenterdynamics.com. Retrieved December 8, 2018.
  47. ↑ Jason Hidalgo (November 16, 2018). "Nevada approves Google's M data center near Las Vegas, M in tax incentives". Rgj.com. Retrieved December 8, 2018.
  48. ↑ Jason Hidalgo (September 16, 2020). "Google to invest $600 million in data center near Reno, gets tax break". Reno Gazette Journal. Retrieved October 26, 2020. With our new data center in Storey County and our expanded investment in our Henderson site, Google will have two facilities in Nevada, bringing our total investment to over $1.88 billion.
  49. ↑ "Google Cloud region in Salt Lake City now open". Google Cloud Blog. Retrieved July 10, 2023.
  50. ↑ "New GCP Region in Seoul". Google Cloud Blog. Retrieved July 10, 2023.
  51. ↑ "Google Cloud expands to Australia with new Sydney region". Google Cloud Blog. Retrieved July 10, 2023.
  52. ↑ "Google launches GCP region in Tel Aviv, Israel". datacenterdynamics.com.
  53. ↑ "Google Cloud region in Tel Aviv Israel now open". Google Cloud Blog. Retrieved July 10, 2023.
  54. ↑ "Google Cloud Toronto region now open". Google Cloud Blog. Retrieved July 10, 2023.
  55. ↑ "New Google Cloud region in Turin, Italy now open". Google Cloud Blog. Retrieved July 10, 2023.
  56. ↑ "Google to Build Cloud Data Centers in Poland". Data Center Knowledge. September 27, 2019.
  57. ↑ "Google kicks off construction on M Alabama data center". Made in Alabama. April 9, 2018. Retrieved August 19, 2019.
  58. ↑ "Die Schweizer Google Cloud Platform zieht zu Green in den Aargau". Swiss IT Magazine (in German). Retrieved July 10, 2023.
  59. ↑ <<Cite web|url=https://www.datacenterknowledge.com/google-alphabet/google-building-cloud-data-centers-close-swiss-banks%7Ctitle=Google%5B%5D Building Cloud Data Centers Close to Swiss Banks||
  60. 1 2 3 4 5 "Introducing 5 new Google Cloud regions". Google Cloud Blog. Retrieved July 10, 2023.
  61. ↑ "Google plant ein großes Rechenzentrum südlich des Flughafens BER". Der Spiegel (in German). September 30, 2022. ISSN   2195-1349 . Retrieved July 10, 2023.
  62. ↑ "Google Cloud investing in Germany with new infrastructure and sustainable energy". Google Cloud Blog. Retrieved July 10, 2023.
  63. ↑ "Berlin-Brandenburg Google Cloud region is now open". Google Cloud Blog. Retrieved February 3, 2024.
  64. ↑ "Google Cloud Platform region updates". Google Cloud Blog. Retrieved July 10, 2023.
  65. ↑ "Google affiliate's latest move signals selection of KC for $600M data center". www.bizjournals.com. Retrieved February 12, 2021.
  66. ↑ "Bringing a new Google Cloud region to Kuwait". Google Cloud Blog. Retrieved July 10, 2023.
  67. 1 2 3 "Announcing new Google Cloud regions in Asia Pacific". Google Cloud Blog. Retrieved July 10, 2023.
  68. ↑ "Announcing a new Google Cloud region in Mexico". Google Cloud Blog. Retrieved July 10, 2023.
  69. ↑ "Google purchases land for new data center in Tainan". Taipei Times. September 12, 2019. Retrieved December 20, 2019.
  70. ↑ "Google to set up data center in Tainan". Focus Taiwan. September 11, 2019. Retrieved December 20, 2019.
  71. ↑ "Google to set up second data center in Taiwan". Taiwan News. September 11, 2019. Retrieved December 20, 2019.
  72. ↑ "Google confirms plans to build 3rd data center in Taiwan". Taiwan News. September 3, 2020. Retrieved September 3, 2020.
  73. ↑ "Our $1 billion investment in a new UK data centre". Google. January 18, 2024. Retrieved January 19, 2024.
  74. ↑ ""Google Stanford Hardware"". Archived from the original on February 9, 1999. Retrieved March 23, 2017.{{cite web}}: CS1 maint: bot: original URL status unknown (link). Stanford University (provided by Internet Archive). Retrieved on July 10, 2006.
  75. ↑ Merlin, Marc (2013). "Case Study: Live upgrading many thousand of servers from an ancient Red Hat distribution to a 10 year newer Debian based one" (PDF). Linux Foundation. Retrieved June 9, 2017.
  76. ↑ Tawfik Jelassi; Albrecht Enders (2004). "Case study 16 — Google". Strategies for E-business. Pearson Education. p. 424. ISBN   978-0-273-68840-2.
  77. 1 2 Computer Architecture, Fifth Edition: A Quantitative Approach, ISBN   978-0123838728; Chapter Six; 6.7 "A Google Warehouse-Scale Computer" page 471 "Designing motherboards that only need a single 12-volt supply so that the UPS function could be supplied by standard batteries associated with each server"
  78. ↑ Google's secret power supplies on YouTube
  79. ↑ Google uncloaks once-secret server, April 1, 2009.
  80. ↑ "Google Sustainability". Google Sustainability.
  81. ↑ "Analytics Press Growth in data center electricity use 2005 to 2010". Archived from the original on January 11, 2012. Retrieved May 22, 2012.
  82. ↑ Google Surpasses Supercomputer Community, Unnoticed?, May 20, 2008.
  83. ↑ Lam, Cedric F.; Liu, Hong; Koley, Bikash; Zhao, Xiaoxue; Kamalov, Valey; Gill, Vijay (2010), "Fiber Optic Communication Technologies: What's Needed for Datacenter Network Operations", Research, vol. 48
  84. ↑ Lam, Cedric F. (2010), FTTH look ahead — technologies & architectures (PDF), p. 4
  85. ↑ "kumara ASN15169", Peering DB
  86. ↑ "Urs Holzle", Speakers, Open Network Summit, archived from the original on May 10, 2012, retrieved May 22, 2012
  87. 1 2 3 Web Search for a Planet: The Google Cluster Architecture (Luiz André Barroso, Jeffrey Dean, Urs Hölzle)
  88. ↑ "Warehouse size computers" (PDF). Archived from the original (PDF) on June 19, 2012. Retrieved May 22, 2012.
  89. ↑ Denis Abt High Performance Datacenter Networks: Architectures, Algorithms, and Opportunities
  90. 1 2 3 Fiach Reid (2004). "Case Study: The Google search engine". Network Programming in .NET. Digital Press. pp.  251–253. ISBN   978-1-55558-315-6.
  91. 1 2 Rich Miller (March 27, 2008). "Google Data Center FAQ". Data Center Knowledge. Archived from the original on March 13, 2009. Retrieved March 15, 2009.
  92. ↑ Brett Winterford (March 5, 2010). "Found: Google Australia's secret data network". ITNews. Retrieved March 20, 2010.
  93. ↑ Singh, Arjun; Ong, Joon; Agarwal, Amit; Anderson, Glen; Armistead, Ashby; Bannon, Roy; Boving, Seb; Desai, Gaurav; Felderman, Bob; Germano, Paulie; Kanagala, Anand (2015). "Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's Datacenter Network". Sigcomm '15. doi: 10.1145/2785956.2787508 . S2CID   2817692.
  94. ↑ Markoff, John; Hansell, Saul. "Hiding in Plain Sight, Google Seeks More Power." New York Times. June 14, 2006. Retrieved on October 15, 2008.
  95. ↑ Google "The Dalles, Oregon Data Center" Retrieved on January 3, 2011.
  96. ↑ Strand, Ginger. "Google Data Center" Harper's Magazine . March 2008. Retrieved on October 15, 2008. Archived August 30, 2012, at the Wayback Machine
  97. ↑ "Stora Enso divests Summa Mill premises in Finland for million". Stora Enso. February 12, 2009. Archived from the original on April 13, 2009. Retrieved December 2, 2009.
  98. ↑ [ dead link ] "Stooora yllätys: Google ostaa Summan tehtaan". Kauppalehti (in Finnish). Helsinki. February 12, 2009. Archived from the original on February 14, 2009. Retrieved February 12, 2009.
  99. ↑ "Google investoi 200 miljoonaa euroa Haminaan". Taloussanomat (in Finnish). Helsinki. February 4, 2009. Retrieved March 15, 2009.
  100. ↑ "Hamina, Finland" . Retrieved April 23, 2018.
  101. ↑ Finland – First Choice for Siting Your Cloud Computing Data Center. Archived July 6, 2013, at the Wayback Machine Accessed August 4, 2010.
  102. ↑ Rory Carroll (October 30, 2013). "Google's worst-kept secret: floating data centers off US coasts". Theguardian.com. Retrieved December 8, 2018.
  103. ↑ Rich Miller (April 29, 2009). "Google Gets Patent for Data Center Barges". Datacenterknowledge.com. Retrieved December 8, 2018.
  104. ↑ Martin Lamonica (September 8, 2008). "Google files patent for wave-powered floating data center". Cnet.com. Retrieved December 8, 2018.
  105. ↑ "Google's ship based datacenter patent application surfaces". Datacenterdynamics.com. September 7, 2008. Retrieved December 8, 2018.
  106. ↑ "Google barge mystery solved: they're for 'interactive learning centers'". Theguardian.com. November 6, 2013. Retrieved December 8, 2018.
  107. ↑ Brandon Bailey (August 1, 2014). "Google confirms selling a mystery barge". San Jose Mercury News . Retrieved April 7, 2015.
  108. ↑ Chris Morran (November 7, 2014). "What Happened To Those Google Barges?". Consumerist . Retrieved January 15, 2017.
  109. ↑ Mark Levene (2005). An Introduction to Search Engines and Web Navigation. Pearson Education. p. 73. ISBN   978-0-321-30677-7.
  110. ↑ "Python Status Update". Artima. January 10, 2006. Retrieved February 17, 2012.
  111. ↑ "Warning". Panela. Blog-city. Archived from the original on December 28, 2011. Retrieved February 17, 2012.
  112. ↑ "Quotes about Python". Python. Retrieved February 17, 2012.
  113. ↑ "Google Architecture". High Scalability. November 22, 2008. Retrieved February 17, 2012.
  114. 1 2 3 Fikes, Andrew (July 29, 2010), "Storage Architecture and Challenges", TechTalk (PDF)[ permanent dead link ]
  115. ↑ "Colossus: Successor to the Google File System (GFS)". SysTutorials. November 29, 2012. Retrieved May 10, 2016.
  116. ↑ Dean, Jeffrey 'Jeff' (2009), "Design, Lessons and Advice from Building Large Distributed Systems", Ladis (keynote talk presentation), Cornell
  117. ↑ Shute, Jeffrey 'Jeff'; Oancea, Mircea; Ellner, Stephan; Handy, Benjamin 'Ben'; Rollins, Eric; Samwel, Bart; Vingralek, Radek; Whipkey, Chad; Chen, Xin; Jegerlehner, Beat; Littlefield, Kyle; Tong, Phoenix (2012), "F1 — the Fault-Tolerant Distributed RDBMS Supporting Google's Ad Business", Research (presentation), Sigmod {{citation}}: CS1 maint: location missing publisher (link)
  118. ↑ "Google alums rev up a new search engine". Los Angeles Times. July 28, 2008. Retrieved September 15, 2021.
  119. 1 2 The Register. Google Caffeine jolts worldwide search machine
  120. ↑ "Google official release note" . Retrieved September 28, 2013.
  121. 1 2 "Google Developing Caffeine Storage System | TechWeekEurope UK". Eweekeurope.co.uk. August 18, 2009. Archived from the original on November 15, 2011. Retrieved February 17, 2012.
  122. ↑ "Developer Guide – Protocol Buffers – Google Code" . Retrieved February 17, 2012.
  123. ↑ http://static.googleusercontent.com/media/research.google.com/en/us/archive/bigtable-osdi06.pdf [ bare URL PDF ]
  124. ↑ windley on (June 24, 2008). "Phil Windley's Technometria | Velocity 08: Storage at Scale". Windley.com. Retrieved February 17, 2012.
  125. ↑ "Message limit – Protocol Buffers | Google Groups" . Retrieved February 17, 2012.
  126. ↑ "Jeff Dean's keynote at WSDM 2009" (PDF). Retrieved February 17, 2012.
  127. ↑ Daniel Peng, Frank Dabek. (2010). Large-scale Incremental Processing Using Distributed Transactions and Notifications. Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation.
  128. ↑ The Register. Google Percolator – global search jolt sans MapReduce comedown
  129. ↑ Chandler Evans (2008). "Google Platform". Future of Google Earth. Madison Publishing Company. p. 299. ISBN   978-1-4196-8903-1.
  130. ↑ Chris Sherman (2005). "How Google Works". Google Power. McGraw-Hill Professional. pp.  10–11. ISBN   978-0-07-225787-8.
  131. ↑ Michael Miller (2007). "How Google Works". Googlepedia. Pearson Technology Group. pp.  17–18. ISBN   978-0-7897-3639-0.
  132. ↑ Gellman, Barton; Soltani, Ashkan (October 30, 2013). "NSA infiltrates links to Yahoo, Google data centers worldwide, Snowden documents say". The Washington Post. Retrieved November 1, 2013.
  133. ↑ Savage, Charlie; Miller, Claire Cain; Perlroth, Nicole (October 30, 2013). "N.S.A. Said to Tap Google and Yahoo Abroad". The New York Times . Retrieved March 9, 2017.
  134. ↑ Gallagher, Sean (October 31, 2013). "How the NSA's MUSCULAR tapped Google's and Yahoo's private networks". Ars Technica . Condé Nast . Retrieved March 9, 2017.
  135. ↑ Miller, Claire Cain (October 31, 2013). "Angry Over U.S. Surveillance, Tech Giants Bolster Defenses". The New York Times . Retrieved March 9, 2017.
  136. ↑ Humphries, Matthew (March 27, 2012). "Google's most efficient data center runs at 95 degrees". geek.com. Archived from the original on June 13, 2016. Retrieved June 13, 2016.
  137. ↑ Hölzle, Urs (December 6, 2016). "We're set to reach 100% renewable energy — and it's just the beginning". The Keyword Google Blog. Retrieved December 8, 2016.
  138. ↑ Statt, Nick (December 6, 2016). "Google just notched a big victory in the fight against climate change". The Verge . Vox Media . Retrieved December 8, 2016.
  139. ↑ Etherington, Darrell (December 7, 2016). "Google says it will hit 100% renewable energy by 2017". TechCrunch . AOL . Retrieved December 8, 2016.

Further reading

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.