DOME MicroDataCenter

Last updated
microDataCenter 32-way carrier
DOME-microDataCenterFull.jpg
fully populated DOME microDataCenter as shown at SC17, Denver, Nov 2017, containing 24 server and 8 FPGA boards, 10GbE switch, storage, power and cooling - picture shows half of the 2U-rack unit

A microDataCenter contains compute, storage, power, cooling and networking in a very small volume, sometimes also called a "DataCenter-in-a-box". The term has been used to describe various incarnations of this idea over the past 20 years. Late 2017 a very tightly integrated version was shown at SuperComputing conference 2017: the DOME microDataCenter. [1] Key features are its hot-watercooling, fully solid-state and being built with commodity components and standards only.

Contents

DOME project

DOME is a Dutch government-funded project between IBM and ASTRON in form of a public–private partnership to develop technology roadmaps targeting the Square Kilometer Array (SKA), the world's largest planned radio telescope. [2] [3] It will be built in Australia and South Africa during the late 2010s and early 2020s. One of the 7 DOME projects is MicroDataCenter (previously called Microservers) that are small, inexpensive and computationally efficient. [4]

The goal for the MicroDataCenter is the capability to be used both near the SKA antennas to do early processing of the data, and inside supercomputers that will do the big data analysis. These servers can be deployed in very large numbers and in environmentally extreme locations such as in deserts where the antennas will be located and not in only in cooled datacenters.

A common misconception is that microservers offer only low performance. This is caused by the first microservers being based on Atoms or early 32bit ARM cores. The aim of the DOME MicroDataCenter project is to deliver high performance at low cost and low power. A key characteristic of a MicroDataCenter is its packaging: very small form factor that allows short communication distances. This is based on using Microservers, eliminating all unnecessary components by integrating as much as possible from the traditional compute server into a single SoC (Server on a chip). A microserver will not deliver the highest possible single-thread performance, instead, it offers an energy optimized design point at medium-high delivered performance. In 2015, several high performance SoCs start appearing on the market, late 2016 a wider choice is available, such as Qualcomms Hydra. [5]

At server level, the 28 nm T4240 based microserver card offers twice the operations per Joule compared to an energy optimized 22 nm Finfet XEON-E3 1230Lv3 based server, while delivering 40% more aggregate performance. The comparison is at server board and not at chip level. [6]

Design

In 2012 a team at IBM Research Zürich led by Ronald P. Luijten started pursuing a very computational dense, and energy efficient 64-bit computer design based on commodity components, running Linux. [7] [8] A system-on-chip (SoC) design where most necessary components would fit on a single chip would fit these goals best, and a definition of "microserver" emerged where essentially a complete motherboard (except RAM, boot flash and power conversion circuits) would fit on chip. ARM, x86 and Power ISA-based solutions were investigated and a solution based on Freescale's Power ISA-based dual core P5020 / quad core P5040 processor came out on top at the time of decision in 2012.

The concept is similar to IBM's Blue Gene supercomputers but the DOME microserver is designed around off the shelf components and will run standard operating systems and protocols, to keep development and component costs down. [9]

The complete microserver is based on the same form factor as standard FB-DIMM socket. The idea is to fit 128 of these compute cards within a 19" rack 2U drawer together with network switchboards for external storage and communication. Cooling will be provided via the Aquasar hot water cooling solution pioneered by the SuperMUC supercomputer in Germany. [10]

The designs of the first prototype was released to the DOME user community on July 3, 2014. The P5040 SoC chip, 16 GB of DRAM and a few control chips (such as the PSoC 3 from Cypress used for monitoring, debugging and booting) comprise a complete compute node with the physical dimensions of 133×55 mm. The card's pins are used for a SATA, five Gbit and two 10 Gbit Ethernet ports, one SD card interface, one USB 2 interface, and power. The compute card operates within a 35 W power envelope with headroom up to 70 W. The bill of materials is around $500 for the single prototype. [7] [8] [9] [11] [12]

Late 2013 a new SoC was chosen for the second prototype. Freescale's newer 12 core / full 24 thread T4240 is significantly more powerful and operates within a comparable power envelope to the P5040 at 43W TDP. This new micro server card offers 24 GB DRAM, and be powered as well as cooled from the copper heatspreader. It is being built and validated for the larger scale deployment in the full 2U drawer in early 2017. To support native 10 GbE signalling, the DIMM connector was replaced with the SPD08 connector.

Late 2016, the production version of the T4240 based microserver card was completed. Using the same form factor and the same connector (and thus plug compatible) a second server prototype board based on the NXP (Formerly Freescale) LS2088A SoC (with 8 A72 ARMv8 cores) was completed around the same time. [9] [12] [10] [13]

History

microDataCenter production version
Fully populated DOME 32 way carrier (microdatacenter).jpg
32-way carrier with 24 T4240ZMS servers and 8 FPGA cards

The smallest form factor micro data center technology was pioneered by the DOME micro server team in Zurich. The computing consists of multiple Microservers and the networking consists of at least one Micro switch module. The first live demo of an 8-way prototype system based on P5040ZMS was performed at Supercomputing 2015 as part of the emerging technologies display, [14] followed by a live demo at CeBIT in March 2016. 8 Way HPL was demonstrated at CeBIT, hence named 'LinPack-in-a-shoebox'.

In 2017 the team finished the production version that contains 64 T4240ZMS servers, two 10/40 GbE switches, storage, power and cooling in a 2U rack unit. The picture on lowest right shows the 32-way carrier (half of the 2U rack unit) populated with 24 T4240ZMS servers, 8 FPGA boards, switch, storage power and cooling. This technology increases density 20-fold compared to traditionally packaged datacenter technology while delivering same aggregate performance. This is achieved by a new top-down design, minimizing component count, using an SoC instead of traditional CPU and dense packaging enabled by the use of hot-water cooling. [15]

Related Research Articles

<span class="mw-page-title-main">PowerPC</span> RISC instruction set architecture by AIM alliance

PowerPC is a reduced instruction set computer (RISC) instruction set architecture (ISA) created by the 1991 Apple–IBM–Motorola alliance, known as AIM. PowerPC, as an evolving instruction set, has been named Power ISA since 2006, while the old name lives on as a trademark for some implementations of Power Architecture–based processors.

Floating point operations per second is a measure of computer performance in computing, useful in fields of scientific computations that require floating-point calculations.

<span class="mw-page-title-main">IBM Blue Gene</span> Series of supercomputers by IBM

Blue Gene was an IBM project aimed at designing supercomputers that can reach operating speeds in the petaFLOPS (PFLOPS) range, with low power consumption.

Cell is a 64-bit multi-core microprocessor microarchitecture that combines a general-purpose PowerPC core of modest performance with streamlined coprocessing elements which greatly accelerate multimedia and vector processing applications, as well as many other forms of dedicated computation.

<span class="mw-page-title-main">MareNostrum</span> Supercomputer in the Barcelona Supercomputing Center

MareNostrum is the main supercomputer in the Barcelona Supercomputing Center. It is the most powerful supercomputer in Spain, one of thirteen supercomputers in the Spanish Supercomputing Network and one of the seven supercomputers of the European infrastructure PRACE.

The PowerPC 400 family is a line of 32-bit embedded RISC processor cores based on the PowerPC or Power ISA instruction set architectures. The cores are designed to fit inside specialized applications ranging from system-on-a-chip (SoC) microcontrollers, network appliances, application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs) to set-top boxes, storage devices and supercomputers.

<span class="mw-page-title-main">Microserver</span>

A data center 64 bit microserver is a server class computer which is based on a system on a chip (SoC). The goal is to integrate all of the server motherboard functions onto a single microchip, except DRAM, boot FLASH and power circuits. Thus, the main chip contains more than only compute cores, caches, memory interfaces and PCI controllers. It typically also contains SATA, networking, serial port and boot FLASH interfaces on the same chip. This eliminates support chips at the board level. Multiple microservers can be put together in a small package to construct dense data center.

In computing, performance per watt is a measure of the energy efficiency of a particular computer architecture or computer hardware. Literally, it measures the rate of computation that can be delivered by a computer for every watt of power consumed. This rate is typically measured by performance on the LINPACK benchmark when trying to compare between computing systems: an example using this is the Green500 list of supercomputers. Performance per watt has been suggested to be a more sustainable measure of computing than Moore's Law.

<span class="mw-page-title-main">PERCS</span>

PERCS is IBM's answer to DARPA's High Productivity Computing Systems (HPCS) initiative. The program resulted in commercial development and deployment of the Power 775, a supercomputer design with extremely high performance ratios in fabric and memory bandwidth, as well as very high performance density and power efficiency.

QPACE is a massively parallel and scalable supercomputer designed for applications in lattice quantum chromodynamics.

The National Center for Computational Sciences (NCCS) is a United States Department of Energy (DOE) Leadership Computing Facility that houses the Oak Ridge Leadership Computing Facility (OLCF), a DOE Office of Science User Facility charged with helping researchers solve challenging scientific problems of global interest with a combination of leading high-performance computing (HPC) resources and international expertise in scientific computing.

<span class="mw-page-title-main">POWER8</span> 2014 family of multi-core microprocessors by IBM

POWER8 is a family of superscalar multi-core microprocessors based on the Power ISA, announced in August 2013 at the Hot Chips conference. The designs are available for licensing under the OpenPOWER Foundation, which is the first time for such availability of IBM's highest-end processors.

<span class="mw-page-title-main">Xeon Phi</span> Series of x86 manycore processors from Intel

Xeon Phi is a discontinued series of x86 manycore processors designed and made by Intel. It was intended for use in supercomputers, servers, and high-end workstations. Its architecture allowed use of standard programming languages and application programming interfaces (APIs) such as OpenMP.

<span class="mw-page-title-main">Aquasar</span> Supercomputer system from IBM Research

Aquasar is a supercomputer prototype created by IBM Labs in collaboration with ETH Zurich in Zürich, Switzerland and ETH Lausanne in Lausanne, Switzerland. While most supercomputers use air as their coolant of choice, the Aquasar uses hot water to achieve its great computing efficiency. Along with using hot water as the main coolant, an air-cooled section is also included to be used to compare the cooling efficiency of both coolants. The comparison could later be used to help improve the hot water coolant's performance. The research program was first termed to be: "Direct use of waste heat from liquid-cooled supercomputers: the path to energy saving, emission-high performance computers and data centers." The waste heat produced by the cooling system is able to be recycled back in the building's heating system, potentially saving money. Beginning in 2009, the three-year collaborative project was introduced and developed in the interest of saving energy and being environmentally-safe while delivering top-tier performance.

iDataCool is a high-performance computer cluster based on a modified IBM System x iDataPlex. The cluster serves as a research platform for cooling of IT equipment with hot water and efficient reuse of the waste heat. The project is carried out by the physics department of the University of Regensburg in collaboration with the IBM Research and Development Laboratory Böblingen and InvenSor. It is funded by the German Research Foundation (DFG), the German state of Bavaria, and IBM.

IBM Power microprocessors are designed and sold by IBM for servers and supercomputers. The name "POWER" was originally presented as an acronym for "Performance Optimization With Enhanced RISC". The Power line of microprocessors has been used in IBM's RS/6000, AS/400, pSeries, iSeries, System p, System i, and Power Systems lines of servers and supercomputers. They have also been used in data storage devices and workstations by IBM and by other server manufacturers like Bull and Hitachi.

<span class="mw-page-title-main">POWER9</span> 2017 family of multi-core microprocessors by IBM

POWER9 is a family of superscalar, multithreading, multi-core microprocessors produced by IBM, based on the Power ISA. It was announced in August 2016. The POWER9-based processors are being manufactured using a 14 nm FinFET process, in 12- and 24-core versions, for scale out and scale up applications, and possibly other variations, since the POWER9 architecture is open for licensing and modification by the OpenPOWER Foundation members.

DOME is a Dutch government-funded project between IBM and ASTRON in form of a public-private-partnership focussing on the Square Kilometre Array (SKA), the world's largest planned radio telescope. SKA will be built in Australia and South Africa. The DOME project objective is technology roadmap development that applies both to SKA and IBM. The 5-year project was started in 2012 and is co-funded by the Dutch government and IBM Research in Zürich, Switzerland and ASTRON in the Netherlands. The project ended officially on 30 September 2017.

<span class="mw-page-title-main">QPACE2</span> Massively parallel and scalable supercomputer

QPACE 2 is a massively parallel and scalable supercomputer. It was designed for applications in lattice quantum chromodynamics but is also suitable for a wider range of applications..

Tesla Dojo is a supercomputer designed and built by Tesla for computer vision video processing and recognition. It is used for training Tesla's machine learning models to improve its Full Self-Driving (FSD) advanced driver-assistance system. According to Tesla, it went into production in July 2023.

References

  1. ↑ Paul Alcorn (November 22, 2017). "SuperComputing 2017 Slideshow". Tom's Hardware.
  2. ↑ "ASTRON & IBM Center for Exascale Technology". Archived from the original on 2020-01-25. Retrieved 2014-07-02.
  3. ↑ Square Kilometer Array: Ultimate Big Data Challenge
  4. ↑ ASTRON & IBM Center for Exascale Technology - Microservers
  5. ↑ "Qualcomm Hydra". Archived from the original on 2017-04-17. Retrieved 2017-04-16.
  6. ↑ [“Energy-Efficient Microserver Based on a 12-Core 1.8GHz 188K-CoreMark 28nm Bulk CMOS 64b SoC for Big-Data Applications with 159 GB/s/L Memory Bandwidth System Density”, R.Luijten et al., ISSCC15, San Francisco, Feb 2015]
  7. 1 2 "IBM high-density μServer demonstration platform leveraging PPC, Linux and hot-water cooling" (PDF). Archived from the original (PDF) on 2014-07-14. Retrieved 2014-07-02.
  8. 1 2 Hemsoth, Nicole (April 11, 2014). "Big Science, Tiny Microservers: IBM Research Pushes 64-Bit Possibilities". HPCwire.
  9. 1 2 3 Morgan, Timothy Prickett (July 7, 2014). "IBM DOME Microserver Could Appeal To Enterprises". EnterpriseAI.
  10. 1 2 "NLeSC signs DOME agreement with IBM and ASTRON". January 15, 2014 – via Flickr.
  11. ↑ "The IBM-ASTRON 64bit μServer demonstrator for SKA" (PDF). Archived from the original (PDF) on 2014-07-14. Retrieved 2014-07-02.
  12. 1 2 "The IBM and ASTRON 64bit μServer for DOME". Archived from the original on 2014-07-14. Retrieved 2014-07-03.
  13. ↑ "IBM to present the incredible shrinking supercomputer". PCWorld.
  14. ↑ "DOME Hot-Water Cooled Multinode 64 Bit Microserver Cluster". Archived from the original on 2016-01-27. Retrieved 2017-12-05.
  15. ↑ DOME microDataCenter project
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.