NCAR-Wyoming Supercomputing Center

Last updated
NCAR building in 2014 NSF-supported Research Facilities (15326178104).jpg
NCAR building in 2014

The NCAR-Wyoming Supercomputing Center (NWSC) is a high-performance computing (HPC) and data archival facility located in Cheyenne, Wyoming, that provides advanced computing services to researchers in the Earth system sciences. [1]

Contents

NWSC provides researchers’ needs for computing, data analysis, and scientific visualization resources combined with powerful data management capabilities to support finer Earth system model resolution, increased model complexity, better statistics, more predictive power, and longer simulation times. [2] [3] The data storage and archival facility [4] at NWSC holds unique historical climate records and a wealth of other scientific data. [5] Scientists at U.S. universities and research institutions access NWSC resources remotely via the Internet from desktop or laptop computers.

History

Inside the supoercomputer command center within NCAR in 2014 NSF-supported Research Facilities (15947767422).jpg
Inside the supoercomputer command center within NCAR in 2014

The NWSC data center is funded by the National Science Foundation (NSF) and the State of Wyoming, and is operated by the National Center for Atmospheric Research. It was created through a partnership [6] of the University Corporation for Atmospheric Research (UCAR), the State of Wyoming, the University of Wyoming, Cheyenne LEADS, [7] the Wyoming Business Council, and Cheyenne Light Fuel and Power Company (now named Black Hills Corporation). Consistent with NCAR’s mission, this supercomputing center is a leader in energy efficiency, incorporating the newest and most efficient designs and technologies available. Planning [8] [9] for this data center began in 2003, groundbreaking [10] at the North Range Business Park in Cheyenne took place in June 2010, and computing operations began in October 2012. [11]

Sustainability and energy efficiency

The facility design is based on modular and expandable spaces that can be adapted for computing system upgrades. Its sustainable design makes it 89% more efficient than a typical data center and up to 10% more efficient than state-of-the-art data centers operating in 2010. [12] [13] Almost 92% of the energy it uses goes directly to its core purpose of powering supercomputers [14] to enable scientific discovery. Part of its efficiency comes from the regionally integrated design that uses Wyoming’s climate to provide natural cooling during 96% of the year and local wind energy [15] that supplies at least 10% of its power. [16] The main energy source used is coal. [17] The NWSC achieved LEED Gold certification [18] [19] for its sustainable design and construction. In 2013 it won first place [20] for Facility Design Implementation in the Uptime Institute’s Green Enterprise IT awards. [21] This award recognizes pioneering projects and innovations that significantly improve energy productivity and resource use in information technology. In June 2013, the NWSC won the Datacenter Dynamics [22] North American ‘Green’ Data Center award [23] [24] for demonstrated sustainability in the design and operation of facilities. [25]

The center currently has a total of 153,000 square feet with 24,000 square feet of raised floor modules for supercomputing systems in its expandable design. [26] It incorporates numerous resource conservation features to reduce its environmental impact. Water consumption at the NWSC is reduced by about 40% compared to traditional designs by using innovative technologies, specialized cooling tower equipment, and low-flow plumbing fixtures. Waste heat from the systems is recycled to pre-heat components in the power plant, to heat the office spaces, and to melt snow and ice on outdoor walkways and rooftops. Windows supply natural light; combined with room occupancy sensors, the building saves 20-30% in lighting and electricity compared to typical office buildings. A building automation system saves energy by continuously optimizing pumps, fans, and controls that heat or cool only occupied areas of the facility. During construction, sustainable practices were used with emphasis on recycled and locally sourced materials. [27] [28]

Supercomputing resources

NSF grants for computing, data, and scientific visualization resources are allocated to researchers who investigate the Earth system through simulation. The current HPC environment includes two petascale supercomputers, data analysis and visualization servers, an operational weather forecasting system, an experimental supercomputing architecture platform, a centralized file system, a data storage resource, and an archive of historical research data. [29] All computing and support systems required for scientific workflows are attached to the shared, high-speed, central file system to improve scientific productivity and reduce costs by analyzing and visualizing their data files in place at the NWSC. [30]

Supercomputer: Yellowstone

The Yellowstone supercomputer in 2014. NSF-supported Research Facilities (15947776872).jpg
The Yellowstone supercomputer in 2014.

In 2012, the Yellowstone supercomputer [31] was installed in the NWSC as its inaugural HPC resource. Yellowstone is an IBM iDataPlex [32] cluster consisted of 72,288 Intel Sandy Bridge EP processor cores in 4,518 16-core nodes, each with 32 gigabytes of memory. [33] All nodes are interconnected with a full fat tree Mellanox FDR InfiniBand network. [34] Yellowstone has a peak performance of 1.504 petaflops and has demonstrated a computational capability of 1.2576 petaflops as measured by the High-Performance LINPACK (HPL) benchmark. [35] It debuted as the world’s 13th fastest computer [36] in the November 2012 ranking by the TOP500 organization. Also in November 2012, Yellowstone debuted as the 58th most energy efficient supercomputer in the world [37] by operating at 875.34 megaflops per watt as ranked by the Green500 organization. Yellowstone is expected to remain in production operation through the end of 2017.

Supercomputer: Cheyenne

Becoming operational in 2017, the 5.34-petaflops Cheyenne supercomputer is currently providing more than three times the computational capacity of Yellowstone. Cheyenne is an SGI ICE XA system with 4,032 dual-socket scientific computation nodes running 145,152, 18-core 2.3-GHz Intel Xeon E5-2697v4 (Broadwell) processing cores and has 315 terabytes of memory. [38] Interconnecting these nodes is a Mellanox EDR InfiniBand network with 9-D enhanced hypercube topology that performs with a latency of only 0.5 microsecond. [39] Cheyenne runs the SUSE Linux Enterprise Server 12 SP1 operating system. [40] Similar to Yellowstone, Cheyenne’s design and configuration will provide balanced I/O and exceptional computational capacity for the data-intensive needs of its user community. [41] Cheyenne debuted as the world's 20th most powerful computer in the November 2016 Top500 ranking. [42]

Supercomputer: unnamed

A new super computer was announced on January 27, 2021, capable of 20 quadrillion calculations per second, said to be 3.5 times faster than the current system in operation at the facility. Schedule to be operational in early 2022. [43]

Data analysis and visualization clusters: Geyser and Caldera

The Geyser and Caldera clusters are specialized data analysis and visualization resources within the data-centric Yellowstone environment. The Geyser data analysis server is a 640-core cluster of 16 nodes, each with 1 terabyte of memory. With its large per-node memory, Geyser is designed to facilitate large-scale data analysis and post-processing tasks, including 3D visualization, with applications that do not support distributed-memory parallelism. [44] The Caldera computational cluster has 256 cores in 16 nodes, each with 64 gigabytes of memory and two Graphics Processing Units (GPUs) for use as either computational processors or graphics accelerators. Caldera’s two NVIDIA Tesla GPUs [45] per node support parallel processing, visualization activities, and development and testing of general-purpose GPU (GPGPU) code.

Operational forecasting system for Antarctic weather: Erebus

The center also houses a separate, smaller IBM iDataPlex cluster named Erebus [46] to support the operational forecasts of the NSF Office of Polar Programs [47] Antarctic Mesoscale Prediction System [48] (AMPS). [49] Erebus has 84 nodes similar to Yellowstone’s, an FDR-10 InfiniBand interconnect, and a dedicated 58-terabyte file system. If needed, Yellowstone will run Erebus’ daily weather forecasts for the Antarctic continent to ensure that the worldwide community of users receives these forecasts without interruption. [50]

Experimental supercomputing architecture platform: Pronghorn

Pronghorn’s architecture has promise for meeting the Earth system sciences’ demanding requirements for data analysis, visualization, and GPU-assisted computation. As part of a partnership between Intel, IBM, and NCAR, this exploratory system is being used to evaluate the effectiveness of the Xeon Phi coprocessor’s Many Integrated Core (MIC) architecture for running climate, weather, and other environmental applications. If these coprocessors prove beneficial to key NCAR applications, they can be easily added to the standard IBM iDataPlex nodes in Yellowstone as a cost-effective way to extend its capabilities.

Pronghorn has 16 dual-socket IBM x360 nodes [51] featuring Intel’s Xeon Phi 5110P coprocessors [52] and 2.6-gigahertz Intel Sandy Bridge (Xeon E5-2670) cores. [53] The system has 64 gigabytes of DDR3-1600 memory per node (63 GB usable memory per node) and is interconnected with a full fat tree Mellanox FDR InfiniBand network.

Centralized file system: GLADE

Geyser, Caldera, and Yellowstone all mount the central file system named GLobally Accessible Data Environment [54] (GLADE), which provides work spaces common to all HPC resources at NWSC for computation, analysis, and visualization. This allows users to analyze data files in place, without sending large amounts of data across a network or creating duplicate copies in multiple locations. GLADE provides centralized high-performance file systems spanning supercomputing, data post-processing, data analysis, visualization, and HPC-based data transfer services. [55] GLADE also hosts data from NCAR’s Research Data Archive, [56] NCAR’s Community Data Portal, [57] and the Earth System Grid [58] that curates CMIP5/AR5 data. The GLADE central disk resource has a usable storage capacity of 36 petabytes as of February 2017. [59] GLADE has a sustainable aggregate I/O bandwidth of more than 220 gigabits per second.

Data archival service: HPSS

Archival data storage at the NWSC is provided by a High Performance Storage System (HPSS) that consists of tape libraries with storage capacity of 320 petabytes. These scalable, robotic systems consist of six Oracle StorageTek SL8500 tape libraries using T10000C tape drives [60] with an I/O rate of 240 megabits per second.

Climate data archive: RDA

NWSC’s data-intensive computing strategy includes a full suite of community data services. NCAR develops data products and services that address the future challenges of data growth, preservation, and management. The Research Data Archive [61] (RDA) contains a large collection of meteorological and oceanographic datasets that support scientific studies in climate, weather, hydrology, Earth system modeling, and other related sciences. [62] It is an open resource; the global research community also uses it.

Educational projects

The NWSC also serves an educational role. [63] Its public outreach program features the NWSC visitor center [64] that explains the science goals and the technology of NCAR and the University of Wyoming. [65] [66] NCAR's higher education internship program [67] places two engineering interns at the NWSC each summer.

The facilities at NWSC are being used in a research collaboration [68] [69] with Colorado State University, Oak Ridge National Laboratory, Lagrange Systems, [70] and NCAR to produce resilient resource management strategies for HPC environments, increase the number of researchers and scientific problems that can use HPC, and help achieve sustainable computing at extreme scales within realistic power budgets. [71]

See also

Related Research Articles

<span class="mw-page-title-main">Supercomputer</span> Type of extremely powerful computer

A supercomputer is a computer with a high level of performance as compared to a general-purpose computer. The performance of a supercomputer is commonly measured in floating-point operations per second (FLOPS) instead of million instructions per second (MIPS). Since 2017, there have existed supercomputers which can perform over 1017 FLOPS (a hundred quadrillion FLOPS, 100 petaFLOPS or 100 PFLOPS). For comparison, a desktop computer has performance in the range of hundreds of gigaFLOPS (1011) to tens of teraFLOPS (1013). Since November 2017, all of the world's fastest 500 supercomputers run on Linux-based operating systems. Additional research is being conducted in the United States, the European Union, Taiwan, Japan, and China to build faster, more powerful and technologically superior exascale supercomputers.

<span class="mw-page-title-main">MareNostrum</span> Supercomputer in the Barcelona Supercomputing Center

MareNostrum is the main supercomputer in the Barcelona Supercomputing Center. It is the most powerful supercomputer in Spain, one of thirteen supercomputers in the Spanish Supercomputing Network and one of the seven supercomputers of the European infrastructure PRACE.

<span class="mw-page-title-main">National Center for Atmospheric Research</span> US federally funded research and development center

The US National Center for Atmospheric Research is a US federally funded research and development center (FFRDC) managed by the nonprofit University Corporation for Atmospheric Research (UCAR) and funded by the National Science Foundation (NSF). NCAR has multiple facilities, including the I. M. Pei-designed Mesa Laboratory headquarters in Boulder, Colorado. Studies include meteorology, climate science, atmospheric chemistry, solar-terrestrial interactions, environmental and societal impacts.

The Texas Advanced Computing Center (TACC) at the University of Texas at Austin, United States, is an advanced computing research center that provides comprehensive advanced computing resources and support services to researchers in Texas and across the US. The mission of TACC is to enable discoveries that advance science and society through the application of advanced computing technologies. Specializing in high performance computing, scientific visualization, data analysis & storage systems, software, research & development and portal interfaces, TACC deploys and operates advanced computational infrastructure to enable computational research activities of faculty, staff, and students of UT Austin. TACC also provides consulting, technical documentation, and training to support researchers who use these resources. TACC staff members conduct research and development in applications and algorithms, computing systems design/architecture, and programming tools and environments.

<span class="mw-page-title-main">National Energy Research Scientific Computing Center</span> Supercomputer facility operated by the US Department of Energy in Berkeley, California

The National Energy Research Scientific Computing Center (NERSC), is a high-performance computing (supercomputer) National User Facility operated by Lawrence Berkeley National Laboratory for the United States Department of Energy Office of Science. As the mission computing center for the Office of Science, NERSC houses high performance computing and data systems used by 9,000 scientists at national laboratories and universities around the country. NERSC's newest and largest supercomputer is Perlmutter, which debuted in 2021 ranked 5th on the TOP500 list of world's fastest supercomputers.

<span class="mw-page-title-main">TOP500</span> Database project devoted to the ranking of computers

The TOP500 project ranks and details the 500 most powerful non-distributed computer systems in the world. The project was started in 1993 and publishes an updated list of the supercomputers twice a year. The first of these updates always coincides with the International Supercomputing Conference in June, and the second is presented at the ACM/IEEE Supercomputing Conference in November. The project aims to provide a reliable basis for tracking and detecting trends in high-performance computing and bases rankings on HPL, a portable implementation of the high-performance LINPACK benchmark written in Fortran for distributed-memory computers.

<span class="mw-page-title-main">Irish Centre for High-End Computing</span> National high-performance computing centre in Ireland

The Irish Centre for High-End Computing (ICHEC) is the national high-performance computing centre in Ireland. It was established in 2005 and provides supercomputing resources, support, training and related services. ICHEC is involved in education and training, including providing courses for researchers.

<span class="mw-page-title-main">Pleiades (supercomputer)</span>

Pleiades is a petascale supercomputer housed at the NASA Advanced Supercomputing (NAS) facility at NASA's Ames Research Center located at Moffett Field near Mountain View, California. It is maintained by NASA and partners Hewlett Packard Enterprise and Intel.

The National Center for Computational Sciences (NCCS) is a United States Department of Energy (DOE) Leadership Computing Facility that houses the Oak Ridge Leadership Computing Facility (OLCF), a DOE Office of Science User Facility charged with helping researchers solve challenging scientific problems of global interest with a combination of leading high-performance computing (HPC) resources and international expertise in scientific computing.

<span class="mw-page-title-main">Tianhe-1</span> Supercomputer

Tianhe-I, Tianhe-1, or TH-1 is a supercomputer capable of an Rmax of 2.5 peta FLOPS. Located at the National Supercomputing Center of Tianjin, China, it was the fastest computer in the world from October 2010 to June 2011 and was one of the few petascale supercomputers in the world.

<span class="mw-page-title-main">National Computational Infrastructure</span> HPC facility in Canberra, Australia

The National Computational Infrastructure is a high-performance computing and data services facility, located at the Australian National University (ANU) in Canberra, Australian Capital Territory. The NCI is supported by the Australian Government's National Collaborative Research Infrastructure Strategy (NCRIS), with operational funding provided through a formal collaboration incorporating CSIRO, the Bureau of Meteorology, the Australian National University, Geoscience Australia, the Australian Research Council, and a number of research intensive universities and medical research institutes.

<span class="mw-page-title-main">National Computer Center for Higher Education (France)</span>

The National Computer Center for Higher Education (CINES), based in Montpellier, is a public institution of French administrative character placed under the supervision of the Ministry of Research and Innovation (MESRI), and created by decree in 1999. CINES offers an IT service used for public research in France. It is one of the major national centers for computing power supply for research in France.

<span class="mw-page-title-main">Tsubame (supercomputer)</span> Series of supercomputers

Tsubame is a series of supercomputers that operates at the GSIC Center at the Tokyo Institute of Technology in Japan, designed by Satoshi Matsuoka.

<span class="mw-page-title-main">Supercomputing in Europe</span> Overview of supercomputing in Europe

Several centers for supercomputing exist across Europe, and distributed access to them is coordinated by European initiatives to facilitate high-performance computing. One such initiative, the HPC Europa project, fits within the Distributed European Infrastructure for Supercomputing Applications (DEISA), which was formed in 2002 as a consortium of eleven supercomputing centers from seven European countries. Operating within the CORDIS framework, HPC Europa aims to provide access to supercomputers across Europe.

<span class="mw-page-title-main">Xeon Phi</span> Series of x86 manycore processors from Intel

Xeon Phi was a series of x86 manycore processors designed and made by Intel. It was intended for use in supercomputers, servers, and high-end workstations. Its architecture allowed use of standard programming languages and application programming interfaces (APIs) such as OpenMP.

<span class="mw-page-title-main">Yellowstone (supercomputer)</span>

Yellowstone was the inaugural supercomputer at the NCAR-Wyoming Supercomputing Center (NWSC) in Cheyenne, Wyoming. It was installed, tested, and readied for production in the summer of 2012. The Yellowstone supercomputing cluster was decommissioned on December 31, 2017, being replaced by its successor Cheyenne.

<span class="mw-page-title-main">Appro</span> American technology company

Appro was a developer of supercomputing supporting High Performance Computing (HPC) markets focused on medium- to large-scale deployments. Appro was based in Milpitas, California with a computing center in Houston, Texas, and a manufacturing and support subsidiary in South Korea and Japan.

<span class="mw-page-title-main">Cray XC40</span> Supercomputer manufactured by Cray

The Cray XC40 is a massively parallel multiprocessor supercomputer manufactured by Cray. It consists of Intel Haswell Xeon processors, with optional Nvidia Tesla or Intel Xeon Phi accelerators, connected together by Cray's proprietary "Aries" interconnect, stored in air-cooled or liquid-cooled cabinets. The XC series supercomputers are available with the Cray DataWarp applications I/O accelerator technology.

The Cheyenne supercomputer at the NCAR-Wyoming Supercomputing Center (NWSC) in Cheyenne, Wyoming began operation as one of the world’s most powerful and energy-efficient computers. Ranked in November 2016 as the 20th most powerful computer in the world by Top500, the 5.34-petaflops system is capable of more than triple the amount of scientific computing performed by NCAR’s previous supercomputer, Yellowstone. It also is three times more energy efficient than Yellowstone, with a peak computation rate of more than 3 billion calculations per second for every watt of energy consumed.

The Cray XC50 is a massively parallel multiprocessor supercomputer manufactured by Cray. The machine can support Intel Xeon processors, as well as Cavium ThunderX2 processors, Xeon Phi processors and NVIDIA Tesla P100 GPUs. The processors are connected by Cray's proprietary "Aries" interconnect, in a dragonfly network topology. The XC50 is an evolution of the XC40, with the main difference being the support of Tesla P100 processors and the use of Cray software release CLE 6 or 7.

References

  1. NCAR-Wyoming Supercomputing Center commissioned, FY2011 NCAR Annual Report. Retrieved 2012-10-16.
  2. Advancing the geosciences through simulation, FY2010 NCAR Annual Report. Retrieved 2012-12-20.
  3. NCAR supercomputer ready for research projects, Wyoming Public Media website. Retrieved 2012-10-16.
  4. Data Support for Climate and Weather Research, National Center for Atmospheric Research website. Retrieved 2013-06-12.
  5. CISL Research Data Archive, NCAR-CISL Research Data Archive website. Retrieved 2013-06-12.
  6. A Public-Private Success Story, Interview, A look at the coalition behind the NCAR-Wyoming Supercomputing Center. NCAR-UCAR AtmosNews website. Retrieved 2012-10-16.
  7. The Cheyenne-Laramie County Corporation for Economic Development, Cheyenne LEADS website. Retrieved 2012-06-11.
  8. Establishing a Petascale Collaboratory for the Geosciences: Scientific Frontiers, 2005. Ad Hoc Committee and Technical Working Group for a Petascale Collaboratory for the Geosciences. A Report to the Geosciences Community. UCAR/JOSS. 80 pp.
  9. Establishing a Petascale Collaboratory for the Geosciences: Technical and Budgetary Prospectus, 2005. Technical Working Group and Ad Hoc Committee for a Petascale Collaboratory for the Geosciences. A Report to the Geosciences Community, UCAR/JOSS. 56 pp.
  10. Breaking Ground on a Groundbreaking Center, NCAR-UCAR AtmosNews website. Retrieved 2012-07-20.
  11. Climate change research gets petascale supercomputer, Computerworld website. Retrieved 2012-10-16.
  12. Green Technology - Raising the Bar in Data Center Efficiency, NCAR-Wyoming Supercomputing Center website. Retrieved 2013-06-19. Source of these estimates is "Basis of Design," an internal construction planning document by H&L Architecture, California Data Center Design Group, Rumsey Engineering, and RMH Group Inc., dated 2010-02-25.
  13. Designing a New Type of Data Center, Data Center Knowledge website. Retrieved 2013-06-19.
  14. Green Technology – Raising the Bar in Data Center Efficiency, NCAR-Wyoming Supercomputing Center website. Retrieved 2013-06-10. Source of this estimate is "Basis of Design," an internal construction planning document by H&L Architecture, California Data Center Design Group, Rumsey Engineering, and RMH Group Inc., dated 2010-02-25.
  15. Happy Jack Windpower, Project. Duke Energy website. Retrieved 2013-06-19.
  16. NCAR Weather Supercomputer Comes Online, Data Center Knowledge website. Retrieved 2013-06-19.
  17. "Meet 'Cheyenne', the coal-powered climate predicting supercomputer". wattsupwiththat.com. 2017-02-22. Retrieved 2020-12-25.
  18. NCAR-Wyoming Supercomputing Center Fact Sheet, NCAR-UCAR AtmosNews website. Retrieved 2012-10-15.
  19. Climate Research Spurred By Green Design of Supercomputing Center, Engineering News Record (ENR) Mountain States website. Retrieved 2013-06-27.
  20. World-class Supercomputing Center Designed to Achieve 1.08 PUE, 2013 Green Enterprise IT Award Winner, Facility Design-Implementation, Uptime Institute website. Retrieved 2013-06-12.
  21. First Place: NCAR-Wyoming Supercomputing Center Recognized for Outstanding Design Implementation, NCAR-UCAR AtmosNews website. Retrieved 2013-04-03.
  22. DatacenterDynamics North American awards ceremony, Datacenter Dynamics website. Retrieved 2013-08-02.
  23. NCAR-Wyoming Supercomputing Center Wins National Data Center Award, Engineering News Record (ENR) Mountain States website. Retrieved 2013-08-02.
  24. NWSC named ‘Green’ Data Center of the Year, NCAR-UCAR AtmosNews website. Retrieved 2013-08-02.
  25. The ‘Green’ Data Center, Datacenter Dynamics website. Retrieved 2013-08-02.
  26. NCAR-Wyoming Supercomputing Center Fact Sheet, NCAR-UCAR AtmosNews website. Retrieved 2012-10-15.
  27. NCAR and the Green Supercomputing Facility of the Future, Reuters website. Retrieved 2012-09-20.
  28. First Place: NCAR-Wyoming Supercomputing Center recognized for outstanding design implementation, NCAR-UCAR AtmosNews website. Retrieved 2013-04-20.
  29. Statistical Analysis of Massive Data Streams: Proceedings of a Workshop, 2004. The National Academies Press website. Retrieved 2013-06-16.
  30. New data service speeds the progress of research, FY2010 NCAR Annual Report. Retrieved 2012-12-20.
  31. Yellowstone Supercomputer Sports Massive Xeon E5 Array, Go Parallel website at sourceforge.net. Retrieved 2012-09-01.
  32. IBM System x iDataPlex dx360 M4, IBM website. Retrieved 2013-06-20.
  33. Yellowstone Archived 2015-12-21 at the Wayback Machine , NCAR Computational and Information Systems Laboratory website. Retrieved 2012-11-20.
  34. System overview, NCAR Computational and Information Systems Laboratory website. Retrieved 2013-06-20.
  35. Yellowstone data-intensive computing environment, FY2012 CISL Annual Report. Retrieved 2012-12-20.
  36. TOP500 List – November 2012, TOP500 Supercomputer Sites. Retrieved 2012-12-01.
  37. The Green500 List – November 2012 Archived 2016-06-06 at the Wayback Machine , The Green500 website. Retrieved 2013-06-28.
  38. Cheyenne - SGI ICE XA, Xeon E5-2697v4 18C 2.3GHz, Infiniband EDR, Top500 website. Retrieved 2017-02-28.
  39. Cheyenne: NCAR’s Next-Generation Data-Centric Supercomputing Environment, CISL website. Retrieved 2017-02-28.
  40. Cheyenne - SGI ICE XA, Xeon E5-2697v4 18C 2.3GHz, Infiniband EDR, Top500 website. Retrieved 2017-02-28.
  41. Provide Supercomputing Resources, FY2016 CISL Annual Report. Retrieved 2017-02-28.
  42. NCAR Launches Five-Petaflop Supercomputer, Top500 website. Retrieved 2017-02-28.
  43. . AP. 27 January 2021 [New supercomputer in Wyoming to rank among world's fastest New supercomputer in Wyoming to rank among world's fastest]. Retrieved 28 January 2021.{{cite news}}: Check |url= value (help); Missing or empty |title= (help)
  44. System overview, NCAR Computational and Information Systems Laboratory website. Retrieved 2013-06-20.
  45. NVIDIA Tesla GPUs Power World's Fastest Supercomputer, TweakTown website. Retrieved 2013-06-25.
  46. New supercomputer in Wyoming aids Antarctic Safety: More-detailed weather forecasts to guide takeoffs, landings, NCAR-UCAR AtmosNews website. Retrieved 2013-06-20.
  47. Polar Programs (PLR), NSF website. Retrieved 2013-02-24.
  48. The Antarctic Mesoscale Prediction System (AMPS), NCAR Mesoscale and Microscale Meteorology website. Retrieved 2012-12-20.
  49. A new model: Antarctic weather forecasting system switches to latest global tool, The Antarctic Sun website. Retrieved 2013-06-25.
  50. Advancing Antarctic Science with AMPS, FY2010 NCAR Annual Report. Retrieved 2012-12-20.
  51. IBM introduces new Opteron superclusters for HPC, HPCwire website. Retrieved 2013-06-25.
  52. Intel Xeon Phi coprocessor 5110P: Highly parallel processing to power your breakthrough innovations, Intel website. Retrieved 2013-06-25.
  53. Intel Xeon Processor E5-2670 (20M Cache, 2.60 GHz, 8.00 GT/s Intel QPI), Intel website. Retrieved 2013-06-25.
  54. New data service speeds the progress of research, FY2010 NCAR Annual Report. Retrieved 2012-12-20.
  55. Globally Accessible Data Environment, FY2012 CISL Annual Report. Retrieved 2012-12-20.
  56. CISL Research Data Archive, Research Data Archive website. Retrieved 2012-09-01.
  57. Community Data Portal, Gateway to Data for the Geosciences, Community Data Portal website. Retrieved 2012-09-01.
  58. ESG Gateway at the National Center for Atmospheric Research, Earth System Grid website. Retrieved 2012-09-01.
  59. System overview, NCAR Computational and Information Systems Laboratory website. Retrieved 2013-06-20.
  60. StorageTek T10000C Tape Drive, Oracle website. Retrieved 2013-06-25.
  61. Data support for climate and weather research, FY2010 NCAR Annual Report. Retrieved 2012-12-20.
  62. CISL Research Data Archive, Research Data Archive website. Retrieved 2013-04-11.
  63. NWSC education and outreach, FY2012 CISL Annual Report. Retrieved 2012-12-20.
  64. Visitor Center, NCAR-Wyoming Supercomputing Center website. Retrieved 2013-02-20.
  65. NWSC visitor center exhibits, FY2012 CISL Annual Report. Retrieved 2012-12-20.
  66. Young scientists, NCAR-Wyoming Supercomputing Center website. Retrieved 2013-02-20.
  67. SIParCS Home, CISL Summer Internships in Parallel Computational Science website. Retrieved 2013-06-03.
  68. Grant helps CSU professors design green supercomputers, The Fort Collins Colorodoan website. Retrieved 2013-06-26.
  69. Colorado State University Engineering Professors Receive Award to Design Green Supercomputers, Colorado State University Department of Public Relations News & Information website. Retrieved 2013-06-26.
  70. Reliable, Scalable, Critical: A New Traffic Optimization Solution, Lagrange Systems website. Retrieved 2013-07-01.
  71. Research Spending & Results, Award Detail, Sudeep Pasricha, PI". Research.gov grants management website for the National Science Foundation. Retrieved 2013-06-25.

Coordinates: 41°07′44″N104°53′51″W / 41.1289°N 104.8974°W / 41.1289; -104.8974