The Computer Language Benchmarks Game

Last updated

The Computer Language Benchmarks Game (formerly called The Great Computer Language Shootout) is a free software project for comparing how a given subset of simple algorithms can be implemented in various popular programming languages.

Contents

The project consists of:

Supported languages

Due to resource constraints, only a small subset of common programming languages are supported, up to the discretion of the game's operator. [1]

List of supported languages

Metrics

The following aspects of each given implementation are measured: [2]

It is common to see multiple solutions in the same programming language for the same problem. This highlights that within the constraints of a given language, a solution can be given which is either of high abstraction, is memory efficient, is fast, or can be parallelized better.

Benchmark programs

It was a design choice from the start to only include very simple toy problems, each providing a different kind of programming challenge. [3] This provides users of the Benchmark Game the opportunity to scrutinize the various implementations. [4]

History

The project was known as The Great Computer Language Shootout until 2007. [5]

A port for Windows was maintained separately between 2002 and 2003. [6]

The sources have been archived on GitLab. [7]

There are also older forks on GitHub. [8]

The project is continuously evolving. The list of supported programming languages is updated approximately once per year, following market trends. Users can also submit improved solutions to any of the problems or suggest testing methodology refinement. [9]

Caveats

The developers themselves highlight the fact that those doing research should exercise caution when using such microbenchmarks:

[...] the JavaScript benchmarks are fleetingly small, and behave in ways that are significantly different than the real applications. We have documented numerous differences in behavior, and we conclude from these measured differences that results based on the benchmarks may mislead JavaScript engine implementers. Furthermore, we observe interesting behaviors in real JavaScript applications that the benchmarks fail to exhibit, suggesting that previously unexplored optimization strategies may be productive in practice.

Impact

The benchmark results have uncovered various compiler issues. Sometimes a given compiler failed to process unusual, but otherwise grammatically valid constructs. At other times, runtime performance was shown to be below expectations, which prompted compiler developers to revise their optimization capabilities.

Various research articles have been based on the benchmarks, its results and its methodology. [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [ excessive citations ]

See also

Related Research Articles

<span class="mw-page-title-main">Java (programming language)</span> Object-oriented programming language

Java is a high-level, class-based, object-oriented programming language that is designed to have as few implementation dependencies as possible. It is a general-purpose programming language intended to let programmers write once, run anywhere (WORA), meaning that compiled Java code can run on all platforms that support Java without the need to recompile. Java applications are typically compiled to bytecode that can run on any Java virtual machine (JVM) regardless of the underlying computer architecture. The syntax of Java is similar to C and C++, but has fewer low-level facilities than either of them. The Java runtime provides dynamic capabilities that are typically not available in traditional compiled languages.

<span class="mw-page-title-main">Java virtual machine</span> Virtual machine that runs Java programs

A Java virtual machine (JVM) is a virtual machine that enables a computer to run Java programs as well as programs written in other languages that are also compiled to Java bytecode. The JVM is detailed by a specification that formally describes what is required in a JVM implementation. Having a specification ensures interoperability of Java programs across different implementations so that program authors using the Java Development Kit (JDK) need not worry about idiosyncrasies of the underlying hardware platform.

In computing, just-in-time (JIT) compilation is compilation during execution of a program rather than before execution. This may consist of source code translation but is more commonly bytecode translation to machine code, which is then executed directly. A system implementing a JIT compiler typically continuously analyses the code being executed and identifies parts of the code where the speedup gained from compilation or recompilation would outweigh the overhead of compiling that code.

SIGPLAN is the Association for Computing Machinery's Special Interest Group (SIG) on programming languages. This SIG explores programming language concepts and tools, focusing on design, implementation, practice, and theory. Its members are programming language developers, educators, implementers, researchers, theoreticians, and users.

BeanShell is a small, free, embeddable Java source interpreter with object scripting language features, written in Java. It runs in the Java Runtime Environment (JRE), dynamically executes standard Java syntax and extends it with common scripting conveniences such as loose types, commands, and method closures, like those in Perl and JavaScript.

HotSpot, released as Java HotSpot Performance Engine, is a Java virtual machine for desktop and server computers, developed by Sun Microsystems which was purchased by and became a division of Oracle Corporation in 2010. Its features improved performance via methods such as just-in-time compilation and adaptive optimization. It is the de facto Java Virtual Machine, serving as the reference implementation of the Java programming language.

JRuby is an implementation of the Ruby programming language atop the Java Virtual Machine, written largely in Java. It is free software released under a three-way EPL/GPL/LGPL license. JRuby is tightly integrated with Java to allow the embedding of the interpreter into any Java application with full two-way access between the Java and the Ruby code.

<span class="mw-page-title-main">Java (software platform)</span> Set of computer software and specifications

Java is a set of computer software and specifications that provides a software platform for developing application software and deploying it in a cross-platform computing environment. Java is used in a wide variety of computing platforms from embedded devices and mobile phones to enterprise servers and supercomputers. Java applets, which are less common than standalone Java applications, were commonly run in secure, sandboxed environments to provide many features of native applications through being embedded in HTML pages.

In software development, the programming language Java was historically considered slower than the fastest third-generation typed languages such as C and C++. In contrast to those languages, Java compiles by default to a Java Virtual Machine (JVM) with operations distinct from those of the actual computer hardware. Early JVM implementations were interpreters; they simulated the virtual operations one-by-one rather than translating them into machine code for direct hardware execution.

The Dynamic Language Runtime (DLR) from Microsoft runs on top of the Common Language Runtime (CLR) and provides computer language services for dynamic languages. These services include:

Eclipse OpenJ9 is a high performance, scalable, Java virtual machine (JVM) implementation that is fully compliant with the Java Virtual Machine Specification.

<span class="mw-page-title-main">Da Vinci Machine</span> Sun Microsystems project

The Da Vinci Machine, also called the Multi Language Virtual Machine, was a Sun Microsystems project aiming to prototype the extension of the Java Virtual Machine (JVM) to add support for dynamic languages.

<span class="mw-page-title-main">Clojure</span> Dialect of the Lisp programming language on the Java platform

Clojure is a dynamic and functional dialect of the programming language Lisp on the Java platform.

Nashorn is a JavaScript engine developed in the Java programming language originally by Oracle and later by the OpenJDK Community. It relies on the support for dynamically typed languages on the Java Platform Nashorn was included with Java 8 through JDK 14.

Java bytecode is the instruction set of the Java virtual machine (JVM), the language to which Java and other JVM-compatible source code is compiled. Each instruction is represented by a single byte, hence the name bytecode, making it a compact form of data.

asm.js is a subset of JavaScript designed to allow computer software written in languages such as C to be run as web applications while maintaining performance characteristics considerably better than standard JavaScript, which is the typical language used for such applications.

<span class="mw-page-title-main">GraalVM</span> Virtual machine software

GraalVM is a Java Development Kit (JDK) written in Java. The open-source distribution of GraalVM is based on OpenJDK, and the enterprise distribution is based on Oracle JDK. As well as just-in-time (JIT) compilation, GraalVM can compile a Java application ahead of time. This allows for faster initialization, greater runtime performance, and decreased resource consumption, but the resulting executable can only run on the platform it was compiled for.

Eclipse Deeplearning4j is a programming library written in Java for the Java virtual machine (JVM). It is a framework with wide support for deep learning algorithms. Deeplearning4j includes implementations of the restricted Boltzmann machine, deep belief net, deep autoencoder, stacked denoising autoencoder and recursive neural tensor network, word2vec, doc2vec, and GloVe. These algorithms all include distributed parallel versions that integrate with Apache Hadoop and Spark.

References

  1. ↑ "The Computer Language Benchmarks Game". benchmarksgame-team.pages.debian.net/benchmarksgame. Retrieved 29 May 2018.
  2. ↑ "How programs are measured – The Computer Language Benchmarks Game". benchmarksgame-team.pages.debian.net/benchmarksgame/. Retrieved 29 May 2018.
  3. ↑ "Why toy programs? – The Computer Language Benchmarks Game". benchmarksgame-team.pages.debian.net/benchmarksgame. Retrieved 29 May 2018.
  4. ↑ "n-body description (64-bit Ubuntu quad core) – Computer Language Benchmarks Game". benchmarksgame-team.pages.debian.net/benchmarksgame. Retrieved 29 May 2018.
  5. ↑ "Trust, and verify – Computer Language Benchmarks Game". benchmarksgame-team.pages.debian.net/benchmarksgame. Retrieved 29 May 2018.
  6. ↑ "The Great Win32 Computer Language Shootout". Dada.perl.it. Retrieved 13 December 2017.
  7. ↑ "archive-alioth-benchmarksgame". salsa.debian.org/benchmarksgame-team. Retrieved 29 May 2018.
  8. ↑ Thiel, Sebastian (24 October 2017). "benchmarksgame-cvs-mirror: A git mirror of the benchmarksgame cvs repository". GitHub . Retrieved 13 December 2017.
  9. ↑ "Contribute your own program – Computer Language Benchmarks Game". benchmarksgame-team.pages.debian.net/benchmarksgame. Retrieved 29 May 2018.
  10. ↑ Kevin Williams; Jason McCandless; David Gregg (2009). "Dynamic Interpretation for Dynamic Scripting Languages" (PDF). Retrieved 25 March 2017.{{cite journal}}: Cite journal requires |journal= (help)
  11. ↑ Tobias Wrigstad; Francesco Zappa Nardelli; Sylvain Lebresne Johan; Ostlund Jan Vitek (January 17–23, 2009). Integrating Typed and Untyped Code in a Scripting Language (PDF). POPL’10. Madrid, Spain. Retrieved 25 March 2017.
  12. ↑ Lerche, Carl (April 17–18, 2009). Write Fast Ruby: It's All About the Science (PDF). Golden Gate Ruby Conference. San Francisco, California. Retrieved 25 March 2017.
  13. ↑ J. Shirako; D. M. Peixotto; V. Sarkar; W. N. Scherer III (2009). Phaser Accumulators: a New Reduction Construct for Dynamic Parallelism (PDF). IEEE International Symposium on Parallel & Distributed Processing. Retrieved 25 March 2017.
  14. ↑ Rajesh Karmani and Amin Shali and Gul Agha (2009). "Actor frameworks for the JVM platform: A Comparative Analysis" (PDF). In Proceedings of the 7th International Conference on the Principles and Practice of Programming in Java. Retrieved 26 March 2017.
  15. ↑ Brunthaler Stefan (2010). Inline Caching Meets Quickening. European Conference on Object-Oriented Programming (ECOOP). Vol. Object-Oriented Programming. pp. 429–451. doi:10.1007/978-3-642-14107-2_21.
  16. ↑ Prodromos Gerakios; Nikolaos Papaspyrou; Konstantinos Sagonas (January 23, 2010). Race-free and Memory-safe Multithreading: Design and Implementation in Cyclone (PDF). Proceedings of the 5th ACM SIGPLAN workshop on Types in language design and implementation. Madrid, Spain. pp. 15–26. Retrieved 25 March 2017.
  17. ↑ Slava Pestov; Daniel Ehrenberg; Joe Groff (October 18, 2010). Factor: A Dynamic Stack-based Programming Language (PDF). DLS 2010. Reno/Tahoe, Nevada, USA. Retrieved 25 March 2017.
  18. ↑ Andrei Homescu; Alex Suhan (October 24, 2011). HappyJIT: A Tracing JIT Compiler for PHP (PDF). DLS’11. Portland, Oregon, USA. Retrieved 25 March 2017.
  19. ↑ Vincent St-Amour; Sam Tobin-Hochstadt; Matthias Felleisen (October 19–26, 2012). Optimization Coaching – Optimizers Learn to Communicate with Programmers (PDF). OOPSLA’12. Tucson, Arizona, USA. Retrieved 25 March 2017.
  20. ↑ Wing Hang Li; David R. White; Jeremy Singer (September 11–13, 2013). JVM-Hosted Languages: They Talk the Talk, but do they Walk the Walk? (PDF). Proceedings of the 2013 International Conference on Principles and Practices of Programming on the Java Platform: Virtual Machines, Languages, and Tools. Stuttgart, Germany. pp. 101–112. Retrieved 25 March 2017.
  21. ↑ Aibek Sarimbekov; Andrej Podzimek; Lubomir Bulej; Yudi Zheng; Nathan Ricci; Walter Binder (October 28, 2013). Characteristics of Dynamic JVM Languages (PDF). VMIL ’13. Indianapolis, Indiana, USA. Retrieved 25 March 2017.
  22. ↑ Bradford L. Chamberlain; Ben Albrecht; Lydia Duncan; Ben Harshbarger (2017). Entering the Fray: Chapel's Computer Language Benchmark Game Entry (PDF). Retrieved 25 March 2017.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.