List of benchmarking methods and software tools

Last updated

Benchmarking requires the use of specific valuation methods. With evaluation is meant the level of achieving the target for a particular evaluation item. There are general "methods" respectively approaches as well as IT-supported "software tools" that enable an effective and efficient work.

Contents

The following is a list of notable methods and benchmarking software tools.

Benchmarking methods

There are many benchmarking methods each having different analytical focus. The methods are mostly known and will be shown in the following summary. [1]

MethodsNotes
Matrix technology
Comparison tables
Graphs: Pie chart, Bar chart and Histogram
SWOT analysis
Potential/resources-analysis
Price/performance ratio
Potential analysis
Life cycle analysis
market growth/market share portfolio
market attractiveness/competitive strength portfolio
Portfolio attractiveness customer/supplier position
Technology/resource strength
Market position/technology position portfolio
Contribution margin/cost development portfolio
Price/customer satisfaction portfolio
Revenue share/revenue portfolio
Spider web diagram

Benchmarking software tools

There are a number software tools that allow the support of different kinds of benchmarking types.

SoftwarePublisherPlatformsNotes
Baromitr.comSubscribe Labs Incweb-basedThis web-based software platform allows any peer group benchmark any performance area of interest. Data is kept secure, identities are kept anonymous and results are shared in a log-in only environment with time-series visualizations and detailed views.
BenchmarkIndexWinning Moves Ltdweb-based
Combo BenchmarkCompare to Compete Online Benchmarkingweb-based databaseThis web-based database is suitable for groups of competitors to benchmark individual performance against group performance. All process and performance benchmarks can be processed in this software, providing interesting analysis tools and complete benchmarking report.
GobenchINDEC GmbH & Co. KGweb-based databaseThe web-based database supports different kinds of benchmarking categories (product, process, competitor / customer, reverse engineering, marketing, patents, technologies, innovations, ...) and allows reams of analysing possibilities [2]
PowerStats.comPowerStats Limited, Auckland, NZweb-based, automated, self-servicePowerStats is a highly automated platform intended to allow groups of competitors to contribute, manage and visualize data for purposes of benchmarking. Participants load raw data on a regular basis, and PowerStats creates relevant KPIs in real-time, making them available via self-service charts and tables via its online interface.
Workload Simulator (WSim) IBM mainframe server WSim simulates one or many network terminal(s) to load a mainframe computer system by executing programmed scripts, for functional testing, system testing, regression testing, capacity management, benchmarking and stress testing. It is a re-packaged, subset version of IBM's Teleprocessing Network Simulator. [3] :19–22
Value Lifecycle ManagerSAPweb-based, automated, self-serviceSAP Value Lifecycle Manager (VLM) version = Based on an internal system that has been in use for more than 10 years. After registering, it is self-service for survey capture and provides personalized dashboards. The idea being that you can estimate the value of a business initiative.

Benchmarking HPC Clusters

There are numerous suites for examining the performance of a High Performance Computing cluster, including

See also

Related Research Articles

<span class="mw-page-title-main">Supercomputer</span> Type of extremely powerful computer

A supercomputer is a computer with a high level of performance as compared to a general-purpose computer. The performance of a supercomputer is commonly measured in floating-point operations per second (FLOPS) instead of million instructions per second (MIPS). Since 2017, supercomputers have existed which can perform over 1017 FLOPS (a hundred quadrillion FLOPS, 100 petaFLOPS or 100 PFLOPS). For comparison, a desktop computer has performance in the range of hundreds of gigaFLOPS (1011) to tens of teraFLOPS (1013). Since November 2017, all of the world's fastest 500 supercomputers run on Linux-based operating systems. Additional research is being conducted in the United States, the European Union, Taiwan, Japan, and China to build faster, more powerful and technologically superior exascale supercomputers.

Benchmarking is the practice of comparing business processes and performance metrics to industry bests and best practices from other companies. Dimensions typically measured are quality, time and cost.

The rational unified process (RUP) is an iterative software development process framework created by the Rational Software Corporation, a division of IBM since 2003. RUP is not a single concrete prescriptive process, but rather an adaptable process framework, intended to be tailored by the development organizations and software project teams that will select the elements of the process that are appropriate for their needs. RUP is a specific implementation of the Unified Process.

<span class="mw-page-title-main">OpenMP</span> Open standard for parallelizing

OpenMP is an application programming interface (API) that supports multi-platform shared-memory multiprocessing programming in C, C++, and Fortran, on many platforms, instruction-set architectures and operating systems, including Solaris, AIX, FreeBSD, HP-UX, Linux, macOS, and Windows. It consists of a set of compiler directives, library routines, and environment variables that influence run-time behavior.

<span class="mw-page-title-main">IBM Information Management System</span> Joint hierarchical database made by IBM

The IBM Information Management System (IMS) is a joint hierarchical database and information management system that supports transaction processing.

<span class="mw-page-title-main">Computer-aided software engineering</span> Software Quality Engineering Practices

Computer-aided software engineering (CASE) was a domain of software tools used to design and implement applications. CASE tools were similar to and were partly inspired by Computer-Aided Design (CAD) tools used for designing hardware products. CASE tools were intended to help develop high-quality, defect-free, and maintainable software. CASE software was often associated with methods for the development of information systems together with automated tools that could be used in the software development process.

<span class="mw-page-title-main">High-performance computing</span> Computing with supercomputers and clusters

High-performance computing (HPC) uses supercomputers and computer clusters to solve advanced computation problems.

Essbase is a multidimensional database management system (MDBMS) that provides a platform upon which to build analytic applications. Essbase began as a product from Arbor Software, which merged with Hyperion Software in 1998. Oracle Corporation acquired Hyperion Solutions Corporation in 2007. Until late 2005 IBM also marketed an OEM version of Essbase as DB2 OLAP Server.

<span class="mw-page-title-main">POWER7</span> 2010 family of multi-core microprocessors by IBM

POWER7 is a family of superscalar multi-core microprocessors based on the Power ISA 2.06 instruction set architecture released in 2010 that succeeded the POWER6 and POWER6+. POWER7 was developed by IBM at several sites including IBM's Rochester, MN; Austin, TX; Essex Junction, VT; T. J. Watson Research Center, NY; Bromont, QC and IBM Deutschland Research & Development GmbH, Böblingen, Germany laboratories. IBM announced servers based on POWER7 on 8 February 2010.

<span class="mw-page-title-main">Benchmark (computing)</span> Comparing the relative performance of computers by running the same program on all of them

In computing, a benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, normally by running a number of standard tests and trials against it.

SPEC INT is a computer benchmark specification for CPU integer processing power. It is maintained by the Standard Performance Evaluation Corporation (SPEC). SPEC INT is the integer performance testing component of the SPEC test suite. The first SPEC test suite, CPU92, was announced in 1992. It was followed by CPU95, CPU2000, and CPU2006. The latest standard is SPEC CPU 2017 and consists of SPEC speed and SPE Crate.

NAS Parallel Benchmarks (NPB) are a set of benchmarks targeting performance evaluation of highly parallel supercomputers. They are developed and maintained by the NASA Advanced Supercomputing (NAS) Division based at the NASA Ames Research Center. NAS solicits performance results for NPB from all sources.

In software engineering, profiling is a form of dynamic program analysis that measures, for example, the space (memory) or time complexity of a program, the usage of particular instructions, or the frequency and duration of function calls. Most commonly, profiling information serves to aid program optimization, and more specifically, performance engineering.

VMmark is a freeware virtual machine benchmark software suite from VMware, Inc. The suite measures the performance of virtualized servers while running under load on a set of physical hardware. VMmark was independently developed by VMware.

<span class="mw-page-title-main">Lis (linear algebra library)</span>

Lis is a scalable parallel software library for solving discretized linear equations and eigenvalue problems that mainly arise in the numerical solution of partial differential equations by using iterative methods. Although it is designed for parallel computers, the library can be used without being conscious of parallel processing.

<span class="mw-page-title-main">Computer cluster</span> Set of computers configured in a distributed computing system

A computer cluster is a set of computers that work together so that they can be viewed as a single system. Unlike grid computers, computer clusters have each node set to perform the same task, controlled and scheduled by software. The newest manifestation of cluster computing is cloud computing.

<span class="mw-page-title-main">Computer architecture</span> Set of rules describing computer system

In computer science and computer engineering, computer architecture is a description of the structure of a computer system made from component parts. It can sometimes be a high-level description that ignores details of the implementation. At a more detailed level, the description may include the instruction set architecture design, microarchitecture design, logic design, and implementation.

HPC Challenge Benchmark combines several benchmarks to test a number of independent attributes of the performance of high-performance computer (HPC) systems. The project has been co-sponsored by the DARPA High Productivity Computing Systems program, the United States Department of Energy and the National Science Foundation.

<span class="mw-page-title-main">Tachyon (software)</span>

Tachyon is a parallel/multiprocessor ray tracing software. It is a parallel ray tracing library for use on distributed memory parallel computers, shared memory computers, and clusters of workstations. Tachyon implements rendering features such as ambient occlusion lighting, depth-of-field focal blur, shadows, reflections, and others. It was originally developed for the Intel iPSC/860 by John Stone for his M.S. thesis at University of Missouri-Rolla. Tachyon subsequently became a more functional and complete ray tracing engine, and it is now incorporated into a number of other open source software packages such as VMD, and SageMath. Tachyon is released under a permissive license.

<span class="mw-page-title-main">Singularity (software)</span> Free, cross-platform and open-source computer program

Singularity is a free and open-source computer program that performs operating-system-level virtualization also known as containerization.

References

  1. Peter Kairies: So analysieren Sie Ihre Konkurrenz. expert Verlag, Renningen 2001, 3-8169-1977-4.
  2. Anette von Ahsen (Hrsg.): Bewertung von Innovationen im Mittelstand. Springer, Berlin-Heidelberg 2010, 978-3-642-01699-8.
    • IBM Workload Simulator ~ User's Guide (PDF). First Edition. IBM. August 2002. SC31-8948-00. Retrieved on October 29, 2015.