Benchmarking requires the use of specific valuation methods. With evaluation is meant the level of achieving the target for a particular evaluation item. There are general "methods" respectively approaches as well as IT-supported "software tools" that enable an effective and efficient work.
The following is a list of notable methods and benchmarking software tools.
There are many benchmarking methods each having different analytical focus. The methods are mostly known and will be shown in the following summary. [1]
Methods | Notes |
---|---|
Matrix technology | |
Comparison tables | |
Graphs: Pie chart, Bar chart and Histogram | |
SWOT analysis | |
Potential/resources-analysis | |
Price/performance ratio | |
Potential analysis | |
Life cycle analysis | |
market growth/market share portfolio | |
market attractiveness/competitive strength portfolio | |
Portfolio attractiveness customer/supplier position | |
Technology/resource strength | |
Market position/technology position portfolio | |
Contribution margin/cost development portfolio | |
Price/customer satisfaction portfolio | |
Revenue share/revenue portfolio | |
Spider web diagram | |
This section needs expansion. You can help by adding to it. (August 2011) |
There are a number software tools that allow the support of different kinds of benchmarking types.
Software | Publisher | Platforms | Notes |
---|---|---|---|
Baromitr.com | Subscribe Labs Inc | web-based | This web-based software platform allows any peer group benchmark any performance area of interest. Data is kept secure, identities are kept anonymous and results are shared in a log-in only environment with time-series visualizations and detailed views. |
BenchmarkIndex | Winning Moves Ltd | web-based | |
Combo Benchmark | Compare to Compete Online Benchmarking | web-based database | This web-based database is suitable for groups of competitors to benchmark individual performance against group performance. All process and performance benchmarks can be processed in this software, providing interesting analysis tools and complete benchmarking report. |
Gobench | INDEC GmbH & Co. KG | web-based database | The web-based database supports different kinds of benchmarking categories (product, process, competitor / customer, reverse engineering, marketing, patents, technologies, innovations, ...) and allows reams of analysing possibilities [2] |
PowerStats.com | PowerStats Limited, Auckland, NZ | web-based, automated, self-service | PowerStats is a highly automated platform intended to allow groups of competitors to contribute, manage and visualize data for purposes of benchmarking. Participants load raw data on a regular basis, and PowerStats creates relevant KPIs in real-time, making them available via self-service charts and tables via its online interface. |
Workload Simulator (WSim) | IBM | mainframe server | WSim simulates one or many network terminal(s) to load a mainframe computer system by executing programmed scripts, for functional testing, system testing, regression testing, capacity management, benchmarking and stress testing. It is a re-packaged, subset version of IBM's Teleprocessing Network Simulator. [3] : 19–22 |
Value Lifecycle Manager | SAP | web-based, automated, self-service | SAP Value Lifecycle Manager (VLM) version = Based on an internal system that has been in use for more than 10 years. After registering, it is self-service for survey capture and provides personalized dashboards. The idea being that you can estimate the value of a business initiative. |
There are numerous suites for examining the performance of a High Performance Computing cluster, including
A supercomputer is a computer with a high level of performance as compared to a general-purpose computer. The performance of a supercomputer is commonly measured in floating-point operations per second (FLOPS) instead of million instructions per second (MIPS). Since 2017, supercomputers have existed which can perform over 1017 FLOPS (a hundred quadrillion FLOPS, 100 petaFLOPS or 100 PFLOPS). For comparison, a desktop computer has performance in the range of hundreds of gigaFLOPS (1011) to tens of teraFLOPS (1013). Since November 2017, all of the world's fastest 500 supercomputers run on Linux-based operating systems. Additional research is being conducted in the United States, the European Union, Taiwan, Japan, and China to build faster, more powerful and technologically superior exascale supercomputers.
Benchmarking is the practice of comparing business processes and performance metrics to industry bests and best practices from other companies. Dimensions typically measured are quality, time and cost.
The rational unified process (RUP) is an iterative software development process framework created by the Rational Software Corporation, a division of IBM since 2003. RUP is not a single concrete prescriptive process, but rather an adaptable process framework, intended to be tailored by the development organizations and software project teams that will select the elements of the process that are appropriate for their needs. RUP is a specific implementation of the Unified Process.
OpenMP is an application programming interface (API) that supports multi-platform shared-memory multiprocessing programming in C, C++, and Fortran, on many platforms, instruction-set architectures and operating systems, including Solaris, AIX, FreeBSD, HP-UX, Linux, macOS, and Windows. It consists of a set of compiler directives, library routines, and environment variables that influence run-time behavior.
The IBM Information Management System (IMS) is a joint hierarchical database and information management system that supports transaction processing.
Computer-aided software engineering (CASE) was a domain of software tools used to design and implement applications. CASE tools were similar to and were partly inspired by Computer-Aided Design (CAD) tools used for designing hardware products. CASE tools were intended to help develop high-quality, defect-free, and maintainable software. CASE software was often associated with methods for the development of information systems together with automated tools that could be used in the software development process.
High-performance computing (HPC) uses supercomputers and computer clusters to solve advanced computation problems.
Essbase is a multidimensional database management system (MDBMS) that provides a platform upon which to build analytic applications. Essbase began as a product from Arbor Software, which merged with Hyperion Software in 1998. Oracle Corporation acquired Hyperion Solutions Corporation in 2007. Until late 2005 IBM also marketed an OEM version of Essbase as DB2 OLAP Server.
POWER7 is a family of superscalar multi-core microprocessors based on the Power ISA 2.06 instruction set architecture released in 2010 that succeeded the POWER6 and POWER6+. POWER7 was developed by IBM at several sites including IBM's Rochester, MN; Austin, TX; Essex Junction, VT; T. J. Watson Research Center, NY; Bromont, QC and IBM Deutschland Research & Development GmbH, Böblingen, Germany laboratories. IBM announced servers based on POWER7 on 8 February 2010.
In computing, a benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, normally by running a number of standard tests and trials against it.
SPEC INT is a computer benchmark specification for CPU integer processing power. It is maintained by the Standard Performance Evaluation Corporation (SPEC). SPEC INT is the integer performance testing component of the SPEC test suite. The first SPEC test suite, CPU92, was announced in 1992. It was followed by CPU95, CPU2000, and CPU2006. The latest standard is SPEC CPU 2017 and consists of SPEC speed and SPE Crate.
NAS Parallel Benchmarks (NPB) are a set of benchmarks targeting performance evaluation of highly parallel supercomputers. They are developed and maintained by the NASA Advanced Supercomputing (NAS) Division based at the NASA Ames Research Center. NAS solicits performance results for NPB from all sources.
In software engineering, profiling is a form of dynamic program analysis that measures, for example, the space (memory) or time complexity of a program, the usage of particular instructions, or the frequency and duration of function calls. Most commonly, profiling information serves to aid program optimization, and more specifically, performance engineering.
VMmark is a freeware virtual machine benchmark software suite from VMware, Inc. The suite measures the performance of virtualized servers while running under load on a set of physical hardware. VMmark was independently developed by VMware.
Lis is a scalable parallel software library for solving discretized linear equations and eigenvalue problems that mainly arise in the numerical solution of partial differential equations by using iterative methods. Although it is designed for parallel computers, the library can be used without being conscious of parallel processing.
A computer cluster is a set of computers that work together so that they can be viewed as a single system. Unlike grid computers, computer clusters have each node set to perform the same task, controlled and scheduled by software. The newest manifestation of cluster computing is cloud computing.
In computer science and computer engineering, computer architecture is a description of the structure of a computer system made from component parts. It can sometimes be a high-level description that ignores details of the implementation. At a more detailed level, the description may include the instruction set architecture design, microarchitecture design, logic design, and implementation.
HPC Challenge Benchmark combines several benchmarks to test a number of independent attributes of the performance of high-performance computer (HPC) systems. The project has been co-sponsored by the DARPA High Productivity Computing Systems program, the United States Department of Energy and the National Science Foundation.
Tachyon is a parallel/multiprocessor ray tracing software. It is a parallel ray tracing library for use on distributed memory parallel computers, shared memory computers, and clusters of workstations. Tachyon implements rendering features such as ambient occlusion lighting, depth-of-field focal blur, shadows, reflections, and others. It was originally developed for the Intel iPSC/860 by John Stone for his M.S. thesis at University of Missouri-Rolla. Tachyon subsequently became a more functional and complete ray tracing engine, and it is now incorporated into a number of other open source software packages such as VMD, and SageMath. Tachyon is released under a permissive license.
Singularity is a free and open-source computer program that performs operating-system-level virtualization also known as containerization.