Scalability testing

Last updated

Scalability testing is the testing of a software application to measure its capability to scale up or scale out in terms of any of its non-functional capability.

Contents

Performance, scalability and reliability testing are usually grouped together by software quality analysts.

The main goals of scalability testing are to determine the user limit for the web application and ensure end user experience, under a high load, is not compromised. One example is if a web page can be accessed in a timely fashion with a limited delay in response. Another goal is to check if the server can cope i.e. Will the server crash if it is under a heavy load? [1]

Dependent on the application that is being tested, different parameters are tested. If a webpage is being tested, the highest possible number of simultaneous users would be tested. [2] Also dependent on the application being tested is the attributes that are tested - these can include CPU usage, network usage or user experience.

Successful testing will project most of the issues which could be related to the network, database or hardware/software. [3]

Creating a scalability test

When creating a new application, it is difficult to accurately predict the number of users in 1, 2 or even 5 years. Although an estimate can be made, it is not a definite number. An issue with an increasing number of users is that it can create new areas of failure. For example, if you have 100,000 new visitors, it's not just access to the application that could be a problem; you might also experience issues with the database where you need to store all the data of these new customers. [4]

Increment loads

This is why when creating a scalability test, it is important to scale up in increments. These steps can be split into small, medium and high loads.

We must scale up in increments as each stage tests a different aspect. Small loads ensure the system functions as it should on a basic level. Medium loads test the system can function at its expected level. High loads test the system can cope with a high load. [5]

Test environment

The environment should be constant throughout testing in order to provide accurate and reliable results. If the testing is a success, we should see a proportional change in performance. For example, if we double the users on the system, we should see a drop in performance of 50%. [6]

Alternatively, if measuring system statistics such as memory or CPU usage over time, this may have a different graph that is not proportional as users are not being plotted on either axis.

Outcomes of scalability testing

Figure 1. The graph shows resource usage (memory) over time Figure 1 Memory usage vs Time.png
Figure 1. The graph shows resource usage (memory) over time

Once we have collected the data from our various stages, we can begin to plot the results on various graphs to show the results. However, the graphs can vary depending on what is being plotted.

Unproportional outcome

In Figure 1, we can see a graph showing a resources usage (in this case, memory) over time. The graph is not proportionate but can still be considered a passed test as initially there is a ramp up phase as the system begins to run, however, as more users are added, there is little change in memory usage. [7] This means that the current memory capacity can cope with all 3 stages of the test.

Proportional outcome

Figure 2. This graph shows the average time taken to execute a report plotted against the number of users Figure 2 User experience graph with trendline.png
Figure 2. This graph shows the average time taken to execute a report plotted against the number of users

In figure 2, we can see a more proportional increase, comparing the number of users to the time taken to execute a report. With a low load of 20 users, the average time is 5.5 seconds, as we increase the load to medium (40 users) and a high load (60 users), the average time increases to 9.5 and 18 seconds respectively. [6]

In some cases, there may be changes that have to be made to the server software or hardware. Once the necessary upgrades have been made, we must re-run the tests to ensure the upgrades have been effective in addressing the issues previously raised. [8]

When we have a proportional outcome, there are no bottlenecks as we scale up and increase the load the system is placed under. [9]

Vertical and horizontal scaling

As a result of scalability testing, upgrades can be required to software and hardware. These upgrades can be split into vertical or horizontal scaling.

Vertical scaling, also known as scaling up, is the process of replacing a component with a device that is generally more powerful or improved. For example, replacing a processor with a faster one. Horizontal scaling, also known as scaling out is setting up another server for example to run in parallel with the original so they share the workload. [10]

Advantages and disadvantages

There are advantages and disadvantages to both methods of scaling. Although scaling up may be simpler, the addition of hardware resources can result in diminishing returns. [11] This means that every time we upgrade the processor for example, we do not always get the same level of benefits as the previous change.

However, horizontal scaling can be extremely expensive, not only the cost of entire systems such as servers, but we must also take into account their regular maintenance costs . [11]

Related Research Articles

<span class="mw-page-title-main">BIOS</span> Firmware for hardware initialization and OS runtime services

In computing, BIOS is firmware used to provide runtime services for operating systems and programs and to perform hardware initialization during the booting process. The BIOS firmware comes pre-installed on an IBM PC or IBM PC compatible's system board and exists in some UEFI-based systems to maintain compatibility with operating systems that do not support UEFI native operation. The name originates from the Basic Input/Output System used in the CP/M operating system in 1975. The BIOS originally proprietary to the IBM PC has been reverse engineered by some companies looking to create compatible systems. The interface of that original system serves as a de facto standard.

<span class="mw-page-title-main">Database</span> Organized collection of data in computing

In computing, a database is an organized collection of data stored and accessed electronically through the use of a database management system. Small databases can be stored on a file system, while large databases are hosted on computer clusters or cloud storage. The design of databases spans formal techniques and practical considerations, including data modeling, efficient data representation and storage, query languages, security and privacy of sensitive data, and distributed computing issues, including supporting concurrent access and fault tolerance.

<span class="mw-page-title-main">Mainframe computer</span> Large computer

A mainframe computer, informally called a mainframe or big iron, is a computer used primarily by large organizations for critical applications like bulk data processing for tasks such as censuses, industry and consumer statistics, enterprise resource planning, and large-scale transaction processing. A mainframe computer is large but not as large as a supercomputer and has more processing power than some other classes of computers, such as minicomputers, servers, workstations, and personal computers. Most large-scale computer-system architectures were established in the 1960s, but they continue to evolve. Mainframe computers are often used as servers.

In software quality assurance, performance testing is in general a testing practice performed to determine how a system performs in terms of responsiveness and stability under a particular workload. It can also serve to investigate, measure, validate or verify other quality attributes of the system, such as scalability, reliability and resource usage.

<span class="mw-page-title-main">Symmetric multiprocessing</span> The equal sharing of all resources by multiple identical processors

Symmetric multiprocessing or shared-memory multiprocessing (SMP) involves a multiprocessor computer hardware and software architecture where two or more identical processors are connected to a single, shared main memory, have full access to all input and output devices, and are controlled by a single operating system instance that treats all processors equally, reserving none for special purposes. Most multiprocessor systems today use an SMP architecture. In the case of multi-core processors, the SMP architecture applies to the cores, treating them as separate processors.

Scalability is the property of a system to handle a growing amount of work. One definition for software systems specifies that this may be done by adding resources to the system.

In computing, stress testing can be applied to either hardware or software. It it used to determine the maximum capability of a computer system and is often used for purposes such as scaling for production use and ensuring reliability and stability. Stress tests typically involve running a large amount of resource-intensive processes until the system either crashes or nearly does so.

Essbase is a multidimensional database management system (MDBMS) that provides a platform upon which to build analytic applications. Essbase began as a product from Arbor Software, which merged with Hyperion Software in 1998. Oracle Corporation acquired Hyperion Solutions Corporation in 2007. Until late 2005 IBM also marketed an OEM version of Essbase as DB2 OLAP Server.

<span class="mw-page-title-main">IBM System z9</span> Line of mainframe computers

IBM System z9 is a line of IBM mainframe computers. The first models were available on September 16, 2005. The System z9 also marks the end of the previously used eServer zSeries naming convention. It was also the last mainframe computer that NASA ever used.

Since the rise of the personal computer in the 1980s, IBM and other vendors have created PC-based IBM-compatible mainframes which are compatible with the larger IBM mainframe computers. For a period of time PC-based mainframe-compatible systems had a lower price and did not require as much electricity or floor space. However, they sacrificed performance and were not as dependable as mainframe-class hardware. These products have been popular with mainframe developers, in education and training settings, for very small companies with non-critical processing, and in certain disaster relief roles.

Performance engineering encompasses the techniques applied during a systems development life cycle to ensure the non-functional requirements for performance will be met. It may be alternatively referred to as systems performance engineering within systems engineering, and software performance engineering or application performance engineering within software engineering.

Stress testing is a software testing activity that determines the robustness of software by testing beyond the limits of normal operation. Stress testing is particularly important for "mission critical" software, but is used for all types of software. Stress tests commonly put a greater emphasis on robustness, availability, and error handling under a heavy load, than on what would be considered correct behavior under normal circumstances.

The multi-stage booting process of Linux is in many ways similar to the BSD and other Unix-style boot processes, from which it derives.

In computing, virtualization or virtualisation is the act of creating a virtual version of something at the same abstraction level, including virtual computer hardware platforms, storage devices, and computer network resources.

<span class="mw-page-title-main">Computer cluster</span> Set of computers configured in a distributed computing system

A computer cluster is a set of computers that work together so that they can be viewed as a single system. Unlike grid computers, computer clusters have each node set to perform the same task, controlled and scheduled by software.

Dynamic Infrastructure is an information technology concept related to the design of data centers, whereby the underlying hardware and software can respond dynamically and more efficiently to changing levels of demand. In other words, data center assets such as storage and processing power can be provisioned to meet surges in user's needs. The concept has also been referred to as Infrastructure 2.0 and Next Generation Data Center.

Linux on IBM Z or Linux on zSystems is the collective term for the Linux operating system compiled to run on IBM mainframes, especially IBM Z / IBM zSystems and IBM LinuxONE servers. Similar terms which imply the same meaning are Linux/390, Linux/390x, etc. The three Linux distributions certified for usage on the IBM Z hardware platform are Red Hat Enterprise Linux, SUSE Linux Enterprise Server, and Ubuntu.

In computer science, in-memory processing (PIM) is a computer architecture for processing data stored in an in-memory database. In-memory processing improves the power usage and performance of moving data between the processor and the main memory. Older systems have been based on disk storage and relational databases using Structured Query Language, which are increasingly regarded as inadequate to meet business intelligence (BI) needs. Because stored data is accessed much more quickly when it is placed in random-access memory (RAM) or flash memory, in-memory processing allows data to be analyzed in real time, enabling faster reporting and decision-making in business.

<span class="mw-page-title-main">Oracle NoSQL Database</span>

Oracle NoSQL Database is a NoSQL-type distributed key-value database from Oracle Corporation. It provides transactional semantics for data manipulation, horizontal scalability, and simple administration and monitoring.

<span class="mw-page-title-main">SAP HANA</span> Database management system by SAP

SAP HANA is an in-memory, column-oriented, relational database management system developed and marketed by SAP SE. Its primary function as the software running a database server is to store and retrieve data as requested by the applications. In addition, it performs advanced analytics and includes extract, transform, load (ETL) capabilities as well as an application server.

References

  1. ↑ "Planning for Load Testing". docs.oracle.com. Retrieved 2015-10-23.
  2. ↑ "Scalability Testing". Performance Blog. Retrieved 2015-10-25.
  3. ↑ Joshi, Prateek. "Why Do We Need Performance Testing?". Perpetual Enigma. Retrieved 2015-10-25.
  4. ↑ "Discovering the right metrics for scalability testing". www.theserverside.com. Retrieved 2015-10-25.
  5. ↑ 'Cytowski' 'Bernardini', 'Maciej' 'Matteo' (2013). "Prac Scalability Analysis" (PDF). Partnership for Advanced Computing in Europe.
  6. 1 2 "IBM Cognos Proven Practices: Designing a Successful Performance and Scalability Test for IBM Cognos BI". www.ibm.com. 2011-11-17. Retrieved 2015-10-25.
  7. ↑ "Enterprise performance and test results" (PDF). Serena. 2011.
  8. ↑ "Scalability Testing: Checking Whether a Site Performance Can Scale Up". support.smartbear.com. Retrieved 2015-10-28.
  9. ↑ "The Netflix Tech Blog: Benchmarking Cassandra Scalability on AWS - Over a million writes per second". techblog.netflix.com. Retrieved 2015-11-04.
  10. ↑ Bondi, André (2014). Foundations of Software and System Performance Engineering: Process, Performance Modeling, Requirements, Testing, Scalability, and Practice. Section 11.2. ISBN   0321833821.
  11. 1 2 "Scalability Testing" (PDF). Comp Nus Education.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.