In-memory database

Last updated

An in-memory database (IMDb, or main memory database system (MMDB) or memory resident database) is a database management system that primarily relies on main memory for computer data storage. It is contrasted with database management systems that employ a disk storage mechanism. In-memory databases are faster than disk-optimized databases because disk access is slower than memory access and the internal optimization algorithms are simpler and execute fewer CPU instructions. Accessing data in memory eliminates seek time when querying the data, which provides faster and more predictable performance than disk. [1] [2]

Contents

Applications where response time is critical, such as those running telecommunications network equipment and mobile advertising networks, often use main-memory databases. [3] IMDBs have gained much traction, especially in the data analytics space, starting in the mid-2000s – mainly due to multi-core processors that can address large memory and due to less expensive RAM. [4] [5]

A potential technical hurdle with in-memory data storage is the volatility of RAM. Specifically in the event of a power loss, intentional or otherwise, data stored in volatile RAM is lost. [6] With the introduction of non-volatile random-access memory technology, in-memory databases will be able to run at full speed and maintain data in the event of power failure. [7] [8] [9]

ACID support

In its simplest form, main memory databases store data on volatile memory devices. These devices lose all stored information when the device loses power or is reset. In this case, IMDBs can be said to lack support for the "durability" portion of the ACID (atomicity, consistency, isolation, durability) properties. Volatile memory-based IMDBs can, and often do, support the other three ACID properties of atomicity, consistency and isolation.

Many IMDBs have added durability via the following mechanisms:

Some IMDBs allow the database schema to specify different durability requirements for selected areas of the database – thus, faster-changing data that can easily be regenerated or that has no meaning after a system shut-down would not need to be journaled for durability (though it would have to be replicated for high availability), whereas configuration information would be flagged as needing preservation.

Hybrids with on-disk databases

While storing data in-memory confers performance advantages, it is an expensive method of data storage. An approach to realising the benefits of in-memory storage while limiting its costs is to store the most frequently accessed data in-memory and the rest on disk. Since there is no hard distinction between which data should be stored in-memory and which should be stored on disk, some systems dynamically update where data is stored based on the data's usage. [10] This approach is subtly different from caching, in which the most recently accessed data is cached, as opposed to the most frequently accessed data being stored in-memory.

The flexibility of hybrid approaches allow a balance to be struck between:

In the cloud computing industry the terms "data temperature", or "hot data" and "cold data" have emerged to describe how data is stored in this respect. [11] Hot data is used to describe mission-critical data that needs to be accessed frequently while cold data describes data that is needed less often and less urgently, such as data kept for archiving or auditing purposes. Hot data should be stored in ways offering fast retrieval and modification, often accomplished by in-memory storage but not always. Cold data on the other hand can be stored in a more cost-effective way and is accepted that data access will likely be slower compared to hot data. While these descriptions are useful, "hot" and "cold" lack concrete definitions. [11]

Manufacturing efficiency provides another reason for selecting a combined in-memory/on-disk database system. Some device product lines, especially in consumer electronics, include some units with permanent storage, and others that rely on memory for storage (set-top boxes, for example). If such devices require a database system, a manufacturer can adopt a hybrid database system at lower and upper cost, and with less customization of code, rather than using separate in-memory and on-disk databases, respectively, for its disk-less and disk-based products.

The first database engine to support both in-memory and on-disk tables in a single database, WebDNA, was released in 1995.

Storage memory

Another variation involves large amounts of nonvolatile memory in the server, for example, flash memory chips as addressable memory rather than structured as disk arrays. A database in this form of memory combines very fast access speed with persistence over reboots and power losses. [12]

Notable In-memory Databases

See also

Notes

  1. ↑ "Definition: in-memory database". WhatIs.com. Retrieved 19 January 2013.
  2. ↑ Michael Vizard. "The Rise of In-Memory Databases". Slashdot. Archived from the original on 1 February 2013. Retrieved 19 January 2013.
  3. ↑ "TeleCommunication Systems Signs up as a Reseller of TimesTen; Mobile Operators and Carriers Gain Real-Time Platform for Location-Based Services". Business Wire. 2002-06-24.
  4. ↑ "Falling RAM Prices Drive In-Memory Database Surge". SAP. Archived from the original on 4 November 2013. Retrieved 19 January 2013.
  5. ↑ "Rise of In-Memory Databases Impacts Wide Range of Jobs". Dice.com. July 13, 2012.
  6. ↑ "In-memory computing: what happens when the power goes out?". 18 February 2015. Retrieved March 10, 2017.
  7. ↑ Historically, RAM was not used as a persistent data store and therefore data loss in these instances was not an issue.Whole-system Persistence with Non-volatile Memories http://research.microsoft.com/apps/pubs/default.aspx?id=160853
  8. ↑ The Bleak Future of NAND Flash Memory http://research.microsoft.com/apps/pubs/default.aspx?id=162804
  9. ↑ AGIGARAM NVDIMM saves data through system failure https://www.embedded.com/electronics-products/electronic-product-reviews/real-time-and-performance/4422291/AGIGARAM-NVDIMM-saves-data-through-system-failure
  10. ↑ "Teradata enters the in-memory fray, intelligently ZDNet". ZDNet . Retrieved July 28, 2017.
  11. 1 2 "What's the Diff: Hot and Cold Data Storage" . Retrieved July 28, 2017.
  12. ↑ "Truly these are the GOLDEN YEARS of Storage."
  13. ↑ "What is SAP HANA?". SAP. Retrieved 2024-08-01.
  14. ↑ "Oracle TimesTen In-Memory Database".

Related Research Articles

<span class="mw-page-title-main">Computer data storage</span> Storage of digital data readable by computers

Computer data storage or digital data storage is a technology consisting of computer components and recording media that are used to retain digital data. It is a core function and fundamental component of computers.

<span class="mw-page-title-main">Computer memory</span> Computer component that stores information for immediate use

Computer memory stores information, such as data and programs, for immediate use in the computer. The term memory is often synonymous with the terms RAM,main memory, or primary storage. Archaic synonyms for main memory include core and store.

A RAM drive is a block of random-access memory that a computer's software is treating as if the memory were a disk drive. RAM drives provide high-performance temporary storage for demanding tasks and protect non-volatile storage devices from wearing down, since RAM is not prone to wear from writing, unlike non-volatile flash memory. They are in a sense the reverse of virtual memory: RAM drive uses a volatile fast memory as if it's a nonvolatile slow memory. Virtual memory is the opposite.

In database systems, durability is the ACID property that guarantees that the effects of transactions that have been committed will survive permanently, even in case of failures, including incidents and catastrophic events. For example, if a flight booking reports that a seat has successfully been booked, then the seat will remain booked even if the system crashes.

Non-volatile memory (NVM) or non-volatile storage is a type of computer memory that can retain stored information even after power is removed. In contrast, volatile memory needs constant power in order to retain data.

The Write Anywhere File Layout (WAFL) is a proprietary file system that supports large, high-performance RAID arrays, quick restarts without lengthy consistency checks in the event of a crash or power failure, and growing the filesystems size quickly. It was designed by NetApp for use in its storage appliances like NetApp FAS, AFF, Cloud Volumes ONTAP and ONTAP Select.

Semiconductor memory is a digital electronic semiconductor device used for digital data storage, such as computer memory. It typically refers to devices in which data is stored within metal–oxide–semiconductor (MOS) memory cells on a silicon integrated circuit memory chip. There are numerous different types using different semiconductor technologies. The two main types of random-access memory (RAM) are static RAM (SRAM), which uses several transistors per memory cell, and dynamic RAM (DRAM), which uses a transistor and a MOS capacitor per cell. Non-volatile memory uses floating-gate memory cells, which consist of a single floating-gate transistor per cell.

A hybrid drive is a logical or physical computer storage device that combines a faster storage medium such as solid-state drive (SSD) with a higher-capacity hard disk drive (HDD). The intent is adding some of the speed of SSDs to the cost-effective storage capacity of traditional HDDs. The purpose of the SSD in a hybrid drive is to act as a cache for the data stored on the HDD, improving the overall performance by keeping copies of the most frequently used data on the faster SSD drive.

Replication in computing involves sharing information so as to ensure consistency between redundant resources, such as software or hardware components, to improve reliability, fault-tolerance, or accessibility.

A NetApp FAS is a computer storage product by NetApp running the ONTAP operating system; the terms ONTAP, AFF, ASA, FAS are often used as synonyms. "Filer" is also used as a synonym although this is not an official name. There are three types of FAS systems: Hybrid, All-Flash, and All SAN Array:

  1. NetApp proprietary custom-build hardware appliances with HDD or SSD drives called hybrid Fabric-Attached Storage
  2. NetApp proprietary custom-build hardware appliances with only SSD drives and optimized ONTAP for low latency called ALL-Flash FAS
  3. All SAN Array build on top of AFF platform, and provide only SAN-based data protocol connectivity.
<span class="mw-page-title-main">Solid-state drive</span> Computer storage device with no moving parts

A solid-state drive (SSD) is a type of solid-state storage device that uses integrated circuits to store data persistently. It is sometimes called semiconductor storage device, solid-state device, and solid-state disk.

<span class="mw-page-title-main">Read-only memory</span> Electronic memory that cannot be changed

Read-only memory (ROM) is a type of non-volatile memory used in computers and other electronic devices. Data stored in ROM cannot be electronically modified after the manufacture of the memory device. Read-only memory is useful for storing software that is rarely changed during the life of the system, also known as firmware. Software applications, such as video games, for programmable devices can be distributed as plug-in cartridges containing ROM.

<span class="mw-page-title-main">Random-access memory</span> Form of computer data storage

Random-access memory is a form of electronic computer memory that can be read and changed in any order, typically used to store working data and machine code. A random-access memory device allows data items to be read or written in almost the same amount of time irrespective of the physical location of data inside the memory, in contrast with other direct-access data storage media, where the time required to read and write data items varies significantly depending on their physical locations on the recording medium, due to mechanical limitations such as media rotation speeds and arm movement.

DataBlitz is a general purpose main memory database management system, developed by Lucent Bell Labs Research from 1993 to 1995. It replaced various home-grown database products used throughout Lucent beginning in 1997.

This glossary of computer hardware terms is a list of definitions of terms and concepts related to computer hardware, i.e. the physical and structural components of computers, architectural issues, and peripheral devices.

The term is used for two different things:

  1. In computer science, in-memory processing (PIM) is a computer architecture in which data operations are available directly on the data memory, rather than having to be transferred to CPU registers first. This may improve the power usage and performance of moving data between the processor and the main memory.
  2. In software engineering, in-memory processing is a software architecture where a database is kept entirely in random-access memory (RAM) or flash memory so that usual accesses, in particular read or query operations, do not require access to disk storage. This may allow faster data operations such as "joins", and faster reporting and decision-making in business.
<span class="mw-page-title-main">SingleStore</span> Database management system

SingleStore is a proprietary, cloud-native database designed for data-intensive applications. A distributed, relational, SQL database management system (RDBMS) that features ANSI SQL support, it is known for speed in data ingest, transaction processing, and query processing.

A NVDIMM or non-volatile DIMM is a type of persistent random-access memory for computers using widely used DIMM form-factors. Non-volatile memory is memory that retains its contents even when electrical power is removed, for example from an unexpected power loss, system crash, or normal shutdown. Properly used, NVDIMMs can improve application performance and system crash recovery time. They enhance solid-state drive (SSD) endurance and reliability.

Solid-state storage (SSS) is non-volatile computer storage that has no moving parts; it uses only electronic circuits. This solid-state design dramatically differs from the commonly-used competing technology of electromechanical magnetic storage which uses moving media coated with magnetic material. Generally, SSS is much faster but more expensive for the same amount of storage.

<span class="mw-page-title-main">Apache Ignite</span>

Apache Ignite is a distributed database management system for high-performance computing.

References

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.