NoSQL

Last updated

NoSQL (originally referring to "non-SQL" or "non-relational") [1] is an approach to database design that focuses on providing a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases. Instead of the typical tabular structure of a relational database, NoSQL databases house data within one data structure. Since this non-relational database design does not require a schema, it offers rapid scalability to manage large and typically unstructured data sets. [2] NoSQL systems are also sometimes called "Not only SQL" to emphasize that they may support SQL-like query languages or sit alongside SQL databases in polyglot-persistent architectures. [3] [4]

Contents

Non-relational databases have existed since the late 1960s, but the name "NoSQL" was only coined in the early 2000s, [5] triggered by the needs of Web 2.0 companies. [6] [7] NoSQL databases are increasingly used in big data and real-time web applications. [8]

Motivations for this approach include simplicity of design, simpler "horizontal" scaling to clusters of machines (which is a problem for relational databases), [5] finer control over availability, and limiting the object-relational impedance mismatch. [9] The data structures used by NoSQL databases (e.g. key–value pair, wide column, graph, or document) are different from those used by default in relational databases, making some operations faster in NoSQL. The particular suitability of a given NoSQL database depends on the problem it must solve. Sometimes the data structures used by NoSQL databases are also viewed as "more flexible" than relational database tables. [10]

Many NoSQL stores compromise consistency (in the sense of the CAP theorem) in favor of availability, partition tolerance, and speed. Barriers to the greater adoption of NoSQL stores include the use of low-level query languages (instead of SQL, for instance), lack of ability to perform ad hoc joins across tables, lack of standardized interfaces, and huge previous investments in existing relational databases. [11] Most NoSQL stores lack true ACID transactions, although a few databases have made them central to their designs.

Instead, most NoSQL databases offer a concept of "eventual consistency", in which database changes are propagated to all nodes "eventually" (typically within milliseconds), so queries for data might not return updated data immediately or might result in reading data that is not accurate, a problem known as stale read. [12] Additionally, some NoSQL systems may exhibit lost writes and other forms of data loss. [13] Some NoSQL systems provide concepts such as write-ahead logging to avoid data loss. [14] For distributed transaction processing across multiple databases, data consistency is an even bigger challenge that is difficult for both NoSQL and relational databases. Relational databases "do not allow referential integrity constraints to span databases". [15] Few systems maintain both ACID transactions and X/Open XA standards for distributed transaction processing. [16] Interactive relational databases share conformational relay analysis techniques as a common feature. [17] Limitations within the interface environment are overcome using semantic virtualization protocols, such that NoSQL services are accessible to most operating systems. [18]

History

The term NoSQL was used by Carlo Strozzi in 1998 to name his lightweight Strozzi NoSQL open-source relational database that did not expose the standard Structured Query Language (SQL) interface, but was still relational. [19] His NoSQL RDBMS is distinct from the around-2009 general concept of NoSQL databases. Strozzi suggests that, because the current NoSQL movement "departs from the relational model altogether, it should therefore have been called more appropriately 'NoREL'", [20] referring to "not relational".

Johan Oskarsson, then a developer at Last.fm, reintroduced the term NoSQL in early 2009 when he organized an event to discuss "open-source distributed, non-relational databases". [21] The name attempted to label the emergence of an increasing number of non-relational, distributed data stores, including open source clones of Google's Bigtable/MapReduce and Amazon's DynamoDB.

Types and examples

There are various ways to classify NoSQL databases, with different categories and subcategories, some of which overlap. What follows is a non-exhaustive classification by data model, with examples: [22]

TypeNotable examples of this type
Key–value cache Apache Ignite, Couchbase, Coherence, eXtreme Scale, Hazelcast, Infinispan, Memcached, Redis, Velocity
Key–value store Azure Cosmos DB, ArangoDB, Amazon DynamoDB, Aerospike, Couchbase, ScyllaDB
Key–value store (eventually consistent) Azure Cosmos DB, Oracle NoSQL Database, Riak, Voldemort
Key–value store (ordered) FoundationDB, InfinityDB, LMDB, MemcacheDB
Tuple store Apache River, GigaSpaces, Tarantool, TIBCO ActiveSpaces, OpenLink Virtuoso
Triplestore AllegroGraph, MarkLogic, Ontotext-OWLIM, Oracle NoSQL database, Profium Sense, Virtuoso Universal Server
Object database Objectivity/DB, Perst, ZODB, db4o, GemStone/S, InterSystems Caché, JADE, ObjectDatabase++, ObjectDB, ObjectStore, ODABA, Realm, OpenLink Virtuoso, Versant Object Database
Document store Azure Cosmos DB, ArangoDB, BaseX, Clusterpoint, Couchbase, CouchDB, DocumentDB, eXist-db, Google Cloud Firestore, IBM Domino, MarkLogic, MongoDB, RavenDB, Qizx, RethinkDB, Elasticsearch, OrientDB
Wide-column store Azure Cosmos DB, Amazon DynamoDB, Bigtable, Cassandra, Google Cloud Datastore, HBase, Hypertable, ScyllaDB
Native multi-model database ArangoDB, Azure Cosmos DB, OrientDB, MarkLogic, Apache Ignite, [23] [24] Couchbase, FoundationDB, Oracle Database
Graph database Azure Cosmos DB, AllegroGraph, ArangoDB, InfiniteGraph, Apache Giraph, MarkLogic, Neo4J, OrientDB, Virtuoso
Multivalue database D3 Pick database, Extensible Storage Engine (ESE/NT), InfinityDB, InterSystems Caché, jBASE Pick database, mvBase Rocket Software, mvEnterprise Rocket Software, Northgate Information Solutions Reality (the original Pick/MV Database), OpenQM, Revelation Software's OpenInsight (Windows) and Advanced Revelation (DOS), UniData Rocket U2, UniVerse Rocket U2

Key–value store

Key–value (KV) stores use the associative array (also called a map or dictionary) as their fundamental data model. In this model, data is represented as a collection of key–value pairs, such that each possible key appears at most once in the collection. [25] [26]

The key–value model is one of the simplest non-trivial data models, and richer data models are often implemented as an extension of it. The key–value model can be extended to a discretely ordered model that maintains keys in lexicographic order. This extension is computationally powerful, in that it can efficiently retrieve selective key ranges. [27]

Key–value stores can use consistency models ranging from eventual consistency to serializability. Some databases support ordering of keys. There are various hardware implementations, and some users store data in memory (RAM), while others on solid-state drives (SSD) or rotating disks (aka hard disk drive (HDD)).

Document store

The central concept of a document store is that of a "document". While the details of this definition differ among document-oriented databases, they all assume that documents encapsulate and encode data (or information) in some standard formats or encodings. Encodings in use include XML, YAML, and JSON and binary forms like BSON. Documents are addressed in the database via a unique key that represents that document. Another defining characteristic of a document-oriented database is an API or query language to retrieve documents based on their contents.

Different implementations offer different ways of organizing and/or grouping documents:

Compared to relational databases, collections could be considered analogous to tables and documents analogous to records. But they are different – every record in a table has the same sequence of fields, while documents in a collection may have fields that are completely different.

Graph

Graph databases are designed for data whose relations are well represented as a graph consisting of elements connected by a finite number of relations. Examples of data include social relations, public transport links, road maps, network topologies, etc.

Graph databases and their query language
NameLanguage(s)Notes
AllegroGraph SPARQL RDF triple store
Amazon Neptune Gremlin, SPARQL Graph database
ArangoDB AQL, JavaScript, GraphQL Multi-model DBMS Document, Graph database and Key-value store
Azure Cosmos DB Gremlin Graph database
DEX/Sparksee C++, Java, C#, Python Graph database
FlockDB Scala Graph database
IBM Db2 SPARQL RDF triple store added in DB2 10
InfiniteGraph Java Graph database
JanusGraph Java Graph database
MarkLogic Java, JavaScript, SPARQL, XQuery Multi-model document database and RDF triple store
Neo4j Cypher Graph database
OpenLink Virtuoso C++, C#, Java, SPARQL Middleware and database engine hybrid
Oracle SPARQL 1.1 RDF triple store added in 11g
OrientDB Java, SQLMulti-model document and graph database
OWLIM Java, SPARQL 1.1 RDF triple store
Profium Sense Java, SPARQL RDF triple store
RedisGraph Cypher Graph database
Sqrrl Enterprise Java Graph database
TerminusDB JavaScript, Python, datalog Open source RDF triple-store and document store [28]

Performance

The performance of NoSQL databases is usually evaluated using the metric of throughput, which is measured as operations/second. Performance evaluation must pay attention to the right benchmarks such as production configurations, parameters of the databases, anticipated data volume, and concurrent user workloads.

Ben Scofield rated different categories of NoSQL databases as follows: [29]

Data modelPerformanceScalabilityFlexibilityComplexityData IntegrityFunctionality
Key–value storehighhighhighnonelowvariable (none)
Column-oriented storehighhighmoderatelowlowminimal
Document-oriented storehighvariable (high)highlowlowvariable (low)
Graph databasevariablevariablehighhighlow-med graph theory
Relational databasevariablevariablelowmoderatehigh relational algebra

Performance and scalability comparisons are most commonly done using the YCSB benchmark.

Handling relational data

Since most NoSQL databases lack ability for joins in queries, the database schema generally needs to be designed differently. There are three main techniques for handling relational data in a NoSQL database. (See table Join and ACID Support for NoSQL databases that support joins.)

Multiple queries

Instead of retrieving all the data with one query, it is common to do several queries to get the desired data. NoSQL queries are often faster than traditional SQL queries so the cost of additional queries may be acceptable. If an excessive number of queries would be necessary, one of the other two approaches is more appropriate.

Caching, replication and non-normalized data

Instead of only storing foreign keys, it is common to store actual foreign values along with the model's data. For example, each blog comment might include the username in addition to a user id, thus providing easy access to the username without requiring another lookup. When a username changes however, this will now need to be changed in many places in the database. Thus this approach works better when reads are much more common than writes. [30]

Nesting data

With document databases like MongoDB it is common to put more data in a smaller number of collections. For example, in a blogging application, one might choose to store comments within the blog post document so that with a single retrieval one gets all the comments. Thus in this approach a single document contains all the data you need for a specific task.

ACID and join support

A database is marked as supporting ACID properties (Atomicity, Consistency, Isolation, Durability) or join operations if the documentation for the database makes that claim. However, this doesn't necessarily mean that the capability is fully supported in a manner similar to most SQL databases.

DatabaseACIDJoins
Aerospike YesNo
Apache Ignite YesYes
ArangoDB YesYes
Amazon DynamoDB YesNo
Couchbase YesYes
CouchDB YesYes
IBM Db2 YesYes
InfinityDB YesNo
LMDB YesNo
MarkLogic YesYes [nb 1]
MongoDB YesYes [nb 2]
OrientDB YesYes [nb 3]
  1. Joins do not necessarily apply to document databases, but MarkLogic can do joins using semantics. [31]
  2. MongoDB did not support joining from a sharded collection until version 5.1. [32]
  3. OrientDB can resolve 1:1 joins using links by storing direct links to foreign records. [33]

See also

Related Research Articles

<span class="mw-page-title-main">Database</span> Organized collection of data in computing

In computing, a database is an organized collection of data or a type of data store based on the use of a database management system (DBMS), the software that interacts with end users, applications, and the database itself to capture and analyze the data. The DBMS additionally encompasses the core facilities provided to administer the database. The sum total of the database, the DBMS and the associated applications can be referred to as a database system. Often the term "database" is also used loosely to refer to any of the DBMS, the database system or an application associated with the database.

A relational database is a database based on the relational model of data, as proposed by E. F. Codd in 1970. A database management system used to maintain relational databases is a relational database management system (RDBMS). Many relational database systems are equipped with the option of using SQL for querying and updating the database.

Object–relational impedance mismatch creates difficulties going from data in relational data stores to usage in domain-driven object models. Object-orientation (OO) is the default method for business-centric design in programming languages. The problem lies in neither relational nor OO, but in the conceptual difficulty mapping between the two logic models. Both are logical models implementable differently on database servers, programming languages, design patterns, or other technologies. Issues range from application to enterprise scale, whenever stored relational data is used in domain-driven object models, and vice versa. Object-oriented data stores can trade this problem for other implementation difficulties.

SPARQL is an RDF query language—that is, a semantic query language for databases—able to retrieve and manipulate data stored in Resource Description Framework (RDF) format. It was made a standard by the RDF Data Access Working Group (DAWG) of the World Wide Web Consortium, and is recognized as one of the key technologies of the semantic web. On 15 January 2008, SPARQL 1.0 was acknowledged by W3C as an official recommendation, and SPARQL 1.1 in March, 2013.

An entity–attribute–value model (EAV) is a data model optimized for the space-efficient storage of sparse—or ad-hoc—property or data values, intended for situations where runtime usage patterns are arbitrary, subject to user variation, or otherwise unforeseeable using a fixed design. The use-case targets applications which offer a large or rich system of defined property types, which are in turn appropriate to a wide set of entities, but where typically only a small, specific selection of these are instantiated for a given entity. Therefore, this type of data model relates to the mathematical notion of a sparse matrix. EAV is also known as object–attribute–value model, vertical database model, and open schema.

<span class="mw-page-title-main">Database model</span> Type of data model

A database model is a type of data model that determines the logical structure of a database. It fundamentally determines in which manner data can be stored, organized and manipulated. The most popular example of a database model is the relational model, which uses a table-based format.

A document-oriented database, or document store, is a computer program and data storage system designed for storing, retrieving and managing document-oriented information, also known as semi-structured data.

<span class="mw-page-title-main">MarkLogic Server</span>

MarkLogic Server is a document-oriented database developed by MarkLogic. It is a NoSQL multi-model database that evolved from an XML database to natively store JSON documents and RDF triples, the data model for semantics. MarkLogic is designed to be a data hub for operational and analytical data.

A graph database (GDB) is a database that uses graph structures for semantic queries with nodes, edges, and properties to represent and store data. A key concept of the system is the graph. The graph relates the data items in the store to a collection of nodes and edges, the edges representing the relationships between the nodes. The relationships allow data in the store to be linked together directly and, in many cases, retrieved with one operation. Graph databases hold the relationships between data as a priority. Querying relationships is fast because they are perpetually stored in the database. Relationships can be intuitively visualized using graph databases, making them useful for heavily inter-connected data.

<span class="mw-page-title-main">Couchbase Server</span> Open-source NoSQL database

Couchbase Server, originally known as Membase, is a source-available, distributed multi-model NoSQL document-oriented database software package optimized for interactive applications. These applications may serve many concurrent users by creating, storing, retrieving, aggregating, manipulating and presenting data. In support of these kinds of application needs, Couchbase Server is designed to provide easy-to-scale key-value, or JSON document access, with low latency and high sustainability throughput. It is designed to be clustered from a single machine to very large-scale deployments spanning many machines.

A cloud database is a database that typically runs on a cloud computing platform and access to the database is provided as-a-service. There are two common deployment models: users can run databases on the cloud independently, using a virtual machine image, or they can purchase access to a database service, maintained by a cloud database provider. Of the databases available on the cloud, some are SQL-based and some use a NoSQL data model.

<span class="mw-page-title-main">Amazon DynamoDB</span> NoSQL database service

Amazon DynamoDB is a fully managed proprietary NoSQL database offered by Amazon.com as part of the Amazon Web Services portfolio. DynamoDB offers a fast persistent key–value datastore with built-in support for replication, autoscaling, encryption at rest, and on-demand backup among other features.

The following is provided as an overview of and topical guide to databases:

<span class="mw-page-title-main">Apache Drill</span> Open-source software framework

Apache Drill is an open-source software framework that supports data-intensive distributed applications for interactive analysis of large-scale datasets. Built chiefly by contributions from developers from MapR, Drill is inspired by Google's Dremel system. Drill is an Apache top-level project. Tom Shiran is the founder of the Apache Drill Project. It was designated an Apache Software Foundation top-level project in December 2016.

<span class="mw-page-title-main">Oracle NoSQL Database</span> Distributed database

Oracle NoSQL Database is a NoSQL-type distributed key-value database from Oracle Corporation. It provides transactional semantics for data manipulation, horizontal scalability, and simple administration and monitoring.

In the field of database design, a multi-model database is a database management system designed to support multiple data models against a single, integrated backend. In contrast, most database management systems are organized around a single data model that determines how data can be organized, stored, and manipulated. Document, graph, relational, and key–value models are examples of data models that may be supported by a multi-model database.

Azure Cosmos DB is a globally distributed, multi-model database service and offered by Microsoft. It is designed to provide high availability, scalability, and low-latency access to data for modern applications. Unlike traditional relational databases, Cosmos DB is a NoSQL database, which means it can handle unstructured and semi-structured, in addition to structured, data types.

<span class="mw-page-title-main">RocksDB</span> Embedded key-value database

RocksDB is a high performance embedded database for key-value data. It is a fork of Google's LevelDB optimized to exploit multi-core processors (CPUs), and make efficient use of fast storage, such as solid-state drives (SSD), for input/output (I/O) bound workloads. It is based on a log-structured merge-tree data structure. It is written in C++ and provides official language bindings for C++, C, and Java. Many third-party language bindings exist. RocksDB is free and open-source software, released originally under a BSD 3-clause license. However, in July 2017 the project was migrated to a dual license of both Apache 2.0 and GPLv2 license. This change helped its adoption in Apache Software Foundation's projects after blacklist of the previous BSD+Patents license clause.

Amazon DocumentDB is a managed proprietary NoSQL database service that supports document data structures, with some compatibility with MongoDB version 3.6 and version 4.0. As a document database, Amazon DocumentDB can store, query, and index JSON data. It is available on Amazon Web Services. As of March 2023, AWS introduced some compliance with MongoDB 5.0 but lacks time series collection support.

An Ordered Key-Value Store (OKVS) is a type of data storage paradigm that can support multi-model database. An OKVS is an ordered mapping of bytes to bytes. It is a more powerful paradigm than Key-Value Store because OKVS allow to build higher level abstractions without the need to do full scans. An OKVS will keep the key-value pairs sorted by the key lexicographic order. OKVS systems provides different set of features and performance trade-offs. Most of them are shipped as a library without network interfaces, in order to be embedded in another process. Most OKVS support ACID guarantees. Some OKVS are distributed databases. Ordered Key-Value Store found their way into many modern database systems including NewSQL database systems.

References

  1. http://nosql-database.org/ "NoSQL DEFINITION: Next Generation Databases mostly addressing some of the points : being non-relational, distributed, open-source and horizontally scalable".
  2. https://www.ibm.com/topics/nosql-databases "What is a NoSQL database?"
  3. "NoSQL (Not Only SQL)". NoSQL database, also called Not Only SQL
  4. Fowler, Martin. "NosqlDefinition". many advocates of NoSQL say that it does not mean a "no" to SQL, rather it means Not Only SQL
  5. 1 2 Leavitt, Neal (2010). "Will NoSQL Databases Live Up to Their Promise?" (PDF). IEEE Computer . 43 (2): 12–14. doi:10.1109/MC.2010.58. S2CID   26876882.
  6. Mohan, C. (2013). History Repeats Itself: Sensible and NonsenSQL Aspects of the NoSQL Hoopla (PDF). Proc. 16th Int'l Conf. on Extending Database Technology.
  7. "Amazon Goes Back to the Future With 'NoSQL' Database". WIRED. 19 January 2012. Retrieved 6 March 2017.
  8. "RDBMS dominate the database market, but NoSQL systems are catching up". DB-Engines.com. 21 November 2013. Retrieved 24 November 2013.
  9. NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence. Addison-Wesley Educational Publishers Inc, 2009, ISBN   978-0321826626.
  10. Vogels, Werner (18 January 2012). "Amazon DynamoDB – a Fast and Scalable NoSQL Database Service Designed for Internet Scale Applications". All Things Distributed. Retrieved 6 March 2017.
  11. Grolinger, K.; Higashino, W. A.; Tiwari, A.; Capretz, M. A. M. (2013). "Data management in cloud environments: NoSQL and NewSQL data stores" (PDF). Aira, Springer. Retrieved 8 January 2014.
  12. "Jepsen: MongoDB stale reads". Aphyr.com. 20 April 2015. Retrieved 6 March 2017.
  13. "Large volume data analysis on the Typesafe Reactive Platform". Slideshare.net. 11 June 2015. Retrieved 6 March 2017.
  14. Fowler, Adam. "10 NoSQL Misconceptions". Dummies.com. Retrieved 6 March 2017.
  15. "No! to SQL and No! to NoSQL | So Many Oracle Manuals, So Little Time". Iggyfernandez.wordpress.com. 29 July 2013. Retrieved 6 March 2017.
  16. Chapple, Mike. "The ACID Model". about.com. Archived from the original on 29 December 2016. Retrieved 26 September 2012.
  17. Fiore, S. (2011). Grid and cloud database management. Springer Science & Business Media. p. 210.
  18. Lawrence, Integration and virtualization of relational SQL and NoSQL systems including MySQL and MongoDB (2014). "Integration and virtualization of relational SQL and NoSQL systems including MySQL and MongoDB". International Conference on Computational Science and Computational Intelligence 1.
  19. Lith, Adam; Mattson, Jakob (2010). "Investigating storage solutions for large data: A comparison of well performing and scalable data storage solutions for real time extraction and batch insertion of data" (PDF). Göteborg: Department of Computer Science and Engineering, Chalmers University of Technology. p. 70. Retrieved 12 May 2011. Carlo Strozzi first used the term NoSQL in 1998 as a name for his open source relational database that did not offer a SQL interface[...]
  20. "NoSQL Relational Database Management System: Home Page". Strozzi.it. 2 October 2007. Retrieved 29 March 2010.
  21. "NoSQL 2009". Blog.sym-link.com. 12 May 2009. Archived from the original on 16 July 2011. Retrieved 29 March 2010.
  22. Strauch, Christof. "NoSQL Databases" (PDF). pp. 23–24. Retrieved 27 August 2017.
  23. https://apacheignite.readme.io/docs Ignite Documentation
  24. https://www.infoworld.com/article/3135070/data-center/fire-up-big-data-processing-with-apache-ignite.html fire-up-big-data-processing-with-apache-ignite
  25. Sandy (14 January 2011). "Key Value stores and the NoSQL movement". Stackexchange. Retrieved 1 January 2012. Key–value stores allow the application developer to store schema-less data. This data usually consists of a string that represents the key, and the actual data that is considered the value in the "key–value" relationship. The data itself is usually some kind of primitive of the programming language (a string, an integer, or an array) or an object that is being marshaled by the programming language's bindings to the key-value store. This structure replaces the need for a fixed data model and allows proper formatting.
  26. Seeger, Marc (21 September 2009). "Key-Value Stores: a practical overview" (PDF). Marc Seeger. Retrieved 1 January 2012. Key–value stores provide a high-performance alternative to relational database systems with respect to storing and accessing data. This paper provides a short overview of some of the currently available key–value stores and their interface to the Ruby programming language.
  27. Katsov, Ilya (1 March 2012). "NoSQL Data Modeling Techniques". Ilya Katsov. Retrieved 8 May 2014.
  28. "TerminusDB an open-source in-memory document graph database". terminusdb.com. Retrieved 16 December 2021.
  29. Scofield, Ben (14 January 2010). "NoSQL - Death to Relational Databases(?)" . Retrieved 26 June 2014.
  30. "Moving From Relational to NoSQL: How to Get Started". Couchbase.com. Retrieved 11 November 2019.
  31. "Can't do joins with MarkLogic? It's just a matter of Semantics! - General Networks". Gennet.com. Archived from the original on 3 March 2017. Retrieved 6 March 2017.
  32. "Sharded Collection Restrictions". docs.mongodb.com. Retrieved 24 January 2020.
  33. "SQL Reference · OrientDB Manual". OrientDB.com. Retrieved 24 January 2020.

Further reading