RocksDB

Last updated
RocksDB
Original author(s) Dhruba Borthakur
Developer(s) Meta Platforms (was Facebook, Inc.)
Initial releaseMay 2012;11 years ago (2012-05)
Stable release
8.8.1 [1]   OOjs UI icon edit-ltr-progressive.svg / 17 November 2023
Repository
Written in C++
Operating system Windows, macOS, Linux, FreeBSD, OpenBSD, Solaris, AIX
Platform Cross-platform
Type Embedded database
License Apache 2.0 or GPL 2
Website rocksdb.org

RocksDB is a high performance [2] [3] [4] [5] [6] embedded database for key-value data. It is a fork of Google's LevelDB optimized to exploit multi-core processors (CPUs), and make efficient use of fast storage, such as solid-state drives (SSD), for input/output (I/O) bound workloads. It is based on a log-structured merge-tree (LSM tree) data structure. It is written in C++ and provides official language bindings for C++, C, and Java. Many third-party language bindings exist. RocksDB is free and open-source software, released originally under a BSD 3-clause license. [7] [8] [9] However, in July 2017 the project was migrated to a dual license of both Apache 2.0 and GPLv2 license. [10] This change helped its adoption in Apache Software Foundation's projects after blacklist of the previous BSD+Patents license clause. [11] [12]

Contents

RocksDB is used in production systems at various web-scale enterprises [13] including Facebook, Yahoo!, [14] and LinkedIn. [15]

Features

RocksDB, like LevelDB, stores keys and values in arbitrary byte arrays, and data is sorted byte-wise by key or by providing a custom comparator.

RocksDB provides all of the features of LevelDB, plus:

and others: List of RocksDB features that are not in LevelDB.

RocksDB is not an SQL database (although MyRocks combines RocksDB with MySQL). Like other NoSQL and dbm stores, it has no relational data model, and it does not support SQL queries. Also, it has no direct support for secondary indexes, however a user may build their own internally using Column Families or externally. Applications use RocksDB as a library, as it provides no server or command-line interface.

History

RocksDB was created at Facebook by Dhruba Borthakur [26] [27] in April 2012, as a fork of LevelDB with the initial stated goal of improving performance for server workloads. [28] [29]

Integration

As an embeddable database, RocksDB can be used as a storage engine within a larger database management system (DBMS). For example, Rockset uses RocksDB [30] mostly for analytical data processing.

Alternative backend

The following projects have been started to replace or offer alternative storage engines for already-established database systems with RocksDB:

ArangoDB

ArangoDB has added RocksDB to its previous storage engine ("mmfiles"). [31] Starting with ArangoDB 3.4, RocksDB will be the default storage engine in ArangoDB. [32]

Cassandra

Cassandra on RocksDB can improve the performance of Apache Cassandra significantly (3-4 times faster in general, 100 times faster in some use-cases).[ citation needed ] The Instagram team at Facebook developed and open-sourced their code, along with benchmarks of their performance results. [33]

MariaDB

MariaDB can use the MyRocks storage engine (which is forked from RocksDB) since MariaDB 10.2.5 (Alpha status) [34] and stable since MariaDB 10.2.16 in 2018. [35]

MongoDB

The MongoRocks project provides a storage module for MongoDB where the storage engine is RocksDB. [36] [37] [38]

A related program is Rocks Strata, a tool written in Go, which allows managing incremental backups of MongoDB when RocksDB is used as the storage engine. [39]

MySQL

The MyRocks project created a new RocksDB-based storage engine for MySQL. [40] [41] In-depth details about MyRocks were presented at Percona Live 2016. [42]

Oxigraph

Oxigraph is a graph database implementing the SPARQL standard, based on RocksDB

UKV

The UKV [43] project allows users to use RocksDB on par with LevelDB as the underlying key-value store. It represents a shared abstraction for create, read, update and delete (CRUD) operations common to every storage engine. It augments it with structured bindings for several high-level languages, including Python, Java, and Go.

Embedded

The following database systems and applications have chosen to use RocksDB as their embedded storage engine:

Ceph's BlueStore

The Ceph's BlueStore storage layer uses RocksDB for metadata management in OSD devices. [44]

Apache Flink uses RocksDB to store checkpoints. [45]

FusionDB

FusionDB [46] uses RocksDB as its storage engine for XML, Key/Value, and JSON. [47]

LogDevice LogsDB

LogDevice's LogsDB is built atop RocksDB. [48]

Manhattan

The Manhattan Distributed Key-Value Store has used RocksDB as its primary engine to store Twitter data since 2018. [49]

Rockset

The Rockset service that is used for operational data analytics uses RocksDB as its storage engine. [50]

SSDB

The ssdb-rocks [51] project uses RocksDB as the storage engine for the SSDB [52] NoSQL Database.

TiDB

The TiDB [53] project uses RocksDB as its storage engine. [54]

YugabyteDB

The YugabyteDB [55] database uses a modified version of RocksDB as part of its DocDB storage engine

Third-party language bindings

Third-party programming language bindings available for RocksDB include:

Related Research Articles

The following tables compare general and technical information for a number of relational database management systems. Please see the individual products' articles for further information. Unless otherwise specified in footnotes, comparisons are based on the stable versions without any add-ons, extensions or external programs.

A time series database is a software system that is optimized for storing and serving time series through associated pairs of time(s) and value(s). In some fields, time series may be called profiles, curves, traces or trends. Several early time series databases are associated with industrial applications which could efficiently store measured values from sensory equipment, but now are used in support of a much wider range of applications.

A document-oriented database, or document store, is a computer program and data storage system designed for storing, retrieving and managing document-oriented information, also known as semi-structured data.

An embedded database system is a database management system (DBMS) which is deeply integrated with an application software; it is built into the software or an application. It is a broad term which includes:

HBase is an open-source non-relational distributed database modeled after Google's Bigtable and written in Java. It is developed as part of Apache Software Foundation's Apache Hadoop project and runs on top of HDFS or Alluxio, providing Bigtable-like capabilities for Hadoop. That is, it provides a fault-tolerant way of storing large quantities of sparse data.

MongoDB is a source-available cross-platform document-oriented database program. Classified as a NoSQL database program, MongoDB uses JSON-like documents with optional schemas. MongoDB is developed by MongoDB Inc. and current versions are licensed under the Server Side Public License (SSPL). MongoDB is a member of the MACH Alliance.

NoSQL is an approach to database design that focuses on providing a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases. Instead of the typical tabular structure of a relational database, NoSQL databases house data within one data structure. Since this non-relational database design does not require a schema, it offers rapid scalability to manage large and typically unstructured data sets. NoSQL systems are also sometimes called "Not only SQL" to emphasize that they may support SQL-like query languages or sit alongside SQL databases in polyglot-persistent architectures.

LevelDB is an open-source on-disk key-value store written by Google fellows Jeffrey Dean and Sanjay Ghemawat. Inspired by Bigtable, LevelDB source code is hosted on GitHub under the New BSD License and has been ported to a variety of Unix-based systems, macOS, Windows, and Android.

A cloud database is a database that typically runs on a cloud computing platform and access to the database is provided as-a-service. There are two common deployment models: users can run databases on the cloud independently, using a virtual machine image, or they can purchase access to a database service, maintained by a cloud database provider. Of the databases available on the cloud, some are SQL-based and some use a NoSQL data model.

<span class="mw-page-title-main">Apache Drill</span> Open-source software framework

Apache Drill is an open-source software framework that supports data-intensive distributed applications for interactive analysis of large-scale datasets. Built chiefly by contributions from developers from MapR, Drill is inspired by Google's Dremel system. Drill is an Apache top-level project. Tom Shiran is the founder of the Apache Drill Project. It was designated an Apache Software Foundation top-level project in December 2016.

The Yahoo! Cloud Serving Benchmark (YCSB) is an open-source specification and program suite for evaluating retrieval and maintenance capabilities of computer programs. It is often used to compare the relative performance of NoSQL database management systems.

SequoiaDB is a multi-model NewSQL database.

Presto is a distributed query engine for big data using the SQL query language. Its architecture allows users to query data sources such as Hadoop, Cassandra, Kafka, AWS S3, Alluxio, MySQL, MongoDB and Teradata, and allows use of multiple data sources within a query. Presto is community-driven open-source software released under the Apache License.

<span class="mw-page-title-main">MyRocks</span>

MyRocks is open-source software developed at Facebook in order to use MySQL features with RocksDB implementations. It is based on Oracle MySQL 5.6.

<span class="mw-page-title-main">DBeaver</span> Multi-platform database administration software

DBeaver is a SQL client software application and a database administration tool. For relational databases it uses the JDBC application programming interface (API) to interact with databases via a JDBC driver. For other databases (NoSQL) it uses proprietary database drivers. It provides an editor that supports code completion and syntax highlighting. It provides a plug-in architecture that allows users to modify much of the application's behavior to provide database-specific functionality or features that are database-independent. This is a desktop application written in Java and based on Eclipse platform.

TiDB is an open-source NewSQL database that supports Hybrid Transactional and Analytical Processing (HTAP) workloads. It is MySQL compatible and can provide horizontal scalability, strong consistency, and high availability. It is developed and supported primarily by PingCAP and licensed under Apache 2.0, though it is also available as a paid product. TiDB drew its initial design inspiration from Google's Spanner and F1 papers.

An Ordered Key-Value Store (OKVS) is a type of data storage paradigm that can support multi-model database. An OKVS is an ordered mapping of bytes to bytes. It is a more powerful paradigm than Key-Value Store because OKVS allow to build higher level abstractions without the need to do full scans. An OKVS will keep the key-value pairs sorted by the key lexicographic order. OKVS systems provides different set of features and performance trade-offs. Most of them are shipped as a library without network interfaces, in order to be embedded in another process. Most OKVS support ACID guarantees. Some OKVS are distributed databases. Ordered Key-Value Store found their way into many modern database systems including NewSQL database systems.

<span class="mw-page-title-main">Trino (SQL query engine)</span>

Trino is an open-source distributed SQL query engine designed to query large data sets distributed over one or more heterogeneous data sources. Trino can query datalakes that contain open column-oriented data file formats like ORC or Parquet residing on different storage systems like HDFS, AWS S3, Google Cloud Storage, or Azure Blob Storage using the Hive and Iceberg table formats. Trino also has the ability to run federated queries that query tables in different data sources such as MySQL, PostgreSQL, Cassandra, Kafka, MongoDB and Elasticsearch. Trino is released under the Apache License.

HammerDB is an open source database benchmarking application developed by Steve Shaw. HammerDB supports databases such as Oracle, SQL Server, Db2, MySQL and MariaDB. HammerDB is written in TCL and C, and is licensed under the GPL v3.

References

  1. ↑ Error: Unable to display the reference properly. See the documentation for details.
  2. ↑ "Performance Benchmarks". GitHub . Retrieved November 29, 2015.
  3. ↑ "Benchmarking the leveldb family". 7 July 2014. Retrieved March 10, 2016.
  4. ↑ "Comparing LevelDB and RocksDB, take 2". 27 April 2015. Retrieved March 10, 2016.
  5. ↑ "Benchmarking LevelDB vs. RocksDB vs. HyperLevelDB vs. LMDB Performance for InfluxDB". 20 June 2014. Retrieved March 10, 2016.
  6. ↑ Golan-Gueta, Guy; Bortnikov, Edward; Hillel, Eschar; Keidar, Idit (April 21, 2015). "Scaling concurrent log-structured data stores". Proceedings of the Tenth European Conference on Computer Systems. pp. 1–14. doi:10.1145/2741948.2741973. ISBN   9781450332385. S2CID   5849146.
  7. ↑ "Facebook's latest open source effort: a flash-powered database called RocksDB". 21 November 2013. Retrieved March 10, 2016.
  8. ↑ "Under the Hood: Building and open-sourcing RocksDB". Facebook . Retrieved March 10, 2016.
  9. ↑ "RocksDB - Facebook's Database Now Open Source" . Retrieved March 10, 2016.
  10. ↑ "GitHub pull request". GitHub . Retrieved July 20, 2017.
  11. ↑ "Apache says 'no' to Facebook code libraries". The Register . Retrieved July 20, 2017.
  12. ↑ "GitHub issue". GitHub . Retrieved July 20, 2017.
  13. ↑ "Users.md". GitHub . Retrieved December 1, 2015.
  14. ↑ "RocksDB on Steroids" . Retrieved March 10, 2016.
  15. ↑ "Benchmarking Apache Samza: 1.2 million messages per second on a single node" . Retrieved March 10, 2016.
  16. ↑ "RocksDB transactions". GitHub. Retrieved 2016-04-04.
  17. ↑ "How to backup RocksDB?". GitHub . Retrieved 2017-07-19.
  18. ↑ "Checkpoints". GitHub . Retrieved 2017-07-19.
  19. ↑ "Column families in RocksDB". GitHub. Retrieved 2016-04-04.
  20. ↑ "RocksDB bloom filters". GitHub. Retrieved 2016-04-04.
  21. ↑ "RocksDB TTL support". GitHub. Retrieved 2016-04-04.
  22. ↑ "Universal compaction". GitHub. Retrieved 2016-04-04.
  23. ↑ "RocksDB merge operator". GitHub. Retrieved 2016-04-04.
  24. ↑ "RocksDB perf context and IO stats context". GitHub. Retrieved 2016-04-04.
  25. ↑ "Spatial indexing in RocksDB". rocksdb.org. Retrieved 2018-07-19.
  26. ↑ "First commit where RocksDB diverges from LevelDB". GitHub . May 10, 2012. Retrieved March 15, 2016.
  27. ↑ "Rocksdb readme file". GitHub . Nov 30, 2012. Retrieved March 15, 2016.
  28. ↑ "The History of RocksDB". November 24, 2013. Retrieved March 10, 2016.
  29. ↑ Borthakur, Dhruba (November 22, 2013). "RocksDB: A High Performance Embedded Key-Value Store for Flash Storage - Data@Scale". YouTube . Retrieved March 10, 2016. ... The story of why we decided to do RocksDB ...
  30. ↑ Dhoot, Sandeep (2019-06-27). "How We Use RocksDB at Rockset". rockset.com. Retrieved 2023-03-01.
  31. ↑ "Comparing new RocksDB and MMFiles storage engines".
  32. ↑ "RC1 ArangoDB 3.4 - Whats new?". 6 September 2018.
  33. ↑ "Open-sourcing a 10x reduction in Apache Cassandra tail latency". 5 March 2018.
  34. ↑ "MyRocks". MariaDB KnowledgeBase. Retrieved 2019-04-28.
  35. ↑ "MariaDB 10.2.16 Release Notes".
  36. ↑ "mongodb-partners/mongo-rocks". GitHub . 29 October 2021.
  37. ↑ "Integrating RocksDB with MongoDB" . Retrieved July 19, 2018.
  38. ↑ "MongoDB + RocksDB at Parse" . Retrieved December 1, 2015.
  39. ↑ "facebookgo/rocks-strata". GitHub . 31 October 2021.
  40. ↑ "facebook/mysql-5.6". GitHub . 2 November 2021.
  41. ↑ "MyRocks: MySQL on RocksDB" (PDF). Retrieved November 29, 2015.
  42. ↑ "MyRocks Deep Dive". 19 April 2016. Retrieved May 9, 2016.
  43. 1 2 3 4 5 "unum-cloud/ukv". GitHub . 28 December 2022.
  44. ↑ "Storage Devices -- Ceph Documentation".
  45. ↑ "Apache Flink 1.8 Documentation: State Backends". ci.apache.org. Retrieved 2019-08-11.
  46. ↑ "FusionDB". Evolved Binary.
  47. ↑ "The Design and Implementation of FusionDB" (PDF). XML Prague.
  48. ↑ "LogDevice: a distributed data store for logs". Mark Marchukov, Facebook. 31 August 2017.
  49. ↑ "Adopting RocksDB within Manhattan". Twitter . 28 December 2022.
  50. ↑ "How we use RocksDB at Rockset". rockset.com. Retrieved 2019-07-10.
  51. ↑ "ideawu/ssdb-rocks". GitHub . 21 August 2021.
  52. ↑ https://ssdb.io
  53. ↑ "pingcap/tidb". GitHub . 4 November 2021.
  54. ↑ "TiDB Internal (I) - Data Storage". Shen Li. 11 July 2017.
  55. ↑ "How We Built a High Performance Document Store on RocksDB?". 20 February 2019.
  56. ↑ "warrenfalk/rocksdb-sharp". GitHub . 28 September 2021.
  57. ↑ "chicken-rocksdb".
  58. ↑ "b1naryth1ef/rocksdb". GitHub . 22 October 2019.
  59. ↑ "urbint/rox". GitHub . September 2021.
  60. ↑ "leo-project/erocksdb". GitHub . September 2021.
  61. ↑ "barrel-db/erlang-rocksdb".
  62. ↑ "tecbot/gorocksdb". GitHub . 29 October 2021.
  63. ↑ "rocksdb-haskell: Haskell bindings to RocksDB".
  64. ↑ "RocksJava". GitHub .
  65. ↑ "rocksdb". 25 March 2022.
  66. ↑ "rocksdb". GitHub .
  67. ↑ "iabudiab/ObjectiveRocks". GitHub . 2 August 2021.
  68. ↑ "OCaml bindings for RocksDB". GitHub . 8 October 2021.
  69. ↑ "An OCaml RocksDb binding using ocaml-ctypes". GitHub . 28 September 2020.
  70. ↑ "RocksDB - Perl extension for RocksDB - metacpan.org".
  71. ↑ "Photonios/rocksdb-php". GitHub . 11 August 2021.
  72. ↑ "SWI-Prolog interface for RocksDB".
  73. ↑ "stephan-hof/pyrocksdb". GitHub . 27 October 2021.
  74. ↑ "rocksdb-ruby - RubyGems.org - your community gem host".
  75. ↑ "spacejam/rust-rocksdb". GitHub . 2 November 2021.
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.