It was recently announced that Yandex released the source code for its DBMS, "YDB" , which implements support for the SQL dialect and ACID transactions.
The DBMS was built from the ground up and initially developed with the goal of ensuring fault tolerance , automatic recovery from failures, and scalability. It's worth noting that Yandex launched operational YDB clusters, including more than 10 nodes, storing hundreds of petabytes of data and handling millions of distributed transactions per second.
Main novelties of YDB
One of YDB's key features is its use of a relational data model with tables. YQL ( YDB Query Language) is used to query and define the data schema; it's a dialect of SQL adapted for working with large, distributed databases . When creating a storage schema, a tree-like grouping of tables is supported, similar to the directories in a file system. An API is provided for working with data in JSON format.
Also noteworthy is its ability to create fault-tolerant configurations that continue to function when disks, nodes, racks, and even individual data centers fail. YDB supports synchronous deployment and replication across three availability zones while maintaining cluster health in the event of a zone failure.
Compatibility with data access via scan queries, designed to perform ad-hoc analytical queries on the database, executed in read-only mode and returning a grpc flow.
Furthermore, it also stands out for storing data directly on block devices using the native PDisk component and the VDisk layer. In addition to VDisk, DSProxy runs, which analyzes disk availability and performance to exclude them if problems are detected.
Other notable features include:
- A flexible architecture that allows you to build various services on top of YDB, right down to virtual block devices and persistent queues. Suitability for different types of workload: OLTP and OLAP (analytical queries).
- Support for multi-user (multi-tenant) and serverless configurations.
- Ability to authenticate clients. Users can create their own virtual clusters and databases on a common shared infrastructure, considering resource consumption in terms of number of requests and data size, or by renting/reserving certain computing resources and storage space.
- Possibility to adjust the useful life of the records for the automatic deletion of obsolete data.
- Interacting with the DBMS and submitting requests is done using the command line interface, the integrated web interface, or the YDB SDK, which provides libraries for C++, C# (.NET), Go, Java, Node.js, PHP and Python.
- Automatically recover from failures with minimal delay to applications and automatically maintain specified redundancy when storing data.
- Automatic creation of indexes on the primary key and the ability to define secondary indexes to improve the efficiency of arbitrary column access.
- Horizontal scalability. As the load and size of data stored grows, the cluster can be expanded simply by connecting new nodes. Compute and storage tiers are separate, allowing you to scale compute and storage separately. The DBMS itself monitors the even distribution of data and load, taking into account the available hardware resources. It is possible to implement geographically distributed configurations that cover multiple data centers in different parts of the world.
- Support for a strong consistency model and ACID transactions when processing queries that span multiple nodes and tables. To improve performance, you can selectively disable consistency checking.
- Automatic data replication, automatic partitioning (partitioning, sharding) when size or load increases, and automatic load and data balancing between nodes.
Finally, it is worth noting that YDB is used in Yandex projects, the code is written in C/C++ and is distributed under the Apache 2.0 license. You can find the source code, as well as more details about it, at the following link.