YTsaurus is a distributed storage and processing platform for big data with support for the MapReduce model.
Some days ago Yandex unveiled through one announced the opening of the source code of the YTsauru platform, which is used for distributed storage and processing of large amounts of data, which supports data manipulation using the MapReduce paradigm, SQL query engine, distributed file system and NoSQL storage in key-value format.
YTsaurus used on the Yandex infrastructure to efficiently use the computing power of supercomputers of the company The platform can scale to clusters of more than 10 nodes, covering up to a million processors and thousands of GPUs (for machine learning tasks).
Isolated containers running on physical servers can be used as cluster units. Storage can contain exabytes of data located on various media such as hard drives, SSDs, NVME, and RAM.
The cluster supports dynamic addition and removal of nodes, redundancy (no single point of failure), automatic replication, active cluster software upgrades, and automatic redundancy recovery in case of node failure.
Three types of clusters are supported: compute clusters (for massively parallel processing of big data using MapReduce operations), clusters for pivot tables and key-value storage, and geographically distributed clusters.
A platform-based service can provide the means to store and process data for tens of thousands of users. Typical YTsaurus applications in Yandex include storing information about ad network users, training machine learning models, forming a search index, and building a data warehouse for services like Yandex Taxi. , Food, Lavka and deliveries
Of the basic use cases it is mentioned:
- Batch processing: MapReduce and SPYT (Apache Spark as a computing engine on data in YTsaurus) for the processing of structured and semi-structured data: records or financial transactions.
- Ad hoc analysis: Fast queries via CHYT (a group of ClickHouse servers on YTsaurus compute nodes) without copying data to a separate analytics system. ODBC and JDBC with the ability to connect BI for visualization.
- OLTP Tasks: Real-time transactional work with key-value storage: for example, user profile storage, ad display, or stream processing.
- Machine learning: Manage GPU clusters to train models with billions of parameters.
- Meta information storage: Transactional storage of metainformation and reliable service of distributed locks.
- Creation of data warehouses and ETL for multi-tier data processing using typical tools: Apache Spark, SQL, MapReduce.
While on the part of the main elements of architecture, the following are mentioned:
- Distributed file system and Cypress fault-tolerant tree-based metainformation storage.
- Scheduler for distributed computing with support for the MapReduce model, as well as advanced basic operations.
- Horizontal scalability of IT operations.
- Isolation of computing resources and possibility to allocate certain computing resources (CPU, GPU, RAM) in different proportions.
- PivotTables for creating OLTP storage, support for MVCC-based storage, transactions, the ability to delete data after expiration, and message queues for streaming data processing over PivotTables.
- API and libraries for programming languages ​​C++, Python, Java, Go.
- A web interface for users and administrators that supports navigation through a tree-like storage.
Finally if you are interested in knowing more about it, you should know that the project's code is written in C/C++ and is open under the Apache 2.0 license. You can check the details in the following link
El GitHub repository contains the server code for YTsaurus, the deployment framework used by k8s, a web interface to the system, and client SDKs for popular programming languages ​​such as C++, Java, Go, and Python.