Micron Technology (a company specializing in the production of DRAM and flash memory) announced the introduction of a new engine called "HSE" (Heterogeneous-memory Storage Engine), which was developed taking into account the specific characteristics of use in SSD units based on NAND flash (X100, TLC, QLC 3D NAND) or read-only memory (NVDIMM).
The engine is designed as a library for embedding in other applications and supports key-value data processing . The HSE code is written in C and distributed under the Apache 2.0 license.
Among the applications of the engine are low-level data storage in NoSQL DBMS, software-defined storage (SDS) such as Ceph and Scality RING, platforms for processing large amounts of data (Big Data), high-performance computing (HPC) systems, Internet of Things (IoT) devices, and solutions for machine learning systems.
HSE is optimized not only for maximum performance but also to ensure the durability of various SSD classes. High speed is achieved through a hybrid storage model: the most relevant data is stored in the cache, reducing the number of disk accesses.
As an example of integrating the new engine into third-party projects, a document-oriented version of the MongoDB DBMS was prepared, which was translated to use HSE.
Technologically, HSE is based on an additional core module, mpool , which implements a specialized interface for storing objects for solid-state drives, taking into account their capabilities and characteristics, thus enabling fundamentally different speed and durability features. Mpool is also an open-source development by Micron Technology, released concurrently with HSE, but it stands out as a separate infrastructure project. Mpool assumes the use of persistent memory and zone storage, but currently only traditional SSDs are supported.
Performance tests using the YCSB (Yahoo Cloud Serving Benchmark) package showed a significant performance increase when using 2 TB of storage with 1 KB data block processing. A particularly significant performance increase was observed in the test with a uniform distribution of read and write operations.
For example, MongoDB with the HSE engine proved to be approximately 8 times faster than the version with the standard WiredTiger engine, and the RocksDB DBMS engine outperformed HSE by more than 6 times. Excellent performance indicators are also evident in the tests, which featured 95% read operations and only 5% changes or additions.
Another test performed involves only read operations, it shows a profit of about 40%. The increase in survivability of SSDs during write operations compared to the RocksDB-based solution is estimated at 7 times.
Key Features of HSE:
- Support for standard and advanced operators to process data in key / value format;
- Full transaction support and with the ability to isolate storage segments by creating snapshots (snapshots can also be used to maintain separate collections in a store).
- Ability to use cursors to traverse data in snapshot-based representations.
- A data model optimized for mixed load types in a single repository.
- Flexible mechanisms to manage storage reliability.
- Customizable data orchestration schemes (distribution through different types of memory present in the repository).
- A library with a C API which can be dynamically linked to any application.
The ability to scale to terabytes of data and hundreds of billions of keys in storage. - Effective processing of thousands of parallel operations.
- Significant increase in bandwidth, reduced latency, and increased read / write for various types of workloads compared to typical workarounds.
- The ability to use different classes of SSDs in the same storage to optimize performance and durability.
You can access the engine code from the following link.