wZD is a powerful storage server efficiently, designed pfor large data storage systems with small and large files for mixed use and drastically reduces a large number of files in a compact form thate looks like a normal WebDAV server from the outside.
The server is written in Go language which uses a modified version of the BoltDB database as a back-end to save and distribute any number of small and large files, NoSQL keys / values, in a compact form within micro Bolt databases (files), with distribution of files and values in BoltDB databases depending on the number of directories or subdirectories and the general structure of the directories.
The server can significantly reduce the number of small files in regular or clustered file systems with full lock bracket. Backed by wZD developers, the cluster stores about 250 million small files spread across 15 million directories on the MooseFS FS cluster.
About wZD
wZD makes it possible to move (file) the contents of directories to files in BoltDB format and then distribute these files from these files (or put files into files using the PUT method), significantly reducing the number of files in the file system and reducing metadata storage overhead.
To increase the efficiency of large file processing, such files can be stored separately from Bolt files.
Such an approach allows to organize the storage of a large number of small files, without resting on the limit of the number of inodes in the filesystem.
The server can also be used as a NoSQL database for data in key / value format (with partitions based on directory structure) or to distribute pre-generated html or json documents from the database.
In terms of performance, loading and writing data using Bolt files leads to an increase in latency of approximately 20-25% when reading and 40-50% when writing. The smaller the file size, the smaller the difference in latency.
Of the key features, the following stand out:
- multithreading
- Multiple servers for fault tolerance and load balancing
- Maximum transparency for the user or developer
- Supported HTTP methods: GET, HEAD, PUT, and DELETE
- Manage read and write behavior through client headers
- Support for customizable virtual hosts.
- Linear read / write scaling using pooled file systems
- Effective methods of reading and writing data.
- Supports CRC data integrity when writing or reading
- Range and Accept-Ranges, If-None-Match, and If-Modifed-Since headers support
- Store and share 10.000 times more files than inodes on any Posix-compatible file system, depending on directory structure
- Support for adding, updating, deleting files and values, and delayed compacting of Bolt files
- Allows the server to be used as a NoSQL database, with easy fragmentation based on the directory structure
- Bolt file support for selective reading of a certain number of bytes of a value
- Easy fragmentation of data into thousands or millions of Bolt files based on directory structure
- Mixed mode support, with ability to save large files separately from Bolt files
- Support for getting a list or number of keys in a directory, including non-unique ones
- Includes a multi-threaded wZA filer to migrate files without stopping the service
Regarding the limitations of the current version: there is no support for Multipart, the POST method, the HTTPS protocol, folders for programming languages, recursive deletion of directories, there is no support for mounting the structure in the file system via WebDAV or FUSE, the files are stored under a system user.
Finally if you are interested in knowing more about it about wZD as well as the instructions and requirements for its installation you can consult the details In the following link.