The release of new libmdbx library version 0.11.7, the launch stands out for the migration of the project to the GitFlic service after GitHub administration removed libmdbx along with a number of other projects on April 15, 2022 without any warning or explanation, while also blocking access to many developers associated with companies that fell under US sanctions.
From the user's point of view, all the pages, the repository and the forks of the project suddenly turned into a “404” page, with no possibility of communication and without finding out the reasons.
Unfortunately, almost all versions have been lost, in which there were many questions with detailed answers, as well as many discussions. The loss of this information is the only objective damage that the GitHub administration managed to inflict on the project, although partial copies of the discussions remain available in the archive.org archive.
The loss of the built-in CI infrastructure and scripts (available for free to OpenSource projects) forced us to review, unify, and eliminate a little technical debt. Now CI has been restored to almost the same volume (about 100 build configurations), with the exception of building and running tests for all BSD and Solaris variants. Tellingly, after GitHub's actions, no clarifications or notifications were received, apart from a reminder of the need for payment.
Main new features of libmdbx 0.11.7
Since the latest news about the release of libmdbx v0.11.3, in addition to recovering from GitHub actions, the following improvements and fixes are worth noting:
A total of 185 changes were made to 89 files, approximately 3300 lines were added, approximately 4100 were removed. Removed primarily due to purging of already useless tech files associated with GitHub and dependent services.
Added a fix for a detected inconsistency effect/defect in the merged page and cache of the buffer in the Linux kernel. On systems where the page and buffer caches are truly unified, it doesn't make sense for the kernel to waste memory on two copies of data by writing to an already memory-allocated file. Therefore, the data that is written becomes visible through the memory allocation before the write() system call completes, even if the data has not yet been written to disk.
Altogether other behavior is not rational, because with a delayed merge, you still have to catch locks for page lists, copy data, or adjust PTEs. Thus, the unspoken rule of consistency has been in place since 1989, when the unified buffer cache appeared in SVR4. As a result, finding weird bugs in busy libmdbx production scenarios has been a lot of work. First, reproducing the problem, then verifying hypotheses and checking improvements.
We can now confidently say that the issue was reliably identified, located, and removed, despite the complexity and specificity of the playback scenario. Also, the work of the bypass mechanism was confirmed by one of the Erigon (Ethereum) developers, in his case, in the debug build, the protection was triggered as a regression due to an additional assertion check.
It should be noted that in the context of the widespread use of libmdbx in working projects, it is fundamentally more important to ensure reliable operation, rather than to find out whether it is a bug or a feature, and whether such consistency can be relied upon especially without looking for the causes of inconsistency within the Linux kernel. So, here we are talking about fixing an issue that could affect users.
Fixed a regression of the EXDEV bug (link between devices) when hot copying a database without compaction to another file system, both through the API and with the mdbx_copy utility.
Kris Zyp has implemented support for libmdbx on Deno.
Fixed handling of value set by MDBX_opt_rp_augment_limit option when processing large transactions on large databases. Previously, due to a bug, unnecessary actions could be performed, sometimes impacting performance on Ethereum implementations (Erigon/Akula/Silkworm) and Binance Chain projects.
Many bugs have been fixed, including those for the C++ API and fixed many build issues on rare and exotic configurations. A complete list of all significant improvements is available on the ChangeLog.
Finally, if you are interested in knowing more about it, you can consult the details In the following link.