|
This very long series of columns (part 1, part 2, part 3, part 4, part 5 y part 6) is going to have an ending here. I could expand on other elements of the semantic desktop, but I will not be able to answer the arguments that are given in loads of guides scattered throughout the Internet that advise on the best ways to deactivate the semantic desktop, because it would eat up memory by the piece . |
Don't worry, the only way you can have a capture like the one you see on a system where Nepomuk works properly like mine is to do the same as me: download 1 GB in text PDF files that have 13 million of addresses and run them through the Nepomuk indexer (not kidding, I did). Nor do I think that you handle PDFs of literally thousands of pages (for my work as a lawyer I must handle the History of the Chilean Constitution, 10 PDFs of 1.200 pages of text each), so, in a correctly configured system, you should not see this ever catch.
Now let's get serious and technical. It is not time for yet another tutorial on “disable Nepomuk to get good performance”, but for the first tutorial on the Internet on “how to get great performance with Nepomuk on”. Pay attention.
Prerequisites
Maybe this should have come first in my guide, and I'm sorry, but I just had to justify why activating Nepomuk (which was what I did in the first six installments) before telling you how to activate it correctly. So, we will take a walk on what is needed, and then we will configure.
First of all, we need to be strict about the distributions we will use. Here are very clear requirements: distributions that like to incorporate obsolete software do not work for KDE, and that unfortunately includes Debian. Thanks to the enormous work of Rex Dieter, leader of the Fedora KDE team, there is a set of packages with KDE 4.10 for Red Hat Enterprise Linux 6, so if you need KDE and a stable distribution like stone, the option is RHEL 6, or a RHEL 6 clone like CentOS, with that repository enabled.
Second, you have to be careful how KDE is packaged, because KDE is extremely sensitive to poor packaging. Until recently, Kubuntu was notorious for making grotesque packaging errors, mixing unsupported versions of required packages, resulting in a horrible Kubuntu experience and people wondering why Nepomuk was so slow and memory hungry when in fact it was the fault of the packer. Nepomuk and Akonadi transmission chain is this (using project names from projects.kde.org and latest versions)
kdelibs (4.10.4)
nepomuk-core (4.10.4)
kde-runtime (4.10.4)
nepomuk-widgets (4.10.4)
shared-desktop-ontologies (0.10.0)
sopranos (2.9.1)
akonadi (1.9.2)
Attention to the last 3: they do not depend on the KDE version used, and must be the last available, even when using a stable point version. The rule is: KDE uses the latest stable version of these packages in its stable branch, and git snapshots in its beta branches. Many extra KDE update repositories update KDE, but not these last three packages, which causes serious problems.
Added to this is Strigi, recently taken from Nepomuk, which was a real headache for all who tried to pack it. New versions weren't advertised properly, and Ubuntu didn't package new versions of this program for a long time, to the point that I had to make a fuss on Sebastian Trüg's blog to get it fixed. Fortunately, this problem is largely over, and Strigi is not updated much anymore, which eliminates the packaging problem.
For this reason, I recommend Chakra as a good index distribution. Manuel Tortosa, the KDE packager for Chakra, knows all this, and therefore the quality of the packages is good, and the experience with Nepomuk and Akonadi, under Chakra, is also good. Chakra has some serious limitations, like it doesn't default to packages that depend on GTK +, but it's a good start.
Also, as we will see next, I highly recommend a distribution that has already made the switch from MySQL to MariaDB. We will see why later.
Preparing the ground
Once we make sure that we meet all the prerequisites, and as long as we have a clean system, we are going to make some changes to the default settings.
Akonadi
We are going to place the following lines in the .local / share / akonadi / mysql.conf file.
sync_binlog = 1 innodb_flush_log_at_trx_commit = 1
If this file does not exist, we will start Akonadi to create it, and then we will close it. On console:
akonadictl start akonadictl stop
For this? MySQL (or MariaDB) is the database that supports Akonadi, and MySQL doesn't like sudden interrupts. In the event of any system crash or power outage, MySQL will introduce errors into the Akonadi database, and these errors, accumulated, will end up unhinging KMail, making its use unbearably slow. These options mean that each transaction is immediately written to disk, minimizing the risks of corruption in Akonadi in the event of a system crash or corruption. This option causes errors with certain versions of MySQL, but it works great with MariaDB.
Kernel
We are going to raise file monitoring to the maximum limit, to substantially improve Nepomuk's performance. The following option in the /etc/sysctl.conf file will do the job
fs.inotify.max_user_watches = 524288
After these two things, we will activate Nepomuk. This is done in System Preferences | Desktop search. Let's keep memory usage at the default settings and turn on email indexing. Don't forget to check out the tips in part 1 on how we can speed up indexing, and after that… check out the rest of the guides, to enjoy the semantic desktop!
Maintenance
What if we were unable to prevent Akonadi database corruptions and Nepomuk is running slow? There is still one line of defense that KDE 4.10 implemented: Nepomuk Cleaner, in addition to the little-known self-cleaning tools that Akonadi has.
$akonadictl vacuum: "Vacuum" the Akonadi database. By aspiration, understand: all entries that are not reflected in a resource are removed.
$akonadictl fsck: Try to fix the corruptions of Akonadi databases. This doesn't always work, so you need to prevent them from happening in the first place. How? With the options that we already saw.
$nepomukcleaner: It is a set of scripts prepared by Vishesh Handa to clean the Nepomuk database, which he converted to a graphical interface. Hit the "Start" button and forget about it. Running this program is mandatory if one is updating the KDE version.
With all the trinkets, on a 64-bit system, and with experimental Akonadi resources, the sum of Nepomuk and Akonadi consumes around 350MB of RAM. Much for some, but adequate, in my view, for the tremendous functionality that is gained.
But Nepomuk still runs too slow for my liking. What I do?
Wait a while. KDE 4.11 incorporates beastly performance increases for Nepomuk. This is not any kind of exaggeration: according to Vishesh Handa's figures, we are talking about 5 times the performance of KDE 4.10 in writing to the database, and more than 7 times in reading, all this, on average . The changes to be seen in KDE 4.11 are massive and will allow Nepomuk to be used, at last, as an alternative for those applications that require databases.
Also, the bug that prevents the correct launch of the Akonadi-Nepomuk connector has already been fixed in the 4.11 branch, and the Nepomuk cleaner will see great improvements. We will have a new Office file indexer, and we will be able to enjoy other tools that will be revealed later.
Hopefully this guide, I repeat, the only one you will find on how to achieve spectacular performance with Nepomuk activated, will help you have a smooth setup, allowing you to do what we saw in past installments and much, much more. Thank you for following me through all these installments, and many thanks to Pablo Castagnino, for publishing this series. See you soon.
Excellent contribution Ernesto! Thanks for sharing all these deliveries, they have helped me a lot to better understand how KDE works (in which I am just getting started) and how to take better advantage of all the tools it provides.
Regards!
Hi Ernesto,
Thank you very much for all the deliveries, they have been very very helpful. I'm using KDE 4.11 on Arch and I confirm it, nepomuk is a beast, indexing never used more than 7% of my old AMD Dual-Core.
By the way, I have read that for those who have 4 or more gigabytes of RAM it is best to assign Nepomuk at least 500 megabytes, in this way the CPU consumption is reduced and the speed of the system -in opening programs, copying of files, etc.- increases. What do you think about this?
regards
Hi Ernesto,
Thank you very much for the series of articles, very complete and detailed and they help us to get the most out of KDE and its tools. On the other hand, I want to ask you a question, I tried with the two Akonadi commands; "Akonadictl vacuum" and "akonadictl fsck" when I use it as a normal user it does not return anything and when using it with sudo it returns this
D-Bus session bus is not available!
0: akonadictl(_Z11akBacktracev+0x34) [0x417c24]
and a series of data similar to the 2nd line (which I did not want to add because it is shorter), what does this mean? If when using it as a normal user it does not return anything, does it mean that the database is healthy? or this command does not work as administrator?
I reiterate my thanks and congratulations for the articles and I hope that your work as a lawyer has to do with a constitutional change in Chile, which has long been necessary.
In my experience, with a 64-bit system, the sweet spot is 128MB allocation to Nepomuk (the rest of the roughly 350MB comes from Akonadi, MySQL, and related services). Going beyond that will simply give Nepomuk more memory to eat, without that translating into real performance increases.
That parameter was fully justified in the days of KDE 4.7, where services were not optimized and Virtuoso, the database behind Nepomuk, required massive amounts of memory to function. It's not like that.
Hello Carlos, while Ernesto answers, I will tell you about my experience with OpenSuse. Indeed and as you say, it opts for KDE, which translates into a stable, reliable and highly integrated environment with the system, as for the updates, these arrive about a month after being released by the KDE team, however you can update to the current version from the semi-official repositories maintained by the community, so far I have updated from there and I have not had problems, but there is a detail and it is the one that Ernesto points out in his guide regarding packaging and versions available, for example the official and semi-official KDE repositories in OpenSuse contain version 2.9.0 of the Soprano package, but it is only that because the other packages are in their latest version, I can solve the above quickly by searching and downloading the version 2.9.1 on the OpenSuse Build Service website (something like a Software Center). At first I commented that the fact that Suse chose KDE as the default environment resulted in a reliable and stable system, I say this comparing with the previous distribution that I use with KDE: Fedora, which worked faster than OpenSuse but was charged as I pay tribute to stability, several times I suffered from crashes and the occasions that I activated Nepomuk and Akonadi were nothing more than to receive notifications of errors, forced reboots and consumption of resources.
At present, and largely thanks to this series of articles, I have a stable, fast and highly integrated system.
Best regards.
Hi Carlos,
It seems to me that all of us who have come to Opensuse from another distro find the issue of repositories and their administration a bit confusing, but as you say, it is a matter of being attentive and having patience because after a time you end up understanding it and appreciating its benefits, it is also true that it is more difficult to find documentation for Opensuse compared to Debian, Ubuntu or Arch, however the community is very active and there are a good number of blogs dedicated largely to OpenSuse (This distro generates a very particular fanaticism) and on the IRC channel they are always ready to help you.
It is not my intention to distort the topic of the post and I hope to have the permission of Pablo and Ernesto to leave you some links that can be of help to you, personally they helped me a lot. the first is from an article that talks about the repositories in Suse and how to manage them to obtain a stable, complete and updated system and the other is a page by a blogger who dedicates a good part to talking about Opensuse.
1.- http://www.diversidadyunpocodetodo.blogspot.com/2012/11/opensuse-build-service-one-click-install-repositorios-paquetes.html
2.- http://www.victorhckinthefreeworld.wordpress.com/
I trust they will be of help to you.
Best regards.
Hi Ernesto:
Could you get a little more wet on optimal Linux distributions for the KDE desktop?
You speak of Red Hat, is this applicable to Fedora? OpenSuse seems to opt for KDE for its default desktop. How is this distribution? Mageia is also on the same wavelength. Maybe PcLinuxOS? Thanks in advance.
Thanks Moscosov. I am currently with OpenSuse. I come from Debian and I am new to OpenSuse (different repositories, tools to install programs, etc.) and although I am testing with it, it is establishing itself as a distribution. I find it very stable, practically everything is fine (except for some problem in finding documentation and "solutions" to problems, which in Debian was more within reach). Now, what I am clear about, since I installed it replacing Gnome, is that I am sticking with KDE. And it is to the point that between choosing a distribution in front of a desktop, I prioritize the benefits of using the desktop over those that the distribution could have by itself.
Package management in Debian seems simpler to me: main, contrib, non-free and there you have practically everything; in opensuse it costs me more (pacman, factory ...), although I think it's a matter of getting used to it and being a bit attentive. But I already said, if this is the penalty for having a KDE with better performance and use, I change the distribution. That is why I also raised others like Mageia, which inherits KDE as a desktop from Mandriva and its ease of making everything work out of the box. I also like that behind it it has a Debian-style foundation (I respect that, but I'm back to Canonical; and I have qualms about Novell ... and a little about Fedora).
That said, Moscosov, thanks for the report.
Greetings.
For nothing in the world use them with sudo, it doesn't work.
It is normal that nothing is returned. What you should do once you launch those commands is wait. If you launch a system monitor after launching those commands (something that can be done in KDE by pressing Control + Esc) you will notice that MySQL's CPU usage goes up after that; that implies that Akonadi is working. Leave it alone.
1. While this article was out, Soprano version 2.9.2 came out, so you're going to have to pay OBS a new visit.
2. It is never a good idea to use plain and simple Fedora with KDE. Go to http://kde-redhat.sourceforge.net/ and activate the Yum repository that appears there. Rex does a good job of patching KDE for Fedora, but you don't see much of it because it often takes time for its packages to reach the main repository.
3. I've gotten too wet, I've recommended a specific one (Chakra Linux). I've never gotten SuSE to work very well for me, and it's because SuSE doesn't get along very well, at least, with my graphics card's proprietary drivers (all I get are generalized system panics), but the feedback I've heard is that their KDE desktops are top notch.
4. What I'm definitely going to get wet with is which distribution not to: Debian. Suffice it to say that Debian Sid has KDE 4.8.4 as the latest version available. It's fine to have "stable" versions, but KDE 4.10.2 (the latest available for RHEL, we're comparing stable vs. stable distributions) gives it 2 dozen laps, and Debian Sid, Debian's "unstable" repository, should have at least less KDE 4.10.4.
If you really want to install Debian with KDE, the two available options (incorporate the ZorinOS repositories, or mix the experimental repositories) invite so much work because yes, the best thing in that case is to simply erase the hard disk or solid state drive and install CentOS or Scientific. Besides that the Debian team was the mastermind of 1 and a half years of slow KDE for all those who did not compile Strigi from git trees, as I had to learn to do.
Mageia? No, for a simple reason: once they release a version of KDE, they stay with it forever. It happened to me with Mandriva that I had to go to "Mandriva International Backports" to get a point update, and the team behind MIB decided not to support Mageia, but to migrate to ROSA Linux (with some pretty explicit anti-Mageia graphics). So if it is the Mandriva family, ROSA Linux is the choice over Mageia.
Finally, the fact that KDE requires distributions to be always up-to-date really makes roll-out distros really shine. My options were to go from Gentoo to Arch with KDEmod, and from there straight to Chakra (which is the spiritual continuation of KDEmod), and I have not regretted it. However, Arch Linux takes a lot of work to maintain.
In short, for color tastes. In most cases, the best KDE distribution is the one you are using, because you are used to its idiosyncrasies. You just have to know which distribution to use, and, if you use a distribution, what to do to improve the KDE experience (for example, in SuSE switching to the KDE Distro repository: Stable is a better idea than staying with the default packages).
Thanks for the answer Ernesto, I did the test again with the commands and it worked perfectly, the system is luxurious thanks to the tips that you have given.
Hopefully you continue with this series of articles deepening the use of Akonadi and Nepomuk or in other areas of KDE.
Thank you very much.
Best regards.
Thank you very much Ernesto. You are right, I have been min-monitoring the system with 128 megabytes allocated to Nepomuk and I do not notice significant differences in performance compared to my previous configuration.
By the way, great blog, congratulations.
I was thinking of answering here, but I decided better to send an email to Pablo and respond with another column. Hopefully it will come out soon.
OpenSuSE has nothing to do with Novell anymore. Novell split in 2: the part that remained with the name was bought by Attachmate, while SuSE went back to being a more or less independent company.
Read it, you ask several questions but they are all related.
1. If you decide to try a new distribution it is best to a) use nepomukbackup to back up your labels and custom data; b) install the distribution to test as a new user; c) restore the backup (that can be done in System Preferences | Desktop Search, go to the third tab). Backing up the custom data and re-indexing everything will give you a functional equivalent of the "portability of semantic information" you're looking for.
In addition, some time ago, there were experiments to write the Nepomuk tags in their metadata (EXIF, ID3) back to the files themselves, but they were not finished.
2. For the same reason, the main limitation is that the labels cannot be moved between computers, but depend on the Nepomuk database.
3. The best thing, as I explained, is to use a distribution that has the latest stable version of KDE. KDE 4.10.4 is that, and any distribution that doesn't have that version doesn't qualify. Other than that, the best distro to use KDE is the one you're using, that's as a basic principle. I personally recommend Chakra, for its large Spanish-speaking community, the good quality of its KDE package, and the fact that all packages compiled for Chakra have their GNOME dependencies removed as much as possible, which avoids unnecessary memory consumption and simplifies fed up with the support.
You have to use the import of the backup with care because it deletes what you have to leave what was in the copy, at least that happened to me a few days ago (luckily it was a new installation and it affected only three files). Chakra then is my choice (although it is always convenient to have an auxiliary distro out there to solve specific problems that Chakra cannot, due to lack of packages, as happened to me with the mvconv tool).
Thanks for the clarification, in that sense I think it is best to contain Nepomuk within the user folder to avoid problems (I have a shared home directory for everyone in which the photographs are located and in another the music with writing rights ; integrating the digikam and amarok database with nepomuk in this case would certainly give problems every time a user changes, for example, the rating of a file via dolphin / nepomuk).
I made the recommendations to force indexing, I am surprised now that I check the result by doing searches. I think it must have taken 1 to 3 hours to index about 16.000 files (many of them text), whereas I was expecting it to take a few, many hours more, much has improved since the old KDE 4.6 (last time I had tried). Very happy with the result. I'll apply the rest of the guides because I think I can get a lot out of this. Thank you.
This is why outdated versions of KDE 😉 won't work. I wouldn't be exaggerating if I told you that Nepomuk in KDE 4.10 is about 20 times the speed of Nepomuk in KDE 4.6, and KDE 4.11 claims to be between 5 and 7 times the speed of KDE 4.10. Sum.
Hi Ernesto.
I have used Gnome for a couple of years and decided to give KDE a try, I was especially interested in the benefits nepomuk offers. After following your tutorials I cannot enable this service.
I have the boxes checked, and the messages in bold are the following:
"Enable Nepomuk Semantic Desktop": Desktop search services are active.
"Activate Nepomuk file indexer": The file indexing service is not running.
"Activate email indexer": Ready to index data (0%).
In the "Details" option, indicate Files: Calculating and Emails: Calculating. I choose to refresh but nothing happens.
You can help me define why I can't activate nepomuk. I use Sabayon as an operating system.
Thank you very much for your attention and help.
regards