Amarok Blog

This is going to be the first of a series of posts about «spring cleaning» of your home directory. We’re also in the right season, so I’m not that Off Topic for now

Why do I care about having a clean home directory? Well, it vastly depends on my setup, but I think this is common enough to grant some discussion about it. I have my /home in a partition that is set up with DM to be replicated on my two harddrives, providing me a basic RAID1 setup for that single partition; this allows me to be relatively safe from a harddisk crash, for what concerns my important data, like SSH and GPG keys, configuration files, mail and so on.

The problem with this is that everything that gets written to my home directory has to be written on two disks, and is often a performance drawback; for this reason, I tend to scatter the non-essential data (like repository checkouts and similar) in different partitions, as they also don’t require much backup most of the times. This also brings me to hate the software that uses my home directory to save cache data, because it ends up using RAID1 for disposable data that I wouldn’t want to have backed up together with really important data.

So, this series of posts are going to explain how I try to keep my home directory clean from cache data, in part to help someone else that might want to do the same, in part for me to remember how and why I did something

One of the first services that I thought of, using data in my home directory, was spamassassin; while the amount of spam mail I receive has now decreased a lot since I left Gentoo (as I’m not in 10 aliases), I still receive quite a bit, so I’m not yet ready to remove my local SpamAssassin filter; it’s probably a sane idea especially since for xine-lib I’m going to repeat my email address over and over at every commit

SpamAssassin saves some data in ~/.spamassassin, namely the bayesian tokes database, the automatic whitelist and your extra preferences. As I don’t have extra per-user preferences (I use SpamAssassin in a single-user environment), I don’t need those, but I do need bayes and awl to work. Since I already have Amarok using PostgreSQL in this box, I decided to use PostgreSQL to also save SpamAssassin data.

Unfortunately, as it is the ebuild does not allow you to easily add postgres support, but this is probably going to be fixed in the future; I have a better ebuild in my overlay ( git://flameeyes.is-a-geek.org/overlay.git ) and I’ll see to send the changes to Perl team now; in the mean time, the things to change are not that much.

The documentation on setting up SpamAssassin with SQL backend can be found on SpamAssassin Wiki, and it applies to PostgreSQL as well as MySQL, even if some things has to be changed around, nothing major though.
During this post I’ll assume that both PostgreSQL and SpamAssassin are only reachable on localhost, and that you don’t need extra security concerns like a password to the database or something like that.

First of all, stop SpamAssassin (if your mail system is not mission critical) and start backing up the bayesian database:

% sudo /etc/init.d/spamd stop

% sa-learn --backup > sa-bayes-backup

This will create a sa-bayes-backup file with the bayesian token currently saved in your home directory in a Berkley DB file.

After this, change the useflags for mail-filter/spamassassin: disable the berkdb useflag and enable the postgres useflag; ignore the warning currently thrown by the ebuild that the bayesian filter needs the DB_File module, it works just as fine with PostgreSQL as backend, but you have to configure it. You might also want to enable the doc useflag, as right now it’s unfortunately controlling the installation of user-serviceable documentation; in alternative, just get an extracted copy of SpamAssassin’s tarball to use as a reference.

Now, it’s time to create the user and the database to store the data into.

% sudo -u postgres -i

postgres % createuser spamassassin

postgres % createdb -O spamassassin spamassassin

postgres % bzcat /usr/share/doc/spamassassin-3*/sql/bayes_pg.sql.bz2 | \

  psql -U spamassassin spamassassin

postgres % bzcat /usr/share/doc/spamassassin-3*/sql/awl_pg.sql.bz2 | \

  psql -U spamassassin spamassassin

You could also use per-user preferences stored in SQL backend if you really need them; as I don’t need them, I instead edited /etc/conf.d/spamd replacing the -c option (which forces spamd into creating per-user configuration files if missing) with -x (which says to spamd to ignore per-user options, that is just what I need.

Now it’s time to set up the database connection from SpamAssassin; although the ebuild suggests to use the secrets.cf file, that is not readable by users, to configure the connection to the database, if you plan to use sa-learn from your user, you might prefer to just enable it in a world-readable file, especially if you don’t have any security concerns on the use of the spamassassin PostgreSQL database; this is what I have done anyway:

bayes_store_module      Mail::SpamAssassin::BayesStore::PgSQL

bayes_sql_dsn           DBI:Pg:dbname=spamassassin;host=localhost

bayes_sql_username      spamassassin

bayes_sql_override_username     spamassassin



auto_whitelist_factory  Mail::SpamAssassin::SQLBasedAddrList

user_awl_dsn            DBI:Pg:dbname=spamassassin;host=localhost

user_awl_sql_username   spamassassin

At this point, SpamAssassin will only use PostgreSQL for its databases, so you can just remove your ~/.spamassassin directory, it will not be recreated. Let’s then start PostgreSQL (or make sure it’s started already, and then restore the Bayes database:

% sudo /etc/init.d/postgresql start

% sa-learn --restore sa-bayes-backup

Now you could restart spamd and have your system back already, but there is one problem with the current ebuild (the one in my overlay does not need this change though): it does not depend on PostgreSQL. From one side it’s correct, you might not be using the localhost pgsql to store the data, so in that case you don’t have to care to start spamd after postgresql, but if you’re going to use a local configuration, you certainly don’t want spamd to start before the PostgreSQL database is up, so you have to edit the /etc/init.d/spamd script, and add in the depend() function, a simple use postgresql line; add postgresql to your default runlevel, and that should be it.

At this point you’re set, just restart your spamd, and it won’t use your homedirectory to store cache data anymore!

% sudo /etc/init.d/spamd start

Beside the ABI breakage that is needed to reduce the structures’ size, that as I blogged yesterday helps quite a bit, another of my currently worked on branches is the FFmpeg integration.

Now, xine-lib has used FFmpeg since ever, and it has an internal copy of it (although on Gentoo you use the external copy of it to reduce the size of the code, and avoid having double security problems), but that copy requires rewriting the build system from FFmpeg’s own makefiles to Makefile.am files, which is far from trivial and quite error prone.

It also requires to clone in configure.ac the tests for the features that are tested in FFmpeg’s own configure script, which is a waste of time when using external FFmpeg most of the time, and needs to be maintained over the long run as they might be modified by FFmpeg developers, and we can’t just copy them 1:1 as they don’t use autoconf either.

To solve this issue, I’ve been working before to implement a sort of Chinese Wall, so that FFmpeg’s build system could be called from xine’s automake build system without need to accommodate the Makefiles every time an upgrade is needed.

It wasn’t that difficult to begin with, but there was an obstacle with the make dist command that is used to create the distribution tarball, as there was no way to seemingly merge its processing with FFmpeg’s build system; today I then implemented a dist-hook that takes the list of files to add for FFmpeg support to the tarball. It requires an extra step during the update of the internal copy, but that’s far from being a problem, as it’s certainly more straightforward than it was to update the previous FFmpeg copy.

With this problem fixed, I was finally able to start working on getting distcheck working and thus trying to get a working branch out of it. The result is quite good, as I was able to get a distcheck running already, and I’m now working on actually implementing Miguel’s idea to disable the “uncommon” audio/video decoders present in FFmpeg; his work on the current 1.1 series is basically half-implemented, and probably never to be fixed, as it would require to change all the Makefile.am so that they take into consideration the huge amount of conditionals currently present in FFmpeg; I think once it’s completed I’ll just ask Miguel to give up on having it working in 1.1 series, as it’s unlikely to ever work decently… well, it might even work, but refreshing FFmpeg would require a huge amount of work, and we are understaffed already.

And by working on this branch, and on this idea, I already discovered two bugs in FFmpeg that I’ve locally patched.. the patches are now waiting (as usual) in ffmpeg-devel to be applied.

Hopefully, this will also be integrated in 1.2 series, that should then start being interesting…

So, after xine finally moved to Mercurial for xine-lib management, I’ve decided to start working on those things that required me to branch out, at least on some of them that is; the first one I was able to tackle down was ffmpeg_integration, that now works fine beside the dist target.

And then I moved working on the structures, applying pahole to all the structures in libxine, even those comprising the public ABI of the library, as I could just break the ABI when needed, rather than limiting to the local structures of the plugins. Some of these changes applied to structures that are not part of the public ABI, so I ended up merging them to the main repository already, and will be present in 1.1.5, even if they are mostly bytes-size changes, that nobody beside me should care about.

But then tonight I gone looking for the 32 buttons that are/were present as an inline array in one of the video overlay structures; I was going to change the inline array with a dynamic array or with an array of pointers, so that the memory was going to be used only when actually used..

It was a sour surprise to find out that the array was never used at all in the code, and it wasn’t used on frontends either, and by removing it, the size of the structure in which it was dropped from 86KB to 40 bytes.. and then the video_overlay_s structure dropped from over 4MB to about 3KB… finally with it removed, there was also a 10MB of memory usage cut down during xine-lib runtime playback, 1/3 of the memory usage when playing an mp3 file:

(before the change, about 1.1.4 release)

(after the change, on my nopadding branch)

For starters, this does seem quite good, don’t you think?

So, as of yesterday, thanks to Darren Salt and Reinhard Tartler, xine finally moved out of CVS dark age, and came into Mercurial’s territory.

As you can see on Debian’s Mercurial repositories page we have quite a few xine repositories: together with xine-lib, Darren also moved his gxine repositories so they are all together.

I’ve already started doing some little work, which shown a little bug on CIA.vc yesterday that was ignoring my commits; fortunately Micah corrected the problem right away.. thanks Micah!

One of the tasks I was able to take care of today was the update of the FFmpeg integration branch, that now is up to date with main xine-lib-1.1 branch. Unfortunately Binutils seems to have broken FFmpeg so I’m unable to finish building and thus checking if it actually works as intended.

Anyway, soon I’ll see to post more updates about it.