Flameeyes

Non so che altro titolo dare a questo post, se non, appunto «Discarica Abusiva», perché di questo tratta.

Mi è venuto in mente di scriverne dopo aver parlato l’altra sera con X-Drum su #gentoo-it su Freenode, visto che stavo informandomi su dove portare a far smaltire alcuni rottami che ho qui in casa (la vecchia lavastoviglie e un gruppo di continuità totalmente partito; alla fine domani chiamerò la società che si occupa dei servizi ambientali per questa zona, Vesta, per fissare un appuntamento per il ritiro, era così semplice, ma non è un fatto conosciuto che questo ritiro è pure gratuito).

Beh, non si tratta di una novità; quando ancora ero in seconda media (quindi ormai circa 8 anni), sono iniziati i lavori necessari alla costruzione delle fognature comunali per la zona dove abito (Santa Lucia/Tarù, alla periferia di Zelarino, entroterra di Venezia, Mestre). Beh per qualche motivo dopo due o tre anni di lavoro, la prima impresa è stata mandata a casa, e il cantiere che avevano aperto a non tanti metri dalla rete del mio giardino, in mezzo ad un campo, è stato posto sotto sequestro. È arrivata un’altra ditta che ha aperto un altro cantiere di fianco, ed è andata avanti con i lavori fino a non molti anni fa.

Beh, visto che il primo cantiere era sotto sequestro e intoccabile, e il secondo cantiere aveva gente che andava e veniva tutti i giorni, la sbarra di accesso al campo è rimasta aperta per molti mesi, e dove c’erano i container, rimasti per anni incustoditi e intoccati (solo poco prima che il secondo cantiere finisse sono stati spostati due camion ribaltabili che erano rimasti là, carichi di benzina pure, in mezzo ad un campo arso dal sole tutta l’estate), si è cominciata a formare una discarica abusiva.

Le foto, cortesia di un mio amico, Alberto Chinellato, che ha collaborato in passato con il quotidiano La Nuova Venezia, e «caporedattore» di un giornalino di zona, sono tutte da vedere:

Una visione d’insieme della discarica.

Se andate su Flickr trovate anche le altre foto panoramiche.

Ora, non è che sia difficile da vedere, visto che si vede dalla strada, non è difficile da sapere dove si trova, se un cantiere è sotto sequestro, ci si aspetta che le forze dell’ordine sappiano dove sia, e soprattuto è cosa nota almeno alla Polizia municipale (i vigili urbani) visto che sono stati contattati da me e da mia madre più volte, e sicuramente anche da altri abitanti della zona (anche se noi siamo effettivamente i più esposti essendo a occhio meno di 100 metri dal nostro giardino).

Per circa due anni l’erba nel campo non è stata tagliata, e come potete vedere dalle foto, tutt’ora ci sono piante ed erbacce che crescono da quello che è rimasto, fornendo un ambiente ideale per pantegane e altri animali non propriamente salubri.

In aggiunta, poiché fino a qualche mese fa la sbarra era ancora aperta, la «stradina» di accesso al campo era diventata una meta nota di auto che cercavano un posto per infrattarsi (e vi assicuro che la zona di romantico non ha assolutamente nulla quindi traetene voi le deduzioni). A tal proposito, Alberto ha anche scritto un articolo sul giornalino di cui sopra:

La copertina del Quattrogatti (giornalino locale) con l’articolo di Alberto riguardo alla discarica.

(in aggiunta potete trovare una scansione in DjVu ad alta qualità per la lettura direttamente sul mio server – sorry ma questo era l’unico formato che avesse una resa accettabile, visto che non ho il sorgente originale da convertire in PDF).

Ora, dopo otto anni che non succede nulla, vediamo se la rete può aiutare ad avere una risposta fattiva nella soluzione di questo problema… specie perché ci sono già altre discariche abusive qua attorno di cui nessuno si occupa.

So today the allocated projects for the various organisations accepted in Google Summer of Code were released, and I decided to skim through them to see if there was something interesting; especially with xine-lib in mind.

First, FFmpeg projects, as FFmpeg is for good or bad xine’s heart, and improvement in it is certainly going to do good to xine project too. There are four decoders proposed and accepted: RV40 (Real Video), E-AC3 (used in some kind of HD media if I remember correctly), Dirac and QCELP. I sincerely never heard of the last one, and of Diract) I heard only as an experiment from BBC; I prepared the ebuild for the Dirac library when I was trying to support it in VLC, but as there was not a simple way to test it I left it masked in the tree for a while; the package is still there, masked, but someone else probably took it over; with further experience, I should have put it in an overlay.

While really interesting, RV40 and E-AC3 doesn’t sound that appealing for the target of xine users; first the Real demuxer needs an overdue overhaul, second E-AC3 only makes sense if we also start supporting HD media, which I don’t think we’d be doing soon enough. I was hoping for a Monkey’s Audio decoder, but that was not the case neither this year.

Users interested in having Monkey’s audio support in Amarok and other xine-based players might consider the idea to start a Chinese wall reverse engineering for it from the macport library, also if some description is present already, and might be enough to start working on it; I’m not keen on going to do it myself though; beside having other things to do, I don’t really know where to start at the moment.. bribes might make me reconsider but remember that almost anything I do goes to help in some way…

Now instead FreeBSD has interesting ideas, I’m quite interested in support for the Apple’s MacBooks (hoping that they will also consider MacBookPro), as I’m an user for it; the TCP/IP regressions testing might also help, as there has been a few TCP/IP problems in the past that might get solved once and forever with this, but what is really interesting is the bintools project to replace parts of binutils with BSD-licensed variants.. interesting not much for its usefulness, but just to see the mess when you add bintools’s commands to binutils’s and elfutils’s … reinventing the wheel, year after year.

For what concerns KDE I think we’ll see the results only in a few months, but I’m happy to see that there are projects to improve Kopete’s MSN and Jabber protocols. I just hope that in KDE4 Kopete is going to get more maintenance and won’t be allowed to fall behind every year to an obsolete state (Jingle support was nice… but it was left incomplete and never update up to now!). And I don’t intend to forget Mike’s project on Kontact’s blogging support.. that will be neat for a blog-addicted like me!

For X.org, the server-side XCB support is something that I’ve read about for a few months now in the xcb mailing list and seems like it’s going to be quite useful to reduce the possibility of a mistake in the X server code, which is something we all rely upon nowadays (by the way, since X.Org was founded, using X started to be less a pain that it was before, the no-configuration startup is nice, too bad xorg.conf still uses that obscure format ).

When it comes to GCC instead, the projects are more technicalities, but there is at least a speedup project that all Gentoo users should welcome easily, and the SEH support (that by the way comes from a friend of mine, hey Hyp if you’re reading! ), that will be one less reason not to use MinGW32 rather than VC++.

And not directly related to xine, but coming from one of xine’s contributors in the last months, there’s a project for Wine to support Solaris.. I wish him good luck, as that might make wine’s code even better, as usually portability helps cleaning up after yourself.

Okay now let’s hope these projects will all be completed during Summer of Code and maintained for a long time afterward!

Finally, yesterday I received the package from Amazon JP!

Let’s ignore the fact that on a ?110 order I had to pay ?41 for custom duties (oh well, the cost of a single CD I ordered in a shop around here I asked before was way higher anyway), and I’ll also try not to repeat too often the ridiculousness that DHL Express was able to ship the package in three days from Narita to Marcon (Venice) and then it took three more days to ship from there to my home (because last Saturday they didn’t deliver, and the package was received there just Friday evening); all things considered, four working days is not a bad delivery time.

Now as crazy as I am, I do like some Japanese music… I started loving L’Arc~en~Ciel music when I heard Driver’s High as opening for Great Teacher Onizuka anime. I was able to buy myself Awake some time ago, and yesterday I received ark in the special 15th anniversary edition, CD with DVD, so I was more than happy!

Now this doesn’t mean I passed the whole day listening to music, eh.. I was actually working for part of the time on finishing the details of my (now expired) job, then I took a shopping tour to a bookshop in Mestre, looking for something from Dario Fo (with scarce results, I was able only to find a single book of him), but of course this without forgetting working on xine; as I said, mercurial is something useful.

The 1.2 series, on which I’m focusing at the moment, is now spotting a cleaner support for packed attributes, and I’m working on adding Doxygen documentation to the code; currently I’m converting documentation in the buffer.h file, just because it’s the one I found most difficult to understand myself.

Unfortunately, emacs support for Doxygen seems to be pretty much non-existant, and even the doxymacs extension does a lousy job, as it fails to highlight the Doxygen comments properly. Add to that that emacs insists on putting two spaces in front of every line after an extern "C" { block (that has to stay there as it’s a public header, although nowadays I think it should be C++ code that should use extern "C" { before includes, rather than polluting the C header files), and you can see why I haven’t converted the whole xine-lib yet.

Unfortunately, I’m not yet sure on how many of the improvements I hoped for I’ll be able to implement in 1.2 series, considering that I’d hope for 1.2 to be released more or less with KDE4 release (before if possible, if not possible not too much afterward), as the 10MB improvement in memory usage is a killer feature.

I’m also trying now to handle a few things in a slightly different way, for instance I’ll be trying to move all of the contributed code in the contrib/ subdirectory, rather than having it scattered among xine proper sources. The problem is that having to deal with both build system and code changes makes quite more difficult to track down the changes, but this should not be a big problem on the long run.

And for who’s curious, yes, I also have an Amazon JP wishlist but it’s mostly for my own use, as the customs duties are a problem.

Tonight I finally started the true 1.2 series branch on xine-lib’s repository; this branch is the merge of the nopadding and the ffmpeg_integration branches, which means it starts with two big improvements: 10MB memory saving and a simpler to upgrade FFmpeg snapshot, with handling for disabling uncommon decoders, just like Miguel wanted.

Now, if you looked well enough at the graphs you posted, you already know what I’m talking about, if you didn’t, well here there’s a new graph, created using the -pq option at xine so that the logo is not displayed once the mp3 finished playing, removing the spurious spike at the end of the graph.

The yellow are you see it’s the memory created by _x_fifo_buffer_new called by _x_video_decoder_init function. That’s right, there’s a huge amount of memory allocated by the video decoder initialisation function when playing an audio only mp3.

Okay, so what the problem is here? First, as the code cannot tell if what is playing is audio only or contains video too beforehand, it has to initialise the video driver every time too; this is quite normal though, nothing to be a problem.

The problem is that the video initialisation initialise automatically also a number of buffers, by default 500, of size 8KB.. even if they never gets used. It also allocates a single memory area for the buffers, 2KB-aligned, which is not exactly the cheapest way to allocate memory.

Now, this in general should be okay, shouldn’t it? Most of the times you’re playing video with xine, and so it should be fine, but there are cases when using this method will just waste a lot of memory, especially if the user tried to be smart and increased the number of buffers to something that isn’t exactly sane…

I’ll now be working on checking if it’s possible to just allocate the buffers when they are needed; in that case the graph will certainly appear different, as the memory will increase less steeply, and would just run down together at the end.

Let’s return to work (although I should be finding a way to clean up the mess that is my home office here)…

This is going to be the first of a series of posts about «spring cleaning» of your home directory. We’re also in the right season, so I’m not that Off Topic for now

Why do I care about having a clean home directory? Well, it vastly depends on my setup, but I think this is common enough to grant some discussion about it. I have my /home in a partition that is set up with DM to be replicated on my two harddrives, providing me a basic RAID1 setup for that single partition; this allows me to be relatively safe from a harddisk crash, for what concerns my important data, like SSH and GPG keys, configuration files, mail and so on.

The problem with this is that everything that gets written to my home directory has to be written on two disks, and is often a performance drawback; for this reason, I tend to scatter the non-essential data (like repository checkouts and similar) in different partitions, as they also don’t require much backup most of the times. This also brings me to hate the software that uses my home directory to save cache data, because it ends up using RAID1 for disposable data that I wouldn’t want to have backed up together with really important data.

So, this series of posts are going to explain how I try to keep my home directory clean from cache data, in part to help someone else that might want to do the same, in part for me to remember how and why I did something

One of the first services that I thought of, using data in my home directory, was spamassassin; while the amount of spam mail I receive has now decreased a lot since I left Gentoo (as I’m not in 10 aliases), I still receive quite a bit, so I’m not yet ready to remove my local SpamAssassin filter; it’s probably a sane idea especially since for xine-lib I’m going to repeat my email address over and over at every commit

SpamAssassin saves some data in ~/.spamassassin, namely the bayesian tokes database, the automatic whitelist and your extra preferences. As I don’t have extra per-user preferences (I use SpamAssassin in a single-user environment), I don’t need those, but I do need bayes and awl to work. Since I already have Amarok using PostgreSQL in this box, I decided to use PostgreSQL to also save SpamAssassin data.

Unfortunately, as it is the ebuild does not allow you to easily add postgres support, but this is probably going to be fixed in the future; I have a better ebuild in my overlay ( git://flameeyes.is-a-geek.org/overlay.git ) and I’ll see to send the changes to Perl team now; in the mean time, the things to change are not that much.

The documentation on setting up SpamAssassin with SQL backend can be found on SpamAssassin Wiki, and it applies to PostgreSQL as well as MySQL, even if some things has to be changed around, nothing major though.
During this post I’ll assume that both PostgreSQL and SpamAssassin are only reachable on localhost, and that you don’t need extra security concerns like a password to the database or something like that.

First of all, stop SpamAssassin (if your mail system is not mission critical) and start backing up the bayesian database:

% sudo /etc/init.d/spamd stop

% sa-learn --backup > sa-bayes-backup

This will create a sa-bayes-backup file with the bayesian token currently saved in your home directory in a Berkley DB file.

After this, change the useflags for mail-filter/spamassassin: disable the berkdb useflag and enable the postgres useflag; ignore the warning currently thrown by the ebuild that the bayesian filter needs the DB_File module, it works just as fine with PostgreSQL as backend, but you have to configure it. You might also want to enable the doc useflag, as right now it’s unfortunately controlling the installation of user-serviceable documentation; in alternative, just get an extracted copy of SpamAssassin’s tarball to use as a reference.

Now, it’s time to create the user and the database to store the data into.

% sudo -u postgres -i

postgres % createuser spamassassin

postgres % createdb -O spamassassin spamassassin

postgres % bzcat /usr/share/doc/spamassassin-3*/sql/bayes_pg.sql.bz2 | \

  psql -U spamassassin spamassassin

postgres % bzcat /usr/share/doc/spamassassin-3*/sql/awl_pg.sql.bz2 | \

  psql -U spamassassin spamassassin

You could also use per-user preferences stored in SQL backend if you really need them; as I don’t need them, I instead edited /etc/conf.d/spamd replacing the -c option (which forces spamd into creating per-user configuration files if missing) with -x (which says to spamd to ignore per-user options, that is just what I need.

Now it’s time to set up the database connection from SpamAssassin; although the ebuild suggests to use the secrets.cf file, that is not readable by users, to configure the connection to the database, if you plan to use sa-learn from your user, you might prefer to just enable it in a world-readable file, especially if you don’t have any security concerns on the use of the spamassassin PostgreSQL database; this is what I have done anyway:

bayes_store_module      Mail::SpamAssassin::BayesStore::PgSQL

bayes_sql_dsn           DBI:Pg:dbname=spamassassin;host=localhost

bayes_sql_username      spamassassin

bayes_sql_override_username     spamassassin



auto_whitelist_factory  Mail::SpamAssassin::SQLBasedAddrList

user_awl_dsn            DBI:Pg:dbname=spamassassin;host=localhost

user_awl_sql_username   spamassassin

At this point, SpamAssassin will only use PostgreSQL for its databases, so you can just remove your ~/.spamassassin directory, it will not be recreated. Let’s then start PostgreSQL (or make sure it’s started already, and then restore the Bayes database:

% sudo /etc/init.d/postgresql start

% sa-learn --restore sa-bayes-backup

Now you could restart spamd and have your system back already, but there is one problem with the current ebuild (the one in my overlay does not need this change though): it does not depend on PostgreSQL. From one side it’s correct, you might not be using the localhost pgsql to store the data, so in that case you don’t have to care to start spamd after postgresql, but if you’re going to use a local configuration, you certainly don’t want spamd to start before the PostgreSQL database is up, so you have to edit the /etc/init.d/spamd script, and add in the depend() function, a simple use postgresql line; add postgresql to your default runlevel, and that should be it.

At this point you’re set, just restart your spamd, and it won’t use your homedirectory to store cache data anymore!

% sudo /etc/init.d/spamd start

Beside the ABI breakage that is needed to reduce the structures’ size, that as I blogged yesterday helps quite a bit, another of my currently worked on branches is the FFmpeg integration.

Now, xine-lib has used FFmpeg since ever, and it has an internal copy of it (although on Gentoo you use the external copy of it to reduce the size of the code, and avoid having double security problems), but that copy requires rewriting the build system from FFmpeg’s own makefiles to Makefile.am files, which is far from trivial and quite error prone.

It also requires to clone in configure.ac the tests for the features that are tested in FFmpeg’s own configure script, which is a waste of time when using external FFmpeg most of the time, and needs to be maintained over the long run as they might be modified by FFmpeg developers, and we can’t just copy them 1:1 as they don’t use autoconf either.

To solve this issue, I’ve been working before to implement a sort of Chinese Wall, so that FFmpeg’s build system could be called from xine’s automake build system without need to accommodate the Makefiles every time an upgrade is needed.

It wasn’t that difficult to begin with, but there was an obstacle with the make dist command that is used to create the distribution tarball, as there was no way to seemingly merge its processing with FFmpeg’s build system; today I then implemented a dist-hook that takes the list of files to add for FFmpeg support to the tarball. It requires an extra step during the update of the internal copy, but that’s far from being a problem, as it’s certainly more straightforward than it was to update the previous FFmpeg copy.

With this problem fixed, I was finally able to start working on getting distcheck working and thus trying to get a working branch out of it. The result is quite good, as I was able to get a distcheck running already, and I’m now working on actually implementing Miguel’s idea to disable the “uncommon” audio/video decoders present in FFmpeg; his work on the current 1.1 series is basically half-implemented, and probably never to be fixed, as it would require to change all the Makefile.am so that they take into consideration the huge amount of conditionals currently present in FFmpeg; I think once it’s completed I’ll just ask Miguel to give up on having it working in 1.1 series, as it’s unlikely to ever work decently… well, it might even work, but refreshing FFmpeg would require a huge amount of work, and we are understaffed already.

And by working on this branch, and on this idea, I already discovered two bugs in FFmpeg that I’ve locally patched.. the patches are now waiting (as usual) in ffmpeg-devel to be applied.

Hopefully, this will also be integrated in 1.2 series, that should then start being interesting…

So, after xine finally moved to Mercurial for xine-lib management, I’ve decided to start working on those things that required me to branch out, at least on some of them that is; the first one I was able to tackle down was ffmpeg_integration, that now works fine beside the dist target.

And then I moved working on the structures, applying pahole to all the structures in libxine, even those comprising the public ABI of the library, as I could just break the ABI when needed, rather than limiting to the local structures of the plugins. Some of these changes applied to structures that are not part of the public ABI, so I ended up merging them to the main repository already, and will be present in 1.1.5, even if they are mostly bytes-size changes, that nobody beside me should care about.

But then tonight I gone looking for the 32 buttons that are/were present as an inline array in one of the video overlay structures; I was going to change the inline array with a dynamic array or with an array of pointers, so that the memory was going to be used only when actually used..

It was a sour surprise to find out that the array was never used at all in the code, and it wasn’t used on frontends either, and by removing it, the size of the structure in which it was dropped from 86KB to 40 bytes.. and then the video_overlay_s structure dropped from over 4MB to about 3KB… finally with it removed, there was also a 10MB of memory usage cut down during xine-lib runtime playback, 1/3 of the memory usage when playing an mp3 file:

(before the change, about 1.1.4 release)

(after the change, on my nopadding branch)

For starters, this does seem quite good, don’t you think?

So, as of yesterday, thanks to Darren Salt and Reinhard Tartler, xine finally moved out of CVS dark age, and came into Mercurial’s territory.

As you can see on Debian’s Mercurial repositories page we have quite a few xine repositories: together with xine-lib, Darren also moved his gxine repositories so they are all together.

I’ve already started doing some little work, which shown a little bug on CIA.vc yesterday that was ignoring my commits; fortunately Micah corrected the problem right away.. thanks Micah!

One of the tasks I was able to take care of today was the update of the FFmpeg integration branch, that now is up to date with main xine-lib-1.1 branch. Unfortunately Binutils seems to have broken FFmpeg so I’m unable to finish building and thus checking if it actually works as intended.

Anyway, soon I’ll see to post more updates about it.

So, I’ve taken two days totally off from almost any kind of communication, I tried to relax, and now I feel a bit better. I still do think I haven’t been able to do much good beside my work on Gentoo, but I’m not ready yet to give up, even if it’s hard to continue, I will continue. At least for a while.

Unfortunately my current UPS setup is not going to fly, X-Drum in #gentoo-it informed me last night of the incompatibility between PSUs with active PFC and UPSes with stepped sinusoidal wave.. and the new PSU I bought two months ago has active PFC. The result is that I need a new UPS, of the Smart UPS series from APC, which will cost me ?420, sigh.

Talking about the topic in subject, two days ago I analysed most of the xine-lib plugins with pahole (with a patch to fix an integer overflow in the offset counter (the author wasn’t expecting structures bigger than 64KiB, but this in xine-lib is not rare), and I do have at least one good news: FFmpeg decoding of non-mpeg video streams was taking 1MB of memory for a libmpeg2 buffer that was not going to be used; I’ve now fixed this so that the structure is only allocated and initialised when needed, so decoding will take 1MB of memory less than before, on next xine-lib release.

Unfortunately, I’ve found similar mistakes in design in other structures, most of which are public, so part of the libxine ABI, and thus I can’t fix them in the 1.1 series, not unless there are good reasons and good results to achieve by breaking it. But, next week we’re going to move to Mercurial, thanks to Darren Salt and Reinhard Tartler who are helping with the migration (for who’s wondering about hosting, it will likely be on Alioth, if they accept us), so I can branch out and fix the stuff.. and then merge back in either 1.1 or 1.2 as we feel needed.

One of the structures I surely will be refactoring is the video overlay structure.. that has a size over 4MiB as it is, which explains why the function to initialise video overlay consumes 5MiB of memory right after xine initialisation, even when playing a sound file. By instancing the structures only when really needed, and making sure that there aren’t holes around, it should be possible to reduce drastically the memory used up by xine.

Another thing in my TODO list is, as I said already, rewriting the plugins cache code. I will also try to provide a simple way to regenerate this cache globally, so that for instance it can be installed directly by the ebuild, without asking users to regenerate the cache by theirselves, and sharing it in memory (through mmap) between users too.

To help this, I’ll also see to change the way the plugins are handled, by using where possible inline arrays for the names and description of the plugins, rather than pointers, allowing to share the structures in memory, where this does not waste too many bytes.

Anyway, I still need to relax a bit more because I can’t really rest lately, and I do need some rest if I am to carry on.

Edit: thanks to Mike Arthur for letting me remember I got the phrase wrong, the permalink will have to stay wrong to avoid confusion though :/

So today’s news are that a bunch of people who never seemed to care about ALSA (sorry Tony but iirc you said that yourself) now will be the new ALSA maintainers. And their first action is to flush down the toilet the months of work I’ve put in making sure that the official alsa-driver package works.

Okay, well, their responsibility.. but again, what about those people who need to use alsa-driver? While Josh (nightmorph) agrees that it’s more than 2.4, Daniel seems to care about it just for 2.4 and those few extra drivers on it. Tell you what: neither my boxes can run in-kernel ALSA, as they both die with it.

And has any of them tried to contact me? I told to them already that the ALSA maintainer’s guide is incomplete, were they able to fill the gaps that I haven’t been able to update before leaving without asking? Maybe.

But with all due respect, Steve wasn’t able to find time to proxy-commit the ebuilds I already prepared for ALSA, Petteri did it.

Do they know about the six bugs that I’ve opened on the ALSA bugs tracker that are patched in Gentoo?

For the way it has been handled, I do think Daniel is just trying to get revenge for my handling of ALSA, sorry Daniel, I respect you, but you’re dead set on the wrong way to see the issue this time. And I’m pretty sure that it will create problems.

I don’t think I handled ALSA all that bad, I didn’t get any help by any other developer, and Daniel tried to push me off track once or twice, but there weren’t many users complaining about ALSA not working… while we all know what happened on the 1.0.8 to 1.0.9 update (or do we? rather, do they? I was there, not sure where they were on that one); genstef failed already once trying to take over ALSA from me, by the way, and if it wasn’t for my proxied commit, ALSA 1.0.14_rc3 wouldn’t have been in the tree most likely (released 6 March, my ebuild was ready on 7th, Petteri committed it at 17 March after leaving some time to see if someone else was going to pick it up).

Anyway, with the future in hands of those devs, I think I won’t restrain myself from saying “I told you so” when, in a couple of months, you’ll hear users complaining about their VIA82xx card or their HDA card not working. Bets are open on when the first signs of something broken will appear.

Oh, someone in there is going to actually take care of what I left incomplete (ld10k1 init script), or is that not cool enough?

Today I was finally able to talk with Vir (Matthias Kretz, the Phonon guy), and I got a partially good news, but also a pretty bad news.

The good thing is that there will be a notification daemon in KDE4, that will use Phonon, so that not every application that requires notification will have to load libxine (and all its damned plugins); it might be loaded on the fly on a file dialog to preview a multimedia file though, which means it’s still going to be used by more than just a couple of applications at a time.

So the result is that we are in the need to branch xine-lib to make more consistent changes to the codebase for a 1.2 series, like for instance a shared cache for the plugins, created – if at all possible – during make install phase, as well as configuration options caching. It’s a massive amount of work that has to go in that direction, so we can make more data in the plugins constant, reducing the chances of having to make the RSS pages dirty, so reducing the overall size of xine-lib’s memory occupation.

Other changes includes the ability to choose the demuxer through a shared mime type, rather than by extension, which would allow not to have to load all the demuxers one by one when we’re passing through them, and a way for the plugin to tell the library if it supports ways to check what it can demux other than by content (some plugins fall back to a by-content detection when they don’t support an extension decoding); plus there is the always-improvable support for seeking, right now there is only one choice: an input plugin can seek or cannot, simple as that, while there can be difference grades of seeking: no seeking at all (sequential), slow seeking (you can seek, but you shouldn’t just jump around to find what the format is, an example of this is HTTP streaming), usual seeking (which is what most plugins already implement), and time seeking (MMS and RTSP support this).

RTSP support is also quite lacking, and it could certainly get improved by using libnemesi (Luca knows I tried already but as it is the structure of xine is not good enough); and another external library we might as well use is libavformat from FFmpeg: right now we only support libavcodec and, to some extents, libpostproc from FFmpeg project, while we could easily make use of libavformat as an extra demuxer plugin, as that can certainly support a lot of formats without us needing to take care of them one by one.

Unfortunately, I don’t think we’ll be able to provide much of this with a 1.2 series right away, especially since we don’t have many developers around at the moment, but that can be hoped.

Now, to return on the title of my blog, I’ve decided to play a bit with pahole, a nice software coming from Arnaldo Carvalho de Melo, a kernel developer, that is able to analyse the dwarf data of binaries (them being final executeables, libraries or kernels) to identify the structures used internally, and their content, to find their size and the eventual padding holes that would allow to reduce their size.

Well, a simple run of it, shown that there is at least one structure video_overlay_object_t (which is also used in video_ovelray_event_t) that takes more than 80KB of space. I’m not sure how many of these structures are loaded, but even a single one is a lot of space used, and I’m quite sure most of it is wasted; as far as I can see, it is used for the overlaid buttons in DVD menus (and other kind of menus I suppose, although I don’t think we currently support Matroska’s menus, although I’m not sure of it); even if that is the case, I don’t really see often DVD menus with 32 buttons, and as every button take more than 2KB, it should have probably be improved by using a button list rather than a static array, allocating the buttons only when effectively needed.

As I said there is a lot of work to do, especially when it comes to memory, but hopefully as soon as we have a new DVCS the work might become easier. Of course switching to a distributed mode will require some adjustment time, as we all need to learn Mercurial (well beside Darren that is), and there will be a lot to do to let others involved, but for instance then the VDR patches from Reinhard Nissl would just become a different branch of xine-lib, that could easily be merged from time to time (today I merged two more patches from him, and I’m sure more will come in the future).

I suppose right now the main question is: where will we host the sources? I suppose the best thing would be some hosting provider on this side of the Atlantic to avoid USA patents, but it’s still something we haven’t taken a sure decision, one idea was to ask on Alioth but it’s all up to be seen.

If the switch goes well, I could also see us moving away from SF.net’s tracker system, that is pretty much unusable, but there isn’t much choice about it, I don’t intend using Mantis again in my life (I used it for NoX-Wizard, do you remember Fabrizio?); Scarab looks interesting, but I’m not sure where we could find an hosting provider with JSP support for no price (or a cheap one).

Oh well, time to go to sleep soon now. If you’re interested in pahole, you can find an ebuild for it in my overlay (I know, gitarella is having trouble, I’m afraid that the Ruby update screwed up both gitarella and ServoFlame, so my services are down until I find time (and will) to fix them.

Okay, tonight just a quick post. First of all, totally unrelated with the title (okay still distributed stuff, but not a nightware), GIT was discarded as option for xine-lib’s future Version Control System, we instead decided to go with Mercurial because of its availability on a wider range of platforms (GIT is not officially supported on Solaris at least, and Cogito does not work at 100% on FreeBSD); I also cleared up my previous doubt about it, both speed and features of the two softwares seems to be at the same level, so portability has been the breaking requirement. Darren converted the CVS already, creating a bunch of repository (due to branching), and it also got support for the user ID to author name translation (which means you’ll see the commit coming from Diego ‘Flameeyes’ Pettenò rather than from an anonymous dgp85).

With reference to my distributed nightmare, instead, I want to thank Mike (vapier) a TON, his last version of nfs-utils (1.0.12-r2) finally fixes my problem with FreeBSD clients, so I upgraded Farragut (breaking ServoFlame in the mean time I’m afraid) and now I’m putting Prakesh back into shape so I can start playing with Tomcat again.

So I don’t have to get crazy with Coda, nor I have to buy a new box right now; I’ll eventually do it, as I start to feel this box is slow, but for now it’s fine as it is without more money to go away; this is good because I can then buy the new Yu-Gi-Oh game as soon as it’s released

Anyway, tomorrow it’s working day again, so I should be sleeping, the DST switch threw me off, and I was awake all last night with the jigsaw my sister gave me… 121×80, 3000 pieces.

So, today almost no xine work, actually almost no work on Open Source at all; I had to relax a bit myself, and I had to continue testing stuff for my job before the contract expires (at the end of the month), but I found time to write to xine-devel again, asking to change version control system, because CVS is driving me nuts.

Why do I think xine needs a better version control system? Well, it’s actually easy to say; read my two previous posts, and you a see that there are a lot of details that I think should be fleshed out and fixed, especially in foresee of xine being default Phonon’s engine.

Some of these, like the memory management thing I thought of, aren’t changes that you’d expect in a stable branch, or even on a «future development» branch, as it can be risky, needs a lot of changes, and is not, in its first stages, targeted to anyone but the developer doing the work; one can develop locally, but that means he loses ability to check its paces without losing history, and also is a mess to merge back into the main branches, as you usually end up doing one big commit with everything.

Branching with CVS sucks, or rather it’s the merging that sucks a lot, something I don’t really want to try if I can avoid it. Subversion improves a bit, but I still think it’s flaky, while the whole distributed VCS series that we get nowadays (GIT and Mercurial first), uses branches as main unit of development, rather than repositories. This is quite cool for this kind of tasks.

In addition to this, non-atomic commits are problematic, and the current workflow is a total mess, as the only really active developers in xine lately have been me and Darren Salt (gxine’s author), with Claudio Ciccani working on xine-plugin. Miguel’s commits result to be slightly less than Darren’s because he checked in a few FFmpeg’s bumps, which caused the commit amount to jump up (non-atomic commits, once again). What we need is an adrenaline injection, simplifying the contribution of patches from users, hoping to find new developers or the project will wither.

Now we’ll see what will be the outcome this time around; sincerely I’m not going to accept the answer I got last time about not being yet the time to change SCM, said by the admins, whom are not committing for months now (I didn’t find even a single commit from Mike on the last six months).

On the other hand, to play locally with importing xine into GIT to see the results, I’ve written a nice script that, when ran on a checkout of a CVS repository on SourceForge.net, will give you the authors file suitable for git-cvsimport use. It page-scrapes SourceForge’s users page to get the real name there given (which might not be perfect for users who didn’t update their name after sf switched to UTF-8), and uses the user (at) users.sourceforge.net mail address, so you might want to change it a bit for instance if you know some users will prefer advertising another address, but it should cover well for those developers you can’t contact anymore (disappeared or passed away even). The page-scraping is done through wget and awk (the awk script is inlined in the sh one), and the script is actually one single command pipe (with a loop and a subshell, okay) to the shell, if written properly it can easily be a oneliner.

It might be useful if someone else is thinking about changing from SF.net’s CVS to GIT?

I’m continuing the work to make xine build (and hopefully work) on OS X, that I already started talking about a few days ago.

Again, the main problem of getting xine to build on OS X is to make the build system aware of it; I started the work on this some time ago by letting the configure understand the difference between PowerPC and Darwin, but that was just the first step of a long journey.

After the other night, though, there has been a bit of a stall, as the fixes I committed, cleaning up the configure.ac, ended up using some features coming from autoconf 2.60, so Bastien Nocera (Totem’s author) reported some problems when building with 2.59. As nobody but him seems to have noticed a difference, and as autoconf is just a tool for developers rather than for the users, I’ve updated the pre-requisite declaration on configure.ac and I’m waiting to hear from the “council” of the xine developers to see what they think about it. I suppose that 2.60 macros will be welcome to be used on the long run anyway.

Now, I have to say that the autoconf and automake code I found when I first started fixing xine, before being a developer and just helping out with Gentoo, was really messy, and it still is for the most part, at least for the code that hasn’t been updated yet.
Today I even found an install and uninstall rule for the default headers, that were being rewritten just to change the destination directory (while it would have been enough to just change the name of the headers’ group, and then replaced its foodir variable).

So from this side, overhauling the configure code is a good idea; with the changes I’ve been doing, the time requested to run the configure should be way shortened; unfortunately there are other problems like pkg-config checks that currently miss the AC_MSG_RESULT and a few tests that should be cached.

Adding a test for a GNU assembler wasn’t easy either, as I had to copy for a GNU ld from libtool, and adapt it.. but then Apple’s assembler reports itself as GNU as (1.38), which doesn’t support the syntax I needed, so I had to adapt the macro to consider Apple’s assembler a non-GNU one. Now I could make the Win32 codecs disabled by default when building on OS X with non-GNU assembler (if you were to use GNU binutils and thus GNU assembler, it would build it, probably, not sure if it works though).

This solved the problem I left the other day, but of course there were more to come, or Murphy wouldn’t be so famous. So there I was again to hack on the configure, as the planar post plugin required a byte-alignment instruction (.balign) not available on the Apple as; I then worked differently the code for the ASMALIGN macro from FFmpeg to have it available on the entire source, allowing me to use it where needed, so that the planar plugin built (thanks exg for the help here).

And was this the end? No of course, now there is the greedy deinterlacer, requiring more registers than needed, that intends not to compile on OS X, not even with -fomit-frame-pointer… the code probably has to be rewritten, maybe the big asm chunk split in simpler parts, but I’m not a big ASM expert, so I’m not sure how I’ll do that.

Anyway, I haven’t given up yet, although the road is still long… just never ask me to do this for Windows !

The problem instead is that tonight, talking with eean and leinir on #amarok, I understood there are a lot of pitfalls in the current xine code, that will be evident as soon as Phono is going to be used for system sounds in KDE: the plugin cache does not work, other then being a human readable text file (which makes parsing slow, compared to a binary cache), and crashing on some invalid data (there’s a bug open for that), the fact that configuration entries are never cached makes it pointless for those plugins having a preload flag enabled. Also, the memory waste I talked about makes even more sense now, especially because by using multiple instances, using mmap to read the files improves the situation a lot, as they can share the same memory area for the same system sounds, even between users, and if I can make the mmap being the only source of data for the library, rather than having to copy it around over and over, it will not add extra memory footprint on them.

I think I’ll have really to study more about memory management, if anybody has a suggestion, it’s very welcome.
And seeing the amount of work that I’ll probably have to pour into xine in the next months thanks to KDE 4, donations are even more welcome, especially when they come with a sample of a file not playing correctly to be fixed… better now than when people cries because they can’t use their sounds for the events. (I don’t like saying it this way, but I admit I was a bit upset when an user seemed to demand that xine resumed playing very old Ogg files, saying that the fact the format is crappy does not make the problem any less real… sure it’s a real problem, but I’m already doing quite bit for xine, and I’m almost alone.. and I know I’ll burn out when people will start demanding improvement in xine just because it’s being used by KDE 4!)

[Would it be worth asking again on xine-devel to give me a better SCM to work with? CVS is really giving me creeps, and making it almost impossible to start working on a 1.2 series… or even a separate branch for a new feature, like the memory thing]

First of all thanks to Alex for showing me that is not just my fool idea that there is something wrong in what is currently considered QA in Gentoo.

Then, before entering in the topic of this entry, I want to say that I’m sorry that today Farragut was down for two hours straight… a temporary construction fell over the power?lines on a construction yard (no, luckily not in any house I have to visit myself, I wouldn’t trust them if they can’t even get a temporary storage that does not fall over at the first windy day), and the power company doesn’t seem to think of this area like a priority. This makes it 7 hours and a half of blackout this month.

Now, to get on topic, this is the second part of my series about the reason why I left Gentoo.. it’s not exactly proper, I could have stand this easily, it’s just something that I do think it’s a mistake for Gentoo, and that should be considered as one thing that has to be fixed.

I think this is probably what Daniel missed entirely when he returned. Being a Gentoo developer is not a right, nor a privilege, it’s something else.

You can easily understand why it’s not a right, or rather how it’s not a right: you need to pass a test and the recruitment process to be able to join the Gentoo Developers pool, this is right because of course nobody is entitled to be a developer just because he wants. One has to prove not to be a burden to the developers, to know what he’s doing and to be able to actually work on the tree. It allows to shoo away time wastes and people who are not going to stick around, and just seems to come for.. not sure for what, but you can continue reading what I think the other problem is to complete this view.

Now, about the privilege, I want to let everybody know that there is nothing good at being a developer most of the times. What you get, is a nice email address, which beside having a good impression on you resumé (or on your Curriculum Vitæ if you live on the right side of the pond?) can help you to get some hostile upstream developers more friendly, but this is far from being an advantage, trust me; you get access to the Gentoo Infra, but what for?

All you have, is access to actually do more work, to check and close bugs, to commit stuff, to work on projects, to take care of ebuilds, packages, whatever. You get to follow policies, you get to be blamed if stuff doesn’t work. What you get, is only responsibility.

If you have to work with an hostile upstream for which you need a Gentoo mail address, you are not really looking for a privilege or an advantage, but just for masochism to work on something even more complicated.

The reason why I decided to become a developer, was to give back something; I was (and am) an avid Free Software user, and I wanted to contribute. I was looking for no advantage, I knew I was expecting work and responsibilities, and with this in mind, I joined Gentoo.

But now, why should I be considering that I would get fired if I ask for a specific condition to be met by my employer? I actually already make sure that some conditions stands when I’m working on something, might it be I don’t have to work on Windows, or it might even be that I don’t want to work in team with someone.

When I asked Bryan to take care of enacting geoman’s brag about retiring, I was posing a condition on my work. It wasn’t met, I left; I see no point in calling this a «blackmail», I think Steev probably labelled it most correctly as «ultimatum», but I don’t see it with such strong negative connotations. Considering it’s a volunteer work, that has nothing good to offer me, and for which I only had to offer my time, I find deciding whether to continue on it or not is my prerogative; if that depends on a particular issue, then let it be. If Bryan decided that acting on geoman’s retirement «plan» was okay, I would have just continued doing my job.

Now of course there are projects for which being a developer is a privilege, because you can access secret repositories, secret sources, secret documentation or whatever else, or because you get paid, or not sure for what else. Gentoo is not like this, not anymore. This is what Daniel missed, when he was in charge, being a developer was a privilege, now it is not, so there’s no much point on trying to deny anyone from being a developer to hurt them, the only point is to keep poisonous people from ruining the experience of other developers.

And before I get misinterpreted, I’m not saying that users weren’t thankful, I actually thank everybody who said thanks to me, o who sent me a gift, they were appreciated, and really helped me carrying on with some things I didn’t like So it’s not totally bad, it’s just that it’s not good by definition, being good is an extra, and that works out only if you actually do your share of work for the community. Thank you guys!

? Before I’m told I’m insulting the Americans, as long as North is up, we in Europe are on the right side, while USA is on the left side. It’s a joke, or a pun as you prefer, if you want I can make you a scheme, hm?