Amarok Blog - Flameeyes

My fiddling with OpenJDK

nospam@example.com (Diego Pettenò) — Thu, 10 May 2007 01:38:00 +0000

It might sound strange coming from someone who wrote a post titled The Java t..crap but I’m liking playing with OpenJDK.

Not like my Java skills (pretty much non-existant, I used it only a few times in my life) are improved, but I find myself comfortable when working with most build systems, after you get scons out of the way (note that I didn’t say I like it, but I can do it just fine), and I do have experience with adding support for external libraries, after all, and that is something that OpenJDK might need.

So, let’s start from the start. The first problem was getting to build OpenJDK on Linux. If you try to build it on AMD64 you’ll see it fails, the reason for this is that the list of files to compile on hotspot is too long for the maximum size of arguments available on Linux; this in turn is caused by the long default path for sources that Portage sets. Petteri was able to build it fine because on x86 the architecture name is “i486”, while on AMD64 is, well, “amd64”: the extra character there is enough to make it burn in flames. Kelly O’Hair from Sun is looking after the issue now, but he says it might take a few weeks to get the results out; not a problem, I suppose, in the mean time you might want to try changing PORTAGE_TMPDIR to use /tmp for openjdk (if you have space in the /tmp partition) or you might want to fool the ebuild by renaming the openjdk directory inside $WORKDIR to something shorter, like “ojdk”; actually, a single character less will work just fine.

After getting a working build of OpenJDK, and trying so that it worked fine on Konqueror (it does!), I had to start messing with it, or I wouldn’t have been myself The first thing was to get it to build with my flags correctly; I was able to get something that worked, but there are a few issues, and I’d like to be able to provide a less invasive patch. O’Hair also got me on the right track now, by telling that hotspot and j2se build systems are different, which might call for just using the same method on boths, by ignoring the default C(XX)FLAGS values coming from environment, and instead use the _OTHER variables. This should allow for just four lines changed, if I find where to put the CFLAGS/CXXFLAGS unset.

After this I started thinking of an useful hack to do to at least understand how the build system works; I could have looked at fixing a few warnings, but I thought it would have been more useful to actually start by looking at the build system, that is what I will need to understand to be able to get OpenJDK building on FreeBSD. I’ve seen zlib-1.1.3 sources in the tree, so I decided to find a way to build against external zlib. Easy, quick, interesting and useful.

Why useful? Well, first of all, it would use a more stable version of zlib, and would share its code with the rest of the system (almost every process has a zlib copy loaded one way or another), second of all that would make it non-vulnerable by possible attacks on the zlib code. Unfortunately adding the support for external zlib was not as easy as I thought, but it was mostly because I stupidly forgot FEATURES=keepwork on the environment (I wanted to get the object files output, as I want to see if the executable stack problems that affects Java are due to the architecture, or just a problem of not having the GNU stack markings on the source files).

And while looking at fixing this, and seeing a tempnam() warning at linking stage, I ended up finding a tiny memory leak, as the string coming from tempnam() is never freed. I’ve prepared a patch and sent it, hopefully to the right mailing list.

Other hacks that might be useful would be finding a way to enable or disable ALSA and Motif bindings building, to reduce the size of the final output, and to avoid depending on those to build openjdk, but I wonder if Sun would ever accept such changes.

When I’ll be comfortable enough with both build system and development environment, I’ll see to start working on building OpenJDK on FreeBSD; for what I can read on the build system mailing list, it should be possible, for now, to build OpenJDK using a JDK 1.5, which is available for Gentoo/FreeBSD, even if the best thing would be using 1.6. Probably I’ll have to provide a openjdk-bin package for Gentoo/FreeBSD to use as a seed, afterward.. this of course if the license allows to do this (most of OpenJDK is GPL-2, but there are some binary blobs still present, until they are replaced, it might not be possible to redistribute the binary itself).

Unfortunately a big problem I have to face while hacking at this is that I don’t have any Solaris/OpenSolaris installation, and I suppose that Sun will accept more willingly contributions that don’t break their main target platform

Now, of course, the reasons I have to find this project interesting and to revert my attitude toward Java are not limited to the simple GPL-2 licensing. What I find more attractive, and makes me hope I’m not mistaken on my judgement, is that now it would be possible to support more hardware platforms, making Java a true platform agnostic language (okay, Kaffe help a bit with this, but the results are not really at the same level), and will also allow to support more operating systems. And with more users’ contributions, it will probably also get better optimisations (hey we got guys like FFmpeg devs who are able to optimise the hell out of multimedia codecs, someone might find useful things to improve on OpenJDK too!).

For what concerns Gentoo/FreeBSD, for sure FreeBSD guys have already some build definitions to support FreeBSD; the problem of those are that are likely not GPL-2 as it is, as they were designed for previous JDK versions, so I’d rather not even look at them to avoid legal issues. The assembler code would likely be the same between Linux and FreeBSD, with the same architecture; for SPARC, the code might be just taken out of the Solaris assembler sources, at least on i486, the assembler between Linux and Solaris doesn’t change besides from a few minor places, and the main issue there is the difference in the assembler program used, and thus in the syntax.

Let’s try to make Java a truly Free platform, and allow it to run on as many system as possible, then it might replace Flash and beat the heck out of Silverlight; Sun seems to be in the right mind set… so waste no time!

Avoid SpamAssassin 3.2.0

nospam@example.com (Diego Pettenò) — Wed, 09 May 2007 22:36:00 +0000

Seems like this release has some problems, I’ve been seeing the Junk folder empty for the past few days, and I thought the problem was due to PostgreSQL update (that’s also a damned update, but that’s for another day). Seems like 3.2.0 instead is not scoring anything at all anymore.

Mails that are pure spam are not even considered by 3.2.0 as possible spam, marking them with a 0.0 score, while 3.1.8 finds them just fine.

I’ve downgraded to 3.1.8 and I’ll probably stick with that for a while.

Ah and forgot to count that the update channel for 3.2 is broken too, with two different bugs, so it can’t even be updated.

I hope in a 3.2.1, but I won’t count on it as it is.

By the way, I got some progress on OpenJDK, I’ll probably blog about that later or tomorrow.

OpenJDK and Gentoo/FreeBSD

nospam@example.com (Diego Pettenò) — Tue, 08 May 2007 23:02:00 +0000

So, Betelgeuse blogged about the new partly-free JDK released by Sun Microsystems. Kudos to them for actually starting to fix the Java Trap. Hopefully in the next months we’ll finally see Java taking the place it was designed for in development of platform-agnostic software, and applets in web pages, with the (hoped) availability of it on more platform that are available right now.

Of course I want to get OpenJDK working on Gentoo/FreeBSD as soon as possible, as the FreeBSD Foundation’s “Diablo” project seems to be stale and broken (latest version is 1.5.0-7, I’m not even sure if it’s vulnerable, as I remember some security advisories for sun-jdk for Linux), and I wanted to move to Tomcat for my blog. So I asked Betelgeuse a bit of information about openjdk and downloaded the Java Experimental overlay, and decided to merge openjdk on Linux first.

To get it to work on Gentoo/FreeBSD the first thing is to get the Linux emulation support at least partly working, as a JDK 1.7 is needed as a seed to build OpenJDK (this reminds me a lot of GHC unfortunately). So I’ll work on it until I can get OpenJDK building here.

Unfortunately I can’t build openjdk yet, the build fails with an “Argument list too long” error in an execve() call. I’m not sure what causes it, but I hope to investigate it further in the near future (maybe I should ask Betelgeuse tomorrow after his exam).

Another problem I’ve found with OpenJDK is that the build of hotspot mixes the values of CFLAGS and CXXFLAGS, throwing a lot of warnings about -Wno-pointer-sign and -Wno-format-zero-length that I have in CFLAGS and are not supported by GNU C++ building. I’ve tried fixing this one but I was unable to find the perfect patch to fix it, so I reported the bug with the tentative patch to Sun, hopefully they know their build system better than I do

When I’m able to get OpenJDK building on Linux, and I understand the process well enough to work on it, then I’ll try to do my best so that OpenJDK works on Gentoo/FreeBSD at the best I can. I’m sure there are people from FreeBSD project working on it too, they did this already to get Diablo out, after all, but most of them don’t blog so you can’t really know what they are up to (beside if they are the same who would be releasing the new Diablos, they are probably missing in action for months now).

Who knows, maybe we’ll be able to get OpenJDK working on Gentoo/FreeBSD/SPARC64 too, that would be neat After all, FreeBSD and Solaris aren’t that much different (as the DTrace and the ZFS ports shows) and SPARC64 is supported by Sun for sure

The relaxing method failure

nospam@example.com (Diego Pettenò) — Tue, 08 May 2007 00:56:00 +0000

Not sure if you remember, but I’ve started, since a month and a half ago, trying to relax myself by playing with the Nintendo DS. When I bought I also bought Mario Kart DS (provided with the unit itself) and Pokémon Sapphire (for GBA, but that was needed also to protect the GBA card slot, as the pre-Lite version doesn’t have a dustcover for it). I’m the nerd kind of guy who actually like the Pokémon series, and I was pretty happy to play it through the end; I’ve played emulated the previous generations, this was the first time I could afford a game myself, and so I took the chance.

I don’t actually like racing games, I got Mario Kart just because it was sold together with the unit, but I enjoyed it, especially to try out the Nintendo DS graphic which was quite nice. Of course after a while, playing only two games starts to be boring, so when I had the last cold I asked my syster to pick up Lego StarWars II for me.

Nice game, nice graphic, nice ideas… too bad that starting from Episode V (the second of the three movies reproduced in the game), it starts to feel buggy.. there are times when you can end up stuck in a wall, requiring to end the mission and then restart from scratch. The game is nice so even if bugged, I continued playing, till in Episode VI I got stuck in a big bug and I needed to search for the solution of the game to find how to pass that mission… the solution also referred to the bug, so I did a bit of search, and I found that LucasArts admitted that the game, in its DS version, was shipped incomplete and unfinished. Nice one.

Oh well, as I said the game is interesting on its own so I don’t feel like I totally wasted my money (I also got it relatively cheap), still I would have liked if Nintendo paid more attention to the title that are published worldwide, maybe putting a “Notice: might contain annoying bugs” on the package… okay I know it won’t ever happen, but let’s say that this game didn’t relax me that much.

Then finally the game I was expecting was released, Yu-Gi-Oh Championship 2007, a nice game after all, although I don’t play the TCG, I found it quite interesting to play emulated before, so I gone for it as soon as it was available (via Amazon, as the local suppliers don’t carry it), I received it last Saturday and I started playing around. The nice thing about this game is the online challenge, which finally allows me to play against human opponents: playing against the computer becomes just a run to a better deck every time until you can beat the next opponent.

So there I go, I try to get some decent cards and start getting beaten up by human adversaries, I wasn’t expecting anything different, but I wanted to see some more strategies before working out a deck that might work against the computer but not against an human.

The online play is a nice thing, really cool especially since it’s WiFi-based so you don’t have to stay put while playing, and the stylus control in the game is well designed; but then I’ve started feeling bothered by the repeated “communication errors” I started getting during games.

First I thought it was truly random, then I’ve seen it happened mostly when I was leading the game.. I thought of a coincidence, but when it happened after I did win the game, the opponent coming to 0 life points… I was sure the thing was not that random.

Now, of course Konami can’t disallow people from closing their connection, or powering off the DS when they see they are losing, but… they could at least assign you the bonus points for the match, or at least the 10 match points you get for playing online, when the adversary fails to complete the game!

Even if I like the game, I’m tempted to ask Konami my money back, online playing is poorly designed.

Proceeding with xine

nospam@example.com (Diego Pettenò) — Fri, 04 May 2007 11:39:00 +0000

My changes to xine in the audio conversion branch are improving step by step; now all the decoders should playback fine, with their maximum resolution, by using the 32-bit floating point format as highest upscale. AC3 files are still a problem, although thanks to moesaji I now know that the mysterious function reduces the value basically by a value of 383 units; it’s still a mystery to me as I have no idea why that happens, and I can’t seem to find any documentation about these magic numbes for other stuff but a52dec.

Anyway, beside a52dec, most of other decoders are already fixed for the new interface, and I’ve started working on updating the ALSA audio output plugin; unfortunately the code of that plugin is ugly and complex, and seems to be a great big compendium of bad code style. With my changes in the audio conversion branch, the size of the initialisation function was reduced by 1KB, which is far from being small. It should also reduce the number of strings to translate, as I’ve used two loops rather than unrolling the tests for the capabilities of ALSA devices, which then condensed the number of strings to translate (they are composed during output, rather than being complete literals).

Hopefully the audio conversion branch will be merged into 1.2 series as soon as all the audio output plugins are converted, and a few more audio conversions routines are written. As the conversions are all limited to a single source file, it should also be quite easy to write more optimised routines using SIMD extensions for various architectures.

By working on this I ended up applying a few changes to 1.2 series itself already, by removing a few members from the decoder’s state structure for many plugins, and putting them into the functions they are used (why declare them in a state structure that is allocated in heap and passed, by pointer, everywhere, when they are used only in a single function? Sometimes even in a single branch of a given function.

The nice parts are that for instance NSF files now playback with a smaller memory footprint, as the buffer is allocated only when needed, and in the audio conversion branch also reduces the whole xine-lib memory footprint by two megabytes, this result means that to playback an mp3 file, xine-lib-1.2-audio-conversion takes half the memory xine-lib-1.1 takes! It’s incredible, but it’s true.

There are a lot more improvements that I hope to push into xine-lib-1.2, one of these is the splitting out of the modplug demuxer on its own plugin, so that binary distributions can just package it separately, allowing users wanting mod playback to actually get it, and saving the ones who cares not about mod files from having to initialise libmodplug every time (it takes quite a bit of CPU time, because of the calculation of the samples, I think I’ll post some images about this in the next days).

Oh and I’m actually thinking of reducing even more the code present in the decoders, by handling also the downmixing into the audio output loop (currently at least A52, DTS and Vorbis decoders contains the code to handle that) and by providing the code to handle also channels’ interleaving, as currently the same plugins also contain the same code, once again.

This again is to allow providing optimisations if possible, although I admit I’m not sure which kind of optimisation can be handled here. Still, it should be better to only have a copy of the code, rather than an undefined number…

The mystery of the a52dec decoded samples

nospam@example.com (Diego Pettenò) — Wed, 02 May 2007 17:48:00 +0000

I’ve been working a lot on the audio conversion code for xine-lib, with results that are always more satisfying, although Amarok fails to build on this new version, audio playback improved a lot, and with the code being all self-contained, the option of replacing the functions with more optimised code, using MMX, SSE, AltiVec or whatever other SIMD extension is more feasible.

Unfortunately I’ve hit one issue: beside A52 and DTS having a channel order different than most of other formats, I have an issue with the type of the decoded samples. There’s a function called float_to_int() that I supposed truncated the floating-point values to get 16-bit signed integer samples (that were the only format supported by xine before), but the floats themselves don’t play at all, while floating-point data plays fine for most other formats. I had to look up a lot of documentation to find for sure that floating-point samples, like I supposed before, varies in the interval [-1.0,1.0], so I was doing it alright up to now.

Now, if someone has any idea on what the following code does, I’d be glad to know it. The blah() function (with the same name) is present in xine, MPlayer, FFmpeg, and renamed in Ogle and other software too.

#include <stdio.h>

#include <stdint.h>



static inline int16_t blah (int32_t i) {

  if (i > 0x43c07fff)

    return 32767;

  else if (i < 0x43bf8000)

    return -32768;

  else

    return i - 0x43c00000;

}



int main() {

  uint8_t val_1[4] = { 0xFF, 0xFF, 0xBF, 0x43 };



  return printf("%f\n%d\n", *(float*)(&val_1), blah(*(int32_t*)(&val_1)));

}

The output (on Little-Endian, as val_1 is the byte representation of a float number in Little-Endian) is: 383.999969 and -1; it makes no sense to me, but maybe someone else has a clue..

And for who’s wondering, the byte-value I chosen there is one of the samples decoded by a52dec, so it’s a proper value I’m trying to understand; and for who’s wondering, the proper correspondence for -1 (int16) in float32 format is (approximated) -3.0517e-05.

Edit: thanks Nathan Smith who made me see the spelling error in the title (I still confuse the two).

Audio conversion flies!

nospam@example.com (Diego Pettenò) — Sat, 28 Apr 2007 16:01:00 +0000

After a week spent closing envelopes and attaching labels, I was finally able to get the audio conversion branch of xine-lib 1.2 to play mp3s.

As libmad decodes the mp3s as 32-bit signed integer samples, the direct output cannot be sent directly to PulseAudio when playing it back (and PulseAudio is the only audio output I converted to the new plugin API), so I had to actually implement at least some basic conversion, in this case from almost every integer format to 32-bit floating point samples (which is the simplest conversion, although it’s CPU consuming.

Previously every mp3 file decoded with libmad was truncated to 16-bit signed integer samples, losing precision, and for what I can see amplifying the volume quite a bit, distorting the signal, but with my new code, there’s no precision loss, and the difference is audible.

You can see the difference in the following screenshot:

If you look at the screenshot you will see the stream info dialog windows of xine-ui: on the left there’s the window for xine-lib 1.1 while on the right there’s the one for xine-lib 1.2 audio_conversion branch. As you can see at the bottom of the window, the first one reports 16-bit for the stream, while the new one returns 32-bit as intended.

I can tell you, the difference is great, and this is just the first step! Hopefully it will be merged in xine-lib-1.2 and then it will be able to play hi-fidelity wave files just fine.

Unfortunately before I can actually merge this branch into main xine-lib-1.2, all the output plugins, the post plugins and most of the audio decoding plugins have to be ported, which is a HUGE amount of work for just one person. I’d also require to use some virtual machines to confirm most of the output plugins work afterward, so it will take me a lot of time alone. If you can handle converting and testing the code, that will certainly be appreciated.

Self-signed certificates on the E61: failed.

nospam@example.com (Diego Pettenò) — Fri, 27 Apr 2007 18:16:00 +0000

I’m not sure if you remember, but last Christmas I blogged about my need to find a way to allow my smartphone (a Nokia E61) to automatically accept the certificate file for my IMAP server, to avoid having to accept it every time when I want to check my mails.

Well, I originally thought it was just a problem of format, as a lot of places on the net talked about the need to convert the PEM certificate to a DER certificate, and then download it with the internal browser from a properly-configured webserver (properly meaning the certificate is served with the mime-type application/x-509-ca-cert), but then it wasn’t enough, I thought it was a problem of size of the certificate (over 1024 bits it doesn’t seem to be supported) but it wasn’t the case either, and the last idea I had was that the problem was with the firmware version.

Unfortunately last month when I asked a friend of mine to help me updating the firmware on the phone, the download process on his laptop (with Windows XP) took more than three hours, then we shut everything down, as it was coming late, and the download wasn’t complete yet. I tried upgrading through a (demo) version of Parallels Desktop for Mac OS X, but it didn’t help either, today I tried on vmware-server, and there it worked fine, without the problems with disconnection and reconnection of the USB device that Parallels suffered from (probably while the device was being reconfigured to provide a different interface to check the phone model, serial and firmware). The firmware upgrade process went fine and easy, in about thirty minutes just like Nokia wrote on the site, I’m not sure what was wrong with my friend’s laptop, nor I care much at this point, if I have to be blunt.

But, not even the firmware update helped me.. so I decided to investigate further: the posts on the web about the configuring the E61 (or the USA-marketed E62, which should just be a crippled version of mine) mixes information about the addition of extra Certificate Authority certificates with requests for help with self-signed certificate. As soon as I considered this, light shed into my mind and I found the problem: the Nokia E61 does not support self-signed certificates; this is an absolute, and there doesn’t seem to be a way to get around this. What you can do instead is create your own Certificate Authority, load that certificate on the smartphone, and then use it to generate your own SSL certificate; this solved the problem for me entirely.

If you need an easy way to build a certificate authority certificate, and to create a new certificate signed with that, you might want to look in the openvpn’s sources’ tarball, that contains an easy-rsa directory with a series of scripts that really helps on that matter.

By the way, the reason why I updated firmware was also to enable the VoIP features, so that I can have a landline-like number routed on my phone while I’m at home (or anywhere where a WLAN can be found) allowing me to receive calls directly without passing through my family’s number.

usb.ids, is it still maintained?

nospam@example.com (Diego Pettenò) — Thu, 26 Apr 2007 21:09:00 +0000

Classically, usb.ids file can be found at http://www.linux-usb.org/usb.ids unfortunately there doesn’t seem to be any more updates after last December. I sent to Vojtech a few patches to add the devices I have here at home, but I didn’t get an answer (I know Vojtech usually answers in a day or two).

Since I hate seeing stuff getting obsoleted so easily, I’ve decided to add a bunch of devices to usb.ids, fix an entry (the one for APC UPSes, as now I have two of them to cope with), and maintain my own copy of usb.ids.

Most of the devices I’ve added are things that I have here at home, but they are not limited to, I’ve added some USB sticks of friends of mine who happened to pass by here, and an entry for a Microsoft mouse from Timothy. I suppose that while usb.ids is not fixed in the original location, I can continue maintaining it for my systems at least, by changing the cron jobs’ path to fetch from (it’s also pretty much useless to download the same usb.ids every month).

So my copy of usb.ids is located at http://dev.gentooexperimental.org/~flameeyes/usb.ids (thanks Patrick for the hosting, which I’m also using for a few other stuff like dist tarballs); if you want to add more devices to the list, feel free to mail me at flameeyes@gmail.com with a diff -u, or simple the vendor and product ids of the device, and the complete naming of it.

I know the hardware from the MacBook Pro is missing, I’ll see to add those the next time I boot the laptop on Linux.

Laptop policies on a workstation

nospam@example.com (Diego Pettenò) — Wed, 25 Apr 2007 14:02:00 +0000

As I bought the new UPS, I now have quite some time to work with it while I’m without power, as the two UPSes can keep the system online for about one hour without any kind of interruptions, included network access. This is quite nice, but there is one thing I was thinking of today…

When the workstations are running on UPS’s battery power, it’s comparable to a laptop running on battery, so I could be just using the same method laptops use: reduce the frequency of the CPU. At least Enterprise is able of CPU frequency scaling, so I’ve configured it to switch to power saving profile when the onbattery event is received by apcupsd.

It was actually quite simple, first of all apcupsd runs as root (although it should drop all privileges, but I’m not sure about it), and I have powersave installed and configured to handle this, so I just changed the /etc/apcupsd/onbattery and /etc/apcupsd/offbattery scripts to add a call to powersave binary, with -e Powersave on the onbattery event and -e Performance on the offbattery event; unfortunately I don’t think there is anything else beside slowing the CPU speed during UPS battery supply that can be done on a Workstation

Now I’m going to look to split the connection of the power cables in my home office so that the hardware that would just go in standby while I’m sleeping would be connected to a single multi-socket, which can be disconnected at once, to avoid leaving the hardware consuming power without any reason. I’d like to be able to do the same with the monitors (that otherwise would remain in standby as they can’t be shut down. If I wasn’t using the same UPS for both networking and monitors, I could probably use the UPS to shut them down and turn them back on at my command.

Sigh, I’m afraid there will be a looot of work to do, and I’m sealing envelopes, once again

A nice productive day

nospam@example.com (Diego Pettenò) — Sun, 22 Apr 2007 01:03:00 +0000

I might be sick, or just crazy, or both of them, but I still think I’m quite more productive when I have fever, or the days around that time. Yesterday I had fever, and I was knock out till late afternoon, but then I started feeling better, and I started producing.

First of all, rbot’s init script in my overlay has been updated: Subversion trunk will now create a rbot.pid file inside the bot’s directory without need for start-stop-daemon to create it itself, which means I can finally let rbot fork on its own rather than forcing it to background (which is not really a good thing to do, if it can be avoided). Thanks to tango and jbn for looking after my raw patch.

Then I decided to finish the work with apcupsd; again in my overlay you can find a new ebuild for 3.14.0 based on the one found on bugzilla, but with a new apccontrol file and a totally renewed init script. This script can be multiplied, which means you can have a /etc/init.d/apcupsd.foo link, which would then start apcupsd looking for /etc/apcupsd/foo.conf configuration file. Thanks to apcupsd authors for implementing this feature in 3.14!

I’ve also attached the two UPSes to Farragut instead of Enterprise, as the latter is not a server and might as well be offline when Farragut is still up (for instance this is the case most of the times I’m outside for the whole day, or if there is noone at home); apcupsd on FreeBSD works nicely, and doesn’t require any fiddling with configuration, neither kernel side (the ugen support is built by default) nor with permissions (as the default is to run as root, this might change in the future, but as it is it’s fine to me; I’ll be working on a better handling of permissions on device nodes for Gentoo/FreeBSD, but it’s not in my priority list at the moment). Also here, they work perfectly fine.

I was also able to fix a bug in xine-lib, with mp4/mov files playback that used version 1 rather than version 0 of the media header atom, such as files generated by FFmpeg. The bug was reported on sourceforge already but I wasn’t sure what it actually meant and where to find a sample file; when I generated the same condition by chance here, I decided to take a deeper look; unfortunately MultimediaWiki doesn’t provide much information about that, but I asked Mike to give me an account there so I can try to write something useful, maybe next time someone else needs a mdhd atom description they won’t have to look at the sources of FFmpeg to see how it’s read and generated.

Then tonight I wanted to resume my work on implementing audio conversions inside the audio output loop instead of doing it for every decoder; it’s an hard work as it probably will require rewriting a good deal of code, but it should be rewarding once it’s done. Right now there are a bunch more of flag values for capabilities, so for instance I can say if a drivers supports integer or float 32-bit samples, 64-bit samples, and if it can accept streams in a different endianness. This is important because there’s little point in doing the job of the output plugin, that might handle that transparently, for instance a big-endian stream might be decoded on a little-endian machine, then sent through PulseAudio to a big-endian machine where it will be reproduced: in this setup, xine’s endian reversal of the stream (from big to little endian) would have been superfluous, as PulseAudio would have accepted the big-endian samples, then sent them to the other machine that needed not to reverse them to reproduce them.

Anyway, right now the code is quite fragile, there’s no conversion being done, there are mostly only things that are totally broken out, there are asserts @1 == 0@ used to mark the code that needs to be rewritten. But something works: I was able to remove a lot of code from the musepack decoder, as libmpcdec always produces 32-bit native-endian (or maybe little-endian, I’m not yet sure) floating point samples; previously the decoder converted all the samples back to 16-bit format, and then gave it to the audio output loop to handle? now instead it sends them directly to the output, and as PulseAudio supports 32-bit float samples, they are not converted and play back fine.

Tomorrow I’ll see to work a way to handle upsamping and downsampling of streams, the problem is that it’s not trivial to decide what to do: if a plugin supports 32-bit integer samples, but not 24-bit integer samples, it should probably upscale the 24-bit to 32-bit to avoid losing precision; if it doesn’t it might upscale it to 32-bit float, or maybe downscale it to 16-bit integers. The same applies to channel mode, if the driver doesn’t support stereo output, should it be updated to 4.0 or should it be downgraded to mono?

For sure this time I’m very happy of being working on branches: leaving the code broken for weeks, maybe months, is not something you want to do on the main development branch. And I mean it, because with the changes I’m doing, not only I’ll be changing the ABI of the library itself (well, actually not much, just a couple of structures), but more importantly I’ll be changing the audio output plugins API, as I need to feed them a sample format rather than a bits-per-sample size.

Anyway, this is not going to be something easy to complete, but it will be a noticeable improvement for Amarok users once done, especially because I want to make sure that the capabilities for “mixer” volume and “PCM” volume are cleared up, probably by deprecating one of them, so that Amarok can be changed not to use xine’s software amplification (which also sucks and I also need to rewrite in good part in this branch) if the output plugin actually supports a per-stream volume (like PulseAudio).

Sponsoring, bribing, and comfort words are welcome, as xine-lib’s audio_out code is giving me creeps.

Downtime and new UPS

nospam@example.com (Diego Pettenò) — Fri, 20 Apr 2007 23:57:00 +0000

Sorry for the downtime guys, the update to baselayout 2 on Farragut wasn’t as easy as expected, I have some talk to do with Roy about it

Now, this is probably going to be the fourth night I’m not gonna sleep well, today I had a support request to take care of in the afternoon and I was supposed to go out with a few friends tonight, but I was so tired that I started having fever, so I had to get a raincheck for that.

The good news is that I received the new UPS, an APC SmartUPS 1000VA, that seems to be able to take enterprise and farragut up for 1 hour and a quarter, which is quite good (the other UPS would take the monitors and the network equipment up for also an hour, which is not bad, and should cover most of short-time non-planned outages.

Unfortunately, if you have followed me for a while, you know there’s no flawless hardware acquisition for me; this time the problem comes from the software needed to control the two UPSes; please note that there are two.

First of all, when I started using an UPS, I used apcupsd but then I moved to nut because it had a decent graphical control software (knutclient); what was the problem with this? Well, nut was confused by the two UPSes, that being both APC, shares the same identical vendor and product ID for the USB device. So it was not a nice thing.

I’ve then decided to come back to apcupsd, but the latest version is not in portage (again) so I bumped it locally with the patch on Bugzilla, and added a gnome useflag for the graphical control utility that is now present. This version is important to me, not only because of the utility, but also because with this version is possible to choose the configuration file at runtime, as it’s not hardcoded during build.

This is also important because in previous versions the only way to have more than one UPS being monitored on the same box with apcupsd required to have two different builds on different places. What I want to do now is to rewrite the init script so that it is multiplexed (like rbot, mt-daapd, openvpn and so on), that way I can simply start two apcupsd instances to have all I need (note that one of them won’t be shutting off my box at all, as it would just keep monitoring the UPS that is actually connected to the monitors and networking.

Once Baselayout 2 is more functional on Farragut, I’ll probably move apcupsd instances there, and network it down here, it would be safer on the long run as enterprise might not be up when farragut is.

I still haven’t been able to finish working on rbot’s changes I need, sigh. And don’t get me started on xine-lib, I’m taking a few days off because what I found myself working on now is a veeery bad thing I’m afraid…. (bit per sample transcoding during output).

About libdvdnav and xine

nospam@example.com (Diego Pettenò) — Tue, 17 Apr 2007 23:03:00 +0000

Don’t worry all you people, I wasn’t killed in a bus accident yet; you are lucky, I actually go out quite rarely so I don’t suffer from this kind of problems very often, the worst that can happen would be a health problem, but I’m lucky enough to feel decently well at the moment Of course there is too much heat for my liking and I start to feel how annoying it is at this temperature, especially staying in my home office with two/three boxes online.

Anyway, as I promised, I started looking at the fork of libdvdnav lead by Nico, to import a newer version of libdvdnav in xine, and cleaning up the patches applied.

Unfortunately it wasn’t exactly straightforward to update the libdvdnav version; beside my will to move the sources into the contrib/ subdirectory as I already did for libmpcdec and ffmpeg, Nico started to replace autotools with a build system alike to FFmpeg’s (which is quite fine by me, as I can integrate it nicely enough into xine-lib), so I had to make sure that the build system was up to a minimum support for the features I needed. The main showstopper was the lack of a way to build in a directory different from the original sources directory, but that was quite easy to handle.

But luckily working with Nico is quite enjoyable All the patch I’ve sent for inclusion up to tonight (so excluding the one I sent less than an hour ago) were accepted and committed to the Subversion repository, which means that the build system is usable from xine-lib’s custom base system, that the compiler flags we inject are respected, that the library builds with -Werror-implicit-declarations (the change was needed to libdvdread, Nico took care of sending it up there).

Tonight I rebuilt xine-lib-1.2 with the new libdvdnav, and it’s working nicely. I suppose the reason why external libdvdnav is not working on seeking is that the code was present in cvs (for sf.net’s dvd project) but not released yet. I should probably also add support for .pc files for libdvdnav and then use those to check for its presence, so to require at least a new enough version that doesn’t have seeking problems, but that is not a priority until a release is done.

There is only one problem that has to be considered: the libdvdnav copy in xine-lib is patched with a patch from Bastien Nocera (Totem’s author) which is used to play invalid DVDs (non-encrypted ISO filesystem DVDs; proper DVDs have UDF as file system). That patch as it is won’t be accepted by libdvdnav authors, and sincerely I wouldn’t have accepted it for xine-lib either.

Why this? Well, the patch takes an opaque type, and makes it transparent, copying out of libdvdcss the structure definition. This is fine as long as the structure is not changed on libdvdcss, but the reason why a type is opaque, is just so that you don’t have to put its definition in the ABI, so you can change it without having to deal with software crashing because it was compiled against the previous definition. This patch breaks this assumption with xine, so it’s bad from a good practices point of view.

I’m not sure myself how much sense does it make to use such a stupid solution (sorry Bastien, but re-declaring an opaque type is a stupid solution) to consider invalid media. A better solution would be to always use the files if they are found, rather than using UDF access, with fallback to raw UDF instead. This should work, but I need first some media to test with, and then I have to understand libdvdnav code.

Anyway, the build framework changes are now committed to the xine-lib-1.2-newdvdnav branch, the “only” thing missing there is the libdvdnav code itself; the reason for this is that up to tonight I was still working with patched sources, not with upstream sources, and I wanted to avoid recommitting everything every time. I will probably wait as much as I can before committing the sources themselves, with this “as much as I can” representing ideally the time till I get to have committed to libdvdnav all the changes that make sense for xine-lib (revised Bastien’s patch, and the file descriptor leak patch).

I also want to thank Nico for the work being done toward adding an option to use external libdvdread in libdvdnav, which will certainly help Gentoo and other distributions: you wouldn’t have to duplicate the code between libdvdnav and libdvdread; you’d have one more dependency in the libdvdnav library, but that shouldn’t be much overhead.

And I want to say it officially: there is a lot of code in xine-lib that scares the hell out of me, and I would like to see it killed before it propagates too much; unfortunately overhauling all of it is difficult alone, especially since I might end up having repercussions that I don’t see (like the CDDA failure that didn’t fail for me when I tested – I was using by-extension detection in Amarok – and the DVD failure that still I was unable to reproduce till at least since Darren fixed it).

I start hating spending my time on xine, I hope to be able to continue working on it till it’s enjoyable to do so.

More progress for XDG support in xine

nospam@example.com (Diego Pettenò) — Sun, 15 Apr 2007 00:03:00 +0000

Returning on yesterday’s blog entry, today I’ll see to update you on the status of xine-lib’s XDG support.

Thanks to Mark Nevill, I didn’t have to reinvent the wheel by parsing the various XDG_ variables to check for the directories I have to search in; he already wrote a libxdg-basedir library that takes care of most of it, allowing me to take care of the implementation details.

Now, xine-lib-1.2 branch has an internal copy of libxdg-basedir (two source files, so it’s not a big deal, and I’ve added a switch to use the external copy of it if needed), and uses it to decide where to read and write some files.

For instance, the plugins cache is no more in ~/.xine/catalog.cache but it’s in $XDG_CACHE_HOME/xine-lib/plugins.cache which both makes more sense and can be decided by user to be moved out of its home (for instance I change the value to XDG_CACHE_HOME, but that’s a topic for another post). The CDDB cache is also moved on the cache home, while the fonts are now discovered in the XDG_DATA_DIRS defined paths. Darren also moved channels.conf load from ~/.xine to XDG_CONFIG_HOME, which means it’s loaded in ~/.config/xine-lib by default.

Please note that I’ve been using xine-lib all over rather than xine, so that users know what is generated/handled by the frontends and what is actually part of the library, if they care.

There are still a few changes pending, for instance Darren wants to support loading of a system-level channels.conf, so that it can be put in /etc/xdg and used for all users without having to put it in xine’s configuration for every user.

I’ve also decided to pay more attention to the security side of xine-lib, for instance, after talking with Taviso today, I’ve added a xine_xcalloc function to wrap around calloc, which should avoid possible overflows (there was one in input_dvb); I’ve changed some of the code, but of course it’s not totally cleaned up. xine-lib really should be audited piece by piece for improvement, every time I touch something I end up cleaning it up by adding more const keywords (trying to let the compiler optimise a bit more) and adding/removing/cleaning up code .

One thing that certainly would help would be to put a better wall between contributed code (where we should always do the work on upstream’s side and then merge it back into xine-lib, to avoid getting them out of sync, unless fixing some xine corner case or similar—although there it could certainly help to put a proper wrapping around the two), and the xine proper code that should fixed/cleaned/improved as we go.

Really, while xine actually do its job most of the times, it still suffers from a lot of possible problems just because the code is too old and stratified. I think I should simply undergo an audit file per file trying to fix stuff while I also update the documentation to be doxygen-compatible, but that’s going to take so much time that I’m not sure how realistic is to work on it; beside I don’t have hardware such as a DVB tuner that would allow me to complete the audit (I can’t try that code and I can’t ensure its working state).

On the other hand, tonight I fired up Klothos again; it has been some time since I’ve done any Gentoo/*BSD work, but I’ve lately asked Roy to put me back in bsd@gentoo.org alias, to see if there is work that needs to be done that I can do. Yes, I suppose I’m considering coming back, but if I will do that, it will be on a lower profile; maybe I’ll help Mike, Frysinger, SpanKY and vapier with the ARM architecture, maybe I’ll just decide to take care of BSD alone.

The problem with Klothos is the PSU; the whole box is quite silent, as the UltraSPARC CPU is cooled by a slow and silent fan, but the PSU is annoyingly noisy, and I can’t just put a be quiet PSU like I put on Enterprise because of the Ultra5 casing (it’s a desktop machine, the case is high as the ATX power supply, which means the big fan used by most silent units would be obstructed by it); I could try a passive PSU, as the box shouldn’t be sucking too much power anyway, but I’m not sure about it and I don’t have money to waste (at the moment I’m unemployed).

If anybody has a suggestion about which PSU I could use for such a box, it’s really appreciated. It’s funny that I paid ?35 for a working box, and then paid way more to bring it to a standard worth to be a dev machine… well, I got the SATA controller already in my possession, as well as the SATA hard disk; the DVD reader was also an old one I used, but I had to buy a new ethernet card to avoid using the obnoxious hme0 driver..

Contributed code and FreeDesktop standard

nospam@example.com (Diego Pettenò) — Fri, 13 Apr 2007 13:28:00 +0000

Of course, this entry will talk about xine-lib.

First of all, a note about what I’m trying to do with contributed code in xine-lib for 1.2 series. As I said already, my ffmpeg_integration branch, which is one of the two branches I merged to get the 1.2 branch itself, was born to allow using FFmpeg without changing its buildsystem, and putting it in a contrib/ subdirectory instead of mixing it with xine sources.

Well, as I’ve now started working on providing also some Doxygen documentation, I’ve seen that this is useful also for other projects that are imported in xine-lib, so that I don’t have to blacklist one by one the directories and the files that are not to be parsed for Doxygen generation.

In xine-lib-1.2 hg tip you can now find libmpcdec (that replaced libmusepack) in contrib/libmpcdec/ rather than having it together xine’s actual code (the decoder plugin code) inside src/libmusepack/. I also updated it to the last version available.

Similarly I hope to move other code, like libmad and of course libdvdnav, that was recently taken over by MPlayer developers because of the unresponsiveness of the original development project.

It will take some time to complete all the moves, also because I’ll try to contextually update the code with the last version available for every project beside libmpeg2. This should be a breath of fresh air for xine 1.2.

Talking about FreeDesktop standards, as Darren already changed gxine to abide to XDG Base Directory Specification, he also added better FreeDesktop standard support in xine-lib for configuration and cache files. So right now I’m working on making xine-lib use XDG_CACHE_HOME to store the plugins cache instead of the hardcoded ~/.xine/catalog.cache, which also allows me to move more cache data out of my home directory

By the way, why one earth the description of the permissions to use while creating directory (that is actually a quite logic 0700) is written in the «Referencing this specification» section ?

Discarica abusiva

nospam@example.com (Diego Pettenò) — Thu, 12 Apr 2007 16:06:00 +0000

Non so che altro titolo dare a questo post, se non, appunto «Discarica Abusiva», perché di questo tratta.

Mi è venuto in mente di scriverne dopo aver parlato l’altra sera con X-Drum su #gentoo-it su Freenode, visto che stavo informandomi su dove portare a far smaltire alcuni rottami che ho qui in casa (la vecchia lavastoviglie e un gruppo di continuità totalmente partito; alla fine domani chiamerò la società che si occupa dei servizi ambientali per questa zona, Vesta, per fissare un appuntamento per il ritiro, era così semplice, ma non è un fatto conosciuto che questo ritiro è pure gratuito).

Beh, non si tratta di una novità; quando ancora ero in seconda media (quindi ormai circa 8 anni), sono iniziati i lavori necessari alla costruzione delle fognature comunali per la zona dove abito (Santa Lucia/Tarù, alla periferia di Zelarino, entroterra di Venezia, Mestre). Beh per qualche motivo dopo due o tre anni di lavoro, la prima impresa è stata mandata a casa, e il cantiere che avevano aperto a non tanti metri dalla rete del mio giardino, in mezzo ad un campo, è stato posto sotto sequestro. È arrivata un’altra ditta che ha aperto un altro cantiere di fianco, ed è andata avanti con i lavori fino a non molti anni fa.

Beh, visto che il primo cantiere era sotto sequestro e intoccabile, e il secondo cantiere aveva gente che andava e veniva tutti i giorni, la sbarra di accesso al campo è rimasta aperta per molti mesi, e dove c’erano i container, rimasti per anni incustoditi e intoccati (solo poco prima che il secondo cantiere finisse sono stati spostati due camion ribaltabili che erano rimasti là, carichi di benzina pure, in mezzo ad un campo arso dal sole tutta l’estate), si è cominciata a formare una discarica abusiva.

Le foto, cortesia di un mio amico, Alberto Chinellato, che ha collaborato in passato con il quotidiano La Nuova Venezia, e «caporedattore» di un giornalino di zona, sono tutte da vedere:

Una visione d’insieme della discarica.

Se andate su Flickr trovate anche le altre foto panoramiche.

Ora, non è che sia difficile da vedere, visto che si vede dalla strada, non è difficile da sapere dove si trova, se un cantiere è sotto sequestro, ci si aspetta che le forze dell’ordine sappiano dove sia, e soprattuto è cosa nota almeno alla Polizia municipale (i vigili urbani) visto che sono stati contattati da me e da mia madre più volte, e sicuramente anche da altri abitanti della zona (anche se noi siamo effettivamente i più esposti essendo a occhio meno di 100 metri dal nostro giardino).

Per circa due anni l’erba nel campo non è stata tagliata, e come potete vedere dalle foto, tutt’ora ci sono piante ed erbacce che crescono da quello che è rimasto, fornendo un ambiente ideale per pantegane e altri animali non propriamente salubri.

In aggiunta, poiché fino a qualche mese fa la sbarra era ancora aperta, la «stradina» di accesso al campo era diventata una meta nota di auto che cercavano un posto per infrattarsi (e vi assicuro che la zona di romantico non ha assolutamente nulla quindi traetene voi le deduzioni). A tal proposito, Alberto ha anche scritto un articolo sul giornalino di cui sopra:

La copertina del Quattrogatti (giornalino locale) con l’articolo di Alberto riguardo alla discarica.

(in aggiunta potete trovare una scansione in DjVu ad alta qualità per la lettura direttamente sul mio server – sorry ma questo era l’unico formato che avesse una resa accettabile, visto che non ho il sorgente originale da convertire in PDF).

Ora, dopo otto anni che non succede nulla, vediamo se la rete può aiutare ad avere una risposta fattiva nella soluzione di questo problema… specie perché ci sono già altre discariche abusive qua attorno di cui nessuno si occupa.

A little Summer of Code analysis

nospam@example.com (Diego Pettenò) — Thu, 12 Apr 2007 08:42:00 +0000

So today the allocated projects for the various organisations accepted in Google Summer of Code were released, and I decided to skim through them to see if there was something interesting; especially with xine-lib in mind.

First, FFmpeg projects, as FFmpeg is for good or bad xine’s heart, and improvement in it is certainly going to do good to xine project too. There are four decoders proposed and accepted: RV40 (Real Video), E-AC3 (used in some kind of HD media if I remember correctly), Dirac and QCELP. I sincerely never heard of the last one, and of Diract) I heard only as an experiment from BBC; I prepared the ebuild for the Dirac library when I was trying to support it in VLC, but as there was not a simple way to test it I left it masked in the tree for a while; the package is still there, masked, but someone else probably took it over; with further experience, I should have put it in an overlay.

While really interesting, RV40 and E-AC3 doesn’t sound that appealing for the target of xine users; first the Real demuxer needs an overdue overhaul, second E-AC3 only makes sense if we also start supporting HD media, which I don’t think we’d be doing soon enough. I was hoping for a Monkey’s Audio decoder, but that was not the case neither this year.

Users interested in having Monkey’s audio support in Amarok and other xine-based players might consider the idea to start a Chinese wall reverse engineering for it from the macport library, also if some description is present already, and might be enough to start working on it; I’m not keen on going to do it myself though; beside having other things to do, I don’t really know where to start at the moment.. bribes might make me reconsider but remember that almost anything I do goes to help in some way…

Now instead FreeBSD has interesting ideas, I’m quite interested in support for the Apple’s MacBooks (hoping that they will also consider MacBookPro), as I’m an user for it; the TCP/IP regressions testing might also help, as there has been a few TCP/IP problems in the past that might get solved once and forever with this, but what is really interesting is the bintools project to replace parts of binutils with BSD-licensed variants.. interesting not much for its usefulness, but just to see the mess when you add bintools’s commands to binutils’s and elfutils’s … reinventing the wheel, year after year.

For what concerns KDE I think we’ll see the results only in a few months, but I’m happy to see that there are projects to improve Kopete’s MSN and Jabber protocols. I just hope that in KDE4 Kopete is going to get more maintenance and won’t be allowed to fall behind every year to an obsolete state (Jingle support was nice… but it was left incomplete and never update up to now!). And I don’t intend to forget Mike’s project on Kontact’s blogging support.. that will be neat for a blog-addicted like me!

For X.org, the server-side XCB support is something that I’ve read about for a few months now in the xcb mailing list and seems like it’s going to be quite useful to reduce the possibility of a mistake in the X server code, which is something we all rely upon nowadays (by the way, since X.Org was founded, using X started to be less a pain that it was before, the no-configuration startup is nice, too bad xorg.conf still uses that obscure format ).

When it comes to GCC instead, the projects are more technicalities, but there is at least a speedup project that all Gentoo users should welcome easily, and the SEH support (that by the way comes from a friend of mine, hey Hyp if you’re reading! ), that will be one less reason not to use MinGW32 rather than VC++.

And not directly related to xine, but coming from one of xine’s contributors in the last months, there’s a project for Wine to support Solaris.. I wish him good luck, as that might make wine’s code even better, as usually portability helps cleaning up after yourself.

Okay now let’s hope these projects will all be completed during Summer of Code and maintained for a long time afterward!

Music overdose?

nospam@example.com (Diego Pettenò) — Wed, 11 Apr 2007 06:17:00 +0000

Finally, yesterday I received the package from Amazon JP!

Let’s ignore the fact that on a ?110 order I had to pay ?41 for custom duties (oh well, the cost of a single CD I ordered in a shop around here I asked before was way higher anyway), and I’ll also try not to repeat too often the ridiculousness that DHL Express was able to ship the package in three days from Narita to Marcon (Venice) and then it took three more days to ship from there to my home (because last Saturday they didn’t deliver, and the package was received there just Friday evening); all things considered, four working days is not a bad delivery time.

Now as crazy as I am, I do like some Japanese music… I started loving L’Arc~en~Ciel music when I heard Driver’s High as opening for Great Teacher Onizuka anime. I was able to buy myself Awake some time ago, and yesterday I received ark in the special 15th anniversary edition, CD with DVD, so I was more than happy!

Now this doesn’t mean I passed the whole day listening to music, eh.. I was actually working for part of the time on finishing the details of my (now expired) job, then I took a shopping tour to a bookshop in Mestre, looking for something from Dario Fo (with scarce results, I was able only to find a single book of him), but of course this without forgetting working on xine; as I said, mercurial is something useful.

The 1.2 series, on which I’m focusing at the moment, is now spotting a cleaner support for packed attributes, and I’m working on adding Doxygen documentation to the code; currently I’m converting documentation in the buffer.h file, just because it’s the one I found most difficult to understand myself.

Unfortunately, emacs support for Doxygen seems to be pretty much non-existant, and even the doxymacs extension does a lousy job, as it fails to highlight the Doxygen comments properly. Add to that that emacs insists on putting two spaces in front of every line after an extern "C" { block (that has to stay there as it’s a public header, although nowadays I think it should be C++ code that should use extern "C" { before includes, rather than polluting the C header files), and you can see why I haven’t converted the whole xine-lib yet.

Unfortunately, I’m not yet sure on how many of the improvements I hoped for I’ll be able to implement in 1.2 series, considering that I’d hope for 1.2 to be released more or less with KDE4 release (before if possible, if not possible not too much afterward), as the 10MB improvement in memory usage is a killer feature.

I’m also trying now to handle a few things in a slightly different way, for instance I’ll be trying to move all of the contributed code in the contrib/ subdirectory, rather than having it scattered among xine proper sources. The problem is that having to deal with both build system and code changes makes quite more difficult to track down the changes, but this should not be a big problem on the long run.

And for who’s curious, yes, I also have an Amazon JP wishlist but it’s mostly for my own use, as the customs duties are a problem.

The other big memory waste

nospam@example.com (Diego Pettenò) — Mon, 09 Apr 2007 20:11:00 +0000

Tonight I finally started the true 1.2 series branch on xine-lib’s repository; this branch is the merge of the nopadding and the ffmpeg_integration branches, which means it starts with two big improvements: 10MB memory saving and a simpler to upgrade FFmpeg snapshot, with handling for disabling uncommon decoders, just like Miguel wanted.

Now, if you looked well enough at the graphs you posted, you already know what I’m talking about, if you didn’t, well here there’s a new graph, created using the -pq option at xine so that the logo is not displayed once the mp3 finished playing, removing the spurious spike at the end of the graph.

The yellow are you see it’s the memory created by _x_fifo_buffer_new called by _x_video_decoder_init function. That’s right, there’s a huge amount of memory allocated by the video decoder initialisation function when playing an audio only mp3.

Okay, so what the problem is here? First, as the code cannot tell if what is playing is audio only or contains video too beforehand, it has to initialise the video driver every time too; this is quite normal though, nothing to be a problem.

The problem is that the video initialisation initialise automatically also a number of buffers, by default 500, of size 8KB.. even if they never gets used. It also allocates a single memory area for the buffers, 2KB-aligned, which is not exactly the cheapest way to allocate memory.

Now, this in general should be okay, shouldn’t it? Most of the times you’re playing video with xine, and so it should be fine, but there are cases when using this method will just waste a lot of memory, especially if the user tried to be smart and increased the number of buffers to something that isn’t exactly sane…

I’ll now be working on checking if it’s possible to just allocate the buffers when they are needed; in that case the graph will certainly appear different, as the memory will increase less steeply, and would just run down together at the end.

Let’s return to work (although I should be finding a way to clean up the mess that is my home office here)…

Spring cleaning in your $HOME: spamassassin with SQL backend

nospam@example.com (Diego Pettenò) — Sun, 08 Apr 2007 13:32:00 +0000

This is going to be the first of a series of posts about «spring cleaning» of your home directory. We’re also in the right season, so I’m not that Off Topic for now

Why do I care about having a clean home directory? Well, it vastly depends on my setup, but I think this is common enough to grant some discussion about it. I have my /home in a partition that is set up with DM to be replicated on my two harddrives, providing me a basic RAID1 setup for that single partition; this allows me to be relatively safe from a harddisk crash, for what concerns my important data, like SSH and GPG keys, configuration files, mail and so on.

The problem with this is that everything that gets written to my home directory has to be written on two disks, and is often a performance drawback; for this reason, I tend to scatter the non-essential data (like repository checkouts and similar) in different partitions, as they also don’t require much backup most of the times. This also brings me to hate the software that uses my home directory to save cache data, because it ends up using RAID1 for disposable data that I wouldn’t want to have backed up together with really important data.

So, this series of posts are going to explain how I try to keep my home directory clean from cache data, in part to help someone else that might want to do the same, in part for me to remember how and why I did something

One of the first services that I thought of, using data in my home directory, was spamassassin; while the amount of spam mail I receive has now decreased a lot since I left Gentoo (as I’m not in 10 aliases), I still receive quite a bit, so I’m not yet ready to remove my local SpamAssassin filter; it’s probably a sane idea especially since for xine-lib I’m going to repeat my email address over and over at every commit

SpamAssassin saves some data in ~/.spamassassin, namely the bayesian tokes database, the automatic whitelist and your extra preferences. As I don’t have extra per-user preferences (I use SpamAssassin in a single-user environment), I don’t need those, but I do need bayes and awl to work. Since I already have Amarok using PostgreSQL in this box, I decided to use PostgreSQL to also save SpamAssassin data.

Unfortunately, as it is the ebuild does not allow you to easily add postgres support, but this is probably going to be fixed in the future; I have a better ebuild in my overlay ( git://flameeyes.is-a-geek.org/overlay.git ) and I’ll see to send the changes to Perl team now; in the mean time, the things to change are not that much.

The documentation on setting up SpamAssassin with SQL backend can be found on SpamAssassin Wiki, and it applies to PostgreSQL as well as MySQL, even if some things has to be changed around, nothing major though.
During this post I’ll assume that both PostgreSQL and SpamAssassin are only reachable on localhost, and that you don’t need extra security concerns like a password to the database or something like that.

First of all, stop SpamAssassin (if your mail system is not mission critical) and start backing up the bayesian database:

% sudo /etc/init.d/spamd stop

% sa-learn --backup > sa-bayes-backup

This will create a sa-bayes-backup file with the bayesian token currently saved in your home directory in a Berkley DB file.

After this, change the useflags for mail-filter/spamassassin: disable the berkdb useflag and enable the postgres useflag; ignore the warning currently thrown by the ebuild that the bayesian filter needs the DB_File module, it works just as fine with PostgreSQL as backend, but you have to configure it. You might also want to enable the doc useflag, as right now it’s unfortunately controlling the installation of user-serviceable documentation; in alternative, just get an extracted copy of SpamAssassin’s tarball to use as a reference.

Now, it’s time to create the user and the database to store the data into.

% sudo -u postgres -i

postgres % createuser spamassassin

postgres % createdb -O spamassassin spamassassin

postgres % bzcat /usr/share/doc/spamassassin-3*/sql/bayes_pg.sql.bz2 | \

  psql -U spamassassin spamassassin

postgres % bzcat /usr/share/doc/spamassassin-3*/sql/awl_pg.sql.bz2 | \

  psql -U spamassassin spamassassin

You could also use per-user preferences stored in SQL backend if you really need them; as I don’t need them, I instead edited /etc/conf.d/spamd replacing the -c option (which forces spamd into creating per-user configuration files if missing) with -x (which says to spamd to ignore per-user options, that is just what I need.

Now it’s time to set up the database connection from SpamAssassin; although the ebuild suggests to use the secrets.cf file, that is not readable by users, to configure the connection to the database, if you plan to use sa-learn from your user, you might prefer to just enable it in a world-readable file, especially if you don’t have any security concerns on the use of the spamassassin PostgreSQL database; this is what I have done anyway:

bayes_store_module      Mail::SpamAssassin::BayesStore::PgSQL

bayes_sql_dsn           DBI:Pg:dbname=spamassassin;host=localhost

bayes_sql_username      spamassassin

bayes_sql_override_username     spamassassin



auto_whitelist_factory  Mail::SpamAssassin::SQLBasedAddrList

user_awl_dsn            DBI:Pg:dbname=spamassassin;host=localhost

user_awl_sql_username   spamassassin

At this point, SpamAssassin will only use PostgreSQL for its databases, so you can just remove your ~/.spamassassin directory, it will not be recreated. Let’s then start PostgreSQL (or make sure it’s started already, and then restore the Bayes database:

% sudo /etc/init.d/postgresql start

% sa-learn --restore sa-bayes-backup

Now you could restart spamd and have your system back already, but there is one problem with the current ebuild (the one in my overlay does not need this change though): it does not depend on PostgreSQL. From one side it’s correct, you might not be using the localhost pgsql to store the data, so in that case you don’t have to care to start spamd after postgresql, but if you’re going to use a local configuration, you certainly don’t want spamd to start before the PostgreSQL database is up, so you have to edit the /etc/init.d/spamd script, and add in the depend() function, a simple use postgresql line; add postgresql to your default runlevel, and that should be it.

At this point you’re set, just restart your spamd, and it won’t use your homedirectory to store cache data anymore!

% sudo /etc/init.d/spamd start

Integrating FFmpeg in xine-lib

nospam@example.com (Diego Pettenò) — Sat, 07 Apr 2007 21:04:00 +0000

Beside the ABI breakage that is needed to reduce the structures’ size, that as I blogged yesterday helps quite a bit, another of my currently worked on branches is the FFmpeg integration.

Now, xine-lib has used FFmpeg since ever, and it has an internal copy of it (although on Gentoo you use the external copy of it to reduce the size of the code, and avoid having double security problems), but that copy requires rewriting the build system from FFmpeg’s own makefiles to Makefile.am files, which is far from trivial and quite error prone.

It also requires to clone in configure.ac the tests for the features that are tested in FFmpeg’s own configure script, which is a waste of time when using external FFmpeg most of the time, and needs to be maintained over the long run as they might be modified by FFmpeg developers, and we can’t just copy them 1:1 as they don’t use autoconf either.

To solve this issue, I’ve been working before to implement a sort of Chinese Wall, so that FFmpeg’s build system could be called from xine’s automake build system without need to accommodate the Makefiles every time an upgrade is needed.

It wasn’t that difficult to begin with, but there was an obstacle with the make dist command that is used to create the distribution tarball, as there was no way to seemingly merge its processing with FFmpeg’s build system; today I then implemented a dist-hook that takes the list of files to add for FFmpeg support to the tarball. It requires an extra step during the update of the internal copy, but that’s far from being a problem, as it’s certainly more straightforward than it was to update the previous FFmpeg copy.

With this problem fixed, I was finally able to start working on getting distcheck working and thus trying to get a working branch out of it. The result is quite good, as I was able to get a distcheck running already, and I’m now working on actually implementing Miguel’s idea to disable the “uncommon” audio/video decoders present in FFmpeg; his work on the current 1.1 series is basically half-implemented, and probably never to be fixed, as it would require to change all the Makefile.am so that they take into consideration the huge amount of conditionals currently present in FFmpeg; I think once it’s completed I’ll just ask Miguel to give up on having it working in 1.1 series, as it’s unlikely to ever work decently… well, it might even work, but refreshing FFmpeg would require a huge amount of work, and we are understaffed already.

And by working on this branch, and on this idea, I already discovered two bugs in FFmpeg that I’ve locally patched.. the patches are now waiting (as usual) in ffmpeg-devel to be applied.

Hopefully, this will also be integrated in 1.2 series, that should then start being interesting…

The first improvement in xine with Mercurial

nospam@example.com (Diego Pettenò) — Fri, 06 Apr 2007 22:01:00 +0000

So, after xine finally moved to Mercurial for xine-lib management, I’ve decided to start working on those things that required me to branch out, at least on some of them that is; the first one I was able to tackle down was ffmpeg_integration, that now works fine beside the dist target.

And then I moved working on the structures, applying pahole to all the structures in libxine, even those comprising the public ABI of the library, as I could just break the ABI when needed, rather than limiting to the local structures of the plugins. Some of these changes applied to structures that are not part of the public ABI, so I ended up merging them to the main repository already, and will be present in 1.1.5, even if they are mostly bytes-size changes, that nobody beside me should care about.

But then tonight I gone looking for the 32 buttons that are/were present as an inline array in one of the video overlay structures; I was going to change the inline array with a dynamic array or with an array of pointers, so that the memory was going to be used only when actually used..

It was a sour surprise to find out that the array was never used at all in the code, and it wasn’t used on frontends either, and by removing it, the size of the structure in which it was dropped from 86KB to 40 bytes.. and then the video_overlay_s structure dropped from over 4MB to about 3KB… finally with it removed, there was also a 10MB of memory usage cut down during xine-lib runtime playback, 1/3 of the memory usage when playing an mp3 file:

(before the change, about 1.1.4 release)

(after the change, on my nopadding branch)

For starters, this does seem quite good, don’t you think?

Conversion to Mercurial done

nospam@example.com (Diego Pettenò) — Tue, 03 Apr 2007 10:28:00 +0000

So, as of yesterday, thanks to Darren Salt and Reinhard Tartler, xine finally moved out of CVS dark age, and came into Mercurial’s territory.

As you can see on Debian’s Mercurial repositories page we have quite a few xine repositories: together with xine-lib, Darren also moved his gxine repositories so they are all together.

I’ve already started doing some little work, which shown a little bug on CIA.vc yesterday that was ignoring my commits; fortunately Micah corrected the problem right away.. thanks Micah!

One of the tasks I was able to take care of today was the update of the FFmpeg integration branch, that now is up to date with main xine-lib-1.1 branch. Unfortunately Binutils seems to have broken FFmpeg so I’m unable to finish building and thus checking if it actually works as intended.

Anyway, soon I’ll see to post more updates about it.

Pahole and xine-lib

nospam@example.com (Diego Pettenò) — Sat, 31 Mar 2007 16:18:00 +0000

So, I’ve taken two days totally off from almost any kind of communication, I tried to relax, and now I feel a bit better. I still do think I haven’t been able to do much good beside my work on Gentoo, but I’m not ready yet to give up, even if it’s hard to continue, I will continue. At least for a while.

Unfortunately my current UPS setup is not going to fly, X-Drum in #gentoo-it informed me last night of the incompatibility between PSUs with active PFC and UPSes with stepped sinusoidal wave.. and the new PSU I bought two months ago has active PFC. The result is that I need a new UPS, of the Smart UPS series from APC, which will cost me ?420, sigh.

Talking about the topic in subject, two days ago I analysed most of the xine-lib plugins with pahole (with a patch to fix an integer overflow in the offset counter (the author wasn’t expecting structures bigger than 64KiB, but this in xine-lib is not rare), and I do have at least one good news: FFmpeg decoding of non-mpeg video streams was taking 1MB of memory for a libmpeg2 buffer that was not going to be used; I’ve now fixed this so that the structure is only allocated and initialised when needed, so decoding will take 1MB of memory less than before, on next xine-lib release.

Unfortunately, I’ve found similar mistakes in design in other structures, most of which are public, so part of the libxine ABI, and thus I can’t fix them in the 1.1 series, not unless there are good reasons and good results to achieve by breaking it. But, next week we’re going to move to Mercurial, thanks to Darren Salt and Reinhard Tartler who are helping with the migration (for who’s wondering about hosting, it will likely be on Alioth, if they accept us), so I can branch out and fix the stuff.. and then merge back in either 1.1 or 1.2 as we feel needed.

One of the structures I surely will be refactoring is the video overlay structure.. that has a size over 4MiB as it is, which explains why the function to initialise video overlay consumes 5MiB of memory right after xine initialisation, even when playing a sound file. By instancing the structures only when really needed, and making sure that there aren’t holes around, it should be possible to reduce drastically the memory used up by xine.

Another thing in my TODO list is, as I said already, rewriting the plugins cache code. I will also try to provide a simple way to regenerate this cache globally, so that for instance it can be installed directly by the ebuild, without asking users to regenerate the cache by theirselves, and sharing it in memory (through mmap) between users too.

To help this, I’ll also see to change the way the plugins are handled, by using where possible inline arrays for the names and description of the plugins, rather than pointers, allowing to share the structures in memory, where this does not waste too many bytes.

Anyway, I still need to relax a bit more because I can’t really rest lately, and I do need some rest if I am to carry on.

I will say "I told you that^Wso"

nospam@example.com (Diego Pettenò) — Wed, 28 Mar 2007 10:34:00 +0000

Edit: thanks to Mike Arthur for letting me remember I got the phrase wrong, the permalink will have to stay wrong to avoid confusion though :/

So today’s news are that a bunch of people who never seemed to care about ALSA (sorry Tony but iirc you said that yourself) now will be the new ALSA maintainers. And their first action is to flush down the toilet the months of work I’ve put in making sure that the official alsa-driver package works.

Okay, well, their responsibility.. but again, what about those people who need to use alsa-driver? While Josh (nightmorph) agrees that it’s more than 2.4, Daniel seems to care about it just for 2.4 and those few extra drivers on it. Tell you what: neither my boxes can run in-kernel ALSA, as they both die with it.

And has any of them tried to contact me? I told to them already that the ALSA maintainer’s guide is incomplete, were they able to fill the gaps that I haven’t been able to update before leaving without asking? Maybe.

But with all due respect, Steve wasn’t able to find time to proxy-commit the ebuilds I already prepared for ALSA, Petteri did it.

Do they know about the six bugs that I’ve opened on the ALSA bugs tracker that are patched in Gentoo?

For the way it has been handled, I do think Daniel is just trying to get revenge for my handling of ALSA, sorry Daniel, I respect you, but you’re dead set on the wrong way to see the issue this time. And I’m pretty sure that it will create problems.

I don’t think I handled ALSA all that bad, I didn’t get any help by any other developer, and Daniel tried to push me off track once or twice, but there weren’t many users complaining about ALSA not working… while we all know what happened on the 1.0.8 to 1.0.9 update (or do we? rather, do they? I was there, not sure where they were on that one); genstef failed already once trying to take over ALSA from me, by the way, and if it wasn’t for my proxied commit, ALSA 1.0.14_rc3 wouldn’t have been in the tree most likely (released 6 March, my ebuild was ready on 7th, Petteri committed it at 17 March after leaving some time to see if someone else was going to pick it up).

Anyway, with the future in hands of those devs, I think I won’t restrain myself from saying “I told you so” when, in a couple of months, you’ll hear users complaining about their VIA82xx card or their HDA card not working. Bets are open on when the first signs of something broken will appear.

Oh, someone in there is going to actually take care of what I left incomplete (ld10k1 init script), or is that not cool enough?