Amarok Blog - Flameeyes

My fiddling with OpenJDK

nospam@example.com (Diego Pettenò) — Wed, 09 May 2007 21:38:00 -0400

It might sound strange coming from someone who wrote a post titled The Java t..crap but I’m liking playing with OpenJDK.

Not like my Java skills (pretty much non-existant, I used it only a few times in my life) are improved, but I find myself comfortable when working with most build systems, after you get scons out of the way (note that I didn’t say I like it, but I can do it just fine), and I do have experience with adding support for external libraries, after all, and that is something that OpenJDK might need.

So, let’s start from the start. The first problem was getting to build OpenJDK on Linux. If you try to build it on AMD64 you’ll see it fails, the reason for this is that the list of files to compile on hotspot is too long for the maximum size of arguments available on Linux; this in turn is caused by the long default path for sources that Portage sets. Petteri was able to build it fine because on x86 the architecture name is “i486”, while on AMD64 is, well, “amd64”: the extra character there is enough to make it burn in flames. Kelly O’Hair from Sun is looking after the issue now, but he says it might take a few weeks to get the results out; not a problem, I suppose, in the mean time you might want to try changing PORTAGE_TMPDIR to use /tmp for openjdk (if you have space in the /tmp partition) or you might want to fool the ebuild by renaming the openjdk directory inside $WORKDIR to something shorter, like “ojdk”; actually, a single character less will work just fine.

After getting a working build of OpenJDK, and trying so that it worked fine on Konqueror (it does!), I had to start messing with it, or I wouldn’t have been myself The first thing was to get it to build with my flags correctly; I was able to get something that worked, but there are a few issues, and I’d like to be able to provide a less invasive patch. O’Hair also got me on the right track now, by telling that hotspot and j2se build systems are different, which might call for just using the same method on boths, by ignoring the default C(XX)FLAGS values coming from environment, and instead use the _OTHER variables. This should allow for just four lines changed, if I find where to put the CFLAGS/CXXFLAGS unset.

After this I started thinking of an useful hack to do to at least understand how the build system works; I could have looked at fixing a few warnings, but I thought it would have been more useful to actually start by looking at the build system, that is what I will need to understand to be able to get OpenJDK building on FreeBSD. I’ve seen zlib-1.1.3 sources in the tree, so I decided to find a way to build against external zlib. Easy, quick, interesting and useful.

Why useful? Well, first of all, it would use a more stable version of zlib, and would share its code with the rest of the system (almost every process has a zlib copy loaded one way or another), second of all that would make it non-vulnerable by possible attacks on the zlib code. Unfortunately adding the support for external zlib was not as easy as I thought, but it was mostly because I stupidly forgot FEATURES=keepwork on the environment (I wanted to get the object files output, as I want to see if the executable stack problems that affects Java are due to the architecture, or just a problem of not having the GNU stack markings on the source files).

And while looking at fixing this, and seeing a tempnam() warning at linking stage, I ended up finding a tiny memory leak, as the string coming from tempnam() is never freed. I’ve prepared a patch and sent it, hopefully to the right mailing list.

Other hacks that might be useful would be finding a way to enable or disable ALSA and Motif bindings building, to reduce the size of the final output, and to avoid depending on those to build openjdk, but I wonder if Sun would ever accept such changes.

When I’ll be comfortable enough with both build system and development environment, I’ll see to start working on building OpenJDK on FreeBSD; for what I can read on the build system mailing list, it should be possible, for now, to build OpenJDK using a JDK 1.5, which is available for Gentoo/FreeBSD, even if the best thing would be using 1.6. Probably I’ll have to provide a openjdk-bin package for Gentoo/FreeBSD to use as a seed, afterward.. this of course if the license allows to do this (most of OpenJDK is GPL-2, but there are some binary blobs still present, until they are replaced, it might not be possible to redistribute the binary itself).

Unfortunately a big problem I have to face while hacking at this is that I don’t have any Solaris/OpenSolaris installation, and I suppose that Sun will accept more willingly contributions that don’t break their main target platform

Now, of course, the reasons I have to find this project interesting and to revert my attitude toward Java are not limited to the simple GPL-2 licensing. What I find more attractive, and makes me hope I’m not mistaken on my judgement, is that now it would be possible to support more hardware platforms, making Java a true platform agnostic language (okay, Kaffe help a bit with this, but the results are not really at the same level), and will also allow to support more operating systems. And with more users’ contributions, it will probably also get better optimisations (hey we got guys like FFmpeg devs who are able to optimise the hell out of multimedia codecs, someone might find useful things to improve on OpenJDK too!).

For what concerns Gentoo/FreeBSD, for sure FreeBSD guys have already some build definitions to support FreeBSD; the problem of those are that are likely not GPL-2 as it is, as they were designed for previous JDK versions, so I’d rather not even look at them to avoid legal issues. The assembler code would likely be the same between Linux and FreeBSD, with the same architecture; for SPARC, the code might be just taken out of the Solaris assembler sources, at least on i486, the assembler between Linux and Solaris doesn’t change besides from a few minor places, and the main issue there is the difference in the assembler program used, and thus in the syntax.

Let’s try to make Java a truly Free platform, and allow it to run on as many system as possible, then it might replace Flash and beat the heck out of Silverlight; Sun seems to be in the right mind set… so waste no time!

Avoid SpamAssassin 3.2.0

nospam@example.com (Diego Pettenò) — Wed, 09 May 2007 18:36:00 -0400

Seems like this release has some problems, I’ve been seeing the Junk folder empty for the past few days, and I thought the problem was due to PostgreSQL update (that’s also a damned update, but that’s for another day). Seems like 3.2.0 instead is not scoring anything at all anymore.

Mails that are pure spam are not even considered by 3.2.0 as possible spam, marking them with a 0.0 score, while 3.1.8 finds them just fine.

I’ve downgraded to 3.1.8 and I’ll probably stick with that for a while.

Ah and forgot to count that the update channel for 3.2 is broken too, with two different bugs, so it can’t even be updated.

I hope in a 3.2.1, but I won’t count on it as it is.

By the way, I got some progress on OpenJDK, I’ll probably blog about that later or tomorrow.

OpenJDK and Gentoo/FreeBSD

nospam@example.com (Diego Pettenò) — Tue, 08 May 2007 19:02:00 -0400

So, Betelgeuse blogged about the new partly-free JDK released by Sun Microsystems. Kudos to them for actually starting to fix the Java Trap. Hopefully in the next months we’ll finally see Java taking the place it was designed for in development of platform-agnostic software, and applets in web pages, with the (hoped) availability of it on more platform that are available right now.

Of course I want to get OpenJDK working on Gentoo/FreeBSD as soon as possible, as the FreeBSD Foundation’s “Diablo” project seems to be stale and broken (latest version is 1.5.0-7, I’m not even sure if it’s vulnerable, as I remember some security advisories for sun-jdk for Linux), and I wanted to move to Tomcat for my blog. So I asked Betelgeuse a bit of information about openjdk and downloaded the Java Experimental overlay, and decided to merge openjdk on Linux first.

To get it to work on Gentoo/FreeBSD the first thing is to get the Linux emulation support at least partly working, as a JDK 1.7 is needed as a seed to build OpenJDK (this reminds me a lot of GHC unfortunately). So I’ll work on it until I can get OpenJDK building here.

Unfortunately I can’t build openjdk yet, the build fails with an “Argument list too long” error in an execve() call. I’m not sure what causes it, but I hope to investigate it further in the near future (maybe I should ask Betelgeuse tomorrow after his exam).

Another problem I’ve found with OpenJDK is that the build of hotspot mixes the values of CFLAGS and CXXFLAGS, throwing a lot of warnings about -Wno-pointer-sign and -Wno-format-zero-length that I have in CFLAGS and are not supported by GNU C++ building. I’ve tried fixing this one but I was unable to find the perfect patch to fix it, so I reported the bug with the tentative patch to Sun, hopefully they know their build system better than I do

When I’m able to get OpenJDK building on Linux, and I understand the process well enough to work on it, then I’ll try to do my best so that OpenJDK works on Gentoo/FreeBSD at the best I can. I’m sure there are people from FreeBSD project working on it too, they did this already to get Diablo out, after all, but most of them don’t blog so you can’t really know what they are up to (beside if they are the same who would be releasing the new Diablos, they are probably missing in action for months now).

Who knows, maybe we’ll be able to get OpenJDK working on Gentoo/FreeBSD/SPARC64 too, that would be neat After all, FreeBSD and Solaris aren’t that much different (as the DTrace and the ZFS ports shows) and SPARC64 is supported by Sun for sure

The relaxing method failure

nospam@example.com (Diego Pettenò) — Mon, 07 May 2007 20:56:00 -0400

Not sure if you remember, but I’ve started, since a month and a half ago, trying to relax myself by playing with the Nintendo DS. When I bought I also bought Mario Kart DS (provided with the unit itself) and Pokémon Sapphire (for GBA, but that was needed also to protect the GBA card slot, as the pre-Lite version doesn’t have a dustcover for it). I’m the nerd kind of guy who actually like the Pokémon series, and I was pretty happy to play it through the end; I’ve played emulated the previous generations, this was the first time I could afford a game myself, and so I took the chance.

I don’t actually like racing games, I got Mario Kart just because it was sold together with the unit, but I enjoyed it, especially to try out the Nintendo DS graphic which was quite nice. Of course after a while, playing only two games starts to be boring, so when I had the last cold I asked my syster to pick up Lego StarWars II for me.

Nice game, nice graphic, nice ideas… too bad that starting from Episode V (the second of the three movies reproduced in the game), it starts to feel buggy.. there are times when you can end up stuck in a wall, requiring to end the mission and then restart from scratch. The game is nice so even if bugged, I continued playing, till in Episode VI I got stuck in a big bug and I needed to search for the solution of the game to find how to pass that mission… the solution also referred to the bug, so I did a bit of search, and I found that LucasArts admitted that the game, in its DS version, was shipped incomplete and unfinished. Nice one.

Oh well, as I said the game is interesting on its own so I don’t feel like I totally wasted my money (I also got it relatively cheap), still I would have liked if Nintendo paid more attention to the title that are published worldwide, maybe putting a “Notice: might contain annoying bugs” on the package… okay I know it won’t ever happen, but let’s say that this game didn’t relax me that much.

Then finally the game I was expecting was released, Yu-Gi-Oh Championship 2007, a nice game after all, although I don’t play the TCG, I found it quite interesting to play emulated before, so I gone for it as soon as it was available (via Amazon, as the local suppliers don’t carry it), I received it last Saturday and I started playing around. The nice thing about this game is the online challenge, which finally allows me to play against human opponents: playing against the computer becomes just a run to a better deck every time until you can beat the next opponent.

So there I go, I try to get some decent cards and start getting beaten up by human adversaries, I wasn’t expecting anything different, but I wanted to see some more strategies before working out a deck that might work against the computer but not against an human.

The online play is a nice thing, really cool especially since it’s WiFi-based so you don’t have to stay put while playing, and the stylus control in the game is well designed; but then I’ve started feeling bothered by the repeated “communication errors” I started getting during games.

First I thought it was truly random, then I’ve seen it happened mostly when I was leading the game.. I thought of a coincidence, but when it happened after I did win the game, the opponent coming to 0 life points… I was sure the thing was not that random.

Now, of course Konami can’t disallow people from closing their connection, or powering off the DS when they see they are losing, but… they could at least assign you the bonus points for the match, or at least the 10 match points you get for playing online, when the adversary fails to complete the game!

Even if I like the game, I’m tempted to ask Konami my money back, online playing is poorly designed.

Proceeding with xine

nospam@example.com (Diego Pettenò) — Fri, 04 May 2007 07:39:00 -0400

My changes to xine in the audio conversion branch are improving step by step; now all the decoders should playback fine, with their maximum resolution, by using the 32-bit floating point format as highest upscale. AC3 files are still a problem, although thanks to moesaji I now know that the mysterious function reduces the value basically by a value of 383 units; it’s still a mystery to me as I have no idea why that happens, and I can’t seem to find any documentation about these magic numbes for other stuff but a52dec.

Anyway, beside a52dec, most of other decoders are already fixed for the new interface, and I’ve started working on updating the ALSA audio output plugin; unfortunately the code of that plugin is ugly and complex, and seems to be a great big compendium of bad code style. With my changes in the audio conversion branch, the size of the initialisation function was reduced by 1KB, which is far from being small. It should also reduce the number of strings to translate, as I’ve used two loops rather than unrolling the tests for the capabilities of ALSA devices, which then condensed the number of strings to translate (they are composed during output, rather than being complete literals).

Hopefully the audio conversion branch will be merged into 1.2 series as soon as all the audio output plugins are converted, and a few more audio conversions routines are written. As the conversions are all limited to a single source file, it should also be quite easy to write more optimised routines using SIMD extensions for various architectures.

By working on this I ended up applying a few changes to 1.2 series itself already, by removing a few members from the decoder’s state structure for many plugins, and putting them into the functions they are used (why declare them in a state structure that is allocated in heap and passed, by pointer, everywhere, when they are used only in a single function? Sometimes even in a single branch of a given function.

The nice parts are that for instance NSF files now playback with a smaller memory footprint, as the buffer is allocated only when needed, and in the audio conversion branch also reduces the whole xine-lib memory footprint by two megabytes, this result means that to playback an mp3 file, xine-lib-1.2-audio-conversion takes half the memory xine-lib-1.1 takes! It’s incredible, but it’s true.

There are a lot more improvements that I hope to push into xine-lib-1.2, one of these is the splitting out of the modplug demuxer on its own plugin, so that binary distributions can just package it separately, allowing users wanting mod playback to actually get it, and saving the ones who cares not about mod files from having to initialise libmodplug every time (it takes quite a bit of CPU time, because of the calculation of the samples, I think I’ll post some images about this in the next days).

Oh and I’m actually thinking of reducing even more the code present in the decoders, by handling also the downmixing into the audio output loop (currently at least A52, DTS and Vorbis decoders contains the code to handle that) and by providing the code to handle also channels’ interleaving, as currently the same plugins also contain the same code, once again.

This again is to allow providing optimisations if possible, although I admit I’m not sure which kind of optimisation can be handled here. Still, it should be better to only have a copy of the code, rather than an undefined number…

The mystery of the a52dec decoded samples

nospam@example.com (Diego Pettenò) — Wed, 02 May 2007 13:48:00 -0400

I’ve been working a lot on the audio conversion code for xine-lib, with results that are always more satisfying, although Amarok fails to build on this new version, audio playback improved a lot, and with the code being all self-contained, the option of replacing the functions with more optimised code, using MMX, SSE, AltiVec or whatever other SIMD extension is more feasible.

Unfortunately I’ve hit one issue: beside A52 and DTS having a channel order different than most of other formats, I have an issue with the type of the decoded samples. There’s a function called float_to_int() that I supposed truncated the floating-point values to get 16-bit signed integer samples (that were the only format supported by xine before), but the floats themselves don’t play at all, while floating-point data plays fine for most other formats. I had to look up a lot of documentation to find for sure that floating-point samples, like I supposed before, varies in the interval [-1.0,1.0], so I was doing it alright up to now.

Now, if someone has any idea on what the following code does, I’d be glad to know it. The blah() function (with the same name) is present in xine, MPlayer, FFmpeg, and renamed in Ogle and other software too.

#include <stdio.h>

#include <stdint.h>



static inline int16_t blah (int32_t i) {

  if (i > 0x43c07fff)

    return 32767;

  else if (i < 0x43bf8000)

    return -32768;

  else

    return i - 0x43c00000;

}



int main() {

  uint8_t val_1[4] = { 0xFF, 0xFF, 0xBF, 0x43 };



  return printf("%f\n%d\n", *(float*)(&val_1), blah(*(int32_t*)(&val_1)));

}

The output (on Little-Endian, as val_1 is the byte representation of a float number in Little-Endian) is: 383.999969 and -1; it makes no sense to me, but maybe someone else has a clue..

And for who’s wondering, the byte-value I chosen there is one of the samples decoded by a52dec, so it’s a proper value I’m trying to understand; and for who’s wondering, the proper correspondence for -1 (int16) in float32 format is (approximated) -3.0517e-05.

Edit: thanks Nathan Smith who made me see the spelling error in the title (I still confuse the two).

Audio conversion flies!

nospam@example.com (Diego Pettenò) — Sat, 28 Apr 2007 12:01:00 -0400

After a week spent closing envelopes and attaching labels, I was finally able to get the audio conversion branch of xine-lib 1.2 to play mp3s.

As libmad decodes the mp3s as 32-bit signed integer samples, the direct output cannot be sent directly to PulseAudio when playing it back (and PulseAudio is the only audio output I converted to the new plugin API), so I had to actually implement at least some basic conversion, in this case from almost every integer format to 32-bit floating point samples (which is the simplest conversion, although it’s CPU consuming.

Previously every mp3 file decoded with libmad was truncated to 16-bit signed integer samples, losing precision, and for what I can see amplifying the volume quite a bit, distorting the signal, but with my new code, there’s no precision loss, and the difference is audible.

You can see the difference in the following screenshot:

If you look at the screenshot you will see the stream info dialog windows of xine-ui: on the left there’s the window for xine-lib 1.1 while on the right there’s the one for xine-lib 1.2 audio_conversion branch. As you can see at the bottom of the window, the first one reports 16-bit for the stream, while the new one returns 32-bit as intended.

I can tell you, the difference is great, and this is just the first step! Hopefully it will be merged in xine-lib-1.2 and then it will be able to play hi-fidelity wave files just fine.

Unfortunately before I can actually merge this branch into main xine-lib-1.2, all the output plugins, the post plugins and most of the audio decoding plugins have to be ported, which is a HUGE amount of work for just one person. I’d also require to use some virtual machines to confirm most of the output plugins work afterward, so it will take me a lot of time alone. If you can handle converting and testing the code, that will certainly be appreciated.

Self-signed certificates on the E61: failed.

nospam@example.com (Diego Pettenò) — Fri, 27 Apr 2007 14:16:00 -0400

I’m not sure if you remember, but last Christmas I blogged about my need to find a way to allow my smartphone (a Nokia E61) to automatically accept the certificate file for my IMAP server, to avoid having to accept it every time when I want to check my mails.

Well, I originally thought it was just a problem of format, as a lot of places on the net talked about the need to convert the PEM certificate to a DER certificate, and then download it with the internal browser from a properly-configured webserver (properly meaning the certificate is served with the mime-type application/x-509-ca-cert), but then it wasn’t enough, I thought it was a problem of size of the certificate (over 1024 bits it doesn’t seem to be supported) but it wasn’t the case either, and the last idea I had was that the problem was with the firmware version.

Unfortunately last month when I asked a friend of mine to help me updating the firmware on the phone, the download process on his laptop (with Windows XP) took more than three hours, then we shut everything down, as it was coming late, and the download wasn’t complete yet. I tried upgrading through a (demo) version of Parallels Desktop for Mac OS X, but it didn’t help either, today I tried on vmware-server, and there it worked fine, without the problems with disconnection and reconnection of the USB device that Parallels suffered from (probably while the device was being reconfigured to provide a different interface to check the phone model, serial and firmware). The firmware upgrade process went fine and easy, in about thirty minutes just like Nokia wrote on the site, I’m not sure what was wrong with my friend’s laptop, nor I care much at this point, if I have to be blunt.

But, not even the firmware update helped me.. so I decided to investigate further: the posts on the web about the configuring the E61 (or the USA-marketed E62, which should just be a crippled version of mine) mixes information about the addition of extra Certificate Authority certificates with requests for help with self-signed certificate. As soon as I considered this, light shed into my mind and I found the problem: the Nokia E61 does not support self-signed certificates; this is an absolute, and there doesn’t seem to be a way to get around this. What you can do instead is create your own Certificate Authority, load that certificate on the smartphone, and then use it to generate your own SSL certificate; this solved the problem for me entirely.

If you need an easy way to build a certificate authority certificate, and to create a new certificate signed with that, you might want to look in the openvpn’s sources’ tarball, that contains an easy-rsa directory with a series of scripts that really helps on that matter.

By the way, the reason why I updated firmware was also to enable the VoIP features, so that I can have a landline-like number routed on my phone while I’m at home (or anywhere where a WLAN can be found) allowing me to receive calls directly without passing through my family’s number.

usb.ids, is it still maintained?

nospam@example.com (Diego Pettenò) — Thu, 26 Apr 2007 17:09:00 -0400

Classically, usb.ids file can be found at http://www.linux-usb.org/usb.ids unfortunately there doesn’t seem to be any more updates after last December. I sent to Vojtech a few patches to add the devices I have here at home, but I didn’t get an answer (I know Vojtech usually answers in a day or two).

Since I hate seeing stuff getting obsoleted so easily, I’ve decided to add a bunch of devices to usb.ids, fix an entry (the one for APC UPSes, as now I have two of them to cope with), and maintain my own copy of usb.ids.

Most of the devices I’ve added are things that I have here at home, but they are not limited to, I’ve added some USB sticks of friends of mine who happened to pass by here, and an entry for a Microsoft mouse from Timothy. I suppose that while usb.ids is not fixed in the original location, I can continue maintaining it for my systems at least, by changing the cron jobs’ path to fetch from (it’s also pretty much useless to download the same usb.ids every month).

So my copy of usb.ids is located at http://dev.gentooexperimental.org/~flameeyes/usb.ids (thanks Patrick for the hosting, which I’m also using for a few other stuff like dist tarballs); if you want to add more devices to the list, feel free to mail me at flameeyes@gmail.com with a diff -u, or simple the vendor and product ids of the device, and the complete naming of it.

I know the hardware from the MacBook Pro is missing, I’ll see to add those the next time I boot the laptop on Linux.

Laptop policies on a workstation

nospam@example.com (Diego Pettenò) — Wed, 25 Apr 2007 10:02:00 -0400

As I bought the new UPS, I now have quite some time to work with it while I’m without power, as the two UPSes can keep the system online for about one hour without any kind of interruptions, included network access. This is quite nice, but there is one thing I was thinking of today…

When the workstations are running on UPS’s battery power, it’s comparable to a laptop running on battery, so I could be just using the same method laptops use: reduce the frequency of the CPU. At least Enterprise is able of CPU frequency scaling, so I’ve configured it to switch to power saving profile when the onbattery event is received by apcupsd.

It was actually quite simple, first of all apcupsd runs as root (although it should drop all privileges, but I’m not sure about it), and I have powersave installed and configured to handle this, so I just changed the /etc/apcupsd/onbattery and /etc/apcupsd/offbattery scripts to add a call to powersave binary, with -e Powersave on the onbattery event and -e Performance on the offbattery event; unfortunately I don’t think there is anything else beside slowing the CPU speed during UPS battery supply that can be done on a Workstation

Now I’m going to look to split the connection of the power cables in my home office so that the hardware that would just go in standby while I’m sleeping would be connected to a single multi-socket, which can be disconnected at once, to avoid leaving the hardware consuming power without any reason. I’d like to be able to do the same with the monitors (that otherwise would remain in standby as they can’t be shut down. If I wasn’t using the same UPS for both networking and monitors, I could probably use the UPS to shut them down and turn them back on at my command.

Sigh, I’m afraid there will be a looot of work to do, and I’m sealing envelopes, once again

A nice productive day

nospam@example.com (Diego Pettenò) — Sat, 21 Apr 2007 21:03:00 -0400

I might be sick, or just crazy, or both of them, but I still think I’m quite more productive when I have fever, or the days around that time. Yesterday I had fever, and I was knock out till late afternoon, but then I started feeling better, and I started producing.

First of all, rbot’s init script in my overlay has been updated: Subversion trunk will now create a rbot.pid file inside the bot’s directory without need for start-stop-daemon to create it itself, which means I can finally let rbot fork on its own rather than forcing it to background (which is not really a good thing to do, if it can be avoided). Thanks to tango and jbn for looking after my raw patch.

Then I decided to finish the work with apcupsd; again in my overlay you can find a new ebuild for 3.14.0 based on the one found on bugzilla, but with a new apccontrol file and a totally renewed init script. This script can be multiplied, which means you can have a /etc/init.d/apcupsd.foo link, which would then start apcupsd looking for /etc/apcupsd/foo.conf configuration file. Thanks to apcupsd authors for implementing this feature in 3.14!

I’ve also attached the two UPSes to Farragut instead of Enterprise, as the latter is not a server and might as well be offline when Farragut is still up (for instance this is the case most of the times I’m outside for the whole day, or if there is noone at home); apcupsd on FreeBSD works nicely, and doesn’t require any fiddling with configuration, neither kernel side (the ugen support is built by default) nor with permissions (as the default is to run as root, this might change in the future, but as it is it’s fine to me; I’ll be working on a better handling of permissions on device nodes for Gentoo/FreeBSD, but it’s not in my priority list at the moment). Also here, they work perfectly fine.

I was also able to fix a bug in xine-lib, with mp4/mov files playback that used version 1 rather than version 0 of the media header atom, such as files generated by FFmpeg. The bug was reported on sourceforge already but I wasn’t sure what it actually meant and where to find a sample file; when I generated the same condition by chance here, I decided to take a deeper look; unfortunately MultimediaWiki doesn’t provide much information about that, but I asked Mike to give me an account there so I can try to write something useful, maybe next time someone else needs a mdhd atom description they won’t have to look at the sources of FFmpeg to see how it’s read and generated.

Then tonight I wanted to resume my work on implementing audio conversions inside the audio output loop instead of doing it for every decoder; it’s an hard work as it probably will require rewriting a good deal of code, but it should be rewarding once it’s done. Right now there are a bunch more of flag values for capabilities, so for instance I can say if a drivers supports integer or float 32-bit samples, 64-bit samples, and if it can accept streams in a different endianness. This is important because there’s little point in doing the job of the output plugin, that might handle that transparently, for instance a big-endian stream might be decoded on a little-endian machine, then sent through PulseAudio to a big-endian machine where it will be reproduced: in this setup, xine’s endian reversal of the stream (from big to little endian) would have been superfluous, as PulseAudio would have accepted the big-endian samples, then sent them to the other machine that needed not to reverse them to reproduce them.

Anyway, right now the code is quite fragile, there’s no conversion being done, there are mostly only things that are totally broken out, there are asserts @1 == 0@ used to mark the code that needs to be rewritten. But something works: I was able to remove a lot of code from the musepack decoder, as libmpcdec always produces 32-bit native-endian (or maybe little-endian, I’m not yet sure) floating point samples; previously the decoder converted all the samples back to 16-bit format, and then gave it to the audio output loop to handle? now instead it sends them directly to the output, and as PulseAudio supports 32-bit float samples, they are not converted and play back fine.

Tomorrow I’ll see to work a way to handle upsamping and downsampling of streams, the problem is that it’s not trivial to decide what to do: if a plugin supports 32-bit integer samples, but not 24-bit integer samples, it should probably upscale the 24-bit to 32-bit to avoid losing precision; if it doesn’t it might upscale it to 32-bit float, or maybe downscale it to 16-bit integers. The same applies to channel mode, if the driver doesn’t support stereo output, should it be updated to 4.0 or should it be downgraded to mono?

For sure this time I’m very happy of being working on branches: leaving the code broken for weeks, maybe months, is not something you want to do on the main development branch. And I mean it, because with the changes I’m doing, not only I’ll be changing the ABI of the library itself (well, actually not much, just a couple of structures), but more importantly I’ll be changing the audio output plugins API, as I need to feed them a sample format rather than a bits-per-sample size.

Anyway, this is not going to be something easy to complete, but it will be a noticeable improvement for Amarok users once done, especially because I want to make sure that the capabilities for “mixer” volume and “PCM” volume are cleared up, probably by deprecating one of them, so that Amarok can be changed not to use xine’s software amplification (which also sucks and I also need to rewrite in good part in this branch) if the output plugin actually supports a per-stream volume (like PulseAudio).

Sponsoring, bribing, and comfort words are welcome, as xine-lib’s audio_out code is giving me creeps.

Downtime and new UPS

nospam@example.com (Diego Pettenò) — Fri, 20 Apr 2007 19:57:00 -0400

Sorry for the downtime guys, the update to baselayout 2 on Farragut wasn’t as easy as expected, I have some talk to do with Roy about it

Now, this is probably going to be the fourth night I’m not gonna sleep well, today I had a support request to take care of in the afternoon and I was supposed to go out with a few friends tonight, but I was so tired that I started having fever, so I had to get a raincheck for that.

The good news is that I received the new UPS, an APC SmartUPS 1000VA, that seems to be able to take enterprise and farragut up for 1 hour and a quarter, which is quite good (the other UPS would take the monitors and the network equipment up for also an hour, which is not bad, and should cover most of short-time non-planned outages.

Unfortunately, if you have followed me for a while, you know there’s no flawless hardware acquisition for me; this time the problem comes from the software needed to control the two UPSes; please note that there are two.

First of all, when I started using an UPS, I used apcupsd but then I moved to nut because it had a decent graphical control software (knutclient); what was the problem with this? Well, nut was confused by the two UPSes, that being both APC, shares the same identical vendor and product ID for the USB device. So it was not a nice thing.

I’ve then decided to come back to apcupsd, but the latest version is not in portage (again) so I bumped it locally with the patch on Bugzilla, and added a gnome useflag for the graphical control utility that is now present. This version is important to me, not only because of the utility, but also because with this version is possible to choose the configuration file at runtime, as it’s not hardcoded during build.

This is also important because in previous versions the only way to have more than one UPS being monitored on the same box with apcupsd required to have two different builds on different places. What I want to do now is to rewrite the init script so that it is multiplexed (like rbot, mt-daapd, openvpn and so on), that way I can simply start two apcupsd instances to have all I need (note that one of them won’t be shutting off my box at all, as it would just keep monitoring the UPS that is actually connected to the monitors and networking.

Once Baselayout 2 is more functional on Farragut, I’ll probably move apcupsd instances there, and network it down here, it would be safer on the long run as enterprise might not be up when farragut is.

I still haven’t been able to finish working on rbot’s changes I need, sigh. And don’t get me started on xine-lib, I’m taking a few days off because what I found myself working on now is a veeery bad thing I’m afraid…. (bit per sample transcoding during output).

About libdvdnav and xine

nospam@example.com (Diego Pettenò) — Tue, 17 Apr 2007 19:03:00 -0400

Don’t worry all you people, I wasn’t killed in a bus accident yet; you are lucky, I actually go out quite rarely so I don’t suffer from this kind of problems very often, the worst that can happen would be a health problem, but I’m lucky enough to feel decently well at the moment Of course there is too much heat for my liking and I start to feel how annoying it is at this temperature, especially staying in my home office with two/three boxes online.

Anyway, as I promised, I started looking at the fork of libdvdnav lead by Nico, to import a newer version of libdvdnav in xine, and cleaning up the patches applied.

Unfortunately it wasn’t exactly straightforward to update the libdvdnav version; beside my will to move the sources into the contrib/ subdirectory as I already did for libmpcdec and ffmpeg, Nico started to replace autotools with a build system alike to FFmpeg’s (which is quite fine by me, as I can integrate it nicely enough into xine-lib), so I had to make sure that the build system was up to a minimum support for the features I needed. The main showstopper was the lack of a way to build in a directory different from the original sources directory, but that was quite easy to handle.

But luckily working with Nico is quite enjoyable All the patch I’ve sent for inclusion up to tonight (so excluding the one I sent less than an hour ago) were accepted and committed to the Subversion repository, which means that the build system is usable from xine-lib’s custom base system, that the compiler flags we inject are respected, that the library builds with -Werror-implicit-declarations (the change was needed to libdvdread, Nico took care of sending it up there).

Tonight I rebuilt xine-lib-1.2 with the new libdvdnav, and it’s working nicely. I suppose the reason why external libdvdnav is not working on seeking is that the code was present in cvs (for sf.net’s dvd project) but not released yet. I should probably also add support for .pc files for libdvdnav and then use those to check for its presence, so to require at least a new enough version that doesn’t have seeking problems, but that is not a priority until a release is done.

There is only one problem that has to be considered: the libdvdnav copy in xine-lib is patched with a patch from Bastien Nocera (Totem’s author) which is used to play invalid DVDs (non-encrypted ISO filesystem DVDs; proper DVDs have UDF as file system). That patch as it is won’t be accepted by libdvdnav authors, and sincerely I wouldn’t have accepted it for xine-lib either.

Why this? Well, the patch takes an opaque type, and makes it transparent, copying out of libdvdcss the structure definition. This is fine as long as the structure is not changed on libdvdcss, but the reason why a type is opaque, is just so that you don’t have to put its definition in the ABI, so you can change it without having to deal with software crashing because it was compiled against the previous definition. This patch breaks this assumption with xine, so it’s bad from a good practices point of view.

I’m not sure myself how much sense does it make to use such a stupid solution (sorry Bastien, but re-declaring an opaque type is a stupid solution) to consider invalid media. A better solution would be to always use the files if they are found, rather than using UDF access, with fallback to raw UDF instead. This should work, but I need first some media to test with, and then I have to understand libdvdnav code.

Anyway, the build framework changes are now committed to the xine-lib-1.2-newdvdnav branch, the “only” thing missing there is the libdvdnav code itself; the reason for this is that up to tonight I was still working with patched sources, not with upstream sources, and I wanted to avoid recommitting everything every time. I will probably wait as much as I can before committing the sources themselves, with this “as much as I can” representing ideally the time till I get to have committed to libdvdnav all the changes that make sense for xine-lib (revised Bastien’s patch, and the file descriptor leak patch).

I also want to thank Nico for the work being done toward adding an option to use external libdvdread in libdvdnav, which will certainly help Gentoo and other distributions: you wouldn’t have to duplicate the code between libdvdnav and libdvdread; you’d have one more dependency in the libdvdnav library, but that shouldn’t be much overhead.

And I want to say it officially: there is a lot of code in xine-lib that scares the hell out of me, and I would like to see it killed before it propagates too much; unfortunately overhauling all of it is difficult alone, especially since I might end up having repercussions that I don’t see (like the CDDA failure that didn’t fail for me when I tested – I was using by-extension detection in Amarok – and the DVD failure that still I was unable to reproduce till at least since Darren fixed it).

I start hating spending my time on xine, I hope to be able to continue working on it till it’s enjoyable to do so.

More progress for XDG support in xine

nospam@example.com (Diego Pettenò) — Sat, 14 Apr 2007 20:03:00 -0400

Returning on yesterday’s blog entry, today I’ll see to update you on the status of xine-lib’s XDG support.

Thanks to Mark Nevill, I didn’t have to reinvent the wheel by parsing the various XDG_ variables to check for the directories I have to search in; he already wrote a libxdg-basedir library that takes care of most of it, allowing me to take care of the implementation details.

Now, xine-lib-1.2 branch has an internal copy of libxdg-basedir (two source files, so it’s not a big deal, and I’ve added a switch to use the external copy of it if needed), and uses it to decide where to read and write some files.

For instance, the plugins cache is no more in ~/.xine/catalog.cache but it’s in $XDG_CACHE_HOME/xine-lib/plugins.cache which both makes more sense and can be decided by user to be moved out of its home (for instance I change the value to XDG_CACHE_HOME, but that’s a topic for another post). The CDDB cache is also moved on the cache home, while the fonts are now discovered in the XDG_DATA_DIRS defined paths. Darren also moved channels.conf load from ~/.xine to XDG_CONFIG_HOME, which means it’s loaded in ~/.config/xine-lib by default.

Please note that I’ve been using xine-lib all over rather than xine, so that users know what is generated/handled by the frontends and what is actually part of the library, if they care.

There are still a few changes pending, for instance Darren wants to support loading of a system-level channels.conf, so that it can be put in /etc/xdg and used for all users without having to put it in xine’s configuration for every user.

I’ve also decided to pay more attention to the security side of xine-lib, for instance, after talking with Taviso today, I’ve added a xine_xcalloc function to wrap around calloc, which should avoid possible overflows (there was one in input_dvb); I’ve changed some of the code, but of course it’s not totally cleaned up. xine-lib really should be audited piece by piece for improvement, every time I touch something I end up cleaning it up by adding more const keywords (trying to let the compiler optimise a bit more) and adding/removing/cleaning up code .

One thing that certainly would help would be to put a better wall between contributed code (where we should always do the work on upstream’s side and then merge it back into xine-lib, to avoid getting them out of sync, unless fixing some xine corner case or similar—although there it could certainly help to put a proper wrapping around the two), and the xine proper code that should fixed/cleaned/improved as we go.

Really, while xine actually do its job most of the times, it still suffers from a lot of possible problems just because the code is too old and stratified. I think I should simply undergo an audit file per file trying to fix stuff while I also update the documentation to be doxygen-compatible, but that’s going to take so much time that I’m not sure how realistic is to work on it; beside I don’t have hardware such as a DVB tuner that would allow me to complete the audit (I can’t try that code and I can’t ensure its working state).

On the other hand, tonight I fired up Klothos again; it has been some time since I’ve done any Gentoo/*BSD work, but I’ve lately asked Roy to put me back in bsd@gentoo.org alias, to see if there is work that needs to be done that I can do. Yes, I suppose I’m considering coming back, but if I will do that, it will be on a lower profile; maybe I’ll help Mike, Frysinger, SpanKY and vapier with the ARM architecture, maybe I’ll just decide to take care of BSD alone.

The problem with Klothos is the PSU; the whole box is quite silent, as the UltraSPARC CPU is cooled by a slow and silent fan, but the PSU is annoyingly noisy, and I can’t just put a be quiet PSU like I put on Enterprise because of the Ultra5 casing (it’s a desktop machine, the case is high as the ATX power supply, which means the big fan used by most silent units would be obstructed by it); I could try a passive PSU, as the box shouldn’t be sucking too much power anyway, but I’m not sure about it and I don’t have money to waste (at the moment I’m unemployed).

If anybody has a suggestion about which PSU I could use for such a box, it’s really appreciated. It’s funny that I paid ?35 for a working box, and then paid way more to bring it to a standard worth to be a dev machine… well, I got the SATA controller already in my possession, as well as the SATA hard disk; the DVD reader was also an old one I used, but I had to buy a new ethernet card to avoid using the obnoxious hme0 driver..

Contributed code and FreeDesktop standard

nospam@example.com (Diego Pettenò) — Fri, 13 Apr 2007 09:28:00 -0400

Of course, this entry will talk about xine-lib.

First of all, a note about what I’m trying to do with contributed code in xine-lib for 1.2 series. As I said already, my ffmpeg_integration branch, which is one of the two branches I merged to get the 1.2 branch itself, was born to allow using FFmpeg without changing its buildsystem, and putting it in a contrib/ subdirectory instead of mixing it with xine sources.

Well, as I’ve now started working on providing also some Doxygen documentation, I’ve seen that this is useful also for other projects that are imported in xine-lib, so that I don’t have to blacklist one by one the directories and the files that are not to be parsed for Doxygen generation.

In xine-lib-1.2 hg tip you can now find libmpcdec (that replaced libmusepack) in contrib/libmpcdec/ rather than having it together xine’s actual code (the decoder plugin code) inside src/libmusepack/. I also updated it to the last version available.

Similarly I hope to move other code, like libmad and of course libdvdnav, that was recently taken over by MPlayer developers because of the unresponsiveness of the original development project.

It will take some time to complete all the moves, also because I’ll try to contextually update the code with the last version available for every project beside libmpeg2. This should be a breath of fresh air for xine 1.2.

Talking about FreeDesktop standards, as Darren already changed gxine to abide to XDG Base Directory Specification, he also added better FreeDesktop standard support in xine-lib for configuration and cache files. So right now I’m working on making xine-lib use XDG_CACHE_HOME to store the plugins cache instead of the hardcoded ~/.xine/catalog.cache, which also allows me to move more cache data out of my home directory

By the way, why one earth the description of the permissions to use while creating directory (that is actually a quite logic 0700) is written in the «Referencing this specification» section ?