|
Thursday, May 10. 2007
It might sound strange coming from someone who wrote a post titled The Java t..crap but I’m liking playing with OpenJDK.
Not like my Java skills (pretty much non-existant, I used it only a few times in my life) are improved, but I find myself comfortable when working with most build systems, after you get scons out of the way (note that I didn’t say I like it, but I can do it just fine), and I do have experience with adding support for external libraries, after all, and that is something that OpenJDK might need.
So, let’s start from the start. The first problem was getting to build OpenJDK on Linux. If you try to build it on AMD64 you’ll see it fails, the reason for this is that the list of files to compile on hotspot is too long for the maximum size of arguments available on Linux; this in turn is caused by the long default path for sources that Portage sets. Petteri was able to build it fine because on x86 the architecture name is “i486”, while on AMD64 is, well, “amd64”: the extra character there is enough to make it burn in flames. Kelly O’Hair from Sun is looking after the issue now, but he says it might take a few weeks to get the results out; not a problem, I suppose, in the mean time you might want to try changing PORTAGE_TMPDIR to use /tmp for openjdk (if you have space in the /tmp partition) or you might want to fool the ebuild by renaming the openjdk directory inside $WORKDIR to something shorter, like “ojdk”; actually, a single character less will work just fine.
After getting a working build of OpenJDK, and trying so that it worked fine on Konqueror (it does!), I had to start messing with it, or I wouldn’t have been myself  The first thing was to get it to build with my flags correctly; I was able to get something that worked, but there are a few issues, and I’d like to be able to provide a less invasive patch. O’Hair also got me on the right track now, by telling that hotspot and j2se build systems are different, which might call for just using the same method on boths, by ignoring the default C(XX)FLAGS values coming from environment, and instead use the _OTHER variables. This should allow for just four lines changed, if I find where to put the CFLAGS/CXXFLAGS unset.
After this I started thinking of an useful hack to do to at least understand how the build system works; I could have looked at fixing a few warnings, but I thought it would have been more useful to actually start by looking at the build system, that is what I will need to understand to be able to get OpenJDK building on FreeBSD. I’ve seen zlib-1.1.3 sources in the tree, so I decided to find a way to build against external zlib. Easy, quick, interesting and useful.
Why useful? Well, first of all, it would use a more stable version of zlib, and would share its code with the rest of the system (almost every process has a zlib copy loaded one way or another), second of all that would make it non-vulnerable by possible attacks on the zlib code. Unfortunately adding the support for external zlib was not as easy as I thought, but it was mostly because I stupidly forgot FEATURES=keepwork on the environment (I wanted to get the object files output, as I want to see if the executable stack problems that affects Java are due to the architecture, or just a problem of not having the GNU stack markings on the source files).
And while looking at fixing this, and seeing a tempnam() warning at linking stage, I ended up finding a tiny memory leak, as the string coming from tempnam() is never freed. I’ve prepared a patch and sent it, hopefully to the right mailing list.
Other hacks that might be useful would be finding a way to enable or disable ALSA and Motif bindings building, to reduce the size of the final output, and to avoid depending on those to build openjdk, but I wonder if Sun would ever accept such changes.
When I’ll be comfortable enough with both build system and development environment, I’ll see to start working on building OpenJDK on FreeBSD; for what I can read on the build system mailing list, it should be possible, for now, to build OpenJDK using a JDK 1.5, which is available for Gentoo/FreeBSD, even if the best thing would be using 1.6. Probably I’ll have to provide a openjdk-bin package for Gentoo/FreeBSD to use as a seed, afterward.. this of course if the license allows to do this (most of OpenJDK is GPL-2, but there are some binary blobs still present, until they are replaced, it might not be possible to redistribute the binary itself).
Unfortunately a big problem I have to face while hacking at this is that I don’t have any Solaris/OpenSolaris installation, and I suppose that Sun will accept more willingly contributions that don’t break their main target platform 
Now, of course, the reasons I have to find this project interesting and to revert my attitude toward Java are not limited to the simple GPL-2 licensing. What I find more attractive, and makes me hope I’m not mistaken on my judgement, is that now it would be possible to support more hardware platforms, making Java a true platform agnostic language (okay, Kaffe help a bit with this, but the results are not really at the same level), and will also allow to support more operating systems. And with more users’ contributions, it will probably also get better optimisations (hey we got guys like FFmpeg devs who are able to optimise the hell out of multimedia codecs, someone might find useful things to improve on OpenJDK too!).
For what concerns Gentoo/FreeBSD, for sure FreeBSD guys have already some build definitions to support FreeBSD; the problem of those are that are likely not GPL-2 as it is, as they were designed for previous JDK versions, so I’d rather not even look at them to avoid legal issues. The assembler code would likely be the same between Linux and FreeBSD, with the same architecture; for SPARC, the code might be just taken out of the Solaris assembler sources, at least on i486, the assembler between Linux and Solaris doesn’t change besides from a few minor places, and the main issue there is the difference in the assembler program used, and thus in the syntax.
Let’s try to make Java a truly Free platform, and allow it to run on as many system as possible, then it might replace Flash and beat the heck out of Silverlight; Sun seems to be in the right mind set… so waste no time!
Wednesday, May 9. 2007
Seems like this release has some problems, I’ve been seeing the Junk folder empty for the past few days, and I thought the problem was due to PostgreSQL update (that’s also a damned update, but that’s for another day). Seems like 3.2.0 instead is not scoring anything at all anymore.
Mails that are pure spam are not even considered by 3.2.0 as possible spam, marking them with a 0.0 score, while 3.1.8 finds them just fine.
I’ve downgraded to 3.1.8 and I’ll probably stick with that for a while.
Ah and forgot to count that the update channel for 3.2 is broken too, with two different bugs, so it can’t even be updated.
I hope in a 3.2.1, but I won’t count on it as it is.
By the way, I got some progress on OpenJDK, I’ll probably blog about that later or tomorrow.
Tuesday, May 8. 2007
So, Betelgeuse blogged about the new partly-free JDK released by Sun Microsystems. Kudos to them for actually starting to fix the Java Trap. Hopefully in the next months we’ll finally see Java taking the place it was designed for in development of platform-agnostic software, and applets in web pages, with the (hoped) availability of it on more platform that are available right now.
Of course I want to get OpenJDK working on Gentoo/FreeBSD as soon as possible, as the FreeBSD Foundation’s “Diablo” project seems to be stale and broken (latest version is 1.5.0-7, I’m not even sure if it’s vulnerable, as I remember some security advisories for sun-jdk for Linux), and I wanted to move to Tomcat for my blog. So I asked Betelgeuse a bit of information about openjdk and downloaded the Java Experimental overlay, and decided to merge openjdk on Linux first.
To get it to work on Gentoo/FreeBSD the first thing is to get the Linux emulation support at least partly working, as a JDK 1.7 is needed as a seed to build OpenJDK (this reminds me a lot of GHC unfortunately). So I’ll work on it until I can get OpenJDK building here.
Unfortunately I can’t build openjdk yet, the build fails with an “Argument list too long” error in an execve() call. I’m not sure what causes it, but I hope to investigate it further in the near future (maybe I should ask Betelgeuse tomorrow after his exam).
Another problem I’ve found with OpenJDK is that the build of hotspot mixes the values of CFLAGS and CXXFLAGS, throwing a lot of warnings about -Wno-pointer-sign and -Wno-format-zero-length that I have in CFLAGS and are not supported by GNU C++ building. I’ve tried fixing this one but I was unable to find the perfect patch to fix it, so I reported the bug with the tentative patch to Sun, hopefully they know their build system better than I do
When I’m able to get OpenJDK building on Linux, and I understand the process well enough to work on it, then I’ll try to do my best so that OpenJDK works on Gentoo/FreeBSD at the best I can. I’m sure there are people from FreeBSD project working on it too, they did this already to get Diablo out, after all, but most of them don’t blog so you can’t really know what they are up to (beside if they are the same who would be releasing the new Diablos, they are probably missing in action for months now).
Who knows, maybe we’ll be able to get OpenJDK working on Gentoo/FreeBSD/SPARC64 too, that would be neat After all, FreeBSD and Solaris aren’t that much different (as the DTrace and the ZFS ports shows) and SPARC64 is supported by Sun for sure
Tuesday, May 8. 2007
Not sure if you remember, but I’ve started, since a month and a half ago, trying to relax myself by playing with the Nintendo DS. When I bought I also bought Mario Kart DS (provided with the unit itself) and Pokémon Sapphire (for GBA, but that was needed also to protect the GBA card slot, as the pre-Lite version doesn’t have a dustcover for it). I’m the nerd kind of guy who actually like the Pokémon series, and I was pretty happy to play it through the end; I’ve played emulated the previous generations, this was the first time I could afford a game myself, and so I took the chance.
I don’t actually like racing games, I got Mario Kart just because it was sold together with the unit, but I enjoyed it, especially to try out the Nintendo DS graphic which was quite nice. Of course after a while, playing only two games starts to be boring, so when I had the last cold I asked my syster to pick up Lego StarWars II for me.
Nice game, nice graphic, nice ideas… too bad that starting from Episode V (the second of the three movies reproduced in the game), it starts to feel buggy.. there are times when you can end up stuck in a wall, requiring to end the mission and then restart from scratch. The game is nice so even if bugged, I continued playing, till in Episode VI I got stuck in a big bug and I needed to search for the solution of the game to find how to pass that mission… the solution also referred to the bug, so I did a bit of search, and I found that LucasArts admitted that the game, in its DS version, was shipped incomplete and unfinished. Nice one.
Oh well, as I said the game is interesting on its own so I don’t feel like I totally wasted my money (I also got it relatively cheap), still I would have liked if Nintendo paid more attention to the title that are published worldwide, maybe putting a “Notice: might contain annoying bugs” on the package… okay I know it won’t ever happen, but let’s say that this game didn’t relax me that much.
Then finally the game I was expecting was released, Yu-Gi-Oh Championship 2007, a nice game after all, although I don’t play the TCG, I found it quite interesting to play emulated before, so I gone for it as soon as it was available (via Amazon, as the local suppliers don’t carry it), I received it last Saturday and I started playing around. The nice thing about this game is the online challenge, which finally allows me to play against human opponents: playing against the computer becomes just a run to a better deck every time until you can beat the next opponent.
So there I go, I try to get some decent cards and start getting beaten up by human adversaries, I wasn’t expecting anything different, but I wanted to see some more strategies before working out a deck that might work against the computer but not against an human.
The online play is a nice thing, really cool especially since it’s WiFi-based so you don’t have to stay put while playing, and the stylus control in the game is well designed; but then I’ve started feeling bothered by the repeated “communication errors” I started getting during games.
First I thought it was truly random, then I’ve seen it happened mostly when I was leading the game.. I thought of a coincidence, but when it happened after I did win the game, the opponent coming to 0 life points… I was sure the thing was not that random.
Now, of course Konami can’t disallow people from closing their connection, or powering off the DS when they see they are losing, but… they could at least assign you the bonus points for the match, or at least the 10 match points you get for playing online, when the adversary fails to complete the game!
Even if I like the game, I’m tempted to ask Konami my money back, online playing is poorly designed.
Friday, May 4. 2007
My changes to xine in the audio conversion branch are improving step by step; now all the decoders should playback fine, with their maximum resolution, by using the 32-bit floating point format as highest upscale. AC3 files are still a problem, although thanks to moesaji I now know that the mysterious function reduces the value basically by a value of 383 units; it’s still a mystery to me as I have no idea why that happens, and I can’t seem to find any documentation about these magic numbes for other stuff but a52dec.
Anyway, beside a52dec, most of other decoders are already fixed for the new interface, and I’ve started working on updating the ALSA audio output plugin; unfortunately the code of that plugin is ugly and complex, and seems to be a great big compendium of bad code style. With my changes in the audio conversion branch, the size of the initialisation function was reduced by 1KB, which is far from being small. It should also reduce the number of strings to translate, as I’ve used two loops rather than unrolling the tests for the capabilities of ALSA devices, which then condensed the number of strings to translate (they are composed during output, rather than being complete literals).
Hopefully the audio conversion branch will be merged into 1.2 series as soon as all the audio output plugins are converted, and a few more audio conversions routines are written. As the conversions are all limited to a single source file, it should also be quite easy to write more optimised routines using SIMD extensions for various architectures.
By working on this I ended up applying a few changes to 1.2 series itself already, by removing a few members from the decoder’s state structure for many plugins, and putting them into the functions they are used (why declare them in a state structure that is allocated in heap and passed, by pointer, everywhere, when they are used only in a single function? Sometimes even in a single branch of a given function.
The nice parts are that for instance NSF files now playback with a smaller memory footprint, as the buffer is allocated only when needed, and in the audio conversion branch also reduces the whole xine-lib memory footprint by two megabytes, this result means that to playback an mp3 file, xine-lib-1.2-audio-conversion takes half the memory xine-lib-1.1 takes! It’s incredible, but it’s true.
There are a lot more improvements that I hope to push into xine-lib-1.2, one of these is the splitting out of the modplug demuxer on its own plugin, so that binary distributions can just package it separately, allowing users wanting mod playback to actually get it, and saving the ones who cares not about mod files from having to initialise libmodplug every time (it takes quite a bit of CPU time, because of the calculation of the samples, I think I’ll post some images about this in the next days).
Oh and I’m actually thinking of reducing even more the code present in the decoders, by handling also the downmixing into the audio output loop (currently at least A52, DTS and Vorbis decoders contains the code to handle that) and by providing the code to handle also channels’ interleaving, as currently the same plugins also contain the same code, once again.
This again is to allow providing optimisations if possible, although I admit I’m not sure which kind of optimisation can be handled here. Still, it should be better to only have a copy of the code, rather than an undefined number…
Wednesday, May 2. 2007
I’ve been working a lot on the audio conversion code for xine-lib, with results that are always more satisfying, although Amarok fails to build on this new version, audio playback improved a lot, and with the code being all self-contained, the option of replacing the functions with more optimised code, using MMX, SSE, AltiVec or whatever other SIMD extension is more feasible.
Unfortunately I’ve hit one issue: beside A52 and DTS having a channel order different than most of other formats, I have an issue with the type of the decoded samples. There’s a function called float_to_int() that I supposed truncated the floating-point values to get 16-bit signed integer samples (that were the only format supported by xine before), but the floats themselves don’t play at all, while floating-point data plays fine for most other formats. I had to look up a lot of documentation to find for sure that floating-point samples, like I supposed before, varies in the interval [-1.0,1.0], so I was doing it alright up to now.
Now, if someone has any idea on what the following code does, I’d be glad to know it. The blah() function (with the same name) is present in xine, MPlayer, FFmpeg, and renamed in Ogle and other software too.
#include <stdio.h>
#include <stdint.h>
static inline int16_t blah (int32_t i) {
if (i > 0x43c07fff)
return 32767;
else if (i < 0x43bf8000)
return -32768;
else
return i - 0x43c00000;
}
int main() {
uint8_t val_1[4] = { 0xFF, 0xFF, 0xBF, 0x43 };
return printf("%f\n%d\n", *(float*)(&val_1), blah(*(int32_t*)(&val_1)));
}
The output (on Little-Endian, as val_1 is the byte representation of a float number in Little-Endian) is: 383.999969 and -1; it makes no sense to me, but maybe someone else has a clue..
And for who’s wondering, the byte-value I chosen there is one of the samples decoded by a52dec, so it’s a proper value I’m trying to understand; and for who’s wondering, the proper correspondence for -1 (int16) in float32 format is (approximated) -3.0517e-05.
Edit: thanks Nathan Smith who made me see the spelling error in the title (I still confuse the two).
|
