apachelogger

....sitting in front of my laptop, looking at launchpad because I was waiting for builds to finish.

Compile clusters are so much nicer -.-

I love LinuxTag!

Stephan already blogged about how I was sleeping on the table.
So, here comes the background story:
After hours of hard work on Project Neon, an Amarok project to create nightly binaries of the latest Amarok version (I was sorting out some underlying issues in the base components in preperation for the openSUSE build service integration) , and as I was somewhat tired and it also was quite late in the night (or rather early in the morning) I choose to go to bed, though one shouldn't call that 'go', anyway, I didn't find one within the critical 10 seconds and chose the next best option: table + chair + me in between (I better don't tell you about the bathtub

My head is telling me that I got too much beer, so please excuse any spelling/grammar errors.
*looking for something to drink*

So...
First part of exams done! *yay*

But meanwhile Stephan invited kwwii for a bit of karaoke @ LinuxTag in Berlin. Kwwii of course agreed, which made Stephan wondering what will happen with records that go straight into my hands.

Well, first of all, I was practising karaoke yesterday (instead of learning for today's maths exam), so i am almost not the best karaoke singer ever

Anyway, the problem is, I have absolutely no sense of embarrassment so I will not only play eventual records on Radio Amarok, I will make them play 24/7 post them onthe 15 blogs I own and make them part of Project Neon.

That reminds me...
Neon is undergoing major refacturing, mainly in order to provide KDE nightly packages for Kubuntu (thanks to Kevin for pushing me so hard ) . Currently the source fetching component is work again (at least to the low profile testing I did) and the ubuntu component is close to be finished, so maybe apt-get soon tells us about kde-nightly

Just for the record, amarok-nightly will use a different package tree in order to preserve the current tree size.

Also new is the Phonon KCM bundling

For comments on comments from yesterday's post please read at the very bottom.

Let's get started with the technical details.


I am not at home right now, so I can't use my awesome paper+pencil-drawing(tm) for this explaination so bear with me in case some parts are a bit too complex (I actually created most code while operating on ballmer's peek, so it's as difficult for me to understand it as it might be for you

Anyway, the Neon framework, which is responsbile for the important parts of the nightly build process (I am considering building as not important here ) is written in Ruby (omg!).

http://websvn.kde.org/trunk/extragear/multimedia/amarok/supplementary_scripts/neon

Those of you who had a look at the upcoming Amarok release script will notice quite some similarities, most of them are caused by the fact that I absolutely hate scrolling in files so both source trees consist of some dirs and a couple of files named according to their use (surprise!).

Neon basically consist of 4 components which more or less rely on each other:

• neon.rb/libneon.rb as core component
• fetcher.rb is handling the collection of sources - more on that later
• publisher.rb is basically a wrapper for various ways of source distribution (e.g. tarball upload via FTP)
• distro.rb is a wrapper for distributions
  

neon.rb is responsible for very very very basic tasks, like setting necessary variables/constants, deciding which parts fetcher.rb actually has to fetch and how to publish the tarballs as well as which distribution processes to invoke (well, and cleanup work, but that might eventually get it's own file).

The first real work that is done, is checking which source trees need to be fetched (Qt gets built once a month - Strigi, TagLib, KDELibs and KDEBase are built once a week - and only Amarok is built once a day). Qt and KDELibs are actually just downloaded from the KDE snapshots on ftp.kde.org, the others get fetched directly from SVN (kdebase, or rather kdebaseruntime is a very cut down version which only ships with stuff that is necessary to run Amarok - the Xine Phonon backend for example).

In case all the SVN magic finished without problems the publishers would kick in and pull the newly created tarballs to some source distribution server (ftp/web server).

Then the distributions get their source packages. All distribution realted tasks are within special files located in neon/distros/ - the only working one is Kubuntu right now, but the processes are the same for most package types anyway.

First it pulls a copy of the source trees, and prepares everything as necessary for the package type/distribution, once that is done, it uploads to a remote build server (in case of Kubuntu this is done package by package to prevent complete meltdowns of the repository in case of an issue.

The most important part is that fetcher.rb created an array listing all fetched source trees (including their SVN revision number), so that all distributions can create an appropriate version string (Kubuntu is using DATE+svnREVISON-0amarok1).

There are a couple of guidelines all distribution packages should follow, for example they should require few to none maintenance. Ultimately the only reason one would have to edit the packaging, once everything works properly, is to make it work for new distribution releases.

Again Kubuntu as example: source packages are created according to the soure tarballs provided by fetcher.rb, they get thrown in a build deamon and run threw some automated cmake/kde build scripts and one gets _one resulting binary per source package.

Also, all packages should come with development headers and debugging symbols (I guess it's pretty obvious why that is

To sum that up: the most tricky part is probably to get the packaging right - general information for distributions are on the wiki page.

Everyone who is interessted in contribution, either contact me personally or use the Neon mailing list

------------------
Regarding Comments:

OpenSuse build service
Good thing, but it's really up to the maintainers where they want to build. For Kubuntu it makes most sense to use the Launchpad Personal Package Archive, for openSUSE however we will of course use the OSBS

Ubuntu -> Debian
Not much of a problem, with a bit of tuning they could actually rely on the same debian directories. The only tricky part is the remote build server, I guess using the openSUSE build service is a good idea?

-dev packages
The first public release of Neon had quite some dependencies on -dev packages, they are there because Neon is also meant to help developers join Amaork development. I removed them from the deps stack (strigi, taglib, kdelibs, kdebaseruntime, amarok) they should disappear from qt at some rebuild as well.
For the developers there will be a seperate package for all necessary -dev packages. Should be available soonish.

Missing Icons?
Should be fixed now.

Wrong Colouring!
I can't reproduce the issue, and actually thought it was fixed months ago. If anyone gets hold of information why this appears please leave a mail at the Neon mailing list

How seperate is it really?
Let's say it that way: the possability that Neon will cause issues with any existing KDE/Amarok/Qt configuration is close to not existing at all.
This also includes your collection.db

For comments on comments from yesterday's post please read at the very bottom.



Let's get started with the technical details.





I am not at home right now, so I can't use my awesome paper+pencil-drawing(tm) for this explaination so bear with me in case some parts are a bit too complex (I actually created most code while operating on ballmer's peek, so it's as difficult for me to understand it as it might be for you



Anyway, the Neon framework, which is responsbile for the important parts of the nightly build process (I am considering building as not important here ) is written in Ruby (omg!).



http://websvn.kde.org/trunk/extragear/multimedia/amarok/supplementary_scripts/neon



Those of you who had a look at the upcoming Amarok release script will notice quite some similarities, most of them are caused by the fact that I absolutely hate scrolling in files so both source trees consist of some dirs and a couple of files named according to their use (surprise!).



Neon basically consist of 4 components which more or less rely on each other:

• neon.rb/libneon.rb as core component
• fetcher.rb is handling the collection of sources - more on that later
• publisher.rb is basically a wrapper for various ways of source distribution (e.g. tarball upload via FTP)
• distro.rb is a wrapper for distributions
  
  

neon.rb is responsible for very very very basic tasks, like setting necessary variables/constants, deciding which parts fetcher.rb actually has to fetch and how to publish the tarballs as well as which distribution processes to invoke (well, and cleanup work, but that might eventually get it's own file).



The first real work that is done, is checking which source trees need to be fetched (Qt gets built once a month - Strigi, TagLib, KDELibs and KDEBase are built once a week - and only Amarok is built once a day). Qt and KDELibs are actually just downloaded from the KDE snapshots on ftp.kde.org, the others get fetched directly from SVN (kdebase, or rather kdebaseruntime is a very cut down version which only ships with stuff that is necessary to run Amarok - the Xine Phonon backend for example).



In case all the SVN magic finished without problems the publishers would kick in and pull the newly created tarballs to some source distribution server (ftp/web server).



Then the distributions get their source packages. All distribution realted tasks are within special files located in neon/distros/ - the only working one is Kubuntu right now, but the processes are the same for most package types anyway.



First it pulls a copy of the source trees, and prepares everything as necessary for the package type/distribution, once that is done, it uploads to a remote build server (in case of Kubuntu this is done package by package to prevent complete meltdowns of the repository in case of an issue.



The most important part is that fetcher.rb created an array listing all fetched source trees (including their SVN revision number), so that all distributions can create an appropriate version string (Kubuntu is using DATE+svnREVISON-0amarok1).



There are a couple of guidelines all distribution packages should follow, for example they should require few to none maintenance. Ultimately the only reason one would have to edit the packaging, once everything works properly, is to make it work for new distribution releases.



Again Kubuntu as example: source packages are created according to the soure tarballs provided by fetcher.rb, they get thrown in a build deamon and run threw some automated cmake/kde build scripts and one gets one resulting binary per source package.



Also, all packages should come with development headers and debugging symbols (I guess it's pretty obvious why that is



To sum that up: the most tricky part is probably to get the packaging right - general information for distributions are on the wiki page.



Everyone who is interessted in contribution, either contact me personally or use the Neon mailing list



------------------

Regarding Comments:



OpenSuse build service

Good thing, but it's really up to the maintainers where they want to build. For Kubuntu it makes most sense to use the Launchpad Personal Package Archive, for openSUSE however we will of course use the OSBS



Ubuntu -> Debian

Not much of a problem, with a bit of tuning they could actually rely on the same debian directories. The only tricky part is the remote build server, I guess using the openSUSE build service is a good idea?



-dev packages

The first public release of Neon had quite some dependencies on -dev packages, they are there because Neon is also meant to help developers join Amaork development. I removed them from the deps stack (strigi, taglib, kdelibs, kdebaseruntime, amarok) they should disappear from qt at some rebuild as well.

For the developers there will be a seperate package for all necessary -dev packages. Should be available soonish.



Missing Icons?

Should be fixed now.



Wrong Colouring!

I can't reproduce the issue, and actually thought it was fixed months ago. If anyone gets hold of information why this appears please leave a mail at the Neon mailing list



How seperate is it really?

Let's say it that way: the possability that Neon will cause issues with any existing KDE/Amarok/Qt configuration is close to not existing at all.

This also includes your collection.db

A least 5 people ask us every day when Amarok 2 will be ready and when it will be easy to install without compiling it - since yesterday we can tell them: there you go!

Project Neon makes it possible for YOU and me and everyone (kind of, at least

Neon is meant to provide nightly Amarok builds ....
which means that it generates new Amarok packages for various distributions (currently only Kubuntu, openSUSE is in the queue though) ever day so that everyone can install them for whatever reason (testing, checking out the latest development, ...).
... So the main aim of Neon is clearly to provide a way to install the latest Amarok development version.

After installing amarok-nightly you will find an entry in your application menu, I'd like to call that well integrated with the operating system ;-), but the nifty thing about it is, that amarok-nightly will run without problems along a production system, it is stored in a completely unrelated path and all configurations are (or rather should) be stored in it's own directory.

This makes it possible to check out Amarok 2 once a day while running Amarok 1 to get the usual music entertainment.

More information on Neon and how to use it are available on it's wiki page.

Later I will write a more technical post, explaining how it works and how to get other distributions supported.