You are here
Project Neon - Explained
May 6, 2008 - 17:37 — apachelogger
For comments on comments :-D from yesterday's post please read at the very bottom.
Let's get started with the technical details.
I am not at home right now, so I can't use my awesome paper+pencil-drawing(tm) for this explaination so bear with me in case some parts are a bit too complex (I actually created most code while operating on ballmer's peek, so it's as difficult for me to understand it as it might be for you ;-)
Anyway, the Neon framework, which is responsbile for the important parts of the nightly build process (I am considering building as not important here :-P) is written in Ruby (omg!).
http://websvn.kde.org/trunk/extragear/multimedia/amarok/supplementary_scripts/neon
Those of you who had a look at the upcoming Amarok release script will notice quite some similarities, most of them are caused by the fact that I absolutely hate scrolling in files so both source trees consist of some dirs and a couple of files named according to their use (surprise!).
Neon basically consist of 4 components which more or less rely on each other:
The first real work that is done, is checking which source trees need to be fetched (Qt gets built once a month - Strigi, TagLib, KDELibs and KDEBase are built once a week - and only Amarok is built once a day). Qt and KDELibs are actually just downloaded from the KDE snapshots on ftp.kde.org, the others get fetched directly from SVN (kdebase, or rather kdebaseruntime is a very cut down version which only ships with stuff that is necessary to run Amarok - the Xine Phonon backend for example).
In case all the SVN magic finished without problems the publishers would kick in and pull the newly created tarballs to some source distribution server (ftp/web server).
Then the distributions get their source packages. All distribution realted tasks are within special files located in neon/distros/ - the only working one is Kubuntu right now, but the processes are the same for most package types anyway.
First it pulls a copy of the source trees, and prepares everything as necessary for the package type/distribution, once that is done, it uploads to a remote build server (in case of Kubuntu this is done package by package to prevent complete meltdowns of the repository in case of an issue.
The most important part is that fetcher.rb created an array listing all fetched source trees (including their SVN revision number), so that all distributions can create an appropriate version string (Kubuntu is using DATE+svnREVISON-0amarok1).
There are a couple of guidelines all distribution packages should follow, for example they should require few to none maintenance. Ultimately the only reason one would have to edit the packaging, once everything works properly, is to make it work for new distribution releases.
Again Kubuntu as example: source packages are created according to the soure tarballs provided by fetcher.rb, they get thrown in a build deamon and run threw some automated cmake/kde build scripts and one gets _one_ resulting binary per source package.
Also, all packages should come with development headers and debugging symbols (I guess it's pretty obvious why that is ;-)
To sum that up: the most tricky part is probably to get the packaging right - general information for distributions are on the wiki page.
Everyone who is interessted in contribution, either contact me personally or use the Neon mailing list
------------------
Regarding Comments:
OpenSuse build service
Good thing, but it's really up to the maintainers where they want to build. For Kubuntu it makes most sense to use the Launchpad Personal Package Archive, for openSUSE however we will of course use the OSBS ;-)
Ubuntu -> Debian
Not much of a problem, with a bit of tuning they could actually rely on the same debian directories. The only tricky part is the remote build server, I guess using the openSUSE build service is a good idea?
-dev packages
The first public release of Neon had quite some dependencies on -dev packages, they are there because Neon is also meant to help developers join Amaork development. I removed them from the deps stack (strigi, taglib, kdelibs, kdebaseruntime, amarok) they should disappear from qt at some rebuild as well.
For the developers there will be a seperate package for all necessary -dev packages. Should be available soonish.
Missing Icons?
Should be fixed now.
Wrong Colouring!
I can't reproduce the issue, and actually thought it was fixed months ago. If anyone gets hold of information why this appears please leave a mail at the Neon mailing list
How seperate is it really?
Let's say it that way: the possability that Neon will cause issues with _any_ existing KDE/Amarok/Qt configuration is close to not existing at all.
This also includes your collection.db :-)
Let's get started with the technical details.
I am not at home right now, so I can't use my awesome paper+pencil-drawing(tm) for this explaination so bear with me in case some parts are a bit too complex (I actually created most code while operating on ballmer's peek, so it's as difficult for me to understand it as it might be for you ;-)
Anyway, the Neon framework, which is responsbile for the important parts of the nightly build process (I am considering building as not important here :-P) is written in Ruby (omg!).
http://websvn.kde.org/trunk/extragear/multimedia/amarok/supplementary_scripts/neon
Those of you who had a look at the upcoming Amarok release script will notice quite some similarities, most of them are caused by the fact that I absolutely hate scrolling in files so both source trees consist of some dirs and a couple of files named according to their use (surprise!).
Neon basically consist of 4 components which more or less rely on each other:
- neon.rb/libneon.rb as core component
- fetcher.rb is handling the collection of sources - more on that later
- publisher.rb is basically a wrapper for various ways of source distribution (e.g. tarball upload via FTP)
- distro.rb is a wrapper for distributions
The first real work that is done, is checking which source trees need to be fetched (Qt gets built once a month - Strigi, TagLib, KDELibs and KDEBase are built once a week - and only Amarok is built once a day). Qt and KDELibs are actually just downloaded from the KDE snapshots on ftp.kde.org, the others get fetched directly from SVN (kdebase, or rather kdebaseruntime is a very cut down version which only ships with stuff that is necessary to run Amarok - the Xine Phonon backend for example).
In case all the SVN magic finished without problems the publishers would kick in and pull the newly created tarballs to some source distribution server (ftp/web server).
Then the distributions get their source packages. All distribution realted tasks are within special files located in neon/distros/ - the only working one is Kubuntu right now, but the processes are the same for most package types anyway.
First it pulls a copy of the source trees, and prepares everything as necessary for the package type/distribution, once that is done, it uploads to a remote build server (in case of Kubuntu this is done package by package to prevent complete meltdowns of the repository in case of an issue.
The most important part is that fetcher.rb created an array listing all fetched source trees (including their SVN revision number), so that all distributions can create an appropriate version string (Kubuntu is using DATE+svnREVISON-0amarok1).
There are a couple of guidelines all distribution packages should follow, for example they should require few to none maintenance. Ultimately the only reason one would have to edit the packaging, once everything works properly, is to make it work for new distribution releases.
Again Kubuntu as example: source packages are created according to the soure tarballs provided by fetcher.rb, they get thrown in a build deamon and run threw some automated cmake/kde build scripts and one gets _one_ resulting binary per source package.
Also, all packages should come with development headers and debugging symbols (I guess it's pretty obvious why that is ;-)
To sum that up: the most tricky part is probably to get the packaging right - general information for distributions are on the wiki page.
Everyone who is interessted in contribution, either contact me personally or use the Neon mailing list
------------------
Regarding Comments:
OpenSuse build service
Good thing, but it's really up to the maintainers where they want to build. For Kubuntu it makes most sense to use the Launchpad Personal Package Archive, for openSUSE however we will of course use the OSBS ;-)
Ubuntu -> Debian
Not much of a problem, with a bit of tuning they could actually rely on the same debian directories. The only tricky part is the remote build server, I guess using the openSUSE build service is a good idea?
-dev packages
The first public release of Neon had quite some dependencies on -dev packages, they are there because Neon is also meant to help developers join Amaork development. I removed them from the deps stack (strigi, taglib, kdelibs, kdebaseruntime, amarok) they should disappear from qt at some rebuild as well.
For the developers there will be a seperate package for all necessary -dev packages. Should be available soonish.
Missing Icons?
Should be fixed now.
Wrong Colouring!
I can't reproduce the issue, and actually thought it was fixed months ago. If anyone gets hold of information why this appears please leave a mail at the Neon mailing list
How seperate is it really?
Let's say it that way: the possability that Neon will cause issues with _any_ existing KDE/Amarok/Qt configuration is close to not existing at all.
This also includes your collection.db :-)