Entries by Jeff Mitchell

We told you it was coming. Sure, that was a while back, so you probably thought we forgot about it.  Or maybe you thought we were simply playing politics, tossing empty promises to our users.

Well...you were wrong.

It may be a bit later than planned -- we wanted to have it in time for 2.1, but it didn't happen -- but as of revision 984572, there is now support for storing an Amarok database on a MySQL server instead of the embedded MySQL database.  There's no configuration dialog in the GUI yet, but it's pretty simple to set up, as explained below.  All you have to do is add a few things into your amarokrc file and make a valid user on the MySQL server instance of your choice -- you don't even need to create the database yourself.  (In fact, you shouldn't -- you should let Amarok create the database so we can ensure that the character set and collation are set right.)

Here's how to do it.

1. Update to at least r984572 (of course, updating to the latest revision is probably your best bet).


2. Wipe your build dir clean and rebuild.  Not necessarily necessary, but as 47 files were changed in that commit, it's not a bad idea.


3. After install, run kbuildsycoca4 --noincremental, just in case. 


4. On your MySQL server, run a command like: "GRANT ALL ON amarokdb.* TO 'amarokuser'@'localhost' IDENTIFIED BY 'mypassword'; FLUSH PRIVILEGES;"  Be sure to substitute for "amarokdb", "amarokuser", "localhost", and "mypassword" as appropriate.


5. Open up your amarokrc file, usually in ~/.kde4/share/config/amarokrc.  Add a [MySQL] section:
   [MySQL]
   UseServer=true
   Database=amarokdb
   Host=localhost
   Password=mypassword
   User=amarokuser


6. Close the file and start Amarok.  It should create the database and start a scan of your files.  If you want to switch back to the embedded collection, simply set "UseServer" to false.

Pretty easy!  Be sure to let me know if you have problems -- file a bug and assign it to "mitchell" at a domain of "kde" plus a dot plus "org".

A heads-up on something new in Amarok SVN (and coming in 2.2 for those of you not living on the bleeding edge):

We've had various bug reports over the years relating to character sets and collation, causing issues with matching searches for music or mis-sorted items.  Well, hopefully no longer.

When you update to 2.2 (recent SVN users, see the note at the end of this post), your Amarok database and tables will be converted to use the 'utf8' character set and 'utf8_unicode_ci' collation as default for any table or column created from this point on.  Every single text/varchar field will also be converted through a two-step process to use 'utf8' as the character set (the data inside was always UTF-8, but there was a possible mismatch between what the data was and what the database thought it was, if your mysql wasn't built to use 'utf8' by default).  In addition, the character set used when talking to the embedded server (the protocol in the socket) will be 'utf8'.

Fixing this mismatch between what the server might have been using for character set/collation and the data we're putting in there should hopefully ensure that sorting and tags work very well for our users with some files wth non-Latin1 tags (probably just about everybody these days).

 

* Recent SVN users: if your build date is earlier than this post I'd recommend wiping your mysqle directory (not just a full rescan), as the initial commit of the updating code contained a bug that could possibly cause trouble down the line with user playlists...but you bleeding edge users should be expecting database wipes every now and then 

I've blogged about Amarok File Tracking before and there's a lot of information about it on the wiki.  For those that haven't heard about the goodness of embedded file tracking, check out those links.  There are a couple pieces of good news, and one piece of bad news.

The good news: in current SVN (and thus 2.2) the amarok_afttagger executable will also now handle FLAC and various Ogg-contained formats. Another piece of good news - the amarok_afttagger executable is now contained in the amarok-utilities package, and thus can be run on headless machines without X!  And lastly -- AFT now works with user playlists, so you can move your files around (keep in mind the caveats if you're not using embedded AFT tags) and your playlists will always stay current, in addition to statistics and The Playlist.

The bad news? Something is currently a bit broken somewhere deep inside with Observers which means that The Playlist will only update with the correct new URL once (the metadata observers seem to die after that).  This doesn't seem to be AFT specific but rather seems like a bug that AFT is exposing.  Closing Amarok and reopening it will cause the proper new URLs to be in the playlist.  I'm working on trying to fix that.

(Important note: it writes into the FLAC Xiph comment.  This is the only metadata type actually required by the FLAC spec, and thus is the proper place to put it, but a lot of FLAC files incorrectly only have ID3v2 tags, so depending on the tagger you're using you may only see one or the other.)

A long time ago I promised to post an update when I got incremental batch scanning working.  Well, as it turns out, that happened a long time ago too, but I never got around to writing the Wiki page for it.  I've corrected that flaw.

Anyone interested in scanning their collection locally instead of across a network connection, or keeping their collection up-to-date when Amarok is closed, should definitely give it a read!

http://amarok.kde.org/wiki/Batch_Mode

For anyone that hasn't seen the Dot story, check it out.

A friendly reminder: you have just under 24 hours to get your applications updated (if you are a student...you've already submitted it, right?) or to get students to update their applications (if you are a mentor).  Applications that arrive before the deadline will not be penalized (and applications that arrive afterwards won't be accepted at all) so it's not too late to get your SoC on...

Google has just asked all students to ensure that their application is submitted *now*, even if they are not done.  You will still have until 19:00 UTC April 3rd to modify them, but Google is having trouble gauging participation because so many students (in all organizations) are discussing applications with mentors and refining them outside of the official site.  So if you are a student, or mentoring students, please ensure that your applications are submitted ASAP.

Remember: the student application deadline is April 3rd at 19:00 UTC.  If you are a student, you should be checking for comments and revising your application while you still have the chance.  If you are a mentor, you should be checking the applications and commenting on them as appropriate.

...NOW!  Students have until April 3rd to get their applications finalized and turned in.  Good luck!

We've been accepted to participate in GSoC 2009.  Hooray!

This means we have a lot of great work ahead of us.  The first order of business is that everyone that wants to be a mentor needs to sign up at http://socghop.appspot.com.  You'll need a Google account (which does not have to be a Gmail account).  Click the link that says "Apply to become a Mentor" and proceed as instructed.  You should also ensure that you're on the kde-soc-mentor mailing list.

Students: now that we're accepted, get to it with contacting mentors!  The application period opens March 23rd and closes April 3rd, so it's not a huge amount of time to get all the details of your proposals worked out.  Be sure to check the ideas page at http://techbase.kde.org/index.php?title=Projects/Summer_of_Code/2009/Ideas, or contact the appropriate mentor if you have a different idea for a project that you'd like to work on.

I'll be sure to keep everyone informed of events as they occur and deadlines as they arrive.  Here's to a successful summer!

Today the application for KDE to participate in GSoC 2009 was submitted.  They check out our ideas page when they're evaluating applications, so be sure to get your ideas up there!  We'll hear back by March 18th.  Some other upcoming dates:

• March 18-23: Students discuss applications with mentoring organizations (you can and should do this earlier, of course)
• March 23: Student application period opens
• April 3: Student application period closes

Of course, this hinges on us being accepted...but past years have shown that we have a good shot.

Could FLEXlm be one of the world's worst-designed programs?

They've just rechristened it FLEXnet Publisher, and I can only think it's to try to get away from existing FLEXlm stigmas.  It's so bad that according to a VMware engineer I spoke to on the phone a few months ago, in the next release of ESX (whatever the new name is going to be) they're ditching it to go back to their own serial-number based scheme, entirely because of a large amount of hugely negative customer feedback.  This is only one major release after they switched to it.

Let me describe the structure of it (if I were to go into the user interface issues, this would become far too long a post, so maybe another time).  There's a management daemon, called lmgrd, and vendor daemons.  These vendor daemons are (often along with the management daemon) provided to you, often without an installer and simply as a bunch of files.  There's also quite often a node-locking generator (based on things like hostname, MAC address, and all manner of things easily faked in a VM) provided to you by the vendors, which as far as I can tell is different for every vendor.  This is possibly so that they can build in their own criteria, but more likely because the FLEXlm people are lazy and can't be bothered to provide a comprehensive set of criteria on their own.

Now, why is there a vendor daemon and a management daemon?  Good question, and I don't know the authoritative answer, but from using it I can make an educated guess or two: either to allow vendors to validate some part of the licenses in their own manner, or to simply confuse and annoy the hell out of you.  See, the different vendor daemons are built against some version of the FLEXlm software, and they may, or may not, have been built against the same version.  This may, or may not, cause problems, including crashes.  It may explain why every time the license server reboots (thanks, generally, to automatic Windows updates), lmgrd fails to start up successfully.

Now, you can run multiple instances of lmgrd -- if you can figure out how to get it installed so that it starts up on bootup, since there isn't much in the way of help in the official help PDF, and only one of the three vendors whose licenses I deal with actually provided an installer (thanks, VMware, I'll be using your installer long after you've switched off of FLEXlm).  This lets you have each license under a different copy of lmgrd, perhaps so that you can attempt matching versions, but mainly so that if one of the lmgrd instances crashes, it will only take down that license.  (Actually, it's because in this configuration you need to run them on multiple machines, so if one machine goes down, most of your licenses stay up.  But I know what they really meant.)

Alternately, you can combine licenses into a single file, a process that the manual warns is time-consuming and error-prone.

Finally, you can simply have all your licenses be handled by a single lmgrd instance.  Now, this isn't really a bad idea, except for port numbers.

See, each vendor daemon needs to run on its own TCP port number.  You can specify the port number in the license file, except when you can't because some vendor hardcoded it into their software.  If you don't specify a port number in the license file, the vendor daemon will be given a port number starting at 27000 on up, giving you an address like 27000@mylicenses.thissucks.com.  Now, because everyone likes autoconfiguration, if in your program you leave out the port number (giving you something like @mylicenses.thissucks.com), then it will automatically search ports 27000 through 27009 for a matching vendor daemon.  Unfortunately, I haven't seen a single vendor product that doesn't puke on such an address, claiming it is invalid because of faulty validation, even when the FLEXlm manual in front of me says it's perfectly legal.

If you don't then modify your license files to specify ports (which you have to remember to do every time you get a new license file), then you better be aware of when the license server reboots, because the ports are given out in the order that the services come up.  So if you're in my situation, where the first of the vendor daemons always crashes the first time it tries to load on bootup, then the other two will have their normal ports decreased by 1.

I haven't even gotten to things like functions to reread license files that don't actually reread them, a UI that doesn't tell you if a server is running or stopped until you hit the buttons to try to run it or stop it, a status window that displays a large amount of license data in a tiny, non-resizable window, and more.  The product feels like it was developed in 1988 (it was) and is a study in market leader stagnation.  I would be very surprised if they have a single developer left and haven't fired everyone to just sit there and watch the money roll in.

If license servers are necessary for some product you're creating, there are other license servers out there that simply run on a well-known port and don't require any of this idiocy.  Unfortunately, in my experience I've seen more companies switch to FLEXlm than away from it. I hope this isn't an industry-wide trend.

Where's the KDE angle?  There isn't one specifically, other than this: if someone in the KDE community thinks this is a good design, or can see themselves designing such a scheme, please just leave.  Now.  (Don't worry, I'm sure this doesn't apply to any of you. )

A long-requested feature has been a way to decouple Amarok's collection scanning from its GUI.  There are various use-cases for this.  For one, it can actually help us with debugging, by allowing us to control the inputs into the scan parser.  For another, many people have all of their music stored on a single machine, and would like to do the scanning locally where it's fast instead of on their e.g. laptop running Amarok, where it's over wireless and slow.

Yesterday and today (as of r933010) I put half of the solution into trunk.  I say half, because full collection rescans are now supported in batch modes, but I am still working on the methodology for incremental scans (I have a few ideas, but have to sort out which is the most reasonable/doable/makes the most sense). Below, I'll explain how to do it.  Keep in mind that this is designed to be (lightly) scripted, not done by hand...so it can be done by hand (which I did during testing) but it has some safeguards in place so that if you script it, and forget about it, Amarok still works normally.  However, there are actually some interesting things you can do now if you script the scanner...

One important thing to note: the scanner requires various bits of Amarok code, mainly centering around the extra taglib plugins we support.  So you're unlikely to have it work on a machine that doesn't have A2 installed, and certainly won't be able to compile it without the rest of the Amarok source, although you may be able to get it to work in a binary-only fashion with just kdelibs and taglib...YMMV.

So, here's the flow:

1. Run amarokcollectionscanner with the -b or --batch option.  Although this doesn't directly modify the output, it sets some internal flags (some for safety, some to enable other options).  The output goes to stdout; save this output as amarokcollectionscanner_batchfullscan.xml (yes, it must be that specific filename).  You'll probably want to use the -r flag too...check --help for more options.
2. Cool feature: if your scan is of files that have a different mount point on the local machine than the machine that will be using the output, add the --rpath option and give it the mountpoint of the directory on the consuming machine.  For instance, if on the local machine your music is at /opt/music, and this directory (music) is mounted as /mnt/music on the remote machine, run amarokcollectionscanner from /opt and use --batch --rpath /mnt/music.
3. Cool feature: You can scan multiple directories and simply append the output to the end of the file from step #1.  Yes, directly append, including the xml version/encoding header.  Amarok's scan manager will detect and handle this.  You can even scan directories that are not defined as collection directories in Amarok (these entries will work fine, but the next time you do a full rescan from within Amarok without using the saved output, they will be gone).
4. Place the amarokcollectionscanner_batchfullscan.xml file in your Amarok data dir.  Usually, this is ~/.kde4/share/apps/amarok.
5. Trigger a full rescan from inside Amarok (in Settings->Collection).  It should be super-fast.  :-)  Note that this step deletes the amarokcollectionscanner_batchfullscan.xml file.  This is on purpose, so if you want to keep re-using it, you'll have to keep copying it over.

As you can see, it's designed for scripting, but it's not terribly complicated even to do by hand.  I'll post more as I get incremental scanning working, and will at some point put a Wiki page up that has all of these instructions in a less transient form.

After a month's delay, they're done.

In the interim (after the delays mentioned in my last post on this topic), my poor, underpowered desktop machine has endured transcode after transcode (from the original source material) and my Internet connection upload after upload as I tried to figure out just why Blip.tv wouldn't work with X or Y.  (In fact, I can quantify these: X is Vorbis/Theora, which produced awful audio and very desynchronized audio/video upon their conversion to .flv; Y is a lot of things related to the original anamorphic encoding of the videos, which Blip.tv can't handle, and finding the right combination of settings and flags and adjustements to make the aspect ratios come out so that everyone didn't look like Gumby®.)

This was followed by a few days of uploading; Blip seems to max out uploading speeds somewhere between 100kbit and 200kbit, so uploading almost 6GB of data, one chunk at a time, took a bit.

The end result is thirteen videos, each Xvid-encoded (at least it's OSS, although patent-encumbered...see Vorbis/Theora problems above) with a Matroska container.  You can get to them through my show or to individual videos directly (these might not be in strict as-presented order):

• Welcome!
• Diversity in KDE
• Akonadi
• KDE and Business Software
• KDE4 and MS Windows
• libplasma in Applications
• Accelerating Graphics
• CMake / CTest / CPack
• Taming the Leopard
• Bringing the Free Desktop to the Mobile World
• KDE & Distros
• KDE-Games, KDE-Edu, and Avogadro
• A Case for Open-Source Coders in the Enterprise

Enjoy!

Just a note to make people aware that #kde-soc exists. There's basically no activity in there right now, which is rather expected, but as SoC-related activities pick up steam so will the channel.