Entries by Jeff Mitchell

I'm pleased as punch/as a fat cat/etc. to point you to The Dot (specifically here) to see the official announcement and some details. More details will be forthcoming soon (and especially as we get the web site in order). Start clearing your schedule and working on your presentations!

A heads-up: Amarok File Tracking can now use MusicBrainz track identifiers for its embedded IDs. This means people that have used Picard to tag their files but not amarok_afttagger can still get some embedded AFT goodness! It also enables an interesting "mode" because it essentially enables song tracking vs. actual file tracking (which you may or may not want, depending on your particular needs).

Full details are here.

Many KDE developers are on Facebook. A while back I wondered if it would be possible to have an official KDE developers' network on Facebook -- after all, there are networks for schools, jobs, cities, and more (and for many developers, KDE is literally or figuratively a job...)

As it turned out, there was a "Kde" network -- but something was odd. To join a work network you have to have an email address affiliated with the network. KDE owns kde.com and kde.org -- so who was this? The only other "KDE" I could find that seemed like it would be legit was the Kentucky Department of Education, and I rather doubted it was them, because they would likely have used all-uppercase KDE as well. So I started an inquiry with Facebook, trying to figure out if either it was someone squatting on our name (and trademark) or whether it was some legit organization -- in which case, would they mind donating the network to us?

After several months of back-and-forth with the people at Facebook, who were very nice (if a bit slow ), I'm happy to say that we've regained the KDE network (properly capitalized) as our own. I still don't know the whole story as to who was there before, and never will due to their privacy policies, but I'll say this:

• If you were in the "Kde" network before and Facebook asked if you would mind donating it to us, and you did, thanks so much!


• If someone was simply squatting in the "Kde" network before, then thanks, Facebook, for kicking them out!

To join the network, go to Settings -> Networks, and enter KDE and your kde.org email address in the appropriate fields.

I've done some work in trunk over the past week that may have a huge impact on many of you Amarokers. Read on, and if you can do some benchmarks for me, fantastic.

First, the schema/table changes.

1. We've seen some issues where people have, for whatever reason, ended up with InnoDB tables instead of MyISAM tables. This is probably the result of their DB being created long ago before we were explicitly telling the mysqle startup to skip InnoDB. This mainly causes a problem because some columns cannot be as wide as we'd like them to be when using InnoDB. So, the first thing being done is that an ALTER TABLE is being forced on every table to explicitly convert to MyISAM. In addition, ENGINE parameters are now used during table creation to be more explicit in the future.


2. Some of you might have seen complaints in the debug output about indexes not being able to be created due to a max key length, which by default in MySQL is 1000 (compile-time option). So, some columns have had their widths adjusted so that all indexes are now successfully created.

Now, the other changes:

As we added more features, scanning got slow. Like, really slow. You'd spend more time running SQL queries than actually scanning your files. So I've been aiming to change that.

Over the past week I've committed changes that remove, per track, anywhere from 1 to 6 SQL queries. The exact amount is highly dependent on your file set, but there is a minimum of one less SQL query per track. If you've done a lot of file moves and AFT kicks in, it'll be an even more massive speedup. I'm going to try to do some further tuning, but already results are looking positive.

Nikolaj has reported that his scan time went from 68 seconds to 18 seconds -- more than 3x faster. Mikko didn't notice a speedup, but he said that whereas scanning used to peg his CPU at 100%, it no longer does so. What I want to know is: how does this affect *you*?

If you want to help, do the following:

1. Backup your DB. If you're using external MySQL do a mysqldump, if you're using internal MySQLe backup the mysqle folder in the Amarok data directory.


2. Update to a revision from a week ago...say, 995000.


3. Wipe your DB.


4. Start Amarok -- it will do a full scan because of the empty DB. Time it as it does the scan.


5. Repeat steps 3 & 4, so that you can see what the time is like after caching.


6. Update to current trunk (at least 998470).


7. Repeat step 3.
   


8. Repeat steps 4 and 5.

Then leave a reply here with your values. If you watch your CPU during each of the scans, report that here too. Thanks!

Today at the Akademy General Meeting, it was mentioned that Gitorious.org is being seriously looked at as a hosting solution for our Git repositories (as opposed to running an instance of Gitorious ourselves). Since I have been a major part of pushing in that direction, I feel that it would be prudent to make sure that those interested are aware of the relevant discussion and the current status.  So, for those interested, read on.

Please note that this is not a post about why KDE is migrating to Git, why this is a good idea/bad idea/neutral idea, etc. This is purely discussing the hosting aspect of Git.

First, I would encourage you to read this kde-scm-interest mail, which I sent to the list on July 2nd. It goes into a good amount of depth as to why Gitorious.org could be beneficial for us, and the rest of this post will assume that you have read that email and the others in the thread, as it will simply update the information therein.

On Monday a large group of interested people, including KDE sysadmins and the guys from Shortcut AS, went to lunch to discuss the technical issues. The output from that discussion is as follows:

• The vast majority of those present feel that Gitorious.org would be the best choice, with the following being the main reasons:
  




• Shorcut could provide a SLA (Service Level Agreement) guaranteeing a minimum level of service, such as uptime and available bandwidth, providing professional hosting services and easing burden on our system administrators.


• As David Faure noted, user account creation is becoming a large burden on our system administrators, which is not something that we would have to administer if using Gitorious.org.
  




• It should be noted that the above was not a unanimous opinion.


• KDE does have infrastructure and bandwidth; it could keep one read-only Subversion server available for historical reasons, and convert the rest to serve as backups or possibly load-balancers. Or to put it in a more general fashion, KDE can reduce hosting costs (which will likely be covered by sponsors) by working with Shortcut. It is not a question that this could be done, but rather what the right method would be for doing so.


• The Gitorious developers have a feature branch where they have already fixed one or both of the current showstopping bugs relating to rights within the shared Git repository. They have said that this should be merged into mainline within a week (not sure if they meant a week from then, or from the end of GCDS).


• The hosting could be set up in such a way that it can be accessed via git.kde.org.


• Post-commit-hook functionality will be available; the Shortcut guys are currently working with us to determine how we can migrate or emulate pre-commit-hook functionality.
  

We have two projects that are chomping at the bit to get onto Git ASAP: Amarok and TagLib. Amarok will be converted first and will serve as the initial guinea pigs to iron out any issues. Barring any major issues being found, TagLib will be converted in short order.

I hope this gives everyone a better idea of KDE's Git-hosting plans. If you haven't checked out Gitorious.org, I encourage you to do so; it's made huge leaps and bounds in the past six months and has become quite a great tool.

Please direct any questions or feedback to the kde-scm-interest mailing list at: kde-scm-interest at kay dee eee dot ooo arr gee, not to the comments section on this blog.

Yes, another one of my semi-habitual posts about AFT.  Just a short one though.

In revision 992942, I finally fixed a bug that has kept AFT working for the playlist in certain situations (although it had previously been working for both saved user playlists and statistics). This means that if you have a track in the playlist, move it to another location, and it is then scanned in that new location (remember, kids, it uses folder mtime to determine whether to scan a folder, so when in doubt do a "touch ."), the track in the playlist should remain valid and play the song in the new location. As the playlist use case was one of the initial reasons for the development of AFT back in Amarok 1.4, you can imagine I'm happy that it's finally (seeming to be) working again in all scenarios, instead of failing in certain situations.

We told you it was coming. Sure, that was a while back, so you probably thought we forgot about it.  Or maybe you thought we were simply playing politics, tossing empty promises to our users.

Well...you were wrong.

It may be a bit later than planned -- we wanted to have it in time for 2.1, but it didn't happen -- but as of revision 984572, there is now support for storing an Amarok database on a MySQL server instead of the embedded MySQL database.  There's no configuration dialog in the GUI yet, but it's pretty simple to set up, as explained below.  All you have to do is add a few things into your amarokrc file and make a valid user on the MySQL server instance of your choice -- you don't even need to create the database yourself.  (In fact, you shouldn't -- you should let Amarok create the database so we can ensure that the character set and collation are set right.)

Here's how to do it.

1. Update to at least r984572 (of course, updating to the latest revision is probably your best bet).


2. Wipe your build dir clean and rebuild.  Not necessarily necessary, but as 47 files were changed in that commit, it's not a bad idea.


3. After install, run kbuildsycoca4 --noincremental, just in case. 


4. On your MySQL server, run a command like: "GRANT ALL ON amarokdb.* TO 'amarokuser'@'localhost' IDENTIFIED BY 'mypassword'; FLUSH PRIVILEGES;"  Be sure to substitute for "amarokdb", "amarokuser", "localhost", and "mypassword" as appropriate.


5. Open up your amarokrc file, usually in ~/.kde4/share/config/amarokrc.  Add a [MySQL] section:
   [MySQL]
   UseServer=true
   Database=amarokdb
   Host=localhost
   Password=mypassword
   User=amarokuser


6. Close the file and start Amarok.  It should create the database and start a scan of your files.  If you want to switch back to the embedded collection, simply set "UseServer" to false.

Pretty easy!  Be sure to let me know if you have problems -- file a bug and assign it to "mitchell" at a domain of "kde" plus a dot plus "org".

A heads-up on something new in Amarok SVN (and coming in 2.2 for those of you not living on the bleeding edge):

We've had various bug reports over the years relating to character sets and collation, causing issues with matching searches for music or mis-sorted items.  Well, hopefully no longer.

When you update to 2.2 (recent SVN users, see the note at the end of this post), your Amarok database and tables will be converted to use the 'utf8' character set and 'utf8_unicode_ci' collation as default for any table or column created from this point on.  Every single text/varchar field will also be converted through a two-step process to use 'utf8' as the character set (the data inside was always UTF-8, but there was a possible mismatch between what the data was and what the database thought it was, if your mysql wasn't built to use 'utf8' by default).  In addition, the character set used when talking to the embedded server (the protocol in the socket) will be 'utf8'.

Fixing this mismatch between what the server might have been using for character set/collation and the data we're putting in there should hopefully ensure that sorting and tags work very well for our users with some files wth non-Latin1 tags (probably just about everybody these days).

 

* Recent SVN users: if your build date is earlier than this post I'd recommend wiping your mysqle directory (not just a full rescan), as the initial commit of the updating code contained a bug that could possibly cause trouble down the line with user playlists...but you bleeding edge users should be expecting database wipes every now and then 

I've blogged about Amarok File Tracking before and there's a lot of information about it on the wiki.  For those that haven't heard about the goodness of embedded file tracking, check out those links.  There are a couple pieces of good news, and one piece of bad news.

The good news: in current SVN (and thus 2.2) the amarok_afttagger executable will also now handle FLAC and various Ogg-contained formats. Another piece of good news - the amarok_afttagger executable is now contained in the amarok-utilities package, and thus can be run on headless machines without X!  And lastly -- AFT now works with user playlists, so you can move your files around (keep in mind the caveats if you're not using embedded AFT tags) and your playlists will always stay current, in addition to statistics and The Playlist.

The bad news? Something is currently a bit broken somewhere deep inside with Observers which means that The Playlist will only update with the correct new URL once (the metadata observers seem to die after that).  This doesn't seem to be AFT specific but rather seems like a bug that AFT is exposing.  Closing Amarok and reopening it will cause the proper new URLs to be in the playlist.  I'm working on trying to fix that.

(Important note: it writes into the FLAC Xiph comment.  This is the only metadata type actually required by the FLAC spec, and thus is the proper place to put it, but a lot of FLAC files incorrectly only have ID3v2 tags, so depending on the tagger you're using you may only see one or the other.)

A long time ago I promised to post an update when I got incremental batch scanning working.  Well, as it turns out, that happened a long time ago too, but I never got around to writing the Wiki page for it.  I've corrected that flaw.

Anyone interested in scanning their collection locally instead of across a network connection, or keeping their collection up-to-date when Amarok is closed, should definitely give it a read!

http://amarok.kde.org/wiki/Batch_Mode

For anyone that hasn't seen the Dot story, check it out.

A friendly reminder: you have just under 24 hours to get your applications updated (if you are a student...you've already submitted it, right?) or to get students to update their applications (if you are a mentor).  Applications that arrive before the deadline will not be penalized (and applications that arrive afterwards won't be accepted at all) so it's not too late to get your SoC on...

Google has just asked all students to ensure that their application is submitted *now*, even if they are not done.  You will still have until 19:00 UTC April 3rd to modify them, but Google is having trouble gauging participation because so many students (in all organizations) are discussing applications with mentors and refining them outside of the official site.  So if you are a student, or mentoring students, please ensure that your applications are submitted ASAP.

Remember: the student application deadline is April 3rd at 19:00 UTC.  If you are a student, you should be checking for comments and revising your application while you still have the chance.  If you are a mentor, you should be checking the applications and commenting on them as appropriate.

...NOW!  Students have until April 3rd to get their applications finalized and turned in.  Good luck!