The server that came from the Mist

Here's a little story I wanted to share with you. It's so bizarre, you might not believe it when you read it, but it really happened


Some back story:

A few months ago Amarok had migrated to Git, and then Konversation had migrated too. We had to write a special "commit hook" script for making the commits show up on IRC, via the CIA bot. This script was running on our Amarok server (called "Kollide"), and all was fine. At some point then we migrated our server to new hardware, as the old contract had expired. That also went really well.


What happened later:

At some point the CIA bot started to print everything twice. Yes, every single commit, all duplicated. I guess you can imagine how annoying that was. So we tried to debug this for weeks, looked at all scripts, at Gitorious, and whatnot. Noone was able to spot the bug, it was a mystery. Two days ago then we discussed this problem again in our #kde-git IRC channel, with Jeremy Whiting, and Eike Hein. Basically we were close to giving up on the whole thing. For fun, Jeremy said something like: "Hmm, what happened to your old server, btw?". That got us thinking. Then Eike compared the IP addresses of the server that sent out the commit message, and what did we see there: Two different IPs. What had happened:

The old server (we like to call it our "Shadow Server" now) had continued to run happily, because the hoster had forgotten to switch it off. So it was still running our web site, all sorts of other stuff, plus this little commit hook script!

When we realized what had happened there: There was just one possible reaction: Headdesk


Don't know about you, but we found that quite bizarre and funny, as noone had anticipated that (not even our admin)