Frictionless Updates

published 27 Aug 2013

One area of development that has both interested me and consumed a lot of my thought time over the years has been the deployment process for osu! (and games in general). As far as deployment is concerned, I have got things down to a very concise process at my end, allowing me to push a variety of new builds/updates for osu! out to you guys with a few key-presses. In this post, I’d like to focus on the other side of the picture – what you see when an update is available, and how that update is applied to your game.

Let’s begin by understanding how updates are seen by end users, and the various methods of deployment that can change this perception.

When a user clicks the icon of a game, they want to play the game. Anything that gets in the way of this should be avoided from a developer’s perspective. Most games force users to update before the game starts, via an enforced patcher – sometimes referred to as a launcher – that serves as a gateway to starting the game (as seen in every MMO, League of Legends, Starcraft 2 etc.). When updates are only being released once a week – or much less in many cases – I can only see this as a horribly inefficient approach which inconveniences users to no end. I’ll go out on a limb here and say that most gamers’ minds have been trained over the years to accept the added 5-20 seconds spent at the patcher (only to be told their game is up-to-date) as something which must exist. Add up this wasted time and we’re probably looking at some big numbers.

launcher patching is zzz

(I have to make a call out to Blizzard who have recently changed their game patchers to have multiple update “levels”, allowing the game to start without applying non-critical updates. This is an improvement, but the launcher is very hard to understand as a result, with stages such as “optimising”, which should IMHO not be the concern of the end-user.)

Ever since the beginning of osu!, I have avoided this method of updating. Until recently, osu! would only launch the patcher if an update was found. The user would always be running osu! and in the game ready to play. In this scenario, unless a critical update is released, the user is able to get into their game instantly without interruption, and start playing.

This may seem like a huge improvement, but I have learnt over the years that this backfires to a certain extent. Because osu! users are generally used to the gratification of being able to interact with the game so quickly, when they are interrupted by the game closing to run the patcher, they are more aggravated than they may have been with a “launcher” style patch process. Even though the update-as-an-afterthought I had been using is more efficient overall, due to the way users perceive interruptions, it seemed to be shedding more of a negative light on updates than I expected, with regular storms of complaints in in-game chat following the release of patches which bring new features and bug fixes players should be pumped to receive.

So it was time to re-think the update process. I am a fond user of Chrome as a browser and have to applaud them for their awesome update process, which goes something like this:

  • The browser runs a daemon in the background which checks for updates on what we will assume is an hourly basis.
  • When an update is found, files are patched in the background.
  • If the browser isn’t running, the patch can be fully applied, and next time the user opens the browser they are magically (and usually without being aware) on the latest version!
  • If the browser is running, Chrome’s menu button changes from the usual black icon to a glowing green, letting the user know that an update is present.
  • Should the user choose to do so, they can complete the update process from within that menu (triggering a browser restart), else the browser will update itself next time it is closed.

Due to the slickness of this process, I decided to switch osu! across to something similar. I’m not too fond of running a daemon in the background, so had to skip out on this part (which does make the final solution a touch less slick), but moving forwards this can be implemented if osu! gets to the point of running in the background for other (cool) reasons :).

The updater now lives in the main menu, in the form of a subtle message in the bottom-left corner while downloading/applying updates, and a spinning arrow to notify when osu! can be restarted to finish the process. There are no progress bars or external applications involved – it all lives inside osu! itself.

spinny things

If the user wants to play on their older version, they can continue to do so. There is nothing forcing them to update. On the other hand, if they want the latest and greatest fixes/feature additions, they have the one-click option to update their game. The process is very quick (as the patch has already been applied in the background, and is differential using binary diff patching), just requiring a restart of the client.

From what I can tell, this has seen an improvement in how users perceive updates, and there is no longer a outbreak of complaints in chat whenever an update is made available. There are still some areas I would like to improve in the future, though.

  • At the moment, if osu! has an update ready to apply on next startup, but another update is released since then, it isn’t smart enough to check for this before running. In this case, it takes another restart of osu! if the user wishes to apply the next update.
  • If an update is found before osu! finishes launching to the menu (for people that have large beatmap collections, for instance), it would be best to apply it at that point, since it will add nominal time to the startup process.
  • There are rare cases which the external updater is still launched as a fail-safe. I think I have ironed all these bugs out, but I still see a small number of users having this happen to them. Eventually I want to deprecate the external updater (osume.exe) completely, so this will need to be resolved.
  • Consider the daemon option. Updating in the background would be an amazing improvement from a user experience point of view. If I was to take this direction, rather than running a daemon on startup (which is a bit obnoxious), osu! would just remain in memory after “exited” by the user (until system reboot) and keep things warm. Memory footprint would be very small in this state, of course.

Any thoughts on the update process? Any issues or suggestions? Let me know, since I’m always looking to improve!


The Last Two Months in osu!

published 03 Jul 2013

I’ve held off writing these (originally) weekly posts because I feel like gathering the required content to make them interesting – screenshots, links and such – to be quite a large time sink, which I would rather spend on making things happen. I also feel like you guys deserve more updates. So here’s a brief summary of all the awesomeness I (and others) have been doing, mostly behind the scenes:

  • The update system was rewritten. osu! should now be able to update in the background without running the external updater app (osume). I have a blog post specifically on the reasoning behind this change and a lot more detail written and almost ready to post, so check that out when it appears.

    update now!

  • The whole osu! infrastructure has needed to scale with the increasing user base. Database load was edging closer towards saturation, so I went performed a bunch of software and hardware optimisation with a very fine comb. This involved deploying more read-only slaves, tweaking indices for better write performance, switching table engines of some high-write tables, altering MySQL configuration on the master, re-partitioning some tables, and making more room on SSDs for hot IO paths. This really needs a full post to understand the scale and effort which goes into keeping a system like osu! running (on my own).

    database load

  • To make space on SSDs, I finally took steps to move replay data out of the database. Many of my database servers operate with no HDD storage, which means irregularly accessed replay data was taking up valuable storage space (those replays ain’t small by any means). I tested riak and mongodb, but finally settled with a hosted solution: Amazon S3. This relieves the task of maintenance of storage, and adds redundancy which hasn’t existed to the level it should have until now. As a result, I have also started storing the top 50 replays for every beatmap, rather than top 40.

  • The osu! main menu now has a shiny visualisation! The osu! cookie now glows as your music gets more intense. It also pulses more as the song gets louder.


  • In order to make audio transitions and loading more performant and flexible, I rewrote a large portion of the osu! audio framework. This isn’t yet in a public release, but should be coming soon. Will help decrease load time on song select previews and making the transition into gameplay a lot more smooth. Also allows for multiple audio tracks playing at once, which may be useful for storyboarding.

  • Resolution switching has also received some attention, making switches from fullscreen to windowed quicker and smarter. Handling of borderless window is now done so as its own mode, which won’t overwrite your window resolution settings.

  • The cause of some replays breaking (misses where the player didn’t actually miss) has been found and patched. Unfortunately it was an error with the replay data itself, so some existing replays are going to be broken for eternity. I tried to hack in a fix for them, but it’s just not worth it due to the potential of causing weirdness on other replays. Might be best to just get a list of the remaining broken replays and edit the replay data.

  • I rewrote my localisation toolkit to allow almost automatic string extraction from osu!. An initial test run of it saw a double in the number of localisable strings. Expect this to keep increasing as I get the impulse to make more available. Note that these updated localisations are not currently in the public release due to some show-stopper bugs in other places.

  • I’ve been working quietly on a new version of pp (ppv2) which will make the ranking system a lot more understandable, real-time, and applicable to lower level players. It also shouldn’t jump around as much. I hope to make it completely open so critics can suggest improvements, and they will be applied as necessary. ppv2 has its own processor which is quite a beast, and can handle recalculations of users in real-time without almost no overhead. Getting this live will mark the end of daily pp processing which currently eats a hell of a lot of processing power.

  • I’ve been working on a new beatmap modding infrastructure and generally rethinking the whole process. You can read more here and see a portion of the system in action here. I haven’t had the time to push this out for real-world use yet, but I really want to this month.

  • I have a branch that finally compiles without XNA, and runs under .NET4.5 (or anything in between). The eventual plan is to move forward from .NET2.0, as there are huge performance improvements with the newer releases – many of which will see less “lag spikes” that some people experience. This will mark the death of DirectX support, but don’t worry; OpenGl support will be vastly improved and tested before this happens.

  • Experimental replay scrubbing is available on the test build, but was broken recently with the above audio framework changes. I’ll fix that soon, so jump on test build and give it a whirl if you are curious. This won’t be available on public in the next release, but maybe sometime soon after. When it works ;).

  • The osu!api has gone live, giving developers access to some of the data osu! has built up over the years. It is currently quite minimal, but this is intended as I plan on only adding new API calls which people will find useful to create interesting services and apps. If you want to request any additions to the API, file an issue on GitHub.

  • Disqus is now available on all beatmap pages, allowing for discussion outside the forum (which in the case of ranked maps is not so active). It makes use of SSO (Single Sign-On), meaning you can post using your existing osu! account. Feels very integrated and nice to use. And it’s threaded!

  • A mapping contest has run and ended, currently in the judging stage. While it didn’t go as smoothly as hoped, we to plan on having more in the near future to make up for it.

A few milestones:

  • 10,000 consecutive users peak (5th May).
  • 500,000 active users (10th June).
  • 100,000 likes on facebook (1st July).
  • 1,500,000,000 plays (3rd July).
  • Approaching 3 million registered accounts!

I know I’ve still missed many things from the above lists. Apologies for that; I will strive to get more regular updates up so you get a more granular look at what is going on. osu! is growing at a pretty crazy speed and I’m doing my best to keep up. Hopefully you guys can agree :).

If you’d like me to write about any of the above dot points in more detail, leave a comment! I have a lot of detail I can add, but don’t want to bore you all with technical blabber that no one cares about.


Optimal Database Backups

published 31 May 2013

No one enjoys database backups. They usually involve a load spike and a lot of table locking (even in best-case scenarios) which can be felt on live servers. Some sites bring their services down to perform backups, others slow to a halt. It is a very important aspect of running an online service, and finding an optimal and elegant solution is usually very specific to the infrastructure and nature of services being offered.

I am very serious about keeping live backups. The osu! database is replicated to a slave server, providing a real-time fallback in case the main server happened to fail. This is already ample to handle any server software/hardware issues – for instance if a drive was to fail. This alone does unfortunately leave the possibility of human mistake – where the database is damaged internally – which while I’d love to say doesn’t happen, is generally unavoidable (especially when expanding the size of the team which is working with database access). In the case of a human mistake, both the master and slave server’s data is in a bad state, making slaves of this nature useless.

In the case of human error, my current backup solution was to take database snapshots from the slave server. This results in very minimal effect on the front-facing service, as user based actions rarely require a query to the slave database, but the storage requirements, the IO requirements and the general clunkiness of snapshots has always bugged me. It also means that as backups were only made once a week, in the case we needed to recover data it may be up to seven days old, which is not acceptable.

Incremental snapshots is one way to avoid this pitfall, but does require all database tables to be of InnoDB engine. I regularly test InnoDB (or in this day and age, XtraDB) but am still getting better overall performance with the arguably less reliable MyISAM, so this is not an option.

Introduce a delayed slave to the equation. This is a separate server which is initialised as a slave to the master database, but maintains a time distance from the live data. This is easily done using the Percona Toolkit’s pt-slave-delay, which runs as a daemon and allows specifying a period for which it should delay sql operations by.

There are a few amazing advantages here:

  • There is no added load to any of the live servers, apart from the network overhead of streaming binlogs.
  • It is a continuous backup. You can’t get better than this. No snapshots to worry about; only the assurance that you can always recover.
  • Because binlogs are always sent instantly, this slave instance can also be replayed to any particular point in history within the delayed duration. So if it is running 24 hours behind by default, it could be asked to catch up to 12 hours, or even removing the delay – making it a potential real-time backup slave in case of failures.
  • If you already have replication setup, initialising the new slave can be done with zero front-facing impact by using an existing slave as the point of initialisation.

To initialise the pt-slave-delay command, it’s as simple as ensuring replication is started, then specifying the delay and check interval. I am currently using the following, which should be run at system startup if you want it to persist:

mysql -e 'start slave;'
pt-slave-delay --delay 24h --interval 5s --no-continue localhost

Take note that while replication is stopped, you will now be able to see how many seconds behind the server is using the SHOW SLAVE STATUS command. As I regularly use this for monitoring the slave delay, I had to use an alternative method to find the delay. For me, the easiest way was to select the MAX(timestamp) from a table with high activity and compare this to CURRENT_TIMESTAMP as follows:


I really enjoy databases and the optimisation of them at low and high levels. osu! is still relatively simple when it comes to database infrastructure but it is rapidly expanding. Keeping up with the increasing load is an interesting and very fun process. I hope to post more articles like this delving into the slightly more technical side of things going forward.


I just found out that as of MySQL5.6 (which I am actually running, so have switched to this method) you no longer need the pt-slave-delay script as this is built-in functionality. You can add a delay with one simple command (make sure to STOP SLAVE; first):

CHANGE MASTER TO MASTER_DELAY = 14400; --delays 4 hours

SUPER IMPORTANT NOTE: If you are delaying further back than the master has stored in binary logs, running a CHANGE MASTER TO like this will cause the world to fall apart, as it resets all slave relay logs. Make sure to carefully read the documentation – specifically:

CHANGE MASTER TO deletes all relay log files and starts a new one, unless you specify RELAY_LOG_FILE or RELAY_LOG_POS. In that case, relay log files are kept; the relay_log_purge global variable is set silently to 0.

p.s. I haven’t forgotten about the “this week in osu!” series, but some of the things i planned on writing about have been lost in my forgetful mind. I’ll try and knock one out along with the next public release, which I am hard at work on getting finalised. I am trying to livestream as much as I can, so if you are interested in the development of osu!, make sure to tag along and say hi in chat :).


distraction-reduced iphone

published 17 Apr 2013

I recently tweeted about a life-hack article titled “The distraction-free iPhone (or ‘Why I’m happier since I disabled Safari’)”. The general gist is that our phones are taking over our lives, and acting as not only a procrastination device, but a distraction from what is going on around us.

“Checking email, checking Twitter, checking news. Wondering if something interesting was happening anywhere in the world. Wondering if anybody was thinking about me.”

Although I love how powerful our phones have become, this is definitely something I have started to feel over the last couple of years. It is especially disturbing when going out to lunch with fellow software engineers and only making conversation when someone comes across an interesting tweet, or an amusing image on instagram. This consumption of information is somewhat of an addiction, and while I tend to focus on consuming only what is relevant from the business side of things – I avoid the social aspects of twitter and similar services – am still guilty of indulging in it.

On reading this article I decided it was definitely a time for a change. Due to my commitments to osu!, osu!stream and puush, I knew that I could not completely lock myself out of services which brought incoming real-time communications which could require immediate response, so after some very deep consideration I began to form an action plan. I felt it was worth writing up exactly what I have decided on, and my reasoning behind each decision.

So, this is my iPhone’s only home screen:

my new home screen

Using Springtomize (a jailbreak tweak) I am able to hide not only icon labels, but any apps which I don’t want appearing on my home screen. I am also able to disable spotlight and pagination completely. Using this technique, I am able to hide Tweetbot and Facebook while still receiving incoming notifications via notification centre. There is no way to access these application (which would be the main sources of information) as long as I do not receive incoming communications.

Let’s run through each of the apps I have left visible and installed, from top-left to bottom-right:

  • Messages my primary method of communication with nearby friends.
  • authy two factor authentication for services such as google/gmail, cloudflare, dropbox etc.
  • 1password password manager. technically also a web browser, but i don’t use this in anything but emergencies.
  • Photos because people always want to look at your photos, right?
  • Fantastical my calendar app of choice. quickly able to schedule events and check what is coming up.
  • puush easiest way to get photos off the phone when necessary (i don’t like to sync with a PC).
  • osu!stream mainly just for display or showing people when they ask about it. i considered removing this but it’d just feel wrong.
  • Clock alarms are handy.
  • Google Maps maps are also handy.
  • Naver en-jp Dictionary the simplest yet most comprehensive english-japanese dictionary (imho).
  • Reeder this is a weird one, since it’s a 100% procrastination tool. it is an exception to the rule because i only use it before sleeping to get my daily dose of news and comics.
  • Simplenote cloud-synced memo.
  • Mailbox i actually prefer Apple Mail for serious mail, which is why i chose to use mailbox. it limits what i can do, and has a very optimal workflow for scheduling mail for later perusal.
  • Clear the absolute best to-do / bucket list app out there.
  • Music i couldn’t live without music.

You may be wondering where system setting are. I actually have a few more apps hidden in an invisible folder, which are utility-class apps that are necessary for maintenance of the phone itself (Cydia / App Store / Settings) or useful in emergency situations (iTap Remote Desktop / Prompt SSH / Dropbox). Out of sight, out of mind, until the day they are necessary.

So far I have not looked back. It sure does limit what I can do with my phone, but I don’t see this being an issue. I have already noticed a reduction in the number of times I mindlessly reach for my phone when I don’t actually have a particular task I need to perform.

I urge others to give this a try. You don’t need to go the full way (like Jake in the original article) but why not try cutting back what you use your phone for, and spending more time paying attention to what is going on around you; interacting with those that are close to you; taking in more of life. We already spend so many hours in front of a PC screen, after all.


This week (month) in osu!

published 02 Apr 2013

Been working so hard on new stuff that I’ve basically forgotten all the things I have done and need to post about. Let’s get started with the big one: the osu! UI and default skin graphics refresh!

Over the course of two long weeks, I spent basically every minute awake in photoshop going through all 421 osu! sprites and doubling the dimensions of each and allowing pixel perfect display at 2048x resolutions. Yep, osu! is now “HD”, if you will! Where I was no longer happy with the old design, or where vector data was not available, they were re-created from scratch. This ended up being over half of the resources.

lots of resources

Let me start by saying that I couldn’t have done this alone. A huge thanks to RBRat3 for helping out with a lot of the icons which did not have vector data, including (but not limited to) the main menu icons, the editor control icons and the mod icons.

I also took this opportunity to not only split the resources into two separate resource collections (UI and gameplay), but also decided to distribute in the compressed PNG format rather than raw bitmaps. This is a trade-off allowing for faster patching and less disk usage while adding a load-time performance overhead. This overhead worked out to around 5% and does not affect runtime performance – plus it is only relevant when using the default skin, so all the people using custom skins won’t even notice a difference – so I figured it was beneficial overall.

I will be distributing the elements of this skin for use as templates and/or modification in the near future. The only reason I haven’t yet is because there are a few parts left which are not finished, including Taiko / CtB / osu!mania modes. Keep a watch on the skinning forum for the template release in the near future.

BUT WAIT, this update didn’t just stop at sprite changes! While changing all these sprites, I found so many other things along the way that I was no longer happy with, and took the liberty of changing them too. Let’s try and make a list!

  • The follow points (those dots which display between hitobjects) are now directional. In the new default skin, they become arrows which – in my testing – really help with map readability.

    direction follow points

  • Hit explosions are visually much smaller, allowing increased readability and reducing the cases they overlap future hitobjects. They also have a really nice glowy effect and a true particle system!

    new hit explosions

    You might also notice that they will briefly appear top-most, but quickly fade to a new position underneath hitobjects, helping with readability on streams.

    new hit explosions #2

  • The ranking popup dialog which used to show your online ranking has been moved so it is no longer a popup. It is now accessible by scrolling downwards, or clicking the handy button at the bottom of the screen.

    online ranking button

  • Special mode selection screen is gone. You can now change modes from song select using the new “Mode” menu! This makes a lot more sense to me these days, where the special modes are basically treated the same as osu! mode.

    mode selection

  • Scrollable views can now be dragged. Also scrolling is a lot smoother, with improved acceleration/inertia handling (please don’t sue me for patent infringement apple <3).

  • The back button has been unified across the game (it used to behave slightly differently in some cases).

  • The background has been removed from osu!direct. It now uses the default menu-background or a user replacement if one exists.

I’ve probably forgotten some other stuff, but I have a few more posts up my sleeve, so will tag it on the end of one of those if I do happen to recall it. I’ve already received quite a lot of feedback on the new design, but feel free to leave anything you have to say in the comments here as well :).