05 Jun 2015
I’m writing this for the second time, since it somehow disappeared the first time :(. That’ll teach me for trusting some chrome based markdown editor. stubbornly returns to sublime text 3
In recent times I look at the user-facing changes and and often disappointed in what I see. My own role in osu! has changed from being able to rapidly implement new features to only just being able to keep up with what is going on each day. The rare occasions I’m able to implement something new are therefore very enjoyable as a result, but I digress.
The ongoing tasks which I choose to complete myself rather than delegate remain that way for a reason. I believe each and every one of them requires my own attention. Delegating tasks is great and all, but there are two issues with it:
- Delegating things that come up infrequently is often less efficient. An example would be emails about conventions, which are always different in nature. Sometimes people will want merchandise, sometimes they will want to promote osu! by presenting a panel. Teaching someone how to respond to these emails would require writing a manual of every possible scenario, or being able to trust their judgement, which leads on to the second issue..
- I find it hard to trust people. Not trust as in confidentiality, but trust as in the ability to diligently handle a scenario in the same way I would myself. I am pretty sure I am yet to meet someone with the same values and level of attention-to-detail that I have. While this may sound like I’m bragging, if you ever run a company of your own you will understand this feeling.
Today was one of those days I spent catching up on mostly mundane run-of-the-mill tasks. You’re still interested? Okay then!
- Fixed a loophole in the account recovery process that was abused overnight. Accounts can now be set into a state where users cannot initiate password recovery and must contact support.
- Purged my inbox backlog down from 188 to 55. I read emails as the come in and when necessary reply within a minute or two, but less important emails are snoozed until a later date. I find it easier to process these in one large batch rather than as-they-come. This is what my inbox looks like now. Might take a few beers tomorrow to get through the remainder ;).
- Brought company accounts up-to-date. Basically distributing money into the correct accounts so people are paid and servers are kept online and purring. I have an accountant to handle things at the end of the year, but daily financials are still my own responsibility. Need to attend to them once a week else they pile up quickly.
- Got acceptance email for AWS Aurora preview. Not sure if I will use this going forward, but couldn’t resist the opportunity to test it for free! Spun up two beefy 16 core 108gb instances which I will be deploying the full osu! database to for performance testing over the coming weeks.
- Figured out osu!keyboard stock issues. We needed to reserve a few keybaords for Taiko World Cup winners, but due a number of faulty units being found during initial quality checking, we ran a bit short.
- Approved and finalised payment for the next osu!tablet production run, this time for 2,000 units! We still have stock from the last run too. I think I have finally got my head around the convoluted processes involved in international manufacturing and shipping, so stock shortages should be a thing of the past.
- Our knowledge base is now fully stocked with content, hopefully helping users before they unnecessarily post to the forum!
So let’s briefly revisit something that happened last week: OVH had a downtime which lasted several hours as the result of a cable being cut. This meant the sudden loss of all our beatmap download mirror traffic! As I was “enjoying” an Owl City concert at the time, I had @nekodex - who you may also know due to his musical contributions to osu! - help me get a new mirror up and running.
osu! beatmap mirrors are deployed as a single file php script with no external dependencies. They use an s3 endpoint to retrieve maps, cache them locally and serve them to users using a secure checksum system. The script is not only self-containing, but updates can be pushed out in an automated process. Each mirror is monitored by the central osu! web server and mirrors which are unavailable or underperforming are automatically removed from the available list.
We had things back up and running in under 20 minutes as a result, but when things like this go haywire it is always wise to add countermeasures to ensure the same thing doesn’t happen again. I have started investigating where I will be adding some additional mirrors in order to add geographical redundancy to the system. As I don’t want to introduce any more complexity to the accounting side of the infrastructure setup of osu! (already across four datacenters!!) they will likely be placed in OVH’s European datacenters.
Going forward, I would love to open the mirroring system up to the public again, letting people lend their spare server bandwidth to the good of the osu! community. This would require some logic to ensure users were linked up with the closest mirror to their physical location, of course. Think a mini-CDN just for osu!. Alternatively maybe I should activate the dormant p2p code that has been sitting inside osu! for several years… ^^;
Have a good weekend. Play lots of osu!.