published 24 Aug 2015

I somehow managed to miss more than one post last week. Let’s play some more catch-up!

  • Following up on the database performance lead I thought I found last Thursday: after spending Friday working through it, trying to understand what was going on – which ended with us setting up a test KVM environment to replicate DigitalOcean’s – I feel like I have more questions than answers. Safe to say, KVM is adding some weird, magical, invisible overhead to MySQL.
  • I spent a good portion of the weekend and today focused on osu!stream. Turned out Apple changed the way paid app contracts work causing all in-app purchases to suddenly fail! This means I need to redeploy the app. Not such a simple task when you are one year behind SDK changes (some that have seriously broken display and input subsystems). I plan on submitting an update for it this week, along with a full-fledged blog post about the future of osu!stream. Quite a touchy subject for me.
  • There’s some experimental code up on the new forums to make scrolling upwards in threads more pleasant… we hope. Check it out and leave some feedback in the comments.
  • Been trying to track down the cause of some users seeing endless update loops. I could have sworn that the updater was flawless, but alas. If you are experiencing issues with updating, make sure to get on the public slack #cuttingedge channel and highlight the fuck out of me.
  • The osu!mania World Cup is ongoing. Make sure to tune in and see how polished the tournament streaming setup is!


published 20 Aug 2015

Was in bed with a fever yesterday. All the muscles in my body were in pain and I honestly didn’t get much done. Quite rare for me, but these things happen!

Caught up today late into the night:

  • Finally seem to have figured out the cause of our master DB instance hitting a performance wall! The culprit is looking to be related to the sheer number of connections we are servicing per second (in excess of 800 at peak). Have a few potential solutions which I started preparing for testing today. More to come after I actually test them.
  • Finally getting closer than ever to the next public release (like, an actual one). A few minor bugs left to fix, but thanks to some amazing optimisation effort by Tom94 things are running buttery smooooth.
  • Finally sourced some osu!tablet pens! Should have them stocked on the osu!store for people wanting a spare or replacement after throwing their pens ON THE GROUND. Pricing should be much more reasonable than Amazon (hoping for less than $15 including shipping).

It is now more than ever that we need your feedback on cutting-edge! Report on slack, but make sure to also post in the cutting-edge thread so your feedback is not missed!



published 18 Aug 2015

Missed a day yesterday as I collapsed in bed so tired I completely forgot I was doing this blogging thing!

Spent the day catching up on the weekend’s email and slack conversations, meeting an osu! player who is going to help out in the near future with expanding our overall search capabilities and going out to see Porter Robinson’s amazing “Worlds” live tour.

Still playing the catch-up game today, working through the list of outstanding pull-requests as we get closer and closer to the next public release. If you haven’t done so yet, make sure to give cutting-edge a try - performance is getting better by the day! If you aren’t getting better performance than on the stable release stream, you should definitely let us know (in #cuttingedge on slack) so we can follow up with you personally and set things right.

Personally I’m seeing a gain of 10% over DirectX on stable. Nothing huge, but it’s an improvement over what we had, which is what matters.



published 14 Aug 2015

Today we managed to merge the OpenGL ES implementation, which brings a compatibility mode for users having no luck with OpenGL. A huge props to smoogi and Tom94 for getting this branch working so quickly and smoothly! If you’re interested in the specifics, make sure to check out this guest post by smoogi.

Even if you weren’t having issues, I’d highly suggest giving cutting-edge another try as performance should be better than ever.

I asked for infrastructure-related questions yesterday but only got one, so let’s answer the unrelated questions as well!

@ppy What OS is running on the osu! Servers? –byteturtle

Web, database, memcache and everything linux run Debian (sid or jessie). Bancho runs Windows Server 2012r2 for maximum .NET performance. There are also some linux VMs running Ubuntu for compatibility with pre-packaged deployments like Sentry.

@ppy Btw, when Osu will be going open-source ? –neskeii

We’re constantly making progress towards the open-source goal. I don’t want to set unrealistic expectations, but my personal goal is “this year”.

@ppy What does “multiple scores per-user-per-beatmap” refer to exactly? Does it affect the pp system in any way? –elim2g

It will mean you can have one score for each mod combination per map. Your highest pp for each beatmap will be taken. This means there’s much less chance that you will lose pp on a particular map due to playing it with a different mod combination and scoring higher.

@ppy Will there be many triangles in osu!next? –Houtarousu

Yes, in some form.

@ppy Will the geki and katu for taiko be removed soon? Since they are totally useless there. –Abrian_MA

On the results screen or? They really need to be removed if they are still displayed, yes.

@ppy Plans for abekobe mod and random mod for taiko? –Abrian_MA

No idea what that is, but I do have plans to add lots of new mods (fun, not score modifiers).



published 13 Aug 2015

Today marks the completion of a gruelling 78 hour database schema change to support multiple scores per-user-per-beatmap. Why did it take so long and what did it involve, you ask? It was an index change on the main osu!-mode scores table, rewriting 38gb of data in an asynchronous process to avoid any table-locking. While pt-online-schema-change makes this easy to do - and tries its best not to degrade database performance - there were still times of increased load resulting in score processing delays. I invested some time in improving the queue mechanism to ensure scores are always received (with pp awarded later) wherever possible.

Continued MySQL 5.7RC roll-out on a slave database (used for country and mod-specific rankings, as well as being the food source for Elasticsearch). Trying to get the upgrade process as perfect as possible as I move to deploying it on the primary slave and master, hopefully next week. Also used this as an opportunity to do performance testing under standard load - quite important as the previous 5.7.x build I tested resulted in HUGE performance regressions.

Yesterday I invested some time in adding log output for the update process. The output is verbose enough to diagnose any update issues people may come across, and will help progress towards my ultimate goal of making a 100% fail-safe install/update process, bootstrapped inside osu!.exe itself. I’d say it’s already around 99% infallible, but as long as users are reporting infinite update loops I won’t be happy.

Haven’t done a Q&A for a while, so let’s make it happen tomorrow. Ask me anything about infrastructure, databases or servers. Tweet with the #FridayQA tag so I can easily gather your questions!