Was in bed with a fever yesterday. All the muscles in my body were in pain and I honestly didn’t get much done. Quite rare for me, but these things happen!

Caught up today late into the night:

• Finally seem to have figured out the cause of our master DB instance hitting a performance wall! The culprit is looking to be related to the sheer number of connections we are servicing per second (in excess of 800 at peak). Have a few potential solutions which I started preparing for testing today. More to come after I actually test them.
• Finally getting closer than ever to the next public release (like, an actual one). A few minor bugs left to fix, but thanks to some amazing optimisation effort by Tom94 things are running buttery smooooth.
• Finally sourced some osu!tablet pens! Should have them stocked on the osu!store for people wanting a spare or replacement after throwing their pens ON THE GROUND. Pricing should be much more reasonable than Amazon (hoping for less than $15 including shipping).

It is now more than ever that we need your feedback on cutting-edge! Report on slack, but make sure to also post in the cutting-edge thread so your feedback is not missed!

Missed a day yesterday as I collapsed in bed so tired I completely forgot I was doing this blogging thing!

Spent the day catching up on the weekend’s email and slack conversations, meeting an osu! player who is going to help out in the near future with expanding our overall search capabilities and going out to see Porter Robinson’s amazing “Worlds” live tour.

Still playing the catch-up game today, working through the list of outstanding pull-requests as we get closer and closer to the next public release. If you haven’t done so yet, make sure to give cutting-edge a try - performance is getting better by the day! If you aren’t getting better performance than on the stable release stream, you should definitely let us know (in #cuttingedge on slack) so we can follow up with you personally and set things right.

Personally I’m seeing a gain of 10% over DirectX on stable. Nothing huge, but it’s an improvement over what we had, which is what matters.

Today we managed to merge the OpenGL ES implementation, which brings a compatibility mode for users having no luck with OpenGL. A huge props to smoogi and Tom94 for getting this branch working so quickly and smoothly! If you’re interested in the specifics, make sure to check out this guest post by smoogi.

Even if you weren’t having issues, I’d highly suggest giving cutting-edge another try as performance should be better than ever.

I asked for infrastructure-related questions yesterday but only got one, so let’s answer the unrelated questions as well!

@ppy What OS is running on the osu! Servers? –byteturtle

Web, database, memcache and everything linux run Debian (sid or jessie). Bancho runs Windows Server 2012r2 for maximum .NET performance. There are also some linux VMs running Ubuntu for compatibility with pre-packaged deployments like Sentry.

@ppy Btw, when Osu will be going open-source ? –neskeii

We’re constantly making progress towards the open-source goal. I don’t want to set unrealistic expectations, but my personal goal is “this year”.

@ppy What does “multiple scores per-user-per-beatmap” refer to exactly? Does it affect the pp system in any way? –elim2g

It will mean you can have one score for each mod combination per map. Your highest pp for each beatmap will be taken. This means there’s much less chance that you will lose pp on a particular map due to playing it with a different mod combination and scoring higher.

@ppy Will there be many triangles in osu!next? –Houtarousu

Yes, in some form.

@ppy Will the geki and katu for taiko be removed soon? Since they are totally useless there. –Abrian_MA

On the results screen or? They really need to be removed if they are still displayed, yes.

@ppy Plans for abekobe mod and random mod for taiko? –Abrian_MA

No idea what that is, but I do have plans to add lots of new mods (fun, not score modifiers).

Today marks the completion of a gruelling 78 hour database schema change to support multiple scores per-user-per-beatmap. Why did it take so long and what did it involve, you ask? It was an index change on the main osu!-mode scores table, rewriting 38gb of data in an asynchronous process to avoid any table-locking. While pt-online-schema-change makes this easy to do - and tries its best not to degrade database performance - there were still times of increased load resulting in score processing delays. I invested some time in improving the queue mechanism to ensure scores are always received (with pp awarded later) wherever possible.

Continued MySQL 5.7RC roll-out on a slave database (used for country and mod-specific rankings, as well as being the food source for Elasticsearch). Trying to get the upgrade process as perfect as possible as I move to deploying it on the primary slave and master, hopefully next week. Also used this as an opportunity to do performance testing under standard load - quite important as the previous 5.7.x build I tested resulted in HUGE performance regressions.

Yesterday I invested some time in adding log output for the update process. The output is verbose enough to diagnose any update issues people may come across, and will help progress towards my ultimate goal of making a 100% fail-safe install/update process, bootstrapped inside osu!.exe itself. I’d say it’s already around 99% infallible, but as long as users are reporting infinite update loops I won’t be happy.

Haven’t done a Q&A for a while, so let’s make it happen tomorrow. Ask me anything about infrastructure, databases or servers. Tweet with the #FridayQA tag so I can easily gather your questions!

Hey there, I’m smoogipooo and I’ll be hosting your daily progress report for today.

Preface

As osu! moves forward with OpenGL one of the biggest challenges we face is maintaining compatibility for older systems. With osu! and especially with OpenGL we strive to maintain compatibility until the statistics show that >99% of the userbase won’t be impacted by a change. Statistics have shown that this is not the case for the move to OpenGL, and we have also received numerous reports of OpenGL not working well on specific systems.

The cause

All the problematic systems had one common trait - the GPU manufacturer’s drivers were not installed and the only Display Adapter in the Device Manager was the Microsoft Basic Device Adapter and likewise gl_info.txt showed GDI Generic as the renderer. We realized that the problems indicated that osu! was being rendering through software. Upon diving into OpenTK I found that while all the graphics contexts had the GENERIC_FORMAT flag, none of the graphics contexts had the GENERIC_ACCELERATED flag which is required to obtain hardware acceleration. This type of acceleration is called MCD acceleration, and surprisingly enough Quake 3 managed to obtain MCD acceleration for reasons still unknown to me (black magic?).

The solution

For some cases it is enough to alert the users to install their drivers, however for many other cases the driver is incompatible with the user’s Operating System. Unable to reach a solution to this, a few days ago I mentioned the possibility of translating OpenGL to DirectX as DirectX is supported by Microsoft and compatibiliy won’t be a problem. Upon looking at how Google Chrome and Mozilla Firefox do WebGL I was directed towards ANGLE which seems to provide exactly this translation.

Well… Not exactly - there are a few caveats:

• All OpenGL code must be changed to OpenGL ES 2.0, as ANGLE only supports ES 2.0 and 3.0.
  • This means that there is absolutely no fixed-function pipeline and everything must be vertex buffered - there is no immediate mode. osu! still uses the fixed-function pipeline for many of its core functions.
  • Slider rendering must be completely rewritten as they made extensive use of the fixed-function pipeline.
• Vertices must be buffered and sprites must be batched in order to achieve the same level of performance as Cutting Edge. Tom94’s recent efforts with this has brought performance above Cutting Edge on the OpenGL renderer!
  • This also brings us closer to fixing sliders however we will explore slider rendering through shaders to further improve performance.
• Shaders need to be rewritten to conform to both OpenGL ES and DirectX standards, complete with loop unrolling. Bloom is currently working but I haven’t had as much success with blur.
  • The shader framework has also undergone quite a few changes to accommodate their new usage throughout the game.

This is not a replacement for OpenGL and will be offered as a compatibility mode only for systems that can’t otherwise obtain a hardware accelerated renderer through OpenGL. OpenGL will provide better performance than DirectX through ANGLE and will remain as our choice of graphics library.

Conclusion

The project is currently in its final stages - buffering and batching are working, shaders are mostly working and most primitive elements are drawing. Buffering and batching everything is already cleaning up the codebase quite nicely so we’re enthusiastic about that too. All that’s left is to fix up sliders and clean-up any remaining code. I leave you with a pretty nice (in my opinion) effect I discovered after the red and blue channels of textures were inverted: