20150813

published 13 Aug 2015

Today marks the completion of a gruelling 78 hour database schema change to support multiple scores per-user-per-beatmap. Why did it take so long and what did it involve, you ask? It was an index change on the main osu!-mode scores table, rewriting 38gb of data in an asynchronous process to avoid any table-locking. While pt-online-schema-change makes this easy to do - and tries its best not to degrade database performance - there were still times of increased load resulting in score processing delays. I invested some time in improving the queue mechanism to ensure scores are always received (with pp awarded later) wherever possible.

Continued MySQL 5.7RC roll-out on a slave database (used for country and mod-specific rankings, as well as being the food source for Elasticsearch). Trying to get the upgrade process as perfect as possible as I move to deploying it on the primary slave and master, hopefully next week. Also used this as an opportunity to do performance testing under standard load - quite important as the previous 5.7.x build I tested resulted in HUGE performance regressions.

Yesterday I invested some time in adding log output for the update process. The output is verbose enough to diagnose any update issues people may come across, and will help progress towards my ultimate goal of making a 100% fail-safe install/update process, bootstrapped inside osu!.exe itself. I’d say it’s already around 99% infallible, but as long as users are reporting infinite update loops I won’t be happy.

Haven’t done a Q&A for a while, so let’s make it happen tomorrow. Ask me anything about infrastructure, databases or servers. Tweet with the #FridayQA tag so I can easily gather your questions!

comments

20150812

published 12 Aug 2015

Hey there, I’m smoogipooo and I’ll be hosting your daily progress report for today.

Preface

As osu! moves forward with OpenGL one of the biggest challenges we face is maintaining compatibility for older systems. With osu! and especially with OpenGL we strive to maintain compatibility until the statistics show that >99% of the userbase won’t be impacted by a change. Statistics have shown that this is not the case for the move to OpenGL, and we have also received numerous reports of OpenGL not working well on specific systems.

The cause

All the problematic systems had one common trait - the GPU manufacturer’s drivers were not installed and the only Display Adapter in the Device Manager was the Microsoft Basic Device Adapter and likewise gl_info.txt showed GDI Generic as the renderer. We realized that the problems indicated that osu! was being rendering through software. Upon diving into OpenTK I found that while all the graphics contexts had the GENERIC_FORMAT flag, none of the graphics contexts had the GENERIC_ACCELERATED flag which is required to obtain hardware acceleration. This type of acceleration is called MCD acceleration, and surprisingly enough Quake 3 managed to obtain MCD acceleration for reasons still unknown to me (black magic?).

The solution

For some cases it is enough to alert the users to install their drivers, however for many other cases the driver is incompatible with the user’s Operating System. Unable to reach a solution to this, a few days ago I mentioned the possibility of translating OpenGL to DirectX as DirectX is supported by Microsoft and compatibiliy won’t be a problem. Upon looking at how Google Chrome and Mozilla Firefox do WebGL I was directed towards ANGLE which seems to provide exactly this translation.

Well… Not exactly - there are a few caveats:

  • All OpenGL code must be changed to OpenGL ES 2.0, as ANGLE only supports ES 2.0 and 3.0.
    • This means that there is absolutely no fixed-function pipeline and everything must be vertex buffered - there is no immediate mode. osu! still uses the fixed-function pipeline for many of its core functions.
    • Slider rendering must be completely rewritten as they made extensive use of the fixed-function pipeline.
  • Vertices must be buffered and sprites must be batched in order to achieve the same level of performance as Cutting Edge. Tom94’s recent efforts with this has brought performance above Cutting Edge on the OpenGL renderer!
    • This also brings us closer to fixing sliders however we will explore slider rendering through shaders to further improve performance.
  • Shaders need to be rewritten to conform to both OpenGL ES and DirectX standards, complete with loop unrolling. Bloom is currently working but I haven’t had as much success with blur.
    • The shader framework has also undergone quite a few changes to accommodate their new usage throughout the game.

This is not a replacement for OpenGL and will be offered as a compatibility mode only for systems that can’t otherwise obtain a hardware accelerated renderer through OpenGL. OpenGL will provide better performance than DirectX through ANGLE and will remain as our choice of graphics library.

Conclusion

The project is currently in its final stages - buffering and batching are working, shaders are mostly working and most primitive elements are drawing. Buffering and batching everything is already cleaning up the codebase quite nicely so we’re enthusiastic about that too. All that’s left is to fix up sliders and clean-up any remaining code. I leave you with a pretty nice (in my opinion) effect I discovered after the red and blue channels of textures were inverted:

comments

20150811

published 11 Aug 2015

Another day spent mostly putting out database fires. Ongoing database migration efforts mean we are dealing with around double the normal system load, which has caused previously fixed performance bottlenecks to rear their ugly heads once again.

While I have not yet had a chance to completely write up the problem, those who have been following from the start will know what I’m talking about. I’m not sure when I will find the effort to fully explain the amount of effort which has been thrown at this particular issue, but it’s safe to say it will continue until we find the cause.

Today I figured our upgrade path from Percona Server 5.6.x to MySQL 5.7.x. Not as straightforward as it could be, due to the official MySQL upgrade process not playing nice with the custom changes that Percona makes to information_schema and performance_schema, but I got it working in the end.

Code changes to accompany the ongoing database migration that will work hand-in-hand to support multiple scores per user per beatmap are still ongoing. Today I fixed some shortcomings which still existed in bancho.

comments

20150810

published 10 Aug 2015

Hope you are all having a productive and energetic start to your weeks.

Another quick Monday update, as I spent most of today catching up on completely uninteresting tasks that I will spare you the details of. This week will hopefully bring some exciting changes to the table which have taken a while to get ready for public consumption.

I can say that we may have made a breakthrough in graphics compatibility for the next osu! release (currently what you see on cutting-edge). While we are testing internally, the whole team is hopeful that this will bring us as close as we are going to get to 100% compatibility.

More to come on that soon, by means of a first-time guest post :).

comments

20150808

published 08 Aug 2015

fireworks over tokyo bay, just an hour ago!

comments