20150814

published 14 Aug 2015

Today we managed to merge the OpenGL ES implementation, which brings a compatibility mode for users having no luck with OpenGL. A huge props to smoogi and Tom94 for getting this branch working so quickly and smoothly! If you’re interested in the specifics, make sure to check out this guest post by smoogi.

Even if you weren’t having issues, I’d highly suggest giving cutting-edge another try as performance should be better than ever.

I asked for infrastructure-related questions yesterday but only got one, so let’s answer the unrelated questions as well!

@ppy What OS is running on the osu! Servers? –byteturtle

Web, database, memcache and everything linux run Debian (sid or jessie). Bancho runs Windows Server 2012r2 for maximum .NET performance. There are also some linux VMs running Ubuntu for compatibility with pre-packaged deployments like Sentry.

@ppy Btw, when Osu will be going open-source ? –neskeii

We’re constantly making progress towards the open-source goal. I don’t want to set unrealistic expectations, but my personal goal is “this year”.

@ppy What does “multiple scores per-user-per-beatmap” refer to exactly? Does it affect the pp system in any way? –elim2g

It will mean you can have one score for each mod combination per map. Your highest pp for each beatmap will be taken. This means there’s much less chance that you will lose pp on a particular map due to playing it with a different mod combination and scoring higher.

@ppy Will there be many triangles in osu!next? –Houtarousu

Yes, in some form.

@ppy Will the geki and katu for taiko be removed soon? Since they are totally useless there. –Abrian_MA

On the results screen or? They really need to be removed if they are still displayed, yes.

@ppy Plans for abekobe mod and random mod for taiko? –Abrian_MA

No idea what that is, but I do have plans to add lots of new mods (fun, not score modifiers).

comments

20150813

published 13 Aug 2015

Today marks the completion of a gruelling 78 hour database schema change to support multiple scores per-user-per-beatmap. Why did it take so long and what did it involve, you ask? It was an index change on the main osu!-mode scores table, rewriting 38gb of data in an asynchronous process to avoid any table-locking. While pt-online-schema-change makes this easy to do - and tries its best not to degrade database performance - there were still times of increased load resulting in score processing delays. I invested some time in improving the queue mechanism to ensure scores are always received (with pp awarded later) wherever possible.

Continued MySQL 5.7RC roll-out on a slave database (used for country and mod-specific rankings, as well as being the food source for Elasticsearch). Trying to get the upgrade process as perfect as possible as I move to deploying it on the primary slave and master, hopefully next week. Also used this as an opportunity to do performance testing under standard load - quite important as the previous 5.7.x build I tested resulted in HUGE performance regressions.

Yesterday I invested some time in adding log output for the update process. The output is verbose enough to diagnose any update issues people may come across, and will help progress towards my ultimate goal of making a 100% fail-safe install/update process, bootstrapped inside osu!.exe itself. I’d say it’s already around 99% infallible, but as long as users are reporting infinite update loops I won’t be happy.

Haven’t done a Q&A for a while, so let’s make it happen tomorrow. Ask me anything about infrastructure, databases or servers. Tweet with the #FridayQA tag so I can easily gather your questions!

comments

20150812

published 12 Aug 2015

Hey there, I’m smoogipooo and I’ll be hosting your daily progress report for today.

Preface

As osu! moves forward with OpenGL one of the biggest challenges we face is maintaining compatibility for older systems. With osu! and especially with OpenGL we strive to maintain compatibility until the statistics show that >99% of the userbase won’t be impacted by a change. Statistics have shown that this is not the case for the move to OpenGL, and we have also received numerous reports of OpenGL not working well on specific systems.

The cause

All the problematic systems had one common trait - the GPU manufacturer’s drivers were not installed and the only Display Adapter in the Device Manager was the Microsoft Basic Device Adapter and likewise gl_info.txt showed GDI Generic as the renderer. We realized that the problems indicated that osu! was being rendering through software. Upon diving into OpenTK I found that while all the graphics contexts had the GENERIC_FORMAT flag, none of the graphics contexts had the GENERIC_ACCELERATED flag which is required to obtain hardware acceleration. This type of acceleration is called MCD acceleration, and surprisingly enough Quake 3 managed to obtain MCD acceleration for reasons still unknown to me (black magic?).

The solution

For some cases it is enough to alert the users to install their drivers, however for many other cases the driver is incompatible with the user’s Operating System. Unable to reach a solution to this, a few days ago I mentioned the possibility of translating OpenGL to DirectX as DirectX is supported by Microsoft and compatibiliy won’t be a problem. Upon looking at how Google Chrome and Mozilla Firefox do WebGL I was directed towards ANGLE which seems to provide exactly this translation.

Well… Not exactly - there are a few caveats:

  • All OpenGL code must be changed to OpenGL ES 2.0, as ANGLE only supports ES 2.0 and 3.0.
    • This means that there is absolutely no fixed-function pipeline and everything must be vertex buffered - there is no immediate mode. osu! still uses the fixed-function pipeline for many of its core functions.
    • Slider rendering must be completely rewritten as they made extensive use of the fixed-function pipeline.
  • Vertices must be buffered and sprites must be batched in order to achieve the same level of performance as Cutting Edge. Tom94’s recent efforts with this has brought performance above Cutting Edge on the OpenGL renderer!
    • This also brings us closer to fixing sliders however we will explore slider rendering through shaders to further improve performance.
  • Shaders need to be rewritten to conform to both OpenGL ES and DirectX standards, complete with loop unrolling. Bloom is currently working but I haven’t had as much success with blur.
    • The shader framework has also undergone quite a few changes to accommodate their new usage throughout the game.

This is not a replacement for OpenGL and will be offered as a compatibility mode only for systems that can’t otherwise obtain a hardware accelerated renderer through OpenGL. OpenGL will provide better performance than DirectX through ANGLE and will remain as our choice of graphics library.

Conclusion

The project is currently in its final stages - buffering and batching are working, shaders are mostly working and most primitive elements are drawing. Buffering and batching everything is already cleaning up the codebase quite nicely so we’re enthusiastic about that too. All that’s left is to fix up sliders and clean-up any remaining code. I leave you with a pretty nice (in my opinion) effect I discovered after the red and blue channels of textures were inverted:

comments

20150811

published 11 Aug 2015

Another day spent mostly putting out database fires. Ongoing database migration efforts mean we are dealing with around double the normal system load, which has caused previously fixed performance bottlenecks to rear their ugly heads once again.

While I have not yet had a chance to completely write up the problem, those who have been following from the start will know what I’m talking about. I’m not sure when I will find the effort to fully explain the amount of effort which has been thrown at this particular issue, but it’s safe to say it will continue until we find the cause.

Today I figured our upgrade path from Percona Server 5.6.x to MySQL 5.7.x. Not as straightforward as it could be, due to the official MySQL upgrade process not playing nice with the custom changes that Percona makes to information_schema and performance_schema, but I got it working in the end.

Code changes to accompany the ongoing database migration that will work hand-in-hand to support multiple scores per user per beatmap are still ongoing. Today I fixed some shortcomings which still existed in bancho.

comments

20150810

published 10 Aug 2015

Hope you are all having a productive and energetic start to your weeks.

Another quick Monday update, as I spent most of today catching up on completely uninteresting tasks that I will spare you the details of. This week will hopefully bring some exciting changes to the table which have taken a while to get ready for public consumption.

I can say that we may have made a breakthrough in graphics compatibility for the next osu! release (currently what you see on cutting-edge). While we are testing internally, the whole team is hopeful that this will bring us as close as we are going to get to 100% compatibility.

More to come on that soon, by means of a first-time guest post :).

comments