These days, we have a lot of highly skilled people working on osu!. It’s no longer the era of everything-is-100%-done-by-peppy. One thing I aim to do with this blog effort is to give everyone on the team a voice. With yesterdays post being pretty exhausting for me, I was happy to take a break today when Tom94 offered to fill in :).

Hi everyone, Tom94 here with a guest post for today.

As many of you probably know already, we are currently pushing osu! towards running exclusively on top of the OpenGL graphics API rather than supporting DirectX and OpenGL alongside each other. This has multiple benefits, such as better cross platform support (hi, Mac and Linux users!) and less effort to implement and optimize visual features. This is not what I want to go in-depth about, though. Today I want to talk about a particular issue with osu!’s OpenGL implementation that has plagued us for a while: Users running on Intel integrated graphics chips mysteriously experienced severely worse performance than when they were running on DirectX.

Luckily, since I own a device with such a graphics chip inside I could test and narrow down where the issue came from. After many hours of troubleshooting and trying various complicated solutions without success, I found that the graphics chip did not support the graphics settings osu! wanted it to use and instead fell back to using the first setting it could find. Turns out this setting included 4x anti-aliasing, a very expensive technique for potentially improving image quality, but one which osu! cannot even profit from. As ridiculous as it may sound, we fixed a long standing huge performance problem simply by turning off anti-aliasing. Nobody noticed it was on in the first place, because in osu! there isn’t any noticeable benefit to it. At the end of the day by changing a single line of code the performance on Intel hardware increased by more than twofold in most cases. And it doesn’t even stop there. Another issue – blurry lines within the editor – which we were just as puzzled about ended up being fixed by this change as well.

This illustrates how when developing software it is the norm that something unexpected happens. Surprisingly often after hours of looking for a bugfix only a very minor change is needed. The challenge is to find this particular small change rather than trying to cover up an issue by patching it up with the programming equivalent of duct tape.

One positive aspect this whole ordeal brought with it was the addition of an in-game overlay designed to detect and troubleshoot issues with performance. It can be toggled via Ctrl+F11 when running the beta or cutting-edge version of the game and helps us greatly to figure out which parts of the game are running slowly. Here is an example of how it looks like when rapidly scrolling through the song selection.

Other interesting fixes that recently happened are:

• Fixed clicks sometimes not being handled if osu! ran at a too high framerate. Yes, this is sort of a first-world problem. A rounding error in the timing code made the game sometimes think objects have been clicked before they have been drawn to the screen. This is yet another very short fix which thankfully was a lot easier to find than the one above.

• Improved the way spinners work, fixing various things in the process. Auto now doesn’t score wrong amounts with the DoubleTime and HalfTime mods and the spinner animation is no longer capped at 60 frames per second.

• The hover sound loop in song select is now finally gone!

I’ve also been optimizing the pp processor to be ready for future things to come, but this is material for another blog post. :) If there are any questions I’ll gladly answer them in the comments!

So let’s have a talk about something quite relevant today: bans and the way they are handled in osu!. Firstly, they are not handled by me except in edge-case scenarios (where unique circumstances exist and other team members cannot make a judgement on their own). I used to handle ban enquiries but the workload has increased to a level that is basically full-time work for one to two people.

There’s far too many intricacies to how we handle bans and appeals, so I’m going to address what I think are the most important facts in a very general way this time. If you want more information on a particular facet that I haven’t explained, then let me know in the comments below and (assuming I am able to without giving out confidential information) I will respond in a future post.

Firstly, while we throw the word “bans” around, it is very rarely that we ban accounts on osu! anymore. There are two levels of account restrictions which are silenced and restricted flags. Internally there are many other finer controls over how a user is limited, but we rarely use them anymore.

Silenced

The one which most of you are familiar with, silenced mode, puts a user in a muted state. They will be unable to respond on chat, join multiplayer games or post on the forums, but can continue to submit scores, check rankings, read chat and browse the site as per usual. We use this mode to warn people when they break rules, or when we suspect the user is about to do something that we can prevent with preemptive action. The most common offences to warrant a silence are chat or forum vandalism. Using BanchoBot commands we are able to stop users in their tracks before they harm the experience of others. Quite often this is done automatically by our trusty BanchoBot.

Silences start at five minutes in length, and double each time the user re-offends. In rare scenarios we do apply more than five minutes on the first silence, if we think the user is highly likely to reoffend straight after that five minute period is over (or if they have done something much more serious). They stack for up to four weeks, after which they disappear from the user’s profile. After going four weeks without offending, the user’s history will be reset (to the public) and with it, the length of future silences will also be reset.

Restricted

Restricted mode on the other hand is a much more serious flag to be applied to a user’s account. On top of it applying all the restrictions of the silenced flag, it also stops the user from appearing to others in score rankings, while also stopping performance points updates and other various internal processes. This mode can be applied by admins, but most of the time it is automatically set by our foul-play detection systems.

When we used to apply actual account bans, I didn’t want to make any mistakes and falsely destroy people’s legitimate accounts, so there were no automated bans placed. With the introduction of restricted mode on the other hand, we could put accounts into a frozen invisible-but-preserved state. This made automation possible, and changed our workload as admins from applying bans to removing false-positive restrictions. With the number of restrictions we see, this makes a lot of sense.

The restricted flag is used primarily for cases of foul-play. These are the ones that cause community drama (cheating, multi-accounting, account sharing, boosting, sale of accounts etc.). It is applied when we need to act in order to protect the core values of osu!.

How does the automated system apply restricted flags? That’s not something I can share with you, for hopefully obvious reasons. What I can say is that false positives are well under 1%. On releasing accounts from restricted mode, they are returned untouched with all newly set records correctly applied. When we fail to resolve a false positive in under 24 hours, we’ll at very least be highly apologetic, and in extreme cases reimburse the user for their lost time.

So then, why do you all see so many complaints about people being banned when they are innocent? About people being falsely accused of multi-accounting when they are saying they actually have siblings? Usually, they are lying. In the cases they aren’t lying, they are usually omitting information (ie. they insist they aren’t cheating, but they have 20 accounts hidden away).

So the, why do people say the osu! support staff are ignoring them? Usually, they are repeat offenders, trying to come back for months or years on end while hiding their identity each time. Why do they say we are rude to them? They probably deserve it. I closely monitor the responses to tickets by our support team, and trust me, I bring attention to even the smallest mistake (I’d tell you to just ask one of them, but we protect their identities to avoid harassment!). People are very quick to contact me by email if they think they were dealt with unfairly, and if they were, I will fix the situation to the best of my ability. 95 times out of 100, I will tell the user what the support team told them: to fuck off.

We try to give honest responses where other game support teams would simply ignore the user. This means that our responses seem “unprofessional” in many cases, but hey, would you rather “unprofessional”, or silence?

I plan on covering individual topics in more detail, such as why we don’t allow multiple accounts, in what cases we use special means such as score rollbacks, what counts as cheating, when appeals are given extra priority and much more. I’d like to clear up this stuff so it’s not taboo, in a hope to stop people putting so much time and effort into creating discussions around it, and instead enjoying the game for what it is.

Today’s Worklog:

• Managed to severely break the osu! client build server with some changes to php error reporting. While trying to track down the issue, I added a new bug which saw me spending almost two hours actually figuring out the underlying issue. Quite depressing, as I wrote this particular system to be resilient to any error. Except this one ;).
• As I recently hooked up the old (current) osu! web to sentry (we use a self-deploy as the pricing of the hosted service is stupid), I spent some time doing general maintenance based on the most prominent error reports. Fixed some lingering issues in the process, like profile A/S/SS ranks not being reverted when a map is disqualified.
• Added a toggle in the osu! client to choose whether automatic downloads include videos. Live on cutting-edge.
• Updated the localisation updater build tool to correctly OAuth with the google sheets API. They shut down the basic auth API last week, and while I already solved that for bancho (it uses it to fetch the FAQ), the localisation updater tool was still a tad broken.
• Followed up the next tablet order. Figuring the logistics of how we are going to store 2,000 tablets this time, as opposed to the usual order of 1,000. On top of that, the person I have helping me with the store is on holiday for a couple of weeks, during which the tablets are likely to arrive!! Lots of planning to be done…

I spent the morning catching up on emails I put off over the weekend. This included accounting follow-ups, payment system integration follow-ups and the usual catch-up on cheaters and foul-play. You don’t even want to know how much time goes into the last of those!

Today’s Worklog:

• Brought bm4 and bm5 (the current two active mirrors) up-to-date with Debian 8 (jessie). Also took the opportunity to switch bm4 over to hhvm, as it was still using php-fpm for some reason.
• Added support for regional mirrors. Currently this is done by providing a CSV list of ISO 3166-1 Alpha 2 country names in the database row for a mirror. If present, that mirror will be used exclusively for the regions specified. An entry of null means it will behave normally, for countries with no exclusive mirror specified.
• Added the ability to remove mirrors from the mirror rotation. Until now, mirrors would automatically be disabled when they are not available, but in the disabled state they would build up a queue of commands to run when they return. For mirrors that aren’t really planning on returning, this queue gets really full really fast (think purging WIP beatmaps each time the creator updates them), so rather than having to remove the mirror entries from the database, I wanted a way to stop them receiving queued events and being checked for recovery.
• Updated the mirror-side download script to have more verbose checking on whether the files it received from Amazon S3 are complete and correct. If they aren’t, they are re-downloaded each time a user requests them until they are found to be the right size.
• Commissioned osu!’s first EU mirror server. It’s still in a state of initialising, as it synchronises almost 1tb of data over, but this doesn’t affect its ability to serve the full EU player base. As with alll beatmap mirrors, the server specs are dual core latest-gen xeons with 64gb RAM (think file cache) and two 2tb drives running in RAID1 for boosted performance and redundancy. 500mbit traffic burstable to 1gbit.
• Continued work on the new modular store product page which will allow for username changes to happen. As the new osu!web infrastructure is still in early stages, progress takes a while as we plan exactly how to go about implementing particular scenarios. I hope to open the new website source up to the public in the coming weeks (it was made with this in mind). Keep a watch out :).

If you’re in the EU, let me know how the new mirror is treating you (your downloads should be coming from bm6.ppy.sh; let me know if otherwise)! Are you getting better speeds than before?

Next on the mirror list is a server in Oceania/South-East Asia. Something like South Korea or Singapore would work nice, I think.

I spent Saturday deciding whether I should continue daily posting on weekends. Until last year I would work through every weekend as if they didn’t exist, but due to recent changes in my daily life and work style – specifically actually working in an office with other osu! people around rough work hours – I tend to focus my energy into weekdays and give myself a bit of time off on weekends now. Not to say I get nothing done; keeping osu! alive requires constant monitoring of the servers and community. Even when I take a day off, I will spend one or two hours catching up on things, and this number can shoot up like crazy if there are any fires are put out!

I’ll probably touch back on the topic of time management and efficiency many times over the course of this blogging, since I believe it is the most important thing in life. Time is the only truly finite resource to a single human, after all.

I decided it’s best for the longevity of this posting structure that I only do one update each weekend. I will likely focus on a specific story or element of work/osu!/technology which I think may be interesting to you guys, while also sharing anything cool I get up to with my time (because why not!).

This weekend I managed to go to a club to see my favourite DJ-san (Nhato, of course), visited a local festival and spent most of Sunday at a starbucks working through email and fiddling with the username change code that I’ve been working on.

Here’s to a productive week ahead of us!

I’m writing this for the second time, since it somehow disappeared the first time :(. That’ll teach me for trusting some chrome based markdown editor. stubbornly returns to sublime text 3

In recent times I look at the user-facing changes and and often disappointed in what I see. My own role in osu! has changed from being able to rapidly implement new features to only just being able to keep up with what is going on each day. The rare occasions I’m able to implement something new are therefore very enjoyable as a result, but I digress.

The ongoing tasks which I choose to complete myself rather than delegate remain that way for a reason. I believe each and every one of them requires my own attention. Delegating tasks is great and all, but there are two issues with it:

• Delegating things that come up infrequently is often less efficient. An example would be emails about conventions, which are always different in nature. Sometimes people will want merchandise, sometimes they will want to promote osu! by presenting a panel. Teaching someone how to respond to these emails would require writing a manual of every possible scenario, or being able to trust their judgement, which leads on to the second issue..
• I find it hard to trust people. Not trust as in confidentiality, but trust as in the ability to diligently handle a scenario in the same way I would myself. I am pretty sure I am yet to meet someone with the same values and level of attention-to-detail that I have. While this may sound like I’m bragging, if you ever run a company of your own you will understand this feeling.

Today was one of those days I spent catching up on mostly mundane run-of-the-mill tasks. You’re still interested? Okay then!

Today’s Worklog:

• Fixed a loophole in the account recovery process that was abused overnight. Accounts can now be set into a state where users cannot initiate password recovery and must contact support.
• Purged my inbox backlog down from 188 to 55. I read emails as the come in and when necessary reply within a minute or two, but less important emails are snoozed until a later date. I find it easier to process these in one large batch rather than as-they-come. This is what my inbox looks like now. Might take a few beers tomorrow to get through the remainder ;).
• Brought company accounts up-to-date. Basically distributing money into the correct accounts so people are paid and servers are kept online and purring. I have an accountant to handle things at the end of the year, but daily financials are still my own responsibility. Need to attend to them once a week else they pile up quickly.
• Got acceptance email for AWS Aurora preview. Not sure if I will use this going forward, but couldn’t resist the opportunity to test it for free! Spun up two beefy 16 core 108gb instances which I will be deploying the full osu! database to for performance testing over the coming weeks.
• Figured out osu!keyboard stock issues. We needed to reserve a few keybaords for Taiko World Cup winners, but due a number of faulty units being found during initial quality checking, we ran a bit short.
• Approved and finalised payment for the next osu!tablet production run, this time for 2,000 units! We still have stock from the last run too. I think I have finally got my head around the convoluted processes involved in international manufacturing and shipping, so stock shortages should be a thing of the past.
• Our knowledge base is now fully stocked with content, hopefully helping users before they unnecessarily post to the forum!

So let’s briefly revisit something that happened last week: OVH had a downtime which lasted several hours as the result of a cable being cut. This meant the sudden loss of all our beatmap download mirror traffic! As I was “enjoying” an Owl City concert at the time, I had @nekodex - who you may also know due to his musical contributions to osu! - help me get a new mirror up and running.

osu! beatmap mirrors are deployed as a single file php script with no external dependencies. They use an s3 endpoint to retrieve maps, cache them locally and serve them to users using a secure checksum system. The script is not only self-containing, but updates can be pushed out in an automated process. Each mirror is monitored by the central osu! web server and mirrors which are unavailable or underperforming are automatically removed from the available list.

We had things back up and running in under 20 minutes as a result, but when things like this go haywire it is always wise to add countermeasures to ensure the same thing doesn’t happen again. I have started investigating where I will be adding some additional mirrors in order to add geographical redundancy to the system. As I don’t want to introduce any more complexity to the accounting side of the infrastructure setup of osu! (already across four datacenters!!) they will likely be placed in OVH’s European datacenters.

Going forward, I would love to open the mirroring system up to the public again, letting people lend their spare server bandwidth to the good of the osu! community. This would require some logic to ensure users were linked up with the closest mirror to their physical location, of course. Think a mini-CDN just for osu!. Alternatively maybe I should activate the dormant p2p code that has been sitting inside osu! for several years… ^^;

Have a good weekend. Play lots of osu!.