So let’s have a talk about something quite relevant today: bans and the way they are handled in osu!. Firstly, they are not handled by me except in edge-case scenarios (where unique circumstances exist and other team members cannot make a judgement on their own). I used to handle ban enquiries but the workload has increased to a level that is basically full-time work for one to two people.

There’s far too many intricacies to how we handle bans and appeals, so I’m going to address what I think are the most important facts in a very general way this time. If you want more information on a particular facet that I haven’t explained, then let me know in the comments below and (assuming I am able to without giving out confidential information) I will respond in a future post.

Firstly, while we throw the word “bans” around, it is very rarely that we ban accounts on osu! anymore. There are two levels of account restrictions which are silenced and restricted flags. Internally there are many other finer controls over how a user is limited, but we rarely use them anymore.

Silenced

The one which most of you are familiar with, silenced mode, puts a user in a muted state. They will be unable to respond on chat, join multiplayer games or post on the forums, but can continue to submit scores, check rankings, read chat and browse the site as per usual. We use this mode to warn people when they break rules, or when we suspect the user is about to do something that we can prevent with preemptive action. The most common offences to warrant a silence are chat or forum vandalism. Using BanchoBot commands we are able to stop users in their tracks before they harm the experience of others. Quite often this is done automatically by our trusty BanchoBot.

Silences start at five minutes in length, and double each time the user re-offends. In rare scenarios we do apply more than five minutes on the first silence, if we think the user is highly likely to reoffend straight after that five minute period is over (or if they have done something much more serious). They stack for up to four weeks, after which they disappear from the user’s profile. After going four weeks without offending, the user’s history will be reset (to the public) and with it, the length of future silences will also be reset.

Restricted

Restricted mode on the other hand is a much more serious flag to be applied to a user’s account. On top of it applying all the restrictions of the silenced flag, it also stops the user from appearing to others in score rankings, while also stopping performance points updates and other various internal processes. This mode can be applied by admins, but most of the time it is automatically set by our foul-play detection systems.

When we used to apply actual account bans, I didn’t want to make any mistakes and falsely destroy people’s legitimate accounts, so there were no automated bans placed. With the introduction of restricted mode on the other hand, we could put accounts into a frozen invisible-but-preserved state. This made automation possible, and changed our workload as admins from applying bans to removing false-positive restrictions. With the number of restrictions we see, this makes a lot of sense.

The restricted flag is used primarily for cases of foul-play. These are the ones that cause community drama (cheating, multi-accounting, account sharing, boosting, sale of accounts etc.). It is applied when we need to act in order to protect the core values of osu!.

How does the automated system apply restricted flags? That’s not something I can share with you, for hopefully obvious reasons. What I can say is that false positives are well under 1%. On releasing accounts from restricted mode, they are returned untouched with all newly set records correctly applied. When we fail to resolve a false positive in under 24 hours, we’ll at very least be highly apologetic, and in extreme cases reimburse the user for their lost time.

So then, why do you all see so many complaints about people being banned when they are innocent? About people being falsely accused of multi-accounting when they are saying they actually have siblings? Usually, they are lying. In the cases they aren’t lying, they are usually omitting information (ie. they insist they aren’t cheating, but they have 20 accounts hidden away).

So the, why do people say the osu! support staff are ignoring them? Usually, they are repeat offenders, trying to come back for months or years on end while hiding their identity each time. Why do they say we are rude to them? They probably deserve it. I closely monitor the responses to tickets by our support team, and trust me, I bring attention to even the smallest mistake (I’d tell you to just ask one of them, but we protect their identities to avoid harassment!). People are very quick to contact me by email if they think they were dealt with unfairly, and if they were, I will fix the situation to the best of my ability. 95 times out of 100, I will tell the user what the support team told them: to fuck off.

We try to give honest responses where other game support teams would simply ignore the user. This means that our responses seem “unprofessional” in many cases, but hey, would you rather “unprofessional”, or silence?

I plan on covering individual topics in more detail, such as why we don’t allow multiple accounts, in what cases we use special means such as score rollbacks, what counts as cheating, when appeals are given extra priority and much more. I’d like to clear up this stuff so it’s not taboo, in a hope to stop people putting so much time and effort into creating discussions around it, and instead enjoying the game for what it is.

Today’s Worklog:

• Managed to severely break the osu! client build server with some changes to php error reporting. While trying to track down the issue, I added a new bug which saw me spending almost two hours actually figuring out the underlying issue. Quite depressing, as I wrote this particular system to be resilient to any error. Except this one ;).
• As I recently hooked up the old (current) osu! web to sentry (we use a self-deploy as the pricing of the hosted service is stupid), I spent some time doing general maintenance based on the most prominent error reports. Fixed some lingering issues in the process, like profile A/S/SS ranks not being reverted when a map is disqualified.
• Added a toggle in the osu! client to choose whether automatic downloads include videos. Live on cutting-edge.
• Updated the localisation updater build tool to correctly OAuth with the google sheets API. They shut down the basic auth API last week, and while I already solved that for bancho (it uses it to fetch the FAQ), the localisation updater tool was still a tad broken.
• Followed up the next tablet order. Figuring the logistics of how we are going to store 2,000 tablets this time, as opposed to the usual order of 1,000. On top of that, the person I have helping me with the store is on holiday for a couple of weeks, during which the tablets are likely to arrive!! Lots of planning to be done…

I spent the morning catching up on emails I put off over the weekend. This included accounting follow-ups, payment system integration follow-ups and the usual catch-up on cheaters and foul-play. You don’t even want to know how much time goes into the last of those!

Today’s Worklog:

• Brought bm4 and bm5 (the current two active mirrors) up-to-date with Debian 8 (jessie). Also took the opportunity to switch bm4 over to hhvm, as it was still using php-fpm for some reason.
• Added support for regional mirrors. Currently this is done by providing a CSV list of ISO 3166-1 Alpha 2 country names in the database row for a mirror. If present, that mirror will be used exclusively for the regions specified. An entry of null means it will behave normally, for countries with no exclusive mirror specified.
• Added the ability to remove mirrors from the mirror rotation. Until now, mirrors would automatically be disabled when they are not available, but in the disabled state they would build up a queue of commands to run when they return. For mirrors that aren’t really planning on returning, this queue gets really full really fast (think purging WIP beatmaps each time the creator updates them), so rather than having to remove the mirror entries from the database, I wanted a way to stop them receiving queued events and being checked for recovery.
• Updated the mirror-side download script to have more verbose checking on whether the files it received from Amazon S3 are complete and correct. If they aren’t, they are re-downloaded each time a user requests them until they are found to be the right size.
• Commissioned osu!’s first EU mirror server. It’s still in a state of initialising, as it synchronises almost 1tb of data over, but this doesn’t affect its ability to serve the full EU player base. As with alll beatmap mirrors, the server specs are dual core latest-gen xeons with 64gb RAM (think file cache) and two 2tb drives running in RAID1 for boosted performance and redundancy. 500mbit traffic burstable to 1gbit.
• Continued work on the new modular store product page which will allow for username changes to happen. As the new osu!web infrastructure is still in early stages, progress takes a while as we plan exactly how to go about implementing particular scenarios. I hope to open the new website source up to the public in the coming weeks (it was made with this in mind). Keep a watch out :).

If you’re in the EU, let me know how the new mirror is treating you (your downloads should be coming from bm6.ppy.sh; let me know if otherwise)! Are you getting better speeds than before?

Next on the mirror list is a server in Oceania/South-East Asia. Something like South Korea or Singapore would work nice, I think.

I spent Saturday deciding whether I should continue daily posting on weekends. Until last year I would work through every weekend as if they didn’t exist, but due to recent changes in my daily life and work style – specifically actually working in an office with other osu! people around rough work hours – I tend to focus my energy into weekdays and give myself a bit of time off on weekends now. Not to say I get nothing done; keeping osu! alive requires constant monitoring of the servers and community. Even when I take a day off, I will spend one or two hours catching up on things, and this number can shoot up like crazy if there are any fires are put out!

I’ll probably touch back on the topic of time management and efficiency many times over the course of this blogging, since I believe it is the most important thing in life. Time is the only truly finite resource to a single human, after all.

I decided it’s best for the longevity of this posting structure that I only do one update each weekend. I will likely focus on a specific story or element of work/osu!/technology which I think may be interesting to you guys, while also sharing anything cool I get up to with my time (because why not!).

This weekend I managed to go to a club to see my favourite DJ-san (Nhato, of course), visited a local festival and spent most of Sunday at a starbucks working through email and fiddling with the username change code that I’ve been working on.

Here’s to a productive week ahead of us!

I’m writing this for the second time, since it somehow disappeared the first time :(. That’ll teach me for trusting some chrome based markdown editor. stubbornly returns to sublime text 3

In recent times I look at the user-facing changes and and often disappointed in what I see. My own role in osu! has changed from being able to rapidly implement new features to only just being able to keep up with what is going on each day. The rare occasions I’m able to implement something new are therefore very enjoyable as a result, but I digress.

The ongoing tasks which I choose to complete myself rather than delegate remain that way for a reason. I believe each and every one of them requires my own attention. Delegating tasks is great and all, but there are two issues with it:

• Delegating things that come up infrequently is often less efficient. An example would be emails about conventions, which are always different in nature. Sometimes people will want merchandise, sometimes they will want to promote osu! by presenting a panel. Teaching someone how to respond to these emails would require writing a manual of every possible scenario, or being able to trust their judgement, which leads on to the second issue..
• I find it hard to trust people. Not trust as in confidentiality, but trust as in the ability to diligently handle a scenario in the same way I would myself. I am pretty sure I am yet to meet someone with the same values and level of attention-to-detail that I have. While this may sound like I’m bragging, if you ever run a company of your own you will understand this feeling.

Today was one of those days I spent catching up on mostly mundane run-of-the-mill tasks. You’re still interested? Okay then!

Today’s Worklog:

• Fixed a loophole in the account recovery process that was abused overnight. Accounts can now be set into a state where users cannot initiate password recovery and must contact support.
• Purged my inbox backlog down from 188 to 55. I read emails as the come in and when necessary reply within a minute or two, but less important emails are snoozed until a later date. I find it easier to process these in one large batch rather than as-they-come. This is what my inbox looks like now. Might take a few beers tomorrow to get through the remainder ;).
• Brought company accounts up-to-date. Basically distributing money into the correct accounts so people are paid and servers are kept online and purring. I have an accountant to handle things at the end of the year, but daily financials are still my own responsibility. Need to attend to them once a week else they pile up quickly.
• Got acceptance email for AWS Aurora preview. Not sure if I will use this going forward, but couldn’t resist the opportunity to test it for free! Spun up two beefy 16 core 108gb instances which I will be deploying the full osu! database to for performance testing over the coming weeks.
• Figured out osu!keyboard stock issues. We needed to reserve a few keybaords for Taiko World Cup winners, but due a number of faulty units being found during initial quality checking, we ran a bit short.
• Approved and finalised payment for the next osu!tablet production run, this time for 2,000 units! We still have stock from the last run too. I think I have finally got my head around the convoluted processes involved in international manufacturing and shipping, so stock shortages should be a thing of the past.
• Our knowledge base is now fully stocked with content, hopefully helping users before they unnecessarily post to the forum!

So let’s briefly revisit something that happened last week: OVH had a downtime which lasted several hours as the result of a cable being cut. This meant the sudden loss of all our beatmap download mirror traffic! As I was “enjoying” an Owl City concert at the time, I had @nekodex - who you may also know due to his musical contributions to osu! - help me get a new mirror up and running.

osu! beatmap mirrors are deployed as a single file php script with no external dependencies. They use an s3 endpoint to retrieve maps, cache them locally and serve them to users using a secure checksum system. The script is not only self-containing, but updates can be pushed out in an automated process. Each mirror is monitored by the central osu! web server and mirrors which are unavailable or underperforming are automatically removed from the available list.

We had things back up and running in under 20 minutes as a result, but when things like this go haywire it is always wise to add countermeasures to ensure the same thing doesn’t happen again. I have started investigating where I will be adding some additional mirrors in order to add geographical redundancy to the system. As I don’t want to introduce any more complexity to the accounting side of the infrastructure setup of osu! (already across four datacenters!!) they will likely be placed in OVH’s European datacenters.

Going forward, I would love to open the mirroring system up to the public again, letting people lend their spare server bandwidth to the good of the osu! community. This would require some logic to ensure users were linked up with the closest mirror to their physical location, of course. Think a mini-CDN just for osu!. Alternatively maybe I should activate the dormant p2p code that has been sitting inside osu! for several years… ^^;

Have a good weekend. Play lots of osu!.

Many people say that it’s the vocal minorities that express the most negativity, and while this is definitely the case, I still aim for the unattainable 100% satisfaction. I hold myself to high standards that I have learned over the years I cannot expect from anyone else. This is actually one reason I am aiming to move osu! to an open-source model, but that story is for another day!

I go out of my way to answer every single email that comes to my personal address. The answers may be brief, but I take in every single word of feedback that users have to offer. I also check the forums, reddit, BBS and basically anywhere people are discussing osu!.

While some may say this is a complete waste of time and focus - and this may be true - I feel it’s something I cannot avoid if I want to keep up-to-date with the temperature of the community. Unique things happen almost every day, quite often which need multiple team members’ input to resolve, so it’s important to stay on the watch for ongoing and upcoming events.

Remember that osu! is actually a very simple game at its core. The gameplay has rarely changed over the years, while the ecosystem around that gameplay is constantly developing. The drive for more accessibility, more stats to track, better ways to communicate with friends is strong. The social and community aspects of osu! really make it feel to me less game, more web service.

Not to say we don’t have stuff in the pipeline to spice up the gameplay too ;).

Today’s Changes:

• The forum search system now defaults to matching all keywords, rather than any. The sorting of results returned by the back-end (elasticsearch, for those playing along at home) has also been changed to relevance rather than freshness. Sorting from newest to oldest is still applied at the forum front-end as a last step. This should hopefully provide for much more relevant results. Once the new site is up, search will be overhauled and provided in a similar way to how you use the facebook search box. For now, let’s make do!
• The help forum now shows only the “Chat with Support” button when someone is available. There are way too many posts in the help forum that could be answered in a single line (profile being in wrong mode, pp not updating etc.). While this is a band-aid measure while we get a full knowledge base system ready, it should help.
• Began work on the osu!web (2.0) site for the first time in a long while. I have made a promise with myself to not add any new features to the old web, so the new username changing system is going to have to be done here.
• Spent a good portion of last night getting my dev environment set up correctly for osu!web development. The new site is being written completely from scratch in Laravel 5. Learning a new framework as I go, already I already wrote the web store in it quite some months back.
• In order to support username changes, which are going to be presented as an item on the store, we first need support for store items with infinite stock and custom display implementations. I made good progress on building the framework for this.

For those freaking out about the username change being a store items, here’s how it’ll work:

• First Change: free (with supporter)
• Second Change: $8 (equiv. 2 months supporter)
• Third Change: $16 (equiv. 4 months supporter)
• Fourth Change: $32 (equiv. 8 months supporter)

I’m sure you can see the pattern here. It will continue to double, likely with a cap at around $100. This may seem like an expensive purchase, but the idea here is to stop people from recklessly changing their name too many times. We discussed the cost amongst team members and decided doubling each time was the best approach to achieve this.

As for history, you will be able to see all previous usernames that a user was known as in their profile, as you’d expect.