20150611 /deploy/

published 11 Jun 2015

switch release streams from the options menu

As most of you may be aware of, the osu! build system got a lot more advanced earlier this year with the introduction of Release Streams. The basic idea behind these is that we can have multiple streams of osu! clients, allowing users to choose how close to the edge of their seat they wish to sit. While from time-to-time we have added and removed temporary testing streams (for things like testing a new experimental framework), there are generally three release streams available at all times:

Stable

The default release stream. Stable builds are generally, as their name implies, stable. There shouldn’t be any new bugs that find their way into these releases (in theory!). Stable builds are released as often as possible, but generally this ends up being between 1-8 weeks between releases. My personal aim is one release a month, but depending on the state of code and how deep in shit we are, this sometimes is not possible.

Beta

Beta releases are a mid-ground, offering inquisitive users early access to new features and fixes that are in the pipeline, without the possibility of the world falling apart on them. The beta release stream is safe for anyone willing to report bugs they may find, rather than raging about it.

If you don’t mind helping, please stay on the Beta stream. The more people we have testing this, the less chance bugs will find their way into stable releases, affecting the masses.

Beta releases happen every Friday automatically, or sooner if we decide there’s a need to get a hot-fix out to users rapidly.

Cutting-Edge

This is where things get interesting. You’ll be exposed to changes made to the osu! code-base in real-time. If we fuck up, it could fuck your osu! up badly. But you’re already here so you know these risks, right?! In exchange for the extra risks, we will give you our ears 24/7 and work with you to fix any issues you come across. You will also get access to some preview features that aren’t quite ready for prime-time, such as the Target Practice mod, the ability to fast-forward replays at varying speeds and more to come.

Cutting edge releases happen as we make changes, multiple times a day. We won’t force you to come join us on the edge, but we do appreciate those who do!

But How?

So… how do these streams work internally? We use git for source control, with two main branches of code: one for development (master) and one for stable releases (release). When any osu!dev makes changes, they first create a new branch of their own to isolate their changes. Once they are happy with what they’ve done, and want to share it with the team, they will submit a pull request. This is basically a way of saying “Hey, I’m ready for you guys to come check my code out!”.

Once at least one other osu!dev has reviewed a pull request, it is merged into the master branch. For those unfamiliar with the term “merge”, this basically means the code is moved from the developer’s private branch into the development branch, and considered to be finalised. The pull request and the master branch become one, and the pull request is destroyed.

Every time a pull request is successfully merged into master, a cutting-edge release is triggered by the build server (we use teamcity for .NET, buildkite for everything else). And that’s all there is to cutting edge releases!

Beta and stable releases are a slightly different story. As we don’t want to push either of these out until changes have had a while to simmer with cutting-edge testers and devs alike, it is up to me to decide when to merge master into release. Even then, performing the merge alone doesn’t do anything; it just prepares the changes for future distribution.

Both the Beta and Stable release streams are based on the release branch; the only difference is their triggers. Beta, as mentioned before, will trigger automatically every Friday. This is to space the builds out evenly and ensure users are not bombarded with constant updates. Stable releases are completely manual, and only triggered when I am completely happy with the state of the release branch.

If all this seems very complex, just thing of the three streams as a way of filtering out bugs from the majority of users. Cutting-Edge users get hit by most of them, and they are fixed before they hit Beta. Beta users (a larger user base) then get to confirm that we have actually covered all bases. Assuming beta users don’t find any issues, everyone is happy and we go ahead with a stable. Should any issues be found, it’s back to cutting-edge for a round of bug fixes!

I attempted to sum all this up into a pretty diagram. Click for full size!

release streams in a can

There’s a lot more complexity not covered in this article that I will share in the future!

Automating deployment makes us very productive. The ability to push fixes out with just two clicks and having it completely integrated into our existing git/github workflows is really efficient and just doesn’t get old :).

comments

20150611 /team94/

published 10 Jun 2015

These days, we have a lot of highly skilled people working on osu!. It’s no longer the era of everything-is-100%-done-by-peppy. One thing I aim to do with this blog effort is to give everyone on the team a voice. With yesterdays post being pretty exhausting for me, I was happy to take a break today when Tom94 offered to fill in :).

Hi everyone, Tom94 here with a guest post for today.

As many of you probably know already, we are currently pushing osu! towards running exclusively on top of the OpenGL graphics API rather than supporting DirectX and OpenGL alongside each other. This has multiple benefits, such as better cross platform support (hi, Mac and Linux users!) and less effort to implement and optimize visual features. This is not what I want to go in-depth about, though. Today I want to talk about a particular issue with osu!’s OpenGL implementation that has plagued us for a while: Users running on Intel integrated graphics chips mysteriously experienced severely worse performance than when they were running on DirectX.

Luckily, since I own a device with such a graphics chip inside I could test and narrow down where the issue came from. After many hours of troubleshooting and trying various complicated solutions without success, I found that the graphics chip did not support the graphics settings osu! wanted it to use and instead fell back to using the first setting it could find. Turns out this setting included 4x anti-aliasing, a very expensive technique for potentially improving image quality, but one which osu! cannot even profit from. As ridiculous as it may sound, we fixed a long standing huge performance problem simply by turning off anti-aliasing. Nobody noticed it was on in the first place, because in osu! there isn’t any noticeable benefit to it. At the end of the day by changing a single line of code the performance on Intel hardware increased by more than twofold in most cases. And it doesn’t even stop there. Another issue – blurry lines within the editor – which we were just as puzzled about ended up being fixed by this change as well.

This illustrates how when developing software it is the norm that something unexpected happens. Surprisingly often after hours of looking for a bugfix only a very minor change is needed. The challenge is to find this particular small change rather than trying to cover up an issue by patching it up with the programming equivalent of duct tape.

One positive aspect this whole ordeal brought with it was the addition of an in-game overlay designed to detect and troubleshoot issues with performance. It can be toggled via Ctrl+F11 when running the beta or cutting-edge version of the game and helps us greatly to figure out which parts of the game are running slowly. Here is an example of how it looks like when rapidly scrolling through the song selection.

Frame-time display

Other interesting fixes that recently happened are:

  • Fixed clicks sometimes not being handled if osu! ran at a too high framerate. Yes, this is sort of a first-world problem. A rounding error in the timing code made the game sometimes think objects have been clicked before they have been drawn to the screen. This is yet another very short fix which thankfully was a lot easier to find than the one above.

  • Improved the way spinners work, fixing various things in the process. Auto now doesn’t score wrong amounts with the DoubleTime and HalfTime mods and the spinner animation is no longer capped at 60 frames per second.

  • The hover sound loop in song select is now finally gone!

I’ve also been optimizing the pp processor to be ready for future things to come, but this is material for another blog post. :) If there are any questions I’ll gladly answer them in the comments!

comments

20150609 /bans/

published 09 Jun 2015

So let’s have a talk about something quite relevant today: bans and the way they are handled in osu!. Firstly, they are not handled by me except in edge-case scenarios (where unique circumstances exist and other team members cannot make a judgement on their own). I used to handle ban enquiries but the workload has increased to a level that is basically full-time work for one to two people.

you're 99% fucked if you see this

There’s far too many intricacies to how we handle bans and appeals, so I’m going to address what I think are the most important facts in a very general way this time. If you want more information on a particular facet that I haven’t explained, then let me know in the comments below and (assuming I am able to without giving out confidential information) I will respond in a future post.

Firstly, while we throw the word “bans” around, it is very rarely that we ban accounts on osu! anymore. There are two levels of account restrictions which are silenced and restricted flags. Internally there are many other finer controls over how a user is limited, but we rarely use them anymore.

Silenced

The one which most of you are familiar with, silenced mode, puts a user in a muted state. They will be unable to respond on chat, join multiplayer games or post on the forums, but can continue to submit scores, check rankings, read chat and browse the site as per usual. We use this mode to warn people when they break rules, or when we suspect the user is about to do something that we can prevent with preemptive action. The most common offences to warrant a silence are chat or forum vandalism. Using BanchoBot commands we are able to stop users in their tracks before they harm the experience of others. Quite often this is done automatically by our trusty BanchoBot.

Silences start at five minutes in length, and double each time the user re-offends. In rare scenarios we do apply more than five minutes on the first silence, if we think the user is highly likely to reoffend straight after that five minute period is over (or if they have done something much more serious). They stack for up to four weeks, after which they disappear from the user’s profile. After going four weeks without offending, the user’s history will be reset (to the public) and with it, the length of future silences will also be reset.

Restricted

Restricted mode on the other hand is a much more serious flag to be applied to a user’s account. On top of it applying all the restrictions of the silenced flag, it also stops the user from appearing to others in score rankings, while also stopping performance points updates and other various internal processes. This mode can be applied by admins, but most of the time it is automatically set by our foul-play detection systems.

When we used to apply actual account bans, I didn’t want to make any mistakes and falsely destroy people’s legitimate accounts, so there were no automated bans placed. With the introduction of restricted mode on the other hand, we could put accounts into a frozen invisible-but-preserved state. This made automation possible, and changed our workload as admins from applying bans to removing false-positive restrictions. With the number of restrictions we see, this makes a lot of sense.

The restricted flag is used primarily for cases of foul-play. These are the ones that cause community drama (cheating, multi-accounting, account sharing, boosting, sale of accounts etc.). It is applied when we need to act in order to protect the core values of osu!.

How does the automated system apply restricted flags? That’s not something I can share with you, for hopefully obvious reasons. What I can say is that false positives are well under 1%. On releasing accounts from restricted mode, they are returned untouched with all newly set records correctly applied. When we fail to resolve a false positive in under 24 hours, we’ll at very least be highly apologetic, and in extreme cases reimburse the user for their lost time.

So then, why do you all see so many complaints about people being banned when they are innocent? About people being falsely accused of multi-accounting when they are saying they actually have siblings? Usually, they are lying. In the cases they aren’t lying, they are usually omitting information (ie. they insist they aren’t cheating, but they have 20 accounts hidden away).

So the, why do people say the osu! support staff are ignoring them? Usually, they are repeat offenders, trying to come back for months or years on end while hiding their identity each time. Why do they say we are rude to them? They probably deserve it. I closely monitor the responses to tickets by our support team, and trust me, I bring attention to even the smallest mistake (I’d tell you to just ask one of them, but we protect their identities to avoid harassment!). People are very quick to contact me by email if they think they were dealt with unfairly, and if they were, I will fix the situation to the best of my ability. 95 times out of 100, I will tell the user what the support team told them: to fuck off.

We try to give honest responses where other game support teams would simply ignore the user. This means that our responses seem “unprofessional” in many cases, but hey, would you rather “unprofessional”, or silence?

I plan on covering individual topics in more detail, such as why we don’t allow multiple accounts, in what cases we use special means such as score rollbacks, what counts as cheating, when appeals are given extra priority and much more. I’d like to clear up this stuff so it’s not taboo, in a hope to stop people putting so much time and effort into creating discussions around it, and instead enjoying the game for what it is.

Today’s Worklog:

  • Managed to severely break the osu! client build server with some changes to php error reporting. While trying to track down the issue, I added a new bug which saw me spending almost two hours actually figuring out the underlying issue. Quite depressing, as I wrote this particular system to be resilient to any error. Except this one ;).
  • As I recently hooked up the old (current) osu! web to sentry (we use a self-deploy as the pricing of the hosted service is stupid), I spent some time doing general maintenance based on the most prominent error reports. Fixed some lingering issues in the process, like profile A/S/SS ranks not being reverted when a map is disqualified.
  • Added a toggle in the osu! client to choose whether automatic downloads include videos. Live on cutting-edge.
  • Updated the localisation updater build tool to correctly OAuth with the google sheets API. They shut down the basic auth API last week, and while I already solved that for bancho (it uses it to fetch the FAQ), the localisation updater tool was still a tad broken.
  • Followed up the next tablet order. Figuring the logistics of how we are going to store 2,000 tablets this time, as opposed to the usual order of 1,000. On top of that, the person I have helping me with the store is on holiday for a couple of weeks, during which the tablets are likely to arrive!! Lots of planning to be done…
comments

20150608 /eu/

published 08 Jun 2015

I spent the morning catching up on emails I put off over the weekend. This included accounting follow-ups, payment system integration follow-ups and the usual catch-up on cheaters and foul-play. You don’t even want to know how much time goes into the last of those!

Today’s Worklog:

  • Brought bm4 and bm5 (the current two active mirrors) up-to-date with Debian 8 (jessie). Also took the opportunity to switch bm4 over to hhvm, as it was still using php-fpm for some reason.
  • Added support for regional mirrors. Currently this is done by providing a CSV list of ISO 3166-1 Alpha 2 country names in the database row for a mirror. If present, that mirror will be used exclusively for the regions specified. An entry of null means it will behave normally, for countries with no exclusive mirror specified.
  • Added the ability to remove mirrors from the mirror rotation. Until now, mirrors would automatically be disabled when they are not available, but in the disabled state they would build up a queue of commands to run when they return. For mirrors that aren’t really planning on returning, this queue gets really full really fast (think purging WIP beatmaps each time the creator updates them), so rather than having to remove the mirror entries from the database, I wanted a way to stop them receiving queued events and being checked for recovery.
  • Updated the mirror-side download script to have more verbose checking on whether the files it received from Amazon S3 are complete and correct. If they aren’t, they are re-downloaded each time a user requests them until they are found to be the right size.
  • Commissioned osu!’s first EU mirror server. It’s still in a state of initialising, as it synchronises almost 1tb of data over, but this doesn’t affect its ability to serve the full EU player base. As with alll beatmap mirrors, the server specs are dual core latest-gen xeons with 64gb RAM (think file cache) and two 2tb drives running in RAID1 for boosted performance and redundancy. 500mbit traffic burstable to 1gbit.
  • Continued work on the new modular store product page which will allow for username changes to happen. As the new osu!web infrastructure is still in early stages, progress takes a while as we plan exactly how to go about implementing particular scenarios. I hope to open the new website source up to the public in the coming weeks (it was made with this in mind). Keep a watch out :).

bm6 in action

If you’re in the EU, let me know how the new mirror is treating you (your downloads should be coming from bm6.ppy.sh; let me know if otherwise)! Are you getting better speeds than before?

Next on the mirror list is a server in Oceania/South-East Asia. Something like South Korea or Singapore would work nice, I think.

comments

20150607 /weekends/

published 07 Jun 2015

I spent Saturday deciding whether I should continue daily posting on weekends. Until last year I would work through every weekend as if they didn’t exist, but due to recent changes in my daily life and work style – specifically actually working in an office with other osu! people around rough work hours – I tend to focus my energy into weekdays and give myself a bit of time off on weekends now. Not to say I get nothing done; keeping osu! alive requires constant monitoring of the servers and community. Even when I take a day off, I will spend one or two hours catching up on things, and this number can shoot up like crazy if there are any fires are put out!

I’ll probably touch back on the topic of time management and efficiency many times over the course of this blogging, since I believe it is the most important thing in life. Time is the only truly finite resource to a single human, after all.

I decided it’s best for the longevity of this posting structure that I only do one update each weekend. I will likely focus on a specific story or element of work/osu!/technology which I think may be interesting to you guys, while also sharing anything cool I get up to with my time (because why not!).

This weekend I managed to go to a club to see my favourite DJ-san (Nhato, of course), visited a local festival and spent most of Sunday at a starbucks working through email and fiddling with the username change code that I’ve been working on.

Here’s to a productive week ahead of us!

comments