a quick update

published 26 May 2014

I have so much I want to post about, but no time to do so. Let’s continue with quick updates because it’s much better than nothing at all!

  • I’ve decided to focus on open-sourcing the whole of osu! as soon as possible. This means that my previous post mentioning needing new team members is temporarily on hold while I restructure things. Moving forward by cleaning up internals, and releasing some side-projects ahead of time.
  • 16 person multiplayer is coming in an update over the next week. Yay!
  • Tablet orders are going out at a rate of around 50-60 a day. Still plenty in stock, but I’m limiting how many are available each day due to the speed we can ship at.
  • The new store is live in all its glory! I have done my best to make it accessible on all devices down to smartphones. Tablets, stickers and plushies are available, with more products coming soon! Excite.
  • Work continues on the new site. I am focusing on the development of this myself, in an attempt to get beatmap modding into a better state ASAP.
  • The osu! site now uses an image proxy for profile user pages and forum posts. This means it is now 100% SSL traffic, which should make your browser a lot happier (no more broken lock icons in your URL bar). It also means people can’t snoop on you accessing their profiles, and should speed things up a fair bit.
  • Images which are being proxied are also being lazy-loaded. This should reduce the bandwidth you waste on the site by a whole heap. Lazy-loading is done in a way such that it shouldn’t affect your normal browsing experience.
  • I moved the complete admin team to Slack for internal communication. It has been the biggest breakthrough in ages as far as I’m concerned. Managed to get everthing integrated in once place, allowing us to handle support tickets, in-game reports, even reply to forum threads without touching the forum itself! It’s 100% linked to bancho and allows us to moderate from smartphones and receive push notifications without a cumbersome IRC client running on low-powered hardware.
  • Catch the Beat World Cup is currently running and we are streaming it each weekend on the osu!live channel. Make sure to tune in! It’s a lot of fun to watch even if you aren’t a CtB player.
  • This amusing conversation happened.
  • A teacher started using osu! in his classroom. Curious stuff.
  • I may be making a guest appearance at a large upcoming convention!

I need to make an important post about my intentions when it comes to open-sourcing osu!, so look forward to that in the coming weeks. A lot of people seem confused and worried about the implications, but you shouldn’t be. I will not fail you!

comments

a quick update

published 01 May 2014

I’m very aware that I haven’t been posting much here. It mainly boils down to the fact that as long as I have things to do that are directly contributing value to osu!, I avoid spending my time writing. I have a few posts written up in a semi-complete state, but haven’t had a chance to finalise them yet. Don’t go away; I will make things happen eventually.

So what has happened over the last four months?

  • I’ve been working on expanding the osu! team. If you think you have what it takes (highly motivated, ambitious, fast-learning) then get in touch.
  • Second batch of tablets arrived this week, and I have been busy busy writing up a web store from scratch to handle the sales. Coming this weekend at latest.
  • Moved the new osu! website to run completely on hhvm. Have been sinking a lot of time into this to ensure it is a strong framework ready to handle rapid expansion in the future.
  • Working on getting many more products to the coming osu!store, including plushies, stickers, better quality t-shirts and more!
  • Setup continuous integration for osu! client builds. I can now create new test/public builds without having a dev environment on my local PC, which is pretty amazing.
  • Moved all storage to Amazon S3. Any of osu!’s infrastructure can now be deployed within a few minutes without having to worry about locally stored content. This is pretty huge!
  • Refreshed the server which bancho runs on to allow for future expansion.
  • Further automated server monitoring and response to better handle issues that previously required manual intervention.

For those wondering about osu!stream (see my last post), I haven’t been able to post an update as the issues are still on-going and I don’t believe it is in my interest to release details until I reach some kind of resolution. Apologies to those who have been waiting to hear more.

I have been pretty busy in real-life, travelling back to Australia land to attend (and handle videography) for a wedding, renewing visas and bringing the books up-to-date to file my tax return. Things should be a bit calmer over the next six months, so expect more soon.

I’ll try and throw some update posts like this to keep you guys in the loop of what I am up to.

comments

Apple IP infringement policy woes (aka where is osu!stream?!)

published 13 Jan 2014

Update: I posted this to reddit and got some amazing advice in the comments. I highly recommend you read the discussion threads there as they cover a lot of ground.

First of all, happy new year everyone! I hope you are having a great start to 2014. It’s been a bit of a rocky start for me, and due to a number of reasons my productivity (measured in output on the visible spectrum) has been at an all-time low. While a lot of you that follow me closely will already know what I am about to convey, I write this with the hope that when I find a resolution, it will be helpful to someone out there that encounters a similar situation.

osu!stream

Let’s return to December 2012, six months after osu!stream’s July release. With the help of a small team we had released a couple of new song packs, accompanied by a 1.1 release fixing many bugs and adding a number of graphical enhancements and UI improvements. Out of the blue, I received an email from the somewhat mysterious appstorenotices@apple.com address.

Date: 12th December, 2012 (10:57)
From: App Store Notices <appstorenotices@apple.com>
To: Dean Herbert <pe@ppy.sh>
 
Dear Dean,
 
**Please include APP23992 in the subject line of any future correspondence on this matter.**
 
On 11/26/2012, we received a notice from Alek Nils that Alek Nils believes the app listed below infringes their intellectual property rights. In particular, Alek Nils believes you are infringing their copyright. Please see their comments below.
 
Developer: ppy
Provider: Dean Herbert|1357346609
App Title: osu!stream
Apple ID: 436952197
 
Comments from Complainant: Stolen Graphics, I already sent you guys 3 emails with all the links, so I guess u have those. Read through the email.  My Email add.: Alek112@hotmail.com
	-
 
You can reach Alek Nils through Alek Nils (email: Alek112@hotmail.com), copied on this email.
 
We look forward to receiving written assurance that your application does not infringe Alek Nils's rights, or that the parties are taking steps to promptly resolve the matter.  Please keep us apprised of your progress.
 
Should you choose to remove your application, (for example, while you make any necessary changes) use the steps provided below.

Having created the majority of the game graphics myself, I was astounded at what I was reading. The particular graphic, while not stated specifically in this initial communication is the “spinner circle”.

spinner circle

The accusation made by the complainant was quite preposterous on many levels, so let’s briefly address that and get it out of the way.

  • The design is based off the Xi-Style skin for osu! PC. This was made by the skilled XiaoUnlimited (now Xiao) in 2008, and can be seen here. Before using this design in osu!stream, I received permission from Xiao (and he can be seen listed in the game credits since day one). While I do not have my original correspondence with Xiao, I did contact him recently to get a statement showing his approval.
  • While I did borrow Xiao’s spiral motif, the centre and outer circle portions are straight from the osu! default skin, which was 100% my own work.
  • The design cited as being “stolen” has since been recreated from scratch twice, in order to support devices with higher resolutions. The latest version was made in POV-Ray, and source is available here, courtesy of mm201.
  • The complainant “Alek Nils” – actually Alek Nilsen – is a third party who created a number of “compilation” skins using other skinners’ graphics – often without their permission. He is claiming to have created a graphic that he in no way has any rights to.

Hopefully by now you can agree that the claim is fraudulent, and we can move on.

Initially I followed up as per Apple’s loose procedure to try and reach a resolution, but it was very obvious that the complainant’s idea of a ‘resolution’ was nothing short of having the app being removed from the App Store. The conversation continued for months back and forth with the AppStoreNotices department as I attempted to explain this to them. At no point did they offer any alternate solutions or advice on how to resolve the dispute. They also took around one month for each reply, to a degree of accuracy that suggested it was a predetermined period for these kind of follow-ups.

Let me add that we did at multiple points try and resolve the issue directly with Alek. A full transcript of the email conversation can be found here.

During this time, the complainant – obviously unhappy with Apple doing nothing to inflict harm on me – took to spamming my blog and email accounts, launching denial-of-service attacks on osu! and other malicious behaviour. This was a significant time sink, but as per my usual strategy I blocked and ignored this as much as possible. I also avoided making any further direct contact with the complainant.

Travel forward to the 4th of January 2014, when Apple sent me a sudden notice that they are removing osu!stream from the App Store. Again, they offer no method of resolution and tell me to talk with the complainant directly.

Date: 4th January, 2014 (04:56)
From: App Store Notices <appstorenotices@apple.com>
To: pe@ppy.sh
 
Dear Dean,
 
We regret that the dispute regarding the app listed below could not be resolved amicably between the parties.  We have removed your application from the App Store.  For any questions relating to this matter, please contact Alek Nils directly.  For any technical questions, please contact iTunes Connect: www.apple.com/itunes/go/itunesconnect/contactus.
 
Developer: ppy
Provider: Dean Herbert|1357346609
App Title: osu!stream
Apple ID: 436952197
-
 
Sincerely,
Victoria

removed

I could probably end this post here and leave you in absolute shock at the appalling manner in which these cases are handled, but let’s persevere: what could I have done differently to resolve this issue without the app being removed?

Probably not much (but I’m open to suggestions).

  • As per the last two years, I should have continued to respond to Apple. I was very busy during the period when I got the escalation email – and on top of that it was from a no-reply address – so I did not send my usual under-five-minute response to Apple. They may have assumed I was ignoring their email as a result (as silly as this sounds).
  • I should have turned to the legal system sooner. I naively believed that due to the lack of evidence supporting this claim, it would never result in anything like this. In reflection, this assumption was a bad one to make.

The current things I have done to try and resolve this issue, with no progress as of yet:

  • Replied several times to AppStoreNotices via email, giving a full history and evidence of the fraudulent nature of the claim.
  • Contacted Apple Developer Support via phone multiple times. Was told they can only contact AppStoreNotices by email via the same process I take, and therefore would not be of any help.
  • Contacted iTunes Connect support. See above.
  • Asked friends who have contacts high up in Apple. They are powerless across country/department borders and therefore cannot help.
  • Sent a plea to Tim Cook directly, in a hope he will correct the wrong which has been done.
  • Submitted a new build to the App Store stating the issue was resolved. Is stuck in perpetual “In Review” status.
  • Contacted a lawyer. Told that this should be happening under DMCA law but isn’t. Very hard to follow up as Apple gives no detail as to their terms or process. I read through all the documents available on Apple’s site but none mention the full process for IP infringements. The only reference anywhere is the submission form for these complaints. Please note that App Store complaints differ from all standard copyright issues, which are filed under DMCA (why?!).

I will update this post as I work towards a resolution. I am open to any advice; please leave a comment or if you wish, contact me privately at pe@ppy.sh. Thanks to everyone that has already shown their support in many ways.

Finally, an apology to those which are unable to download osu!stream, restore their purchased songs, purchase new song packs, or restore the app itself from a backup or on to a new device. I will try my best to get things straight again, and hopefully have a few updates to the app itself which will make up for lost time.

comments

I Have A Few Posts Waiting For Some Final Touches

published 28 Dec 2013

I have a few posts waiting for some final touches, but as I am likely to be busy going into the new year with osu! updates, osu!tablet shipping and a bit of travel, I hope you can enjoy this in the mean time!

comments

Stopping a DDoS

published 19 Sep 2013

Anyone who is a regular osu! player will be well aware of the troubles I have been experiencing over the last few months keeping the servers online. Daily DDoS attacks have meant constant interruptions across the board, but with the majority focused on Bancho – the server-side component of osu! responsible for multiplayer, chat, user presence and providing osu! with up-to-date player stats and ranking details. I thought it would be interesting (and hopefully beneficial to someone in the future) to write up my experience of combatting such an attack to the best of my ability.

First, let’s go into the knowns of the attack.

If you are not familiar with what a DDoS attack is, I highly suggest reading up on them before continuing to read this post. I am expecting a lot of questions asking why I can’t just “block the IPs” of the attacker or similar, and while I will try and answer that, you will probably get better answers reading wikipedia due to the sheer scope of what can be involved in such an attack.

The attacker

DDoS attacks are regularly used as a way of bringing attention to a specific issue, group, or single user’s demands. They force a service to become aware of whatever the attackers want, holding it at random to an extent. For this reason, it is often the case that the user or group committing the attack will publicly take responsibility for it, and provide proof that it is indeed them performing it, in order to gain recognition.

In this case, even while going out of my way to find out who was responsible for the attacks, it was very unclear until late into the picture. Via relayed IM logs, I was eventually able to get an idea of who was responsible, and what they wanted from us. As I expected, a user had been banned for cheating – in this case over a period of many years – and wanted all of their accounts unbanned. This is obviously a demand which I would never agree on.

The power source

A single user is usually not capable of launching an attack that would take down a service like osu! without another party managing the botnet and/or servers responsible for providing the bandwidth to launch the attack. In this case, the attacker was making use of multiple publicly available “stresser” or “booter” services, which provide a web interface in front of the infrastructure required to launch attacks. This allows an attack to be launched by simply entering a target IP address, port, attack type, length and hitting the “GO” button.

a stresser

These services usually charge between $3-20 an hour depending on their reliability and strength. They sit under the legal veil of being “stress testers” which are made to be used on servers you own to test how they will stand against an attack. They usually contain no contact information and are very clearly geared towards users with different intentions.

pricing

It is safe to say from periods of analysis (where a small subset of the data is logged and parsed during an attack) that there were both large spanning botnets and a few high-powered servers involved. Whether these were compromised servers, or servers rented by the “stresser” services themselves, they were capable of reaching attack velocities up to – and in a few cases exceeding – 10Gbit/s. This is a sizeable force to deal with.

polish

The target (osu!)

osu! is run from a number of diverse locations around the world, with database slaves and download mirrors distributed for performance and redundancy. The core servers are all rented at Softlayer’s SJC datacentre. After years of searching for a datacentre which manages to just do-it-right, I ended up with Softlayer, and I have been impressed with their reliability and support 95% of the time. The pricing is above what you would pay elsewhere, but they offer benefits such as private networking, portable IP addresses and free PPTP VPN access which others do not provide.

The osu! website has been sitting behind CloudFlare for over six months now. I was initially skeptic about using a service like CloudFlare, as it is adding an extra unknown between your service and the internet which you have very little control over – if something was to go wrong at their end, I have no power to fix it. While in the last six months this has happened occasionally, the overall result of switching CloudFlare on has been very, VERY positive. I would love to go into the specifics of this in another article.

CloudFlare can handle DDoS attacks. They can handle, mitigate and cut off the source at a level datacentres may not be able to do. They have a knowledge of how attacks happen and how they can be stopped with minimal consequence and downtime. During the period of attacks on osu!, the website did not flinch once. The attackers either knew they had no chance of messing with CloudFlare, or tried and failed to cause any harm. Unfortunately for us, Bancho is a completely TCP-based protocol, running over port 13381 with a custom protocol I engineered specifically for osu!. As CloudFlare only handle HTTP traffic, putting Bancho behind CloudFlare was simply not an option.

Bancho, sitting in Softlayer’s datacentre, is guaranteed a certain level of protection that is offered with all servers, in the form of a Cisco Guard firewall. While renting these devices permanently is outside of my limited budget, Softlayer are kind enough to dynamically reroute all traffic through one should they detect an incoming attack. Once this occurs, the firewall will intercept and filter traffic, delivering a clean stream of data to the end server for 24 hours, after which the conditions are re-assessed and the device is usually removed. Should attacks keep up, Softlayer also reserve the right to null route you server’s IP, deeming it useless for 24 hours (with no traffic ever reaching it). Cisco Guard and null routing is done on a per-IP basis, which allows a bit of flexibility should multiple IPs be assigned to a single server. This turned out to be very useful during the initial stages of the attack.

Other osu! services – such as download mirrors – were not heavily targeted by these attacks. Even if they were, it is minimum-impact and easy to re-route to another location. There are also several mirror run by other kind people which provide downloads should the official mirror go down.

Testing the waters

Long before the recent wave of attacks even started, there were occasional DDoS attacks detected against osu!. At the time, I assumed these to be random – sometimes people can be looking to test their botnet out, or hitting an IP which used to belong to another service. It is easy to see the these were no mistake, and in hindsight were the prologue to the main period of attack. The first of such attacks was on May 19th.

The force to deal with

Starting around July 5th, I began noticing an increase in the number of incoming attacks. This is most easily seen on a graph of incoming traffic to the server running bancho:

incoming traffic to bancho

Note that each of these spikes was usually a series of independent attacks on that day, and that while the maximum traffic shown on this graph is 1Gbit, they regularly exceeded this, but Cisco Guard kicks in at this point so it is not visible here.

Course of action

Most datacentres are not fit for dealing with DDoS attacks. 99% of them will resort to null routes as a solution to clients under attack. Softlayer offers firewalls which have DDoS protection, but due to the size of the attacks, even with such protection added the IP endpoints would likely be null-routed to protect the larger network, and reduce the effect on other clients sharing the same routing infrastructure.

There are services which offer DDoS mitigation, by placing a “proxy” between your server and the internet and eating the DDoS traffic. The price of these range from $50 to upwards of $10,000 a month. For the level of cover required by osu!, we are looking at the expensive end of the scale. For what it’s worth, I did try – and am still using for bancho’s IRC gateway – Staminus, which offers cheaper options that null-route on a very fine scale, making recovery fast after the attacks stop, for a relatively affordable price.

It was time for thinking outside the box. We need a solution which will not only stop these attacks, but prevent them from happening again in the future.

I have long wanted to add UDP support to bancho, allowing for faster round-trip times and lower overheads when establishing connections, but in this case UDP would not help. Instead, let’s consider adding HTTP support. Why HTTP? Because CloudFlare!

Firstly, if you are following closely you are thinking one of two things right now:

  1. So you’re completely re-hauling osu! to use a REST approach?
  2. What? But bancho is a streaming protocol! You’re crazy! You’re doing it wrong!

While I would love to try the first option – and who knows, maybe eventually this will happen – I was looking for a quick solution, which could be implemented in a few days maximum. Rewriting from scratch with a fresh protocol and architecture in this amount of time is just not feasible. So let’s move to the crazy option. Piping a streaming protocol over HTTP.

It may not be as crazy as it sounds. These days HTTP widely supports keep-alive, which means a single TCP connection can be used to transport multiple requests. This reduces the connection establishment time drastically. All that remains is the overhead that comes with HTTP headers, which can be reduced by not including any headers which would not be used by bancho. Including the bare minimum headers is still necessary, such as the HTTP protocol version and transfer type.

Even so, in order to establish the responsiveness of a streaming connection over a non-streaming protocol would require quite frequent sends – hopefully at least once a second. We can consider two cases here: one where the client requests something of the server and expects a response, and a second where the server has a waiting command/request of the client. The majority of osu! requests are initiated from the client-side, so we can optimise with the first case in mind.

Case 1: client has a request of the server

In this case, we can treat the request basically as a REST request. Assuming there is no existing request to the server, we can instantly send a new HTTP request, and wait for on the response. As we are encapsulating a stream here, we don’t want to send a request if there’s an outstanding request, as this could cause weirdness to all ends.

Case 2: server has a response waiting for the client

If the response is already being waited on by the client, there is likely already an open HTTP connection. If not, we resort to polling from the client. Depending on the current state of the client, polling will occur every 1-20 seconds. If the user hasn’t moved their mouse in a while, or the osu! window is inactive, the polling interval will scale back over time, resulting in less unnecessary traffic. When active, the perceivable latency added by polling is next to zero, as the previous poll is kept waiting on a response to the extent of the polling interval (within reasonable limitations). This means we always have one HTTP request open waiting on a response.

Before starting on this approach, I contacted CloudFlare stating what I wanted to do, and their thoughts on the matter. I was particularly concerned about the number of requests this would cause, and also whether they permitted this kind of usage of their service. Their response was to make sure that connections were not kept open for long periods, and to upgrade my plan to a higher tier (Business / Enterprise) to account for the load. I was already aware of their policies on long-standing connections, so planned for this from the start. Upgrading my account was the least of my concerns, and still cheaper than any other DDoS mitigation option of this scale. We are good to go!

Path to recovery

Launching was fairly painless and went without any hitches. I was thoroughly impressed with the deployment, and how smoothly everything went.

a lot of requests

CloudFlare handled the new load like a boss, and the attacks stopped. After some quick iterations optimising the poll timings, bandwidth usage was lower than TCP connections used to be, which surprised me, but was strangely comforting at the same time.

The resolution of this issue was so sudden that there really isn’t much more to say. CloudFlare are a power to be reckoned with, and are doing a great part in protecting the internet1.

Final thoughts

It is now a month after deploying this solution. Things are still running smoothly, and we haven’t “seen” an attack since. Note that this doesn’t mean there haven’t been attacks. One downside of being behind CloudFlare is unless they report a fault on their twitter/status page, you are totally unaware of what is going on on the other side. There have been very short periods of reduced traffic, and debugging these cases is quite frustrating when you are unable to see exactly what is or is not being blocked from hitting your servers. Based on the infrequency of this happening, I choose to have some faith for now.

As I said previously, adding a service like CloudFlare in front of web servers is adding another unknown. It is therefore important to know that routing is reliable and support is there. CloudFlare have not disappointed on either of these, offering support turn-around times of <30 minutes in most every case, with a knowledgeable engineer rather than some outsourced level 1 tech guy (more than I can say for most datacentres out there). Their routing is amazing, nothing more to say there.

While I’ve tried to go through everything in great detail, there is a whole lot more to this story, my implementation of the new bancho protocol, and what I have learnt over the last couple of months. If you want to know more about any specific facet, please leave a comment!

  1. Some of the services used to launch these DDoS attacks from are sitting behind CloudFlare, protecting themselves. I reported these to CloudFlare but it seems as though they will not act against a service unless they match very specific criteria. You’d think they would be against sites used to attack CloudFlare itself, but hey, who am I to decide that ^^; 

comments