20150608 /eu/

published 08 Jun 2015

I spent the morning catching up on emails I put off over the weekend. This included accounting follow-ups, payment system integration follow-ups and the usual catch-up on cheaters and foul-play. You don’t even want to know how much time goes into the last of those!

Today’s Worklog:

  • Brought bm4 and bm5 (the current two active mirrors) up-to-date with Debian 8 (jessie). Also took the opportunity to switch bm4 over to hhvm, as it was still using php-fpm for some reason.
  • Added support for regional mirrors. Currently this is done by providing a CSV list of ISO 3166-1 Alpha 2 country names in the database row for a mirror. If present, that mirror will be used exclusively for the regions specified. An entry of null means it will behave normally, for countries with no exclusive mirror specified.
  • Added the ability to remove mirrors from the mirror rotation. Until now, mirrors would automatically be disabled when they are not available, but in the disabled state they would build up a queue of commands to run when they return. For mirrors that aren’t really planning on returning, this queue gets really full really fast (think purging WIP beatmaps each time the creator updates them), so rather than having to remove the mirror entries from the database, I wanted a way to stop them receiving queued events and being checked for recovery.
  • Updated the mirror-side download script to have more verbose checking on whether the files it received from Amazon S3 are complete and correct. If they aren’t, they are re-downloaded each time a user requests them until they are found to be the right size.
  • Commissioned osu!’s first EU mirror server. It’s still in a state of initialising, as it synchronises almost 1tb of data over, but this doesn’t affect its ability to serve the full EU player base. As with alll beatmap mirrors, the server specs are dual core latest-gen xeons with 64gb RAM (think file cache) and two 2tb drives running in RAID1 for boosted performance and redundancy. 500mbit traffic burstable to 1gbit.
  • Continued work on the new modular store product page which will allow for username changes to happen. As the new osu!web infrastructure is still in early stages, progress takes a while as we plan exactly how to go about implementing particular scenarios. I hope to open the new website source up to the public in the coming weeks (it was made with this in mind). Keep a watch out :).

bm6 in action

If you’re in the EU, let me know how the new mirror is treating you (your downloads should be coming from bm6.ppy.sh; let me know if otherwise)! Are you getting better speeds than before?

Next on the mirror list is a server in Oceania/South-East Asia. Something like South Korea or Singapore would work nice, I think.

comments

20150607 /weekends/

published 07 Jun 2015

I spent Saturday deciding whether I should continue daily posting on weekends. Until last year I would work through every weekend as if they didn’t exist, but due to recent changes in my daily life and work style – specifically actually working in an office with other osu! people around rough work hours – I tend to focus my energy into weekdays and give myself a bit of time off on weekends now. Not to say I get nothing done; keeping osu! alive requires constant monitoring of the servers and community. Even when I take a day off, I will spend one or two hours catching up on things, and this number can shoot up like crazy if there are any fires are put out!

I’ll probably touch back on the topic of time management and efficiency many times over the course of this blogging, since I believe it is the most important thing in life. Time is the only truly finite resource to a single human, after all.

I decided it’s best for the longevity of this posting structure that I only do one update each weekend. I will likely focus on a specific story or element of work/osu!/technology which I think may be interesting to you guys, while also sharing anything cool I get up to with my time (because why not!).

This weekend I managed to go to a club to see my favourite DJ-san (Nhato, of course), visited a local festival and spent most of Sunday at a starbucks working through email and fiddling with the username change code that I’ve been working on.

Here’s to a productive week ahead of us!

comments

20150605 /progress/

published 05 Jun 2015

I’m writing this for the second time, since it somehow disappeared the first time :(. That’ll teach me for trusting some chrome based markdown editor. stubbornly returns to sublime text 3

In recent times I look at the user-facing changes and and often disappointed in what I see. My own role in osu! has changed from being able to rapidly implement new features to only just being able to keep up with what is going on each day. The rare occasions I’m able to implement something new are therefore very enjoyable as a result, but I digress.

The ongoing tasks which I choose to complete myself rather than delegate remain that way for a reason. I believe each and every one of them requires my own attention. Delegating tasks is great and all, but there are two issues with it:

  • Delegating things that come up infrequently is often less efficient. An example would be emails about conventions, which are always different in nature. Sometimes people will want merchandise, sometimes they will want to promote osu! by presenting a panel. Teaching someone how to respond to these emails would require writing a manual of every possible scenario, or being able to trust their judgement, which leads on to the second issue..
  • I find it hard to trust people. Not trust as in confidentiality, but trust as in the ability to diligently handle a scenario in the same way I would myself. I am pretty sure I am yet to meet someone with the same values and level of attention-to-detail that I have. While this may sound like I’m bragging, if you ever run a company of your own you will understand this feeling.

Today was one of those days I spent catching up on mostly mundane run-of-the-mill tasks. You’re still interested? Okay then!

Today’s Worklog:

  • Fixed a loophole in the account recovery process that was abused overnight. Accounts can now be set into a state where users cannot initiate password recovery and must contact support.
  • Purged my inbox backlog down from 188 to 55. I read emails as the come in and when necessary reply within a minute or two, but less important emails are snoozed until a later date. I find it easier to process these in one large batch rather than as-they-come. This is what my inbox looks like now. Might take a few beers tomorrow to get through the remainder ;).
  • Brought company accounts up-to-date. Basically distributing money into the correct accounts so people are paid and servers are kept online and purring. I have an accountant to handle things at the end of the year, but daily financials are still my own responsibility. Need to attend to them once a week else they pile up quickly.
  • Got acceptance email for AWS Aurora preview. Not sure if I will use this going forward, but couldn’t resist the opportunity to test it for free! Spun up two beefy 16 core 108gb instances which I will be deploying the full osu! database to for performance testing over the coming weeks.
  • Figured out osu!keyboard stock issues. We needed to reserve a few keybaords for Taiko World Cup winners, but due a number of faulty units being found during initial quality checking, we ran a bit short.
  • Approved and finalised payment for the next osu!tablet production run, this time for 2,000 units! We still have stock from the last run too. I think I have finally got my head around the convoluted processes involved in international manufacturing and shipping, so stock shortages should be a thing of the past.
  • Our knowledge base is now fully stocked with content, hopefully helping users before they unnecessarily post to the forum!

So let’s briefly revisit something that happened last week: OVH had a downtime which lasted several hours as the result of a cable being cut. This meant the sudden loss of all our beatmap download mirror traffic! As I was “enjoying” an Owl City concert at the time, I had @nekodex - who you may also know due to his musical contributions to osu! - help me get a new mirror up and running.

osu! beatmap mirrors are deployed as a single file php script with no external dependencies. They use an s3 endpoint to retrieve maps, cache them locally and serve them to users using a secure checksum system. The script is not only self-containing, but updates can be pushed out in an automated process. Each mirror is monitored by the central osu! web server and mirrors which are unavailable or underperforming are automatically removed from the available list.

We had things back up and running in under 20 minutes as a result, but when things like this go haywire it is always wise to add countermeasures to ensure the same thing doesn’t happen again. I have started investigating where I will be adding some additional mirrors in order to add geographical redundancy to the system. As I don’t want to introduce any more complexity to the accounting side of the infrastructure setup of osu! (already across four datacenters!!) they will likely be placed in OVH’s European datacenters.

Going forward, I would love to open the mirroring system up to the public again, letting people lend their spare server bandwidth to the good of the osu! community. This would require some logic to ensure users were linked up with the closest mirror to their physical location, of course. Think a mini-CDN just for osu!. Alternatively maybe I should activate the dormant p2p code that has been sitting inside osu! for several years… ^^;

Have a good weekend. Play lots of osu!.

comments

20150604 /vocal/

published 04 Jun 2015

Many people say that it’s the vocal minorities that express the most negativity, and while this is definitely the case, I still aim for the unattainable 100% satisfaction. I hold myself to high standards that I have learned over the years I cannot expect from anyone else. This is actually one reason I am aiming to move osu! to an open-source model, but that story is for another day!

I go out of my way to answer every single email that comes to my personal address. The answers may be brief, but I take in every single word of feedback that users have to offer. I also check the forums, reddit, BBS and basically anywhere people are discussing osu!.

While some may say this is a complete waste of time and focus - and this may be true - I feel it’s something I cannot avoid if I want to keep up-to-date with the temperature of the community. Unique things happen almost every day, quite often which need multiple team members’ input to resolve, so it’s important to stay on the watch for ongoing and upcoming events.

Remember that osu! is actually a very simple game at its core. The gameplay has rarely changed over the years, while the ecosystem around that gameplay is constantly developing. The drive for more accessibility, more stats to track, better ways to communicate with friends is strong. The social and community aspects of osu! really make it feel to me less game, more web service.

Not to say we don’t have stuff in the pipeline to spice up the gameplay too ;).

Today’s Changes:

  • The forum search system now defaults to matching all keywords, rather than any. The sorting of results returned by the back-end (elasticsearch, for those playing along at home) has also been changed to relevance rather than freshness. Sorting from newest to oldest is still applied at the forum front-end as a last step. This should hopefully provide for much more relevant results. Once the new site is up, search will be overhauled and provided in a similar way to how you use the facebook search box. For now, let’s make do!
  • The help forum now shows only the “Chat with Support” button when someone is available. There are way too many posts in the help forum that could be answered in a single line (profile being in wrong mode, pp not updating etc.). While this is a band-aid measure while we get a full knowledge base system ready, it should help.
  • Began work on the osu!web (2.0) site for the first time in a long while. I have made a promise with myself to not add any new features to the old web, so the new username changing system is going to have to be done here.
  • Spent a good portion of last night getting my dev environment set up correctly for osu!web development. The new site is being written completely from scratch in Laravel 5. Learning a new framework as I go, already I already wrote the web store in it quite some months back.
  • In order to support username changes, which are going to be presented as an item on the store, we first need support for store items with infinite stock and custom display implementations. I made good progress on building the framework for this.

For those freaking out about the username change being a store items, here’s how it’ll work:

  • First Change: free (with supporter)
  • Second Change: $8 (equiv. 2 months supporter)
  • Third Change: $16 (equiv. 4 months supporter)
  • Fourth Change: $32 (equiv. 8 months supporter)

I’m sure you can see the pattern here. It will continue to double, likely with a cap at around $100. This may seem like an expensive purchase, but the idea here is to stop people from recklessly changing their name too many times. We discussed the cost amongst team members and decided doubling each time was the best approach to achieve this.

As for history, you will be able to see all previous usernames that a user was known as in their profile, as you’d expect.

early design mock-up

comments

20150603 /transparency/

published 03 Jun 2015

In 2007 when I first started publicly releasing osu!, I would release daily changelog updates and daily builds, with hugely visible changes and massive new features. Fast-forward to today, where finding the time and (more importantly) motivation to write a blog post is near impossible.

It’s not that I don’t want to. I just feel my time is beter spent elsewhere. Writing for me is something I enjoy, but it’s also a distraction from the never-ending list of tasks I have floating around my brain. I end up prioritising what to me feels most beneficial to the osu! user base.

But it’s time to take a step back.

In the recent months, there seems to be a group of users trying to destroy the osu! team’s image. They make it seem like we ignore support requests, break the game and are not interested in the community at large. While I could probably address this by meeting these people on their level (reddit seems to be the central campfire for bitching these days), I would rather stay away from such negativity.

At the same time, I want to give myself and the rest of the osu!team a stronger voice. I want to assure the user-base of osu! that we are making forward progress, addressing issues as they arise and in genearl doing a lot behind the scenes to keep things running smoothly that is (intentionally) hidden from players.

I’m going to start updating this blog with shorter posts focusing mostly on what I and the rest of the osu! team are up to on a day-to-day basis. Setting aside around 20 minutes a day to get something posted - even if it is very brief.

  • Updated username changing code to only free up usernames after all other checks have been completed (diallowed usernames etc.).
  • Installed new SSL certificate for *.ppy.sh (old one was expiring in 2 months).
  • Fixed “recent activity” not updating correctly for score events. This broke in a rewrite of the score submission code I completed yesterday.
  • FInished removing all usage of mdb2 in favour of mysqli.
  • Finished integration of a new payment system “¥Coins” which makes it much easier for Japanese users to support osu!.
  • Added a new column to all score tables to store whether a replay is available or not. SHould make things a bit more straightforward than the current system of storing this information in a separate table. Is a step forward to sending this out to the client to allow for more than top50 replays. Adding a single tinyint to the scores tables took a total of 9 hours! For people interested in how I’m doing it with no downtime, this is the answer.
  • Fixed beatmap playcounts not being updated anywhere near as often as they should have. This was actually a really silly one, and also means that our overall playcount (currently sitting at 3 billion) is heavily deflated. Here I was thinking we didn’t grow much recently as the time it took from 1 to 2 billion was the same as from 2 to 3, when in fact it was likely half or less.
  • Continued investigation into ongoing mysql performance issues (I’ll wall-of-text about this after I solve it), this time focusing on network IO. Added some extra config settings to bancho to allow dynamic disabling of the most network-intensive queries. Didn’t help.
  • Created a new monitoring dashboard for real-time monitoring. My default one provides a great overview of the last day, but I needed something I could have up on my second monitor while performing system maintenance or applying what could be breaking-changes.
  • Corrected some issues with username change history storage. This is something I recently added to chain into the ability to change usernames more than once. Coming soon!

Along with these kinds of bulleted lists, I will try to gradually expose more details on things going on behind the scenes that I believe you guys may find interesting. Any suggestions or questions are welcome in the comments section. I will either reply in to your comments or address them in the next day’s post.

Oh, and for those of you that enjoyed my osu!weekly posts focusing on news around the community, make sure you are following along with the osu!weekly news post series. This weekly posting by Tasha has basically replaced what I was trying to do previously.

comments