Releasing code in a live environment

Running a massively multiplayer game is difficult. You need good up-time, but also need to continuously update the game and add new content. The server is busy. We’ve had over 900 million page views in our three years since launch and we currently average 800 page views a minute with over 1.2 million a day – and that’s only active pages being viewed; not including game notifications, passive chat, or any ingame popups. We’ve just done our 815th releaseto live; I imagine even our players don’t know we release something on average every 1.4 days. Some are minor bug fixes, others full UI refreshes, of which we have done 3:

We are available on many platforms, with an HTML5 website game, a Chrome Webstore App, Firefox App, a Facebook game, and Windows 8 App; all of which run on the same single sharded world, and all of which need to be kept up to date and in sync. We have to ensure with each change everything keeps working on all the different platforms, different browsers and different devices – Illyriad even works on a Kindle Touch! We run a full Microsoft stack, have changed CDN 4 times, and during these three years we’ve had 1 day of cumulative downtime, mostly for security patches, although our longest period was for completely moving hosting providers to Hivelocity for our dedicated servers and Windows Azure for our cloud based needs and CDN – lots of live data to move. All through this the players keep playing. Blissfully unaware of all the changes going on. Sending their troops on 58 million combat missions, their traders on 41 million missions and building their 33 million buildings, all the while chatting 213 million words together. Each update is transparent and doesn’t interrupt the players’ play time for patching. People live busy lives, and they have chosen to give their precious time to play the game and that must be respected. They don’t want to spend that time waiting for downloads, patching and updates. They just want to play. Maintaining a uninterrupted service, with huge concurrency and transparent updates is hard, but its worth it!