WebGL Experiments: Texture Compression

Lilli Thompson from Google asked us how we were doing texture decompression in the pixel shaders and what algorithm we were using. We thought we would share our answer…

Texture compression was a bit of journey – as no one at Illyriad had ever implemented anything in 3d before; to us texture compression was mostly a tick box on a graphics card.

It started when we found out our 90MB of jpegs expanded to 2GB of on-board memory and we were worried we’d made a terrible mistake, as this was certainly beyond the possibilities of low-end hardware! Half of it this was due to Three.js keeping a reference to the image and the image also being copied to the GPU process – so essentially the required texture memory doubled.

Dropping the reference Three.js held after the texture was bound to WebGL resolved this. I’m not sure how this will play out with context lost events – as I assume we will have lost the texture at that point – but local caching in the file system and reloading may help with recreating them at speed.

With 1GB of memory remaining we were faced with three choices – either deciding what were were trying to do wasn’t possible; reducing the texture sizes and losing fidelity or trying to implement texture de-compression in the shader. Naturally we opted for the last.

We were originally planning to use 24bit S3TC/DX1; however this proved overly complex in the time we had available as the pixel shaders have no integer masking or bitshifts and everything needs to be worked in floats. The wonders we could unleash with binary operators and type casting (not conversion) – but I digress…

In the end we compromised on 256 colour pallettized textures (using AMD’s The Compressonator to generate P8 .DDS textures). This reduced the texture to one byte per pixel – not as small or high colour as DX1 – but already 4 times smaller than our original uncompressed RGBA textures.

It took a while to divine the file format; which we load via XMLHttpRequest into an arraybuffer. The files have 128 bytes of header which we ignore, followed by the 256×4 byte palette which we load into a lookup table texture RGBA. The rest we load into a Luminance texture. Both textures need to use NearestFilter sampling and not use mipmapping to be interpreted sensibly.

We have created our own compressed texture loaders – the colour texture loader looks a little like this:

[code lang=”javascript”]
Illyriad.TextureCompColorLoader = function (path, width, height, uniforms) {
var texture = new THREE.DataTexture(0, 1, 1, THREE.LuminanceFormat,
(new THREE.UVMapping()), THREE.RepeatWrapping, THREE.RepeatWrapping,
THREE.NearestFilter, THREE.NearestFilter);

var request = new XMLHttpRequest();
request.open("GET", path, true);
request.responseType = "arraybuffer";

// Decode asynchronously
request.onload = function () {
if (request.status == 200) {
var imageDataLength = request.response.byteLength – width * height;
uniforms.tColorLUT.texture = new THREE.DataTexture(
new Uint8Array(request.response, 128, 256 * 4),
256, 1, THREE.RGBAFormat, (new THREE.UVMapping()),
THREE.ClampToEdgeWrapping, THREE.ClampToEdgeWrapping,
THREE.NearestFilter, THREE.NearestFilter);
uniforms.tColorLUT.texture.needsUpdate = true;
texture.image = { data: new Uint8Array(request.response, imageDataLength),
width: width, height: height };
texture.needsUpdate = true;
}
}
request.send();
return texture;
}
[/code]

When we first did the decompression in the pixel shader, it was very blocky as we had turned off filtering to read the correct values from the texture. To get around this we had to add our own bilinearSample function to do the blending for us. In this function it uses the diffuse texture with the colour look up table and using the texture size and texture pixel interval samples the surrounding pixels. The other gotcha is that the lookup texture is in BGRA format so the colours need to be swizzeled. This makes that portion of the shader look like this:

[code]
uniform sampler2D tDiffuse;
uniform sampler2D tColorLUT;

uniform float uTextInterval;
uniform float uTextSize;

vec3 bilinearSample(vec2 uv, sampler2D indexT, sampler2D LUT)
{
vec2 tlLUT = texture2D(indexT, uv ).xx;
vec2 trLUT = texture2D(indexT, uv + vec2(uTextInterval, 0)).xx ;
vec2 blLUT = texture2D(indexT, uv + vec2(0, uTextInterval)).xx;
vec2 brLUT = texture2D(indexT, uv + vec2(uTextInterval , uTextInterval)).xx;

vec2 f = fract( uv.xy * uTextSize );
vec4 tl = texture2D(LUT, tlLUT).zyxw;
vec4 tr = texture2D(LUT, trLUT).zyxw;
vec4 bl = texture2D(LUT, blLUT).zyxw;
vec4 br = texture2D(LUT, brLUT).zyxw;
vec4 tA = mix( tl, tr, f.x );
vec4 tB = mix( bl, br, f.x );
return mix( tA, tB, f.y ).xyz;
}

void main()
{
vec4 colour = vec4(bilinearSample(vUv,tDiffuse,tColorLUT),1.0);

[/code]

This performs fairly well; and certainly better than when your computer feels some virtual memory is required because you are using too much! However, I’m sure on-board graphics card decompression should be swifter and hopefully open up the more complex S3TC/DX1-5 compression formats.

There is a major downside however with decompressing this way in the pixel shader. You have to turn off mipmapping! Not only does turning off mipmapping cause a performance hit as you always have to read the full-size textures – but more importantly it doesn’t look good. In fact in the demo – we had to use full-size textures for the grass so we could apply mipmapping as otherwise in the distance it was a wall of static!

Unfortunately, as far as I’m aware, WebGL while you can create mipmaps with generateMipmap – you can’t supply your own. Again, real compressed textures should help here.

EDIT: Benoit Jacob has pointed out this is possible by passing a non-zero ‘level’ parameter to texImage2D – one to look into.

Some caveats on the demo:

  • Obviously even 90MB of jpeg textures is far too much – the production version will be substantially smaller, as we are being a bit smarter on how we will be using them.
  • This has been a learning process both for us and Quantic Arts (who are used to boxed set games).
  • This was a tester to see the upper limits of what we can do in WebGL, so we haven’t been focusing on optimization yet.
  • We will be reworking the obj models to reduce their download size substantially.
  • The way the game works is that no one player will need all the textures at once (the time between queuing a building and it’s actual appearance in the game allows us to download the models/texture)

So the actual game requirements will be much much lower.

Sound: Enhancing the Illyriad experience with audio

In Illyriad we will be using audio in three ways:

1. Motif background music

We will be mixing these in and out of the game environment dependent upon the map focus on biome. Some examples could be Mediterranean music, jungle music, desert music or polar music. We also plan to have some generic musical motifs and themes that enhance the game experience, such as triumphal themes that linger after you win a battle or trepidation themes that build anticipation before a diplomatic event such as sabotage or assassination. We are generally writing specific music for all of these themes as opposed to licencing them. We are also exploring the Web Audio API for this purpose – almost treating it as a sequencer – so that we can slave a standardized or fractional BPM to the game server clock and use all the clever tricks. For example, we may increase tempo of the music with the player’s proximity to the event outcome.

Weird, I know, but during the midlife crisis that impelled me to start Illyriad, I also released a reasonably successful album of ambient and dance music.  So I’m pretty familiar with creating soundscapes and – whilst I haven’t yet used it in anger – the toolset I see in the Web Audio API looks pretty comprehensive. I would bet my bottom dollar that someone will build (if they haven’t already) a full Web Audio API sequencer, and I can also see (the equivalent of) VST plugins and softsynths being sold on the Chrome webstore.

2. Notification/event audio

These are the standardized sounds that occur during gameplay based on triggers such as event queuing or occurrences. For example, starting building construction in game with sound of carpenters sawing or the messenger noise you get when you receive an email. We’re looking at making some of these specific to the game races, so the new mail sound for receiving an in game message is different for Elves and Orcs. Some of these are being worked on currently by James Bell, our audio engineer. He is out in the field recording the sounds at historic battle re-enactment events around the UK so that we can really get it right!

We’re also looking to mix these on the fly in the browser. A battle notification audio event would be made up of many different audio snippets in the mix. For example, a player who attacks a pack of lions, using his cavalry, in a savannah environment that wins the battle will get a very different sound effect notification from a player who attacks a horde of skeleton warriors, using his bowmen, in the arctic north that loses the battle.

3. Environmental audio

Different from the motif music, these are specifically audio snippets (some looped, some one-off or randomly triggered) that are geo-specific, especially within the WebGL 3D environment. If you go up close to the grinding cog-wheels of a working flourmill or approach the river bank, the audio source is located (in the stereo and depth pan) in reference to your viewing position and that of the sound source with volume controlled by proximity.

Some of these effects are environment proximity-related and others are time-slaved (such as the dawn chorus of birds or the frogs and crickets of dusk) or environment-slaved (for example, wind, rain or thunder) as the game world progresses and changes.

WebGL Experiments: Illyriad’s 3d Town

What can WebGL do? Can it do what we want? We were wondering.. and so decided to put it to the test…

To test the upper bounds of WebGL we put together a rough and ready demo [caution – it’s bandwidth hungry].  It’s very rough, not optimized and currently only runs on Chrome [working in all browsers that support WebGL is our priority]; but that’s kind of the point – its a technology tester to ensure we weren’t making a mistake.

The results speak of themselves – it definitely proves itself!  Sure it needs a bit more polish, but we are now confident that the actual in-game libraries we are building have a lot of head room to use. Below are a couple screenshots of the town during the day:


And another at night:

Of course there were many trials along the way and things that didn’t quite work as we’d planned as can seen below:

   

We learnt the importance of GPU compressed textures and had to write a pixel shader decompressor of our own, as WebGL doesn’t currently support them natively – but with a cost.  The loss of mip-maping this causes it can clearly be seen; and we will have to work around this if they are not supported soon.

Overall we are very pleased with the result, which you can check out here.  Remember to press space to unlock your mouse to look around – if you aren’t fond of reading on-screen instructions 😉

Naturally this is just a taster of what we have waiting in the wings. We’ll look to provide some follow-up blog posts about the techniques and tools being used in this early experiment including:

  • Web Audio API
  • Pixel shader texture decompression
  • Deferred shading

3rd Party Libraries in use

Illyriad: HTML5 WebGL Preview 2

We thought we’d up the game on the last demo and show a bit of lighting and animation.

A high level flour mill from Illyriad seemed an obvious choice!

Again this is purely HTML5 using the WebGL canvas and JavaScript; no plugins were used. We use mrdoob’s excellent Three.js library:

[youtube=https://www.youtube.com/watch?v=0bZmt3yQzmE]

We’ve included the full browser window in this recording to show that it is in fact running in a Chrome browser window. While it happily runs at 1080p, we’ve had to record at a much lower resolution as the screen capture utility slows everything down… Alas.

Illyriad: HTML5 WebGL Preview

We’ve done a bit of bloging about the “now”, but what about the future?

Well here at Illyriad we’ve been experimenting with WebGL and I can tell you we are impressed with how its shaping up.

Here’s a little taster of what we’ve been trying:

[youtube=https://www.youtube.com/watch?v=F22IvhUtrmM]

Its slightly shaky – but that’s my mouse movement not any lag from WebGL. Looks like I might never be a brain surgeon… haha

HTML5 Canvas: Creating a screenshot from multiple canvases in JavaScript

Programmatically creating screenshots from JavaScript of your canvas element is fairly straightforward; but how do you do it when your game or scene is composed of multiple overlaid canvases?

This example makes use of the canvas caching and it links into the animation loop for starting and stopping – your mileage may vary.

For simplicity, lets say we have 4 canvas elements in use, all of the same size (500×500) used in the following order:

  1. A far background: canvasFar
  2. A near background for a touch of Parallax scrolling: canvasNear
  3. A layer for player characters and other objects: canvasSprites
  4. A layer for special effects: canvasEffects

[code lang=”javascript”]
function TakeScreenshot() {
StopAnimating(); // Stop the animation loop

var bufferScreenshot = CreateCanvasCached("Screenshot");
bufferScreenshot.height = 500; bufferScreenshot.width = 500;
var contextScreenshot = bufferScreenshot.getContext("2d");
// Draw the layers in order
contextScreenshot.drawImage(
document.getElementById("canvasFar"), 0, 0, 500, 500);
contextScreenshot.drawImage(
document.getElementById("canvasNear"), 0, 0, 500, 500);
contextScreenshot.drawImage(
document.getElementById("canvasSprites"), 0, 0, 500, 500);
contextScreenshot.drawImage(
document.getElementById("canvasEffects"), 0, 0, 500, 500);

// Save to a data URL as a jpeg quality 9
var imgUrl = bufferScreenshot.toDataURL("image/jpeg", .9);

StartAnimating(); // Restart the animation loop
return imgUrl;
}
[/code]

Now calling TakeScreenshot() will return a data URL that contains the image – with this you can:

  • Set the src of an image element or css background
  • Open it in a new window to display it as an image (unfortunately you can’t set the content-disposition or filename of a data URL for automatic download)
  • Send it to your sever to base64 decode and create an image file which you can then to popup a download or embed in a page which can then be shared, tweeted, posted to Google+ or Facebook.
  • Use it as source image for more canvas fun or a WebGl texture

UPDATE: There is a toBlob() method on Canvas that allows the saving of generated files on the client-side which is coming in Chrome 14, hopefully other browsers will follow suit.

This should all work fine.  However if you are using images from different domains or a multiple sub-domains – either by using a CDN or for paralleling requests across domain names you will receive a:

SECURITY_ERR: DOM Exception 18

This is a security protection to ensure your code isn’t grabbing images it shouldn’t be, and sending them back to your server. I’ll post about how to work with this in a future post.

(Fix) Memory Leaks: Animating HTML5 Canvas

If you create canvas elements dynamically either for off-screen composition or sprite manipulation, when this is in an animation loop running hundreds of times a second it can start to leak memory.

I’m not entirely sure why this is; but it’s quite widespread in many browsers including Webkit. Perhaps the animation causes the browser to register as continually active which prevents the garbage collector from running – as it will clean it up if you stop the animation and change DOM elements e.g. from an Ajax update. But that’s just wild speculation…

What we are interested in is how to fix it, and also achieve some performance gains from not continuously creating DOM elements.

The trick is to create a function which caches these canvas elements and reuses them. The easiest way is to have a create canvas function that handles all this for you. To keep a handle on the different canvases in use in Illyriad, we use a named canvas element to ensure the correct one is returned. You can call them whatever is relevant:

[code lang=”javascript” title=”Create cached canvas”]
var CachedCanvases = new Object(); // Canvas element cache

function CreateCanvasCached(name) {
if (!CachedCanvases[name]) {
var canvas = document.createElement(‘canvas’);
CachedCanvases[name] = canvas;
return canvas;
}
return CachedCanvases[name];
};
[/code]

Using this is very simple, whenever you create a canvas dynamically that is just used for composition just (i.e. you won’t attach it to the DOM) use the cache function:

[code lang=”javascript”]
var SpriteCompositionCanvas = CreateCanvasCached("sprite");
SpriteCompositionCanvas.width = 64; SpriteCompositionCanvas.height = 64;
var SpriteCompositionContext = SpriteCompositionCanvas.getContext("2d");

[/code]

I’ll do another post on some of the wonderful composition operations you can do using canvas; when this technique will become very important.

If you are updating your page using Ajax and programmatically adding and removing the display of canvas elements you will want to clear the canvas cache when you change page by adding a cache clean up to your ajax page unload function.

[code lang=”javascript”]
function CleanupCachedCanvasses() {
var delList = new Array();
for (name in CachedCanvases) {
delList.push(name);
}
for (i = 0; i < delList.length; i++) {
delete CachedCanvases[delList[i]];
}
delete delList;
}
[/code]

Now you should be good to go!

HTML5 Canvas: Animation Loop – Only work when you have to

Animating HTML5 Canvas be it 2d or WebGl can put a strain on the end user’s browser.

Trying to draw more animation frames than the browser can handle can lock it up or cause it to terminate your script. Also if the user is viewing a different tab or is in another window; burning the user’s CPU on animation they don’t even see is just squandering their resources.

Luckily, much like debouncing and throttling for user input and ajax calls, there is something we can do. HTML5 provides a neat solution for animation.  The requestAnimationFrame function allows you to put in a call-back which gets executed when the browser is ready to draw another frame. This allows the browser to throttle the animation based on the current CPU load and whether the canvas is on screen or not, resulting in a far better performance and a much lower resource usage.

However, some older browsers don’t support requestAnimationFrame, but we can use the following code to detect if they do, and if they don’t, run though some earlier implementations and finally a fall-back method to ensure we can still use the function:

[code lang=”javascript” title=”Setting up timing function”]
if (!window.requestAnimationFrame) {
window.requestAnimationFrame = (function () {
return window.webkitRequestAnimationFrame ||
window.mozRequestAnimationFrame ||
window.oRequestAnimationFrame ||
window.msRequestAnimationFrame ||
function ( callback,element) {
window.setTimeout(callback, 1000 / 60); // Fallback timeout
};
})();
}
[/code]

To use this function we need to create an animation loop. Also if we separate our animated elements from our static elements (e.g. background) into different canvases, we only need to redraw the static elements when they change which is far less frequently.

For this we create two functions renderStatic and renderAnim; which are outside the scope of this article, but there are plenty of canvas examples to draw on. When the static elements require redrawing we just set the requiresRedraw variable to true.

Putting this altogether we end up with a animation loop looking similar to the following:

[code lang=”javascript” title=”Setting up animation callback”]
var isAnimating = false; // Is animation on or off?
var animateRunning = false; // Are we in the animation loop?
var requiresRedraw = true; // Do the static elements need to be redrawn?

function StartAnimating() { // Start animating/drawing
isAnimating = true;
if (!animateRunning) Draw(); // Only draw if we are not already drawing
}
function StopAnimating() { // Stop animating/drawing
isAnimating = false;
}

function Draw() {
if (isAnimating) { // Only draw if we are drawing
animateRunning = true;
try {
if (requiresRedraw) {
requiresRedraw = false;
renderStatic(); // function defined elsewhere
// which draws static elements
}
renderAnim(); // function defined elsewhere
// which draws animated elements
} catch (e) {
if (window.console && window.console.log)
window.console.log(e); // for debugging
}
requestAnimationFrame(Draw);
animateRunning = false;
}
}
[/code]

To start the drawing we can use StartAnimating() and to stop we can use StopAnimating()

Following this pattern should help you improve the performance of your HTML5 canvas app – while also reducing the load on your user’s computer. A double benefit!

How To: Installing WordPress for IIS7

So you fancy doing some blogging? How do you install WordPress for II7 and above? This should show you how to install WordPress in Internet Information Services 7.

First, fire up the Web Platform Installer as this is the easiest way to start. Choose the Applications tab and click “Add” next to WordPress:

Choosing WordPress in the Web Platform Installer
Choose WordPress in the Web Platform Installer

The click “Install”. If you are asked whether to install any extra needed components (e.g. PHP, MySQL etc.) – confirm this.

Choose to install MySQL locally

At the next screen choose to install MySQL locally; unless you are going to run it across multiple servers or have an install of MySQL you can use elsewhere.

Confirm install, with required components (e.g. PHP)

Confirm the install at the next screen. The little crosses threw me a bit, ticks might have been better…

Choose admin password for MySQL

At the next screen you need to create an Admin password for MySQL. Remember what you enter as this password as you will need it later – click continue.

Wait for install to complete

The install of all the various components, and any service packs etc. can take a little while so be patient…

You may have to restart a couple times

You may be asked to restart a couple times, depending on what bits needed to be installed. After a reboot, if the installation doesn’t continue straight away – just fire up the Web Platform Installer again and it will continue.

You may receive installation errors

If you receive any installation errors; check the logs, however this is likely to be from restarting at inappropriate times. Just reselect WordPress to install – your list of extra extensions should be shorter – and hopefully the install will now complete without errors.

Choose IIS website setup details (click for larger)

At the next screen you choose how your new WordPress site will be accessed from the internet (click to see larger version):

  • Website: Either Choose a “New Website” if you want it to go from the root of a domain or subdomain; or choose an existing site if you want it off a subdriectory.
  • ‘WordPress’ application name: Use / if you are creating a new website and want it in the root, else choose the subdirectory you want it accessed from the Internet at.
  • Web Site Name: This is the both the Application pool and the name of the Site in IIS Manager, unless you are creating a subdirectory – when it is just the name of the App Pool.
  • Physical path: This is the actual physical directory on disk where your WordPress files will be held.
  • IP Address: Unless you want WordPress to only respond on a particular IP address – just leave this as “All Unassigned”
  • Host Name: This is the actual domain that WordPress will respond to, so set it to the subdomain or domain you want to use e.g. blog.yoursite.com or www.myblog.com etc.

After filling in the details you want to use, click continue.

Enter the database login credentials you provided earlier (click for larger)

At the next screen you need to use the database password you entered earlier for the Database Administrator Password. Scrolling further down:

Enter details for a new user WordPress will use (click for larger)

Choose a database User Name and password for the install to create. Note: this is not the user name you will be using, but the user WordPress will use to connect to the database. If you installed the database locally, put localhost in for the Database Server.

Scrolling further down:

Add some strengthening phrases (click for larger)

Enter a database name to use, this could be the name of the blog or the website name you used earlier. Next you need to enter a set of Unique Key phrases. These are to strengthen the encryption for passwords and authentication. Go wild, you don’t need to remember these.

Click continue:

Installation complete, click launch WordPress

You’ve now installed WordPress and just need to do a final configuration step, so click the “Launch WordPress” link on this page.

Configure WordPress and setup new user (click for larger)

Now you just need to fill in a few more details on you new WordPress website.

  • Site Title: This is the title of you blog “My blog” or whatever you want it to be known as.
  • Username: This is the Username you will use to actually login to your WordPress site.
  • Password: Choose a password
  • Your E-mail: And enter your email address

Click install WordPress…

Congratulations! You should now how WordPress successfully installed and ready to use. Just log in at the next screen providing the login credentials you just supplied:

JavaScript: Don’t spam your server, Debounce and Throttle

If you make server call-backs or update the browser GUI based on user interaction you can end up doing far more updates than necessary…

In fact on one occasion, during Illyriad-beta testing, we started tripping our server’s dynamic IP request rate throttling and banning our own IP address – D’oh!

Common situations where over responding to events are mousemove, keypress, resize and scroll; but there are a whole host of situations where this applies.

We update our pages dynamically via ajax, based on user clicks around the user interface. In this situation if the user makes several page navigation requests in quick succession – the only important one to be made is the last, as for all the previous ones the data will be thrown away. So why make the requests in the first place? It’s just unnecessary server load.

Equally, if a user is moving an mouse around a map and you are updating UI coordinates for every pixel change the mouse makes, it potentially a huge number of DOM updates far beyond the perception of a user; for no real benefit, but it will cause a degradation in performance.

Initially, we controlled the situation using complex setups of setTimeout clearTimeout and setInterval. However, as the various places we made use of this became more varied, with different sets of needs, the code became more and more complex and difficult to maintain.

Luckily, it was around this time we discovered Ben Alman’s jQuery plugins debounce and throttle; which are extremely powerful – yet simple to use of and cover a whole host of permutations.

Throttling

Using his  jQuery $.throttle, you can pass a delay and function to $.throttle to get a new function, that when called repetitively, executes the original function no more than once every delay milliseconds.

Throttling can be especially useful for rate limiting execution of handlers on events like resize and scroll. Check out the example for jQuery throttling.

Debouncing

Using his jQuery $.debounce, you can pass a delay and function to $.debounce to get a new function, that when called repetitively, executes the original function just once per “bunch” of calls, effectively coalescing multiple sequential calls into a single execution at either the beginning or end.

Debouncing can be especially useful for rate limiting execution of handlers on events that will trigger AJAX requests. Check out the example for jQuery debouncing.

In fact its well worth checking out all Ben Alman’s jQuery Projects as they are full of little wonders, that you soon are sure how you did without them!