Why and How We Migrated Babylon.js to Azure: CORS, Gzip, and IndexedDB
You’re working for a startup. Suddenly that hard year of coding is paying off – with success comes more growth and demand for your web app to scale.
In this tutorial, I want to humbly use one of our more recent “success stories” around our WebGL open-source gaming framework, babylon.js and its website: babylonjs.com. We’ve been excited to see so many web gaming devs try it out. But to keep up with the demand, we knew we needed a new web hosting solution.
While this tutorial focused on Microsoft Azure, many of the concepts apply to various solutions you might prefer. We’re going to see also the various optimizations we’ve put in place to limit as much as possible the output bandwidth from our servers to your browser.
But the experience on the website was still good enough. Loading our scenes wasn’t done at stellar speed but users weren’t complaining that much.
However, recently, a cool guy decided to share our work on Hacker News. We were really happy for such news! But look at what happened on the site’s connections:
Game over for our little server! It slowly stopped working and the experience for our users was really bad. The IIS server was spending its time serving large static assets and images, and the CPU usage was too high. As we were about to launch the Assassin’s Creed Pirates WebGL experience project running on babylon.js, it’s was time to switch to a more scalable professional hosting by using a cloud solution.
But before reviewing our hosting choices, let’s briefly talk about the specifics of our engine and website:
1 – Everything is static on our website. We currently don’t have any server-side code running.
2 – Our scenes (.babylon JSON files) and textures (.png or .jpeg) files could be very big (up to 100 MB). This means that we absolutely needed to activate gzip compression on our “.babylon” scene files. Indeed, in our case, the pricing is going to be indexed a lot on the outgoing bandwidth.
3 – Drawing into the WebGL canvas needs special security checks. You can’t load our scenes and textures from another server without CORS enabled for instance.
Credits: I’d like to special thank Benjamin Talmard, one of our French Azure Technical Evangelist who helped us move to Azure.
4. Here comes the awesomeness of the tooling. As I was logged into VS using the Microsoft Account bound to my Azure subscription, the wizard let me simply choose the web site on which I’d like to deploy.
No need to worry about complex authentication, connection string or whatever.
“Next, Next, Next & Publish” and a couple of minutes later, at the end of the uploading process of all our assets and files, the web site was up & running!
On the configuration side, we wanted to benefit from the cool autoscale service. It would have helped a lot in our previous Hacker News scenario.
First, your instance has be configured in “Standard” mode in the “Scale” tab.
Then, you can choose up to how many instances you’d like to automatically scale, in which CPU conditions and also on which scheduled times. In our case, we’ve decided to use up to 3 small instances (1 core, 1.75 GB memory) and to auto-spawn a new instance if the CPU goes over 80% of its utilization. We will remove one instance if the CPU drops under 60%. The autoscaling mechanism is always on in our case, we haven’t set some specific scheduled times.
The idea is really to only pay for what you need during specific timeframes and loads. I love the concept. With that, we would have been able to handle previous peaks by doing nothing thanks to this Azure service! This what I call a service.
You’ve got also a quick view on the autoscaling history via the purple chart. In our case, since we’ve moved to Azure, we never went over 1 instance up to now. And we’re going to see below how to minimize the risk into falling into an autoscaling.
To conclude on the web site configuration, we wanted to enable automatic gzip compression on our specific 3D engine resources (.babylon & .babylonmeshdata files). This was critical to us as it could save up to 3x the bandwidth and thus… the price.
Web Sites are running on IIS. To configure IIS, you need to go into the web.config file. We’re using the following configuration in our case:
As you can see, the green log lines imply that everything works well.
Here is a sample case where it will fail. If you try to load our scenes from our blob storage directly from your localhost machine (or any other domain), you’ll get these errors in the logs:
In conclusion, if you see that your calling domain is not found in the “Access-Control-Allow-Origin” header with an “Access is denied” just after that, it’s because you haven’t set properly your CORS rules. It is very important to control your CORS rules; otherwise, anyone could use your assets, and thus your bandwidth, and thus costing money without letting you know!
I know what you’re thinking. Am I going to do that for all my files?!? No, you could work on building a tool or post-build script that would automate that. For instance, here is a little command-line tool I’ve built:
It’s that simple! In my case, I need to simply change the following URL:
http://yoda.blob.core.windows.net/wwwbabylonjs/Scenes
to
http://az612410.vo.msecnd.net/wwwbabylonjs/Scenes
Note that you can customize this CDN domain to your own if you want to.
Thanks to that, we are able to serve you our 3D assets in a very fast way as you’ll be served from one of the node locations listed here: Azure Content Delivery Network (CDN) Node Locations.
Our web site is currently hosted on the Northern Europe Azure datacenter. But if you’re coming from Seattle, you’ll ping this server just to download our basic index.html, index.js, index.css files and a couple of screenshots. All the 3D assets will be served from the Seattle node just near you!
Note: All our demos are using the fully optimized experience (blob storage using gzip, CDN & DB caching).
153 items and 43.33 MB received.
But if you’ve accepted to let babylonjs.com to “use additional storage on your computer”, here is what you’ll see the second time you’ll load the same scene:
1 item and 348 bytes!
We’re just checking if the manifest file has changed. If not, we’re loading everything from the DB and we’re saving 43+ MB of bandwidth.
For instance, this approach is being used in the Assassin’s Creed Pirates games:
Let’s think about that:
- the game launches almost immediately once it has been loaded once as the assets are served directly from the local DB
- your web storage is less stressed and less bandwidth is used—costing you less money!
Now that will satisfy both your users and your boss!
Introduction
Babylon.js is a personal project we’ve been working on for over a year now. As it’s a personal (i.e. our time and money), we’ve hosted the website, textures & 3d scenes on a relatively cheap hosting solution using a small, dedicated Windows/IIS machine. The project started in France, but was quickly under the radar of several 3D and web specialists around the globe as well as some gaming studios. We were happy about the community’s feedback but the traffic was manageable! For instance, between Feb 2014 and April 2014, we had an average of 7K+ users/months with an average of 16K+ pages/viewed/months. Some of the events we’ve been speaking at have generated some interesting peaks:
But the experience on the website was still good enough. Loading our scenes wasn’t done at stellar speed but users weren’t complaining that much.
However, recently, a cool guy decided to share our work on Hacker News. We were really happy for such news! But look at what happened on the site’s connections:
Game over for our little server! It slowly stopped working and the experience for our users was really bad. The IIS server was spending its time serving large static assets and images, and the CPU usage was too high. As we were about to launch the Assassin’s Creed Pirates WebGL experience project running on babylon.js, it’s was time to switch to a more scalable professional hosting by using a cloud solution.
But before reviewing our hosting choices, let’s briefly talk about the specifics of our engine and website:
1 – Everything is static on our website. We currently don’t have any server-side code running.
2 – Our scenes (.babylon JSON files) and textures (.png or .jpeg) files could be very big (up to 100 MB). This means that we absolutely needed to activate gzip compression on our “.babylon” scene files. Indeed, in our case, the pricing is going to be indexed a lot on the outgoing bandwidth.
3 – Drawing into the WebGL canvas needs special security checks. You can’t load our scenes and textures from another server without CORS enabled for instance.
Credits: I’d like to special thank Benjamin Talmard, one of our French Azure Technical Evangelist who helped us move to Azure.
Step 1: Moving to Azure Web Sites & the Autoscale service
As we’d like to spend most of our time writing code and features for our engine, we don’t want losing time on the plumbing. That’s why, we immediately decided to choose a PaaS approach and not a IaaS one. Moreover, we liked Visual Studio integration with Azure. I can almost do everything from my favorite IDE. And even if babylon.js is hosted on Github, we’re using Visual Studio 2013, TypeScript and Visual Studio Online to code our engine. As a note for your project, you can get Visual Studio Community and an Azure Trial for free. Moving to Azure took me approximately 5 min: 1. I created a new Web Site in the admin page: http://manage.windowsazure.com (could be done inside VS too). 2. I took the right changeset from our source code repository matching the version that was currently online. 3. I right-clicked the Web project in the Visual Studio Solution Explorer.
4. Here comes the awesomeness of the tooling. As I was logged into VS using the Microsoft Account bound to my Azure subscription, the wizard let me simply choose the web site on which I’d like to deploy.
No need to worry about complex authentication, connection string or whatever.
“Next, Next, Next & Publish” and a couple of minutes later, at the end of the uploading process of all our assets and files, the web site was up & running!
On the configuration side, we wanted to benefit from the cool autoscale service. It would have helped a lot in our previous Hacker News scenario.
First, your instance has be configured in “Standard” mode in the “Scale” tab.
Then, you can choose up to how many instances you’d like to automatically scale, in which CPU conditions and also on which scheduled times. In our case, we’ve decided to use up to 3 small instances (1 core, 1.75 GB memory) and to auto-spawn a new instance if the CPU goes over 80% of its utilization. We will remove one instance if the CPU drops under 60%. The autoscaling mechanism is always on in our case, we haven’t set some specific scheduled times.
The idea is really to only pay for what you need during specific timeframes and loads. I love the concept. With that, we would have been able to handle previous peaks by doing nothing thanks to this Azure service! This what I call a service.
You’ve got also a quick view on the autoscaling history via the purple chart. In our case, since we’ve moved to Azure, we never went over 1 instance up to now. And we’re going to see below how to minimize the risk into falling into an autoscaling.
To conclude on the web site configuration, we wanted to enable automatic gzip compression on our specific 3D engine resources (.babylon & .babylonmeshdata files). This was critical to us as it could save up to 3x the bandwidth and thus… the price.
Web Sites are running on IIS. To configure IIS, you need to go into the web.config file. We’re using the following configuration in our case:
<system.webServer>
<staticContent>
<mimeMap fileExtension=".dds" mimeType="application/dds" />
<mimeMap fileExtension=".fx" mimeType="application/fx" />
<mimeMap fileExtension=".babylon" mimeType="application/babylon" />
<mimeMap fileExtension=".babylonmeshdata" mimeType="application/babylonmeshdata" />
<mimeMap fileExtension=".cache" mimeType="text/cache-manifest" />
<mimeMap fileExtension=".mp4" mimeType="video/mp4" />
</staticContent>
<httpCompression>
<dynamicTypes>
<clear />
<add enabled="true" mimeType="text/*"/>
<add enabled="true" mimeType="message/*"/>
<add enabled="true" mimeType="application/x-javascript"/>
<add enabled="true" mimeType="application/javascript"/>
<add enabled="true" mimeType="application/json"/>
<add enabled="true" mimeType="application/atom+xml"/>
<add enabled="true" mimeType="application/atom+xml;charset=utf-8"/>
<add enabled="true" mimeType="application/babylonmeshdata" />
<add enabled="true" mimeType="application/babylon"/>
<add enabled="false" mimeType="*/*"/>
</dynamicTypes>
<staticTypes>
<clear />
<add enabled="true" mimeType="text/*"/>
<add enabled="true" mimeType="message/*"/>
<add enabled="true" mimeType="application/javascript"/>
<add enabled="true" mimeType="application/atom+xml"/>
<add enabled="true" mimeType="application/xaml+xml"/>
<add enabled="true" mimeType="application/json"/>
<add enabled="true" mimeType="application/babylonmeshdata" />
<add enabled="true" mimeType="application/babylon"/>
<add enabled="false" mimeType="*/*"/>
</staticTypes>
</httpCompression>
</system.webServer>
This solution is working pretty well and we even noticed that the time to load our scenes has been reduced compared to our previous host. I’m guessing this is thanks to the better infrastructure and network used by Azure datacenters.
However, I have been thinking about moving into Azure for a while now. And my first idea wasn’t to let web sites instances serving my large assets. Since the beginning, I was more interested in storing my assets into the blob storage better designed for that. It would offer us also a possible CDN scenario.
Step 2: Moving assets into Azure Blob Storage, enabling CORS, gzip support & CDN
The primary reason for using blob storage in our case is to avoid loading the CPU of our web site instances to serve them. If everything is being served via the blob storage except a few html, js & css files, our web site instances will have few chances to autoscale. But this raises two problems to solve: 1. As the content will be hosted on another domain name, we will fall into the cross-domain security problem. To avoid that, you need to enable CORS on the remote domain (Azure Blob Storage) 2. Azure Blob Storage doesn’t support automatic gzip compression. And we don’t want to lower the CPU web site usage if in exchange we’re paying 3x time the price because of the increased bandwidth!Enabling CORS on blob storage
CORS on blob storage has been supported for a few months now. This article, Windows Azure Storage: Introducing CORS, explains how to use Azure APIs to configure CORS. On my side, I didn’t want to write a small app to do that. I’ve found one on the web already written: Cynapta Azure CORS Helper – Free Tool to Manage CORS Rules for Windows Azure Blob Storage. I then just enabled the support for GET and proper headers on my container. To check if everything works as expected, simply open your F12 developer bar and check the console logs:
As you can see, the green log lines imply that everything works well.
Here is a sample case where it will fail. If you try to load our scenes from our blob storage directly from your localhost machine (or any other domain), you’ll get these errors in the logs:
In conclusion, if you see that your calling domain is not found in the “Access-Control-Allow-Origin” header with an “Access is denied” just after that, it’s because you haven’t set properly your CORS rules. It is very important to control your CORS rules; otherwise, anyone could use your assets, and thus your bandwidth, and thus costing money without letting you know!
Enabling gzip support on our blob storage
As I was telling you before, Azure Blob Storage doesn’t support automatic gzip compression. It seems to be also the case of competitors’ solutions like S3. You’ve got two options to work around that: 1. gzip yourself the files on the client before uploading, upload it in the blob storage using your classic tools and set the “content-encoding” header to “gzip”. This solution works but only for browsers supporting gzip (is there still a browser not supporting gzip anyway?) 2. gzip yourself the files on the client & upload two versions in the blob storage: one with the default .extension and one with the .extension.gzip, for instance. Set up a handler on the IIS side that will catch the HTTP request from the client, check for the header “accept-encoding” set to “gzip” and serve the appropriate files based on this support. You’ll find more details on the code to implement in this article: Serving GZip Compressed Content from the Azure CDN. In our case, I don’t know any browser supporting WebGL and not gzip compression. So if the browser doesn’t support gzip, there’s no real interest in going further as this probably means that WebGL is not supported either. I’ve then chosen the first solution. As we don’t have a lot of scenes and we’re not producing a new one every day, I’m currently using this manual process: 1. Using 7-zip, I’m compressing the .babylon files on my machine using gzip encoding and “compression level” to “fastest”. The other compression levels seems to generate issues in my tests. 2. I upload the file using CloudBerry Explorer for Microsoft Azure Cloud Storage. 3. I manually set the HTTP header “content-encoding” to “gzip” with CloudBerry.
I know what you’re thinking. Am I going to do that for all my files?!? No, you could work on building a tool or post-build script that would automate that. For instance, here is a little command-line tool I’ve built:
string accountName = "yoda";
string containerName = "wwwbabylonjs";
string accountKey = "yourmagickey";
string sceneTextContent;
// First argument must be the directory into the Azure Blob Container targeted
string directory = args[0];
try
{
StorageCredentials creds = new StorageCredentials(accountName, accountKey);
CloudStorageAccount account = new CloudStorageAccount(creds, useHttps: true);
CloudBlobClient client = account.CreateCloudBlobClient();
CloudBlobContainer blobContainer = client.GetContainerReference(containerName);
blobContainer.CreateIfNotExists();
var sceneDirectory = blobContainer.GetDirectoryReference(directory);
string[] filesArgs = args.Skip(1).ToArray();
foreach (string filespec in filesArgs)
{
string specdir = Path.GetDirectoryName(filespec);
string specpart = Path.GetFileName(filespec);
if (specdir.Length == 0)
{
specdir = Environment.CurrentDirectory;
}
foreach (string file in Directory.GetFiles(specdir, specpart))
{
string path = Path.Combine(specdir, file);
string sceneName = Path.GetFileName(path);
Console.WriteLine("Working on " + sceneName + "...");
CloudBlockBlob blob = sceneDirectory.GetBlockBlobReference(sceneName);
blob.Properties.ContentEncoding = "gzip";
blob.Properties.ContentType = "application/babylon";
sceneTextContent = System.IO.File.ReadAllText(path);
var bytes = Encoding.UTF8.GetBytes(sceneTextContent);
using (MemoryStream ms = new MemoryStream())
{
using (GZipStream gzip = new GZipStream(ms, CompressionMode.Compress, true))
{
gzip.Write(bytes, 0, bytes.Length);
}
ms.Position = 0;
Console.WriteLine("Gzip done.");
blob.UploadFromStream(ms);
Console.WriteLine("Uploading in " + accountName + "/" + containerName + "/" + directory + " done.");
}
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex);
}
To use it, I could do:
UploadAndGzipFilesToAzureBlobStorage Scenes/Espilit C:\Boulot\Babylon\Scenes\Espilit\*.babylon*
to push a scene containing multiple files (our incremental scenes with muliples .babylonmeshdata files)
Or simply:
UploadAndGzipFilesToAzureBlobStorage Scenes/Espilit C:\Boulot\Babylon\Scenes\Espilit\Espilit.babylon
to push a unique file.
To check that gzip was working as expected using this solution, I’m using Fiddler. Load your content from your client machine and check in the network traces if the content returned is really compressed and can be uncompressed:
Enabling CDN
Once you’ve done the two previous steps, you just need to click on a single button in the Azure administration page to enable CDN and map it to your blob storage:
It’s that simple! In my case, I need to simply change the following URL:
http://yoda.blob.core.windows.net/wwwbabylonjs/Scenes
to
http://az612410.vo.msecnd.net/wwwbabylonjs/Scenes
Note that you can customize this CDN domain to your own if you want to.
Thanks to that, we are able to serve you our 3D assets in a very fast way as you’ll be served from one of the node locations listed here: Azure Content Delivery Network (CDN) Node Locations.
Our web site is currently hosted on the Northern Europe Azure datacenter. But if you’re coming from Seattle, you’ll ping this server just to download our basic index.html, index.js, index.css files and a couple of screenshots. All the 3D assets will be served from the Seattle node just near you!
Note: All our demos are using the fully optimized experience (blob storage using gzip, CDN & DB caching).
Step 3: Using HTML5 IndexedDB to avoid re-downloading the assets
Optimizing loading times and controlling output bandwidth costs is not just about server-side. You can also build some logic client-side to optimize things. Fortunately, we’ve done that since v1.4 of our babylon.js engine. I’ve explained in great details how I’ve implemented the support for IndexedDB in this article: Using IndexedDB to handle your 3D WebGL assets: sharing feedbacks & tips of Babylon.JS and you’ll find how to activate it in babylon.js on our wiki: >Caching the resources in IndexedDB. Basically, you just have to create a .babylon.manifest file matching the name of the .babylon scene, then set what you’d like to cache (textures and/or JSON scene). That’s it. For instance, check what’s going on with the Hill Valley demo scene. First time you’re loading it, here are the requests sent:
153 items and 43.33 MB received.
But if you’ve accepted to let babylonjs.com to “use additional storage on your computer”, here is what you’ll see the second time you’ll load the same scene:
1 item and 348 bytes!
We’re just checking if the manifest file has changed. If not, we’re loading everything from the DB and we’re saving 43+ MB of bandwidth.
For instance, this approach is being used in the Assassin’s Creed Pirates games:
Let’s think about that:
- the game launches almost immediately once it has been loaded once as the assets are served directly from the local DB
- your web storage is less stressed and less bandwidth is used—costing you less money!
Now that will satisfy both your users and your boss!
This article is part of the web dev tech series from Microsoft. We’re excited to share Microsoft Edge and the new EdgeHTML rendering engine with you. Get free virtual machines or test remotely on your Mac, iOS, Android, or Windows device @ http://dev.modern.ie/(dpe)