Hi All,

Saw this rather interesting snippet come through on the Twitter feed this morning.

http://mobilitydigest.com/icloud-brought-to-you-by-microsoft-and-amazon/

Clicking through shows you some partial HTTP messages (some stuff is blurred out)

Certainly looks like iCloud is using both Azure Blob storage and S3.

They are probably using Blob storage for the ability to use Shared Access Signatures for file upload, but, without seeing the full URL in the HTTP request that’s a bit of a guess.

My guess is that the call to the iCloud servers for authorizePut will be fetching a SAS and then this is being used in the PUT request to the Blob storage endpoint.

There is a header in there called AuthorizationBSharedKey. I certainly hope that’s not the storage account shared key for the Azure storage account! But again, without seeing the full messages I can’t really tell. It’s certainly not a standard Azure header, but, it does have a somewhat worrying name.

Anyone able to pull the headers in full for some analysis? chris(at)syringe.net.nz

Chris

So there’s not a whole lots of stuff on the interweb that gets you started with drives and SQL Compact 4 is only just out into the open. My need was for a lightweight database for a sample app I’m building with Chris Klug for our session at Tech Ed New Zealand.

I thought I’d run quickly through how I got things going.

Getting the Bits
I grabbed the bits using the Microsoft Web Platform installer. I wanted two things. 1) SQL Compact 4 CTP1 and; 2) The Web Matrix tool.
It’s hopefully fairly obvious why I needed SQL Compact. The reason I wanted Web Matrix is because it’s currently the only tool I could find that will allow me to create and edit SQL Compact databases.

I’ll be using Visual Studio for much of this post; I’m guessing y’all know how to get your mittens on that now.

Creating a Database
To build our database we’ll be using Web Matrix. It doesn’t have a specific database designer that is independent of the site editor so we’ll need to create a site and then create it in the context of our site.

We’ll start from a template.

…and create an empty site.

Choose Databases from the left hand nav.

Choose to create a new database from the ribbon

Double click to select tables. And right click to add a new table

Create a basic table definition with a ContactID and ContactName. Use the Ribbon controls to create new columns. Use a BigInt identity column for the ID and make it the Primary Key.

Save the table as Contact

C:\Users\Chris Auld\Documents\My Web Sites\Empty Site\App_Data

We need some data in our database. We could ahve used Web Matrix to do this but I wanted to show another tool that allows us to execute raw SQL against the DB as it may be more suited to some developers.

The tool in question is the SQL CE Toolbox (you’ll need the latest version which is a standalone application)
Grab it from here: http://sqlcetoolbox.codeplex.com/
Don’t just install the VIsual Studio add-in version as at the time of writing it doesn’t support the v4 sdf format.

Open the tool and right click to add a new connection

Open the database file and test the connection

Expand the tree to show our Contact table. Right click and choose ‘Script as Insert’. This will create a template insert script for us.
Enter some sensible (or stupid if you prefer) values and then execute the query. I didn’t attempt to insert the identity column and so removed that from the query.

Before we leave this tool click the Estimated Plan toolbar button. As long as you have SQL Server Management Studio installed it’ll open that tool and hand over the query plan text.

That’s right folks. That is a graphical query plan graph of the query for my embedded database. That’s pretty damn cool!

Getting Your Database Into Windows Azure Storage

Being a database we want our *.sdf file sitting in a WIndows Azure drive. There are a bunch of ways we could do this:

- We could put it into our service package, create a drive on startup, copy to the drive and then do a lazy check each time we start so it’s only done once.
-We could create the whole VHD on the client machine and upload that to storage. This removes any of the once only init code from our application. THis is the approach I chose to use

We need to create a VHD. Hooray for Windows 7- this is trivial. If you are a hard-ass like Steve Marx then you can use a batch command to do this but you all know that I’m not really a hard ass so I’ll do the draggy droppy clicky clicky thing.

Open Disk Management. I’m a fan of the search box in the start menu- type ‘Disk Management’ and then select ‘Create and format hard disk partitions’

Use the Action menu to Create VHD. Stick it somewhere useful like the desktop as we’ll need to grab it later. You’ll want to choose a fixed size- I chose to make it 100MB. You’ll only pay for the actual data you store on disk (not the empty stuff) but more on that shortly.

Once created you’ll need to Initialize the disk with an MBR

.. and Format as NTFS

\

The drive will now be available in Windows Explorer and we can copy our *.sdf file across from the My Web Sites directory to our new VHD

Once we’ve copied it over we can go back and Detach the VHD

Now we can upload the VHD to Blob storage. Before you crack out your favourite Azure storage client (I like Cloud Studio from Cerebrata) you’ll need to want to check that it supports sparse Page Blob uploads. Cloud Studio doesn’t so we’ll use the VHDUpload Tool from here: http://blogs.msdn.com/b/windowsazurestorage/archive/2010/04/11/using-windows-azure-page-blobs-and-how-to-efficiently-upload-and-download-page-blobs.aspx I took the liberty of building the tool: VHDUpload

More in the Part 2 of this post tomorrow.

The Windows Azure Management API uses x509 certificates to authenticate callers. In order to make a call to the API you need to have a certificate with both public and private key at the client and and the public key uploaded into the Azure portal. But, if you then want to call the management API from your Windows Azure VMs then you’ll also need to install the cert into the instances by defining them in the service definition This post will show you how.

I found it a bit of a pain to get going so here’s my simple guide. I used this to setup the certs for my favourite open source Azure toolkit Lokad-Cloud. We’ll be creating a self signed certificate, then uploading that certificate into the Windows Azure management portal. Finally we’ll add the certificate to our service model to ensure that Windows Azure installs the certificate into our VM instance when it is started.

Here’s the approach in pictures so you can follow along.

1. Create a self signed certificate in the IIS7 Manager
   Open IIS7 Manager
    
2. Expand the node for your local machine
    
3. Double Click Server Certificates
   
4. Choose Create Self-Signed Certificate
   
5. Give it a friendly name
   

You’re all done in IIS7 Manager. It’s just created a new Certificate and added this into the trusted root certification authorities on your machine. This is a highly trusted location for the cert so do be careful if you ever export it with the private key included.

We need to export it with the public key included so that we can upload it to the Windows Azure Portal.

1. Run CertMgr.msc - just type it in full into the Start box.
   
2. Expand 'Trusted Root Certificates Authorities –> Certificates"’
   
3. Find your cert using the friendly name column and right click and choose Export
   
4. Choose the option to NOT export the public key. You should generally avoid exporting the public key if at all possible*.
5. Choose to export as a DER Encoded binary
6. Save it somewhere useful – we stick ours into source control
7. Choose Export again, but, this time choose to export the private key
   Leave the PFX options all unchecked
   
8. Give it a sensible name.

You’re all done exporting. Now we need to upload it to Azure.

1. Go to the Windows Azure Portal
2. Choose the appropriate project
   
3. Click the Account tab across the top of the page
   
4. Click Manage My API Certificates
   
5. Browse to and upload your *.cer file you created earlier
   
6. You should now see the certificate listed against the account.
   Note that it’ll be listed using the Subject rather than the friendly name. So you’ll need to identify it by the thumbprint if you have multiple uploaded from the same machine. Azure Team: Can we please have the friendly name listed here?
   

Now we’re ready to use our certificate from the Windows Azure Tools for Visual Studio

1. Open your Windows Azure Project – We’re using Lokad-Cloud here
2. Expand the Cloud Project to show the Roles.
   
3. Right click the Web Role and choose properties
   
   This will open the Snazzy Windows Azure graphical UI. This is much nice than editing the Model XML by hand.
4. Choose the Certificates Tab
   
5. In our case there is already a certificate entry defined by the default Lokad Model definition.
   Click the elipses (…) at the end of the thumbnail column to open the certificate chooser dialog.
   
6. Choose the cert from the dialog
   
7. Repeat for the worker role

The last thing we need to do is upload the certificate into our cloud service before we can upload our packages.

1. Create a new Cloud Service
2. Browse to the bottom of the page to find certificates and choose Manage
   
3. Browse for your certificate (including the private key)
4. Punch in your password and hit upload
5. Confirm that you’re all uploaded.
   
   

You’re all done. Now you can happily deploy the app to Windows Azure along with having your certificate deployed into your Azure instances as well. This means that your Azure roles can now call the management API themselves.

*There will be times when you need to export the public key. We’ve had to do that in this scenario as we actually want to install the private key into our Azure instances. Another good example here is in a shared development environment. You have the option of either sharing a certificate among everyone or each uploading your own public key. Because we’re using the lokad tool and this explicitly supports just a single cert we’ve actually exported our cert (password protected) into our source control system so all developers can install the same private key.

If you have done an export and want to install the private cert onto a new machine then you’ll need to add it into the Local Computer –> Personal store.

1. Run MMC by typing MMC in the start run box
2. Add the Certificates Snapin
   
   
3. Choose Computer Account
   
4. Choose local computer
5. Open the Personal Store and choose More Actions>Import
   
6. Browse to and import your certificate.

The certificate should now be visible in the Windows Azure Tools for VS.NET cert selector dialog.

I’ve mentioned this at a heap of sessions I’ve presented at and have never managed to get around to posting it.

So. Attached is my Windows Azure Pricing spreadsheet.

Pretty simple. has three worksheets.

1. A simple Table storage calculator that determines the cost of table storage.
   This includes overhead calculation – i.e. what is the ration of name to value in your name/value pairs
   It also checks your key lengths fall within the 260 char URI limit and a few other things.
2. A model of session state pricing.
   Basically shows that SQL Azure will typically be a cheaper option for Session state storage (or anything with little data on disk and high read/write counts).
3. A detailed (very detailed) model of PhluffyPhotos.
   

For more detail please see my MIX10 session on Storage in Windows Azure Platform

http://live.visitmix.com/MIX10/Sessions/EX11

Much like Wade Wegner I think that SQL Azure is the jewel in the cloud for Microsoft. None of the other vendors have anything like it. While it can be a bit sticker-shock-ish given that nominally a gig in SQL Azure is 65 times the price of a gig in Azure storage, once you actually run some real world scenario models it turns out to be really well priced. Think of a SQL Azure instance not as 1GB (or 10GB… but I see no reason to use 10GB partitions) of storage but rather as the cheapest damn fully backed up and HA relational database solution you’ll find anywhere- and yes FOSSers I include your stuff in that calculation; no greasy haired, under washed and over WoWed engineers needed here.

Anyway… I digress.

A major PITA in using SQL Azure is that the tooling is tantalizingly close to be OK… but in many ways it just doesn’t work. Dumping a SQL script and then re-creating the DB in SQL Azure is a painful exercise in find and replace- check out the hands on lab on Migrating a DB to the cloud in the Azure training kit for the gory details…

Wade has a blog post up about a freebie tool written by George Huey that automates this process for you. Essential for your Azure kit bag. It will parse out all the unsupported stuff.

I ran it up and gave it a nice brutal challenge…. the AdventureWorksLT script that’s used in the aforementioned lab. This includes both schema and data and is a decent effort to parse. The tool churned for a good 3 or 4 minutes…. but I got a script out! The original script includes some real curve-balls like XML Indexes and some tables with data to populate but no clustered index (SQL Azure needs a clustered index before you can insert into a table). I don’t expect it’ll get everything right., but, let’s take a look at how it does against my hand crafted script…..

DOH!

It doesn’t support cut and paste or saving of the script yet so I’ll need to go and change the source first… More reporting back from me later. This looks really promising and I’m confident it’s going to solve 90% of the pain points I’ve been hitting trying to move complex (hell even simple) databases to the cloud.

While there are certainly situations where it makes sense to have a natural PartitionKey when working with Azure storage there are other times when all you really want is a simple way to bucket up your data into equal bins. The usual approach to partitioning is going to be some sort of hash function but if you decide to use a Guid as the RowKey for your data you’ve basically got a nice collision resistant equal distribution already, you just need to turn it into a partition key.

I was sitting on the plane back from Singapore having a bit of a think about this. Given that we can represent our Guid as a 128 bit Interger we can probably just do RowKey % PartitionCount  and get a nice simple ordinal for each partition.

So after my birthday dinner I did what any dedicated birthday boy would do and broke out Visual Studio for a bit of a hack around..

First problem was the ‘128 bit integer’ as .NET doesn’t have a native BigInt type. A quick bit of Tiwtter asking and @adjames suggested the BigInteger class in .NET 4.0, but, given this is Azure there’s no .NET 4.0 support quite yet. A bit of Binging (is that a verb yet?) found some posts on StackOverflow and an implementation of a BigInteger class on CodePlex.

A quick console application confirmed that my thinking on the plane was right.

            static
            void Main(string[]
args) { int[] counts = newint[]{0,0,0,0,0,0,0,0,0,0};
DateTime start = DateTime.Now; for (int i
= 0; i < 1000000; i++) { Guid g = Guid.NewGuid(); BigInteger b = new BigInteger(g.ToByteArray());
BigInteger c = new BigInteger(10); //Number
of partitionsint p = BigInteger.ToInt32(BigInteger.Abs(b
% c)); //Console.WriteLine(g.ToString() + " : " + p.ToString()); counts[p]
+= 1; } DateTime end = DateTime.Now; TimeSpan duration = end - start; Console.WriteLine("Took:
" + duration.TotalMilliseconds + " milliseconds");
Console.WriteLine(counts.ToString(",")); Console.ReadKey();
}

Running this confirmed that my RowKey values would be evenly distributed across the 10 partitions- my concern here was that the Guid algorithm might not be quite up to the task but all seems good.

6500ms for a million rows doesn’t look too bad on the face of it. I’m sure there are plenty of performance optimizations to be eeked out, but, they’ll pale into insignificance compared to a round trip to Azure storage via the load balancer. What I do need to test is that it’s not more efficient to rehash the Guid into 64 bits and then calculate the modulo. But that’s for another night- jaded now and hoping to do 100km on the roadie in the morning.

Originally post on the Intergen Blog

I’ve been spending a bunch of time of late thinking, working and talking about Cloud Computing platforms. There’s plenty of smoke and even a little fire too.

If you’ve been listening to the IT media for the past 12 months: Cloud Computing means many things to many different people but I tend to list 3 key attributes of most Cloud Platforms.

1. Elasticity
   The ability to turn capacity on and off as and when required
2. Infinite Scale (or the illusion thereof)
   The ability to scale up/out your solution as much as you want. It’s illusory in that Microsoft or Amazon do have a limited data centre resource, but it’s unlikely your application will be able to come even close to saturating it.
3. Pay for what you use
   In the cloud you only pay for the user accounts, or processor hours, or gigabytes of storage/traffic that you actually use. To a degree this goes along with elasticity- though the level of granularity is key.

From the various vendors there are Cloud Platform offerings with differing degrees of what I tend to think of as vertical integration.

• Software as a Service
  These are the likes of the Microsoft BPOS offerings or Salesforce.com. They’re fully packaged services- rock up with your credit card and you can have a seat of Salesforce or an Exchange mailbox. Your unit of scale is basically a user account- have as many user accounts as you want for as much scale as you want.
• Infrastructure as a Service
  At the other end of the spectrum are the big virtual machine clouds run by people like Amazon.com (EC2) or Rackspace (Mosso). Your unit of scale here is a Virtual Machine (complete with operating system). I also group things like Amazon S3 and Azure Storage in here. You can have as many virtual machines as you want in EC2- but, remember that scale doesn’t come for free, you’re still going to be looking after your operating system; keeping it patched and so forth.
• Platform as a Service
  This is the greyest of the categories. We’re talking here about platforms that provide you with some sort of abstraction atop multiple servers allowing you to scale out (elastically). The sorts of things that fall in here are Windows Azure Compute, Google Gears and Microsoft Live mesh. Arguably Salesforce, CRM Online and SharePoint Online fit in here to a degree too. The unit of scale here is a more abstract concept- these platforms generally require you to build a scale ready application from the outset; by taking this hit up front you get the benefits of not having to worry about things like the underlying operating system management.

So if that’s our taxonomy why do I say that elasticity is the key for early adopters?

Elasticity is the one thing that the cloud vendors can do that you can’t achieve in any other way. Being able to efficiently support users who want to turn things off an on all the time requires both scale and diversity. Scale, because you need to have a large enough customer base that a user turning off a hundred or so machines represents a mere fraction of your total capacity rather than the vast majority of it. Diversity, because you want to have different workloads with different peak demands running on your gear.

The parallels here with more traditional utilities should be obvious. The reason that the electricity system works is scale and diversity. Save for some very large industrial customers, who tend to have different contractual terms, dropping 90% of the demand for a given customer really doesn’t make a big difference to a utility. Their scale is such that individual customer load changes tend to wash each other out. Yes there are national changes in demand over the day but your power company turns capacity on and off in large (and relatively efficient to start/stop) generating units. One of the reasons that this model works is that the utility customers are a diverse bunch- if their entire customer base were residential customers, all of whom cooked roast lamb every night and all at 6pm exactly then the model wouldn’t work quite so well would it?

We’ve got a pretty nice data centre setup at Intergen. We can run applications very efficiently- we have a PUE of 1.62. We can run applications at great scale- need 1000 cores on a 12 month contract we can do that. We can keep you going 24/7/365- we’ve not had an outage since we moved to our new Terrace premises. What I can’t offer you though is 1000 cores for 4 hours one morning because you’ve got a big online sale happening- we simply don’t have the scale nor the customer diversity to make that happen. I’d hazard a guess that there isn’t an internal enterprise data centre in the world geared up for this sort of request. This is the sweet spot of the cloud vendors, particularly the Infrastructure/Platform as a Service people.

It basically comes down to your periodicity and differential of demand. If your application needs 100 cores during working hours and 10 outside working hours you’re a good candidate for the cloud but other options might suffice. If your application needs 10 cores for 360 days of the year and 100 cores for 5 days in that period then my submission is that cloud is your only sensible option here. Highly elastic workloads can only be cost effectively delivered through cloud computing.

So a post that came through on RSS this morning from Ryan Dunn got me motivated to jot down some notes on a topic I’ve bee thinking about for a while. How best to deploy into Windows Azure. I’ll caveat this post with the fact that at the moment the Windows Azure deployment story is fairly simplistic- I don’t think we’ll be deploying via the web portal come RTM.

So in Windows Azure you basically have two types of account, Storage and Compute. Within a compute account you effectively have  two sets of instances that are grouped behind two URLs. These are referred to as ‘Staging’ and ‘Production’. So you might have two URLs that look something like this:
http://foobar.cloudapp.net
http://<guid>.cloudapp.net

For all intents and purposes there is no difference between the instances that sit in behind these URLs. Basically the Azure Load Balancers are just pointing requests at the two different groups of servers. When you then push the ‘big blue button’ to do a production release you are basically just re-configuring the rules on the load balancer to swap the URLs around. This leads me to my point from the post title. In Windows Azure the idea of Staging really is much more akin to the military idea of a staging area. It’s the place where you marshal the troops ready to charge over the trenches.

The ‘Staging’ deployment in Windows Azure basically gives us an area to marshall our instances before we make them live on the production URL. It means that we can achieve a Zero downtime deployment model because the machines are warm when we start routing traffic to them. It also means that you might not be able to  (or probably shouldn’t)  treat the Staging Deployment like you might traditionally treat a staging server in a more traditional on premise deployment. In an on premise staging server you’ll usually take your application, deploy it into staging, test it (under limited load) then redeploy it into production- this is a very different model.

Looking at the image above you’ll see I have 1 instance in my Staging Deployment and 2 Instances in my production deployment.  If I do a swap of these then we’ll end up with 1 instance in production and 2 in staging- basically halving the capacity of our production system. This would be bad.

You should use your Staging deployment as a true staging area. You should configure it to run the same number of instances. You should give it the same configuration settings (where possible given code/data change vectors). In this way when you flick the switch and route traffic at those warm servers you’ll be ready to rock at full capacity.

What does this mean for your Windows Azure development lifecycle? Here’s how I’d configure my accounts for a real world scenario aiming to achieve zero downtime upgrades. I’m going to *gloss over* the considerations of upgrading your data layer and leave that for another blog post.

I’d configure myself with 3 storage accounts and 2 compute accounts for my application.
foobarDevStorage, foobarStageStorage, foobarProdStorage
foobarStage, foobarProd

I’d dev my project on my local box using the local computer fabric and local storage. In some situations I’d use the cloud for my Dev storage instead.

I’d treat my Stage accounts like a traditional Staging server. In other words I’d configure it with a couple of instances and deploy my package up there for pre-release testing. In reality I’d probably never use the Stage.Production Deployment on my staging server instead contenting myself with the http://<guid>.cloudapp.net URL. When I’m ready to do a production release I’d take the same package I used in Stage and deploy it into the Prod account. I’d use a Production config file and my Prod.Staging Deployment in the foobarProd account would be configured to run the full number of instances- i.e. if my production site runs on 30 instances my Prod.Staging deployment would need to be running 30 instances. I’d then to a quick^^ *smoke test* on the Prod.Staging deployment before finally swapping the Prod.Production and Prod.Staging deployments and basically routing traffic at my new deployment. I’d then suspend my Prod.Staging deployment because I’ll be paying to have those instances running.

That’s my current thinking anyway. All subject to change as Windows Azure changes over the coming month.

Now that’s the easy bit! Hot swapping the compute instances like this is easy because we are (or at least should be) building them using a stateless server pattern.

The harder part is achieving a hot swap upgrade when you need to change your data tier. Because the data tier is stateful we need to consider consistency of the data. This is a problem in on-premise models too and one that does not have an easy solution. Maybe the Azure team will come up with some ‘magic’ that helps here??? I’ve got some thoughts on how it might be done but none of them are particularly elegant or developed yet. keep your eye out for a post on it sometime in the future.

If this stuff appears useful to people (including the footnote below) I might bang up a sample and a web cast. Let me know in the comments.

P.S. Ryan: Looking forward to seeing that tool!
^^ In reality I’d probably want each instance to have serviced at least one request in order to have JIT’ed everything and spun up. This means, for a 30 instance deployment, that I’d really have to slam the server with some load and probably have some way (setting a flag in the Application.OnStart event?) of ensuring that I had hit 30 unique servers. Easiest way to generate the load would probably be with a Windows Azure Worker role :-)

So at almost all the Windows Azure events I’ve run around the world recently we’ve always gotten into a discussion about hooking Windows Azure applications up to on-premise systems. Indeed in a recent demo I wrote for a CRM + Windows Azure session I showed how to wire up Windows Azure applications to your on-premise CRM server. Often you’ll be using the cloud for what I call forward processing.

You should use caution here. Your Windows Azure application has the ability to scale almost infinitely. Your on-premise application does not. If you wire your Windows Azure web role directly to your on-premise system, say by making a direct Web Service call, and then your site gets slash dotted you’ve basically built your self a massive cloud hosted Distributed Denial of Service engine!

What you need to ensure that you do is buffer all of your requests from the cloud to your on-premise system via a queue of some sort. You may choose to use a Windows Azure Storage Queue for this or you may choose to use the new Queue and Router capabilities in the .NET Service bus. This means that if your Azure Web Role suddenly gets a huge amount of traffic the requests to your back end system will be queued and processed only as fast as the on premise application is able to handle them- the last thing you want is for a simple client self service application in the cloud to bring down a core business application on premise.

Obviously doing this is going to require that the call to the backend system is capable of being made asynchronously. If it’s a one way operation then it’s easy, if it’s a two way operation then getting data back to your user is generally going to be a case of having your worker role right back to to an Azure Table or indeed having a notification queue for each user that your worker role can enqueue the result onto.

In summary: If you wire your Windows Azure application directly to your on-premise application you are dicing with danger.

So I’ve been head down and arse up this week; I’ve barely had enough time to get on top of my email inbox let alone my unread blog entries. Finally made some progress this evening at YVR and now on NZ83 en route YVR-AKL.

So while I was underwater a stink kicked up around a thing called the ‘Cloud Manifesto’. Meant to be being released Monday, it was leaked here (all 6 pages of it) a few days early. From my reading between the lines of the thinly veiled blog post by Steve Martin from Microsoft it looks like the document has been written, by an as yet undisclosed group, and is being farmed around a bunch of companies for them to ‘sign up’. There is a posting on the Cloud Computing Interop Forum (on Google Groups) by Reuven Cohen (who I’ve never heard of) who is the ‘Creator of the Enomalism Elastic Computing Platform’ (which I have to admit I’ve never heard of either). Reuven is also involved in Cloud Camp (which I have heard of). I tell you all this and encourage you to go and have a sniff around some of the links above and some of the press coverage so far so that you have some context before continuing.

http://www.techcrunchit.com/2009/03/26/out-of-order-20/ Steve Gillmor from TechCrunch

http://blogs.zdnet.com/BTL/?p=15341 – Amazon expressing their antipathy towards the whole thing

http://www.eweek.com/c/a/Cloud-Computing/Microsoft-Calls-for-Open-Cloud-Standards-538212/ Darryl Taft

I provide the above to give some background. Now for some of my thoughts.

The Emperors New Clothes*

So reading the document itself most of it feels so obvious it barely warrants saying- it’s PR puffery really. I’m going to cherry pick a few statements from the document to discuss.

“We believe that these core principles are rooted in the belief that cloud computing should be as open as all other IT technologies.  “

I hate to break it, but, other IT technologies aren’t that open. As I’ll set out below I’m a big believer that in most cases openness and interoperability are emergent aspects of technology and not something you should really set out to engineer.

“To reassure their customers, cloud providers must offer a high degree of transparency into their operations.”

So I just don’t know how realistic this is. Certainly Microsoft and even more so Google are highly protective of their data center operation details. A measure of security through obscurity is important here still I think. I would also disagree with the authors that moving data into a shared infrastructure necessarily exposes one to more potential for unauthorized exposure.

“Cloud providers must use and adopt existing standards wherever appropriate.  The IT industry has invested heavily in existing standards and standards organizations; there is no need to duplicate or reinvent them. “

So this is bang on target and to be honest the biggest thing that the Cloud Providers can do to ensure *useful* openness is to ensure they use existing standards wherever possible. But do we need this sort of industry bickering to state the bleeding obvious?

“Any community effort around the open cloud should be driven by customer needs, not merely the technical needs of cloud providers, and should be tested or verified against real customer requirements.  “

Would be interesting to know what level of customer involvement there was in the Manifesto document? Were folks like SmugMug (who are my favourite example of a great Cloud operated business) or enterprise customers (I know there is at least one airline using EC2 for some of their stuff) involved? It all feels like a bit much of a simplistic puff piece at the moment- though according to the CCIP group post ‘major players’ have been involved.

“Cloud computing standards organizations, advocacy groups, and communities should work together and stay coordinated, making sure that efforts do not conflict or overlap. “

Haven’t really gotten off to the best start in this regard to be honest.

This document is meant to begin the conversation, not define it.

I’m afraid I agree with a number of the commentators (both vested interests and ‘independent’ voices) at the top of this post. It was a pretty shitty way to ‘begin the conversation’.

Who’s Actually Behind It All

So a big question in my mind is who is actually driving this thing?

To be honest if it were just a group of bit players stroking and stoking their egos then I don’t actually think it would have generated the level of interest and posturing (from MS and Amazon to date) that we’ve seen.

Gillmor looks at the obvious candidates, IBM and Google (Amazon having declared they’re outside the tent). It seems inconceivable that Microsoft and Amazon would not be invited to be involved in this at the earliest opportunity. Even if it were the ‘Anything But Microsoft Brigade’ you’d still expect Amazon to be seated at the table- hell their model of a high scale application operator selling their dog food to others embodies for me much of what the cloud is about.

I have no idea how Cohen can, with a straight face, say:

“Given the nature of this document we have attempted to be
as inclusive as possible inviting most of the major names in technology to
participate in the initial draft.”

Given that this Manifesto appears to be largely driven by East Coast companies I can’t help but think that IBM have got their finger in this somehow. It’ll be somewhat ironic if this the case, as a key proponent of the standardization of Office Open XML (now ISO29500) I’ve had first hand knowledge of IBM’s ‘do as we say’; might this be their ‘not as we do’ moment?

I guess all will be revealed come Monday.

On The Substantive Matter- Or Why I Only Kind Of Give a Crap About Interop in the Cloud

Standards stifle innovation. There, I said it. Friends of mine will probably bitch at me for being so blunt, but, I genuinely believe that standardizing technologies is, in most cases, best left until they have reached a reasonable level of maturity.

Once something is standardized pace becomes glacially slow and the ability for innovators to recover a return on intellectual property becomes nigh on impossible. There are, of course, some business whom this model is de rigueur- I guess we’ve got to wait until Moday to find out more.

For the most part we’ll be building our applications using the same protocols and technologies that we’ve always done. At least for the stuff that spins my wheels, high scale applications, there’s not a lot that’s new in the cloud; it’s really just an interesting new way to deploy and dynamically scale the same architectures I’ve been working on since I was building SaaS apps in the Dot Bomb days.

The new stuff really comes down to the cloud fabric^ and the management thereof and frankly I think it’s too early to think about standards. Amazon scales at the unit of a virtual machine, Microsoft does the same but with sugar on the top to hide the fact that it’s really Windows at all. Let’s see what sort of models shake out best for customers before we try and anoint one king.

I Guess We Wait Until Monday

As someone looking to ship a pretty significant Cloud delivered app later this year what I really want is;

• For my Cloud provider of choice to ship their bits!
• For my travel schedule to slow down a bit- my Tripit stats are truly horrendous and I’m flying to Bangalore Monday morning.
• For the Manifesto authors, whoever and wherever they might be to read a little Hayek.

Let’s see was happens Monday.

*So worth nothing this wasn’t my turn of phrase. Used here first but I liked it so much I pinched it!
^Call it what you will, this is the term I use when talking about Windows Azure at least.

So I’ve been doing a bunch of presenting around the Cloud Recently and also starting to discuss it as a platform with a number of our customers.

I thought I’d put together a blog post on some of the key reading I’m getting people to do to come up to speed.

http://berkeleyclouds.blogspot.com/2009/02/above-clouds-released.html
Is an academic white paper out of Berkeley. Well worth a read and I’d probably start your journey here.

It’s also important, no matter which platform you choose, that you have a good understanding of how to build horizontally scalable systems and that you understand why and how to partition your stateful data tier.
I quite like the recent O’Reilly book Beautiful Architecture  for an overview of this with worked examples

Other stuff that’s worth reading include:

The Primer on Scale from Wikipedia is useful
http://en.wikipedia.org/wiki/Scale_out#Scale_vertically_vs._horizontally

Choosing Partition Keys
Some Thoughts on Choosing Partition Keys in Windows Azure's Table Storage

And Also from Dare: On ‘Trusting’ The Cloud
Asking should we trust the cloud is like asking should we trust horseless carriages

If STS’s, IP’s RP’s and other identity acronyms are Dutch to you then Vittorio’s article on Claims is well worth a read
Claims and Identity, On-Premise and Cloud Solutions

So you may find yourself having issues calling into the development fabric web roles. The error message will be something like ‘te server actively refused the connection’. There are some issues with IPv6. The development fabric will only listen on open host and I think what was happening is that the requests appear to be coming from the IPv6 address rather than the IPv4 one3.

Anyway, I disabled IPv6 and commented the IPv6 localhost entry from the *.hosts file. All worked after this.

So one of the cool things you can do in Windows Azure storage is just enter the URL of a public Blob in your browser and retrieve it via an HTTP GET. This makes sense as in REST, a GET and a URI will return the data at the URI. In the case of an Azure Blob this means the bytes stored returned with the appropriate content type in the header.

You can also do thinks like retrieve a list of the Blobs in a given container like this:

It would be nice to be able to do the same with Azure Table storage. A quick glance at the SDK leads one to think it’s as simple as

http://myaccount.table.core.windows.net/Tables

But this will fail on both the Cloud Storage and Local Storage. This is because

“The request must include the Authorization header and the Date header, as specified in Authentication Schemes .”

So that sounds simple enough- I just cracked open Fiddler thinking I’d just punch in the headers. But, these headers are actually fairly complex. Well, ms-x-Date is simple enough, but, the Authentication headers involve both canonicalization and signing of the request.

I’d love to give you a simple solution so you can just browse tables, but, I haven’t got one. I hope that at least this saves you from spending too much time trying.

So a question came up in the Azure training session today about what the timeout was before a ‘Dequeued’ message was returned to the quen unless it was ‘Removed’. Windows Azure is designed to ensure your messages will always be completed so it returns a dequeud message to a queu if it has not be permanently removed after a period of time.

I knew it had to be configurable… but had no idea exactly how.

The answer is here in the SDK

http://msdn.microsoft.com/en-us/library/dd179396.aspx

or, if you are using the StorageClient sample code from .NET (and if you are using .NET then you really should be!) then this post from Neill Kidd should help

http://blogs.msdn.com/neilkidd/archive/2008/12/07/poison-in-windows-azure.aspx

// Get message with 60 sec visibility
Message msg = queue.GetMessage(60);

This means to ‘Dequeue’ a message only place that message back on the queue if RemoveMessage has not been called within 60 seconds.

So based on the questions I have been receiving at the various Azure Services Platform events around the world I am starting to put together a list of blog posts to do as I get time. Please feel free to suggest other posts.

• What Fits Where
  I probably need to do a post that talks a bit about which bits of the Azure Services Platform fit into what sort of businesses and business models. Windows Azure won’t cure world hunger, but, it will be a great platform to use if you’re going to build the next Facebook. There’s not been a lot out of Microsoft with these sort of details yet, so, hopefully this post will go someway to answering these questions.
• Forward Caching in Windows Azure
  This post will talk about how to use the Local Storage in Windows Azure to cache data. Why you would do it. What data you should store in there. What data you should not store in there.
• What Being Stateless Means
  What does building a stateless Web Role or Worker Role mean? What are some common traps where people might slip into ‘statefulness’?
• Multithreading and Parallelization in Windows Azure Worker Roles
  When should you spawn multiple threads vs When you should create new instances. Will discuss things like performance bounds. CPU Bound vs IO Bound vs Network Bound.
• How To Implement A Plug-in Model In Windows Azure
  Because the Windows Azure deployment model is on a Package basis it’s not possible to add new DLLs once an application has initially being deployed into a Compute project. The solution is, of course, to load the DLLs remotely into Cloud Blob storage and then dynamically load the types using a plugin pattern.

Had plenty of conversations over the past few days in Mumbai with people about scale out strategies. About the difference between stateless and stateful portions of your application and about some ideas around choosing horizontal partitioning strategies.

Waddayaknow… I boot up RSS Bandit this morning and Dare has a great post on this.

Covers many of the things we discuss this week.

One of the things I focussed on a bunch this week is having some knowledge of the distribution of your data when choosing a good partition key.
You want a horizontal partitioning key (partition key in Azure Table Storage) that ideally DOESN’T fit a normal distribution.
Think of it like this. If you partition your data using say the first letter of the user name then you’re gonna end up with a whole bunch in your ‘S’ partition and not a lot in your ‘Z’ partition.

Anyway. A great post and well worth reading.

Building Scalable Databases: Pros and Cons of Various Database Sharding Schemes

So I am running an Azure Training course here in Mumbai today and people are asking about good blogs to read WRT the Azure Services Platform.

Here is the OPML export for my Azure folder from RSS Bandit.

If you are building applications with Windows Azure you will know that your Web Role and Worker Role instances need to be stateless. If you’ve been architecting high scale (read horizontally scaling server farms) then you’ll be used to this concept- you’ll know about the Sql Session state provider and so forth.

If you are using Azure you obviously will not be using the Sql provider. Instead you should look at the AspProvider example in the SDK. This uses Azure Blob and Table storage to store your session state.

So one of the funky things about the .NET Services (was Biztalk serivces) part of Windows Azure is the Connectivity Service.

This makes it easier for WCF to communicate bidirectionally through NATs.

It works by initially working through the cloud hosted relay and then attempting to establish a direct connection. The direct connection approach is pretty cool.

Obviously if it’s inside the same subnet they just talk direct, but, if it’s punching through NAT devices it requires some differnet approaches.

The approach that Clemens et al have taken here is the idea of NAT Port Prediction. Basically it involes ‘guessing which ports will be opened outbound, firing a packet at roughly the same time between both ends of the connection and ‘hopefully’ snapping a socket open.

High level details are here:
http://nutss.gforge.cis.cornell.edu/pub/imc05-tcpnat/

Quite a cool approach, Clemens assures us that it’s used a bunch already by the likes of Live Messenger and others.