Syringe.Net.Nz

Windows Azure Drives + SQL Compact 4 – Part #1

2010-07-20T20:24:02.02425+12:00

So there’s not a whole lots of stuff on the interweb that gets you started with drives and SQL Compact 4 is only just out into the open. My need was for a lightweight database for a sample app I’m building with Chris Klug for our session at Tech Ed New Zealand.

I thought I’d run quickly through how I got things going.

Getting the Bits
I grabbed the bits using the Microsoft Web Platform installer. I wanted two things. 1) SQL Compact 4 CTP1 and; 2) The Web Matrix tool.
It’s hopefully fairly obvious why I needed SQL Compact. The reason I wanted Web Matrix is because it’s currently the only tool I could find that will allow me to create and edit SQL Compact databases.

I’ll be using Visual Studio for much of this post; I’m guessing y’all know how to get your mittens on that now.

Creating a Database
To build our database we’ll be using Web Matrix. It doesn’t have a specific database designer that is independent of the site editor so we’ll need to create a site and then create it in the context of our site.

We’ll start from a template.

…and create an empty site.

Choose Databases from the left hand nav.

Choose to create a new database from the ribbon

Double click to select tables. And right click to add a new table

Create a basic table definition with a ContactID and ContactName. Use the Ribbon controls to create new columns. Use a BigInt identity column for the ID and make it the Primary Key.

Save the table as Contact

Close Web Matrix- our database has been created and saved into a subfolder of my documents. Mine was at

C:\Users\Chris Auld\Documents\My Web Sites\Empty Site\App_Data

We need some data in our database. We could ahve used Web Matrix to do this but I wanted to show another tool that allows us to execute raw SQL against the DB as it may be more suited to some developers.

The tool in question is the SQL CE Toolbox (you’ll need the latest version which is a standalone application)
Grab it from here: http://sqlcetoolbox.codeplex.com/
Don’t just install the VIsual Studio add-in version as at the time of writing it doesn’t support the v4 sdf format.

Open the tool and right click to add a new connection

Open the database file and test the connection

Expand the tree to show our Contact table. Right click and choose ‘Script as Insert’. This will create a template insert script for us.
Enter some sensible (or stupid if you prefer) values and then execute the query. I didn’t attempt to insert the identity column and so removed that from the query.

Before we leave this tool click the Estimated Plan toolbar button. As long as you have SQL Server Management Studio installed it’ll open that tool and hand over the query plan text.

That’s right folks. That is a graphical query plan graph of the query for my embedded database. That’s pretty damn cool!

Getting Your Database Into Windows Azure Storage

Being a database we want our *.sdf file sitting in a WIndows Azure drive. There are a bunch of ways we could do this:

- We could put it into our service package, create a drive on startup, copy to the drive and then do a lazy check each time we start so it’s only done once.
-We could create the whole VHD on the client machine and upload that to storage. This removes any of the once only init code from our application. THis is the approach I chose to use

We need to create a VHD. Hooray for Windows 7- this is trivial. If you are a hard-ass like Steve Marx then you can use a batch command to do this but you all know that I’m not really a hard ass so I’ll do the draggy droppy clicky clicky thing.

Open Disk Management. I’m a fan of the search box in the start menu- type ‘Disk Management’ and then select ‘Create and format hard disk partitions’

Use the Action menu to Create VHD. Stick it somewhere useful like the desktop as we’ll need to grab it later. You’ll want to choose a fixed size- I chose to make it 100MB. You’ll only pay for the actual data you store on disk (not the empty stuff) but more on that shortly.

Once created you’ll need to Initialize the disk with an MBR

.. and Format as NTFS

The drive will now be available in Windows Explorer and we can copy our *.sdf file across from the My Web Sites directory to our new VHD

Once we’ve copied it over we can go back and Detach the VHD

Now we can upload the VHD to Blob storage. Before you crack out your favourite Azure storage client (I like Cloud Studio from Cerebrata) you’ll need to want to check that it supports sparse Page Blob uploads. Cloud Studio doesn’t so we’ll use the VHDUpload Tool from here: http://blogs.msdn.com/b/windowsazurestorage/archive/2010/04/11/using-windows-azure-page-blobs-and-how-to-efficiently-upload-and-download-page-blobs.aspx I took the liberty of building the tool: VHDUpload

More in the Part 2 of this post tomorrow.

Windows Azure Certificates for Self management Scenarios

2010-05-03T21:18:56.510125+12:00

The Windows Azure Management API uses x509 certificates to authenticate callers. In order to make a call to the API you need to have a certificate with both public and private key at the client and and the public key uploaded into the Azure portal. But, if you then want to call the management API from your Windows Azure VMs then you’ll also need to install the cert into the instances by defining them in the service definition This post will show you how.

I found it a bit of a pain to get going so here’s my simple guide. I used this to setup the certs for my favourite open source Azure toolkit Lokad-Cloud. We’ll be creating a self signed certificate, then uploading that certificate into the Windows Azure management portal. Finally we’ll add the certificate to our service model to ensure that Windows Azure installs the certificate into our VM instance when it is started.

Here’s the approach in pictures so you can follow along.

Create a self signed certificate in the IIS7 Manager
Open IIS7 Manager
Expand the node for your local machine
Double Click Server Certificates
Choose Create Self-Signed Certificate
Give it a friendly name

You’re all done in IIS7 Manager. It’s just created a new Certificate and added this into the trusted root certification authorities on your machine. This is a highly trusted location for the cert so do be careful if you ever export it with the private key included.

We need to export it with the public key included so that we can upload it to the Windows Azure Portal.

Run CertMgr.msc - just type it in full into the Start box.
Expand 'Trusted Root Certificates Authorities –> Certificates"’
Find your cert using the friendly name column and right click and choose Export
Choose the option to NOT export the public key. You should generally avoid exporting the public key if at all possible*.
Choose to export as a DER Encoded binary
Save it somewhere useful – we stick ours into source control
Choose Export again, but, this time choose to export the private key
Leave the PFX options all unchecked
Give it a sensible name.

You’re all done exporting. Now we need to upload it to Azure.

Go to the Windows Azure Portal
Choose the appropriate project
Click the Account tab across the top of the page
Click Manage My API Certificates
Browse to and upload your *.cer file you created earlier
You should now see the certificate listed against the account.
Note that it’ll be listed using the Subject rather than the friendly name. So you’ll need to identify it by the thumbprint if you have multiple uploaded from the same machine. Azure Team: Can we please have the friendly name listed here?

Now we’re ready to use our certificate from the Windows Azure Tools for Visual Studio

Open your Windows Azure Project – We’re using Lokad-Cloud here
Expand the Cloud Project to show the Roles.
Right click the Web Role and choose properties

This will open the Snazzy Windows Azure graphical UI. This is much nice than editing the Model XML by hand.
Choose the Certificates Tab
In our case there is already a certificate entry defined by the default Lokad Model definition.
Click the elipses (…) at the end of the thumbnail column to open the certificate chooser dialog.
Choose the cert from the dialog
Repeat for the worker role

The last thing we need to do is upload the certificate into our cloud service before we can upload our packages.

Create a new Cloud Service
Browse to the bottom of the page to find certificates and choose Manage
Browse for your certificate (including the private key)
Punch in your password and hit upload
Confirm that you’re all uploaded.

You’re all done. Now you can happily deploy the app to Windows Azure along with having your certificate deployed into your Azure instances as well. This means that your Azure roles can now call the management API themselves.

*There will be times when you need to export the public key. We’ve had to do that in this scenario as we actually want to install the private key into our Azure instances. Another good example here is in a shared development environment. You have the option of either sharing a certificate among everyone or each uploading your own public key. Because we’re using the lokad tool and this explicitly supports just a single cert we’ve actually exported our cert (password protected) into our source control system so all developers can install the same private key.

If you have done an export and want to install the private cert onto a new machine then you’ll need to add it into the Local Computer –> Personal store.

Run MMC by typing MMC in the start run box
Add the Certificates Snapin
Choose Computer Account
Choose local computer
Open the Personal Store and choose More Actions>Import
Browse to and import your certificate.

The certificate should now be visible in the Windows Azure Tools for VS.NET cert selector dialog.

Windows Azure Pricing Calculator Spreadsheet

2010-04-21T09:06:10.883125+12:00

I’ve mentioned this at a heap of sessions I’ve presented at and have never managed to get around to posting it.

So. Attached is my Windows Azure Pricing spreadsheet.

Pretty simple. has three worksheets.

A simple Table storage calculator that determines the cost of table storage.
This includes overhead calculation – i.e. what is the ration of name to value in your name/value pairs
It also checks your key lengths fall within the 260 char URI limit and a few other things.
A model of session state pricing.
Basically shows that SQL Azure will typically be a cheaper option for Session state storage (or anything with little data on disk and high read/write counts).
A detailed (very detailed) model of PhluffyPhotos.

For more detail please see my MIX10 session on Storage in Windows Azure Platform

http://live.visitmix.com/MIX10/Sessions/EX11

Chris and Dave do Denniston

2010-01-08T06:42:29.458+13:00

Denniston is a former coal mining area situated on an alpine plateau about 30km North of Westport on the West Coast of New Zealand (Wikipedia here). It was actually an active mining area right up until the 1960’s. In recent years it has fallen under the ambit of the Department of Conservation. Along with the Buller Cycling club they have been building out a bunch of Mountain Bike tracks around the area.

You can pull details on the cycling area from the DoC site and from the Buller Cycling club.
http://www.doc.govt.nz/parks-and-recreation/places-to-visit/west-coast/buller-area/denniston/activities/mountain-bike-tracks/
http://www.cyclebuller.co.nz/index.php?option=com_content&view=category&layout=blog&id=19&Itemid=3

Dave and I decided we’d try and visit the tracks as part of our summer South Island road trip. We drove in via Murchison as we did some paddling along the way. We had a couple of days in the Murchison region where the water levels were well up. Paddled a few runs that I’ve done before as well as a new run down the mangles that started with about a 2m waterfall- pulled quite a crowd when we ran that one for some reason… didn’t seem that difficult.

We drove into Westport the evening before we planned to ride and found ourselves a nice motel (Buller Bridge Motel) with free WiFi. Got up early and headed up the hill. This place really is quite the plateau. The hill rises steeply off the ocean almost to an altitude of about 650m at the carpark. The views were pretty good for us; on a really fine and still day I can imagine you’d be able to see all the way up to Karamea.

Once we got to the top we had a bit of a potter around looking at some of the old mining ruins. Denniston is famous for the Denniston Incline which is a frighteningly steep, two pitch, coal railway that literally goes straight up the side of the hill. Check out this video from the NZ Archives for an idea of what it was like.

Then we headed up the hill to the Museum Car park which is the designated starting place for the Mountain Biking trails. While getting organized we saw the Google Car driving around- yes, even in the middle of bloody nowhere there is Google!

We started out with the Ropers Hotel Circuit.
http://www.cyclebuller.co.nz/attachments/011_MTB%20-%20Denniston%20-%20Ropers%20Hotel%20Circuit.pdf
Straight away the riding was quite different to anything we’d ridden before in New Zealand. Lots of slick rock and ledges. Was quite fast riding and reasonably hard on the suspension. A 6” trail bike is ideal- we had a Trance and a Mojo. The last part of this track before it hits the road is walking only- for ecological reasons rather than ride-ability. You definitely want to do this track in the predominant direction indicated on the map as it would be a pain to walk up that hill.

Next we rode out on one of the longer trails. This was nominally on 4WD track but I’d challenge most people to take their 4WD there and get it back in one piece. The track was Sullivans Circuit and it went off the other side of the Plateau back down towards Westport.
http://www.cyclebuller.co.nz/attachments/011_MTB%20-%20Denniston%20-%20Sullivans%20Circuit.pdf

This track had plenty of challenging riding. Lots of large rocks and ledges. it was all too easy to go far faster than 0.1mm of lycra really should justify.

Finally we rode the Miners and Drill track circuits. Again, lots of hairy riding with a bunch of sketchy single track, some of which we both had to walk.

There was still plenty of sign of the old coal mining stuff and indeed the coal itself.

Despite only doing about 30km (in 3.15hr!) we were in need of a beer once we got back to the car.

Here’s the GPS dump

And the full TCX file of my Garmin 310XT is here:
http://connect.garmin.com/activity/21786909

Interesting Stats from 2009

2010-01-01T16:42:44.36425+13:00

Thought it would be interesting to do a blog post of my stats for 2009.

It was a pretty hellish year for travel. It’s the first full year that I’ve used TripIt which has proved to be a really useful tool. TripIt also provides travel stats.

Now not all of this was work travel (I had a couple of overseas holidays) but it was still a pretty full on year.

This is borne out in my exercise stats. Obviously my most exercised location was Wellington, but, I also did a whole heap in Issaquah, Washington and also in Bangalore, India.

I remained pretty steady over the course of the year in terms of the amount of exercise I did. It was still really hard to maintain anything approaching a good training program though.

Once again it was a shitty year for me skiing wise, though these stats do miss out a bit of skiing I had in January. I also did far less mountain biking than I would have liked to.

I did do over 300hr of exercise over the year and travelled 2700km. I also burned 65,000 calories.

Will be interesting to see what I can manage this year.

Microsoft Office Extensions to the Open XML File Format (ISO29500) Specification

2009-10-22T12:37:28.609375+13:00

So the question was asked today in my Open XML Development for Office 2010 and beyond as to whether the Word 2010 Extensions documentation was available anywhere.

I had to take an action item to follow up and find this.

It can be found here: http://msdn.microsoft.com/en-us/library/dd773189.aspx

My Hopeless Gym Experience at the Mandalay Bay Spa

2009-10-20T03:43:18.65625+13:00

So… I genuinely object to paying US$20/day to use the hotel gym. But, needs must sometimes and so it was that I found myself at the Mandalay Bay Spa $20 the poorer this morning. Been working reasonably hard in the build up to the K1 road race in a couple of weeks so needed to get some time in on the stationary trainer.

Well, what an utterly useless experience that was. I got inside the Gym to find that;

They had two consumer grade exercycles and that was it as far as bikes went.
None of the cardio equipment was available anyway

So even if I had been able to get on the equipment it wouldn’t have been much good for the Interval workout I had planned- I ripped the crank off a consumer grade exercycle in Bangalore this year so it’s just downright dangerous.

In the end I threw my toys out of the cot, got myself a refund of my fees and went for a run up the strip.

Think I’ll try and find someone friendly at the Luxor for the Gym tomorrow and will not be staying at the Mandalay Bay for MIX10 next year that’s for sure.

Smashed a new PB on my hill repeat ride

2009-09-23T08:16:12.53125+12:00

Got out for hills this morning on the road bike. Turned into a shorter ride, but, I pushed the 2nd interval really hard (100% MHR) and in doing so smashed about 7% off my PB up the short pinch climb I do.

The average power was 450 Watts over the 4min 31 seconds. Pity the hill wasn’t a touch longer as it meant my new PB 5 minute power ended up being only 431 Watts i.e. 30 seconds of my rest break at the top was included.. Goal for the next few weeks is to really work on lifting my power profile. I think I’ll probably look at doing Peak 1’w and 5’w on Happy Valley road as it certainly appears to help going up a hill. At the moment my 1’w isn’t even on the chart and I’d like to work on getting my entire profile into the Cat 4 region over the next few weeks in the lead up to the K1 race. Means I need to be targeting 350 Watts for 60 minutes which I think is going to be the hardest bit.

The other interesting thing is to see just how much temperature variation affects my (supposedly temperature compensating) barometric altimeter on the Polar s625x. All 5 of those intervals should be the same height, but, there is a definite trend downward over the set.

Essential Tools for SQL Azure Development

2009-09-03T23:12:12.21875+12:00

Much like Wade Wegner I think that SQL Azure is the jewel in the cloud for Microsoft. None of the other vendors have anything like it. While it can be a bit sticker-shock-ish given that nominally a gig in SQL Azure is 65 times the price of a gig in Azure storage, once you actually run some real world scenario models it turns out to be really well priced. Think of a SQL Azure instance not as 1GB (or 10GB… but I see no reason to use 10GB partitions) of storage but rather as the cheapest damn fully backed up and HA relational database solution you’ll find anywhere- and yes FOSSers I include your stuff in that calculation; no greasy haired, under washed and over WoWed engineers needed here.

Anyway… I digress.

A major PITA in using SQL Azure is that the tooling is tantalizingly close to be OK… but in many ways it just doesn’t work. Dumping a SQL script and then re-creating the DB in SQL Azure is a painful exercise in find and replace- check out the hands on lab on Migrating a DB to the cloud in the Azure training kit for the gory details…

Wade has a blog post up about a freebie tool written by George Huey that automates this process for you. Essential for your Azure kit bag. It will parse out all the unsupported stuff.

I ran it up and gave it a nice brutal challenge…. the AdventureWorksLT script that’s used in the aforementioned lab. This includes both schema and data and is a decent effort to parse. The tool churned for a good 3 or 4 minutes…. but I got a script out! The original script includes some real curve-balls like XML Indexes and some tables with data to populate but no clustered index (SQL Azure needs a clustered index before you can insert into a table). I don’t expect it’ll get everything right., but, let’s take a look at how it does against my hand crafted script…..

DOH!

It doesn’t support cut and paste or saving of the script yet so I’ll need to go and change the source first… More reporting back from me later. This looks really promising and I’m confident it’s going to solve 90% of the pain points I’ve been hitting trying to move complex (hell even simple) databases to the cloud.

Rebuild Day: Post #1, software audit and backup

2009-08-09T10:35:53.75875+12:00

Today is rebuild day for my primary machine, a Lenovo x61t. Moving to Win7 RTM and Office 2010 Tech Preview.

First up is pulling all the Data off my C drive.

This mainly means the data out of my profile folder. A key thing to remember here is to check your filesystem Virtualization folders for stuff that’s been put there by poorly written software. I usually backup to a hard drive but today I’m going to just push it across to my D drive partition- I made the decision last build to run two partitions again which is a move I’m pleased with.

Then it’s time to do an audit of the Program Files folder to determine which of the crap that I’ve installed over the past 7 months I actually need to install again. here is my list, not necessarily in order. Hopefully it might give readers a few pointers.

Polar Pro Trainer
WKO+
Lightroom
Life Cam
Intelli Point
WinRAR
7-Zip
iTunes
Live Mesh
Office 2010
VS2k8 + SPs
Expression Studio 3
Twhirl/Tweetdeck
Star Alliance Travel Desk
IIS + other goodies via Web Platform Installer
SQL 2k8 inc Books online
DevExpress Controls + Addins
Azure SDK
.NET Services SDK
Process Explorer
Sony Vegas
TortoiseSVN
Live Writer/Messenger
eTrust + RAS Bits
K-Lite Codec Pack
Zoom H Series Driver
SnagIt
Camtasia

Simple Partitioning with Windows Azure Table Storage

2009-08-08T19:50:00.69625+12:00

While there are certainly situations where it makes sense to have a natural PartitionKey when working with Azure storage there are other times when all you really want is a simple way to bucket up your data into equal bins. The usual approach to partitioning is going to be some sort of hash function but if you decide to use a Guid as the RowKey for your data you’ve basically got a nice collision resistant equal distribution already, you just need to turn it into a partition key.

I was sitting on the plane back from Singapore having a bit of a think about this. Given that we can represent our Guid as a 128 bit Interger we can probably just do RowKey % PartitionCount and get a nice simple ordinal for each partition.

So after my birthday dinner I did what any dedicated birthday boy would do and broke out Visual Studio for a bit of a hack around..

First problem was the ‘128 bit integer’ as .NET doesn’t have a native BigInt type. A quick bit of Tiwtter asking and @adjames suggested the BigInteger class in .NET 4.0, but, given this is Azure there’s no .NET 4.0 support quite yet. A bit of Binging (is that a verb yet?) found some posts on StackOverflow and an implementation of a BigInteger class on CodePlex.

A quick console application confirmed that my thinking on the plane was right.

            static
            void Main(string[]
args) { int[] counts = newint[]{0,0,0,0,0,0,0,0,0,0};
DateTime start = DateTime.Now; for (int i
= 0; i < 1000000; i++) { Guid g = Guid.NewGuid(); BigInteger b = new BigInteger(g.ToByteArray());
BigInteger c = new BigInteger(10); //Number
of partitionsint p = BigInteger.ToInt32(BigInteger.Abs(b
% c)); //Console.WriteLine(g.ToString() + " : " + p.ToString()); counts[p]
+= 1; } DateTime end = DateTime.Now; TimeSpan duration = end - start; Console.WriteLine("Took:
" + duration.TotalMilliseconds + " milliseconds");
Console.WriteLine(counts.ToString(",")); Console.ReadKey();
}

Running this confirmed that my RowKey values would be evenly distributed across the 10 partitions- my concern here was that the Guid algorithm might not be quite up to the task but all seems good.

6500ms for a million rows doesn’t look too bad on the face of it. I’m sure there are plenty of performance optimizations to be eeked out, but, they’ll pale into insignificance compared to a round trip to Azure storage via the load balancer. What I do need to test is that it’s not more efficient to rehash the Guid into 64 bits and then calculate the modulo. But that’s for another night- jaded now and hoping to do 100km on the roadie in the morning.