While there are certainly situations where it makes sense to have a natural PartitionKey when working with Azure storage there are other times when all you really want is a simple way to bucket up your data into equal bins. The usual approach to partitioning is going to be some sort of hash function but if you decide to use a Guid as the RowKey for your data you’ve basically got a nice collision resistant equal distribution already, you just need to turn it into a partition key.

I was sitting on the plane back from Singapore having a bit of a think about this. Given that we can represent our Guid as a 128 bit Interger we can probably just do RowKey % PartitionCount  and get a nice simple ordinal for each partition.

So after my birthday dinner I did what any dedicated birthday boy would do and broke out Visual Studio for a bit of a hack around..

First problem was the ‘128 bit integer’ as .NET doesn’t have a native BigInt type. A quick bit of Tiwtter asking and @adjames suggested the BigInteger class in .NET 4.0, but, given this is Azure there’s no .NET 4.0 support quite yet. A bit of Binging (is that a verb yet?) found some posts on StackOverflow and an implementation of a BigInteger class on CodePlex.

A quick console application confirmed that my thinking on the plane was right.

            static
            void Main(string[]
args) { int[] counts = newint[]{0,0,0,0,0,0,0,0,0,0};
DateTime start = DateTime.Now; for (int i
= 0; i < 1000000; i++) { Guid g = Guid.NewGuid(); BigInteger b = new BigInteger(g.ToByteArray());
BigInteger c = new BigInteger(10); //Number
of partitionsint p = BigInteger.ToInt32(BigInteger.Abs(b
% c)); //Console.WriteLine(g.ToString() + " : " + p.ToString()); counts[p]
+= 1; } DateTime end = DateTime.Now; TimeSpan duration = end - start; Console.WriteLine("Took:
" + duration.TotalMilliseconds + " milliseconds");
Console.WriteLine(counts.ToString(",")); Console.ReadKey();
}

Running this confirmed that my RowKey values would be evenly distributed across the 10 partitions- my concern here was that the Guid algorithm might not be quite up to the task but all seems good.

6500ms for a million rows doesn’t look too bad on the face of it. I’m sure there are plenty of performance optimizations to be eeked out, but, they’ll pale into insignificance compared to a round trip to Azure storage via the load balancer. What I do need to test is that it’s not more efficient to rehash the Guid into 64 bits and then calculate the modulo. But that’s for another night- jaded now and hoping to do 100km on the roadie in the morning.

Way cool. @RotoruaNZ Tweeted me the other day after I posted some photos of Mountain Biking up here.

Asked if they could use one of my photos on their Mountain Biking page. Check out Phil Ross on the dipper with the *loud* strobe action going on.

http://rotoruanz.com/ 

If you’re one of my overseas readers I can’t comment the City of Rotorua to you enough. I’ve dreamed of living there since I was at highschool and it’s still 100% on my 5 year roadmap. It’s close to the ski fields, has phenomenal mountain biking (the Redwoods), great kayaking (Kaituna and Wairoa) and great air links for frequent business travellers like me (Can go direct to AKL, WLG, CHC and ZQN).

The only query/quibble I have with the Destination Rotorua site is why on earth is it hosted on the other side of the world!?! See the trace-route below. Looks like it’s in Orlando!
Looks like their site uses PHP so they’ll probably be a good candidate for the Apache version of the Runtime Page Optimizer http://www.getrpo.com/Product/Apache. Running the RPO site tester it reckons it should knock a couple of seconds off the New Zealand load times.

C:\Users\Chris>tracert www.rotoruanz.com

Tracing route to rotoruanz.com [66.7.213.144]
over a maximum of 30 hops:

  1    13 ms    <1 ms     1 ms  …
  2    <1 ms    <1 ms    <1 ms  …
  3     1 ms     2 ms     1 ms  wlgrtr1-65.intergen.org.nz [202.126.87.65]
  4     4 ms     1 ms     2 ms  ihwrtr1-129.intergen.net.nz [202.126.87.129]
  5     2 ms     5 ms     3 ms  32.114.216.13
  6   158 ms   150 ms   148 ms  165.87.71.190
  7   150 ms   148 ms   148 ms  12.127.33.6
  8   150 ms   152 ms   158 ms  cr2.sffca.ip.att.net [12.122.136.74]
  9   148 ms   148 ms   156 ms  ggr3.sffca.ip.att.net [12.122.136.17]
10   150 ms   149 ms   154 ms  att-gw.sanfran.level3.net [192.205.33.78]
11   157 ms   162 ms   161 ms  vlan89.csw3.sanjose1.level3.net [4.68.18.190]
12   176 ms   152 ms   160 ms  ae-83-83.ebr3.sanjose1.level3.net [4.69.134.233]

13   151 ms   155 ms   167 ms  ae-2.ebr3.losangeles1.level3.net [4.69.132.10]
14   148 ms   157 ms   148 ms  ae-63-63.csw1.losangeles1.level3.net [4.69.137.3
4]
15   160 ms   163 ms   161 ms  ae-62-62.ebr2.losangeles1.level3.net [4.69.137.1
7]
16   183 ms   181 ms   192 ms  ae-3.ebr3.dallas1.level3.net [4.69.132.78]
17   202 ms   181 ms   180 ms  ae-93-93.csw4.dallas1.level3.net [4.69.136.166]

18   190 ms   197 ms   181 ms  ae-91-91.ebr1.dallas1.level3.net [4.69.136.133]

19   215 ms   214 ms   224 ms  ae-1-14.bar2.orlando1.level3.net [4.69.137.153]

20   220 ms   218 ms   210 ms  ae-9-9.car2.orlando1.level3.net [4.69.133.69]
21   212 ms   210 ms   211 ms  hostdime.car2.orlando1.level3.net [4.79.118.38]

22   211 ms   211 ms   213 ms  dime151.dizinc.com [66.7.213.144]

Trace complete.

So the session videos are up for MIX09.

It’s video and screen casts …. which means you can watch me deftly dancing across the stage in my yellow Crocs. Both sessions are a bunch of fun! Well worth taking a look at.

Building Accessible RIAs in Microsoft Silverlight
http://sessions.visitmix.com/MIX09/T65M

State of the Art in Web Site Design on Microsoft SharePoint
http://sessions.visitmix.com/MIX09/C20F

So VSeWSS doesn’t provide a Visual Studio project type to create Themes.

So we have to start out with a SharePoint –> Blank Project.

Then we add a Root File

Then we need to create a structure something like this (if we want to be nicely multilingual aware)
 

So creating these by hand is a bit of a pain in the arse… especially if you have to do it during a presentation (come to my presentation at MIX09! ‘State of the Art in Web Site Design on Microsoft SharePoint’).

So I wrote a little Macro that creates these folders for you… thought it might be useful to some people.

          Sub TemporaryMacro() Dim themeName AsString =
InputBox("Enter the template name") Dim rootItem As ProjectItem
= DTE.Solution.Projects.Item(1).ProjectItems.Item(2) Dim templateItem As ProjectItem
= rootItem.ProjectItems.AddFolder("TEMPLATE") templateItem.ProjectItems.AddFolder("THEMES").ProjectItems.AddFolder(themeName)
templateItem.ProjectItems.AddFolder("LAYOUTS").ProjectItems.AddFolder("1033")
_ .ProjectItems.AddFolder("IMAGES").ProjectItems.AddFolder(themeName) End Sub

Hope it’s helpful for people.

So there are plenty of posts around the web about how to get the best performance out of Virtual Machines. I thought I’d jot down what I do. I use VMs mostly for running training and demos- so my usage approach is optimized that way.

First let me say I am a Virtual PC guy. I’ve tried VMWare, I’ve tried Hyper-V, I’ve tried Virtual Server 2005 R2. In the end I just find VPC 2007 SP1 the best balance of performance and convenience. I use VMs mostly for presenting and I do so off a less than uber grunty machine- I run my VMs on a Lenovo x61t Notebook- it’s not the fastest machine in the world but it is tiny, light and versatile- all grat for regular travellers like me. My Vista Index looks like this

You’ll note I’ve invested in the important things- 300GB 7200RPM HDD and 4GB of RAM. Despite only being a 1.6Ghz Processor it still gets a 4.5.

Anyway, I digress. So here’s how I run my VPCs.

I do some basic Host machine tweaks per: http://support.microsoft.com/Default.aspx?id=840193
Basically this is just configuring my Virus Scanner (CA eTrust) to ignore VPC and my VHD and other related files.

I follow some of the guest tweaks that Andrew Connell has collated here.

My key secret is my drive configuration.
I put my VHDs onto a fast Flash key. I’ve got a 32GB Patriot XT and I just got an OCZ ATV 32GB. Both are fast reading drives at about 30 Megabytes per Second..
I then configure my VMC so that the *.vmc file is on my primary HDD spindle (the 7200RPM drive above) and I turn on Undo disks. This means that all the write traffic is on the 7200RPM platter and the read access is split between this main drive and the thumb drive. The result is great performance. I’ll probably swap my primary drive to an SSD later this year which will probably mean a bit of a rethink.

YMMV.. but this works well for me.

Whoops! Got to Xero this morning and got this.

Looking at the address bar it appears that it’s sent me to the custom error page. And then the custom error page has thrown an unhandled exception which has been bubbled to the default error handler.

If ever there was a case for using ‘On Error Resume Next’ then Custom ASP.NET Error pages are it . Throwing an unhandled exceptin from your nice Custom Error page gives bad UX.

On the plus side, the Xero support people are just super and the actual application really rocks too.

Had plenty of conversations over the past few days in Mumbai with people about scale out strategies. About the difference between stateless and stateful portions of your application and about some ideas around choosing horizontal partitioning strategies.

Waddayaknow… I boot up RSS Bandit this morning and Dare has a great post on this.

Covers many of the things we discuss this week.

One of the things I focussed on a bunch this week is having some knowledge of the distribution of your data when choosing a good partition key.
You want a horizontal partitioning key (partition key in Azure Table Storage) that ideally DOESN’T fit a normal distribution.
Think of it like this. If you partition your data using say the first letter of the user name then you’re gonna end up with a whole bunch in your ‘S’ partition and not a lot in your ‘Z’ partition.

Anyway. A great post and well worth reading.

Building Scalable Databases: Pros and Cons of Various Database Sharding Schemes

So Ivan Towlson has written up a great post on C# covariance and contravariance in .NET 4.0

http://hestia.typepad.com/flatlander/2008/12/c-covariance-and-contravariance-by-example.html

I know that I burnt my fingers on this issue doing some ASP.NET MVC stuff a few weeks ago so I’m looking forward to getting my hands on the 4.0 bits to use in Anger.

So one of the funky things about the .NET Services (was Biztalk serivces) part of Windows Azure is the Connectivity Service.

This makes it easier for WCF to communicate bidirectionally through NATs.

It works by initially working through the cloud hosted relay and then attempting to establish a direct connection. The direct connection approach is pretty cool.

Obviously if it’s inside the same subnet they just talk direct, but, if it’s punching through NAT devices it requires some differnet approaches.

The approach that Clemens et al have taken here is the idea of NAT Port Prediction. Basically it involes ‘guessing which ports will be opened outbound, firing a packet at roughly the same time between both ends of the connection and ‘hopefully’ snapping a socket open.

High level details are here:
http://nutss.gforge.cis.cornell.edu/pub/imc05-tcpnat/

Quite a cool approach, Clemens assures us that it’s used a bunch already by the likes of Live Messenger and others.

So not sure if I’m allowed to talk about this… but bugger it… here goes.

So the guys @ Aptimize (http://www.getrpo.com) have put together an add in for Fiddler that basically runs their Page Optimizer in the Fiddler proxy.

So this lets me basically ty the RPO on any site. The actual page load times are all up the wazoo due to all the traffic going *slowly* through fiddler…. but check out the YSlow goodness.

RPO Off, Microsoft.com gets a grade F (34)

RPO On, Microsoft.com gets a grade A (94)

Sometimes you just want to do this

update foo set bar=123 order by foo.lastmodifiedtimestamp

Why.... well you may rely on the order of the timestamp.

It's suprprisingly hard to do but is possible using CTEs.

Idea pinched from here. Reposted in search of better Live Search ranking for the obvious search query 'controlling update order in SQL Server'

Feel free to take a look at the comments to the last post as this is a followup. You may want to ignore the snipey content devoid comments from our friend in the NZOSS community.

Herewith a follow up post that hopefully addresses the substantive questions that were actually raised (thanks Stu)

Sorry for the delay. I've been busy trying to get a high quality specification progressed through the ISO standards process. Oh and I've also managed to get outside to do some skiing in the Montana backcountry.

The issue is that the GPL aims to enforce the distribution of any derived work under the GPL also.

I do not want to release my applications under the GPL and inparticular I do not want to release any Open Source code I write under the GPL as I do not believe in the 'Copyleft' philosophy to which it subscribes.

Now that's fine. As a general rule I avoid GPL code like the plague (we do use LGPL code in some of our products). In fact our contracts at Kognition included a clause requireing neither party to the agreement to provide GPL code to the other.

So the question then comes to can I implement ODF without having to derive my work from any GPL based code.
My feeling is that even looking at the code for say OpenOffice will get me into trouble.
Likewise decompiling the code will be problematic.

I am actually comfortable reverse engineering by observation for features like 'blink', I do not believe that is going to breach copyright in the work.

But the question is, will reverse engineering by observation be sufficient. And to be honest I just don't know the answer to that question. I don't really see myself spending that much time working with ODF as I tend to agree with The Burton Report as to its likely levels of adoption and indeed the likely market segments to adopt it- selling software to people who are philosophically opposed to paying for software is unlikely to be a sustainable business. That said I did find a very interesting bit of commentary on the web about just this problem quite recently.

http://www.gnome.org/projects/gnumeric/announcements/1.8/gnumeric-1.8.shtml

"The Gnumeric team does not envision using the OpenDocument Format as its native format.

The spreadsheet part of ODF, in its current form, is ill defined and has many, many problems. For example: (1) there is no meaningful discussion of what functions a spreadsheet should support and what they should do. Without that, there is little point in trying to move a spreadsheet from one program to another; (2) there is no provision for sharing formulas between cells; (3) there is no implementation -- writing an ODF exporter consists of reverse-engineering OpenOffice to see what parts of the standard it can handle. (Note: the preceding comments relate to the spreadsheet part of ODF only; we do not have an informed opinion on ODF for word processing documents, for example.)

We may revisit this decision in the future, should the situation improve. In the meantime, we will strive to maintain a reasonable importer and exporter."

Those guys look to have actually broached the problem and to be honest that kinda answers my question. If I can't realisitically use ODF without reverse-engineering OpenOffice then I'm pretty much stuffed in terms of writing a GPL free implementation.

JB, Darryl and myself are doing a big road trip around New Zeaand in February showing off some of the great new bits that are shipping this year.

We're going to build and deploy a full blown application live on stage. And we're not talking about some basic console application here... this thing is going to be as mission ready as we can do in two hours! In particular it's going to be hosted across a pair of clustered virtual servers- it's basically the sort of setup that we'd be happy to pick up and drop into the IntergenData Center.

Some of the things we'll be showing off...

• Live mapping
• AJAX Enabled WCF calls
• Virtualization with Windows Server 2008
• IIS7
• Powershell Scripting for data center and VM administration
• ADO.NET Entities
• Spatial Queries with SQL 2008
• Mirroring with SQL 2008

... and a whole lot of other cool things.

I don't want to give the game away yet and tell you what we're building, but, I'll give you a clue.....

You'll be able to keep an eye on where the three Amigos are going Mountain Biking and Whitewater Kayaking and Rafting and Skydiving and things as we make our way around the country.

Thought I'd chunk a quick note up here about a cool sample we built recently.

It shows off using .NET to put together Office Open XML documents (ECMA 376).

It can pull odwn your IIS log files and process these into things like spreadsheets.

It shows just how easy it is to build up office documents as XML.

http://www.codeplex.com/IISAnalyzer

ScottGu has just blogged about the fact they they are making the source to the .NET Framework available so that you can step into it from your debugger.

http://weblogs.asp.net/scottgu/archive/2007/10/03/releasing-the-source-code-for-the-net-framework-libraries.aspx

I just love it how he and his team just stick this sorta cool stuff on their Blogs rather than going theough marketing (and LCA???)

This was originally posted by meon our Intergen Company Blog. Reposted here for completeness.

Service Oriented Architecture: everyone is still talking about it. Recently there was a bit of a blog-o-thread happening among a bunch of US analysts about Microsoft’s SOA strategy. I refer in particular to articles by Lorraine Lawson at IT Business Edge and Dana Gardner at ZDNet.

Gardner is right when he says ‘SOA is a style and conceptual computing framework, not a reason to upgrade.’ Yet, maybe through lack of time at the coalface, he thinks that massive upgrades are required to put this framework in place. The answer couldn’t be further from the truth. The vision of SOA, that is systems demarcating themselves into atomic reusable services accessible by explicitly defined interfaces and open protocols, is easily implementable in the old faithful .NET 1.0 of several years ago. Sure you may not be able to pass transaction scope across interface boundaries or rely on non-transport level message delivery guarantees, but if people are required to upgrade in pursuit of these features then it must surely be called for what it is - the evolution and maturation of the entire SOA ecosystem - there were no vendors offering this six years ago, let alone Microsoft.

Even among its chief critics, Microsoft is considered a key initiator and implementer of the broad set of  WS-* standards. The irony of the criticism levelled at Microsoft in the above articles is that, unlike the other vendors mentioned, the Microsoft stack does not require expensive application servers (beyond the operating system and the cost of that pales against say WebSphere) to actually make the conceptual framework of SOA a concrete reality.

Criticism is levelled at the Connected Systems Division vision of agility in SOA applications. Having seen the future, to an extent, this is somewhere that I think shows great promise. It’s arguable whether we’ll actually be able to compartmentalize enterprise applications much longer. Increasingly I see organisations where our approach, even in relation to users on the ground, is about surfacing features out of a broader IT ecosystem within the organisation.  The future of IT agility will centre on the recombination of existing services and function more than it will the creation of new ones.

Lawson’s focus is a lot closer to the coal face. However twice in her post she notes some association between what she calls ‘Open Code’ and SOA. For example:

“I think what’s confusing matters is Microsoft’s inability, thus far, to reconcile SOA’s demands for open code and standards with a business model that’s thrived on proprietary solutions.”

This could not be further from the truth. While SOA certainly demands Open Standards it most explicitly frees us from the need to have ‘Open Code’. By making the boundaries of the application explicit and well described we are freed from knowing anything of their internal workings - if communication with a service requires us to see the ‘Open Code’ then we have surely failed. The SOA landscape and vision is very much one of highly optimised proprietary systems communicating by way of poorly optimised, but open, protocols. As Microsoft moves from a Desktop and Operating Systems company to being a broader vendor of both those tools and more vertically focused platforms it is inevitable that those teams, at the very least, will see the value in exposing their revenue generating applications for use by other vendors platforms. To ignore the need to have Sharepoint Server or Microsoft CRM participate in a broader Serviced ecosystem is to ignore a good portion of the market who don’t buy into the Windows Everywhere vision.

While analysts in glass towers gaze at their Service Oriented Navels, there are a good chunk of us out in the world making at least part of their utopian vision a reality. They would do well to talk amongst us in addition to themselves every once in a while.