The image says it all. Take a look at the full post Powerful/Free Download: Refactor! for ASP.NET 2.2 on Mark Millers blog. Thanks to my colleague Brian for passing this along. We have some tinkering to do in Visual Studio tomorrow morning I would think :)

Thomas Jespersen wrote a post about Carbonite and how much he likes the service which made me revisit the idea of online backup. I originally rejected the idea due to privacy, performance, and pricing concerns.

I can't claim that I have much to hide on my computer but I still don't like the idea of transmitting everything I've ever created digitally over the Internet to a service where I basically only have their word for them not being evil and not misusing my data.

Secondly I do have a lot of stuff to backup. I've been very careful with my data during the years and I'm proud to say that I've not lost anything of value or importance for more than five years. Of course the reason for that is an event which happened one day 5 * 365 - 1 days ago :) Luckily I didn't have a whole lot to lose but I learned my lesson. Additionally I'm now using the PC for so much more than just playing around. Specifically my wife would murder me if we ever lost a digital photo and of course I'd weep myself to sleep every night should I lose stuff related to work.

For a long period of time I've been using FolderShare to back my stuff up to multiple locations both on site and off site. FolderShare is a tool for synchronizing folders and therein lies both the power and a very real danger in that deleting, say every digital photo in one location, deletes everything in the other locations as well. Not the very best of backup strategies but it did do the trick for me for a while and for free too.

So back to Thomas Jespersen who reminded me of the online options for backing up. I had taken a look at Carbonite but rejected it as I said. His post however made me rethink that decision because losing everything is much worse than people gaining insight into my personal stuff. Also it's worth mentioning that the online backup services really aren't insecure in that they offer encryption of everything before the data is sent across the wire so one would need to purposely pursue getting access to the data which to me is highly unlikely.

Finally there's pricing which has been a major barrier for me. I'm not prepared to pay hundreds of dollars each year to back my stuff up remotely when I can buy hard drives in the amount of terabytes for that same amount of money and carry them off site. Sure the convenience factor isn't too high on the manual hard drive switch-a-roo thing but still worth considering.

Luckily two service providers do offer pricing which is justifiable and even affordable. Both Mozy and Carbonite offer an unlimited backup plan for $4.95 a month and with rebates you can go even lower if you go for a full year or two.  Also a major factor is the unlimited storage when compared to other backup provides who want major bucks for anything above 10 gigabytes.

So I've basically chosen Carbonite and Mozy for evaluation based on two factors: Price and storage space.

Features: Carbonite 0 / Mozy 1

Carbonite is a pretty bare bones product: you use Windows Explorer to set which folders to backup which is very handy. Each backed up folder gets a little blue dot to indicate that Carbonite is protecting it. From that point on Carbonite runs all by itself no need to think more about it.

With Mozy you will be dealing with multiple backup sets. Backupsets make a lot of sense in the initial backup where you have a lot of stuff you need to backup; some more important than other. I created a backupset for important stuff and lumped everything else into a different set and ran a manual backup of the first on to ensure that documents, photos, code, etc. was backed up before spending time backing up my iTunes library.

Backupsets can be configured by using the physical file structure or by rules. The rules feature is pretty neat because you could create a global backupset for everything related to say code and include projects, code files, solutions and so forth in the set and not worry where it is actually stored. Very appealing to my developer mind which likes generic solutions :) For me the feature is useful to exclude .p2p files which are created by FolderShare for synchronization. Carbonite has a similar feature but it requires you to navigate to the file, right-click and select exclude file type.

Mozy has a nice versioning feature which allows you to go back to a previous version, very nice for those of us used to source control :)

Ease of Use: Carbonite 1 / Mozy 0

Both providers offer a downloadable client which run on your computer unattended. For a  solution that just works Carbonite has the advantage as you don't really need to set anything up for it to work. The user interface is very simple which both a strength and a weakness.

Mozy on the other hand brings a lot of settings you can tweak which is nice for a person like me who like to tweak and tinker with the stuff. For the average person though I think Carbonite comes out ahead, you can't mess anything up because there's nowhere to get a the settings.

Configuration: Carbonite 0 / Mozy 1

As mentioned Mozy offers a lot of configuration options which is great though some of the options seem strange. For example they have put a slider in which you can move between two points: Faster Computer / Faster Backups. My guess is that the slider is for controlling the encoding process but it's not what I would call obvious and I certainly don't get the point of it in this day and age where most of us are equipped with dual core machines. Still nice to have the option I guess.

Mozy offers a lot more configurability in terms of what you're backing up. Mozy has the notion of backup sets which is basically a collection of stuff you want to backup. It's nice to be able to have stuff separated so you get the most important stuff backup up first in the initial backup. In the long run however it's not a very useful feature and Mozy seems to spend forever on populating the backup sets whenever you need to display the client interface.

Network Performance: Carbonite 0 / Mozy 1

This is the big one for me. I have a lot of stuff I need backed up and when I have unlimited storage available I want everything backed up damn it! On paper unlimited storage looks like a good deal but in reality it's very constrained by the network. My connection is a 20/20 mbit fiber optical connection so it shouldn't be a problem to backup everything. The service however limits this, the question is: how much does it limit it? For Carbonite I'm able to get about  1 mbit/sec when backing up which means that it takes forever for me to backup 10 GB.

With Mozy I get anywhere from 1 - 4 mbit, mostly 4 so I have to go with Mozy on this one. Basically the 4x speed increase means the world in difference and I can actually use the service like I want to. One strange thing I've found is that Mozy will actually stop using the network when it encodes (encrypts) data which is unfortunate when the service is network constricted.

Support

With a product like Carbonite which caters to inexperienced computer users ease of use is essential and when all else fails support needs to be in place to help out the user. I tried contacting Carbonite support in regards to the low speeds I experienced and got only a reference to their FAQ back after days of waiting. In spite of providing them with a very detailed description of my setup, connection speeds, etc. they felt that providing me with their FAQ was the best course of action. Of course I'd already read their FAQ and concluded that the problem wasn't on my end of the line. Disappointing to say the least. No followups to ensure that the problem was fixing happened either.

Issues

While testing Carbonite I came across a weird issue with iTunes where my iTunes Library files was corrupted twice after installing Carbonite. I've never experienced this kind of problem with iTunes before not even running on the Windows Vista betas. I can't say for sure that there is a problem with Carbonite and iTunes; all I can say is that since uninstalling Carbonite I have not experienced the problem again.

Summary: Carbonite 1 / Mozy 3

It's not everyday that you get the chance to sit down with one of the .NET rock stars and pick their brain. Nonetheless that's exactly the chance we got this week when Patrick Tisseghem swung by the office for a beer and a chat. We've been wanting to hire Patrick and U2U for a training engagement for a while but our schedules just didn't fit. Luckily circumstances meant that Patrick was in Aarhus for another engagement and offered to swing by the office if we'd provide some beer for the session. Naturally we did :)

We first learned about Patrick and U2U at TechEd 2006 in Barcelona where he gave a talk on creating a CMS site with SharePoint Server 2007 (a publishing site in MOSS terminology). We were very impressed with the demo as he got a site up and running in 45 minutes with nothing but the SharePoint Designer, no Visual Studio needed. Very convincing. You can read my reaction to his session in my post entitled TechEd 2006: Web Content Management and Microsoft Office SharePoint 2007. The other thing I remember very clearly from that session is that I've never encountered a living person who said "Microsoft Office SharePoint Server 2007" so many times in such a short time span :)

Back to our session with Patrick. There was no formal plan for our chat and what a great thing that turned out to be. Basically we spent three hours talking about our ideas and how we envisioned them implemented in SharePoint. He in turn let us know how we could go about leveraging the platform. Great stuff and making it even better was the informal tone. Patrick is a very knowledgeable person when it comes to SharePoint and the .NET platform in general but what made his views even more convincing was the fact that he had a very ... lets say realistic image of what SharePoint is and what it isn't. I was actually surprised that he had such a balanced view of the product, he even went as far as calling some of the features "samples" which is right on the mark to my mind.

Of course I couldn't help but ask about the future for SharePoint to which Patrick had a very interesting answer: Basically we'll see even more functionality moved to SharePoint making it the "one" platform for the future. Now I got to admit that I do have a problem with the "the can be only one" way of thinking. I'm worried that making SharePoint the Swiss-army knife of web software will dilute the value of the product. When trying to be all encompassing you are very likely to miss the mark by a huge margin and not do any of the areas well.

Also SharePoint is a hugely expensive product at this point and rightly so: It addresses a wide audience. My fear is that SharePoint will become prohibitively expensive to a market such as the Danish where we're dealing with many small to medium sized businesses and not very many large companies. From my point of view markets are going to open up for ISVs who target these markets because Microsoft seems oblivious to these markets. Of course this can be remedied by licensing so it will be very interesting to see what Microsoft does next.

Finally I'm concerned that complexity will bog down developers with nitty gritty details, abstraction this, abstraction that. SharePoint is a complex product as it is, adding more stuff to the product will naturally need to be accommodated by more complex architectural solutions which we need to understand to be able to go in a customize according to the wishes of our customers. In this day and age where agile methodologies are all the rage we need to select products which support this way of thinking and SharePoint is rapidly moving away from this.

Enough about me ranting about SharePoint. I like the product make no mistake about it but I do think that solution providers need a broader perspective on the standard platforms available out there in order to be successful. This has always been the case but in a market such as the Danish this is even more true.

We covered a lot of ground and even got around to discussing Silverlight and AJAX. Silverlight has been announced to the public recently and of course we were eager to know more about it. Now I have to admit that I didn't particularly see Silverlight as something relevant to us since we're focused on doing tight backend solutions. Designers provide the layouts and UIs we need and we implement them. That was true until I learned that Silverlight actually includes a small version of the .NET Framework running on the client. Version 1.0 doesn't have this but the alpha of version 1.1 does and with it I see a point to actually investing time and effort in learning about the technology. Simply put we're now able to do web UI running on the client in C# and VB both an exciting and slightly confusing prospect because we have some overlap with AJAX functionality wise. Another thing for my todo list I guess :)

We have a guy coming on soon who has changed career path from designer to .NET; I think I need to turn him loose on Silverlight and see what he can do :)

Our sales guy needed some Commerce Server picture and did what everybody does: Search Google Images. Little did he know ... :)

Google Image Search for "Commerce Server"

I just finished reading Eric Sink on the Business of Software yesterday and I was actually annoyed when I ran out of pages: I wanted to know even more! :) The premise for the book is Eric Sinc's work with SourceGear a company he founded. SourceGear which develops the source control system called Vault, an alternative to Visual Source Safe.The content of the book is made up from blog posts and MSDN articles he did on running a MicroISV.

In his own words SourceGear is, "a boring but profitable small ISV selling developer tool", and that is basically what you get from the book: Not boring content but solid practical guidance on how to go about areas developers normally don't think about (or even care that much about). He covers areas such as Marketing, Accounting, Hiring, Strategy, Pricing, and Sales.

Initially I was skeptical to the notion of turning blog posts into a book, much like Joel Spolsky did with his books. Let me just tell you right off that bat that the model actually works and it works well. Basically you get everything presented in a coherent manner which makes the thoughts and point so much more accessible than what you find in a blog. Reading blogs often gives you a disjointed look into the person due to the fact that most of us usually reads lots of blogs on a daily basis which makes it hard to separate the content. Not so with the book. Also there's still something to be said about sitting back on the couch and enjoying an hour with some good old fashioned paper :)

By no means is this a comprehensive work to allow you to run a business. What it does give you is an insight into some of the hard learned lessons that Eric Sink has been taught over the years by running SourceGear. It's accessible and has a humorous tone which often lacks in books for our kind of audience.

Eric's style of writing is engaging and interesting and I even got the feeling that he learned a thing or two along the way making it even more compelling to know what happens on the next page.

It's not a huge read so I would suggest that you read if you find yourself at all interested in other aspects of development than just writing the code, doing solution architecture, or even infrastructure architecture. For me it provided a nice additional insight into the workings of my surroundings here at the office and a fresh perspective on why some things work the way they do.

For you guys here at Vertica reading this: The book is on the book shelf if you want more ;)

Working on a particular data intensive project has thought me a lot about SQL optimization both about structure optimization and stored procedure optimizations. We all know about introducing redundancy and indexes to improve read performance but I've picked up a couple of tricks from our resident SQL expert I thought I'd pass along. Please bear in mind that these are only my observations of how our systems behaves your mileage may vary.

Main Problem Area

For our web solutions the performance issues we generally encounter with an application are usually related to the data tier. I can recall maybe two instances where the application tier was the culprit otherwise always the data tier. When diagnosing performance issues you can save yourself a lot of time by eliminating the frontend and going directly to the database. Also determining whether you have a problem in the data tier is pretty straight forward using the Profiling tools of SQL Server. Granted my experience with Windows application is limited but I would imagine that the likelihood of more issues related to memory and CPU usage cropping up is higher but you'll probably still need to deal with some kind of database if you're doing anything in the business.

Temporary Tables / Table Variables

My number one tip for performance optimization when dealing with large amounts of data is to try to limit the number of rows you're dealing with. It's a nobrainer using filters to do so but my .NET brain needed some time to get used to the idea of doing it using temporary tables and tables variables. For some reason I had a hard time dealing with the fact that I should pull data out of a comfortable data structure and stick into a data variable somewhere; maybe because I'm used to avoiding transferring a lot of data across the database boundary into my application. Keeping in mind that we don't actually move the data outside the database and only store the temporary tables for the duration of the execution of our stored procedure it makes a lot of sense to pick out the rows we need and only work on the smaller set.

I had a case yesterday where a query would execute for almost a solid two minutes, actually 1 minute and 40 seconds. The case was a view joined onto itself to do some rolling calculations. By extracting my set from the view once and placing it into a table variable I was able to squeeze execution down to  6 seconds. That's an improvement in the order of 16x people! :)

Default Parameter Values vs. Execution Plans

An interesting case came up a while back where when we were trying to figure out why the users were experiencing long execution times when doing a particular operation. However much we tried we were unsuccessful at reproducing the problem. Whenever we'd run the procedure the performance was great and very much in line with what we were expecting to see. Doing the operation from the interface ourselves did yield the same lack of performance. Basically we were seeing great performance when executing the procedure directly in Management Studio and poor performance when execution the same procedure from the application UI.

Eventually we figured it out: It turned out that the stored procedure had some default parameter values set for a couple of optional search parameters. The application UI would not pass these parameters to the stored procedure and SQL Server would select an execution plan optimized for the default scenario. We were passing all the parameters to the procedure in Management Studio and were causing SQL Server to select a more optimized execution plan thus giving us the better performance and the impression that "something was happening between ASP.NET and SQL Server".

Removing the default parameter values and always passing in the same number of parameters fixed the issue. Probably more a gotcha that anything else :)

WITH RECOMPILE

Ah WITH RECOMPILE, I remember it fondly from some years ago when I was studying for the SQL Server 2000 exam and I thought to myself, "I'm NEVER going to need that". Well it turns out that I did need it and the scenario was a text book example too. Basically we have an application in production which visualizes work load for an entire week. When the week starts out the users start scheduling that work for individual work days attaching a date and the actual work needing to be handled. Basically the result of this is that the dataset Monday morning is vastly different from the dataset Tuesday when most of the work is scheduled and good to go.

The main take away from this is this: When your datasets change a lot from execution to execution you should look into adding WITH RECOMPILE to stored procedures working on the dataset.

IF Statements

Yes IF statements! The rule of thumb is that they're evil. Ok maybe not but I've seen a couple of instances where the IF statement actually affected performance in a negative way because SQL Server was unable to select the best execution plan. Also in most cases you can embed the IF logic in a single SQL statement using the where clause like so

Redundancy

I know that redundancy is a pretty well known technique but I still want to mention it because you can gain massive performance increased by sprinkling a little bit of redundancy into your schemas. The basic idea is to avoid doing calculation by storing the result but course this introduces the possible of data which contradicts each other so tread carefully.

For this one we had data structures containing status history for inventory. For each item we knew when it had been set to a specific status so we just went ahead and used that existing knowledge to determine inventory status for a particular item. Not what I would call a speed demon once we started adding aprox. 150,000 items to the inventory. Simply adding the current status code to the item gave us many times the performance we were seeing and it scales well.

VIEW

Views are a great way of simplifying queries but unfortunately we can't use filtering in many cases so we end up with a lot of views assembled from various table. Before we can limit the resultset SQL Server will have to give us back the entire thing which isn't a very cost effective way of doing things. We tried to create very general purpose views but that just got us in the situation I just described so instead we went with more specific views with lots more filtering inside them. Alternately simply move the results from the view into a temporary store which also gave us some great results.

In many cases we need to join a view onto itself in order to do rolling summations to determine what data is available in the future. Basically a kind of what if calculation without actually going ahead and updating anything. This is a very intensive process where the general purpose views needed to be assembled a couple of times before we could get our results back. Limiting the resultset was the only option to actually make this stuff perform at acceptable levels.

Finally

This is not a very scientific SQL optimization post nor is it intended to be. You could probably pick apart my observations and start talking about the what, the why, exclusive lock this, dirty read that, etc. but instead think of it as a more practical look at SQL optimization.I bring you this post with some examples because you might have an easier time determining what action to take if you face situation similar to the ones I just described.

For me starting out with a SQL optimization is pretty vexing because looking at the SQL statement reveals nothing. I really need to get into the meat of things before the possibilities start to reveal themselves. Once I pry open the statement points of issue blossom before my very eyes to reveal where I can get my results. The moment when I see a query speed up from minutes to mere seconds is probably the most satisfying part of my job. Quick wins don't come along very often in software development but SQL affords some of the best ones :)

Well it's been out for a while but I was having trouble finding out what is new in this release from the main site. Jon Galloway has the lowdown on that one:

• Multiple Database support! Now you can generate a full DAL for as many databases as you like.
• Enterprise Library 3.0 Support. Just added this in and it works nicely.
• All-new command-line tool. You can now use our command-line tool (called SubCommander) to do all kinds of fun things, like generate your code and version your database. You can even hook into it using Visual Studio's External Tools - this will allow you to generate your project's code with the click of a button, and it will use your project settings (look for a blog post on Rob's blog).
• Improved querying. You can now use our Query tool to run OR's, IN's, and aggregates. You can even type what you want to see:
  IDataReader rdr = new SubSonic.Query("Products").WHERE("CategoryID = 5").ExecuteReader();
  We've also renamed many of our methods (well, we've added aliases) to make the query more readable. You can now use WHERE, AND, OR, IN, BETWEEN_AND, etc. to make your calls that much more readable.
• New Controls. You can now use our Smart Dropdown, which loads itself:
  <subsonic:DropDown id=mySmarty runat=server tablename="categories" />
  You can also use our new ManyToMany Checkbox list helper to both list and save information for many to many relationships:
  <subsonic: ManyManyList id=myList runat=server MapTableName="Product_Category_Map" PrimaryTableName="Products" PrimaryKeyValue="1" ForeignTableName="Categories" />
• A new AutoScaffold page that you can drop right into your project to admin all your tables. This thing reads your tables and creates scaffolds for you automagically (thanks Eric!).
• A completely reworked code-generation system that uses an ASP-style templating system. You can now code up your templates like you would an ASP page (also just like CodeSmith). Test them in your web site to make sure they work, then replace (or add to) the bits that get generated at runtime. You can override our templates by specifying a template directory in the web.config:
  <SubSonicService defaultProvider="Northwind" fixPluralClassNames="false" templateDirectory="D:\PathToMyTemplates">
• Regular Expression Naming Engine. If you don't like what our convention is, then you can use your own with some simple regex. Pass a simple string, or a dictionary replacement and all of your tables/views will be renamed as you specify.
• Query Inspection. Want to know what's happening to your query, and how long it's taking? You can simply use the new Inspect() method which outputs the results and statistics to HTML for you to review.
• Improved Trace/Debug. We've added tracing to (almost) every facet of SubSonic, so if you turn tracing on you can see what SubSonic's trying to do. We're always adding to this and if you see something we've missed, let us know :).
• Improved Scaffolding. Your foreign key values are now pulled into the GridView and displayed instead of their keys. Eric figured this one out - go dude!

Do not play this game at work, if you have a deadline, or if you need to go to the toilet real bad. It will suck up your time like you would have never thought possible.

"A live action puzzle game written for fun in flash, based on a Warcraft 3 custom map. You have to stop your enemies, or 'creeps', from travelling all the way across the screen. Tower pieces can be purchased and placed on the map to kill the creeps before they make it across."

Desktop Tower Defense

I got turned on to Google Co-op when reading a post on some blog I can't remember the name of off the top of my head but I didn't really get until later when I got around to listening to the Dan Appleman episode on .NET Rocks.

The problem with my first introduction to Google Co-op was that it only searched a single site. Now what Dan has done is basically to add multiple sites to his custom search and thereby he has create SearchDotNet an engine searching only quality .NET sites for content with Dan Appleman as a kind of editor-in-chief. Check out his post SearchDotNet.com – Google custom search for .NET developers.

This really opened my eyes to the possibilities of Google Co-op. I've been toying with the idea of creating a couple of sites with the main purpose of aggregating great content but I never got around to it due to time constraints. Now thanks to Google I can create the sites easily and I can do with very little HTML.

First thing everybody should do is to integration Google search into their blogs if they haven't already done so and then go ahead and start creating searches for their favorite sites. I know I have a couple I visit when I'm trying to solve some kind of problem.

Great news to the people annoyed by the fact that Team System isn't available to everyone: Some of the unit testing features from Team System will start appearing in Visual Studio Professional in the Orcas release. Also Microsoft is looking into adding more stuff from Team System into Visual Studio so be sure to voice your opinion on that.

"Due to popular demand we have decided to add the majority of the unit testing features of Team System to the Pro Sku of Visual Studio.  With the release of Orcas, the support for authoring, generating and running unit tests, will become available to the users of the Pro Sku of Visual Studio. Pro customers will also take advantage of the some of the unit testing improvements we have added into Orcas"

[Unit Testing Trickling into Pro!]

I'm still thinking that we need to see much more especially with projects like TestDriven.NET and NCover providing so much more. Also Team Foundation Server is a big part of the equation. Of course we have the ability to duck tape together an open source solution which does approximately the same thing. At this point I would even argue that an open source solution would be better but taking the future into consideration the open source solution will never be able to keep up with Team System simply due to the fact that the one thing Microsoft does extremely well is product integration. They have all the basics down and they have been moving on the round two for some time now.

To create awareness of Mozilla Develop Center, Mozilla Corporation has created a very cute desktop background for us. Check it out. I just had to add it as my own desktop image :)

Get larger versions of the background image.

The final release of Enterprise Library 3.0 is upon us. I've have been looking forward to this release for some time and the timing couldn't be better as I'm currently architecting two new solutions which will benefit significantly of the new features offered by EntLib 3.0 such as the Validation Block also the Policy Injection Block looks very exciting. Can't wait to get my hands dirty with those two. Of course the ever popular logging and data access blocks are still included, using those is just a no brainer as far as I'm concerned.

Here's what program manager Tom Hollander has to say:

"If you've been keeping up with the Community Technology Preview releases, there shouldn't be anything too surprising in the final release, although the overall quality should be much higher. All of the major new features have already been described on this blog or in other places, but here is a quick summary of the most significant changes"

[Just Released! Enterprise Library 3.0 - April 2007]

Download Enterprise Library 3.0 - April 2007