AJ's blog

August 14, 2011

RIA with Silverlight–The Business Perspective

If you read this, chances are that you are a developer and that you like Silverlight. And why not? Exciting platform, great features, outstanding tooling. But! If you’re a corporate developer, have you sold it to your management yet? If not, this post is for you.

Silverlight is for RIA, and the domain of RIA applications is largely intranet or closed/controlled extranet user groups. This again is what is usually found in larger enterprise companies. Companies that usually have a vested interest in controlling their environment. And in terms of bringing software into production and of operations and maintenance afterwards, every new platform is one platform to many.

So, the odd developer comes along and talks about this great new technology. Does the management care? Probably not. What does it care about? Simple. Money! Money, as in costs for deployment and user support, hardware and licenses to get the stuff up and running, operations and developer training, maintenance. And money as in savings in the respective areas and – the cornerstone, as the business usually pays the bill – impact on the business. All usually subsumed under the term ROI.

About a year ago, I finished an analysis looking into RIA with Silverlight, conducted for a major customer. Not from the point of view of the developer, but that of business people, operations, and IT management:

So, let’s look briefly at each aspect…

User/Business perspective…

The business doesn’t exactly care for the platform Silverlight itself; it cares for its business benefits. Benefits as in improved user experience, streamlined business workflows, office integration, and so on. And since we had some lighthouse projects with Silverlight we were able to collect some customers’ voices:

“This [streamlining with Silverlight] would reduce a [...] business process [...] from ~10 min to less than a minute.”

“Advanced user experience of Silverlight UI helps raising acceptance of new CRM system in business units”

“I was very impressed of the prototype implementation […] with Silverlight 3. Having analyzed the benefits of this technology I came to the conclusion that I want the […] development team to start using Silverlight as soon as possible. [...]”

This is also confirmed by the typical research companies, like Gartner or Forrester:

“Firms that measure the business impact of their RIAs say that rich applications meet or exceed their goals” (Forrester)

Operations perspective…

In production, the benefit of Silverlight applications (compared with respective conventional web based applications) is reduced server and network utilization.

For example, we had a (small but none-trivial) reference application at our disposal, which was implemented in ASP.NET as well as Silverlight (as part of an analysis to check the feasibility of Silverlight for LOB applications). We measured a particular use case with both implementations – starting the application and going through 10 steps, including navigation, searches, and selections. Both applications were used after a warm-up phase, meaning that the .xap file, as well as images and other static files had already been cached.

The particular numbers don’t matter, what matters is the difference between the amount of data that has been exchanged for each step (in case of navigations none at all for Silverlight). For the single steps:

And accumulated over time:

A ratio of roughly a tenth of the network utilization is quite some achievement – considering that the Silverlight application wasn’t even optimized to use local session state and caching, it should be even higher.

This should have a direct impact on the number of machines you need in your web farm. Add the fact that session state management on the client drastically reduces the demand for ASP.NET session state – usually realized with a SQL Server (Cluster) – there is yet another entry on the savings list.

On the down side is the deployment of the Silverlight plugin. For managed clients – especially if outsourcing the infrastructure comes into play – this may very well become a showstopper.

IT Management perspective…

With respect to development and maintenance, what IT Management should care about includes things like ability to deliver the business demands, development productivity, bug rates in production, costs for developer training, and so on.

Actually all areas in which Silverlight can shine, compared with other RIA technologies, and with the typical mix of web technologies as well:

  • Rich, consistent, homogenous platform
    • .NET Framework (client and server), Visual Studio, Debugger, C#
    • Reduced technology mix, less technology gaps, less broad skill demands
  • Improved code correctness and quality…
    • compiler checks, unit testing, code coverage, debugging, static code analysis, in-source-documentation, …
  • Improved architecture and code
    • Clean concepts, coding patterns, clear separation of client code, lead to better architectures
    • Powerful abstractions lead to less code (up to 50% in one project), less complexity, less errors

Customers’ voices in this area:

“between our desktop app and the website, we estimate 50% re-use of code”

“a .NET developer can pretty much be dropped into a SL project. […] This is a huge deal […]”

“As alternative for Silverlight we considered Flash. […] only Silverlight could provide a consistent development platform (.NET/C#). […]”

 

Conclusion…

Taking all this together, and considering that enterprise companies usually have the tooling and test environments (well…) readily available, this all adds up to something like the following bill:

RIA Return on Invest

Whether the bill looks the same for your company or for one particular project, of course, depends on many things. Especially nowadays with all the hubbub around HTML5 and mobile applications (without any relevant Silverlight support). But if RIA is what you need, the Silverlight will quite often yield far more benefits than any other option.

Still, you need to do your own evaluation. However, I hope to have given you some hints on what you might focus on, if you want to sell technology to the people who make platform decisions in your company.

The actual analysis was fairly detailed and customer specific. But we also prepared a neutralized/anonymized version, which we just made available for download (pdf). (Also directly at SDX.)

That’s all for now folks,
AJ.NET

kick it on DotNetKicks.com

June 3, 2010

MCPD – Enterprise Application Developer 3.5

Filed under: .NET, .NET Framework, Software Developers, WCF — ajdotnet @ 4:43 pm

I finally managed to upgrade my MCPD/EA from .NET 2.0 to .NET 3.5. And since my last post on the topic is still requested relatively frequently, I thought an update might be due.

Again, I took the upgrade exams, but since they distinguish the topics pretty well (each topic is covered in a separated section during the questionnaire and has to be completed before moving on to the next), the assessments should be valid for single exams as well – even if the single exams may shift emphasis somewhat.

Logistics

The upgrade is available to those who have the MCPD/EA on .NET 2.0. (This implies that there is no direct upgrade from the older MCSD, sorry Daniel.)

There are two exams that cover the ground of 5 single exams:

  • Exam 70-568: UPGRADE: Transition your MCPD Enterprise Application Developer Skills to MCPD Enterprise Application Developer 3.5, Part 1
  • Exam 70-569: UPGRADE: Transition your MCPD Enterprise Application Developer Skills to MCPD Enterprise Application Developer 3.5, Part 2

You can learn more about these and other upgrades here.

Part 1 includes the following three parts and respective certifications:

  • Microsoft Certified Technology Specialist: .NET Framework 3.5, ADO.NET Applications
  • Microsoft Certified Technology Specialist: .NET Framework 3.5, ASP.NET Applications
  • Microsoft Certified Technology Specialist: .NET Framework 3.5, Windows Forms Applications

Part 2 adds another Technology Specialist part and the “Designing and Developing” part that concludes the MCPD:

  • Microsoft Certified Technology Specialist: .NET Framework 3.5, Windows Communication Foundation Applications
  • Designing and Developing Enterprise Applications Using the Microsoft .NET Framework 3.5

All of which concludes the “Microsoft Certified Professional Developer: Enterprise Application Developer 3.5”

More information about single exams is available here.

The Exam Parts

While the last upgrade included a major remapping of the certification scheme, this one is merely keeping up with technological changes. The single parts vary in the amount of change they have undergone and the degree to which they can be considered being up-to-date.

  • The ASP.NET part was no big surprise, containing additional questions for ASP.NET AJAX. I’ll call that “reasonably up-to-date”.
     
  • The WinForms part hasn’t really changed its content at all. Hardly surprising, since WinForms itself haven’t changed. What has changed with .NET 3.5 though, is the introduction of WPF, a whole new UI technology. WPF is completely missing from the 3.5 exams. Hence, seen as Windows development part, this can hardly be called up-to-date.
     
  • The ADO.NET part. Well. I would call that one “unreasonably up-to-date”. It is still mainly focused on DataSets in great detail. Now, I can understand that LINQ2SQL doesn’t qualify for an exam with ADO.NET Entity Framework around the corner. But still! DataSets have gone out-off-fashion even before LINQ2SQL and I really had to scratch some layers of dust from that knowledge.
     
  • The WCF Part is probably the most advanced. This used to be the “Building Services” exam, covering remoting and WSE, but those topics are completely gone. This exam is solely focused on WCF, and you have to have a solid understanding about hosting options, configuration, and security to pass this exam.
     
  • Designing and Developing hasn’t changed in style, but it incorporates current technologies. Unlike the respective other exams, it actually addresses WPF and ADO.NET Entity Framework as technology choices.

Generally speaking, each and every of these exam parts can be solved by people with reasonable experience in the topic. That should cover ASP.NET, WinForms, and “Designing and Developing” quite nicely. Whether you put ADO.NET in that bag is up to you, but dusting the DataSet knowledge shouldn’t be a showstopper anyway. The only part that is more demanding is the WCF part. Be aware that the questions cover WCF capabilities in general; having created one or the other WCF service or reference in Visual Studio and twisted the configuration somewhat is certainly not going to get you through this exam.

Looking Ahead

.NET 4.0 and Visual Studio 2010 have been released weeks ago and usually it takes Microsoft about a year to prepare the next increment of certifications. So the current 3.5 certification is going to stay valuable for quite some time. Still, a little crystal ball gazing might be fun:

In terms of technologies, .NET 4.0 does not only contain brand new stuff, but also incorporates some technologies that have been around for some time, but not formally part of the .NET Framework. Together with the gaps noted above we have the following list of suspects:

  • In the ADO.NET area:
    • LINQ 2 SQL (leaving it out so far may have been lack of time rather than a deliberate choice after all)
    • ADO.NET Entity Framework
  • In the ASP.NET area:
    • ASP.NET MVC
    • ASP.NET Dynamic Data
    • perhaps jQuery
  • In the Windows Development area:
    • WPF
    • Parallel Extensions (PLINQ, TPL)
  • In the Designing and Developing area:
    • Visual Studio Architecture features
    • Testing features

I do hope that we will see a substitution of content in the ADO.NET area. Please Microsoft, no more DataSets! And it wouldn’t be a surprise at all if Microsoft dropped WinForms in favor of WPF, adding a little parallel stuff. ASP.NET on the other hand will certainly grow, as ASP.NET WebForms are not substituted but complemented by the new additions. The new Visual Studio features may result in in just one or the other additional question. But then, they might not, as they are not universally available.

Anyway, this is reasonably manageable for those people trying to keep their certificates up-to-date. For people trying to get their first certification it certainly adds up; but then, they may start with Technology Specialist or the more specific MCPD certificates and build on that.

What parts didn’t I mention? Silverlight is obviously missing from the list. And don’t tell me, it’s not relevant for enterprise development, for it certainly is! But I still can live with that, for it isn’t formally part of the .NET Framework and you have to draw the line somewhere. The same applies to Team Foundation Server.

On the other hand, how can you call yourself a certified “Enterprise Application Developer”, not knowing about TFS or Silverlight?

That’s all for now folks,
AJ.NET

September 27, 2009

PDC 08 – Recap

Filed under: Fun, Miscellaneous, Software Developers — ajdotnet @ 2:13 pm

So, PDC09 is not far away. But, sadly, I’m not going to make it this time.

I had thought for a longer time about posting some pictures from PDC08 as kind of a retrospective – but you know how it is, other things coming up, not much time, well.

And then this nut showed up, hijacked my blog and posted this nonsense about some crime that should have happened last PDC? Come on! Here’s the list of what I can only call a real good example of real dumb accusations:

Let me stress that again: There is no proof whatsoever to this story! As I see it, all these accusations are groundless, all people addressed are innocent victims. So, please, do not accuse anyone of installing LINUX on somebody’s machine if you don’t have proof. Much less on the grounds of this story! ;-)

All that remains for me is some complementary pictures:

 Windows Azure was THE big announcement at PDC08, and it’s going to be one of the major topic for PDC09. Microsoft really has something here that might change a lot in the long run. If they remove the two road blockers, that is: One, they need to make this infrastructure available for enterprise customers to lower the barrier to entry. Two, they need to do what they always did: attract developers and create a prosperous community of enthusiasts around Azure. To achieve that, they need to offer the platform for free. All with reasonable constraints of course. And this is just my opinion.   

 

Oslo, and M, was another major topic. But I can’t help thinking that this was mainly because it was the “Don Box-Topic”. It’s been relatively quiet around it since then, and unfortunately this also includes the somewhat related “Dublin”, which I think is a very important step in the services area.

 

Hands-On-Labs was as always the best way to lay your hands on the stuff you heard about for the first time just this very morning. If you had the time.

 

Talking about the location, the first and the last thing you saw about PDC was L.A. …

 

the Convention Center…

 

… and if you didn’t use the shuttle system, it was probably Pico Station, featuring a great view at Downtown.

However, I have been attending 3 PDCs so far, all of them in L.A. I’d really like to see some other city next time? (Microsoft, I could give you a list of areas in the U.S., I haven’t visited so far ;-) ).

So much for my little retrospective. CU at PDCnext… .

That’s all for now folks,
AJ.NET

kick it on DotNetKicks.com

August 1, 2009

Posting Guards: Guard Classes explained

Filed under: .NET, .NET Framework, C#, Software Developers, Software Development — ajdotnet @ 11:53 am

A colleague writing an article about code contracts recently asked me about some useful links about Guard classes, something I have been advertising for some time. ‘Sure’, I thought — and was taught otherwise.

There are certainly references to Guard classes. For example this one hinting at an implementation from MS Patterns&Practices, although an outdated one and no online documentation available. And this one talking about using Guard classes in combination with LINQ.

Still, nothing explaining the concept, nothing on Wikipedia, nothing anywhere. Can it be that Guard classes are far too simple to be worth mentioning? I don’t think so, because while the implementation certainly is simple, I wouldn’t have had to explain the concept that often if that where the reason.

So, what are Guard classes?

First of all, let’s make sure we’re taking about the same thing. Judging from the name, Guard classes are about guarding against something, but this can mean just about anything. Actually I found two incarnations: One, guarding against concurrent access, i.e. some kind of locks. And two, guarding against the passing of invalid parameters into a method – which is what this post is about. (You may have come across Guard classes by the name of ArgChecker, ArgumentHelper, or something similar, or as single helper methods appearing where they don’t belong.)

Straight to the matter: Have a look at the following example:

IEnumerable<Employee> FindCustomers1(Guid companyID, EmployeeFilter filter)
{
    IEnumerable<Employee> result= null;
    if (companyID!=Guid.Empty) // only hit DB if we have to
    {
        using (DatabaseDataContext context = new DatabaseDataContext())
        {
            result = from e in context.Employees
                         where e.FirstName == filter.FirstName && e.LastName == filter.LastName
                         select e;
        }
    }
    return result;
}

While probably doing its job properly, this method isn’t exactly defensive. The issue should be apparent: What happens if the caller passed null for the filter? Not that he is expected to, quite the opposite, but anyway?

You could try to handle that in code and it would probably lead to some ugly code, as Abby rightly complained about. And to be blunt, that approach is wrong right from the start. If caller is not supposed to pass null values into our method, why should we clutter our code with conditions that aren’t supposed to happen anyway.

The better solution is getting rid of unwanted null values by making them illegal. And not only by politely asking in the documentation (and please RTFM); rather make it unmistakably apparent, punish any violation, make it impossible to bypass. In other words, check the parameter and throw an ArgumentException.

On second thought, the same reasoning applies also to the properties of our filter object, so we may end up writing the following code, prone to becomes ugly itself:

if (filter == null)
    throw new ArgumentNullException(„filter“);
if (filter.FirstName == null)
    throw new ArgumentNullException(„filter.FirstName“);
if (filter.LastName == null)
    throw new ArgumentNullException(„filter.LastName“);

And we actually did expect some content, so why not go ahead and do some other vanity checks:

if (filter.FirstName == „“)
    throw new ArgumentNullException(„filter.FirstName“);
if (filter.LastName == „“)
    throw new ArgumentNullException(„filter.LastName“);

But wait. An empty string is certainly not null. Unfortunately there is no StringEmptyArgumentException. Should we use an ArgumentOutOfRangeException? Or an InvalidOperationException, not an ArgumentException at all? Derive a new one? ArgumentOutOfRangeException may look nicely to me, but my team mate just settled with ArgumentNullException, which is at the least an inconsistency… .

And what the heck, this is way too much code for nothing anyway. And so we end up not checking our parameters and hoping for the best…?

Now consider this code:

IEnumerable<Employee> FindCustomers2(Guid companyID, EmployeeFilter filter)
{
    Guard.AssertNotNull(filter, „filter“);
    Guard.AssertNotEmpty(filter.FirstName, „filter.FirstName“);
    Guard.AssertNotEmpty(filter.LastName, „filter.LastName“);
    
    IEnumerable<Employee> result = null;
    if (companyID != Guid.Empty) // only hit DB if we have to
    {
        using (DatabaseDataContext context = new DatabaseDataContext())
        {
            result = from e in context.Employees
                         where e.FirstName == filter.FirstName && e.LastName == filter.LastName
                         select e;
        }
    }
    return result;
}

We just replaced 5 ugly conditions (10 LOC) with 3 calls. Let me rephrase that: We replaced a bunch of code detailing some implementation with 3 lines revealing the intention – way more readable. Wow! By centralizing the checks, we were able to provide more convenience (putting the null and string empty check in one method), we ensured consistent behavior, we improved the calling code considerably. And finally, we even added some more information to the exception, as you will see in the implementation:

[DebuggerStepThrough]
static public void AssertNotEmpty(string arg, string paramName)
{
    if (arg == null)
        throw new ArgumentNullException(paramName,
            „Argument ‚“ + (paramName ?? „<missing>“) + „‘ should not be NULL!“);

    if (arg.Length == 0)
        throw new ArgumentOutOfRangeException(paramName, arg,
            „Argument ‚“ + (paramName ?? „<missing>“) + „‘ should not be empty!“);
}

You don’t want to put that code in every method. But you do want to call the Guard method, don’t you?

Now that we have established what Guard classes are, let’s look at …

What are Guard classes about?

So far I introduced Guard classes form the coding level, merely as convenient helper classes. But there’s some broader meaning to them, namely in the following areas:

  • Contracting
  • Self preservation
  • Robustness

Contracting.

Contracting as a means has gained attention as important aspect of SOA service design, and also as development approach, namely design by contract. Contracting may be a major undertaking if it comes to SOA services, yet it can also be employed on a much lower level, designing the public interface of a class.

A little simplified, a contract of a class or service consists of

  • Functional contract: Specifies the operations, the semantics, required call sequences, runtime behavior, error conditions and behavior. Some of this may be expressed using code, some may not.
  • Data contract: Details parameters and return values, in other words the data that is exchanged during the operations. It defines the legal shape and values of the data, restricting it first by data types, secondly by additional demands, such as stating which parameter is mandatory, which content is required, and which values it is limited to beyond what the data type mandates. Also interdependencies between fields, and so on.
  • Some other aspects, but that’s not relevant right now.

As you see, the data contract goes beyond what the type system is capable to enforce. And Guard classes may serve as a means to express some of those aspects in code. Thus, Guard classes serve to enforce the contract of a class. And since these calls are that obvious, the code is self-documenting its preconditions quite nicely.

Note 1: Of course this makes only sense with regard to the technical contract that has to be met by the calling code, i.e. the developer, not a business contact that has to be met by the user. See my post on error handling and responsibilities for more on that topic.

Note 2: In terms of design by contract, Guard classes express the preconditions of an operation, something that has been baked into other programming languages right from the start.

Self preservation.

If you are working in a team, you inadvertedly have to work with other team members. Good ones, bad ones. And you may want to protect yourself against the later… .

Face it: If a NullReferenceException arises, the developer who wrote that particular code fragment – mayhap you – is held responsible (shot on first sight, so to speak). Even if it turned out later that the calling code passed some illegal null value… . Well, problem solved, but the stigma sticks, and anyone will only remember that the exception was thrown in your code.

How do you protect yourself against that? Guard classes. If the calling code just passed some crap data – despite you having told that guy what to pass a hundred times – just throw it right back into his face. That’s what ArgumentExceptions are for: complaining loud and clear about someone unduly misusing your code. Be the accusator, not the accused!

BTW: If you’re on the callers side and the method you just called returned a value, you may achieve something similar with Debug.Assert.

For example if you had written that code above, and I knew you to be a sloppy developer, I would certainly check the return value. And if I were as mean as you are sloppy, I would deliberately call it passing Guid.Empty as companyID. And if you dared to hand back that pesky null instead of an empty enumeration I would … .

Well, never mind. You‘re not sloppy, I‘m not mean, and I only put that bug in to stress the fact that Guard classes only work up the call chain, not down. (And yes, it’s a bug to return null in the above case. Methods supposed to be used in a LINQ context have to comply with LINQ demands, which are based on functional programming, which boils down to „no nulls please!“)

Robustness (or: making errors apparent).

Someone passed null or some other unwanted value to a method. There are actually two bad things that might happen:

  1. The application may crash further down.
  2. The wrong value may affect some data, but the application continues to work.

Number one may be bad, but at least the application crashes (which is a good thing!). Still, in this case Guard classes will help you identify the error far earlier, possibly avoiding a situation where you’ll have to hunt for the root cause of a problem. This should at least help diagnosing the problem.

Number two is far worse. It’s actually the very situation that you would want to avoid at any cost: A bug that stays unnoticed. A bug that occasionally produces wrong data, and that lingers around long enough to seriously affect data consistency to a degree that renders all data useless, because you have no way of knowing which data was affected and which not. Technically just a tiny slip, but in the long run these bugs are the most harmful.

To guard against these bugs Guard classes put up another safety net. Make sure the data coming in matches the specification.

Essentially both issues neatly demonstrate what „fail early, fails fast“ is about. Making errors apparent, prevent them from being obscured and from causing damage further down the line.

And that’s it…

Quite some task for a tiny little helper – but really worth the effort. Should you need some kick-start to employ this concept yourself, I posted the code for a Guard class here. Use it, enhance it, employ it to make you life easier.

HIH!

That’s all for now folks,
AJ.NET

kick it on DotNetKicks.com

July 26, 2009

Playing game…

We’ve defined the problem, we’ve established the rules, now we are ready to actually play the game. In other words: We will employ consistent and sufficient exception management in an ASP.NET application. No less!

Note: What follows is merely a sample implementation, just to drive the ball home, but it should turn my earlier musings into a little more practical advice. And since this is only meant to highlight the principle, not to provide a production ready solution, I’ll keep it simple.

Exception classes

First let’s try to classify our exceptions, depending on the supposed effect. This classification is sort of the driving force behind the strategy: who is responsible, or more to the point how to tell the responsible people…

A quick reminder on what our goal is:

  1. The user has to solve everything that results from his user user interaction in due process. This includes invalid input, insufficient permissions, or other business conditions that interrupt the current functionality.
  2. The administrator is responsible for infrastructure issues. This includes every call into the „outside world“, such as databases, services, file system. Issues here are manifold, including:
    • unavailable database or service
    • errors during some operation, e.g. unexpected a constraint violations
    • erroneous configuration
  3. Everything else: Bugs. Developer. You! ;-)

So consequently:

  • If the user is responsible, we just tell him on the page and let him choose what to do.
  • If the administrator is responsible, we show a general error page, telling the user that the application is not available, and of course we write some event log entry.
  • If the developer is responsible, we do the same, but we tell the administrator explicitly to forward the event log information to the development team.

I’ll spare you the boilerplate exception implementation details, suffice it to say that we have a UserIsResponsibleException for “user exceptions” and a AdminIsResponsibleException for “admin exceptions”, respectively. Names that wouldn’t make it into my production code, but that’s what their intention is, anyway. Oh, and add a ConcurrencyException exception for respective issues, they may or may not be user exceptions, depending on the context.

Now let’s put the handlers in place. These reside in the UI layer, and from there we can work through the architectural layers towards the databases.

User error handlers

User exceptions have to be handled on the page level. Whenever you call into some business layer method, say from a button click event handler, you wrap the call in a try/catch(respective exceptions). The exception is then presented to the user, and eaten, meaning it is not thrown along. Something like this:

protected void btnConcurrentUpdate_Click(object sender, EventArgs e)
{
    try
    {
        BusinessLogic bl = new BusinessLogic();
        bl.UpdateDatabaseWithConcurrencyIssue("some content");
        lblConcurrentUpdate.Text = "did it.";
    }
    catch (UserIsResponsibleException ex)
    {
        // just give feedback and we’re done
        PresentError(ex);
    }
    catch (ConcurrencyException ex)
    {
        // just give feedback and we’re done
        PresentError(ex);
    }
}

The way of presenting the error message is something that has to be decided per application. You could write the message to some status area on the page, you could send some java script to pop up a message box. You could even provide a validator that checks for a recent user exception and works with the validation summary for the feedback.

Statistics:

  • Exception classes: Write once.
  • Presentation: Write once.

The try/catch code is simple enough, yet you have to do it again and again, and should the need for another user exception arise (e.g. some user errors can be handled on the page, some others require leaving the page), the logic is spread across our pages. Thus it makes sense to factor the actual handling into a separate method, in a helper class, or a page base class. Now, catching specific exceptions is still part of our job and that cannot be factored out. To solve this we have three options:

  1. Build the exception hierarchy accordingly and catch the base class for user exceptions.
  2. Catch all exceptions and throw the unwanted exceptions on.
  3. Live with the problem.

I decided against option 1 because I didn’t want the concurrency exception to be derived from some user exception. Option 3 is what we wanted to avoid in the first place. So my solution is number 2, in the name of ease of use and maintenance, but at the expense of language supported exception handling features:

protected void btnConcurrentUpdate_Click(object sender, EventArgs e)
{
    try
    {
        BusinessLogic bl = new BusinessLogic();
        bl.UpdateDatabaseWithConcurrencyIssue("some content");
        lblConcurrentUpdate.Text = "did it.";
    }
    catch (Exception ex)
    {
        // since C# does not support exception filters, we need to catch
        // all exceptions for generic error handling; throw those on, that
        // have not been handled…
        if (!HandleUserExceptions(ex))
            throw;
    }
}

This is boilerplate and unspecific enough to copy&paste it around. The generic handler may look like this:

bool HandleUserExceptions(Exception ex)
{
    if (ex is ConcurrencyException)
    {
        // just give feedback and we’re done
        PresentError(ex);
        return true;
    }
    
    if (ex is UserIsResponsibleException)
    {
        // just give feedback and we’re done
        PresentError(ex);
        return true;
    }
    
    // other errors throw on…
    return false;
}

Statistics:

  • Handler method: Write once.
  • Catching exceptions: Every relevant location, boilerplate copy&paste

Please note that exception filters might make this a little less coarse, but this is one of the rare occasions, where a CLR feature is not available to C# programmers. Anyway, we just factored all logic into a centralized method which achieves exactly what we asked for.

But wait… . Wouldn’t the Page.Error event provide a better anchor? It’s already there and no need to add try/catch around every line of code. Look how nice:

protected void btnConcurrentUpdate_Click(object sender, EventArgs e)

    BusinessLogic bl = new BusinessLogic(); 
    bl.UpdateDatabaseWithConcurrencyIssue("some content"); 
    lblConcurrentUpdate.Text = "did it.";
}

protected override void OnError(EventArgs e)
{
    // raise events
    base.OnError(e);
    
    var ex = Server.GetLastError();
    
    // if we don’t clear the error, ASP.NET
    // will continue and call Application.Error
    if (HandleUserExceptions(ex))
        Server.ClearError();
}

Unfortunately at the time the error event is raised, the exception will already have disrupted the page life cycle and rendering has been broken. It may help in case you need a redirect, but not if the page should continue to work.

“Other” error handlers

So, the page handles user exceptions, what about admin and system exceptions? The page doesn’t care, so they wind up the chain and ASP.NET will eventually pass them to the application, via the Application.Error event, available in the global.asax. This is the “last chance exception handler” or LCEH for short.

One thing we did want to do was writing the exception into the event log. I’ll leave actually writing into the event log to you, just remember that creating an event log source requires respective privileges, e.g. running as admin. Just a minor deployment hurdle.

protected void Application_Error(object sender, EventArgs e)
{
    Exception ex = Server.GetLastError();
    
    // unpack the exception
    if (ex is HttpUnhandledException)
        ex = ex.InnerException;
    Session["exception"] = ex;
    
    // write detailed information to the event log…
    WriteInformationToEventLog(ex);
    
    // tell user something went wrong
    Server.Transfer("Error.aspx");
}

private void WriteInformationToEventLog(Exception ex)
{
    // an ID as reference for the user and the admin
    ex.Data["ErrorID"] = GenerateEasyToRememberErrorID();
    
    string whatToDoInformation;
    if (ex is AdminIsResponsibleException)
        whatToDoInformation = "Admin, you better took care of this!";
    else
        whatToDoInformation = "Admin, just sent the information to the developer!";
    
    // put as much information as possible into the eventlog,
    // e.g. write all exception properties into one big exception report,
    // including the Exception.Data dictionary and inner exceptions.
    // also include information about the request, the user, etc.

    …
}

As you can see, there are some details to keep in mind. One, the exception has been wrapped by the ASP.NET runtime, and that exception class doesn’t tell you anything worthwhile. Second, passing information to the error page may be a little tricky in cases where the request has not yet (or not at all) established the session state. The easier approach is to let ASP.NET handle the redirection, yet at the loss of information. Three, be careful regarding the requests: depending on the IIS version you may get only .aspx requests, or any request including static resources. Thus, with this simplistic implementation a missing image will send you to the error page depending on the IIS version and registered handlers.

I would also like to encourage you to spend some time on collecting information for the event log message. A simple ex.ToString() doesn’t reveal that much information if you look closely. You should invest some time to add information about the current request (URL, etc.), the user (name, permissions), and server state (app domain name, to detect recycling issues). And use reflection to get all information from the exception, especially including the Exception.Data dictionary. Also go recursively through the inner exceptions. The more information you provide, the better.

I also hinted at an error ID. It is probably a good idea to give the user some ID to refer to when he talks to the admin. This way, the admin will know exactly what to look for in the event log (especially in multiple event logs of a web farm).

Finally, the exception class tells us whether we should include an “Admin, do your job!” or “Kindly forward this issue to your fellow developer, brownies welcome, too.” message.

And by the way: This is the one and only location in our application that writes to the event log!

Statistics:

  • Global error handler: Write once.

Done. We have the handlers in place. Handlers for user exceptions, admin issues, and any other exception that might be thrown. Now we need to take care that error causes are mapped to the respective exception class.

Business layer

The business logic is being called from the UI layer, and calls into the data access layer. Thus it has to check parameters coming in, as well as data handed back from the data source.

Data coming in hast to comply with the contract of the business class, and I’d like to stress the fact that not every data is legal. Meaning the business logic is not expected to accept every crap the UI layer might want to pass. A search method that takes a filter class as parameter? No harm in defining this parameter mandatory and require the UI to pass an instance, even if it is empty. And a violation is not the user’s problem, it’s the programmer’s fault! In these cases we throw an ArgumentException, and that’s it. The LCEH will take care of the rest, blaming the UI developer that is.

public IEnumerable<Customer> SearchCustomer(SearchCustomerFilter filter)
{
    // preconditions have to be met by the calling code
    Guard.AssertNotNull(filter, "filter");
    
    // input values have to be provided by the user
    if (string.IsNullOrEmpty(filter.Name))
        throw new UserIsResponsibleException("provide a filter value");
    
    DatabaseAccess db = new DatabaseAccess();
    return db.SearchCustomer(filter);
}

Any issue with the content of the data, i.e. the information the user provides, is a user issue. Usually this includes data validations and if you follow the rule book you’ll have to repeat everything the UI already achieved via validators. Whether you do that or just provide additional checks, e.g. dependencies between fields, is something I leave to you. Anyway, throw a user exception if the validation fails.

Calling the data access layer may also result in thrown exceptions. Usually there is no need to catch them at all. Period.

Finally the data you get from the database may ask for some interruption. A customer database may tell you that this particular customer doesn’t pay his dues, thus no business with him. In this case you throw a user exception in order to interrupt the current business operation.

And that’s it.

Statistics:

  • Business validations and interruptions: As the business logic demands…

“Uhh, that’s it? And I used to spend so much time on error handlers in my business logic…”. Nice, isn’t it? Just business logic, no unrelated technical demand.

Data access layer

The data access layer is where many exceptions come from. If you call a data source (web service, database, file system, whatever), the call may fail because the service is unavailable. Or it may fail because the called system reports an error (constraint violation, sharing violation, …). It may time out. Some exceptions won’t require any special handling and you can leave them alone. Some need to be promoted to admin or user exceptions, or they may require special handling for some reason. You should  translate those exceptions to respective application exceptions. The reason is  that you don’t want to bleed classes specific to a data source technology (e.g. LINQ 2 SQL exceptions) into the upper layers, introducing undue dependencies.

Luckily all this tends to be as boilerplate as can be. Call/catch/translate. With always the same catches. So, again, we can provide a “once and for all” (at least per data source technology) exception handler, like this one for the ADO.NET Entity Framework:

public IEnumerable<Customer> SearchCustomer(SearchCustomerFilter filter)
{
    try
    {
        using (DatabaseContainer db = new DatabaseContainer())
        {
            //…
        }
    }
    catch (DataException ex)
    {
        TranslateException(ex);
        // any exception not translated is thrown on…
        throw;
    }
}

And the respective translation method:

static void TranslateException(Exception ex)
{
    // database not available, command execution errors, mapping errors, …
    if (ex is EntityException)
        throw new AdminIsResponsibleException("…", ex);
    
    // translate EF concurrency exception to application concurrency exception
    if (ex is OptimisticConcurrencyException)
        throw new ConcurrencyException("…", ex);
    
    // constraint violations, etc.
    // this may be caused by inconsistent data
    if (ex is UpdateException)
        throw new AdminIsResponsibleException("…", ex);
}

Statistics:

  • Translation method: Write once.
  • Catching exceptions: Every relevant location, boilerplate copy&paste

Conclusion:

Done. Let’s sum up the statistics:

  • Write once (at least partly reusable in other applications):
    • Application exception classes
    • Last chance exception handler (LCEH)
    • Handler method for user errors, including presenting the error
    • Exception translation method in DAL
  • Boilerplate copy&paste
    • Catching exceptions in the UI, call handler
    • Catching exceptions in the DAL, call translation
  • Application code that still needs a brain ;-)
    • Business validations and interruptions: As the business logic demands…

And this is all we have to do with regard to exception handling in our application. I’ll let that speak for itself… .

This post concludes this little series about the state of affairs regarding error handling in an application. I do not claim to have presented something particularly new or surprising. Rather the opposite, this or some similar strategy should be considered best practice. The reason to even start this series was the fact that I regularly encounter code that does exception handling … questionably. And I also regularly met people who look at exception handling only in the scope of the current code fragment, not in light of a broader approach that spans the application.

If you need further motivation to employ proper error handling in the first place, look here.
A little more on error management in long running, asynchronous scenarios can be found here.

That’s all for now folks,
AJ.NET

kick it on DotNetKicks.com

July 19, 2009

The Rules of the Game

With the last post I tried to establish the problem: The fact that exception handling needs more than try/catch. That it needs a proper exception management strategy, which should be part of the application architecture. And I paid special attention to the paradox demand that exception management should work when everything else fails.

Now, I cannot provide THE strategy, for there is no such thing. But I may present one feasibly approach that has served me well, quite a few times. I’ll simplify quite a bit, but it is nonetheless a complete approach that may serve as starting point for more complex demands.

OK, let’s play…

To define the playing field: The following is about classical n-Tier ASP.NET web applications, and I have decided to represent user induced issues, like business validation errors, as exceptions.

Some Groundwork

Exceptions are used for different error conditions, but also for non-erroneous situations, which leads to a surprising variety:

  • Exceptions being thrown as result to infrastructure issues. E.g. IOException, SecurityException. They may be caused by invalid configurations, by unavailable infrastructure resources, or for other reasons.
  • Exceptions representing errors reported by external services, like database constraint violations or SOAP exceptions.
  • Exceptions being thrown because the caller did not meet the expectations of the called code, e.g. a violation of a contract. (Usually ArgumentExceptions).
  • Exceptions being thrown because a developer’s lapse. NullReferenceException.
  • Exceptions being thrown to sidestep the current flow of control, e.g. HttpResponse.Redirect() throws a ThreadAbortException to prevent an ASP.NET page from further unnecessary processing.
  • Business validation errors. The business logic decides that – despite proper input validation – something is not quite as it should be. Not exactly an error in a technical sense, but the processing simply cannot go on.
  • User concurrency issues. Like some user wants to update some data that has been changed by someone else in the meantime. And “last one wins” may not be good enough.

Please note that while Basic Rule Number 1 for exception handling (no abuse of exceptions for non-exceptional conditions) is as valid as ever, there are certainly exceptions (no pun intended) to that rule. For example, aborting the request upon redirection certainly makes sense; (ab)using an exception to do so may violate said rule, yet it is far more efficient and convenient than any other alternative. The same may be true for business validation errors. Not exceptional, yet exceptions may be the most convenient way to accommodate them.

This is a (perhaps surprising) variety and may even be incomplete – actually that variety is a major contributor to the complexity of exception handling. And yet, it is only half of the equation. Exceptions have cause and effect, and I found that tackling exception handling from the opposite angle, the effect that is, has certain advantages.

But before we come to that, there’s a question that has to be answered…

What is the purpose, the intention of throwing exceptions? (Seriously!)

This is a question one should ask at least once, even so it may seem a bit superfluous. For the answer does not include some of the usually mentioned suspects: Preventing data inconsistencies? Let’s simplify a bit and state that this is the job of transactions. Gracefully shut down? An error happened, what do I care how the application goes down.

Actually it’s far simpler: An error happened! Something which the developer didn’t anticipate or couldn’t cope with. The primary purpose of exceptions is to communicate that situation. And exception handling in turn it is about notifying everyone involved in the appropriate way, telling them what they are supposed to do now, either to compensate the issue or to prevent it from happening again. In other words: It’s about responsibilities emerging out of this error.

And this is what my strategy revolves around: Responsibilities.

An Exception Handling Strategy

Back to cause and effect, and let’s start by the later one. More to the point, the desired effect, driven by what is actually asked for in case of an error (rather than what could be done). This will lead to a strategy that is more simple, easier to understand, and easier to implement.

As I mentioned ‘responsibilities’ the first question is kind of obvious:

Question #1: Who should be responsible for what?

This is no esoteric question aiming at pieces of architecture or code. Rather it’s asking which person has to shoulder the work (or the blame, if you like):

  1. The Administrator: He has to solve problems caused by infrastructure issues (such as invalid configuration entries, unavailable databases, and so on).
  2. The User himself: Let him deal with errors caused by invalid input, other users working simultaneously with the same data, or in some other way inherent to the intended usage of the product. RTFM!
  3. The developer (YOU!): Anything else falls back on the developer’s desk.

Obviously the last point is the inconvenient part: Every exception defaults to that category, unless the developer explicitly “tells” someone else that he is responsible. Let’s stress that: An error caused by an invalid configuration entry is the developer’s problem – unless he explicitly tells the administrator that he should take care of this.

So, now we are back where we started right at the beginning, at the developer’s desk. The difference is that the developer now has some specific goal: Lay the blame on someone else’s doorstep!

This is actually a very important point. Making someone else responsible is in the developer’s own interest. It helps him avoiding unnecessary and annoying future work. Quite motivating if you ask me.

This makes the next question obvious:

Question #2: How does the developer tell the user or the administrator that he is to take responsibility?

The answer is simple: Provide the necessary feedback, e.g. a message box or an event log entry; in more detail:

  • In case of a user error you tell him with a message box, a popup, within some message area or on a special “you did wrong!” page.
    You don’t tell the admin, because he doesn’t care for typos. Neither do you as developer.
  • In case of an infrastructure issue you show a standard message to the user, telling him to call the admin and come back later. You don’t want to include details, because they might include security relevant information, such as the always cited database connection string.
    Additionally you have to give the admin all information he needs, usually within the event log.

In both cases it’s important that you provide not only the information that an error has occurred (that much is obvious, isn’t it?). You need to include information on how to solve it. Otherwise it will eventually again be up to you, the developer, to solve the issue. That’s why it makes sense not to stop at “couldn’t save data” or “data source not available”. Rather invest some work – in your own self-interest – in providing “Customer data could not be save because it is locked by user ‘XYZ’” or “Database not available, connection string:=…”.

  • Remember the last case: In case of technical issues for which the developer is responsible, you still have to tell the user and the admin something. The user should again get some standard error page (not the ASP.NET error page!), the admin should get an event log entry that tells him to notify the developer. This entry should include as much information as possible for a post mortem diagnosis.

So, essentially this paragraph revolved around presenting an exception, at the UI level where the information leaves the area of your application code. It’s quite simple to implement that, because you can rely on centralized features: ASP.NET provides a global error handler that can be used to provide standard event log entries. Presenting user errors on a page should be boilerplate as well, the only distinction being whether the current page and data is still valid (just show a message box) or not (redirect to some business error page). Nice, easy, and no rocket science.

Of course in order to do this we need a means of distinguishing those three cases. For this, all you need is respective exception classes, two actually. One BlameTheUserException and another BlameTheAdminException. You don’t need a BlameTheDeveloperException, because he is already to blame for every other exception, no need to stress that ;-).

Question #3: Who to put exceptions to work… ?

If the developer’s goal is to blame someone else, then this goal can guide him answering the more detailed questions, such as, where in his code should he be catching exceptions (where not)? When should he throw exceptions (when not)? What about logging? And so on. Everything else follows as consequence…

First about throwing exceptions. Yes, not only the system throws exceptions; you may do the same, and no need to be shy about it. Sprinkle your methods with guard calls throwing argument exceptions, they imply coding errors on the callers side. If the code allows certain conditions that are logically impossible, guard yourself against that impossibility. Throwing exceptions in exceptional cases is no problem at all. Actually it is by far the preferred way to react this harshly, rather than obscuring erroneous conditions because of falsely understood modesty. Fail early, fail often!

Handling exceptions, as in solving the problem once and for ever is of course possible. If it is possible – which is in my experience rarely the case. (It had to be mentioned, though.)

Enriching exception information is a more frequent task. Whenever it makes sense for the post mortem diagnosis, you should add context information to the exception. Adding the actual SQL statement and the connection string to a database error, adding the user name and the accessed resource to security exceptions. This is valuable, if not necessary, to diagnose the root cause of the problem. It’s not even necessary to wrap the exception to do that, since the exception class has a dictionary Data for this very purpose.

Promoting exceptions: Once your code is able to decide that a certain error is an administration or user problem, you should wrap it in the respective class, thus “promoting” it.
This sentence implies the fact that not every location is suited to make that decision. The database layer should probably never decide whether a particular error is the consequence of some erroneous user action; this usually depends on the business context in which it is called, and should be decided there. Infrastructure issues on the other hand happen very low in the call chain and couldn’t even be decided further up.
The real work here lies in actually putting in the necessary handlers. However they tend to gather in classes that deal with the outside world, the database, configuration, some service, or whatever. They also tend to be very boilerplate.

That’s it. … Wait! … That’s not possible!

Question #4: Can that really be all?

Actually this is it. No “every method should…”. No “layer specific exceptions…”. No “log it just in case…”. Actually whenever I tell people how to implement an exception handling strategy in an existing application, the majority of the work is removing exception handlers. Sometimes life can be so easy.

In case it escaped you: If the developer doesn’t have to do something to employ proper exception management, he can’t do anything wrong. We don’t have to rely on him to properly handle his own bugs. We just solved a paradoxon!

Going further, I even consider it an issue by itself if you did too much exception handling!

For example “shouldn’t we at least log the error, just in case?” No, you shouldn’t. The global error handler is responsible for doing that, so there is no need to log it. Worse, if the calling code decided to handle the situation and carry on, there is by definition no application issue at all. And yet the event log would show one, waking the admins for no reason, or flooding the event log and obscuring the one important entry.

So what? That’s a strategy?

Yes it is. Check the demands the initial questions if you like, they’re all accounted for. Take some file that doesn’t exist. Either you did nothing to anticipate that situation, then an IOException will ensue. Neither user nor admin are taking care of this, so it’s back on your desk. Or you did handle it and threw a BlameTheAdminException.

Of course I simplified quite a bit, but not at the expense of the concept, only in terms of features. For example you may need to have different reactions for the user, some errors calling for a message box, other calling for a redirect. Similarly some unavailable service may allow retrying rather than requiring aborting the whole application. Or you may need additional exception classes to handle them in code. Anyway, nothing of this affects the strategy as such.

Once you’ve established such a strategy, every decision you make on the code level becomes easier, and more confident. And what’s more, in my experience it streamlines error handling, makes it more robust, and frequently simplifies the code by removing unnecessary exception handling. Because quite often if someone asks for exception handling, the answer is “already taken care of, nothing to do”.

Getting back to baseball, every player on the field now knows exactly what he is supposed to do. Some do all the work, most do nothing most of the time… .

PS: The ending anecdote: http://leedumond.com/blog/the-greatest-exception-handling-wtf-of-all-time/

That’s all for now folks,
AJ.NET

kick it on DotNetKicks.com

July 12, 2009

Playing Ball With Exceptions

Ever came across an ASP.NET error page because of a missing configuration entry? Been faced with the question “shouldn’t we at least catch the exception and log it? Just to make sure?” or some statement like “you should catch every exception and throw a new one”? Stumbled over the notorious catch-all or a class whose methods all do exception handling a bit differently? Well, those are symptoms of a missing error management strategy.

It’s actually quite frustrating how often that happens. What’s more, many developers don’t even notice that there is something amiss; they’re quite content dealing with exception on a method by method basis, without taking the bigger picture into account. Or they are very well aware that there is something amiss, but they don’t get it. They know they have to do something, so they set out to do a bit here, a bit there, but not with a clear goal in mind, nothing consistent, only aiming to treat the symptoms and quite regularly overdoing it.

I really don’t remember what first brought the analogy with baseball, of all things, to my mind. Perhaps it was the notion that ‘try’, ‘catch’ and ‘throw’ could be outcries during some ball game, combined with my lack of understanding of baseball (a bit stretching, but that’s how mind twists work). Perhaps it was the idea that many developers do exception handling like they would play baseball if they had never heard of the game.

Stretching or not, that analogy is something I can work with. (And my apologies to everyone who loves and understands that game for everything that follows! ;-))

A Quick Evaluation of … Baseball.

Baseball, like all games has rules. Rules that put the various players into roles they have to fulfill. Most importantly, rules you have to learn if you want to play along. Some are fairly basic “mechanics” of the game: The pitcher throws in the general direction of the batter and the catcher does what his title implies (or so wikipedia tells me).
Some rules are obvious by looking at the game: If the batter hits that ball, he drops the bat and runs, while those guys on the field are trying to catch the ball.
Some rules are rather convoluted and you may have to study some booklet to grasp them; like what the hell is an inning and how do they determine who is winning?

That’s about as much as I understand about baseball. But then, would I want to play baseball, I knew I had a lot to learn before even thinking of entering the field. What rules apply to pitching. What roles do those guys on the field have, what are their responsibilities. How to score points. When to move on to the next round. When does the game end. Under what conditions do the rules change. What is considered foul play and when does it lead to aborting the game.

And this is a surprisingly similar situation to exception handling… . Except that many who play that game play it as if there isn’t even the need for rules, much less asked for them (or have been told about them – which is telling something, too).

Back to Exceptions

The mechanics of exceptions revolve around try/catch/throw and the implied syntax, exception hierarchies, and understanding the runtime behavior. Knowing that much can be expected from every developer and is usually not an issue at all. It is however only just the mechanics, and merely a starting point to proper exception handling.

In the same way the current situation in a game does play a role in baseball (e.g. the batter’s decisions on whether to play safely or to take some risk), the location and situation of a particular code fragment plays a role in the decision on what to do with regard to exceptions. Dealing with them only in the scope of the current code fragment is simply too shortsighted. One has to ask about the overall application strategy to exception handling and how that respective fragment fits into that strategy.

The caveat is: You have to have such a strategy. And defining that strategy should be part of coming up with an application architecture. Yes, exception handling is an architectural topic, not a coding issue.

Exceptions vs. Errors vs. Complexity

As I said, exception handling is an aspect of static and dynamic application architecture, just like security, transaction handling, state management, whatever. But there’s one thing that sets exception handling apart, and that’s dealing with the unexpected, the contradiction of having to deal with what you didn’t think of in the first place.

Exceptions come into play under three different conditions:

  • First, as a means to break the current control flow, say to abort a request upon redirection to another page. This is intended behavior in a regular situation, not even an error.
  • Secondly in cases that disrupt the logically correct operation, but you actually planned for that disruption. Say you rely on a file to exist but you know that someone might screw up, so you handled that case, despite the fact that during regular operations it shouldn’t happen at all. This is an error, but an intended behavior in an expected irregular situation.
  • The third condition is the worst of all: Something unintended happens, something you cannot do anything about. It may arise as ArgumentException (telling you that the developer screwed up) or may be as TypeLoadException (telling you that the environment is on quicksand). In any case this is a purely unintended error and the only thing you can assume about the state of your application afterwards is, that it is not safe to assume anything.

An application architecture should provide provide rules on how to handle these conditions. Add questions like how to deal with the consequences, or deal breaker like parallelism, asynchronicity, or loose coupling, as well as reasonable demands regarding data consistency and error compensation… . And this becomes quite a complex task.

The paradoxical situation however is: Complexity is the very source of errors. Adding complexity is like pouring gasoline into the fire. As developers we already have to deal with the complexity of the problem domain. An exception management strategy should kick in if we fail in that regard. And if he is already doing something wrong, how can we rely on him to do something other (that isn’t even in his line of thinking) right? We can’t!

So, a good exception handling strategy needs to solve that contradiction: The need to deal with different exception conditions on one side. Making things easy for the developers, at the same time not relying on them to do things correctly on the other. Not a small task, indeed. And perhaps this is the very reason that exception handling is usually dealt with so poorly.

Stay Tuned…

OK, this was a lengthy introduction and the actual story is even longer. However I thought it prudent to point out the bigger problem and generally raise the awareness for the topic. Of course I also enjoyed playing with that baseball analogy… ;-). Anyway, in the interest of some readers not interested in such musings I split this one in two distinct posts. This one to explain the problem and make you curious, the next one to provide a solution. Stay tuned if I caught your attention.

PS: Special thanks to Karl for the permission to use his pictures…

PPS: There’s actually more to say about baseball than just baseball. Have a look here, German readers may find a more thorough explanation here. And no, it’s got nothing to do with software development ;-)

That’s all for now folks,
AJ.NET

kick it on DotNetKicks.com

March 29, 2009

Visual Studio 2010 Architecture Edition

Today I’d like to share another left over from the SDX Talk I mentioned earlier: Basically some screenshots from Visual Studio 2010 Architecture Edition (VSArch from now on). Don’t expect something fancy if you already know VSArch, I just couldn’t find all that much information on the Web beyond the two screenshots on the Microsoft site.

The main new things within VSArch include the Architecture Explorer, UML Support, and the Layer Diagram.

Architecture Explorer

Note: To make the following more tangible I loaded a sample project I use regularly as test harness and took respective screenshots while I analyzed it. Click the images for a larger view…

The Architecture Explorer is about getting a better view into existing code. Whether you join a project that is under way, whether you have lost control over your code, or whether you just need to spice up your documentation. Architecture Explorer helps you by visualizing your solution artifacts and dependencies. Artifacts include the classical code artifacts (classes, interfaces, etc.), as well as whole assemblies, files, and namespaces.

Architecture Explorer lets you select those artifacts, display graphs with dependencies, and even navigate along those dependencies and in and out of detail levels.

The following screenshot shows VSArch. The docked pane on the bottom contains the Architecture Explorer that acts as “navigation and control center”. This is where you select your artifacts and visualizations. It could certainly use some improvement from a usability perspective, but it does the job anyway.

vsarch_assemblies.jpg

The screenshot shows two different visualizations of the assembly dependencies in my solution, a matrix view and a directed graph. Just to stress the fact: This was generated out of the solution, by analyzing the project dependencies.

The next screenshot shows a mixture of various artifacts, including classes, interfaces, even files, across parts of or the whole solution.

vsarch_artifacts.jpg

Depending on what filters you set, this graph could give you a high level overview of certain artifacts and their dependencies. For example you could easily spot hot spots, like the one class your whole system depends upon. Or make sure the dependencies are nicely managed via interfaces and find undue relationships. Even spot unreferenced and therefore dead graphs.

Once you go one level deeper, you may want to cluster the artifacts by some category.

vsarch_relationship.jpg

The image shows again artifacts and their dependencies, but this time grouped by the project to which they belong. It also shows what kind of relationship a line represents and lets you navigate along that dependency.

The Architecture Explorer should help getting a better understanding of your code. It helps you to detect code smells or may guide your refactoring.

UML Support

Yes, UML like in, well UML. Not extensively, but it includes activity diagram, component diagram, (logical) class diagram, sequence diagram, and use case diagram. I didn’t spend much time investigating them, just drew some diagrams in order to take the screen shots. Generally I can say that Microsoft can draw boxes and lines (big surprise here) but there is a lingering feeling that those diagram editors may not be finished yet (again, hardly surprising on a CTP).

Creating a new diagram is easy enough. Just create a new project of type “Modeling Project” and add an item:

vsarch_dialog.jpg

Everything starts with a use case, so here is our use case diagram:

vsarch_usecase.jpg

One can draw the diagram as he likes. As you can see from the context menu, there is something being worked on. Namely the “Link to Artifacts” entry shows the Architecture Explorer, yet I couldn’t quite figure out what’s behind this. Also note the validate entries which didn’t do very much, but we’ll see them later in the Layer Diagram.

Next on the list is activity diagrams:

vsarch_activity.jpg

Works as expected, no surprises, no hidden gems that I’ve found.

The same is true for the component diagram:

vsarch_component.jpg

Just a diagram, no surprises.

The logical class diagram gets more interesting:

vsarch_logicalclass.jpg

As you can see, it contains very .NETy stuff like enumerations. It also has these menu entries that hint on more to come in the future — right now the selected menu entry brings up the error message asking for a stereotype, yet I didn’t even find a way to set those. Also the editor may still need some work, e.g. one cannot drag classes in and out of packages.

As a side note: The relation between this logical class diagram and the already existing class diagram escapes me. At least they are a little redundant.

Next on the list is the sequence diagram. Rather than drawing one myself I reverse engineered the existing code:

vsarch_sequence.jpg

Quite nice and again, used this way it can help you documenting or just plain understanding existing code.

Note: If you want to try that yourself, the CTP has a bug: You need to have a modeling project and at least one diagram before the menu entry “Generate Sequence Diagram” appears. And while you will be presented with a dialog asking what call depth to analyze, it usually works only for one level.

Layer Diagram.

Now for the most dreadfully looking diagram (though Microsoft has a more colorful one on its site…): Some boring connected blocks, meant to represent the layers of your architecture.

vsarch_layer.jpg

Actually this is one of the most interesting features for any architect and dev lead: It’s a living beast! :evil: 

You can add assemblies as well as other artifacts to the bleak boxes. Afterwards you can actually validate whether the dependencies between those artifacts match or violate the dependencies implied by the diagram. In the screenshot you can see that I deliberately misplaced an assembly and consequently got a respective error. Using this feature an architect can ensure that all layer related architectural decisions are honored during development.

To conclude…

The Architecture Explorer is certainly a worthwhile feature and I also like the validation feature of the Layer Diagram. That’s certainly something new and not to be found in other products.

Generating sequence diagrams is nice but it remains to be seen whether this will allow roundtrip engineering. The logical class diagram doesn’t yet meet my expectations and it’s not quite clear to me how it will evolve. The other diagrams? Well, they just work. However in this group is nothing exciting for you if you already have another modeling tool like Enterprise Architect (no advertising intended, just happens to be the one I’ve used recently…). And a dedicated tool probably will provide a more complete UML coverage. UML 2.0 has 13 types of diagrams, including state diagrams, which is in my opinion the biggest gap in VSArch UML support.

Anyway, if that caught your attention and your interested in more details there are two options: One, download the CTP and try for yourself. Two, if you want it more condensed and avoid the hassle with a VPC, watch a video with VSArch at work. For that there are two links I can provide:

  1. Peter Provost’s talk at the PDC. Go to the timeline on the PDC site, search for TL15 and you should find “TL15 Architecture without Big Design Up Front”, which is about VSArch, despite the title. His talk was the role model for my analysis of VSArch, yet seeing it live could still give better insights.
  2. Visual Studio Team System 2010 Week on Channel 9 has a bunch of videos, especially the “Architecture Day” ones. “top down” and “bottom up” show VSArch at work.

The final question however will be if all those features are compelling enough to actually buy the … Visual Studio Team Suite (i.e. the “you get everything” package). Why not the Architecture Edition? Well, if you are a developer as well as an architect, the Architecture Edition lacks too much in the other areas. Given that there is usually quite a monetary gap between dev edition and team suite, that gap might very well be used to buy a 3rd party UML tool instead… .

That’s all for now folks,
AJ.NET

kick it on DotNetKicks.com

March 21, 2009

Going parallel…

The so-called „many core shift“ is happening. It’s not a thing of the future, it’s not „just around the corner“, it has already begun. And it will change our developers’ life.

Last week we had some customer events, containing some talks about PDC and other stuff and how it will affect the near term future. Among other things I tried to describe the many core shift as well as its consequences. Curiously the audience was largely aware of the facts, yet the consequences had yet escaped the vast majority. So let me try to repeat the gist and see whether this is maybe a common symptom…

What’s the many core shift, anyway?

Moore’s Law states that the number of transistors on an integrated circuits doubles every two years. Until not long ago, and accompanied by more complex designs and higher clock speeds, that meant faster CPUs. This was sometimes called the „free lunch for developers“, because if one happened to write a slow application… not that anyone ever would ;-) , all he had to do was wait two years and it ran twice as fast.

However this evolution has reached its physical limits (clock speed, power consumption, etc.). Yet, still the number doubles… .
So instead of building faster and more complex CPUs, the manufacturers started placing more CPUs, read cores, on a chip. It started 2006 with Intel’s dual cores, today you won’t find a single core desktop machine anymore. High end consumer machines come with quad cores, and servers with 16 cores (delivered as 4 quad cores). Have a look at the extrapolation:

cores

The “lower” line shows cores on a CPU, starting 2006 with 2 cores, while the steeper one assumes 4 CPU sockets. And just in case the conclusion escaped you: Five years from now we will have between 32 and 128 cores. And remember, we are talking “consumer grade stuff”, that is the box under your desk, not something special! Impressive?

So that’s the many core shift. But what does it mean?

Well, it probably means that today’s software runs a bit faster. Not much, mind you, certainly not the 32 times faster a 64 core machine is supposed to be compared to my dual core. Why is that? Well, have a look at the following task manager of a 64 core machine:

taskmgr_64
(That’s a fake of course, but have a look at Marks’s blog for a real one.)

Now look at your own desktop and count the open applications. Outlook, Word, perhaps PowerPoint, Internet Explorer, Acrobat Reader? OK, say half a dozen applications, add 10 more for the OS stuff actually doing something. That’s 16 applications, using the upper row of cores, perhaps even to 100% and yearning for more, while the other 3 rows just sit there and twiddle their thumbs. That sad truth is: Most of today’s applications simply are not capable of employing these cores appropriately. Consequence: In order to leverage these cores we have to change the way we write our software!

Two questions come to mind: Do we actually need that kind of processing power? And if so, how do we open it up?

Seriously, does the average user need 64 cores?

Well, yes he does. If for no other reason, he did need the increase in processing power during the last years, why should that change?

So, what does he need it for? Gamers are always at the forefront of processing power demand. We have the generally increasing demand in UI technology: 3D, animations, visual effects are becoming mainstream with windows, WPF and Silverlight. The trend to digital photography has had its effect on the demand for graphics software. On my dual core, DXO needs about 1 hour per 150 pictures, so there’s certainly room for improvement (I brought 2500 pics from my last vacation in Tanzania. Do the math ;-) ). Background encryption, compression, virus scanning, etc. also add up.

Even if you are an “ordinary business user”: Word just sits and waits for your input most of the time? Well, open a non trivial 100 page document and see how long Word takes for pagination, spell checking, or updating the TOC. Change the formatting and watch again. So while Word mostly does nothing exciting, there are „burst times“ when it could really need those cores.

And I did not even mention Visual Studio and the compiler yet…

How do I, Joe Developer, put 64 cores to good use?
And how do I make sure the app doesn’t degrade on an old dual core beyond reasonable limits?

Here we are right at the center of the problem: Multithreading is not exactly something new, we’ve had that for more than 20 years on PC’s now. So why do I even have to ask that question? It’s because we didn’t actually use multithreading within our applications if we didn’t have to. Because it’s laborious, error prone, awkward. You have to deal with thread synchronization, race conditions, dead locks, error management, communication between threads. You can’t debug it, tracing doesn’t help very much either. In short: It’s a pain in the …, well.

So let’s face it: Most developers have avoided multithreading altogether (perhaps the lucky ones). And those who did do multithreading probably did it just for optimizations in very distinct areas.

But what we need to leverage those cores is quite the opposite: We need multithreading to become mainstream, kind of ubiquitous. For that it needs to be easier to employ parallelism. Complexity has to be pushed out of our code into the platform. Somewhat like nobody thinks any longer about virtual memory (while some of us are old enough to remember the days of physical addressing).

In other words: In order to deal with the parallelization demands, we need new patterns, libraries, and tools.

Microsoft is going to give us a first delivery on that with the next Visual Studio 2010 and .NET wave. Optimized runtimes (e.g. the thread pool), better tools (e.g. the debugger) and not the least, new libraries, introducing new patterns (e.g. Task Parallel Library). There’s more in the unmanaged world (e.g. Parallel Patterns Library), more on the server side (CCR), more on the language front (F#), more in the research area (transactional memory).
Microsoft even devoted a whole “developer center” to parallel computing (look there for more details). And quite rightly so, because there is no single solution to parallelization, it comes in different flavors (e.g. data parallelism vs. task based parallelism) and we can expect further developments in this area in the future.

Also it’s noteworthy that the OSes, namely Windows Server 2008 R2 and Windows 7 which share the same kernel, can manage 256 cores. Compared to what they supported before this is quite a jump.

Conclusion

So, parallelization is here to stay and we are going to have to deal with it. If anything, the trend is going to accelerate. It’s reasonable to assume that eventually processor manufacturers will trade single core performance for number of cores, i.e. put more but less capable cores on a chip, in order to save power consumption (green IT and mobility being two other major trends).

Looking even further, the many core shift may reach a break even where standard desktop systems will cease to profit from additional cores (how parallel can you become after all?), the problems of memory access may limit the amount of cores. Asynchronous multi cores may evolve, e.g. having cores optimized for certain tasks…

See, there’s a lot to look forward to. Our profession certainly remains interesting :-)

PS: See Being Parallel? for some thoughts on whether today’s multithreaded client software will profit from additional cores…

That’s all for now folks,
AJ.NET

kick it on DotNetKicks.com

December 20, 2008

Explaining Azure…

… is something I did explicitly not want to do!

Not that it is not necessary to do so. It’s just that I expected a lot of bloggers, especially from Microsoft itself, trying to spread the news and foster understanding of what is ahead of us. Well, the Microsoft folks became kind of hushed, as if ducking down and counting the shrapnels after having thrown the bomb at the PDC. So I changed my mind…

Preface…

Given what Microsoft unveiled at the PDC — A new vision, a new strategy, a new technology stack, and a mish mash of existing, sometimes overlapping, not yet consistent, much less complete applications and services — it’s no wonder that it took me some time to grasp the idea. And when I thought I might have understood the gist I was still, well, unsure of whether I had gotten it all right.
The breakthrough came at the ask-the-experts. More to the point I had the chance to talk to Ori Amiga (the guy that did the “BB04 Live Services: A Lap around the Live Framework and Mesh Services” talk). Other than joking about the various ways to pronounce “Azure” and the fact that Americans always manage to get it wrong (sorry guys ;-) — and sorry Ori, hope I didn’t give you away too badly :o) ), this little chat really turned “suspected functionality” into “understood technology” (at least I do hope so…).

Explaining Azure worked not on explaining the ever present Azure picture as is. It worked on developing the pieces of the picture, bit by bit, and relating it to other concepts. And since I worked for me, I thought I might share the gist of that conversation in much the same style, hoping I’m providing new insights rather than reiterating already available information. Actually I’m going to use the very sheet of paper we (mostly Ori) drew throughout said discussion. 

Enough of the preliminaries, here we go…

First of all, keep in mind that there’s two Azures: Windows Azure and the Azure Services Platform. They are not the same and neither presents the full picture. I’ll try to dissect that picture layer by layer, like a cream gateau…

It starts with the cake bottom: Windows Azure

Windows Azure is the basic “infrastructure” (to avoid the term “Operating System” for now) to run applications on, highlighted in the following picture in red:

azure.1.windows_azure

That includes computing capabilities, basic storage, management of applications (i.e. deployment, including upgrading), and operations (e.g. handling failures). These concepts are abstractions from the underlying OS (Windows 2008 actually), machines, and storage devices.

The terms to think of are service instances rather than processes or (virtual) machines. This is similar to the way virtual memory abstracts physical memory. While any memory access obviously has to happen on physical memory, the virtual memory manager is free to relocate it, even to swap it out to disk. This not only makes applications independent of the amount of physical memory, it also optimizes resource usage, allowing other applications to use the memory my application has claimed but does not access at the moment.

Yet, while Microsoft Azure is one level of abstraction above the machine’s OS, it has similar concepts (highlighted in the above picture in blue):

  • computing ~ job scheduling, etc. (NT Kernel);
  • storage ~ file system (NTFS);
  • management ~ application installer;
  • operations ~ task manager, event manager, etc..

Thus it is quite feasible to call Windows Azure an Operating System for the datacenter (or for the cloud if you’re a marketing guy), even if that may not match exactly what you learned in university about operating systems.

The implications of deploying an application to Windows Azure (i.e. how the application has to be built and how the fabric manages it) is actually quite interesting … but a whole new blog post, thus I will skip that for now.

Let’s move on to the second layer of the cake: The Azure Services Platform

A barebone OS like Windows Azure would be of limited use if it were not completed with other general purpose services. Which services exactly that includes may be debatable, yet, again, the similarity with our local environment may help depicting the features we as developers have come to expect from the platform we are developing on: database service, user accounts, IPC, etc..

Again, the following picture highlights the services in red and similarities in blue:

azure.2.azure_services_platform

Microsoft decided that the following services may be good ones to start with:

  • .NET Services: basic infrastructure services for application security (access control), application communication (Service Bus), and workflow (three guesses…?).
  • SQL Services: database related stuff; not exactly a SQL Server, but aspiring to be…
  • Live Services: All around social applications (community, devices, etc.)
  • Core application services: This is a set of higher level application services, such as SharePoint Services and CRM Services (explicitly not including the UI!). In my opinion they are there because they were readily available, not because they are particularly necessary.

Oh, and not to forget the reoccurring three dots in the PDC slides; those dots tell us that these are not closed and sealed sets. Actually Microsoft said that every major server application will eventually be made available on Azure.

Now for the chocolate and the cream: the applications running on the Azure Services Platform

While Ori included the application layer in the platform, any PDC slide puts it on top:

azure.3.applications

Where you put the label is of no consequence anyway, because this is no more than a logical hierarchy. However, please don’t misinterpret this hierarchy by assuming that those applications have to run on Azure and only applications running on Azure can leverage the Azure services! Au contraire!

If you have an application deployed on Azure it is (technically speaking) no different from other services. The difference only lies in the purpose, or the consumer if you wish. And still applications and services are free to call any services, not just those running on Azure. Likewise if your application is running on your local machine or network it can use services deployed on Azure to store data or integrate with whatever, that’s fine as well. Actually the best example for this flexibility is coming from Microsoft itself: Live Mesh.

The cherry on the cream: Live Mesh

Technically speaking the Live Mesh Desktop and Live Services are just another set of applications and services running on the Azure Services Platform, complemented with applications running somewhere else and using those services. This limited view however would miss much of the capabilities of Live Mesh, and the way it enhances the platform.

azure.4.live_mesh

Live Mesh aims no less than to connect people, devices, and applications. Live Services contains services for identity (LiveID), presence, etc., and Mesh Services to maintain users, devices, applications, and — a corner stone — synchronization. The resource model organizes mesh objects (data, news, etc.) in feeds and entries, which in turn are subject to synchronization among the applications being “deployed” to Live Mesh. “Deploying” an application means either actual deployment on Azure, or storing it for (seamless) installation on your local device, via Live Mesh Client, offline capable if built to be.

Still with me? Well, to make this mess, pardon mesh, a little more tangible, let’s recap the example of Ori’s talk (PPTX WMV, images taken from there):


livemesh_app


livemesh_mobile

He had his media center PC connected to the Live Mesh, advertising its meta data, like favorites, recordings, etc.. That information was synchronized to the Live Desktop (running in the browser), so he could pick a TV show there and “start” recording. (Actually that “start” was some little piece of data, synchronized back to the media center PC which in turn did, surprise!, start recording.) He then started a locally installed application that showed the TV guide and had the typical red recording sign right at the respective TV show. That application was offline capable, so he could have planed his TV recordings on the airplane and have it synchronized when he gets back online. Finally he also showed the same TV Guide on his mobile phone simulation.
The only thing missing was integration with other people, but I think it was in a key note where they showed an application that allowed sharing of film critics with some friends.
All in all, that’s what I call ubiquitous computing!

Spicing the cake: Developer tools

This part is actually not in the picture, but it’s no less essential: Where does your application or 3rd party code fit in? How does it get there? Those parts in red may be your application or service:

azure.5.development

It’s actually quite easy: You can write applications that run on Azure and provide services (just like the basic services Microsoft provides), or an UI (just like Microsoft’s applications). You can access any of those services from the cloud or your local application, no matter whether Microsoft provided it or someone else. And you can Mesh-enable any of these applications and services as you like. This is an open platform!

Also Microsoft provides a simulated local environment for Azure, called development fabric, along with Visual Studio integration. Thus it is possible to develop your application locally, test it locally, and only afterwards deploy it to the cloud.
Regarding Live Mesh Matthias has more information on developing with Live Framework.

Postscript…

That’s it. You can find the complete untampered drawing here. That scrawl at the bottom of the drawing is actually Ori’s signature, but you should attribute any error, misunderstanding and adverse opinion in this post to me.

Finally two links for some alternate explanations (already repeated a thousand times over, but what the heck, they’re good):

That’s all for now folks,
AJ.NET

kick it on DotNetKicks.com

Older Posts »

The Shocking Blue Green Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 244 other followers