AJ's blog

July 28, 2006

Anatomy of a bug…

Filed under: .NET, .NET Framework, ASP.NET, C#, Software Development — ajdotnet @ 5:31 pm

I should probably write that post as long as the blame is fresh and still hurts‚Ķ ūüė≥

The Problem:
We encounterd problems with our ASP.NET 1.1 application under load conditions (of course this happened in production‚Ķ 8O). The application became “less stable” than it used to be. Using a load test tool we could easily reproduce the problems in our development environment. One simulated user works fine. Two users fine as well. 10 concurrent users and we encountered the firts problems, 50 and we had pages with 25% to 35% error rate.
Note that we had load tested our application before, albeit only the major release before the current one. Unfortunately we had done quite some changes, so it was reasonable to asume that one of those changes was the root cause, and yet, the problem might have been there before (as it related to timing issues) and just didn’t manifest itself. But let’s look at some details first.

The Situation:
We user pages that contained user controls (employing our own template mechanism) which in turn contained components (System.ComponentModel.Component). These components provided the data to be shown at the page and they had to be initialized. In Page.OnInit we called the base class method (which should initialize the user controls and create the components) and afterwards our component initialization. Some snippets:

Code in SomeUserControl:

private void InitializeComponent() 
{ 
    this.components = new System.ComponentModel.Container(); 
    this.MyDataComponent = new DataComponent(this.components); 
    this.MyDataComponent.Name = "MyDataComponent"; 
}

Code in our page class:

protected override void OnInit(EventArgs e) 
{ 
    base.OnInit (e); 
    Controller.OnPageInit(); 
    // the controller calls ComponentManager.InitComponents()... 
}

With one single user (i.e. thread) active at a time it worked smoothly. When we debugged the application under load test conditions some arkward exceptions occured that gave us the impression that the user controls were initializes asynchonoulsy.

1. the components beeing created in InitializeComponent()  were null. If we stopped in the debugger and set the instruction pointer back a few lines (or wrote a loop that just slept a few msec and tried to access the component until it actually got hold of it), the component would eventually appear.

Code in our page class:

public ArrayList GetComponents() 
{ 
    if (_components==null) 
    { 
        bool ret= FindComponents(); 
        int loopCount= 1; 
        while(ret==false) 
        { 
            System.Threading.Thread.Sleep(10); 
            ++loopCount; 
            ret= FindComponents(); 
            if (loopCount>100) // 10*100 = 1 sec. 
                break; 
        } 
        else if (loopCount>1) 
            Trace.WriteLine( 
                "Page needed several attempts to get components: " 
                + loopCount); 
    } 
    return _components; 
}

Usually we had 2, 3, 4, 5, 6 loops at most.

2. (different error condition after some changes): The Name property of the component (being set in InitializeComponent() ) is accessed during initialization. We checked the property in code and it had the default value (“<undefined>”). We stopped within the debugger and the value was correct.

string name= component.Name.ToLower(); 
if (name=="<undefined>") 
    throw new Exception("<undefined>"); 
    // breakpoint on throw => debugger show correct name.

Ergo: The page/user control is obviously being initialized asynchronously, ASP.NET on the other hand guarantees that a page is processed by exactly one thread. We didn’t start any threads, so we had a contradiction. This obviously had to be a bug within the ASP.NET runtime‚Ķ. . How likely could that be? Right. The probability would be close to 0. (If you don’t agree you may want to have a look at Testing ASP.NET 2.0 and Visual Web Developer.)

The Search
We finally decided to strip down the project to isolate the root cause of our problem. To accomplish that we had quite some work to do. Our applications consists of a substantial number of pages, user controls, components, business interfaces, business objects, etc., and additionaly some sophisticated core funtionalities… . Eventually our application was down to one page with one user control and the bug still reproducable. Next in line was getting rid of the various libraries that we used. Bang! Spring.NET! (The factory framework we used: http://www.springframework.net/.)

Spring.NET? The call chain looked like Page –> Controller –> ComponentManager, with the component manager iterating over the components on the page. And the component manager was created using Spring.NET. And some further research¬†revealed that there have been multi threading issues with Spring.NET, see here¬†for example. So we had a bug in Spring.NET. Just a quick look at the documentation‚Ķ yes, looks good‚Ķ. yes‚Ķ. err‚Ķ you’re kidding, aren’t you‚Ķ. Ahhrrggghhhh!
Some time later (about 2 hours) we had recovered… ūüė°

The Cause
Spring.NET uses a configuartion file in which the classes are registered for instantiation. One can also specify that a class shall only be instantiated once (i.e. as singletons), effectively¬†returning the same object for each subsequent request. And –¬†Heads up!¬†– the default is singleton=”true‚Ķ . Nobody in our team would have guessed that.

The consequence was that all pages used the same component manager which used its back reference to the page (that was changed constantly) to process the components. This not only explained the exceptions, it also may have supplied the pages with data of another page of another user.

A simple addition of singleton=”false” in that config file solved our problem.

The Conclusion
So, what do we learn from all this?

1. Load test your application! You never know what your application will do under stress.
2. Take this as a rule: The bug is not within the .NET Framework or one of the (widely used) libraries (or JRE if you happen to be from the other camp). The bug is in your code!
3. RTFM! Explicitely specify configuration values and don’t rely on default values – especially with libraries you use. What a feasible “default” is depends on subjective opinions and those can vary from developer to developer‚Ķ (q.e.d.)

There’s also some positive experience (not for the first time): Any problem – no matter how intricate it appears to be – can be tracked down eventually – given enough time and the right people.

That all for now folks,
AJ.NET

July 23, 2006

typeof(Array).GetInterfaces() changes behaviour in .NET 2.0

Filed under: .NET, .NET Framework, C# — ajdotnet @ 12:47 pm

When we migrated an existing project from .NET 1.1 to .NET 2.0 Gerhard (who owns http://www.objectmapper.net/) noted that his mapper suddenly had problems. The cause was an interesting change in the CLR/CTS:

Suppose you were using the interfaces an array exposes like that:

int[]  arrayOfInt = new int[]{1,4,5};

Type[] interfaces = arrayOfInt.GetType().GetInterfaces();

int    i          = interfaces.Length;  // number of supported interfacesobject o          = arrayOfInt;

int    b          = 0;  // counter for implemented interfaces

if (o is ICollection)

    ++b;

if (o is IList)

    ++b;

if (o is IEnumerable)

    ++b;

In .NET 1.1 the resulting value of i was 0. No interfaces supported by arrays, sorry. No, wait! b was 3! So it supports interfaces, but it doesn’t tell!

One may be inclined to call this a bug…. ūüėČ

Anyway, run the same code in .NET 2.0 and the result of i is 7. The interfaces returned are:

  • {Name = “ICloneable” FullName = “System.ICloneable”}
  • {Name = “IList” FullName = “System.Collections.IList”}
  • {Name = “ICollection” FullName = “System.Collections.ICollection”}
  • {Name = “IEnumerable” FullName = “System.Collections.IEnumerable”}
  • {Name = “IList`1” FullName = “System.Collections.Generic.IList`1[[System.Int32, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]”}
  • {Name = “ICollection`1” FullName = “System.Collections.Generic.ICollection`1[[System.Int32, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]”}
  • {Name = “IEnumerable`1” FullName = “System.Collections.Generic.IEnumerable`1[[System.Int32, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]”}¬†

Noteworthy that it implements the old non-generic collection interfaces as well as the new generic ones.

That’s all for now folks,
AJ.NET

July 19, 2006

Is memory shuffling the culprit?

Filed under: .NET, .NET Framework — ajdotnet @ 8:34 pm

In one project we encountered a performance problem with storing files in an Oracle database (and this post is in essence not relateded to Oracle or databases in general): It took around ~45 sec (2MB file) and ~100 sec (4MB file) to store the files as Blobs. The files were read into on big byte array and handed over to ADO.NET. Naively thinking, even if this occupied the whole machine (i.e. scalability goodbye), in terms of performance this should probably be burstmode-fast. Obviously this was not the case and it turned out that the problem was caused somewhere in the ADO.NET data provider for Oracle. 

Between Gerhard (with his object mapper) and Carl (being the Oracle specialist), I (being somewhat experienced with .NET) was involved in tracking the problem down.

My wild guess was that the data provider is just a layer above the oracle client lib (C-Lib), in which case it would move a memory block of 2 or 4 MB respectively via P/Invoke. This would include special GC handling (large object heap), pinning, marshaling, memcopy, physical memory allocations, page faults, etc.. From this assumption I deduced that quite a few of the possible issues would vanish if we moved several small memory blocks, say 4kB, instead. In other words a streaming or a chunking approach might help.

It was just a guess, yet Gerhard proved that it actually did solve the problem. If you are interested in the Oracle specifics I recommend reading Gerhards post… .

There’s actually two lessons to learn:

  1. P/Invoke and marshaling of large amounts of memory can be quite costly in terms of performance. Streaming/chunking ist much more efficient (and in case of server side applictions much more scaleable).
  2. As anyone with some knowledge about logical reasing knows: A false statement may lead to a correct conclusion. So I cannot be sure about statement #1. It may be the false presumption that actually led to a working solution.

Well, right now I’ll have to live with that unsatisfying¬†situation. “Hier seh’ ich nun, ich armer Tor, und bin so klug als wie zuvor…” (German saying, not translatable…)

Thats all for now folks,
AJ.NET

July 17, 2006

ASP.NET 2.0 DataBinding Examined

Filed under: .NET, .NET Framework, ASP.NET, C#, Software Development — ajdotnet @ 9:54 pm

The databinding in ASP.NET is something I have been working on earlier (see http://www.alexander-jung.net/artikel/databinding/default.htm, partly available in english). So it was quite logical to check out the new ASP.NET 2.0 features.

Databindung falls in two categories: Simple binding and complex binding. I’ll talk only about simple binding in this post, that’s more than enough ground to cover anyway. Perhaps complex binding and datasources will be covered later (if I’ve got somethinig worthwile to say).

First the facts, then the assessment. Let’s start with ASP.NET 1.x to be able to see the differences…

Simple binding is used to bind properties to data. This could be the text property of a label, the title property of an image, or any other simple property of any other server control. If you set the property of a control with ASP.NET 1.x and Visual Studio .NET 2003 the designer would generate code fragements like this:

Single control and within datagrid:

<asp:TextBox id=TextBox2 runat="server"

    Text='<%# DataBinder.Eval(timer1, "Interval") %>'/><asp:TextBox id=TextBox3 runat="server"

    Text='<%# DataBinder.Eval(Container, "DataItem.Name") %>'/>

The first argument to DataBinder.Eval is a component (in this case a timer which makes close to no sense in a web application…) or in case the control is contained in a list control (grid, repeater, …) it may be Container to denote the current row. Behind the scenes the ASP.NET runtime would generate code like this (stripped down for brevity):

private Control __BuildControl__control7()

{

    TextBox __ctrl;

    __ctrl = new TextBox();

    this.__control7 = __ctrl;

    __ctrl.ID = "TextBox3";

    __ctrl.DataBinding += new EventHandler(this.__DataBind__control7);

    return __ctrl;

}public void __DataBind__control7(object sender, EventArgs e)

{

    DataGridItem Container;

    TextBox target;

    target = ((TextBox)(sender));

    Container = ((DataGridItem)(target.BindingContainer));

    target.Text = Convert.ToString(DataBinder.Eval(Container, "DataItem.LegalName"));

}

As you can see, it’s all based on code generation and it only works one way. To get the data out of the textbox you need to roll up the sleves and prepare for some work.

Now let’s look at ASP.NET 2.0 with Visual Studio 2005:

With simple binding, one can no longer choose the artificial property “(DataBindings)” (there is a new one, named “(Expressions)”, but that’s not directly related to databinding in the sense we’re just covering). Rather one has to use the smart tag (the small little triangle at the upper right corner) “Edit DataBindings‚Ķ”. This smart tag however is only available for controls that are contained in a DataBoundControl. For controls contained in list controls this is the case and therefore no difference. For single controls though one needs a new parent control, FormView in this case.
Apart from this the output of the designer looks quite similar:

within formview:
<asp:TextBox ID=”TextBox2″ runat=”server” Text=’<%# Bind(“id”) %> Enabled=”False” ReadOnly=”True”/>
within gridview:
<asp:TextBox ID=”TextBox3″ runat=”server” Text=’<%# Bind(“name”) %> />

There are some noteable differences: First (and of no consequence at all) is the shortened syntax. Second, there is two flavours: Eval and Bind. Eval is the same one way databinding as in 1.x, Bind supports two way databinding. The third difference is the setting of the enabled and readonly properties. This is due to the fact that formviews supports different templates (just like the grid). Now let’s look at the generated code:

private global::TextBox @__BuildControl__control13() {

    global::TextBox @__ctrl;

    @__ctrl = new global::TextBox();

    @__ctrl.TemplateControl = this;

    @__ctrl.ApplyStyleSheetSkin(this);

    @__ctrl.ID = "TextBox2";

    @__ctrl.Enabled = false;

    @__ctrl.ReadOnly = true;

    @__ctrl.DataBinding += new EventHandler(this.@__DataBinding__control13);

    return @__ctrl;

}public void @__DataBinding__control13(object sender, EventArgs e) {

    TextBox dataBindingExpressionBuilderTarget;

    FormView Container;

    dataBindingExpressionBuilderTarget = ((TextBox)(sender));

    Container = ((FormView)(dataBindingExpressionBuilderTarget.BindingContainer));

    if ((this.Page.GetDataItem() != null)) {

        dataBindingExpressionBuilderTarget.Text = Convert.ToString(this.Eval("id"),

            Globalization.CultureInfo.CurrentCulture);

    }

}

public IOrderedDictionary @__ExtractValues__control12(Control @__container) {

    OrderedDictionary @__table;

    TextBox TextBox2;

    TextBox2 = ((TextBox)(@__container.FindControl("TextBox2")));

    […] // other controls ommited

@__table = new OrderedDictionary();

    if ((TextBox2 != null)) {

        @__table["id"] = TextBox2.Text;

    }

    […] // other values ommited

    return @__table;

}

Now, the Eval part works essentially the same way it did in 1.x, registering a DataBinding event handler on the textbox and setting the value. Vice versa we have one method for the formview control that takes the values of all bound child controls and puts them into a dictionary. Again it’s fully based on code generation.

Assessment:

Now let’s see whether the new databinding lives up to the expectations. Well, there’s good news and bad news…

The good part: Imagine a web site structured like a “classical web site” (i.e. one containing lists, simple input forms, etc.). With ASP.NET 2.0 one can implement that without leaving the designer (provided the data is readily available). It’s actually quite amazing how far you can get without having to type code. And due to the code generation approach it’s fast and efficient and any faults will show up during compile time. If you ask me, that’s better than ever.

The bad part: Imagine a web site that is different in one or the other respect. It may have a UI that’s structured like a windows form with several user controls, each providing their own formview, yet they should work in unison. It may be that you shy away of the maintenance effort for several templates per formview and would prefer a meta data driven frontend. This could be controls that can render themselfs readonly rather than having to maintain two different templates. Or the data structures you use provide enough meta information to create validators dynamically.
Now you’re struck. You simply don’t have the informations you need. The textbox never knows which field it is bound to, so how do you retrive the meta data you need? The formview behaves at if it where the only one working exclusively on one whole row/entity, so how do you synchronize multiple formviews, each working on a partition of an entity?
The answer is: You can’t. Full stop.
The consequence: Forget your good ideas – or roll your own databinding.

Has Microsoft delivered something good?
In terms of functionality I think they have done quite a good job, they could hardly have delivered more.

Could Microsoft have done better?
In terms of design and how to surface the features to the developer they don’t live up to my expectations. In fact I wouldn’t call the databinding mechanism a framework feature because it lacks framework quality, in this case extensibility points. This is even more annoying since I know the can do better. MFC used code generation (read preprocessor macros) and yet was extensible; ATL used templates and yet was extensible. Other areas in ASP.NET use interfaces, methods, provider patterns – all valid extension points. I know they could have done better.

That’s all for now folks,
AJ.NET

SOA Starting Point

Filed under: SOA, Software Architecture — ajdotnet @ 9:25 pm

As SOA (along with Web Services) is one of my favourite topics, I think it prudent to define a starting point with my first post. (Actually this is not exacly my first “post”, since I used to have some of the following information on my web site, yet I cleaned that part just recently.) I should point out that this post refers to conceptual things rather than implementation or other topics.

Now, the usual starting point would be to come up with a definition. Another one. Adding to the not so tiny number of already existing definitions… .¬†No way!¬† If you need a definition I recommend Reger Sessions ObjectWatch Newsletter #45: “What is a Service-Oriented Architecture (SOA)?” (October 7, 2003, http://www.objectwatch.com/newsletters/issue_45.htm), which also does away with some misconceptions.

Anyway, the starting point for me lies a little earlier than the term “SOA”. It’s got to do with the “S” in SOA: Pat Hellands work on “Autonomous computing: Fiefdoms and Emissaries” (2002, Microsoft Webcast, http://microsoft.com/usa/Webcasts/ondemand/892.asp — and I could not find it, the link is broken. I know that there are also PPTs from Pat about this topic, yet I couldn’t find any of them publicly available either. If anyone knows some links, please tell me!)
The value of this work lies in doing away with classic (read clients/server like) patterns of data exchange.

Also dating earlier than the term “SOA” is another issue of Rogers newsletters: “The Software Fortress Model: A Next Generation Model for Describing Enterprise Software Architectures” (#36, 17.11.2001, http://www.objectwatch.com/newsletters/issue_36.htm). OK, I know, the link is also broken. But I sent Roger an email asking when/whether it will work again and he has also written a book about that topic [UPDATE]It took but a weekend and #36 was online again :-D. Thanks, Roger! [END UPDATE].
The reason I think this a valuable work is that it defined the landscape of systems/services at the advent of SOA. Many of the issues with implementing a SOA can be understood better with the SFM in mind.

What a starting point. Based on information I cannot deliver :-(… . But it gets better.

Again it was Pat Helland who brought up Metropolis, “A metaphor for the evolution of information technology into the world of service-oriented architectures.” (04/2004, http://www.architecturejournal.net/2004/issue2/aj2metrop.aspx, there was more information on his personal web site, yet since he kind of went out of business, his site is down. What a pity.)
Speaking of evolution, I see Metropolis as the evolution of Rogers SFM in a SOA world where the various services work together and become a living, almost kind of social, interacting eco system.

And back to Roger (it’s like ping pong, isn’t it ;-)): He just published a white paper about “A Better Path to Enterprise Architectures” (04/2004, http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/sessfin00.asp, or on his website). He also uses the “analogy of an enterprise architecture to a city plan”, but he takes it further and examins how to evolve the enterprise (rather than the single service) without loosing control.

Well, that’s it so far. These two guys have done tremendous work to describe the large, interdepended, and grossly complex eco systems that we as developers and architects have to deliver, maintain, evolve, and master. To my relief so far, the reality today is (mostly) still some iteratations behind these prospects. However – as Roger pointed out with his examples of failures – this may be the case because we (as an industry) couldn’t yet deliver this vision of eco systems.

That’s all for now folks,
AJ.NET

July 15, 2006

Intel’s Core 2 Extreme & Core 2 Duo: The Empire Strikes Back

Filed under: Miscellaneous — ajdotnet @ 11:18 am

Now, this is my first post, and it covers a quite uncommon topic for me for it relates to hardware (which is something I tend to find boring). Yet it struck me that yesterday I knew which OS my next machine will probably run (Vista – despite its DRM nuisance), but today I know which processor (familly) will be at the heart of that machine.

The full article that came to my attention can be found here. The gist is that the new Core 2 processors obsolete Intels current Pentium D line and put AMD into the second league. Some quotes:

  • Unfortunately AMD won’t have an architectural update of the Athlon 64 X2 until sometime in 2007 or 2008, thus its only response to Intel’s Core 2 lineup today is to also reduce pricing.
  • As you will soon see, Intel’s new Core 2 lineup has basically made all previous Intel processors worthless.
  • At the high end, the Core 2 Extreme X6800 was just under 36% faster than the Athlon 64 FX-62. In fact, even the $316 E6600 was around 18% faster than AMD’s fastest. To add even more insult to injury the slowest Core 2 Duo in the test, the 1.86GHz E6300 is barely slower than AMD’s fastest Athlon 64 X2.
  • The old Intel lineup of Pentium D processors is truly an embarrassment. Only the Extreme Edition 965 is remotely competitive and even then it can barely outperform the $183 E6300.
  • these power numbers heavily favor Intel.
  • Intel’s Core 2 Extreme X6800 didn’t lose a single benchmark in our comparison; not a single one. In many cases, the $183 Core 2 Duo E6300 actually outperformed Intel’s previous champ: the Pentium Extreme Edition 965. In one day, Intel has made its entire Pentium D lineup of processors obsolete. Intel’s Core 2 processors offer the sort of next-generation micro-architecture performance leap that we honestly haven’t seen from Intel since the introduction of the P6.

There is essentially two things that thrill me:

  1. A huge step forward in terms of performance at the right time, i.e. at the advent of Vista (and nobody thinks this timing is anything but pure accidental ;-))
  2. No more need to distinguish between desktop and laptop processors. More to the point, no need to choose between fast but power consuming heating chips or striped down versions.

So, whoever is thinking of a new PC as his next christmas present, (s)he should tell Santa that it better had one of these new processors.

Bye,
AJ.NET

July 14, 2006

Finally…

Filed under: Miscellaneous — ajdotnet @ 3:00 pm

.. I decided to inflict my 0,02‚ā¨ on the world. Let’s see how far this gets and how long I am motivated to keep this blog up and running….

Blog at WordPress.com.