Cache as cache can!

Hi again,

Caching. Probably the optimization startegy that is employed most often.
Caching is good. Caching immediately speeds up the code. This guy seems like taking a long time? Go, cache it! Cache like crazy. Well, not quite.
Caching is bad. Caching introduces memory pressure. Caching forces you to maintain cache integrity and if not done correctly and exhaustive it will cause arkward data inconsistencies. And it isn’t worth the effort anyway, since when the guy comes back and asks for the same data (if at all), the cached data will be long outdated. Well, not quite either.

So what’s the matter with caching?
Caching is a tool and like every tool it has to be handled correctly. Using a hammer to iron your shirt might look like it worked, yet it will surely smash the buttons. (Which you’ll notice when this shirt is the last clean one you have and you need it for the job interview because you got fired for some foul caching strategy you implemented in the flagship software of your previous employer… 🙂 ).

Here’s the problem:
Caching is easy. Real easy. Putting a hastable in place or using the ASP.NET cache functionality is no big deal – and the code will be faster, if only at first glance. So it’s actually quite tempting to put caching in place.

But like a good drink, caching has its potential adverse effects, especially if applied improperly, such as:

• The application is eating up more and more memory because your homegrown caching using a static hastable will grow steadily.
• The web application will support considerably less concurrent users due to memory preasure.
• The web application will become slower because it invokes the garbage collector or does app domain recyclings more often.
• In some extreme cases you will even see OutOfMemoryExceptions. (Try running BizTalk under heavy load if you’ve never seen one of these guys.)
• You’ll get inconsistent data because not all code lines manipulating the data have been updated to also remove/update the outdated cache entry.
• Best case: Your application just isn’t faster because noone actually used the cached data. Well, startup probably takes even longer.
• Oh boy, did you forget to synchronize access to the cache?
• …

And here’s the deal:
Blindly caching is bad. Sensible caching is still a powerfull tool. Caching is an optimization strategie and should be treated as such. Measure first, find the bottlenecks, decide upon the optimization strategy (yes, caching is not the only one) and then possibly apply caching.
Lemma: You can’t do this right from the beginning, you need a functional complete version of your application. Thus caching should be a tabu until you reach a first functional iteration in development. And since caching is that easy be adamant about not caching until you reach that state!

Note that there may be caching allready in place (e.g. within the database) so allways measure subsequent data requests as well. Also don’t measure the performance alone, measure potential cache hits and misses before implementing caching. After all, putting something in a cache only makes sense if it is requested again afterwards with reasonable frequency.

If you have identified a longer list of possible candidates, don’t go and cache them all. Look for the spots that provide the most revenue. Remember, caching introduces memory pressure.

Here are some additional general hints:

• Begin with a well-defined caching strategy that clearly states what is to be cached, when, and where. Otherwise you will have a client that caches what came from an HTML fragment cache that contains business data cached at the UI layer, which in turn resulted from database data cached in the business layer that came from the database cache. Caching already cached data is particullarly bad, so you should decide carefully at which layer caching should be applied.
• Homegrown caching (e.g. a hastable) should be used sparingly. It is ok for stable data (e.g. reflection data) that cannot change afterwards. But make sure the cache does not grow infinitely and don’t forget to use a reader/writer lock.
• Use a real cache implementation (e.g. System.Web.Caching.Cache) where possible, one that lets the cached data expire. Leverage expiration strategies (e.g. absolute or sliding expiration) depending on the measured hit and miss statistics.
• Make sure your caching strategy isn’t invalidated by load balancing strategies. The cache hit rate can only degrade in a web farm.
• As a corollary: Don’t expect certain data to be in the cache.

Your caching strategy should answer a few questions and help other people in the project to understand why caching is used in one place but not in the other. It should include criterias for:

• choosing caching candidates
• selecting caching locations
• deciding upon cache time and scope

Let’s see what the options for these topics are:

Good caching candidates:

Not all data is a good candidate for caching. Good candidates usually fullfill (most of) these criterias:

• Aquiring the information takes a considerable amount of time.
• Caching the information does not use precious resources (i.e. too much memory, open connections, additional processing time due to de/serialization, …)
• The probability that this information is requested several times afterwards within a reasonable time span is high.
• The probability that this information is changed – especially concurrently – (and thus the cache invalidated) is low.
• Physical layer transistions: Whenever the transistion has to cross process boundaries, networks with bandwidth constraints, data transformation, security checks, etc., the probability to gain performance by avoiding the transition is high.

Good caching locations:

In order to avoid double caching you need to decide where to cache and where not. Good locations include:

• Layer transitions: They are quite good candidates since layers (should) provide a clean and well-defined interface. You may be able to introduce a lightweight caching layer between two layers, thus removing all caching logic from the layers themselves. This works particular good if you already use factory patterns to access the next layer.
• Singleton classes: If you have channeled access to certain information through some kind of class (proxy, helper, singleton, …) this class provides the very spot to employ caching. Within this class you can control cache access, cache invalidation, etc.. The user of this class doesn’t even need to know the data is cached.

Caching time and expiration:

Most data becomes outdated or unused after some time. It is sensible to decide when to throw the data away.

• Inifinitely stable data: Data that can’t change during runtime (e.g. reflection data, machine information, or configuration data within the web.config) could be cached infinitely (i.e. as long as the app domain lives). However you should make sure that this data does not grow infinitely in size and remains a good caching candidate over time.
• Data subject to occasional nonsignificant changes: Some data may change rarely and if it does, the change does not have to be applied immediately. A typical example is changes to passwords or configurations that become effective within the next 15 minutes. This data is a good candidate for caching with absolute expiration and requires no effort to maintain cache consistency. The changing part however (e.g. some administration screens) need to be able to bypass the cache to get access to the current values.
• Data subject to occasional significant changes: If changes happen only rarely but need to be applied immediately (e.g. when changing some key tables) there is the option to deliberately throw away the whole cache data rather than going at length to maintain cache integrity.
• Data subject to frequent changes: If the data shall be cached but it is likely that it might be changed during cache time you’ve got the worst possible case. You need to maintain cache integrity, i.e. every operation affecting the data needs to update the cache accordingly (i.e. update the cached data or simply remove it).
• Data subject to concurrent changes: If data may be changed in the database or backendsystem concurrently by other users you cannot avoid going to the database with each data request. However there are still some options:
  • If the data is needed more than once during one web request (e.g. several parts in a web page of a CRM system need the current customer), you may employ a page cache (thus one data request per web request is made rather than one for each part). This cache is quite simple as it does not have to deal with expiration (it dies with the page), concurrency, and only rarely with cache consistency.
  • It may be the case that the database supports some kind of “the data hasn’t changed” feedback which is considerably faster than asking for the data itself. In this case you have the means to cache the data and still maintain cache integrity.
  • It may also be the case that the database or backend system supports some kind of event that tells you some data has changed. The ASP.NET caching supports cache dependencies to do just that, supporting dependencies on files, other cache items, and notably SQL Server 2005.

Caching scope:

Depending on the scope of the data you have different options for the location of the cache store:

• Caching on the client: Infinitely stable data is best cached at the client. This includes static files (images, scriptfiles, static HTML files, …) in web applications (which is only some configuration in IIS) but also stable key tables and business data for smart client applications.
• Caching for the whole application: Stable data that cannot be sent to the client and data that is user independent can be cached in the global application cache without user reference. Key tables are a perfect example.
• Caching for the single user: Data that is user dependent can still be cached. Examples would be user rights calculated based on his roles, personalization information, etc.. One may use the global cache with user-dependent cache key or the user session for this data.
• Request cache: Data that is used several times during a request but otherwise needs to be up-to-date can very well be cached during the processing of a request. A simple hashtable within the page may be enough.

Special cases:

There are some special cases in ASP.NET applications that I mention for completeness:

• Caching of view state: View state in ASP.NET pages may become quite big, grid controls are especially heavy-weighted in this respect. This is an issue for low speed connections. You can mitigate the problem with HTTP compression, yet storing the view state on the server rather than sending it to the client may prove to be more effcient. ASP.NET 2.0 already supports that.
• Caching of session state: Session state should be held out-of-proc to avoid problems with app domain restarts or web farms. However, accessing the state data requires inter process communication, serialization, and optionally database access. As long as you only do reads (usually the majority of the calls) you could cache the session data (provided you don’t have load balancing in place). It’s a rare situation but if you have heavy-weighted or complex session data you may bennefit from this.

Finally done. I cannot believe it. This is probably the final post about performance for quite some time. What was intended as one little innocent postling became a grown-up, mature familly. Perhaps they’ll reproduce in the future but for now the offspring has to grow up and prosper. Hopefully in one of your projects, I would be glad.

That’s all for now folks,
AJ.NET

Advertisement