We Need a New Paradigm for Web Analytics

I have been studying how to make our web analytics better for reporting dynamic campaign data. We have WebTrends as our analytics, but and on this blog I use Google Analytics for my personal analytics solution.

From an architecture perspective, the basic way an analytics engine gets its data is two ways:

1. A javascript that sends a message to the engine with some tracking data (this is how Google Analytics works for example)
2. A standard web log file is injested (this is how WebTrends typically works)

Both tend to log the static URL plus any additional HTTP information available, e.g. the referrer, query parameters, geographic information, browser information, etc.

All of this works fine for a site built in 1999 when static URLs were the driving force behind a web site. But in an ASP.NET or any other server side generated web site, trying to store all this traffic data in a client side URL is pretty limiting. Consider the following scenarios:

1. You have an ASP.NET forms based application which gets posted back to the same URL over and over again as the form gets edited. The URL doesn't change but lots of information is changing in the background.

2. You have an ASP.NET page which is dynamically built out page fragments such as a portal page.

3. You want to associate a non-page related event to a web session along with the page data you are tracking. For example, suppose that you have a checkout process and you want to register that an order was created and link that to your web page analytics so that you can see which preceding pages link to an order.

In all these cases, the traditional web analytics approach doesn't work. If you look at how WebTrends suggests to do campaign tracking, this gives you an idea on how to fake out the software to track these events. You can store query parameters to identify variables, events, etc. and then get WebTrends to report against them. However, this isn't really a feasible solution for the above scenarios in that a) you don't necessarily want to expose data back on the URL; b) you don't want to have to constantly dynamically re-write every URL in your application; c) if you have a complicated set of objects your tracking then writing out a serialized objected isn't feasible either.

To me this is a real opportunity for Microsoft and the .NET framework in the future. It would be a snap for them to implement it - they already most of the pieces in place and could offer a very open, developer friendly approach to web analytics. This would allow for some very funky tracking in the background and provide the developer the flexibility to log a whole bunch of information as needed. The trick in building the solution is performance - the reason why WebTrends works on web logs is because its fast and means no performance hit on your web traffic. But at least in our case, we would sacrifice a minor performance hit to get more powerful data - we get relatively small amounts of traffic but understanding the patterns in high amounts of detail is very powerful in improve conversion, analyze campaign effectiveness, etc.

Has anyone come across a product that approaches web analytics this way?