White House Data Feeds are Baby Steps

The White House announced to great fan fare that it would be releasing data feeds of its public information collected by hundreds of agencies within the US government.  They have launched the first iteration of this service.

My interest in this feed is to use this data to build mash-ups, visualization applications, and web applications that use this data.  There are many such applications that use existing data feeds such as Amazon data, Google Maps data, Microsoft Live data, etc. in interesting ways.   Amazon in particular has had an XML based REST API available for years (in fact Amazon sells a book on how to use it)  For example, Frucall provides a mobile comparison application that uses a variety of publically available APIs to compare products across retailers and present them to your mobile device. 

So with hundreds of data feeds, imagine the visualization, comparison, calculation and searching applications that could be built on US government data.  As well, imagine then directing these applications to mobile devices, XBOX 360, IPhones, Blackberries, etc.  in formats that work ideally for those platforms.

Unfortunately, based on what has been launched, this vision is not really feasible.  Here is what I can see from what is currently available:

• There are only about 50 data feeds.  Given the number of US agencies, this is a very small sample of the data available.
• The site provides data in a variety of formats such as XML, CSV, KML (google earth) and ESRI (GIS data).  The feed that would be useful for apps is the XML feed, as explained by their own site (“Better suited for consumption by automated applications capable of handling raw XML files”).  There are only 8 feeds in XML format available.
• The data is not a feed, they are files in most cases. If you click on the links, you get either a zip file containing some XML data or a site with some links to XML files.
• In some cases, the XML files are RSS feeds.  While this might be great if you want content, the “data” is really just unstructured content.
• There are no APIs, no REST and each feed has a completely different format.  There is zero consistency even for things like unique identifiers, delivery format, or XML structure.  There are also generally no available XML schemas – you just get raw XML.
• Its not clear how data is published in a timely fashion.  In most cases, you get a directory of files but there is no information on how often they are published.

So while its great to see the information being public, the US government has a long way to go before they have a useful API that could be used for application development purposes.  Given that Googgle. Expedia, Amazon, etc. have had these APIs for many years, its a lot of fanfare and not a lot delivered for this service debut – dumping a bunch of files onto a directory was a victory in 1999, not in 2009.

Business Intelligence is More Than Just a Fancy Graph

"Business Intelligence" is a hot topic these days. There are companies that produce business intelligence products such as Microsoft, Business Objects, Dundas Charts, and so on that focus on visualization of data with the approach that by providing a simplified view of complex data it will enable key stakeholders to make decisions.

A good example is Dundas Charts - the company produces an amazing set of widgets and controls for building dashboards, charts, and visualization of data.



The price is right too - you can buy these fancy BI controls for a license of about $1000.

However, in my experience with Business Intelligence applications, I have come to the following basic conclusion:

Visualizing data is relatively easy.
Obtaining the data in an enterprise environment is hard.
Defining what the data means is even harder.

Let me give you an example. Wait Times are calculated in Ontario and presented to the public in order to show accountability for the funds being invested into improving access to surgeries.

Here is what the data looks like:



As you can see, the indicators are quite simple. The data could have been visualized in a number of different ways such as a red-yellow-green style KPI, a guage, charts, etc. Implementing this would simply involve taking a control library like Dundas Charts and feeding it the above data to get a graphical representation. However, visualization is actually not that valuable - the average person does not need a graph to understand a basic set of numbers.

Obtaining this particular set of data is hard. It requires the synthesis of raw data coming in through automated feeds from about 150 hospitals into a central database that is then scrubbed to match patient records and wait times records together. The cost of building this application to collect the data was millions of dollars - the cost of producing the PDF file containing the numbers is significantly less.

Defining the original business rules, definitions and targets for how wait times data was to be collected was even harder. What does a "Wait" actually mean? There are in Ontario two different wait periods (called Wait One and Wait Two). Wait One is defined as the time it takes for you to get the appointment with the doctor who provide the diagnosis. Wait Two is defined as the time it takes between the point where the doctor provides the diagnosis and the point where you get the procedure. This is what the current Wait Times application tracks - Wait Two data. Different provinces have different definitions and track different wait intervals. In addition, you will notice that only certain types of procedures are represented. These were chosen through government priority - another period of business analysis that took several years to define, prioritize and fund. Similarly, the "targets" were defined through another complex process using clinical experts who invested significant time to analyze and provide recommendations on what was the clinically appropriate targets.

If you are looking at a report, a set of numbers, etc. you are looking at the tip of the iceberg - the amount of energy, work and thought that goes into defining the data and then obtaining it is enormous in comparison to building a fancy graph. In the case of Wait Times, producing the report now is done by running automated processes. The cost of the underlying system to obtain the data cost millions of dollars and took years to implement.

So the next time you look at a report, appreciate the amount of work that has gone into it. And as an IT professional, the value to the customer is in helping them to define the data and then obtain it from a variety of data sources. Produce the actual report is simple in comparison.