White House Data Feeds are Baby Steps

The White House announced to great fan fare that it would be releasing data feeds of its public information collected by hundreds of agencies within the US government.  They have launched the first iteration of this service.

My interest in this feed is to use this data to build mash-ups, visualization applications, and web applications that use this data.  There are many such applications that use existing data feeds such as Amazon data, Google Maps data, Microsoft Live data, etc. in interesting ways.   Amazon in particular has had an XML based REST API available for years (in fact Amazon sells a book on how to use it)  For example, Frucall provides a mobile comparison application that uses a variety of publically available APIs to compare products across retailers and present them to your mobile device. 

So with hundreds of data feeds, imagine the visualization, comparison, calculation and searching applications that could be built on US government data.  As well, imagine then directing these applications to mobile devices, XBOX 360, IPhones, Blackberries, etc.  in formats that work ideally for those platforms.

Unfortunately, based on what has been launched, this vision is not really feasible.  Here is what I can see from what is currently available:

• There are only about 50 data feeds.  Given the number of US agencies, this is a very small sample of the data available.
• The site provides data in a variety of formats such as XML, CSV, KML (google earth) and ESRI (GIS data).  The feed that would be useful for apps is the XML feed, as explained by their own site (“Better suited for consumption by automated applications capable of handling raw XML files”).  There are only 8 feeds in XML format available.
• The data is not a feed, they are files in most cases. If you click on the links, you get either a zip file containing some XML data or a site with some links to XML files.
• In some cases, the XML files are RSS feeds.  While this might be great if you want content, the “data” is really just unstructured content.
• There are no APIs, no REST and each feed has a completely different format.  There is zero consistency even for things like unique identifiers, delivery format, or XML structure.  There are also generally no available XML schemas – you just get raw XML.
• Its not clear how data is published in a timely fashion.  In most cases, you get a directory of files but there is no information on how often they are published.

So while its great to see the information being public, the US government has a long way to go before they have a useful API that could be used for application development purposes.  Given that Googgle. Expedia, Amazon, etc. have had these APIs for many years, its a lot of fanfare and not a lot delivered for this service debut – dumping a bunch of files onto a directory was a victory in 1999, not in 2009.

Billboard Gone Bad

I was downtown the other day and snapped this with my cell phone camera.  Apparently, this bill board needs Windows Activation!

SharePoint is not a Database!

As users of SharePoint start using lists they start taking advantage of a very intuitive and highly configurable information management tool.  Users who have been using Excel or Access in the past start seeing SharePoint as a kind of rapid application development platform for building structured data applications.  However, this is where SharePoint’s list architecture breaks down because simply:

SharePoint is not a database!

Part of the problem is that business users designing lists are not familiar with basic database concepts - they just want to store their information.  Part of the problem is false advertising by consultants and/or Microsoft that SharePoint is a miracle platform that can do anything - it does some things well but replacing your SQL server or even Access database isn't one of them.  The other problem is there is no really good way for surfacing bi-directionally data from a database into SharePoint – the business data catalogue works only in read only mode and does not allow for updates back to the underlying data store.

Here are a list of some features that a database has that SharePoint does not support - the moment you need these features you will need a real database.

• Primary And foreign keys: one of the most basic concepts in a database is the enforcement of a unique identifier (either system generated or user supplied) that can identify a record and link across table in parent child relationships.
• Transactions: if two changes are required to go together and the second change fails you want the first change to be undone (rolled back).
• SQL language: complex queries can me written in code using a standard language.  Queries can be saved and repurposed.
• Stored procedures: database programmers can write complex routines that can be called as reusable code blocks. 
• Indexing and query optimization: used to improve performance, indexing allows the database designer to pre-index specific fields that are frequently used in queries to improve performance.  In addition, most databases have optimization engines that based on what you are trying to fetch will optimize how the data is retrieved.
• Large data and binary fields: most modern databases allow you to store large binary files such as video files, large volumes of text, images, etc. in the database.  SharePoint can be made to store large files as documents but you only get one per record and a generic binary object. 
• Access outside of SharePoint: lists are accessible outside of SharePoint but only through XML or programming interfaces.  Lists are not great data stores to be used for line of business applications written independently of SharePoint because the integration is relatively poor in comparison to running a SQL query on a database.
• Cascading deletes: if a parent record is deleted, its children should also be deleted.  In most databases, this can be configured to happen automatically or else reject the delete of the parent until the children are deleted first.  This eliminates the risk of orphan child records.

These features are considered basic in databases – they have been around for decades and even the most primitive databases such as Access or FileMaker Pro have most of these features. 

Lists are great for flat, changing content especially when attached to unstructured documents.  At least in the current version, lists are not substitutes for databases and shouldn’t be considered so.  If you need a database, then you are better off building a proper database and using SharePoint as a front-end for surfacing that data through reports, dashboards, etc. or as a host for data entry components such as InfoPath forms or custom build web parts.      

Hotspot Shield and Hulu No Longer Work Together

Sites like Pandora and Hulu block IP addresses not originating from the United States.  The solution for the past 6 months or so has been to use Hotspot Shield which proxies your IP address behind an US address.

As of today, this no longer works.  When you load up a video, you get the following:

Pandora still works thankfully…

NOTE: There seems to be a solution to this problem – thanks to the comment below!  If you clean out your browser cache and then reconnect to hotspot shield it seems to work.  I suspect this is a temporary solution until Hulu starts blocking this new batch of servers.

When is the old world media going to get that the Internet is global and find a solution for all this rights territory nonsense?

Why SharePoint Projects Fail

Microsoft Office SharePoint Server 2007 (affectionately known in our circles as “MOSS”) is now at what should be at the peak of adoption.  The product has been on the market since 2007 and there is a new version coming in a year.

Yet with many of our customers, we find that MOSS has been implemented but barely gets off the ground.  There a few reasons why this seems to be happening:

• MOSS is championed by IT and then too tightly controlled.  For example, we have customers who have installed MOSS but will only use it for out of the box collaboration – no custom code, no document management, no BI, etc.  If you want an out of the box team site, then that’s part of the “service” but otherwise you will have to wait.
• MOSS starts in a single department and then never makes it out.  A department picks MOSS for their needs but then there is not enough support or sponsorship in the organization to roll it out further.
• MOSS is implemented skunk works style on a single box.  The implementation is more successful than originally envisioned and what was a “proof of concept” starts getting treated like production.  This sets up the assumption that MOSS is “Rapid Development” and eventually the box falls over under strain.
• MOSS is implemented by IT but there has been no planning for adoption and training.   MOSS sits there with a bunch of services – users simply don’t know how to use them and go on using other more reliable forms of collaboration such as email, file shares, etc. 
• The implementation gets mired in legal, HR, or branding issues as corporate citizens start to have the ability to create their own content.  Similar to the IT clamp down, the organization clamps down on the services that MOSS can provide such as blogs, wikis, My Sites, etc. because they do not have a good model for managing the risk of employees contributing their own content.

Why does this happen?  What is missing from most MOSS implementations?  In my experience, there are a few common causes:

• Too much involvement by IT, not enough involvement from business stakeholders.
• Too much focus on MOSS as a generic toolbox (document management, collaboration, search, etc.), not enough focus on MOSS as a way to automate or enable true business services.
• Poor infrastructure planning to enable scaling to the enterprise.
• Lack of training, adoption and change management.
• Lack of a killer app for the particular needs of the business
• Not enough support from executive champions
• No governance model to set expectations on how MOSS will be operated, changed and maintained.

If you are an IT Director or department champion for SharePoint, I strongly encourage you to think about these issues BEFORE you start an implementation.  We see many cases where organizations start thinking about these issues after the first implementation has already failed and the organization has soured on the platform because there was not enough investment in managing the project and an eager IT department ready to roll out a service.

Separating the Humans from the Robots

 

With a new Terminator movie coming out, I started thinking about the ways that on the Internet now we need to try and distinguish the humans from the robots.  There is already a “war” going on out there between humans and sophisticated robots for control of Facebook, Hotmail, Gmail and other web 2.0 services.

One of the more annoying “fronts” in this war is CAPTCHA (“Completely Automated Public Turing Test to Tell Computers and Humans Apart”).  This reverse Turing test is what is used on many sites to test to see if you are human and not some automated robot.

The problem is that these tests are like antibiotics – the more they get used the more sophisticated the robots.  Early CAPTCHA tests were easily broken by bots so they have evolved into more and more sophisticated tests.  Now the number of hoops and the readability of these tests are so bad that human beings can barely pass them.

The challenge is how to make a test easily answerable by a human being but yet heuristically difficult enough to foil a software program. 

Google is experimenting with new techniques to try and simplify the experience of verification – see below.

Even using this technique, Google has to specifically pick the right images:

However, while the images will be randomly rotated, they will be carefully selected. First, Google will exclude images for which its own computers can readily identify the top, such as photographs showing landscapes with blue sky (easily detected), text (easily recognized) or portraits of people (there are many facial recognition applications on the market). Then, it will screen out images that humans find too difficult to orient (for example, abstract art, or overhead views, that don't have a readily identifiable top) by conducting a sort of opinion poll.

Google is relying not just on individual results but collective results.  The heuristic picture composed from millions of clicks is what will power the analytics so that Google can reliably detect bots instead of humans.

Live Earth Bird’s Eye View

One of the cooler features that sets Live Earth/Maps apart from Google Maps is the birds eye view. 

Take a look at this building in Google Maps:

Now take a look at the Bird’s Eye view of the same building in Live Earth/Maps.

 

 

 

The four views are available if you hit the rotate button in the interface – these are actual photographs taken from different angles.

The difference in understanding heights of buildings, features around buildings, etc. is significant – imagine if you’re doing some house hunting and you see a neighbourhood you like.  Wouldn’t you like to see the increased detail and vantage points that you get from this orthogonal view instead of just the top down view you get in Google maps?