White House Data Feeds are Baby Steps

The White House announced to great fan fare that it would be releasing data feeds of its public information collected by hundreds of agencies within the US government.  They have launched the first iteration of this service.

My interest in this feed is to use this data to build mash-ups, visualization applications, and web applications that use this data.  There are many such applications that use existing data feeds such as Amazon data, Google Maps data, Microsoft Live data, etc. in interesting ways.   Amazon in particular has had an XML based REST API available for years (in fact Amazon sells a book on how to use it)  For example, Frucall provides a mobile comparison application that uses a variety of publically available APIs to compare products across retailers and present them to your mobile device. 

So with hundreds of data feeds, imagine the visualization, comparison, calculation and searching applications that could be built on US government data.  As well, imagine then directing these applications to mobile devices, XBOX 360, IPhones, Blackberries, etc.  in formats that work ideally for those platforms.

Unfortunately, based on what has been launched, this vision is not really feasible.  Here is what I can see from what is currently available:

• There are only about 50 data feeds.  Given the number of US agencies, this is a very small sample of the data available.
• The site provides data in a variety of formats such as XML, CSV, KML (google earth) and ESRI (GIS data).  The feed that would be useful for apps is the XML feed, as explained by their own site (“Better suited for consumption by automated applications capable of handling raw XML files”).  There are only 8 feeds in XML format available.
• The data is not a feed, they are files in most cases. If you click on the links, you get either a zip file containing some XML data or a site with some links to XML files.
• In some cases, the XML files are RSS feeds.  While this might be great if you want content, the “data” is really just unstructured content.
• There are no APIs, no REST and each feed has a completely different format.  There is zero consistency even for things like unique identifiers, delivery format, or XML structure.  There are also generally no available XML schemas – you just get raw XML.
• Its not clear how data is published in a timely fashion.  In most cases, you get a directory of files but there is no information on how often they are published.

So while its great to see the information being public, the US government has a long way to go before they have a useful API that could be used for application development purposes.  Given that Googgle. Expedia, Amazon, etc. have had these APIs for many years, its a lot of fanfare and not a lot delivered for this service debut – dumping a bunch of files onto a directory was a victory in 1999, not in 2009.

Separating the Humans from the Robots

 

With a new Terminator movie coming out, I started thinking about the ways that on the Internet now we need to try and distinguish the humans from the robots.  There is already a “war” going on out there between humans and sophisticated robots for control of Facebook, Hotmail, Gmail and other web 2.0 services.

One of the more annoying “fronts” in this war is CAPTCHA (“Completely Automated Public Turing Test to Tell Computers and Humans Apart”).  This reverse Turing test is what is used on many sites to test to see if you are human and not some automated robot.

The problem is that these tests are like antibiotics – the more they get used the more sophisticated the robots.  Early CAPTCHA tests were easily broken by bots so they have evolved into more and more sophisticated tests.  Now the number of hoops and the readability of these tests are so bad that human beings can barely pass them.

The challenge is how to make a test easily answerable by a human being but yet heuristically difficult enough to foil a software program. 

Google is experimenting with new techniques to try and simplify the experience of verification – see below.

Even using this technique, Google has to specifically pick the right images:

However, while the images will be randomly rotated, they will be carefully selected. First, Google will exclude images for which its own computers can readily identify the top, such as photographs showing landscapes with blue sky (easily detected), text (easily recognized) or portraits of people (there are many facial recognition applications on the market). Then, it will screen out images that humans find too difficult to orient (for example, abstract art, or overhead views, that don't have a readily identifiable top) by conducting a sort of opinion poll.

Google is relying not just on individual results but collective results.  The heuristic picture composed from millions of clicks is what will power the analytics so that Google can reliably detect bots instead of humans.

Live Earth Bird’s Eye View

One of the cooler features that sets Live Earth/Maps apart from Google Maps is the birds eye view. 

Take a look at this building in Google Maps:

Now take a look at the Bird’s Eye view of the same building in Live Earth/Maps.

 

 

 

The four views are available if you hit the rotate button in the interface – these are actual photographs taken from different angles.

The difference in understanding heights of buildings, features around buildings, etc. is significant – imagine if you’re doing some house hunting and you see a neighbourhood you like.  Wouldn’t you like to see the increased detail and vantage points that you get from this orthogonal view instead of just the top down view you get in Google maps?

Adsense and Evil Java

Adsense in general does a good job of targeting ads based on the content of your site.  I have adsense running on my blog and I find that it does a reasonable job of putting ads that are somehow connected to my content.  If I’m writing about Microsoft Project, the reader gets ads on Microsoft Project.

Adsense seems to have one basic problem – it cannot distinguish between positive and negative content.  Here is some content that I found on a random site:

Java is EVIL, it's the work of the devil, made to collect your
souls and to turn you into brainless slaves of hell.
If you like Java, you will lose your soul to be damned
for eternal torment in hell.Demons and devils will feast on your soul
for all eternity.
So stop using it now to save your soul.
Or the holy inquisition will come for you and burn your flesh to save
your soul, because the devil must not collect more souls or nobody can
prevent Armageddon.
Java is evil, using it is BLASPHEMY , only heretics and
witches like Java.And heretics and witches shall BURN !
Java = programming language written by SATAN in Hell's Kitchen in 666 minutes ! ! !

I of course didn’t write this and for the record, I have used Java for many many years and I don’t think I’m going to Hell.  I posted this to prove a point – Adsense will now show ads that promote java products.  This is a sample of what Adsense delivered on this site:

So the question is for advertisers – do you really want your company being linked to content such as this simply because it has a few key words in it?  Is there a way to heuristically analyze content to filter this out (or even better, target the negative content for example to anti-Java advertisers?)

Online Trends for 2009

Here are some online trends to look out for in 2009.

Advertising Takes a Beating

With the sluggishness of the US economy, advertising will decline significantly and online advertising in particular will be hit hard. Online advertising will still continue to grow but at a much slower pace.

This slowdown in growth will put pressure on web 2.0 applications with lots of content but lower conversion rates. Applications such as Facebook, Flickr, YouTube, etc. will need to focus more heavily on monetizing all those collaboration interactions that millions of people now do for free every day.

There will also be a continued emphasis on Cost Per Action instead of Cost Per Click to link advertised costs to conversion. As the cost of online advertising increases (for example the cost of buying keywords) there will be increased pressure on Google and other advertising platforms to show direct linkage between advertising and conversion to revenue. This is where online advertising as a channel has had an advantage over traditional media channels.

Internet TV is Finally a Reality

The "watch video on your computer" revolution started about 10 years ago, but until this year watching the equivalent of television via the Internet really wasn't possible. Instead, users have sacrificed the video quality that even low definition television provides for increased collaboration, crowd created content, etc. In addition, the selection for mainstream content has been limited online until the last year or so.

In 2009, the compromise on quality and selection will fade away. Video sites such as YouTube, Facebook, etc. are all upgrading their video streaming so that the videos uploaded by users are streamed at much higher resolution. In addition, sites such as Hulu now stream hundreds of mainstream television programs in almost broadcast quality. While not able to yet compete with HDTV, video on the Internet will now be viable as a full-screen, sit in the living room type experience.

Part of the challenge has been the source video - consumer digital cameras tended to be lower resolution video than television. However, the latest digital cameras produce very good video quality including my Panasonic camera that can shoot HD video in wide screen format.

Web 2.0 will Become Mainstream

Don Tapscott's new book on the Net Generation points out that when technology becomes boring and we stop talking about it then the next level of innovation is possible. Web 2.0 will simply become The Internet as usual and we'll stop talking about it as this new found technology revolution (until we think of the next one of course).

Applications like YouTube, Facebook, etc. are now so mainstream that your mother uses them. Applications such as Facebook, MySpace, YouTube, etc. have all already demographically shifted to the mainstream from being in the hands of the power users, the young and the affluent.

Mobile Applications Will See Massive Growth

With the launch of the IPhone and its inevitable competitors, the phone now becomes a viable application platform. IPhone is as significant for mobile applications as IPod was for downloadable music.

Another mobile device seeing large sales is the Amazon Kindle. It allows users to purchase books and view them in a mobile device. The idea of e-books has been around for over 10 years but it takes a device like the Kindle with its complete vertical integration to drive the adoption.

Geotagging and Live/Google Earth

As the amount of content increases, one of the simplest ways to filter content is through geography. In addition, with the maturity and increased use of mapping interfaces such as Live Earth, Google Earth, Google Maps, etc. both as destination sites and as mash-ups, geotagging of content, pictures, videos, and people will be a vital way of filtering content in 2009.

One of the simplest examples is the integration of Panaramio with Google Maps. In Google Maps if you look up any address it will show you related photos from Panaramio, all geotagged on upload by location.

Change of Strategy From The RIAA/MPAA

For the past several years, the global culture industry has been taking people to court over copyright infringement. Approximately 30,000 people have been sued in the US for copyright infringement over the past 5 years. The impact on file sharing has been no real impact on file sharing but has made the RIAA one of the most hated organizations worldwide.

It appears that after 5 years of harassment law suits, the RIAA has decided to change its strategy and use a softer approach. In addition, new technology and business partnerships will allow the media conglomerates to start making money off their content no matter who uploads it. For example, Universal now stands to make over $100 million in ad revenue through a deal inked with YouTube that uses search terms to identify user contributed music videos that map to Universal's artists.

Casual Entertainment Over Hard Core Gaming

The success of the Nintendo Wii (sold out again this Christmas!) as well as other casual games such as the Sims and Spore will continue to provide online entertainment for the masses. Collaboration, casual games and customization will continue to push the mainstream entertainment market. Games like RockStar, Wii Fit, etc. will continue to provide mainstream entertainment while leaving more hard core games to cater to the hard core 18-30 male demographic.

See the above note as well on platforms like IPhone and Facebook - these are perfect jumping points for casual gaming as they cater to an audience that might play a game of solitaire but isn't going to be spending hours with Call of Duty.

The casual gaming market is still in massive growth mode and this will continue in 2009.

Difference, Story Telling and Community

This isn't a trend, but it will continue to be true in 2009 as it always has been: story telling, difference and community building continue to drive the content creation process.

The revolutionary part of the Internet is the ease at which these tools can be employed for someone to tell a story, create some innovative content and enable a community. The growth of blogs shows that everyone has something to say and the size of the audience it generates is how well it connects to its community of interest. Determining "success" depends on the objective - it may simply be distributing family pictures or it may be making your opinion heard by millions as Barak Obama did in 2008.

The Key Challenge in 2009: Conversion

I have a friend who is very passionate about informing parents about Toy Recalls. She has an active blog, she uses Twitter and Facebook and she is able to get her message out to several hundred viewers a day. When we discussed her "success criteria", she seemed pleased that she had grown this community from zero but her next question was, "ok so now what?". This is I think the key question for all this collaboration activity going on - how do we convert the energy of all this discussion, file sharing, photo sharing, tagging, blogging, etc. into some sort of concrete action so that its simply not just hanging out (although that is not a bad goal in itself).

It's as if we have got people off the couch and started them talking again. So now the community is sharing, talking, collaborating. So now what happens? How do the tools available provide methods to convert this community to action?

This is the future growth opportunity for application platforms like Facebook and you can see the evolution of these tools going in this direction already - applications like the Causes application for example are a first attempt to translate a community of action into a concrete action (in this case donating money to charity).

Live Earth vs. Google Earth - Which One Is Better?

I decided to run a comparison between Google Earth and Live Earth. Here are some of my findings.

Google Earth runs only as a separate executable. Live Earth runs as an ActiveX control within the live maps interface. Microsoft's approach is more integrated but it also means that you cannot run the application full-screen.

Google Earth seems to find locations better especially when they are vague. For example, I typed in "yonge and dundas toronto ontario" and Google Earth found the intersection. Live Earth couldnt find it and simply suggested businesses which weren't even at that location.

Google Earth's ability to handle misspelled words is much better as well. For example, I typed in "young street toronto ontario" and it was still able to find Yonge Street as an exact match. Microsoft Live Earth could not find it.

Microsoft Live Earth splits the search box between businesses and locations. The search assumes that you will split the business name and location into the two boxes. For example, I typed in "eaton centre toronto ontario" into the business box and it asked me for a location. In Google Earth, typing in the same search query finds me the exact match.

The search box in Google Earth is on the left hand side. In Microsoft Live Earth the search box is at the top of the screen. This makes a big difference if you are using a wide screen monitor. Microsoft's map ends up being very wide. Google Earth's map ends up being nice and square.

Microsoft's satellite pictures seem to be more up to date and detailed. Looking at downtown Toronto, the Google Earth satellite picture shows certain buildings under construction which I know are now complete while the Microsoft picture shows them finished. See also the picture of Monarch Park below and you'll notice a significant difference in picture quality and resolution.

Microsoft Live Earth

Google Earth

Both Microsoft Live Earth and Google Earth have 3D modeled buildings. However, Microsoft Live Earth's are vastly superior both in number and in quality. Google Earth shows about 5-10 buildings in Toronto where Microsoft shows hundreds.

Google Earth: Not Much of a City Scape

Live Earth: Now That's a Downtown!

Both applications take up a large amount of RAM to run. Looking at Task Manager, Google Earth was taking up to 800 megabytes of RAM and Internet Explorer was taking up to 600 megabytes. I assume this must be for caching of data.

For smaller towns, both applications seem to be equally poor. For example, searching for Port Hope, Ontario yields a poor grainy image in both cases likely because the high resolution satellite imagery is simply not available.

Google Earth

Google Earth is much richer in terms of content overlays and more flexible in being able to turn them on and off. My personal favourite is the Panaromio overlay which geo-maps personal photographs to locations from hundreds of thousands of users around the world.

Microsoft Live Earth has a feature called "Birds Eye" which does a map of photographs to the map taken from different angles. This feature is not available at all in Google Earth.

Microsoft Live Earth has very nice traffic reporting features. I was able to check the 401 highway in Toronto, see the busiest spots to avoid and also it provides construction updates as well. Google Earth does not have this feature at all.

Both applications provide directions. However, Google Earth has a feature that allows you to "play" the route. The camera will follow the route through the directions showing you in 3D how to drive to your destination.

Is Google Analytics Having Trouble Keeping Up?

Since I started this blog, I have been using Google Analytics to measure traffic to the site. In general, its a fabulous service and works really well, until recently.

I've noticed that in the past month or so, Google Analytics seems to have trouble keeping up with the traffic. I've noticed that the numbers reported are lower than expected and stay that way for 1-2 days and then as presumably the engine crawls through the right logs in queue, it catches up.

One way to tell for sure whether you're having a similar problem is compare your Google Analytics numbers to your Ad Sense numbers. Ad Sense seems to be more accurate and is updated in real time. Google Analytics is queue based and so tends to need some time to catch up. I've found in recent days that Ad Sense gives me a better indication of short term stats than Analytics.

For my blog, this is not a huge problem. But this is yet again a mark against the "software as a service" model - would you want to trust your corporate infrastructure to a queue in the cloud that can get bogged down because of an influx of unexpected growth?

Ad Words and Negative Content

The great site Worse Than Failure had a few images today of examples where legitimate ads were being shown side by side with negative content:

Having ad sense on my blog, I have noticed the same thing. I wrote a couple posts on the failure of Canada's tax software, and if you go to this post the ads displayed are for tax software!

Isn't this a pretty basic requirement for online advertising, e.g. don't put my ad for Levi's Jeans on a web page where the content is "I hate Levi's Jeans!"? Why after 10+ years of online advertising has the experts at Google and others figured out how to targets ads to the right content. In my example, the right ads would be those marketing to people reading about the failures of the tax system, e.g. tax lawyers, H&R block, etc. and not tax software providers.

I'm Somehow Ranked 6 for the Google Phrase "Buy Books"

I have an Amazon affiliate store (see the link to the right). It has a few books in it, but not much more. I have never received a sale through this link.

Yet my friend Jordan pointed out today that if you search on "Buy Books" the link to my store shows up as ranked number 6 in Google's list!

How is this possible? What does this really say about Google's ability to filter results? Is it simply because I'm a random Amazon store?

Google Spreadsheets has less features than Excel for Windows 3.1

I was playing with Google Spreadsheets today and I found it lacking in some basic features. There are features that I remember being in Windows 3.1 Excel (Version 2.0?) that are missing in Google:

• Data extrapolation
• Double click a column to auto-size it
• Copying and pasting a column and having it re-size properly
• If you have a border that is boxed on all sides and then click the left border, it doesn't do anything.
• Clicking on ctrl to add non-contiguous cells together.
• Field names
• Autosorting of data
• Pivot tables
• Spell check
• Validation support
• Group and Outline
• Ability to fit print results on a page, see print preview in the spreadsheet

These are basic functions that I use all the time. In addition to these pretty basic features that are missing, the overall feeling of doing data entry is very slow - responsiveness is something that needs to be instant in a spreadsheet and there are too many operations that feel too slow for serious number crunching work.

So while Google Spreadsheet seems like a nice idea, its not something I could ever see replacing Excel.

See also my post previously on these types of "Internet applications" replacing desktop applications - looking at the current crop of examples, even a PocketPC running Windows CE has more application power than what's currently the best of the Internet applications.

Desktop R.I.P? Don't Count on It!

There is an interesting article in this month's Wired with the title, Desktop R.I.P. The argument goes that as more people use applications like Google, Yahoo Mail, Flickr, SalesForce, etc. the need for a desktop will go away and that we'll all live in this Internet-based world where we can have a virtual desktop sitting on some hosted server or servers.

Personally, I don't think this is going to happen for a number of reasons:

1. Hosted applications are fundamentally rented software. In most cases, the rent also scales with the load, e.g. number of users, number of transactions, etc. While this is also true with desktop/server software (maintenance fees, upgrade fees, etc.) there is typically more flexibility in options.

2. Hosted cloud based applications are a return to the "dumb-terminal" concept that has been thrown around and largely rejected for the past 30 years. Why have a $1500 machine on your desk when all it does is act like a green-screen terminal? The PC revolution over the past 20-30 years has shown that having some local hardware makes a huge impact on the experience of software.

3. There are large software segments that will never go cloud, for example games. Do you think that Electronic Arts is going to be creating the equivalent of Google Desktop anytime soon?

4. Desktops are cheap. Servers are expensive. I can buy a desktop for $1199 at MDG. The equivalent Dell server is about $3600, albeit with better power management, smaller footprint, etc. However, in general the economic models don't make much sense - why give up a cheap, non-redundant client and push all that computing to highly redundant servers that are double or triple the price for the equivalent power?

In addition, desktops are getting cheaper. If the economic engine was that desktops were getting more expensive, then a hosted model could come in as a low-cost alternative. But desktops get cheaper every year and in the case of mobile devices, video game consoles, etc. they're subsidized. So why throw away all that local computing power?

5. The desktop is no longer a single desktop - a desktop is now a TV appliance, a mobile phone, a traditional desktop, an XBOX 360, a kiosk, etc. While it is possible to boil all these down to a browser based experience, the experience is pretty lack-luster and compromises the power of these devices in their local context.

Forget desktops - just look at your blackberry and compare your blackberry email built into the client vs. going to mobile.msn.com on your WAP browser. Which one would you choose?

6. Your data sitting on someone else's servers - for the majority of killer apps, this is a non-starter, especially if the data is in the United States. If you host your CRM data in a US data centre, you've now exposed your entire customer database to the reach of the Patriot Act.

So I think the idea of the desktop death watch is premature at best. In fact, it was supposed to die a while ago with the network computer. The Network Computer was supposed to kill the desktop in the mid-90s. The exact same vision as Google's - why have that expensive desktop running local apps when you could replace with a terminal and run everything in the cloud (and of course pay Oracle/Sun a fee for it). Oracle and Sun in the mid 90's were the Google of today - they had publicly declared war on Microsoft and used the exact same arguments - Windows is too expensive, too flaky, too hard to maintain, etc. so just give up on desktops and move to the cloud where everything will be secure, cheap, etc.

How many Network Computers do you see kicking around today?