White House Data Feeds are Baby Steps

The White House announced to great fan fare that it would be releasing data feeds of its public information collected by hundreds of agencies within the US government.  They have launched the first iteration of this service.

My interest in this feed is to use this data to build mash-ups, visualization applications, and web applications that use this data.  There are many such applications that use existing data feeds such as Amazon data, Google Maps data, Microsoft Live data, etc. in interesting ways.   Amazon in particular has had an XML based REST API available for years (in fact Amazon sells a book on how to use it)  For example, Frucall provides a mobile comparison application that uses a variety of publically available APIs to compare products across retailers and present them to your mobile device. 

So with hundreds of data feeds, imagine the visualization, comparison, calculation and searching applications that could be built on US government data.  As well, imagine then directing these applications to mobile devices, XBOX 360, IPhones, Blackberries, etc.  in formats that work ideally for those platforms.

Unfortunately, based on what has been launched, this vision is not really feasible.  Here is what I can see from what is currently available:

• There are only about 50 data feeds.  Given the number of US agencies, this is a very small sample of the data available.
• The site provides data in a variety of formats such as XML, CSV, KML (google earth) and ESRI (GIS data).  The feed that would be useful for apps is the XML feed, as explained by their own site (“Better suited for consumption by automated applications capable of handling raw XML files”).  There are only 8 feeds in XML format available.
• The data is not a feed, they are files in most cases. If you click on the links, you get either a zip file containing some XML data or a site with some links to XML files.
• In some cases, the XML files are RSS feeds.  While this might be great if you want content, the “data” is really just unstructured content.
• There are no APIs, no REST and each feed has a completely different format.  There is zero consistency even for things like unique identifiers, delivery format, or XML structure.  There are also generally no available XML schemas – you just get raw XML.
• Its not clear how data is published in a timely fashion.  In most cases, you get a directory of files but there is no information on how often they are published.

So while its great to see the information being public, the US government has a long way to go before they have a useful API that could be used for application development purposes.  Given that Googgle. Expedia, Amazon, etc. have had these APIs for many years, its a lot of fanfare and not a lot delivered for this service debut – dumping a bunch of files onto a directory was a victory in 1999, not in 2009.

Proposed ISP Tax Shows Canadian Content Producers Do Not Understand the Internet

Decades ago in Canada, funds were created to encourage television production of Canadian Content. These programs were funded by the cable companies and regulated by the CRTC. These funds were created to share "Canadian Stories" in a sea of American television content that because of their inherent scale could be produced easier and cheaper.

The CRTC is now entertaining a proposal to create a similar fund through taxation of ISPs (and therefore all Internet subscribers) to fund Canadian television producers so they can continue to produce "Canadian Stories" created by Professional Content Creators. This proposal is supported by industry lobby groups such as ACTRA, Society of Composers, SOCAN, etc.

These folks want to tax every Internet subscriber (e.g. practically everyone in Canada) 3% of revenue (which of course means the price will go up by 3% to compensate) from the ISPs that provide us Internet Access.
This proposal shows a complete misunderstanding of the difference between a regulated, limited, and centralized Television Industrial Complex model and the Internet - a fundamentally distributed, decentralized and long tail model of content distribution.

Here are some fundamental differences that these lobbyists fail to acknowledge in their proposal:

• In the television age, the definition of "Content" was professional created, broadcast content. The assumption in this proposal is that Colin Mochrie, the CBC, and other CanCon stars own the culture and the MILLIONS of content creators on Facebook, YouTube, etc. in Canada are not qualified to tell their own stories.
• In the television age, the definition of "Content" is VIDEO content. On the Internet, there is no limitation and content can be everything from maps, images, videos, flash, applications, etc. Content can also be a mix from a wide variety of sources - how do you regulate content creation when it is mashed up collections from all over the world?
• In the television age, there were major barriers to content creators because of the limited shelf space. Canadians need a way to get on the air simply because they couldn't compete with American content producers. But on the Internet, this model makes no sense because anyone can put up a web site, the costs are cheap and the number of channels are unlimited. What is the barrier to making money on the Internet other that POPULARITY? There is a simple way for Canadians to support their own culture - visit the web site and support it through ad revenue like everyone else does on the Internet.
• In the television age, exporting content to other countries was extremely limited because the distribution channels were controlled by cable channels in the country. The Internet has no such barriers - if Canadians want to share their stories they can gain an audience world wide without any restriction.- In the television age, there is no integration of "Application" and "Content". Monetization of content was through a single model - distribution broadcast supported through advertising. In the Internet, the two are mixed together and there are many monetization models available. The monetization mechanisms are also distributed, e.g. I can be creating "Canadian" content that is monetized through Google AdSense in the United States. The selling, re-packaging and syndication of content through dozens of channels is both an opportunity and a challenge for Canadian content producers to claim for themselves. But the solution isn't to bypass these channels and simply tax the consumer - this is just a return to the simplified television monetization model.
• In the television age, the justification for this fund was that in order to compete with American content producers there needed to be funds to create a level playing field and encourage the telling of "Canadian Stories" because the distribution channel for content was limited both in capacity (number of channels) and revenue potential (only the top 10-20 television shows make the money). However, the Internet market is completely the opposite - the market is a long-tail, unlimited distribution channel and there are lots of examples where relatively small content distributors are making money by targeting micro-niche audiences by keeping their costs relatively low and providing a highly valuable target audience for advertisers or by providing content through subscription that funds the cost.

The CRTC is in the business of regulating traditional centralized content broadcasting and is therefore bought into the fundamental model assumptions of the television age. Canadian Television Producers who are in favour of this model seem to be wanting to keep their existing model and apply it to the Internet, because if they simply wanted to make money from content there are easier more direct Internet models available that do not require a tax on consumers. The Canadian Producers advocating for this proposal and the CRTC if it accepts it simply are trying to implement the old television age model and apply to the new Internet content model so that these folks do not have to migrate to a new way of creating, distributing, monetizing content. In addition, the entire proposal continues to question the fundamental value of funding "Professional Content Producers" as the bastion of Canadian culture when there are potentially millions of Canadians who are creating "Canadian Stories" themselves on a daily basis through blogs, Facebook, etc. Do we really need Gordon Lightfood, Colin Mochrie, the CBC, etc. to have a monopoly on culture creation when anyone in Canada can easily tell their own stories?

Online Trends for 2009

Here are some online trends to look out for in 2009.

Advertising Takes a Beating

With the sluggishness of the US economy, advertising will decline significantly and online advertising in particular will be hit hard. Online advertising will still continue to grow but at a much slower pace.

This slowdown in growth will put pressure on web 2.0 applications with lots of content but lower conversion rates. Applications such as Facebook, Flickr, YouTube, etc. will need to focus more heavily on monetizing all those collaboration interactions that millions of people now do for free every day.

There will also be a continued emphasis on Cost Per Action instead of Cost Per Click to link advertised costs to conversion. As the cost of online advertising increases (for example the cost of buying keywords) there will be increased pressure on Google and other advertising platforms to show direct linkage between advertising and conversion to revenue. This is where online advertising as a channel has had an advantage over traditional media channels.

Internet TV is Finally a Reality

The "watch video on your computer" revolution started about 10 years ago, but until this year watching the equivalent of television via the Internet really wasn't possible. Instead, users have sacrificed the video quality that even low definition television provides for increased collaboration, crowd created content, etc. In addition, the selection for mainstream content has been limited online until the last year or so.

In 2009, the compromise on quality and selection will fade away. Video sites such as YouTube, Facebook, etc. are all upgrading their video streaming so that the videos uploaded by users are streamed at much higher resolution. In addition, sites such as Hulu now stream hundreds of mainstream television programs in almost broadcast quality. While not able to yet compete with HDTV, video on the Internet will now be viable as a full-screen, sit in the living room type experience.

Part of the challenge has been the source video - consumer digital cameras tended to be lower resolution video than television. However, the latest digital cameras produce very good video quality including my Panasonic camera that can shoot HD video in wide screen format.

Web 2.0 will Become Mainstream

Don Tapscott's new book on the Net Generation points out that when technology becomes boring and we stop talking about it then the next level of innovation is possible. Web 2.0 will simply become The Internet as usual and we'll stop talking about it as this new found technology revolution (until we think of the next one of course).

Applications like YouTube, Facebook, etc. are now so mainstream that your mother uses them. Applications such as Facebook, MySpace, YouTube, etc. have all already demographically shifted to the mainstream from being in the hands of the power users, the young and the affluent.

Mobile Applications Will See Massive Growth

With the launch of the IPhone and its inevitable competitors, the phone now becomes a viable application platform. IPhone is as significant for mobile applications as IPod was for downloadable music.

Another mobile device seeing large sales is the Amazon Kindle. It allows users to purchase books and view them in a mobile device. The idea of e-books has been around for over 10 years but it takes a device like the Kindle with its complete vertical integration to drive the adoption.

Geotagging and Live/Google Earth

As the amount of content increases, one of the simplest ways to filter content is through geography. In addition, with the maturity and increased use of mapping interfaces such as Live Earth, Google Earth, Google Maps, etc. both as destination sites and as mash-ups, geotagging of content, pictures, videos, and people will be a vital way of filtering content in 2009.

One of the simplest examples is the integration of Panaramio with Google Maps. In Google Maps if you look up any address it will show you related photos from Panaramio, all geotagged on upload by location.

Change of Strategy From The RIAA/MPAA

For the past several years, the global culture industry has been taking people to court over copyright infringement. Approximately 30,000 people have been sued in the US for copyright infringement over the past 5 years. The impact on file sharing has been no real impact on file sharing but has made the RIAA one of the most hated organizations worldwide.

It appears that after 5 years of harassment law suits, the RIAA has decided to change its strategy and use a softer approach. In addition, new technology and business partnerships will allow the media conglomerates to start making money off their content no matter who uploads it. For example, Universal now stands to make over $100 million in ad revenue through a deal inked with YouTube that uses search terms to identify user contributed music videos that map to Universal's artists.

Casual Entertainment Over Hard Core Gaming

The success of the Nintendo Wii (sold out again this Christmas!) as well as other casual games such as the Sims and Spore will continue to provide online entertainment for the masses. Collaboration, casual games and customization will continue to push the mainstream entertainment market. Games like RockStar, Wii Fit, etc. will continue to provide mainstream entertainment while leaving more hard core games to cater to the hard core 18-30 male demographic.

See the above note as well on platforms like IPhone and Facebook - these are perfect jumping points for casual gaming as they cater to an audience that might play a game of solitaire but isn't going to be spending hours with Call of Duty.

The casual gaming market is still in massive growth mode and this will continue in 2009.

Difference, Story Telling and Community

This isn't a trend, but it will continue to be true in 2009 as it always has been: story telling, difference and community building continue to drive the content creation process.

The revolutionary part of the Internet is the ease at which these tools can be employed for someone to tell a story, create some innovative content and enable a community. The growth of blogs shows that everyone has something to say and the size of the audience it generates is how well it connects to its community of interest. Determining "success" depends on the objective - it may simply be distributing family pictures or it may be making your opinion heard by millions as Barak Obama did in 2008.

The Key Challenge in 2009: Conversion

I have a friend who is very passionate about informing parents about Toy Recalls. She has an active blog, she uses Twitter and Facebook and she is able to get her message out to several hundred viewers a day. When we discussed her "success criteria", she seemed pleased that she had grown this community from zero but her next question was, "ok so now what?". This is I think the key question for all this collaboration activity going on - how do we convert the energy of all this discussion, file sharing, photo sharing, tagging, blogging, etc. into some sort of concrete action so that its simply not just hanging out (although that is not a bad goal in itself).

It's as if we have got people off the couch and started them talking again. So now the community is sharing, talking, collaborating. So now what happens? How do the tools available provide methods to convert this community to action?

This is the future growth opportunity for application platforms like Facebook and you can see the evolution of these tools going in this direction already - applications like the Causes application for example are a first attempt to translate a community of action into a concrete action (in this case donating money to charity).

The Growing Phenomenon of Internet

This brings me back - I found this old CBC new video circa 1993 on this new fangled "Internet".



In 1993, I was in 3rd year university and had one of the few accounts thanks to some connections through my parents who were academics. In those days, only grad students were allowed Internet accounts. This was pre-WWW where unix terminals were using protocols such as email, ftp, gopher and veronica. Mosaic (which then became Netscape) wouldn't come out for another year or so.

An another interesting historical note - "CRS" standard for Canada Remote Systems which was one of the leading Bulletin Board Systems in Toronto. They were a commercial BBS with hundreds of phone lines allowing for local chat. They figured out and tried to hook up to the new Internet in order to avoid extinction - unsuccessfully. CRS got wiped out with the rest of BBS's as the Internet took over.