SharePoint vs. Confluence Comparison

I was at a client site this week and they currently have Confluence and are moving to SharePoint 2007 and wanted to understand the differences between the two products.  I have personally implemented both products in different environments and like both of them.  Each as its strengths and weaknesses.

Here is a side by side comparison:

	Confluence	SharePoint 2007
Company	Made by Atlassian in Australia.   In my experience, Atlassian has wicked customer service, a great support network, etc.	Made by Microsoft.  SharePoint is one of Microsoft’s most successful products and Microsoft has made a major commitment to the product.  Support in my experience is standard product support but doesn’t have the tender love and care you get from Atlassian staff.
Consulting Support	Limited, but in some markets quite good.  Look for open-source experts as Atlassian has strong ties to the open source community.  The software itself is Java based.	Huge consulting market from small shops to major consulting companies such as Accenture.  One of the major pluses of the platform is the talent pool to help implement and configure it.
Price	Free if you are a non-profit company and you ask them nicely.  Server license is available ranging from $800-12000 depending on number of users.	Pricing starts at about $5000 but for intranet cost is per named user at about $75-94 a user.  200 users would cost about $20,000 vs. about $4,000 for confluence.
Search	Good with confluence content and Microsoft documents.  No federated search model included.  Did you mean feature allows for synonym searches.	Enterprise level search engine with relevance ranking, synonyms, large scale searches (up to 500,000 documents easily and up to 50 million in theory), and federation model.  Out of the box search can index other web sites, file servers, etc. and plugins through third party products can index other systems such as OpenText, Oracle, SAP, etc.
Look and feel customization	If you’re changing the logo or basic colours you’re fine.  If you want to create a whole new interface, you will find it a challenge.  Confluence uses SiteMesh templates which are a set of XML/HTML files that need to be customized directly.   In my direct experience (and I have done lots of HTML editing), wading through the library of templates is not a trivial undertaking.	SharePoint supports customization at multiple levels through separation of layout, look and feel, and master pages.  SharePoint Designer is provided as a GUI based tool for creating new themes as well.  Template system is significantly simpler than Confluence in my experience to customize.
WIKI	Stronger than SharePoint in supporting WIKI mark-up.  Rich text editor is not bad but not as strong as SharePoint.	WIKI template provided as a site template.  WIKI mark-up is limited to embedding links in text, but rich text editor make WIKI mark-up less needed in most cases.
Security	LDAP based.  Authorization model is 100% stored in Confluence, so groups need to managed within the product.  Single Sign On is not provided.	SharePoint is Active Directory based by default.    Single Sign On is provided with Windows using Internet Explorer – users already logged into Windows are automatically logged into SharePoint.  Groups in SharePoint can map to Active Directory groups which makes management somewhat easier than Confluence.
Web Parts	Confluence supports a plugin model and there are some available.  Product availability is small in comparison to the third party web part market for SharePoint.	Long list of web parts are supplied out of the box for SharePoint.  Massive third party market for Web Parts including integration with major vendors such Oracle, SAP, OpenText, etc.  Web Parts can be easily built in .NET if custom parts are required.
Document Management	Version control, linking and embeddable.  However, there is no check-in/out, content types, workflow, etc.	Much stronger in SharePoint including features such as check-in/out, content types, meta-data, integration with Office 2007, workflow, authorization models, etc. that are typical of an enterprise document management system.
Integration with Office	No support.	Very strong integration as long as you’re on Office 2007.
Other Features	Blogs, discussion forums, etc. are provided.	Too many features to name, but SharePoint is an enterprise portal with dozens of additional features including: reporting integration, dashboards, dozens of web parts, workflow, forms server,  etc.  See this link for a complete list of features.

Comparing the two products is a bit of an apples to orange comparison because the markets for each are quite different.  Confluence is primarily a collaboration tool for small to medium size enterprises.  SharePoint is an enterprise portal that includes collaboration but also dozens of other portal features.  

In my direct experience, SharePoint is more user friendly for end-users – creating pages, changing content, managing documents seems to be more intuitive than in Confluence.  If you’re a power WIKI user then its less of an issue but I handed it to an HR department and found they struggled with it.  SharePoint requires a lot of pre-configuration to get it right but if its set up right I find the adoption tends to be easier.

If you really want a strong WIKI collaboration tool and don’t have much money then Confluence is a very good product.  The support you will get from Atlassian is awesome and they are constantly improving it.  However, if you’re looking at enterprise collaboration, document management, search, etc. then in my experience you may be looking at an upgrade to SharePoint in the long run as your user base becomes more sophisticated.

SharePoint is not a Database!

As users of SharePoint start using lists they start taking advantage of a very intuitive and highly configurable information management tool.  Users who have been using Excel or Access in the past start seeing SharePoint as a kind of rapid application development platform for building structured data applications.  However, this is where SharePoint’s list architecture breaks down because simply:

SharePoint is not a database!

Part of the problem is that business users designing lists are not familiar with basic database concepts - they just want to store their information.  Part of the problem is false advertising by consultants and/or Microsoft that SharePoint is a miracle platform that can do anything - it does some things well but replacing your SQL server or even Access database isn't one of them.  The other problem is there is no really good way for surfacing bi-directionally data from a database into SharePoint – the business data catalogue works only in read only mode and does not allow for updates back to the underlying data store.

Here are a list of some features that a database has that SharePoint does not support - the moment you need these features you will need a real database.

• Primary And foreign keys: one of the most basic concepts in a database is the enforcement of a unique identifier (either system generated or user supplied) that can identify a record and link across table in parent child relationships.
• Transactions: if two changes are required to go together and the second change fails you want the first change to be undone (rolled back).
• SQL language: complex queries can me written in code using a standard language.  Queries can be saved and repurposed.
• Stored procedures: database programmers can write complex routines that can be called as reusable code blocks. 
• Indexing and query optimization: used to improve performance, indexing allows the database designer to pre-index specific fields that are frequently used in queries to improve performance.  In addition, most databases have optimization engines that based on what you are trying to fetch will optimize how the data is retrieved.
• Large data and binary fields: most modern databases allow you to store large binary files such as video files, large volumes of text, images, etc. in the database.  SharePoint can be made to store large files as documents but you only get one per record and a generic binary object. 
• Access outside of SharePoint: lists are accessible outside of SharePoint but only through XML or programming interfaces.  Lists are not great data stores to be used for line of business applications written independently of SharePoint because the integration is relatively poor in comparison to running a SQL query on a database.
• Cascading deletes: if a parent record is deleted, its children should also be deleted.  In most databases, this can be configured to happen automatically or else reject the delete of the parent until the children are deleted first.  This eliminates the risk of orphan child records.

These features are considered basic in databases – they have been around for decades and even the most primitive databases such as Access or FileMaker Pro have most of these features. 

Lists are great for flat, changing content especially when attached to unstructured documents.  At least in the current version, lists are not substitutes for databases and shouldn’t be considered so.  If you need a database, then you are better off building a proper database and using SharePoint as a front-end for surfacing that data through reports, dashboards, etc. or as a host for data entry components such as InfoPath forms or custom build web parts.      

Why SharePoint Projects Fail

Microsoft Office SharePoint Server 2007 (affectionately known in our circles as “MOSS”) is now at what should be at the peak of adoption.  The product has been on the market since 2007 and there is a new version coming in a year.

Yet with many of our customers, we find that MOSS has been implemented but barely gets off the ground.  There a few reasons why this seems to be happening:

• MOSS is championed by IT and then too tightly controlled.  For example, we have customers who have installed MOSS but will only use it for out of the box collaboration – no custom code, no document management, no BI, etc.  If you want an out of the box team site, then that’s part of the “service” but otherwise you will have to wait.
• MOSS starts in a single department and then never makes it out.  A department picks MOSS for their needs but then there is not enough support or sponsorship in the organization to roll it out further.
• MOSS is implemented skunk works style on a single box.  The implementation is more successful than originally envisioned and what was a “proof of concept” starts getting treated like production.  This sets up the assumption that MOSS is “Rapid Development” and eventually the box falls over under strain.
• MOSS is implemented by IT but there has been no planning for adoption and training.   MOSS sits there with a bunch of services – users simply don’t know how to use them and go on using other more reliable forms of collaboration such as email, file shares, etc. 
• The implementation gets mired in legal, HR, or branding issues as corporate citizens start to have the ability to create their own content.  Similar to the IT clamp down, the organization clamps down on the services that MOSS can provide such as blogs, wikis, My Sites, etc. because they do not have a good model for managing the risk of employees contributing their own content.

Why does this happen?  What is missing from most MOSS implementations?  In my experience, there are a few common causes:

• Too much involvement by IT, not enough involvement from business stakeholders.
• Too much focus on MOSS as a generic toolbox (document management, collaboration, search, etc.), not enough focus on MOSS as a way to automate or enable true business services.
• Poor infrastructure planning to enable scaling to the enterprise.
• Lack of training, adoption and change management.
• Lack of a killer app for the particular needs of the business
• Not enough support from executive champions
• No governance model to set expectations on how MOSS will be operated, changed and maintained.

If you are an IT Director or department champion for SharePoint, I strongly encourage you to think about these issues BEFORE you start an implementation.  We see many cases where organizations start thinking about these issues after the first implementation has already failed and the organization has soured on the platform because there was not enough investment in managing the project and an eager IT department ready to roll out a service.

Live Earth Bird’s Eye View

One of the cooler features that sets Live Earth/Maps apart from Google Maps is the birds eye view. 

Take a look at this building in Google Maps:

Now take a look at the Bird’s Eye view of the same building in Live Earth/Maps.

 

 

 

The four views are available if you hit the rotate button in the interface – these are actual photographs taken from different angles.

The difference in understanding heights of buildings, features around buildings, etc. is significant – imagine if you’re doing some house hunting and you see a neighbourhood you like.  Wouldn’t you like to see the increased detail and vantage points that you get from this orthogonal view instead of just the top down view you get in Google maps?

How To Use Microsoft Project: Task Type

One of the more confusing aspects of Microsoft Project is Task Type.  Task Type can be set to "Fixed Units", "Fixed Work", or "Fixed Duration" for each task.

How to Set Task Type

You can set task type in a couple ways:

1. If you select a task or tasks and go to task information (under project in the menu or right click and its there as well) you will find task type in the advanced tab.

2. If you go to window and click Split when in the GANTT chart view, you get a Task Form at the bottom where you can set the Task Type.

  If you select all your tasks, you can set the Task Information for every task to the same Task Type – this will lead to less confusion when you are adjusting your plan because all the task types will be consistently set.

Effort Driven

There is also a check box (which is checked by default) that says that tasks are “Effort Driven”.  When this is checked, it is assumed that work stays constant even if you adjust the other two measures.  So for example, if you change a 40 hour task from 1 week duration to 2 weeks duration, project will keep the 40 hours of work but decrease the number of resources required to compensate (only 50% of a resource in this case).

This may not be what you want and I find particularly with using Fixed Duration having this turned off is highly desirable because want to adjust the work, not the units.  For example, what I want is to tell project, “I want to schedule this task with 1 person for 2 weeks” and have it calculate work to be 80 hours.  If I adjust the schedule to be 3 weeks, I want the work to increase by 50%.  Similarly, if I am working in Fixed Duration and I double the resources, I want the duration to stay constant and for the work to double.  This will only happen if “Effort Driven” is turned off. 

How Task Type Works

Any task can be adjusted using three measures: Units, Duration or Work.  Because each measure is dependent on the other, at least one of the other two must be impacted.  By changing the task type, you are telling project which measure you are going to adjust and which measure should be automatically re-calculated.  Below is a chart that shows how Project reacts based on the task type:

 

Field that you change	Field calculated if the task is Fixed Units	Field calculated if the task is Fixed Work	Field calculated if the task is Fixed Duration
Work	Duration is recalculated	Duration is recalculated	Units are recalculated
Duration	Work is recalculated	Units are recalculated	Work is recalculated
Units	Duration is recalculated	Duration is recalculated	Work is recalculated

Basic Tips for Using Task Type Efficiently

Here are some basic tips to using Task Type efficiently and to avoid confusion:

• Pick the task type that makes sense for you and use it consistently.  I tend to think in Duration with Effort Driven turned off first and what I want is to calculate the number of hours (and therefore cost) required to do the work.  Other PMs like thinking in work instead and want to adjust the schedule instead.  Pick an approach before you start and be consistent with it.
• You can switch the task type without impacting your plan.  Therefore, feel free to change the task type for the specific scenarios below.

Scenarios to use Specific Task Types

Fixed Duration

I used Fixed Duration when I have a good understanding of the schedule and the team and I want to calculate cost/effort.  A good example of this is doing a plan for requirements gathering interviews.  I know that I have 2 weeks of interviews, 1 week of documentation and 2 days of client review and I know that I need a Business Analyst and a Technical Architect in each one at 100%.  Fixed Duration works really well because I can adjust the schedule really easily and the work effort is calculated for me.

Another good use for Fixed Duration is % allocations for overhead resources.  Project Managers for example tend to be used on an overhead basis for the duration of the project on a fairly constant basis.  If the project takes longer you will need more project management.  It’s also easier to think of assigning a PM at 30% to a project for the entire duration than to say, “I need 85 hours of project management for this project).  With Fixed Duration, you can simply assign the PM for the entire duration and adjust the % allocation and the work effort is automatically calculated.

Fixed Work 

A good use of Fixed Work is when you want to try and fit a fixed amount of hours into a fixed schedule.  By fixing the work, you can adjust the number of resources you need to impact the duration.  For example, let’s assume you have 1280 hours of work and you want to try and jam that work into a fixed 8 week period.  With Fixed Work turned on, you can set the allocation of resources to 400% in order fit the fixed amount of work into a duration you wish.

Fixed Units

I generally find that using Fixed Units is my least desirable option simply because in most real world scenarios adjusting the number of resources required is the most flexible option.

One scenario where Fixed Units might be appropriate is for department planning.  In a department, the number of fixed units is constant (the number of full time employees) and as a Director you want to know how much work your team can produce.  What tends to happen in these scenarios is the queue of work is constant and the duration flexes – you end up telling the poor person at the end of the priority list that their project won’t get done until next year.

In this case, setting Fixed Units would allow you to lock in the resource pool based on the size of your department and then adjust the work to calculate the resulting schedule.  Based on the schedule, you would then be able to tell your stakeholders how many projects can be done in the coming year.

I hope that provides some guidance on how to use this key function in Microsoft Project – I have found even very senior Project Managers do not fundamentally understand how Duration, Effort and Units work together and how to model them in Microsoft Project.  If you understand the conceptual framework and the rules that Microsoft Project uses, you will find that there are less errors in your plan, less confusion and less tearing your hair out when you find Project adjusts your numbers in ways you didn’t intend.

Applying the Software Development Life Cycle in a Web 2.0 Context

For the past 20 years or so, the "Software Development Life Cycle" has been the process for defining, designing, constructing, and deploying applications. Various methodologies for building software have evolved over time, e.g. Waterfall methods, spiral method, rational unified process, extreme programming, etc. that prescribe different processes for development of software. However, all of them regardless of methodology have some common basic assumptions that I believe have now changed in a Web 2.0 world. These basic assumptions are as follows:

• Software development involves engineering code artifacts that implement some sort of requirements specification. In waterfall methods, the requirements are defined up front. In more iterative or agile methods the requirements are iterated through cycles. But in all these methodologies, the primary purpose of developing software is custom developing software based on these requirements.
• The "Application" is a deployable, atomic set of artifacts that can be deployed through a series of steps to a particular environment. The "Application" is used by a client that engages with the application. Traditionally, the user of the application does not get to change or redefine the application except through very rigidly defined configuration mechanisms.
• Software development historically has required a large amount of testing in order to ensure that a) there are no bugs in the software and b) the software complies with requirements. Software is tested before it is released to users.
• Software development is done by a team of experts that understand how the software works, how it can be modified and how to manage the overall development cycle. Software developers and end-users of the software do not really collaborate on building the software - end-users consume the software as it is released.

In the media, the old traditional form of Push media is being replaced by user generated content, collaboration, and Pull from users. The distinction between producers and consumers has been blurred so that online news hubs become more like information radiators than centralized distribution points.

Software development, in my opinion, is going through a similar revolution to the mainstream media. It is happening in a slower fashion because the software "industrial complex" is still highly engaged in generating software through the traditional means and because it generally works especially for large enterprise applications. However, in certain spaces and where consumers of software are engaged, the traditional model of a software development team pushing out software that is consumed by a user is starting to break down.

In order to make my point, let me give you a few examples:

• Software applications can be created via "Mash-ups" that allow end-users to combine existing application components into new applications.
• Portal applications such as Microsoft SharePoint are blurring the distinction between application components and user modifiable objects. For example, a Document Library in SharePoint part of an application or is it a user object?
• Cloud based services are now being released and combined in a variety of different methods.
• Clients such as Microsoft Office 2007 are now sophisticated enough to serve as application development platforms in themselves. For example, a traditional "reporting application" can now be built quite easily using Excel 2007 pointing at a data warehouse.

In addition, users expectations have changed dramatically has Web 2.0 has encouraged them to collaborate, blog, share information and build their own customized spaces. When a 12 year old can over night create their own blog, organize a protest online, share pictures, integrate Google Maps, and add dozens of widgets to their personalized home page, the idea of waiting 3 months for the next software "release" by an ivory tower software team quaint at best and non-responsive at worst.

In the corporate world, IT departments are struggling with this challenge in application spaces that encourage collaboration and customization with end-users. Business users do not want to wait 3 months for a static report - they want a drill down BI tool that allows them to build their own "application" (whatever that now means). In the portal world, IT departments are struggling with how much control they provide their end-users to redefine the underlying structure of meta-data, document types, user profiles, etc. In the SharePoint world for example, its difficult to tell where the "application" stops and user generated content begins because the platform is designed to encourage modification and collaboration by office workers.

On the other hand, the risks and challenges that the software development life cycle has helped solve over the past 20 years have not disappeared because we have moved from a code centric, monolithic, and IT controlled software development cycle to a collaborative, distributed and user centric model:

• How are changes "deployed" when an office user can simply make a change in production to a content type, a work flow, a user profile definition?
• How are the impacts of changes "tested"? For example, I can change the layout of my blog, add widgets to it, etc. but can I really test those changes before they become visible to the world (in my blog software, I have the ability to "preview" which is sort of a test but not quite the same as a proper QA environment)?
• How are application componentized when changes are done by potentially hundreds of users working collaboratively? What constitutes a "release"? For example, on Facebook, I create a Facebook Page that has a set of content, some groups, some events, etc. but there is no way to package these changes up into a deployable unit.
• How are changes versioned and backed-up? I have this problem with my blog for example. If I make changes to my blog "application" through changing the layout, modifying the HTML, etc. there is no easy way for me to preserve these changes as a version. I cannot roll-back to a previous version either.

Software development methodologies have good answers to these challenges and over the past 20 years have matured to address these issues through test driven development, source code control, build management, project management, quality assurance, etc.

As an industry, we have a similar challenge to the mainstream media - how do we adapt these worth while processes so that we can continue to deliver high quality software to our customers while adapting to an environment where our users are going to be creating software with us in a highly collaborative manner? How do we recreate the software development life cycle for a Web 2.0 world?

Key IT Challenges for 2009

Here are some of the key IT challenges I see for 2009 based on feedback from our customers, the media, and my own understanding of key technologies coming on the scene. In addition, these challenges reflect the broader economy which is going to go through a global recession in 2009, impacting IT budgets in a number of different ways (see my previous post on this issue).


Challenge #1: Maximizing Existing IT Investments

In the past 2 years, there have been a significant amount of investments made in new technology platforms that were supposed to empower office workers to higher productivity and less dependance on IT. Products like content management systems, portals, data warehouses, analytics, CRM, ERP, etc. all have become increasingly mainstream over the past couple years as the economy boomed and CIOs were looking to make long term investments.

This year, in this time of restraint, I see a key challenge for all IT departments in maximizing existing investments that were made over the past two years. For example, how many companies have invested in SharePoint and are only using it to power their intranet? I have many customers who are looking to see how can they can justify their SharePoint platform investment through more advanced applications. For consultants, this is an opportunity to sell more vertically integrated, business centric solutions and move away from selling the latest technology and this will continue for at least another year.

Challenge #2: The Proliferation of Virtual Machines

The virtual machine as a server platform has really hit the IT mainstream. Virtualization has been around for years but thanks to a very mature platform with VMWare ESX and an upcoming push from Microsoft Hyper-V, the use of virtual machines is expanding rapidly.

Virtual servers have started replacing physical ones, first with non-core systems such as development servers to now in some cases full production environments.

One of the key challenges with Virtualization is that because it becomes so easy to spin up a new server, the number of servers proliferates, leading to VM sprawl. Management of virtual servers doesn't easily map from the physical world especially when doing tasks such as asset management (how do you manage a virtual asset). IT managers will spend a lot of time in 2009 coming to terms with these challenges.

Challenge #3: Web 2.0 Goes Mainstream

As technologies go mainstream, the language, terms and concepts that were reserved for leading edge technologists land in the hands of business managers, stakeholders and consumers. Unfortunately, they get them typically without the deep technical or even conceptual understanding.

This happened a couple years ago with the concept of "Portals". Everyone wanted a portal. Portals went from being these esoteric technologies to jargon words in marketing magazines. The same is happening now with concepts like "Blogs", "Wikis", "Web 2.0", etc.

I'm now receiving RFPs from government agencies with requests to "use Web 2.0 technologies" with no rationale, purpose or objectives. Some manager has now heard of Facebook and so now wants it on their web site. As technology consultants, the challenge will be seperating the myth from the fact, selling the real effort it takes to use these technologies effectively and trying to guide stakeholders to realize relevance from what has now become a Fad.

Challenge #4: Storage

Even a few years ago, it was a big deal to have a Storage Area Network (SAN). SANs were reserved for large organizations with lots of data. With the proliferation of media files, volumes of emails and documents, and no discipline when it comes to archiving in most organizations, managing the ever increasing demand for more storage is becoming a major headache. With hard drives getting increasingly cheaper the perception that storage is nearly free is pervasive (I can go buy a terrabyte backup drive now at BestBuy for about $100). Unfortunately, corporate storage is nowhere near as cheap when the requirement includes high performance, highly redundant and low power consuming. For example, at my last organization we spent almost $50,000 implementing a SAN in order to gain faster I/O access for our core databases. The SAN has 28 drives in it, is highly redundant, etc. but even with its several terrabytes of storage it will fill up rather quickly if not managed well.

Challenge #5: Search and Document Management

I mean search here in the broadest sense - finding stuff. Go try and find a document on your corporate intranet. Try and locate a file on your file server. Try and understand the taxonomy or meta-data structure for your document repository (assuming your organization has even defined one).

Creating documents is now so easy that they have proliferated. In addition, there is typically in most organizations very poor meta-data strategies for indexing data and immature archiving strategies to remove old documents. For example, at my last position we had a file server with about 4 terrabytes of documents. We did a scan of the thousands of documents on that server and found that over 80% had not been touched in the past 2 years. Even if they are not being accessed they act as an information clog when trying to find important documents.

Document management used to require very expensive and proprietary software such as OpenText, Documentum, HummingBird, etc. and was reserved for companies such as law firms who had very regulated and structured document repository. But with lower cost tools such as SharePoint you can quite easily now implement some decent meta-data taxonomies and start to catalogue, version control and archive documents in a methodical way.

The biggest challenge isn't the technology - its getting commitment from the business to organize the thousands of old documents and migrate them into the new structure. This is a massive effort and requires significant involvement from the business. While there are some tools to help with this effort, the clean-up effort of document migration is still a continuing challenge.

Corporate IT Departments Trends for 2009

Corporate IT departments are, at the best of times, the internal consulting organization for their organizations and provide strategic technology services to help propel the overall success of the business. The role and objectives of the corporate IT department will change in 2009 as the economy continues to lag, technologies mature and organizations that saw "Web 2.0" as bleeding edge now start to adopt some of more mainstream applications of online technologies.

Here are some of the key trends I see in 2009 for the corporate IT department. For consultants such as myself, this will have a major influence on how we can continue to provide high value to our internal IT partners by providing services and solutions that contribute to the value proposition for the corporate IT department in 2009.

Corporate IT Departments Will Lag Behind in Upgrades

With budgets tightening, corporate IT departments will take the hit in upgrades for both hardware and software. IT departments will be asked to do more with less and those departments who may have planned on an Office 2007 upgrade, a SharePoint 2003 to 2007 conversion project, a .NET framework 1.1 to 2.0/3.0/3.5 upgrade project, server refreshes, etc. will be asked to defer them to save costs.

Scrutiny on Capital and Cash Flow

Typically, large scale expenditures can be amortized over a 3-5 year period. For example, if a Director IT needs to purchase $500K in new servers, they can typically budget these over a 3-5 year amortization period so they don't get hit with the entire cost in year one. However, from a cash flow perspective, the money still gets spent and in cases where cash is tight (e.g. credit crunches) these types of cash outlays will be dramatically curtailed.

As a result, this will lead to arrangements with providers where costs can be paid on a monthly basis, e.g. leases, software as a service, outsourcing, financing, fee for service, etc. where cash flows out of the organization in a more even way than a large capital expenditure.

Long Term Consulting Under Attack, Short Term Consulting Will Increase

As budgets tighten, the following tends to happen during the budget planning cycle:

1. Full time hires are deferred
2. High cost consultants are targeted
3. Projects have to show very quick ROI

If you are a $200 / hr consultant that has been working on a project for 2 years and the project will continue for another 2 years, then expect that cost to be reviewed and for department managers to try and squeeze out consultants and replace them with cheaper internal resources.

However, because full-time hires become blocked, the only way to get new work done in the short term tends to be through direct consulting costs. In addition, because the business does not have long term visibility they will be hesitant to take on a full-time resource as a long term commitment. If the business stakeholder can justify the ROI in the short term, they will tend to outsource the work simply because the Corporate IT department cannot hire any full time resources to do the work. In addition, outsourcing the work as a fixed cost to a consulting firm is less risk and the cost is directly attached to the cost of the project, making it more tangible for a business stakeholder than a resource hire.

Incremental Optimization

For both revenue and profit, the goal for 2009 will be to optimize as much as possible. For example, technologies such as SEO, analytics, data mining, reporting, CRM (especially for those who already have an existing platform), channel management, marketing optimization, etc. will sell very well to organizations who have over the past 2-3 years have been growing their customer base but now need to focus on retention, optimization of marketing dollars and efficiency. Technologies that can help do this will continue to get the green light as they produce ROI faster than brand new initiatives and they are less risk for a CIO who is under budget pressure.

Layoffs for Many, Indispensable Positions for a Few

In tough times, there are typically layoffs. Depending on the speed of action required, layoffs are many times done without proper planning and project/application coverage (the worst case is where you see companies having to hire back the people they just laid off on contract!). For those who are laid off, it means that they are now looking for a job. But for those who are left standing, they can find themselves now indespensable where there used to be three people who knew how to fix an application and now there is only one. In some cases, the one left is a consultant!

For those who are in this position, it provides leverage to those left behind. If you are the only developer who now has access to source code and can run a build, you were kept because they couldn't afford to dump you. If you can stretch to grab the work that was being done by other developers and demonstrate the flexibility and tenacity to help clean up the mess inevitably left behind by these changes then this will be looked at favourably by the company.

For consultants, there are opportunities to fill the gaps left behind in the rush to lay off staff through short term and long term consulting arrangements, especially if the consultant has had previous experience with the applications in the organization. There are also opportunities to pitch alternatives to lay-offs without adequate transition through outsourcing arrangements. A service company can provide flexibility in resource ramp-up or ramp down that is difficult for an internal department to manage especially in a time of crisis. In addition, a consulting company has the advantage of being able to work in stealth, allowing for a Corporate IT Director for example to quietly outsource key functions to an outsource provider while planning a lay off so as to avoid disruption and continue service levels.

Opportunity for Entrepreneurialism

There is an opportunity for IT departments to re-invigorate through a focus on entrepreneurialism. For those IT leaders who can show the business a clear plan to short term revenue, increased profit, etc. this will tend to get more attention than more themes such as scalability, efficiency, etc. Entrepreneurs focus on delivering high value services on a shoe-string budget - this is what is required in an economic down turn.

Business stakeholders will also be more apt to listen when they are also hungrier for more revenue. For example, many sales businesses will find their traditional sources of revenue tapped out and will be looking around for new revenue sources. If the Corporate IT Department has some creativity and can partner effectively, they can help provide some great new ideas for using Web 2.0 technologies, creating online campaigns, optimizing CRM data, using data mining techniques, etc. to squeeze more revenue out of existing channels or to open up new ones.

For consultants, this is the ONLY way to sell business in 2009 - either demonstrate how to squeeze more revenue/profit out of existing channels or show how a very small investment in a new channel can lead to large growth potential. Be prepared to take risks and be entrepreneurial with your customers in how they pay for your services, how much risk you take on, etc. and this will also be attractive to customers who are hesitant to spend money.

Focus on Business Solutions that Make Revenue or Profit, Not Technology

This is always the case with business stakeholders, but in times of restraint this will be even more pronounced: Corporate IT Departments who are focused on generating concrete business centric solutions that create revenue or profit opportunities will be prioritized.

If you are a Corporate IT Department or one of its service providers and your focus is heavy technology buzzword projects like: SOA, Cloud Computing, Web 2.0, Architecture, Software Development Life Cycle, Enterprise Data Warehouse, ERP, etc. then you will be in trouble in 2009. For example, at one of the banks here in Canada for example, I know that their list of thirty major IT has been cut to less than ten for 2009.

Projects where existing horizontal technologies in place are now leveraged to create additional value will have significantly more priority. For example, we have many customers who have previously bought Microsoft SharePoint 2007 but aren't leveraging it effectively or maximizing its use. Incremental investments in adding verticalized and targeted applications on top of this already existing investment will show quicker pay off than investments in additional infrastructure technologies.

Business Intelligence is More Than Just a Fancy Graph

"Business Intelligence" is a hot topic these days. There are companies that produce business intelligence products such as Microsoft, Business Objects, Dundas Charts, and so on that focus on visualization of data with the approach that by providing a simplified view of complex data it will enable key stakeholders to make decisions.

A good example is Dundas Charts - the company produces an amazing set of widgets and controls for building dashboards, charts, and visualization of data.



The price is right too - you can buy these fancy BI controls for a license of about $1000.

However, in my experience with Business Intelligence applications, I have come to the following basic conclusion:

Visualizing data is relatively easy.
Obtaining the data in an enterprise environment is hard.
Defining what the data means is even harder.

Let me give you an example. Wait Times are calculated in Ontario and presented to the public in order to show accountability for the funds being invested into improving access to surgeries.

Here is what the data looks like:



As you can see, the indicators are quite simple. The data could have been visualized in a number of different ways such as a red-yellow-green style KPI, a guage, charts, etc. Implementing this would simply involve taking a control library like Dundas Charts and feeding it the above data to get a graphical representation. However, visualization is actually not that valuable - the average person does not need a graph to understand a basic set of numbers.

Obtaining this particular set of data is hard. It requires the synthesis of raw data coming in through automated feeds from about 150 hospitals into a central database that is then scrubbed to match patient records and wait times records together. The cost of building this application to collect the data was millions of dollars - the cost of producing the PDF file containing the numbers is significantly less.

Defining the original business rules, definitions and targets for how wait times data was to be collected was even harder. What does a "Wait" actually mean? There are in Ontario two different wait periods (called Wait One and Wait Two). Wait One is defined as the time it takes for you to get the appointment with the doctor who provide the diagnosis. Wait Two is defined as the time it takes between the point where the doctor provides the diagnosis and the point where you get the procedure. This is what the current Wait Times application tracks - Wait Two data. Different provinces have different definitions and track different wait intervals. In addition, you will notice that only certain types of procedures are represented. These were chosen through government priority - another period of business analysis that took several years to define, prioritize and fund. Similarly, the "targets" were defined through another complex process using clinical experts who invested significant time to analyze and provide recommendations on what was the clinically appropriate targets.

If you are looking at a report, a set of numbers, etc. you are looking at the tip of the iceberg - the amount of energy, work and thought that goes into defining the data and then obtaining it is enormous in comparison to building a fancy graph. In the case of Wait Times, producing the report now is done by running automated processes. The cost of the underlying system to obtain the data cost millions of dollars and took years to implement.

So the next time you look at a report, appreciate the amount of work that has gone into it. And as an IT professional, the value to the customer is in helping them to define the data and then obtain it from a variety of data sources. Produce the actual report is simple in comparison.

Live Earth vs. Google Earth - Which One Is Better?

I decided to run a comparison between Google Earth and Live Earth. Here are some of my findings.

Google Earth runs only as a separate executable. Live Earth runs as an ActiveX control within the live maps interface. Microsoft's approach is more integrated but it also means that you cannot run the application full-screen.

Google Earth seems to find locations better especially when they are vague. For example, I typed in "yonge and dundas toronto ontario" and Google Earth found the intersection. Live Earth couldnt find it and simply suggested businesses which weren't even at that location.

Google Earth's ability to handle misspelled words is much better as well. For example, I typed in "young street toronto ontario" and it was still able to find Yonge Street as an exact match. Microsoft Live Earth could not find it.

Microsoft Live Earth splits the search box between businesses and locations. The search assumes that you will split the business name and location into the two boxes. For example, I typed in "eaton centre toronto ontario" into the business box and it asked me for a location. In Google Earth, typing in the same search query finds me the exact match.

The search box in Google Earth is on the left hand side. In Microsoft Live Earth the search box is at the top of the screen. This makes a big difference if you are using a wide screen monitor. Microsoft's map ends up being very wide. Google Earth's map ends up being nice and square.

Microsoft's satellite pictures seem to be more up to date and detailed. Looking at downtown Toronto, the Google Earth satellite picture shows certain buildings under construction which I know are now complete while the Microsoft picture shows them finished. See also the picture of Monarch Park below and you'll notice a significant difference in picture quality and resolution.

Microsoft Live Earth

Google Earth

Both Microsoft Live Earth and Google Earth have 3D modeled buildings. However, Microsoft Live Earth's are vastly superior both in number and in quality. Google Earth shows about 5-10 buildings in Toronto where Microsoft shows hundreds.

Google Earth: Not Much of a City Scape

Live Earth: Now That's a Downtown!

Both applications take up a large amount of RAM to run. Looking at Task Manager, Google Earth was taking up to 800 megabytes of RAM and Internet Explorer was taking up to 600 megabytes. I assume this must be for caching of data.

For smaller towns, both applications seem to be equally poor. For example, searching for Port Hope, Ontario yields a poor grainy image in both cases likely because the high resolution satellite imagery is simply not available.

Google Earth

Google Earth is much richer in terms of content overlays and more flexible in being able to turn them on and off. My personal favourite is the Panaromio overlay which geo-maps personal photographs to locations from hundreds of thousands of users around the world.

Microsoft Live Earth has a feature called "Birds Eye" which does a map of photographs to the map taken from different angles. This feature is not available at all in Google Earth.

Microsoft Live Earth has very nice traffic reporting features. I was able to check the 401 highway in Toronto, see the busiest spots to avoid and also it provides construction updates as well. Google Earth does not have this feature at all.

Both applications provide directions. However, Google Earth has a feature that allows you to "play" the route. The camera will follow the route through the directions showing you in 3D how to drive to your destination.

LiveEarth not in the top 10?!

Whoever organized LiveEarth really missed the blogosphere. A few points of evidence:

1. There are only 2049 blog posts on LiveEarth?
2. LiveEarth as a search term isn't hitting the Technorati top 10.
3. The LiveEarth facebook group only has 678 members.

Whoever is responsible for creating Buzz on the blogsophere for LiveEarth is doing a poor job - Paris Hilton and the IPhone are doing much better than LiveEarth, a supposedly global change event.

I'm going back to enjoying the music on the old fashion technology called the Television! :)

Problems with Live Earth Broadcast

Compared to the Live8 broadcast that AOL did, the LiveEarth broadcast is a disappointment. The resolution of the video is significantly poorer and yet I'm finding that the stream is having a lot of problems handling the load. There are a lot of blackouts, frequent rebuffering of content and there have already been complaints that the site only works in Internet Explorer.

This could have been a fantastic showcase for Silverlight - with the quick download of the plugin Microsoft could have showcased its new technology including some fantastic video streaming.

But instead we see choppy video, disconnected audio and very slow response rate on the interface.

What a missed opportunity!

Why Agile Software Projects are Good for Customers

We have two software projects that are going on at the Foundation. Project #1 is just finishing. It was not managed by me and it was done using a traditional waterfall death march type methodology. Project #2 is being managed by me and we're using a traditional Agile project management approach.

Comparing the two projects really shows the different in the benefits and obligations of the customer in an agile project:

• Project #1 is 100% over original estimate for no apparent reason. The customer gave up trying to even get a date from because they haven't seen any deliverables, don't even really understand the scope of work and just want it done.
• In Project #1, there was about 1 month just spent trying to define requirements. There were dozens of meetings trying to nail down field mappings with no code being written in the meantime (which starts to explain why the project is late). The right people were never in the room and the developers got different answers from different people.
• In Project #1, despite the investment in time, the output in terms of requirements documentation and certainty of scope was quite poor. Despite the investment, the requirements still changed up until the last week before testing including major business rule corrections.
• In Project #2, we've taken a traditional Agile PM approach. We include the customer in every meeting (usually once a week) and we are using a combination of use cases and wire frames to document requirements. This is the first time in the organization that the customers have seen either format so its a bit of a learning curve in terms of how to read them. But it took about an hour of reading through use cases before the customer fundamentally got the value of such a document.
• In Project #2, I got my lead technical architect to start prototyping on day one. By week 2 of the project, we already have a prototype writing in .NET using SQL Server 2005. The data model is about 50% done. But before even the documentation is done we have a working model.
• In Project #2 for project management we've taken an Iteration Cycle of 2 weeks as the basis for development. We have an architecture/planning/requirements spike for the first 2-3 weeks but have already started coding. Every Iteration has a plan to deliver a list of use case implementations and related wire frames as fully functional code. We have formal show and tell sessions at the end of each cycle.
• In Project #2, we have a proper project plan with real estimation. Estimation is done at the wire frame and use case level and tasks are broken down to about 1-3 days in duration. In my experience anything more granular is a waste of time and anything more becomes unpredictable and subject to error.
• In Project #2, we will be using Test Driven Development and Customer Testing all the way through the project as well as Continuous Integration.

The experience so far from a customer perspective has been really interesting so far:

• Customer participation on Project #2 has been extremely high and involved. The customer role has been really well defined and our stakeholders have been good at providing the right level of input.
• There is a good trust relationship and a sense of team - it isn't a US vs. Them approach. We're not delivering software for a customer - we're all as a team delivering on a project.
• Our ability to shift priorities and change the schedule is highly responsive. For example, our marketing department is working with the team to figure out the content for the application and its taking a lot more time than expected. So we've started on the back end logistics sections and left the front-end until the end in order to give the marketing folks the most leeway. By having check ins every two weeks we can easily adjust.
• In addition, because the application is built in vertical slices instead of horizontal ones, we don't have to have 100% of one tier done before moving to the next one. We can simply move from front end to back and hook together some disconnected pieces as they come available.
• The project is simply more fun and more of a team based approach.

Some recommendations to software project teams in treatment of their customers (whether internal or external) in an agile context:

• Don't patronize your customer because they've never done software projects before. Don't treat them like idiots because they've never read use cases or UML diagrams or database models before. If you take an hour to introduce the concepts and travel light, you'll find that customers "get it" rather quickly and then see the value of these requirements tools.
• Always remember that the customer is the end user and not a stakeholder representing an end user. For example, many Directors of Marketing act as the "customer" in web project teams. They're not. They're representing a customer. Everyone on the team needs to always focus on the end customer and stakeholders always have to be able to filter out their own biases and represent the end user successfully.
• Customers will buy into an agile project management approach if it produces results - period. If you cannot deliver software on time or its poor quality, an Agile approach won't save you and will quickly degenerate. Shorter iteration cycles means there is no where to hide, so talent/experience and ability to deliver are critical early success factors.
• Show and Tells are critical to the Agile Project. We use to have them once a week on Fridays. Make Show and Tells Fun! We use to dim the lights and use the projector and have popcorn and snacks. Even if its some boring application, hype it up and celebrate it.
• Customers should be the leading evangelists as well - some customers have been taught to be the evil task masters and try to crush development teams because they think it gets the team to work harder. I can tell you from the other side of the team that it doesn't work - developers work less, produce poorer quality and flip the bozo bit on you in short order.
• One of they values of an Agile project team is transparency and team work. Transparency is a double edged sword - customers have to be prepared to hear the bad news with the good and to help the team prioritize effectively. Good customers develop a relationship with the team that allows them to be trusted - its critical in order to encourage honest dialogue.

Overall, so far Project #2 is doing much better than Project #1. It has lots of risks and we may not be able to deliver all the functions asked for in the time line. But the requirements are well understood, communication is open and use cases are prioritized. We've already had conversations about what parts of the scope could drop if needed and contingency plans if we go late. Customers are in the loop at all times and a core member of the team.

How to Use Microsoft Project Efficiently

I've worked with a lot of project managers who don't really know how to use MS Project efficiently. I was trained well on MS Project because in the consulting business, you have to be able to build million dollar project plans in days for sales pitches. Our PMO figured out a number of ways of using Microsoft Project more efficiently.

Here are my recommendations:

1. Get rid of the GANTT chart by dragging it to the right. Use the extra space for some additional columns.

2. Go to Window and click on Split. This will bring up the Task Form dialogue box at the bottom. You will find that entering data in the task form is much more efficient and visible than any other method.

3. Go to Tools->Options and click on Schedule. Change the Default Task Type to Fixed Duration and click off the box marked "New Tasks are Effort Driven". The reason is that when you're estimating, you'll find in most cases its easier to think in duration and allocating a % of a persons time across that duration than thinking about person hours. For example, instead of thinking of a task taking 25 hours, its easier to think about a task taking 3 days at 100%. Its also easier to think about a task as taking 2 people x 3 days, especially if you're days are 7.5 hours (do you think of a task as taking 50% of a week or 18.75 hours?)

4. Don't put in any absolute dates in your plans. Make all your dates based on dependencies except for the basic project start date.

5. For tasks that you want to go in parallel, use the SS predecessor type to make sure that multiple tasks start at the same time.

6. Go to Format->Text Styles and click on Item to Change. Select Critical Path and change the colour to something obvious (I use BOLD RED). This will help you in making sure that every task has the appropriate predecessor set - if it doesn't, you'll notice that the critical path stops half way through your project plan. It should highlight tasks from the beginning to the very end of your project.

7. Lets assume you have a project plan that is way past your target timeline and you want to now parallel your effort by adding more people. Many project managers will start creating streams of tasks in parallel with complex dependencies. But here is a simpler way if all you want to know is how many people you need to complete the work on time - simply take your tasks, change them to fixed work and increase the percentage to 400%, 500%, etc. until you get your timeline in the ball park. If its 500%, that means you need 5 full time developers to get the work done.

NOTE: the one draw back to this approach is that it doesn't tell you when each task will be done. But in my experience, this is not that useful as development projects tend to have 50 tasks that can largely be done at any time over a longer period and you simply spread the load across a pool of developers until the work is done. If there are specific tasks that are dependent, then set them up as a seperate stream. But if its simply a pool of tasks that need to get done this technique is much faster and simpler to manage.

8. Don't allocate overhead resources to tasks (e.g. your PM, your TA, your QA lead, your stakeholders, etc.) unless you have very specific requirements. In most cases, you can estimate an overhead resource at a % allocation across an entire project phase. For example, I'll allocate a PM at 30% for the duration of the project, a TA for 20%, etc. This again is a bit of a cheat - if you want to plan that a PM is going to be busy at 100% on Wednesday and 20% on Thursday and 40% on Friday, then this technique fails. In my experience, this is not the way overhead resources work and in the end having your project plan reflect this micro-detail is not worth the work to maintain it.

9. Here are a few ways to calculate contingency:

• Click on Insert Column and pick Cost1. Right click on it and select customize field. Then click on Formula and type in Cost*1.10. This will calculate contingency at 10%. If you want 25% contingency then use Cost1*1.25. Make sure that Calculation for Task and Summary Rows has Rollup selected and Sum.
  
  NOTE: There is a bug in Project 2003 where it sometimes does recalculate fields properly. If you notice this, then go to Tools->Options->Calculation and click Calculate Now. This will recalculate the field.
  
  


• Create a task called Contingency and set a price for it. This is good for setting an arbitrary contingency amount.


• Create a task called Contingency and assign a resource called Contingency at sufficient hours to drive up the price. The advantage of this approach over the one above is that it means your total hours assigned also has contingency built it and your hourly rate stays consistent with what it would be without the contingency assigned.


• Create a task called Schedule Contingency and set the duration to an amount of time with no resources attached. Bury the task where you think the schedule risk is going to occur - I usually put it into the QA phase of the project as this is the mostly likely area where you'll need the contingency.

10. Save often. I've had too many cases where Project crashes and there is no automatic recovery. In addition, there is only one level of Undo in Project 2003 (this is fixed in 2007) so if you make some significant changes its tough to revert them.

11. Use the keyboard as much as possible. Learn keyboard shortcuts. Don't drag things around when you can copy and paste. Don't use your mouse to indent when you can click on Alt-Shift-Right Arrow. Typing in predecessors manually instead of clicking on the link task button in the toolbar generally is faster.

Hopefully, that will help speed up your MS Project work dramatically. If you work this way, you'll find that planning with MS Project is feasible. If you're doing it the traditional way by dragging grantt bars around you'll find a major speed improvement working this way.

Developing an Analytics Platform for Microsoft .NET

A few weeks ago, I posted an article on a proposed new approach to analytics. I argued that we needed a new paradigm for analytics that goes beyond the static page analysis model found in major engines such as WebTrends, Google Analytics, etc.

So this weekend I started designing one in .NET and now have a pretty decent tracking model. I started with an article I found on .NET and tracking. I took Vikram's basic ideas and expanded them a little bit to add some more general concepts.

The basic architecture looks like this:



The terminology on the diagram comes from Domain Driven Design by Eric Evans.

I've tried developing this as a pretty basic framework and made it as light as possible so that if my site gets a lot it won't take a lot of a) memory or b) load to persist the data. But the architecture achieves what I wanted to achieve, namely it allows me to save event data based on userid, sessionid and affiliate referrer and to specify the nature of those events based on programming events (e.g. a transaction happening in the business domain) instead of simply tracking a logged URL.

Once the data is logged, we will design a data-mart that will suck the data out of the log and process it into some decent OLAP cubes for reporting purposes.

If you want to send me feedback on the ideas, please feel free to leave your comments...

Google Spreadsheets has less features than Excel for Windows 3.1

I was playing with Google Spreadsheets today and I found it lacking in some basic features. There are features that I remember being in Windows 3.1 Excel (Version 2.0?) that are missing in Google:

• Data extrapolation
• Double click a column to auto-size it
• Copying and pasting a column and having it re-size properly
• If you have a border that is boxed on all sides and then click the left border, it doesn't do anything.
• Clicking on ctrl to add non-contiguous cells together.
• Field names
• Autosorting of data
• Pivot tables
• Spell check
• Validation support
• Group and Outline
• Ability to fit print results on a page, see print preview in the spreadsheet

These are basic functions that I use all the time. In addition to these pretty basic features that are missing, the overall feeling of doing data entry is very slow - responsiveness is something that needs to be instant in a spreadsheet and there are too many operations that feel too slow for serious number crunching work.

So while Google Spreadsheet seems like a nice idea, its not something I could ever see replacing Excel.

See also my post previously on these types of "Internet applications" replacing desktop applications - looking at the current crop of examples, even a PocketPC running Windows CE has more application power than what's currently the best of the Internet applications.

MIX07 72 Hour Conversation Archive

Here is a list of the posts I made throughout the MIX07 conference. These were done in near real time as sessions ended. Enjoy!

• The Impact of Bloggers on Public Relations
• The Future of N-Tier Architectures
• Silverlight 1.1 and .NET Development
• Evolution of Declarative Programming in .NET
• LINQ finally brings O/R Mapping to .NET
• Dynamic Data Controls
• Feedback on Expression Web

MIX07 Update: The Impact of Bloggers on Public Relations

CIO.com reports that there was a disappointing talk at the MIX07 conference on "digital entertainment". I didn't attend they keynote (preferring to take the time to put up some blog posts instead) but the immediate reaction shows the power of bloggers controlling news.

Before even the talk was concluded, people were blogging about how poor the talk was in near real time. It shows how important the blogging community has become in the following ways:

1. News is released in real time by thousands of bloggers. MIX07 and Silverlight both made the top ten searches in Technorati. Managing this spin cycle in real time becomes increasingly important for any PR department or agency.

2. The news dies quickly. I was blogging every 3-4 hours, and my the time my post was up on Technorati, the last one was already buried off the first couple pages. The best thing you can do from a PR perspective is to flood the network with news - this helps to bump older content down the list and traffic to those news items dies as a result. I have personally seen a massive drop-off even with my own posts after about 24 hours in traffic coming from news search engines such as Digg, Shoutwire, Technorati, etc.

3. Reading is happenning in real time as well. I was blogging throughout the MIX07 and I could see through analytics each post was getting traffic within minutes of being posted. Its hard to tell whether these readers are people at the conference or those simply visiting from the outside and experiencing it vicariously, but if you can generate enough awareness there will be a large amount of readers who will grab your message in near real time.

4. Its a bit hard to understand the impact of blogging on conference goers themselves. For example, did enough people start reading blog entries or news sitting in the talk that they got the message to walk out? Given the number of blackberries and wireless laptops in the room, its a possibility.

MIX07 Update: The Future of N-Tier Architectures

As a Director of IT, one of my principle jobs is to recruit talent. During our interview process, we start by asking a candidate about a project they designed, coded and deployed and we present them this picture:



The interview discussion focuses on answering what replaces the question mark - what is the appropriate N-Tier Architecture for their application. You can use this question for any developer because its open to multiple solutions - the junior developer might answer that there is an ASP.NET page that connects using the supplied wizards to an ADO.NET connection that then talks to the database. For a more seasoned architect, they should answer with a much richer domain model, database access layer, proxies for web service or other distributed calls, monitoring and configuration management layers, class factories, etc.

That's why the question is an interesting question for interviewers - it forces the discussion to question design decisions. Why did they use ADO.NET? Why did they choose web services instead of remoting? Why did they choose to keep code in separate objects instead of in the code-behind files? Why did they choose particular design patterns, etc.?

As web technologies have evolved is that they have focused on adding maturity in the tiers. A classic ASP site with some ASP page code, some VB 6.0 objects and a connection to the database through ODBC is now an ASP.NET page, a code-behind file, some domain model objects, some ADO datasets, web service proxy classes, etc. Now there are dozens of options on implementation of that middle-tier and the role of the architect is to decide amongst a selection of options which is appropriate for the particular application being built.

What has happened as these technologies have evolved is an explosion of tiers. In the last enterprise application I managed, we had the following tiers: 1) reverse proxy; 2) HTTP handlers/modules; 3) ASP.NET page; 4) code-behind files; 5) value objects or business entities; 6) web service client stub classes; 7) web service handler classes; 8) data access layer objects; 9) Microsoft Enterprise Library; 10) the Database!

At least it was still easier than using classical J2EE! :)

What seems to be happening as Microsoft technologies evolve is a resolving of this tier explosion through a few different strategies:

1. Use of LINQ to simplify data access and consolidate the data tier into some SQL like statements.

2. Code generation techniques for automating plumbing code (the LINQ designer for example will build your domain model simply by pointing at a schema).

3. Smarter web controls that require less code - Dynamic Data Controls for example.

4. Integration of .NET into Silverlight to put more smarts into the client instead of requiring web service calls for every function.

5. Greater focus on Plain Old .NET Objects - LINQ in particular moves from the heavy Dataset based approach that ADO.NET uses as value objects and can instead populate simple lightweight classes that the developer can design as required.

6. Greater focus on declarative languages - LINQ and also use of IronPython and PHP for example. Instead of focusing on imperative languages that specify the algorithms on how to do instructions, a declarative language allows the developer to focus on business rule declaration and specify what is required and delegate implementation to the framework.

7. On the client side, dramatic improvements in rendering in Silverlight especially in 3D based applications.

All these developments are promising. If the technology works as advertised then development of basic plumbing operations such as CRUD operations with a database, web service calls, etc. should be dramatically easier to code and maintain.

However, "works as advertised" is always the challenge. Developers who have worked with O/R mappers for years have experience with frameworks that claim to hide away details but then also hide away performance bottlenecks, poor implementation strategies and developers are increasingly dependent on the framework to work in the background.

And from what I have seen at the conference so far, its really too early to tell - we're still at the "Just use the wizard" stage where Microsoft demos a cool technology by running a dinky unrealistc demo. Don't be the fool to take the sample code and consider it ready for production without some serious design review!

MIX07 Update: Silverlight 1.1 and .NET Development

I went to a session this morning on Silverlight 1.1 Alpha which is the next version of the product. It features a number of interesting features:

1. Complete CLR .NET based framework - write your Silverlight applications in VB.NET, C#, IronPython, etc. using standard .NET classes. What ships with Silverlight is a stripped down version of the .NET framework in order to shrink the plug in size. For example, XML document reading is included, but XSD validation and XPATH are not included.

2. Silverlight runs 100% client side all the managed code runs within the plug in and will run the same on a Macintosh as it does on the PC (although I have heard this before with Java applets and it never quite worked 100%).

3. Serving Silverlight apps can be done any web server that can serve files. So if you want to deploy your apps to an Apache server, no problem.

4. If you choose to serve your Silverlight apps from ASP.NET you get some advantages and tight integration with your ASP.NET page.

5. Debugging support is built in, including debugging via a proxy client on a Macintosh. Debugging is standard Visual Studio debugging with breakpoints, stepping through, etc.

6. LINQ for objects is supported with LINQ for databases (DLINQ) and LINQ for XML (XLINQ) on the road map.

7. There is the ability to interop between HTML and Silverlight. You can access the HTML DOM from Silverlight, and you can expose your objects in Silverlight to javascript by using a simple [Scriptable] attribute.

8. Web services support is very strong with wizards for generating all the stub code including support for Windows Communication Foundation (WCF).

In general, I was pretty impressed and the 1.1 version is the obvious target for traditional .NET developers. From my perspective, the killer application for Silverlight isn't the animation sitting on web page that makes some animation go in as static content. Where its going to pulverize Flash is where you have .NET application developers who want a rich user experience without having to learn an entirely new programming language in ActionScript. Even in 1.1, the CLR in Silverlight seems much more robust than typical ActionScript code and more importantly for .NET developers is familiar territory.