SharePoint vs. Confluence Comparison

I was at a client site this week and they currently have Confluence and are moving to SharePoint 2007 and wanted to understand the differences between the two products.  I have personally implemented both products in different environments and like both of them.  Each as its strengths and weaknesses.

Here is a side by side comparison:

	Confluence	SharePoint 2007
Company	Made by Atlassian in Australia.   In my experience, Atlassian has wicked customer service, a great support network, etc.	Made by Microsoft.  SharePoint is one of Microsoft’s most successful products and Microsoft has made a major commitment to the product.  Support in my experience is standard product support but doesn’t have the tender love and care you get from Atlassian staff.
Consulting Support	Limited, but in some markets quite good.  Look for open-source experts as Atlassian has strong ties to the open source community.  The software itself is Java based.	Huge consulting market from small shops to major consulting companies such as Accenture.  One of the major pluses of the platform is the talent pool to help implement and configure it.
Price	Free if you are a non-profit company and you ask them nicely.  Server license is available ranging from $800-12000 depending on number of users.	Pricing starts at about $5000 but for intranet cost is per named user at about $75-94 a user.  200 users would cost about $20,000 vs. about $4,000 for confluence.
Search	Good with confluence content and Microsoft documents.  No federated search model included.  Did you mean feature allows for synonym searches.	Enterprise level search engine with relevance ranking, synonyms, large scale searches (up to 500,000 documents easily and up to 50 million in theory), and federation model.  Out of the box search can index other web sites, file servers, etc. and plugins through third party products can index other systems such as OpenText, Oracle, SAP, etc.
Look and feel customization	If you’re changing the logo or basic colours you’re fine.  If you want to create a whole new interface, you will find it a challenge.  Confluence uses SiteMesh templates which are a set of XML/HTML files that need to be customized directly.   In my direct experience (and I have done lots of HTML editing), wading through the library of templates is not a trivial undertaking.	SharePoint supports customization at multiple levels through separation of layout, look and feel, and master pages.  SharePoint Designer is provided as a GUI based tool for creating new themes as well.  Template system is significantly simpler than Confluence in my experience to customize.
WIKI	Stronger than SharePoint in supporting WIKI mark-up.  Rich text editor is not bad but not as strong as SharePoint.	WIKI template provided as a site template.  WIKI mark-up is limited to embedding links in text, but rich text editor make WIKI mark-up less needed in most cases.
Security	LDAP based.  Authorization model is 100% stored in Confluence, so groups need to managed within the product.  Single Sign On is not provided.	SharePoint is Active Directory based by default.    Single Sign On is provided with Windows using Internet Explorer – users already logged into Windows are automatically logged into SharePoint.  Groups in SharePoint can map to Active Directory groups which makes management somewhat easier than Confluence.
Web Parts	Confluence supports a plugin model and there are some available.  Product availability is small in comparison to the third party web part market for SharePoint.	Long list of web parts are supplied out of the box for SharePoint.  Massive third party market for Web Parts including integration with major vendors such Oracle, SAP, OpenText, etc.  Web Parts can be easily built in .NET if custom parts are required.
Document Management	Version control, linking and embeddable.  However, there is no check-in/out, content types, workflow, etc.	Much stronger in SharePoint including features such as check-in/out, content types, meta-data, integration with Office 2007, workflow, authorization models, etc. that are typical of an enterprise document management system.
Integration with Office	No support.	Very strong integration as long as you’re on Office 2007.
Other Features	Blogs, discussion forums, etc. are provided.	Too many features to name, but SharePoint is an enterprise portal with dozens of additional features including: reporting integration, dashboards, dozens of web parts, workflow, forms server,  etc.  See this link for a complete list of features.

Comparing the two products is a bit of an apples to orange comparison because the markets for each are quite different.  Confluence is primarily a collaboration tool for small to medium size enterprises.  SharePoint is an enterprise portal that includes collaboration but also dozens of other portal features.  

In my direct experience, SharePoint is more user friendly for end-users – creating pages, changing content, managing documents seems to be more intuitive than in Confluence.  If you’re a power WIKI user then its less of an issue but I handed it to an HR department and found they struggled with it.  SharePoint requires a lot of pre-configuration to get it right but if its set up right I find the adoption tends to be easier.

If you really want a strong WIKI collaboration tool and don’t have much money then Confluence is a very good product.  The support you will get from Atlassian is awesome and they are constantly improving it.  However, if you’re looking at enterprise collaboration, document management, search, etc. then in my experience you may be looking at an upgrade to SharePoint in the long run as your user base becomes more sophisticated.

SharePoint is not a Database!

As users of SharePoint start using lists they start taking advantage of a very intuitive and highly configurable information management tool.  Users who have been using Excel or Access in the past start seeing SharePoint as a kind of rapid application development platform for building structured data applications.  However, this is where SharePoint’s list architecture breaks down because simply:

SharePoint is not a database!

Part of the problem is that business users designing lists are not familiar with basic database concepts - they just want to store their information.  Part of the problem is false advertising by consultants and/or Microsoft that SharePoint is a miracle platform that can do anything - it does some things well but replacing your SQL server or even Access database isn't one of them.  The other problem is there is no really good way for surfacing bi-directionally data from a database into SharePoint – the business data catalogue works only in read only mode and does not allow for updates back to the underlying data store.

Here are a list of some features that a database has that SharePoint does not support - the moment you need these features you will need a real database.

• Primary And foreign keys: one of the most basic concepts in a database is the enforcement of a unique identifier (either system generated or user supplied) that can identify a record and link across table in parent child relationships.
• Transactions: if two changes are required to go together and the second change fails you want the first change to be undone (rolled back).
• SQL language: complex queries can me written in code using a standard language.  Queries can be saved and repurposed.
• Stored procedures: database programmers can write complex routines that can be called as reusable code blocks. 
• Indexing and query optimization: used to improve performance, indexing allows the database designer to pre-index specific fields that are frequently used in queries to improve performance.  In addition, most databases have optimization engines that based on what you are trying to fetch will optimize how the data is retrieved.
• Large data and binary fields: most modern databases allow you to store large binary files such as video files, large volumes of text, images, etc. in the database.  SharePoint can be made to store large files as documents but you only get one per record and a generic binary object. 
• Access outside of SharePoint: lists are accessible outside of SharePoint but only through XML or programming interfaces.  Lists are not great data stores to be used for line of business applications written independently of SharePoint because the integration is relatively poor in comparison to running a SQL query on a database.
• Cascading deletes: if a parent record is deleted, its children should also be deleted.  In most databases, this can be configured to happen automatically or else reject the delete of the parent until the children are deleted first.  This eliminates the risk of orphan child records.

These features are considered basic in databases – they have been around for decades and even the most primitive databases such as Access or FileMaker Pro have most of these features. 

Lists are great for flat, changing content especially when attached to unstructured documents.  At least in the current version, lists are not substitutes for databases and shouldn’t be considered so.  If you need a database, then you are better off building a proper database and using SharePoint as a front-end for surfacing that data through reports, dashboards, etc. or as a host for data entry components such as InfoPath forms or custom build web parts.      

Why SharePoint Projects Fail

Microsoft Office SharePoint Server 2007 (affectionately known in our circles as “MOSS”) is now at what should be at the peak of adoption.  The product has been on the market since 2007 and there is a new version coming in a year.

Yet with many of our customers, we find that MOSS has been implemented but barely gets off the ground.  There a few reasons why this seems to be happening:

• MOSS is championed by IT and then too tightly controlled.  For example, we have customers who have installed MOSS but will only use it for out of the box collaboration – no custom code, no document management, no BI, etc.  If you want an out of the box team site, then that’s part of the “service” but otherwise you will have to wait.
• MOSS starts in a single department and then never makes it out.  A department picks MOSS for their needs but then there is not enough support or sponsorship in the organization to roll it out further.
• MOSS is implemented skunk works style on a single box.  The implementation is more successful than originally envisioned and what was a “proof of concept” starts getting treated like production.  This sets up the assumption that MOSS is “Rapid Development” and eventually the box falls over under strain.
• MOSS is implemented by IT but there has been no planning for adoption and training.   MOSS sits there with a bunch of services – users simply don’t know how to use them and go on using other more reliable forms of collaboration such as email, file shares, etc. 
• The implementation gets mired in legal, HR, or branding issues as corporate citizens start to have the ability to create their own content.  Similar to the IT clamp down, the organization clamps down on the services that MOSS can provide such as blogs, wikis, My Sites, etc. because they do not have a good model for managing the risk of employees contributing their own content.

Why does this happen?  What is missing from most MOSS implementations?  In my experience, there are a few common causes:

• Too much involvement by IT, not enough involvement from business stakeholders.
• Too much focus on MOSS as a generic toolbox (document management, collaboration, search, etc.), not enough focus on MOSS as a way to automate or enable true business services.
• Poor infrastructure planning to enable scaling to the enterprise.
• Lack of training, adoption and change management.
• Lack of a killer app for the particular needs of the business
• Not enough support from executive champions
• No governance model to set expectations on how MOSS will be operated, changed and maintained.

If you are an IT Director or department champion for SharePoint, I strongly encourage you to think about these issues BEFORE you start an implementation.  We see many cases where organizations start thinking about these issues after the first implementation has already failed and the organization has soured on the platform because there was not enough investment in managing the project and an eager IT department ready to roll out a service.

Applying the Software Development Life Cycle in a Web 2.0 Context

For the past 20 years or so, the "Software Development Life Cycle" has been the process for defining, designing, constructing, and deploying applications. Various methodologies for building software have evolved over time, e.g. Waterfall methods, spiral method, rational unified process, extreme programming, etc. that prescribe different processes for development of software. However, all of them regardless of methodology have some common basic assumptions that I believe have now changed in a Web 2.0 world. These basic assumptions are as follows:

• Software development involves engineering code artifacts that implement some sort of requirements specification. In waterfall methods, the requirements are defined up front. In more iterative or agile methods the requirements are iterated through cycles. But in all these methodologies, the primary purpose of developing software is custom developing software based on these requirements.
• The "Application" is a deployable, atomic set of artifacts that can be deployed through a series of steps to a particular environment. The "Application" is used by a client that engages with the application. Traditionally, the user of the application does not get to change or redefine the application except through very rigidly defined configuration mechanisms.
• Software development historically has required a large amount of testing in order to ensure that a) there are no bugs in the software and b) the software complies with requirements. Software is tested before it is released to users.
• Software development is done by a team of experts that understand how the software works, how it can be modified and how to manage the overall development cycle. Software developers and end-users of the software do not really collaborate on building the software - end-users consume the software as it is released.

In the media, the old traditional form of Push media is being replaced by user generated content, collaboration, and Pull from users. The distinction between producers and consumers has been blurred so that online news hubs become more like information radiators than centralized distribution points.

Software development, in my opinion, is going through a similar revolution to the mainstream media. It is happening in a slower fashion because the software "industrial complex" is still highly engaged in generating software through the traditional means and because it generally works especially for large enterprise applications. However, in certain spaces and where consumers of software are engaged, the traditional model of a software development team pushing out software that is consumed by a user is starting to break down.

In order to make my point, let me give you a few examples:

• Software applications can be created via "Mash-ups" that allow end-users to combine existing application components into new applications.
• Portal applications such as Microsoft SharePoint are blurring the distinction between application components and user modifiable objects. For example, a Document Library in SharePoint part of an application or is it a user object?
• Cloud based services are now being released and combined in a variety of different methods.
• Clients such as Microsoft Office 2007 are now sophisticated enough to serve as application development platforms in themselves. For example, a traditional "reporting application" can now be built quite easily using Excel 2007 pointing at a data warehouse.

In addition, users expectations have changed dramatically has Web 2.0 has encouraged them to collaborate, blog, share information and build their own customized spaces. When a 12 year old can over night create their own blog, organize a protest online, share pictures, integrate Google Maps, and add dozens of widgets to their personalized home page, the idea of waiting 3 months for the next software "release" by an ivory tower software team quaint at best and non-responsive at worst.

In the corporate world, IT departments are struggling with this challenge in application spaces that encourage collaboration and customization with end-users. Business users do not want to wait 3 months for a static report - they want a drill down BI tool that allows them to build their own "application" (whatever that now means). In the portal world, IT departments are struggling with how much control they provide their end-users to redefine the underlying structure of meta-data, document types, user profiles, etc. In the SharePoint world for example, its difficult to tell where the "application" stops and user generated content begins because the platform is designed to encourage modification and collaboration by office workers.

On the other hand, the risks and challenges that the software development life cycle has helped solve over the past 20 years have not disappeared because we have moved from a code centric, monolithic, and IT controlled software development cycle to a collaborative, distributed and user centric model:

• How are changes "deployed" when an office user can simply make a change in production to a content type, a work flow, a user profile definition?
• How are the impacts of changes "tested"? For example, I can change the layout of my blog, add widgets to it, etc. but can I really test those changes before they become visible to the world (in my blog software, I have the ability to "preview" which is sort of a test but not quite the same as a proper QA environment)?
• How are application componentized when changes are done by potentially hundreds of users working collaboratively? What constitutes a "release"? For example, on Facebook, I create a Facebook Page that has a set of content, some groups, some events, etc. but there is no way to package these changes up into a deployable unit.
• How are changes versioned and backed-up? I have this problem with my blog for example. If I make changes to my blog "application" through changing the layout, modifying the HTML, etc. there is no easy way for me to preserve these changes as a version. I cannot roll-back to a previous version either.

Software development methodologies have good answers to these challenges and over the past 20 years have matured to address these issues through test driven development, source code control, build management, project management, quality assurance, etc.

As an industry, we have a similar challenge to the mainstream media - how do we adapt these worth while processes so that we can continue to deliver high quality software to our customers while adapting to an environment where our users are going to be creating software with us in a highly collaborative manner? How do we recreate the software development life cycle for a Web 2.0 world?

Corporate IT Departments Trends for 2009

Corporate IT departments are, at the best of times, the internal consulting organization for their organizations and provide strategic technology services to help propel the overall success of the business. The role and objectives of the corporate IT department will change in 2009 as the economy continues to lag, technologies mature and organizations that saw "Web 2.0" as bleeding edge now start to adopt some of more mainstream applications of online technologies.

Here are some of the key trends I see in 2009 for the corporate IT department. For consultants such as myself, this will have a major influence on how we can continue to provide high value to our internal IT partners by providing services and solutions that contribute to the value proposition for the corporate IT department in 2009.

Corporate IT Departments Will Lag Behind in Upgrades

With budgets tightening, corporate IT departments will take the hit in upgrades for both hardware and software. IT departments will be asked to do more with less and those departments who may have planned on an Office 2007 upgrade, a SharePoint 2003 to 2007 conversion project, a .NET framework 1.1 to 2.0/3.0/3.5 upgrade project, server refreshes, etc. will be asked to defer them to save costs.

Scrutiny on Capital and Cash Flow

Typically, large scale expenditures can be amortized over a 3-5 year period. For example, if a Director IT needs to purchase $500K in new servers, they can typically budget these over a 3-5 year amortization period so they don't get hit with the entire cost in year one. However, from a cash flow perspective, the money still gets spent and in cases where cash is tight (e.g. credit crunches) these types of cash outlays will be dramatically curtailed.

As a result, this will lead to arrangements with providers where costs can be paid on a monthly basis, e.g. leases, software as a service, outsourcing, financing, fee for service, etc. where cash flows out of the organization in a more even way than a large capital expenditure.

Long Term Consulting Under Attack, Short Term Consulting Will Increase

As budgets tighten, the following tends to happen during the budget planning cycle:

1. Full time hires are deferred
2. High cost consultants are targeted
3. Projects have to show very quick ROI

If you are a $200 / hr consultant that has been working on a project for 2 years and the project will continue for another 2 years, then expect that cost to be reviewed and for department managers to try and squeeze out consultants and replace them with cheaper internal resources.

However, because full-time hires become blocked, the only way to get new work done in the short term tends to be through direct consulting costs. In addition, because the business does not have long term visibility they will be hesitant to take on a full-time resource as a long term commitment. If the business stakeholder can justify the ROI in the short term, they will tend to outsource the work simply because the Corporate IT department cannot hire any full time resources to do the work. In addition, outsourcing the work as a fixed cost to a consulting firm is less risk and the cost is directly attached to the cost of the project, making it more tangible for a business stakeholder than a resource hire.

Incremental Optimization

For both revenue and profit, the goal for 2009 will be to optimize as much as possible. For example, technologies such as SEO, analytics, data mining, reporting, CRM (especially for those who already have an existing platform), channel management, marketing optimization, etc. will sell very well to organizations who have over the past 2-3 years have been growing their customer base but now need to focus on retention, optimization of marketing dollars and efficiency. Technologies that can help do this will continue to get the green light as they produce ROI faster than brand new initiatives and they are less risk for a CIO who is under budget pressure.

Layoffs for Many, Indispensable Positions for a Few

In tough times, there are typically layoffs. Depending on the speed of action required, layoffs are many times done without proper planning and project/application coverage (the worst case is where you see companies having to hire back the people they just laid off on contract!). For those who are laid off, it means that they are now looking for a job. But for those who are left standing, they can find themselves now indespensable where there used to be three people who knew how to fix an application and now there is only one. In some cases, the one left is a consultant!

For those who are in this position, it provides leverage to those left behind. If you are the only developer who now has access to source code and can run a build, you were kept because they couldn't afford to dump you. If you can stretch to grab the work that was being done by other developers and demonstrate the flexibility and tenacity to help clean up the mess inevitably left behind by these changes then this will be looked at favourably by the company.

For consultants, there are opportunities to fill the gaps left behind in the rush to lay off staff through short term and long term consulting arrangements, especially if the consultant has had previous experience with the applications in the organization. There are also opportunities to pitch alternatives to lay-offs without adequate transition through outsourcing arrangements. A service company can provide flexibility in resource ramp-up or ramp down that is difficult for an internal department to manage especially in a time of crisis. In addition, a consulting company has the advantage of being able to work in stealth, allowing for a Corporate IT Director for example to quietly outsource key functions to an outsource provider while planning a lay off so as to avoid disruption and continue service levels.

Opportunity for Entrepreneurialism

There is an opportunity for IT departments to re-invigorate through a focus on entrepreneurialism. For those IT leaders who can show the business a clear plan to short term revenue, increased profit, etc. this will tend to get more attention than more themes such as scalability, efficiency, etc. Entrepreneurs focus on delivering high value services on a shoe-string budget - this is what is required in an economic down turn.

Business stakeholders will also be more apt to listen when they are also hungrier for more revenue. For example, many sales businesses will find their traditional sources of revenue tapped out and will be looking around for new revenue sources. If the Corporate IT Department has some creativity and can partner effectively, they can help provide some great new ideas for using Web 2.0 technologies, creating online campaigns, optimizing CRM data, using data mining techniques, etc. to squeeze more revenue out of existing channels or to open up new ones.

For consultants, this is the ONLY way to sell business in 2009 - either demonstrate how to squeeze more revenue/profit out of existing channels or show how a very small investment in a new channel can lead to large growth potential. Be prepared to take risks and be entrepreneurial with your customers in how they pay for your services, how much risk you take on, etc. and this will also be attractive to customers who are hesitant to spend money.

Focus on Business Solutions that Make Revenue or Profit, Not Technology

This is always the case with business stakeholders, but in times of restraint this will be even more pronounced: Corporate IT Departments who are focused on generating concrete business centric solutions that create revenue or profit opportunities will be prioritized.

If you are a Corporate IT Department or one of its service providers and your focus is heavy technology buzzword projects like: SOA, Cloud Computing, Web 2.0, Architecture, Software Development Life Cycle, Enterprise Data Warehouse, ERP, etc. then you will be in trouble in 2009. For example, at one of the banks here in Canada for example, I know that their list of thirty major IT has been cut to less than ten for 2009.

Projects where existing horizontal technologies in place are now leveraged to create additional value will have significantly more priority. For example, we have many customers who have previously bought Microsoft SharePoint 2007 but aren't leveraging it effectively or maximizing its use. Incremental investments in adding verticalized and targeted applications on top of this already existing investment will show quicker pay off than investments in additional infrastructure technologies.

Business Intelligence is More Than Just a Fancy Graph

"Business Intelligence" is a hot topic these days. There are companies that produce business intelligence products such as Microsoft, Business Objects, Dundas Charts, and so on that focus on visualization of data with the approach that by providing a simplified view of complex data it will enable key stakeholders to make decisions.

A good example is Dundas Charts - the company produces an amazing set of widgets and controls for building dashboards, charts, and visualization of data.



The price is right too - you can buy these fancy BI controls for a license of about $1000.

However, in my experience with Business Intelligence applications, I have come to the following basic conclusion:

Visualizing data is relatively easy.
Obtaining the data in an enterprise environment is hard.
Defining what the data means is even harder.

Let me give you an example. Wait Times are calculated in Ontario and presented to the public in order to show accountability for the funds being invested into improving access to surgeries.

Here is what the data looks like:



As you can see, the indicators are quite simple. The data could have been visualized in a number of different ways such as a red-yellow-green style KPI, a guage, charts, etc. Implementing this would simply involve taking a control library like Dundas Charts and feeding it the above data to get a graphical representation. However, visualization is actually not that valuable - the average person does not need a graph to understand a basic set of numbers.

Obtaining this particular set of data is hard. It requires the synthesis of raw data coming in through automated feeds from about 150 hospitals into a central database that is then scrubbed to match patient records and wait times records together. The cost of building this application to collect the data was millions of dollars - the cost of producing the PDF file containing the numbers is significantly less.

Defining the original business rules, definitions and targets for how wait times data was to be collected was even harder. What does a "Wait" actually mean? There are in Ontario two different wait periods (called Wait One and Wait Two). Wait One is defined as the time it takes for you to get the appointment with the doctor who provide the diagnosis. Wait Two is defined as the time it takes between the point where the doctor provides the diagnosis and the point where you get the procedure. This is what the current Wait Times application tracks - Wait Two data. Different provinces have different definitions and track different wait intervals. In addition, you will notice that only certain types of procedures are represented. These were chosen through government priority - another period of business analysis that took several years to define, prioritize and fund. Similarly, the "targets" were defined through another complex process using clinical experts who invested significant time to analyze and provide recommendations on what was the clinically appropriate targets.

If you are looking at a report, a set of numbers, etc. you are looking at the tip of the iceberg - the amount of energy, work and thought that goes into defining the data and then obtaining it is enormous in comparison to building a fancy graph. In the case of Wait Times, producing the report now is done by running automated processes. The cost of the underlying system to obtain the data cost millions of dollars and took years to implement.

So the next time you look at a report, appreciate the amount of work that has gone into it. And as an IT professional, the value to the customer is in helping them to define the data and then obtain it from a variety of data sources. Produce the actual report is simple in comparison.