Applying the Software Development Life Cycle in a Web 2.0 Context

For the past 20 years or so, the "Software Development Life Cycle" has been the process for defining, designing, constructing, and deploying applications. Various methodologies for building software have evolved over time, e.g. Waterfall methods, spiral method, rational unified process, extreme programming, etc. that prescribe different processes for development of software. However, all of them regardless of methodology have some common basic assumptions that I believe have now changed in a Web 2.0 world. These basic assumptions are as follows:

• Software development involves engineering code artifacts that implement some sort of requirements specification. In waterfall methods, the requirements are defined up front. In more iterative or agile methods the requirements are iterated through cycles. But in all these methodologies, the primary purpose of developing software is custom developing software based on these requirements.
• The "Application" is a deployable, atomic set of artifacts that can be deployed through a series of steps to a particular environment. The "Application" is used by a client that engages with the application. Traditionally, the user of the application does not get to change or redefine the application except through very rigidly defined configuration mechanisms.
• Software development historically has required a large amount of testing in order to ensure that a) there are no bugs in the software and b) the software complies with requirements. Software is tested before it is released to users.
• Software development is done by a team of experts that understand how the software works, how it can be modified and how to manage the overall development cycle. Software developers and end-users of the software do not really collaborate on building the software - end-users consume the software as it is released.

In the media, the old traditional form of Push media is being replaced by user generated content, collaboration, and Pull from users. The distinction between producers and consumers has been blurred so that online news hubs become more like information radiators than centralized distribution points.

Software development, in my opinion, is going through a similar revolution to the mainstream media. It is happening in a slower fashion because the software "industrial complex" is still highly engaged in generating software through the traditional means and because it generally works especially for large enterprise applications. However, in certain spaces and where consumers of software are engaged, the traditional model of a software development team pushing out software that is consumed by a user is starting to break down.

In order to make my point, let me give you a few examples:

• Software applications can be created via "Mash-ups" that allow end-users to combine existing application components into new applications.
• Portal applications such as Microsoft SharePoint are blurring the distinction between application components and user modifiable objects. For example, a Document Library in SharePoint part of an application or is it a user object?
• Cloud based services are now being released and combined in a variety of different methods.
• Clients such as Microsoft Office 2007 are now sophisticated enough to serve as application development platforms in themselves. For example, a traditional "reporting application" can now be built quite easily using Excel 2007 pointing at a data warehouse.

In addition, users expectations have changed dramatically has Web 2.0 has encouraged them to collaborate, blog, share information and build their own customized spaces. When a 12 year old can over night create their own blog, organize a protest online, share pictures, integrate Google Maps, and add dozens of widgets to their personalized home page, the idea of waiting 3 months for the next software "release" by an ivory tower software team quaint at best and non-responsive at worst.

In the corporate world, IT departments are struggling with this challenge in application spaces that encourage collaboration and customization with end-users. Business users do not want to wait 3 months for a static report - they want a drill down BI tool that allows them to build their own "application" (whatever that now means). In the portal world, IT departments are struggling with how much control they provide their end-users to redefine the underlying structure of meta-data, document types, user profiles, etc. In the SharePoint world for example, its difficult to tell where the "application" stops and user generated content begins because the platform is designed to encourage modification and collaboration by office workers.

On the other hand, the risks and challenges that the software development life cycle has helped solve over the past 20 years have not disappeared because we have moved from a code centric, monolithic, and IT controlled software development cycle to a collaborative, distributed and user centric model:

• How are changes "deployed" when an office user can simply make a change in production to a content type, a work flow, a user profile definition?
• How are the impacts of changes "tested"? For example, I can change the layout of my blog, add widgets to it, etc. but can I really test those changes before they become visible to the world (in my blog software, I have the ability to "preview" which is sort of a test but not quite the same as a proper QA environment)?
• How are application componentized when changes are done by potentially hundreds of users working collaboratively? What constitutes a "release"? For example, on Facebook, I create a Facebook Page that has a set of content, some groups, some events, etc. but there is no way to package these changes up into a deployable unit.
• How are changes versioned and backed-up? I have this problem with my blog for example. If I make changes to my blog "application" through changing the layout, modifying the HTML, etc. there is no easy way for me to preserve these changes as a version. I cannot roll-back to a previous version either.

Software development methodologies have good answers to these challenges and over the past 20 years have matured to address these issues through test driven development, source code control, build management, project management, quality assurance, etc.

As an industry, we have a similar challenge to the mainstream media - how do we adapt these worth while processes so that we can continue to deliver high quality software to our customers while adapting to an environment where our users are going to be creating software with us in a highly collaborative manner? How do we recreate the software development life cycle for a Web 2.0 world?