Software Development Metaphors

There are lots of discussions on Software Development "methodologies". They are codified sets of practices on how to do software development "correctly". When you get down to the root of each of these methodologies, the root of all of them is a metaphor. Before you adopt a methodology such as XP, Scrum, RUP, Software Factory, etc. you should really ask yourself what metaphor you believe your particular software development practices fit into and what metaphors fit your particular organization's culture. If you start there instead of at the "here is my fancy new toolbox of processes" then adoption will be higher.

So here is a list of popular metaphors used to explain the practice of creating software...

The Traditional Software Development Metaphors

Software Development as Factory

• Heavy emphasis on frameworks (assembly line) and components (widgets)
• Use in large scale development projects (hundreds of developers) or highly distributed projects (outsourced)
• Highly centralized control of processes, tools, etc.

Software Development as Engineering

• Heavy focus on estimation techniques
• Fixes made in planning are less expensive than after “construction” has started.
• Highest project management priority is being “in control”.
• Software development as a science, e.g. objective, measurable.
• Belief that Software Development is an immature or imprecise version of engineering.

Software Development as Model/Architecture

• Architect as “GOD”
• Code is simply implementation of a model
• If the software doesn’t work, either the model is wrong or the code doesn’t implement the model properly.
• Tooling can automatically generate application from the model.
• Software models a business, a process, etc. in design. Goal of software is to accurately reflect real life processes.
• Need specialized modeling languages, expertise, and tools

Software Development as Workflow/Process

• Projects as repeatable processes.
• Workflow is defined up front, and people are compartmentalized.
• Emphasis on distribution, encapsulation and hand-off.
• Project managers are process champions – they make sure the rules are followed.

These metaphors have formed the bedrock of software development. They represent the traditional way of viewing software development as well as project management in general. They have enabled a whole toolbox of techniques, tools and templates for managing projects and software development.

The Radical Software Development Metaphors

These metaphors have become in vogue thanks to various agile methodologies, reactions against the traditional view of software and frankly a lot of failures for the traditional metaphors to come to grips with the complexities of creating software.

Software Development as Craft/Voodoo/Martial Art

• Focus on simplicity, elegance, and craftsmanship
• Toolbox approach
• Apprentice / Master / Guru model
• Software is highly dependent on people
• Software “Design” provides guide to picking solutions

Software Development as Game Playing

• Games are processes without the prescription
• Games can be open or closed, cooperative or competitive
• Rules of the game are enforced, but outcome is unknown
• Games also include collaborative activities such as sports, playing music, dance, etc.

Software Development as Book Writing/Composition/Poetry

• Iteration, drafts, reviews, etc.
• Focus on creativity within an established structure.
• Objective is known, but product is not well defined.
• Challenges in forcing creative process into rigid scope, time lines and budget are recognized.

Software Development as Learning/Experiment/Invention

• Modeling in software is about learning
• Documentation, requirements gathering is a means to collective learning
• Focus on establishment of common language
• Prototyping, experiment, measuring as a way to learn

Which one fits your culture, your organization needs, and your skill set? If you're a start-up company, then the software factory metaphor might not be the best fit. However, if you're building printer drivers with third party outsourcers in India, then perhaps it is the right model.

If you start with basic metaphors, the toolbox and methodologies such as RUP, XP, Scrum, Waterfall, etc. start to make more sense - they're expressions of the underlying metaphor. In addition, if you want to evangelize a new software development methodology (for example, those who are trying to evangelize the use of XP in a traditional RUP organization) starting with a discussion around metaphors and basic understanding of software development might help the discussion, especially with non-technical users.

One last suggestion - this discussion is a great interview topic for Project Managers. If you want to hire an "Agile" PM, then start the interview with the list of metaphors and ask them which one they believe fits their style of working. If they pick one metaphor, then get them to explain how their PM toolbox (GANTT charts, risk matrices, status reports, etc.) fit into the metaphor. The traditional toolbox can be used in an agile context if the underlying metaphor is right - there are ways to use MS Project even if you believe that Software Development is more like writing a book then building a bridge.

Project Planning in an Agile Project

Tate Stuntz has an interesting post on his blog arguing that Project Plans are obsolete and hard to maintain when doing Agile projects. In my experience on agile projects, project planning and Gantt chart making definitely has to change, but there are still ways I think you can make MS Project and Gantt charts work for you in an agile context. Here are my recommendations based on my experience:

• Learn to use MS Project efficiently (see my previous post for recommendations). I've seen PM's take 2 weeks to build a project plan - if this is you then trying to work this way in a 2 week iteration cycle means you're not moving fast enough. You have to be at the point where you could re-boot your project plan in a matter of hours so that you can adjust quickly.
• Use MS Project for planning and not for tracking. If you're working with short iteration cycles, you'll see improved tracking by simply management by walking around. In addition, measuring project velocity is more valuable than percentage complete since in an Agile project the finish line is not always defined.
• Use a project plan for iteration planning at a macro-level and not for iteration tracking within an iteration cycle. What I like doing is maintaining a project plan with a simple list of user stories in each 2 week iteration. That allows me to track dependencies between user stories pretty easily. At review time at the end of an iteration, we discuss the next iteration with our stakeholders and reset expectations by re-forecasting the plan (which takes no more than an hour of work). That gives stakeholders a good understanding of what we currently see as in and out of scope and when features may be coming up. This is highly valuable and I find is critical for buy in for stakeholders - if the response from the team is simply, "we can only see to the end of the next iteration cycle" this tends to make business users very nervous.
• If you need pretty Gantt charts, one of my favourite methods is to use Visio - it provides pretty Gantt charts for presentations without the fuss of Project.
• I tend to use the whole team as a big pool for planning iterations. This gets you out of the micro-detail of having to assign one user story to one person, another user story to another person and so on. If I have 4-5 developers on a team, that gives me 70 hours x 4-5 developers for a total iteration budget of 280-350 hours. I can then do a good estimate of each user story that includes planning, development and QA with the team. We simply add up a bunch of these stories into the budget and that becomes the iteration. The way to do this in MS Project is to take your steps and instead of assigning individuals assigned to each task and running them in parallel try simply taking a generic role of "developer" and assigning them at 500% to every task in sequence. You can then tell how much time you're going to spend across the entire team without having to fuss with all the micro-dependencies.
• For preliminary planning, use low-tech Gantt chart tools such as whiteboards, post-it notes, etc. I can create a Gantt chart without MS Project using a white board, some markers and post-it notes. This works great without the overhead of the tooling getting in the way. If you want to preserve it, simply take a digital photo of it and store the picture.

I think if you follow some of these techniques as a Project Manager, you can still sleep at night without feeling completely out of control and yet move faster and leaner. In addition, you can use project planning to help sell the concept of Agile to stakeholders who aren't as comfortable with the XP style of development and give them the comfort of a traditional project plan without locking yourself into the traditional bureaucratic approach to Project Management.

Low Tech Approaches to Requirements - Part 3

For those who missed Parts 1 and 2, see the following previous posts:

• Part 1: Different Techniques on using non-tech tools for requirements gathering
• Part 2: Using existing tech tools in simpler ways to speed up requirements gathering.

One of my staff sent me this image today and I thought it was a great example of how to use low-tech approaches to conversations. He simply took a screen shot of a screen, threw it into MS Paint and then started making changes with the developer beside him.

Between them in 5-10 minutes they had a "language" for what the marks on the screen mean:

• Red lines through something mean to get rid of it
• A box with text means to insert a new column
• A line on a checkbox means that on checking it something will happen
• etc.

So instead of using UML, fancy wireframes, or some bureacractic documentation process, try simply taking a screen capture and throwing it into Paint!

Test Driven Development When Writing New Code

I have been doing TDD for several years now, and I can still not get around the recommendation to test first to the point where it breaks the code, write enough code to pass the test, and then go to the next test. In my experience this tends to work with mature code bases where there is a chunk of existing code that is being changed. However, I'm not sure this approach works for new applications that are being built from scratch. In my experience, a less purist approach seems to work for me:

• Write enough code to get something at least working minimally. If you think of a use case, I'm not sure there is much point in writing gobs of test cases before you even have a primary path through the use case. The XP methodology talks about an architectural or design spike, but in my experience there is also a coding spike. For example, it might be the first iteration that instead of writing code + tests we just write as much code as we can to push through the primary path as widely as possible instead of trying to be exhaustive in our testing.
  
• I use at least a basic unit test in NUnit as an entry point in all code debugging. That enforces the practice of having tests but also provides a useful way to gain entry into the code without web pages, windows forms, or a global main method. Using TestDriven.NET, you can simply right-click a test and run it right in studio using the debugger, NUnit, NCover, etc. and I find this higly efficient when writing new code.
  
• There does seem to be a drop off point where writing more and more tests provides less value, as in any testing effort. I'm seen some overly puritanical approaches to TDD where it was insisted that every boundary condition was to be coded up as a unit test no matter how unlikely the scenario or risk. In my experience there is a point where coding these scenarios is not worth the benefit of the test. In addition, I have found that in some cases it is still easier and better to manually test something than to try and code it up as a test.
  
• I definately like the idea of using build scripts and NUnit tests when making modifications to code. However, In my experience the tests actually don't change as much as the underlying code. In some cases there are new requirements but in many cases they are implementation changes. As long as the existing tests are still green after the change, then no new tests are required.
  
• There seems to be a trade-off between quick progress and the feeling of "being on a roll" and the pace of TDD which tends to be plodding but regular. Again, I think a less purist approach is best - at some points you feel your creative juices flowing and you want to plow through 2-3 pages of code in one go without writing exhaustive unit tests for each little method. Other times if you're being more cautious you want to test the living hell out of every line of code. As a mature developer, you have to feel the right pattern depending on the code you're writing.
  
• Lastly, I think its critical that at a minimum unit tests are written before code is checked into source control. That way if you have continuous integration the unit tests will run on every build. It seems also just a courtesy - you might have insight on why you left out some tests but your team mates might not. So before checking in code make sure there are tests to go with it. I find also that the unit tests are really a good description in code of the requirements so for new code its really helpful to have them as a guide.
  

For code changes in a larger mature application, I like the traditional TDD process of making new tests so that they fail, writing code to fix the tests and then regression. But for coding applications from scratch or starting a new major code project, I find a looser style tends to produce better results.

So with any agile technique, don't get to ideological about the approach - in my experience, TDD is best used as an approach and not a rigid perscription.

Agile Tools for .NET Part 2

This is a follow up to my previous post on Agile Tools for .NET. I got to the office and I noticed a few more tools that I missed in my previous post.

Confluence

Atlassian's professional WIKI solution, it works great with JIRA and provides a really nice WIKI solution. For those who want to travel light with documentation, WIKI's provide a nice fit for technical documentation, build documentation, internal designs, etc. as they easily accomodate rapid changes, are completely web based and allow for automatic version tracking.

Data Dictionary Creator

One of the principles of Agile is make your code self-documenting. What about making your database self-documenting? You can do this in SQL 2005 in T-SQL by adding meta-data and descriptions to each field in your database. The Data Dictionary Creator tool does the same thing through a nice GUI and allows for export and import of the documentation. You can then document your database and store the meta-data in source safe for version control and safe keeping. In addition, you can export the descriptions to an excel file for instant documentation!

Wisdom Screen Hunter

Not really a development tool, but how many times have you been creating documents and wanted a quick screen capture? Forget using the PrntScrn and fiddling in paint - use this tool. Its free and allows you to capture any arbitrary rectangular area as well as any active window. Simply hit F6 and off you go.

Agile Tools for .NET

The following is a list of tools for doing agile based development with Microsoft .NET. There are quite a few tools out there that will accelerate your use of continuous integration, test driven development, source control integration, etc.

I have used all these tools - most I use on a daily basis. They are highly recommend and most are either low cost or free.

JIRA

Best tool for task management. For more details see my review of JIRA from a previous post. It's bugzilla on steroids and is much better and simpler than most windows based tools. It's java based, but there is still no reason not to use it for .NET based development.

NUNIT

NUnit is an implementation of the XUNIT framework for .NET. It allows you to create unit test classes that can be run by either a GUI or console application. NUnit will also integrate into your build process so that on each build, you run through your automated tests to make sure they still pass.

NANT

NANT is a port of the ANT build tool from Java. NANT allows for build scripts based on simple XML files. NANT can do things such as compile code, deploy assemblies, manage directories, start and stop processes, run executables, etc.

NCover

NCover is a code coverage tool for .NET. It will instrument your assemblies when they are built to analyze each line of code in your assembly and whether it was actually executed (exercised). If you combine this with NUnit, you can then tell how much your unit tests actually go through every path of code.

dotTrace or Ants Profiler

Either tool will do, but personally I like Ants Profiler better and you can get it as a bundle with Ants Loader for cheaper. Both tools do the same thing - they profile either memory or performance data for your executing code. You can profile an ASP.NET web site, a Winform application or a Windows Service. What you will get in the case of memory profiling is the ability to snapshot at any point of execution a picture of what is in memory. This can help with objects not getting garbage collected properly or where you're running into memory issues. For performance profiling, the tool will tell you the amount of time each line of code took to execute. This can be useful for performance testing in that you can see the breakdown of time for a method to isolate the particular lines of code that are taking the longest.

Cruise Control .NET

Cruise Control is a port of the java application of the same name. It is a continuous integration server which will fire a build on every check into your source code repository. It can run either a NANT script or an MSBuild script or simply tell Visual Studio 2005 to do a build. It then stores the status of each build in a little intranet site that you can monitor and it can email you notifications as well. This provides an automated way to integrate code and test to make sure it builds and passes all your unit tests.

My Generation

My Generation is a code generator. It ships with a number of templates for generating O/R mapping code for frameworks such as Gentle, NHibernate, DAAB, etc.

TestDriven .NET

This is one of my favourite tools. It allows you to integrate your unit testing right into studio. Want to run one of your NUnit Tests? Just right click on the test code and up comes a Run with option that you can then run with the debugger, NCover, NUnit, etc. The one I use most is run with Debugger as it means you can step through the code hit by each of your unit tests.

ANKH

Ankh is a subversion add-in for Visual Studio 2005. For those who are using subversion as your source code control system (and I would recommend subversion in general over Visual Source Safe) it allows you to update and commit code from the solution explorer window. My only problem with this tool is that its quite slow especially if you have large projects, so you may find it easier to check in with external tools after a while. But for those who are just getting used to subversion, its a great first step.

Getting to the Finish Line in an Agile Project

One of the great potentials of Agile software development is flexibility in changing requirements. We currently use agile methodologies and have scheduled 2 week iterations. We have show and tells with stakeholders every two weeks where we show them where things are at and gather new user stories, functionality changes and other feedback to plan for the next iteration.

One of the challenges we have found with this approach is that the project quickly turns into perpetual development. One of the advantages of a traditional waterfall model is a solid arc in development and a very specific deliverable - you have a good definition of when you're done. In an environment where the requirements evolve, the time lines are fluid and the stakeholders are not that experienced at software projects we are finding that this means that the project never seems to end! Every two week iteration, stakeholders simply pile on new requirements and the launch date drifts away.

In a consulting environment, this isn't such a bad thing if you're on a time and materials engagement - the client simply gets to spend their money as long as they want. But in an internal development team, this poses a challenge because most stakeholders aren't really aware of how much money they are spending and they simply keep monopolizing the team. In the meantime, there are other projects or priorities queueing up in the background.

Based on what we have found works, here are some practical recommendations on how to combat this problem:

• Establish clear business objectives and as many solid functional requirements as possible. There is a happy medium between XP style iterative programming and a rigid waterfall model. Try to put some acceptance criteria that can be measured so that the entire team has a common understanding of the target.
• Remember the law of triple constraints - of cost, resources, and time changing any of these impacts the others. Make sure that stakeholders are aware of the real cost (even if it is an internal cost) through an internal budgeting model. In our organization, we have a shared cost model in IT but we can allocated the shared cost to any department we choose. In cases where we see departments monopolizing our time we can allocate more cost to them. This gives internal stakeholders an incentive to prioritize and conserve resources.
• Make stakeholders compete with each other. This is pure politics but it works. The easiest way to stop a project is to simply tell another stakeholder that they're waiting in the queue behind someone else. There will be a pretty quick meeting between stakeholders where one will start to put a lot of pressure on the other to get their project done quickly or to get out of line altogether.
• Time box and resource constraint the project explicitly. We do this quite a bit with reporting needs which seem to continually evolve ad nauseum. When we start these types of projects, we establish a budget of 2-3 iterations and that's the budget. The requirements can evolve but there continues to be pressure on the stakeholder to get it right the first time or else lose their window.
• Always have a phased approach with Phase 2 and 3 already planned in Phase 1. This gives the stakeholder the ability to still feel they're going to get a change at some point but perhaps can live with launching without it.
• Be vigilant and communicate often about what's currently in and out of scope. I'm not sure whether stakeholders intentionally play dumb or whether people simply don't hear bad news but I've experienced several times the case where three meetings in a row we hear questions about a feature that we've already ruled out of scope and there is genuine surprise from the stakeholder that they're not getting a particular feature even if its been already discussed, documented, etc.
• Have really strong leaders in QA. QA Leads, especially in the latter stages of projects, become crucial at communicating and organizing priorities. A great QA Lead can manage stakeholders on a daily bug review meeting and can handle the constant re-negotiation of last minute feature requests, low priority bug fixes, etc. and keep the process moving towards launch. If the project loses velocity during the stage it opens the door to ever increasing scope change. If the project has high velocity then everyone on the team feels like they better prioritize because time is running out and the project is going to launch without them. This helps crystallize priorities and puts good pressure on stakeholders to commit to the minimum work required to launch.

I've been in projects where we sat at 90% for 5 months because we kept stalling as we closed in on the finish line. In addition, I've been on projects with internal teams where the stakeholders simply got used to having a software team at their disposal week after week and their was never any pressure to launch anything or to stop working.

As a project manager, your goal is to launch - 90% complete is really 0% complete especially if it's still just running in development or still buggy in test.

Implementing an Agile Change Control Process

What is Change Control?

Change Control traditionally is a project management process for managing change. The typical method for managing change in a classic PMP project is a Change Control form that is filled out containing the scope of the change, impact to the project, etc. This is then approved and incorporated into the master project plan, charter, etc.



In addition, Change Control processes are used by operations departments to manage the deployment and promotion of code from one environment to another. The Change Control Form provides the documentation for an operations team to deploy, operate and monitor the change.

Change Control and Agile

This traditional view of Change Control is at odds with Agile methodologies for the following reasons:

1. Change is viewed as a non-typical event, e.g. the project was running smoothly until someone threw a change into the project. Agile projects by definition are based on the assumption that changes are typical events and aren't special.

2. Change requests are simply Waterfall designs in miniature. Requirements for the change must be documented up front and analyzed. Stakeholders must approve of changes before they are made with the assumption that requirements, scope, risk, and design can be formally documented ahead of making a code change.

3. Traditional Change Control processes are document heavy. The process is designed as much to capture and document change as it is to make effective decisions. Change control forms are kept around forever with dates and identifiers so that someone could find out who approved a change. In many cases, these are also printed out, resulting in very large binders of paper with signatures for every change.

4. In an agile project, one of the objectives is to flatten the team and reduce hierarchies so that people can more effectively communicate. However, traditional change control processes require specialized knowledge, special powers of approval and a hierarchical command and control structure. Project Managers are usually the writers of change control forms, but then they must get approval from "Stakeholders" who will bless the change.

Here is a good example of a traditional Change Control process - have a look at Bell South's Change Control Process Guide. At 120 pages, it is a great example of how bloated a large formal change control process can get when Project Managers are working in a bureaucratic environment. Just managing the process documentation alone takes a massive investment - just look at the 20 pages of version numbers of this document and you'll see that someone's full-time job must be simply keeping the guide up to date!

Change Control isn't a bad thing - the original objectives behind formal Change Control are actually in line with the objectives of agile methodologies:

1. Define change and anticipate change early instead of leaving it to the end.
2. Providing a process for incorporating change during development of a project.
3. Communicate scope, risks and impacts of a change.
4. Provide the appropriate review and approval process to authorize a change.

It isn't that change control is bad - it's that the evolution of formal project methodologies has led to very bureaucratic, documentation heavy processes. In addition, keep in mind that most of these processes were invented before the incorporation of digital collaboration tools such as email, Wikis, sharepoint, JIRA, Word, etc. and so the typical change control process has artifacts from a non-digital era.

Implementing an Agile Change Control Process

Change Control can be used effectively in an Agile environment if it is sufficiently light weight, non-bureaucratic and collaborative. Here are my recommendations based on what I have seen work effectively on Agile and non-Agile projects:

1. Use an agile methodology such as XP, Crystal, Scrum, etc. that has change management built into its fundamental process. In Scrum, for example, all work is effectively a change and is logged as a "change request" in a work backlog. In XP the changes are logged as User Stories. Iterations allow for changes to happen on a regular basis - they're not special change events.

2. Use a generic task tracking system such as JIRA to log any change. In JIRA, you don't need a separate "bug tracking" system that is different than your change tracking system. In addition, you can take a change all the way from requirements to deployment without re-entering information. You could do the same thing with low tech tools such as Excel or Word simply by having a work log that is updated regularly.

3. If you have specific legal or regulatory reasons to track changes then try to understand the specific minimum requirements. I've worked in a government environment for example where all changes needed to be documented so that they could be audited. But the level of detail that was incorporated into each Change Control form became ridiculous - an auditor probably doesn't really need to see a change in an individual SQL script, a deployment checklist that includes changing ODBC connection strings, etc.!

4. Eliminate as many approvals as possible. We had a change control form that had 10 signatures on it - I was one of them! I used to get these forms across my desk documenting some change to an application that I had never even seen, had no idea how to measure the risk of the change and yet I was expected to "approve" it. Even worse, the signature had to be on paper so there was a constant stream of people running around with pieces of paper trying to find me to get my signature. Eliminate all this process and identify the minimum number of people who need to be involved in any change.

5. If you're going to have a Change Control Form/Process, keep in as light as possible. Your form should be no more than 2 pages and should be as automated as possible. We currently use JIRA to automate all our tickets. In previous environments, we've used Outlook forms. I've also seen this done use Word with Macros enabled to provide drop downs, automatic fill-in, etc. Keep your process light as well - if you need 2 hours of training to teach someone how to make a code change, then something is wrong!

6. Keep Operations teams involved from day one. One of the primary reasons why change control forms are deployed is because Operations teams get tired of having code dumped on them without documentation or involvement. The typical solution is to put in a gate keeping process to keep the developers from deploying something unsafe to production. This is a legitimate role, but change control processes typically provide artificial security - simply because a developer has filled in the proper form doesn't mean their code works as advertised. Instead, have an Operations Lead that is an integral part of the project from day one who will have a specific understanding of the project and create the right level of documentation, deployment guides, test cases, etc. and be testing these along the way. This is far more effective than a large form being written at the end of the cycle.

7. Keep stakeholders involved throughout the project. The reason why Change Control forms need to be so heavily documented is because the key stakeholders have not been involved all along the way. If stakeholders are seeing working software using Show and Tell techniques, then Change Control simply becomes regular involvement in the project.

8. Use multiple models of communication. Instead of having a single Change Control Form that forces people to write in a specific way, allow for multiple models of communication that could include meetings, power point presentations, white boards, Visio diagrams, etc.

Based on the above principles, here is my current favorite flavour of Change Control:

1. We build our requirements using use case documents that living documents. We currently use Word because its dumb simple and stakeholders can easily open it and change it. We use power point for wire frames and Visio for almost everything else.

2. Revisions of documents get stored in source code control for safe keeping and history. However, we're not dogmatic about it and in my experience the value of these minor revisions is usually not significant - most stakeholders want to see the current version. We keep them in our repository so that everything is stored together and document revisions are tied to code revisions.

3. We have iterations every 2 weeks that include review meetings with the entire team including stakeholders. We do a show and tell of all working code and we use the meeting to prioritize and change. We typically do this by opening up documents on the spot, but sometimes we use pen and paper and transcribe the changes right after the meeting.

4. We use JIRA for day to day tracking of tasks. We treat all changes, bugs, feature requests and tickets the same way. We have a very simple work flow that moves all tasks from open to resolved to closed. Our list of changes can be pulled directly from JIRA as a list of changes. The list of outstanding bugs can be also pulled at any time so we don't need a specific final bug review meeting - we will have been reviewing them on a daily basis.

5. When we're getting close to deployment, we use white board based checklists to capture the minute details (e.g. make sure the ODBC driver is configured properly) that need to be done to move to production. We check these off by simply crossing them off or wiping them from the list. We have 15-20 minute stand-up meetings that includes the Operations Lead so that we're communicating effectively with Operations and they know exactly where things are at as we move into production.

6. Just before doing a deployment to production, I have a meeting with executive stakeholders with a very high level presentation in power point. It presents the current status, any risks that they need to understand, confidence in going, and high level operations plan. Executives typically don't care about whether the SQL is right - they want to know that the team has confidence that it can deploy safely, has followed the appropriate testing processes and that the team is on standby with a contingency plan in case things go wrong.

There are lots of variations to this process and many different tool combinations. You can scale the process up or down depending on the size and formality of the project. In addition, if you're working in a specific regulatory context you may need to be producing more formal documents, approvals, audit trails, etc. My suggestion is always prioritize these below communication and if you need to do this for regulatory purposes then do so but keep the work as minimized as possible. Never substitute change control processes for good communication, team collaboration, stakeholder involvement, etc. In addition, remember that Change Control is a means and not an end - working software is the objective and should be focus for the team.

How to be an Effective "Stake Holder"

Part 1- Introduction

Every software project has "Stakeholders". There are other terms used for this role such as "Internal Customer", "Steering Committee Member", "Department Head", "Boss", "Sponsor", "Champion", etc. Sometimes stakeholders are formally identified (usually by assigning them to a project directly) but there are also lots of informal stakeholders as well.

While the context for these articles is software development projects, in my experience being an effective Stakeholder is consistent across all projects and the same guidelines apply consistently for other projects as well, e.g. marketing projects, facilities projects, etc. As a Director of Information Technology, I play the role of Stakeholder on some projects and been on the delivery side on other projects. I've worked on projects that are internally delivered as well as worked as an external consultant. So I have seen the Stakeholder role from a number of different angles.

This introductory post gives you an introduction to the role of the Stakeholder and what is expected from you if you have been assigned as one. In future posts, I'll provide some specific techniques, guidelines and suggestions for managing and influencing project teams for Stakeholders.

Your Boss Comes to You with a New Project...

Here are some really bad but unfortunately typical ways that Stakeholders get assigned:

• Your boss has been assigned the role and doesn't want to do it. So he or she delegates it to you. You can continue to delegate down if you choose to do so. I've actually had this happen to the point where the original stakeholder was the VP of Marketing and by the time the project had started a junior copy editor showed up saying, "I was told that I'm to represent marketing on this project"!
• You're in a role of authority and someone on the project thinks you should be consulted. For example, you're the Director of Marketing and someone in HR decides that they want to change the web site to have job postings. You get roped in because you've issued the policy that no one changes the web site without your input. You're now a Stakeholder!
• The project is half way through and someone realizes they need your input. This happens to Operations teams a lot - 80% of the way through development, someone on the team realizes that they need 10 more servers! As the person owning this non-budgeted purchase you're now a Stakeholder!
• You're asked to join or are already part of a committee that regularly meets to discuss general issues. For example, many organizations have branding committees, privacy committees, project review committees, etc. Sometimes projects will make the entire committee a Stakeholder. I had this experience once working on a project building a government web site - there was a permanent "web committee" with 7 members on it who had to meet on every issue and come to consensus on every change. The entire committee was a Stakeholder.

If you find yourself getting labelled as a Stakeholder in any of these scenarios, beware! They are high risk and put increased risk on the project.

Recommendations

Here are some recommendations on when to sign-up to be a Stakeholder.

• Be empowered and qualified to be a stakeholder. The primary role of a Stakeholder is to make decisions and provide input. If the project is expecting a VP level Stakeholder and they get the VP's secretary, they're not getting a stakeholder who is qualified or empowered to make decisions.
• Avoid Stakeholder proxies. Stakeholder proxies are people who show up to all the meetings, collect a lot of notes but then cannot make decisions on their own. If the answer to questions asked of you is typically, "Well I need to ask my boss whether we're allowed to do that" then tell your boss that the project really needs the boss to be the Stakeholder, not you.
• Be prepared to commit a minimum of 2-3 hours a week to the project. If you cannot commit to the time, then having you in the loop just creates additional communications overhead for the project team.
• If you are primary stakeholder or if you are providing extensive documentation, business expertise, or testing support then expect this time commitment to be considerably higher. For example, we had a project to deliver a new B2B order entry system. One of our stakeholders was the person who ran the B2B business. During requirements gathering, she was involved in the project about 20-30 hours a week. During development, she was involved about 10 hours a week to clarify requirements, approve changes, etc. During testing, she was involved 40-50 hours a week to help test key functionality, provide clarification on missing requirements, facilitate testing with other people in her department, etc.
• Stakeholders should be established up front and given ample warning of expectations. You should know you're on a project from the beginning even if you won't be needed until further down the line. In addition, you should have a context for the entire project and not just be isolated to your area.
• Beware of people randomly assigning you as a Stakeholder through conversations in the hall, informal emails, etc. It may come back at you if the project turns south.
• The inverse of the above is don't assign yourself as a Stakeholder simply because you're in a position of power. Simply inserting yourself into a project is not an effective way to be a Stakeholder. In addition, don't take credit for projects where you just went to a single meeting - that's not being a Stakeholder!
• If you work in a traditional departmental structure, switching to a project based model will require radical changes in how department heads interact with project teams. Many Directors assume that if a single resource of their department is on any project, they automatically get to have a say in every project decision. Even worse, some Directors simply reserve the right to sit on the sidelines and snipe at the project teams at any time at random. You'll see this in emails coming from Directors that start with, "I noticed this project is trying to use MY resource (equipment, people, brains, etc.) and before I ALLOW YOU to do so, I must REVIEW AND APPROVE this project". Don't assume that just because you're a Director you're automatically a Stakeholder on every project!
• Avoid projects where there are many Stakeholders, especially if there is no Stakeholder hierarchy or specific areas of ownership. Here is a good sign of too many stakeholders - if there are more stakeholders than project team members!
• Try to be on projects where you have a personal interest in their success. This creates investment, accountability and passion. Part of your role as a Stakeholder is to be a champion, so having a personal stake in the success of the project makes a huge difference for you and the project team. If your career is on the line because of a project, do you think you're going to be a great Stakeholder and do what it takes to be responsive? Lesson to executives - make your Directors bonuses tied to project targets, goals and successes and not just the running of the department.

Depending on where you work, you will more or less control over the projects in which you are responsible. However, all this means is that the techniques you use are more or less political There are lots of ways to get on and off a project - some of them explicit and some of them implicit (More on this in future posts). In addition, if you have no choice but to be the Stakeholder for a project then there are things you can do to improve your contribution to the project and ensure a greater chance of success.

Stay tuned for future posts for more recommendations on how to be a better Stakeholder...

Why Agile Software Projects are Good for Customers

We have two software projects that are going on at the Foundation. Project #1 is just finishing. It was not managed by me and it was done using a traditional waterfall death march type methodology. Project #2 is being managed by me and we're using a traditional Agile project management approach.

Comparing the two projects really shows the different in the benefits and obligations of the customer in an agile project:

• Project #1 is 100% over original estimate for no apparent reason. The customer gave up trying to even get a date from because they haven't seen any deliverables, don't even really understand the scope of work and just want it done.
• In Project #1, there was about 1 month just spent trying to define requirements. There were dozens of meetings trying to nail down field mappings with no code being written in the meantime (which starts to explain why the project is late). The right people were never in the room and the developers got different answers from different people.
• In Project #1, despite the investment in time, the output in terms of requirements documentation and certainty of scope was quite poor. Despite the investment, the requirements still changed up until the last week before testing including major business rule corrections.
• In Project #2, we've taken a traditional Agile PM approach. We include the customer in every meeting (usually once a week) and we are using a combination of use cases and wire frames to document requirements. This is the first time in the organization that the customers have seen either format so its a bit of a learning curve in terms of how to read them. But it took about an hour of reading through use cases before the customer fundamentally got the value of such a document.
• In Project #2, I got my lead technical architect to start prototyping on day one. By week 2 of the project, we already have a prototype writing in .NET using SQL Server 2005. The data model is about 50% done. But before even the documentation is done we have a working model.
• In Project #2 for project management we've taken an Iteration Cycle of 2 weeks as the basis for development. We have an architecture/planning/requirements spike for the first 2-3 weeks but have already started coding. Every Iteration has a plan to deliver a list of use case implementations and related wire frames as fully functional code. We have formal show and tell sessions at the end of each cycle.
• In Project #2, we have a proper project plan with real estimation. Estimation is done at the wire frame and use case level and tasks are broken down to about 1-3 days in duration. In my experience anything more granular is a waste of time and anything more becomes unpredictable and subject to error.
• In Project #2, we will be using Test Driven Development and Customer Testing all the way through the project as well as Continuous Integration.

The experience so far from a customer perspective has been really interesting so far:

• Customer participation on Project #2 has been extremely high and involved. The customer role has been really well defined and our stakeholders have been good at providing the right level of input.
• There is a good trust relationship and a sense of team - it isn't a US vs. Them approach. We're not delivering software for a customer - we're all as a team delivering on a project.
• Our ability to shift priorities and change the schedule is highly responsive. For example, our marketing department is working with the team to figure out the content for the application and its taking a lot more time than expected. So we've started on the back end logistics sections and left the front-end until the end in order to give the marketing folks the most leeway. By having check ins every two weeks we can easily adjust.
• In addition, because the application is built in vertical slices instead of horizontal ones, we don't have to have 100% of one tier done before moving to the next one. We can simply move from front end to back and hook together some disconnected pieces as they come available.
• The project is simply more fun and more of a team based approach.

Some recommendations to software project teams in treatment of their customers (whether internal or external) in an agile context:

• Don't patronize your customer because they've never done software projects before. Don't treat them like idiots because they've never read use cases or UML diagrams or database models before. If you take an hour to introduce the concepts and travel light, you'll find that customers "get it" rather quickly and then see the value of these requirements tools.
• Always remember that the customer is the end user and not a stakeholder representing an end user. For example, many Directors of Marketing act as the "customer" in web project teams. They're not. They're representing a customer. Everyone on the team needs to always focus on the end customer and stakeholders always have to be able to filter out their own biases and represent the end user successfully.
• Customers will buy into an agile project management approach if it produces results - period. If you cannot deliver software on time or its poor quality, an Agile approach won't save you and will quickly degenerate. Shorter iteration cycles means there is no where to hide, so talent/experience and ability to deliver are critical early success factors.
• Show and Tells are critical to the Agile Project. We use to have them once a week on Fridays. Make Show and Tells Fun! We use to dim the lights and use the projector and have popcorn and snacks. Even if its some boring application, hype it up and celebrate it.
• Customers should be the leading evangelists as well - some customers have been taught to be the evil task masters and try to crush development teams because they think it gets the team to work harder. I can tell you from the other side of the team that it doesn't work - developers work less, produce poorer quality and flip the bozo bit on you in short order.
• One of they values of an Agile project team is transparency and team work. Transparency is a double edged sword - customers have to be prepared to hear the bad news with the good and to help the team prioritize effectively. Good customers develop a relationship with the team that allows them to be trusted - its critical in order to encourage honest dialogue.

Overall, so far Project #2 is doing much better than Project #1. It has lots of risks and we may not be able to deliver all the functions asked for in the time line. But the requirements are well understood, communication is open and use cases are prioritized. We've already had conversations about what parts of the scope could drop if needed and contingency plans if we go late. Customers are in the loop at all times and a core member of the team.

Agile Just Feels Natural

Have done agile development for most of my career, I find it interesting when I first introduce the concepts to new developers. We're just starting to introduce it to people in my department, and we have got started with reading through the usual standard books by Ambler, Fowler, Beck, et al and I've start acting as the Agile PM on some key software projects.

What I find interesting is when I introduce it to developers, the universal reaction seems to be the following:

1. What a breath of fresh air - this is how we should have doing software all along.
2. I've already been using these techniques but didn't have a methodology to explain it and wasn't rewarded for it. I find that developers who have been raised in the classical SDLC have secretly rebelled against it and agile methodologies simply validate the rebellion.
3. The biggest flaw as most software developers experience with the traditional waterfall model (or worse the traditional death march just get it done by a fixed deadline approach) is that they now have some confidence in their work and ability to deliver against timelines.
4. When I find developers revert back to their old bad habits, they almost always run back into trouble and then slap themselves in the head of going back to hiding their risks, not being iterative and letting projects slip into death marches.

See my next post on why Agile is Great for Customers...

Low Tech Approaches to Requirements Gathering - Part 2

For those who missed Part 1, I wrote on how to use low tech tools such as paper, Post-IT Notes, whiteboards, etc. to gather requirements.

In this post, I'll describe how to better use tools such as Office, Project, Visio, etc. to gather requirements.

Here are my recommendations about approaching tools such as these in an agile context:

1. Use "Just Enough" tools. Don't use Rational Rose for UML diagramming if Visio will do quite nicely.

2. Buy the absolutely best tools for the job. In theory, you can write .NET applications using Notepad and command lines tools and save a few bucks on licenses. But the productivity will be poor and learning curve will be worse. Don't allow your organization to scrimp on tool costs - in my experience if its the right tool for job, paying the $500-1000 for a particular tool is worth the cost.

3. Invest in tools that are standardized. If you're working in Java, use Eclipse. If you're working in .NET, using Visual Studio. If you're an Oracle DBA, get Toad in your office. If you want a source control tool, get VSS, Subversion or CVS and not Seapine. That way when you go hiring for new developers, they've probably used your tools before.

4. Get really good at using the tools. It sounds like a basic suggestion, but I don't know how many project managers I've worked with don't really understand Microsoft Project. If you set up project in particular ways such as exposing the dependency columns, splitting the window to have the resource allocation on the bottom instead on the side, getting rid of the GANTT chart altogether, etc. productivity goes up considerably. Same thing with Word - the first thing I do with any document sent to me is look at how many Styles are being used and typically I find hundreds of them. Whomever wrote the document didn't understand how styles work, why they improve productivity, etc.

5. Buy tools that are easy to learn. Again, another dumb suggestion but I look for tools that a junior developer or programmer can understand. In many cases, this means buying more tools as the simplest ones can be single purpose. I've rarely come across a monster tool that works as a really good Swiss Army Knife for all occasions. The profiler I use though is stupid simple - its from RedGate and you load it up, point it at your project and start profiling in about 30 seconds. Its dumb simple to learn and cheap as well.

6. Have everyone have access to all the tools and have everyone knowledgeable about them. For example, I've worked in organization where only the Senior Project Managers were "allowed" to have Microsoft Project. If you weren't a Senior PM, then you had to either use Excel or find a Senior Project Manager to help you build a project plan. Technical resources should be able to open up a project plan as well - if you want your team to be accountable for deliverables, telling them that they're not important enough to see the plan sends the wrong message!

7. Automate work as much as possible. Use Continuous Integration, Test Driven Development, build scripts, O/R mapping layers, code generators, documentation generators, etc. where appropriate to automate drudgery work. Invest in time sheet systems, collaboration spaces, source code control systems, etc. where appropriate instead of tracking things manually.

8. Remember that every document, artifact and tool costs money to buy and to maintain. Every software license has maintenance costs. Every document takes up space on a file server. Artifacts have to be managed, secured, backed-up, etc. So travel light as product just enough documentation. Use just enough tools to do the job which also usually means use the ones that are the most productive. Remember that emails are documents as well - if you're finding that you're getting hundreds of emails a day, figure out how to get your team to be more concise, more action oriented and spend the majority of their time producing deliverables instead of documents!

9. Get the fastest computers you can afford. Get two monitors minimum for every developer. Get a good video card. If you have to wait 20 minutes to do a build, trying to do agile development is going to be tough. If paying an extra $500-1000 will cut that build time to 5 minutes then I would pay the money as it will result in massive productivity improvements on a daily basis.

Low Tech Approaches to Requirements Gathering: Part 1

One of the tenents of agile development is just enough documentation. I would recommend adding another suggestion: just enough tooling.

The boss comes to you and says, we need a project plan for this large software project. Budget is about $1 million and I need it done in 6 months. Figure out what is required, how many resources you need and what kind of scope can be delivered.

So how many of you have already reached for Microsoft Project?

Forget it for now - come back to it after you've tried some more low tech approaches. Here are some suggestions in order of really low to medium tech approach:

1. Talking: before you start visualizing, diagraming, or white boarding, start talking! Some people talk and diagram at the same time (I do) but I've been to too many meetings where an architect gets up to the front of the room and the team sits there for 30 minutes waiting for the person to finish his fancy whiteboard picture.

2. Piece of paper: I love white boards, but they're hard to carry to a bar, a coffee shop, an airplane, etc. So keep a sketchpad, a napkin, etc. and use it as your white board.



3. White board: the number one tool for an agile PM. If you want to record your white board, take a digital picture of it!

Note: Make your white boards portable. Get the white boards that are on wheels or make them small enough that you can move them from desk to desk. We used to have our release checklist on white boards and it worked great - we could leave it up permanently (without the PLO in a meeting room) and then we would erase things off the board as they got done. This also stops the need for dedicated planning rooms - simply bring the tools with you.



4. Big sheets of paper and Post-IT Notes: Ever done a web site map with a big sheet of paper, some markers and post-it notes? Try it and you'll never go back to using Visio to do site maps again.

Note: If you want to be really advanced, use coloured Post-IT Notes. Get Red, Yellow, Green and you can use them to connote priority, level of difficulty, etc.

5. Paper templates: if you find yourself redrawing, create some templates in word and then print them off. For example, don't draw yourself a calendar each time - simply print off one in outlook and then start writing on it.

All of these techniques mean no use of a computer (except for create a couple templates). You can use these techniques in any environment that has access to an Office Depot.

Low tech approaches to planning, requirements gathering, design, etc. have some specific benefits:

1. No license costs: while this isn't such an issue with Project, if your development tool standard is Rational Rose, Together Soft or some other expensive IDE where every user must fork out thousands to build a UML diagram, a project plan or a document then using lower tech approaches will have dramatic cost savings.

2. Encourage collaboration: if you get people to talk, to collaborate and to physically draw, use construction paper, or write the collaboration happens naturally.

3. Take away bias: If you're a technical architect, you think in objects. If you're a database designer, you think in tables, stored procedures, queries, etc. If you're a project manager, you think in gantt charts, dependencies and resources. So which toolset do you want to work with together? Post-it notes makes everything give up their biases and focus on real requirements.

4. Speed: How fast are you really with Project, Visio, Word, etc. I'm a power-user on all these tools but I still find whiteboards are faster.



5. Freedom to reboot: People get attached to their work. In addition, if you've spent the past 2 hours making sure the lines all connect properly to the boxes in Visio or making sure that you've leveled your resources in Project you really don't want to relive that experience in the same afternoon. But whiteboards are really easy to reboot and people don't get as attached to Post-IT notes.

6. Multiple models: If you use computer based tools, its more difficult to use multiple models. You need Visio for your wireframes, you need Word for some documents, you need Project for your gantt, etc. The amount of tools required to support a multiple-model approach. Alternatively, you get sucked into the idea of using one BIG GRANDIOSE TOOL - enter in the Rational Rose, MDL, Case Tools, Heavy IDEs, etc. that try to be all things to all people and cost a fortune in the process.

One of the challenges with low-tech approaches is how to translate them into digital, cleaned-up format for posterity. Here are some basic suggestions:

1. Don't bother: every document requires upkeep, so don't bother capturing every design decision in a fancy document unless its really necessary. Low-tech documentation is more about the process of thinking through drawing then creating fancy documents. Keep the top 10% of what comes out and dump the rest.

2. Take a picture: take a picture of your white board - you've now got documentation! Put it on your WIKI, store in Word documents with notes, etc. and you're good to go.

3. Data-entry: sometimes you want to capture the notes, diagrams, etc. in a digital file so it can be further manipulated. So at the end of the meeting, take the 20 minutes it requires in most cases to data-enter it using the right tool. Better yet, get your junior or admin assistant to do it for you - why have a $100K technical architect creating circles in Visio when a $40K secretary could do it for you?

Note: this is probably an out-sourcing opportunity as well. Do some white boarding, take a picture and then send it overnight to diagrammed in India. Walk in the next morning with your fancy new diagram complete with colour coding and code stubs created!

Once you've done some design work in a low-tech manner, then use a computer based tool if you need it. See tomorrow for part 2 where I discuss how to use the computer in a low tech way and also how to use the existing tools more efficiently.

Classic Presentation from Uncle Bob Martin

I saw Bob Martin at a Software Development conference a couple years ago and loved his presentation. He's one of the founders of the Agile Manifesto and one of the smartest old-school architects in the business.

He's just posted his classic presentation on The Principles of Agile Design. This is a must see for any Technical Architect.