Agile vs. Traditional software methodologies for successful government IT procurement

Although many people would not agree with me, I personally think that standalone agile methods would not be as successful as the UK Government think they might be, when applied to public sector organisations. In my opinion, in order to successfully tackle the problem and reduce the risk of project failure, a mixture of agile and traditional software methodologies should be put into practice for a number of reasons.

As always happens, large IS projects are exposed to adverse influences, the so called project risks, which may result in delaying the timely completion of the project, increasing the cost or reducing the final quality of the product. Project risks affect all aspects of a software project: the organization, the personnel, the technology etc. and can prevent the intended benefit of the project being realised.

All these different types of risk at which a complex IS project is exposed to, should be taken into account by public sector organisations. Some of these risks can be successfully reduced or even eliminated by using the right software development methodology and it is up to the UK Government to make a decision between agile and traditional methods, or as I suggest, a mixture of both.

Traditional software methodologies

In essence, traditional software development methodologies have a tendency to first plan out a large part of the software process in great detail for a long span of time. That is why these methodologies are plan-driven in which work begins with the elicitation, analysis and documentation of a complete and stable set of requirements, followed by architectural and high level design development and inspection. Furthermore, traditional methodologies are process oriented and based on sequential series of steps. All processes would normally consist of certain tasks that must be performed by different members of the project team (managers, designers, coders, testers) and for each of these tasks there is a well-defined procedure. The process steps are defined in detail early in the project, and project goals remain relatively stable throughout the execution of the project. Moreover, plan-driven approaches view documentation as a key element and prescribe documents that need to be developed in advance of actual coding. These documents tend to be numerous and more formal, comprehensive in nature and describing the system in as much detail as possible. Traditional software development methodologies normally put customer feedback and testing at the last stage of their project lifecycle. In the traditional approach, a project is successful if it is on time, on budget and goes according to the plan. Traditional development tends to be successful in an environment where all properties of the end product are specified in detail early in development and provides a clear model of a fixed end result. It assumes that a project is predictable and therefore it is possible to plan and design it from the beginning to the end. The plan is typically optimized for the end product and making changes at various stages of development can require completed work to be started over. That is why predictive teams have difficulties changing direction and future refactoring would be expensive.

Agile methodologies

On the other hand, one of the key constructs distinguishing agile methods from traditional ones is that they are designed not to predict, but accept and respond to change at any stage of the project. Since it makes it very difficult to provide a set of stable requirements in today’s volatile and constantly changing environment, the agile approach is to break the dependency on requirements stability and come up with a process that takes into account changes. Furthermore, agile methodologists do not try to plan for things that may not even happen, because software projects cannot be accurately predicted far into the future and there are too many variables to take into account. Agile methodologies focus on the talents and skills of individuals and yield processes to specific people and teams which results in much more decentralized approach, spreading the decision making to the developers much more. From this perspective, all process steps are dynamically determined based on changes in the environment and the collective experiences of the developers. That is why agile methodologies rely on tacit knowledge within a team as opposed to documentation. They try to keep documentation to minimal, since excess documentation creates a waste of time in producing and reviewing the documents. Agile documentation may consist of formal but may also include more informal documents such as story cards and white board captures. Moreover, rather than reading or writing numerous documents, agile methods put emphasis on face-to-face communication and deploy daily meetings which contributes to the rapidness of sharing ideas. Regarded as a people-oriented approach, agile involves customer feedback on a regular basis and instead of measuring success by cost and deadline, questions whether or not the customer got the software that he or she wanted.

table

Challenges

However, agile methodologies could face serious challenges when applied to centralised and bureaucratic structures that are typical of public sector organizations, since they are hierarchy-driven, highly-procedural, based on strict rules and operating within a regulated environment. People within these organizations are accustomed to precise patterns of working that require high degree of job specializations while decision-making is traditionally deferred up the management line. Therefore, the bureaucratic organisation’s inherent culture would not match the agile approaches and it would be very difficult to introduce them successfully in a place where traditional organisation structure has been used for years. Overcoming this challenge will require a significant cultural shift and many bureaucratically-minded people will find this uncomfortable for a number of reasons.

First, management in centralized and bureaucratic organizations may be uncomfortable with not having a final commitment date of delivery with a bottom line cost. Agile projects have no fixed price or fixed schedule and projects are open-ended and evolve as requirements change. Therefore it would become harder for a manager to accept this technique as he would rather know the total cost of the project and the overall project schedule beforehand.

Second, within a bureaucratic context, the intrinsic nature of management-driven responsibility inhibits the fast, authoritative decision-making activities that are crucial for an agile process. Since “bureaucracy is the enemy of speed’ as Martin puts it, agile methods may face serious troubles within bureaucratic structures, unsuited to dynamic business processes. Moreover, agile development requires collaboration without centralized control, so that team members could be able to communicate spontaneously with each other and with stakeholders, which would not be possible within the strict bureaucratic environment. What is more, enabling agile behaviour requires a great dose of team freedom in self-organizing efficiently without any supervision, which would not be accepted by the centralized management.

Third, bureaucratic organizations would not tolerate the little emphasis on documentation inherent to all agile methodologies. Regarding the software creation of complex systems, not having comprehensive documents and models to judge the progress of the whole project from, would not be accepted by the senior management. Moreover, most agile techniques do not support traditional walkthroughs and code inspections during the life-cycle, and may be inappropriate for life-critical IS systems.
Finally, agile processes are applicable to small or medium sized teams and may have difficulties handling larger teams. With larger teams that are typical for public sector organisations, the number of communication lines that have to be maintained can reduce the effectiveness of practices such as informal face-to-face communications and review meetings. Likewise, the process of self-organizing a project team, which is usual for an agile method may take too long for large teams.

Conclusion

To summarize, while agile methods are used in applications that can be built quickly and do not require extensive quality assurance, critical, reliable and safe systems are more suited to a traditional methodology. While fixed procedures and static working patterns are not beneficial in a volatile and changing business environment, responding to change can be resolved using an agile method, because practices defined in agile methods allow for better handling the changes, such as constant feedback from the customer. On the other hand, traditional software development methodologies scale better to large projects, but do not provide tangible results until the end of the project lifecycle. Previous experience has shown that more traditional structured methods are no longer that effective for the increasingly volatile and dynamic nature of current business environment. Consequently, more agile methods have evolved to provide more flexible approach. However, today’s bureaucratic organization’s inherent culture may not match the development technique adopted by agile methodologies. For all these reasons, should the UK government decide to use either standalone agile or traditional methods, there has to be a trade-off.
As seen above, both agile and traditional methodologies have their strengths and weaknesses. People usually follow either one of these methodologies, but agile and traditional methods can be combined when needed. If the UK Government manages to grasp the advantages of both techniques and apply them to ICT procurement, the results should be satisfying. Talking about a situation in which the combination of agile and traditional methods would make sense is if a large Government project can be divided into a number of relatively independent smaller subprojects with each part being implemented by an agile team. However, there has to be a higher-level project management to coordinate the teams, which can be managed with a more “traditional” approach. Moreover, formal techniques may be used in an agile way to handle critical pieces of software to increase quality and confidence. Furthermore, agile methods can be applied to rapidly changing lower-critical modules, while planned approach can be used for relatively stable, high-critical modules. Therefore, if the flexibility of agile methods is combined with the disciplined processes of traditional methods to break the project down into manageable chunks, the Government may succeed in blending the advantages of these two techniques. It is evident that a climate of trust, co-operation, collaborative and flexible working practices coupled with authoritative fast decision-making and clearly defined roles are necessary if this practice is to succeed.

To conclude, successful ICT procurement and delivery needs a mixture between agile and traditional methods to promote a controlled, structured but yet flexible development method, which will be suited to the uncertainty, and the continually changing business requirements. Nowadays, successful organisations are ambidextrous, so the UK Government needs to combine the advantages of both agile and traditional software development methodologies to cope with the problem and reduce the risk of project failure.