The waterfall methodology is always bad
This shorter blog post may be a little harsh, but it is absolutely all right sometimes to shake up our beliefs or the status quo.
Occasionally, I read on my Twitter that waterfall technology has some advantages over other software development methodologies, and it is absolutely suitable for some kinds of projects. And many articles claim the same.
The waterfall is very deep-rooted to the idea that we can have a fixed size and fix-scoped project. I am referring the situation where the client tells you their requirements, and you are able to give them the price and term of delivery. It is also from time to time believed that it is even possible to get a better result using this methodology than in a more agile-oriented approach.
I spent a lot of time on projects where my team and I were using the waterfall methodology. And it was always the pain. On our first project, we were really focused on the preparation and analytics phase. We even had separate roles such as business analytics, designers (drawing UML and class diagrams), and finally, developers and testers. And it was a mess with many change requests after clients checked the first version of the software. In each project which followed we took a path that was more and more informal and less waterfall-ish, but it was always the same old story.
Today it is not clear to me where the waterfall might be suitable. Even with the incredibly detailed and up-front requirements, including also a design or wireframe model too, the changes are inevitable. Simply , this is the nature of software. Furthermore, as I see it, it is due to our language. We polished our perfect plan that describes our new future project, and everyone in the room seems happy. What can go wrong?
The dangerous culprit there is the lack of awareness that our language is a rather vague tool to express our intention. All of us have unique viewpoints and experience, and everyone actually understands words and phrases in a slightly different manner. The project on paper looks good because everyone sees it in their own way.
Business and the waterfall — the best friends
As far as I am concerned, many years ago people in software development were convinced that business and waterfall were closely bound together. I assume it was most probably because the industry developing software was a young industry, so we borrowed the project-building approaches to build a project from non-tech companies or projects. Fixed price and fixed scope was really a thing in those days. I am absolutely sure that those days the rate of unsuccessful software projects was really high. In order to deal with that, some smart guys wrote the following agile manifesto.
Agile manifesto
Agile manifesto was a great and admirable attempt to shake our industry, our beliefs, and business customs. It was a huge step in the right direction, and I am grateful that it happened. Since a lot has been already written about the Agile manifesto, i am going to refrain from any comments. You can make yourself familiar with its basic principles by checking this link.
Nowadays, agile movement is thriving, which is why I have supposed the waterfall belongs to the past. Yet, the myth of its usefulness persists even up to now. Why is it so important to stop saying that the waterfall might be suitable for some projects?
Software nature
I am firmly convinced we need to accept the concealed nature of software development.
Software development is an inherently “agile”.
That, I guess, is the main root of the problem of the waterfall. Consequently, we need to start viewing our work as agile too. For a little more sophisticated application than an utterly simple static web site, you are obliged to tackle the complexity related to an unexplored area and problems.
Let us imagine that you have received the perfect requirements from your client or business analysis. And surprisingly, even with these requests, you sometimes reach a dead end.
It seems to me that the “fixed price and fixed scope” approach is far more successful in real engineering. For example, if your job is to build a house, you are in a much more stable world than in software development. Maybe I might be wrong, but the immaturity of our industry could be at fault.
Looking at our industry, you keep coming across an immense number of new frameworks, new devices, new screens, new business models, changing apis, etc. IT is an extremely dynamic.
Moreover, even if we were able to freeze our dynamic world somehow, there is still another real threat to our estimations. Up to the present day, there is an increasingly growing number of regulations and legal obligations imposed on our industry. And these changes cannot be taken into account while making estimations. Inevitably, what you need is a complex software that would be capable of running for years on end without any maintenance.
To give you an example, the “line of business” software (tailored applications like ERP, CRM, etc.) is often just a description of processes in the real world. And anytime a process is changed, the software has to be modified as well. And the sooner, the better.
The planning fallacy
In addition, the waterfall is particularly dangerous due to our mental bias, “the planning fallacy.” the term was the first coined by Daniel Kahneman and Amos Tversky in 1979. We, people society, are terrible at planning a complex thing, no matter whether you are making an estimation for nuclear plants or a new website for a small bakery on the corner. Just study the nuclear plant’s estimated costs vs. real costs and you will see a nice illustration of this bias.
In spite of knowing the planning fallacy for many years, we are still kind of pretending that our next estimation of a large project will definitely be better :)
Bearing this in mind, may we not have fixed prices?
Yes, you can. Provided you prefer fixed prices, buy a prefabricated solution. Nowadays, software as a service (SAAS) handles many simple or more complex problems. Whoever is satisfied with a simple static page without any marketing or support had better look for ready-to-use products. Not a customized solution.
Would you like a CRM without any consultation? Without the necessity to have all processes within your entire company reviewed?. You might choose from the wide range of various CRM SAAS products. The software world is pacing to commoditize whatever can be commoditized.
Our role as developers requires extension into consultancy. We need to expand beyond our traditional role of mere developers into consultancy. What your client needs today is a partner not just a provider of software. How does this relate to the waterfall? Unknown and hidden issues can not be estimated ahead. And both our clients and we need to accept that.
Is the waterfall always bad?
I regret to say yes. The waterfall methodology is not better than any other more agile approach. And our current unfortunate state of affairs calls for an extra commitment and dedication to communicate these ideas to our clients as well as to other developers and project managers. I am absolutely convinced that our effort will bear fruit.
