The following is an excerpt from The Complete Software Developer’s Career Guide by John Sonmez. To get the entire book delivered to your inbox, go here.
Are you ready to put on your boxing gloves and enter the ring?
Are you ready to be confused?
Are you ready to endlessly debate semantics? To hire expensive consultants to tell you what you are doing wrong and coach your team to higher levels by getting everyone “certified?”
Well, welcome to the world of software development methodologies.
We don’t have cake, but you are welcome to have a donut at our daily stand-up meetings.
Perhaps nothing causes more arguments in the software development community than the selection of software development methodologies and their intended implementation.
Software development methodologies define the processes we use to build software.
Some methodologies are fairly lightweight and don’t tell you much besides a set of principles to stand by.
Other methodologies—like Extreme Programming—are extremely prescriptive and tell you exactly how you should build your software and run your entire team.
In this post, we are going to start from the past and look at the old model of development: the waterfall process, which is actually still in use in many organizations today.
Then, we’ll jump right into the biggest fad in software development today: the quasi-defined, everyone-is-doing-it-but-no-one-is-doing-it-right methodology known as Agile development.
Finally, I’ll introduce you to the basics of what I consider the 3 main implementations of Agile development which are being used today.
I have to warn you, I’m not going to attempt to cover every existing software development methodology that has ever existed.
Instead, I’m going to use my author’s privilege again and tell you what I think you need to know.
When I was first learning about software development, the traditional waterfall process was a given.
That’s just how software was built.
We didn’t call it waterfall.
We didn’t make fun of it.
We just accepted it as the way to develop software and we did it—well, as best as we could.
(Note: this doesn’t mean there weren’t other software development methodologies out there. There were plenty of them; it’s just that they weren’t widely known or used, and many of them were only more formal ways of doing the waterfall methodology.)
Waterfall development is much like it sounds. It’s building software one step at a time, with each step falling down into the next lower step until everything is downstream.
Built into the waterfall development methodology is the Software Development Life Cycle (SDLC).
This same SDLC shows up in just about every methodology.
In waterfall development, the SDLC is sequential.
In fact, you could say that waterfall development is just following the SDLC step-by-step, nothing more.
Yeah, you know me.
I’m down with SDLC.
What is it?
It’s a sequence of developing software that goes from requirements analysis, to software design, to implementation, testing, deployment, and then finally maintenance.
You progress through each phase and you only move forward, never back.
There is some overlap between phases.
Let’s briefly talk about each phase so that you can be down with SDLC.
In this phase, you gather all the requirements for the software.
What should it do? What features should it have?
What should it look like?
How should it behave?
You might gather this information by talking to customers or stakeholders, or by just making it up yourself, but you need to know what you are going to build before you build it. (Although as you enter the world of software development, if you haven’t already, you’ll quickly discover this is not the case in reality.)
Now that you know the requirements of the software you are trying to build, it’s time to figure out how you are going to build it.
At this phase, you take those requirements and turn them into architectural design for the system, low-level algorithms, and UML diagrams (if you like), and generally decide how the system is going to be built and work together.
The level of detail for software design is up for debate, but some level of design is always necessary.
In traditional waterfall approaches, you typically have what is known as big upfront design, which means that most of the details are planned out at a very low level during this phase.
If you are going to always go forward and never backward and you have a fixed schedule, designing everything up front seems to make sense, but in practical reality, requirements change, and there are so many unforeseen circumstances that this rarely works out.
Ok, now it’s actually time to code.
This phase of software development is supposed to be all about writing that code.
Here you turn the design from the last phase into actual working code.
I really don’t think much more needs to be said about that here, so… moving on…
Ah, so you have a beautiful piece of code that is elegantly designed, perfectly implemented and shines like a thousand suns.
Then, some greasy-haired tester comes along and ruins it all by showing you how you deviated from the requirements and that none of your code actually works.
This is the testing phase.
The testers have been creating their test plans, writing their test cases, and preparing for this moment from the second they first saw you laughing gleefully in your cubicle as you enjoyed yourself, writing your code—no one should be that happy.
Now the testers run their tests, find the bugs, and after some debate and much shenanigans, you fix as many of them as possible until everyone agrees it’s time to move on.
Now it’s time to see if this baby actually works.
If you have multiple components that were developed separately, you strap them together and you could call that phase “integration,” but either way you’ve got to deploy that code. Put it out there in the wild.
This might mean deploying the code to a server and then nervously flipping the switch, saying, “We’re live.”
This might mean making the gold standard CD that you’ll ship out to all your customers.
Today, it might mean uploading your app to the app store.
Either way, you are now putting the software—and all your hopes and dreams—into the customer’s hands.
Good luck.
I hope you really did fix those bugs…
Oh, you thought you were done when you put that software out into production and shipped it to your customers?
Moved on to the next big thing already?
Not so fast. Most projects end up in the maintenance phase longer than any other phase.
Once the software is out there in the wild, you still have to support it.
Now you’ll be fixing bugs that customers find, adding new features, and keeping everything running smoothly.
Later on, an IT support specialist will be responsible for the upkeep of this system – troubleshooting problems, ensuring compatibility with upgrades, and providing internal technical support.
This phase lasts as long as the software is still being used and supported.
And that’s all there really is to the SDLC: get the requirements, design the software, build it, test it, deploy it and maintain it until the company goes out of business or some young whippersnapper gets the idea of beginning all over again and rewriting it.
Agile really is the big game changer when it comes to software development.
Before Agile came along, most development projects were using some kind of waterfall development process, whether they acknowledged it or not.
What I mean by this is that they were sequentially building their software, going from one phase of the SDLC to another.
While some development projects were using some iterative approaches before the Agile movement took effect, and were breaking the SDLC into smaller cycles, that idea wasn’t formalized until Agile.
So what exactly is Agile?
Well, even years after its introduction, we still don’t really know.
Agile is a bit amorphous. You have to understand the history to understand why.
It all started at The Lodge at Snowbird Ski Resort in Utah.
Basically, a bunch of authors from different development methodologies met together with industry leaders to try and find some common ground for how software development should be done in the industry.
The original group had 17 members, who discussed some of the problems affecting software development at the time, and together they formed what is known as the Agile Manifesto, which you can read below:
We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on
the right, we value the items on the left more.
It is based on 12 principles, defined below:
Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.
Welcome changing requirements, even late in
development. Agile processes harness change for
the customer’s competitive advantage.
Deliver working software frequently, from a
couple of weeks to a couple of months, with a
preference to the shorter timescale.
Business people and developers must work
together daily throughout the project.
Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.
The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation.
Working software is the primary measure of progress.
Agile processes promote sustainable development.
The sponsors, developers, and users should be able
to maintain a constant pace indefinitely.
Continuous attention to technical excellence
and good design enhances agility.
Simplicity—the art of maximizing the amount
of work not done—is essential.
The best architectures, requirements, and designs
emerge from self-organizing teams.
At regular intervals, the team reflects on how
to become more effective, then tunes and adjusts
its behavior accordingly.
I think the principles better indicate what Agile is really about, rather than the manifesto itself.
So as you can see, Agile isn’t really a methodology itself, but rather it defines at a very high level how software development should be done, and is a sort of parent to other methodologies that are considered Agile in nature.
Agile ignited the idea that software should be developed and delivered incrementally and embraced the idea that requirements can and should change during the development of the software.
It also redefined some of the relationships between various members of software development teams by valuing face-to-face communication and self-organizing teams over heavy documentation and strict protocols.
Like I said, some development organizations were already doing some of these things and following methodologies which embraced one or more aspects of the Agile principles, but for the most part, the world at large was still mostly developing software with the traditional waterfall approach.
Before we dive into a few of the current popular Agile methodologies, let’s take a brief moment to talk about why the waterfall approach sounds good on paper, but doesn’t seem to work very well in reality.
One of the biggest issues with waterfall development is that requirements change—or rather, they are not known until later in the project.
If you try and take a step-by-step approach to developing software and attempt to get all the requirements upfront, it’s not going to bode well for you if suddenly those requirements change or new ones are discovered.
If you’ve already fully designed the architecture of the system and you have to change something, but you are in the process of implementing it, you either have to scrap a bunch of what you already did and go backwards, or you have to put your foot down and say no.
So, either the project slips or you build the wrong thing and piss off the customer.
As a software developer, it would be nice if nothing changed.
It would be nice if we could go and gather all the requirements, designs, or solutions and then implement them.
But life doesn’t work out that way.
Really, Agile is about acknowledging that fact, accepting it, and building software development processes that work with that constraint, not against it.
That’s Agile.
Now let’s talk about some of the popular Agile development methodologies out there.
I’m not going to cover all of them—and to be honest with you, most teams that claim to be following one of these methodologies aren’t really following them.
They are following them in name, but not really in practice.
Most teams do what I would classify as some Agile-like development methodology.
So where does Scrum come in?
Scrum itself was created by Ken Schwaber and Jeff Sutherland somewhat simultaneously in the early 1990s.
In 1995 they wrote a joint paper which defined the Scrum methodology, merging their two approaches.
Scrum is a formalized and prescriptive methodology that defines specific roles on a software development team, the workflow for developing the software, and what specific meeting should take place in each iteration of development, also known as a sprint.
There are three main roles in Scrum.
First, the product owner acts as the voice of the customer and ultimately determines what the priority of the work should be, as well as communicating with the rest of the business, other stakeholders and customers.
The development team not only writes the code but also performs the analysis, design, testing, and all the other tasks associated with delivering the software.
And finally, the Scrum master acts like a coach for the team by helping remove any impediments that would slow down the team, communicating with the product owner, and facilitating the Scrum process.
The basic idea behind Scrum is that software development is broken up into smaller iterations called sprints, which have a set amount of work that is locked down and done in that timeframe. The results are then incrementally delivered to the customer at the end of each sprint.
All of the features that need to be developed for the software are put together into what is known as a product backlog. (These are basically like the requirements of the system.)
The product backlog is prioritized, and during every sprint, a sprint backlog is created where a certain set of product backlog items are pulled to be worked on during that sprint—which usually lasts one or two weeks.
At the beginning of each sprint, a planning meeting is held where backlogs are pulled into the current sprint, and the current team estimates the effort level of completing the backlogs.
Technically, the team is supposed to commit to getting all of those backlogs done during the sprint, but I’ve found that rarely happens in practice. (Commitment is difficult.)
Every day there is a quick stand-up meeting called a Scrum where everyone gives a really quick report.
The idea of the Scrum meeting is to keep everyone informed and remove any impediments that might be slowing down progress.
A Scrum meeting happens at the same time in the same location each day, and each team member answers three questions:
I’ve found it quite useful to require personal commitments for question 2.
So, I would change it to “what will I commit to doing today that will help the team reach the sprint goal?”
This subtle change makes quite a bit of a difference—in my experience.
During the sprint, the team works together to complete all the backlog items in that sprint, and a burndown chart is usually used to track the progress and velocity of the team in completing the backlog items.
The burndown chart tracks the remaining hours, story points, difficulty points, or whatever characterization method is being used to track the work left in the sprint.
When the sprint is over, a review is conducted where the functionality completed during the sprint is demonstrated to the stakeholders.
Finally, a retrospective meeting is held where the team reflects on the past sprint and develops ideas for improvement in the next sprint.
When Scrum is executed correctly, I find it to be an extremely effective and valuable methodology for developing software.
Unfortunately, I’ve found that in reality it is often not followed very closely, and many allowances are made to compensate for failure or to try and game the system.
I’ve written up an extensive article on what I feel are the failings of Scrum in general, so I won’t go into details here, but I think it’s worth talking about something I’ve already mentioned, which I think is the biggest reason Scrum teams fail to have the success they should.
Commitment.
Having been a Scrum master for several teams, where I coached many organizations on implementing Scrum, I have found that the biggest thing which tends to kill an otherwise successful Scrum implementation is the lack of commitment—both by the team and at a personal level.
It is really easy to pull backlog items into a sprint and agree to get them done if all goes well and according to plan.
It’s a much more difficult thing to actually commit to getting them done.
Without the concept of commitment, the standards of accountability decline and the sprint loses meaning because the work pulled into the sprint is only a pipedream and is not reliable.
It is very much like creating a to-do list of things you are going to do in a day, trying your best to get them done every day and most days failing to complete the list.
The list itself starts to become meaningless over time and you start to wonder why you have it at all.
It’s much more effective to make commitments that you fulfill 99% of the time, because then you can trust yourself and you, in turn, can be trusted.
I could talk quite a bit more on this subject, but I think you get the point.
While Scrum is a fairly formalized and prescriptive methodology—at least in workflow and organization—Kanban is not.
Kanban is similar, but it’s a much more loosely defined methodology, based more on principles than on instruction.
Kanban has its origins in the Toyota Production System and lean manufacturing.
Originally Kanban was created as a way to limit the work in production for manufacturing, which results in increased efficiency and less inventory.
When applied to software development, Kanban is primarily focused on what is known as a Kanban board.
A Kanban board is just a simple board with several columns in it, which represent the various stages of work as it flows through the development process.
The idea here is to make all the work being done on a project visible and to limit the total amount of work being done at one time (known as WIP, or work in progress) in order to identify and decrease bottlenecks.
Like Scrum, Kanban is based on the idea of self-organizing teams which are multi-disciplinary.
Kanban is easily applied to existing systems and processes and acts as a way to formalize and visualize the flow of work through those systems via the publicly visible Kanban board.
Kanban is very much focused on the idea of continuous improvement via feedback loops.
There is no defined way that software development teams use Kanban, so the process is going to vary from team to team.
Typically, you might expect that the team would have some kind of a backlog or list of work that needs to be done, and that work would be prioritized.
Then, someone on the team would pick up new work to be done and add it to the Kanban board.
The work would move across the board as it progresses from stage to stage.
Perhaps the work starts in analysis and design and then progresses to development and then moves to testing and finally deployment, but there may be various steps in-between or other ways or organizing the work.
I’ve always been a big fan of Kanban.
In fact, I use a variation of Kanban myself for most of the work I do.
But I’ve always felt like there should be a bit more structure around the process.
Awhile back, I wrote up my own formalized version of Kanban which I called Kanbanand.
The goal was to create a little more structure and prescription in both the workflow and development processes.
The last Agile methodology we are going to discuss in this post is one of my favorites because it is so prescriptive and it promotes an extremely high-level of professionalism and rigor around software development.
Extreme Programming, or XP, was first created by Kent Beck around 1996; he released his first book detailing the process in 1999, Extreme Programming Explained.
XP took many of the best practices at the time, like unit testing, test-driven development, object-oriented programming, and a high-quality customer focus, and elevated them to a level that some people would call extreme—hence the name.
Due to its extreme and rigorous nature, it never caught on all that much, although some teams still practice it today. (Scrum is by far more common—or at least variations of Scrum that I like to call Scrumbut.)
|Hey John| You just said Scrumbut, what is that? It sounds interesting; is that some kind of a delicious, flakey pastry?
Ah, Scrumbut.
Yes that.
It’s what I call it when a team says they are doing Scrum, followed by a long pause and then they say “but” and give a long list of exceptions to the Scrum process that they aren’t doing.
Then they make all manners of excuses as to why they are doing things their way instead of just following the damn process like it was written!
As a consultant or a software developer trying to coach teams on Scrum or follow the Scrum process, it’s as frustrating as hell and makes you want to pull your hair out.
The biggest problem I have with Scrumbut though, is that it is a way to get around the pain in the Scrum process–the very pain that makes it effective.
It’s sort of like if you were trying to cook something on a stove. You turned on the stove, it was hot and you burned your hand because you weren’t using hot pads to grab the handle of the pan. So you said, “you know what, I like everything about cooking with the stove except for the heat. The heat is just no good. Let’s cook on the stove, but let’s not use any heat.”
If instead you said, “hmm… the handle of that pan was hot, maybe we should put on some gloves or use a hot pad next time,” you’d make much more progress.
But, alas, most software development teams trying to implement Scrum just turn off the gas and wonder why their food isn’t cooking.
That’s Scrumbut.
I’m not bitter about it at all.
XP, like other Agile methodologies, embraces change and utilizes short development cycles or iterations to allow for change and the evolution of software over time.
The development process for an XP project is centered around an extremely focused set of disciplines.
XP practitioners will plan out what they are going to work on—much like Scrum—and then assign various points to the work to be done.
When the work is actually done, it begins with testing.
Acceptance tests define the done criteria which the work item must pass in order to be considered complete.
Before any actual real code is written, unit tests are created which define exactly what the code should do given various circumstances, subsequently driving the actual development of the code.
XP relies heavily on the idea of pair programming where two developers sit together and jointly work on all code that is being created.
The goal is always to design and implement features as simple as possible, with the current needs in mind, not the needs of the future.
The idea is that the code can evolve to handle more complex situations as they arise, rather than trying to optimize prematurely or provide extra flexibility, which often comes at the cost of added complexity.
The concept of collecting code ownership and coding standards is very important to the practice of XP.
XP is even so prescriptive as to require that no developers participate on overtime on the project.
As you can imagine, XP draws a large amount of criticism, and it is extremely difficult to follow without having a team in which all members are dedicated to the principles and practices.
To an outsider, XP can seem very much like a programming cult.
But I have to say I personally like XP and find it extremely effective when implemented correctly.
I’ve just always had an extremely difficult time convincing managers and development teams to fully adopt the process.
Let’s be honest here: most software development teams you work on will either be professing to follow some methodology and not really following it or will pretend to not be following any formal methodology at all.
I used to get pretty upset by this.
I used to get up on my soapbox and preach the values of following Scrum to the letter, or the benefits of actually implementing XP rather than just pretending to.
Or I would ask, “What methodology are you using to develop your software?” and when I get a an answer of “We don’t have one,” I’d lose my shit.
What I’ve come to realize is that a specific methodology is not as important as having some kind of repeatable and measurable process that can evolve and adapt.
It’s great if your team is doing Scrum and improving upon how they do Scrum.
But it’s also great if your team is taking some stuff from Scrum and taking some stuff from Kanban and XP and doing their own process that works—so long as there is some kind of process.
That is the key. A repeatable and defined process.
So, as you are learning about methodologies, I’d urge you to consider a few final points.
First, a repeatable process is more important than a specific methodology.
And second, just because a methodology isn’t listed here doesn’t mean it’s not a valid methodology for software teams to use.
Rather than trying to cover every software development methodology in existence—and there are many—I’ve tried to give you a basic understanding of the methodologies that I feel most software development processes today either utilize or heavily borrow from.
John Sonmez is the author of the perennial top-selling Soft Skills: The Software Developer’s Life Manual and the founder of Simple Programmer. In The Complete Software Developer’s Career Guide, John shares the principles and knowledge that took him from teenage hacker to highly paid senior development and consulting positions—and by age 33, early retirement and a second career as an entrepreneur.
Today he runs the hugely popular Simple Programmer blog and YouTube channel, where he helps millions of developers every year to master the career and life skills that made all the difference in his success.
Ever wonder how some companies make product updates feel like the highlight of your day? …
Picture this: You’re in the middle of a hectic workday, balancing strategic decisions with daily…
Ever wish customer feedback came with subtitles? With the right feedback analytics tools, you can…
Survey design is the backbone of effective data collection, enabling businesses, product managers and researchers…
Wondering how to master Jira’s vast capabilities for strategic project/product success? Epics are the key…
In this article, we walk you through the ultimate in-app feedback how to strategy, including…
