Why and How Software Projects Spiral Out Of Control.

Each year, billions of dollars are wasted on failed projects.

To put numbers on this:

  1. Up to 50% of projects fail.
  2. These failure costs the U.S. economy around $50-150 billion per year. Globally, this number is significantly higher.
  3. According to McKinsey, around 17% of projects fail so badly they threaten the continued existence of the organization itself!

However, the key question is:


Why is this acceptable?

It's interesting to look at industries where failure is not so commonplace, such as the airline industry. If your plane had a 50% failure rate, you most likely would decide that flying isn't for you.

So why do airlines get things right almost all of the time, while software seems to go wrong almost half of the time? In 2017, the airline industry had over 4 billion passengers and suffered zero casualties, and generally reports fatal incident rates of 0.06 per million flights, which works out at one death for every 16 million flights that take place.

One of the key differences between the two industries is in the approach by senior management and leadership. Airplanes are incredibly complex machines, and there is nobody alive that will pretend to understand how every single system and subsystem (and all their redundancies) works on.

In addition, any change or addition to a plane design is thoroughly tested and reviewed, and every failure is investigated to ensure that the root causes are fixed permanently.

What Often Happens.

Often a project starts quite well, with the team making good progress on the initial feature set, but week by week the project becomes more complex, the backlog grows faster than any features can be completed, and the scope expands until it makes a mockery of your initial estimates.

It can feel that you've walked into a trap and that the project is trying to eat you alive.

So why does this often happen? Why are even experienced executives, project managers, and developers often surprised by how quickly things can go wrong?

The core issue, we believe, is that many stakeholders believe that a product of project can be simply defined by a set of high-level features written down, and that every so often you can add a feature or remove one, and that you can understand the complexity of the project at a quick glance.

This couldn't be further from the truth.

Any non-trivial software project will have a depth and complexity that will be far greater than the ability of any single individual to understand it fully.

Let me repeat that more simply:

Nobody understands your project or product fully.

While this may seem overly pessimistic, I will shortly give you an example of how a seemingly small feature can have far more depth that what is initially apparent, but the key idea to keep in mind is that there is no such thing as simplicity. There is only the perception of simplicity.

"Add a Client" Feature.

At Mäd, we're quite the project management nerds, and so we spun up another company called Bloo, that dedicated to building a project management application that we can use for ourselves, and also sell for monthly fee to anyone who also wants to use it.

At one point, we already had a web and mobile application that had several features such as:

  • Projects - A wrapper to keep different projects separate from each other.
  • Discussions - The ability to post up and reply to discussions in a forum-like manner.
  • Updates - Giving a daily status update so everyone knows what's going on.
  • People - Adding and removing team members from projects, and seeing who's where.

Obviously, as a consultancy, we have clients. We needed a feature to be able to add clients to projects. The initial brief:

The ability to type an email address into a text field and invite a client to join one project.

This seems rather straight-forward, and it might just include:

  • A field to type the email.
  • An email sent to the client.
  • A link for them to sign up to our company account.

While this may appear to be trivial, we then have several more considerations:

  • New User Type - Client account and team member accounts will have certain key differences, as we won't give as much access to our systems to a client as we would to a team member. This then means that we are dealing with a new user type with different permissions.
  • Tagging clients - We have a large and growing team and also our clients also add new team members. So we need a way to "tag' client profiles to ensure that our team knows that this particular person in the project belongs to the client side.
  • Duplication Issues - What if this particular client has already been added to another project in Mäd, and our team member is not aware of this? This then implies that we need to search our existing user database in our company in real time as our team members type the client email to verify if that client has already been added to another project and doesn't require a new sign up link. In addition, we then have to think about how we handle this particular situation? Do we auto-invite them to the new project, do they need to accept, how do we notify them? We chose to auto-invite them but notify them via email. And now that we're talking about notifications...
  • Notification and Activity - In our project management system, we also have an activity feed, and then this had to be notified to inform everyone that a new person from the client side had been added to the project.
  • Clients adding new team members - Of course, the feature that we built to add clients also has to be used by the clients themselves, so they can invite their own additional team members, which means this had to be modified further as we do not want clients to be able to add people to our company as Mäd team members or administrators!

In fact, there were actually several potential flows just for this feature of typing an email into a field and sending a welcome email.

Let's  consider when John (in Mäd Company) wants to add Erika to Project A.

  1. Erika is not in the system at all. She will receive the standard invite email with a sign up link to join the Mäd company account.
  2. Erika is not in the Mäd company account, but is already using Bloo in her company. The same flow applies, but because Erika is already in Bloo, we require a different email template as the wording "Sign Up" doesn't make sense to her, as she has already signed up to use the software.
  3. Erika is in the Mäd company account as a client, but not in that particular project. Erika's name will be available on the dropdown list as John types her email out as she is part of our client database already.
  4. Erika is actually already in the project. We'll need to give an error to John, and inform him that Erika is already in the project.
  5. Erika has already received an invitation, but has not yet accepted yet. This even has subflows, as our invitation links expire after a week, so if the link hasn't expired, we will send the same link but extend it for a week. If the link has expired, then we will generate a new link and email that again.

In addition, there are also a few nice-to-have features that we could add:

  • Can we filter just the client in the people list of a project, so our team members can see them without seeing their own team members?
  • If a client is already using Bloo and wants to invite their own team members from their company, should we also enable real-time search on their team member database in their company so they can just pick their team from the autosuggest menu?
  • Also, instead of just adding one client or team member at a time, are we able to add multiple team members, each which may have a different status on the above five options, and still handle all the invites seamlessly for the end user?

We still haven't included details for this feature such as ensuring positive and negative tests, various types of validations, and so on, but we can see that this is already far deeper and more in-depth than the initial feature:

The ability to type an email address into a text field and invite a client to join one project.

And the key point is that you cannot really know how long a feature will take to build until you start getting really into the details, and this is where most software projects go wrong.

They are started with sometimes nothing more than a list of bullet points by management, and then timelines and budgets need to somehow be created from these bullet points, and these timelines and budgets are nothing but )(hopefully) well-educated guesses.

A Better Way.

The problem is not that software is complex, many other things are also very complex (i.e. launching rockets into space!) but it is more about the way that we decide to uncover the complexity, and accepting that fact that we must, at some point, uncover this complexity.

Often, what happens is that we give some high-level objectives to the developers, and then the build something, we test it, and realize that there are lots of holes in the software, and we ask them to build a second version, and eventually we get the application we want:

Not the best way.

However, this also gives us several crappy versions of software in the interim period.

A better approach is to map out what you want at a high-level, and then dig deeper to get the second level of features, and then dig deeper to get the third level of features, and then we can get a pretty good ida of what we need to do:

A much better way of doing things.

To be able to achieve this, we actually need to do two key things:

  1. Have a good way to list out the features and sub-features so that they can be easily represented.
  2. A way to make an analysis the outcome of point #1 and then cut and add to it effectively.

And this really doesn't have to be done in a high tech way. A meeting in a room with a whiteboard and several discussions will most likely do the trick, followed by a write up of all the points into a shared document where it can be further refined.

Here are some of the in-progress work photos of the previously discussed "invite-a-client" feature.

The key things to remember:

  • Prioritize speed - there is no need for high-quality mockups or designs as this point, they just waste time and make everyone feel good for no reasons.
  • Focus - As you being to draw, you might find other problems worth reviewing or trying to solve, and you may see that you are making things too complex for the end user. Where possible, try to always sacrifice internal simplicity for external (i.e. end user) simplicity. The ideal scenario is where you can have both.

Question to ask.

As you go through the above process, there is quite a large number of questions that need to be asked (nobody said that this was going to be simple!) for each part of the flow. If these questions uncover or reveal new steps or requirements, then you'll need to add them to your flows.

Depending on your business, you may find that there are additional business-specific questions that you need to ask each time as well.

You can find a list of questions to ask here in our Standard Operating Procedure.

Your Choice

While the above may seem a lot of work, we have to remember that software will always have complexity that you will not be able to see by just glancing at a feature list.

So, to build a product you will have to tackle this complexity, but it is about choosing how to do it, and, more importantly, when.

You've got two choices:

  1. You can build a flowchart or rough mockups as explained above and question your assumptions and see what comes out, and then keep iterating until everything is clear. Or,
  2. You can start to get a team to build your product, and then uncover these issues with each versions, and it will take you many months to learn the same things you could have learned in a week with a whiteboard and a marker.
Emanuele Faja

Emanuele Faja

CEO at Mäd. Emanuele brings on board a passion for minimalist Italian design and a 21st-century management approach. He consistently helps clients create and maintain market leader positions.

Read More
Why and How Software Projects Spiral Out Of Control.
Share this