Complexity II: What Is Complexity?

In part one we introduced the idea of complexity in software product design and covered some examples. In this part, we’ll talk about what complexity actually is, how to define it, and - most importantly - how to identify it in your work.

Defining Complexity

We’re not talking about algorithmic complexity here, nor managing complexity in software organisations or architectures. We’re talking about the kind of complexity that ruins products, burns engineering time, and makes users feel stupid.

Complexity increases the cognitive load required on a user in order to successfully operate your software. You can break this down into many dimensions:

Number of choices a user has to make
Frequency of those choices
Number of variables they must hold in their head at once
How many things each choice can affect
Where those effects occur (locally vs globally, immediately vs hidden)
Amount of prior knowledge needed to make the “right” choice
Ambiguity - how confident a user can be that they’re doing the right thing
Reversibility - whether a user can undo a thing if they mess up

Each one of these probably constitutes its own chapter in a textbook, or perhaps its own textbook. Complexity is a complex space. 😉

The kind of complexity we’re discussing is not “how complicated the code is.” It’s how much the user must know, notice, remember, choose, or fear they might break.

If you are after a more technical definition, then my best attempt is that the complexity of your software product is proportional to the size and cyclomatic complexity of the state-space representation of the user-facing¹ parts of your software.

Fancy enough for you? Ok, let’s move on.

Flexibility Generates Complexity

Let’s take a simple user goal: add two numbers together.

One way you could do this is with a spreadsheet. However, with a spreadsheet, the user can put numbers wherever they want. They can reference other sheets. They can SUM, COUNT, XOR, OFFSET, VLOOKUP, and even accidentally open a portal to #ERROR! if they nest parentheses wrong.

It’s powerful. It’s flexible. But it’s complex.

Now, let’s picture a simple calculator - one you might remember from school. You can punch in numbers, you can add things and get a result, but there’s still plenty of other things we can do, and perhaps by accident if we are not accustomed to such a tool. It’s not as flexible as the spreadsheet, so it’s simpler; but we’re not at the ground level yet.

Now let’s picture a dedicated sum application with a nice little GUI. Two boxes. One button. You type numbers, you click, it adds. It can only do one thing. It’s rigid. It’s predictable. It’s simple.

So, what makes the spreadsheet complex? Not the math. Not the UI. It’s the flexibility.

A spreadsheet can model almost anything, which means the user is responsible for knowing which thing they’re currently modeling. They must know:

where the numbers live
whether a formula reaches into other sheets
whether a change will break some other dependent calculation
how the architecture of the sheet hangs together

The calculator is simpler, requiring fewer of these things - the mental model required to operate it is smaller. But our tidy little sum app requires none of this. There is nothing to remember, nothing to accidentally break, no invisible consequences.

This is the key insight:

Flexibility generates complexity. Constraint generates simplicity.

In the ideal, we, as software engineers, are often taking a complex, flexible, general purpose system like a database and then constraining it, as a sculptor cuts away at a marble slab, until all that’s left facing the user is the sum app; precisely what the user needs - and nothing more.

Why Engineers Rebuild Their Tools

When engineers model a problem space, we often model it in full fidelity. We see all the edge cases, all the weird states, all the interactions. We see the truth of how the system works.

And then - out of the purest and noblest intentions - we give all of that truth to the user. We don’t want to hide anything. We don’t want to trick them. We want to give them the “real” model of the problem. So we build products that mirror the complexity of the underlying domain; we give them a spreadsheet when all they needed was sum.

We think we’re being transparent. But really, we’re giving our cognitive load to the user - and typically they are not software experts, like we.

Remember, our job is to manage complexity on behalf of our user - that means taking responsibility for some of it. If the user can see all of the complexity, it’s likely we’re not managing things well. When we go to a restaurant, we don’t eat in the kitchen. When we buy a car, we don’t tour the factory.

To return to the sculpture analogy, if we simply move our “marble slab” into some DBMS, rather than sculpting it - and then we build an API over it that more-or-less replicates the DBMS’s API, then we build a web application over this API that more-or-less replicates a database client (one menu item per table with basic list and CRUD capabilities), then we aren’t managing complexity at all.

If we do this, we’re actually just adding incidental complexity on our side, since we have a bunch of new stuff to maintain, but without delivering any real value to our end user by managing their complexity. It’s lose-lose. It should have stayed in a spreadsheet.

This is how software ends up as expensive and limited database clients. This is how project management software turns into a taxonomy of fifteen ticket types and four overlapping priority systems. This is how bad software is made.

If, at this point, you’re thinking “but hang on - doesn’t our API and web application constrain the DBMS? MySQL is far more flexible and powerful than the CRUD interface you describe”. You’re right, I confess, constraint alone does not guarantee you are managing complexity effectively. It’s also about translating our model into the user’s mental model.

Bridging The Mental Gap

Software engineers and users stand on either side of a river. On one bank, the engineer, with a precise understanding of how a problem is solved. On the other, the user, with an often imprecise understanding of their goal. Complexity, then, is the river between.

We manage complexity by building a bridge; by defining the problem we’re going to solve. This is the design activity that decides how far across the river we’re going to build, and therefore how much complexity we’ll take on our side, and how much we’ll leave for our users.

So we have three concepts:

The solution. This is where engineering lives, it’s the how. One side of River Complexity.
The problem. This is where design lives, it’s the what. The bridge over the river.
The goal. This is where the user lives, it’s the why. The other side of the river.

If you only build on your side of the river, that means users have to wade across the river of complexity to engage with your software. Your job is to build a bridge from your mental model to your users’ so they can cross the river safe, comfortable and dry.

Good design is that bridge:

Databases do this: they hide bytes and present tables.
Browsers do this: they hide TCP and present documents.
Game engines do this: they hide quaternions and present gizmos.

The art of design is crafting a problem that software can solve that enables the user in their goal. It does this by:

deciding which truths to reveal and which to hide (constraint); and
which truths to translate into something the user already understands (translating²).

Signs Of Poor Complexity Management

Here are some obvious signs to keep an eye out for - this is not an exhaustive list, and it doesn’t automatically indicate poor complexity management:

your UI is not so different from a database client (read more)
your UI models exactly the shape of the underlying problem domain (read more)
you are trying to give users the same power you have (read more)
you are building generic solutions in order to “future proof” everything
you can’t clearly explain how your work affects a user
you are constantly “refactoring” or dealing with “tech debt” unaligned with business goals

Here are just a handful of key 🚩s to keep an eye out for. If you or your team are doing one or more of these, there’s a good chance you are injecting incidental complexity instead of contributing genuine value by managing your users’ complexity.

Look Through Your User’s Eyes

A spreadsheet is limitless, and therefore can be overwhelming. Our sum app is constrained, and therefore more approachable. Most products should land somewhere between the two - but much closer to the sum app than most engineers think.

Recall engineers live on the other side of the river of complexity from their users. Now imagine civil engineers building a bridge that only goes halfway across the water. It’s a bit of a silly image, don’t you think? But, typically, many software engineers are at most throwing out a bit of old rope while users struggle against the current.

If you can, take time to interact with your real users and practice this skill - engineers who can think like their users stand head and shoulders above their peers.

Good software product design happens when you see the world as your user sees it, not as you do³.

The total user experience is far more than just the user interface. ↩
The same principle applies when communicating with non-technical folks - the art of cross specialty communication is managing the complexity of the discussion by speaking or writing in the stakeholder’s world-model, rather than yours. ↩
If you are thinking “but I see the problem as it actually is” then you have work to do. You don’t have the model of the world. You have a model of the world. Your strength as an engineer (and as a person) will grow the moment you can hold multiple models at once. A more accurate world-model includes the world-models of other people. ↩