September 11, 2014

Empiricism and complexity in software development

Definition of complexity

The Oxford dictionary defines complexity as:

“Consisting of many interconnecting parts or elements; intricate”

This certainly describes the software we build today.


The definition of deterministic is given as:

“Algorithm, model, procedure, process, etc., whose resulting behavior is entirely determined by its initial state and inputs, and which is not random or stochastic. Processes or projects having only one outcome are said to be deterministic their outcome is ‘pre-determined.’ A deterministic algorithm, for example, if given the same input information will always produce the same output information.”

While we hope the code we write is deterministic, the process we use to build code can not be. Things that stop the process of writing code being deterministic:

• Business factors will change
• Different developers will bring different expertise and style
• People learn to do things better the second time
• Technology advances make different solutions possible
• Technology advances make old solutions obsolete
• The human factor adds an element of randomness to the process, such as what people are there at any given meeting, what the mood of the teams are, who is ill that day etc.


The systems we build are complex and the process used to build them are non-deterministic.

Historical approach to software development

Historically, a waterfall methodology was used to solve software problems. The waterfall approach assumes that building software is deterministic. As a result, plans overrun, budgets overrun and expectations are sadly not met. Waterfall based software development does not deliver to expectations because of this underlying assumption, and the knock on effect is loss of trust, money and relationships.

This is the single root cause why waterfall delivery does not work.

How best to solve complex non-deterministic problems

Science has been solving complex and non-deterministic problems for centuries with the empirical approach. The empirical process is based up empiricism which is defined as:

“Empiricism is a theory which states that knowledge comes only or primarily from sensory experience.”

Scientific method is defined as:

The scientific method is a body of techniques for investigating phenomena, acquiring new knowledge, or correcting and integrating previous knowledge.

In addition from wikipedia:

“The chief characteristic which distinguishes the scientific method from other methods of acquiring knowledge is that scientists seek to let reality speak for itself.”

If we apply Scientific method to software development we must use empiricism and gain knowledge from primary or sensory experience, we must use the scientific method to let reality speak for itself and correct any previous errors and integrate any new insights into previous knowledge.

Agile practices

Agile practices are based around the assumption that software is complex and non-deterministic. They are based around empiricism and the scientific method.
Scrum is an empirical process tool. Scrum has 3 pillars:

1. Transparency
2. Inspection
3. Adaption

The three pillars give us a cyclical way to move towards our completed software. If we make everything we do transparent, we can inspect it and then we can learn and adapt it to take on the new information. We can apply this to the process itself AND learning about the software product in terms of features.

These cycles of improvement naturally happen with the team to the cadence of the sprint and release cycle.

Kanban has the concept of small incremental changes through feedback loops. This has the same effect as the Scrum cycle of improvement and follows the empiricist approach.

Extreme Programming details small iterations in development cycles and feedback loops to incrementally make improvements.

Lean has the principle of ‘Amplify Learning’ which advocates small iterations of development to provide learning to improve software.


Agile methodologies are based upon the underlying assumption that software is complex and the process to create software is non-deterministic. Agile methodologies use empirical and scientific process to solve these problems.


Simon Powers
Simon Powers is an Agile Coach specialising in large scale transformations and agile adoption. He has a background in very large enterprise architecture which has led on to organisational design and agile process refinement. Simon is the founder of Adventures with Agile.