On efficient software development

03.03.2025 Permalink

In my previous post I mentioned the subject of efficiency as one of the five dimensions one could use to find spots where a software development project could improve, but I did not elaborate on that.

This article provides an overview of what I consider as important areas where software project efficiency is based on. For me efficient work includes and enables

People and the process

Let's start with the involved humans and the overall process framework. You need to have skilled, motivated and empowered people on your project. If you combine these with clearly stated project goals and a process that enables learning (like every iterative-incremental process framework does), you might just let the team run unattended. That's easy advice and the process part is really not hard to fulfill, but the people part is. In most cases you'll have a mix of experienced professionals and others that are still learning.

So, just in case you're not so sure if the team that you were able to put together choses efficient ways of getting the work done, here are some more specific ideas.

Quality, Simplicity, Complexity

The quality of your programming language, tools, frameworks and libraries is important. Avoid things that are buggy or not fit for the purpose. If you hear people complain about the need for workarounds, reasons to double check, low performance or high effort to adapt to recent changes, help the team to get rid of the cause and replace it with something better.

There is a strong positive correlation between quality and simplicity. Obviously it is easier to create a mature library or tool for one well-defined problem than for a mix of problems. If the scope of a solution is wide, web frameworks being a typical example, the necessary maturity will only come with production level use of the product (hopefully by others) and time passed.

There are, however, intrinsically complex problems that the team might face. I would count system distribution, a transactional database, product variants management or a text processing editor as examples where dragons are lurking. Some of those problems are avoidable without compromising the project objectives, others are not. The first choice should be avoidance, the second is the selection of an existing mature solution. In rare cases the team needs to come up with its very own design and implementation to solve the problem. Such cases should be treated as sources of risk with respect to budget, schedule and quality and need high attention.

Another potential source of complexity is source code and project organization: a high number of dependencies slows the team down, this applies to code as well as to an organization. Whenever there is a justifiable chance to get rid of a dependency the team should decide to do so. The idea of a high degree of decoupling is key here.

Speaking of complexity reduction by decoupling source code, I consider a micro services architecture as a potential false friend. It is true that each service implementation on its own is simple in the sense above. But the team is stepping into the realm of distributed systems, pulling in a number of problems like loosing transactional behaviour, overall availabilty, performance issues caused by extensive remote communication, maintaining runtime environment configuration consistency and so on. A distributed system can be a reasonable choice to fulfill certain non-functional requirements. But code decoupling alone does not justify such a choice. There are means like static code analysis or reviews that help avoid the proverbial "big ball of mud".

Automation

To get a version of a system up and running from implementation to deployment to testing and documentation there is a huge potential for automation of software development tasks.

Model-based software development

In case your programming language requires much ceremony to express certain aspects of your implementation (e.g. Java), a diagram- or text-based abstraction of these aspects in combination with a code generator might be justified. This type of automation requires special skills, has quite some impact on the tool chain and will only amortize in a medium to large system. I'd rather prefer a programming language where internal DSLs are a natural and emergent phenomenon or not needed at all (e.g. Clojure), but often the programming language is not a choice but a given.

Continuous integration / continuous deployment

In contrast to model-driven software development there are areas of automation that promise to quickly pay off even for small projects and systems. These are automated library dependency resolution, scripted software builds, continuous integration and deployment. Today, these practices are even summarized under their own brand name: DevOps. I hope their application is nowadays as self-evident as the use of a version control system. I'd say: a must have.

Test automation

While the definition of test cases is a demanding intellectual human task the execution of a set of these tests is a typical task for a machine. However, test code is code, which needs to be maintained just as every other piece of code. In other words: there is a cost attached to having automated tests. Therefore, careful analysis is necessary to decide where and how to apply automated testing. I consider the following criteria for making decisions when looking at a specific part of the system:

In general, the higher the testing level is on the testing pyramid the more expensive is the creation and maintenance of an automated test suite, unit tests usually being the cheapest.

In addition keep in mind:

Here are some examples regarding automated testing of typical parts of a system together with a brief assessment:

Documentation

A fourth area of automation can be found where documentation needs to be provided:

Infrastructure automation

Another area: A team creating a distributed system will have to deploy the subsystems to distinct nodes, which have to know each other in order to cooperate. For integration or testing purposes there is usually more than the production system, and the task of consistently configuring all subsystems on all these different runtime environments is often tedious and error prone. So this makes another area for automation of the infrastructure setup using formal descriptions of necessary containers, software versions and so forth.

And last but not least, monitoring automation should be applied to collect and evaluate data and events from production systems in order to learn about performcance, chances for optimization and all kinds of other numbers.

Risk-based application of software engineering methods

If you take a look at the 15+ knowledge areas of software engineering methods in SWEBOK you might feel a bit intimidated.

As a software professional I'd like to be acquainted with the methods considered best practice, but does this mean that a team should implement all of these on its project?

To find a way to an answer, imagine an efficient project organization for a very special case: Creating a software tool for yourself. You are the sponsor, sole source of requirements, programmer, tester and user. Even in such a case there are certain practices you might want to keep, like using a tool for version control or automated build and some unit testing. But you wouldn't consider a stakeholder analysis, or explicit requirements or test management, let alone a formal configuration management audit.

In this special case there is no need for many of such tasks because there is no risk involved if you skip the formal execution of these tasks. To the contrary: you would waste your precious time for no other reason than maybe a good conscience.

Therefore my advice is to regard process frameworks and well-known methods as store shelf with offers. The team should take a look at each item and ask itself: "What happens if we go without?" It should pick only those ideas that will pay off in terms of risk reduction, or add very little overhead.

There are, however, some disciplines where the likelihood of damage is high and its detection usually happens too late to keep the project out of serious trouble. These ideas should only be ignored if the team has good reasons to be sure it will not be bitten by possible consequences.

Among them is usability engineering, i.e. learning about the context of users, getting a deep understanding of how they need to do their work and testing the software accordingly by letting real users work with it. I've seen enough projects where the team and client representatives had only a deceptive idea of how the product would be used, leading to poor productivity in the field. Building the wrong tool for users is most efficient only if your goal is to throw time and money out of the window.

Another area where I would usually advise against its omission is the elicitation of architectural drivers like non-functional requirements, system context, constraints and so forth. Wrong technical decisions could be the cause for a major re-write of parts of the product in the future, once the mismatches become obvious.

And a third area whose omission might lead to an unpleasant surprise late in the game is overall software testing with realistic data (in terms of quantity and characteristics/variants). Because the devil is in the details, numerous edge cases that a system must handle before it can be used in production might only become visible when real-life data is processed.

In addition a team should treat defects rigorously because it is well-known that system parts that show unwanted behaviour in some respect are potentially also infested with problems in other respects. Defects should therefore be treated with a zero-tolerance mindset. They have to be tracked using bug tickets and need to be eliminated as soon as possible.

Whenever a team is about to skip any of the above it should have good reasons to do so.

In closing

To sum it all up: to work efficiently requires skilled people that quickly learn what is necessary to meet the projects objectives. They pick the right tools, abandon the bells and whistles and strive for a high degree of automation where it pays off. Not surprisingly, this observation applies to software development as well as to any other creative or artisan work.