On efficient software development

03.03.2025 Permalink

In my previous post I mentioned the subject of efficiency as one of the five dimensions one could use to find spots where a software development project could improve, but I did not elaborate on that.

This article provides an overview of what I consider as important areas where software project efficiency is based on. For me efficient work includes and enables

reduced waste of time,
avoidance of rework, and
less stress for humans.

People and the process

Let's start with the involved humans and the overall process framework. You need to have skilled, motivated and empowered people on your project. If you combine these with clearly stated project goals and a process that enables learning (like every iterative-incremental process framework does), you might just let the team run unattended. That's easy advice and the process part is really not hard to fulfill, but the people part is. In most cases you'll have a mix of experienced professionals and others that are still learning.

So, just in case you're not so sure if the team that you were able to put together choses efficient ways of getting the work done, here are some more specific ideas.

Quality, Simplicity, Complexity

The quality of your programming language, tools, frameworks and libraries is important. Avoid things that are buggy or not fit for the purpose. If you hear people complain about the need for workarounds, reasons to double check, low performance or high effort to adapt to recent changes, help the team to get rid of the cause and replace it with something better.

There is a strong positive correlation between quality and simplicity. Obviously it is easier to create a mature library or tool for one well-defined problem than for a mix of problems. If the scope of a solution is wide, web frameworks being a typical example, the necessary maturity will only come with production level use of the product (hopefully by others) and time passed.

There are, however, intrinsically complex problems that the team might face. I would count system distribution, a transactional database, product variants management or a text processing editor as examples where dragons are lurking. Some of those problems are avoidable without compromising the project objectives, others are not. The first choice should be avoidance, the second is the selection of an existing mature solution. In rare cases the team needs to come up with its very own design and implementation to solve the problem. Such cases should be treated as sources of risk with respect to budget, schedule and quality and need high attention.

Another potential source of complexity is source code and project organization: a high number of dependencies slows the team down, this applies to code as well as to an organization. Whenever there is a justifiable chance to get rid of a dependency the team should decide to do so. The idea of a high degree of decoupling is key here.

Speaking of complexity reduction by decoupling source code, I consider a micro services architecture as a potential false friend. It is true that each service implementation on its own is simple in the sense above. But the team is stepping into the realm of distributed systems, pulling in a number of problems like loosing transactional behaviour, overall availabilty, performance issues caused by extensive remote communication, maintaining runtime environment configuration consistency and so on. A distributed system can be a reasonable choice to fulfill certain non-functional requirements. But code decoupling alone does not justify such a choice. There are means like static code analysis or reviews that help avoid the proverbial "big ball of mud".

Automation

To get a version of a system up and running from implementation to deployment to testing and documentation there is a huge potential for automation of software development tasks.

Model-based software development

In case your programming language requires much ceremony to express certain aspects of your implementation (e.g. Java), a diagram- or text-based abstraction of these aspects in combination with a code generator might be justified. This type of automation requires special skills, has quite some impact on the tool chain and will only amortize in a medium to large system. I'd rather prefer a programming language where internal DSLs are a natural and emergent phenomenon or not needed at all (e.g. Clojure), but often the programming language is not a choice but a given.

Continuous integration / continuous deployment

In contrast to model-driven software development there are areas of automation that promise to quickly pay off even for small projects and systems. These are automated library dependency resolution, scripted software builds, continuous integration and deployment. Today, these practices are even summarized under their own brand name: DevOps. I hope their application is nowadays as self-evident as the use of a version control system. I'd say: a must have.

Test automation

While the definition of test cases is a demanding intellectual human task the execution of a set of these tests is a typical task for a machine. However, test code is code, which needs to be maintained just as every other piece of code. In other words: there is a cost attached to having automated tests. Therefore, careful analysis is necessary to decide where and how to apply automated testing. I consider the following criteria for making decisions when looking at a specific part of the system:

Algorithmic or logic complexity
Stability of the requirements that guided the implementation
Risk of breaking existing functionality
Effort involved in setting up the tests
Cost or damage in case of bugs or unwanted behaviour
Effect on ease of refactoring or even rigidity regarding change

In general, the higher the testing level is on the testing pyramid the more expensive is the creation and maintenance of an automated test suite, unit tests usually being the cheapest.

In addition keep in mind:

Trying to make pieces of code available for unit tests fosters decoupling, which is a good thing.
Having some automated tests that don't just target single functions or classes in isolation, but include the integration of some important parts of the software brings a lot of confidence when refactoring is due. In other words: the absence of such tests could be a constant obstacle to fixing broken windows.

Here are some examples regarding automated testing of typical parts of a system together with a brief assessment:

Non-trivial calculations or logic are almost always worth covering with unit tests, even if requirements could change.
Testing database access that is implemented in a CRUD (create, read, update, delete) fashion is in many cases not justified.
But complex database queries, subject to change in the future, could be rewarding test objects, even if there is some effort involved to prepare the database data.
A user interface (UI) that is used by thousands of people in production, but is not subject to change anymore, might go without automated test coverage.
A user interface in the early stages might not be stable enough before usability tests have been conducted. The existence of automated UI tests could then create barriers regarding the improvement of the UI, which is eventually detrimental for usability.
But once existing UI functionality is confirmed and used in production by a significant number of users AND the product is still being extended, automated UI testing becomes a must have.
A public REST API of a system that an unknown number of clients depend upon might be a very worthwhile test object.

Documentation

A fourth area of automation can be found where documentation needs to be provided:

A very common practice is the generation of API documentation from comments written in source code.
For HTTP based REST APIs the use of OpenAPI merges documentation with a tool for experimental exploration of a machine-machine interface by humans, an optimal combination to make the API learnable.
A database schema is a useful starting point and important ground for understanding how a system works. When the DDL carries comments they can be used to auto-generate a kind of poor mans data dictionary. In addition the references between tables could be used to auto-generate an overview diagram.

Infrastructure automation

Another area: A team creating a distributed system will have to deploy the subsystems to distinct nodes, which have to know each other in order to cooperate. For integration or testing purposes there is usually more than the production system, and the task of consistently configuring all subsystems on all these different runtime environments is often tedious and error prone. So this makes another area for automation of the infrastructure setup using formal descriptions of necessary containers, software versions and so forth.

And last but not least, monitoring automation should be applied to collect and evaluate data and events from production systems in order to learn about performcance, chances for optimization and all kinds of other numbers.

Risk-based application of software engineering methods

If you take a look at the 15+ knowledge areas of software engineering methods in SWEBOK you might feel a bit intimidated.

As a software professional I'd like to be acquainted with the methods considered best practice, but does this mean that a team should implement all of these on its project?

To find a way to an answer, imagine an efficient project organization for a very special case: Creating a software tool for yourself. You are the sponsor, sole source of requirements, programmer, tester and user. Even in such a case there are certain practices you might want to keep, like using a tool for version control or automated build and some unit testing. But you wouldn't consider a stakeholder analysis, or explicit requirements or test management, let alone a formal configuration management audit.

In this special case there is no need for many of such tasks because there is no risk involved if you skip the formal execution of these tasks. To the contrary: you would waste your precious time for no other reason than maybe a good conscience.

Therefore my advice is to regard process frameworks and well-known methods as store shelf with offers. The team should take a look at each item and ask itself: "What happens if we go without?" It should pick only those ideas that will pay off in terms of risk reduction, or add very little overhead.

There are, however, some disciplines where the likelihood of damage is high and its detection usually happens too late to keep the project out of serious trouble. These ideas should only be ignored if the team has good reasons to be sure it will not be bitten by possible consequences.

Among them is usability engineering, i.e. learning about the context of users, getting a deep understanding of how they need to do their work and testing the software accordingly by letting real users work with it. I've seen enough projects where the team and client representatives had only a deceptive idea of how the product would be used, leading to poor productivity in the field. Building the wrong tool for users is most efficient only if your goal is to throw time and money out of the window.

Another area where I would usually advise against its omission is the elicitation of architectural drivers like non-functional requirements, system context, constraints and so forth. Wrong technical decisions could be the cause for a major re-write of parts of the product in the future, once the mismatches become obvious.

And a third area whose omission might lead to an unpleasant surprise late in the game is overall software testing with realistic data (in terms of quantity and characteristics/variants). Because the devil is in the details, numerous edge cases that a system must handle before it can be used in production might only become visible when real-life data is processed.

In addition a team should treat defects rigorously because it is well-known that system parts that show unwanted behaviour in some respect are potentially also infested with problems in other respects. Defects should therefore be treated with a zero-tolerance mindset. They have to be tracked using bug tickets and need to be eliminated as soon as possible.

Whenever a team is about to skip any of the above it should have good reasons to do so.

In closing

To sum it all up: to work efficiently requires skilled people that quickly learn what is necessary to meet the projects objectives. They pick the right tools, abandon the bells and whistles and strive for a high degree of automation where it pays off. Not surprisingly, this observation applies to software development as well as to any other creative or artisan work.

Five dimensions of software project health

08.02.2025 Permalink

A software project is usually a complex endeavor spanning several months or years, requiring expensive expenditure of human labour. Many aspects are unique: the customer, the product, the composition of specialists, the challenges, chances and risks and so forth.

So it's no wonder that each project follows its own path, makes its own rules, distributes responsibilities in different ways. Many projects may officially adhere to a documented process framework like Scrum or whatever a company has decided to use. But at the operational level, they have to work in their own style.

So how can one judge if a project team is on its way to success or headed into trouble, when adherence to a process framwork (or lack thereof) is no useful indicator?

In order to examine for when a project team needs to review the way it works, I found much benefit in the following five dimensions:

Product Quality

If the product released by the team is in poor shape then something needs to change. Poor quality can manifest itself in frequent occurrence of bugs, change requests that point to fundamental traits, users complaining that their work is hampered, low performance on reasonably sized inputs or high effort to implement seemingly simple changes. The cause of poor quality may be time pressure, a low level of expertise or morale, a lack of understanding about user needs or simply skipping standard quality assurance steps like reviews or tests.

Stakeholder Satisfaction

The outcome of a software project usually affects different groups of people. For example there is the paying sponsor, subject matter experts, one or more user groups, possibly a work council, IT operations and last but not least the development team. Each group sees the project from its own angle, each individual has its own perception. The likelihood that something is actually broken is high when one or more stakeholder representatives express their concern.

Transparency

The management overseeing the project will have a hard time trusting the project team when its work is kind of opaque. Good project management will bring light to actual progress, budget consumption, spotted risks and problems, next milestones, task assignments and so forth. If the management feels a lack of transparency then the project team should change its habits.

Predictability

The more uncertainty there is the harder predictions are, especially those concerning the future. Uncertainty in the very early days of a project is usually high, therefore one important first task is some exploration to reduce choices and risks. However, at some point in time the project team will have to provide estimates about effort and attainable milestones. Going forward, if the team repeatedly misses milestones that were set based on its own estimates something needs to change. The project may have dependencies which it can't control, ever changing requirements, or the methods for the collection of numbers about the past and creating estimates for the future might need improvement. In any case the project management must adapt its modus operandi to regain their stakeholders trust.

Efficiency

Finally, even if all people affected by the product feel satisfied, the project team is able to live up to its promises, the management feels like it has enough insight and control and the product meets its requirements, there's a chance that time and budget are wasted for no good reason. Consequently, questioning habits, tools and technologies should be part of a well-formed closer look at how the product is developed from requirements to release.

In conclusion

A regular retrospective, as it is recommended in Scrum projects, is a good occasion to think about how the team is doing in light of the five dimensions explained above.

If it ain't broken then don't fix it. If the team is successful according to the five dimensions, then congratulate them and let them do their work, because it simply doesn't matter if the way they work complies to a given process framework.

Emacs Org Mode and Getting Things Done (GTD)

30.09.2023 Permalink

A few hundred years ago only a small minority of human beings were able to read and write, enabling them to exchange and preserve information. However, to live a more or less happy life most people didn't need this skill.

Today the picture is completely different. Finding, consuming, processing and creating information is front and center for many jobs. In addition, many professions require highly specialized knowledge and experience. To be productive as an organization effective communication among specialized workers is important. Those who are nowadays unable to fluently read or write are effectively excluded from the information age and won't be able to work in these jobs.

To sum it up: literacy - a human cultural achievement which was once dispensable for the majority of humans - has become a vital basic skill.

Why personal task and knowledge management?

It is an interesting question if there are practices today that you could consider as dispensable and are therefore not widespread, but which might be vital requirements in the future, similar to what literacy has become today.

Effective personal management of a high number of tasks and amounts of information might be one of those skills.

As an illustration: Creating software is certainly one of these jobs that are typical for the information age. Everyday many of us are faced with a constant influx of information and requests to do something, and we, too, often need the help of co-workers or information and decisions by clients. Yet, since our brain cannot handle two or more things simultaneously, creating a new solution (or a part thereof) requires times of undivided focus, where we need to stop any distraction from the outside and from ourselves, in order to truly understand the problem and formulate a possible solution.

Therefore the active management of my own time, tasks and information seems to me as if its mastery is already today a big advantage to be collaborative and productive without putting your health or the quality of your work at risk. Those who face a high load of tasks and/or information influx and try to do without any kind of disciplined approach seem to be at a disadvantage.

From paper-based notebooks to Emacs and GTD

I started my personal task management journey while working on a long-running software project around the 2000s by using a single A4-sized notebook. I had to learn a considerable amount of project specific information and needed to remember all of my own tasks, as well as commitments by others. So I tried to write down as much as possible during and after meetings and looked into my notes several times a day to make sure that I won't miss anything. This habit was far from being perfect but it brought me through the time without going nuts. Since then a paper-based notebook remained a necessity for organizing my work. For recurring items I tried calendar events with mixed success. However, I did not use any kind of task organization for my private life.

Fast-forward to today a lot has changed. I started using Emacs for programming in Clojure in 2012. With Emacs comes Org Mode but I used it merely for occasional note taking, ignoring its vast capabilities. And only one or two years ago, as I felt that I needed to improve my personal habits a bit, I came across "Getting Things Done" (GTD), an approach published by David Allen in 2001. Although there are nice online guides available which present the essential ideas, the book itself is worth reading and contains quite some insights from Allens work as a consultant in these matters. I decided to give the method a try and adopted it for organizing my entire professional and my private life.

Org Mode and Getting Things Done: a mapping

To implement GTD with Emacs and Org you need a conceptual mapping of Allens ideas and suggestions to terms and features that Org Mode offers.

Here is what I came up with:

A GTD item, usually a task, is an Org headline level 2 or 3 starting with a TODO keyword.
GTDs lists for "next action", "waiting for" and "someday/maybe" can be mapped to a keyword like NEXT, WAITING and MAYBE.
The Org Mode category nicely matches a GTD project or area like for example a customer, a specific hobby, domestic work, gardening, bicycle maintenance etc. By default an Org file establishes a category, so a reasonable approach would be to have one distinct Org file for each project or area. Then Org agenda is the key to bring everything together in one view with a timeline.
The numerous lists GTD encourages you to maintain can be realized by using Org Mode tags in conjunction with org-sparse-tree.
In addition, Org Mode tags with a leading @ character can symbolize a GTD context, like a place or a person.
org-capture is a literal solution for putting new stuff into an in-tray file, stored there for a subsequent organizing step. I maintain an inbox--todos.org file for this purpose.
Org headlines can carry a deadline or scheduled date. Together with the Org agenda view you have a very effective tickler file.
For regularly recurring tasks you can attach a repeater specification to a date.
Org Mode also supports checklists that are mentioned in GTD to support yourself when conducting more complex recurring tasks.
And if a piece of your stuff is not actionable then it could be valuable information that you want to store for future reference. Because of this I maintain several *--notes.org files that keep all the non-task stuff.

Find my Emacs/Org configuration on Github.

Effects on my daily life

Managing my whole stuff, professional and private, only takes me 5 to 10 minutes each day. I do a brief review once a week, usually during the weekend. Putting everything into Org and using the Org agenda creates clarity in my life about "what's next". Now I can literally see in the Org agenda if the upcoming days promise to be stressful or easy, and I can shift things to balance the load, taking my personal level of energy into account. If I have to wait for someone before I can continue on a task I use the WAITING state and move the scheduled date to some days in the future. If a task is running longer I attach notes to it so I can easily recall later what the status is. Because of this it is no burden to have 2, 5 or 10 projects in parallel, because I am allowed to forget almost everything about them until a related task shows up in my agenda again.

This "permission to forget" regularly gives me inner peace, so I can take my time and exclusively focus on finding a solution for a customer, or to go on auto-pilot when getting things done at home.

Being sure to never forget anything and being able to react to forseeable congestions gives me so much confidence and control that I eventually started new personal projects. The mental relaxation helps me getting more things done than before, without increasing my level of stress.

My quest for the ideal keyboard switch

01.06.2021 Permalink

Last year I built my first ergonomic mechanical keyboard with brown (tactile) switches. After an initial phase of training I am very happy with the layout and keymap. Typing has become much more relaxed, there's no need to ever look at the keyboard to find a key, and my hands always remain in the same position.

There is actually a wide variety of keyboard switches out there so as I was striving for the perfect personal keyboard experience I was curious about how other switches felt like.

So, I used the last 12 months to build 4 more boards with the exact same design but different colors of acrylic plates, a pair made of poplar plywood and different switches. Here's how they turned out:

Dark anthracite acrylic plates, Cherry MX Red switches (linear 45g, lubed). Never thought that I appreciate linear switches that much.

Translucent acrylic plates, covered by a stickerbomb, again Cherry MX Red switches (linear 45g, lubed). Initially, I didn't plan to cover the board with stickers. But unfortunately, the glue created an ugly look on the translucent plates so I used the stickers as a rescue.

Poplar plywood clear coated, Cherry MX Black switches (linear 60g, lubed). This very light wood makes the keyboard weigh only 280g. It is very quiet and I use it while sitting on the couch.

Poplar plywood yellow painted, Kailh Speed Copper switches (tactile, 50g, shorter travel distance, lubed, dampening O-rings). Actually much more tactile than any MX Brown or clone. The typing sound is considerable though. This is now my daily driver in the office.

And here's my first build again with white acrylic plates. I replaced the brown switches with Kailh BOX White switches (click-bar, 55g, shorter travel distance, dampening O-rings) to see if I like the sound and feeling. To me, in terms of clickyness it is by far superior to any click-jacket based switch like MX Blue or clones.

While it is fun and relaxing to build these boards it also served a serious purpose: find the ideal switch for me. Impressions from typical switch testers are limited because you can't do actual work with one of the switches.

What did I learn about switches and what type do I like best?

MX Browns or clones are very popular and I used them myself for many years, but they are not really tactile. It is worthwhile to compare them to linears or more 'modern' switches like Kailh Speed Copper.
MX Blues, known to be the clicky switches, are actually mushy compared to click-bar based MX style switches of the Kailh BOX series. If you're into clicky switches then I strongly recommend to give them a try.
My fingers tend to bottom out on linears, so I always get the 'clack' and a physical feedback, plus I like the smoothness. Thus, I prefer a lighter linear switch over a clicky one for fast writing. For a more relaxed and quieter operation in the evening I find MX Blacks with their higher resistance more pleasant.
I am fastest on Kailh Copper Speed, and their tactility often prevents me from bottoming out, so the actual travel distance is by far shorter compared to all other switches I used so far. However, they're by no means quiet switches.
I don't play any computer games. Therefore I can't tell which type of switch might be the best for me in such a setup. The general opinion is that light linears are the typical choice for gamers.

My very personal summary for the time being: MX Blacks are my leisure time switches, but for serious writing I use Copper Speed or MX Red. But who knows? There are still many more switches to try, so maybe I'll do some more builds just to find out.

Six years of professional Clojure development

10.05.2021 Permalink

Over the last couple of years me and my colleagues here at doctronic have been busy creating and maintaining more than a dozen of individual software systems for our clients.

We have the privilege to almost exclusively use Clojure and ClojureScript, as this is still the valid strategic technology decision made by doctronic shortly before I joined them in 2015. In this post I'd like to list some of the strengths and weaknesses in everyday real-life use of the language and its ecosystem.

Stability of the language and libraries: Anyone who ever created software of practical value for paying customers knows that reliability of core language constructs and libraries is important. It supports predictability, and eventually saves you from insanity whenever you want to benefit from performance improvements or bug fixes. Upgrading to a new version of a library or even Clojure itself is risk-free. The authors and maintainers really do care about stability. If you listened to talks of Rich Hickey or Stuart Halloway you know that this is no coincidence. "Don't break things!" is part of the culture.

Code size: If you compare some of the idiomatic code snippets in Clojure with the equivalents in more popular languages you might conject that whole Clojure code bases might in general be smaller compared to what you can achieve in other languages. Now, after having worked more than 20 years with imperative OO languages and more than 5 years with Clojure I can firmly state that you'll achieve the same functionality with a fraction of the amount of code. This impacts costs: My feeling is that we now build systems with only half of the staff that I used to have on projects with a comparable scope.

Finding staff: Clojure's popularity is still far, far behind imperative general purpose languages like Java or Python, so fewer people feel encouraged to invest the time to learn the concepts and gain practice. It seems that this makes hiring harder. doctronic addressed this challenge by positioning itself as a supporter of the language in Germany through organizing and sponsoring the annual :clojureD conference. This helps greatly with staffing. But even if a company has a lower visibility, the fact that it uses Clojure will attract aficionados of advanced programming languages. If these developers do enjoy their workplace they will usually stay around longer with their employer. So I'd argue that you gain quality in skills and a lower turnover rate, but it might take more patience to build up a team of Clojure developers.

Teaching the language to apprentices: doctronic constantly employs apprentices so I was able to accompany some of them and help them learn Clojure. Young minds with very few programming experience are remarkably fast in picking up the language and become productive, it seems. I assume there are two main reasons for this: There's actually a whole lot of things they don't need to master compared to OO land. And they don't need to unlearn anything, in other words, they don't need to overcome old habits of doing things.

Some discipline required: Clojure is a language that gives us a wide range of options to implement solutions. And it imposes very little ceremony. For example, there is no static type system that checks your code upon compilation, but you can get some of the benefits for your data at runtime by using Clojure spec. This also serves as documentation and possibly a generator for test data. It is up to the team to decide when and where more checks, restrictions and tests have to be added. If the team misses out on this the code base might be harder to understand and maintain than necessary. In essence: with great freedom comes great responsibility.

Navigating the code: When maintaining or extending an existing code base you often need to find specific places where a function or a piece of data is referenced. To support this kind of search for specific identifiers it is important that naming is self-explaining and consistent, even more so because there are no links established via a type system. To mitigate this weakness in Clojure, you should use qualified keywords whereever possible.

Documentation becomes more important: When you're trying to understand a function that invokes other, non-core functions you'll need a clear idea what kind of data these functions expect and return. Without any data type declarations the code itself often doesn't reveal how the data that flows through you functions looks like. You'll need either Clojure spec or more thorough documentation to mitigate this problem. It is not so rare that you need to test functions of interest isolated in the REPL to get a clear idea how the data looks like.

The promise of purity: Any practical software application must cause some side-effects. So even if Clojure strongly encourages developers to create pure functions your systems will contain substantial parts that either depend on some environment or cause side-effects. The trick is to separate these pieces sharply and keep the unpure parts small. Without any discipline you could easily spread side-effects everywhere and end up with the typical imperative mess. Clojure makes it easy to create a mostly pure implementation but it does not enforce this. To ensure this and other qualities the teams at doctronic conduct code reviews for almost every commit before it is promoted into the master branch.

Reusability: Here's where Clojure really shines. Because most functions are pure and operate on few common data structures (mostly maps, vectors and sets) it is very easy to write reusable code. We created many internal libraries just by separating the candidate functions into their own namespaces within the regular project source tree. To finally establish the library we move the code from the project repo into a new Git repo and include the resulting library Jar in the project.clj as dependency. Thus, we have a very lean process that results in production-quality reusable assets.

Startup time: Starting a fresh JVM with a Clojure Uber-Jar to bring up an application takes noticable time. I assume that JVM class loading causes this delay. So if you consider to create a command line tool that has a short net execution time you wouldn't want this runtime overhead.

Runtime stability: Clojure applications are in general very stable. I remember that in bigger Java projects from my past we always had to do some load testing in order to detect programming flaws like memory leaks or race conditions. With Clojure, we do load testing only to find real performance issues.

Ok, time to come to an end before this post becomes too long. The list is perhaps not complete, but I guess I included the most important aspects that recently popped into my mind. When I decided to fully jump on this train in 2015 I expected more unpleasant surprises. Now, six years later I have proven to myself that Clojure and ClojureScript are a practical and sound choice for real world software development, and I still enjoy using them.