Why do we even test?

Everybody, everywhere is constantly testing. Most subscription services, from newspapers to video streaming, offer free trials so that potential customers can test the service before committing to it. The goal is to convince users that the service is indeed worth the money and prove that it meets all (or at least most) of their expectations.

Using such trials or samples to test products is so common that it seems almost crazy to ask this question: Why do we even test?

While developing OnMCU (my new startup for embedded testing automation), I’ve been thinking about how to approach systems engineering for embedded devices. Especially regarding testing software of embedded systems, I encountered a few more interesting questions:

• What is the goal of testing?
• What can testing do and what can it not do?
• And since in the real world things inevitably go wrong at some point: How should errors be handled?

With the last one leading to an even more fundamental question: What is an error even?

Fortunately, many smart people, like for example the famous Edsger Dijkstra, have been asking the same questions and they came up with answers. This blog post is a small collection of what I have found to be the most interesting ideas, trying to give credit wherever possible.

#What is the Point of Testing?

tl;dr:

• Tests should be used to prove correctness, not to find bugs
• Tests can only prove the presence of bugs, not their absence
• Program components need to be structured with testing in mind — otherwise the program is not likely to be correct

The goal of test is to prove correctness, not to find bugs. If you study the quality movement you’ll see this precept has revolutionized manufacturing. Alas, it hasn’t seeped into software engineering.

— Jack Ganssle (https://www.ganssle.com/tem/tem500.html)

In general, I think, this is a good point to start. Write tests with the goal of proving correctness under all possible circumstances. For unit tests this usually boils down to testing a procedure against an expected set of inputs. By expected I do not necessarily mean the “wanted” set of inputs — you need to expect the unwanted and unexpected, as well. In practice, unfortunately, this is easier said than done. We are working with actual hardware and hardware itself is, to some extent, undefined behavior.

A sensor may die, a connection may corrode or some ominous cosmic ray may flip your precious bit. Not all of this can be tested and “in the wild”, things (or users!) will happen you never thought of. In the end, we have to settle for statistical statements about the likelihood of things going sideways. Which brings me to a quote by the famous inventor of the Dijkstra algorithm, Edsger Dijkstra:

Recalling that our true concern is with really large programs, we observe as an aside that the size itself requires a high confidence level for the individual program components. If the chance of correctness of an individual component equals p, the chance of correctness of a whole program, composed of N such components, is something like $P = p^N$ As $N$ will be very large, $p$ should be very, very close to $1$ if we desire $P$ to differ significantly from zero!

— Edsger Dijkstra (https://www.cs.utexas.edu/~EWD/transcriptions/EWD02xx/EWD249/EWD249.html)

Especially for embedded applications, we might want to assign weights to these probabilities. In most, if not all, applications, not every single part of the program is super critical. So it is generally a good idea to identify the most critical parts, isolate them and increase the probability of them being correct.

Imagine a washing machine: one very critical part is ensuring the door cannot be opened while the machine is operational. We might want to ensure this not only in software, but even in hardware. Safety first! When the machine has finished its job, an LED might light up and short sound of success may beep out of a speaker. Not a critical feature at all — most of the time, nobody will hear it anyway. So we might only want to ensure that this part of the program, should it fail, does not block or crash the rest of the program. And call it a day.

And since

Program testing can be used to show the presence of bugs, but never to show their absence

— Edsger Dijkstra (https://www.cs.utexas.edu/~EWD/transcriptions/EWD02xx/EWD249/EWD249.html)

we need to have more testing wherever we need to show more bugs’ presence.

What do we learn from this? Proving correctness is inherently difficult. And even though testing is a powerful tool, it is not capable of proving a program’s correctness. Still, we should write tests as if it were.

#What is an Error?

tl;dr: Errors can be “bugs” (programming errors) or “forks in the control flow” (unexpected/unwanted inputs)

When people talk about errors, they might mean two entirely different things.

According to the Cambridge Dictionary an error may be

something done or written by accident that is not correct, not accurate, or does not give the right result

or

a mistake, esp. in a way that can be discovered as wrong

The first one is an error, as sometimes found by the compiler. It is often called a “bug”, but to be precise, it is a “programmer error”.

The second one is a little more difficult, although it is the more common usage of “error” in programming. We can think of this “mistake” as a “Fork in the Control Flow” (Tim McNamara, IEEE SE Radio #644), where the program receives an input that is unexpected (or not understandable, i.e., some precondition is not satisfied) and thus deviates from the “ideal” program flow. For example, the program tries to open a file that does not exist. This might not be the intended or “good” case, but it is something that can be handled (e.g., by creating the file).

In a sense, such a “mistake” can become a “bug” if it is not handled correctly. ^[1]

I think Dijkstra is right to a large extent when he writes that

We could, for instance, begin with cleaning up our language by no longer calling a bug a bug but by calling it an error. It is much more honest because it squarely puts the blame where it belongs, viz. with the programmer who made the error. The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking is intellectually dishonest as it disguises that the error is the programmer’s own creation.

— Edsgar Dijkstra (https://www.cs.utexas.edu/~EWD/transcriptions/EWD10xx/EWD1036.html)

because only identifying the source of an error allows us to take appropriate measures.

#To Panic or Not to Panic

Talking about appropriate measures: what does this even mean?

Deciding on what to do often boils down to the question of “to panic or not to panic?”, that is, should the program shut down (somewhat gracefully) or can the error be handled in a way that allows for the program to continue (the aforementioned “fork in the control flow”).

This is not an easy question to answer. When writing a small program for oneself, maybe a couple of hundred lines of code, sophisticated error handling may not be necessary. Often it is even okay to just let the program crash. In libraries or programs that are used by other people, error handling needs to be more than crashing.

As a general rule, you should panic only when this is literally the only way to go. There are not many occasions where this is true. More often than not, the same error will occur on the next run of the program, so just restarting it will not help for long.

#Your Input!

• Who (and whose ideas) have I missed?
• What are your biggest pain points with testing?
• What has been your greatest testing success?
• Is there an argument to be made about testing from a Bayesian or information theoretic perspective? That would seem quite natural to me, but I haven’t come up with a good idea about that. Testing reduces uncertainty, so it reduces entropy by giving us more prior information about the system. What implications does that have?

If you have any remarks, comments or ideas, drop me an e-mail to #anVsaWFuQG9ubWN1LmNvbQ==