TDD in Clojure, part 4 — The Dark side of TDD

This is the fourth and final part of this series. The goal of this series was to share some insights on how to use Test Driven Development (TDD) in Clojure as I learn this language while ramping up as a software engineer at Nubank.

In Part 1 I presented the idea of using quadratic equations to learn about TDD, gave some contextualization on what is TDD and why it is important. I also showed how to set up the environment to start coding.

Part 2 is my personal favorite part so far. It was about the Detroit school of TDD, but also deals with some important topics while learning Clojure, such as Java Interop and an introduction to Schemas.

In Part 3 I talked about London School of TDD and the use of mocks. Personally I didn't liked it as much as the others because our proposed problem (quadratic equations) was a bit simplistic to deep dive into some of the more interesting aspects of London School. I hope to be able to revisit this theme in future articles and come up with better explanations.

What lies ahead

There are a bunch of topics that I still want to cover:

• Boundary testing — What are the limits of TDD and how boundary testing is important to cover blindspots.
• Code coverage — How to measure your code coverage, and why you shouldn't rely on it too much.
• Custom schemas — how to use plumatic/schema for more than type checking.
• Generative testing — A powerful tool to complete your test suit.

This series was written with an incremental approach in mind, and I am constantly making references to content of the previous posts. Please keep that in mind before going beyond this part and take some time to read the other parts if it becomes hard to follow.

Limits of TDD

“There is no silver bullet”. You probably read this sentence many times in you career as a software engineer. If not, you will.

It is probably true, since it has been repeated for more than 30 years:

It comes from a paper written by Fred Brooks, and can be found on his famous book, the Mythical Man Month. It states:

there is no single development, in either technology or management technique, which by itself promises even one order of magnitude [tenfold] improvement within a decade in productivity, in reliability, in simplicity.

Obviously, the same can be said about TDD. While it is a wonderful technique that can improve your code quality and the ability to deliver fully testable code, there are some important blind spots that everyone should be aware of.

TDD May lead to a false sense of security, as you wrote tests for every single line of production code. Some metrics, such as Code Coverage can be misgiven.

Get our code coverage

In Clojure we can measure our code coverage using Cloverage (https://github.com/cloverage/cloverage).

It is really easy and straight forward to use. Basically you only need to add the dependency to your project.clj file and execute

lein cloverage

in the command line.

I'll use the code I made in part 3 as a starting point. You can find it at:

GitHub - the-sidh/tdd-clojure-london

The code is already updated with 2 small changes:

1 — I added lein-cloverage as a dependency on project.clj

(defproject london-quadratic "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0"
            :url "https://www.eclipse.org/legal/epl-2.0/"}
  :dependencies [[org.clojure/clojure "1.10.3"]
                 [prismatic/schema "1.2.0"]
                 [nubank/mockfn "0.6.0"]
                 [lein-cloverage "1.2.2"]] <- HERE
  :repl-options {:init-ns london-quadratic.core}
  :plugins [[lein-cloverage "1.2.2"]])

2- I had to make some java type juggling on the function quadratic-formula to have Cloverage working fine. For some reason, the line

sqrt-discriminant (.sqrt discriminant (new MathContext 10))

was breaking the lines coverage count. I think it is something to do with the not-so-new version of the sqrt that requires a MathContext.

I used Math/sqrt instead, and had to convert from BigDecimal to Double, and back to BigDecimal again. In the end I got:

Changes on lines 9 and 10

The conversion are made on Lines 9 and 10. Since we have tests covering our whole project, when I run the tests I can spot instantly if this change had any side effect. It didn't, and that is great!

When I run `lein cloverage` I got:

Cloverage reports 100% lines covered

It also gave me some fancy HTML reports:

Fancy HTML report

Everything seems alright. I have 100% coverage for my lines, and if all my lines are covered by tests, and all my tests passes, I have no bugs, right?

Wrong…

The Dark side Of TDD

This 100% lines coverage brings the false sense of security I mentioned above. Take a look into what tests we have, so far, for the quadratic-formula function:

Besides the tests that ensure that we are validating the input, the only test that is validating the actual business rule is

(testing "given a =1, b=-1 and c = -12 when calculating the roots of the quadratic equation should return a set containing 4 and -3"
  (providing [(discriminant 1M -1M -12M) 49M]
             (is (= #{4M -3M} (formula/quadratic-formula 1M -1M -12M)))))

We are testing only for one combination of parameters. More importantly, looking at line 13, you will find out that we are only testing the result of discriminant as 49.

Source: https://codingjourneyman.com/2015/06/29/testing-is-a-developer-job/

This lack of tests is disturbing the force.

Boundary Testing come to rescue

In "Pragmatic Unit Testing in Java 8 with JUnit", the authors describe Boundary Unit Tests as

(testing) the edges around the happy-path cases where things often go wrong

In our case, what should happen if it discriminant was ZERO? Or if it was a negative number? Let's find out!

Discriminant equals to ZERO

I'll first add a test to verify what happens if the discriminant is ZERO. I know, back from the days in school, that the result should be a set with only one item, since the quadratic formula is:

Quadratic Formula

And the square root of zero is zero, both branches, plus and minus, will give the same result.

As we chose to use a set as the return of function, and sets, by definition, don't allow duplicates, I expect to get only one item in the set instead of two items with the same value.

For this test I'll use A=4, B=-4 and C=-1, as the result should be a set containing 0.5.

(testing "given the discriminant is Zero should return a set with only one item"
  (providing [(discriminant 4M -4M -1M) 0M]
             (is (= #{0.5M} (formula/quadratic-formula 4M -4M -1M)))))

When I run the test, it fails with an error:

expected: (= #{0.5M} (formula/quadratic-formula 4M -4M -1M))
actual: java.lang.IllegalArgumentException: Duplicate key: 0.5

Here is our first bug! If by chance, the discriminant is zero, clojure.lang.PersistentHashSet don't allow the second value to be inserted in the set.

Clojure sets works differently for sets in other JVM based languages, like Kotlin, the language I was using for some years before joining nubank. Kotlin silently ignores the duplication. Clojure chose to raise an IllegalArgumentException.

We can deal with this many ways. The easiest way I found was simple workaround (did I heard gambiarra?) :

I used the function "set". It receives a collection as argument, and safely convert it to a set. I will replace

#{(-> minus-b (- sqrt-discriminant) (/ two-times-a))
  (-> minus-b (+ sqrt-discriminant) (/ two-times-a))}

with

(set [(-> minus-b (- sqrt-discriminant) (/ two-times-a))
      (-> minus-b (+ sqrt-discriminant) (/ two-times-a))])

and ended up with:

When I run my tests again, they will pass:

Discriminant is negative

Now let's see what happen if the discriminant is a negative number.

In this case, my math teacher taught me that the results are not real numbers, since they can't allow the square root of a negative number.

Debatably, we could raise an exception in this case. Instead, I prefer returning an empty set. It is more in tune with the idea of creating reusable high order functions.

I came up with this test:

(testing "given the discriminant is negative should return an empty set"
  (providing [(discriminant 1M -4M 5M) -4M]
             (is (= #{} (formula/quadratic-formula 1M -4M 5M)))))

When I try to run this test, I get a VERY weird error:

expected: (= #{} (formula/quadratic-formula 1M -4M 5M))
actual: java.lang.NumberFormatException: Character N is neither a decimal digit number, decimal point, nor “e” notation exponential mark.

The REPL is a great tool to understand what happens in clojure.

First, I'll evaluate what happens when I try to calculate the square root of -4 using Math/sqrt.

NaN stands for Not A Number. We can see this behavior documented on the javadoc page for Math class:

Returns the correctly rounded positive square root of a double value. Special cases:

If the argument is NaN or less than zero, then the result is NaN.

If the argument is positive infinity, then the result is positive infinity.

If the argument is positive zero or negative zero, then the result is the same as the argument.

Otherwise, the result is the double value closest to the true mathematical square root of the argument value.

If we take this NaN thing and try to create a BigDecimal with it, we get that same error:

There are many ways to solve this. I really like the behavior of BigDecimal when we try to get a square root of a negative number. It throws a nice ArithmeticException at us. My approach will be create an function that emulates this behavior and use try-catch to deal with this in my quadratic-formula function:

This is a common limitation of TDD and it is a trap you should avoid.

In part 1, I introduced the 3 step cycle of TDD:

1-Write test

2-Write production code

3-Refactor

Doing TDD doesn't mean that you will not have to create additional tests after the 3 steps cycle is completed!

Boundary testing is a mindset that we all must in mind, regardless of doing or not TDD. Writing your tests and code following TDD cycle and writing boundary tests are complementary techniques.

More about Code Coverage

Before moving on to the last 2 topics I'll cover in the series, allow me one more rant against Code Coverage.

I have worked on more than one project, in different companies, that at some point had Code Coverage as one engineering OKR. For god’s sake, I remember even suggesting it more than once.

My current opinion is that code coverage should used only by tech engineerings to spot improvement opportunities in the current test suit. It’s use to measure quality is arguable, as I demonstrated above, and definitely should not be used as tool to measure productivity.

One example: If you have a project that has 80% of production code coverage by tests, the 20% that is not covered by are blind spots, and test coverage tolls will help to spot this.

Don’t trust blindly on the other 80%, however. Use techniques such as generative testing and boundary testing to make sure your tests are comprehensive.

Using schemas to enforce parameters validation

In part 2, I used plumatic/schema to ensure our arguments were BigDecimals, allowing our functions to run safely, since they rely on functions from that type.

Way back in part 1, I asked my readers to bear with me:

These are all examples of complete quadratic equations or it’s standard form, since a, b and c are not zero. The code we are going to write will only solve equations in this form only (bear with me, there will be a good reason for this).

The "good reason" was proving the opportunity to use schemas for more than type checking. I'll use it to make sure that A, B and C are BigDecimals and are not ZERO.

First I'll create a pure function that evaluates if a given number is ZERO. To do so I'll start by creating a new test. A good place for this function may be the file core.clj, that we kept empty so far.

Then I'll create the production code:

Both function and production are pretty simple and don't require additional comments. When I run the new test, it passes.

Applying the idea of boundary testing, I should create at least two additional tests, to evaluate values bigger and lesser than zero:

Running all new tests, everything seems to be okay:

I'll leave my new function to rest for a while, and will create a test to evaluate if I an exception is raised if A is equals to ZERO:

(testing "given a is zero should throw an exception"
  (is (thrown? ExceptionInfo (formula/quadratic-formula 0M -1M -12M))))

The test, as expected fails

FAIL in (quadratic-test) (formula_test.clj:28)
given a is zero should throw an exception
expected: (thrown? ExceptionInfo (formula/quadratic-formula 0M -1M -12M))
actual: nil

since my function is only validating for BigDecimals, and A equals Zero causes the function to return nil (since A is used as a divisor in the formula).

Take a look at the function:

(s/defn quadratic-formula [a :- BigDecimal. <--HERE
                           b :- BigDecimal
                           c :- BigDecimal]
  (try
    ...))

What I need to do is to define a more specific schema, and replace BigDecimal after :- for this new schema.

I'll do that by creating a new name space and defining my custom schema there:

The "constrained" function allows the creation for a schema with an additional condition. We provide our pure function "not-zero" as the additional condition on top of being a BigDecimal.

Now I can use this customized schema to validate "A":

s/defn quadratic-formula [a :- coefficient/bigdecimal-not-zero
                           b :- BigDecimal
                           c :- BigDecimal]
  (...)

A few TDD cycles later, I'll have all my parameters covered, and comprehensive tests:

When I run my tests, everything goes according to the plan:

Going one Step further with generative tests

So far I have been doing "example-based-tests". I provide some examples of inputs that I, beforehand, knew the expected result. This way of testing works really well for TDD and for boundary testing, but we can go one step further and do property-based-tests.

The idea is to run our functions against a set of random values, evaluating broader conditions.

In my case, I can think in calling quadratic-formula one hundred times, with random BigDecimals as A, B and C, and compare the size of the results against the three possible options :

• Empty set, if the Discriminant happens to be negative.
• One element, if the Discriminant happens to be ZERO.
• Two elements, if the Discriminant happens to be positive.

Clojure has a really nice support for this, with test.check.

To use it, the first step is to include the dependency in my project.clj file

(defproject london-quadratic "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  :url "http://example.com/FIXME"
  :license {:name "EPL-2.0 OR GPL-2.0-or-later WITH Classpath-exception-2.0"
            :url "https://www.eclipse.org/legal/epl-2.0/"}
  :dependencies [[org.clojure/clojure "1.10.3"]
                 [prismatic/schema "1.2.0"]
                 [nubank/mockfn "0.6.0"]
                 [lein-cloverage "1.2.2"]
                 [org.clojure/test.check "1.1.1"]] <--HERE
  :repl-options {:init-ns london-quadratic.core}
  :plugins [[lein-cloverage "1.2.2"]])

Now I can write new test in formula_test.clj for this:

This small excerpt of code do a lot!

(def non-zero-big-decimal
  (gen/fmap #(bigdec %) (gen/such-that #(not (= 0 %)) gen/large-integer)))

Here I am defining a custom generator. It will:
gen/large-integer -> generate a random large integer number

then:

(gen/such-that #(not (= 0 %)) -> disregard the ones that are equals to zero

and finally:

(gen/fmap #(bigdec %) -> map it to a bigdecimal.

With this function ready, I can use it to generate the parameters

(defspec size-is-expected 100
         (prop/for-all [a non-zero-big-decimal
                        b non-zero-big-decimal
                        c non-zero-big-decimal]
                      (contains? #{0 1 2} (count (london-quadratic.formula/quadratic-formula 1M -1M -12M))))

defspec size-is-expected 100 -> The specification will be executed one hundred times

(prop/for-all [a non-zero-big-decimal
b non-zero-big-decimal
c non-zero-big-decimal] -> assigns the result of the custom generator to a, b and c

(contains? #{0 1 2} (count (london-quadratic.formula/quadratic-formula a b c)) -> counts the number of elements in the set returned by quadratic-formula and tries to match it against the possible results: 0, 1 or 2.

When I execute the test, it runs 100 times:

There is still a lot about generative tests. As other topics, such as mocks, my goal was to introduce the concept and how it can be used in the context of this minimal study project. I strongly suggest using the documentation of the projects as a start point to learn more that you can do with both mockfn and Test check.

Complete Source Code

The complete source code for the final part is avaiable on my GitHub space:

GitHub - the-sidh/tdd-clojure-final

We still have 100% lines coverage, but who cares?

Conclusion

With this part I conclude my basic study on how to apply TDD in Clojure. I hope it is useful to someone besides myself.

Please feel free to reach out for feedback, suggestions for improvement or follow up questions.

Thank you a lot for reading.

References

Building Nubank

The Mythical Man-Month - Wikipedia

GitHub - cloverage/cloverage: Clojure test coverage tool

GitHub - plumatic/schema: Clojure(Script) library for declarative data description and validation

test.check/README.md at master · clojure/test.check

Pragmatic Unit Testing in Java 8 with JUnit