Testing

{:toc}

Strategy for Bessy

The faster a problem is caught, the cheaper it is to fix (because the humans forget what they did over time). We shall employ many layers to achieve this:

TODO - update as we get these things implemented

Unit Testing

For Bessy Python unittest tool seems to be sufficient for now. As always with Python, there’s lots of alternatives.

Bessy Unity leverages Unity’s Test Runner to run unit tests.

Python Setup

Unit tests live in their own file, the file name must start with “test_”, for example “test_classification.py”. The test name should correspond to the class or unit or topic that is being tested.

The unittest tool follows the same basic paradigm of JUnit, CppUnit, XCTest, etc. A “unit test” consists of a set of test functions that check a small portion of functionality. Each test is run independently from a known state:

import unittest

class TestNumMethods(unittest.TestCase):

   def test_abs_of_positive_is_positive(self):
      value = 50.0
      self.assertEqual(abs(value), 50.0);

   def test_abs_of_negative_is_positive(self):
      value = -100.0
      self.assertEqual(abs(value), 100.0);

You can detect and run all tests like this:

python -m unittest -v

test_abs_of_negative_is_positive (mymodule.test_num.TestNumMethods) ... ok
test_abs_of_positive_is_positive (mymodule.test_num.TestNumMethods) ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.000s

Unity Setup

TODO - explain this… note, there are failing and flaky tests at on Windows.

Unit Test Style

Writing tests is a bit different from writing production code. We want tests to be easily understood when an assertion fails. That means tests should strive for readability over compactness. Key things to observe:

In the example below, note how it’s clear what the test is checking and that there is only one assertion:

def test_can_make_dog_happy(self):
   dog = Dog("fido")                 ## arrange
   dog.praise()                      ## act
   self.assertTrue(dog.isHappy)      ## assert

If we also wanted to tests that Dog wags its tail and leaps when happy, it’s better to create a new test. While that duplicates code, single assertions make it obvious why a test is failing. This is a key difference between test code and production code. We prefer tests to be explicit at the expense of duplication.

Test driven development

Tests are a powerful tool for guiding software development. By writing a test before you write production code, you are forced to think about how code is used by others. For example, how will the function be called? how are the results provided? how do I tell that the function did what it is supposed to do?

The process is simple:

The hard part is having the discipline to follow the process. Avoid the temptation to write all the tests first. Don’t have someone else write the tests. This robs you of feedback from the tests. Just write one test at a time. Each test will provide a little bit of feedback on the chosen design. It’s very normal to get about 3 tests into development and realize that your design needs to change. Let the tests guide you.

When should you use TDD? When writing production code or complicated code. If you’re just hacking something together that’s going to be thrown away in two days, it’s probably not worth it.

Bessy challenge

For Bessy, a lack of unit tests is a challenge because:

Code that was created without unit tests is hard to work with. Creating that first failing test may require significant refactoring, which increases the risk of injecting a bug.

A good strategy when working on code like this is to create a few tests around the area that you will be working. Think of the tests as setting up scaffolding before making changes. These tests can help detect unintended change. Once the scaffolding is up, resume regular test first development can begin - start with a failing test.

Further Reading

Foundations

The biggest hurdle to unit testing is having software that can be tested. All the canned examples on the internet skip this part making it seem way easier than it actually is. The first time you try to add a test to existing code, it’s going to be really hard because the code wasn’t designed for testability. This is a normal part of the experience, and unfortunately, folks give up at this point.

To be proficient at unit testing, you need a solid understanding of how to structure code so that it can be tested. The following books are excellent places to start.

Bob Martin’s book on the “SOLID” principles is a great intro to software design. Dependency Inversion or Injection (the D in SOLID) is a key technique in designing software that can be unit tested. Read the section about SOLID principles. The latter half of the book is about library/package design principles and can be skipped.

Kent Beck’s book on unit testing is really all you need. His example uses Java, but the concept is applicable to any language. Step through the first section, it’s well worth it.

Hands on

These videos do a decent job of showing the TDD technique in real time, but don’t get into design topics. The examples are done using a similar stack to Think2Switch (JavaScript, Node.js and Jest). There’s also a repository with example code so you can go through the “katas” on your own. The first example is a classic software interview question:

When writing tests, it helps to follow a common format (Greg Wilding update this):

Mocking (Greg Wilding update)

Further reading

This video reflects on the software industry’s adoption of unit testing / TDD and common misconceptions and pitfalls. The presenter assumes previous experience with unit testing.