Making the Jasper build faster and more reliable with Docker and other stuff

Maybe a little embarrassingly, most of my efforts in Jasper this summer has been around making all the automated test suites run faster and far more reliably. I first invested a dreadful amount of time toward making the main xUnit test library run tests in parallel.

Next, Jasper has extensions that integrate Consul, RabbitMQ, Sql Server, and Postgresql via Marten. As you can probably imagine, that makes integration testing a little bit of a mess with all the infrastructure dependencies. The prevailing trend — at least in my circles — these days is to try to leverage Docker for infrastructure needs within build automation. To that end, I first tried to use the Docker.Net library in the manner I wrote about in *A* way to use Docker for integration tests (which I need to update because our team had to tweak for CI usage).

That worked okay, but you’d still hit occasional timeouts and there were a few tests that needed both one of the databases and Rabbit MQ running. Instead, I switched to just having a tiny Docker compose file like this:

version: '3'
services:
  postgresql:
    image: "clkao/postgres-plv8:latest"
    ports:
     - "5433:5432"
  consul:
    image: "consul:latest"
    ports:
     - "8500:8500"
     - "8600:8600"
  rabbitmq:
    image: "rabbitmq:latest"
    ports:
     - "5672:5672"
  sqlserver:
    image: "microsoft/mssql-server-linux:2017-latest"
    ports:
     - "1433:1433"
    environment:
     - "ACCEPT_EULA=Y"
     - "SA_PASSWORD=P@55w0rd"
     - "MSSQL_PID=Developer"

which I’m showing mostly to call out “that was easy, why didn’t I do this ages ago?” In my build script for Jasper, it just makes sure to call “docker-compose up -d” once before running any of the integration test suites.

I also folded all the test libraries for the various integrations into a single test library called “IntegrationTests,” but used the xUnit [Collection] attributes and base context classes to effectively allow each of the extension libraries to be tested in parallel threads, but single threaded within the set of tests targeting each extension library.

All told, the combination of:

1. Upgrading to netcoreapp2.1 for all test libraries
2. Ensuring that the main xUnit testing library can run specifications in parallel
3. Integrating the Docker infrastructure set up inside the build script itself
4. Letting the tests for the Consul, Postgresql, MSSQL, and RabbitMQ integrations run in parallel in the combined integration testing library
5. Moving any “flaky” tests that were vulnerable to timing or threading issues to Storyteller where it’s more robust
6. Fixing the real underlying issues in the code around some of the “flaky” tests:-(

made the automated build far faster and more reliable than it was before I started that effort and I’m one of those people who thinks that’s extremely important for the success of a project.