Continuous testing with Cantata

Within a DevOps pipeline Cantata provides automated continuous unit and integration testing. Cantata tests can be run automatically every time code is checked-in to verify new or changed code. Cantata is easy to fit into DevOps pipeline and offers:

All tool options are available for direct command line invocation, lending itself to automation in scripts, makefiles, batch files, and Jenkins declarative pipelines within a DevOps or Continuous Integration/ Continuous Delivery framework.

Test infrastructure as code

Cantata test cases, test harnesses and stubs and wrappers are all generated as clean code. Cantata’s test scripts are generated as C or C++ code, they are simple to read, well structured and easy to maintain.

Tests written in the same language as the code under test make it intuitive to edit and customize test scripts. This format enables the tests to be maintained alongside the code under test. If your infrastructure is also maintained as code, this guarantees repeatable, deterministic results every time you build and run your unit and integration tests.

Impact Analysis

Impact Analysis can be used to automatically re-run only the Cantata tests impacted by code changes, a technique sometimes referred to as ‘change based testing’. This is done by calculating the transitive closure of a software change to determine which code files have changed and which particular tests have been impacted (touched) by those files. This means that only the relevant subset of tests need to be re-run, reducing the complexity and execution time of the pipeline.

Cantata in a DevOps Pipeline

The Cantata test framework can be used for multiple optional steps within the static analysis and software test stages in a DevOps Pipeline. With workflow triggered by a commit to a repository, these steps with their example quality gates are described below:

Static metrics

Generated at build time Cantata static metrics provide useful information on non-functional qualities of the source code, such as its complexity and ongoing maintainability.

Build and run impacted unit tests

Where an existing suite of unit tests is available only impacted tests need to be built and run. Cantata measures structural code coverage achieved and produces certifiable results data.

Auto-generate missing tests

The Cantata AutoTest feature automatically generates test cases to exercise all code paths according to a structural coverage metric. This can be used on code with no existing unit tests to generate a set of baseline tests. For code with existing unit tests AutoTest can be used to generate extra tests as needed to achieve the required code coverage. Cantata can also automatically generate a test case per function as a useful start point for test creation.

Requirements-based unit tests

Bi-directional trace relationships between Cantata test cases and requirements can be established within the Cantata UI. Data for this is then available via a REST API enabling verification of these relationships as a quality gate within a DevOps pipeline.

Build and run impacted integration tests

Cantata integration tests can also be run within a DevOps pipeline. The unique Cantata Wrapping feature provides powerful integration testing by enabling calls to be intercepted as an addition to mocking or stubbing which simulates calls.

Requirements-based integration tests

Data is also available via REST API for trace relationships between integration test cases and requirements.