SmingTest

An extensible test framework for Sming, with integrated profiling and scheduling support.

Getting started

A Sample test skeleton project is provided which you can copy to provide a starting point.

See the Sming /tests/HostTests test application for a more comprehensive example.

  • A test module contains all code related to testing a specific Component, Library or framework module

  • If a module contains more than one file then place all files in a sub-directory

  • Each module must have a registration function:

  • Each module contains one or more test groups:

    • Each group is a class inheriting from TestGroup

    • Add a call to registerGroup() to the registration function for each test class

  • Keep each group brief. Use multiple, simpler groups if necessary.

  • The group TestGroup::execute() method is called to run the tests. On return the group is considered to have been successfully completed and destroyed.

  • Call TEST_ASSERT at appropriate points; passing false will fail the test and abort the process, displaying the source location of the failure.

  • If a test fails then additional details may be shown before calling TEST_ASSERT(false).

  • For asynchronous testing calling TestGroup::pending() before returning. When the tests have been completed call TestGroup::complete(). (See /tests/HostTests/modules/Timers.cpp for an example.)

The following macros are added for ease of importing tests from other frameworks:

What happens

The registerGroup() function creates a factory function which is added to the SmingTest::Runner::groupFactories list.

The test runner creates, executes and destroys each group in turn, and deals with scheduling.

Notes

Tests are run with DEBUG_VERBOSE_LEVEL at WARNING level, so debug_i statements will not normally be shown. Tests can use other debug_X functions as required, or Serial print methods.

Tests should compile and run for all architectures.

API Documentation

Defines

REGISTER_TEST(name)

Provides consistent global name for test factory function.

Test modules should use this macro to implement factory function:

#include <SmingTest.h>

class SampleTest: public TestGroup {
   ...
};

void REGISTER_TEST(sample)
{
   registerGroup<SampleTest>();
}
Parameters:

  • name – Name of test

Functions

template<class GroupClass>
void registerGroup()

Register a factory function (a lambda) to create a given TestGroup class.

Template Parameters:

Class – to be registered

namespace SmingTest

Typedefs

typedef TestGroup *(*Factory)()

Factory function to create a TestGroup class.

using Callback = Delegate<void()>

Variables

Runner runner
class Runner
#include <SmingTest.h>

Public Functions

Runner()
inline void setGroupIntervalMs(unsigned ms)
inline void addFactory(Factory factory)
void execute(Callback onComplete, unsigned initialDelayMs = 0)
void groupComplete(TestGroup *group)

Protected Functions

void runNextGroup()

Private Types

enum class State

Values:

enumerator stopped
enumerator waiting

In between tests.

enumerator running

Private Members

Vector<Factory> groupFactories

List of registered class factories.

Timer taskTimer
unsigned taskIndex = {0}
State state = {State::stopped}
NanoTime::Time<uint32_t> totalTestTime
unsigned testCount = {0}
unsigned failureCount = {0}
unsigned groupIntervalMs = {500}
Callback onComplete

Internal check macros

INTERNAL_CHECK(expr, verbose)
INTERNAL_CHECK2(res, expr, verbose)
INTERNAL_CHECK_EQ(a, b, verbose)
INTERNAL_CHECK_NEQ(a, b, verbose)

Check an expression, print message for success or failure (verbose)

CHECK(expr)

Check expression evaluates to true.

Parameters:

  • expr

CHECK2(res, expr)

Provide separate test result and expression.

Parameters:

  • res – Result of test

  • expr – Expression to display

CHECK_EQ(a, b)

Check two values are the same.

Parameters:

  • a

  • b

CHECK_NEQ(a, b)

Check two values are not the same.

Parameters:

  • a

  • b

Check an expression, but only print message on failure

REQUIRE(expr)

Check expression evaluates to true.

Parameters:

  • expr

REQUIRE2(res, expr)

Provide separate test result and expression.

Parameters:

  • res – Result of test

  • expr – Expression to display

REQUIRE_EQ(a, b)

Check two values are the same.

Parameters:

  • a

  • b

REQUIRE_NEQ(a, b)

Check two values are not the same.

Parameters:

  • a

  • b

Defines

TEST_ASSERT(result)

Check a test result.

Note

Failure generates an assertion so when run in the host emulator the process fails.

Parameters:

  • result – true if test was successful, false on failure

class TestBase
#include <TestBase.h>

Base class supporting verification for test assertions.

Subclassed by TestGroup

Public Functions

inline virtual ~TestBase()
virtual bool testVerify(bool res, const TestParam &param)

Print result of a test.

Parameters:

  • res – Result of the operation

  • param – Details of the test for display

Return values:

bool – Same as res

inline bool test_verify(bool res, const char *expr, const String &value1, const String &value2, bool verbose)
template<typename V>
inline std::enable_if<std::is_arithmetic<V>::value, bool>::type test_verify(bool res, const char *expr, const V &value1, const V &value2, bool verbose)
template<typename V>
inline std::enable_if<!std::is_same<V, String>::value && !std::is_arithmetic<V>::value, bool>::type test_verify(bool res, const char *expr, const V &value1, const V &value2, bool verbose)
inline virtual void fail(const char *func)
struct TestParam
#include <TestBase.h>

Contains details for test verification.

Public Members

const char *expr

Text of expression generated by the macro.

String value1

First value in comparison, or expected result.

String value2

Second value in comparison (optional)

bool verbose

true to always emit message, false only for errors

Defines

startTest(s)
TEST_CASE_1_ARG(name)

Start a test item.

Use like this:

TEST_CASE("My Test", "description") {
      ...
}

Note: Description is optional.

TEST_CASE_2_ARGS(name, desc)
GET_3RD_ARG(arg1, arg2, arg3, ...)
TEST_CASE_ARG_CHOOSER(...)
TEST_CASE(...)
class TestGroup : public TestBase
#include <TestGroup.h>

Class to simplify generation of begin/end messages for a test group.

Public Types

enum class State

Values:

enumerator running
enumerator pending
enumerator complete
enumerator failed

Public Functions

inline TestGroup(const String &name)
void commenceTest()
virtual void execute() = 0

Implement this method to define the test.

Note

If tests are asynchronous, call pending() before returning and call complete() when the group has completed execution (e.g. via timer callback, etc.)

void startItem(const String &tag, const String &description = nullptr)

Note the start of a test item within a group.

virtual void fail(const char *func) override

Called when test fails to identify location.

inline const String &getName()
inline NanoTime::Time<uint32_t> elapsedTime() const
inline State getState() const
void initialiseAndExecute()

Called by test runner.

Protected Functions

inline void pending()

Call to mark test as pending so it will be executed asynchronously Call complete() when test is finished.

void complete()

Call to complete pending (asynchronous) test.

Private Members

String name
State state = {State::running}
OneShotFastUs groupTimer
jmp_buf exception

References

SoC support

  • esp32

  • esp32c2

  • esp32c3

  • esp32s2

  • esp32s3

  • esp8266

  • host

  • rp2040

  • rp2350