Black Box Regression tests

From ErikaWiki

Revision as of 12:42, 31 October 2011 by Paolo.gai (Talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

This page documents about the Black Box regression tests implemented for ERIKA Enterprise.

Main idea

The main idea is expressed by the following points:

  • the system allows the specification of a set of tests
    • each test is a black box test
      • each test contains a set of assertions
      • each assertion checks both the execution order as well as return values of functions
      • the test is successful when the test executed in the proper order
  • the tests are compiled for a target platform
    • for each test is possible to specify for which architecture it works (variable worksfor in conf.in)
    • for each test it is possible to specify a set of values for the testing
      • typical example: a test must work for BCC1, BCC2, ECC1, ECC2, both standard and extended error checking
      • the system will test all the possible combination of all the variables (in the case above, they will be 8 combinations)
      • please refer the variable conf in conf.in
  • the test generates a debugger configuration (e.g. for Lauterbach)
    • the debugger configuration is responsible to run all tests
    • each run does the following:
      • loads the application
      • sets a breakpoint in a specific position (typically EE_assert_last
      • runs the application
      • if the application does not stop, then it is a failure :-)
      • if the application stops, then a variable is tested. If it is ok, it means that all the execution of all the assertions are correct
    • all the debugger output is returned and stored in a log file.
  • the tests currently implement the MODISTARC tests, plus some other tests to check the behavior of the main context change and interrupt functions.

Important files

  • all files are stored inside the testcase directory
  • the testcase/makefile if run without parameters gives an idea of the possible commands
  • each test is stored in a separate directory
    • conf.in which contains the settings that explains how to run the test
      • note that the conf.in settings are handled as #defines in the OIL and.c file
    • code.c is the test source code
    • appl.oil is the OIL configuration file
  • common
    • contains all the scritps
    • important things to look at are the subdirectories, one for each architecture, and in particular the test.mk which contains information of the commands run at each phase.

More info

Some more information can be found at Running the MODISTARC regression tests.

Personal tools