Regression tests on ScicoLab
I would like to thank Ramine Nikoukhah, Roberto Bucher, Simone Mannori, and Dario Di Stefano for their help in setting up this environment. Without them, I simply couldn't do that. PJ
The number of blocks supported by the ScicosLab-FLEX package is constantly growing. For that reason, we designed a regression test mechanism similar to the testcase scripts available on ERIKA Enterprise to be able to always check that all blocks compiles without problems.
The idea is that before every release we will be able to test the compilation of a set of example diagrams, hopefully containing most of the supported blocks, ensuring in that way that the system at least compiles.
As a future work, we will also need to check the results of the execution of the various tests (but this is more difficult since it involves the target).
How to use it
Preparing the environment
- install the following packages in your Cygwin installation
- do a checkout from the ERIKA svn using the command
- move the testcase directory in the installation dir of ScicosLab
- on my PC, it is under c:\Evidence\scicoslab-44b7
Running the tests
- open a Cygwin XTerm window
- cd in the testcase directory
- type "make". the system will ask to export some environment variables. Each message ends with printing a command, which is typically the right one to type :-)
- when done, type again make. You will get some help message. Again, the last line is the command to type.
- type "make ARCH=flex" to compile the examples for the FLEX boards.
The following is a report of what happens on my machine (only one example is present). Note: it may take a while to compile!!!
pj@tom ~ $ cd /cygdrive/c/Evidence/scicoslab-44b7/testcase/ pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase $ make ERROR: Missing environment variable SCIBASE ...maybe you can try with the following command: export SCIBASE=/cygdrive/c/Evidence/scicoslab-44b7/testcase/.. pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase $ export SCIBASE=/cygdrive/c/Evidence/scicoslab-44b7/testcase/.. pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase $ make --------------------------------------------------- ScicosLab testcases --------------------------------------------------- This testcase will compile all the ScicosLab tests for all the configuration with a name starting with a given prefix. usage: make ARCH=prefix make clean make supported ARCH=prefix where - "DIRS" are the list of examples you want to test - "DISTCC=distcc" if not set in your bash_profile, allows to run tests using distcc - "prefix" is one of: - "flex" - ScicosLab FLEX codegen ... and others - the "clean" rule removes all the */out* directories - the "supported" rule prints for each testcase if the particular ARCHitecture is supported Examples: make ARCH=flex pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase $ make ARCH=flex RM tmp directory EXPERIMENT flex_led/ BOARD=BOARDFLEX CLEAN flex_led/ BOARD=BOARDFLEX CODEGEN flex_led/ BOARD=BOARDFLEX COMPILE flex_led/ BOARD=BOARDFLEX ALL ended successfully! pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase $
Analyzing the results
- The testcase directory contains a set of subdirectory, each one containing a test.cos file
- the test.cos file can be compiled in various ways. each time the test.cos is compiled, a directory is created
- each created directory is inside the directory containing the original test.cos file
- there is a "flex_led" directory
- inside the "flex_led" directory another directory is created, named "out_flex_BOARD_BOARDFLEX" (for the naming of the directory see below)
- the "out_flex_BOARD_BOARDFLEX" directory contains a copy of teh original .cos file, plus the result of the code generation as it was done from the graphical interface. in our case, we have a "test_scig" subdirectory containing the code generated by ScicosLab for the "test.cos" file.
- the "test_scig" directory contains the generated code, compiled using the dsPIC toolchain.
- as a result of the compilation a "Debug" directory is created inside "test_scig"
- all the output of the various phases are stored in two files into the "out_flex_BOARD_BOARDFLEX" directory. In particular:
- scicoslab_log.txt contains the log of the commandline execution of ScicosLab
- compile_log.txt contains the log of the compilation of the generated application
The analyze target
The analyze target can be used to make an analysis of the coverage of the tests. Just run
make analyze ARCH=flex
And you will get the following results:
- tmp/globalblocklist.txt, which contains the pair (test_directory, block_contained), that lists the blocks used in each example
- tmp/globalpalettelist.txt, which contains the pair (palette, block_contained), that lists the blocks contained in each palette added by the Scicos-FLEX project
- tmp/globalcoverage.txt, which contains a text report with, for each block, the examples and the palettes containing the specific block. This file is the most important because it ensures that each block has been compiled at least once.
The Hudson target
This target is used during the integration with Hudson.
The idea is the following:
- the examples are compiled in different way, depending on their conf.in file
- on the same directory, the various architectures are built by using, for each architecture,
- at the end, the various tests compiled will be all there
- all the compilations done correctly will produce a doneflag.txt file
- then we run
- the hudson target will check the existence of all the doneflag.txt files generated by all possible compilations, plus it will check that all blocks has at least an example including it
- the hudson target will generate an XML file in tmp/results.xml containing the results of the checks
- if a block should be ignored, it should be listed into hudson_ignoreblocks.txt
How to add a new test
Adding a new test is quite simple. You need to do the following:
Create the directory
- each directory must contain only one .cof file.
- the name of the .cof file must be test.cos
Create and tag the design
- the test.cos must have a superblock tagged with the ID "cg" ("c"ode "g"enerator)
- to tag the superblock you need to do the following:
- click on the superblock to select it
- right click on the superblock. A pop-up menu appears
- select Block Properties --> Identification
- a window titled "Set Block Identification" appears, with a textbox named "ID"
- enter "cg" into the textbox.
- press Ok
- save the design
- to tag the superblock you need to do the following:
We need to tag each test.cos! The rationale is thate ach design may contain more than one superblock. For that reason, the script needs to know exactly which is the block to be generated.
Make the conf.in file
You need to prepare a "conf.in" file to be put in the test directory.
The quick way: USE THE FOLLOWING CONTENT FOR THE conf.in FILE
conf = BOARD=BOARDFLEX worksfor = flex
The following gives the explanation of the content of each line.
Basically, the two lines stands for:
- worksfor: is the toolchain used (FLEX, RTAI, ...)
- conf: is the specific template/configuration used (example: the FLEX board can be compiled on different boards with different "templates")
the conf line
- contains a set of values which are used to compile the test.
- the test is compiled with a combination of all the specified values
- do not use the underscore "_" inside the values.
conf = CONF=B1,B2,E1,E2 STATUS=STD,EXT
Means that 8 tests will be done:
- CONF=B1 STATUS=STD
- CONF=B2 STATUS=STD
- CONF=E1 STATUS=STD
- CONF=E2 STATUS=STD
- CONF=B1 STATUS=EXT
- CONF=B2 STATUS=EXT
- CONF=E1 STATUS=EXT
- CONF=E2 STATUS=EXT
Also means that 8 subdirectories will be created, named
with the respective configuration inside.
For the ScicosLab tests we currently use only one variable, "BOARD".
The typical value for BOARD is BOARDFLEX, resulting in:
conf = BOARD=BOARDFLEX
for most examples.
If in the future we find that an example can work for more than one type of hardware, then the line will be like the following, meaning that the same design is compiled for more than one configuration:
conf = BOARD=BOARDFLEX,BOARDXXX,RTAI,RTAI_GEA
Remember, no underscores in the names above!
In order for the various boards to work, you will need to modify the file testcase/common/flex_codegen/run_compile.sh
the worksfor line
The idea is that the regression system will work for various toolchains. examples:
and each toolchain may have its own scripts to make sure the compilation works correctly.
Currently, we support only one toolchain, "flex". for taht reason, the line will look like:
worksfor = flex
If a demo can be compiled with more than one toolchain, we need to separate the toolchains by spaces. Example:
worksfor = flex rtai
The list of supported toolchains can be obtained by typing "make" in the "testcase" directory.
The toolchains are then specified in files named "test.mk" under subdirectories of the testcase/common directory. For example, look at testcase/common/flex_codegen/test.mk
Fill the directory
- put the "test.cos" in the directory
- put the "conf.in" in the directory
The script automatically recognizes the presence of a test by the fact that the "conf.in" file exists.