From ErikaWiki

Latest revision as of 13:53, 12 September 2011

Acknowledgements

I would like to thank Ramine Nikoukhah, Roberto Bucher, Simone Mannori, and Dario Di Stefano for their help in setting up this environment. Without them, I simply couldn't do that. PJ

Rationale

The number of blocks supported by the ScicosLab-FLEX package is constantly growing. For that reason, we designed a regression test mechanism similar to the testcase scripts available on ERIKA Enterprise to be able to always check that all blocks compiles without problems.

The idea is that before every release we will be able to test the compilation of a set of example diagrams, hopefully containing most of the supported blocks, ensuring in that way that the system at least compiles.

As a future work, we will also need to check the results of the execution of the various tests (but this is more difficult since it involves the target).

How to use it

Preparing the environment

• install the following packages in your Cygwin installation
  • procmail
  • gcc
• do a checkout from the ERIKA svn using the command
  • svn+ssh://TUXFAMILY_USERNAME@svn.tuxfamily.org/svnroot/erika/erikae/repos/scilab_codegen/trunk/testcase
• move the testcase directory in the installation dir of ScicosLab
  • on my PC, it is under c:\Evidence\scicoslab-44b7

Running the tests

• open a Cygwin XTerm window
• cd in the testcase directory
• type "make". the system will ask to export some environment variables. Each message ends with printing a command, which is typically the right one to type :-)
• when done, type again make. You will get some help message. Again, the last line is the command to type.
• type "make ARCH=flex" to compile the examples for the FLEX boards.

The following is a report of what happens on my machine (only one example is present). Note: it may take a while to compile!!!

pj@tom ~
$ cd /cygdrive/c/Evidence/scicoslab-44b7/testcase/

pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase
$ make
ERROR: Missing environment variable SCIBASE
...maybe you can try with the following command:
export SCIBASE=/cygdrive/c/Evidence/scicoslab-44b7/testcase/..

pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase
$ export SCIBASE=/cygdrive/c/Evidence/scicoslab-44b7/testcase/..

pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase
$ make
---------------------------------------------------
ScicosLab testcases
---------------------------------------------------

This testcase will compile all the ScicosLab tests
for all the configuration with a name starting with
a given prefix.

usage:

make ARCH=prefix
make clean
make supported ARCH=prefix

where 

- "DIRS" are the list of examples you want to test
- "DISTCC=distcc" if not set in your bash_profile, 
  allows to run tests using distcc

- "prefix" is one of:

  - "flex" - ScicosLab FLEX codegen
  ... and others

- the "clean" rule removes all the */out* directories

- the "supported" rule prints for each testcase if the particular ARCHitecture
  is supported

Examples:

make ARCH=flex

pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase
$ make ARCH=flex
RM tmp directory
EXPERIMENT flex_led/ BOARD=BOARDFLEX
CLEAN flex_led/ BOARD=BOARDFLEX
CODEGEN flex_led/ BOARD=BOARDFLEX
COMPILE flex_led/ BOARD=BOARDFLEX
ALL ended successfully!

pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase
$ 

Analyzing the results

• The testcase directory contains a set of subdirectory, each one containing a test.cos file
• the test.cos file can be compiled in various ways. each time the test.cos is compiled, a directory is created
• each created directory is inside the directory containing the original test.cos file

Example:

• there is a "flex_led" directory
• inside the "flex_led" directory another directory is created, named "out_flex_BOARD_BOARDFLEX" (for the naming of the directory see below)
• the "out_flex_BOARD_BOARDFLEX" directory contains a copy of teh original .cos file, plus the result of the code generation as it was done from the graphical interface. in our case, we have a "test_scig" subdirectory containing the code generated by ScicosLab for the "test.cos" file.
• the "test_scig" directory contains the generated code, compiled using the dsPIC toolchain.
• as a result of the compilation a "Debug" directory is created inside "test_scig"

• all the output of the various phases are stored in two files into the "out_flex_BOARD_BOARDFLEX" directory. In particular:
  • scicoslab_log.txt contains the log of the commandline execution of ScicosLab
  • compile_log.txt contains the log of the compilation of the generated application

The analyze target

The analyze target can be used to make an analysis of the coverage of the tests. Just run

make analyze ARCH=flex 

And you will get the following results:

• tmp/globalblocklist.txt, which contains the pair (test_directory, block_contained), that lists the blocks used in each example
• tmp/globalpalettelist.txt, which contains the pair (palette, block_contained), that lists the blocks contained in each palette added by the Scicos-FLEX project
• tmp/globalcoverage.txt, which contains a text report with, for each block, the examples and the palettes containing the specific block. This file is the most important because it ensures that each block has been compiled at least once.

The Hudson target

This target is used during the integration with Hudson.

The idea is the following:

• the examples are compiled in different way, depending on their conf.in file
• on the same directory, the various architectures are built by using, for each architecture,

make ARCH=xxx

• at the end, the various tests compiled will be all there
• all the compilations done correctly will produce a doneflag.txt file
• then we run

make hudson

• the hudson target will check the existence of all the doneflag.txt files generated by all possible compilations, plus it will check that all blocks has at least an example including it
• the hudson target will generate an XML file in tmp/results.xml containing the results of the checks
• if a block should be ignored, it should be listed into hudson_ignoreblocks.txt

How to add a new test

Adding a new test is quite simple. You need to do the following:

Create the directory

• each directory must contain only one .cof file.
• the name of the .cof file must be test.cos

Create and tag the design

• the test.cos must have a superblock tagged with the ID "cg" ("c"ode "g"enerator)
  • to tag the superblock you need to do the following:
    • click on the superblock to select it
    • right click on the superblock. A pop-up menu appears
    • select Block Properties --> Identification
    • a window titled "Set Block Identification" appears, with a textbox named "ID"
    • enter "cg" into the textbox.
    • press Ok
    • save the design

We need to tag each test.cos! The rationale is that each design may contain more than one superblock. For that reason, the script needs to know exactly which is the block to be generated.

Make the conf.in file

You need to prepare a "conf.in" file to be put in the test directory.

The quick way: USE THE FOLLOWING CONTENT FOR THE conf.in FILE

conf = BOARD=BOARDFLEX
worksfor = flex

The following gives the explanation of the content of each line.

Basically, the two lines stands for:

• worksfor: is the toolchain used (FLEX, RTAI, ...)
• conf: is the specific template/configuration used (example: the FLEX board can be compiled on different boards with different "templates")

the conf line

• contains a set of values which are used to compile the test.
• the test is compiled with a combination of all the specified values
• do not use the underscore "_" inside the values.

Example:

conf = CONF=B1,B2,E1,E2 STATUS=STD,EXT

Means that 8 tests will be done:

• CONF=B1 STATUS=STD
• CONF=B2 STATUS=STD
• CONF=E1 STATUS=STD
• CONF=E2 STATUS=STD
• CONF=B1 STATUS=EXT
• CONF=B2 STATUS=EXT
• CONF=E1 STATUS=EXT
• CONF=E2 STATUS=EXT

Also means that 8 subdirectories will be created, named

• out_CONF_B1_STATUS_STD
• out_CONF_B2_STATUS_STD
• out_CONF_E1_STATUS_STD
• out_CONF_E2_STATUS_STD
• out_CONF_B1_STATUS_EXT
• out_CONF_B2_STATUS_EXT
• out_CONF_E1_STATUS_EXT
• out_CONF_E2_STATUS_EXT

with the respective configuration inside.

For the ScicosLab tests we currently use only one variable, "BOARD".

The typical value for BOARD is BOARDFLEX, resulting in:

conf = BOARD=BOARDFLEX

for most examples.

If in the future we find that an example can work for more than one type of hardware, then the line will be like the following, meaning that the same design is compiled for more than one configuration:

conf = BOARD=BOARDFLEX,BOARDXXX,RTAI,RTAI_GEA

Remember, no underscores in the names above!

In order for the various boards to work, you will need to modify the file testcase/common/flex_codegen/run_compile.sh

the worksfor line

The idea is that the regression system will work for various toolchains. examples:

• RTAI
• Xenomai
• FLEX
• ...

and each toolchain may have its own scripts to make sure the compilation works correctly.

Currently, we support only one toolchain, "flex". for taht reason, the line will look like:

worksfor = flex

If a demo can be compiled with more than one toolchain, we need to separate the toolchains by spaces. Example:

worksfor = flex rtai

The list of supported toolchains can be obtained by typing "make" in the "testcase" directory.

The toolchains are then specified in files named "test.mk" under subdirectories of the testcase/common directory. For example, look at testcase/common/flex_codegen/test.mk

Fill the directory

• put the "test.cos" in the directory
• put the "conf.in" in the directory

The script automatically recognizes the presence of a test by the fact that the "conf.in" file exists.