Regression tests on ScicoLab

From ErikaWiki

(Difference between revisions)
Jump to: navigation, search
Line 141: Line 141:
You need to prepare a "conf.in" file to be put in the test directory.
You need to prepare a "conf.in" file to be put in the test directory.
-
The quick way: '''USE THE FOLLOWING CONTENT'''
+
'''The quick way: USE THE FOLLOWING CONTENT FOR THE conf.in FILE'''
<pre>
<pre>
conf = BOARD=BOARDFLEX
conf = BOARD=BOARDFLEX
worksfor = flex
worksfor = flex
</pre>
</pre>
 +
 +
The following gives the explanation of the content of each line.
 +
 +
Basically, the two lines stands for:
 +
* '''worksfor''': is the toolchain used (FLEX, RTAI, ...)
 +
* '''conf''': is the specific template/configuration used (example: the FLEX board can be compiled on different boards with different "templates")
=== the conf line ===
=== the conf line ===
Line 193: Line 199:
</pre>
</pre>
-
'''Remember, no underscores in the names!'''
+
'''Remember, no underscores in the names above!'''
In order for the various boards to work, you will need to modify the file testcase/common/flex_codegen/run_compile.sh
In order for the various boards to work, you will need to modify the file testcase/common/flex_codegen/run_compile.sh
 +
=== the worksfor line ===
 +
 +
The idea is that the regression system will work for various toolchains. examples:
 +
* RTAI
 +
* Xenomai
 +
* FLEX
 +
* ...
 +
 +
and each toolchain may have its own scripts to make sure the compilation works correctly.
 +
 +
Currently, we support only one toolchain, "flex". for taht reason, the line will look like:
 +
 +
<pre>
 +
worksfor = flex
 +
</pre>
 +
 +
If a demo can be compiled with more than one toolchain, we need to separate the toolchains by spaces. Example:
 +
 +
<pre>
 +
worksfor = flex rtai
 +
</pre>
 +
 +
The list of supported toolchains can be obtained by typing "make" in the "testcase" directory.
 +
 +
The toolchains are then specified in files named "test.mk" under subdirectories of the testcase/common directory. For example, look at testcase/common/flex_codegen/test.mk
== Fill the directory ==
== Fill the directory ==

Revision as of 14:39, 10 October 2010

Contents

Acknowledgements

I would like to thank Ramine Nikoukhah, Roberto Bucher, Simone Mannori, and Dario Di Stefano for their help in setting up this environment. Without them, I simply couldn't do that. PJ

Rationale

The number of blocks supported by the ScicosLab-FLEX package is constantly growing. For that reason, we designed a regression test mechanism similar to the testcase scripts available on ERIKA Enterprise to be able to always check that all blocks compiles without problems.

The idea is that before every release we will be able to test the compilation of a set of example diagrams, hopefully containing most of the supported blocks, ensuring in that way that the system at least compiles.

As a future work, we will also need to check the results of the execution of the various tests (but this is more difficult since it involves the target).

How to use it

Preparing the environment

  • install the following packages in your Cygwin installation
    • procmail
    • gcc
  • do a checkout from the ERIKA svn using the command
    • svn+ssh://TUXFAMILY_USERNAME@svn.tuxfamily.org/svnroot/erika/erikae/repos/scilab_codegen/trunk/testcase
  • move the testcase directory in the installation dir of ScicosLab
    • on my PC, it is under c:\Evidence\scicoslab-44b7

Running the tests

  • open a Cygwin XTerm window
  • cd in the testcase directory
  • type "make". the system will ask to export some environment variables. Each message ends with printing a command, which is typically the right one to type :-)
  • when done, type again make. You will get some help message. Again, the last line is the command to type.
  • type "make ARCH=flex" to compile the examples for the FLEX boards.

The following is a report of what happens on my machine (only one example is present). Note: it may take a while to compile!!!

pj@tom ~
$ cd /cygdrive/c/Evidence/scicoslab-44b7/testcase/

pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase
$ make
ERROR: Missing environment variable SCIBASE
...maybe you can try with the following command:
export SCIBASE=/cygdrive/c/Evidence/scicoslab-44b7/testcase/..

pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase
$ export SCIBASE=/cygdrive/c/Evidence/scicoslab-44b7/testcase/..

pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase
$ make
---------------------------------------------------
ScicosLab testcases
---------------------------------------------------

This testcase will compile all the ScicosLab tests
for all the configuration with a name starting with
a given prefix.

usage:

make ARCH=prefix
make clean
make supported ARCH=prefix

where 

- "DIRS" are the list of examples you want to test
- "DISTCC=distcc" if not set in your bash_profile, 
  allows to run tests using distcc

- "prefix" is one of:

  - "flex" - ScicosLab FLEX codegen
  ... and others

- the "clean" rule removes all the */out* directories

- the "supported" rule prints for each testcase if the particular ARCHitecture
  is supported

Examples:

make ARCH=flex

pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase
$ make ARCH=flex
RM tmp directory
EXPERIMENT flex_led/ BOARD=BOARDFLEX
CLEAN flex_led/ BOARD=BOARDFLEX
CODEGEN flex_led/ BOARD=BOARDFLEX
COMPILE flex_led/ BOARD=BOARDFLEX
ALL ended successfully!

pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase
$ 

Analyzing the results

  • The testcase directory contains a set of subdirectory, each one containing a test.cos file
  • the test.cos file can be compiled in various ways. each time the test.cos is compiled, a directory is created
  • each created directory is inside the directory containing the original test.cos file

Example:

  • there is a "flex_led" directory
  • inside the "flex_led" directory another directory is created, named "out_flex_BOARD_BOARDFLEX" (for the naming of the directory see below)
  • the "out_flex_BOARD_BOARDFLEX" directory contains a copy of teh original .cos file, plus the result of the code generation as it was done from the graphical interface. in our case, we have a "test_scig" subdirectory containing the code generated by ScicosLab for the "test.cos" file.
  • the "test_scig" directory contains the generated code, compiled using the dsPIC toolchain.
  • as a result of the compilation a "Debug" directory is created inside "test_scig"
  • all the output of the various phases are stored in two files into the "out_flex_BOARD_BOARDFLEX" directory. In particular:
    • scicoslab_log.txt contains the log of the commandline execution of ScicosLab
    • compile_log.txt contains the log of the compilation of the generated application


How to add a new test

Adding a new test is quite simple. You need to do the following:

Create the directory

  • each directory must contain only one .cof file.
  • the name of the .cof file must be test.cos

Create and tag the design

  • the test.cos must have a superblock tagged with the ID "cg" ("c"ode "g"enerator)
    • to tag the superblock you need to do the following:
      • click on the superblock to select it
      • right click on the superblock. A pop-up menu appears
      • select Block Properties --> Identification
      • a window titled "Set Block Identification" appears, with a textbox named "ID"
      • enter "cg" into the textbox.
      • press Ok
      • save the design

We need to tag each test.cos! The rationale is thate ach design may contain more than one superblock. For that reason, the script needs to know exactly which is the block to be generated.

Make the conf.in file

You need to prepare a "conf.in" file to be put in the test directory.

The quick way: USE THE FOLLOWING CONTENT FOR THE conf.in FILE

conf = BOARD=BOARDFLEX
worksfor = flex

The following gives the explanation of the content of each line.

Basically, the two lines stands for:

  • worksfor: is the toolchain used (FLEX, RTAI, ...)
  • conf: is the specific template/configuration used (example: the FLEX board can be compiled on different boards with different "templates")

the conf line

  • contains a set of values which are used to compile the test.
  • the test is compiled with a combination of all the specified values
  • do not use the underscore "_" inside the values.

Example:

conf = CONF=B1,B2,E1,E2 STATUS=STD,EXT

Means that 8 tests will be done:

  • CONF=B1 STATUS=STD
  • CONF=B2 STATUS=STD
  • CONF=E1 STATUS=STD
  • CONF=E2 STATUS=STD
  • CONF=B1 STATUS=EXT
  • CONF=B2 STATUS=EXT
  • CONF=E1 STATUS=EXT
  • CONF=E2 STATUS=EXT

Also means that 8 subdirectories will be created, named

  • out_CONF_B1_STATUS_STD
  • out_CONF_B2_STATUS_STD
  • out_CONF_E1_STATUS_STD
  • out_CONF_E2_STATUS_STD
  • out_CONF_B1_STATUS_EXT
  • out_CONF_B2_STATUS_EXT
  • out_CONF_E1_STATUS_EXT
  • out_CONF_E2_STATUS_EXT

with the respective configuration inside.

For the ScicosLab tests we currently use only one variable, "BOARD".

The typical value for BOARD is BOARDFLEX, resulting in:

conf = BOARD=BOARDFLEX

for most examples.

If in the future we find that an example can work for more than one type of hardware, then the line will be like the following, meaning that the same design is compiled for more than one configuration:

conf = BOARD=BOARDFLEX,BOARDXXX,RTAI,RTAI_GEA

Remember, no underscores in the names above!

In order for the various boards to work, you will need to modify the file testcase/common/flex_codegen/run_compile.sh

the worksfor line

The idea is that the regression system will work for various toolchains. examples:

  • RTAI
  • Xenomai
  • FLEX
  • ...

and each toolchain may have its own scripts to make sure the compilation works correctly.

Currently, we support only one toolchain, "flex". for taht reason, the line will look like:

worksfor = flex

If a demo can be compiled with more than one toolchain, we need to separate the toolchains by spaces. Example:

worksfor = flex rtai

The list of supported toolchains can be obtained by typing "make" in the "testcase" directory.

The toolchains are then specified in files named "test.mk" under subdirectories of the testcase/common directory. For example, look at testcase/common/flex_codegen/test.mk

Fill the directory

  • put the "test.cos" in the directory
  • put the "conf.in" in the directory

The script automatically recognizes the presence of a test by the fact that the "conf.in" file exists.

Personal tools