Regression tests on ScicoLab

From ErikaWiki

Revision as of 14:07, 10 October 2010 by Paolo.gai (Talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Contents

Rationale

The number of blocks supported by the ScicosLab-FLEX package is constantly growing. For that reason, we designed a regression test mechanism similar to the testcase scripts available on ERIKA Enterprise to be able to always check that all blocks compiles without problems.

The idea is that before every release we will be able to test the compilation of a set of example diagrams, hopefully containing most of the supported blocks, ensuring in that way that the system at least compiles.

As a future work, we will also need to check the results of the execution of the various tests (but this is more difficult since it involves the target).

How to use it

Preparing the environment

  • install the following packages in your Cygwin installation
    • procmail
    • gcc
  • do a checkout from the ERIKA svn using the command
    • svn+ssh://TUXFAMILY_USERNAME@svn.tuxfamily.org/svnroot/erika/erikae/repos/scilab_codegen/trunk/testcase
  • move the testcase directory in the installation dir of ScicosLab
    • on my PC, it is under c:\Evidence\scicoslab-44b7

Running the tests

  • open a Cygwin XTerm window
  • cd in the testcase directory
  • type "make". the system will ask to export some environment variables. Each message ends with printing a command, which is typically the right one to type :-)
  • when done, type again make. You will get some help message. Again, the last line is the command to type.
  • type "make ARCH=flex" to compile the examples for the FLEX boards.

The following is a report of what happens on my machine (only one example is present). Note: it may take a while to compile!!!

pj@tom ~
$ cd /cygdrive/c/Evidence/scicoslab-44b7/testcase/

pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase
$ make
ERROR: Missing environment variable SCIBASE
...maybe you can try with the following command:
export SCIBASE=/cygdrive/c/Evidence/scicoslab-44b7/testcase/..

pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase
$ export SCIBASE=/cygdrive/c/Evidence/scicoslab-44b7/testcase/..

pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase
$ make
---------------------------------------------------
ScicosLab testcases
---------------------------------------------------

This testcase will compile all the ScicosLab tests
for all the configuration with a name starting with
a given prefix.

usage:

make ARCH=prefix
make clean
make supported ARCH=prefix

where 

- "DIRS" are the list of examples you want to test
- "DISTCC=distcc" if not set in your bash_profile, 
  allows to run tests using distcc

- "prefix" is one of:

  - "flex" - ScicosLab FLEX codegen
  ... and others

- the "clean" rule removes all the */out* directories

- the "supported" rule prints for each testcase if the particular ARCHitecture
  is supported

Examples:

make ARCH=flex

pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase
$ make ARCH=flex
RM tmp directory
EXPERIMENT flex_led/ BOARD=BOARDFLEX
CLEAN flex_led/ BOARD=BOARDFLEX
CODEGEN flex_led/ BOARD=BOARDFLEX
COMPILE flex_led/ BOARD=BOARDFLEX
ALL ended successfully!

pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase
$ 

Analyzing the results

  • The testcase directory contains a set of subdirectory, each one containing a test.cos file
  • the test.cos file can be compiled in various ways. each time the test.cos is compiled, a directory is created
  • each created directory is inside the directory containing the original test.cos file

Example:

  • there is a "flex_led" directory
  • inside the "flex_led" directory another directory is created, named "out_flex_BOARD_BOARDFLEX" (for the naming of the directory see below)
  • the "out_flex_BOARD_BOARDFLEX" directory contains a copy of teh original .cos file, plus the result of the code generation as it was done from the graphical interface. in our case, we have a "test_scig" subdirectory containing the code generated by ScicosLab for the "test.cos" file.
  • the "test_scig" directory contains the generated code, compiled using the dsPIC toolchain.
  • as a result of the compilation a "Debug" directory is created inside "test_scig"
  • all the output of the various phases are stored in two files into the "out_flex_BOARD_BOARDFLEX" directory. In particular:
    • scicoslab_log.txt contains the log of the commandline execution of ScicosLab
    • compile_log.txt contains the log of the compilation of the generated application




How to add a new test

Personal tools