Regression tests on ScicoLab
From ErikaWiki
Contents |
Rationale
The number of blocks supported by the ScicosLab-FLEX package is constantly growing. For that reason, we designed a regression test mechanism similar to the testcase scripts available on ERIKA Enterprise to be able to always check that all blocks compiles without problems.
The idea is that before every release we will be able to test the compilation of a set of example diagrams, hopefully containing most of the supported blocks, ensuring in that way that the system at least compiles.
As a future work, we will also need to check the results of the execution of the various tests (but this is more difficult since it involves the target).
How to use it
Preparing the environment
- install the following packages in your Cygwin installation
- procmail
- gcc
- do a checkout from the ERIKA svn using the command
- svn+ssh://TUXFAMILY_USERNAME@svn.tuxfamily.org/svnroot/erika/erikae/repos/scilab_codegen/trunk/testcase
- move the testcase directory in the installation dir of ScicosLab
- on my PC, it is under c:\Evidence\scicoslab-44b7
Running the tests
- open a Cygwin XTerm window
- cd in the testcase directory
- type "make". the system will ask to export some environment variables. Each message ends with printing a command, which is typically the right one to type :-)
- when done, type again make. You will get some help message. Again, the last line is the command to type.
- type "make ARCH=flex" to compile the examples for the FLEX boards.
The following is a report of what happens on my machine (only one example is present). Note: it may take a while to compile!!!
pj@tom ~ $ cd /cygdrive/c/Evidence/scicoslab-44b7/testcase/ pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase $ make ERROR: Missing environment variable SCIBASE ...maybe you can try with the following command: export SCIBASE=/cygdrive/c/Evidence/scicoslab-44b7/testcase/.. pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase $ export SCIBASE=/cygdrive/c/Evidence/scicoslab-44b7/testcase/.. pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase $ make --------------------------------------------------- ScicosLab testcases --------------------------------------------------- This testcase will compile all the ScicosLab tests for all the configuration with a name starting with a given prefix. usage: make ARCH=prefix make clean make supported ARCH=prefix where - "DIRS" are the list of examples you want to test - "DISTCC=distcc" if not set in your bash_profile, allows to run tests using distcc - "prefix" is one of: - "flex" - ScicosLab FLEX codegen ... and others - the "clean" rule removes all the */out* directories - the "supported" rule prints for each testcase if the particular ARCHitecture is supported Examples: make ARCH=flex pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase $ make ARCH=flex RM tmp directory EXPERIMENT flex_led/ BOARD=BOARDFLEX CLEAN flex_led/ BOARD=BOARDFLEX CODEGEN flex_led/ BOARD=BOARDFLEX COMPILE flex_led/ BOARD=BOARDFLEX ALL ended successfully! pj@tom /cygdrive/c/Evidence/scicoslab-44b7/testcase $
Analyzing the results
- The testcase directory contains a set of subdirectory, each one containing a test.cos file
- the test.cos file can be compiled in various ways. each time the test.cos is compiled, a directory is created
- each created directory is inside the directory containing the original test.cos file
Example:
- there is a "flex_led" directory
- inside the "flex_led" directory another directory is created, named "out_flex_BOARD_BOARDFLEX" (for the naming of the directory see below)
- the "out_flex_BOARD_BOARDFLEX" directory contains a copy of teh original .cos file, plus the result of the code generation as it was done from the graphical interface. in our case, we have a "test_scig" subdirectory containing the code generated by ScicosLab for the "test.cos" file.
- the "test_scig" directory contains the generated code, compiled using the dsPIC toolchain.
- as a result of the compilation a "Debug" directory is created inside "test_scig"
- all the output of the various phases are stored in two files into the "out_flex_BOARD_BOARDFLEX" directory. In particular:
- scicoslab_log.txt contains the log of the commandline execution of ScicosLab
- compile_log.txt contains the log of the compilation of the generated application