Chapter 17
Testing

The CGAL test suite is a way to test the compilation and execution of CGAL programs automatically (i.e., without user interaction) on a number of different platforms. Developers should, of course, thoroughly test their code on their own development platform(s) before submitting it. The test suite serves as a way to test on additional platforms not available to the developer.

17.1   What a test suite for a package should contain

The test suite helps the developer(s) of a package to

• detect compilation problems on the various platforms
• detect runtime problems
• check the correctness of the algorithms in the package

That does not mean that the test suite is a platform for initial testing of code. New code should be tested on different platforms by the developer before submission.

It is strongly recommended for a test suite of a package to

• Cover the complete code of the package; every (member) function should be called at least once. (See Section 17.2 for a description of a tool you can use to test for code coverage.)
• Use more than one instantiation of templated functions or classes.
• A lot of classes in CGAL can be parametrized by traits classes, so that they are usable with different kernels. In such cases more than one kernel should be used for testing.
• Use pre- and postcondition checkers wherever it is possible.

17.2   Using the code coverage tool gcov

The tool gcov can be used together with the GNU C++ compiler to test for code coverage in your programs and may be helpful when you create your CGAL test suite programs. You can find a complete guide to this tool in the GNU on-line documentation at http://gcc.gnu.org/onlinedocs/gcc-2.95.2/gcc_6.html. If you want to use the code coverage tool gcov, you have to compile your programs with the options -fprofile-arcs and -ftest-coverage. Here is a simple example:

   #include<iostream>

   using namespace std;

   void fu(int val)
   {
    int w,v=0;
    if (val==0) {
     cout << "val == 0!\n";
     for(w=0;w<100;w++) v=v+w;
    }
    else {
     cout << "val != 0!\n";
     for(w=0;w<10;w++) v=v+w;  
    }
 
    cout << "v:" << v << "\n";
   }

   int main()
   {
     fu(0);
     return 0;
   }

   g++ -fprofile-arcs -ftest-coverage -o test test.C
   test
   gcov test.C  

                #include<iostream>
                
                using namespace std;
                
                void fu(int val)
           1    {
           1     int w,v=0;
           1     if (val==0) {
           1      cout << "val == 0!\n";
           1      for(w=0;w<100;w++) v=v+w;
                 }
      ######     else {
      ######      cout << "val != 0!\n";
      ######      for(w=0;w<10;w++) v=v+w;  
                }
                 
           1     cout << "v:" << v << "\n";
                }
                
                int main()
           1    {
           1      fu(0);
           1      return 0;
                }

17.3   Test suite directory

The test suite is located in the directory test of the internal releases of CGAL. This directory is not part of external releases. The directory test contains:

• a script run_testsuite that is (not surprisingly) used to run the test suite.
• a subdirectory for every package included in the internal release. These subdirectories are created from the test directories of the packages by copying the source, include, and input files from these directories and adding makefiles and cgal_test scripts where needed. See Section 4.1 for more information about the proper structure of the test directory for a package.
• a subdirectory with a name that ends in _Examples for every package that was submitted with an examples directory (Section 4.3)
• a subdirectory with a name that ends in _Demo for every package that was submitted with a demo directory (Section 4.4)

The test suite will attempt to compile all the programs in the subdirectories of test and to run all except the demo programs (which usually require user interaction) by using the cgal_test scripts (Sections 4.1 and 5.5) and will save the results in files in the package subdirectories (Section 17.6). Even if a program fails to compile or run, the test suite will continue.

17.4   Test suite input

Input to programs in the test suite can be supplied in three different ways:

data files in the data directory

As described in Section 4.1, a package's test directory may contain a subdirectory data that contains input files for the test programs.

*.cin files

If a test program program.C requires input from standard input (i.e., cin), you should put a file called program.cin in the test directory. The test suite will then execute the program using the command

         ./program < program.cin
         

command-line arguments supplied in the cgal_test script

You are discouraged from using this option to give input values to your programs since it requires you to edit and submit a cgal_test script; see Section 5.5.

However, if a test program program.C absolutely requires command-line parameters, you should do the following. Use create_cgal_test to create the script cgal_test. This file contains an entry of the form

         compile_and_run program 
         

Just put the command-line parameters for program at the end of this line:

         compile_and_run program  arg1 arg2 ..
         

The test suite will then execute the program using the command

         ./program <arg1> <arg2> ...
         

17.5   Running the test suite

The test suite is run using the run_testsuite script that is distributed with every internal release in the test directory. There are several ways you can customize this script to meet you needs:

• Add additional compiler and linker flags by setting the variables TESTSUITE_CXXFLAGS and TESTSUITE_LDFLAGS at the top of the script. These variables are prepended to CXX_FLAGS and LDFLAGS, respectively, in the test suite makefiles. So, for example, if you have a directory experimental/include/CGAL containing new or experimental CGAL files, you can do the following: TESTSUITE_CXXFLAGS="-Iexperimental/include" and in this way test with your new files without overwriting the originals.

• Export additional environment variables by ading lines to the run_testsuite script. As an example, it will be demonstrated how to export the LD_LIBRARY_PATH by editing run_testsuite.
  1. Add the line LD_LIBRARY_PATH=<your library path> to the script.
  2. Append LD_LIBRARY_PATH to the line export PLATFORM CGAL_MAKEFILE TESTSUITE_CXXFLAGS TESTSUITE_LDFLAGS in the script.

  After this, the programs from the test suite will be run using the LD_LIBRARY_PATH that was specified in step 1.

• Run the test suite on more than one platform by adding a line at the bottom of the script of the form

             run_testsuite <include makefile>
        

  for every platform that you wish to test. Just substitute for <include makefile> the appropriate include makefiles that were generated during installation. (Don't forgot to use the full path name for the makefile!) By default, the last line in the file is

             run_testsuite $CGAL_MAKEFILE
        

  so you need not make any changes if you run the testsuite on only one platform and have set the CGAL_MAKEFILE environment variable properly.

After these steps you are ready to run the test suite. It can be run in two different ways:

./run_testsuite

The test suite will run the tests from all test directories. This may take a considerable amount of time.

./run_testsuite <dir1> <dir2> ...

The test suite will run only the test programs in the test directories

<dir1> <dir2> ... 

To run an entire CGAL test suite automatically, including downloading of an internal release, configuration, and installation of the library, you can use the autotest_cgal script described in Section 5.6.

17.6   Files generated by the test suite

The testsuite will generate the following output files:

• <testdir>/ErrorOutput_<platform>

  This file contains two lines for every program that was tested on platform <platform> in the test directory <testdir>. The first line tells if the compilation was successful and the second line tells if the execution was successful (i.e., the program returned the value 0). (See Section 4.1 for more details.)

• <testdir>/ProgramOutput.<program>.<platform>

  This file contains the console output from the test program <program.C> run on platform <platform>.

• <testdir>/CompilerOutput_<platform>

  This file contains the compiler output from platform <platform> for all programs.

• error.txt

  This is just a concatenation of all the ErrorOutput files that were generated during the last run of the test suite.

17.7   Test suite results

17.8   Requirements and recommendations

Requirements:

• Test your code thoroughly before submitting it.
• Obey the directory structure detailed in Section 4.1
• Check the test suite results for your package regularly.

Recommendations:

• Write test suite programs that use more than one instantiation of templated functions and classes, call every member function at least once, and use more than one kernel.
• Use pre- and postcondition checkers.
• Use gcov to test your code for coverage.
• Don't submit a makefile for your test suite unless you need to do something very special to compile or link your program. If you find you want to do something very special in your makefile, think long and hard about whether it's really necessary or not.
• Don't submit the script cgal_test with your package.