How to Use SETTE
This tutorial will demonstrate how to use SETTE. First, please download and build SETTE, its dependencies and the test generator tools (see here). After you are ready, open a terminal and switch to SETTE's directory (if you followed the tutorial, it is sette-tool). Make sure that the sette.jar exists. If not, you have to build it with the ./build.sh command.
Before executing SETTE, the following artifacts must be provided:
- Code snippets (for example our code snippets)
- The executable of the tool you would like to use
- Appropriate configuration in the
sette.propertiesfile
The following figure demonstrates the main phases of SETTE. These are the phases of automatic evaluation:
-
generator: since almost all tools require a customized environment to execution, a runner project shall be generated for each of them. This generation includes the copy of the corresponding source files and libraries, removing any dependency to SETTE (like SETTE annotations) from them and creating the files required by the tool (usually test-drivers and configuration files). -
runner: this is the most time-consuming phase. It runs the tool on all of the code snippets separately in another process, thus SETTE will not affect the behaviour of the test input generator. All the raw outputs (standard and error output) are recorded during this phase. -
parser: parses the raw outputs (which were created in the latter phase) to a common XML format. During this phase it is also decided, whether the tool was terminated successfully, or it was not able to start (N/A), exited with an exception (EX) or reached the timeout (T/M). -
tests-generator: generates a JUnit 3 test suite from the parsed output to make the code coverage analysis available. -
tests-run: runs all the tests with the generated inputs and calculates coverage for all the executions using JaCoCo.
To start SETTE, run the ./sette.sh command. It will display its menu and write information about the tools. If you see an error message, probably the sette.properties configuration file is not correct.
Starting the snippet-browser scenario a GUI window is opened which displays the categorized code snippets and their basic information.
To evaluate a tool, you have to execute the formerly mentioned 5 phases respectively. After you select a scenario, you are asked for which tool you would like to use. After executing all the 5 phases, you can export the results to a CSV format and use it for further analysis.
Please note that SETTE is in development phase, thus it sometimes display extra messages. If the application encounters an error, it stops, so other messages should be ignored by the user. If you have any problem with using SETTE, please do not hesitate to report an issue or contact us.