End to End Testing (Java)

jplag.cli/pom.xml

@@ -52,6 +52,10 @@
             <groupId>de.jplag</groupId>
             <artifactId>chars</artifactId>
         </dependency>
+        <dependency>
+            <groupId>de.jplag</groupId>
+            <artifactId>endToEndTesting</artifactId>
+        </dependency>
         <dependency>
             <groupId>net.sourceforge.argparse4j</groupId>
             <artifactId>argparse4j</artifactId>

jplag.endToEndTesting/README.md

@@ -0,0 +1,105 @@
+# JPlag - End To End Testing
+With the help of the end-to-end module, changes to the detection of JPlag are to be tested.
+With the help of elaborated plagiarisms, which have been worked out from suggestions in the literature on the topic of "plagiarism detection and avoidance", a wide range of detectable change can be covered. The selected plagiarisms are the decisive factor here as to whether a change in recognition can be perceived. 
+
+### References
+These elaborations provide basic ideas on how a modification of the plagiarized source code can look like or be adapted.
+These code adaptations refer to a wide range of changes starting from
+adding/removing comments to architectural changes in the deliverables.
+
+The following elaborations were used to be able to create the plagiarisms with the largest coverage:
+- [Mossad: defeating software plagiarism detection](https://dl.acm.org/doi/abs/10.1145/3428206 "Mossad: defeating software plagiarism detection")
+- [Detecting Source Code Plagiarism on Introductory Programming Course Assignments Using a Bytecode Approach - Oscar Karnalim](https://ieeexplore.ieee.org/abstract/document/7910274 "Detecting Source Code Plagiarism on Introductory Programming Course Assignments Using a Bytecode Approach - Oscar Karnalim")
+- [Detecting Disguised Plagiarism - Hatem A. Mahmoud](https://arxiv.org/abs/1711.02149 "Detecting Disguised Plagiarism - Hatem A. Mahmoud")
+- [Instructor-centric source code plagiarism detection and plagiarism corpus](https://dl.acm.org/doi/abs/10.1145/2325296.2325328 "Instructor-centric source code plagiarism detection and plagiarism corpus")
+
+### Steps Towards Plagiarism
+The following changes were applied to sample tasks to create test cases:
+<ul type="1">
+	<li>Inserting comments or empty lines (normalization level)</li>
+	<li>Changing variable names or function names (normalization level)</li>
+	<li>Insertion of unnecessary or changed code lines (token generation)</li>
+	<li>Changing the program flow (token generation) (statments and functions must be independent from each other)</li>
+		<ul>
+			<li>Variable decleration at the beginning of the program</li>
+			<li>Combining declerations of variables</li>
+			<li>Reuse of the same variable for other functions</li>
+		</ul>
+	<li>Changing control structures</li>
+		<ul>
+			<li>for(...) to while(...)</li>
+			<li>if(...) to switch-case</li>
+		</ul>
+	<li>Modification of expressions</li>
+		<ul>
+			<li>(X < Y) to !(X >= Y) and ++x to x = x + 1</li>
+		</ul>
+	<li>Splitting and merging statements</li>
+		<ul>
+			<li>x = getSomeValue(); y = x- z; to y = (getSomeValue() - Z;</li>
+		</ul>
+	<li>Inserting unnecessary casts</li>
+</ul>
+
+More detailed information about the create as well as about the subject to the issue can be found in the issue [Develop an end-to-end testing strategy](https://github.com/jplag/JPlag/issues/193 "Develop an end-to-end testing strategy").
+
+**The changes listed above have been developed and evaluated for purely scientific purposes and are not intended to be used for plagiarism in the public or private domain.**
+
+Software is according to [§ 2 of the copyright law](https://www.gesetze-im-internet.de/urhg/__2.html "§ 2 of the copyright law") a protected work which may not be plagiarized. 
+
+### JPlag - End To End TestSuite Structure
+The construction of an end to end test is done with the help of the JPlag api. 
+
+``` java 
+[...]
+    /**
+     * This method creates the necessary results as well as models for a test run and summarizes them for a comparison.
+     * @param testClassNames Plagiarized classes names in the resource directorie which are needed for the test
+     * @param testIdentifier name of the testId to load and identify the stored results
+     * @throws IOException is thrown in case of problems with copying the plagiarism classes
+     * @throws ExitException in case the plagiarism detection with JPlag is preemptively terminated would be of the test.
+     */
+    private void runJPlagTestSuite(String[] testClassNames, int testIdentifier) throws IOException, ExitException {
+        String functionName = StackWalker.getInstance().walk(stream -> stream.skip(1).findFirst().get()).getMethodName();
+        TestCaseModel testCaseModel = jplagTestSuiteHelper.createNewTestCase(testClassNames, functionName);
+        JPlagResult jplagResult = new JPlag(testCaseModel.getJPlagOptionsFromCurrentModel()).run();
+
+        for (JPlagComparison jPlagComparison : jplagResult.getAllComparisons()) {
+            assertEquals(testCaseModel.getCurrentJsonModel().getResultModelById(testIdentifier).getResultSimilarity(), jPlagComparison.similarity(),
+                    "The JPlag results [similarity] do not match the stored values!");
+        }
+    }
+[...]
+```
+The created plagiarisms are copied with the JPlagTestSuiteHelper into temporary directories, which can then be checked with JPlag. 
+
+In order to be able to distinguish in which domain of the recognition changes have occurred, fine granular test cases are used. These are composed of the changes already mentioned above. The plagiarism is compared with the original delivery and thus it is possible to detect and test small sections of the recognition. 
+
+The values compared so far to detect a match are limited to the similarity of the JPlag scan. It is already planned to expand the comparative values in order to have more indications of a change. 
+
+The Current Discussion is already taking place and can be found at [End to end testing - "comparative values"](https://github.com/jplag/JPlag/issues/548 "End to end testing - \"comparative values\""). 
+
+The current JPlag scans will be compared with the stored ones.
+This was done by storing the data in a *.json file which is read at the beginning of each test run. 
+
+``` json  
+{
+[...]
+    "function_name": "normalizationLevelTest",
+    "test_results": [
+      {
+        "result_similarity": 100,
+        "test_identifier": 0
+      },
+      {
+        "result_similarity": 100,
+        "test_identifier": 1
+      },
+      {
+        "result_similarity": 100,
+        "test_identifier": 2
+      }
+    ]
+  }
+[...]
+```

jplag.endToEndTesting/pom.xml

@@ -0,0 +1,29 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <parent>
+        <groupId>de.jplag</groupId>
+        <artifactId>aggregator</artifactId>
+        <version>${revision}</version>
+    </parent>
+    <artifactId>endToEndTesting</artifactId>
+
+    <dependencies>
+        <dependency>
+            <groupId>de.jplag</groupId>
+            <artifactId>jplag</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>de.jplag</groupId>
+            <artifactId>frontend-testutils</artifactId>
+            <version>${revision}</version>
+            <type>test-jar</type>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>com.fasterxml.jackson.core</groupId>
+            <artifactId>jackson-databind</artifactId>
+            <version>2.13.3</version>
+        </dependency>
+    </dependencies>
+</project>

jplag.endToEndTesting/src/main/java/de/jplag/end_to_end_testing/constants/TestDirectoryConstants.java

@@ -0,0 +1,32 @@
+package de.jplag.end_to_end_testing.constants;
+
+import java.nio.file.Path;
+
+/**
+ * All constant values that are needed in the test cases or helper classes.
+ */
+public final class TestDirectoryConstants {
+
+    private TestDirectoryConstants() {
+        // private constructor to prevent instantiation
+    }
+
+    /**
+     * Name for the folder to copy the submissions to in order to test them with JPlag
+     */
+    public static final String TEMPORARY_DIRECTORY_NAME = "testing-directory-submission";
+    /**
+     * Create the complete path to the submission files. Here the temporary system path is extended with the
+     * "TEMPORARY_DIRECTORY_NAME", which is predefined in this class.
+     */
+    public static final String TEMPORARY_SUBMISSION_DIRECTORY_NAME = Path.of("target", TEMPORARY_DIRECTORY_NAME).toString();
+
+    /**
+     * Base path to the created plagiarism and the main file located in the project resources.
+     */
+    public static final Path BASE_PATH_TO_JAVA_RESOURCES_SORTALGO = Path.of("src", "test", "resources", "java", "sortAlgo");
+    /**
+     * Base path to the saved results of the previous tests in a *.json file
+     */
+    public static final Path BASE_PATH_TO_JAVA_RESULT_JSON = Path.of("src", "test", "resources", "results", "JavaResult.json");
+}

jplag.endToEndTesting/src/main/java/de/jplag/end_to_end_testing/helper/JPlagTestSuiteHelper.java

@@ -0,0 +1,120 @@
+package de.jplag.end_to_end_testing.helper;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
+import java.util.Arrays;
+import java.util.List;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import de.jplag.end_to_end_testing.constants.TestDirectoryConstants;
+import de.jplag.end_to_end_testing.model.JsonModel;
+import de.jplag.end_to_end_testing.model.TestCaseModel;
+import de.jplag.options.LanguageOption;
+
+/**
+ * This helper class deals with creating the test cases as well as copying and deleting for the end-to-end tests. The
+ * required plagiarisms are copied from the resource folder to a temporary location, thus creating a folder structure
+ * that can be tested by JPlag. Models are instantiated here and required information for the tests is loaded.
+ */
+public class JPlagTestSuiteHelper {
+
+    private static final Logger logger = LoggerFactory.getLogger(JPlagTestSuiteHelper.class);
+
+    private String[] resourceNames;
+    private List<JsonModel> resultModel;
+    private LanguageOption languageOption;
+
+    /**
+     * Helper class for the endToEnd tests. In this class the necessary resources are loaded, prepared and copied for the
+     * tests based on the passed parameters. An instance of this class loads all necessary paths and properties for a test
+     * run with the specified language
+     * @param languageOption for loading language-specific resources
+     * @throws IOException is thrown for all problems that may occur while parsing the json file. This includes both reading
+     * and parsing problems.
+     */
+    public JPlagTestSuiteHelper(LanguageOption languageOption) throws IOException {
+        this.languageOption = languageOption;
+        this.resourceNames = new File(TestDirectoryConstants.BASE_PATH_TO_JAVA_RESOURCES_SORTALGO.toString()).list();
+
+        this.resultModel = JsonHelper.getResultModelFromPath();
+        logger.debug("temp path at [{}]", TestDirectoryConstants.TEMPORARY_SUBMISSION_DIRECTORY_NAME);
+    }
+
+    /**
+     * creates all necessary folder paths and objects for a test run. Also searches for the stored previous results of the
+     * test in order to compare them with the current results.
+     * @param classNames Array of class names with language specific extension to be prepared for a test.
+     * @return comparison results saved for the test
+     * @throws IOException Exception can be thrown in cases that involve reading, copying or locating files.
+     */
+    public TestCaseModel createNewTestCase(String[] classNames, String functionName) throws IOException {
+        createNewTestCaseDirectory(classNames);
+        JsonModel resultJsonModel = resultModel.stream().filter(jsonModel -> functionName.equals(jsonModel.getFunctionName())).findAny().orElse(null);
+        return new TestCaseModel(TestDirectoryConstants.TEMPORARY_SUBMISSION_DIRECTORY_NAME, resultJsonModel, languageOption);
+    }
+
+    /**
+     * The copied data should be deleted after instance closure
+     * @throws IOException if an I/O error occurs
+     */
+    public void clear() throws IOException {
+        logger.debug("Class instance was cleaned!");
+        deleteCopiedFiles(new File(TestDirectoryConstants.TEMPORARY_SUBMISSION_DIRECTORY_NAME));
+    }
+
+    /**
+     * Copies the passed filenames to a temporary path to use them in the tests
+     * @throws IOException Exception can be thrown in cases that involve reading, copying or locating files.
+     */
+    private void createNewTestCaseDirectory(String[] classNames) throws IOException {
+        // before copying files to the test path, check if all files are in the resource
+        // directory
+        for (String className : classNames) {
+            if (!Arrays.asList(resourceNames).contains(className)) {
+                throw new FileNotFoundException(String.format("The specified class could not be found! [%s]", className));
+            }
+        }
+        // Copy the resources data to the temporary path
+        for (int counter = 0; counter < classNames.length; counter++) {
+            Path originalPath = Path.of(TestDirectoryConstants.BASE_PATH_TO_JAVA_RESOURCES_SORTALGO.toString(), classNames[counter]);
+            Path copiePath = Path.of(TestDirectoryConstants.TEMPORARY_SUBMISSION_DIRECTORY_NAME,
+                    TestDirectoryConstants.TEMPORARY_DIRECTORY_NAME + (counter + 1), classNames[counter]);
+
+            File directory = new File(copiePath.toString());
+            if (!directory.exists()) {
+                directory.mkdirs();
+            }
+
+            Files.copy(originalPath, copiePath, StandardCopyOption.REPLACE_EXISTING);
+            logger.debug("Copy file from [{}] to [{}]", originalPath, copiePath);
+        }
+    }
+
+    /**
+     * Delete directory with including files
+     * @param file Path to a folder or file to be deleted. This happens recursively to the path
+     * @throws IOException if an I/O error occurs
+     */
+    private void deleteCopiedFiles(File folder) throws IOException {
+        File[] files = folder.listFiles();
+        if (files != null) { // some JVMs return null for empty dirs
+            for (File file : files) {
+                if (file.isDirectory()) {
+                    deleteCopiedFiles(file);
+                } else {
+                    Files.delete(file.toPath());
+                    logger.debug("Delete file in folder: [{}]", file);
+                }
+            }
+        }
+        Files.delete(folder.toPath());
+        logger.debug("Delete folder: [{}]", folder);
+    }
+
+}

jplag.endToEndTesting/src/main/java/de/jplag/end_to_end_testing/helper/JsonHelper.java

@@ -0,0 +1,33 @@
+package de.jplag.end_to_end_testing.helper;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.List;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import de.jplag.end_to_end_testing.constants.TestDirectoryConstants;
+import de.jplag.end_to_end_testing.model.JsonModel;
+
+/**
+ * Helper class for serializing and creating all json dependent events.
+ */
+public final class JsonHelper {
+
+    /**
+     * private constructor to prevent instantiation
+     */
+    private JsonHelper() {
+        // For Serialization
+    }
+
+    /**
+     * Parsing the old results in the json file as a list from ResultJsonModel.
+     * @return list of saved results for the test cases
+     * @throws IOException is thrown for all problems that may occur while parsing the json file. This includes both reading
+     * and parsing problems.
+     */
+    public static List<JsonModel> getResultModelFromPath() throws IOException {
+        return Arrays.asList(new ObjectMapper().readValue(TestDirectoryConstants.BASE_PATH_TO_JAVA_RESULT_JSON.toFile(), JsonModel[].class));
+    }
+}

jplag.endToEndTesting/src/main/java/de/jplag/end_to_end_testing/model/JsonModel.java

@@ -0,0 +1,52 @@
+package de.jplag.end_to_end_testing.model;
+
+import java.util.Arrays;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * The ResultJsonModel is the java object for the JavaResult.json file. The object contains all the necessary
+ * information for the comparisons in the test cases between old and new results, which have been stored in the
+ * JavaResult.json file.
+ */
+public class JsonModel {
+    @JsonProperty("function_name")
+    private String functionName;
+    @JsonProperty("test_results")
+    private ResultModel[] results;
+
+    /**
+     * Constructor for the JsonModel. The model is the serialization of the Json file in the form of a Java object.
+     * @param functionName the function name for the associated test results. Used as identifier to search results for the
+     * test cases.
+     * @param results Collection of the results that are in the current json file for certain tests
+     */
+    public JsonModel(String functionName, ResultModel[] results) {
+        this.functionName = functionName;
+        this.results = results;
+    }
+
+    /**
+     * empty constructor in case the serialization contains an empty object to prevent throwing exceptions. this constructor
+     * was necessary for serialization with the Jackson parse extension
+     */
+    public JsonModel() {
+        // For Serialization
+    }
+
+    /**
+     * @return the name of the currently used function stored in json
+     */
+    public String getFunctionName() {
+        return functionName;
+    }
+
+    /**
+     * returns the results for the function that have been stored for the given id.
+     * @param identifier for the comparative values
+     * @return associated comparison values that have been assigned to the identifier
+     */
+    public ResultModel getResultModelById(Integer identifier) {
+        return Arrays.asList(results).stream().filter(resultModel -> identifier.equals(resultModel.getTestId())).findAny().orElse(null);
+    }
+}