Skip to content

Commit

Permalink
Fixing #137 documented the feature
Browse files Browse the repository at this point in the history
  • Loading branch information
baubakg committed Jun 20, 2024
1 parent 362f481 commit 7df31aa
Show file tree
Hide file tree
Showing 8 changed files with 89 additions and 38 deletions.
11 changes: 8 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,17 +260,21 @@ In this case we get :

### Comparing Log Data
As of version 1.11.0 we have introduced the possibility to compare two LogData objects. This is a light compare that checks that for a given key, if it is absent, added or changes in frequency. The method `compare` returns a `LogDataComparison` object that contains the results of the comparison. A comparison can be of three types:
* Added : The entry has been added
* NEW : The entry has been added
* Removed : The entry has been removed
* Changed : The entry has changed in frequency

Apart from this we return the :
* delta : The difference in frequency
* deltaRatio : The difference in frequency as a ratio
* deltaRatio : The difference in frequency as a ratio in %

These values are negative if the values have decreased.

```java
Creating a differentiation report is done with the method `LogData.compare(LogData<T> in_logData)`. This method returns a `LogDataComparison` object that contains the results of the comparison.

#### Creating a Differentiation Report
We can generate an HTML Report where the differences are high-lighted. This is done with the method `LogDataFactory.generateComparisonReport(LogData reference, LogData target, String filename)`. This method will generate an HTML Report detailing the found differences.


## Assertions and LogDataAssertions
As of version 1.0.5 we have introduced the notion of assertions. Assertions can either take a LogData object or a set of files as input.
Expand All @@ -294,6 +298,7 @@ We now have the possibility to export the log data results into a CSV file. The

### 1.11.0 (next version)
- **(new feature)** [#127](https://github.com/adobe/log-parser/issues/127) You can now compare two LogData Objects. This is a light compare that checks that for a given key, if it is absent, added or changes in frequency.
- **(new feature)** [#137](https://github.com/adobe/log-parser/issues/137) We can now generate an HTML report for the differences in log data.
- **(new feature)** [#117](https://github.com/adobe/log-parser/issues/117) You can now include the file name in the result of the analysis.
- **(new feature)** [#123](https://github.com/adobe/log-parser/issues/123) We now log the total number and size of the parsed files.
- [#110](https://github.com/adobe/log-parser/issues/110) Moved to Java 11
Expand Down
Binary file modified diagrams/Log_Parser-Processes.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
49 changes: 36 additions & 13 deletions diagrams/Log_Parser.drawio
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<mxfile host="Electron" modified="2023-10-05T13:03:58.211Z" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/21.5.0 Chrome/112.0.5615.204 Electron/24.5.1 Safari/537.36" etag="BJ7LdRRuA4aZrNys8yPA" version="21.5.0" type="device" pages="3">
<mxfile host="Electron" modified="2024-06-20T15:45:08.208Z" agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/21.5.0 Chrome/112.0.5615.204 Electron/24.5.1 Safari/537.36" etag="-3Xt7U6pxw0lLMGN0c0n" version="21.5.0" type="device" pages="3">
<diagram id="GF7U1PxuMcIbTAeL1bwN" name="Classes">
<mxGraphModel dx="1114" dy="878" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="850" pageHeight="1100" background="#FFFFFF" math="0" shadow="0">
<mxGraphModel dx="1354" dy="912" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="850" pageHeight="1100" background="#FFFFFF" math="0" shadow="0">
<root>
<mxCell id="0" />
<mxCell id="1" parent="0" />
Expand Down Expand Up @@ -72,7 +72,7 @@
</mxGraphModel>
</diagram>
<diagram id="LirX-YWR1LI0kx1sx8zC" name="Processes">
<mxGraphModel dx="1114" dy="878" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" background="#FFFFFF" math="0" shadow="0">
<mxGraphModel dx="1114" dy="912" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="827" pageHeight="1169" background="#FFFFFF" math="0" shadow="0">
<root>
<mxCell id="dgaH53lybmkEFGB52cs7-0" />
<mxCell id="dgaH53lybmkEFGB52cs7-1" parent="dgaH53lybmkEFGB52cs7-0" />
Expand All @@ -92,12 +92,13 @@
<mxGeometry x="440" y="227" width="120" height="60" as="geometry" />
</mxCell>
<mxCell id="dgaH53lybmkEFGB52cs7-35" value="Group By &lt;br&gt;Definition Entry" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.25;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;entryPerimeter=0;fontColor=#393C56;labelBackgroundColor=#FFFFFF;" parent="dgaH53lybmkEFGB52cs7-1" source="dgaH53lybmkEFGB52cs7-42" target="dgaH53lybmkEFGB52cs7-34" edge="1">
<mxGeometry relative="1" as="geometry">
<mxGeometry x="0.589" relative="1" as="geometry">
<mxPoint x="700" y="260" as="sourcePoint" />
<Array as="points">
<mxPoint x="850" y="242" />
<mxPoint x="850" y="70" />
<mxPoint x="850" y="180" />
</Array>
<mxPoint as="offset" />
</mxGeometry>
</mxCell>
<mxCell id="dgaH53lybmkEFGB52cs7-32" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;labelBackgroundColor=#FFFFFF;fontColor=#393C56;" parent="dgaH53lybmkEFGB52cs7-1" edge="1">
Expand All @@ -107,23 +108,33 @@
</mxGeometry>
</mxCell>
<mxCell id="dgaH53lybmkEFGB52cs7-34" value="Group By" style="strokeWidth=2;html=1;shape=mxgraph.flowchart.terminator;whiteSpace=wrap;shadow=1;labelBackgroundColor=#FFFFFF;" parent="dgaH53lybmkEFGB52cs7-1" vertex="1">
<mxGeometry x="920" y="40" width="100" height="60" as="geometry" />
<mxGeometry x="970" y="150" width="100" height="60" as="geometry" />
</mxCell>
<mxCell id="FXQAlGXq-PYeFnHcnt1Y-17" value="&lt;font style=&quot;font-size: 11px&quot;&gt;Filter By &lt;br&gt;Definition Entry&lt;/font&gt;" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.25;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;entryPerimeter=0;labelBackgroundColor=#FFFFFF;strokeColor=#000000;fontSize=12;" parent="dgaH53lybmkEFGB52cs7-1" source="dgaH53lybmkEFGB52cs7-42" target="FXQAlGXq-PYeFnHcnt1Y-16" edge="1">
<mxGeometry x="0.3133" relative="1" as="geometry">
<mxGeometry x="0.5683" relative="1" as="geometry">
<Array as="points">
<mxPoint x="850" y="242" />
<mxPoint x="850" y="290" />
</Array>
<mxPoint as="offset" />
</mxGeometry>
</mxCell>
<mxCell id="CEJ-vFzQ0yJb0CGG5-7o-4" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.75;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;dashed=1;" edge="1" parent="dgaH53lybmkEFGB52cs7-1" source="dgaH53lybmkEFGB52cs7-42" target="CEJ-vFzQ0yJb0CGG5-7o-0">
<mxCell id="CEJ-vFzQ0yJb0CGG5-7o-4" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.75;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;dashed=1;" parent="dgaH53lybmkEFGB52cs7-1" source="dgaH53lybmkEFGB52cs7-42" target="CEJ-vFzQ0yJb0CGG5-7o-0" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="CEJ-vFzQ0yJb0CGG5-7o-5" value="export" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.75;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;dashed=1;" edge="1" parent="dgaH53lybmkEFGB52cs7-1" source="dgaH53lybmkEFGB52cs7-42" target="CEJ-vFzQ0yJb0CGG5-7o-1">
<mxCell id="CEJ-vFzQ0yJb0CGG5-7o-5" value="export" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.75;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;dashed=1;" parent="dgaH53lybmkEFGB52cs7-1" source="dgaH53lybmkEFGB52cs7-42" target="CEJ-vFzQ0yJb0CGG5-7o-1" edge="1">
<mxGeometry relative="1" as="geometry" />
</mxCell>
<mxCell id="C3BIZFhyQSCN-kVbLvhw-1" value="compare &lt;br&gt;two&lt;br&gt;LogData" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.25;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;entryPerimeter=0;" edge="1" parent="dgaH53lybmkEFGB52cs7-1" source="dgaH53lybmkEFGB52cs7-42" target="C3BIZFhyQSCN-kVbLvhw-2">
<mxGeometry x="0.6517" relative="1" as="geometry">
<mxPoint x="850" y="80" as="targetPoint" />
<Array as="points">
<mxPoint x="850" y="242" />
<mxPoint x="850" y="70" />
</Array>
<mxPoint as="offset" />
</mxGeometry>
</mxCell>
<mxCell id="dgaH53lybmkEFGB52cs7-42" value="LogData" style="rounded=0;whiteSpace=wrap;html=1;shadow=1;labelBackgroundColor=#FFFFFF;" parent="dgaH53lybmkEFGB52cs7-1" vertex="1">
<mxGeometry x="600" y="227" width="120" height="60" as="geometry" />
</mxCell>
Expand Down Expand Up @@ -221,7 +232,7 @@
<mxGeometry x="70" y="608.5" width="110" height="63" as="geometry" />
</mxCell>
<mxCell id="FXQAlGXq-PYeFnHcnt1Y-16" value="Filter By /&lt;br&gt;search" style="strokeWidth=2;html=1;shape=mxgraph.flowchart.terminator;whiteSpace=wrap;shadow=1;labelBackgroundColor=#FFFFFF;" parent="dgaH53lybmkEFGB52cs7-1" vertex="1">
<mxGeometry x="920" y="260" width="100" height="60" as="geometry" />
<mxGeometry x="970" y="260" width="100" height="60" as="geometry" />
</mxCell>
<mxCell id="FXQAlGXq-PYeFnHcnt1Y-18" value="JSON" style="shape=document;whiteSpace=wrap;html=1;boundedLbl=1;labelBackgroundColor=#FFFFFF;strokeColor=#000000;fillColor=#FFFFFF;fontSize=12;align=center;" parent="dgaH53lybmkEFGB52cs7-1" vertex="1">
<mxGeometry x="527" y="560" width="110" height="63" as="geometry" />
Expand Down Expand Up @@ -259,17 +270,29 @@
<mxPoint x="330" y="200" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="CEJ-vFzQ0yJb0CGG5-7o-0" value="JSON" style="shape=document;whiteSpace=wrap;html=1;boundedLbl=1;labelBackgroundColor=#FFFFFF;strokeColor=#000000;fillColor=#FFFFFF;fontSize=12;align=center;" vertex="1" parent="dgaH53lybmkEFGB52cs7-1">
<mxCell id="CEJ-vFzQ0yJb0CGG5-7o-0" value="JSON" style="shape=document;whiteSpace=wrap;html=1;boundedLbl=1;labelBackgroundColor=#FFFFFF;strokeColor=#000000;fillColor=#FFFFFF;fontSize=12;align=center;" parent="dgaH53lybmkEFGB52cs7-1" vertex="1">
<mxGeometry x="840" y="350" width="110" height="63" as="geometry" />
</mxCell>
<mxCell id="CEJ-vFzQ0yJb0CGG5-7o-1" value="CSV" style="shape=document;whiteSpace=wrap;html=1;boundedLbl=1;labelBackgroundColor=#FFFFFF;strokeColor=#000000;fillColor=#FFFFFF;fontSize=12;align=center;" vertex="1" parent="dgaH53lybmkEFGB52cs7-1">
<mxCell id="CEJ-vFzQ0yJb0CGG5-7o-1" value="CSV" style="shape=document;whiteSpace=wrap;html=1;boundedLbl=1;labelBackgroundColor=#FFFFFF;strokeColor=#000000;fillColor=#FFFFFF;fontSize=12;align=center;" parent="dgaH53lybmkEFGB52cs7-1" vertex="1">
<mxGeometry x="840" y="440" width="110" height="63" as="geometry" />
</mxCell>
<mxCell id="C3BIZFhyQSCN-kVbLvhw-2" value="Compare" style="strokeWidth=2;html=1;shape=mxgraph.flowchart.terminator;whiteSpace=wrap;shadow=1;labelBackgroundColor=#FFFFFF;" vertex="1" parent="dgaH53lybmkEFGB52cs7-1">
<mxGeometry x="970" y="40" width="100" height="60" as="geometry" />
</mxCell>
<mxCell id="C3BIZFhyQSCN-kVbLvhw-3" value="export" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;dashed=1;exitPerimeter=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;" edge="1" parent="dgaH53lybmkEFGB52cs7-1" source="C3BIZFhyQSCN-kVbLvhw-2" target="C3BIZFhyQSCN-kVbLvhw-4">
<mxGeometry relative="1" as="geometry">
<mxPoint x="730" y="282" as="sourcePoint" />
<mxPoint x="1110" y="70" as="targetPoint" />
</mxGeometry>
</mxCell>
<mxCell id="C3BIZFhyQSCN-kVbLvhw-4" value="HTML" style="shape=document;whiteSpace=wrap;html=1;boundedLbl=1;labelBackgroundColor=#FFFFFF;strokeColor=#000000;fillColor=#FFFFFF;fontSize=12;align=center;" vertex="1" parent="dgaH53lybmkEFGB52cs7-1">
<mxGeometry x="1140" y="37" width="110" height="63" as="geometry" />
</mxCell>
</root>
</mxGraphModel>
</diagram>
<diagram id="6ssjRT5Lo6XObwfs72mm" name="log-parsing">
<mxGraphModel dx="1114" dy="878" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1169" pageHeight="827" math="0" shadow="0">
<mxGraphModel dx="1114" dy="912" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1169" pageHeight="827" math="0" shadow="0">
<root>
<mxCell id="ElcgD5YhV_jEJsSnePJW-0" />
<mxCell id="ElcgD5YhV_jEJsSnePJW-1" parent="ElcgD5YhV_jEJsSnePJW-0" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@

import com.adobe.campaign.tests.logparser.exceptions.IncorrectParseDefinitionException;
import com.adobe.campaign.tests.logparser.exceptions.LogDataExportToFileException;
import com.adobe.campaign.tests.logparser.utils.FileUtils;
import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVPrinter;
import org.apache.logging.log4j.LogManager;
Expand Down Expand Up @@ -393,12 +394,7 @@ public File exportLogDataToCSV(Set<String> in_headerSet, String in_csvFileName)
throws LogDataExportToFileException {
File l_exportFile = new File(in_csvFileName);

if (l_exportFile.exists()) {
log.info("Deleting existing log export file {}...", in_csvFileName);
if (!l_exportFile.delete()) {
throw new LogDataExportToFileException("We were unable to delete the file "+ l_exportFile.getPath());
}
}
FileUtils.cleanFile(l_exportFile);

try (CSVPrinter printer = new CSVPrinter(new FileWriter(in_csvFileName), CSVFormat.DEFAULT)) {
printer.printRecord(in_headerSet);
Expand Down Expand Up @@ -439,7 +435,7 @@ public Map<String, LogDataComparison> compare(LogData<T> in_logData) {

for (String lt_key : in_logData.getEntries().keySet()) {
if (!this.getEntries().containsKey(lt_key)) {
lr_diff.put(lt_key, new LogDataComparison(in_logData.get(lt_key), LogDataComparison.ChangeType.ADDED,
lr_diff.put(lt_key, new LogDataComparison(in_logData.get(lt_key), LogDataComparison.ChangeType.NEW,
0, in_logData.get(lt_key).getFrequence()));
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,6 @@ public Double getDeltaRatio() {


enum ChangeType {
ADDED, REMOVED, MODIFIED
NEW, REMOVED, MODIFIED
};
}
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@

import java.io.File;
import java.io.FileFilter;
import java.nio.charset.Charset;
import java.util.*;
import java.util.stream.Collectors;

Expand Down Expand Up @@ -260,7 +259,7 @@ public static <T extends StdLogEntry> File generateDiffReport(LogData<T> in_logD
sb.append("<tr>");
sb.append("<th>New Errors</th>");
sb.append("<td>");
sb.append(comparisonReport.values().stream().filter(l -> l.getChangeType().equals(LogDataComparison.ChangeType.ADDED)).count());
sb.append(comparisonReport.values().stream().filter(l -> l.getChangeType().equals(LogDataComparison.ChangeType.NEW)).count());
sb.append("</td>");
sb.append("</tr>");
sb.append("<tr>");
Expand Down Expand Up @@ -313,12 +312,7 @@ public static <T extends StdLogEntry> File generateDiffReport(LogData<T> in_logD
sb.append("</body>");
File l_exportFile = new File(in_reportName + ".html");

if (l_exportFile.exists()) {
log.info("Deleting existing log export file {}...", l_exportFile.getPath());
if (!l_exportFile.delete()) {
throw new LogDataExportToFileException("We were unable to delete the file "+ l_exportFile.getPath());
}
}
com.adobe.campaign.tests.logparser.utils.FileUtils.cleanFile(l_exportFile);

try {
//FileUtils.writeStringToFile(l_exportFile, );
Expand All @@ -328,4 +322,5 @@ public static <T extends StdLogEntry> File generateDiffReport(LogData<T> in_logD
}
return l_exportFile;
}

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
/*
* Copyright 2022 Adobe
* All Rights Reserved.
*
* NOTICE: Adobe permits you to use, modify, and distribute this file in
* accordance with the terms of the Adobe license agreement accompanying
* it.
*/

package com.adobe.campaign.tests.logparser.utils;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

import java.io.File;

public class FileUtils {
protected static Logger log = LogManager.getLogger();

/**
* Deletes a file if it exists
* @param in_file a file to delete
*/
public static void cleanFile(File in_file) {
if (in_file.exists()) {
log.info("Deleting existing log export file {}...", in_file.getPath());
in_file.delete();
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,17 @@
*/
package com.adobe.campaign.tests.logparser.core;

import com.adobe.campaign.tests.logparser.exceptions.LogDataExportToFileException;
import com.adobe.campaign.tests.logparser.utils.FileUtils;
import org.hamcrest.Matchers;
import org.testng.annotations.Test;

import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Arrays;
import java.util.List;
import java.util.Map;

import static org.hamcrest.MatcherAssert.assertThat;
Expand Down Expand Up @@ -96,7 +101,7 @@ public void simpleDifferenceEmptyLeftSide() {
l_diff.get("12").getDelta(), Matchers.equalTo(2));
assertThat("We should have the correct key",
l_diff.get("12").getDeltaRatio(), Matchers.equalTo(100.0));
assertThat("We should classify this as removed", l_diff.get("12").getChangeType(), Matchers.equalTo(LogDataComparison.ChangeType.ADDED));
assertThat("We should classify this as removed", l_diff.get("12").getChangeType(), Matchers.equalTo(LogDataComparison.ChangeType.NEW));

assertThat("We should have stored the original logData", l_diff.get("12").getLogEntry(), Matchers.equalTo(l_cubeData2.get("12")));
}
Expand Down Expand Up @@ -157,7 +162,7 @@ public void simpleAddedRemoved() {
assertThat("We should classify this as removed", l_diff.get("12").getChangeType(),
Matchers.equalTo(LogDataComparison.ChangeType.REMOVED));
assertThat("We should classify this as removed", l_diff.get("13").getChangeType(),
Matchers.equalTo(LogDataComparison.ChangeType.ADDED));
Matchers.equalTo(LogDataComparison.ChangeType.NEW));
}

@Test
Expand Down Expand Up @@ -196,9 +201,6 @@ public void simpleAddedRemovedComplex() {

}




private static ParseDefinition fetchSTDDefinition() {
ParseDefinition l_definition = new ParseDefinition("tmp");
final ParseDefinitionEntry l_parseDefinitionEntryKey = new ParseDefinitionEntry("AAZ");
Expand Down

0 comments on commit 7df31aa

Please sign in to comment.