Code

Webanalytic bug#566

Code

Merged

MiniPlayer merged 74 commits intorelease/v1.4.0Hurence/logisland:release/v1.4.0from webanalyticBugHurence/logisland:webanalyticBugCopy head branch name to clipboardApr 29, 2021

+75,685-6,977

Commits

Commits on Nov 23, 2020

• [WIP] tests that shows the bug.

  MiniPlayer

  committed
  7d0ea9b

  Copy full SHA for 7d0ea9b

Commits on Nov 24, 2020

• Fixed incrementalwebsession bug with high frequency batch.

  MiniPlayer

  committed
  6b82d68

  Copy full SHA for 6b82d68

Commits on Nov 25, 2020

• Enable the possibility to use plugin without special packaging. Each plugin module is now installed without repackaging and with repackaging.

  MiniPlayer

  committed
  2df4095

  Copy full SHA for 2df4095

• Added documentation on content of assembly and how to compare it.

  MiniPlayer

  committed
  6747608

  Copy full SHA for 6747608

Commits on Nov 26, 2020

• Added documentation on IncrementalWebSession processor. Proposed another implementation to solve some problems.

  MiniPlayer

  committed
  bba58fc

  Copy full SHA for bba58fc

• Added test for rewind of incrementalWebSession and restart after a fail without commit of the offsets.

  MiniPlayer

  committed
  dcd1fba

  Copy full SHA for dcd1fba

Commits on Nov 27, 2020

• Moved inner class of IncrementalWebSession To their own file. So that code is more readable. (Ensured all test that were green, still are)

  MiniPlayer

  committed
  6ea2b8e

  Copy full SHA for 6ea2b8e

Commits on Dec 1, 2020

• [WIP] new implementation of the incrementalWebSession processor (handling rewind).

  MiniPlayer

  committed
  7407a56

  Copy full SHA for 7407a56

Commits on Dec 4, 2020

• [WIP] implementing the new version of IncrementalWebSession processor.

  MiniPlayer

  committed
  ddf58c0

  Copy full SHA for ddf58c0

Commits on Dec 8, 2020

• [WIP]implemented New IncrementalWebSession processor and adapted IT tests. Implemented queryGet and multiQueryGet methods in Es 7 client service.

  MiniPlayer

  committed
  bdaa9f7

  Copy full SHA for bdaa9f7

• Added prefix and suffix date formatter properties for webevent and web sessions indices. Fixed some issues in elasticsearch queries.

  MiniPlayer

  committed
  3f855f1

  Copy full SHA for 3f855f1

• [WIP]

  MiniPlayer

  committed
  f874865

  Copy full SHA for f874865

Commits on Dec 9, 2020

• Finished implementation of IncrementalWebSession. IT tests are green.

  MiniPlayer

  committed
  65fe579

  Copy full SHA for 65fe579

• Adapted old unit test to integration tests.

  MiniPlayer

  committed
  3b9c3d3

  Copy full SHA for 3b9c3d3

Commits on Dec 10, 2020

• Decoupled Event, Websession from IncrementalWebSession processor instance.

  MiniPlayer

  committed
  02bed7f

  Copy full SHA for 02bed7f

• DOne some TODO. Added a test to verify add method of Events object.

  MiniPlayer

  committed
  71dc65a

  Copy full SHA for 71dc65a

• Implemented BoolQueryRecordRoot, which is a tree query model. It is used to construct boolQuery in elasticsearch7 service. Fixed with that the bug when deleting future sessions in IncrementalWebSes…

  Show description for 87c2718
  MiniPlayer

  committed
  87c2718

  Copy full SHA for 87c2718

• Moved toMap methods from IncrementalWebssesion to Utils. Extracted SessionCheckerResults too from anonymous instance to real classes.

  MiniPlayer

  committed
  42b655f

  Copy full SHA for 42b655f

Commits on Dec 11, 2020

• Fixed tests.

  MiniPlayer

  committed
  d9512d1

  Copy full SHA for d9512d1

Commits on Dec 14, 2020

• [WIP] Added test on time parsing and use same library. Added option to choose Timezone to use for format. Otherwise use system default.

  MiniPlayer

  committed
  107883f

  Copy full SHA for 107883f

• Removed all TODOs in logisland-processor-web-analytics module

  MiniPlayer

  committed
  ecdca58

  Copy full SHA for ecdca58

• Removed some todos.

  MiniPlayer

  committed
  89ed8fe

  Copy full SHA for 89ed8fe

• Fixed DebugStream processor. Using toString method instead of serialization. toString method of StandardRecord is pretty for all types !

  MiniPlayer

  committed
  d9ade5c

  Copy full SHA for d9ade5c

Commits on Dec 15, 2020

• Added two properties in IncrementalWebSession for choosing filednames where to put index names. By default 'es_index' and 'es_type'. IncrementalWebSession does not save events into es anymore. It r…

  Show description for c99d0b2
  MiniPlayer

  committed
  c99d0b2

  Copy full SHA for c99d0b2

Commits on Dec 16, 2020

• Added tests about a bug if an events is delayed. Added in test a counter to ensure there is not too much rewind. Thoses tests are currently not passing.

  MiniPlayer

  committed
  3d7aa64

  Copy full SHA for 3d7aa64

Commits on Dec 17, 2020

• [WIP] Handle case when events does not come in order. Added documentation about it in wiki.

  MiniPlayer

  committed
  acb2357

  Copy full SHA for acb2357

Commits on Jan 5, 2021

• [WIP] Handle case when events does not come in order. Added documentation about it in wiki.

  MiniPlayer

  committed
  893a1a2

  Copy full SHA for 893a1a2

• handle case when events does not come in order (as best effort). We assume the time range of input events is not too large.

  MiniPlayer

  committed
  ae00447

  Copy full SHA for ae00447

Commits on Jan 6, 2021

• Fixed a bug. originalSessionId was not always set to divolteSession. New TestRunner always clone input records when using enqueue so that input REcord are never modified (used in unit test afterwar…

  Show description for 339ab96
  MiniPlayer

  committed
  339ab96

  Copy full SHA for 339ab96

• Handle case when last event has not matching session in es (event from future).

  MiniPlayer

  committed
  1ea3982

  Copy full SHA for 1ea3982

Commits on Jan 7, 2021

• Fixed bugs in multiple complex cases. First event for a divolt session come in another batch after some other event passed. In this case try to fix all already processed events, a conf is available…

  Show description for 9811012
  MiniPlayer

  committed
  9811012

  Copy full SHA for 9811012

• Simplified calculation of new sessions.

  MiniPlayer

  committed
  14cfa07

  Copy full SHA for 14cfa07

Commits on Jan 8, 2021

• Improved speed of It tests in IncrementalWebsessionIt. Spliited some big methode in smaller ones.

  MiniPlayer

  committed
  b4a28fc

  Copy full SHA for b4a28fc

Commits on Jan 11, 2021

• Added java doc on IT tests.

  MiniPlayer

  committed
  0a36c4b

  Copy full SHA for 0a36c4b

• Merge branch 'release/v1.4.0' into webanalyticBug

  MiniPlayer

  committed
  cd84b9f

  Copy full SHA for cd84b9f

Commits on Jan 22, 2021

• Added comment for cleaning logisland-api.

  MiniPlayer

  committed
  b77248a

  Copy full SHA for b77248a

• Modifyes StructuredStream api (reader/writer). Added a module of test for webanal flux.

  MiniPlayer

  committed
  040ca0d

  Copy full SHA for 040ca0d

• Applied base logisland api modifications. Renommed ValidationContext to Configuration and removed unused methods.

  MiniPlayer

  committed
  4888ac4

  Copy full SHA for 4888ac4

• Implemented changes in Engine api. Adding init(), softStop() and renamed others method in ProcessingEngine interface.

  MiniPlayer

  committed
  0055c2b

  Copy full SHA for 0055c2b

• Implemented changes in Stream api. Creating a SparkStreamContext and broadcasting services in engine.

  MiniPlayer

  committed
  09c4791

  Copy full SHA for 09c4791

Commits on Jan 25, 2021

• Implemented changes in Service api. Adding start and stop method in ControllerService interface with default implementation. Removed unused method in AbstractControllerService. Removed KerberosCont…

  Show description for 2f35aee
  MiniPlayer

  committed
  2f35aee

  Copy full SHA for 2f35aee

• Added forgotten SparkStreamContext file which should have been pushed with stream modifications.

  MiniPlayer

  committed
  5562807

  Copy full SHA for 5562807

• Added default start and stop methods to processors.

  MiniPlayer

  committed
  4526b1a

  Copy full SHA for 4526b1a

• Added usage of start/init methods of Processor interface into streams. Even if it does not really do anything at the moment.

  MiniPlayer

  committed
  7d50376

  Copy full SHA for 7d50376

• Fixed usage of isInitialized in EncodeBase64 processor.

  MiniPlayer

  committed
  2b45045

  Copy full SHA for 2b45045

• Modified StructuredStream so that Reader are responsible for creating the right records. Writer to inject in sink from records and the stream apply transformations with processors.

  MiniPlayer

  committed
  00fa22c

  Copy full SHA for 00fa22c

Commits on Feb 4, 2021

• working logisland with confluent schema registry

  michaelsoubra

  committed
  e4cfd39

  Copy full SHA for e4cfd39

Commits on Feb 5, 2021

• serializer cleaning

  michaelsoubra

  committed
  0d75ffa

  Copy full SHA for 0d75ffa

• fix comment

  michaelsoubra

  committed
  db207be

  Copy full SHA for db207be

• remove spark on k8s option from logisland main.yml

  michaelsoubra

  committed
  905efa5

  Copy full SHA for 905efa5

• UT for Confluent Serialiser

  michaelsoubra

  committed
  d31cea7

  Copy full SHA for d31cea7

• - instead of . for kibana

  michaelsoubra

  committed
  f3eda51

  Copy full SHA for f3eda51

Commits on Feb 8, 2021

• Added a module with simple test of kafka with an embedded kafka 0.10.X server. Keeping the other module using kafka 2 for later usage (when we update engine to use kafka 2).

  MiniPlayer

  committed
  3c8d962

  Copy full SHA for 3c8d962

Commits on Feb 9, 2021

• Excluded net.jpountz.lz4:lz4 dependencies from dependence kafka_streaming in spark 2_4_common_engines module that was conflicting when testing in a module. Anyway this is embedded in spark_core lib…

  Show description for d7caa0a
  MiniPlayer

  committed
  d7caa0a

  Copy full SHA for d7caa0a

• Ignore infinite tests.

  MiniPlayer

  committed
  c840e35

  Copy full SHA for c840e35

Commits on Feb 10, 2021

• Working simple pipeline in test module

  MiniPlayer

  committed
  cc691c3

  Copy full SHA for cc691c3

• Moved properties into more specific object, so that we can more easily know what properties is used in each stream or providers.

  MiniPlayer

  committed
  076934f

  Copy full SHA for 076934f

Commits on Feb 11, 2021

• Some cleaning. Reducing parameters needed. Moved some code to init method.

  MiniPlayer

  committed
  27769f0

  Copy full SHA for 27769f0

Commits on Feb 15, 2021

• Use ControllerServiceContext in ControllerServiceLookupSink instead of ControllerServiceCOnfiguration. Fixed a bug in Addfields, dynamic props did not support empty values. Fixed a bug in DATE_TIME…

  Show description for f072117
  MiniPlayer

  committed
  f072117

  Copy full SHA for f072117

• Added some files for test with kafka > 0.10

  MiniPlayer

  committed
  3056718

  Copy full SHA for 3056718

Commits on Feb 16, 2021

• Adjusted conf and used service so that jobs works.

  MiniPlayer

  committed
  904e38d

  Copy full SHA for 904e38d

• Added outputmode option in KafkaStructuredStream service. Added conf to use group by in spark streaming.

  MiniPlayer

  committed
  f7a06bd

  Copy full SHA for f7a06bd

Commits on Feb 24, 2021

• Postion is not a Record anymore. Started new conf of StructuredStream.

  MiniPlayer

  committed
  05c05a1

  Copy full SHA for 05c05a1

Commits on Feb 25, 2021

• Reuse old ways to create ControllerServiceLookup, using ControllerServiceConfiguration instead of context which impose services to be serializable.

  MiniPlayer

  committed
  8c7672e

  Copy full SHA for 8c7672e

• Finalized Structur stream conf and improved bad conf explaination.

  MiniPlayer

  committed
  3ec613c

  Copy full SHA for 3ec613c

• Fixed checkpoint localtion. Each query must have a different checkpointlocation.

  MiniPlayer

  committed
  9ab962e

  Copy full SHA for 9ab962e

• Added an ignored test in laboratory webanal module test.

  MiniPlayer

  committed
  98de40b

  Copy full SHA for 98de40b

Commits on Mar 9, 2021

• Merge remote-tracking branch 'origin/feature/schema_registry_confluent' into webanalyticBug

  Show description for 20d3e53
  

  Enrico Mano

  committed
  20d3e53

  Copy full SHA for 20d3e53

Commits on Mar 11, 2021

• Merge of code for confluent serializer that is not yet serializable

  

  Enrico Mano

  committed
  3be6295

  Copy full SHA for 3be6295

Commits on Mar 12, 2021

• Make ConfluentSerializer serializable for structured streaming

  

  Enrico Mano

  committed
  62cf247

  Copy full SHA for 62cf247

• Change geolocation label to avoid conflict with divolte location

  

  Enrico Mano

  committed
  f36b7b4

  Copy full SHA for f36b7b4

• Translate log message to english

  

  Enrico Mano

  committed
  dc36a5b

  Copy full SHA for dc36a5b

• Removed unecessary processor.

  MiniPlayer

  committed
  b21f2fe

  Copy full SHA for b21f2fe

• Removed not needed extra-jars and not needed dependencies in spark common pom (kafka-schema-registry-client and play-json_2.12)

  MiniPlayer

  committed
  d58275d

  Copy full SHA for d58275d