Webanalytic bug #566
Merged
Webanalytic bug #566
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…plugin module is now installed without repackaging and with repackaging.
…her implementation to solve some problems.
…il without commit of the offsets.
… code is more readable. (Ensured all test that were green, still are)
…ests. Implemented queryGet and multiQueryGet methods in Es 7 client service.
…b sessions indices. Fixed some issues in elasticsearch queries.
…sed to construct boolQuery in elasticsearch7 service. Fixed with that the bug when deleting future sessions in IncrementalWebSession. Moved model stuff of datastore in logisland-api module to model package.
…ssionCheckerResults too from anonymous instance to real classes.
…o choose Timezone to use for format. Otherwise use system default.
…zation. toString method of StandardRecord is pretty for all types !
… where to put index names. By default 'es_index' and 'es_type'. IncrementalWebSession does not save events into es anymore. It return both sessions and events as record instead so that they can be injected into elasticsearch with the BulkAddElasticsearch.
…ter to ensure there is not too much rewind. Thoses tests are currently not passing.
…tion about it in wiki.
…tion about it in wiki.
… the right records. Writer to inject in sink from records and the stream apply transformations with processors.
…X server. Keeping the other module using kafka 2 for later usage (when we update engine to use kafka 2).
…ming in spark 2_4_common_engines module that was conflicting when testing in a module. Anyway this is embedded in spark_core libraries depending on spark version used.
…y know what properties is used in each stream or providers.
…f ControllerServiceCOnfiguration. Fixed a bug in Addfields, dynamic props did not support empty values. Fixed a bug in DATE_TIME_FORMATTER_VALIDATOR, error were not properly catched.
…to use group by in spark streaming.
…viceConfiguration instead of context which impose services to be serializable.
…t' into webanalyticBug # Conflicts: # logisland-core/logisland-engines/logisland-engine-spark_2_X/logisland-engine-spark_2_common/src/main/scala/com/hurence/logisland/engine/spark/KafkaStreamProcessingEngine.scala # logisland-core/logisland-engines/logisland-engine-spark_2_X/logisland-engine-spark_2_common/src/main/scala/com/hurence/logisland/stream/spark/AbstractKafkaRecordStream.scala # logisland-core/logisland-engines/logisland-engine-spark_2_X/logisland-engine-spark_2_common/src/main/scala/com/hurence/logisland/stream/spark/KafkaRecordStreamParallelProcessing.scala # logisland-core/logisland-engines/logisland-engine-spark_2_X/logisland-engine-spark_2_common/src/main/scala/com/hurence/logisland/stream/spark/package.scala # logisland-core/logisland-engines/logisland-engine-spark_2_X/logisland-engine-spark_2_common/src/main/scala/com/hurence/logisland/stream/spark/structured/StructuredStream.scala # logisland-core/logisland-engines/logisland-engine-spark_2_X/logisland-engine-spark_2_common/src/main/scala/com/hurence/logisland/stream/spark/structured/provider/KafkaStructuredStreamProviderService.scala
…mmon pom (kafka-schema-registry-client and play-json_2.12)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixed a bug in incrementalWebSession with high frequency batch. (Added refresh on index of mapping and sessions).
Added an integration test about it.
Modified packaging of plugins so that we can also just import it from another project.