EZP-30463: As an Developer, I want to use Solr Cloud #137

adamwojs · 2019-04-18T11:22:47Z

JIRA: https://jira.ez.no/browse/EZP-30463

Description

This PR introduces support for Solr Cloud. More information https://lucene.apache.org/solr/guide/6_6/solrcloud.html

Configuration

From now on the user can specify data distribution strategy for connection via distribution_strategy option. The possible values are:

standalone - https://lucene.apache.org/solr/guide/6_6/legacy-scaling-and-distribution.html (current implementation)
cloud - Solr Cloud

The default value is standalone for backward compatibility reasons.

Example Solr Cloud configuration :

ez_search_engine_solr:
    endpoints:
        main:
            dsn: '%solr_dsn%'
            core: '%solr_main_core%' 
        en:
            dsn: '%solr_dsn%'
            core: '%solr_en_core%'
        fr:
            dsn: '%solr_dsn%'
            core: '%solr_fr_core%'
        # ...
    connections:
        default:
            distribution_strategy: cloud
            entry_endpoints:
                - main
                - en
                - fr
             # -  ...
            mapping:
                translations:
                    eng-GB: en
                    fre-FR: fr
                    # ...
                main_translations: main

Cluster screenshot:

Document routing

This solution uses the default Solr Cloud document routing strategy: compositeId. In compare to implicit strategy eZ Platform doesn't need to know shards list.

More information: https://lucene.apache.org/solr/guide/6_6/shards-and-indexing-data-in-solrcloud.html#ShardsandIndexingDatainSolrCloud-DocumentRouting

Language specific analysis

The configuration is based on Multi-Core setup so any specific language analysis options could be specified on the collection level.

andrerom · 2019-04-18T13:07:32Z

☁️FTW😉

bin/.travis/init_solr.sh

bundle/DependencyInjection/Configuration.php

bin/.travis/init_solr.sh

andrerom · 2019-05-02T07:50:25Z

.travis.yml

@@ -18,7 +18,8 @@ matrix:
          env: TEST_CONFIG="phpunit-integration-legacy-solr.xml" SOLR_VERSION="6.5.1"  CORES_SETUP="shared"
        - php: 7.2
          env: TEST_CONFIG="phpunit-integration-legacy-solr.xml" SOLR_VERSION="6.6.5"  CORES_SETUP="single" SOLR_CORES="collection1"
-
+        - php: 7.2
+          env: TEST_CONFIG="phpunit-integration-legacy-solr.xml" SOLR_VERSION="6.6.5"  CORES_SETUP="cloud" SOLR_CORES="collection1" SOLR_CLOUD="yes"


hmm, based on logic in getTestConfigurationFile it does not seems like we need both CORES_SETUP and SOLR_CLOUD. CORES_SETUP=cloud seems like it could be detected anywhere needed.

But maybe logic in getTestConfigurationFile is temprary (simple for now) and you intend to allow several different cloud configs right?

bundle/DependencyInjection/EzSystemsEzPlatformSolrSearchEngineExtension.php

lib/Gateway/DistributionStrategy/DocumentRouter/NullDocumentRouter.php

adamwojs · 2019-05-22T12:53:38Z

PR has been updated with significant changes so please re-review 😉

kmadejski

Just one small thing which we discussed before. It seems that it has to be covered in this place.

Besides, looks good to me, great job! 🙂

kmadejski · 2019-05-22T13:17:11Z

bin/.travis/init_solr.sh

+    copy_files ${TEMPLATE_DIR} "${files[*]}"
+
+    # modify solrconfig.xml to remove section that doesn't agree with our schema
+    sed -i.bak '/<updateRequestProcessorChain name="add-unknown-fields-to-the-schema">/,/<\/updateRequestProcessorChain>/d' "${TEMPLATE_DIR}/solrconfig.xml" || exit_on_error "Can't modify file '${TEMPLATE_DIR}/solrconfig.xml'"


I think you have to add this line (https://github.com/ezsystems/ezplatform-solr-search-engine/blob/master/bin/generate-solr-config.sh#L123) here as well even though tests are passing. Without it, you'll notice a significant delay before seeing content changes (eg. breadcrumbs in AdminUI).

Applied in 260c729

andrerom · 2019-05-23T17:49:13Z

bin/.travis/init_solr.sh

 download

+if [ "$SOLR_CLOUD" = "no" ]; then
+    $SCRIPT_DIR/../generate-solr-config.sh \


@adamwojs This script is used to generate config for platform.sh, which is kind of a must. So most likely many of the changes here should be moved into generate-solr-config.s, so script will be able to continued to be used also with cloud config on platform.sh.

@vidarl might be able to provide more info, and also who to talk to there if needed.

You might be right, but let's process with combination platform.sh + Solr Cloud as follow up for this PR. I want to unblock QA on this PR and platform.sh itself is a quite new topic for me 😉

andrerom · 2019-05-23T17:51:26Z

composer.json

@@ -14,7 +14,7 @@
    "prefer-stable": true,
    "require": {
        "php": "^7.1",
-        "ezsystems/ezpublish-kernel": "^8.0@dev",
+        "ezsystems/ezpublish-kernel": "dev-ezp_30463_solr_cloud as 7.5",


Suggested change

"ezsystems/ezpublish-kernel": "dev-ezp_30463_solr_cloud as 7.5",

"ezsystems/ezpublish-kernel": "^7.5.2@dev || ^8.0@dev",

Once kernel pr is merged.

alongosz · 2019-06-05T08:13:10Z

@adamwojs If this is to work with eZ Platform 2.5 LTS you need to rebase against the latest stable branch, master is for 3.0.

m-tyrala

Cloud seems to work fine in various configurations of nodes, shards, and cores. Searching can be performed for various languages (japanese, polish, french, german, english). Searching still works after deleting the shard replica. After deleting Leader replica, another one is taking its place automatically. Documents are distributed quasi equally (distribution scales well with the number of documents).

QA approves

andrerom

+1 But please push 1.7 branch based on 1.6 + composer changes and rebase this on that before merging, as @alongosz said master is 3.0 only now and we need this on 2.5.

adamwojs · 2019-06-21T12:49:44Z

PR has been rebased against the new 1.7 branch (also temporary composer.json changes has been reverted). Last successful CI build https://travis-ci.org/ezsystems/ezplatform-solr-search-engine/builds/548644319

kmadejski · 2019-06-21T12:51:18Z

Good job @adamwojs! It's cool to see this feature 🎉

adamwojs added the Feature label Apr 18, 2019

adamwojs self-assigned this Apr 18, 2019

andrerom suggested changes Apr 29, 2019

View reviewed changes

bin/.travis/init_solr.sh Outdated Show resolved Hide resolved

bundle/DependencyInjection/Configuration.php Outdated Show resolved Hide resolved

andrerom reviewed Apr 29, 2019

View reviewed changes

bin/.travis/init_solr.sh Outdated Show resolved Hide resolved

andrerom reviewed May 2, 2019

View reviewed changes

andrerom reviewed May 6, 2019

View reviewed changes

bundle/DependencyInjection/EzSystemsEzPlatformSolrSearchEngineExtension.php Outdated Show resolved Hide resolved

lib/Gateway/DistributionStrategy/DocumentRouter/NullDocumentRouter.php Outdated Show resolved Hide resolved

adamwojs force-pushed the ezp_30463 branch 2 times, most recently from a1818cf to 7a2d376 Compare May 8, 2019 06:51

adamwojs changed the title ~~[WIP] EZP-30463: As an Developer, I want to use Solr Cloud~~ EZP-30463: As an Developer, I want to use Solr Cloud May 8, 2019

adamwojs requested review from alongosz and kmadejski May 8, 2019 06:51

This was referenced May 8, 2019

EZP-26807: First implementation for SolrCloud support #86

Closed

[WIP]Solr cloud support 2.0 rebased on recent master #127

Closed

adamwojs force-pushed the ezp_30463 branch from 7a2d376 to 73cb349 Compare May 22, 2019 11:35

kmadejski approved these changes May 22, 2019

View reviewed changes

andrerom reviewed May 23, 2019

View reviewed changes

adamwojs added Ready for QA and removed Ready for QA labels May 28, 2019

m-tyrala approved these changes Jun 12, 2019

View reviewed changes

andrerom approved these changes Jun 13, 2019

View reviewed changes

m-tyrala added QA Approved and removed Ready for QA labels Jun 19, 2019

adamwojs changed the base branch from master to 1.7 June 21, 2019 11:34

EZP-30463: As an Developer, I want to use Solr Cloud

f407bba

adamwojs force-pushed the ezp_30463 branch from 260c729 to 0e748c5 Compare June 21, 2019 11:46

adamwojs force-pushed the ezp_30463 branch from 0e748c5 to f407bba Compare June 21, 2019 12:42

adamwojs merged commit af22427 into 1.7 Jun 21, 2019

adamwojs deleted the ezp_30463 branch June 21, 2019 12:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EZP-30463: As an Developer, I want to use Solr Cloud #137

EZP-30463: As an Developer, I want to use Solr Cloud #137

adamwojs commented Apr 18, 2019 •

edited

Loading

andrerom commented Apr 18, 2019

andrerom May 2, 2019

adamwojs commented May 22, 2019

kmadejski left a comment •

edited

Loading

kmadejski May 22, 2019

adamwojs May 28, 2019

andrerom May 23, 2019 •

edited

Loading

adamwojs May 28, 2019

andrerom May 23, 2019

alongosz commented Jun 5, 2019

m-tyrala left a comment •

edited

Loading

andrerom left a comment •

edited

Loading

adamwojs commented Jun 21, 2019

kmadejski commented Jun 21, 2019

	"ezsystems/ezpublish-kernel": "dev-ezp_30463_solr_cloud as 7.5",
	"ezsystems/ezpublish-kernel": "^7.5.2@dev \|\| ^8.0@dev",

EZP-30463: As an Developer, I want to use Solr Cloud #137

EZP-30463: As an Developer, I want to use Solr Cloud #137

Conversation

adamwojs commented Apr 18, 2019 • edited Loading

Description

Configuration

Document routing

Language specific analysis

andrerom commented Apr 18, 2019

andrerom May 2, 2019

Choose a reason for hiding this comment

adamwojs commented May 22, 2019

kmadejski left a comment • edited Loading

Choose a reason for hiding this comment

kmadejski May 22, 2019

Choose a reason for hiding this comment

adamwojs May 28, 2019

Choose a reason for hiding this comment

andrerom May 23, 2019 • edited Loading

Choose a reason for hiding this comment

adamwojs May 28, 2019

Choose a reason for hiding this comment

andrerom May 23, 2019

Choose a reason for hiding this comment

alongosz commented Jun 5, 2019

m-tyrala left a comment • edited Loading

Choose a reason for hiding this comment

andrerom left a comment • edited Loading

Choose a reason for hiding this comment

adamwojs commented Jun 21, 2019

kmadejski commented Jun 21, 2019

adamwojs commented Apr 18, 2019 •

edited

Loading

kmadejski left a comment •

edited

Loading

andrerom May 23, 2019 •

edited

Loading

m-tyrala left a comment •

edited

Loading

andrerom left a comment •

edited

Loading