EZP-31299: Added language mask to LSE search index #2940

michal-myszka · 2020-02-04T13:06:21Z

Question	Answer
JIRA issue	EZP-31299
Bug/Improvement	yes
New feature	yes
Target version	`7.5`
BC breaks	no
Tests pass	yes
Doc needed	no

This PR's changes improvement searching in languages (in translations).

Updated indexing process for legacy SQL search (added language mask to words links)
Updated SQL search engine for searching in translations from provided languages list
Provided search language filter into ContentSearchHitAdapter
Added SQL migration for ezsearch_object_word_link table

Related changes in
ezsystems/ezplatform-admin-ui#1223
ezsystems/ezplatform-solr-search-engine#168

TODO:

Implement feature / fix a bug.
Implement tests.
Fix new code according to Coding Standards ($ composer fix-cs).
Ask for Code Review.
Merge first EZP-31299: Changed condition for "useAlwaysAvailable" ezplatform-solr-search-engine#168 (for tests pass)

eZ/Publish/Core/Pagination/Pagerfanta/ContentSearchHitAdapter.php

…to admin UI)

…xing)

eZ/Publish/Core/Search/Legacy/Content/Common/Gateway/CriterionHandler/FullText.php

…Handler/FullText.php Co-Authored-By: André R. <[email protected]>

data/mysql/schema.sql

data/update/mysql/dbupdate-7.5.5-to-7.5.6.sql

data/mysql/schema.sql

eZ/Publish/API/Repository/Tests/SearchServiceTest.php

eZ/Publish/Core/Search/Legacy/Content/Common/Gateway/CriterionHandler/FullText.php

eZ/Publish/Core/Search/Legacy/Content/Mapper/FullTextMapper.php

alongosz

@michal-myszka I've missed one more thing (seems the scripts for the previous release miss the same thing, but out of scope here):

data/update/mysql/dbupdate-7.5.6-to-7.5.7.sql

data/update/postgres/dbupdate-7.5.6-to-7.5.7.sql

Co-Authored-By: Andrew Longosz <[email protected]>

eZ/Publish/API/Repository/Tests/SearchServiceTest.php

adamwojs · 2020-02-18T08:42:45Z

eZ/Publish/API/Repository/Tests/SearchServiceTest.php

+        usort(
+            $searchHits,
+            function ($a, $b) {
+                return ($a->valueObject->id < $b->valueObject->id) ? -1 : 1;


You can use the spaceship operator here. This opportunity may not happen again 😉

Used (hurray!) :)

eZ/Publish/API/Repository/Tests/SearchServiceTest.php

adamwojs · 2020-02-18T09:20:41Z

eZ/Publish/Core/Search/Legacy/Content/WordIndexer/Gateway/DoctrineDatabase.php

@@ -136,6 +143,8 @@ public function index(FullTextData $fullTextData)
                    'ContentClassAttributeID' => $fullTextValue->fieldDefinitionId,
                    'identifier' => $fullTextValue->fieldDefinitionIdentifier,
                    'integer_value' => $integerValue,
+                    'language_code' => $fullTextValue->languageCode,
+                    'is_main_and_always_available' => $fullTextValue->isMainAndAlwaysAvailable,


The language_code / is_main_and_always_available values needs to added to

ezpublish-kernel/eZ/Publish/Core/Search/Legacy/Content/WordIndexer/Gateway/DoctrineDatabase.php

Lines 155 to 158 in 4d46223

$indexArray[] = ['Word' => $additionalUrlWord,

'ContentClassAttributeID' => $fullTextValue->fieldDefinitionId,

'identifier' => $fullTextValue->fieldDefinitionIdentifier,

'integer_value' => $integerValue, ];

as well.

Right, added

Co-Authored-By: Adam Wójs <[email protected]>

adamwojs · 2020-02-24T07:43:58Z

eZ/Publish/API/Repository/Tests/SearchServiceTest.php

+        bool $useAlwaysAvailable
+    ): SearchResult {
+        if (false === in_array($findMethod, self::AVAILABLE_FIND_METHODS, true)) {
+            throw new \InvalidArgumentException(


Nitpick: missing import and @throws docblock

adamwojs · 2020-02-24T07:44:41Z

eZ/Publish/Core/Pagination/Tests/ContentSearchAdapterTest.php

@@ -17,12 +17,13 @@ class ContentSearchAdapterTest extends ContentSearchHitAdapterTest
    /**
     * @param Query $query
     * @param SearchService $searchService
+     * @param array


Suggested change

* @param array

* @param array $languageFilter

adamwojs · 2020-02-24T07:45:34Z

eZ/Publish/Core/Pagination/Tests/ContentSearchHitAdapterTest.php

@@ -37,9 +37,9 @@ protected function setUp()
     *
     * @return ContentSearchHitAdapter
     */
-    protected function getAdapter(Query $query, SearchService $searchService)
+    protected function getAdapter(Query $query, SearchService $searchService, array $languageFilter = [])


Nitpick: missing @param for $languageFilter

adamwojs · 2020-02-24T07:53:08Z

eZ/Publish/Core/Pagination/Pagerfanta/ContentSearchHitAdapter.php

@@ -65,7 +77,7 @@ public function getSlice($offset, $length)
        $query->limit = $length;
        $query->performCount = false;

-        $searchResult = $this->searchService->findContent($query);
+        $searchResult = $this->searchService->findContent($query, $this->languageFilter);


Correct me if I'm wrong, but it seems that we are missing an assertion ensuring that language Filter is passed to findContent ContentSearchHitAdapterTest.

alongosz · 2020-02-24T08:41:09Z

eZ/Publish/API/Repository/Tests/SearchServiceTranslationLanguageFallbackTest.php

+
+        $setupFactory = $this->getSetupFactory();
+
+        if ($setupFactory instanceof LegacySolrSetupFactory) {


Hmm, this block would look better in a separate data provider.

adamwojs

It's fine from my POV 😉

katarzynazawada · 2020-02-28T12:17:13Z

Tests passed in https://github.com/ezsystems/ezplatform-page-builder/pull/516

lserwatka · 2020-02-28T13:24:37Z

Could you merge it up?

EZP-31299 - Provided search language filter into ContentSearchHitAdapter

0458275

michal-myszka mentioned this pull request Feb 4, 2020

EZP-31299: Added 'search in specific language' feature ezsystems/ezplatform-admin-ui#1223

Merged

8 tasks

michal-myszka requested a review from adamwojs February 4, 2020 13:08

andrerom reviewed Feb 4, 2020

View reviewed changes

eZ/Publish/Core/Pagination/Pagerfanta/ContentSearchHitAdapter.php Outdated Show resolved Hide resolved

adamwojs requested changes Feb 4, 2020

View reviewed changes

eZ/Publish/Core/Pagination/Pagerfanta/ContentSearchHitAdapter.php Outdated Show resolved Hide resolved

Moved "displayLanguage" property from ContentSearchHitAdapter (moved …

ee5488c

…to admin UI)

andrerom approved these changes Feb 8, 2020

View reviewed changes

Michał Myszka added 2 commits February 11, 2020 15:43

Added translation context for FullText legacy (SQL) search (inc. inde…

4d46223

…xing)

CS-fix

f722948

andrerom reviewed Feb 11, 2020

View reviewed changes

eZ/Publish/Core/Search/Legacy/Content/Common/Gateway/CriterionHandler/FullText.php Outdated Show resolved Hide resolved

Michał Myszka and others added 6 commits February 12, 2020 10:33

Fix SQL for Postgres

2afafce

Add migration

592a369

Added tests

7a0fa21

Update eZ/Publish/Core/Search/Legacy/Content/Common/Gateway/Criterion…

7686adf

…Handler/FullText.php Co-Authored-By: André R. <[email protected]>

Fix cs

8c75a86

Added 'language_mask' to SQL schemas

fbc3138

michal-myszka requested review from ViniTou and mikadamczyk February 14, 2020 08:57

michal-myszka changed the title ~~EZP-31299 - Provided search language filter into ContentSearchHitAdapter~~ EZP-31299 - Search in languages Feb 17, 2020

alongosz requested changes Feb 17, 2020

View reviewed changes

data/mysql/schema.sql Show resolved Hide resolved

alongosz requested changes Feb 17, 2020

View reviewed changes

alongosz changed the title ~~EZP-31299 - Search in languages~~ EZP-31299: Added language mask to LSE search index Feb 17, 2020

Michał Myszka added 2 commits February 17, 2020 15:12

After CR changes

31b1c82

After CR changes

f5eb735

alongosz reviewed Feb 17, 2020

View reviewed changes

data/update/mysql/dbupdate-7.5.6-to-7.5.7.sql Show resolved Hide resolved

data/update/postgres/dbupdate-7.5.6-to-7.5.7.sql Show resolved Hide resolved

michal-myszka and others added 2 commits February 17, 2020 16:33

Update data/update/mysql/dbupdate-7.5.6-to-7.5.7.sql

a5c1e64

Co-Authored-By: Andrew Longosz <[email protected]>

Update data/update/postgres/dbupdate-7.5.6-to-7.5.7.sql

960a468

Co-Authored-By: Andrew Longosz <[email protected]>

alongosz approved these changes Feb 18, 2020

View reviewed changes

alongosz requested a review from adamwojs February 18, 2020 08:55

adamwojs reviewed Feb 18, 2020

View reviewed changes

michal-myszka and others added 5 commits February 18, 2020 10:23

Update eZ/Publish/API/Repository/Tests/SearchServiceTest.php

91dfe8f

Co-Authored-By: Adam Wójs <[email protected]>

Update eZ/Publish/API/Repository/Tests/SearchServiceTest.php

afb318d

Co-Authored-By: Adam Wójs <[email protected]>

After CR changes

edced1a

Updated tests

fd5ef69

Additional test data limitation only for Solr

5d0fe77

adamwojs reviewed Feb 24, 2020

View reviewed changes

adamwojs requested review from alongosz and andrerom February 24, 2020 07:54

After CR changes

792a8be

alongosz reviewed Feb 24, 2020

View reviewed changes

adamwojs approved these changes Feb 24, 2020

View reviewed changes

andrerom approved these changes Feb 24, 2020

View reviewed changes

michal-myszka self-assigned this Feb 24, 2020

michal-myszka added the Ready for QA label Feb 26, 2020

alongosz mentioned this pull request Feb 28, 2020

[Poc] EZP-27761: Translations are not being indexed/found by Solr #2074

Closed

katarzynazawada approved these changes Feb 28, 2020

View reviewed changes

katarzynazawada added QA approved and removed Ready for QA labels Feb 28, 2020

lserwatka merged commit 557bdfe into 7.5 Feb 28, 2020

lserwatka deleted the ezp-31299-additional-search-languages-filter branch February 28, 2020 13:24

michal-myszka mentioned this pull request Feb 28, 2020

EZP-31299: Fixed tests after merge #2940 #2962

Merged

1 task

lserwatka pushed a commit that referenced this pull request Mar 3, 2020

EZP-31299: Fixed tests after merge #2940 (#2962)

86fc614

ViniTou mentioned this pull request Mar 4, 2020

[MERGED AUTOMATICALLY] Run for travis when merging up 7.5 #2969

Merged

michal-myszka mentioned this pull request Mar 5, 2020

EZP-31299: [Tests] Added missing language mask to initial test data #2973

Merged

alongosz mentioned this pull request Mar 12, 2020

EZP-31478: Dropped LSE FullText index from clean data #2984

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EZP-31299: Added language mask to LSE search index #2940

EZP-31299: Added language mask to LSE search index #2940

michal-myszka commented Feb 4, 2020 •

edited

Loading

alongosz left a comment

adamwojs Feb 18, 2020

michal-myszka Feb 18, 2020

adamwojs Feb 18, 2020

michal-myszka Feb 18, 2020

adamwojs Feb 24, 2020

adamwojs Feb 24, 2020

adamwojs Feb 24, 2020

adamwojs Feb 24, 2020

alongosz Feb 24, 2020

adamwojs left a comment •

edited

Loading

katarzynazawada commented Feb 28, 2020

lserwatka commented Feb 28, 2020

	$indexArray[] = ['Word' => $additionalUrlWord,
	'ContentClassAttributeID' => $fullTextValue->fieldDefinitionId,
	'identifier' => $fullTextValue->fieldDefinitionIdentifier,
	'integer_value' => $integerValue, ];


		$setupFactory = $this->getSetupFactory();

		if ($setupFactory instanceof LegacySolrSetupFactory) {

EZP-31299: Added language mask to LSE search index #2940

EZP-31299: Added language mask to LSE search index #2940

Conversation

michal-myszka commented Feb 4, 2020 • edited Loading

alongosz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamwojs left a comment • edited Loading

Choose a reason for hiding this comment

katarzynazawada commented Feb 28, 2020

lserwatka commented Feb 28, 2020

michal-myszka commented Feb 4, 2020 •

edited

Loading

adamwojs left a comment •

edited

Loading