"Index on Schedule" some products are not visible

[nuzil]

Preconditions

Magento Version : 2.2.5

ElasticSuite Version : 2.5.15
Elasticsearch: 2.4.6

Environment : Production

Steps to reproduce

0. Indexes is set on "Schedule", cron running every minute
1. Products are imported via Magento REST API -> PUT /V1/products/{SKU}
2. Some products are not visible on a frontend in categories and in search, but some are fine
3. After full reindex All products appared and works as expected

Expected result

1. All products are available on frontend

Actual result

1. Some products are not available

I want to mention, that I'm not 100% sure that problems is on Elasticsuite side, but I want to hear opinions of people where can be a problem and maybe somebody already had this problem before

[romainruaud]

They should be reindexed on the fly when using On Schedule indexing.

Are you sure they are put inside the _cl table during import ? They should, since it's managed by a DB trigger, but it could be a first point to investigate.

Also, are there any error in the /var/log/elasticsearch/elasticsearch.log file ?

[no-response]

This issue was waiting update from the author for too long.
Without any update, we are unfortunately not sure how to resolve this issue.
We are therefore reluctantly going to close this bug for now. Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away!
Thanks for your contribution.

[nuzil]

Hi @romainruaud
Thanks for the ideas. Currently we are covered it with logs, and research why sometimes its still happens.

[HannesJa]

We are currently looking into this as well, although our Magento's version is 2.1.9, and Smile Elasticsuite 2.3.11.

However, after some looking around, it seems that this is not an issue with Smile Elasticsuite, but could be with Magento's way of handling the changelogs.

I found a couple of reported issues about this, and they seem to be related. Take a look at these for example, should they be of any help:
magento/magento2#5836 (comment)

" The problem was that version_id in mview_state table had a value much higher than the autoincrement value in catalog_product_category_cl table. Everytime magento 2 cron runs it checks whether the changelogs tables have higher values than mview_state indexes (version_id). "

As for why this happens that the mview_state -table has much higher versions, than the _cl tables, this might be one reason:
magento/magento2#13577

"When cleaning ALL changelog rows, indexes get confused after a MySQL restart. The changelog version ids can start over at 1, and mview_state will be ahead of it.
This causes incremental indexing to simply not execute."

Edit: Changed Markdown formatting

[romainruaud]

great investigation @HannesJa , thank you for contributing.

@nuzil do you have additional elements ?

Regards

[no-response]

This issue was waiting update from the author for too long.
Without any update, we are unfortunately not sure how to resolve this issue.
We are therefore reluctantly going to close this bug for now. Please don't hesitate to comment on the bug if you have any more information for us; we will reopen it right away!
Thanks for your contribution.

[sedax90]

We have the same problem after imports created with FireGento_FastSimpleImport2. After the upgrade (even just prices), index start to reindex but not all products are correctly processed. It returns no errors but some products are not visible anymore. We have to forcely reindex to view all products.