Improve Directory tree performance of New Media Gallery

[nazarklovanych]

REOPENED OLD PR -> #30629

Description (*)

Scan only directories for directory tree component
The result of using glob instead of iterator with more than 500000 files in media folder

Related Pull Requests

Fixed Issues (if relevant)

1. Fixes Magento 2.4.1 Media Gallery....not ready for prime time. Fails when you have large amounts of data (images) #30596

Manual testing scenarios (*)

1. A lot of images in filesystem -> execute this file to generate empty txt files in media folder https://gist.github.com/Nazar65/61c756ce76afa8c6718bf670794d548c execute in CMD bash glob.sh
2. Set memory_limit in php to 100MB
3. Open Media Galley
4. Media gallery directory tree component renders directory tree component

Questions or comments

Contribution checklist (*)

• Pull request has a meaningful description of its purpose
• All commits are accompanied by meaningful commit messages
• All new or changed code is covered with unit/integration tests (if applicable)
• All automated tests passed successfully (all builds are green)

[m2-assistant]

Hi @nazarklovanych. Thank you for your contribution
Here are some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

• @magento give me test instance - deploy test instance based on PR changes
• @magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

• @magento run all tests - run or re-run all required tests against the PR changes
• @magento run <test-build(s)> - run or re-run specific test build(s)
  For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names. Allowed build names are:

1. Database Compare
2. Functional Tests CE
3. Functional Tests EE,
4. Functional Tests B2B
5. Integration Tests
6. Magento Health Index
7. Sample Data Tests CE
8. Sample Data Tests EE
9. Sample Data Tests B2B
10. Static Tests
11. Unit Tests
12. WebAPI Tests
13. Semantic Version Checker

You can find more information about the builds here

ℹ️ Run only required test builds during development. Run all test builds before sending your pull request for review.

For more details, review the Magento Contributor Guide documentation.

⚠️ According to the Magento Contribution requirements, all Pull Requests must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of Pull Requests happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[nazarklovanych]

@magento run all tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

[nazarklovanych]

@magento run all tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

[nazarklovanych]

@magento run all tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

[nazarklovanych]

@magento run all tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

[nazarklovanych]

@magento run all tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

[nazarklovanych]

@magento run Integration Tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

[nazarklovanych]

@magento run all tests

[magento-automated-testing]

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

[nazarklovanych]

@magento run all tests

[kimurabanyan]

@perfpcs
I commented because I found a performance issue in this PR code. It seems to be related to the problem you have.
I haven't suggested a fix for this issue. I'm not currently using Adobe Stock, so I'm using the old media gallery to work around this issue.

[perfpcs]

@kimurabanyan OK I understand you. Thanks for clarifying. I guess Magento has no one on their team that can fix this properly because most everyone I know all reverted back to old media gallery because the new one is non-functional. Sad!

[interwijs]

@perfpcs, when using the old media gallery I also experience problems in Magento 2.4.4. Images are not visible and the directory tree only shows Storage Root directory with 2 empty subdirectories with no name. When I try to upload an image I get the message: We can't upload the file to the current folder right now. Please try another folder. And when I try to add a directory I get the message: We cannot create the folder under the selected directory. So the old media gallery is also useless right now.

[perfpcs]

@interwijs I have not yet reverted to old media gallery in 2.4.4 but I will try it and see if it also is broke. Seems hard to believe if this is so that there is not more uproar from the community with no workable media gallery. And after 1.5 years of them knowing the issues, still no viable fix. Yet they are worried about merging code that does not even fix the problem. All this is hard to understand.

[pmonosolo]

Hello,

I tried this code and found the problem. First of all, the pattern used in "isPathExcluded->execute" would be as follows if it is Magento default.

 /^(wysiwyg\b(?!-)(?:\/?[^\/]+)*\/?$|catalog[\/]+category\b(?!-)(?:\/?[^\/]+)*\/?$)/

If $relativePath ='catalog', this pattern does not match, so getDirectories is recursively called in the following if statement.

https://github.com/magento/magento2/pull/35404/files#diff-f9876dbb5d6835b068311c3353b9ea2d0b535c7069bf474429822a3db8c960e9R81

The problem here is that under "catalog" there is not only a "category" but also a "product" directory. In the recursively called getDirectories, $relativePath = 'catalog/category' will not be in the same if statement, but 'catalog/product' will be in the if statement and getDirectories will be called again. In other words, getDirectories will be executed in all directories under 'catalog/product'.

This has a huge impact, and the more products we have, the more impact we will have.

UGH...it looks like this is the culprit - this folder needs to be blocked off from being searched as there's no need to access it - its completely automated by the product creation / import processes.

[sprankhub]

This PR fixed the issues for us. Thanks, @nazarklovanych!

[perfpcs]

Still broken for me on 2.4.5-p1

#30596

[sprankhub]

@perfpcs, yes, it is broken in 2.4.5-p1. However, did you try to apply the changes from this PR manually? Did you check if the changes from this PR solve the issue for you?

[perfpcs]

@sprankhub , yes I tried and am using this PR but it only partially works. Editing directory.xml results in no changes to the tree whatsoever so we can never get ONLY the WYSIWYG directory. Also we are missing many of the things that make this useful like the search box, sorting drop downs, etc.

Do you have all these things showing?

[sprankhub]

yes I tried and am using this PR but it only partially works.

Please mind that this PR is about the performance / memory issue with large media directories. If you have different issues, you might want to look into other / open new issues.

Editing directory.xml results in no changes to the tree whatsoever so we can never get ONLY the WYSIWYG directory.

This is not part of this PR.

Also we are missing many of the things that make this useful like the search box, sorting drop downs, etc.

I do see all this:

[engcom-Hotel]

Hello @nazarklovanych,

First of all, thank you for your contribution and collaboration!

We are in the process of reviewing this PR, but it seems there are some conflicts. So we request you to please resolve the conflicts so that we move further with this PR.

Meanwhile, we are moving this PR On Hold, we will pick this up when the conflicts resolve.

Thanks

[engcom-Hotel]

Hello @nazarklovanych,

Have you got a chance to resolve the conflicts of this PR?

Thanks

[engcom-Hotel]

Hi @nazarklovanych,

Thank you for your contribution!

As per the comment here #35404 (comment) we are unable to move further on this PR due to conflicts. Can you please look into it?

Till then we are closing this PR.

Please feel free to reopen or let us know when you are ready to work on it again, and we will be happy to re-open it.

Thank you!