Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Opensearch Webcrawl feature #1615

Closed
gautamkct opened this issue Nov 28, 2021 · 6 comments
Closed

Opensearch Webcrawl feature #1615

gautamkct opened this issue Nov 28, 2021 · 6 comments
Labels
discuss Issues intended to help drive brainstorming and decision making feature New feature or request

Comments

@gautamkct
Copy link

Hi Team,

Wondering if opensearch support Web crawling out of box or any plans to inlcude in near future.

Thanks
Gautam Sharma
@setiah
Copy link
Contributor

setiah commented Nov 28, 2021

It doesn't today, but that's a great feature ask. Having a crawler to automatically pull new content and ingests in an OpenSearch cluster would be very useful for website search use case and compliment #1219 nicely

@setiah setiah added the discuss Issues intended to help drive brainstorming and decision making label Nov 28, 2021
@gautamkct
Copy link
Author

Thanks @setiah for adding this to 1219

@gautamkct
Copy link
Author

Hi Team, Could you please let us know if you are considering this as feature request

@dblock
Copy link
Member

dblock commented Apr 11, 2022

Feels like an entire new plugin/product. Elastic does have a web crawler/app search product.

AFAIK nobody is working on this, feel free to pick it up!

@dblock dblock added the feature New feature or request label Apr 11, 2022
@smacrakis
Copy link

The open source StormCrawler web crawler was ported to OpenSearch a few months ago.
DigitalPebble, the authors of StormCrawler, are an OpenSearch partner and provide consulting services if you need them.

@dblock
Copy link
Member

dblock commented May 30, 2023

I'm going to close this since at least some solutions exist and I don't think crawling should be a feature of core. Thanks everyone!

@dblock dblock closed this as completed May 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
discuss Issues intended to help drive brainstorming and decision making feature New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dblock @gautamkct @setiah @smacrakis