Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support S3-Source Scan for object meta data only #5433

Open
Zhangxunmt opened this issue Feb 13, 2025 · 0 comments
Open

Support S3-Source Scan for object meta data only #5433

Zhangxunmt opened this issue Feb 13, 2025 · 0 comments

Comments

@Zhangxunmt
Copy link

Zhangxunmt commented Feb 13, 2025

Is your feature request related to a problem? Please describe.
The feature request is related to the offline batch ingestion integration with ml-commons. opensearch-project/ml-commons#2891

The S3-Source Scan currently reads the full content of S3 objects. However, for offline ML batch job processing, only object metadata (such as name and size) is required. Reading the entire content can lead to duplicate batch job executions, adding unnecessary overhead.

Describe the solution you'd like
Enhance the S3-Source Scan by introducing an option to retrieve only metadata (e.g., object name, size) instead of reading full object contents. This ensures that batch job processors receive only the required information, preventing redundant job triggers.

Additional context
This improvement would optimize offline batch ingestion while also benefiting future async processors that require metadata without processing full object content.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

No branches or pull requests

1 participant