You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
The feature request is related to the offline batch ingestion integration with ml-commons. opensearch-project/ml-commons#2891
The S3-Source Scan currently reads the full content of S3 objects. However, for offline ML batch job processing, only object metadata (such as name and size) is required. Reading the entire content can lead to duplicate batch job executions, adding unnecessary overhead.
Describe the solution you'd like
Enhance the S3-Source Scan by introducing an option to retrieve only metadata (e.g., object name, size) instead of reading full object contents. This ensures that batch job processors receive only the required information, preventing redundant job triggers.
Additional context
This improvement would optimize offline batch ingestion while also benefiting future async processors that require metadata without processing full object content.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
The feature request is related to the offline batch ingestion integration with ml-commons. opensearch-project/ml-commons#2891
The S3-Source Scan currently reads the full content of S3 objects. However, for offline ML batch job processing, only object metadata (such as name and size) is required. Reading the entire content can lead to duplicate batch job executions, adding unnecessary overhead.
Describe the solution you'd like
Enhance the S3-Source Scan by introducing an option to retrieve only metadata (e.g., object name, size) instead of reading full object contents. This ensures that batch job processors receive only the required information, preventing redundant job triggers.
Additional context
This improvement would optimize offline batch ingestion while also benefiting future async processors that require metadata without processing full object content.
The text was updated successfully, but these errors were encountered: