[DOC] Misleading and unclear documentation for the Spark Connector in the SQL/PPL docs

[salyh]

What do you want to do?

• Request a change to existing documentation
• Add new documentation
• Report a technical problem with the documentation
• Other

Tell us about your request.
Regarding: https://opensearch.org/docs/latest/search-plugins/sql/settings/#spark-connector-settings

• The Spark connector is, according to this comment only supporting AWS EMR Serverless Spark (which means I need to have AWS credentials). This should be made clear in the docs.

• The docs lacks examples how to setup EMR Serverless Spark and OpenSearch and where to provide the configuration (like spark.uri). For an user its unclear how to setup a basic working example.

• Some of the config properties lacks examples and the info which values are valid:

  • spark.uri "The identifier for your Spark data source." is misleading, lacks example and what the default is and wether its mandatory
  • spark.auth.typeIts unclear which values are valid and what the default is and wether its mandatory

• The spark connector docs lacks an reference to https://opensearch.org/docs/latest/dashboards/management/data-sources/ (and potentially https://opensearch.org/docs/latest/dashboards/management/accelerate-external-data/) and an explanation and examples how to add spark as a datasource

• The docs are not coherent with https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/connectors/spark_connector.rst

  • emr.cluster is missing for example

• The ppl example is unclear

POST /_plugins/_ppl
content-type: application/json
{
   "query": "source = my_spark.sql('select * from alb_logs')"
}

To what is my_spark referring to?

Version:
all since Spark connector is supported

What other resources are available?

• Add Docker-compose setting for IT testing opensearch-spark#606 (comment)
• [RFC] OpenSearch and Apache Spark Integration opensearch-spark#4 (comment)
• [FEATURE] Object Storage (S3) Data Ingestion through Streaming Query sql#948 (comment)
• OpenSearch and Spark Integration P1 Demo opensearch-spark#317
• https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst

[Naarcha-AWS]

@salyh: Thanks for submitting this issue! I'll find a dev who can help make the changes you requested.

[dblock]

[Catch All Triage - 1, 2, 3, 4]

[salyh]

@YANG-DB @Naarcha-AWS any update? We need to clarify this to get opensearch-project/opensearch-spark#606 done