Schema refresh failed with Elasticsearch v7.x

[kirkiris]

Issue Summary

After upgrading Elasticsearch to v7.x, I am getting a "Schema refresh failed" error in Create -> Query.
Schema refresh on Elasticsearch v6.x is working fine.

I have changed the logging level to DEBUG but I wasn't able to find much. The only thing I noticed is that when requesting the mappings on ES v7.x, the content_length of the response seems unreasonably small.

ES v7.x

server_1            | [2021-07-26 15:51:27,018][PID:11][DEBUG][urllib3.connectionpool] Starting new HTTPS connection (1): [HOSTNAME REDACTED]
server_1            | [2021-07-26 15:51:27,949][PID:11][DEBUG][urllib3.connectionpool] [HOSTNAME REDACTED] "GET /_mappings HTTP/1.1" 200 9992
server_1            | [2021-07-26 15:51:27,956][PID:11][INFO][metrics] method=GET path=/api/data_sources/1/schema endpoint=datasourceschemaresource status=200 content_type=application/json content_length=61 duration=948.26 query_count=6 query_duration=7.81

ES v6.x

server_1            | [2021-07-26 15:54:15,568][PID:11][DEBUG][urllib3.connectionpool] Starting new HTTPS connection (1): [HOSTNAME REDACTED]
server_1            | [2021-07-26 15:54:16,140][PID:11][DEBUG][urllib3.connectionpool] [HOSTNAME REDACTED] "GET /_mappings HTTP/1.1" 200 19621
server_1            | [2021-07-26 15:54:16,162][PID:11][INFO][metrics] method=GET path=/api/data_sources/2/schema endpoint=datasourceschemaresource status=200 content_type=application/json content_length=113140 duration=606.12 query_count=6 query_duration=7.41

I wasn't able to find anything related to this issue neither in Redash discussion forums nor in GitHub.

Steps to Reproduce

1. Create an Elasticsearch datadource (ES v7.x)
2. Open Query UI
3. Wait Schema Refresh
4. Schema Refresh Failed

Technical details:

• Redash Version: 8.0.0.b32245
• Browser/OS: Chrome/macOS Big Sur
• How did you install Redash: docker-compose

[susodapop]

The ES query runner is deeply buggy. Rewrite effort is here. I'm interested to know if the changes in that version will resolve your issue.

[kirkiris]

Thanks for your reply!

I've tried your suggestion and, apart from some minor UI bugs, it seems to be working (at least the schema refresh part).