vdk-impala: Handle errors on refresh/invalidate metadata

[doks5]

Currently, if a data job executes a query, which fails due to metadata issue, the ImpalaErrorHandler
logic tries to refresh or invalidate the metadata, and re-try the query. If the job has only read
access to the table, an authorization error is raised and the job failes with User Error. This should
not happen, as the refresh/invalidate operation is initiated by vdk, not the user.

This change updates the error handler logic by adding try/except blocks around refresh/invalidate
metadata operations, so that errors that arise from these operations do not cause job failures.

Testing Done: Added test

[antoniivanov]

I do not fully comprehend the problem and it clashes of my (independent) understanding of the overall problem.

Related problems I know about are

A)Impala is eventually consistent and sometimes raises AuthorizationException even if the data job user has been authorized recently. So re-trying with back-off makes sense while waiting for new authorization rules to sync

B)Impala recovery mechanism when querying a table and it detects that the table metadata is out of sync, tries to execute invalidate metadata on that table (to sync it) before retrying the original query. But if it doesn't have "write" permission to that table, invalidate metadata fails. No amount of re-trying would prevent that since the permission would not change. It still would not have "write" permission.

The code seems like a solution for A) but the description seems more like B) So I am confused.

Is there option C I am not seeing ?

[doks5]

I do not fully comprehend the problem and it clashes of my (independent) understanding of the overall problem.

Related problems I know about are

A)Impala is eventually consistent and sometimes raises AuthorizationException even if the data job user has been authorized recently. So re-trying with back-off makes sense while waiting for new authorization rules to sync

B)Impala recovery mechanism when querying a table and it detects that the table metadata is out of sync, tries to execute invalidate metadata on that table (to sync it) before retrying the original query. But if it doesn't have "write" permission to that table, invalidate metadata fails. No amount of re-trying would prevent that since the permission would not change. It still would not have "write" permission.

The code seems like a solution for A) but the description seems more like B) So I am confused.

Is there option C I am not seeing ?

Hi, @antoniivanov

When an authorization error is raised by impala, the message gives indication what type of operation has been attempted.
The case that I'm trying to attempt at solving is when a data job, that has been granted only read access to a table, executes a SELECT * FROM some_table query, the query fails with metadata inconsistency error (maybe some other job which has write access has modified the table just before the "select" query was issued), and the error handling tries to refresh/invalidate the metadata. Because the job has only read access, the refresh/invalidate query would fail with authorization error and trigger a job failure. By just sleeping for a bit and re-trying the initial select query, we can avoid job failures if, in the meantime, the metadata has managed to sync across the impala coordinators.

[doks5]

Hi, @antoniivanov

I've addressed the feedback and updated the comments and PR description.

[antoniivanov]

Looks good to me. Thanks.

I updated the title. The prefix is just the name of the plugin (vdk-impala) without [vdk-plugins]