Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return Optional.empty() instead of rethrowing exception and catch RuntimeException when submiting #824

Merged
merged 5 commits into from
Jan 11, 2024

Conversation

edvaone
Copy link
Contributor

@edvaone edvaone commented Jan 9, 2024

At the moment we are rethrowing the exception in cases there were issues in getting Application ID from the YarnBackend. We should instead return optional empty and just log the exception in order to allow calling code to decide what to do if application Id could not be fetched.

Also we need to add RuntimeException which is thrown sometimes by Hadoop/Spark libs:

java.lang.RuntimeException: Exception in thread "main" java.io.IOException: Could not get block locations. Source file "/user/xxxxx/.sparkStaging/application_1698161930067_305157/xxxxx" - Aborting...block==null
        at org.apache.spark.launcher.OutputRedirector.redirect(OutputRedirector.java:67)
        at java.base/java.lang.Thread.run(Unknown Source)
...

@Minutis Minutis changed the title Save DEAD state if Yarn client throws exception Return Optional.empty() instead of rethrowing exception and catch RuntimeException when submiting Jan 10, 2024
Add specific case for wrapped IOException
Add specific case for wrapped IOException
Add specific case for wrapped IOException
@Minutis Minutis merged commit 1ab90cf into master Jan 11, 2024
2 checks passed
@Minutis Minutis deleted the server/handle-yarn-io-exceptions branch January 11, 2024 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants