Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[🍒][PLUGIN-1855] Add DBErrorDetailsProvider #1919

Merged
merged 1 commit into from
Jan 27, 2025
Merged

[🍒][PLUGIN-1855] Add DBErrorDetailsProvider #1919

merged 1 commit into from
Jan 27, 2025

Conversation

psainics
Copy link
Contributor

@psainics psainics commented Jan 27, 2025 •
edited by jira bot
Loading

🍒 [cherrypick]

Commits :

PR:

ErrorDetailsProvider - Generic Database [Source|Sink] plugin

Jira : PLUGIN-1855

Description

Implement Program Failure Exception Handling in Generic Database Source/Sink plugin to catch known errors

Code change

  • Added DBErrorDetailsProvider.java
  • Modified StructuredRecordUtils.java
  • Modified DBSink.java
  • Modified DBSource.java
  • Modified DBUtils.java

Tests [ MySql ]

  • Test Case (PK Violation)
Raw Logs 

2025-01-27 13:46:49,670 - ERROR [Executor task launch worker for task 0.0 in stage 0.0 (TID 0):o.a.s.u.Utils@98] - Aborting task
io.cdap.cdap.api.exception.WrappedStageException: Stage 'Database2' encountered : io.cdap.cdap.api.exception.ProgramFailureException: Error occurred in the phase: 'Writing' with sqlState: '23000', errorCode: '1062', errorMessage: Duplicate entry '5' for key 'PRIMARY'
  at io.cdap.cdap.etl.common.ErrorDetails.handleException(ErrorDetails.java:77)
  at io.cdap.cdap.etl.spark.io.StageTrackingRecordWriter.close(StageTrackingRecordWriter.java:67)
  at org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.closeWriter(SparkHadoopWriter.scala:373)
  at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$executeTask$1(SparkHadoopWriter.scala:145)
  at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
  at org.apache.spark.internal.io.SparkHadoopWriter$.executeTask(SparkHadoopWriter.scala:135)
  at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$write$1(SparkHadoopWriter.scala:88)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  at org.apache.spark.scheduler.Task.run(Task.scala:136)
  at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:750)
Caused by: io.cdap.cdap.api.exception.ProgramFailureException: Error occurred in the phase: 'Writing' with sqlState: '23000', errorCode: '1062', errorMessage: Duplicate entry '5' for key 'PRIMARY'
  at io.cdap.cdap.api.exception.ProgramFailureException$Builder.build(ProgramFailureException.java:229)
  at io.cdap.cdap.api.exception.ErrorUtils.getProgramFailureException(ErrorUtils.java:198)
  at io.cdap.plugin.db.common.DBErrorDetailsProvider.getProgramFailureException(DBErrorDetailsProvider.java:203)
  at io.cdap.plugin.db.common.DBErrorDetailsProvider.getExceptionDetails(DBErrorDetailsProvider.java:169)
  at io.cdap.cdap.etl.common.ErrorDetails.handleException(ErrorDetails.java:75)
  ... 14 common frames omitted
Caused by: java.sql.BatchUpdateException: Duplicate entry '5' for key 'PRIMARY'
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at com.mysql.cj.util.Util.handleNewInstance(Util.java:192)
  at com.mysql.cj.util.Util.getInstance(Util.java:167)
  at com.mysql.cj.util.Util.getInstance(Util.java:174)
  at com.mysql.cj.jdbc.exceptions.SQLError.createBatchUpdateException(SQLError.java:224)
  at com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchSerially(ClientPreparedStatement.java:853)
  at com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchInternal(ClientPreparedStatement.java:435)
  at com.mysql.cj.jdbc.StatementImpl.executeBatch(StatementImpl.java:794)
  at io.cdap.plugin.db.batch.sink.ETLDBOutputFormat$1.close(ETLDBOutputFormat.java:93)
  at io.cdap.cdap.etl.spark.io.TrackingRecordWriter.close(TrackingRecordWriter.java:46)
  at io.cdap.cdap.etl.spark.io.StageTrackingRecordWriter.close(StageTrackingRecordWriter.java:65)
  ... 13 common frames omitted
Caused by: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry '5' for key 'PRIMARY'
  at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:117)
  at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
  at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953)
  at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1092)
  at com.mysql.cj.jdbc.ClientPreparedStatement.executeBatchSerially(ClientPreparedStatement.java:832)
  ... 18 common frames omitted
2025-01-27 13:46:49,674 - WARN  [Executor task launch worker for task 0.0 in stage 0.0 (TID 0):i.c.p.d.b.s.ETLDBOutputFormat@101] - java.sql.SQLNonTransientConnectionException: No operations allowed after connection closed.
  at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
  at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
  at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
  at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
  at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:73)
  at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:73)
  at com.mysql.cj.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:1862)
  at io.cdap.plugin.db.batch.sink.ETLDBOutputFormat$1.close(ETLDBOutputFormat.java:99)
  at io.cdap.cdap.etl.spark.io.TrackingRecordWriter.close(TrackingRecordWriter.java:46)
  at io.cdap.cdap.etl.spark.io.StageTrackingRecordWriter.close(StageTrackingRecordWriter.java:65)
  at org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.closeWriter(SparkHadoopWriter.scala:373)
  at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$executeTask$2(SparkHadoopWriter.scala:150)
  at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1549)
  at org.apache.spark.internal.io.SparkHadoopWriter$.executeTask(SparkHadoopWriter.scala:135)
  at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$write$1(SparkHadoopWriter.scala:88)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  at org.apache.spark.scheduler.Task.run(Task.scala:136)
  at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:750)
Caused by: com.mysql.cj.exceptions.ConnectionIsClosedException: No operations allowed after connection closed.
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
  at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
  at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
  at com.mysql.cj.NativeSession.checkClosed(NativeSession.java:1171)
  at com.mysql.cj.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:573)
  at com.mysql.cj.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:1816)
  ... 16 more
POST v3/namespaces/{namespace-id}/apps/{app-id}/workflows/DataPipelineWorkflow/runs/{run-id}/classify 
[
{
  "stageName": "Database2",
  "errorCategory": "Plugin-'DB Integrity Constraint Violation'-'Database2'",
  "errorReason": "Duplicate entry '5' for key 'PRIMARY'. For more details, see https://en.wikipedia.org/wiki/SQLSTATE",
  "errorMessage": "Error occurred in the phase: 'Writing' with sqlState: '23000', errorCode: '1062', errorMessage: Duplicate entry '5' for key 'PRIMARY'",
  "errorType": "USER",
  "dependency": "true",
  "errorCodeType": "SQLSTATE",
  "errorCode": "23000",
  "supportedDocumentationUrl": "https://en.wikipedia.org/wiki/SQLSTATE"
}
]

image

@psainics psainics added the build Trigger unit test build label Jan 27, 2025
@psainics psainics self-assigned this Jan 27, 2025
@psainics psainics marked this pull request as ready for review January 27, 2025 12:34
@psainics psainics merged commit 6f82a5e into cdapio:release/2.13 Jan 27, 2025
7 checks passed
@psainics psainics deleted the cherry-pick/3a6bfebe10cc2a4acbbdbd213b37e5a547d5b993 branch January 27, 2025 12:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@itsankit-google itsankit-google itsankit-google approved these changes

Assignees

@psainics psainics

Labels
build Trigger unit test build
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@psainics @itsankit-google