Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle NumberFormatException in debezium while updating sequence max value for sequence on non-integer column #2360

Merged
merged 3 commits into from
Mar 3, 2025
Merged

Handle NumberFormatException in debezium while updating sequence max value for sequence on non-integer column #2360

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e9c4973
Handled the NumberFormatException exception for non-integer values fo…
priyanshi-yb Feb 24, 2025
87682c5
fix test
priyanshi-yb Feb 25, 2025
2007ee4
review comment
priyanshi-yb Feb 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions ...er-voyagerexporter/src/main/java/io/debezium/server/ybexporter/SequenceObjectUpdater.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,8 +129,15 @@ public void processRecord(Record r){
}
String seqName = getFromColumnSequenceMap(schemaOrDbname, r.t.tableName, r.afterValueColumns.get(i));
if (seqName != null){
Long columnValue = Long.valueOf(r.afterValueValues.get(i).toString());
sequenceMax.put(seqName, Math.max(sequenceMax.get(seqName), columnValue));
Long columnValue = null;
String value = r.afterValueValues.get(i).toString();
try {
columnValue = Long.valueOf(value);
sequenceMax.put(seqName, Math.max(sequenceMax.get(seqName), columnValue));
} catch (NumberFormatException e) {
//Skipping the sequences that are not on Integer columns
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add more context in the comment on the type of sequences and why they could fail with a NumberFormatException?

LOGGER.info("Skipping unsupported sequence with non-interger value: '{}'", seqName);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps DEBUG would be better? otherwise this would come up for every single row.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, okay

}
}
}

Expand Down
5 changes: 5 additions & 0 deletions migtests/tests/pg/sequences/export_data_status-report.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,5 +93,10 @@
"table_name": "sales_region",
"status": "DONE",
"exported_count": 1000
},
{
"exported_count": 3,
"status": "DONE",
"table_name": "users"
}
]
7 changes: 7 additions & 0 deletions migtests/tests/pg/sequences/import_data_status-report.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,5 +131,12 @@
"total_count": 1000,
"imported_count": 1000,
"percentage_complete": 100
},
{
"table_name": "public.\"users\"",
"status": "DONE",
"total_count": 3,
"imported_count": 3,
"percentage_complete": 100
}
]
37 changes: 36 additions & 1 deletion migtests/tests/pg/sequences/pg_sequences_automation.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -214,4 +214,39 @@ WITH region_list AS (
amount[1 + mod(n, array_length(amount, 1))],
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this test already running in live migration & offline with BETA_FAST_DATA_EXPORT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only Offline with Beta_fast mode, we don't run pg_sequences test for live-migration currently

'Branch ' || n as branch,
region[1 + mod(n, array_length(region, 1))]
FROM amount_list, region_list, generate_series(1,1000) as n;
FROM amount_list, region_list, generate_series(1,1000) as n;

CREATE SEQUENCE user_code_seq
START WITH 1
INCREMENT BY 1;

CREATE OR REPLACE FUNCTION generate_user_code() RETURNS TEXT AS $$
DECLARE
new_code TEXT;
BEGIN
SELECT 'USR' || LPAD(nextval('user_code_seq')::TEXT, 4, '0')
INTO new_code;
RETURN new_code;
END;
$$ LANGUAGE plpgsql;

CREATE TABLE users (
user_id SERIAL PRIMARY KEY,
user_code TEXT UNIQUE,
name TEXT
);

ALTER TABLE ONLY users ALTER COLUMN user_code SET DEFAULT generate_user_code()

ALTER SEQUENCE public.user_code_seq OWNED BY public.users.user_code;

INSERT INTO users (name)
VALUES ('John Doe');

INSERT INTO users (name)
VALUES ('ABC');

INSERT INTO users (name)
VALUES ('XYZ');

SELECT * FROM users;
7 changes: 4 additions & 3 deletions migtests/tests/pg/sequences/validate
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ EXPECTED_ROW_COUNT['public'] = {
'sales_region' : 1000,
'boston' : 334,
'sydney': 333,
'london': 333
'london': 333,
'users': 3
}

EXPECTED_ROW_COUNT['schema1'] = {
Expand Down Expand Up @@ -82,15 +83,15 @@ EXPECTED_TABLE_SUM['schema4'] = {
}

NUM_TABLES = {
'public': 13,
'public': 14,
'schema1': 5,
'schema2': 2,
'schema3': 1,
'schema4': 1
}

NUM_SEQUENCES = {
'public': 4+3+1, # 2 are implicit because of serial/generated columns
'public': 4+3+1+2, # 2 are implicit because of serial/generated columns
'schema1': 1+3, # 3 are implicit because of serial/generated columns
'schema2': 1,
'schema3': 1,
Expand Down