Added the test scripts for resumption #2117
Conversation
|
|
shubham-yb
requested review from
makalaaneesh,
sanyamsinghal,
ShivanshGahlot and
priyanshi-yb
Does your PR have changes that can cause upgrade issues?
|
migtests/scripts/resumption.py
Outdated
| """ | ||
| Runs the yb-voyager command with support for resumption testing. | ||
| """ | ||
| for attempt in range(1, resumption['max_restarts'] + 1): |
There was a problem hiding this comment.
let's get/define all the configs in the beginning. It will make it easier to understand what all configuration options are involved.
max_restarts = resumption['max_restarts']
min_interrupt_seconds = resumption['min_interrupt_seconds']
...
migtests/scripts/resumption.py
Outdated
| if not output: # Exit if output is empty (end of process output) | ||
| break | ||
| full_output += output | ||
| if time.time() - start_time > 5: |
There was a problem hiding this comment.
why break ? what is 5? seconds? minutes?
There was a problem hiding this comment.
This was written so that we get the output in realtime. Turning back it isn't much helpful in case of automation. Have changed this implementation.
migtests/scripts/resumption.py
Outdated
| # Final import retry logic | ||
| print("\n--- Final attempt to complete the import ---") | ||
|
|
||
| for _ in range(2): |
There was a problem hiding this comment.
Why 2 attempts finally?
There was a problem hiding this comment.
The two attempts and the sleep were to avoid any intermittent issues or system overload. Have removed it.
migtests/scripts/resumption.py
Outdated
| try: | ||
| print("\nVoyager command output:") | ||
|
|
||
| process = subprocess.Popen( |
There was a problem hiding this comment.
nit: separate function for starting command (can be called in above for-loop as well)
migtests/scripts/resumption.py
Outdated
| ) | ||
|
|
||
| # Capture and print output | ||
| for line in iter(process.stdout.readline, ''): |
There was a problem hiding this comment.
in the above for-loop, we're reading both stderr and stdout, here we're only reading stdout. Any particular reason? Would be good to be consistent here (call a common function that captures stdout/stderr)
There was a problem hiding this comment.
Also till when will you keep reading? How long will the loop run?
There was a problem hiding this comment.
The loop was to be run till the command exits and print the output in realtime.
Implemented a separate function for running the command and capturing stdout and stderr.
migtests/scripts/resumption.py
Outdated
| for line in iter(process.stderr.readline, ''): | ||
| print(line.strip()) | ||
| sys.stdout.flush() | ||
| time.sleep(30) |
There was a problem hiding this comment.
It was added to avoid intermittent failures / system overload. Have removed it.
migtests/scripts/resumption.py
Outdated
| print("Final import failed after 2 attempts.") | ||
| sys.exit(1) | ||
|
|
||
| def validate_row_counts(row_count, export_dir): |
There was a problem hiding this comment.
note for future: you can create a common python file that has such helper
functions.
| @@ -0,0 +1,133 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
Assuming that the ONLY change here is that you're specifying ROW_COUNT and essentially making generate_series dynamic.
| schema2.Case_Sensitive_Table: 5000000 | ||
| schema2.case: 5000000 | ||
| schema2.Table: 5000000 | ||
| public.boston: 2500000 |
There was a problem hiding this comment.
where is the code that generates data for all these other tables boston/cust/emp/etc? I only see code for table/case/Case_Sensitive_Table
There was a problem hiding this comment.
Those are being done via the partitions test schema / data
| try: | ||
| process.terminate() | ||
| process.wait(timeout=10) | ||
| except subprocess.TimeoutExpired: |
There was a problem hiding this comment.
let's put some logs/prints to determine if we terminated the process or killed it
Describe the changes in this pull request
Describe if there are any user-facing changes
N/A
How was this pull request tested?
Made the changes to the Jenkins pipeline as well.