[Bug] Timeout not working as expected

[tatiana]

The ticket #74 was not enough for running ray pipelines that take longer. When using 0.3.0a5, the tasks are still timing out after 10min running.

As part of this ticket, we need to create a ray pipeline that takes 20 min and ensure it does not time out after 10 minutes.

[pankajkoti]

After having setup locally, I was not able to reproduce this issue. Upon setting job_timeout_seconds=0 for the SubmitRayJob I was able to see the job running longer than 10 mins. I have reported my findings over Slack to our users & awaiting their confirmation.

Attached is a DAG run that ran for 700s (~11.5 mins) from a DAG task that runs a loop for 700 iterations with a 1 second sleep between each iteration.