control-service: use full url for heartbeat tests and heartbeat tests run in multiple namespaces

[murphp15]

Why

We want to catch regressions in control plane if they are related to using the wrong namespace.

What

We now use different namespace for deployment and control.

How was this tested?

on a previous commit I enabled heartbeat steps and everything ran well.

[antoniivanov]

Maybe we can use the public one https://iaclqhm5xk.execute-api.us-west-1.amazonaws.com ?

Added benefit some tests can run in the gitlab.com runners (instead of ours) .
I've been wanting to run vdk-hearbeat on windows for example.

The downside may be that if Control Service is re-created (deleted and created) the Load Balanced domain name would change and one need to manually update the API gateway (https://github.com/vmware/versatile-data-kit/wiki/Gitlab-CICD#api-gateway)

[murphp15]

Maybe we can use the public one https://iaclqhm5xk.execute-api.us-west-1.amazonaws.com ?

Added benefit some tests can run in the gitlab.com runners (instead of ours) . I've been wanting to run vdk-hearbeat on windows for example.

The downside may be that if Control Service is re-created (deleted and created) the Load Balanced domain name would change and one need to manually update the API gateway (https://github.com/vmware/versatile-data-kit/wiki/Gitlab-CICD#api-gateway)

@tozka I don't agree with using the external url its needless IO on the network and not the pattern we should encourage.

[antoniivanov]

@tozka I don't agree with using the external url its needless IO on the network and not the pattern we should encourage.

You make a valid point, when it comes to jobs deployed / jobs runnning on the cloud ("cloud jobs").

Let me clarify the difference between local command runs and cloud runs.

• Local command runs refer to executing jobs or other CLI commands on the local machine of the user, where the vdk is run on the same system where it's stored.
• On the other hand, cloud runs involve deploying and executing jobs on a cloud-based platform (VDK Kubernetes runtime)

For cloud jobs, indeed, minimizing unnecessary network I/O by avoiding external URLs is a more efficient approach as it reduces unnecessary network traffic and latency and instability.

However vdk-heartbeat is designed to verify local CLI (Command Line Interface) commands and simulate how user uses VDK. For that case, it might be beneficial to use the external URL. The rationale behind this is that these tests simulate the conditions under which end-users would be interacting with our application. Typically, these users would be accessing our services via the external URL. Therefore, using the external URL in tests helps ensure that we're accurately replicating and testing the user experience.

As part of its test vdk-heartbeat also deploys a job and remotely executes it. And also prints the logs of that remote execution. That remote execution likely would use CONTROL_SERVICE_URL that is internal because it's part of the "cloud" but the rest of the vdk-heartbeat which runs commands outside of the cloud should use external address

[murphp15]

@tozka I am only coming back to this PR now.

I am setting the URL only in ini files which are used internally?
Where in this PR do you want me to use the full external URL?

[antoniivanov]

@tozka I am only coming back to this PR now.

I am setting the URL only in ini files which are used internally? Where in this PR do you want me to use the full external URL?

I forgot what was the problem being fixed here? Does it still exists? (perhaps update PR description?)

[DeltaMichael]

@murphp15 Will this affect the frontend deployment and related tests in any way?

Relevant places in the code where the frontend internal URL is used:

https://github.com/vmware/versatile-data-kit/blob/main/projects/frontend/cicd/.gitlab-ci.yml#L201
https://github.com/vmware/versatile-data-kit/blob/main/projects/frontend/cicd/ping_frontend_heartbeat_config.ini#L5

[murphp15]

@DeltaMichael it won't effect the frontend in anyway because deployments never make requests to the frontend.