Stabilise nightly build pipelines #1559
Assignees
Labels
bug
Something isn't working
Comments
antoniivanov
added a commit
that referenced
this issue
Feb 27, 2023
This change is part of fixing vdk-trino which is failing regularly in the nightly builds . See #1559 Until now we used pytest-docker to start service container, this is switching it to testcontainers-python. There are multiple advanrages of using it. * Supposedly Testcontainers is built with performance in mind and uses a variety of techniques to minimize container startup time and resource usage like image reuse, container reuse, asynchronous operations. * It is far more popular lately - https://hugovk.github.io/top-pypi-packages/ is a site I frequently used to decide which library is likely to be more stable/used . It is showing testcontainers is 2000 places ahead of pytest-docker * Testcontainers provides a wider range of features for managing containers, such as starting and stopping them, waiting for them to become available, and exposing their ports to the host system * Testcontainers-Python allows you to define and configure containers using Python code instaed of yaml which pytest-docker required. Testing Done: ran the test suite of vdk-trino a few times and it was more stable and quick. I have not run actual benchmarks and recorded times but it seems abotu twice faster possibly. We will verify this as part of the nightly builds. Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
added a commit
that referenced
this issue
Feb 27, 2023
This change is part of fixing vdk-trino which is failing regularly in the nightly builds . See #1559 Until now we used pytest-docker to start service container, this is switching it to testcontainers-python. There are multiple advanrages of using it. * Supposedly Testcontainers is built with performance in mind and uses a variety of techniques to minimize container startup time and resource usage like image reuse, container reuse, asynchronous operations. * It is far more popular lately - https://hugovk.github.io/top-pypi-packages/ is a site I frequently used to decide which library is likely to be more stable/used . It is showing testcontainers is 2000 places ahead of pytest-docker * Testcontainers provides a wider range of features for managing containers, such as starting and stopping them, waiting for them to become available, and exposing their ports to the host system * Testcontainers-Python allows you to define and configure containers using Python code instaed of yaml which pytest-docker required. Testing Done: ran the test suite of vdk-trino a few times and it was more stable and quick. I have not run actual benchmarks and recorded times but it seems abotu twice faster possibly. We will verify this as part of the nightly builds. Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
added a commit
that referenced
this issue
Feb 27, 2023
This change is part of fixing vdk-trino which is failing regularly in the nightly builds . See #1559 Until now we used pytest-docker to start service container, this is switching it to testcontainers-python. There are multiple advanrages of using it. * Supposedly Testcontainers is built with performance in mind and uses a variety of techniques to minimize container startup time and resource usage like image reuse, container reuse, asynchronous operations. * It is far more popular lately - https://hugovk.github.io/top-pypi-packages/ is a site I frequently used to decide which library is likely to be more stable/used . It is showing testcontainers is 2000 places ahead of pytest-docker * Testcontainers provides a wider range of features for managing containers, such as starting and stopping them, waiting for them to become available, and exposing their ports to the host system * Testcontainers-Python allows you to define and configure containers using Python code instaed of yaml which pytest-docker required. Testing Done: ran the test suite of vdk-trino a few times and it was more stable and quick. I have not run actual benchmarks and recorded times but it seems abotu twice faster possibly. We will verify this as part of the nightly builds. Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
added a commit
that referenced
this issue
Feb 28, 2023
This change is part of fixing vdk-trino which is failing regularly in the nightly builds . See #1559 Until now we used pytest-docker to start service container, this is switching it to testcontainers-python. There are multiple advanrages of using it. * Supposedly Testcontainers is built with performance in mind and uses a variety of techniques to minimize container startup time and resource usage like image reuse, container reuse, asynchronous operations. * It is far more popular lately - https://hugovk.github.io/top-pypi-packages/ is a site I frequently used to decide which library is likely to be more stable/used . It is showing testcontainers is 2000 places ahead of pytest-docker * Testcontainers provides a wider range of features for managing containers, such as starting and stopping them, waiting for them to become available, and exposing their ports to the host system * Testcontainers-Python allows you to define and configure containers using Python code instaed of yaml which pytest-docker required. Testing Done: ran the test suite of vdk-trino a few times and it was more stable and quick. I have not run actual benchmarks and recorded times but it seems abotu twice faster possibly. We will verify this as part of the nightly builds. Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
added a commit
that referenced
this issue
Feb 28, 2023
This change is part of fixing vdk-trino which is failing regularly in the nightly builds . See #1559 Until now we used pytest-docker to start service container, this is switching it to testcontainers-python. There are multiple advanrages of using it. * Supposedly Testcontainers is built with performance in mind and uses a variety of techniques to minimize container startup time and resource usage like image reuse, container reuse, asynchronous operations. * It is far more popular lately - https://hugovk.github.io/top-pypi-packages/ is a site I frequently used to decide which library is likely to be more stable/used . It is showing testcontainers is 2000 places ahead of pytest-docker * Testcontainers provides a wider range of features for managing containers, such as starting and stopping them, waiting for them to become available, and exposing their ports to the host system * Testcontainers-Python allows you to define and configure containers using Python code instaed of yaml which pytest-docker required. Testing Done: ran the test suite of vdk-trino a few times and it was more stable and quick. I have not run actual benchmarks and recorded times but it seems abotu twice faster possibly. We will verify this as part of the nightly builds. Signed-off-by: Antoni Ivanov <[email protected]> vdk-trino: debugging commit Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
added a commit
that referenced
this issue
Feb 28, 2023
This change is part of fixing vdk-trino which is failing regularly in the nightly builds . See #1559 Until now we used pytest-docker to start service container, this is switching it to testcontainers-python. There are multiple advanrages of using it. * Supposedly Testcontainers is built with performance in mind and uses a variety of techniques to minimize container startup time and resource usage like image reuse, container reuse, asynchronous operations. * It is far more popular lately - https://hugovk.github.io/top-pypi-packages/ is a site I frequently used to decide which library is likely to be more stable/used . It is showing testcontainers is 2000 places ahead of pytest-docker * Testcontainers provides a wider range of features for managing containers, such as starting and stopping them, waiting for them to become available, and exposing their ports to the host system * Testcontainers-Python allows you to define and configure containers using Python code instaed of yaml which pytest-docker required. Testing Done: ran the test suite of vdk-trino a few times and it was more stable and quick. I have not run actual benchmarks and recorded times but it seems abotu twice faster possibly. We will verify this as part of the nightly builds. Signed-off-by: Antoni Ivanov <[email protected]> vdk-trino: debugging commit Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
added a commit
that referenced
this issue
Feb 28, 2023
This change is part of fixing vdk-trino which is failing regularly in the nightly builds . See #1559 Until now we used pytest-docker to start service container, this is switching it to testcontainers-python. There are multiple advanrages of using it. * Supposedly Testcontainers is built with performance in mind and uses a variety of techniques to minimize container startup time and resource usage like image reuse, container reuse, asynchronous operations. * It is far more popular lately - https://hugovk.github.io/top-pypi-packages/ is a site I frequently used to decide which library is likely to be more stable/used . It is showing testcontainers is 2000 places ahead of pytest-docker * Testcontainers provides a wider range of features for managing containers, such as starting and stopping them, waiting for them to become available, and exposing their ports to the host system * Testcontainers-Python allows you to define and configure containers using Python code instaed of yaml which pytest-docker required. Testing Done: ran the test suite of vdk-trino a few times and it was more stable and quick. I have not run actual benchmarks and recorded times but it seems abotu twice faster possibly. We will verify this as part of the nightly builds. Signed-off-by: Antoni Ivanov <[email protected]> vdk-trino: debugging commit Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
added a commit
that referenced
this issue
Mar 1, 2023
This change is part of fixing vdk-trino which is failing regularly in the nightly builds . See #1559 Until now we used pytest-docker to start service container, this is switching it to testcontainers-python. There are multiple advanrages of using it. * Supposedly Testcontainers is built with performance in mind and uses a variety of techniques to minimize container startup time and resource usage like image reuse, container reuse, asynchronous operations. * It is far more popular lately - https://hugovk.github.io/top-pypi-packages/ is a site I frequently used to decide which library is likely to be more stable/used . It is showing testcontainers is 2000 places ahead of pytest-docker * Testcontainers provides a wider range of features for managing containers, such as starting and stopping them, waiting for them to become available, and exposing their ports to the host system * Testcontainers-Python allows you to define and configure containers using Python code instaed of yaml which pytest-docker required. Testing Done: ran the test suite of vdk-trino a few times and it was more stable and quick. I have not run actual benchmarks and recorded times but it seems abotu twice faster possibly. We will verify this as part of the nightly builds. Signed-off-by: Antoni Ivanov <[email protected]> vdk-trino: debugging commit Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
added a commit
that referenced
this issue
Mar 1, 2023
This change is part of fixing vdk-trino which is failing regularly in the nightly builds . See #1559 Until now we used pytest-docker to start service container, this is switching it to testcontainers-python. There are multiple advanrages of using it. * Supposedly Testcontainers is built with performance in mind and uses a variety of techniques to minimize container startup time and resource usage like image reuse, container reuse, asynchronous operations. * It is far more popular lately - https://hugovk.github.io/top-pypi-packages/ is a site I frequently used to decide which library is likely to be more stable/used . It is showing testcontainers is 2000 places ahead of pytest-docker * Testcontainers provides a wider range of features for managing containers, such as starting and stopping them, waiting for them to become available, and exposing their ports to the host system * Testcontainers-Python allows you to define and configure containers using Python code instaed of yaml which pytest-docker required. Testing Done: ran the test suite of vdk-trino a few times and it was more stable and quick. I have not run actual benchmarks and recorded times but it seems abotu twice faster possibly. We will verify this as part of the nightly builds. Signed-off-by: Antoni Ivanov <[email protected]> vdk-trino: debugging commit Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
added a commit
that referenced
this issue
Mar 1, 2023
This change is fixing vdk-trino tests which are failing regularly in the nightly builds . See #1559 The tests were failing intermittently for two main reasons * Docker failures which are being fixed by adopting testcontainers (in favour of pytest-docker) * Hidden dependencies between test cases (because different test use same schema and tables on shared db instance) Until now we used pytest-docker to start service container, this is switching it to testcontainers-python. There are multiple advantages of using it. * Supposedly Testcontainers is built with performance in mind and uses a variety of techniques to minimize container startup time and resource usage like image reuse, container reuse, asynchronous operations. * It is far more popular lately - https://hugovk.github.io/top-pypi-packages/ is a site I frequently use to decide which library is likely to be more stable/used . It is showing testcontainers is 2000 places ahead of pytest-docker * Testcontainers provides a wider range of features for managing containers, such as starting and stopping them, waiting for them to become available, and exposing their ports to the host system * Testcontainers-Python allows you to define and configure containers using Python code instead of yaml which pytest-docker required. Testing Done: ran the test suite of vdk-trino a few times and it was more stable and quick. I have not run actual benchmarks and recorded times but it seems about twice faster possibly. We will verify this as part of the nightly builds. Signed-off-by: Antoni Ivanov <[email protected]>
yonitoo
pushed a commit
that referenced
this issue
Mar 1, 2023
This change is fixing vdk-trino tests which are failing regularly in the nightly builds . See #1559 The tests were failing intermittently for two main reasons * Docker failures which are being fixed by adopting testcontainers (in favour of pytest-docker) * Hidden dependencies between test cases (because different test use same schema and tables on shared db instance) Until now we used pytest-docker to start service container, this is switching it to testcontainers-python. There are multiple advantages of using it. * Supposedly Testcontainers is built with performance in mind and uses a variety of techniques to minimize container startup time and resource usage like image reuse, container reuse, asynchronous operations. * It is far more popular lately - https://hugovk.github.io/top-pypi-packages/ is a site I frequently use to decide which library is likely to be more stable/used . It is showing testcontainers is 2000 places ahead of pytest-docker * Testcontainers provides a wider range of features for managing containers, such as starting and stopping them, waiting for them to become available, and exposing their ports to the host system * Testcontainers-Python allows you to define and configure containers using Python code instead of yaml which pytest-docker required. Testing Done: ran the test suite of vdk-trino a few times and it was more stable and quick. I have not run actual benchmarks and recorded times but it seems about twice faster possibly. We will verify this as part of the nightly builds. Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
added a commit
that referenced
this issue
Mar 9, 2023
The use of global variables in GitLab CI can create problems when multiple CI files are being imported, as variables are not scoped to a specific file. This can lead to unexpected variables being set, causing issues in the CI/CD pipeline. This is what is happening wiht quickstart-vdk test - https://gitlab.com/vmware-analytics/versatile-data-kit/-/jobs/3901914540 It expected that there's python installed because it's supposed to use python image but some other file overrides this image to docker and causes the test to fail. This changes quickstart-vdk to use job-level variables. In future changes I will be changing all CIs to use job-level variables or environment variables to avoid conflicts and ensure proper scoping. Or ensure that global variabels and labels are prefixed with the project or plugin name. This is part of #1559 Testing Done: this PR CI Signed-off-by: Antoni Ivanov <[email protected]>
ivakoleva
pushed a commit
that referenced
this issue
Mar 9, 2023
The use of global variables in GitLab CI can create problems when multiple CI files are being imported, as variables are not scoped to a specific file. This can lead to unexpected variables being set, causing issues in the CI/CD pipeline. This is what is happening wiht quickstart-vdk test - https://gitlab.com/vmware-analytics/versatile-data-kit/-/jobs/3901914540 It expected that there's python installed because it's supposed to use python image but some other file overrides this image to docker and causes the test to fail. This changes quickstart-vdk to use job-level variables. In future changes I will be changing all CIs to use job-level variables or environment variables to avoid conflicts and ensure proper scoping. Or ensure that global variabels and labels are prefixed with the project or plugin name. This is part of #1559 Testing Done: this PR CI Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
added a commit
that referenced
this issue
Mar 9, 2023
As part of effort to stabilize builds - #1559 I am removing global variables and names since they can be overwriten by accident across differnet projects. Signed-off-by: Antoni Ivanov <[email protected]>
antoniivanov
added a commit
that referenced
this issue
Mar 9, 2023
As part of effort to stabilize builds - #1559 I am removing global variables and names since they can be overwriten by accident across differnet projects. Signed-off-by: Antoni Ivanov <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
At the moment the nightly build pipeline is very unstable.
The result of this ticket is that nightly builds should always be green when no regressions have been introduced.
Steps To Reproduce
Look at previous nightly runs to get a list of unstable jobs.
The text was updated successfully, but these errors were encountered: