Create Dangerzone environments and run QA tests on them

[apyrgio]

This PR introduces two scripts that can help with Dangerzone development and releases. Those are:

• dev_scripts/env.py: This script creates Dangerzone environments on various Linux platforms. These environments can be used to test Dangerzone and build artifacts.
• dev_scripts/qa.py: This script runs QA tests on the environments that the previous script creates. These QA tests are taken from our release instructions, but they are not fully automated. They act as guard rails for the developer who wants to ensure that Dangerzone works for a platform.

Closes #286
Closes #287

[deeplow]

I had some specific comments about the windows adaptation of the install/windows/build-image.py but I think I have a better suggestion that works on all OSes - install/build-image.py:

from dangerzone.container import get_runtime_name
import subprocess

runtime = get_runtime_name()

def run(cmd):
    subprocess.run(cmd, shell=True)

print("Building container image")
run(f'{runtime} build container --platform linux/amd64 --tag dangerzone.rocks/dangerzone')

print("Saving and compressing container image")
run(f'{runtime} save dangerzone.rocks/dangerzone | gzip > share/container.tar.gz')

print("Looking up the image id")
run(f'{runtime} image ls dangerzone.rocks/dangerzone | grep "dangerzone.rocks/dangerzone" | tr -s \' \' | cut -d\' \' -f3 > share/image-id.txt')

This way we would de-duplicate some of the code and remove the added gzip complexity in python code. I tested on Linux and this python script is 3 times faster than the proposed one.

What do you think about using this one? I can also add this as a separate PR as it's not core this this change.

[deeplow]

I tested this solely on a Qubes system. I haven't tested yet on windows.

Feedback:

• duplication: see this discussion.
• **there could be a way to move back to a previous step. From playing around it appears to be easy to mistakenly move on to the next step.
• have you considered mounting in the containers ~/.bashrc? It oculd be useful to have the history of commands so we don't have to repeatedly type them.
• The build .deb step was much slower for me. It took 27mins. Is this slow performance to be expected?
• In the container there is no PDF viewer, which leads to
  
• when an 'auto' is running and you press any key when the command is running, those keys will get pressed after the command runs. So if you press enter while waiting to see if something happens, then it presses enter on the next commands -- but because enter skips the task, you advance to the next ones. I don't know how to solve this problem easily, but changing the auto from enter to something else less clickable can help.

Some situations where 'auto' failed:

• I've had many 'auto' commands related to lack of space. But that's to be expected. After increasing the size the went away. Because this may be so common, should we add a check for them?

• Failure in auto step "Create Dangerzone QA environment" (fedora 36)

I was unable to find why this was failed, but after repeating the step, it worked.

[...]
  xcb-util-renderutil-0.3.9-20.fc36.x86_64                                      
  xcb-util-wm-0.4.1-22.fc36.x86_64                                              
  xkeyboard-config-2.35.1-1.fc36.noarch                                         
  xml-common-0.6.3-58.fc36.noarch                                               
  xz-5.2.5-9.fc36.x86_64                                                        
  yajl-2.1.0-18.fc36.x86_64                                                     

Complete!
--> de82eca9bdf
STEP 5/10: RUN rm /tmp/dangerzone-0.4.0-1.noarch.rpm
error running container: did not get container start message from parent: EOF
Error: building at STEP "RUN rm /tmp/dangerzone-0.4.0-1.noarch.rpm": slirp4netns failed
Traceback (most recent call last):
  File "/home/user/dangerzone/dev_scripts/env.py", line 509, in <module>
    sys.exit(main())
  File "/home/user/dangerzone/dev_scripts/env.py", line 505, in main
    return args.func(args)
  File "/home/user/dangerzone/dev_scripts/env.py", line 409, in env_build
    env.build(show_dockerfile=args.show_dockerfile)
  File "/home/user/dangerzone/dev_scripts/env.py", line 391, in build
    self.runtime_run("build", "-t", image, build_dir)
  File "/home/user/dangerzone/dev_scripts/env.py", line 235, in runtime_run
    subprocess.run(self.runtime_cmd + list(args), check=True)
  File "/usr/lib64/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['podman', 'build', '-t', 'dangerzone.rocks/fedora:36', PosixPath('/home/user/dangerzone/dev_scripts/envs/fedora/36/build')]' returned non-zero exit status 125.
Traceback (most recent call last):
  File "/home/user/dangerzone/./dev_scripts/qa.py", line 470, in <module>
    sys.exit(main())
  File "/home/user/dangerzone/./dev_scripts/qa.py", line 464, in main
    platform.start()
  File "/home/user/dangerzone/./dev_scripts/qa.py", line 294, in start
    self.build_qa_image()
  File "/home/user/dangerzone/./dev_scripts/qa.py", line 91, in inner
    out = func(self, *args, **kwargs)
  File "/home/user/dangerzone/./dev_scripts/qa.py", line 280, in build_qa_image
    self.run(
  File "/home/user/dangerzone/./dev_scripts/qa.py", line 116, in run
    subprocess.run(args, check=True)
  File "/usr/lib64/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/home/user/dangerzone/dev_scripts/env.py', '--distro', 'fedora', '--version', '36', 'build')' returned non-zero exit status 1.

Other suggestions:

• copy test_docs to the container's home folder
  When testing I found myself wasting quite a bit of time with selecting the documents. If every time the container I had already the test documents in the home directory, I could select them more easily. Additionally, it would be nice if we had a fresh copy of the test_docs directory, since every time convert them, they are moved to the unsafe directory.

• Report generation
  It would be nice at the end if the script generated some markdown that could be pasted on the QA issue for the release about what was tested. See my comment about prompt() on collecting notes about scenarios and then reporting them at the end.

Next steps (possibly future PRs):

• Replace in CI build-* with building with your setup and deprecate packagecloud

[apyrgio]

I had some specific comments about the windows adaptation of the install/windows/build-image.py but I think I have a better suggestion that works on all OSes - install/build-image.py:

8< ... snip ... >8

Just to make sure I understand, do you propose this as a drop-in script for all OSes? Because the gzip executable will not be available on Windows

This way we would de-duplicate some of the code and remove the added gzip complexity in python code. I tested on Linux and this python script is 3 times faster than the proposed one.

That's weird. The script you suggest and the existing Linux one are very similar, so where do you attribute this boost?

What do you think about using this one? I can also add this as a separate PR as it's not core this this change.

In general, having a script for all OSes would be cool, so I'd be down with that, as a separate PR of course. We could even branch this discussion to a new issue.

[deeplow]

I had some specific comments about the windows adaptation of the install/windows/build-image.py but I think I have a better suggestion that works on all OSes - install/build-image.py:
8< ... snip ... >8

Just to make sure I understand, do you propose this as a drop-in script for all OSes? Because the gzip executable will not be available on Windows

Ah. Damn. I forgot about that. That's probably the reason why the windows one was in python. I don't know where I concluded that it worked on all systems, then.

Let's do this in another PR as you suggest.

This way we would de-duplicate some of the code and remove the added gzip complexity in python code. I tested on Linux and this python script is 3 times faster than the proposed one.

That's weird. The script you suggest and the existing Linux one are very similar, so where do you attribute this boost?

Chunk sizes, perhaps?

[apyrgio]

Replying to the rest of the feedback (#289 (review)):

I tested this solely on a Qubes system. I haven't tested yet on windows.

Feedback:

* [ ]  **duplication:** see [this discussion](https://github.com/freedomofpress/dangerzone/pull/289#discussion_r1054373702).

Yeap, juicy subject, let's continue the discussion there.

* [ ]  **there could be a way to move back to a previous step. From playing around it appears to be easy to mistakenly move on to the next step.

We can do it, but is it worth doing it? I think it's one of the things we should do once we use this script more.

* [ ]  have you considered mounting in the containers `~/.bashrc`? It oculd be useful to have the history of commands so we don't have to repeatedly type them.

Good question. You probably refer to the .bash_history file, and I've in fact considered it. Currently, the state of the Dangerzone environment is just the Podman containers (/home/user/.local/share/containers), but that wasn't always the case. I used to mount the whole user's home (/home/user) as a directory, meaning that all user state would persist across executions (containers, bash history, Dangerzone config, you name it).

I dropped it because I need to think a bit if past state can affect subsequent executions. Most probably it won't, but it doesn't hurt us delaying this a bit. Are you ok with that?

* [ ]  The build `.deb` step was much slower for me. It took 27mins. Is this slow performance to be expected?

No, it shouldn't. I haven't experience such slow build times myself. We may need to debug this.

* [ ]  In the container there is no PDF viewer, which leads to
  ![no-pdf-viewer](https://user-images.githubusercontent.com/47065258/209135525-c9663944-58e1-48aa-bef8-b1c554b2b824.png)

I personally install mupdf, which is a light-weight PDF viewer, when running the QA steps. Do you have a PDF viewer that does not bring much dependencies?

* when an 'auto' is running and you press any key when the command is running, those keys will get pressed after the command runs. So if you press enter while waiting to see if something happens, then it presses enter on the next commands -- but because enter skips the task, you advance to the next ones. I don't know how to solve this problem easily, but changing the auto from `enter` to something else less clickable can help.

Good catch. I've mentioned in the corresponding discussion (#289 (comment)) that we can consume the process' stdin before prompting the user, which would solve these cases. We can discuss more there.

Some situations where 'auto' failed:

* I've had many 'auto' commands related to lack of space. But that's to be expected. After increasing the size the went away. Because this may be so common, should we add a check for them?

Hm, this gun shoots in two directions. Imagine the case where a step can run without extra space, but the system is low in resources. In that case, what will this check do? Cancel a reasonable execution? Throw a warning that will be lost in the log messages. I'm not sure how we can make this work, but if you have a suggestion, shoot it. Disclaimer, most of the issues I've encountered where related to low disk space as well, so I get your point.

* Failure in auto step "Create Dangerzone QA environment" (fedora 36)

I was unable to find why this was failed, but after repeating the step, it worked.

Interesting, I haven't seen this before. Is this reproducible? If yes, we can look into it. Else, I'd attribute it to something transient (maybe low disk space).

[deeplow]

* [ ]  have you considered mounting in the containers `~/.bashrc`? It oculd be useful to have the history of commands so we don't have to repeatedly type them.

Good question. You probably refer to the .bash_history file, and I've in fact considered it. Currently, the state of the Dangerzone environment is just the Podman containers (/home/user/.local/share/containers), but that wasn't always the case. I used to mount the whole user's home (/home/user) as a directory, meaning that all user state would persist across executions (containers, bash history, Dangerzone config, you name it).

I dropped it because I need to think a bit if past state can affect subsequent executions. Most probably it won't, but it doesn't hurt us delaying this a bit. Are you ok with that?

Yes, I meant .bash_history. I get the point of keeping the state. But I wouldn't consider keeping past commands in history as state that will be of terrible consequence. If anything it will speed up testing. It's a quality-of-life improvement that I think would make a difference in practice. So I'd insist for us to keep it.

I personally install mupdf, which is a light-weight PDF viewer, when running the QA steps. Do you have a PDF viewer that does not bring much dependencies?

Nope. It can be that one or anything else you prefer.

Hm, this gun shoots in two directions. Imagine the case where a step can run without extra space, but the system is low in resources. In that case, what will this check do? Cancel a reasonable execution? Throw a warning that will be lost in the log messages. I'm not sure how we can make this work, but if you have a suggestion, shoot it. Disclaimer, most of the issues I've encountered where related to low disk space as well, so I get your point.

Just a warning would be my suggestion + the use of colors (bold, yellow). But we can tackle this in the future when we consider colors in the script.

Interesting, I haven't seen this before. Is this reproducible? If yes, we can look into it. Else, I'd attribute it to something transient (maybe low disk space).

Not reproducible. Let's just discard this issue.

[apyrgio]

* [ ]  have you considered mounting in the containers `~/.bashrc`? It oculd be useful to have the history of commands so we don't have to repeatedly type them.

Good question. You probably refer to the .bash_history file, and I've in fact considered it. Currently, the state of the Dangerzone environment is just the Podman containers (/home/user/.local/share/containers), but that wasn't always the case. I used to mount the whole user's home (/home/user) as a directory, meaning that all user state would persist across executions (containers, bash history, Dangerzone config, you name it).
I dropped it because I need to think a bit if past state can affect subsequent executions. Most probably it won't, but it doesn't hurt us delaying this a bit. Are you ok with that?

Yes, I meant .bash_history. I get the point of keeping the state. But I wouldn't consider keeping past commands in history as state that will be of terrible consequence. If anything it will speed up testing. It's a quality-of-life improvement that I think would make a difference in practice. So I'd insist for us to keep it.

Ok, I'll add this.

I personally install mupdf, which is a light-weight PDF viewer, when running the QA steps. Do you have a PDF viewer that does not bring much dependencies?

Nope. It can be that one or anything else you prefer.

Cool, I'll install it as well, both in the dev and the install environment.

Hm, this gun shoots in two directions. Imagine the case where a step can run without extra space, but the system is low in resources. In that case, what will this check do? Cancel a reasonable execution? Throw a warning that will be lost in the log messages. I'm not sure how we can make this work, but if you have a suggestion, shoot it. Disclaimer, most of the issues I've encountered where related to low disk space as well, so I get your point.

Just a warning would be my suggestion + the use of colors (bold, yellow). But we can tackle this in the future when we consider colors in the script.

Got it.

Interesting, I haven't seen this before. Is this reproducible? If yes, we can look into it. Else, I'd attribute it to something transient (maybe low disk space).

Not reproducible. Let's just discard this issue.

Ok.

[apyrgio]

Hi @deeplow. I think I've addressed all the review comments that we talked about, and I fixed some issues tangent to this code. Feel free to give me your opinion on the changes.

[deeplow]

Only minor comments / questions. No need to address them as they are minor.

Approved in general

[deeplow]

I didn't understand this part. Could you explain a bit more the logic? Or give an example where this happens?

[apyrgio]

Sigh, I'm not proud of this code. An example where the problem manifests is here:

dangerzone/dev_scripts/qa.py

Lines 699 to 700 in f0c3c82

	@QABase.task("Build Dangerzone image", ref="REF_BUILD", auto=True)
	def build_container_image(self):

If you look carefully, I use ref="REF_BUILD", instead of ref=REF_BUILD. The reason is that during class creation, REF_BUILD = None. To accomodate for the REF_BUILD that the subclasses later define, I had to resort to a dynamic approach 😞 .

[apyrgio]

I've rebased the branch, and I've removed the Fedora 35 support, so now our build completes successfully.