Add docker storage check #3787

juanvallejo · 2017-03-28T21:17:29Z

Adds docker storage check to ensure that a docker host is using a supported storage driver,
that its thinpool usage is not close to a specified limit, and that Loopback device is not being used (if devicemapper).

Also updates docker_image_availability to reuse code for running only on docker hosts and installing docker-py if needed

cc @brenton @rhcarvalho @sosiouxme

brenton · 2017-03-29T12:10:17Z

I'll be honest, I don't know the difference between overlay and overlay2. Were you able to confirm we intend to support both of them?

rhcarvalho

What's the conceptual difference between this and #3518? There seems to be some overlap.

rhcarvalho · 2017-03-29T12:42:51Z

roles/openshift_health_checker/openshift_checks/docker_storage_check.py

+        return lv[0]
+
+    def exec_cmd(self, cmd_str, task_vars):
+        return self.module_executor("command", {


For new checks you can use self.execute_module instead.

juanvallejo · 2017-03-29T14:33:37Z

@brenton

Were you able to confirm we intend to support both of them?

I reached out to @stevekuznetsov and he was able to confirm that we will only be supporting overlay2. I have updated this check accordingly

juanvallejo · 2017-03-29T14:35:52Z

roles/openshift_health_checker/openshift_checks/docker_storage_check.py

+        return False, ""
+
+    def get_lvs_data(self, task_vars):
+        lvs_cmd = "/sbin/lvs --select vg_name=docker --select lv_name=docker-pool --report-format json"


I attempted to look for a core module to use in place of this command, however the closest one I found was http://docs.ansible.com/ansible/lvol_module.html, which does not seem to return the information needed for this check

juanvallejo · 2017-04-26T19:26:53Z

@rhcarvalho @sosiouxme PTAL

rhcarvalho · 2017-04-27T09:11:55Z

roles/openshift_health_checker/openshift_checks/docker_storage.py

+        if failed:
+            return failed, no_data_msg
+
+        if data_percent > float(self.max_thinpool_data_usage_percent):


When we read max_thinpool_data_usage_percent from task_vars, as the tests show, it might be a string... and we can't always convert a string to float (not tested).
We could be a bit more careful and handle TypeError.

I'd suggest doing that early on when we read from task_vars, not here.

rhcarvalho · 2017-04-27T09:15:38Z

roles/openshift_health_checker/openshift_checks/docker_storage.py

+        return False, ""
+
+    def get_lvs_data(self, task_vars):
+        lvs_cmd = "/sbin/lvs --select vg_name=docker --select lv_name=docker-pool --report-format json"


Can we rely on the names here, docker and docker-pool?

hm, upon looking at other installs, vg_name and lv_name seemed to vary slightly.

A point @sdodson brought up was that we should not be concerned about the total size of a logical volume, as it is automatically expanded once it reaches a certain usage threshold. Instead, we should be looking at the total size of the volume group containing the docker LV.

An idea that comes to mind would be to use vgdisplay (or the more convenient vgs), iterate through all of the volume groups in the output (in my case, I only had one -- docker), until we come across a VG with docker in its name. We can then alter this check to only warn when the VG is nearing capacity rather than a specific logical volume under it. WDYT?

EDIT: At least in my case, I am also able to get the specific VG name for docker LV by checking the contents of /etc/sysconfig/docker-storage-setup

cc @sosiouxme

Somehow I did not notice this message. Err... yeah, technically the VG can be named anything and that's probably what we should be looking at. Might want to look in docker config for what it's using. I'll poke around this tomorrow.

rhcarvalho · 2017-04-27T09:18:02Z

roles/openshift_health_checker/openshift_checks/docker_storage.py

+    def get_lvs_data(self, task_vars):
+        lvs_cmd = "/sbin/lvs --select vg_name=docker --select lv_name=docker-pool --report-format json"
+        result = self.exec_cmd(lvs_cmd, task_vars)
+        failed = False


Why do we need this? Looks like it is only used in the last line of the method.

Thanks, will remove. was leftover after I updated this method

rhcarvalho · 2017-04-27T09:22:21Z

roles/openshift_health_checker/openshift_checks/docker_storage.py

+        failed = False
+
+        if result.get("failed", False):
+            return True, {}, result.get("msg", "")


This method returns 3 non-obvious values and they do undocumented...
I'm guessing "did it fail?", "the lvs data I asked for" and "some error message if any". It's a weird API to have failed as the first argument, and might even be unnecessary to have it.

Alternatively, these methods could return what they are meant to return, and if not raise an exception, that would remove a lot of the conditionals.

rhcarvalho · 2017-04-27T09:25:02Z

roles/openshift_health_checker/openshift_checks/docker_storage_driver.py

+    """Check Docker storage driver compatibility.
+
+    This check ensures that Docker is using a supported storage driver,
+    and that Loopback is not being used (if using devicemapper).


Written version of a mental note: this is another check (after memory and disk checks) that will annoy people creating toy clusters...

cc @brenton @sosiouxme

If you're setting up "toy clusters" with multiple nodes and openshift-ansible, my opinion is that you're out of the "toy cluster" space -- that's more for oc cluster up.

rhcarvalho · 2017-04-27T09:27:03Z

roles/openshift_health_checker/openshift_checks/docker_storage_driver.py

+    def is_active(cls, task_vars):
+        """Skip non-containerized installations."""
+        is_containerized = get_var(task_vars, "openshift", "common", "is_containerized")
+        return super(DockerStorageDriver, cls).is_active(task_vars) and is_containerized


Looks like both checks ask for a

class ...(Containerized, OpenShiftCheck): ...

?

I am not sure what you mean by this, should we create a new mixin rather than read the is_containerized value from task_vars?

That's what I meant, for symmetry with NonContainerized.

I dunno about a Containerized mixin but I think this check should be active under the same conditions as the other one.

rhcarvalho · 2017-04-27T09:28:39Z

roles/openshift_health_checker/openshift_checks/docker_storage_driver.py

+
+        if not self.is_supported_storage_driver(info):
+            msg = "Unsupported Docker storage driver detected. Supported storage drivers: {drivers}"
+            return {"failed": True, "msg": msg.format(drivers=self.storage_drivers)}


This would print something like ...Supported storage drivers: ['foo', 'bar'].
We can make it nicer with ', '.join(self.storage_drivers).

rhcarvalho · 2017-04-27T09:30:00Z

roles/openshift_health_checker/openshift_checks/docker_storage_driver.py

+            msg = "Use of loopback devices is discouraged. Try running Docker with `--storage-opt dm.thinpooldev`"
+            return {"failed": True, "msg": msg}
+
+        return {"changed": False}


If we don't focus on the changed behavior (we not always return it), maybe it would be more consistent to simply don't ever mention it?

rhcarvalho · 2017-04-27T09:30:46Z

roles/openshift_health_checker/test/docker_storage_test.py

+
+
+
+


Lots of empty lines?

rhcarvalho · 2017-04-27T09:32:08Z

roles/openshift_health_checker/openshift_checks/docker_storage.py

+    """Check Docker storage sanity.
+
+    This check ensures that thinpool usage is not
+    close to 100% for a containerized installation.


The percentage threshold is arbitrary, this comment is too specific.

juanvallejo · 2017-05-10T18:30:01Z

aos-ci-test

openshift-bot · 2017-05-10T20:42:03Z

success: "aos-ci-jenkins/OS_3.5_NOT_containerized, aos-ci-jenkins/OS_3.5_NOT_containerized_e2e_tests" for b89bb5d (logs)

openshift-bot · 2017-05-10T20:42:58Z

success: "aos-ci-jenkins/OS_3.6_NOT_containerized, aos-ci-jenkins/OS_3.6_NOT_containerized_e2e_tests" for b89bb5d (logs)

openshift-bot · 2017-05-10T20:44:57Z

success: "aos-ci-jenkins/OS_3.5_containerized, aos-ci-jenkins/OS_3.5_containerized_e2e_tests" for b89bb5d (logs)

openshift-bot · 2017-05-10T20:45:45Z

success: "aos-ci-jenkins/OS_3.6_containerized, aos-ci-jenkins/OS_3.6_containerized_e2e_tests" for b89bb5d (logs)

sosiouxme · 2017-05-18T16:37:55Z

roles/openshift_health_checker/openshift_checks/docker_storage.py

+
+    @classmethod
+    def is_active(cls, task_vars):
+        """Skip non-containerized installations."""


Why would we not do this for non-containerized installations too? Everything that's a node needs supportable Docker storage.

juanvallejo · 2017-05-18T17:54:05Z

@sosiouxme thanks for the feedback. Review comment addressed

sosiouxme · 2017-05-18T18:41:15Z

pyunit test is failing right now.

sosiouxme · 2017-06-07T11:16:56Z

@rhcarvalho any concerns with this or are we clear to merge?

rhcarvalho · 2017-06-07T11:51:26Z

@rhcarvalho any concerns with this or are we clear to merge?

Taking a look now.

rhcarvalho · 2017-06-07T12:06:10Z

roles/openshift_health_checker/openshift_checks/mixins.py

+        if get_var(task_vars, "openshift", "common", "is_atomic"):
+            return "", False, False
+
+        # NOTE: we would like to use "package" module but that fails in a way we can't debug just yet.


I believe it's because "package" is not a module but yet another action plugin (or both?). I've run into this before, looks like the method we invoke to call a module can only resolve to actual modules and not any action plugin, which is a pity.

What we could try is to import and run the action plugin directly.

rhcarvalho · 2017-06-07T12:11:55Z

roles/openshift_health_checker/openshift_checks/mixins.py

+                "Unable to install required packages on this host:\n"
+                "    {deps}\n{msg}"
+            ).format(deps=',\n    '.join(self.dependencies), msg=msg)
+        return msg, result.get("failed") or result.get("rc", 0) != 0, result.get("changed")


Adding one of two variables here would make this more readable.

# ... failed = result.get("failed") or result.get("rc", 0) != 0 changed = result.get("changed") return msg, failed, changed

rhcarvalho · 2017-06-07T12:14:07Z

@juanvallejo @sosiouxme please update the PR description to include the "bonus" that is coming additional to the new health check ;)

rhcarvalho

Is the new check going to bother people on "non-production clusters"? E.g. oc cluster up style. Do we have a way to disable it?

rhcarvalho · 2017-06-07T12:18:33Z

roles/openshift_health_checker/openshift_checks/docker_storage.py

+        info = info["info"]
+
+        # check if the storage driver we saw is valid
+        driver = info.get("Driver", "[NONE]")


Why "[NONE]"?

if the docker info came back without a driver (which seems unlikely, but you know, code defensively) then we're going to say to the user "Docker is using the X driver which isn't supported". What should X be? I don't really care, just thought it would be good to report something distinctly bogus in this unlikely event.

rhcarvalho · 2017-06-07T12:22:30Z

roles/openshift_health_checker/openshift_checks/docker_storage.py

+                )
+                return {"failed": True, "changed": changed, "msg": msg}
+            result = self._check_dm_usage(driver_status, task_vars)
+            result["changed"] = changed


Shouldn't this be something like:

result['changed'] = result.get('changed', False) or changed

?

only if _check_dm_usage ever made changes. it doesn't so... keep it simple?

I get we (don't / might not) even use the value of 'changed' for checks right now, but we "code defensively" in lots of places, why wouldn't we do it here? If that assumption changes (that _check_dm_usage never sets the 'changed' key), it is unlikely that we'll go look at every caller to stop overriding the value.

rhcarvalho · 2017-06-07T12:24:02Z

roles/openshift_health_checker/openshift_checks/docker_storage.py

+            metadata_total=driver_status.get("Metadata Space Total"),
+        )
+
+        for key, value in vals.copy().items():


Why .copy()?

because we're modifying vals in the loop and I heard it was a bad idea to iterate while changing

That's correct...

On a second look, I wonder why we mutate vals, and why do we add a key with the _bytes prefix? I couldn't find anywhere that actually reads back that key?!

Ah, I see I see... we later read "data_used_bytes" and so on.

yes, and vals is merged into the result, so that the details are all in the ansible spew if you want to debug what you got

rhcarvalho · 2017-06-07T12:28:38Z

roles/openshift_health_checker/openshift_checks/docker_storage.py

+        an LVM2 LV as a backing storage pool. The LV is "thin" because it does not use
+        all available storage from its VG, instead expanding as needed; so in considering
+        available space, it is necessary to include space remaining in the VG (which is
+        unfortunately not reported by the storage driver in docker info).


This sounds like a rant, and refers to things that can easily become false statements or be hard to follow.
E.g.:

"What is 'docker info'?" -- I think this is about our custom Ansible module, but a reader of this code may have no idea what that refers too. In particular, it is NOT about the docker info CLI command.

"unfortunately not reported..." -- might be not reported now, but when people will read this, we can't guarantee that is still true. Instead of talking about what something does or doesn't today, focus on what is required. In the future, someone might understand the requirements and find a way how to implemented it differently.

Just trying to give background on how this works and why we're looking for the values we are, which I didn't know myself until recently. So, sounds like it could be improved.

rhcarvalho · 2017-06-07T12:32:54Z

roles/openshift_health_checker/openshift_checks/docker_storage.py

+        except ValueError as err:
+            return {
+                "failed": True,
+                "msg": "Unable to convert thinpool data usage limit to float: {}".format(str(err))


Hmm, with this we cannot tell if the problem was within max_thinpool_data_usage_percent or max_thinpool_metadata_usage_percent.

Also, users may not care about "float". Perhaps a better message would explain that the value in the configuration is invalid (and include the bad value in the message).

I was just lazy, didn't want to duplicate the entire try/except for two almost identical things. I will rework this.

rhcarvalho · 2017-06-07T12:37:30Z

roles/openshift_health_checker/openshift_checks/docker_storage.py

+                "is higher than threshold {:.1f}."
+            ).format(vals["metadata_pct_used"], vals["metadata_threshold"])
+
+        vals["msg"] = msg or "Thinpool usage is below thresholds."


Please include the configured threshold and offending value in the message.

That's the success message. Which BTW the user doesn't see unless they run with -vvv

Aha, didn't realize that... I'm used to see a 'msg' key accompanying a 'failed': True, didn't think of the other case ;-)

rhcarvalho · 2017-06-07T12:39:59Z

roles/openshift_health_checker/openshift_checks/docker_storage.py

+        match = re.match(r'((?:[^-]|--)+)-(?!-)', pool)  # matches up to the first single hyphen
+        if not match:
+            raise OpenShiftCheckException(
+                "Host's docker says it is using LVM storage pool with invalid name '{}'".format(pool)


What makes it valid / invalid?!

"docker says" sounds strange to me ;)

Fair enough :)

This "should not happen" and it would be hard to see how it would be the user's fault if it does, so I blamed it on docker...

rhcarvalho · 2017-06-07T12:43:22Z

roles/openshift_health_checker/openshift_checks/docker_storage.py

+        if not match:
+            raise ValueError("Cannot convert to a byte size: " + string)
+        number, unit = match.groups()
+        return float(number) * units.get(str(unit).lower(), 1)


If the unit is anything that is not in the units dict we may incorrectly assume it is bytes.
I'd rather check explicitly. Empty = bytes, one of the units in the dicts = multiplication factor, otherwise error.

rhcarvalho · 2017-06-07T12:44:37Z

roles/openshift_health_checker/test/docker_storage_test.py

@@ -0,0 +1,214 @@
+import pytest
+
+


Too many blank lines, should be at most one separating groups of imports.

do we have to pylint test files :)

rhcarvalho · 2017-06-07T12:47:05Z

roles/openshift_health_checker/test/docker_storage_test.py

+    assert DockerStorage.is_active(task_vars=task_vars) == is_active
+
+
+non_atomic_tv = {"openshift": {"common": {"is_atomic": False}}}


What does tv stand for?! Ah, task_vars!

Please let's use the full name. It's enough cognitive load to have to know what is "task_vars", let alone figure out _tv -> _task_vars on something that is used only once...

sosiouxme · 2017-06-07T19:07:37Z

Yes this will annoy people on dev clusters where they likely will have devicemapper backed by a loopback device since it's the default setup. Once we include it in the install-time checks they can disable it the same as any check.

Incorporated docker_storage_driver into docker_storage as both need driver info. Corrected storage calculation to include VG free space, not just the current amount in the LV pool. Now makes no assumptions about pool name. Improved user messaging. Factored out some methods that can be shared with docker_image_availability.

sosiouxme · 2017-06-07T23:28:15Z

@rhcarvalho review issues hopefully addressed

sosiouxme · 2017-06-08T02:05:46Z

aos-ci-test

openshift-bot · 2017-06-08T02:51:32Z

success: "aos-ci-jenkins/OS_3.6_NOT_containerized, aos-ci-jenkins/OS_3.6_NOT_containerized_e2e_tests" for a092dff (logs)

openshift-bot · 2017-06-08T02:51:51Z

success: "aos-ci-jenkins/OS_3.5_containerized, aos-ci-jenkins/OS_3.5_containerized_e2e_tests" for a092dff (logs)

openshift-bot · 2017-06-08T02:53:27Z

success: "aos-ci-jenkins/OS_3.6_containerized, aos-ci-jenkins/OS_3.6_containerized_e2e_tests" for a092dff (logs)

openshift-bot · 2017-06-08T02:56:13Z

success: "aos-ci-jenkins/OS_3.5_NOT_containerized, aos-ci-jenkins/OS_3.5_NOT_containerized_e2e_tests" for a092dff (logs)

This avoids unintentionally overriding the value from `True` to `False`.

sosiouxme · 2017-06-09T13:15:57Z

roles/openshift_health_checker/openshift_checks/mixins.py

-        return msg, result.get("failed") or result.get("rc", 0) != 0, result.get("changed")
+        failed = result.get("failed", False) or result.get("rc", 0) != 0
+        changed = result.get("changed", False)
+        return msg, failed, changed


sorry, i see i missed doing this one. thanks.

sosiouxme · 2017-06-09T13:16:09Z

aos-ci-test

rhcarvalho · 2017-06-09T12:48:15Z

roles/openshift_health_checker/openshift_checks/docker_storage.py

+
+    def _check_dm_usage(self, driver_status, task_vars):
+        """
+        Backing assumptions: We expect devicemapper to be backed by an auto-expanding thin pool


Describing the assumptions is great, but what I'm missing here is some more standard info we normally have in a docstring:

https://www.python.org/dev/peps/pep-0257/#one-line-docstrings

The docstring is a phrase ending in a period. It prescribes the function or method's effect as a command ("Do this", "Return that"), not as a description; e.g. don't write "Returns the pathname ...".

...
The docstring for a function or method should summarize its behavior and document its arguments, return value(s), side effects, exceptions raised, and restrictions on when it can be called (all if applicable). Optional arguments should be indicated. It should be documented whether keyword arguments are part of the interface.

Looking at the code I see we construct a dict, modify it and return it... maybe we're just missing a line in the beginning, something like:

"""Check whether data and metadata volume usages are below a given threshold and return ... [Continue with the assumptions...] """

openshift-bot · 2017-06-09T15:15:10Z

success: "aos-ci-jenkins/OS_3.5_NOT_containerized, aos-ci-jenkins/OS_3.5_NOT_containerized_e2e_tests" for 3912fa8 (logs)

openshift-bot · 2017-06-09T15:17:54Z

success: "aos-ci-jenkins/OS_3.6_NOT_containerized, aos-ci-jenkins/OS_3.6_NOT_containerized_e2e_tests" for 3912fa8 (logs)

openshift-bot · 2017-06-09T15:18:13Z

success: "aos-ci-jenkins/OS_3.5_containerized, aos-ci-jenkins/OS_3.5_containerized_e2e_tests" for 3912fa8 (logs)

openshift-bot · 2017-06-09T15:20:46Z

success: "aos-ci-jenkins/OS_3.6_containerized, aos-ci-jenkins/OS_3.6_containerized_e2e_tests" for 3912fa8 (logs)

sosiouxme · 2017-06-09T15:23:02Z

[merge]

openshift-bot · 2017-06-09T15:23:55Z

Evaluated for openshift ansible merge up to 3912fa8

openshift-bot · 2017-06-09T17:15:58Z

continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/merge_pull_request_openshift_ansible/537/) (Base Commit: bbeaa20)

juanvallejo changed the title ~~Add docker storage check~~ [WIP] Add docker storage check Mar 28, 2017

rhcarvalho reviewed Mar 29, 2017

View reviewed changes

juanvallejo force-pushed the jvallejo/docker-storage-check branch from 365593b to 66c072b Compare March 29, 2017 14:31

juanvallejo commented Mar 29, 2017

View reviewed changes

juanvallejo mentioned this pull request Apr 6, 2017

add ram and non-containerized storage preflight check #3518

Merged

juanvallejo force-pushed the jvallejo/docker-storage-check branch from b1fc67b to 3225aa0 Compare April 6, 2017 22:19

juanvallejo force-pushed the jvallejo/docker-storage-check branch 3 times, most recently from 562f23a to e45d630 Compare April 26, 2017 19:26

juanvallejo changed the title ~~[WIP] Add docker storage check~~ Add docker storage check Apr 26, 2017

rhcarvalho suggested changes Apr 27, 2017

View reviewed changes

juanvallejo force-pushed the jvallejo/docker-storage-check branch 2 times, most recently from 437ce63 to c876e8d Compare May 1, 2017 15:03

juanvallejo force-pushed the jvallejo/docker-storage-check branch from c876e8d to b89bb5d Compare May 8, 2017 15:55

juanvallejo force-pushed the jvallejo/docker-storage-check branch from b89bb5d to 6859392 Compare May 10, 2017 19:28

juanvallejo force-pushed the jvallejo/docker-storage-check branch 3 times, most recently from 1d2dc64 to c4c75b8 Compare May 18, 2017 15:24

sosiouxme reviewed May 18, 2017

View reviewed changes

juanvallejo force-pushed the jvallejo/docker-storage-check branch from c4c75b8 to aa19712 Compare May 18, 2017 17:53

rhcarvalho reviewed Jun 7, 2017

View reviewed changes

rhcarvalho suggested changes Jun 7, 2017

View reviewed changes

sosiouxme force-pushed the jvallejo/docker-storage-check branch from d63d245 to bace709 Compare June 7, 2017 23:12

sosiouxme force-pushed the jvallejo/docker-storage-check branch from bace709 to a092dff Compare June 7, 2017 23:27

rhcarvalho approved these changes Jun 9, 2017

View reviewed changes

rhcarvalho added 2 commits June 9, 2017 14:25

Improve code readability

670e939

Consider previous value of 'changed' when updating

3912fa8

This avoids unintentionally overriding the value from `True` to `False`.

sosiouxme reviewed Jun 9, 2017

View reviewed changes

rhcarvalho reviewed Jun 9, 2017

View reviewed changes

openshift-bot merged commit 2f0b5b9 into openshift:master Jun 9, 2017

juanvallejo deleted the jvallejo/docker-storage-check branch June 12, 2017 20:22

		assert DockerStorage.is_active(task_vars=task_vars) == is_active


		non_atomic_tv = {"openshift": {"common": {"is_atomic": False}}}

Add docker storage check #3787

Add docker storage check #3787

Conversation

juanvallejo commented Mar 28, 2017 • edited by sosiouxme Loading

brenton commented Mar 29, 2017

rhcarvalho left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juanvallejo commented Mar 29, 2017

Choose a reason for hiding this comment

juanvallejo commented Apr 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juanvallejo Apr 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevekuznetsov Apr 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juanvallejo commented May 10, 2017

openshift-bot commented May 10, 2017

openshift-bot commented May 10, 2017

openshift-bot commented May 10, 2017

openshift-bot commented May 10, 2017

Choose a reason for hiding this comment

juanvallejo commented May 18, 2017

sosiouxme commented May 18, 2017

sosiouxme commented Jun 7, 2017

rhcarvalho commented Jun 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhcarvalho commented Jun 7, 2017

rhcarvalho left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sosiouxme Jun 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sosiouxme Jun 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sosiouxme commented Jun 7, 2017 • edited Loading

sosiouxme commented Jun 7, 2017

sosiouxme commented Jun 8, 2017

openshift-bot commented Jun 8, 2017

openshift-bot commented Jun 8, 2017

juanvallejo commented Mar 28, 2017 •

edited by sosiouxme

Loading

juanvallejo Apr 28, 2017 •

edited

Loading

stevekuznetsov Apr 28, 2017 •

edited

Loading

sosiouxme Jun 7, 2017 •

edited

Loading

sosiouxme Jun 7, 2017 •

edited

Loading

sosiouxme commented Jun 7, 2017 •

edited

Loading

openshift-bot commented Jun 9, 2017 •

edited

Loading