Fix RPATH and install_name handling on macOS for uninstalled and installed binaries #3691

nirbheek · 2018-06-05T12:19:47Z

When we link to an external library either with find_library() without any dirs:, or with dependency(), we should be able to run uninstalled out of the box without having to set any environment variables or other shenanigans.

This is especially important on macOS because only the system frameworks directory is in the default runtime path, and all other frameworks and libraries need to be found with RPATH or absolute path to the dylib.

I will be adding commits that fix this next.

ignatenkobrain · 2018-06-05T13:53:31Z

mesonbuild/compilers/c.py

+        if isinstance(extra_dirs, str):
+            extra_dirs = [extra_dirs]
+        key = (tuple(self.exelist), libname, tuple(extra_dirs), code, libtype)
+        if not key in self.find_library_cache:


[Flake8]

[E713] test for membership should be 'not in'

mesonbuild/backend/ninjabackend.py

+            # outputs to not be installed).
+            custom_install_dir = True
+        else:
+            custom_install_dir = False


ignatenkobrain · 2018-06-07T22:52:47Z

mesonbuild/mintro.py

-            res[os.path.join(installdata.build_dir, path)] = os.path.join(installdata.prefix, installdir, os.path.basename(path))
+        for t in installdata.targets:
+            res[os.path.join(installdata.build_dir, t.fname)] = \
+                    os.path.join(installdata.prefix, t.outdir, os.path.basename(t.fname))


[Flake8]

[E126] continuation line over-indented for hanging indent

nirbheek · 2018-06-08T00:27:18Z

CCing @ePirat @SoapGentoo @siriobalmelli @tschoonj @ryandesign @mojca @ilovezfs (based on participation in previous PRs and issues)

Reviews and/or comments are welcome. Just reading the commit messages should suffice to understand what the new behaviour will be.

ePirat · 2018-06-08T19:27:38Z

This looks like a good solution to me.

SoapGentoo · 2018-06-10T08:40:23Z

mesonbuild/scripts/depfixer.py

        if new_rpath:
-            subprocess.check_call(['install_name_tool', '-add_rpath', new_rpath, fname],
+            args += ['-add_rpath', new_rpath]
+        # Rewrite -install_name @rpath/libfoo.dylib to /path/to/libfoo.dylib


are you using absolute install_names now instead of relative ones? This would be a major breaking change

are you using absolute install_names now instead of relative ones?

Yes, see commit message.

This would be a major breaking change

Can't break what is already broken ;)

textshell · 2018-06-10T17:13:16Z

mesonbuild/dependencies/base.py

+                libpaths.append(lib[2:])
+                continue
+            elif lib.startswith('-l'):
+                args = self.compiler.find_library(lib[2:], self.env, libpaths, libtype)


My code had an error message for self.compiler is None. I don't remember the exact code paths that triggered that but i think we need this here too.

Yes, I noticed that but I didn't want to add untested or potentially dead code. So I thought I'd skip it and see what breaks so we know what that case is.

I'm pretty sure it's not dead. And i don't think leaving preventable stack traces around for unsuspecting users is a good approach.

I don't like preemptively adding checks for cases without understanding what the consequences of those cases are, since silently taking a bad decision is worse. I'll add an assert telling the user to file a bug.

can you try in a java only (as in meson languages) project? I know that that is nonsensical, but maybe it's an easy way to reproduce it.

I read the code again, and this can happen if the project doesn't have a clike language in the list of languages in project() or add_languages(): rust, csharp, python. So I re-added the check.

textshell · 2018-06-10T17:27:42Z

run_unittests.py

@@ -484,7 +484,7 @@ def setUp(self):
        src_root = os.path.join(os.getcwd(), src_root)
        self.src_root = src_root
        self.prefix = '/usr'
-        self.libdir = os.path.join(self.prefix, 'lib')
+        self.libdir = 'lib'


Is this save? Does any other test use self.libdir?

It is safe w.r.t. passing to meson. Other tests do use self.libdir and those have been changed in the same commit.

textshell · 2018-06-10T17:28:26Z

run_unittests.py

@@ -3198,7 +3198,6 @@ def test_old_gnome_module_codepaths(self):
        self.build()
        mesonbuild.modules.gnome.native_glib_version = None

-    @unittest.skipIf(shutil.which('pkg-config') is None, 'Pkg-config not found.')


Do we really want to hard depend on pkg-config for the test suite on *nix now? I feel that this does not belong into this PR.

I wanted to ensure that the test would not be skipped on the CI just because pkg-config wasn't available. I'll change it to skip it when we're not on CI.

Please do. I think completely removing these skips should be the topic of another PR.

textshell · 2018-06-10T18:09:01Z

run_unittests.py

-
-        TODO: On Windows, this can test whether PATH is set properly
+        PkgConfigDependency) works. On macOS, this workflow works out of the
+        box. On Linux, BSDs, Windows, etc, you need to set extra arguments such


That this does not work on linux and BSDs sounds more like a bug in meson that needs to be fixed?
So i think that comment is misleading.

No, it's not a bug. If you do not install your libraries into the runtime linker paths (ld.so.conf), you need to set the install_rpath: or your libraries won't be found.

It says """Test that uninstalled usage of an external library [...] works""". So maybe that's not what it seems to say? Not sure.

When we link to an external library either with find_library() without any dirs:, or with dependency(), we should be able to run uninstalled out of the box without having to set any environment variables or other shenanigans. This is especially important on macOS because only the system frameworks directory is in the default runtime path, and all other frameworks and libraries need to be found with RPATH or absolute path to the dylib.

This allows us to more aggressively de-dup them, and also sets RPATHs to all libraries that are not in the system linker paths so that binaries can be run uninstalled without any special steps. These RPATHs will be wiped on install, so they do not affect reproducible builds. De-duping: Fixes #2150 Fixes #2118 Fixes #3071 RPATHs: Fixes #314 Fixes #2881 Also fixes the uninstalled usage portions of: #3038 #3077

The install name is used by consumers of the library to find the library at runtime. If it's @rpath/libfoo.dylib, all consumers must manually add the library path to RPATH, which is not what people expect. Almost everyone sets the library install name as the full path to the library, and this is done at install time with install_name_tool.

On macOS, we set the install_name for built libraries to @rpath/libfoo.dylib, and when linking to the library, we set the RPATH to its path in the build directory. This allows all built binaries to be run as-is from the build directory (uninstalled). However, on install, we have to strip all the RPATHs because they point to the build directory, and we change the install_name of all built libraries to the absolute path to the library. This causes the install name in binaries to be out of date. We now change that install name to point to the absolute path to each built library after installation. Fixes #3038 Fixes #3077 With this, the default workflow on macOS matches what everyone seems to do, including Autotools and CMake. The next step is providing a way for build files to override the install_name that is used after installation for use with, f.ex., private libraries when combined with the install_rpath: kwarg on targets.

Our appveyor configuration provides pkg-config when building for mingw, cygwin, msvc, etc. Of course, people manually running the tests won't require pkg-config.

jpakkane · 2018-06-10T22:19:51Z

Overall this is looking good. Nice job.

Eventually we will need some solution to produce relocatable app bundles. Basically this but with an actual API. :D

Maybe a base option with values absolute or bundle? I'd prefer an option with explicit values rather than a boolean, because it is easier to add new values to the former but very difficult for the latter.

nirbheek · 2018-06-10T22:38:38Z

Eventually we will need some solution to produce relocatable app bundles. Basically this but with an actual API. :D

Yeah, we were talking about this on IRC today. Seems like everyone is in favour of it.

Maybe a base option with values absolute or bundle? I'd prefer an option with explicit values rather than a boolean, because it is easier to add new values to the former but very difficult for the latter.

Makes sense to me, and it would be good to build that around an app-based test case (perhaps using meson to build an actual app for the macOS app store) so we know we cover the use-case correctly.

ePirat · 2018-06-10T23:25:21Z

I'm happy to help with the macOS app bundle stuff, I've spent quite a long time investigating them, so if there will be any PR or Bug tracking this, please mention me. 😄

jpakkane · 2018-06-17T12:39:40Z

What's the status of this? Is this ready to merge or do we need the toggle option first to avoid potential breakage?

ePirat · 2018-06-17T12:51:48Z

Whats needed is an option to set the install_name to something else if desired. But that can be done in another PR. IMO the current behavior of meson when installing is broken on macOS and this patch fixes it, so I do not think it would break something.

jpakkane · 2018-06-17T14:41:00Z

LGTM, feel free to merge if you think this is ready.

SoapGentoo · 2018-06-17T23:03:10Z

It would still be nice to get the old behaviour back before cutting a new release as an option. For instance, for building in Anaconda, we need the old behaviour, as it is the only way to make macOS binaries fully relocatable.

nirbheek · 2018-06-18T02:01:05Z

It would still be nice to get the old behaviour back before cutting a new release as an option

That's not really feasible at this point because we have a week to release, and it's not clear whether just a builtin option is sufficient for that, or if we need something more.

The workaround currently is to manually run install_name_tool on the outputs after install; a run target or a script would be a good way to do it.

nirbheek · 2018-06-18T02:01:47Z

LGTM, feel free to merge if you think this is ready.

Someone reported an issue with shared modules to me, will look into that and then merge this.

nirbheek · 2018-06-18T06:33:13Z

Someone reported an issue with shared modules to me, will look into that and then merge this.

This turned out to be something unrelated to Meson; the library the module was linking to had an incorrect install_name, and had been built with cmake.

jon-turney · 2018-06-18T18:28:30Z

After e3757e3, generating for gtkdapp fails with ERROR: Language D does not support library finding. No idea what that's about 😕

See https://travis-ci.org/jon-turney/meson-corpus-test/jobs/393722545#L1265

textshell · 2018-06-18T18:40:12Z

We are missing a test somewhere that a d only project can use pkgconfig dependencies.

ExternalDependency's __init__ currently only whitelists

             # Try to find a compiler that this dependency can use for compiler
             # checks. It's ok if we don't find one.
             for lang in ('c', 'cpp', 'objc', 'objcpp', 'fortran', 'd'):

Can we extend that to D without breaking other parts? Or does this change need a fallback that works without self.compiler.find_library?

nirbheek · 2018-06-19T03:44:45Z

Can we extend that to D without breaking other parts

That list already contains D? And that is actually what's breaking this, because....

ERROR: Language D does not support library finding. No idea what that's about

I looked at the code, and it looks like D doesn't have a find_library() implementation, so the code is incorrectly assuming that:

self.compiler being defined means that the compiler has a find_library() implementation
The find_library() implementation will find C-like libraries, which is what pkg-config is always used for.

I guess we need to differentiate between "C/C++/Objc/Objc++/Fortran" (aka c-abi) and "languages that can consume C/C++/Objc/Objc++ libraries" (c-interop).

See: mesonbuild/meson#3691

jon-turney · 2018-06-27T13:06:54Z

After e3757e3

This commit also seems to regress building gst-transcoder (a subproject of pitivi):

# ninja -C _build GstTranscoder-1.0.gir -v
ninja: Entering directory `_build'
[1/1] /usr/bin/g-ir-scanner -pthread -I/usr/include/gobject-introspection-1.0 -I/usr/include/glib-2.0 -I/usr/lib/i386-linux-gnu/glib-2.0/include --no-libtool --namespace=GstTranscoder --nsversion=1.0 --warn-all --output GstTranscoder-1.0.gir '--add-init-section=extern gboolean gst_init(gint *argc, gchar **argv); gst_init(NULL,NULL);' -I/pitivi/subprojects/gst-transcoder/ -I/pitivi/subprojects/gst-transcoder/_build/ --filelist=/pitivi/subprojects/gst-transcoder/_build/gsttranscoder-1.0@sha/GstTranscoder_1.0_gir_filelist --include=GObject-2.0 --include=Gst-1.0 --include=GstPbutils-1.0 --symbol-prefix=gst_ --identifier-prefix=Gst --cflags-begin -I/usr/include/glib-2.0 -I/usr/lib/i386-linux-gnu/glib-2.0/include -pthread -I/usr/include/gstreamer-1.0 --cflags-end -L/pitivi/subprojects/gst-transcoder/_build/ /usr/lib/i386-linux-gnu/libglib-2.0.so /usr/lib/i386-linux-gnu/libgobject-2.0.so /usr/lib/i386-linux-gnu/libgstreamer-1.0.so /usr/lib/i386-linux-gnu/libgstpbutils-1.0.so --add-include-path=/usr/share/gir-1.0 --library gsttranscoder-1.0
FAILED: GstTranscoder-1.0.gir
/usr/bin/g-ir-scanner -pthread -I/usr/include/gobject-introspection-1.0 -I/usr/include/glib-2.0 -I/usr/lib/i386-linux-gnu/glib-2.0/include --no-libtool --namespace=GstTranscoder --nsversion=1.0 --warn-all --output GstTranscoder-1.0.gir '--add-init-section=extern gboolean gst_init(gint *argc, gchar **argv); gst_init(NULL,NULL);' -I/pitivi/subprojects/gst-transcoder/ -I/pitivi/subprojects/gst-transcoder/_build/ --filelist=/pitivi/subprojects/gst-transcoder/_build/gsttranscoder-1.0@sha/GstTranscoder_1.0_gir_filelist --include=GObject-2.0 --include=Gst-1.0 --include=GstPbutils-1.0 --symbol-prefix=gst_ --identifier-prefix=Gst --cflags-begin -I/usr/include/glib-2.0 -I/usr/lib/i386-linux-gnu/glib-2.0/include -pthread -I/usr/include/gstreamer-1.0 --cflags-end -L/pitivi/subprojects/gst-transcoder/_build/ /usr/lib/i386-linux-gnu/libglib-2.0.so /usr/lib/i386-linux-gnu/libgobject-2.0.so /usr/lib/i386-linux-gnu/libgstreamer-1.0.so /usr/lib/i386-linux-gnu/libgstpbutils-1.0.so --add-include-path=/usr/share/gir-1.0 --library gsttranscoder-1.0
/usr/lib/gcc/i686-linux-gnu/7/include/stddef.h:435: syntax error, unexpected identifier in '  __float128 __max_align_f128 __attribute__((__aligned__(__alignof(__float128))));' at '__float128'
g-ir-scanner: link: cc -o /pitivi/subprojects/gst-transcoder/_build/tmp-introspect2ggjc_ha/GstTranscoder-1.0 /pitivi/subprojects/gst-transcoder/_build/tmp-introspect2ggjc_ha/GstTranscoder-1.0.o -L. -Wl,-rpath,. -Wl,--no-as-needed -lgsttranscoder-1.0 -L/pitivi/subprojects/gst-transcoder/_build/ -Wl,-rpath,/pitivi/subprojects/gst-transcoder/_build/ -lgio-2.0 -lgobject-2.0 -Wl,--export-dynamic -lgmodule-2.0 -pthread -lglib-2.0
/usr/bin/i686-linux-gnu-ld: /pitivi/subprojects/gst-transcoder/_build/tmp-introspect2ggjc_ha/GstTranscoder-1.0.o: undefined reference to symbol 'gst_init'
//usr/lib/i386-linux-gnu/libgstreamer-1.0.so.0: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
linking of temporary binary failed: Command '['cc', '-o', '/pitivi/subprojects/gst-transcoder/_build/tmp-introspect2ggjc_ha/GstTranscoder-1.0', '/pitivi/subprojects/gst-transcoder/_build/tmp-introspect2ggjc_ha/GstTranscoder-1.0.o', '-L.', '-Wl,-rpath,.', '-Wl,--no-as-needed', '-lgsttranscoder-1.0', '-L/pitivi/subprojects/gst-transcoder/_build/', '-Wl,-rpath,/pitivi/subprojects/gst-transcoder/_build/', '-lgio-2.0', '-lgobject-2.0', '-Wl,--export-dynamic', '-lgmodule-2.0', '-pthread', '-lglib-2.0']' returned non-zero exit status 1.
ninja: build stopped: subcommand failed.

compared to:

# git -C /meson reset --hard e3757e3^
HEAD is now at 5467eed1 Test that binaries that use external libraries work
# rm -r _build && meson _build
[...]
# ninja -C _build GstTranscoder-1.0.gir -v
ninja: Entering directory `_build'
[1/1] /usr/bin/g-ir-scanner -pthread -I/usr/include/gobject-introspection-1.0 -I/usr/include/glib-2.0 -I/usr/lib/i386-linux-gnu/glib-2.0/include --no-libtool --namespace=GstTranscoder --nsversion=1.0 --warn-all --output GstTranscoder-1.0.gir '--add-init-section=extern gboolean gst_init(gint *argc, gchar **argv); gst_init(NULL,NULL);' -I/pitivi/subprojects/gst-transcoder/ -I/pitivi/subprojects/gst-transcoder/_build/ --filelist=/pitivi/subprojects/gst-transcoder/_build/gsttranscoder-1.0@sha/GstTranscoder_1.0_gir_filelist --include=GObject-2.0 --include=Gst-1.0 --include=GstPbutils-1.0 --symbol-prefix=gst_ --identifier-prefix=Gst --cflags-begin -I/usr/include/glib-2.0 -I/usr/lib/i386-linux-gnu/glib-2.0/include -pthread -I/usr/include/gstreamer-1.0 --cflags-end -L/pitivi/subprojects/gst-transcoder/_build/ --extra-library=glib-2.0 --extra-library=gobject-2.0 --extra-library=gstreamer-1.0 --extra-library=gstpbutils-1.0 --add-include-path=/usr/share/gir-1.0 --library gsttranscoder-1.0
/usr/lib/gcc/i686-linux-gnu/7/include/stddef.h:435: syntax error, unexpected identifier in '  __float128 __max_align_f128 __attribute__((__aligned__(__alignof(__float128))));' at '__float128'
g-ir-scanner: link: cc -o /pitivi/subprojects/gst-transcoder/_build/tmp-introspecteos3lfhh/GstTranscoder-1.0 /pitivi/subprojects/gst-transcoder/_build/tmp-introspecteos3lfhh/GstTranscoder-1.0.o -L. -Wl,-rpath,. -Wl,--no-as-needed -lgsttranscoder-1.0 -lglib-2.0 -lgobject-2.0 -lgstreamer-1.0 -lgstpbutils-1.0 -L/pitivi/subprojects/gst-transcoder/_build/ -Wl,-rpath,/pitivi/subprojects/gst-transcoder/_build/ -lgio-2.0 -lgobject-2.0 -Wl,--export-dynamic -lgmodule-2.0 -pthread -lglib-2.0

(difference seems to be that g-ir-scanner gets passed absolute paths to libraries, rather than library names with --extra-library)

mojca · 2018-06-27T13:32:11Z

I just wanted to mention that this change seems to mostly work for me, however I'm still experiencing issues when running test on a particular piece of software:
https://gitlab.gnome.org/GNOME/libhttpseverywhere/blob/master/test/meson.build
but it's not yet clear to me whether that's an issue in Meson or an issue with suboptimal meson.build file. What happens is that the library is built and keeps @rpath before it is installed. There is an executable to test that library (by running test/httpseverywhere_test), but that one does not manage to find the library. I need more time and testing though. After installation the library seems to be ok.

ePirat · 2018-06-27T14:18:28Z

@mojca Sounds like the LC_RPATH is not correctly set on the test executable? Could you check with otool -l test/httpseverywhere_test | grep LC_RPATH -C 3 maybe?

See: mesonbuild/meson#3691

meson-internal was introduced in Homebrew#25667 A patch was proposed upstream in mesonbuild/meson#2577, but never merged. It looks like the issues were fixed by mesonbuild/meson#3555 mesonbuild/meson#3691 mesonbuild/meson#3356 Now that we are one year later, and things have settled down, try to use meson everywhere and drop meson-internal, which is outdated.

This works around an undocumented semantic in Meson as regard dylibs' IDs. The fix performed for dyld in 2c058e5 meant that anyone attempting to use relocated libraries within Meson would have their libraries or executables crash at launch, since Meson does not insert any RPATH entries for dependencies, only for build targets in the current project. See: https://gitlab.freedesktop.org/gstreamer/cerbero/-/merge_requests/1478 mesonbuild/meson#3691 (NOTE: for a more comprehensive fix, implement the post-install step on osxuniversalgenerator.py:do_merge inside the copy-pc action.) Part-of: <https://gitlab.freedesktop.org/gstreamer/cerbero/-/merge_requests/1485>

This works around an undocumented semantic in Meson as regard dylibs' IDs. The fix performed for dyld in 2c058e5 meant that anyone attempting to use relocated libraries within Meson would have their libraries or executables crash at launch, since Meson does not insert any RPATH entries for dependencies, only for build targets in the current project. See: https://gitlab.freedesktop.org/gstreamer/cerbero/-/merge_requests/1478 mesonbuild/meson#3691 (NOTE: for a more comprehensive fix, implement the post-install step on osxuniversalgenerator.py:do_merge inside the copy-pc action.) Part-of: <https://gitlab.freedesktop.org/gstreamer/cerbero/-/merge_requests/1491>

nirbheek added the OS:macos Issues specific to Apple Operating Systems like MacOS and iOS label Jun 5, 2018

nirbheek mentioned this pull request Jun 5, 2018

Misc library search fixes #3693

Merged

nirbheek force-pushed the nirbheek/macos-rpath-uninstalled branch from ccb4e04 to 5726a9e Compare June 5, 2018 13:52

ignatenkobrain reviewed Jun 5, 2018

View reviewed changes

nirbheek force-pushed the nirbheek/macos-rpath-uninstalled branch from 5726a9e to 434d702 Compare June 5, 2018 13:58

nirbheek mentioned this pull request Jun 7, 2018

PkgConfigDependency: Also try to eagerly resolve libraries path without static mode. #3414

Closed

nirbheek force-pushed the nirbheek/macos-rpath-uninstalled branch from 434d702 to 837f9ad Compare June 7, 2018 19:26

ignatenkobrain reviewed Jun 7, 2018

View reviewed changes

mesonbuild/backend/ninjabackend.py

# outputs to not be installed).

custom_install_dir = True

else:

custom_install_dir = False

This comment was marked as resolved.

Sign in to view

nirbheek force-pushed the nirbheek/macos-rpath-uninstalled branch 3 times, most recently from 198725f to 5903beb Compare June 7, 2018 22:52

ignatenkobrain reviewed Jun 7, 2018

View reviewed changes

nirbheek changed the title ~~Test that binaries that use external libraries work~~ Fix RPATH and install_name handling on macOS for uninstalled and installed binaries Jun 7, 2018

nirbheek force-pushed the nirbheek/macos-rpath-uninstalled branch 3 times, most recently from 6893a70 to 02eb35e Compare June 8, 2018 00:17

SoapGentoo approved these changes Jun 10, 2018

View reviewed changes

textshell reviewed Jun 10, 2018

View reviewed changes

nirbheek force-pushed the nirbheek/macos-rpath-uninstalled branch from 02eb35e to 7621c83 Compare June 10, 2018 18:37

nirbheek added 4 commits June 11, 2018 00:20

unittests: Assert that we have pkg-config on all CI

176087e

Our appveyor configuration provides pkg-config when building for mingw, cygwin, msvc, etc. Of course, people manually running the tests won't require pkg-config.

nirbheek force-pushed the nirbheek/macos-rpath-uninstalled branch from 7621c83 to 176087e Compare June 10, 2018 18:50

nirbheek merged commit d2d1a7c into master Jun 18, 2018

nirbheek deleted the nirbheek/macos-rpath-uninstalled branch June 18, 2018 06:33

This was referenced Jun 18, 2018

macOS: add libtool compatible versioning and fix rpath issues #2577

Closed

Add static as keyword to find_library #3747

Merged

nirbheek mentioned this pull request Jun 19, 2018

Use c-library languages in dependencies #3757

Merged

mojca added a commit to mojca/macports-ports that referenced this pull request Jun 25, 2018

meson: use git version to fix RPATH handling

aa2a755

See: mesonbuild/meson#3691

mojca mentioned this pull request Jun 25, 2018

meson: use git version to fix RPATH handling macports/macports-ports#2090

Merged

11 tasks

mojca added a commit to macports/macports-ports that referenced this pull request Jun 30, 2018

meson: use git version to fix RPATH handling

ba12033

See: mesonbuild/meson#3691

iMichka mentioned this pull request Aug 24, 2019

meson-internal: delete Homebrew/homebrew-core#43437

Closed

5 tasks

SpaceIm mentioned this pull request Mar 9, 2022

[conan-center] KB-H071 Validate relocable shared libraries conan-io/hooks#403

Draft

1 task

eli-schwartz mentioned this pull request Mar 9, 2022

How to install relocable shared lib on macOS (install_name set to @rpath)? #10099

Open

Fix RPATH and install_name handling on macOS for uninstalled and installed binaries #3691

Fix RPATH and install_name handling on macOS for uninstalled and installed binaries #3691

Conversation

nirbheek commented Jun 5, 2018

Choose a reason for hiding this comment

This comment was marked as resolved.

Choose a reason for hiding this comment

nirbheek commented Jun 8, 2018

ePirat commented Jun 8, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpakkane commented Jun 10, 2018

nirbheek commented Jun 10, 2018

ePirat commented Jun 10, 2018

jpakkane commented Jun 17, 2018

ePirat commented Jun 17, 2018

jpakkane commented Jun 17, 2018

SoapGentoo commented Jun 17, 2018 • edited Loading

nirbheek commented Jun 18, 2018 • edited Loading

nirbheek commented Jun 18, 2018

nirbheek commented Jun 18, 2018

jon-turney commented Jun 18, 2018

textshell commented Jun 18, 2018

nirbheek commented Jun 19, 2018 • edited Loading

jon-turney commented Jun 27, 2018

mojca commented Jun 27, 2018 • edited Loading

ePirat commented Jun 27, 2018

SoapGentoo commented Jun 17, 2018 •

edited

Loading

nirbheek commented Jun 18, 2018 •

edited

Loading

nirbheek commented Jun 19, 2018 •

edited

Loading

mojca commented Jun 27, 2018 •

edited

Loading