GitLab Integration #3327

humitos · 2017-11-28T00:50:30Z

This is the continuation of the work done by @saily at #1870

I updated the code to follow the new codebase and also use the apiv4 endpoints from gitlab. Besides, there are a bunch of style-things.

Missing things:

update webhook method
more testing
send proper context to the sphinx theme

Closes #1441
Closes #1870
Closes #1443

Sync gitlab repos

…ries

Support private repositories and cleanup

…ocs.org into saily-sync-gitlab-repos - Skip test_oauth.py from flake8 since it has a lot of "Too long lines" - Add some style fixes following the new rules - Minimum adaptation for the registry OAuth classes

humitos · 2017-11-28T00:51:35Z

.pre-commit-config.yaml

@@ -26,6 +26,7 @@ repos:
      'flake8-string-format',
      'flake8-tidy-imports',
      'flake8-todo']
+      exclude: 'test_oauth.py'


This file has a lot of long lines that I don't want to fix now :)

humitos · 2017-11-28T00:51:41Z

docs/webhooks.rst


 * Go to the **Settings** page for your project
 * Click **Integrations**
 * For **URL**, use the URL of the integration on Read the Docs, found on the
  :ref:`integration detail page <integration-detail>` page
-* Leave the default **Push events** selected
+* Leave the default **Push events** selected and mark **Tag push events** also


Not sure, but I think we need the Tag push events here also...

Seems like we'd want them. +1

humitos · 2017-11-28T00:52:34Z

readthedocs/builds/utils.py

+# TODO: I think this can be different than `gitlab.com`
+# self.adapter.provider_base_url
+GL_REGEXS = [
+    re.compile('gitlab.com/(.+)/(.+)(?:\.git){1}'),


Are we going to support private gitlab instalation? How this should looks like?

We will, but this integration will be on the .com, we won't put that code here yet.

OK. So, I think the TODO note is good for now. I will leave it there

humitos · 2017-11-28T00:54:04Z

readthedocs/oauth/services/gitlab.py

+        re.escape(urlparse(adapter.provider_base_url).netloc))
+    default_avatar = {
+        'repo': urljoin(settings.MEDIA_URL, 'images/fa-bookmark.svg'),
+        'org': urljoin(settings.MEDIA_URL, 'images/fa-users.svg'),


Do we want these default avatars? Github and Bitbucket are not using nothing similar to this.

I'd say no, we use the github/bitbucket supplied avatar as the only source I believe.

or, alternatively, we edit all to include a default avatar. Perhaps in another PR later?

I will remove the default avatars and its files.

We can make it in another PR for all the services, but I think it doesn't worth it really.

humitos · 2017-11-28T00:56:32Z

readthedocs/oauth/services/gitlab.py

+
+        # The ID or URL-encoded path of the project
+        # https://docs.gitlab.com/ce/api/README.html#namespaced-path-encoding
+        repo_id = '{}%2F{}'.format(owner, repo)


This is enough here but maybe it's better to find a way to get the proper repo:id from gitlab

this is probably fine. Github gives us this information in the api return, but i don't see anything wrong with this as long as owner and repo are supplied from the API.

I found a better way to do this,

# The ID or URL-encoded path of the project # https://docs.gitlab.com/ce/api/README.html#namespaced-path-encoding repo_id = json.loads(project.remote_repository.json).get('id')

It uses the RemoteRepository response saved in our database, which I think it's better than parsing a URL, more specifically in GitLab that the URL could have different domains.

So, I'll try to use the repo id here instead of owner + repo and we can get rid of the GL_REGEX also, since it won't be needed anymore.

The regex is also used at this point:

https://github.com/rtfd/readthedocs.org/blob/4db7339aac7a52d7a35b6359eff8032e8aadf6ae/readthedocs/doc_builder/backends/sphinx.py#L72-L75

So, we will need to figure this out for that case, at least.

humitos · 2017-11-28T00:57:24Z

readthedocs/oauth/services/gitlab.py

+        pass
+
+    # @classmethod
+    # def get_token_for_project(cls, project, force_local=False):


I think this is only needed by Github, but I don't understand how it works yet...

humitos · 2017-11-28T00:58:04Z

readthedocs/settings/base.py

+        'gitlab': {
+            'SCOPE': [
+                'api',
+                'read_user',


Not sure, but maybe the sudo permission is also required.

We'll want to make sure here. Folks are really particular about the permissions we request

I'm not sure to understand their documentation. Seems poor to me: https://docs.gitlab.com/ce/user/profile/personal_access_tokens.html#limiting-scopes-of-a-personal-access-token

So far so good, I was able to connect the account and setup the webhooks using only these two.

humitos · 2017-11-28T15:04:33Z

readthedocs/settings/base.py

@@ -228,15 +231,6 @@ def INSTALLED_APPS(self):  # noqa
    CELERY_CREATE_MISSING_QUEUES = True

    CELERY_DEFAULT_QUEUE = 'celery'
-    # Wildcards not supported: https://github.com/celery/celery/issues/150
-    CELERY_ROUTES = {


I think this is not used anymore. None of those tasks exist.

We've moved to marking the queue on @task declaration

…/gitlab/integration

humitos · 2017-11-29T14:35:22Z

Just a comment about our auto-linting configuration:

@agjohnson I would say that we should disable D202 since I didn't find a way to make it automatically with the tools that I know. Having it enabled will produce a lot of this kind of manual changes: 0ce1702

The other way around is supported by yapf and added automatically: making that space mandatory, c2f16d6#diff-4d8c13c2356e3f836af6d1cbfd975203

humitos · 2017-11-29T14:37:55Z

@agjohnson another thing that it's missing here (but probably for another PR that also change the theme) is the context sent to sphinx at https://github.com/rtfd/readthedocs.org/blob/4db7339aac7a52d7a35b6359eff8032e8aadf6ae/readthedocs/doc_builder/backends/sphinx.py#L112

Let me know how you prefer to work on that.

ericholscher

Looks great to me! I didn't go through the gitlab docs really to see if we're doing everything right, but the code looks great. 👍

ericholscher · 2017-11-29T15:31:42Z

docs/features.rst

@@ -13,7 +13,7 @@ More information can be found in the :doc:`vcs` page.
 Auto-updating
 -------------

-The :doc:`webhooks` page talks about the different ways you can ping RTD to let us know your project has been updated. We have official support for GitHub, and anywhere else we have a generic post-commit hook that allows you to POST to a URL to get your documentation built.
+The :doc:`webhooks` page talks about the different ways you can ping RTD to let us know your project has been updated. We have official support for GitHub, Bitbucket and GitLab, and anywhere else we have a generic post-commit hook that allows you to POST to a URL to get your documentation built.


<3 doc updates.

Credits to @saily, the original author :)

ericholscher · 2017-11-29T15:32:15Z

docs/webhooks.rst


 * Go to the **Settings** page for your project
 * Click **Integrations**
 * For **URL**, use the URL of the integration on Read the Docs, found on the
  :ref:`integration detail page <integration-detail>` page
-* Leave the default **Push events** selected
+* Leave the default **Push events** selected and mark **Tag push events** also


Seems like we'd want them. +1

ericholscher · 2017-11-29T15:34:10Z

readthedocs/oauth/services/base.py

        """
        # TODO Replace this check by keying project to remote repos
        return (
            cls.url_pattern is not None and
-            cls.url_pattern.search(project.repo) is not None
-        )
+            cls.url_pattern.search(project.repo) is not None)


All these are linting changes, I think? This is a bit of what I was worried about w/ the auto-linting, making it hard to catch real changes in the massive linting changes.

Yes, they are linting changes.

I try to make these changes in separated commits but sometimes it's complicated.

I don't know what's the best way for this. I think we can make a huge PR with all the automcatic linting changes and fix the specific one by hand after merging it. That will make PR with linting changes smaller in the future.

But I don't know what's best, honestly (I'm used to review PR with a lot of linting changes because we followed the same as I'm doing here but...)

We discussed this a little. I don't think large commit is the answer yet, this shouldn't be priority. We can however be more careful in making PRs where autolint is a separate commit. This should be more easy to review, but requires some careful git rebasing depending on your workflow (if you are commiting autolint changes with your logic changes)

Or just a separate PR, whatever is easier to manage for your own sake.

ericholscher · 2017-11-29T15:35:55Z

readthedocs/oauth/services/gitlab.py

+                log.debug(
+                    'Not importing %s because mismatched orgs',
+                    fields['name'],
+                )


Wonder if this should be a higher priority? Will this cause breakage for the user?

Honestly. I'm not sure. Github and Bitbucket code do exactly the same.

It's kind of a weird case, because it should happend that user U created the repo R with organization O. Then, user UU tries to import the same repo R but with organization OO and that generates this error, right?

That is correct. Probably no need to surface this any harder, it's a weird edge case.

agjohnson · 2017-11-29T17:16:46Z

another thing that it's missing here

@humitos could you explain this more, i don't quite understand

re: #3327 (comment)

humitos · 2017-11-29T17:42:16Z

@agjohnson we are sending some context to sphinx (github_user, github_repo, etc). So, we will to send the same information about gitlab_ and the theme should have to accept it and do the magic to add the Edit in Gitlab link, and more.

agjohnson · 2017-11-29T22:23:53Z

No need to do this on signal, it can be included in the block above, similar to github/butbucket. But good point, i didn't realize this was missing.

humitos · 2017-11-29T22:40:40Z

I also found there is some helper methods missing. For example: https://github.com/rtfd/readthedocs.org/blob/61dc4b5389197c12c7ebe92a0968eebb551115c2/readthedocs/builds/models.py#L229

agjohnson · 2017-12-01T06:55:21Z

Looks like a few more pieces missing. I'll assign this to @humitos as a WIP until it's wrapped up next week.

…/gitlab/integration

agjohnson

Okay ... :)

One last change, we just need a test case for this GitLab URL in our context data. With that, I swear this will be mergeable.

agjohnson · 2017-12-05T23:34:12Z

readthedocs/builds/models.py

+            path=filename,
+            source_suffix=source_suffix,
+            action=action_string,
+        )


We should test this method

fcasson · 2020-05-21T15:44:44Z

Can anyone advise how to customise the gitlab_url - we have our own deployment. Following https://sphinx-rtd-theme.readthedocs.io/en/stable/configuring.html#file-wide-metadata, I tried to set in conf.py

html_context = {
"display_gitlab": True, # Integrate Gitlab
"gitlab_user": "myuser", # Organization or User
"gitlab_repo": "myrepo", # Repo name
"gitlab_version": "develop", # Branch
"conf_py_path": "/webpages/wiki/", # Path in the checkout to the docs root
"gitlab_url": "https://git.mylocalgitlab.org.uk/"
}

But it doesn't work, it remains set to gitlab.com in the HTML. Did I put this in the wrong place?

EDIT: Fixed. The right variable to use in html_context settings is gitlab_host (I inferred this from snippet on https://docs.readthedocs.io/en/stable/guides/vcs.html though it is not documented explicitly)

humitos · 2020-05-25T09:45:23Z

@fcasson please, next time open a new issue instead of commenting in an old PR. Thanks!

I think you will need DONT_OVERWRITE_SPHINX_CONTEXT set in your project. See https://docs.readthedocs.io/en/stable/guides/feature-flags.html#feature-flags --let me know (via email support) if that makes sense in your case.

Daniel Widerin and others added 25 commits June 16, 2016 07:11

Add GitLab repo sync and webhook support

ef9c792

oauth: gitlab: use unicode whenever format is used

cf5c47c

oauth: gitlab: prevent duplicate entries after sync

2356d84

oauth: gitlab: avoid KeyError when owner is missing

7113e94

oauth: gitlab: setup_webhook must return a tuple

5d7d6c1

oauth: gitlab: set a default avatar if none is returned

f9df35e

oauth: gitlab: use SSH url if repo is private

e4adb98

Added tests for the GitLabService

da7471f

Added some documentation for the Gitlab integration

aec6941

Merge pull request #1 from galileo-press/sync-gitlab-repos

9152bb6

Sync gitlab repos

Improved url_pattern of the GitLabService to support private reposito…

299a2b5

…ries

Fixed linting error

8a10848

Simplified exception handling

97a4fb1

Use PRODUCTION_DOMAIN as webhook id

dff4d7d

Make imports python3 compatible

4fbe6bb

Use the GitLab project ID, not the PRODUCTION_DOMAIN

50285fe

Merge pull request #2 from galileo-press/sync-gitlab-repos

e288bf3

Support private repositories and cleanup

Update urls as mentioned in comments

6c8d1c3

Update webhook docs as mentioned in comments

0cb1ed5

Style changes

c33b691

New interface for paginated results

be5ccd0

Improve gitlab integration's code

7e38e70

Fix sync orgs and webhook create

90e4ba1

Fix webhook URL

1e8a8a8

humitos commented Nov 28, 2017

View reviewed changes

RichardLitt added the PR: work in progress Pull request is not ready for full review label Nov 28, 2017

humitos commented Nov 28, 2017

View reviewed changes

humitos added 2 commits November 28, 2017 10:47

Update webhook workflow for GitLab

2fa1c51

Merge branch 'master' of github.com:rtfd/readthedocs.org into humitos…

39c9899

…/gitlab/integration

humitos added 2 commits November 28, 2017 16:47

Fix isort settings

4e6f4a6

Fix lint

0ce1702

humitos added PR: ready for review and removed PR: work in progress Pull request is not ready for full review labels Nov 29, 2017

humitos requested a review from ericholscher November 29, 2017 15:01

ericholscher approved these changes Nov 29, 2017

View reviewed changes

agjohnson assigned humitos Dec 1, 2017

agjohnson added PR: work in progress Pull request is not ready for full review and removed PR: ready for review labels Dec 1, 2017

This was referenced Dec 4, 2017

Integrate private GitLab installation #3352

Closed

Use default avatars for GitLab/GitHub/Bitbucket integrations (users/organizations) #3353

Closed

humitos added 4 commits December 4, 2017 12:50

Styles / Linting

1e668e1

Context for GitLab on templates

4eec51e

Merge branch 'master' of github.com:rtfd/readthedocs.org into humitos…

1e5200e

…/gitlab/integration

More style

12eae4b

agjohnson reviewed Dec 5, 2017

View reviewed changes

Test for get_gitlab_url

3d276dd

agjohnson approved these changes Dec 7, 2017

View reviewed changes

agjohnson merged commit ba71aa1 into master Dec 7, 2017

agjohnson deleted the humitos/gitlab/integration branch December 7, 2017 17:57

humitos mentioned this pull request Dec 27, 2017

Automatically add "Edit on GitLab" links to documentation pages for repositories hosted on GitLab.com #2910

Closed

GitLab Integration #3327

GitLab Integration #3327

Conversation

humitos commented Nov 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agjohnson Nov 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Nov 29, 2017

humitos commented Nov 29, 2017

ericholscher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agjohnson commented Nov 29, 2017 • edited Loading

humitos commented Nov 29, 2017

agjohnson commented Nov 29, 2017

humitos commented Nov 29, 2017

agjohnson commented Dec 1, 2017

agjohnson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fcasson commented May 21, 2020 • edited Loading

humitos commented May 25, 2020

humitos commented Nov 28, 2017 •

edited

Loading

agjohnson Nov 28, 2017 •

edited

Loading

agjohnson commented Nov 29, 2017 •

edited

Loading

fcasson commented May 21, 2020 •

edited

Loading