Chore/python3 #646

paulineribeyre · 2019-06-18T21:15:30Z

updated dockerfile to use the python 3 base image
did NOT update shibboleth dockerfile to work with python 3
ran 2to3 for python 2 to 3 syntax updates
encoding fixes: lots of encoding issues due to all python 3 strings now being unicode; may have missed some…
Exception.message does not exist in python 3, so the custom Gen3 exceptions (CirrusNotFound, GoogleAPIError, etc) that inherit from Exception don't have a message attribute anymore
removed enum34 dependency because it is not compatible with the standard library's enum on python >3.4
removed conditional imports for python 2/3 compatibility since Fence now only works on python 3
fixed test logic test_update_user_service_account_success

TODO:
pin versions after PRs are merged:

Breaking Changes

Python 3 instead of python 2

Dependency updates

Remove unused flask-postgres-session dependency
Remove enum34 dependency
Use python 3 versions of cirrus (1.0.0), storage-client, cdisutilstest (1.0.0), userdatamodel (2.0.0), cdispyutils (1.0.0) and authutils (4.0.0)

PlanXCyborg · 2019-06-18T21:35:22Z

The style in this PR agrees with black. ✔️

This formatting comment was generated automatically by a script in uc-cdis/wool.

coveralls · 2019-06-18T21:36:06Z

Pull Request Test Coverage Report for Build 7256

69 of 103 (66.99%) changed or added relevant lines in 31 files are covered.
5 unchanged lines in 3 files lost coverage.
Overall coverage remained the same at ?%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
fence/error_handler.py	4	5	80.0%
fence/jwt/keys.py	7	8	87.5%
fence/rbac/client.py	1	2	50.0%
fence/resources/storage/cdis_jwt.py	0	2	0.0%
fence/utils.py	5	8	62.5%
fence/scripting/google_monitor.py	1	5	20.0%
fence/scripting/fence_create.py	0	7	0.0%
fence/resources/aws/boto_manager.py	0	15	0.0%

Files with Coverage Reduction	New Missed Lines	%
fence/resources/google/validity.py	1	88.1%
fence/resources/aws/boto_manager.py	2	20.69%
fence/jwt/keys.py	2	86.96%

Totals
Change from base Build 7250:	0.0%
Covered Lines:
Relevant Lines:	0

💛 - Coveralls

Avantol13 · 2019-07-05T16:56:16Z

Dockerfile


-COPY . /fence
+# number of uwsgi worker processes
+ENV UWSGI_CHEAPER 2


I think this is already set in the base image https://github.com/uc-cdis/cloud-automation/blob/master/Docker/python-nginx/python3.6-alpine3.7/Dockerfile#L168-L169

fence/jwt/keys.py

Avantol13 · 2019-07-08T19:59:48Z

fence/jwt/keys.py

@@ -181,8 +181,13 @@ def public_key_to_jwk(self):
        Return:
            dict: JWK representation of the public key
        """
-        n, e = _rsa_public_numbers(self.public_key)
+        # n, e = _rsa_public_numbers(self.public_key)


I think we can fully remove this

fence/resources/google/validity.py

fence/resources/user/__init__.py

fence/sync/utils.py

tests/google/test_access_utils.py

Avantol13 · 2019-07-08T20:26:45Z

tests/test_import_fence.py

@@ -22,15 +22,15 @@ def reload_modules(module_name):
    # actually exist, even if it does. To do this, patch-delete the attribute
    # and reload all the fence modules.
    fence_submodules = [
-        module for module in sys.modules.keys() if module.startswith(module_name)
+        module for module in list(sys.modules.keys()) if module.startswith(module_name)
    ]
    for module in fence_submodules:
        if sys.modules[module]:
            # SQLAlchemy gets upset when a table is loaded twice, so ignore
            # that.
            try:
                # NOTE: in python3 this should become ``importlib.reload``


Suggested change

# NOTE: in python3 this should become ``importlib.reload``

Avantol13 · 2019-07-08T20:29:43Z

tests/test_logout.py

-import urlparse
-import urllib
+import urllib.request, urllib.parse, urllib.error
+from fence.resources.storage.cdis_jwt import create_session_token


is this needed? it doesn't look like it was there before

it's duplicated :o looks like the imports have been messed up during a rebase to master, thanks will fix

Avantol13 · 2019-07-09T14:06:31Z

README.md

@@ -101,6 +101,7 @@ At the moment, supported IDPs include:
  - InCommon
  - eduGAIN

+Note: the Shibboleth dockerfile image is at https://quay.io/repository/cdis/fence-shib and is NOT compatible with python 3/the latest fence.


is there a specific reason for this?

i didn't update the shibboleth dockerfile to work with python 3 code for now, I discussed with Rudy and he said master fence probably won't be deployed to the commons that's using this dockerfile

paulineribeyre added 13 commits June 11, 2019 11:20

chore(python3): 2to3

4653cea

chore(python3): fix some things

edc944c

chore(python3): fix more things

0e3099f

chore(python3): this test never should have passed...

9aaf024

chore(python3): remove py2 compatibility imports

2aa361c

chore(python3): fix failing oidc tests + other stuff

46fce29

chore(python3): fix some more tests

13ad2ca

chore(python3): Exception.message does not exist in py3

a1cfeac

fix deprecation warning

525ce4c

chore(python3): python 3 dockerfile

79d1da7

chore(python3): fix deployment (squash)

604d105

chore(python3): run integration tests

0099a74

master rebase

83471db

chore(python3): black

8d4be60

paulineribeyre force-pushed the chore/python3 branch from 9c28af5 to 8d4be60 Compare June 19, 2019 16:10

chore(python3): post integration tests fixes

941f912

paulineribeyre force-pushed the chore/python3 branch from cde1d09 to 941f912 Compare June 26, 2019 18:25

paulineribeyre and others added 8 commits June 26, 2019 13:29

master rebase

c3b2e3e

chore(python3): 2to3 + black

545ec0d

use wool check instead of travis

9db625d

codacy fixes

928b382

Merge branch 'master' into chore/python3

11132a9

chore(python3): pin dependencies

0daa0f0

Merge branch 'master' of github.com:uc-cdis/fence into chore/python3

cd1f788

chore(python3): pin storageclient

e73c7fa

Avantol13 requested changes Jul 8, 2019

View reviewed changes

paulineribeyre added 2 commits July 8, 2019 17:24

chore(python3): small fixes

8613c07

master rebase

46af1ba

Avantol13 reviewed Jul 9, 2019

View reviewed changes

chore(python3): dockerrun.bash chmod 644 to 755

71e22c7

Avantol13 approved these changes Jul 9, 2019

View reviewed changes

paulineribeyre added 3 commits July 11, 2019 10:40

trigger jenkins

19f8eb9

Merge branch 'master' into chore/python3

7974071

chore(python3): fix privacy policy tests

addd965

paulineribeyre merged commit 08daa7e into master Jul 12, 2019

paulineribeyre deleted the chore/python3 branch July 12, 2019 15:25

PlanXCyborg mentioned this pull request Oct 21, 2022

Gen3 Monthly Release 2022.10 dcf-interop.kidsfirstdrc.org 1666320713 uc-cdis/cdis-manifest#5281

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Chore/python3 #646

Chore/python3 #646

paulineribeyre commented Jun 18, 2019 •

edited

Loading

PlanXCyborg commented Jun 18, 2019 •

edited

Loading

coveralls commented Jun 18, 2019 •

edited

Loading

Avantol13 Jul 5, 2019

Avantol13 Jul 8, 2019

Avantol13 Jul 8, 2019

Avantol13 Jul 8, 2019

paulineribeyre Jul 8, 2019

Avantol13 Jul 9, 2019

paulineribeyre Jul 9, 2019

Chore/python3 #646

Chore/python3 #646

Conversation

paulineribeyre commented Jun 18, 2019 • edited Loading

Breaking Changes

Dependency updates

PlanXCyborg commented Jun 18, 2019 • edited Loading

coveralls commented Jun 18, 2019 • edited Loading

Pull Request Test Coverage Report for Build 7256

💛 - Coveralls

Avantol13 Jul 5, 2019

Choose a reason for hiding this comment

Avantol13 Jul 8, 2019

Choose a reason for hiding this comment

Avantol13 Jul 8, 2019

Choose a reason for hiding this comment

Avantol13 Jul 8, 2019

Choose a reason for hiding this comment

paulineribeyre Jul 8, 2019

Choose a reason for hiding this comment

Avantol13 Jul 9, 2019

Choose a reason for hiding this comment

paulineribeyre Jul 9, 2019

Choose a reason for hiding this comment

paulineribeyre commented Jun 18, 2019 •

edited

Loading

PlanXCyborg commented Jun 18, 2019 •

edited

Loading

coveralls commented Jun 18, 2019 •

edited

Loading