Fix Authentication issues introduced in 0.14.3, and other minor fixes #256

Chaffelson · 2021-01-19T16:47:35Z

nipyapi-0.16.1-py2.py3-none-any.whl.zip

Revert 0.14.3 changes to Authentication handling which introduced basicAuth support but resulted in some NiFi connections appearing incorrectly as Anonymous
Added simpler basicAuth control to force it via a config switch without changing tokenAuth and other Authorization header behavior during normal usage
nipyapi.config.global_force_basic_auth is now available for use for this purpose
Moved all Security controls in config.py to a common area at the foot of the file
Removed auth_type from security.service_login as it is now redundant
Added controls to handle certificate checking behavior which has become more strict in recently versions of Python3, ssl_verify and check_hostname are now handled
security.set_service_auth_token now has an explicit flag for ssl host checking as well
Fix oversight where improved model serialisation logic was not correctly applied to Registry
Removed unusused parameter refresh from parameters.update_parameter_context
Reduced unecessary complexity in utils.dump with no change in functionality
Updated client gen mustache templates to reflect refactored security and api client code
Minor linting and docstring and codestyle improvements

…icAuth support but resulted in some NiFi connections appearing incorrectly as Anonymous Added simpler basicAuth control to force it via a config switch without changing tokenAuth and other Authorization header behavior during normal usage nipyapi.config.global_force_basic_auth is now available for use for this purpose Moved all Security controls in config.py to a common area at the foot of the file Removed auth_type from security.service_login as it is now redundant Added controls to handle certificate checking behavior which has become more strict in recently versions of Python3, ssl_verify and check_hostname are now handled security.set_service_auth_token now has an explicit flag for ssl host checking as well Fix oversight where improved model serialisation logic was not correctly applied to Registry Removed unusused parameter refresh from parameters.update_parameter_context Reduced unecessary complexity in utils.dump with no change in functionality Updated client gen mustache templates to reflect refactored security and api client code Minor linting and docstring and codestyle improvements

coveralls · 2021-01-19T17:40:59Z

Coverage increased (+0.4%) to 69.11% when pulling 5464363 on fixAuthc into b045251 on main.

iMajna · 2021-01-21T13:56:20Z

Small update regarding PR.

While authentication towards both Nifi and NiFi Registry works now:

Registry Authentication
nipyapi.security.get_service_access_status(service='registry')
OUTPUT:

{'anonymous': False,
 'identity': 'user-1,
 'login_supported': True,
 'resource_permissions': {'any_top_level_resource': {'can_delete': True,
                                                     'can_read': True,
                                                     'can_write': True},
                          'buckets': {'can_delete': True,
                                      'can_read': True,
                                      'can_write': True},
                          'policies': {'can_delete': True,
                                       'can_read': True,
                                       'can_write': True},
                          'proxy': {'can_delete': True,
                                    'can_read': True,
                                    'can_write': True},
                          'tenants': {'can_delete': True,
                                      'can_read': True,
                                      'can_write': True}}}

NiFi Authentication
nipyapi.security.get_service_access_status(service='nifi')
OUTPUT:

{'access_status': {'identity': 'user-1',
                   'message': 'You are already logged in.',
                   'status': 'ACTIVE'}}

Listing Buckets
I can even list buckets:
nipyapi.versioning.list_registry_buckets()
OUTPUT:

4:{'allow_bundle_redeploy': False,
 'allow_public_read': False,
 'created_timestamp': 1607535940892,
 'description': None,
 'identifier': '62878614-fc26-48da-9ca1-9c9d3c3e1255',
 'link': {'href': 'buckets/62878614-fc26-48da-9ca1-9c9d3c3e1255',
          'params': {'rel': 'self'}},
 'name': 'test1',
 'permissions': {'can_delete': True, 'can_read': True, 'can_write': True},
 'revision': None}

Importing Flow version

The issue happens when I try to make import of flow from one registry to another:

 nipyapi.versioning.import_flow_version(
        bucket_id=uniqueBucket.identifier,
        encoded_flow=devExport,
        flow_id=prodFlow.identifier
    )

OUTPUT (DEBUG log):

send: b'GET /nifi-registry-api/buckets/62878614-fc26-48da-9ca1-9c9d3c3e1255/flows HTTP/1.1\r\nHost: registry.prod.host:18433\r\nAccept-Encoding: identity\r\nAccept: application/json\r\nContent-Type: application/json\r\nUser-Agent: Swagger-Codegen/1.0.0/python\r\nAuthorization: Bearer REDACTED_TOKEN2\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Thu, 21 Jan 2021 13:37:34 GMT
header: Content-Type: application/json
header: X-Content-Type-Options: nosniff
header: X-XSS-Protection: 1; mode=block
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: 0
header: Strict-Transport-Security: max-age=31540000 ; includeSubDomains
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: Content-Length: 458
header: Server: Jetty(9.4.19.v20190610)
send: b'GET /nifi-api/system-diagnostics HTTP/1.1\r\nHost: nifi.prod.node:8443\r\nAccept-Encoding: identity\r\nAccept: application/json\r\nContent-Type: application/json\r\nUser-Agent: Swagger-Codegen/1.0.0/python\r\nAuthorization: Bearer REDACTED_TOKEN\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Thu, 21 Jan 2021 13:37:34 GMT
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: X-XSS-Protection: 1; mode=block
header: X-Content-Type-Options: nosniff
header: Strict-Transport-Security: max-age=31540000
header: Cache-Control: private, no-cache, no-store, no-transform
header: Server: Jetty(9.4.19.v20190610)
header: X-ProxiedEntitiesAccepted: true
header: Vary: Accept-Encoding
header: Vary: Accept-Encoding
header: Content-Type: application/json
header: Content-Length: 1901
send: b'GET /nifi-registry-api/swagger/swagger.json HTTP/1.1\r\nHost: registry.prod.host:18433\r\nAccept-Encoding: identity\r\nUser-Agent: Swagger-Codegen/1.0.0/python\r\nContent-Type: application/json\r\n\r\n'
reply: 'HTTP/1.1 401 Unauthorized\r\n'
header: Date: Thu, 21 Jan 2021 13:37:34 GMT
header: Content-Type: text/plain;charset=iso-8859-1
header: X-Content-Type-Options: nosniff
header: X-XSS-Protection: 1; mode=block
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: 0
header: Strict-Transport-Security: max-age=31540000 ; includeSubDomains
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: Transfer-Encoding: chunked
header: Server: Jetty(9.4.19.v20190610)
send: b'GET /nifi-registry-api/buckets/62878614-fc26-48da-9ca1-9c9d3c3e1255/flows HTTP/1.1\r\nHost: registry.prod.host:18433\r\nAccept-Encoding: identity\r\nAccept: application/json\r\nContent-Type: application/json\r\nUser-Agent: Swagger-Codegen/1.0.0/python\r\nAuthorization: Bearer REDACTED_TOKEN2\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Thu, 21 Jan 2021 13:37:35 GMT
header: Content-Type: application/json
header: X-Content-Type-Options: nosniff
header: X-XSS-Protection: 1; mode=block
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: 0
header: Strict-Transport-Security: max-age=31540000 ; includeSubDomains
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: Content-Length: 458
header: Server: Jetty(9.4.19.v20190610)
send: b'GET /nifi-api/system-diagnostics HTTP/1.1\r\nHost: nifi.prod.node:8443\r\nAccept-Encoding: identity\r\nAccept: application/json\r\nContent-Type: application/json\r\nUser-Agent: Swagger-Codegen/1.0.0/python\r\nAuthorization: Bearer REDACTED_TOKEN\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Thu, 21 Jan 2021 13:37:35 GMT
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: X-XSS-Protection: 1; mode=block
header: X-Content-Type-Options: nosniff
header: Strict-Transport-Security: max-age=31540000
header: Cache-Control: private, no-cache, no-store, no-transform
header: Server: Jetty(9.4.19.v20190610)
header: X-ProxiedEntitiesAccepted: true
header: Vary: Accept-Encoding
header: Vary: Accept-Encoding
header: Content-Type: application/json
header: Content-Length: 1901
send: b'GET /nifi-registry-api/swagger/swagger.json HTTP/1.1\r\nHost: registry.prod.host:18433\r\nAccept-Encoding: identity\r\nUser-Agent: Swagger-Codegen/1.0.0/python\r\nContent-Type: application/json\r\n\r\n'
reply: 'HTTP/1.1 401 Unauthorized\r\n'
header: Date: Thu, 21 Jan 2021 13:37:35 GMT
header: Content-Type: text/plain;charset=iso-8859-1
header: X-Content-Type-Options: nosniff
header: X-XSS-Protection: 1; mode=block
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: 0
header: Strict-Transport-Security: max-age=31540000 ; includeSubDomains
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: Transfer-Encoding: chunked
header: Server: Jetty(9.4.19.v20190610)
Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/opt/virtualenv/python3.6/lib64/python3.6/site-packages/nipyapi/versioning.py", line 685, in import_flow_version
    nipyapi.utils.validate_parameters_versioning_support()
  File "/opt/virtualenv/python3.6/lib64/python3.6/site-packages/nipyapi/utils.py", line 521, in validate_parameters_versioning_support
    '0.6', service='registry', bool_response=True)
  File "/opt/virtualenv/python3.6/lib64/python3.6/site-packages/nipyapi/utils.py", line 540, in enforce_min_ver
    if check_version(min_version, service=service) == 1:
  File "/opt/virtualenv/python3.6/lib64/python3.6/site-packages/nipyapi/utils.py", line 492, in check_version
    '/swagger/swagger.json', 'GET', _preload_content=False
  File "/opt/virtualenv/python3.6/lib64/python3.6/site-packages/nipyapi/registry/api_client.py", line 332, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/opt/virtualenv/python3.6/lib64/python3.6/site-packages/nipyapi/registry/api_client.py", line 153, in __call_api
    _request_timeout=_request_timeout)
  File "/opt/virtualenv/python3.6/lib64/python3.6/site-packages/nipyapi/registry/api_client.py", line 355, in request
    headers=headers)
  File "/opt/virtualenv/python3.6/lib64/python3.6/site-packages/nipyapi/registry/rest.py", line 233, in GET
    query_params=query_params)
  File "/opt/virtualenv/python3.6/lib64/python3.6/site-packages/nipyapi/registry/rest.py", line 224, in request
    raise ApiException(http_resp=r)
nipyapi.registry.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 21 Jan 2021 13:37:35 GMT', 'Content-Type': 'text/plain;charset=iso-8859-1', 'X-Content-Type-Options': 'nosniff', 'X-XSS-Protection': '1; mode=block', 'Cache-Control': 'no-cache, no-store, max-age=0, must-revalidate', 'Pragma': 'no-cache', 'Expires': '0', 'Strict-Transport-Security': 'max-age=31540000 ; includeSubDomains', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Security-Policy': "frame-ancestors 'self'", 'Transfer-Encoding': 'chunked', 'Server': 'Jetty(9.4.19.v20190610)'})
HTTP response body: b"Access is denied due to: Unknown user with identity 'anonymous'. Contact the system administrator.\n"

NOTE!
All commands are executed in the same session one after another in debug console

Chaffelson · 2021-01-21T14:05:16Z

Looks like I may have impacted the registry swagger lookup by the auth changes as well, will need to test it.

send: b'GET /nifi-registry-api/swagger/swagger.json HTTP/1.1\r\nHost: registry.prod.host:18433\r\nAccept-Encoding: identity\r\nUser-Agent: Swagger-Codegen/1.0.0/python\r\nContent-Type: application/json\r\n\r\n'
reply: 'HTTP/1.1 401 Unauthorized\r\n'

… environment

Chaffelson · 2021-01-21T15:01:21Z

nipyapi-0.16.1-py2.py3-none-any.whl.zip
Updated distributable with fixed Registry Swagger version check

…sfy py2 and py3 compatibility

… checking

Chaffelson added this to the 0.17.0 milestone Jan 19, 2021

Chaffelson self-assigned this Jan 21, 2021

Add fix for retrieving swagger.json from Registry in an authenticated…

f46a85a

… environment

Set pyUp to ignore Watchdog as it must stay between versions to stati…

26a2a68

…sfy py2 and py3 compatibility

Chaffelson mentioned this pull request Jan 21, 2021

NiFiRegistry unable to login #247

Closed

Chaffelson added 2 commits January 21, 2021 17:26

if Client is not instantiated, optimistically instantiate for version…

3c3b97d

… checking

mistake in instantiating the api_client

5464363

Chaffelson merged commit 0687dbc into main Jan 22, 2021

Chaffelson deleted the fixAuthc branch January 22, 2021 09:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Authentication issues introduced in 0.14.3, and other minor fixes #256

Fix Authentication issues introduced in 0.14.3, and other minor fixes #256

Chaffelson commented Jan 19, 2021

coveralls commented Jan 19, 2021 •

edited

Loading

iMajna commented Jan 21, 2021 •

edited

Loading

Chaffelson commented Jan 21, 2021

Chaffelson commented Jan 21, 2021

Fix Authentication issues introduced in 0.14.3, and other minor fixes #256

Fix Authentication issues introduced in 0.14.3, and other minor fixes #256

Conversation

Chaffelson commented Jan 19, 2021

coveralls commented Jan 19, 2021 • edited Loading

iMajna commented Jan 21, 2021 • edited Loading

Chaffelson commented Jan 21, 2021

Chaffelson commented Jan 21, 2021

coveralls commented Jan 19, 2021 •

edited

Loading

iMajna commented Jan 21, 2021 •

edited

Loading