Introduce endpoints for listing available Thoth s2i base container images #1224
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
sesheta
added
the
size/M
|
/hold requires thoth-station/storages#2201 and new thoth-storages module release |
sesheta
added
the
do-not-merge/hold
fridex
changed the title
fridex
force-pushed
the
thoth-s2i-endpoint
branch
from
d6c3cda to
a4df244
Compare
Pre-Commit Test failed! Click here |
1 similar comment
Pre-Commit Test failed! Click here |
fridex
force-pushed
the
thoth-s2i-endpoint
branch
from
a4df244 to
aa15200
Compare
|
/deploy |
|
Testing in test environment, with #1227 (comment) we should be ready to merge this. Will update the PR once the testing is done. |
Pre-Commit Test failed! Click here |
fridex
force-pushed
the
thoth-s2i-endpoint
branch
from
aa15200 to
faba75b
Compare
2 tasks
|
@pacospace PTAL |
pacospace
reviewed
Feb 1, 2021
| "thoth_s2i_image_name": thoth_s2i_image_name, | ||
| "thoth_s2i_image_version": thoth_s2i_image_version, | ||
| "thoth_s2i": f"{thoth_s2i_image_name}:v{thoth_s2i_image_version}", | ||
| "analysis_id": analyses[-1], # Show only the last, the most recent, one. |
There was a problem hiding this comment.
don't we need .order_by(desc(PackageExtractRun.datetime)) in the query if we want this?
There was a problem hiding this comment.
It will require additional operations on the database side. Simply the last entry inserted will be the one shown on the endpoint. A corner case might be manual insertion of old documents, but that should not happen in a deployment. We can document it in thoth-station/thoth-application#825
There was a problem hiding this comment.
It will require additional operations on the database side. Simply the last entry inserted will be the one shown on the endpoint. A corner case might be manual insertion of old documents, but that should not happen in a deployment. We can document it in thoth-station/thoth-application#825
Yes please let's document it there :)
There was a problem hiding this comment.
Will do. Do we want to have this in meanwhile?
There was a problem hiding this comment.
I think we can go ahead meanwhile, thanks!
There was a problem hiding this comment.
It's present here, but not in the database, I tested new endpoints, but they failed without new schema in the databae :) I'm creating new workflow-helper with new thoth-storages to run schema-update-job from management-api and have new schema in test to test this :)
There was a problem hiding this comment.
There was a problem hiding this comment.
I updated schema in test environment and tested: @fridex
{"name": "sentry_sdk.errors", "levelname": "DEBUG", "module": "tracing", "lineno": 658, "funcname": "_set_initial_sampling_decision", "created": 1612196510.9817915, "asctime": "2021-02-01 16:21:50,981", "msecs": 981.7914962768555, "relative_created": 585848.9122390747, "process": 22, "message": "[Tracing] Discarding <http.server> transaction <generic WSGI request> because it's not included in the random sample (sampling rate = 0.25)"}
{"name": "thoth.user_api.openapi_server", "levelname": "ERROR", "module": "app", "lineno": 1891, "funcname": "log_exception", "created": 1612196511.1965628, "asctime": "2021-02-01 16:21:51,196", "msecs": 196.5627670288086, "relative_created": 586063.6835098267, "process": 22, "message": "Exception on /api/v1/s2i/python [GET]\nTraceback (most recent call last):\n File \"/opt/app-root/lib64/python3.8/site-packages/flask/app.py\", line 2447, in wsgi_app\n response = self.full_dispatch_request()\n File \"/opt/app-root/lib64/python3.8/site-packages/flask/app.py\", line 1952, in full_dispatch_request\n rv = self.handle_user_exception(e)\n File \"/opt/app-root/lib64/python3.8/site-packages/flask_cors/extension.py\", line 165, in wrapped_function\n return cors_after_request(app.make_response(f(*args, **kwargs)))\n File \"/opt/app-root/lib64/python3.8/site-packages/flask/app.py\", line 1821, in handle_user_exception\n reraise(exc_type, exc_value, tb)\n File \"/opt/app-root/lib64/python3.8/site-packages/flask/_compat.py\", line 39, in reraise\n raise value\n File \"/opt/app-root/lib64/python3.8/site-packages/flask/app.py\", line 1950, in full_dispatch_request\n rv = self.dispatch_request()\n File \"/opt/app-root/lib64/python3.8/site-packages/flask/app.py\", line 1936, in dispatch_request\n return self.view_functions[rule.endpoint](**req.view_args)\n File \"/opt/app-root/lib64/python3.8/site-packages/connexion/decorators/decorator.py\", line 48, in wrapper\n response = function(request)\n File \"/opt/app-root/lib64/python3.8/site-packages/connexion/decorators/uri_parsing.py\", line 144, in wrapper\n response = function(request)\n File \"/opt/app-root/lib64/python3.8/site-packages/connexion/decorators/parameter.py\", line 121, in wrapper\n return function(**kwargs)\n File \"/opt/app-root/src/thoth/user_api/api_v1.py\", line 184, in list_s2i_python\n \"analysis_id\": analyses[-1], # Show only the last, the most recent, one.\nIndexError: list index out of range"}
{"name": "sentry_sdk.errors", "levelname": "INFO", "module": "scope", "lineno": 345, "funcname": "_drop", "created": 1612196511.2008724, "asctime": "2021-02-01 16:21:51,200", "msecs": 200.87242126464844, "relative_created": 586067.9931640625, "process": 22, "message": "event processor (<function DedupeIntegration.setup_once.<locals>.processor at 0x7f4c792429d0>) dropped event ({'level': 'error', 'exception': {'values': [{'module': None, 'type': 'IndexError', 'value': 'list index out of range', 'mechanism': {'type': 'logging', 'handled': True}, 'stacktrace': {'frames': [{'filename': 'flask/app.py', 'abs_path': '/opt/app-root/lib64/python3.8/site-packages/flask/app.py', 'function': 'wsgi_app', 'module': 'flask.app', 'lineno': 2447, 'pre_context': [' ctx = self.request_context(environ)', ' error = None', ' try:', ' try:', ' ctx.push()'], 'context_line': ' response = self.full_dispatch_request()', 'post_context': [' except Exception as e:', ' error = e', ' response = self.handle_exception(e)', ' except: # noqa: B001', ' error = sys.exc_info()[1]'], 'vars': {'self': <Flask 'thoth.user_api.openapi_server'>, 'environ': {'wsgi.errors': <gunicorn.http.wsgi.WSGIErrorsWrapper object at 0x7f4c6e74bd30>, 'wsgi.version': (1, 0), 'wsgi.multithread': False, 'wsgi.multiprocess': False, 'wsgi.run_once': False, 'wsgi.file_wrapper': <class 'gunicorn.http.wsgi.FileWrapper'>, 'wsgi.input_terminated': True, 'SERVER_SOFTWARE': 'gunicorn/20.0.4', 'wsgi.input': <gunicorn.http.body.Body object at 0x7f4c6e645790>, 'gunicorn.socket': <socket.socket fd=21, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('172.20.2.38', 8080), raddr=('172.21.8.1', 45520)>, 'REQUEST_METHOD': 'GET', 'QUERY_STRING': '', 'RAW_URI': '/api/v1/s2i/python', 'SERVER_PROTOCOL': 'HTTP/1.1', 'HTTP_ACCEPT': 'application/json', 'HTTP_USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36', 'HTTP_SEC_FETCH_SITE': 'same-origin', 'HTTP_SEC_FETCH_MODE': 'cors', 'HTTP_SEC_FETCH_DEST': 'empty', 'HTTP_REFERER': 'https://test.thoth-station.ninja/api/v1/ui/', 'HTTP_ACCEPT_ENCODING': 'gzip, deflate, br', 'HTTP_ACCEPT_LANGUAGE': 'it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7,es;q=0.6', 'HTTP_HOST': 'test.thoth-station.ninja', 'HTTP_X_FORWARDED_HOST': 'test.thoth-station.ninja', 'HTTP_X_FORWARDED_PORT': '443', 'HTTP_X_FORWARDED_PROTO': 'https', 'HTTP_FORWARDED': 'for=10.36.113.248;host=test.thoth-station.ninja;proto=https', 'HTTP_X_FORWARDED_FOR': '10.36.113.248', 'wsgi.url_scheme': 'http', 'REMOTE_ADDR': '172.21.8.1', 'REMOTE_PORT': '45520', 'SERVER_NAME': '0.0.0.0', 'SERVER_PORT': '8080', 'PATH_INFO': '/api/v1/s2i/python', 'SCRIPT_NAME': '', 'werkzeug.request': <Request 'http://test.thoth-station.ninja/api/v1/s2i/python' [GET]>}, 'start_response': functools.partial(<function _sentry_start_response at 0x7f4c792a59d0>, <bound method Response.start_response of <gunicorn.http.wsgi.Response object at 0x7f4c6e74ba00>>, <Transaction(name='/api/v1.thoth_user_api_api_v1_list_s2i_python', op='http.server', trace_id='505d5e7c2f8f409184f13ad696053ac0', span_id='a23f8dae29db84d6', parent_span_id=None, sampled=False)>), 'ctx': <RequestContext 'http://test.thoth-station.ninja/api/v1/s2i/python' [GET] of thoth.user_api.openapi_server>, 'error': IndexError('list index out of range'), 'e': IndexError('list index out of range')}}, {'filename': 'flask/app.py', 'abs_path': '/opt/app-root/lib64/python3.8/site-packages/flask/app.py', 'function': 'full_dispatch_request', 'module': 'flask.app', 'lineno': 1952, 'pre_context': [' request_started.send(self)', ' rv = self.preprocess_request()', ' if rv is None:', ' rv = self.dispatch_request()', ' except Exception as e:'], 'context_line': ' rv = self.handle_user_exception(e)', 'post_context': [' return self.finalize_request(rv)', '', ' def finalize_request(self, rv, from_error_handler=False):', ' \"\"\"Given the return value from a view function this finalizes', ' the request by converting it into a response and invoking the'], 'vars': {'self': <Flask 'thoth.user_api.openapi_server'>, 'rv': None}}, {'filename': 'flask_cors/extension.py', 'abs_path': '/opt/app-root/lib64/python3.8/site-packages/flask_cors/extension.py', 'function': 'wrapped_function', 'module': 'flask_cors.extension', 'lineno': 165, 'pre_context': [' # Wrap exception handlers with cross_origin', ' # These error handlers will still respect the behavior of the route', \" if options.get('intercept_exceptions', True):\", ' def _after_request_decorator(f):', ' def wrapped_function(*args, **kwargs):'], 'context_line': ' return cors_after_request(app.make_response(f(*args, **kwargs)))', 'post_context': [' return wrapped_function', '', \" if hasattr(app, 'handle_exception'):\", ' app.handle_exception = _after_request_decorator(', ' app.handle_exception)'], 'vars': {'args': (IndexError('list index out of range'),), 'kwargs': {}, 'app': <Flask 'thoth.user_api.openapi_server'>, 'cors_after_request': <function make_after_request_function.<locals>.cors_after_request at 0x7f4c79242d30>, 'f': <bound method Flask.handle_user_exception of <Flask 'thoth.user_api.openapi_server'>>}}, {'filename': 'flask/app.py', 'abs_path': '/opt/app-root/lib64/python3.8/site-packages/flask/app.py', 'function': 'handle_user_exception', 'module': 'flask.app', 'lineno': 1821, 'pre_context': [' return self.handle_http_exception(e)', '', ' handler = self._find_error_handler(e)', '', ' if handler is None:'], 'context_line': ' reraise(exc_type, exc_value, tb)', 'post_context': [' return handler(e)', '', ' def handle_exception(self, e):', ' \"\"\"Handle an exception that did not have an error handler', ' associated with it, or that was raised from an error handler.'], 'vars': {'self': <Flask 'thoth.user_api.openapi_server'>, 'e': IndexError('list index out of range'), 'exc_type': <class 'IndexError'>, 'exc_value': IndexError('list index out of range'), 'tb': <traceback object at 0x7f4c792b0440>, 'handler': None}}, {'filename': 'flask/_compat.py', 'abs_path': '/opt/app-root/lib64/python3.8/site-packages/flask/_compat.py', 'function': 'reraise', 'module': 'flask._compat', 'lineno': 39, 'pre_context': [' import collections.abc as collections_abc', '', ' def reraise(tp, value, tb=None):', ' if value.__traceback__ is not tb:', ' raise value.with_traceback(tb)'], 'context_line': ' raise value', 'post_context': ['', ' implements_to_string = _identity', '', 'else:', ' iterkeys = lambda d: d.iterkeys()'], 'vars': {'tp': <class 'IndexError'>, 'value': IndexError('list index out of range'), 'tb': <traceback object at 0x7f4c792b0440>}}, {'filename': 'flask/app.py', 'abs_path': '/opt/app-root/lib64/python3.8/site-packages/flask/app.py', 'function': 'full_dispatch_request', 'module': 'flask.app', 'lineno': 1950, 'pre_context': [' self.try_trigger_before_first_request_functions()', ' try:', ' request_started.send(self)', ' rv = self.preprocess_request()', ' if rv is None:'], 'context_line': ' rv = self.dispatch_request()', 'post_context': [' except Exception as e:', ' rv = self.handle_user_exception(e)', ' return self.finalize_request(rv)', '', ' def finalize_request(self, rv, from_error_handler=False):'], 'vars': {'self': <Flask 'thoth.user_api.openapi_server'>, 'rv': None}}, {'filename': 'flask/app.py', 'abs_path': '/opt/app-root/lib64/python3.8/site-packages/flask/app.py', 'function': 'dispatch_request', 'module': 'flask.app', 'lineno': 1936, 'pre_context': [' getattr(rule, \"provide_automatic_options\", False)', ' and req.method == \"OPTIONS\"', ' ):', ' return self.make_default_options_response()', ' # otherwise dispatch to the handler for that endpoint'], 'context_line': ' return self.view_functions[rule.endpoint](**req.view_args)', 'post_context': ['', ' def full_dispatch_request(self):', ' \"\"\"Dispatches the request and on top of that performs request', ' pre and postprocessing as well as HTTP exception catching and', ' error handling.'], 'vars': {'self': <Flask 'thoth.user_api.openapi_server'>, 'req': <Request 'http://test.thoth-station.ninja/api/v1/s2i/python' [GET]>, 'rule': <Rule '/api/v1/s2i/python' (GET, OPTIONS, HEAD) -> /api/v1.thoth_user_api_api_v1_list_s2i_python>}}, {'filename': 'connexion/decorators/decorator.py', 'abs_path': '/opt/app-root/lib64/python3.8/site-packages/connexion/decorators/decorator.py', 'function': 'wrapper', 'module': 'connexion.decorators.decorator', 'lineno': 48, 'pre_context': ['', ' else: # pragma: 3 no cover', ' @functools.wraps(function)', ' def wrapper(*args, **kwargs):', ' request = self.api.get_request(*args, **kwargs)'], 'context_line': ' response = function(request)', 'post_context': [' return self.api.get_response(response, self.mimetype, request)', '', ' return wrapper'], 'vars': {'args': (), 'kwargs': {}, 'request': <connexion.lifecycle.ConnexionRequest object at 0x7f4c78ee0130>, 'function': <function list_s2i_python at 0x7f4c791f8040>, 'self': <BaseDecorator>}}, {'filename': 'connexion/decorators/uri_parsing.py', 'abs_path': '/opt/app-root/lib64/python3.8/site-packages/connexion/decorators/uri_parsing.py', 'function': 'wrapper', 'module': 'connexion.decorators.uri_parsing', 'lineno': 144, 'pre_context': [' form = coerce_dict(request.form)', '', ' request.query = self.resolve_query(query)', ' request.path_params = self.resolve_path(path_params)', ' request.form = self.resolve_form(form)'], 'context_line': ' response = function(request)', 'post_context': [' return response', '', ' return wrapper', '', ''], 'vars': {'request': <connexion.lifecycle.ConnexionRequest object at 0x7f4c78ee0130>, 'coerce_dict': <function AbstractURIParser.__call__.<locals>.wrapper.<locals>.coerce_dict at 0x7f4c740891f0>, 'query': {}, 'path_params': {}, 'form': {}, 'function': <function list_s2i_python at 0x7f4c791e7f70>, 'self': <OpenAPIURIParser>}}, {'filename': 'connexion/decorators/parameter.py', 'abs_path': '/opt/app-root/lib64/python3.8/site-packages/connexion/decorators/parameter.py', 'function': 'wrapper', 'module': 'connexion.decorators.parameter', 'lineno': 121, 'pre_context': ['', ' # attempt to provide the request context to the function', ' if pass_context_arg_name and (has_kwargs or pass_context_arg_name in arguments):', ' kwargs[pass_context_arg_name] = request.context', ''], 'context_line': ' return function(**kwargs)', 'post_context': ['', ' return wrapper'], 'vars': {'request': <connexion.lifecycle.ConnexionRequest object at 0x7f4c78ee0130>, 'kwargs': {}, 'request_body': None, 'query': {}, 'arguments': [], 'consumes': ['application/json'], 'function': <function list_s2i_python at 0x7f4c79213a60>, 'has_kwargs': False, 'operation': <connexion.operations.openapi.OpenAPIOperation object at 0x7f4c791ead00>, 'pass_context_arg_name': None, 'pythonic_params': False, 'sanitize': <function parameter_to_arg.<locals>.sanitized at 0x7f4c791e7e50>}}, {'filename': '/opt/app-root/src/thoth/user_api/api_v1.py', 'abs_path': '/opt/app-root/src/thoth/user_api/api_v1.py', 'function': 'list_s2i_python', 'module': 'thoth.user_api.api_v1', 'lineno': 184, 'pre_context': ['', ' entries.append({', ' \"thoth_s2i_image_name\": thoth_s2i_image_name,', ' \"thoth_s2i_image_version\": thoth_s2i_image_version,', ' \"thoth_s2i\": f\"{thoth_s2i_image_name}:v{thoth_s2i_image_version}\",'], 'context_line': ' \"analysis_id\": analyses[-1], # Show only the last, the most recent, one.', 'post_context': [' })', '', ' return {\"s2i\": entries}', '', ''], 'vars': {'GRAPH': <thoth.user_api.openapi_server._GraphDatabaseWrapper object at 0x7f4c79073f10>, 'entries': [], 'thoth_s2i_image_name': None, 'thoth_s2i_image_version': None, 'analyses': []}}]}}]}, 'logger': 'thoth.user_api.openapi_server', 'logentry': {'message': 'Exception on /api/v1/s2i/python [GET]', 'params': ()}, 'extra': {'asctime': '2021-02-01 16:21:51,196'}, 'event_id': '0dacdba2c6444200add341b6bd21937e', 'timestamp': datetime.datetime(2021, 2, 1, 16, 21, 51, 200805), 'breadcrumbs': {'values': [{'message': 'SELECT DISTINCT ON (software_environment.thoth_s2i_image_name, software_environment.thoth_s2i_image_version) software_environment.thoth_s2i_image_name AS software_environment_thoth_s2i_image_name, software_environment.thoth_s2i_image_version AS software_environment_thoth_s2i_image_version \\nFROM software_environment', 'category': 'query', 'data': {}, 'timestamp': datetime.datetime(2021, 2, 1, 16, 21, 51, 137209), 'type': 'default'}, {'message': 'SELECT package_extract_run.analysis_document_id AS package_extract_run_analysis_document_id \\nFROM package_extract_run JOIN software_environment ON software_environment.id = package_extract_run.software_environment_id \\nWHERE software_environment.thoth_s2i_image_name IS NULL AND software_environment.thoth_s2i_image_version IS NULL', 'category': 'query', 'data': {}, 'timestamp': datetime.datetime(2021, 2, 1, 16, 21, 51, 167402), 'type': 'default'}]}, 'transaction': '/api/v1.thoth_user_api_api_v1_list_s2i_python', 'contexts': {'trace': {'trace_id': '505d5e7c2f8f409184f13ad696053ac0', 'span_id': 'a23f8dae29db84d6', 'parent_span_id': None, 'op': 'http.server', 'description': None}, 'runtime': {'name': 'CPython', 'version': '3.8.3', 'build': '3.8.3 (default, Aug 18 2020, 08:56:04) \\n[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]'}}})"}
There was a problem hiding this comment.
I think it is because there are no data yet, maybe a check on analyses[-1], WDYT?
if analyses:
There was a problem hiding this comment.
or also if thoth_s2i_image_name, thoth_s2i_image_version are not null
|
/deploy |
This works once on a PR, sha needs to be explicitly adjusted otherwise. Will merge this and inspect what's going on in test env. |
You can reuse it, it will repush the image on quay with new sha but same tag. Yesterday there was an issue in last step of the pipeline, the image from quay was not propagated to thoth-test-core because imagestreamtag in thoth application was not changed so ArgoCD didn't sync, the second time it worked. I talked with @harshad16 about this yesterday as well :) and using again /deploy worked. |
This was referenced Feb 15, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related Issues and Dependencies
Depends-On: thoth-station/storages#2201
Fixes: #1223
This introduces a breaking change