Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create debian templates for the python 3 flask templates. #26

Closed
wants to merge 5 commits into from
Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions template/python3-flask-debian/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
FROM openfaas/of-watchdog:0.7.2 as watchdog
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the latest version?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
FROM openfaas/of-watchdog:0.7.2 as watchdog
FROM openfaas/of-watchdog:0.7.6 as watchdog

FROM python:3-slim-buster
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect we should pin a specific Python 3 version, if not, folks will run into issues with libraries.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Specific major version)


COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
RUN chmod +x /usr/bin/fwatchdog

ARG ADDITIONAL_PACKAGE
# Alternatively use ADD https:// (which will not be cached by Docker builder)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this comment now

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Alternatively use ADD https:// (which will not be cached by Docker builder)


RUN apt-get update && apt-get --assume-yes install musl-dev gcc make ${ADDITIONAL_PACKAGE}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we adding musl-dev? The whole point of using Debian is to avoid musl and to benefit from all the packages built with glibc.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More used to seeing -qy, how does --assume-yes differ?


# Add non root user
RUN addgroup --system app && adduser app --system --ingroup app
RUN chown app /home/app

USER app

ENV PATH=$PATH:/home/app/.local/bin

WORKDIR /home/app/

COPY index.py .
COPY requirements.txt .

USER root
RUN pip install -r requirements.txt

#build function directory and install user specified componenets.
USER app

RUN mkdir -p function
RUN touch ./function/__init__.py
WORKDIR /home/app/function/
COPY function/requirements.txt .
RUN pip install --user -r requirements.txt

WORKDIR /home/app/

#install function code
USER root

COPY function function
RUN chown -R app:app ./

#configure WSGI server and healthcheck
USER app

ENV fprocess="python index.py"

ENV cgi_headers="true"
ENV mode="http"
ENV upstream_url="http://127.0.0.1:5000"

HEALTHCHECK --interval=5s CMD [ -e /tmp/.lock ] || exit 1

CMD ["fwatchdog"]
Empty file.
7 changes: 7 additions & 0 deletions template/python3-flask-debian/function/handler.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
def handle(req):
"""handle a request to the function
Args:
req (str): request body
"""

return req
Empty file.
33 changes: 33 additions & 0 deletions template/python3-flask-debian/index.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Copyright (c) Alex Ellis 2017. All rights reserved.
# Licensed under the MIT license. See LICENSE file in the project root for full license information.

from flask import Flask, request
from function import handler
#from gevent.wsgi import WSGIServer
from gevent.pywsgi import WSGIServer

app = Flask(__name__)

@app.before_request
def fix_transfer_encoding():
"""
Sets the "wsgi.input_terminated" environment flag, thus enabling
Werkzeug to pass chunked requests as streams. The gunicorn server
should set this, but it's not yet been implemented.
"""

transfer_encoding = request.headers.get("Transfer-Encoding", None)
if transfer_encoding == u"chunked":
request.environ["wsgi.input_terminated"] = True

@app.route("/", defaults={"path": ""}, methods=["POST", "GET"])
@app.route("/<path:path>", methods=["POST", "GET"])
def main_route(path):
ret = handler.handle(request.get_data())
return ret

if __name__ == '__main__':
#app.run(host='0.0.0.0', port=5000, debug=False)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#app.run(host='0.0.0.0', port=5000, debug=False)


Copy link
Contributor

@viveksyngh viveksyngh Apr 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are using gevent based WSGIServer, we should monkey patch standard libraries to make it cooperative with gevent.

from gevent import monkey
monkey.patch_all()

http://www.gevent.org/api/gevent.monkey.html

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... well, ok.

But that kind of represents a change to all of the templates, which is out of the scope of this PR>

That's probably better as a new issue, especially because I at least don't know any implications for using a newer version of gevent..

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain what "we should money patch" means? 🤔 I would say this is also out of scope since the task was to port from Alpine to Debian. Happy for you to raise an issue Vivek if you can explain in a bit more detail there?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gevent is built on top of libevent/libev (for asynchronous I/O) and greenlets (lightweight cooperative multi-threading). Greenlets are based on Stackless Python. Greenlets has no implicit scheduling and we can control exactly when our code runs.

In case of I/O block, the code will automatically switch to other I/O or wait for acknowledgment, however the standard libraries implementation does not work that way.

Monkey patching is required to patch the standard library with compatible cooperative counterparts from gevent package.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that, this is out of scope of this PR. I will raise a separate issue.

http_server = WSGIServer(('', 5000), app)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use

Suggested change
http_server = WSGIServer(('', 5000), app)
http_server = WSGIServer(('0.0.0.0', 5000), app)

binding to 0.0.0.0 resolves this error

Forking - python [index.py]
2020/04/04 14:15:13 Started logging stderr from function.
2020/04/04 14:15:13 Started logging stdout from function.
2020/04/04 14:15:13 OperationalMode: http
2020/04/04 14:15:13 Timeouts: read: 10s, write: 10s hard: 10s.
2020/04/04 14:15:13 Listening on port: 8080
2020/04/04 14:15:13 Writing lock-file to: /tmp/.lock
2020/04/04 14:15:13 Metrics listening on port: 8081
2020/04/04 14:15:13 stderr: Traceback (most recent call last):
2020/04/04 14:15:13 stderr:   File "index.py", line 33, in <module>
2020/04/04 14:15:13 stderr:     http_server.serve_forever()
2020/04/04 14:15:13 stderr:   File "/usr/local/lib/python3.7/site-packages/gevent/baseserver.py", line 367, in serve_forever
2020/04/04 14:15:13 stderr:     self.start()
2020/04/04 14:15:13 stderr:   File "/usr/local/lib/python3.7/site-packages/gevent/baseserver.py", line 305, in start
2020/04/04 14:15:13 stderr:     self.init_socket()
2020/04/04 14:15:13 stderr:   File "/usr/local/lib/python3.7/site-packages/gevent/pywsgi.py", line 1490, in init_socket
2020/04/04 14:15:13 stderr:     StreamServer.init_socket(self)
2020/04/04 14:15:13 stderr:   File "/usr/local/lib/python3.7/site-packages/gevent/server.py", line 146, in init_socket
2020/04/04 14:15:13 stderr:     self.socket = self.get_listener(self.address, self.backlog, self.family)
2020/04/04 14:15:13 stderr:   File "/usr/local/lib/python3.7/site-packages/gevent/server.py", line 157, in get_listener
2020/04/04 14:15:13 stderr:     return _tcp_listener(address, backlog=backlog, reuse_addr=cls.reuse_addr, family=family)
2020/04/04 14:15:13 stderr:   File "/usr/local/lib/python3.7/site-packages/gevent/server.py", line 252, in _tcp_listener
2020/04/04 14:15:13 stderr:     sock = GeventSocket(family=family)
2020/04/04 14:15:13 stderr:   File "/usr/local/lib/python3.7/site-packages/gevent/_socket3.py", line 114, in __init__
2020/04/04 14:15:13 stderr:     self._sock = self._gevent_sock_class(family, type, proto, fileno)
2020/04/04 14:15:13 stderr: OSError: [Errno 97] Address family not supported by protocol
2020/04/04 14:15:13 Forked function has terminated: exit status 1

I got this from locustio/locust#556

http_server.serve_forever()
3 changes: 3 additions & 0 deletions template/python3-flask-debian/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
flask
gevent

2 changes: 2 additions & 0 deletions template/python3-flask-debian/template.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
language: python3-flask
fprocess: python index.py
9 changes: 9 additions & 0 deletions template/python3-flask/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
RUN chmod +x /usr/bin/fwatchdog

ARG ADDITIONAL_PACKAGE
# Alternatively use ADD https:// (which will not be cached by Docker builder)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Alternatively use ADD https:// (which will not be cached by Docker builder)


RUN apk --no-cache add musl-dev gcc make ${ADDITIONAL_PACKAGE}

# Add non root user
Expand All @@ -19,8 +21,11 @@ WORKDIR /home/app/

COPY index.py .
COPY requirements.txt .

USER root
RUN pip install -r requirements.txt

#build function directory and install user specified componenets.
USER app

RUN mkdir -p function
Expand All @@ -31,9 +36,13 @@ RUN pip install --user -r requirements.txt

WORKDIR /home/app/

#install function code
USER root

COPY function function
RUN chown -R app:app ./

#configure WSGI server and healthcheck
USER app

ENV fprocess="python index.py"
Expand Down
49 changes: 49 additions & 0 deletions template/python3-http-debian/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
FROM openfaas/of-watchdog:0.7.2 as watchdog
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
FROM openfaas/of-watchdog:0.7.2 as watchdog
FROM openfaas/of-watchdog:0.7.6 as watchdog

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FROM python:3-slim-buster

COPY --from=watchdog /fwatchdog /usr/bin/fwatchdog
RUN chmod +x /usr/bin/fwatchdog

ARG ADDITIONAL_PACKAGE
# Alternatively use ADD https:// (which will not be cached by Docker builder)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Alternatively use ADD https:// (which will not be cached by Docker builder)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs to be removed. I made this comment earlier


RUN apt-get update && apt-get --assume-yes install musl-dev gcc make ${ADDITIONAL_PACKAGE}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure we should install musl-dev 🤔


# Add non root user
RUN addgroup --system app && adduser app --system --ingroup app
RUN chown app /home/app

USER app

ENV PATH=$PATH:/home/app/.local/bin

WORKDIR /home/app/

COPY index.py .
COPY requirements.txt .
USER root
RUN pip install -r requirements.txt
USER app

RUN mkdir -p function
RUN touch ./function/__init__.py
WORKDIR /home/app/function/
COPY function/requirements.txt .
RUN pip install --user -r requirements.txt

WORKDIR /home/app/

USER root
COPY function function
RUN chown -R app:app ./
USER app

# Set up of-watchdog for HTTP mode
ENV fprocess="python index.py"
ENV cgi_headers="true"
ENV mode="http"
ENV upstream_url="http://127.0.0.1:5000"

HEALTHCHECK --interval=5s CMD [ -e /tmp/.lock ] || exit 1

CMD ["fwatchdog"]
5 changes: 5 additions & 0 deletions template/python3-http-debian/function/handler.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
def handle(event, context):
return {
"statusCode": 200,
"body": "Hello from OpenFaaS!"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we have this echo the request body, like in the default flask template handler

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, I'm matching the existing templates. This is what the default http header does, so I don't want to change that.

I don't want to change anything about these templates across the board. I just want to make the debian ones available and functional.

At least under this PR.

}
Empty file.
69 changes: 69 additions & 0 deletions template/python3-http-debian/index.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
#!/usr/bin/env python
from flask import Flask, request, jsonify
from waitress import serve
import os

from function import handler

app = Flask(__name__)

class Event:
def __init__(self):
self.body = request.get_data()
self.headers = request.headers
self.method = request.method
self.query = request.args
self.path = request.path

class Context:
def __init__(self):
self.hostname = os.environ['HOSTNAME']

def format_status_code(resp):
if 'statusCode' in resp:
return resp['statusCode']

return 200

def format_body(resp):
if 'body' not in resp:
return ""
elif type(resp['body']) == dict:
return jsonify(resp['body'])
else:
return str(resp['body'])

def format_headers(resp):
if 'headers' not in resp:
return []
elif type(resp['headers']) == dict:
headers = []
for key in resp['headers'].keys():
header_tuple = (key, resp['headers'][key])
headers.append(header_tuple)
return headers

return resp['headers']

def format_response(resp):
if resp == None:
return ('', 200)

statusCode = format_status_code(resp)
body = format_body(resp)
headers = format_headers(resp)

return (body, statusCode, headers)

@app.route('/', defaults={'path': ''}, methods=['GET', 'PUT', 'POST', 'PATCH', 'DELETE'])
@app.route('/<path:path>', methods=['GET', 'PUT', 'POST', 'PATCH', 'DELETE'])
def call_handler(path):
event = Event()
context = Context()
response_data = handler.handle(event, context)

resp = format_response(response_data)
return resp

if __name__ == '__main__':
serve(app, host='0.0.0.0', port=5000)
2 changes: 2 additions & 0 deletions template/python3-http-debian/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
flask
waitress
14 changes: 14 additions & 0 deletions template/python3-http-debian/template.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
language: python3-http
fprocess: python index.py
build_options:
- name: dev
packages:
- make
- automake
- gcc
- g++
- subversion
- python3-dev
- musl-dev
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove please

- libffi-dev
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's that package for?

- git
7 changes: 6 additions & 1 deletion template/python3-http/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ RUN chmod +x /usr/bin/fwatchdog

ARG ADDITIONAL_PACKAGE
# Alternatively use ADD https:// (which will not be cached by Docker builder)

RUN apk --no-cache add ${ADDITIONAL_PACKAGE}

# Add non root user
Expand All @@ -22,6 +23,8 @@ COPY index.py .
COPY requirements.txt .
USER root
RUN pip install -r requirements.txt

#build function directory and install user specified componenets.
USER app

RUN mkdir -p function
Expand All @@ -32,12 +35,14 @@ RUN pip install --user -r requirements.txt

WORKDIR /home/app/

#install function code
USER root
COPY function function
RUN chown -R app:app ./

#configure WSGI server and healthcheck
USER app

# Set up of-watchdog for HTTP mode
ENV fprocess="python index.py"
ENV cgi_headers="true"
ENV mode="http"
Expand Down