From f0b4ccc1e345b234a9e37aa8af651d74bd3ffbae Mon Sep 17 00:00:00 2001 From: James Date: Sat, 5 Mar 2022 09:16:48 +1030 Subject: [PATCH] Update python_guide.md * Updated references to 'pythonFile', 'pythonFiles', and 'pythonModule' to 'pyFiles' etc as this is the current CRD API spec * Added reference to the Python Dockerfile provided * General clean up of the documentation --- docs/python_guide.md | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/docs/python_guide.md b/docs/python_guide.md index ea66eec2..576ff2a5 100644 --- a/docs/python_guide.md +++ b/docs/python_guide.md @@ -1,18 +1,23 @@ # Running Python jobs using pyflink API with the Flink Operator -This guide provides how to start a job with a python application using pyflink that is unnecessary to deploy Apache Beam. +This guide demonstrates how to start a job with a python application using pyFlink without deploying Apache Beam ## Prerequisites -Apache Flink is not officially providing any dockerfile including python, you should deploy a dockerfile to pull it into k8s pod. +Apache Flink does not provide any official docker images for pyFlink, you will need to build and host your own image. A sample docker file is provided +in `images/flink/python.Dockerfile` -1. Build a Dockerfile: please follow [DockerSetup#enableing-python](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/#enabling-python) in the flink docs +Alternatively: + +1. Create your own Dockerfile: please follow [DockerSetup#enableing-python](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/#enabling-python) in the flink docs 2. Deploy the Dockerfile to any docker registry ## Starting a job with a python file -You can start a job with a python file by specifying the `.spec.job.pythonFile` property and changing the `.spec.image.name` to yours. -The `.spec.job.pythonFile` is transformed to `python` as an argument in the flink command. +You can start a job with a python file by specifying the `.spec.job.pyFile` property. The `.spec.job.pyFile` is transformed to `python` +as an argument in the flink command. + +Make sure you update `.spec.image.name` to point to your pyFlink Docker Image and registry. ```yaml apiVersion: flinkoperator.k8s.io/v1beta1 @@ -22,16 +27,18 @@ metadata: spec: ... image: - name: your_dockerfile + name: ... job: - pythonFile: "examples/python/table/word_count.py" + pyFile: "examples/python/table/word_count.py" ``` ## Starting a job with one or more python files -If you wrote the application with multiple python files, speicify `.spec.job.pythonModule` and `.spec.job.pythonFiles`. +If you wrote the application with multiple python files, specify `.spec.job.pyModule` and `.spec.job.pyFiles`. These properties are transformed to `pyModule` and `pyFiles` as arguments in the flink command, respectively. +Refer to the [pyFlink CLI Docs](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/cli/#submitting-pyflink-jobs) for further +information. ```yaml apiVersion: flinkoperator.k8s.io/v1beta1 @@ -41,9 +48,9 @@ metadata: spec: ... image: - name: your_dockerfile + name: ... job: - pythonModule: "word_count" - pythonFiles: "examples/python/table" + pyModule: "word_count" + pyFiles: "examples/python/table" ```