Dockerfile and script for running crawl jobs in Google Cloud Run with captcha support #208
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
codders
changed the title
Add additional python script for gcloud run
Updated dependencies
codders
force-pushed
the
feat/google-cloud-run
branch
from
93f0907 to
966ce83
Compare
| @@ -161,15 +161,35 @@ First build the image inside the project's root directory: | |||
| $ docker build -t flathunter . | |||
| ``` | |||
|
|
|||
| **When running a container using the image, a config file needs to be mounted on the container at ```/config.yaml```.** The example below provides the file ```config.yaml``` off the current working directory: | |||
| **When running a container using the image, a config file needs to be mounted on the container at ```/config.yaml``` or configuration has to be supplied using environment variables.** The example below provides the file ```config.yaml``` off the current working directory: | |||
There was a problem hiding this comment.
We should think about separating the README into separate .md-files and only use the main file as index, because it is so large already
There was a problem hiding this comment.
Yeah. That would make sense. If you have a proposal for how you would like to see that split, I would welcome that. At least the basic getting-started instructions should maybe be at the top of the main README, and people who want to know more can dig a little.
alexanderroidl
previously approved these changes
Aug 29, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for Google-Cloud-Run, and adds a docker image that can be launched to trigger a one-time crawl (which can then be setup to run on a schedule).
I have refactored the config handling so that it's possible to pass the relevant config by environment variables.
This needs a bunch of documentation still - opening the PR for discussion