gCloud app not sending notifications #202
Comments
|
k. I took a look at this. I've created a fix in #204 that I will merge in a bit |
|
Please try with the latest code on master. Debug should also be working now |
|
I've pulled from master but all this update this was break the app completely. Can't reach it anymore, getting 502 Bad Gateway. the exception is actually quite similar to #199, with Immobilienscout crashing |
|
Are you deploying with Google App Engine? Or Google Cloud Run? |
|
Google App Engine |
|
Okay. So the thing is... Google App Engine just deploys the python libraries, and it's not possible (as far as I've seen) to get Google Chrome installed as part of that process. So you can deploy to Google App Engine, as long as you don't want to use Immoscout or 2captcha/imagetyperz. I'm currently (as in this second) working on getting a deployment working with Google Cloud Run, which will deploy a docker container that includes Google Chrome. This seems to work better - I'll post some updates when I have them. But of course this might turn out more expensive than App Engine |
|
GAE is cool but since it works flawlessly in Docker (after #199), I though it might be better to focus on that approach since the Webinterface isn't that useful at all because and you still need to configure quite a bit in the config.yaml and elsewhere. Deploying the docker container to a VPS works flawlessly afaik. |
|
For me, the webinterface is critical, as it makes it possible for non-developers to use the service. I have a public instance at https://flathunter.codders.io that hundreds of strangers have been using over the years (and there are about 30 people receiving notifications from it right now), and I'm trying to get it updated to also scrape immoscout. The flathunter software already supports persisting data to Firebase, and I'm kinda hoping that will still work in GCR, but right now I'm also stuck just getting the thing to behave well after it launches. My hope is GCR turns out cheaper than a VPS. But if I don't get this working in the next days then I guess I'll head that route. |
Ah ok I see, you limited the search url's to Berlin. I was also thinking about providing this as a service for friends and family but I found the web interface not so useful for my purpose because you could not configure the search url's through it per telegram receiver id so I would need to do use the config.yaml anyway and deploy multiple services.
Maybe the solution to this issue is to mention the limitations of the GAE approach in the readme and work on alternative deployment solutions like Cloud Run or AWS EC2 (which has more hours in the free tier than GCP) Thanks for all the hard work! |
|
Okay. I got this kinda running now on GCR. The nice thing about GCR is that I'm only paying while the site is running - about 2 minutes in every 10 - which I'm hoping means that I'll stay in the free tier in Google Cloud (or at least it'll be very cheap). But what I learned so far is that it doesn't run with less than 1GB of RAM, and I have the problems you already identified about shipping the config.yaml in the docker image (plus some other GCR issues). I'll open a branch shortly with the changes and then we (@abuchmueller, @codders, @alexanderroidl ?) can collaborate on how to merge it in a way that isn't total trash. Bis morgen! |
|
@codders Thank you for the hard work! It's amazing that you got it running on GCR. By the way, what do you mean by "kinda" having it running? I'd be happy to collaborate on that. |
|
Kinda - I mean, it runs okay and crawls immoscout (and the rest) and puts the data in Firebase, and the website loads. But there's some things that are not ideal.
But if you already want to take a look at dotenv support, that would be super helpful. I'm on the road today so I can't push my changes, but it was just a couple of lines in the Dockerfile and the driver_arguments in the config file so far (plus the GCR config). |
|
Okay - I added a Work-in-progress PR #208 so you can see where I'm headed. There's a refactor of the config processing to handle passing config in the environment, and I'm splitting the 'hunt' / crawl activity from the web-serving activity. There's no reason the webserver has to have 1GB of memory and google chrome embedded just to let people setup and configure their Telegram notifications. Let me know what you think! |
|
fyi - that's merged to main now. You can just pull the latest code and read the docs to try that out. |
thanks a ton! I'll try it out on saturday and report back! I haven't had the time lately to dive more into this |
Very Nice! As promised I report back.
Also a bit confused on the timer: What does |
|
Update: I reverted the version on the Google App Engine to an older build (pre #208) and it works now with in conjunction with the Gcloud Run Job, so that the UI is updated once the job runs. |
|
@abuchmueller - thanks for the report!
How is your python coding? Do you want to have a go at getting the distance matrix API code running? |
I love the distance matrix feature, its the only reason I still run docker locally. I'll take a look at your PR and have a go at it! |
|
Closing this ticket as the original issue is resolved. Please open a new ticket for the other items if you still want to make changes there. Thanks! |
On gcloud the bot runs, but sends no notifications.
I have
Logs:
I use the same config locally as well, and it works without issues. Logger is set to Debug but this is all I can get from
gcloud app logs read.The text was updated successfully, but these errors were encountered: