Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added web interface for Flathunter #6

Merged
merged 14 commits into from
May 27, 2020

Conversation

codders
Copy link

@codders codders commented May 24, 2020

Created a simple flask-based web interface that allows users to see time of last run, and to trigger a run.
The interface can be run locally, or deployed to Google Cloud

@codders codders requested review from mordax7 and tschuehly May 24, 2020 09:37
@codecov-commenter
Copy link

codecov-commenter commented May 24, 2020

Codecov Report

Merging #6 into master will increase coverage by 5.72%.
The diff coverage is 82.83%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master       #6      +/-   ##
==========================================
+ Coverage   59.15%   64.87%   +5.72%     
==========================================
  Files           7       11       +4     
  Lines         426      541     +115     
  Branches       50       62      +12     
==========================================
+ Hits          252      351      +99     
- Misses        159      170      +11     
- Partials       15       20       +5     
Impacted Files Coverage Δ
flathunter/hunter.py 43.63% <66.66%> (+5.40%) ⬆️
flathunter/googlecloud_idmaintainer.py 72.72% <72.72%> (ø)
flathunter/web/views.py 80.00% <80.00%> (ø)
flathunter/idmaintainer.py 90.47% <91.42%> (+1.00%) ⬆️
flathunter/config.py 100.00% <100.00%> (ø)
flathunter/crawl_ebaykleinanzeigen.py 61.42% <100.00%> (ø)
flathunter/crawl_wggesucht.py 63.63% <100.00%> (ø)
flathunter/web/__init__.py 100.00% <100.00%> (ø)
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 77f163f...d194a06. Read the comment docs.

@mordax7
Copy link

mordax7 commented May 26, 2020

Reproduce the error:

  • fetch the PR
  • configure the config.yml(see below)
  • run the web-ui like described in the README.md with python main.py

log snippet:

durations_enabled = "google_maps_api" in self.config and self.config["google_maps_api"]["enable"]
TypeError: argument of type 'Config' is not iterable

my config.yml:

# Should the bot endlessly looop through the URLs?
# Between each loop it waits for <sleeping_time> seconds.
loop:
  active: yes
  sleeping_time: 5

# List the URLs containing your filter properties below.
# Currently supported services: www.immobilienscout24.de and
# www.wg-gesucht.de. List the URLs in the following format:
# urls:
# 	- https://www.immobilienscout24.de/Suche/...
# 	- https://www.wg-gesucht.de/...
urls:
#  - https://www.wg-gesucht.de/wohnungen-in-Munchen.90.2.1.0.html?offer_filter=1&city_id=90&sort_column=0&noDeact=1&categories%5B%5D=2&rent_types%5B%5D=2&sMin=45&rMax=1100&rmMin=2&sin=1&exc=2&img_only=1
#  - https://www.ebay-kleinanzeigen.de/s-wohnung-mieten/muenchen/anbieter:privat/anzeige:angebote/preis:800:1000/c203l6411+wohnung_mieten.etage_i:1,+wohnung_mieten.qm_d:45,+wohnung_mieten.zimmer_d:2,
  - https://www.immobilienscout24.de/Suche/de/bayern/muenchen/wohnung-mieten?haspromotion=false&numberofrooms=2.0-&price=800.0-1000.0&livingspace=45.0-&floor=1-&sorting=2&enteredFrom=result_list
#  - https://www.immowelt.de/liste/muenchen/wohnungen/mieten?roomi=2&prima=1000&wflmi=45&eqid=-205%2C-26&sort=createdate%2Bdesc

excluded_titles:
  - "wg"
  - "tausch"
  - "wochenendheimfahrer"
  - "pendler"
  - "zwischenmiete"

# If an expose includes an address, the bot is capable of
# displaying the distance and time to travel (duration) to
# some configured other addresses, for specific kinds of
# travel.
#
# Available kinds of travel ('gm_id') can be found in the
# Google Maps API documentation, but basically there are:
# 	- "bicyle"
#	- "transit" (public transport)
#	- "driving"
#
# The example configuration below includes a place for
# "John", located at the main train station of munich.
# Two kinds of travel (bicycle and transit) are requested,
# each with a different label. Furthermore a place for
# "Jane" is included, located at the given destination and
# with the same kinds of travel.
durations:
  - name: Hauptbahnhof
    destination: Hauptbahnhof, München
    modes:
      - gm_id: transit
        title: "Öff."

# Multiline message (yes, the | is supposed to be there),
# to format the message received from the Telegram bot.
#
# Available placeholders:
# 	- {title}: The title of the expose
#	- {rooms}: Number of rooms
#	- {price}: Price for the flat
# 	- {durations}: Durations calculated by GMaps, see above
#	- {url}: URL to the expose
message: |
  {title}
  Zimmer: {rooms}
  Größe: {size}
  Preis: {price}
  Ort: {address}

  {url}

# Calculating durations requires access to the Google Maps API.
# Below you can configure the URL to access the API, with placeholders.
# The URL should most probably just kept like that.
# To use the Google Maps API, an API key is required. You can obtain one
# without costs from the Google App Console (just google for it).
google_maps_api:
  key: DEDUCT
  url: https://maps.googleapis.com/maps/api/distancematrix/json?origins={origin}&destinations={dest}&mode={mode}&sensor=true&key={key}&arrival_time={arrival}
  enable: False

# Sending messages using Telegram requires a Telegram Bot configured.
# Telegram.org offers a good documentation about how to create a bot.
# Once you read it, will make sense. Still: bot_token should hold the
# access token of your bot and receiver_ids should list the client ids
# of receivers. Note that those receivers are required to already have
# started a conversation with your bot.
#
# telegram:
#   bot_token: 160165XXXXXXX....
#   receiver_ids:
#       - 12345....
#       - 67890....
telegram:
  bot_token: DEDUCT
  receiver_ids:
    - DEDUCT

codders added 12 commits May 26, 2020 20:57
Added Favicon, Index page, basic CSS
Moved web interface to flathunter.web module
Added initial test case
Updated README.md
Refactored classes to make it possible to run the hunter through flask
Added optional Connection argument to IdMaintainer methods to deal with threading
Display information on status page
Make Config object implement __iter__
@codders codders force-pushed the feat/web-interface branch from f6f49cd to a22838a Compare May 26, 2020 19:31
@codders
Copy link
Author

codders commented May 26, 2020

Reproduce the error:

  • fetch the PR
  • configure the config.yml(see below)
  • run the web-ui like described in the README.md with python main.py

The test suite was passing a dictionary instead of a Config object to hunter, which is why it didn't detect the issue. I've added a fix for that, and a fix for the issue now.

Copy link

@mordax7 mordax7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants