Use one flag to cache all third party data in one directory #131
Conversation
|
Also, I should note -- this removes the older flags ( |
| -u --user-agent=AGENT Override user agent. | ||
| -t --timeout=TIMEOUT Override timeout (in seconds). | ||
| -c --cache-third-parties=DIR Cache third party data, and what directory to cache it in. | ||
| -f --ca-file=PATH Specify custom CA bundle (PEM format) |
There was a problem hiding this comment.
Some of the changes here are from syncing the options in cli.py with the options documented in the README, which required making changes in both directions.
| @@ -3,22 +3,20 @@ | |||
| """pshtt ("pushed") is a tool to test domains for HTTPS best practices. | |||
|
|
|||
| Usage: | |||
| pshtt (INPUT ...) [--output OUTFILE] [--sorted] [--json] [--markdown] [--debug] [--timeout TIMEOUT] [--user-agent AGENT] [--preload-cache PRELOAD] [--cache] [--suffix-cache SUFFIX] [--ca-file PATH] | |||
| pshtt (INPUT ...) [--output OUTFILE] [--sorted] [--json] [--markdown] [--debug] [--timeout TIMEOUT] [--user-agent AGENT] [--cache-third-parties DIR] [--ca-file PATH] | |||
There was a problem hiding this comment.
I also synced the options between the Usage: line here and the list below, as they were slightly out of sync.
| @@ -107,9 +105,7 @@ def main(): | |||
| options = { | |||
| 'user_agent': args['--user-agent'], | |||
| 'timeout': args['--timeout'], | |||
| 'preload_cache': args['--preload-cache'], | |||
| 'suffix_cache': args['--suffix-cache'], | |||
| 'cache': args['--cache'], | |||
There was a problem hiding this comment.
The --cache flag has been unused for a long time -- it's a remnant of a time when this library used requests-cache, but that was removed a long time ago.
There was a problem hiding this comment.
For info on that, see requests-cache/requests-cache#70.
| preload_pending = None | ||
|
|
||
| # Used for determining base domain via Mozilla's public suffix list. | ||
| SUFFIX_CACHE = None | ||
| PUBLIC_SUFFIX_CACHE = None | ||
| PUBLIC_SUFFIX_CACHE_DEFAULT = "public-suffix-list.txt" |
There was a problem hiding this comment.
I didn't provide a way to override the individual filenames within the third party cache directory. Though options are always nice, it doesn't seem like it has enough utility to merit the complexity.
| @@ -1015,28 +1022,38 @@ def did_domain_error(domain): | |||
| ) | |||
|
|
|||
|
|
|||
| def fetch_preload_pending(): | |||
| def load_preload_pending(): | |||
There was a problem hiding this comment.
Standardized on load_ as the prefix for these 3 loading/caching functions, since they were split among load_, create_, and fetch_ before, which seemed slightly confusing.
|
Thanks for the review, @jsf9k! Next up is a PR letting third party cache data be passed in via variable, to support environments like Lambda where you can reuse in-memory data across containers. |
|
Fix the dependabot ignore directive for `github/codeql-action`
This removes the individual flags for the preload list and PSL, and makes one new flag that covers the preload list, PSL, and the preload pending list.
In addition to making @lgarron and the hstspreload.org servers a lot happier, this will make it easier for people/tools that make use of
pshttto rationalize their caching strategy, and easier for thepshttmaintainers to add new third party data sources without adding new flags and approaches to caching.Fixes #99.