- install dependency
pip install -r requirements.txt usage: python dl.py [-h] (-u URL | -f FILE) [-d DEPTH]- url can be a h5ai directory url or a txt file which contains multiple urls
- format of txt file:
<url> <optional depth>
<url> ...
...
The crawler will search (including sib dir) to find the downloadable URLs and confirm before starting to download. Features:
- Download any files from the websiite
- Depth of recursion Control
- Url caching
- Download status tracking
-
- If the download is cancelled, it will skip the downloaded files when re-run