Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crawler doesn't mark invalid URL as failed #286

Closed
tw4l opened this issue Apr 13, 2023 · 1 comment · Fixed by #300
Closed

Crawler doesn't mark invalid URL as failed #286

tw4l opened this issue Apr 13, 2023 · 1 comment · Fixed by #300
Assignees
@tw4l
Labels
bug Something isn't working

Comments

@tw4l
Copy link
Member

tw4l commented Apr 13, 2023

If the crawler is given an invalid URL such as https://doesntexist.webrecorder.net, rather than having a failed URL as one would expect, instead the page loads and runs behaviors, is marked as succesfully done, and has a load state of 4 (BEHAVIORS_DONE) in the resulting pages.jsonl.
@tw4l tw4l added the bug Something isn't working label Apr 13, 2023
@tw4l tw4l self-assigned this Apr 13, 2023
@tw4l tw4l moved this from Triage to Ready for Dev in Webrecorder Projects Apr 13, 2023
@rgaudin
Copy link
Contributor

rgaudin commented Apr 14, 2023

See #207 as well

@tw4l tw4l moved this from Ready for Dev to Dev In Progress in Webrecorder Projects Apr 25, 2023
@tw4l tw4l mentioned this issue Apr 25, 2023
Merged
@tw4l tw4l moved this from Dev In Progress to PR In Review in Webrecorder Projects Apr 25, 2023
@github-project-automation github-project-automation bot moved this from PR In Review to Done! in Webrecorder Projects Apr 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tw4l tw4l

Labels
bug Something isn't working
Projects
Webrecorder Projects
Archived in project
Milestone
No milestone
2 participants
@rgaudin @tw4l