Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ensure exclusions apply to pages that redirect #744

Closed
ikreymer opened this issue Jan 27, 2025 · 0 comments · Fixed by #745
Closed

Ensure exclusions apply to pages that redirect #744

ikreymer opened this issue Jan 27, 2025 · 0 comments · Fixed by #745

Comments

@ikreymer
Copy link
Member

Currently, if an in scope page URL redirects to an out-of-scope page, that page will still be loaded and crawled.
This can be confusing, for example:

  • Suppose there's an exclusion for 'x.com' when crawling 'mydomain.example.com'.
  • 'https://mydomain.example.com/redirect-to-x' returns a 307 to 'x.com', since it is in scope, it will be loaded.
  • The user thinks that 'x.com' exclusion is not working, because page was loaded due to redirect.
ikreymer added a commit that referenced this issue Jan 27, 2025
@ikreymer
apply exclusions to redirects:
aac36e2 
- if redirected page is excluded, block loading of page
- mark page as excluded, don't retry, and don't write to page list
- support generic blocking of pages based on initial page response
- fixes #744
@ikreymer ikreymer mentioned this issue Jan 27, 2025
Merged
@ikreymer ikreymer changed the title Ensure exclusions apply to pages that redirect. Ensure exclusions apply to pages that redirect Jan 27, 2025
@github-project-automation github-project-automation bot moved this from Triage to Done! in Webrecorder Projects Jan 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Webrecorder Projects
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Apply exclusions to redirects webrecorder/browsertrix-crawler
1 participant
@ikreymer