Apply exclusions to redirects

[ikreymer]

• if redirected page is excluded, block loading of page
• mark page as excluded, don't retry, and don't write to page list
• support generic blocking of pages based on initial page response
• fixes Ensure exclusions apply to pages that redirect #744

[ikreymer]

Additional testing:
docker run -p 9037:9037 -v $PWD/crawls:/crawls -it webrecorder/browsertrix-crawler crawl --url "https://tararecuperavel.org/2014/01/09/ilha-de-plastico-chega-a-praia-do-baleal/img_1188/" --screencastPort 9037 --exclude facebook.com --exclude x.com --collection redir-test

page includes share links to x.com and facebook.com, which should now be excluded

Also have test in ./tests/exclude-redirected.test.js

[tw4l]

The manual testing example seems to work for Facebook but not for X - I'm still seeing it redirect to x.com in the screencast and the logs include lines such as:

{
  "timestamp": "2025-01-28T18:14:10.328Z",
  "logLevel": "info",
  "context": "behaviorScript",
  "message": "Behavior log",
  "details": {
    "state": {
      "tweets": 0,
      "images": 0,
      "videos": 0,
      "threads": 1
    },
    "msg": "done!",
    "page": "https://x.com/intent/post?via=TaraRecupervel&related=wordpressdotcom&text=IMG_1188&url=https%3A%2F%2Ftararecuperavel.org%2F2014%2F01%2F09%2Filha-de-plastico-chega-a-praia-do-baleal%2Fimg_1188%2F&mx=2",
    "workerid": 0
  }
}
{
  "timestamp": "2025-01-28T18:14:10.833Z",
  "logLevel": "info",
  "context": "behavior",
  "message": "Run Script Finished",
  "details": {
    "frameUrl": "https://x.com/intent/post?via=TaraRecupervel&related=wordpressdotcom&text=IMG_1188&url=https%3A%2F%2Ftararecuperavel.org%2F2014%2F01%2F09%2Filha-de-plastico-chega-a-praia-do-baleal%2Fimg_1188%2F&mx=2",
    "page": "https://tararecuperavel.org/2014/01/09/ilha-de-plastico-chega-a-praia-do-baleal/img_1188/?share=twitter&nb=1",
    "workerid": 0
  }
}

(pretty printed for legibility)

[tw4l]

Ah, it didn't work because the redirect was still to https://twitter.com/.... With --exclude twitter.com added to the test command, I can confirm behavior is expected in manual testing. Going to take a closer look at the code now.