Dismiss cookie permission, GDPR, and subscription popups before archiving

[pirate]

Might be able to graft some code from: https://www.i-dont-care-about-cookies.eu for hiding modals and overlays.

Also:
https://greasyfork.org/de/scripts/34815-anti-adblock-killer-reek-

[pirate]

This is blocked by:

• Switch all dependencies to pure python and release ArchiveBox pip package #177 (we need pyppeteer to run user scripts to dismiss popups)
• Add ability to run JS scripts during archiving with Playwright/Puppeteer #51 support for user scripted puppeteer flows

[pirate]

Useful scripts for evading bot detection in the future: https://github.com/niespodd/browser-fingerprinting#technical-insights-into-bypassing-bot-detection

[datoslabs]

Hi,

I am curious if there's any interest in the community to incorporate AI/LLM to help solve this issue.

Recently I was working on a simple web scraper / retriever for a LLM project and came across the need to dismiss not only cookie consent but also newsletter subscription pop-ups. After some research on web UI element detection computer vision models and failed POCs, I decided to give LLM a try. I tested in Google AI studio using Gemini 2.0 Flash model; to my surprise, it appears Gemini 2.0 not only is able to detect if a pop-up (or cookie consent banner) exists on the screen, it is also able to correctly identify the XPath to close the pop-up!

The test I choose was artnet.com, it consisted of a newsletter subscription pop-up with a light gray "X" button to dismiss. Due to the light color, most web UI element CV detection models either ignored the button (to close the pop-up) completely or gave it a very low score of 30% confidence level; however, Gemini 2.0 had no problem identifying the button based on the same screen shot. Moreover when I provided it with the corresponding HTML, it was able to correctly identify the XPath which we can use in Playwright/Puppeteer/Selenium to send the click event. Following are the screenshots from one of my tests in Google AI Studio:

In my limited tests, it took 50-70 seconds to "think" (I ran it a couple times across 2 different days and both tests took >>50 seconds to process). I am sure more testing and optimization can be done to improve on this if there's enough interests to incorporate this feature into the project. Also for reference, https://github.com/browser-use/browser-use have incorporated similar technique to close pop-up dialogs; though, it appears that they highlighted each UI controls by adding colored labels and borders before taking a screen shot as an input for the LLM instead relying on LLM to identify the XPath to a specific control.

[GlassedSilver]

Hi,

I am curious if there's any interest in the community to incorporate AI/LLM to help solve this issue.

Recently I was working on a simple web scraper / retriever for a LLM project and came across the need to dismiss not only cookie consent but also newsletter subscription pop-ups. After some research on web UI element detection computer vision models and failed POCs, I decided to give LLM a try. I tested in Google AI studio using Gemini 2.0 Flash model; to my surprise, it appears Gemini 2.0 not only is able to detect if a pop-up (or cookie consent banner) exists on the screen, it is also able to correctly identify the XPath to close the pop-up!

The test I choose was artnet.com, it consisted of a newsletter subscription pop-up with a light gray "X" button to dismiss. Due to the light color, most web UI element CV detection models either ignored the button (to close the pop-up) completely or gave it a very low score of 30% confidence level; however, Gemini 2.0 had no problem identifying the button based on the same screen shot. Moreover when I provided it with the corresponding HTML, it was able to correctly identify the XPath which we can use in Playwright/Puppeteer/Selenium to send the click event. Following are the screenshots from one of my tests in Google AI Studio:

In my limited tests, it took 50-70 seconds to "think" (I ran it a couple times across 2 different days and both tests took >>50 seconds to process). I am sure more testing and optimization can be done to improve on this if there's enough interests to incorporate this feature into the project. Also for reference, https://github.com/browser-use/browser-use have incorporated similar technique to close pop-up dialogs; though, it appears that they highlighted each UI controls by adding colored labels and borders before taking a screen shot as an input for the LLM instead relying on LLM to identify the XPath to a specific control.

This has major privacy implications, maybe optionally this could be fine, but I think a local AI model would suit this purpose better if you want to involve AI at all.

Personally I use Consent-o-Matic in my browsers and it not only auto okays all cookies, it even accepts or denies based on what the site lets you control and what you desire. For example allowing functional cookies for things like embeds but keeping trackers out.

It's truly set and forget, rarely misses and could be a fantastic consideration to source from. It even clicks your choices in a hidden manner.

https://youtu.be/LtgyVYwjlGE

[pirate]

Yes for paid clients I'm already using a mixture of local LLMs and hosted models to detect and solve CAPTCHAs, hide cookie popups, and do QA analysis of captures to determine if there's any obsctructions to seeing the content.

None of it is open source right now because it's the only thing I can charge for to sustain the rest of the project.