Enhance scraper reliability with improved page load detection

[tharropoulos]

Change Summary

What is this?

This PR addresses the inconsistent page scraping behavior where the number of pages found and processed varies between runs. The issue stems from race conditions during page loading, where content might not be fully rendered before scraping begins. By implementing a more robust page load detection mechanism, we ensure consistent scraping results across multiple runs.

Changes

Code Changes:

1. In custom_downloader_middleware.py:

   • Implement dynamic page load detection using WebDriverWait instead of static delays
   • Add proper error handling for timeout scenarios
   • Improve code organization and imports structure

   Changes in detail:

   • Replace time.sleep(spider.js_wait) with WebDriverWait to actively check document readiness
   • Add fallback to original timeout behavior if dynamic check fails
   • Add proper imports for selenium exceptions and WebDriverWait
   • Reorganize imports for better code organization

Demo

Running the scraper now produces consistent results across multiple runs for the same website. The scraper properly waits for the DOM content to be fully loaded before extracting content.

Context

This change addresses issue #75 where the scraper was producing inconsistent results due to race conditions in page loading. The number of pages found would vary between runs because some dynamic content might not have been fully loaded when the scraping occurred.

Previously, the scraper used a static delay which could be:

• Too short: Missing dynamically loaded content
• Too long: Unnecessarily slowing down the scraping process

The new implementation ensures we wait exactly as long as needed for each page to load completely.

PR Checklist

• I have read and signed the Contributor License Agreement.

[jasonbosco]

This PR is now available in 0.12.0.rc6