Support Deno runtime

[tigitz]

Which package is the feature request for? If unsure which one to select, leave blank

crawlee

Feature

I would like to use Deno for my crawling projects.

Motivation

All the features and comparisons with node can be found here: https://deno.com/runtime

Ideal solution or implementation, and any additional constraints

Add some tests in CI to make sure package can be run with Deno runtime

Alternative solutions or implementations

No response

Other context

Here are currently discovered list of Deno issues blocking the support:

• "node:http" doesn't implement "Server.unref()" API denoland/deno#19113
• Cannot read properties of undefined (reading 'timeout') at ClientRequest.setTimeout (ext:deno_node/http.ts:578:27) denoland/deno#19215
• 个��.hk throws error at Object.toASCII (ext:deno_node/punycode.ts:5:16) denoland/deno#19214 (can be manually patched to circumvent the issue)
• ReadError: The server aborted pending request (supposedly at IncomingMessageForClient._destroy (ext:deno_node/http.ts:759:18)) denoland/deno#19238

[B4nan]

I see the unref issue you created on deno side is now resolved, was that the only problem?

FWIW we will most probably switch to native ESM in the next major.

[tigitz]

@B4nan I've updated the list of issue I've posted in the Deno project since then.

With a manual patch, it's now possible to run the current simple usage example:

// Add import of CheerioCrawler
import { RequestQueue, CheerioCrawler } from 'crawlee';

const requestQueue = await RequestQueue.open();
await requestQueue.addRequest({ url: 'https://crawlee.dev' });

// Create the crawler and add the queue with our URL
// and a request handler to process the page.
const crawler = new CheerioCrawler({
    requestQueue,
    // The `$` argument is the Cheerio object
    // which contains parsed HTML of the website.
    async requestHandler({ $, request }) {
        // Extract <title> text with Cheerio.
        // See Cheerio documentation for API docs.
        const title = $('title').text();
        console.log(`The title of "${request.url}" is: ${title}.`);
    }
})

// Start the crawler and wait for it to finish
await crawler.run();

However, I've tested for my own large project in the meantime and I've encountered some issues with enqueueLinks not queuing anything and the processing time being around ~10x higher. Which is definitely not the improvements I expected.

I plan to redo some tests and gather as much info as my expertise allows me to before sharing those findings.

[lloydjatkinson]

I'd also like to see Deno support! If Crawlee ships native ESM I imagine that would be beneficial too.

[maheshbansod]

Would love to use crawlee in my deno project! Is this on the project timeline?

[pkoretic]

Deno 2.x seems to work fine with latest npm crawlee package version.

[vikyw89]

Didnt work on deno 2