cURL-like API for XHR request as a JavaScript library

[warrenseine]

Following up on #3016.

The cURL API is well tested and well known. I'm not sure we could implement it to match all cURL features, but the cURL API definitely covers the XHR browser API, effectively replacing emscripten_wget_*.

Any good soul and/or lazy intern in a company using Emscripten that would be willing to implement that?

[juj]

Adding cURL support sounds ok, but I am not sure if we want to replace or deprecate the current emscripten_wget_... functionality, even though I agree that it is a bit messy, like we discussed in that other issue. I think we could definitely add cURL as a core implemented library, although we don't want to continue autolinking to all core libraries, so it would be preferably activated by a linker flag e.g. -lcurl.js or similar (this scheme was discussed earlier), which would hook into src/library_curl.js where the functionality could be implemented. That way there is a clear step on how to opt in to the library.

[warrenseine]

I'm totally fine with that.

Since there is a code base using emscripten_wget_..., it would be a bad idea to technically replace these functions. I'd suggest to:

1. Implement library_curl.js.
2. Once equivalent, keep both APIs and suggest developers to switch.
3. Once complete, reimplement emscripten_wget_... as a shell around the new API and deprecate it (can we auto-add -lcurl.js?)
4. At some point in the future, maybe a major version, drop the old API.

[stale]

This issue has been automatically marked as stale because there has been no activity in the past 2 years. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.

[jrynkiew]

I would love to see this implemented in the future

[sbc100]

Does the existing emscripten_wget_... API not satisfy your pusposes from the native code side?

If you are you look for an API to use from JS, then I don't see how this has anything to do with emscripten? libcurl is native library but if you just want a JS re-implemenation of libcurl then that doesn't really have any relation to emscripten as far as I can tell. Emscripten is about compiling native code for the web and libcurl (that actual native version probably don't itself compile for the web, but require come kind of re-implemenation on top of web apis (like XHR)).

[warrenseine]

It’s been a while since I opened this issue, but the feature request still makes sense to me. If you use OpenGL in your native code, Emscripten takes care of binding calls to WebGL. The idea is to do the same for libcurl and XHR. It would avoid conditional blocks to use either the native or the Emscripten-specific API. While not strictly necessary, I think this fits with the Emscripten philosophy.

On my end, I’m not using Emscripten these days so I don’t really need this anymore, but it would be a good addition to the project.

[sbc100]

True, I can see that exposing a native libcurl API, implemented in JS might be useful. I'm not sure much of the libcurl API surface is possible to implement without actual TCP/UDP under the hood. Maybe enough?

In any case, I will re-open with "Help Wanted".

[warrenseine]

Yes, for sure there are cURL APIs that can’t be implemented in XHR or fetch, just like there are OpenGL APIs that can’t be implemented in WebGL. It’s still very useful to have automagic bindings for the 90% use cases covered by both platforms.

[jrynkiew]

I'm having trouble implementing emscripten_wget into my code at https://github.com/jrynkiew/WebOS at WebOS/src/cpp/WebOS/WebOS.cpp using static callback functions (I know I'm doing something wrong), but I have these functions wrapped in ifdef(emscripten) for WASM. For Linux or Windows I use a standard cURL call, and it compiles successfully. It would simply be much much easier to implement cross platform compatible code if these call styles were kept the same for native / emscripten usage, even if the functionality was greatly reduced.

Best Regards,
Jeremi

[jrynkiew]

Actually I solved my problem, never occurred to me to use emscripten fetch the same way as cURL :P

I have all my problems sorted, now just some interface errors.
What I did was:

First define the variables needed for cURL and a string buffer for use with emscripten WASM or Windows/Linux binaries (determined at build time)

CURL *curl;
CURLcode res;
std::string readBuffer;

Then the implementation

#if defined(__EMSCRIPTEN__)
    emscripten_fetch_attr_t attr;
    emscripten_fetch_attr_init(&attr);
    strcpy(attr.requestMethod, "GET");
    attr.attributes = EMSCRIPTEN_FETCH_LOAD_TO_MEMORY;
    attr.onsuccess = onLoaded;
    attr.onerror = onError;
    attr.userData = &readBuffer;
    emscripten_fetch(&attr, "http://localhost");
    printf("Wallet Connection in progress....");
    printf(readBuffer.c_str());

#else
    curl = curl_easy_init();
    if(curl) {
        curl_easy_setopt(curl, CURLOPT_URL, "https://jrpc.pl/");
        curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
        curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
        res = curl_easy_perform(curl);
        curl_easy_cleanup(curl);
        printf(readBuffer.c_str());
    }   
#endif

and the callback functions for cURL and emscripten_fetch

static size_t WriteCallback(void *contents, size_t size, size_t nmemb, void *userp)
{
    ((std::string*)userp)->append((char*)contents, size * nmemb);
    return size * nmemb;
}

#if defined(__EMSCRIPTEN__)
    static void onLoaded(emscripten_fetch_t *fetch) 
    {
        printf("Finished downloading %llu bytes from URL %s.\n", fetch->numBytes, fetch->url);
        ((std::string*)(fetch->userData))->append((char*)fetch->data, fetch->totalBytes * fetch->numBytes);
        emscripten_fetch_close(fetch);
    }
    static void onError(emscripten_fetch_t *fetch)
    {
        printf("Connecting to ioPay %s failed, HTTP failure status code: %d.\n", fetch->url, fetch->status);
        ((std::string*)(fetch->userData))->append((char*)fetch->data, fetch->totalBytes * fetch->numBytes);
        emscripten_fetch_close(fetch);
    }
#endif

I don't really know what I'm doing with fetch->totalBytes in the emscripten callback, so I just used it the same way cURL does it, but it seems to work nonetheless.

All the code samples were copied from the curl and emscripten_fetch documentation examples.

I'm not an expert in emscripten, nor C++/C for that matter, but I got it working :)

Having said all that, I still would love to see this implemented as a native library in emscripten :)

[mewalig]

I still would love to see this implemented as a native library in emscripten

Amen

[jjimenezshaw]

Is there any news on this?

[sbc100]

There is no direct emulation of libcurl in emscripten today, and no plans to add it.

If anyone wanted to contribute a libcurl port to ports/contrib that would be most welcome, alternatively it looks easy enough to just ifdef locally as in #3270 (comment)