How to send bulk transactions (i.e. automatic nonce management)

[ruzpuz]

Hello,
I have a scenario where I have to send n transactions of my custom token to n addresses. My plan was to create a batch transaction and send it all together. Is that even possible?

I am successfully sending the token (one transaction at the time) like this

  const valueToSend = ethers.utils.parseUnits(value, numberOfDecimals); 
  const result = await this.contract.transfer(address, valueToSend);

[ricmoo]

This is something I plan to add to the cookbook, but this issue may help get you on the right track:
#319

I will likely include something similar in either ethers or in an ethers extension package, that wraps a Signer and manages the nonce as well as rebroadcasting.

[tlxnithin]

I have been trying something similar and I am getting replacement fee too low error. So you were saying #319 is the only work around??
Also I have observed that web3 doesn't seem to be having this issue, why ethers.js is having this? Just wanted to know, if someone could explain.

[ricmoo]

This should have more to do with your backend than it does with the library. What is your Web3 setup vs. Ethers setup? In both cases are you using a Geth/Parity node?

[tlxnithin]

In both cases, it is Geth node. We have REST API client wrapped around the ethers.js. If I send 10 concurrent request calls, only one succeeds and other nine gets the replacement fee too low error. How do you suggest me to overcome this?
We are using wallet to send signed transaction.

[ricmoo]

In that case you will need to increment the nonce between each transaction, since each transaction when it queries the node will get the same transaction count.

Your Web3 version probably used the eth_sendTransaction API call?

You can do the same thing with ethers too, if you would rather the node manage your nonce, by using provider.getSigner() on the JsonRpcProvider, but then you are exposing private keys via your Geth instance.

[tlxnithin]

I will give that a try. So the difference is in eth_sendTransaction() and eth_sendRawTransaction() ?? Thanks for the quick response.

[ricmoo]

The eth_sendTransaction (which is what JsonRpcSinger.prototype.sendTransaction does) requires the node have a private key, which will be used to sign the transaction. It will look up any missing details (e.g. gas price, nonce), sign it and then eth_sendRawTransaction that signed transaction.

The ethers_sendRawTransaction (which is what provider.sendTransaction does) expects a fully prepared and signed transaction, and it just gets dumped tot eh network. The Wallet takes care of all those looking up missing details (e.g. gas price, nonce). But if the wallet is used too closely back-to-back, it is unaware of the other transactions that are sent to the network.

In the future, I will be adding a helper class to facilitate managing nonces, and rebroadcasting dropped transactions, since a node will only preserve a certain number of future, unconfirmed transactions for a given address...

Does it make sense? The system is very eventually consistent. :)

[tlxnithin]

That's pretty clear explanation. :)
But I seem to have the same issue even if I use provider.getSigner() or wallet to get the contract instance.
I am getting the contract instance and calling one of the contract method as below. Seems to be missing something. Below is what I am trying.

var provider = new ethers.providers.JsonRpcProvider(url);

async function createExample(params){
let signer = provider.getSigner();
let contract = getContractInstance(contractAddress, abi, signer);

let res = await contract.create(params);
// parse result 
}

function getContractInstance(address, abi, signer){
let contract = new ethers.Contract(address, abi, signer);
return contract;
}

Is there any better way of doing this?

[ricmoo]

Actually, now that I think about it, I think that I pre-populate all the values before sending them to eth_sendTransaction in JsonRpcSigner. I probably shouldn't do that, so the node can manage that.

I will fix that and post a new version soon.

Can you post the ABI you are using for the above example? This may help as well: https://docs.ethers.io/ethers.js/html/api-contract.html

[tlxnithin]

https://docs.ethers.io/ethers.js/html/api-contract.html#connecting-to-existing-contracts is exactly what we are doing.

Will be posting the abi soon here.

[tlxnithin]

Here is the ABI :

[
{
"constant": true,
"inputs": [
{
"name": "",
"type": "address"
}
],
"name": "permissions",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "_interfaceId",
"type": "bytes4"
}
],
"name": "supportsInterface",
"outputs": [
{
"name": "",
"type": "bool"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "name",
"outputs": [
{
"name": "",
"type": "string"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "_tokenId",
"type": "uint256"
}
],
"name": "getApproved",
"outputs": [
{
"name": "",
"type": "address"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "ERROR_TOKI_NOT_LOCKED",
"outputs": [
{
"name": "",
"type": "string"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "totalSupply",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "InterfaceId_ERC165",
"outputs": [
{
"name": "",
"type": "bytes4"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "_owner",
"type": "address"
},
{
"name": "_index",
"type": "uint256"
}
],
"name": "tokenOfOwnerByIndex",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "PERMISSION_TO_MODIFY",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "ERROR_ACCESS_RESTRICTED",
"outputs": [
{
"name": "",
"type": "string"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "PERMISSION_TO_MODIFY_PAY_STATUS",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "_from",
"type": "address"
},
{
"name": "_to",
"type": "address"
},
{
"name": "_tokenId",
"type": "uint256"
}
],
"name": "safeTransferFrom",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "_tokenId",
"type": "uint256"
},
{
"name": "_indexFrom",
"type": "uint256"
},
{
"name": "_indexTo",
"type": "uint256"
}
],
"name": "ftHoldersBalances",
"outputs": [
{
"name": "",
"type": "address[]"
},
{
"name": "",
"type": "uint256[]"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "_index",
"type": "uint256"
}
],
"name": "tokenByIndex",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "_tokenId",
"type": "uint256"
}
],
"name": "ownerOf",
"outputs": [
{
"name": "",
"type": "address"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "_tokenId",
"type": "uint256"
}
],
"name": "ftAddress",
"outputs": [
{
"name": "_ftAddress",
"type": "address"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "",
"type": "uint256"
}
],
"name": "nftValues",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "_tokenId",
"type": "uint256"
}
],
"name": "ftHoldersCount",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "_owner",
"type": "address"
}
],
"name": "balanceOf",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": false,
"inputs": [],
"name": "renounceOwnership",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "PERMISSION_SET_PERMISSION",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "_tokenId",
"type": "uint256"
}
],
"name": "nftValue",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "_tokenId",
"type": "uint256"
},
{
"name": "_holder",
"type": "address"
}
],
"name": "ftHolderBalance",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "ERROR_ZERO_ADDRESS",
"outputs": [
{
"name": "",
"type": "string"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "PERMISSION_TO_MODIFY_STATUS",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "owner",
"outputs": [
{
"name": "",
"type": "address"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "symbol",
"outputs": [
{
"name": "",
"type": "string"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "ERROR_MINTING_NFT",
"outputs": [
{
"name": "",
"type": "string"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "",
"type": "uint256"
}
],
"name": "_allTokens",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "PERMISSION_TO_MODIFY_VERIFIER",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "_from",
"type": "address"
},
{
"name": "_to",
"type": "address"
},
{
"name": "_tokenId",
"type": "uint256"
},
{
"name": "_data",
"type": "bytes"
}
],
"name": "safeTransferFrom",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "PERMISSION_TO_CREATE",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "_tokenId",
"type": "uint256"
}
],
"name": "tokenURI",
"outputs": [
{
"name": "",
"type": "string"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "",
"type": "uint256"
}
],
"name": "ftAddresses",
"outputs": [
{
"name": "",
"type": "address"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "PERMISSION_TO_DEACTIVATE",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "_address",
"type": "address"
},
{
"name": "_permission",
"type": "uint256"
}
],
"name": "setPermission",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "ERROR_DISALLOWED",
"outputs": [
{
"name": "",
"type": "string"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "_owner",
"type": "address"
},
{
"name": "_operator",
"type": "address"
}
],
"name": "isApprovedForAll",
"outputs": [
{
"name": "",
"type": "bool"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "_newOwner",
"type": "address"
}
],
"name": "transferOwnership",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"inputs": [
{
"name": "_name",
"type": "string"
},
{
"name": "_symbol",
"type": "string"
}
],
"payable": false,
"stateMutability": "nonpayable",
"type": "constructor"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"name": "previousOwner",
"type": "address"
}
],
"name": "OwnershipRenounced",
"type": "event"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"name": "previousOwner",
"type": "address"
},
{
"indexed": true,
"name": "newOwner",
"type": "address"
}
],
"name": "OwnershipTransferred",
"type": "event"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"name": "from",
"type": "address"
},
{
"indexed": true,
"name": "to",
"type": "address"
},
{
"indexed": true,
"name": "tokenId",
"type": "uint256"
}
],
"name": "Transfer",
"type": "event"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"name": "owner",
"type": "address"
},
{
"indexed": true,
"name": "approved",
"type": "address"
},
{
"indexed": true,
"name": "tokenId",
"type": "uint256"
}
],
"name": "Approval",
"type": "event"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"name": "owner",
"type": "address"
},
{
"indexed": true,
"name": "operator",
"type": "address"
},
{
"indexed": false,
"name": "approved",
"type": "bool"
}
],
"name": "ApprovalForAll",
"type": "event"
},
{
"constant": false,
"inputs": [
{
"name": "",
"type": "address"
},
{
"name": "",
"type": "address"
},
{
"name": "",
"type": "uint256"
}
],
"name": "transferFrom",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "",
"type": "address"
},
{
"name": "",
"type": "uint256"
}
],
"name": "approve",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "",
"type": "address"
},
{
"name": "",
"type": "bool"
}
],
"name": "setApprovalForAll",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "",
"type": "string"
},
{
"name": "",
"type": "string"
},
{
"name": "",
"type": "uint256"
},
{
"name": "",
"type": "address"
},
{
"name": "",
"type": "uint256"
}
],
"name": "mint",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "",
"type": "address"
},
{
"name": "",
"type": "uint256"
}
],
"name": "burn",
"outputs": [
{
"name": "",
"type": "bool"
}
],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "_tokiId",
"type": "string"
},
{
"name": "_assetReference",
"type": "string"
},
{
"name": "_owner",
"type": "address"
},
{
"name": "_beneficiary",
"type": "address"
},
{
"name": "_ccy",
"type": "string"
},
{
"name": "_fungibleTokenSupply",
"type": "uint256"
},
{
"name": "_payDate",
"type": "uint256"
},
{
"name": "_marketId",
"type": "uint256"
}
],
"name": "createToki",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "_tokiId",
"type": "uint256"
}
],
"name": "tokiById",
"outputs": [
{
"name": "",
"type": "uint256[5]"
},
{
"name": "",
"type": "address[2]"
},
{
"name": "",
"type": "string"
},
{
"name": "",
"type": "string"
},
{
"name": "",
"type": "string"
},
{
"name": "",
"type": "bool"
},
{
"name": "",
"type": "bool"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "_tokiId",
"type": "uint256"
},
{
"name": "_newStatus",
"type": "uint256"
}
],
"name": "updateTokiStatus",
"outputs": [
{
"name": "",
"type": "bool"
}
],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "_tokiId",
"type": "uint256"
},
{
"name": "_newPayStatus",
"type": "uint256"
}
],
"name": "updateTokiPayStatus",
"outputs": [
{
"name": "",
"type": "bool"
}
],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "_tokiId",
"type": "uint256"
}
],
"name": "deactivateToki",
"outputs": [
{
"name": "",
"type": "bool"
}
],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "_tokiId",
"type": "uint256"
}
],
"name": "lockTokiToMarketplace",
"outputs": [
{
"name": "",
"type": "bool"
}
],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "_tokiId",
"type": "uint256"
}
],
"name": "tokiLockStatus",
"outputs": [
{
"name": "",
"type": "bool"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
}
]

Please let me know if you find anything to resolve that.

[ricmoo]

I don't see a "create" function in your ABI. In your above example did you mean createToki?

[tlxnithin]

Yeah, Sorry about that. I was not sure about the sharing the ABI and the code we had. So the code was genralised. It was createToki().

[ricmoo]

(The default provider should work, since it connects to both INFURA and Etherscan, and it either doesn't respond, falls back onto the other; I'm wondering if there are Firewall rules blocking it, since when you first fire up an EC2 or something in VPC it has some pretty restrictive IP table...)

(sorry, wrong issue)

[dhl]

Actually, now that I think about it, I think that I pre-populate all the values before sending them to eth_sendTransaction in JsonRpcSigner. I probably shouldn't do that, so the node can manage that.

@ricmoo this is great, thank you.

We ran into the same bulk transactions problem as we are running tests in parallel.

The way we work around this issue is by wrapping an instance of JsonRpcSigner and delegate everything to the wrapped signer instance except for transaction, where I basically duplicated the code in JsonRpcSigner and delete the auto-populated nonce before sending the transaction with eth_sendTransaction.

Are you looking for help with this pre-population task?

[onetom]

In other words, this check could probably just go away:

ethers.js/src.ts/utils/transaction.ts

Lines 228 to 230 in fa68385

	if (tx.nonce == null) {
	tx.nonce = provider.getTransactionCount(from);
	}

We tried our parallel test suite after removing these lines and the nonce related errors went away.

[ricmoo]

Oh, I’ll just remove the call to populateTransaction in JsonRpcSigner. We still need the nonce to be populated in other Signers.

Sorry, I was busy today, but should be able to get it in tomorrow. :)

[tlxnithin]

@dhl Can you share the work around code that you have used if possible?
Thanks in advance. :)

[tlxnithin]

I removed 'nonce' from the array in hexlifyTransaction function in jsonrpcprovider and It worked for me. @ricmoo May be instead of removing call to populateTransaction, removing only nonce will make sense?

[ricmoo]

@tlxnithin I think anything not explicit should be removed, since, for example gasPrice and gasLimit may be better implemented in the eth_sendTransaction than the individual calls are using the eth_gasPrice and eth_estimateGas; for example in the upcoming wallet separation, calls to specific contracts may get different gas prices.

[ricmoo]

This has been added in 4.0.16. Please give it a try and let me know if this solves you issues.

Thanks! :)

[dhl]

@tlxnithin sorry mate, saw your message too late. Looks like @ricmoo just made the custom signer unnecessary.

Thank you so much @ricmoo! Much obliged!

[ricmoo]

Let me know if there are any issues, and feel free to re-open.

Thanks! :)

[tlxnithin]

Thanks a lot guys @ricmoo @dhl

[tlxnithin]

@tlxnithin I think anything not explicit should be removed, since, for example gasPrice and gasLimit may be better implemented in the eth_sendTransaction than the individual calls are using the eth_gasPrice and eth_estimateGas; for example in the upcoming wallet separation, calls to specific contracts may get different gas prices.

👍

[sssubik]

Hey @tlxnithin @ricmoo How can I send a bulk of transactions without having to worry about the nonce? I havent used ether.js but web3.js has problems with it. Could you point me to a documentation or codes explaining me this as its hard to find it in internet.