Payment metadata

[hackaugusto]

Problem Definition

The current API use identifiers to correlate a payment with some service or product. For this to work the nodes (initiator and target) need to use an out-of-band communication channel to agree on the identifier, otherwise the target won't know what the payment is for, this results in a hard to use api.

Solution

Allow the initiator to provide metadata for a given payment, so that applications don't need to use an out-of-band communication and rely on identifiers.

Tasklist

Accept metada data in the transfer API.
Save the metadata as part of the state change that starts the transfer.
Extend the protocol messages to send the metada to the target node.
Extend the events API to expose this metadata.

[palango]

I've got a proof of concept working, now some questions:

• What type should the metadata have and what size do we want to allocate for it? I think bytes should be Ok but I'm unsure what size to allocate for it (and I don't see a way to make it variable sized either)
• Do we want to add a metadata field to the RefundTransfer as well, so that the receiver can send a message backwards as well?

[hackaugusto]

What type should the metadata have and what size do we want to allocate for it? I think bytes should be Ok but I'm unsure what size to allocate for it (and I don't see a way to make it variable sized either)

For the UDP messages we can use a prefix encoding, adding the data length as a prefix to the data itself. I'm not sure how that will play out though, because the code is allocating the buffer before the data is encoded.

For the REST API it may be nicer to allow for arbitrary json, that raises a compatiblity problem though, if a different client uses the format directly. One solution is to add a prefix with the mime encoding of the data.

Do we want to add a metadata field to the RefundTransfer as well, so that the receiver can send a message backwards as well?

The mediators don't realy need to know about this metadata, I would say it's event better if we don't expose this data to unecessary nodes.

We can send it directly from the initiator to the target with the SecretReveal message (well, a new type of message would be even better).

[palango]

For the UDP messages we can use a prefix encoding, adding the data length as a prefix to the data itself. I'm not sure how that will play out though, because the code is allocating the buffer before the data is encoded.

Yeah, something like that should work. Adding a mime-type on top of that should be easy.

The mediators don't realy need to know about this metadata, I would say it's event better if we don't expose this data to unecessary nodes.

That speaks against sending it with the MediatedTransfer, right? Can we encrypt it with the public key of the receiver?

We can send it directly from the initiator to the target with the SecretReveal message (well, a new type of message would be even better).

But then we're back to having a second channel for communication and out-of-band communication.

[hackaugusto]

That speaks against sending it with the MediatedTransfer, right?

Yep

Can we encrypt it with the public key of the receiver?

Not at the moment, we don't have the public key, the simplest way to fix that for the moment is to add the public key into the endpoint registry. @konradkonrad may have other suggestions.

But then we're back to having a second channel for communication and out-of-band communication.

The SecretReveal message is part of the raiden protocol, so I wouldn't consider it out-of-band.

[palango]

Ok, so we basically have two options here:

1. Make public keys for all nodes available, so we can encrypt the metadata and the intermediate nodes cannot access it.
2. Add a new message type for direct communication between nodes, in this case it would be negotiation of an identifier for the transfer.

[hackaugusto]

2. Add a new message type for direct communication between nodes, in this case it would be negotiation of an identifier for the transfer.

Are you talking about a new message that could be used without the payment protocol? That is possible but I think it complicate things, there are quite a few problems:

• should this new message be sent prior or after the payment?
  • if prior, for how long should the node store this metadata in memory? (We don't have indexed tables to query for this metadata from the database)
  • if after, for how long should the node wait for the metadata?
• to avoid attacks this message must be authenticated, which introduces an additional signature

Instead of adding a new generic communication message, I'm proprosing replacing the message at the step 4 below (but not messages from steps 5 and 6) with a new message that has this metadata:

1. Alices sends a MediatedTransfer to Bob with Charlie as target
2. Bob mediates the transfer, and sends a MediatedTransfer to Charlie
3. Charlie sends a SecretRequest to Alice
4. Alice sends a SecretReveal to Charlie
5. Charlie sends a SecretReveal to Bob
6. Bob sends a SecretReveal to Alice
   7/8. Alice sends an Secret message to Bob and Bob to Charlie

[LefterisJP]

I like the idea of using a new in-protocol message much like the SecretReveal sent from initiator to target.

in this case it would be negotiation of an identifier for the transfer.

@palango Why negotiation? It's going to be some payment metadata and the initiator just includes it for identification/misc purposes in the transfer. Does he need to negotiate with the target in-protocol? I don't see why.

Regarding the type of the data, having variable length would be nice but is it really needed? If the metadata are supposed to be text-identifiers/reason-strings only a fixedsize string field would be enough, no?

[hackaugusto]

If the metadata are supposed to be identifiers only a fixedsize string field would be enough, no?

We already have identifiers for transfers, but the identifier alone doesn't make a nice API to build apps on top. This issue was created to solve the usability of the API.

Think about someone writing an application that uses raiden as a payment method, for this to work with the current API (as of acdf354) the payer and the receiving end must use an out-of-band communication channel (e.g. an additional HTTP request), this additional communication step is used to agree on what the payment is for (what service is being payed? Or, what is the content of the online cart?).

The goal here is to make the application developer life easier, allowing him to add arbitrary metadata in the payment with the details his app needs. I think the best approach to solve this problem is to have the metadata delivery only once, when the payment is completed, without sharing this information with the rest of the network.

[palango]

3. Charlie sends a SecretRequest to Alice

Isn't this the wrong direction? I thought we want to send metadata from initiator to target, not the other way around.

@palango Why negotiation? It's going to be some payment metadata and the initiator just includes it for identification/misc purposes in the transfer. Does he need to negotiate with the target in-protocol? I don't see why.

I understood the problem in the following way: The initiator want to pay for some good or resource, but the target doesn't necessarily know which one. Therefore we either need to create a mapping between payment and payment target (with the problems Augusto listed above) or send some metadata describing the reason for the transfer.

[LefterisJP]

@hackaugusto By identifier I meant a text identifier. (It's unfortunate we already use the same term as a number in Raiden). Exactly what you described.

what service is being payed? Or, what is the content of the online cart?)

Wouldn't a fixed size text be enough for that?

[hackaugusto]

Isn't this the wrong direction? I thought we want to send metadata from initiator to target, not the other way around.

Pardon me, wrong bullet number, I'm talking about the reveal secret message that the initiator sends to the target.

[palango]

Pardon me, wrong bullet number, I'm talking about the reveal secret message that the initiator sends to the target.

That sounds like a better solution than sending the metadata during the whole routing.

Wouldn't a fixed size text be enough for that?

I guess it would be enough to provide 128 bytes, but it would be nicer if they could just serialize whatever they have.

[hackaugusto]

I guess it would be enough to provide 128 bytes, but it would be nicer if they could just serialize whatever they have.

We are only limited by the transport maximum message size, on the current implementation that is the UDP MTU, which can be safely set to 576 bytes, but could be larger than 1500 if the full path has jumbo frames.

I think for now it's best to target a conservative size, and only expand once we have defined the message protocol that we are going to use.

[palango]

The 128 was just an arbitrary number. The point I wanted to make was that a flexible size would be most user-friendly.

[hackaugusto]

The 128 was just an arbitrary number. The point I wanted to make was that a flexible size would be most user-friendly.

Yes, Yes, that is fine.

I agree that a variable length is most user friendly, but on the flip side I don't think we should compromise this early on arbitrary length data, since we don't have the messaging protocol completely figured out. So, my proposal is to stick with this arbitrary default of 128, with that default the messages willl be smaller than 576 bytes, allowing new fields to be added.

[naure]

Another compromise would be a fixed size payment identifier or hash, but also define the canonical way of transferring the actual metadata. One way is a dedicated message type. Another way is to let the receiver provide a URL of his choice where the client should send data.