Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

{error, incomplete} while decoding nested maps #49

Closed
comtihon opened this issue Aug 11, 2015 · 5 comments
Closed

{error, incomplete} while decoding nested maps #49

comtihon opened this issue Aug 11, 2015 · 5 comments

Comments

@comtihon
Copy link

I have this java code to encode package:

    Map<String, Object> header = new HashMap<>();
    header.put("msg_type", 1);
    header.put("request_id", 15);

    Map<String, Object> msgMap = new HashMap<>();
    msgMap.put("chat_id", "1476392A-56AE-4849-8BB2-EB9E9C973603");
    msgMap.put("from", 43);

    Map<String, String> message = new HashMap<>();
    message.put("text", "sadf");
    message.put("type", "text");

    msgMap.put("message", message);
    msgMap.put("msg_id", "22A52279-EDD2-430E-ABAB-94ACC894BCE8");
    msgMap.put("to", new int[]{42});

    ByteArrayOutputStream out = new ByteArrayOutputStream();
    MessagePacker packer = MessagePack.newDefaultPacker(out);
    packer.packMapHeader(header.size());
    packer.packString("msg_type");
    packer.packInt(1);
    packer.packString("request_id");
    packer.packInt(15);

    packer.packMapHeader(msgMap.size());
    packer.packString("chat_id");
    packer.packString("1476392A-56AE-4849-8BB2-EB9E9C973603");
    packer.packString("from");
    packer.packInt(43);

    packer.packString("message");
    packer.packMapHeader(message.size());
    packer.packString("text");
    packer.packString("sadf");
    packer.packString("type");
    packer.packString("text");

    packer.packString("msg_id");
    packer.packString("22A52279-EDD2-430E-ABAB-94ACC894BCE8");

    packer.packString("to");
    packer.packArrayHeader(new int[]{42}.length);
    packer.packInt(42);

    packer.close();

    byte[] arr = out.toByteArray();
    StringBuilder erlangBin = new StringBuilder("<<");
    for (byte b : arr)
        erlangBin.append((b < 0 ? b + 256 : b)).append(",");
    erlangBin.setLength(erlangBin.length()-1);
    erlangBin.append(">>");
    System.out.println(erlangBin.toString());

As a result - I got binary package:

<<130,168,109,115,103,95,116,121,112,101,1,170,114,101,113,117,101,115,116,95,105,100,15,133,167,99,104,97,116,95,105,100,217,36,49,52,55,54,51,57,50,65,45,53,54,65,69,45,52,56,52,57,45,56,66,66,50,45,69,66,57,69,57,67,57,55,51,54,48,51,164,102,114,111,109,43,167,109,101,115,115,97,103,101,130,164,116,101,120,116,164,115,97,100,102,164,116,121,112,101,164,116,101,120,116,166,109,115,103,95,105,100,217,36,50,50,65,53,50,50,55,57,45,69,68,68,50,45,52,51,48,69,45,65,66,65,66,45,57,52,65,67,67,56,57,52,66,67,69,56,162,116,111,145,42>>

Then on the Erlang side I am trying to decode it:

1> B = <<130,168,109,115,103,95,116,121,112,101,1,170,114,101,113,117,101,115,116,95,105,100,15,133,167,99,104,97,116,95,105,100,217,36,49,52,55,54,51,57,50,65,45,53,54,65,69,45,52,56,52,57,45,56,66,66,50,45,69,66,57,69,57,67,57,55,51,54,48,51,164,102,114,111,109,43,167,109,101,115,115,97,103,101,130,164,116,101,120,116,164,115,97,100,102,164,116,121,112,101,164,116,101,120,116,166,109,115,103,95,105,100,217,36,50,50,65,53,50,50,55,57,45,69,68,68,50,45,52,51,48,69,45,65,66,65,66,45,57,52,65,67,67,56,57,52,66,67,69,56,162,116,111,145,42>>.
2> {Header, Rest} = msgpack:unpack_stream(B, [{format, map}]).
{#{<<"msg_type">> => 1,<<"request_id">> => 15},
 <<133,167,99,104,97,116,95,105,100,217,36,49,52,55,54,51,
   57,50,65,45,53,54,65,69,45,52,56,...>>}
3> Header.
#{<<"msg_type">> => 1,<<"request_id">> => 15}
4> msgpack:unpack_stream(Rest, [{format, map}]).
{error,incomplete}

Erlang seems to work fine with ordinary map requests, but can't decode nested.

@mururu
Copy link
Member

mururu commented Aug 11, 2015

By default msgpack-erlang use old spec. But msgpack-java uses new spec.
Please use enable_str option to use new spec.

msgpack:unpack_stream(B, [{format, map},{enable_str, true}]).

@mururu mururu closed this as completed Aug 11, 2015
@comtihon
Copy link
Author

Yes, thanks, it helped!
But now all binaries became strings :(

@comtihon
Copy link
Author

The point is, that I get messages through net, decode them and save to database aswell, as get from database and sent to net - so it will be very conveniently for me, if I am able to decode these messages from new spec and all strings were binaries, as without {enable_str, true}. Can it be such an opportunity?

@kuenishi
Copy link
Member

In that case it could be easy to decode binary to construct objects in including strings as binaries, but in encoding, it is hard to tell binary from string. How much could such asymmetric code be tested?

@vincentdephily
Copy link
Contributor

We're hitting this as well, now that other components are sending us new-spec messages. Using {enable_str,true} fixes the {error,incomplete} issue, but

  • switching from binaries to strings has a huge code rewrite cost, and certainly also a performance cost
  • enable_str,true is just working around a bug, the decoder should be able to parse new-spec messages without it.

Please can you suggest a fix that still returns msgpack str as erlang binaries ?

As an aside, it's not clear what the intent of "enable_str" is. Lack of documentation doesn't really help. IMHO we should have these options if we want clear compatibility and flexibility:

  • {format_spec,new|old} (only meaningfull for encoding)
  • {encode_list,str|bin} (throws badarg if format_spec=old and encode_list=str)
  • {decode_str,list|binary}

vincentdephily added a commit to vincentdephily/msgpack-erlang that referenced this issue Feb 2, 2016
vincentdephily added a commit to vincentdephily/msgpack-erlang that referenced this issue Feb 2, 2016
vincentdephily added a commit to vincentdephily/msgpack-erlang that referenced this issue Feb 2, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants