Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decoding only part of a stream? #128

Open
mcclure opened this issue Sep 30, 2020 · 1 comment
Open

Decoding only part of a stream? #128

mcclure opened this issue Sep 30, 2020 · 1 comment

Comments

@mcclure
Copy link

@mcclure mcclure commented Sep 30, 2020

Scenario: I have a small app where I store a binary blob. The binary blob has metadata associated with it, so, I think: I will store first some sort of header, and then the binary blob. I decide that the header should be a msgpack item. So I encode() my header, write() the result to a file, and then write() my binary blob. (There are reasons why I do not simply include the binary blob inside the msgpack item.)

When it comes time to read my file back in, I get a ReadableStream for the file, I call decodeAsync() on the ReadableStream, and… I get an error, Extra 512 of 529 byte(s) found at buffer[17]. Which, yes, that is expected, I put it there.

My only options for decoding msgpack seem to be decode/decodeAsync, which error if there is extra data at the end of the stream; and decodeStream, which understands there can be many consecutive data items but assumes they are all msgpack.
I can decode a single msgpack at the start of a stream by doing for await (const idk of decodeStream(fileStream)) { and then doing a break inside the loop, but if i do this I find fileStream is exhausted (0 remaining bytes), so I cannot resume from the stream following that first msgPack item. (And I don't have a way of knowing how many bytes the first msgPack item was, so I can't even start over from the start and skip past it).
Alternately, I can do the for await trick and attempt to read from the stream inside the loop after msgPack has decoded one item, but this won't work either because ReadableStreams are only allowed to have one Reader at a time.

How should I proceed? It seems like "read one item from this stream, but allow me to still use the stream when you are done with it" is not a particularly outlandish use case, but it does not seem to be supported.

@gfx
Copy link
Member

@gfx gfx commented Oct 4, 2020

I think MessagePack is designed to handle fixed-sized data structures and it's not easy to handle your case.

You can define your own protocol on the top of MessagePack, like this:

[data_lne][data_msgpack][stream_buffer]
uint32     msgpack       chunk of bytes

Thus, the data structure does not depend on a particular MessagePack library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.