Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upDecoding only part of a stream? #128
Comments
|
I think MessagePack is designed to handle fixed-sized data structures and it's not easy to handle your case. You can define your own protocol on the top of MessagePack, like this:
Thus, the data structure does not depend on a particular MessagePack library. |
Scenario: I have a small app where I store a binary blob. The binary blob has metadata associated with it, so, I think: I will store first some sort of header, and then the binary blob. I decide that the header should be a msgpack item. So I encode() my header, write() the result to a file, and then write() my binary blob. (There are reasons why I do not simply include the binary blob inside the msgpack item.)
When it comes time to read my file back in, I get a ReadableStream for the file, I call decodeAsync() on the ReadableStream, and… I get an error,
Extra 512 of 529 byte(s) found at buffer[17]. Which, yes, that is expected, I put it there.My only options for decoding msgpack seem to be decode/decodeAsync, which error if there is extra data at the end of the stream; and
decodeStream, which understands there can be many consecutive data items but assumes they are all msgpack.I can decode a single msgpack at the start of a stream by doing
for await (const idk of decodeStream(fileStream)) {and then doing abreakinside the loop, but if i do this I find fileStream is exhausted (0 remaining bytes), so I cannot resume from the stream following that first msgPack item. (And I don't have a way of knowing how many bytes the first msgPack item was, so I can't even start over from the start and skip past it).Alternately, I can do the
for awaittrick and attempt to read from the stream inside the loop after msgPack has decoded one item, but this won't work either because ReadableStreams are only allowed to have one Reader at a time.How should I proceed? It seems like "read one item from this stream, but allow me to still use the stream when you are done with it" is not a particularly outlandish use case, but it does not seem to be supported.