deduplication

Currently directory contents are uploaded without compression, which makes q blobs larger than they need to be.

Given that directory JSON data is trivially 3x compressible with high throughput using pgzip algorithm, we should enable compression by default and/or make it selectable by policy.

Unfortunately this can't be the default in v1 index format, because compression is done in obj

Is your feature request related to a problem? Please describe.
Currently, MapType are not supported for Spark DataFrames

Describe the solution you'd like
Add support for MapType Spark DataFrame columns

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other co

Right dduper has minimal test script to check basic functionality See ci/gitlab/*.sh . Enhance it add RAID tests.