I just ran into an issue when trying to use to_csv with distributed workers that don't share a file system. I shouldn't have been surprised that writing to a local file system from a distributed worker doesn't work. It shouldn't work. But the error I got was just a File Not Found error. That brought me to:dask/dask#2656 (comment) - which was the answer.

See Table 2 in the original paper found here

The corresponding website can be found here

Clients created with worker_client or get_client don't respect the timeout settings (e.g. distributed.comm.timeouts.connect. The timeout is available to set programmatically, but defaults to 3 rather than falling back to the config file. I think this should be as simple as replacing timeout=3 with timeout=None throughout that code path.

Instructions for how to install pyjanitor via pipenv

Some folks might use pipenv for environment management. The recent update requires a prerelease dependency (black, as menti

Problem description

Our dask update graphs are not properly optimized.

We ussually use dask.dataframe optimization and set ave_width=repartition_ratio for kartothek.io.dask.dataframe.update_dataset_from_ddf graphs. We should return an optimized graph from update_dataset_from_ddf to make our users' life simple.

We already have code that does this, whoever picks this up can ping me

In trying to write tests for #189, I'm finding very difficult to add columns to existing tests, as in some cases like the all_types table, the table is defined in a separate file than the tests and multiple tests try to write to the same table.

Additionally, our test suite doesn't prove that the data that are uploaded are the same as the data downloaded for all types.

We should consider m

The assert_array_shape call is used a lot in REGENIE, and serves both as a runtime check and as documentation for the reader. We should sprinkle it liberally through other functions.