distributed-training

We would like to forward a particular 'key' column which is part of the features to appear alongside the predictions - this is to be able to identify to which set of features a particular prediction belongs to. Here is an example of predictions output using the tensorflow.contrib.estimator.multi_class_head:

{"classes": ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"],
 "scores": [0.068196

I have the same hardware envs, same network, but I could not get the result as you, almost half as you. Any best practices and experience? thanks very much! for bytePS with 1 instance and 8 GPU, I have similar testing result.

proposed by @mryab from #126

Virtual batching and LR scheduling are popular techniques with many applications. It would be nice to have an example of how to implement them with hivemind.

As of 0.8.15, ExpertBackend supports two ways of doing so:

• hacky way: by wrapping over optimizer and implementing LR schedule in opt.step
• orthodox way: by subclassing ExpertBackend and implementing ap

torchtext (as of 0.4.0) adopts torch.utils.data.DataLoader, and the older iterator interface is deprecated. Ensure AdaptDL's AdaptiveDataLoader supports this new torchtext interface for data loading, and port the example transformer code to the new interface. Then, adaptdl.data.iterator can be deprecated/removed.