huggingface/transformers#12276 introduced a new --log_level feature, which now allows users to set their desired log level via CLI or TrainingArguments.

run_translation.py was used as a "model" for other examples.

Now we need to replicate this to all other Trainer-based examples under examples/pytorch/, the 3 changes are

1. importing datasets
2. using `training

In gensim/models/fasttext.py:

    model = FastText(
        vector_size=m.dim,
        vector_size=m.dim,
        window=m.ws,
        window=m.ws,
        epochs=m.epoch,
        epochs=m.epoch,
        negative=m.neg,
        negative=m.neg,
        # FIXME: these next 2 lines read in unsupported FB FT modes (loss=3 softmax or loss=4 onevsall,
        # or model=3 supervi

Is your feature request related to a problem? Please describe.
I typically used compressed datasets (e.g. gzipped) to save disk space. This works fine with AllenNLP during training because I can write my dataset reader to load the compressed data. However, the predict command opens the file and reads lines for the Predictor. This fails when it tries to load data from my compressed files.

Hi
Wikiann dataset needs to have "spans" columns, which is necessary to be able to use this dataset, but this column is missing from huggingface datasets, could you please have a look? thank you @lhoestq

Description
We have to continuously push for quality, thus we always need to improve our testing and test coverage. Any improvement of testing is a great contribution to jina.

If you have no idea how to get started on this, here there are some ideas that could be good to have.

• Increase coverage
  Checking the reports on `https://codecov.io/gh/jina-ai/jin

Hi I would like to propose a better implementation for 'test_indices':

We can remove the unneeded np.array casting:

Cleaner/New:
test_indices = list(set(range(len(texts))) - set(train_indices))

Old:
test_indices = np.array(list(set(range(len(texts))) - set(train_indices)))