pretraining

Recent Advances in Vision and Language PreTrained Models (VL-PTMs)

Papers about pretraining and self-supervised learning on Graph Neural Networks (GNN).

End-to-End recipes for pre-training and fine-tuning BERT using Azure Machine Learning Service

X-modaler is a versatile and high-performance codebase for cross-modal analytics.

Official Pytorch Implementation of: "ImageNet-21K Pretraining for the Masses"(NeurIPS, 2021) paper

Paddle Distributed Training Extended. 飞桨分布式训练扩展包

OpenAI GPT2 pre-training and sequence prediction implementation in Tensorflow 2.0

Transformer with Untied Positional Encoding (TUPE). Code of paper "Rethinking Positional Encoding in Language Pre-training". Improve existing models like BERT.

Research code for EMNLP 2020 paper "HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training"

AAAI-20 paper: Cross-Lingual Natural Language Generation via Pre-Training

PyTorch code for "Unifying Vision-and-Language Tasks via Text Generation" (ICML 2021)

An official implementation for " UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation"

Research Code for NeurIPS 2020 Spotlight paper "Large-Scale Adversarial Training for Vision-and-Language Representation Learning": UNITER adversarial training part

Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations. [2021]

Python source code for EMNLP 2020 paper "Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT".

Emergent Communication Pretraining for Few-Shot Machine Translation

Code for generating a single image pretraining dataset

Dynamic Transfer Learning for Low-Resource Neural Machine Translation

Pre-training and fine-tuning GNN model on source code

Understanding "A Lite BERT". An Transformer approach for learning self-supervised Language Models. (wip)

bert to tensorflow

This repository contains the source code for the Semantic Knowledge Extractor Tool (SKET). SKET is an unsupervised hybrid knowledge extraction system that combines a rule-based expert system with pre-trained machine learning models to extract cancer-related information from pathology reports.

It is very useful for preparing data set of machine learning.

Pretraining on 2015, 2019 and IDRIDs with ResNet 101 and 152 and fine tuning with 2019 dataset only

CMLM预训练的非官方实现

An overview of German transformer models related to language modeling in the field of Natural Language Processing.

Script to pre-train hugginface transformers BART with Tensorflow 2

This is a flexible class for training specific layers of deep neural-nets in an online manner. Supports Keras models.