Transformers-based Encoder-Decoder Models - HuggingFace

Follow the full discussion on Reddit.
Transformer-based encoder-decoder models have become indispensable for seq2seq tasks such as summarization and translation. Recently, there has been a lot of research on different pre-training objectives for transformer-based encoder-decoder models, e.g. T5, Bart, Pegasus, ProphetNet, Marge, etc. However, the model architecture has stayed largely the same.