An easy framework for pretraining lanuguage models

Follow the full discussion on Reddit.
Hi, ML redditors! I and my colleagues( u/seopbo, and more ) made LAnguage framework for Self-Supervised Learning (LASSL). LASSL aims to provide an easy-to-use framework for pretraining language model by only using Huggingface's Transformers and Datasets. Currently, BERT, RoBERTa, GPT2, and ALBERT are provided, and the model will be continuously updated in the future. Also, in order to see the effectiveness of the code, we will continue to add models trained using it to the model hub along with downstream evaluation. We hope it helps whoever want to make their own language models to make it easy and fast.

Comments

There's unfortunately not much to read here yet...

Discover the Best of Machine Learning.

Ever having issues keeping up with everything that's going on in Machine Learning? That's where we help. We're sending out a weekly digest, highlighting the Best of Machine Learning.

Join over 900 Machine Learning Engineers receiving our weekly digest.

Best of Machine LearningBest of Machine Learning

Discover the best guides, books, papers and news in Machine Learning, once per week.

Twitter