Data preprocessing steps for pretraining BERT from scratch

marshmellow77
Sun, 30 Jan 2022 07:38:23 +0000

I believe these two tutorials should be helpful:

Main NLP tasks - Hugging Face Course. This one trains a Causal LM from scratch, whereas you seem more interested in training a Masked LM (e.g. BERT) from scratch. That being said, many of the same principles apply.

How to train a new language model from scratch using Transformers and Tokenizers.

Hope those are helpful!

Cheers
Heiko

Read full topic