patrickvonplaten/opt_metaseq_6700m

This repo let's you run the following checkpoint using facebookresearch/metaseq.

Do the following:

1. Install PyTorch

pip3 install torch==1.10.1+cu113 torchvision==0.11.2+cu113 torchaudio==0.10.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

2. Install Megatron

git clone https://github.com/patrickvonplaten/Megatron-LM.git
cd Megatron-LM
pip3 install six regex
pip3 install -e .

3. Install fairscale

git clone https://github.com/facebookresearch/fairscale.git
cd fairscale
git checkout prefetch_fsdp_params_simple
pip3 install -e .

4. Install metaseq

git clone https://github.com/patrickvonplaten/metaseq.git
cd metaseq
pip3 install -e .

5. Clone this repo (click top right on "How to clone")

6. Run the following:

cd <path/to/cloned/repo>
bash run.sh