# CUDA OOM while saving the model

**URL:** <https://discuss.huggingface.co/t/cuda-oom-while-saving-the-model/40231>\
**Category:** 🤗Transformers\
**Created:** [May 18, 2023, 6:10am UTC](https://discuss.huggingface.co/t/cuda-oom-while-saving-the-model/40231 "2023-05-18T06:10:51Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![aastha6](https://avatars.discourse-cdn.com/v4/letter/a/7bcc69/32.png) [@aastha6](https://discuss.huggingface.co/u/aastha6)\
**Post date:** [May 18, 2023, 6:10am UTC](https://discuss.huggingface.co/t/cuda-oom-while-saving-the-model/40231/1 "2023-05-18T06:10:51Z")

</div>

I am trying to finetune FLAN-t5-XXL using PEFT’s LORA method.

**Training Details -**

```auto
Dataset size = 6k records,
instance_type = AWS's ml.g5.16xlarge.
batch_size = 2,
gradient_accumulation_steps = 2
learning_rate = 1e-3,
num_train_epochs = 1 # Want to change it to 3 and more but chose 1 for experimenting if training completes or not

```

**Training completes with this output -**  
`{'train_runtime': 1364.2004, 'train_samples_per_second': 0.733, 'train_steps_per_second': 0.183, 'train_loss': 1.278140380859375, 'epoch': 1.0}`

But getting CUDA OOM at the point when trying to save the model by `trainer.save_model` call.

**Details of error -**  
ErrorMessage “OutOfMemoryError: CUDA out of memory. Tried to allocate 40.00 MiB (GPU 0; 22.19  
GiB total capacity; 20.34 GiB already allocated; 32.50 MiB free; 20.96 GiB  
reserved in total by PyTorch) If reserved memory is \>\> allocated memory try  
setting max\_split\_size\_mb to avoid fragmentation. See documentation for Memory  
Management and PYTORCH\_CUDA\_ALLOC\_CONF”

Could anyone help me in sorting this out?

Aastha

---

<div class="post-metadata">

**Author:** ![aastha6](https://avatars.discourse-cdn.com/v4/letter/a/7bcc69/32.png) [@aastha6](https://discuss.huggingface.co/u/aastha6)\
**Post date:** [May 18, 2023, 1:35pm UTC](https://discuss.huggingface.co/t/cuda-oom-while-saving-the-model/40231/2 "2023-05-18T13:35:08Z")

</div>

Hi @sgugger ! Do you have any suggestion on how to solve this error ?

---

<div class="post-metadata">

**Author:** ![aastha6](https://avatars.discourse-cdn.com/v4/letter/a/7bcc69/32.png) [@aastha6](https://discuss.huggingface.co/u/aastha6)\
**Post date:** [June 13, 2023, 2:30pm UTC](https://discuss.huggingface.co/t/cuda-oom-while-saving-the-model/40231/3 "2023-06-13T14:30:25Z")

</div>

Found a solution - [CUDA OOM error while saving the model · Issue #16 · philschmid/deep-learning-pytorch-huggingface · GitHub](https://github.com/philschmid/deep-learning-pytorch-huggingface/issues/16)