# Add data augmentation process during training every epoch

**URL:** <https://discuss.huggingface.co/t/add-data-augmentation-process-during-training-every-epoch/3274>\
**Category:** Beginners\
**Created:** [January 19, 2021, 5:46am UTC](https://discuss.huggingface.co/t/add-data-augmentation-process-during-training-every-epoch/3274 "2021-01-19T05:46:54Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![yusukemori](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/yusukemori/32/23302_2.png) [@yusukemori](https://discuss.huggingface.co/u/yusukemori)\
**Post date:** [January 19, 2021, 5:46am UTC](https://discuss.huggingface.co/t/add-data-augmentation-process-during-training-every-epoch/3274/1 "2021-01-19T05:46:55Z")

</div>

Hello,

I’d like to process my training dataset every epoch.  
I want to add random processing as data augmentation, and I want to do it during training, not preprocessing.

I think I can do it with `Trainer`, `DataCollator`, or ` __getitem__ ` of `datasets.arrow_dataset.Dataset`, but where should I do it?

For the evaluation set and test set, I plan to do a preprocess using `datasets.arrow_dataset.Dataset.map`.

Thank you in advance.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [January 19, 2021, 9:14pm UTC](https://discuss.huggingface.co/t/add-data-augmentation-process-during-training-every-epoch/3274/2 "2021-01-19T21:14:07Z")

</div>

The `DataCollator` can help if you have something randomized in the call that returns the batch. A `getitem` in your Dataset can also help, it all depends on what you are trying to do exactly.

The `Trainer` in itself has nothing implemented for data augmentation, so it won’t help you.

---

<div class="post-metadata">

**Author:** ![yusukemori](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/yusukemori/32/23302_2.png) [@yusukemori](https://discuss.huggingface.co/u/yusukemori)\
**Post date:** [January 20, 2021, 3:52am UTC](https://discuss.huggingface.co/t/add-data-augmentation-process-during-training-every-epoch/3274/3 "2021-01-20T03:52:24Z")

</div>

Hi @sgugger

Thank you for the answer!  
I now understand for my purpose I should treat a `DataCollator` or `getitem` in a Dataset, depending on what I am trying to do, and the `Trainer` is not an appropriate place in which data augmentation is done.

Referring to your answer in my another question [How to use Seq2SeqTrainer (Seq2SeqDataCollator) in v4.2.1 - #5 by sgugger](https://discuss.huggingface.co/t/how-to-use-seq2seqtrainer-seq2seqdatacollator-in-v4-2-1/3243/5),  
I’ll consider the best way to implement to get something randomized in my case.

Thank you so much!