# Huggingface datasets streaming problem

**URL:** <https://discuss.huggingface.co/t/huggingface-datasets-streaming-problem/19923>\
**Category:** 🤗Datasets\
**Created:** [July 2, 2022, 7:57am UTC](https://discuss.huggingface.co/t/huggingface-datasets-streaming-problem/19923 "2022-07-02T07:57:05Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![psyche](https://avatars.discourse-cdn.com/v4/letter/p/848f3c/32.png) [@psyche](https://discuss.huggingface.co/u/psyche)\
**Post date:** [July 2, 2022, 7:57am UTC](https://discuss.huggingface.co/t/huggingface-datasets-streaming-problem/19923/1 "2022-07-02T07:57:05Z")

</div>

I trained my model with transformers.Trainer and datasets.load\_dataset.  
When I use the private datasets with streaming options, I got an error message “unauthorized” even if I pushed private tokens. It happens only in the training with transformers.Trainer. I guess there are some problems in certification errors for the dataloader in the Trainer. I want to know the reason why it happens yesterday (before the day I couldn’t see any message like this).

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [July 13, 2022, 2:03pm UTC](https://discuss.huggingface.co/t/huggingface-datasets-streaming-problem/19923/2 "2022-07-13T14:03:41Z")

</div>

Hi! What kind of dataset are you trying to stream? We’ve recently fixed some issues with streaming private audio and vision datasets, so if your dataset falls into that category, you should update the lib to the newest version. Also, make sure to explicitly set `use_auth_token=True` or `use_auth_token="<token>"` in `load_dataset` when loading a private dataset.

---

<div class="post-metadata">

**Author:** ![psyche](https://avatars.discourse-cdn.com/v4/letter/p/848f3c/32.png) [@psyche](https://discuss.huggingface.co/u/psyche)\
**Post date:** [July 14, 2022, 6:50am UTC](https://discuss.huggingface.co/t/huggingface-datasets-streaming-problem/19923/3 "2022-07-14T06:50:35Z")

</div>

Hi, Thanks for your concerns. I tried on my personal text data saved with `.parquet` extension(actually, it uploaded by the function `datasets.DatasetDict.push_to_hub` ). My implementation code is below,

```auto
from datasets import load_dataset

dataset = load_dataset(<repo_id>, use_auth_token=<private_token>, streaming=True)
for d in dataset['train']:
    print(d)
    break # this is for checking

```

it cause the error like this,

/usr/local/lib/python3.7/dist-packages/aiohttp/client\_reqrep.py in raise\_for\_status(self)  
1007 status=self.status,  
1008 message=self.reason,  
 → 1009 headers=self.headers,  
1010 )  
1011

`ClientResponseError: 401, message='Unauthorized', url=URL('https://huggingface.co/datasets/.../train-00000-of-00001-168b451062c67c34.parquet')`

At the first time, I guess it is because of the inserted token. But when I load again the dataset with the `streaming=False` option, it works well.

So, I wondered why this occurs…! (datasets version is “2.3.2”)

---

<div class="post-metadata">

**Author:** ![psyche](https://avatars.discourse-cdn.com/v4/letter/p/848f3c/32.png) [@psyche](https://discuss.huggingface.co/u/psyche)\
**Post date:** [July 18, 2022, 8:17am UTC](https://discuss.huggingface.co/t/huggingface-datasets-streaming-problem/19923/4 "2022-07-18T08:17:59Z")

</div>

I fix this error and pulled it to the [fix-auth-error-private-dataset by hkjeon13 · Pull Request #4699 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/pull/4699)

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [July 22, 2022, 11:50am UTC](https://discuss.huggingface.co/t/huggingface-datasets-streaming-problem/19923/5 "2022-07-22T11:50:25Z")

</div>

we’ll do a new release of `datasets` on monday to include the fix 🙂

---

<div class="post-metadata">

**Author:** ![psyche](https://avatars.discourse-cdn.com/v4/letter/p/848f3c/32.png) [@psyche](https://discuss.huggingface.co/u/psyche)\
**Post date:** [July 23, 2022, 4:52am UTC](https://discuss.huggingface.co/t/huggingface-datasets-streaming-problem/19923/6 "2022-07-23T04:52:05Z")

</div>

Thank you! I’m looking for 🙂

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [July 27, 2022, 9:24am UTC](https://discuss.huggingface.co/t/huggingface-datasets-streaming-problem/19923/7 "2022-07-27T09:24:37Z")

</div>

We just did the release, please update `datasets` and let us know if you still have the issue 🙂

```auto
pip install -U datasets

```