Length error using `map` with datasets

lhoestq
Thu, 25 May 2023 15:11:12 +0000

Hi ! Yes you can remove the other columns with:

laion_ds_batched = laion_ds.map(collate_fn, batched=True, batch_size=8, remove_columns=laion_ds.column_names)