Handling decoding errors such as UnidentifiedImageError

keturn
Sat, 02 Dec 2023 00:04:31 +0000

I’m using the datasets library to load gigant/oldbookillustrations. It’s stored in parquet files, with a couple Image fields plus a handful of other short text fields with assorted metadata.

It loads okay, and many records seem to work just fine, but when I try to filter or map over the whole thing, it throws an UnidentifiedImageError from within the decoder:
traceback (click for more details)

cannot identify image file <_io.BytesIO object at 0x7fa50f2822a0>

How can I fix this or work around it?

Is there a way to install some kind of error handler on the decoder?

How can I identify which records are failing to decode, so that I can troubleshoot further or repair them?

Can I tell my operation to skip the failing record and move on?
env versions (click for more details)