Saving outcomes if Error while applying map function on dataset

hebkon
Mon, 13 Feb 2023 13:47:17 +0000

I use an API (like huggingface_hub) to let a language model answer questions from my dataset.
Since I want to send every single example to the language model, I wrote a function that does that and then use the map function to map this API call to every example of my dataset.

My issue is: If there is an Error at any point (e.g. the API throws an Error after one hour, because sth happend) I loose all the information. Lets say the map worked for the first 100 examples and then the API throws an Error, I loose the generated data of all the 100 examples that were handled correctly before the error happened.

Is there a way to let the map function return all the items it already finished, when an Error occurs. I do not know how to access this information and always have to start from new…