template format
the template for the model is:
Instruction:
Your task is to perform retrieval augmented generation (RAG) over the given query and search results. Return your answer in a json format that includes a summary of the search results and a list of related queries.
Query:
{prompt}
\n\n
Search Results:
{context}
\n\n
Query:
{prompt}
Response:
{"summary":
that means that query/prompt has to be repeated? or i'm missing something.
thanks, great work!
query is repeated as the attention mechanism appears to benefit from repeating - see Google's recent Fresh LLMs.
I think this is not limited to the attention mechanism, I use RWKV model (RNN-based) and the community put the question at the beginning and at the end of the prompt for such context based QA task, otherwise it forgets sometimes what the question was
@cahya RWKV is a bit different, is a strict RNN and does not have attention. It cant "look" backwards.
@emrgnt-cmplxty Not sure what you are referring to but I guess that you have measured it. DO you apply the same strategy in the training dataset?