Hello. I am attempting to experiments with using my own retrieval results and assessing the impact on the performance of the model (for the full-wiki setting)
If I understood correctly, the process would be as follows.
Generate my own version of hotpot_dev_fullwiki_v1.json in which the context entry of the JSON has been modified with my own paragraphs. All the other keys should be left identical. I shall refer to this json file as custom_dev_fullwiki.json.
Likewise for the test set, to create custom_test_fullwiki.json
Then, I would run the preprocessing , training and evaluation commands as specified in the README.
To make meaningful comparison of metrics, once I change my retrieval method and generate a 'v2.json' set of files, I should start again from scratch on preprocessing, then train etc.
Is this correct?
Hello. I am attempting to experiments with using my own retrieval results and assessing the impact on the performance of the model (for the full-wiki setting)
If I understood correctly, the process would be as follows.
Generate my own version of
hotpot_dev_fullwiki_v1.jsonin which thecontextentry of the JSON has been modified with my own paragraphs. All the other keys should be left identical. I shall refer to this json file ascustom_dev_fullwiki.json.Likewise for the test set, to create
custom_test_fullwiki.jsonThen, I would run the preprocessing , training and evaluation commands as specified in the README.
To make meaningful comparison of metrics, once I change my retrieval method and generate a 'v2.json' set of files, I should start again from scratch on preprocessing, then train etc.
Is this correct?