Question about data preprocessing

[KristenZHANG]

Hi, I have a question about the data preprocessing (data_preprocessor.py) and want to seek some help:
When calling the rm_blank of grounding, why the is_shorten is set as True?
Thanks!

[songfeng]

Hi,

Thanks for the question!

In some rare cases, we found that certain grounding content couldn't be matched with a passage even though we could find one when searching manually. Then, it seemed that shorten the grounding content by a little bit could help with the odd issue. So, it is sort of a not-so-elegant workaround that we applied.

Best,
Song