My hypothesis is since the space of loss values from different weights is less informative in low-resource domains due to less data. Getting directions/gradients in the loss space might be less clear, similar to deep RL. While random search would be too inefficient. Evolutionary search over the space of parameters would still be guided by the training loss, but would have less directionality, therefore might stumble on a better solution for low data domains.
Possibly a long shot, but not too complex.
My hypothesis is since the space of loss values from different weights is less informative in low-resource domains due to less data. Getting directions/gradients in the loss space might be less clear, similar to deep RL. While random search would be too inefficient. Evolutionary search over the space of parameters would still be guided by the training loss, but would have less directionality, therefore might stumble on a better solution for low data domains.
Possibly a long shot, but not too complex.