Pre-slice the input datasets spatially in the concurrent Dataset for efficiency

[dfulu]

Pull Request

Description

The PVNetConcurrentDatasetclass loads the entire spatial crop before slicing out windows for each location. This is fine when the input dataset roughly matches the spatial extent of the spread of location. However, I've recently gained a use-case where the input dataset is spatially much larger than the spread of locations. e.g. the dataset covers europe but I'm only interested in the UK.

This PR adds a spatial slice into the __init__() of PVNetConcurrentDataset so that the input datasets are reduced cover only the spatial area needed to create the window slices around all the locations. This reduces the amount of unnecessary data loaded for each sample when the input datasets are much wider than required.

Also:

• Removed references to padding since that was removed a while ago

How Has This Been Tested?

I've run some checks locally and the PVNetConcurrentDataset class is already tested in:

tests/torch_datasets/test_pvnet_dataset.py::test_pvnet_concurrent_dataset

Checklist:

• My code follows OCF's coding style guidelines
• I have performed a self-review of my own code
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• I have checked my code and corrected any misspellings

[felix-e-h-p]

Looking all good, just a couple of comments.

Also, what about a test that fails without the buffer via select_spatial_slice_pixels_multiple?

[dfulu]

Thanks for the review @felix-e-h-p

Also, what about a test that fails without the buffer via select_spatial_slice_pixels_multiple?

I'm not sure I understand. Do you mean we'd use select_spatial_slice_pixels_multiple() with a window size other than 2 so the buffer isn't applied?

[felix-e-h-p]

Thanks for the review @felix-e-h-p

Also, what about a test that fails without the buffer via select_spatial_slice_pixels_multiple?

I'm not sure I understand. Do you mean we'd use select_spatial_slice_pixels_multiple() with a window size other than 2 so the buffer isn't applied?

Ah was thinking more a test that uses window size of 2 and just shows that removing the buffer would cause a failure.

[dfulu]

Ah was thinking more a test that uses window size of 2 and just shows that removing the buffer would cause a failure.

To do that we'd probably need to make the buffer configurable, and then we'd simply be testing that the version where the buffer isn't the default value is incorrect. That doesn't feel like best practice to me