WIP: Add support for raster loader by soniacq · Pull Request #87 · VIDA-NYU/UrbanMapper

soniacq · 2025-12-10T21:36:11Z

📚 Documentation preview 📚: https://UrbanMapper--87.org.readthedocs.build/en/87/

- Implement RasterLoader to load raster files with rasterio - Support loading GeoTIFF, PNG, JP2 and other formats - Modified to add example usage of raster loader - Raster loader returning 3D array

- Implement RasterLoader with block-wise downsampling and polygon geometry - Convert raster pixels into aggregated blocks with spatial geometry - Output GeoDataFrame with pixel_id, coordinates, value, area

…e loader)

…rBase interface.

fabiofelix

This PR is good to go after addressing the comments.

fabiofelix · 2025-12-11T19:09:54Z

This file should be merged with the main UM documentation, e.g., docs/api/loaders.md

Not really, no. This current README is outdated in the current state of this PR. RasterLoader is having a docstring so loaders.md just need:

## ::: urban_mapper.modules.loader.RasterLoader options: heading: "RasterLoader" members: - _load - preview # and whatever else

fabiofelix · 2025-12-11T19:11:41Z

Could you check all the related descriptions? RasterLoader._load always returns a GeoDataframe.

fabiofelix · 2025-12-11T19:12:30Z

This change could be discarded.

simonprovost

Thanks Sonia for this PR! I have made comments because IMHO the state of the implementation is not bullet-proof enough yet. Furthermore:

The two examples appear to have non-cleared cells as showing identical content, yet Git notifies changes. Please let's discard the visualiser as @fabiofelix suggested, but not the loader example. Let's add the loader of a small raster, please, for the sake of the interactive examples on the doc.
That same small raster file can therefore be used to have unit test for this new loader please.
It would also be nice to concentrate all commits of @JUDITH-sketch into one (such as feat: raster basics or whatever else), as there seem to be a lot of exploratory-based commits > 10.

I stay alert to any questions! Cheers.

simonprovost · 2025-12-11T20:10:50Z

            ValueError: If an unsupported format is requested.
        """
        pass
+        pass


simonprovost · 2025-12-11T20:11:08Z

 from urban_mapper.modules.loader.abc_loader import LoaderBase
 from urban_mapper.modules.loader.loaders.csv_loader import CSVLoader
 from urban_mapper.modules.loader.loaders.parquet_loader import ParquetLoader
+from urban_mapper.modules.loader.loaders.raster_loader import RasterLoader  # Importing RasterLoader of the new raster loader module


Inline comment to remove

—— Same for the others please

simonprovost · 2025-12-11T20:12:44Z

        is determined by the file extension. Supported formats include `CSV`, `shapefile`,
-        and `Parquet`.
-
+        and `Parquet`. 


I am not sure if that is a pre-commit-based modif. if not, to revert

simonprovost · 2025-12-11T20:18:47Z

+    def with_options(self, **options,) -> "LoaderFactory":
+        """
+        Set additional key-value options to configure loader behavior.
+        This method allows you to specify arbitrary configuration options, such as block size, resolution, or other loader parameters. These options will be forwarded to the loader upon instantiation.


This docstring is not clear. I would suggest something along the lines of:

Set additional key-value options to configure loader behavior. If a loader supports or requires parameter propagation during instantiation, use `with_options(<parameter_of_interest>=<value_of_your_like>)`. For example, if the loader of interest is `Raste Loader`, you can use `block_size`,`resolution`, and so on. Users are directed to each loader's docstring for more information on such permissible options. See more in `Examples` below. ```

simonprovost · 2025-12-11T20:20:09Z

    @require_attributes(["source_type", "source_data"])
    def load(self) -> gpd.GeoDataFrame:
-        """Load the data and return it as a `GeoDataFrame`.
+        """Load the data and return it as a `GeoDataFrame` or raster object.


If we update the docstring we have got to update the return type of the function, above.

simonprovost · 2025-12-11T20:59:06Z

+                # Latitude/longitude per transformation from CRS to WGS84
+                transformer = Transformer.from_crs(crs, 4326, always_xy=True)


Am not an expert in Raster but just double checking: should we not parametrise this CRS 4326?

simonprovost · 2025-12-11T21:01:13Z

+                gdf = gpd.GeoDataFrame({
+                    'pixel_id': np.arange(len(values)),
+                    'row': rows.flatten(),
+                    'col': cols.flatten(),
+                    'area': areas,
+                    'value': values,
+                    'latitude': lat,
+                    'longitude': lon,
+                    'geometry': geoms
+                }, crs=crs)
+
+                return gdf


Let's create the GDF while returning it instead of variable-instantiate it prior.

simonprovost · 2025-12-11T21:02:22Z

+            raise RuntimeError(f"Error while loading downsampled raster: {e}")
+


raise RuntimeError(f"Error while loading downsampled raster: {e}") from e

simonprovost · 2025-12-11T21:02:50Z

+            ValueError: If the requested format is not supported.
+
+        """
+        # If metadata is not loaded, try to load it


To remove all inline polluting comments please

simonprovost · 2025-12-11T21:03:43Z

+        shape = (
+            self.meta.get("count", "?"),
+            self.meta.get("height", "?"),
+            self.meta.get("width", "?")
+        )
+        dtype = self.meta.get("dtype", "?")
+        crs = self.meta.get("crs", "?")


Let's changed all ? into more readable null-information like N/A please

JUDITH-sketch and others added 18 commits July 21, 2025 18:08

feat: add basic raster data loader support

200883a

- Implement RasterLoader to load raster files with rasterio - Support loading GeoTIFF, PNG, JP2 and other formats - Modified to add example usage of raster loader - Raster loader returning 3D array

feat: add downsampled raster loader returning GeoDataFrame

3981197

- Implement RasterLoader with block-wise downsampling and polygon geometry - Convert raster pixels into aggregated blocks with spatial geometry - Output GeoDataFrame with pixel_id, coordinates, value, area

test: add tests for RasterLoader functionality

7d61fc4

feat : with_options works now

e96cf11

feat : example with time of operations

c2cd777

feat : adapted docstring

91392b9

feat : docstrings and comments translated in english

186ec5b

feat : the tests are not usefull anymore (test a former version of th…

e62f8eb

…e loader)

feat : Readme for explain rasters versions

0c54d20

feat : adding on the branch all the rasters versions and tests versions

68be1bb

feat : adding README

ffe5376

feat : README

3eef1d1

feat : new versions of notebook

68e8170

build(deps): add rasterio to pyproject.toml to support raster loader.

aed79d6

refactor(loader): consolidate raster loader into a single final version

145a8df

Fix (loader): Fix merging issues.

ef33696

fix(loader): propagate with_options and align RasterLoader with Loade…

bcee416

…rBase interface.

Fix (loader): Add examples how to load a raster data.

20831f5

soniacq force-pushed the feat/raster_loader branch from 34a5f39 to 20831f5 Compare December 11, 2025 17:18

fabiofelix requested changes Dec 11, 2025

View reviewed changes

simonprovost self-requested a review December 11, 2025 20:02

simonprovost requested changes Dec 11, 2025

View reviewed changes

		# Latitude/longitude per transformation from CRS to WGS84
		transformer = Transformer.from_crs(crs, 4326, always_xy=True)

		raise RuntimeError(f"Error while loading downsampled raster: {e}")

Conversation

soniacq commented Dec 10, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fabiofelix left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

simonprovost left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

simonprovost Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

simonprovost Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

soniacq commented Dec 10, 2025 •

edited by github-actions Bot

Loading

simonprovost left a comment •

edited

Loading

simonprovost Dec 11, 2025 •

edited

Loading

simonprovost Dec 11, 2025 •

edited

Loading