Add generic sparse field read/write/delete API endpoints

[Copilot]

• Add sparse_field action with GET/PUT/DELETE
• PUT and DELETE: check schema conflict (409 if conflict)
• Tests for all paths including schema conflict for both PUT and DELETE
• Apply black reformatting (line-length=120 as configured in pyproject.toml)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Generic use of sparse metadata fields</issue_title>
<issue_description>Sparse fields have been introduced in #11511 for storing custom fields values for resources.
Handling of such fields is orchestrated by the MetadataManager, following the structure of the json schema, with the addition of multilang fields.

Sparse fields can be anyway used also for "external usage", i.e. for storing data related to dataset but not strictly related to the declared json schema.
For instance, the extraMetadata json field in ResourceBase could be easily and more cleanly stored in a set of sparse fields.

Storing custom data in sparse fields should not conflict with the naming of the metadata fields declared in the json schema, so any access to such fields should be constrained to avoid declared fields.
We may think of an API which grants read access to any sparse field (according to user access permissions to the related resource), but forbids access to sparse fields with declared names.

• GET /resource/<id>/sparse/<sparsekey>
  • check resource read access permission or 404
  • retrieve value
  • 404 on key not found
• PUT /resource/<id>/sparse/<sparsekey> (upsert)
  • check resource read access permission or 404
  • check resource write access permission or 403
  • get the json schema, check if sparsekey is a key of the schema
    • 409 if conflict
  • insert or update sparse field content

Note that this approach, based on jsonschema keys, would also prevent writing sparse key of fields handled by the BaseHander, i.e. fields the are within the ResourceBase table, but this is a wanted behaviour, so there will not be problems when/if some of the base field will be moved to the sparse domain.

</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Generic use of sparse metadata fields #13974

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[cla-bot]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @copilot on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

[etj]

@copilot also add a DELETE call

[cla-bot]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @copilot on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

[Copilot]

@copilot also add a DELETE call

Done. Added DELETE /api/v2/metadata/sparse/<pk>/<sparsekey> in fd7071e:

• 404 if resource not found or user lacks read access
• 403 if user lacks change_resourcebase_metadata
• 404 if the key doesn't exist
• 200 on successful deletion

Also added 5 tests covering all DELETE paths.

[Copilot]

Pull request overview

This PR adds generic sparse field API endpoints to enable reading, writing, and deleting arbitrary resource-related metadata beyond what's declared in the JSON schema. The implementation provides three new REST endpoints (GET, PUT, DELETE) for managing sparse fields, with appropriate permission checks and schema conflict detection.

Changes:

• Added sparse_field action to MetadataViewSet with GET, PUT, and DELETE methods for managing external sparse field data
• Implemented comprehensive test suite covering permission scenarios, schema conflicts, and CRUD operations for sparse fields
• Added security measures to prevent resource enumeration and schema field shadowing

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.

File	Description
geonode/metadata/api/views.py	Implements sparse_field API endpoint with GET/PUT/DELETE operations, permission checks, and schema conflict validation
geonode/metadata/tests/tests.py	Adds SparseFieldApiTests class with comprehensive test coverage for all sparse field operations and edge cases

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Database operations (update_or_create on line 188) are not wrapped in try-except blocks. If a database integrity error occurs (e.g., due to value length exceeding max_length or other constraint violations), it will result in an unhandled exception and a 500 Internal Server Error response instead of a more appropriate 400 Bad Request. Consider wrapping database operations in try-except blocks to catch IntegrityError, DataError, or ValidationError and return appropriate error responses.

[Copilot]

The sparsekey parameter is not validated for length or content. Since the SparseField model has a name field with max_length=64, attempting to use a key longer than 64 characters will result in a database error rather than a clear 400 Bad Request response. Consider adding validation to ensure sparsekey doesn't exceed 64 characters and only contains valid characters.

[Copilot]

No test case validates the scenario when the 'value' field is missing from the PUT request body. The current implementation would accept request.data without a 'value' key and set the field to None (due to request.data.get("value", None)). Consider adding a test to verify the behavior when the value key is absent, or alternatively, add validation to require the value field in the request.

Suggested change

	@patch("geonode.security.registry.PermissionsHandlerRegistry.user_has_perm", return_value=True)
	@patch("geonode.metadata.manager.metadata_manager.get_schema")
	def test_put_missing_value_field_creates_sparse_field_with_null(self, mock_get_schema, mock_perm):
	"""
	Ensure that a PUT request without a 'value' key is accepted and
	results in a SparseField whose value is None (current behavior).
	"""
	mock_get_schema.return_value = {"properties": {"custom_key": {}}}
	url = self._url(self.resource.pk, "custom_key")
	# Intentionally omit 'value' from the payload
	response = self.client.put(url, data={}, format="json")
	self.assertEqual(response.status_code, status.HTTP_200_OK)
	sf = SparseField.objects.get(resource=self.resource, name="custom_key")
	self.assertIsNone(sf.value)

[cla-bot]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @copilot on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

[etj]

@copilot as per the UPDATE case, the DELETE should not delete keys belonging to the jsonschema. Add check and related tests.

[Copilot]

@copilot as per the UPDATE case, the DELETE should not delete keys belonging to the jsonschema. Add check and related tests.

Done in 4cdaa01. DELETE now checks the JSON schema before deleting — returns 409 if the key matches a declared schema property, identical to the PUT behaviour. Test added in test_delete_schema_conflict_returns_409.

[cla-bot]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @copilot on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

[etj]

@copilot apply "black" reformatting as configured in the project

[cla-bot]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @copilot on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

[Copilot]

@copilot apply "black" reformatting as configured in the project

Done in 6868852. Applied black==24.4.2 with line-length=120 (as configured in pyproject.toml) to both changed files.

[etj]

@copilot you implemented the path as `url_path=r"sparse/(?P\d+)/(?P[A-Za-z0-9_-]+)",' not as described in the original issue, why? just answer, do not perform code changes for now.

[Copilot]

@copilot you implemented the path as `url_path=r"sparse/(?P\d+)/(?P[A-Za-z0-9_-]+)",' not as described in the original issue, why? just answer, do not perform code ch...

Two differences from the issue description:

1. Base path /api/v2/metadata/sparse/ instead of /resource/<id>/sparse/: The issue described the endpoint conceptually. I placed it inside the existing MetadataViewSet (rooted at /api/v2/metadata/) because that's where all metadata-related endpoints already live in GeoNode. There is no /resource/ top-level API router in the codebase.

2. [A-Za-z0-9_\-]+ instead of [^/]+: This came from a resolved bot review suggestion in the first round of review comments, which I accepted. The original pattern I used was indeed [^/]+ (any non-slash character), which is closer to the issue's implied intent of allowing any key name.

[codecov]

Codecov Report

❌ Patch coverage is 2.39521% with 163 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.29%. Comparing base (7f6c030) to head (6868852).
⚠️ Report is 4 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (7f6c030) and HEAD (6868852). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (7f6c030)	HEAD (6868852)
	7	6

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #13975       +/-   ##
===========================================
- Coverage   74.07%   52.29%   -21.78%     
===========================================
  Files         950      950               
  Lines       56826    56964      +138     
  Branches     7719     7733       +14     
===========================================
- Hits        42093    29791    -12302     
- Misses      13044    25787    +12743     
+ Partials     1689     1386      -303     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.