Skip to content

Per-Category BulkData Configuration #215

@bburda

Description

@bburda

Summary

The current BulkDataStore uses a flat vector<string> for categories and a single global max_upload_bytes limit. The original plan for #210 included a BulkDataCategoryConfig struct with richer per-category configuration (access modes, per-category size limits, source/upload path templates, human-readable names). This was intentionally deferred to keep the initial PR scoped, but should be implemented as a follow-up.


Proposed solution

1. Per-Category Config Struct

Replace vector<string> categories with a BulkDataCategoryConfig struct:

struct BulkDataCategoryConfig {
  std::string id;           // Short ID (used in URLs): "calibration", "firmware"
  std::string name;         // Human-readable display name: "Calibration Data"
  std::string access;       // "read-only", "write-only", "read-write" (default: "read-write")
  size_t max_file_size{0};  // Per-category max upload size (0 = use global limit)
  std::string source_path;  // Template for pre-existing files: "/data/{entity_id}/cal/"
  std::string upload_path;  // Template for uploads: "/uploads/{entity_id}/{category}/"
};

2. Behavior Changes

  • Access mode enforcement: handle_upload rejects uploads to read-only categories (400). handle_delete rejects deletes on read-only categories. write-only categories reject list/download.
  • Per-category size limits: max_file_size per category overrides the global max_upload_bytes when set.
  • Source path mapping: Categories with source_path expose pre-existing files on disk (not uploaded via API). Useful for calibration data managed by external tooling.
  • Upload path templates: Categories with upload_path store uploads in a custom location instead of the default {storage_dir}/{entity_id}/{category}/ layout.
  • Human-readable names: Expose name field in the GET /bulk-data categories list response.

3. Configuration (gateway_params.yaml)

bulk_data:
  storage_dir: "/tmp/ros2_medkit_bulkdata"
  max_upload_size: "50MB"
  categories:
    - id: calibration
      name: "Calibration Data"
      access: read-write
      max_file_size: "10MB"
    - id: firmware
      name: "Firmware Images"
      access: read-write
      max_file_size: "100MB"
    - id: logs
      name: "System Logs"
      access: read-only
      source_path: "/var/log/ros2/{entity_id}/"

4. API Response Change

GET /api/v1/{entity-path}/bulk-data returns objects instead of strings:

{
  "items": [
    {"id": "rosbags", "name": "Rosbag Recordings", "access": "read-only"},
    {"id": "calibration", "name": "Calibration Data", "access": "read-write"},
    {"id": "firmware", "name": "Firmware Images", "access": "read-write"}
  ]
}

Additional context

Related: #210 (Bulk Data Upload & Delete)

Files to Modify

  • include/ros2_medkit_gateway/bulk_data_store.hpp — Add BulkDataCategoryConfig struct, update constructor
  • src/bulk_data_store.cpp — Per-category validation, path resolution
  • src/http/handlers/bulkdata_handlers.cpp — Access mode checks in upload/delete/download
  • src/gateway_node.cpp — Parse new YAML config format
  • config/gateway_params.yaml — Add example config
  • docs/api/rest.rst — Update categories response schema
  • docs/configuration.rst — Document new config options
  • Unit tests for access mode enforcement and per-category limits

Acceptance Criteria

  • BulkDataCategoryConfig struct with all fields
  • Access mode enforced on upload, delete, and download
  • Per-category max_file_size overrides global limit
  • source_path mapping for pre-existing files
  • Unit tests for each access mode
  • Integration test for config-driven category setup
  • Docs updated

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions