Skip to content

fix(project): reduce memory consumption by using optimized CouchDB view#3985

Draft
bibhuti230185 wants to merge 1 commit intoeclipse-sw360:mainfrom
siemens:fix/project-cache-memory-optimization
Draft

fix(project): reduce memory consumption by using optimized CouchDB view#3985
bibhuti230185 wants to merge 1 commit intoeclipse-sw360:mainfrom
siemens:fix/project-cache-memory-optimization

Conversation

@bibhuti230185
Copy link
Contributor

@bibhuti230185 bibhuti230185 commented Mar 25, 2026

Summary

This PR optimizes memory consumption in the project clearing cache by replacing repository.getAll() with a dedicated CouchDB view that fetches only cache-relevant fields.

Problem

The cachedAllProjectsIdMap in ProjectDatabaseHandler was consuming ~83% of heap memory due to caching all project fields (including heavy fields like descriptions, external IDs, security assessments, etc.) for every project in the system.

Solution

  • Created a new CouchDB view forClearingCache in ProjectRepository that emits only the 12 fields required for clearing state cache operations
  • Updated getRefreshedAllProjectsIdMap() to use the optimized view instead of repository.getAll()
  • This reduces memory footprint by approximately 80% while maintaining full functionality

Fields included in the optimized view:

  • _id - Project identifier
  • linkedProjects - Project hierarchy relationships
  • releaseIdToUsage - Release dependencies
  • createdBy, projectResponsible, leadArchitect - User permissions
  • moderators, contributors - Permission groups
  • businessUnit - Access control
  • visbility - Visibility settings
  • clearingState - Clearing workflow state
  • attachments - Attachment metadata

Changes Made

  1. ProjectRepository.java - Added forClearingCache view with map/reduce functions
  2. ProjectDatabaseHandler.java - Updated cache refresh logic to use new view
  3. ProjectRepositoryTest.java - Added comprehensive unit tests (4 test cases)

Issue: N/A (Performance optimization identified through heap dump analysis)

Suggest Reviewer

@GMishx - This change affects the project clearing cache mechanism

How To Test?

Unit Tests

mvn test -pl backend/common -Dtest=ProjectRepositoryTest

All 4 new test cases verify:

  1. All projects are returned by the view
  2. Required fields are properly included
  3. Minimal projects (with few fields) are handled correctly
  4. Project hierarchy traversal works as expected

Integration Testing

  1. Start SW360 with the changes
  2. Create multiple projects with various configurations (linked projects, releases, etc.)
  3. Perform operations that trigger cache refresh:
    • View project clearing state
    • Navigate project hierarchy
    • Check project permissions
  4. Monitor heap usage - should see significant reduction in memory consumption
  5. Verify that all clearing-related functionality works as before

Memory Profiling (Recommended)

  1. Take a heap dump before the change
  2. Take a heap dump after the change
  3. Compare ProjectDatabaseHandler.cachedAllProjectsIdMap memory usage
  4. Expected result: ~80% reduction in size

Checklist

Must:

  • All related issues are referenced in commit messages and in PR
  • Code follows SW360 coding standards and patterns
  • Added comprehensive unit tests for new view functionality
  • Uses CouchDB design document pattern for views
  • Maintains backward compatibility (no API changes)
  • Code is AI-assisted (GitHub Copilot for test case generation)

Additional Notes

  • No database migration required - CouchDB views are created automatically on first access
  • No API changes - This is purely an internal optimization
  • Backward compatible - Existing functionality remains unchanged
  • Performance impact - Positive only (reduces memory, no functional changes)
  • Risk assessment - Low (isolated change with comprehensive tests)

…ew for project cache

Signed-off-by: Bibhuti Bhusan Dash <bibhuti230185@gmail.com>
@bibhuti230185 bibhuti230185 changed the title fix(project): reduce memory consumption by using optimized CouchDB vi… fix(project): reduce memory consumption by using optimized CouchDB view Mar 25, 2026
@GMishx GMishx added WIP work in progress needs code review needs general test This is general testing, meaning that there is no org specific issue to check for labels Mar 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs code review needs general test This is general testing, meaning that there is no org specific issue to check for WIP work in progress

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants