feat: multi-keys design for GSI

[hackmajoris]

Changes

This PR implements a complete refactoring to DynamoDB Single-Table Design using the new Multi-Key (Composite Keys) GSI feature announced by AWS.

AWS Multi-Key GSI Blog: https://aws.amazon.com/blogs/database/multi-key-support-for-global-secondary-index-in-amazon-dynamodb/

Database Architecture

Single-Table Design with Multi-Entity Support

Table: glad-entities

• Primary Key: entity_id (Partition Key only)
• Entity Types: User, Skill (master), UserSkill

Entity Key Patterns

User:       USER#<username>
Skill:      SKILL#<skill_id>
UserSkill:  USERSKILL#<username>#<skill_id>

Global Secondary Indexes (GSI)

1. ByUser - User Profile + Skills

• PK: Username
• SK: EntityType
• Use Case: Query all data for a user (profile + skills in one query)

2. ByEntityType - Entity Listing

• PK: EntityType
• SK: SkillName
• Use Case: List all users, all master skills, etc.

3. BySkillID - Denormalization Sync

• PK: skill_id
• SK: Username
• Use Case: Find all UserSkills referencing a master skill (for sync operations)

4. SkillsByLevel - Skill Search (Multi-Key Composite)

• PK: SkillName
• SK: ProficiencyLevel
• Additional Sort Key: YearsOfExperience
• Use Case: Find "Expert Python developers with 5+ years" in a single query

5. SkillsByCategory - Category Browsing

• PK: EntityType
• SK: Category
• Use Case: List all skills in "Programming" category

Data Model

3-Entity Model

1. User (Profile)

type User struct {
    Username     string    // Denormalization key
    Name         string
    PasswordHash string
    Email        string
    CreatedAt    time.Time
    UpdatedAt    time.Time
    
    // DynamoDB Keys
    EntityID   string    // USER#<username>
    EntityType string    // "User"
}

2. Skill (Master Catalog)

type Skill struct {
    SkillID     string    // Immutable ID (e.g., "python")
    SkillName   string    // Display name (e.g., "Python")
    Description string
    Category    string    // Programming, Cloud, DevOps, etc.
    Tags        []string
    CreatedAt   time.Time
    UpdatedAt   time.Time
    
    // DynamoDB Keys
    EntityID   string    // SKILL#<skill_id>
    EntityType string    // "Skill"
}

3. UserSkill (User's Skills with Denormalized Data)

type UserSkill struct {
    // References
    Username        string    // FK to User
    SkillID         string    // FK to Skill (immutable)
    
    // Denormalized from Skill (for query performance)
    SkillName       string
    Category        string
    
    // User-specific data
    ProficiencyLevel    ProficiencyLevel  // Beginner, Intermediate, Advanced, Expert
    YearsOfExperience   int
    Notes               string
    CreatedAt           time.Time
    UpdatedAt           time.Time
    
    // DynamoDB Keys
    EntityID   string    // USERSKILL#<username>#<skill_id>
    EntityType string    // "UserSkill"
}

Key Features Implemented

1. Master Skills Catalog

• Centralized skill metadata (names, categories, descriptions)
• Skills have immutable skill_id and mutable display metadata
• UserSkills reference master skills via skill_id
• Denormalization pattern: SkillName and Category copied to UserSkills for query performance

2. Repository Layer Refactoring

New Repository Interfaces:

• UserRepository - User CRUD operations
• SkillRepository - UserSkill CRUD operations
• MasterSkillRepository - Master skill catalog management
• Repository - Unified interface combining all three

Implementations:

• DynamoDBRepository - Production implementation using AWS SDK
• MockRepository - In-memory implementation for local development

3. Centralized Key Generation

File: cmd/app/internal/models/keys.go

Single source of truth for all entity key construction:

func BuildUserEntityID(username string) string
func BuildMasterSkillEntityID(skillID string) string
func BuildUserSkillEntityID(username, skillID string) string

4. Master Skill Lookup Service

Before:

// Hardcoded temporary values
skillID := skillName
category := "Uncategorized"

After:

// Lookup from master catalog
masterSkill, err := s.masterSkillRepo.GetMasterSkill(skillID)
skill, err := models.NewUserSkill(
    username,
    masterSkill.SkillID,      // From master
    masterSkill.SkillName,    // From master
    masterSkill.Category,     // From master
    proficiencyLevel,
    yearsOfExperience,
)

5. Comprehensive Validation

Skill ID Validation:

• Format: lowercase, alphanumeric, dashes only
• Length: 1-50 characters
• Example valid IDs: python, aws-lambda, react-js

Category Validation:

• Whitelist of 10 predefined categories
• Categories: Programming, Cloud, DevOps, Database, Frontend, Backend, Mobile, Data, Security, Other

Skill Name Validation:

• Length: 2-100 characters

Implementation Details

Files Created

Database Layer:

• cmd/app/internal/database/constants.go - Table and GSI name constants
• cmd/app/internal/database/entity_keys.go - Entity key constants (deprecated, moved to models)
• cmd/app/internal/database/master_skill_repository.go - Master skill operations
• cmd/app/internal/database/user_skill_repository.go - User skill operations
• cmd/app/internal/database/user_repository.go - User operations (extracted from monolithic file)
• cmd/app/internal/database/skill_repository.go - Skill repository interface

Models:

• cmd/app/internal/models/keys.go - Centralized entity key builders
• cmd/app/internal/models/skill.go - Master skill domain model with validation
• cmd/app/internal/models/user_skill.go - User skill domain model

Services:

• cmd/app/internal/service/master_skill_service.go - Master skill business logic

Handlers:

• cmd/app/internal/handler/master_skill_handler.go - Master skill API endpoints

Documentation:

• cmd/app/testdata/dynamo-db-multi-keys-queries.md - Query examples and test data
• docs/api-testing/api-test.http - API testing examples
• docs/api-testing/http-client.env.json - Environment configuration

Files Modified

Core Application:

• cmd/app/main.go - Updated service initialization with both repositories
• cmd/app/integration_test.go - Updated test setup for new repository structure

Database Layer:

• cmd/app/internal/database/dynamodb.go - Refactored to interface implementation
• cmd/app/internal/database/factory.go - Repository factory pattern
• cmd/app/internal/database/mock.go - Fixed key generation, added master skills support
• cmd/app/internal/database/mock_test.go - Updated tests for skillID parameter

Models:

• cmd/app/internal/models/user.go - Updated to use centralized key builders
• cmd/app/internal/models/user_skill.go - Added denormalization fields

Services:

• cmd/app/internal/service/skill_service.go - Implemented master skill lookup

Handlers:

• cmd/app/internal/handler/user_handler.go - Updated for new repository interfaces
• cmd/app/internal/handler/user_handler_test.go - Test updates

Infrastructure:

• deployments/app/cdk.go - Complete GSI definitions with multi-key support
• cmd/app/internal/dto/dto.go - Added DTOs for master skills

Documentation:

• README.md - Added Multi-Key GSI reference

Files Deleted (Outdated Documentation)

• docs/README-DYNAMODB-DESIGN.md
• docs/dynamodb-quick-reference.md
• docs/dynamodb-single-table-design-plan.md
• docs/entity-addition-protocol.md

Testing

Test Files Updated

• cmd/app/internal/database/mock_test.go - Updated to use skillID parameter
• cmd/app/integration_test.go - Updated repository initialization

Query Performance Examples

Find Expert Python Developers with 5+ Years

aws dynamodb query \
  --table-name glad-entities \
  --index-name SkillsByLevel \
  --key-condition-expression "SkillName = :skill AND ProficiencyLevel = :level" \
  --filter-expression "YearsOfExperience >= :years" \
  --expression-attribute-values '{
    ":skill": {"S": "Python"},
    ":level": {"S": "Expert"},
    ":years": {"N": "5"}
  }'

Get User Profile + All Skills (Single Query)

aws dynamodb query \
  --table-name glad-entities \
  --index-name ByUser \
  --key-condition-expression "Username = :user" \
  --expression-attribute-values '{":user": {"S": "john"}}'

List All Master Skills in Programming Category

aws dynamodb query \
  --table-name glad-entities \
  --index-name SkillsByCategory \
  --key-condition-expression "EntityType = :type AND Category = :cat" \
  --expression-attribute-values '{
    ":type": {"S": "Skill"},
    ":cat": {"S": "Programming"}
  }'

Design Patterns Applied

1. Single-Table Design - One table for all entity types
2. Denormalization - SkillName/Category copied to UserSkills for query performance
3. Composite Keys - Multi-attribute sort keys for complex queries
4. Repository Pattern - Clean abstraction over data access
5. Factory Pattern - Repository creation based on environment
6. Builder Pattern - Centralized entity key construction
7. Validation Pattern - Input validation in domain models

🔗 References

• AWS Multi-Key GSI Blog
• DynamoDB Best Practices
• Single-Table Design

---