- 0. Overview
- 1. Core Generic Types
- 2. Generic Function Patterns
- 3. Best Practices
This document defines the generic types and patterns used throughout the NovusPack API. It provides the technical specifications for type-safe generic implementations across all API modules.
Note: For collection operations (filtering, mapping, searching, aggregation), NovusPack uses samber/lo instead of custom generic functions.
See samber/lo Usage Standards for detailed guidelines on when and how to use samber/lo functions.
- samber/lo Usage Standards - Guidelines for using samber/lo collection operations
- Core Package Interface - Generic types and interfaces
- File Management API - Generic tag operations and collections
- Package Compression API - Generic strategy patterns and compression concurrency
- Streaming and Buffer Management - Generic buffer pools and streaming concurrency
- Security Validation API - Generic encryption patterns and type-safe security
This section describes the core generic types used throughout the API.
The Option type provides type-safe handling of optional values.
// Option provides type-safe optional configuration values
type Option[T any] struct {
value T
set bool
}// Set sets the option value.
func (o *Option[T]) Set(value T)// Get returns the value and a boolean indicating if the value is set.
func (o *Option[T]) Get() (T, bool)// GetOrDefault returns the value if set, otherwise returns the default value.
func (o *Option[T]) GetOrDefault(defaultValue T) T// IsSet returns true if the option has a value set.
func (o *Option[T]) IsSet() bool// Clear clears the option value.
func (o *Option[T]) Clear()// Create an option
var opt Option[string]
opt.Set("hello")
// Get value with default
value := opt.GetOrDefault("default")
// Check if set
if opt.IsSet() {
val, _ := opt.Get()
fmt.Println(val)
}The Result type provides type-safe handling of operations that may fail.
// Result represents a value that may be an error.
type Result[T any] struct {
value T
err error
}// Ok creates a successful Result with the given value.
func Ok[T any](value T) Result[T]// Err creates a failed Result with the given error.
func Err[T any](err error) Result[T]// Unwrap returns the value and error from the Result.
func (r Result[T]) Unwrap() (T, error)// IsOk returns true if the Result contains a value (no error).
func (r Result[T]) IsOk() bool// IsErr returns true if the Result contains an error.
func (r Result[T]) IsErr() bool// Success case
result := Ok("success")
if result.IsOk() {
value, _ := result.Unwrap()
fmt.Println(value)
}
// Error case
result := Err[string](errors.New("failed"))
if result.IsErr() {
_, err := result.Unwrap()
fmt.Println(err)
}PathEntry represents a minimal file or directory path.
PathEntry is a shared type used by both FileEntry and PathMetadataEntry to represent path information. It contains only the path string itself - no metadata, permissions, or symlink information.
Path metadata (permissions, timestamps, ownership, tags, symlinks) is stored separately in metadata structures (see Package Metadata API - Path Metadata System) to allow the same file content to have different permissions at different paths.
This section describes the PathEntry structure for path management.
// PathEntry represents a minimal file or directory path
type PathEntry struct {
// PathLength is the length of the path in bytes (UTF-8)
// Must match the actual length of the Path field
PathLength uint16
// Path is the UTF-8 encoded file or directory path (not null-terminated)
// Must be valid UTF-8 and non-empty (after trimming whitespace)
// All paths MUST be stored with a leading "/" to ensure full path references (see Package Path Semantics in api_core.md)
// The root path "/" represents the package root
// All paths use forward slashes (/) as separators, regardless of source platform
Path string
}// Validate performs validation checks on the PathEntry
// Returns error if PathLength doesn't match Path length, or if Path is empty/invalid
func (p *PathEntry) Validate() error// Size returns the total size of the PathEntry in bytes
// Formula: 2 (PathLength) + PathLength (Path)
func (p *PathEntry) Size() int// ReadFrom reads a PathEntry from the provided io.Reader
// Implements io.ReaderFrom interface
// Returns number of bytes read and any error encountered
func (p *PathEntry) ReadFrom(r io.Reader) (int64, error)// WriteTo writes a PathEntry to the provided io.Writer
// Implements io.WriterTo interface
// Returns number of bytes written and any error encountered
func (p *PathEntry) WriteTo(w io.Writer) (int64, error)// GetPath returns the path string as stored (Unix-style with forward slashes)
func (p *PathEntry) GetPath() string// GetPathForPlatform returns the path string converted for the specified platform
// On Windows, converts forward slashes to backslashes
// On Unix/Linux, returns the path as stored (with forward slashes)
func (p *PathEntry) GetPathForPlatform(isWindows bool) stringThe binary format for PathEntry is a minimal variable-length structure containing only the path information.
Field Layout (in order):
| Field | Size | Type | Description |
|---|---|---|---|
| PathLength | 2 bytes | uint16 | Length of path in bytes (UTF-8 encoded) |
| Path | variable | UTF-8 string | File or directory path (not null-terminated) |
Total Size: 2 + PathLength bytes
Byte Order: All multi-byte integers are stored in little-endian format.
- PathLength: 2-byte unsigned integer (little-endian) specifying the number of bytes in the Path field
- Path: UTF-8 encoded string, exactly PathLength bytes, not null-terminated
Path Metadata: Permissions, timestamps, ownership, tags, and symlink information for paths are stored separately in metadata structures (see Package Metadata API - Path Metadata System). This design allows the same file content to have different permissions and metadata at different paths.
For path storage rules, normalization, display, and extraction behavior, see Package Path Semantics in the Core Package Interface API.
The path storage rules cover:
- Path normalization rules (separator normalization, leading slash requirement, dot segment canonicalization, Unicode normalization, path length limits)
- Path normalization on storage
- Path display and extraction behavior
- Case sensitivity rules
- Path length validation utilities
The Validate() method performs the following checks:
- Path Length Consistency:
PathLengthmust exactly match the byte length of thePathfield - Path Non-Empty:
Pathmust not be empty or contain only whitespace (after trimming) - UTF-8 Validity:
Pathmust be valid UTF-8 (enforced by Go's string type) - Path Format:
Pathmust begin with/(leading slash is mandatory for all stored paths)- Exception: The root path
/itself is valid (as a directory)
- Exception: The root path
- Path Separators:
Pathmust use forward slashes (/) as separators, not backslashes (\) - Unicode Normalization:
Pathmust be normalized to NFC form (see Unicode Normalization) - No Null Bytes:
Pathmust not contain null bytes (\x00) - No Trailing Slash for Files: File paths must not end with
/(directories must end with/)
If any validation check fails, Validate() returns a PackageError with ErrTypeValidation.
This section describes the usage context for generic types.
FileEntry.Pathsis a slice ofPathEntrystructures- The first path entry (index 0) is considered the primary path
- Additional paths (index 1+) are secondary paths that point to the same content
- Multiple paths enable efficient storage of hard links and symbolic links
- Each path can have its own metadata (permissions, timestamps, ownership) stored in
PathMetadataEntry - All paths MUST be stored with a leading
/(see Package Path Semantics)
PathMetadataEntry.Pathis a string path- Directory paths must end with
/(forward slash) - The path represents the path location in the package hierarchy
Cross-Reference: For information about how PathEntry is used within FileEntry structures in the package file format, see Package File Format - Path Entries.
Use samber/lo for all collection operations.
NovusPack uses samber/lo for collection operations including filtering, mapping, searching, aggregation, and duplicate detection.
No custom collection interface is needed.
For collection operation patterns and examples, see samber/lo Usage Standards.
Use native Go data structures with samber/lo for operations.
NovusPack uses native Go data structures (slices, maps) with samber/lo functions for operations.
No custom wrapper types are needed.
Use native map[K]V with samber/lo functions:
import "github.com/samber/lo"
// Extract keys
keys := lo.Keys(myMap)
// Extract values
values := lo.Values(myMap)
// Extract entries (key-value pairs)
entries := lo.Entries(myMap)
// Transform map entries
transformed := lo.MapEntries(myMap, func(k K, v V) (K2, V2) {
return transformKey(k), transformValue(v)
})Use native map[T]bool or slices with samber/lo functions:
import "github.com/samber/lo"
// Remove duplicates from slice
unique := lo.Uniq(items)
// Remove duplicates by key
unique := lo.UniqBy(items, func(item T) K { return item.Key() })
// Check if slice contains value
contains := lo.Contains(items, value)Use samber/lo aggregation functions:
import "github.com/samber/lo"
// Sum values
total := lo.SumBy(items, func(item T) int { return item.Size() })
// Count matching items
count := lo.CountBy(items, func(item T) bool { return item.IsValid() })
// Reduce/accumulate
result := lo.Reduce(items, func(acc U, item T, _ int) U {
return accumulate(acc, item)
}, initialValue)See samber/lo Usage Standards for more patterns and examples.
Generic strategy pattern for processing different data types.
// Strategy defines a generic strategy pattern for processing different data types.
type Strategy[T any, U any] interface {
Process(ctx context.Context, input T) (U, error)
Name() string
Type() string
}Generic validation system for type-safe validation.
// Validator defines a generic validation interface for type-safe validation.
type Validator[T any] interface {
Validate(value T) error
}This section describes the ValidationRule structure for validation operations.
// ValidationRule represents a single validation rule
type ValidationRule[T any] struct {
Name string
Predicate func(T) bool
Message string
}// Returns *PackageError on failure
func (r *ValidationRule[T]) Validate(value T) errorGeneric concurrency and thread safety patterns for type-safe concurrent operations.
// WorkerPool manages concurrent workers for any data type
type WorkerPool[T any] struct {
mu sync.RWMutex
workers []*Worker[T]
workChan chan Job[T]
done chan struct{}
config *ConcurrencyConfig
}// Worker represents a single worker in the pool
type Worker[T any] struct {
mu sync.RWMutex
id int
workChan chan Job[T]
done chan struct{}
strategy Strategy[T, T]
}// Job represents a unit of work for any data type
type Job[T any] struct {
ID string
Data T
Result chan Result[T]
Context context.Context
Priority int
}// ConcurrencyConfig defines thread safety and worker management settings
type ConcurrencyConfig struct {
// Worker management
MaxWorkers int // Maximum parallel workers (0 = auto-detect)
WorkerTimeout time.Duration // Worker timeout for graceful shutdown
WorkerBufferSize int // Worker channel buffer size (0 = auto-calculate)
// Thread safety
UseMutex bool // Use mutex for shared state protection
UseRWMutex bool // Use read-write mutex for better read performance
LockTimeout time.Duration // Lock acquisition timeout
// Resource management
MaxConcurrentOps int // Maximum concurrent operations (0 = no limit)
ResourcePoolSize int // Resource pool size for workers
}// ThreadSafetyMode defines the level of thread safety guarantees
type ThreadSafetyMode int
const (
ThreadSafetyNone ThreadSafetyMode = iota // No thread safety guarantees
ThreadSafetyReadOnly // Read-only operations are safe
ThreadSafetyConcurrent // Concurrent read/write operations
ThreadSafetyFull // Full thread safety with synchronization
)This section describes generic concurrency methods and patterns.
// Start initializes and starts the worker pool
// Returns *PackageError on failure
func (p *WorkerPool[T]) Start(ctx context.Context) error// Stop gracefully shuts down the worker pool
// Returns *PackageError on failure
func (p *WorkerPool[T]) Stop(ctx context.Context) error// SubmitJob submits a job to the worker pool
// Returns *PackageError on failure
func (p *WorkerPool[T]) SubmitJob(ctx context.Context, job Job[T]) error// GetWorkerStats returns current worker pool statistics
func (p *WorkerPool[T]) GetWorkerStats() WorkerStats// ProcessConcurrently processes multiple items concurrently using the worker pool
func ProcessConcurrently[T any](ctx context.Context, items []T, processor Strategy[T, T], config *ConcurrencyConfig) ([]Result[T], error)Generic configuration patterns for type-safe configuration management.
// Config provides type-safe configuration for any data type
type Config[T any] struct {
ChunkSize Option[int64]
MaxMemoryUsage Option[int64]
CompressionLevel Option[int]
Strategy Option[Strategy[T, T]]
Validator Option[Validator[T]]
}This section describes the ConfigBuilder structure for building configurations.
// ConfigBuilder provides fluent configuration building
type ConfigBuilder[T any] struct {
config *Config[T]
}See NewConfigBuilder Function (Streaming Configuration) for the complete function definition.
// WithChunkSize sets the chunk size for the configuration.
func (b *ConfigBuilder[T]) WithChunkSize(size int64) *ConfigBuilder[T]// WithMemoryUsage sets the memory usage limit for the configuration.
func (b *ConfigBuilder[T]) WithMemoryUsage(usage int64) *ConfigBuilder[T]// WithCompressionLevel sets the compression level for the configuration.
func (b *ConfigBuilder[T]) WithCompressionLevel(level int) *ConfigBuilder[T]// WithStrategy sets the processing strategy for the configuration.
func (b *ConfigBuilder[T]) WithStrategy(strategy Strategy[T, T]) *ConfigBuilder[T]// WithValidator sets the validator for the configuration.
func (b *ConfigBuilder[T]) WithValidator(validator Validator[T]) *ConfigBuilder[T]// Build constructs and returns the final configuration.
func (b *ConfigBuilder[T]) Build() *Config[T]This section describes generic function patterns used throughout the API.
Use samber/lo for all collection operations.
NovusPack uses samber/lo for all collection operations including filtering, mapping, searching, and aggregation.
This provides better performance, wider adoption, and reduces maintenance burden.
See also Collection Operations and Data Structure Operations for additional patterns.
Use these samber/lo functions for collection operations:
- Filtering:
lo.Filter()orlo.Reject() - Mapping:
lo.Map() - Searching:
lo.Find(),lo.Contains(), orlo.IndexOf() - Aggregation:
lo.Reduce(),lo.SumBy(), orlo.CountBy() - Duplicate Detection:
lo.UniqBy()orlo.FindDuplicates() - Validation:
lo.EveryBy()orlo.SomeBy()
import "github.com/samber/lo"
// Filter items
filtered := lo.Filter(items, func(item T, _ int) bool {
return item.IsValid()
})
// Map transformation
mapped := lo.Map(items, func(item T, _ int) U {
return transform(item)
})
// Find element
found, ok := lo.Find(items, func(item T) bool {
return item.Matches(criteria)
})
// Sum aggregation
total := lo.SumBy(items, func(item T) int {
return item.Size()
})
// Reduce operation
result := lo.Reduce(items, func(acc U, item T, _ int) U {
return accumulate(acc, item)
}, initialValue)See samber/lo Usage Standards for detailed guidelines and patterns.
Generic functions for type-safe validation operations.
// ValidateWith validates a single value using a validator
// Returns *PackageError on failure
func ValidateWith[T any](ctx context.Context, value T, validator Validator[T]) error// ValidateAll validates multiple values using a validator
func ValidateAll[T any](ctx context.Context, values []T, validator Validator[T]) []error// ComposeValidators creates a validator that runs multiple validators
func ComposeValidators[T any](validators ...Validator[T]) Validator[T]Error Handling: Validation functions return errors using the structured error system.
Errors use NewPackageError or WrapErrorWithContext with typed error context structures.
See Error Handling for details on generic error context helpers.
Generic factory functions for creating type-safe instances.
See NewTag Function for the complete function definition.
// NewBufferPool creates a new buffer pool for the specified type
func NewBufferPool[T any](config *BufferConfig) *BufferPool[T]// NewConfigBuilder creates a new configuration builder
func NewConfigBuilder[T any]() *ConfigBuilder[T]NewTag: Used in FileEntry API - Tag TypeNewBufferPool: Used in Streaming and Buffer ManagementNewConfigBuilder: Used throughout the API for configuration management
This section describes best practices for using generic types and patterns.
- Use descriptive type parameter names:
T,U,Vfor simple cases - Use meaningful names for complex cases:
Key,Value,Element - Use simple, descriptive names without "Generic" prefix:
Option[T],Config[T],WorkerPool[T] - Let the generic syntax
[T any]indicate genericity, not the name itself
- Use the most restrictive constraint that works
- Prefer
comparableoveranywhen possible - Use interface constraints for behavior requirements
Use any (or no constraint) when defining a type parameter constraint and:
- The type parameter doesn't need any specific operations
- The type is only used for storage or passing through
- No comparison, ordering, or specific method calls are needed
- Examples:
Option[T any],Result[T any],Config[T any]
Avoid using any as a concrete parameter type or return type in exported APIs.
Prefer typed structs, dedicated sum types, or generic functions over any inputs.
Use comparable when:
- The type needs to be used in map keys or compared with
==or!= - The type is used in set operations or duplicate detection
- Examples: Map key types, set element types, types used in comparison operations
Note: GetTag[T] uses any constraint, not comparable, because:
- The key is a
stringparameter, not typeT - The implementation only performs type assertion, not comparison
- Using
anyallowsGetTag[any]("key")when the tag type is unknown
Use interface constraints when:
- The type needs to implement specific methods
- Behavior requirements must be enforced at compile time
- Examples:
Strategy[T, U]where T and U might need specific interfaces
Example interface showing how interface constraints can be used:
// Example interface: demonstrates interface constraint pattern
// This is NOT an actual API type - it's shown for illustration only
type Serializable interface {
Serialize() ([]byte, error)
}See Strategy Interface for the full definition.
The actual Strategy interface uses T any (unconstrained).
This example shows how you could add interface constraints if needed:
// Example: A hypothetical constrained version of Strategy
// The actual Strategy interface uses T any (see line 558)
type ConstrainedStrategy[T Serializable, U any] interface {
Process(ctx context.Context, input T) (U, error)
}The canonical Strategy definition uses T any to allow maximum flexibility.
Always document why a specific constraint is chosen:
- Explain the operations that require the constraint
- Provide examples of valid and invalid type instantiations
- Note any performance or type safety implications
Avoid any as a concrete type in exported APIs unless the API is explicitly an untyped boundary.
Prefer compile-time type safety by representing variability with named types.
Common alternatives include:
- Use a typed struct for known fields and options.
- Use a dedicated sum type (tagged union) for "one of N" values.
- Use generic functions to keep callers typed and avoid runtime type assertions.
- Use
[]byteorjson.RawMessagefor opaque payloads when the format is external and untyped.
- Use
Result[T]for operations that may fail - Use
Option[T]for optional values - Always use generic versions for type safety
- When using
samber/lofunctions, ensure error messages provide sufficient context (use explicit loops when index information is needed for debugging)
When returning errors from generic functions, use the generic error context helpers from the structured error system:
AddErrorContext[T any](err error, key string, value T) error: Add type-safe context to errorsGetErrorContext[T any](err error, key string) (T, bool): Retrieve type-safe context from errorsNewPackageError[T any](errType ErrorType, message string, cause error, context T) *PackageError: Create structured errors with typed contextWrapErrorWithContext[T any](err error, errType ErrorType, message string, context T) *PackageError: Wrap errors with typed context
For complete documentation, see Structured Error System.
Validation functions should use typed error context when returning validation errors:
See ValidationErrorContext Structure for the complete structure definition.
Example usage:
// Example: Validation error with typed context
// Note: ValidationErrorContext is defined in api_signatures.md
type ExampleValidationErrorContext struct {
Field string
Value interface{}
Expected string
}
err := NewPackageError(ErrTypeValidation, "validation failed", nil, ExampleValidationErrorContext{
Field: "path",
Value: path,
Expected: "non-empty string",
})Use MapError[T, U] to transform error contexts when needed.
See MapError Function.
Example context type for demonstrating error transformation:
// Example context type: demonstrates error transformation pattern
// This is NOT an actual API type - it's shown for illustration only
type OldContext struct {
Path string
}
// Example context type: demonstrates error transformation pattern
// This is NOT an actual API type - it's shown for illustration only
type NewContext struct {
FilePath string
}
// Example: transform from OldContext to NewContext
transformedErr := MapError(err, func(old OldContext) NewContext {
return NewContext{FilePath: old.Path}
})- Document type parameter constraints
- Provide usage examples
- Document generic function usage patterns and type safety benefits
- Reference
samber/lousage guidelines when documenting collection operations
- Test with multiple type instantiations
- Use type-specific test cases
- Verify compile-time type safety
Type safety is always a priority in the NovusPack API. All functions that can benefit from generics should use generic type parameters. Since this is v1 of the API, there are no backward compatibility concerns.
Use generic versions for:
- Type safety with compile-time checks
- Working with multiple types that share the same operations
- Avoiding type assertions or
interface{}conversions - Performance benefits from avoiding runtime type checks
- Examples:
GetFileEntryTag[T],SetFileEntryTag[T],EncryptFile[T],GetFileEntryTagsByType[T]
Even for single well-known types, prefer generics when they provide type safety benefits. The performance overhead of generics in Go is minimal and the type safety benefits are significant.
Important: Go 1.25 and earlier versions do not support generic methods on non-generic receiver types.
This means that while you can define generic methods on generic types (e.g., func (p *WorkerPool[T]) Start(...)),
you cannot define generic methods on non-generic types (e.g., func (fe *FileEntry) GetTag[T any](...)).
- Tag operations on
FileEntryandPathMetadataEntryare implemented as standalone generic functions rather than methods. - Function names are prefixed with the type name to avoid confusion:
GetFileEntryTag[T],GetPathMetaTag[T], etc. - This is a known limitation documented in the API specifications and does not affect functionality.
// ❌ Not supported in Go 1.25:
func (fe *FileEntry) GetTag[T any](key string) (*Tag[T], error)✅ Supported - standalone generic function pattern:
See GetFileEntryTag Function for the complete function definition.
The function signature follows the pattern: func GetFileEntryTag[T any](fe *FileEntry, key string) (*Tag[T], error)
If future Go versions add support for generic methods on non-generic types, the NovusPack API may be updated to use methods. However, the current function-based approach will remain supported for backward compatibility.
All tag operations use typed tags exclusively. The tag system provides both collection-level and individual tag operations.
See GetFileEntryTags Function for the complete function definition.
See GetFileEntryTagsByType Function for the complete function definition.
See AddFileEntryTags Function for the complete function definition.
See SetFileEntryTags Function for the complete function definition.
See GetFileEntryTag Function for the complete function definition.
See AddFileEntryTag Function for the complete function definition.
See SetFileEntryTag Function for the complete function definition.
See GetPathMetaTags Function for the complete function definition.
See GetPathMetaTagsByType Function for the complete function definition.
See GetPathMetaTag Function for the complete function definition.
See AddPathMetaTag Function for the complete function definition.
See SetPathMetaTag Function for the complete function definition.
- Use
GetFileEntryTags()when you need all tags or are iterating over tags. Returns([]*Tag[any], error), returns*PackageErroron failure (corruption, I/O). - Use
GetFileEntryTagsByType[T]()when you need all tags of a specific type. Returns([]*Tag[T], error), returns*PackageErroron failure (corruption, I/O). - Use
GetFileEntryTag[T]()when you need type-safe access to a specific tag. Returns(*Tag[T], error)where(nil, nil)means the tag was not found (normal case) and(nil, error)means an underlying error occurred (corruption, I/O). If you don't know the tag type, useGetFileEntryTag[any](fe, "key")to retrieve the tag and inspect itsTypefield to determine the actual type. - Use
AddFileEntryTag[T]()when adding a new tag (will error if key already exists). - Use
SetFileEntryTag[T]()when updating an existing tag (will error if key does not exist). - Use
AddFileEntryTags()when adding multiple new tags at once (will error if any key already exists). - Use
SetFileEntryTags()when updating multiple existing tags at once (will error if any key does not exist).