Refactor/calypr clients

[matthewpeterkort]

AI generated PR changes summary. Read it over it is quite accurate for what has changed.

Technical Changelog: Local refactor vs Remote develop

This document details the technical differences between your current local branch (refactor/calypr-clients) and the remote develop branch.

1. Architectural Overhaul: Client Modularization

The monolithic 

client package has been decomposed into domain-specific clients managed by a central g3client.

• New g3client Orchestrator:
  • Introduced g3client.Gen3Interface and g3client.Gen3Client.
  • Delegation Pattern: Instead of direct methods (e.g., g3.DeleteRecord), the client now exposes sub-clients:
    • g3.Fence().DeleteRecord(...)
    • g3.Indexd().GetObject(...)
    • g3.Sower().DispatchJob(...)
• Dependency Injection: The Gen3Client now injects RequestInterface, FenceInterface, etc., making unit testing with mocks significantly easier.

2. System-Wide Logging Migration (log/slog)

The entire logging subsystem has been rewritten to use Go's standard structured logging library (log/slog).

• logs/factory.go:
  • Replaced: io.Writer based logging replaced with slog.Handler chains.
  • New: NewProgressHandler wraps the root handler to coordinate with progress bars.
  • Changed: New() now returns *Gen3Logger (wrapping *slog.Logger) instead of the custom *TeeLogger.
• Usage Changes:
  • Calls like logger.Printf replaced with logger.InfoContext(ctx, ...) or logger.WarnContext(ctx, ...).
  • Structured attributes now supported in log messages.

3. Configuration & Authentication Logic

Refined validation logic in conf/validate.go to be more granular and informative.

• Token Validation:
  • IsValid broken down into IsTokenValid and IsCredentialValid.
  • New Logic: IsCredentialValid checks both AccessToken and APIKey independently.
  • Edge Case: Explicitly handles cases where access_token is invalid but api_key is valid (triggering refresh logic elsewhere).
• Feedback: Added warnings via Logger when tokens are nearing expiration (previously just returned a formatted string).

4. Breaking API & Field Renames

Several core data structures have been standardized, causing breaking changes for consumers of these structs.

Upload Package (upload/)

• RetryObject & FileUploadRequestObject:
  • FilePath → Renamed to SourcePath (clarifies intent vs destination).
  • Filename → Renamed to ObjectKey (standard S3/object storage terminology).

Download Package (download/)

• ManifestObject:
  • ObjectID → Renamed to GUID (standard Gen3 terminology).
• downloader.go:
  • AskForConfirmation now returns an error on user abortion instead of calling logger.Fatal directly, allowing better cleanup/exit handling.
  • Renames flag handling: Now warnings are logged via WarnContext instead of standard print.

5. New Features

• Sower Client (sower/): First-class support for dispatching and monitoring Sower jobs (DispatchJob, Status, Output).
• Token Refresh Helper: g3client.EnsureValidCredential added to centrally handle the "check validity -> if invalid but API key good -> refresh -> save" loop.

6. Project Structure Moves

Feature	Old Location	New Location
Configuration	client/conf	conf/
Upload Logic	client/upload	upload/
Download Logic	client/download	download/
Request builder	client/request	request/
Mocks	client/mocks	mocks/

[bwalsh]

Use fmt.Errorf and fmt.Sprintf for better performance and readability.

[bwalsh]

@matthewpeterkort This is a big PR. Trying to understand all of it. Can you give me some starting points? eg New code vs moved code?

[bwalsh]

Summary slog handler

✅ This branch centralizes progress reporting in context.Context and introduces new slog handler plumbing (including a progress-forwarding handler) to integrate log messages with progress callbacks.

issues

❌ Log source attribution is still incorrect for wrapper methods. Failed/Succeeded call their *Context counterparts, which then call logWithSkip(..., 3, ...). That skip value points the source to the wrapper method (or deeper) instead of the original caller, undermining the stated goal of accurate source attribution. Consider increasing the skip for these wrapper paths (e.g., Failed/Succeeded should pass a higher skip, or logWithSkip should accept an extra skip delta for wrappers).

https://github.com/calypr/data-client/blob/refactor/calypr-clients/logs/tee_logger.go#L56-L204

suggestions

❓ The progress handler currently forwards only r.Message (and omits level/attrs). If progress consumers need richer diagnostics, consider including r.Level and relevant attributes in the emitted ProgressEvent payload (or serialize the record).

https://github.com/calypr/data-client/blob/refactor/calypr-clients/logs/handler.go#L27-L41

[bwalsh]

Summary new/updated tests

🔍 What I looked at

download/transfer_test.go — progress callback usage updated to come from context.Context, and fake client updated to satisfy the new Sower() interface requirement.

upload/multipart_test.go — progress callback now passed via context, and OID explicitly set to match new progress plumbing; fake client updated for Sower() interface.

logs/logger_test.go — base logger test updated to use slog.Logger to match the new WithBaseLogger type signature.

⚖️ Findings

✅ No issues found with the new test updates. The changes align with the new context-based progress plumbing and updated logger configuration:

The download progress tests now correctly attach the callback via context, consistent with the production code’s common.WithProgress usage.

The multipart upload test now sets both progress callback and OID in context, matching the new flow used in multipart upload progress events.

Logger tests correctly reflect the new *slog.Logger type for WithBaseLogger.

[matthewpeterkort]

@matthewpeterkort This is a big PR. Trying to understand all of it. Can you give me some starting points? eg New code vs moved code?

the sower client code was taken from Forge and moved in here. The indexd client code was taken from git-drs and moved in here. The fence client code was ripped out of the existing ./upload and ./download business logic directories and rewritten as a dedicated client.

See summary at top