Skip to content

API.md

Latest commit

 

History

History
1180 lines (984 loc) · 29.4 KB

API.md

File metadata and controls

1180 lines (984 loc) · 29.4 KB
Raw

Kohaku Hub API Documentation

Last Updated: January 2025

This document explains how Kohaku Hub's API works, the data flow, and all available endpoints.

System Architecture

graph TB
    subgraph Client["Client Layer"]
        CLT["Client<br/>(huggingface_hub, git, browser)"]
    end

    subgraph Entry["Entry Point"]
        NGX["Nginx (Port 28080)<br/>- Serves static files<br/>- Reverse proxy"]
    end

    subgraph App["Application Layer"]
        API["FastAPI (Port 48888)<br/>- Auth & Permissions<br/>- HF-compatible API<br/>- Git Smart HTTP"]
    end

    subgraph Storage["Storage Backend"]
        LFS["LakeFS<br/>- Git-like versioning<br/>- Branch management<br/>- Commit history"]
        DB["PostgreSQL/SQLite<br/>- User data<br/>- Metadata<br/>- Deduplication<br/>- Synchronous with db.atomic()"]
        S3["MinIO/S3<br/>- Object storage<br/>- LFS files<br/>- Presigned URLs"]
    end

    CLT -->|HTTP/Git/LFS| NGX
    NGX -->|Static files| CLT
    NGX -->|/api, /org, resolve| API
    API -->|REST API async| LFS
    API -->|Sync queries with db.atomic| DB
    API -->|Async| S3
    LFS -->|Stores objects| S3
Loading

Core Concepts

File Size Thresholds

graph TD
    Start[File Upload] --> Check{File size > 10MB?}
    Check -->|No| Regular[Regular Mode]
    Check -->|Yes| LFS[LFS Mode]
    Regular --> Base64[Base64 in commit payload]
    LFS --> Presigned[S3 presigned URL]
    Base64 --> FastAPI[FastAPI processes]
    Presigned --> Direct[Direct S3 upload]
    FastAPI --> LakeFS1[LakeFS stores object]
    Direct --> Link[FastAPI links S3 object]
    Link --> LakeFS2[LakeFS commit with physical address]
Loading

Note: The LFS threshold is configurable via KOHAKU_HUB_LFS_THRESHOLD_BYTES (default: 10MB = 10,000,000 bytes). Can also be set per-repository.

Storage Layout

S3 Bucket Structure:

s3://hub-storage/
  │
  ├── hf-model-org-repo/        ← LakeFS managed repository
  │   └── main/                 ← Branch
  │       ├── config.json
  │       └── model.safetensors
  │
  └── lfs/                      ← LFS objects (content-addressable)
      └── ab/                   ← First 2 chars of SHA256
          └── cd/               ← Next 2 chars
              └── abcd1234...   ← Full SHA256 hash

Upload Workflow

Overview

sequenceDiagram
    participant Client
    participant API as FastAPI
    participant LakeFS
    participant S3

    Note over Client,S3: Phase 1: Preupload Check
    Client->>API: POST /preupload (file hashes & sizes)
    API->>API: Check DB for existing SHA256
    API-->>Client: Upload mode (regular/lfs) & dedup info

    alt Small Files (<10MB)
        Note over Client,S3: Phase 2a: Regular Upload
        Client->>API: POST /commit (base64 content)
        API->>LakeFS: Upload object
        LakeFS->>S3: Store object
    else Large Files (>=10MB)
        Note over Client,S3: Phase 2b: LFS Upload
        Client->>API: POST /info/lfs/objects/batch
        API->>S3: Generate presigned URL
        API-->>Client: Presigned URL
        Client->>S3: PUT file (direct upload)
        Client->>API: POST /commit (lfsFile entry)
        API->>LakeFS: Link physical address
    end

    Note over Client,S3: Phase 3: Commit
    API->>LakeFS: Commit with message
    LakeFS-->>API: Commit ID
    API-->>Client: Commit URL & OID
Loading

Step 1: Preupload Check

Purpose: Determine upload mode and check for duplicates

Endpoint: POST /api/{repo_type}s/{repo_id}/preupload/{revision}

Request:

{
  "files": [
    {
      "path": "config.json",
      "size": 1024,
      "sha256": "abc123..."
    },
    {
      "path": "model.bin",
      "size": 52428800,
      "sha256": "def456..."
    }
  ]
}

Response:

{
  "files": [
    {
      "path": "config.json",
      "uploadMode": "regular",
      "shouldIgnore": false
    },
    {
      "path": "model.bin",
      "uploadMode": "lfs",
      "shouldIgnore": true    // Already exists!
    }
  ]
}

Step 2: Commit

Purpose: Atomically commit all changes to the repository

Endpoint: POST /api/{repo_type}s/{repo_id}/commit/{revision}

Format: NDJSON (Newline-Delimited JSON)

Example Payload:

{"key":"header","value":{"summary":"Add model files","description":"Initial upload"}}
{"key":"file","value":{"path":"config.json","content":"eyJtb2RlbCI6...","encoding":"base64"}}
{"key":"lfsFile","value":{"path":"model.bin","algo":"sha256","oid":"abc123...","size":52428800}}
{"key":"deletedFile","value":{"path":"old_config.json"}}

Operation Types:

Key Description Usage
header Commit metadata Required, must be first line
file Small file (inline base64) For files ≤ 10MB
lfsFile Large file (LFS reference) For files > 10MB, already uploaded to S3
deletedFile Delete a single file Remove file from repo
deletedFolder Delete folder recursively Remove all files in folder
copyFile Copy file within repo Duplicate file (deduplication-aware)

Download Workflow

sequenceDiagram
    participant Client
    participant API as FastAPI
    participant LakeFS
    participant S3

    Note over Client,S3: Optional: HEAD request for metadata
    Client->>API: HEAD /resolve/{revision}/{filename}
    API->>LakeFS: Stat object
    LakeFS-->>API: Object metadata (SHA256, size)
    API-->>Client: Headers (ETag, Content-Length, X-Repo-Commit)

    Note over Client,S3: Download: GET request
    Client->>API: GET /resolve/{revision}/{filename}
    API->>LakeFS: Get object metadata
    API->>S3: Generate presigned URL
    API-->>Client: 302 Redirect (presigned URL)
    Client->>S3: Direct download
    S3-->>Client: File content

    Note over Client: No proxy - direct S3 download
Loading

Database Schema

erDiagram
    USER ||--o{ REPOSITORY : owns
    USER ||--o{ SESSION : has
    USER ||--o{ TOKEN : has
    USER ||--o{ SSHKEY : has
    USER }o--o{ USER : member_of
    USER ||--o{ REPOSITORY_LIKE : likes
    USER ||--o{ DOWNLOAD_SESSION : downloads
    REPOSITORY ||--o{ FILE : contains
    REPOSITORY ||--o{ COMMIT : has
    REPOSITORY ||--o{ STAGING_UPLOAD : has
    REPOSITORY ||--o{ REPOSITORY_LIKE : liked_by
    REPOSITORY ||--o{ DOWNLOAD_SESSION : tracked
    REPOSITORY ||--o{ DAILY_REPO_STATS : has_stats
    COMMIT ||--o{ LFS_OBJECT_HISTORY : references

    USER {
        int id PK
        string username UK
        string normalized_name UK
        boolean is_org
        string email UK
        string password_hash
        boolean email_verified
        boolean is_active
        bigint private_quota_bytes
        bigint public_quota_bytes
        bigint private_used_bytes
        bigint public_used_bytes
        string full_name
        text bio
        blob avatar
        datetime avatar_updated_at
        datetime created_at
    }

    REPOSITORY {
        int id PK
        string repo_type
        string namespace
        string name
        string full_id
        boolean private
        int owner_id FK
        bigint quota_bytes
        bigint used_bytes
        int lfs_threshold_bytes
        int lfs_keep_versions
        text lfs_suffix_rules
        int downloads
        int likes_count
        datetime created_at
    }

    FILE {
        int id PK
        int repository_id FK
        string path_in_repo
        int size
        string sha256
        boolean lfs
        boolean is_deleted
        int owner_id FK
        datetime created_at
        datetime updated_at
    }

    COMMIT {
        int id PK
        string commit_id
        int repository_id FK
        string repo_type
        string branch
        int author_id FK
        int owner_id FK
        string username
        text message
        text description
        datetime created_at
    }

    TOKEN {
        int id PK
        int user_id FK
        string token_hash UK
        string name
        datetime last_used
        datetime created_at
    }

    SESSION {
        int id PK
        string session_id UK
        int user_id FK
        string secret
        datetime expires_at
        datetime created_at
    }

    SSHKEY {
        int id PK
        int user_id FK
        string key_type
        text public_key
        string fingerprint UK
        string title
        datetime last_used
        datetime created_at
    }

    STAGING_UPLOAD {
        int id PK
        int repository_id FK
        string repo_type
        string revision
        string path_in_repo
        string sha256
        int size
        string upload_id
        string storage_key
        boolean lfs
        int uploader_id FK
        datetime created_at
    }

    LFS_OBJECT_HISTORY {
        int id PK
        int repository_id FK
        string path_in_repo
        string sha256
        int size
        string commit_id
        int file_id FK
        datetime created_at
    }

    REPOSITORY_LIKE {
        int id PK
        int repository_id FK
        int user_id FK
        datetime created_at
    }

    DOWNLOAD_SESSION {
        int id PK
        int repository_id FK
        int user_id FK
        string session_id
        int time_bucket
        int file_count
        string first_file
        datetime first_download_at
        datetime last_download_at
    }

    DAILY_REPO_STATS {
        int id PK
        int repository_id FK
        date date
        int download_sessions
        int authenticated_downloads
        int anonymous_downloads
        int total_files
        datetime created_at
    }
Loading

API Endpoint Summary

Repository Operations

Endpoint Method Auth Description
/api/repos/create POST ✓ Create new repository
/api/repos/delete DELETE ✓ Delete repository
/api/repos/move POST ✓ Move/rename repository
/api/{type}s GET ○ List repositories (respects privacy)
/api/{type}s/{id} GET ○ Get repo info
/api/{type}s/{id}/tree/{rev}/{path} GET ○ List files
/api/{type}s/{id}/revision/{rev} GET ○ Get revision info
/api/{type}s/{id}/paths-info/{rev} POST ○ Get info for specific paths
/api/users/{username}/repos GET ○ List all repos for a user/org (grouped by type)

File Operations

Endpoint Method Auth Description
/api/{type}s/{id}/preupload/{rev} POST ✓ Check before upload
/api/{type}s/{id}/commit/{rev} POST ✓ Atomic commit
/{id}/resolve/{rev}/{file} GET ○ Download file
/{id}/resolve/{rev}/{file} HEAD ○ Get file metadata
/{type}s/{id}/resolve/{rev}/{file} GET ○ Download file (with type)
/{type}s/{id}/resolve/{rev}/{file} HEAD ○ Get file metadata (with type)

LFS Operations

Endpoint Method Auth Description
/{id}.git/info/lfs/objects/batch POST ✓ LFS batch API
/api/{id}.git/info/lfs/verify POST ✓ Verify upload

Commit History

Endpoint Method Auth Description
/{type}s/{namespace}/{name}/commits/{branch} GET ○ List commits on a branch with pagination

Branch and Tag Management

Endpoint Method Auth Description
/{type}s/{namespace}/{name}/branch POST ✓ Create a new branch
/{type}s/{namespace}/{name}/branch/{branch} DELETE ✓ Delete a branch
/{type}s/{namespace}/{name}/tag POST ✓ Create a new tag
/{type}s/{namespace}/{name}/tag/{tag} DELETE ✓ Delete a tag

Settings Management

Endpoint Method Auth Description
/api/users/{username}/settings PUT ✓ Update user settings
/api/organizations/{org_name}/settings PUT ✓ Update organization settings
/{type}s/{namespace}/{name}/settings PUT ✓ Update repository settings (private, gated, LFS settings)
/api/{type}s/{namespace}/{name}/lfs/settings GET ○ Get repository LFS settings

Social Features

Likes:

Endpoint Method Auth Description
/api/{type}s/{namespace}/{name}/like POST ✓ Like a repository
/api/{type}s/{namespace}/{name}/like DELETE ✓ Unlike a repository
/api/{type}s/{namespace}/{name}/like GET ○ Check if current user liked repository
/api/{type}s/{namespace}/{name}/likers GET ○ List users who liked repository
/api/users/{username}/likes GET ○ List repositories user has liked

Statistics & Trending:

Endpoint Method Auth Description
/api/{type}s/{namespace}/{name}/stats GET ○ Get repository statistics (downloads, likes)
/api/{type}s/{namespace}/{name}/stats/recent GET ○ Get recent download statistics (time series)
/api/trending GET ○ Get trending repositories

Avatars:

Endpoint Method Auth Description
/api/users/{username}/avatar POST ✓ Upload user avatar
/api/users/{username}/avatar GET ○ Get user avatar image
/api/users/{username}/avatar DELETE ✓ Delete user avatar
/api/organizations/{org_name}/avatar POST ✓ Upload organization avatar
/api/organizations/{org_name}/avatar GET ○ Get organization avatar image
/api/organizations/{org_name}/avatar DELETE ✓ Delete organization avatar

Quota Management

Endpoint Method Auth Description
/api/quota/{namespace} GET ✓ Get namespace quota information
/api/quota/{namespace} PUT ✓ Set namespace quota
/api/quota/{namespace}/recalculate POST ✓ Recalculate namespace storage usage
/api/quota/{namespace}/public GET ○ Get public quota info (permission-based)
/api/quota/{namespace}/repos GET ✓ List namespace repositories with storage breakdown
/api/quota/repo/{type}/{namespace}/{name} GET ○ Get repository quota information
/api/quota/repo/{type}/{namespace}/{name} PUT ✓ Set repository quota
/api/quota/repo/{type}/{namespace}/{name}/recalculate POST ✓ Recalculate repository storage

Invitations

Endpoint Method Auth Description
/api/invitations/org/{org_name}/create POST ✓ Create organization invitation
/api/invitations/{token} GET ✗ Get invitation details
/api/invitations/{token}/accept POST ✓ Accept invitation
/api/invitations/{token} DELETE ✓ Delete/cancel invitation
/api/invitations/org/{org_name}/list GET ✓ List organization invitations

SSH Keys

Endpoint Method Auth Description
/api/user/keys GET ✓ List user's SSH keys
/api/user/keys POST ✓ Add new SSH key
/api/user/keys/{key_id} GET ✓ Get SSH key details
/api/user/keys/{key_id} DELETE ✓ Delete SSH key

Validation

Endpoint Method Auth Description
/api/validate/check-name POST ✗ Check if username/org/repo name is available
/api/validate-yaml POST ✗ Validate YAML content

Authentication Operations

Endpoint Method Auth Description
/api/auth/register POST ✗ Register new user
/api/auth/login POST ✗ Login and create session
/api/auth/logout POST ✓ Logout and destroy session
/api/auth/verify-email GET ✗ Verify email with token
/api/auth/me GET ✓ Get current user info
/api/auth/tokens GET ✓ List user's API tokens
/api/auth/tokens/create POST ✓ Create new API token
/api/auth/tokens/{token_id} DELETE ✓ Revoke API token

External Token Operations (Fallback System)

Endpoint Method Auth Description
/api/fallback-sources/available GET ✗ List available fallback sources
/api/users/{username}/external-tokens GET ✓ List user's external tokens (masked)
/api/users/{username}/external-tokens POST ✓ Add/update external token
/api/users/{username}/external-tokens/{url} DELETE ✓ Delete external token
/api/users/{username}/external-tokens/bulk PUT ✓ Bulk update external tokens

Organization Operations

Endpoint Method Auth Description
/org/create POST ✓ Create new organization
/org/{org_name} GET ✗ Get organization details
/org/{org_name}/members GET ○ List organization members
/org/{org_name}/members POST ✓ Add member to organization
/org/{org_name}/members/{username} DELETE ✓ Remove member from organization
/org/{org_name}/members/{username} PUT ✓ Update member role
/org/users/{username}/orgs GET ✗ List user's organizations

Git Operations

Endpoint Method Auth Description
/{namespace}/{name}.git/info/refs GET ○ Git service advertisement
/{namespace}/{name}.git/HEAD GET ○ Get HEAD reference
/{namespace}/{name}.git/git-upload-pack POST ○ Clone/fetch/pull
/{namespace}/{name}.git/git-receive-pack POST ✓ Push (in development)

Utility Operations

Endpoint Method Auth Description
/api/whoami-v2 GET ✓ Get detailed current user info
/api/version GET ✗ Get API version information
/health GET ✗ Health check
/ GET ✗ API information

Auth Legend:

  • ✓ = Required
  • ○ = Optional (public repos)
  • ✗ = Not required

New Features Documentation

Repository Likes

Like a repository:

POST /api/models/org/model/like
Authorization: Bearer YOUR_TOKEN

Response:

{
  "success": true,
  "message": "Repository liked successfully",
  "likes_count": 42
}

Check if liked:

GET /api/models/org/model/like

Response:

{
  "liked": true
}

List likers:

GET /api/models/org/model/likers?limit=50

Response:

{
  "likers": [
    {
      "username": "alice",
      "full_name": "Alice Developer"
    }
  ],
  "total": 42
}

Statistics and Trending

Get repository stats:

GET /api/models/org/model/stats

Response:

{
  "downloads": 1234,
  "likes": 42
}

Get recent statistics (time series):

GET /api/models/org/model/stats/recent?days=30

Response:

{
  "stats": [
    {
      "date": "2025-01-15",
      "downloads": 45,
      "authenticated": 30,
      "anonymous": 15,
      "files": 120
    }
  ],
  "period": {
    "start": "2024-12-16",
    "end": "2025-01-15",
    "days": 30
  }
}

Get trending repositories:

GET /api/trending?repo_type=model&days=7&limit=20

Response:

{
  "trending": [
    {
      "id": "org/hot-model",
      "type": "model",
      "downloads": 5000,
      "likes": 200,
      "recent_downloads": 1500,
      "private": false
    }
  ],
  "period": {
    "start": "2025-01-08",
    "end": "2025-01-15",
    "days": 7
  }
}

Avatar Management

Upload avatar:

POST /api/users/alice/avatar
Authorization: Bearer YOUR_TOKEN
Content-Type: multipart/form-data

file: [image binary data]

Features:

  • Accepts JPEG, PNG, WebP, GIF
  • Maximum input size: 10MB
  • Automatically resizes to fit 1024x1024
  • Center crops to square
  • Converts to JPEG format
  • Output quality: 95%

Response:

{
  "success": true,
  "message": "Avatar uploaded successfully",
  "size_bytes": 245678
}

Get avatar:

GET /api/users/alice/avatar

Returns JPEG image with cache headers.

Quota Management

Get quota information:

GET /api/quota/alice
Authorization: Bearer YOUR_TOKEN

Response:

{
  "namespace": "alice",
  "is_organization": false,
  "quota_bytes": 10737418240,
  "used_bytes": 1234567890,
  "available_bytes": 9502850350,
  "percentage_used": 11.5
}

Set quota:

PUT /api/quota/alice
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json

{
  "quota_bytes": 10737418240
}

Repository-specific quota:

GET /api/quota/repo/model/org/my-model

Response:

{
  "repo_id": "org/my-model",
  "repo_type": "model",
  "namespace": "org",
  "quota_bytes": 1073741824,
  "used_bytes": 524288000,
  "available_bytes": 549453824,
  "percentage_used": 48.8,
  "effective_quota_bytes": 1073741824,
  "namespace_quota_bytes": 10737418240,
  "namespace_used_bytes": 5368709120,
  "namespace_available_bytes": 5368709120,
  "is_inheriting": false
}

Storage breakdown for namespace:

GET /api/quota/org/repos
Authorization: Bearer YOUR_TOKEN

Response:

{
  "namespace": "org",
  "is_organization": true,
  "total_repos": 15,
  "repositories": [
    {
      "repo_id": "org/large-model",
      "repo_type": "model",
      "name": "large-model",
      "private": false,
      "quota_bytes": null,
      "used_bytes": 5368709120,
      "percentage_used": 50.0,
      "is_inheriting": true,
      "created_at": "2025-01-01T00:00:00Z"
    }
  ]
}

Invitations

Create organization invitation:

POST /api/invitations/org/my-org/create
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json

{
  "email": "newuser@example.com",
  "role": "member",
  "max_usage": null,
  "expires_days": 7
}

Response:

{
  "success": true,
  "token": "abc123...",
  "invitation_link": "http://hub.example.com/invite/abc123...",
  "expires_at": "2025-01-22T12:00:00Z",
  "max_usage": null,
  "is_reusable": false
}

Reusable invitation (10 uses):

{
  "role": "member",
  "max_usage": 10,
  "expires_days": 30
}

Accept invitation:

POST /api/invitations/{token}/accept
Authorization: Bearer YOUR_TOKEN

SSH Keys

Add SSH key:

POST /api/user/keys
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json

{
  "title": "My Laptop",
  "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIB... user@host"
}

Response:

{
  "id": 42,
  "title": "My Laptop",
  "key_type": "ssh-ed25519",
  "fingerprint": "SHA256:abc123...",
  "created_at": "2025-01-15T12:00:00.000000Z",
  "last_used": null
}

Supported key types:

  • ssh-rsa
  • ssh-dss
  • ecdsa-sha2-nistp256
  • ecdsa-sha2-nistp384
  • ecdsa-sha2-nistp521
  • ssh-ed25519

Name Validation

Check if name is available:

POST /api/validate/check-name
Content-Type: application/json

{
  "name": "my-new-repo",
  "namespace": "org",
  "type": "model"
}

Response (available):

{
  "available": true,
  "normalized_name": "my_new_repo",
  "conflict_with": null,
  "message": "Repository name is available"
}

Response (conflict):

{
  "available": false,
  "normalized_name": "my_new_repo",
  "conflict_with": "org/My-New-Repo",
  "message": "Repository name conflicts with existing repository: My-New-Repo (case-insensitive)"
}

LFS Settings

Get repository LFS settings:

GET /api/models/org/model/lfs/settings

Response:

{
  "lfs_threshold_bytes": 5000000,
  "lfs_threshold_bytes_effective": 5000000,
  "lfs_threshold_bytes_source": "repository",
  "lfs_keep_versions": 10,
  "lfs_keep_versions_effective": 10,
  "lfs_keep_versions_source": "repository",
  "lfs_suffix_rules": [".safetensors", ".bin"],
  "lfs_suffix_rules_effective": [".safetensors", ".bin"],
  "server_defaults": {
    "lfs_threshold_bytes": 10000000,
    "lfs_keep_versions": 5
  }
}

Update repository settings with LFS:

PUT /models/org/model/settings
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json

{
  "lfs_threshold_bytes": 5000000,
  "lfs_keep_versions": 10,
  "lfs_suffix_rules": [".safetensors", ".bin", ".gguf"]
}

Content Deduplication

Kohaku Hub implements content-addressable storage for LFS files:

Same file uploaded to different repos:

Repo A: myorg/model-v1
  └─ model.bin (sha256: abc123...)

Repo B: myorg/model-v2
  └─ model.bin (sha256: abc123...)

S3 Storage:
  └─ lfs/ab/c1/abc123...  ← SINGLE COPY
         ▲          ▲
         │          │
    Repo A      Repo B
    (linked)    (linked)

Benefits:
  - Save storage space
  - Faster uploads (skip if exists)
  - Efficient for model variants

Error Handling

Kohaku Hub uses HuggingFace-compatible error headers:

HTTP Response Headers:
  X-Error-Code: RepoNotFound
  X-Error-Message: Repository 'org/repo' not found

Error Codes:

Code HTTP Status Description
RepoNotFound 404 Repository doesn't exist
RepoExists 400 Repository already exists
RevisionNotFound 404 Branch/commit not found
EntryNotFound 404 File not found
GatedRepo 403 Need permission
BadRequest 400 Invalid request
ServerError 500 Internal error

These error codes are parsed by huggingface_hub client to raise appropriate Python exceptions.

Performance Considerations

Download Tracking

KohakuHub implements smart download tracking:

Session Deduplication:

  • Downloads are grouped into 15-minute sessions
  • Multiple files downloaded in the same session count as 1 download
  • Uses session ID + time bucket for deduplication

Benefits:

  • Accurate download counts (git clone = 1 download, not N file downloads)
  • Trending calculations based on unique sessions
  • Efficient storage (one record per session)

Recommended S3 Providers

Provider Best For Pricing Model Notes
Cloudflare R2 High download Free egress, $0.015/GB storage Best for public datasets
Wasabi Archive/backup $6/TB/month, free egress* *if download < storage
MinIO Self-hosted Free (your hardware/bandwidth) Full control, privacy
AWS S3 Enterprise Pay per GB + egress Most features, expensive egress
Backblaze B2 Budget $6/TB storage, $0.01/GB egress Good for mixed workloads

Recommendation for KohakuHub:

  • Development: MinIO (included in docker-compose)
  • Public Hub: Cloudflare R2 (free egress saves costs)
  • Private/Enterprise: Self-hosted MinIO or AWS S3 with VPC endpoints

External Token API (User Fallback Tokens)

Users can configure their own tokens for external fallback sources to access private repositories.

List Available Sources

Public endpoint - no authentication required

GET /api/fallback-sources/available

Response:

[
  {
    "url": "https://huggingface.co",
    "name": "HuggingFace",
    "source_type": "huggingface",
    "priority": 1
  }
]

List User's External Tokens

GET /api/users/{username}/external-tokens
Authorization: Bearer YOUR_TOKEN

Response (tokens are masked):

[
  {
    "url": "https://huggingface.co",
    "token_preview": "hf_a***",
    "created_at": "2025-01-22T10:30:00Z",
    "updated_at": "2025-01-22T10:30:00Z"
  }
]

Add/Update External Token

POST /api/users/{username}/external-tokens
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json

{
  "url": "https://huggingface.co",
  "token": "hf_abc123xyz"
}

Response:

{
  "success": true,
  "message": "External token saved"
}

Notes:

  • If token exists for this URL, it will be updated
  • Token is encrypted before storage (AES-256)
  • User can only manage their own tokens

Delete External Token

DELETE /api/users/{username}/external-tokens/https%3A%2F%2Fhuggingface.co
Authorization: Bearer YOUR_TOKEN

Response:

{
  "success": true,
  "message": "External token deleted"
}

Note: URL must be URL-encoded in path

Bulk Update External Tokens

Replace all external tokens at once:

PUT /api/users/{username}/external-tokens/bulk
Authorization: Bearer YOUR_TOKEN
Content-Type: application/json

{
  "tokens": [
    {"url": "https://huggingface.co", "token": "hf_abc123"},
    {"url": "https://other-hub.com", "token": "token456"}
  ]
}

Response:

{
  "success": true,
  "message": "Updated 2 external tokens"
}

Notes:

  • Deletes tokens not in the new list
  • Atomic operation (all or nothing)

Using External Tokens in Requests

Authorization Header Format:

Bearer <auth_token>|<url1>,<token1>|<url2>,<token2>...

Examples:

  1. API token + external token:
curl -H "Authorization: Bearer my_api_token|https://huggingface.co,hf_abc123" \
  http://localhost:28080/api/models/org/model
  1. Session auth + external token:
# Frontend automatically sends: "Bearer |https://huggingface.co,hf_abc123"
  1. Anonymous + external token:
curl -H "Authorization: Bearer |https://huggingface.co,hf_abc123" \
  http://localhost:28080/api/models/facebook/gpt2

Token Priority:

  1. Authorization header tokens (highest - per-request override)
  2. Database tokens (medium - user preferences)
  3. Admin tokens (lowest - server defaults)

Configuration:

# Required: Encryption key
export KOHAKU_HUB_DATABASE_KEY="$(openssl rand -hex 32)"

# Optional: Require auth for fallback
export KOHAKU_HUB_FALLBACK_REQUIRE_AUTH=false  # Default: false