Feature Request: Native Support for Remote Includes (url:, git:, gitraw:)

[genro]

Feature Request: Native Support for Remote Includes (url:, git:, gitraw:)

Repository: genropy/gtext
Type: Enhancement
Priority: Medium

---

Problem Statement

Currently, gtext only supports local file includes via include blocks. To include remote files, users must resort to workarounds like:

```include
cli: curl -s https://raw.githubusercontent.com/org/repo/main/path/to/file.md

This approach has several limitations:
1. **Security complexity**: Requires configuring CLI rules for curl
2. **Platform dependency**: Relies on curl being installed
3. **Verbose**: Adds unnecessary complexity to templates
4. **No caching**: Downloads file every time
5. **Auth handling**: Manual token management for private repos

---

## Proposed Solution

Add native support for three new include types:

### 1. `url:` - Generic HTTP/HTTPS Support

**Syntax:**
```markdown
```include
url: https://example.com/path/to/file.md

**Features:**
- Direct HTTP/HTTPS fetching
- Automatic caching (configurable TTL)
- Support for custom headers (authentication)
- Follows redirects
- Validates SSL certificates

**Configuration:**
```bash
# Allow specific domains
gtext config :url add_rule "https://raw.githubusercontent.com/*" allow --global
gtext config :url add_rule "https://gist.githubusercontent.com/*" allow --global

# Set cache TTL (default: 1 hour)
gtext config :url set cache_ttl 3600

# Add auth headers
gtext config :url set header "Authorization: Bearer ${GITHUB_TOKEN}"

---

2. git: - Git Repository Support

Syntax:

```include
git: https://github.com/org/repo.git/path/to/file.md@main

**Features:**
- Clone/fetch from git repositories
- Sparse checkout (only needed files)
- Branch/tag/commit support via `@ref`
- Local caching with git's built-in mechanisms
- Automatic authentication via git credentials

**Examples:**
```markdown
<!-- Latest from main branch -->
```include
git: https://github.com/org/repo.git/docs/api.md@main

git: https://github.com/org/repo.git/CHANGELOG.md@v1.2.0

git: https://github.com/org/repo.git/README.md@a1b2c3d

**Configuration:**
```bash
# Set cache directory
gtext config :git set cache_dir "~/.cache/gtext/git"

# Allow specific repos
gtext config :git add_rule "https://github.com/myorg/*" allow --global

# Set default branch
gtext config :git set default_branch "main"

---

3. gitraw: - GitHub/GitLab Raw URL Shorthand

Syntax:

```include
gitraw: github.com/org/repo/path/to/file.md@main

OR

gitraw: gitlab.com/org/repo/path/to/file.md@main

**Features:**
- Syntactic sugar for GitHub/GitLab raw URLs
- Automatically constructs correct raw URL
- Supports private repos (uses configured tokens)
- Same caching as `url:`

**Under the hood:**

gitraw: github.com/org/repo/docs/api.md@main
↓
https://raw.githubusercontent.com/org/repo/main/docs/api.md

**Configuration:**
```bash
# Configure GitHub token for private repos
gtext config :gitraw set github_token "${GITHUB_TOKEN}"

# Configure GitLab token
gtext config :gitraw set gitlab_token "${GITLAB_TOKEN}"

# Allow specific orgs
gtext config :gitraw add_rule "github.com/myorg/*" allow --global

---

Use Cases

1. Shared Command Templates (Our Use Case)

Current workaround:

```include
cli: curl -s https://raw.githubusercontent.com/softwellsrl/meta-softwell-developers/main/.claude-templates/commands/checkdoc.md

**With `gitraw:`:**
```markdown
```include
gitraw: github.com/softwellsrl/meta-softwell-developers/.claude-templates/commands/checkdoc.md@main

### 2. Documentation Aggregation

```markdown
# Project Overview

```include
gitraw: github.com/myorg/project-a/README.md@main

API Reference

gitraw: github.com/myorg/project-b/docs/api.md@v2.0.0

Common Guidelines

url: https://internal.company.com/guidelines.md

### 3. Cross-Repository Templates

```markdown
# Standard Header

```include
git: https://github.com/myorg/templates.git/header.md@main

Project-Specific Content

[content here]

Standard Footer

git: https://github.com/myorg/templates.git/footer.md@main

---

## Implementation Details

### Security Model

All remote includes follow gtext's **secure-by-default** policy:

1. **Explicit allowlisting**: Only allowed domains/repos can be accessed
2. **Token isolation**: Auth tokens stored securely, never in templates
3. **Cache validation**: Checksums verify cached content integrity
4. **Sandboxing**: Included content is treated as untrusted until validated

### Caching Strategy

~/.cache/gtext/
├── url/
│ └── [hash-of-url]/
│ ├── content
│ └── metadata.json
├── git/
│ └── [repo-hash]/
│ └── [sparse checkout of needed files]
└── gitraw/
└── [symlink to url/ for reuse]

**Cache invalidation:**
- `url:` - TTL-based (configurable, default 1 hour)
- `git:` - On branch/tag change or manual `gtext cache clear`
- `gitraw:` - Same as `url:`

### API Design

```python
# New extension modules
class UrlIncludeExtension(Extension):
    def include(self, url: str, headers: dict = None) -> str:
        # Fetch, cache, return content
        pass

class GitIncludeExtension(Extension):
    def include(self, git_url: str, ref: str = "main") -> str:
        # Clone/fetch, checkout ref, return content
        pass

class GitRawIncludeExtension(Extension):
    def include(self, gitraw_url: str, ref: str = "main") -> str:
        # Convert to raw URL, delegate to UrlIncludeExtension
        pass

---

Breaking Changes

None - This is purely additive functionality.

---

Alternative Approaches Considered

1. Keep cli: curl approach

Rejected: Too verbose, platform-dependent, manual auth

2. Add only url:

Rejected: Doesn't leverage git's auth and caching for repos

3. Add only git:

Rejected: Overkill for single-file includes, slower for simple cases

---

Acceptance Criteria

• url: includes work with public HTTPS URLs
• url: respects security allowlist rules
• url: caches content with configurable TTL
• url: supports custom headers for authentication
• git: clones/fetches repos and includes files
• git: supports branch/tag/commit refs via @
• git: uses sparse checkout for efficiency
• git: respects git's credential system
• gitraw: converts to proper GitHub raw URLs
• gitraw: converts to proper GitLab raw URLs
• gitraw: supports private repos via tokens
• All three types respect security rules
• Cache mechanism works correctly
• Documentation updated with examples
• Tests cover all scenarios

---

Benefits

1. Cleaner templates: No more cli: curl workarounds
2. Better security: Built-in auth and allowlisting
3. Performance: Intelligent caching
4. Cross-platform: No dependency on external tools
5. Team collaboration: Easy sharing of templates via repos
6. Version control: Pin to specific commits/tags

---

Timeline Suggestion

• Phase 1 (MVP): url: support only
• Phase 2: Add gitraw: (builds on url:)
• Phase 3: Add full git: support

---

Requested by: Softwell Team
Context: Building shared Claude Code command templates across organization