Refactoring: Unified Command Syntax and Enhanced Command System

# Command System Refactoring

## Overview

This issue proposes a major refactoring of gtext's command syntax to achieve:
- Unified syntax across all contexts (files and CLI)
- All commands as first-class citizens (not "protocols" vs "modifiers")
- Enhanced security with command profiles
- New `self:` protocol for document self-reference
- AI command support with automatic marker management

## 🎯 Core Decisions

### 1. Unified Syntax: `command:`

**Decision:** All commands use `command:` syntax (colon AFTER the name) everywhere.

**In .gtext files:**
```markdown
```include
cli: date
static: file.txt
tldr: render: static: doc.md
```
```

**In CLI commands:**
```bash
gtext config cli: add_rule "date" allow
gtext config static: add_rule "*.md" allow
gtext config tldr: add_rule "*" deny
```

**Spacing:** Only AFTER colons (`cli: arg` not `cli : arg`)

---

### 2. All Are "Commands" (Semantic Shift)

**Current:** "Protocols" (cli, static, glob) vs "Modifiers" (expand, tldr, translate)

**New:** All are composable commands at the same level.

**Command types:**

**Source commands (protocols):**
- `cli:` - execute shell command
- `static:` - read file
- `glob:` - pattern matching files
- `self:` - (NEW) reference to current document

**Transformation commands:**
- `render:` - (RENAMED from `expand:`) recursively process includes
- `tldr:` - AI summarization
- `translate[lang]:` - AI translation

---

### 3. `static:` Mandatory (No Fallback)

**Decision:** Remove implicit `static:` fallback.

**Before (implicit):**
```markdown
```include
file.txt          # Treated as static:
```
```

**After (explicit):**
```markdown
```include
file.txt          # ❌ ERROR
static: file.txt  # ✅ OK
```
```

**Reason:** Explicit syntax, no ambiguity, more secure.

---

### 4. Command Pipeline (Composable)

**Decision:** Commands are composable in pipelines.

**Syntax:** `cmd1: cmd2: cmd3: arg`

**Execution:** Right to left (arg → cmd3 → cmd2 → cmd1)

**Examples:**
```markdown
```include
translate[it]: render: static: doc.md.gtext
tldr: self: *
translate[en]: cli: python gen.py
```
```

**Pipeline execution:**
1. `static: doc.md.gtext` → reads file
2. `render:` → processes includes in file
3. `translate[it]:` → translates result to Italian

---

### 5. `self:` Protocol (NEW)

**Decision:** Document itself is a protocol.

**Semantics:** `self` ALWAYS passes the entire document to the command. The text after `self` is incorporated into the AI prompt, NOT used as a filter.

**Examples:**
```markdown
```include
tldr: self
```
→ Prompt: "Summarize this document"
→ Input: entire document

```include
tldr: self condizioni generali e clausole di rescissione
```
→ Prompt: "Extract and summarize 'condizioni generali' and 'clausole di rescissione', dividing by section"
→ Input: entire document

```include
translate[it]: self introduzione
```
→ Prompt: "Translate the 'introduzione' section to Italian"
→ Input: entire document
```

**Implementation:** Deferred queue (see Decision 9)

---

### 6. Technical Markers (Internal Only)

**Decision:** AI commands wrap output in technical markers that are automatically removed at the end of rendering.

**Purpose:**
- Prevent recursive summarization (self: skips marked blocks)
- Track AI-generated content
- No configuration needed

**Marker format (internal, not visible to user):**
```markdown

AI summary...



Translation...

```

**Rendering flow:**
1. Render document with AI commands → wrap output with markers
2. `self:` commands skip marked blocks (avoid recursion)
3. Final cleanup step → remove ALL markers → clean output

**Result:** User never sees markers, output is always clean.

---

### 7. `render:` Behavior

**Decision:** `render:` pass-through silently if no includes found.

**Behavior:**
- If content has ```include` blocks → process them recursively
- If no includes → pass content as-is (no warning)

**Examples:**
```markdown
render: static: template.gtext  # Has includes → processes them
render: static: plain.txt       # No includes → passes through
```

---

### 8. Multi-Profile Security System

**Decision:** Configuration has multiple named profiles, one active as "current".

**Config structure:**
```json
{
  "current": "paranoid",
  "profiles": {
    "paranoid": {
      "cli:": {"rules": []},
      "static:": {"rules": []},
      "glob:": {"rules": []},
      "tldr:": {"rules": []},
      "translate:": {"rules": []},
      "render:": {"rules": [{"pattern": "*", "action": "allow"}]},
      "self:": {"rules": [{"pattern": "*", "action": "allow"}]}
    },
    "default": {
      "cli:": {"rules": []},
      "static:": {"rules": []},
      "glob:": {"rules": []},
      "tldr:": {"rules": [{"pattern": "*", "action": "allow"}]},
      "translate:": {"rules": [{"pattern": "*", "action": "allow"}]},
      "render:": {"rules": [{"pattern": "*", "action": "allow"}]},
      "self:": {"rules": [{"pattern": "*", "action": "allow"}]}
    },
    "norules": {
      "cli:": {"rules": [{"pattern": "*", "action": "allow"}]},
      "static:": {"rules": [{"pattern": "*", "action": "allow"}]},
      "glob:": {"rules": [{"pattern": "*", "action": "allow"}]},
      "tldr:": {"rules": [{"pattern": "*", "action": "allow"}]},
      "translate:": {"rules": [{"pattern": "*", "action": "allow"}]},
      "render:": {"rules": [{"pattern": "*", "action": "allow"}]},
      "self:": {"rules": [{"pattern": "*", "action": "allow"}]}
    }
  }
}
```

**Predefined profiles:**
- `paranoid` - Everything blocked except render/self (DEFAULT at first install)
- `default` - AI allowed, filesystem/shell blocked
- `norules` - Everything allowed (testing/dev)

**CLI commands:**
```bash
# Profile management
gtext config profile list
gtext config profile select paranoid
gtext config profile create my-dev
gtext config profile show
gtext config profile show paranoid

# Operations on current profile
gtext config cli: add_rule "date" allow
gtext config show
```

**User can edit predefined profiles** - they're just starting points.

---

### 9. Deferred Queue for `self:`

**Decision:** Use deferred queue for `self:` processing.

**Implementation:**
```
During render:
  - Process all commands EXCEPT self:
  - Collect all self: commands in a queue with position markers

After render complete:
  - Process deferred queue with full document
  - Replace position markers with AI output
  - Apply marker cleanup
```

**Advantages:**
- `self:` can appear anywhere in document
- Always sees complete rendered document
- No recursive issues with markers

---

## 📋 Implementation Plan

### Phase 1: Parser Refactoring (include.py)
- Remove implicit `static:` fallback
- Change parsing from `:modifier:protocol:` to `command:command:protocol:`
- Add support for spaces after `:` (lstrip)
- Implement explicit syntax validation
- Rename "modifier" → "command" in code

### Phase 2: `self:` Protocol (NEW)
- Add `self` to PROTOCOLS
- Implement `_handle_self(selector, base_dir, context)`
- Implement deferred queue system
- Text after `self` goes into AI prompt

### Phase 3: Marker System (NEW)
- Create `MarkerManager` class
- Technical marker generation with UUID
- Wrap AI command output automatically
- Skip marked blocks in `self:`
- Final cleanup step to remove markers

### Phase 4: Multi-Profile System
- Extend `config.py` for profiles
- Implement profile management commands
- Create predefined profiles (paranoid, default, norules)
- Set `paranoid` as initial profile

### Phase 5: CLI Refactoring (cli.py)
- Change argparse from `:cli` to `cli:`
- Update all config commands
- Update help text

### Phase 6: Security Extension
- Extend `is_command_allowed` for all commands
- Add pipeline validation (check each command)
- Update config for new commands

### Phase 7: Documentation
- Update SECURITY.md with new syntax
- Update QUICKSTART-SECURITY.md
- Update README.md with examples
- Update docstrings

### Phase 8: Tests
- Update ~40-50 tests for explicit `static:`
- Test `self:` protocol
- Test pipeline composition
- Test marker system
- Test profile management
- Test security for AI commands

---

## 🎯 Breaking Changes

**This is an alpha release - no users affected.**

**Changes:**
1. CLI syntax: `:cli` → `cli:`
2. File syntax: implicit `static:` removed
3. `expand:` → `render:` (rename)

**Migration:** Not needed (alpha phase)

---

## 📊 Files Affected

**Major changes:**
- `gtext/extensions/include.py` - Complete parser refactoring
- `gtext/config.py` - Profile system
- `gtext/cli.py` - CLI syntax change
- `gtext/markers.py` - NEW module

**Minor changes:**
- `tests/test_*.py` - 40-50 tests updated
- Documentation files

**Estimated effort:** 2-3 days

---

## ✅ Success Criteria

- [ ] All commands use `command:` syntax (files and CLI)
- [ ] `static:` mandatory (no fallback)
- [ ] Pipeline commands work (`cmd1: cmd2: arg`)
- [ ] `self:` protocol functional with deferred queue
- [ ] Markers wrap AI output and are removed at end
- [ ] Profile system works (paranoid default)
- [ ] All tests pass (171+ tests)
- [ ] Documentation updated
- [ ] No regression in existing features

---

**Related:** #4 (Security system)
**Depends on:** PR #5 merged to main

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactoring: Unified Command Syntax and Enhanced Command System #6

Command System Refactoring

Overview

🎯 Core Decisions

1. Unified Syntax: `command:`

2. All Are "Commands" (Semantic Shift)

3. `static:` Mandatory (No Fallback)

7. `render:` Behavior

8. Multi-Profile Security System

9. Deferred Queue for `self:`

📋 Implementation Plan

Phase 1: Parser Refactoring (include.py)

Phase 2: `self:` Protocol (NEW)

Phase 3: Marker System (NEW)

Phase 4: Multi-Profile System

Phase 5: CLI Refactoring (cli.py)

Phase 6: Security Extension

Phase 7: Documentation

Phase 8: Tests

🎯 Breaking Changes

📊 Files Affected

✅ Success Criteria

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Refactoring: Unified Command Syntax and Enhanced Command System #6

Description

Command System Refactoring

Overview

🎯 Core Decisions

1. Unified Syntax: command:

2. All Are "Commands" (Semantic Shift)

3. static: Mandatory (No Fallback)

7. render: Behavior

8. Multi-Profile Security System

9. Deferred Queue for self:

📋 Implementation Plan

Phase 1: Parser Refactoring (include.py)

Phase 2: self: Protocol (NEW)

Phase 3: Marker System (NEW)

Phase 4: Multi-Profile System

Phase 5: CLI Refactoring (cli.py)

Phase 6: Security Extension

Phase 7: Documentation

Phase 8: Tests

🎯 Breaking Changes

📊 Files Affected

✅ Success Criteria

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. Unified Syntax: `command:`

3. `static:` Mandatory (No Fallback)

7. `render:` Behavior

9. Deferred Queue for `self:`

Phase 2: `self:` Protocol (NEW)