Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,9 @@ These are absolute rules — never violate them:
- **Terminal masking sync block buffering** (experimental): Shielded Terminal buffers PTY output by detecting DEC 2026 sync block markers (`\x1b[?2026h`/`\x1b[?2026l`). Complete sync blocks are masked atomically. Non-sync data uses 30ms timeout buffer. This matches claude-chill's approach.
- **Terminal masking ANSI-aware matching**: `maskTerminalOutput()` strips ANSI escape codes AND all whitespace (spaces, tabs, newlines) before regex matching. Ink word-wraps long keys with `\r\n` + indentation; stripping all whitespace allows regex to match keys across visual line breaks. Structural characters (ANSI + whitespace) within matched ranges are preserved in output via `extractStructural()`.
- **Terminal masking node-pty loading**: Triple fallback: (1) `require('node-pty')`, (2) `require(vscode.env.appRoot + '/node_modules.asar.unpacked/node-pty')`, (3) `require(vscode.env.appRoot + '/node_modules/node-pty')`. Falls back to `child_process.spawn` line-mode terminal.
- **System-wide masking** (experimental): Uses `AXObserver` + `kAXFocusedUIElementChangedNotification` / `kAXValueChangedNotification` to monitor focused text elements across all apps. `AXBoundsForRange` provides precise multi-line text coordinates. NSPanel overlay with `ignoresMouseEvents = true` for click-through. Settings toggle (`systemWideMasking` in UserDefaults), only active when both setting AND Demo Mode are ON.
- **System-wide masking performance**: 10ms initial scan delay, 30ms debounce for changes, 500ms polling fallback. Actual processing 2-6ms. Immediate scan on app switch. Text cache (`lastScannedText`) prevents redundant overlay updates.
- **System-wide masking coordinate conversion**: AX API uses top-left origin, AppKit uses bottom-left. Convert via `primaryScreenHeight - axY - height`. Multi-screen: use primary screen height as reference.

## Documentation

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,8 +169,8 @@ See the [docs/](docs/) directory for detailed specifications:
- [x] Linked Key Groups (sequential paste with ⌘V→Tab automation, Settings UI CRUD, pre-fetch Keychain)
- [x] Clipboard Capture (⌃⌥⌘V hotkey, 22 built-in patterns, 3-tier confidence routing, whitespace-tolerant)
- [x] 🧪 Terminal masking — Shielded Terminal (node-pty proxy, DEC 2026 sync block buffering, ANSI-aware masking, Claude Code compatible)
- [x] 🧪 System-wide masking — Accessibility overlay (AXObserver + AXBoundsForRange, NSPanel click-through overlay, Settings toggle)
- [ ] API Key rotation & deployment sync
- [ ] System-wide masking (Accessibility API)

## Contributing

Expand Down
1 change: 1 addition & 0 deletions docs/01-product-spec/implementation-status.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
| ~~`⌃⌥⌘V` capture clipboard~~ | Spec §4.4 | ~~中~~ | ✅ HotkeyManager → ClipboardEngine.detectKeysInClipboard() → 內建 22 種 pattern 偵測 → confidence 三階路由 → VaultManager 存儲 |
| ~~Linked Key Groups (sequential paste)~~ | Spec §6.3 | ~~中~~ | ✅ `LinkedGroup`/`GroupEntry`/`SequentialPasteEngine` 完成、Settings UI 群組管理(CRUD)、`request_paste_group` IPC handler |
| ~~Terminal masking (Shielded Terminal)~~ | Spec §3.2 | ~~中~~ | 🧪 實驗性。node-pty proxy + DEC 2026 sync block buffering + ANSI-aware masking。Known limitation: Rewind 確認頁部分洩漏 |
| ~~System-wide masking (Accessibility overlay)~~ | Spec §3.3 | ~~中~~ | 🧪 實驗性。AXObserver + AXBoundsForRange + NSPanel overlay。Settings 可開關,Demo Mode 附加功能。Known limitation: 僅掃描 focused element、copy/paste 不安全 |
| Shortcut conflict detection | Spec §4.4 | 低 | |
| Import / Export vault | Spec §9.1 | 低 | |

Expand Down
136 changes: 136 additions & 0 deletions docs/14-system-wide-masking-advanced.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
# System-Wide Masking:進階安全方案(社群貢獻方向)

## 現況

目前的 System-wide masking 使用 **Accessibility overlay**(`AXObserver` + `NSPanel`),是視覺遮蔽方案:
- ✅ 螢幕上看不到 key
- ❌ `⌘C` 複製仍可取得原始 key
- ❌ 螢幕錄影軟體可能錄到 overlay 下方的原始畫面(取決於 window level 順序)

## 更安全的方向:ScreenCaptureKit + Virtual Camera

最安全的方案是在**畫面輸出層**攔截,讓原始 key **從未出現在任何輸出中**。

### 概念架構

```
[macOS 螢幕]
↓ ScreenCaptureKit
[擷取畫面 frame (CGImage/IOSurface)]
[OCR 偵測 API key 位置] ← 可複用現有的 pattern matching
[在 frame 上繪製遮蔽區塊]
[輸出到 Virtual Camera / 螢幕分享]
```

### 適用場景

| 場景 | overlay 方案 | ScreenCaptureKit 方案 |
|------|------------|---------------------|
| OBS 直播 | ⚠️ overlay 可能不在錄製範圍 | ✅ 輸出已遮蔽 |
| Google Meet 螢幕分享 | ⚠️ 取決於 window level | ✅ virtual camera 輸出已遮蔽 |
| 螢幕錄影 | ⚠️ 同上 | ✅ 錄到的就是遮蔽後的 |
| Copy/Paste | ❌ 原始 key 仍可複製 | ❌ 同樣無法防止(不在畫面層) |

### 技術元件

#### 1. ScreenCaptureKit 擷取

```swift
import ScreenCaptureKit

// 取得可擷取的螢幕
let content = try await SCShareableContent.current
let display = content.displays.first!

// 建立 filter(擷取整個螢幕)
let filter = SCContentFilter(display: display, excludingWindows: [])

// 建立串流
let config = SCStreamConfiguration()
config.width = display.width
config.height = display.height
config.pixelFormat = kCVPixelFormatType_32BGRA

let stream = SCStream(filter: filter, configuration: config, delegate: self)
try stream.addStreamOutput(self, type: .screen, sampleHandlerQueue: .global())
try await stream.startCapture()
```

#### 2. OCR 偵測 key 位置

兩種方案:

**方案 A:Vision framework(Apple 原生 OCR)**
```swift
import Vision

func detectKeys(in image: CGImage) -> [(String, CGRect)] {
let request = VNRecognizeTextRequest { request, error in
guard let results = request.results as? [VNRecognizedTextObservation] else { return }
for observation in results {
let text = observation.topCandidates(1).first?.string ?? ""
// 複用 MaskingCoordinator 的 pattern matching
let matches = maskingCoordinator.shouldMask(text: text)
// observation.boundingBox → 螢幕座標
}
}
request.recognitionLevel = .fast // 即時處理需要 fast mode
let handler = VNImageRequestHandler(cgImage: image)
try? handler.perform([request])
}
```

**方案 B:已知座標直接遮蔽(搭配 AX API)**

不用 OCR — 複用現有 `SystemMaskingService` 的 `AXBoundsForRange` 座標,直接在擷取的 frame 上繪製遮蔽區塊。這比 OCR 快得多。

```swift
// 從 SystemMaskingService 取得已偵測的 key 座標
let keyRects = systemMaskingService.getActiveOverlayRects()

// 在 frame 上繪製遮蔽
let context = CGContext(data: ..., width: ..., height: ...)
for rect in keyRects {
context.setFillColor(CGColor.white)
context.fill(rect)
// 可選:繪製 masked text
}
```

#### 3. Virtual Camera 輸出

使用 [CoreMediaIO DAL Plugin](https://developer.apple.com/documentation/coremediaio) 或 [OBS Virtual Camera](https://obsproject.com/) 將處理後的 frame 作為虛擬攝影機輸出。

第三方選項:
- [mac-virtual-camera](https://github.com/pjb/mac-virtual-camera) — Swift 實作
- OBS Studio 的 Virtual Camera API

### 已踩過的坑(省下你的時間)

從 overlay 方案開發中學到的經驗:

1. **AXBoundsForRange 是精確的** — 不需要估算字元寬度,直接用 AX API 取得多行文字的精確螢幕座標
2. **座標轉換** — AX 用左上角原點,AppKit/CoreGraphics 用左下角,轉換公式:`appKitY = primaryScreenHeight - axY - height`
3. **多螢幕** — 用 primary screen height 作為轉換基準
4. **效能基準** — Pattern matching 只要 0.1-0.3ms,AX 查詢 + overlay 更新 2-6ms。ScreenCaptureKit 方案的瓶頸會在 OCR(如果用的話)
5. **30ms debounce 足夠** — 人眼感知閃爍的閾值約 50ms,30ms debounce 夠快
6. **App 切換要立即掃描** — 不能等 debounce,否則切回時 key 會閃現 500ms+
7. **Overlay ID 要唯一** — 同一個 keyId 的多個出現需要不同的 overlay ID(用 keyId + match index)
8. **NSHostingView 要重用** — 每次 `new NSHostingView` 會造成 view 累積

### 建議實作順序

1. **先做 OBS 插件**(最簡單)— 寫一個 OBS Source Plugin,從 DemoSafe Core 取得 key 座標(via IPC),在 OBS 的畫面上繪製遮蔽區塊
2. **再做 Virtual Camera**(中等)— 用 ScreenCaptureKit 擷取 + AX 座標繪製遮蔽 + DAL Plugin 輸出
3. **最後做 OCR**(最複雜)— 如果 AX API 座標不可用(某些 app 不支援),用 Vision framework OCR 作為 fallback

### 相關檔案

| 檔案 | 可複用內容 |
|------|---------|
| `Services/Accessibility/SystemMaskingService.swift` | AXObserver 設定、focused element 掃描、AXBoundsForRange 座標取得 |
| `Services/Masking/MaskingCoordinator.swift` | Pattern matching 引擎(`shouldMask()`) |
| `Views/Overlay/SystemOverlayController.swift` | 座標轉換 (`convertAXToAppKit`) |
1 change: 1 addition & 0 deletions docs/en/01-product-spec/implementation-status.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
| ~~`⌃⌥⌘V` capture clipboard~~ | Spec §4.4 | ~~Medium~~ | ✅ HotkeyManager → ClipboardEngine.detectKeysInClipboard() → 22 built-in patterns → 3-tier confidence routing → VaultManager store |
| ~~Linked Key Groups (sequential paste)~~ | Spec §6.3 | ~~Medium~~ | ✅ `LinkedGroup`/`GroupEntry`/`SequentialPasteEngine` complete, Settings UI group management (CRUD), `request_paste_group` IPC handler |
| ~~Terminal masking (Shielded Terminal)~~ | Spec §3.2 | ~~Medium~~ | 🧪 Experimental. node-pty proxy + DEC 2026 sync block buffering + ANSI-aware masking. Known limitation: Rewind confirmation page partial leak |
| ~~System-wide masking (Accessibility overlay)~~ | Spec §3.3 | ~~Medium~~ | 🧪 Experimental. AXObserver + AXBoundsForRange + NSPanel overlay. Settings toggle, Demo Mode enhancement. Known limitation: focused element only, copy/paste unsafe |
| Shortcut conflict detection | Spec §4.4 | Low | |
| Import / Export vault | Spec §9.1 | Low | |

Expand Down
121 changes: 121 additions & 0 deletions docs/en/14-system-wide-masking-advanced.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# System-Wide Masking: Advanced Secure Approach (Community Contribution Guide)

## Current State

The current system-wide masking uses **Accessibility overlay** (`AXObserver` + `NSPanel`) — a visual-only approach:
- ✅ Key not visible on screen
- ❌ `⌘C` copy still captures the original key
- ❌ Screen recording software may capture the original beneath the overlay

## More Secure Direction: ScreenCaptureKit + Virtual Camera

The most secure approach intercepts at the **frame output layer**, ensuring the original key **never appears in any output**.

### Architecture

```
[macOS Screen]
↓ ScreenCaptureKit
[Capture frame (CGImage/IOSurface)]
[Detect API key positions] ← Reuse existing pattern matching
[Draw masking blocks on frame]
[Output to Virtual Camera / Screen Share]
```

### Use Case Comparison

| Scenario | Overlay Approach | ScreenCaptureKit Approach |
|----------|-----------------|--------------------------|
| OBS livestream | ⚠️ Overlay may not be in capture | ✅ Output already masked |
| Google Meet screen share | ⚠️ Depends on window level | ✅ Virtual camera output masked |
| Screen recording | ⚠️ Same issue | ✅ Recording captures masked version |
| Copy/Paste | ❌ Original key still copyable | ❌ Same (not in frame layer) |

### Technical Components

#### 1. ScreenCaptureKit Capture

```swift
import ScreenCaptureKit

let content = try await SCShareableContent.current
let display = content.displays.first!
let filter = SCContentFilter(display: display, excludingWindows: [])

let config = SCStreamConfiguration()
config.width = display.width
config.height = display.height
config.pixelFormat = kCVPixelFormatType_32BGRA

let stream = SCStream(filter: filter, configuration: config, delegate: self)
try stream.addStreamOutput(self, type: .screen, sampleHandlerQueue: .global())
try await stream.startCapture()
```

#### 2. Key Position Detection

**Option A: Vision Framework OCR**
```swift
import Vision

func detectKeys(in image: CGImage) -> [(String, CGRect)] {
let request = VNRecognizeTextRequest { request, error in
guard let results = request.results as? [VNRecognizedTextObservation] else { return }
for observation in results {
let text = observation.topCandidates(1).first?.string ?? ""
let matches = maskingCoordinator.shouldMask(text: text)
// observation.boundingBox → screen coordinates
}
}
request.recognitionLevel = .fast // Real-time needs fast mode
}
```

**Option B: AX API Coordinates (Recommended — No OCR needed)**

Reuse existing `SystemMaskingService`'s `AXBoundsForRange` coordinates to draw masking blocks directly on captured frames. Much faster than OCR.

```swift
let keyRects = systemMaskingService.getActiveOverlayRects()
let context = CGContext(data: ..., width: ..., height: ...)
for rect in keyRects {
context.setFillColor(CGColor.white)
context.fill(rect)
}
```

#### 3. Virtual Camera Output

Use [CoreMediaIO DAL Plugin](https://developer.apple.com/documentation/coremediaio) or [OBS Virtual Camera](https://obsproject.com/).

Third-party options:
- [mac-virtual-camera](https://github.com/pjb/mac-virtual-camera) — Swift implementation
- OBS Studio Virtual Camera API

### Lessons Learned (From Overlay Implementation)

1. **AXBoundsForRange is precise** — returns exact multi-line text screen coordinates, no estimation needed
2. **Coordinate conversion** — AX uses top-left origin, AppKit/CG uses bottom-left: `appKitY = primaryScreenHeight - axY - height`
3. **Multi-monitor** — use primary screen height as conversion reference
4. **Performance baseline** — pattern matching: 0.1-0.3ms, AX query + overlay: 2-6ms. ScreenCaptureKit bottleneck would be OCR (if used)
5. **30ms debounce is sufficient** — human flicker perception threshold is ~50ms
6. **Immediate scan on app switch** — don't wait for debounce, or key flashes for 500ms+
7. **Unique overlay IDs** — same keyId with multiple occurrences needs distinct IDs (keyId + match index)
8. **Reuse NSHostingView** — creating new views on every update causes accumulation

### Suggested Implementation Order

1. **OBS Plugin first** (easiest) — write an OBS Source Plugin that gets key coordinates from DemoSafe Core via IPC, draw masking on OBS scene
2. **Virtual Camera next** (moderate) — ScreenCaptureKit capture + AX coordinates + DAL Plugin output
3. **OCR last** (complex) — Vision framework OCR as fallback when AX API coordinates unavailable

### Reusable Files

| File | Reusable Content |
|------|-----------------|
| `Services/Accessibility/SystemMaskingService.swift` | AXObserver setup, focused element scanning, AXBoundsForRange |
| `Services/Masking/MaskingCoordinator.swift` | Pattern matching engine (`shouldMask()`) |
| `Views/Overlay/SystemOverlayController.swift` | Coordinate conversion (`convertAXToAppKit`) |
18 changes: 16 additions & 2 deletions packages/swift-core/DemoSafe/App/AppState.swift
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ final class AppState: ObservableObject {
let sequentialPasteEngine: SequentialPasteEngine
let toolboxState: ToolboxState
let toolboxController: FloatingToolboxController
let systemOverlayController: SystemOverlayController
let systemMaskingService: SystemMaskingService

private var cancellables = Set<AnyCancellable>()

Expand All @@ -34,6 +36,8 @@ final class AppState: ObservableObject {
self.hotkeyManager = HotkeyManager(maskingCoordinator: maskingCoordinator)
self.toolboxState = ToolboxState(vaultManager: vaultManager)
self.toolboxController = FloatingToolboxController()
self.systemOverlayController = SystemOverlayController()
self.systemMaskingService = SystemMaskingService(maskingCoordinator: maskingCoordinator, overlayController: systemOverlayController)

// Sync isDemoMode bidirectionally with MaskingCoordinator
maskingCoordinator.$isDemoMode
Expand Down Expand Up @@ -85,6 +89,14 @@ final class AppState: ObservableObject {
isDemoMode.toggle()
maskingCoordinator.isDemoMode = isDemoMode
maskingCoordinator.broadcastState()

// Sync system-wide masking with Demo Mode (only if enabled in settings)
let systemMaskingEnabled = UserDefaults.standard.bool(forKey: "systemWideMasking")
if isDemoMode && systemMaskingEnabled {
systemMaskingService.start()
} else {
systemMaskingService.stop()
}
}

func switchContext(contextId: UUID) {
Expand Down Expand Up @@ -116,18 +128,20 @@ final class AppState: ObservableObject {
// MARK: - Private — Hotkey Wiring

private func wireHotkeyCallbacks() {
// Toolbox show (hold start)
// Toolbox show (hold start) — also enables peek mode for system overlays
hotkeyManager.onToolboxShow = { [weak self] in
guard let self else { return }
self.toolboxState.reset()
self.toolboxState.isVisible = true
self.systemMaskingService.setPeekMode(true)
let mouseLocation = NSEvent.mouseLocation
self.toolboxController.show(near: mouseLocation)
}

// Toolbox release (hold end)
// Toolbox release (hold end) — also disables peek mode
hotkeyManager.onToolboxRelease = { [weak self] in
guard let self else { return }
self.systemMaskingService.setPeekMode(false)
self.toolboxState.handleRelease { [weak self] keyId in
self?.copyKey(keyId: keyId)
}
Expand Down
Loading
Loading