Skip to content

feat: add TwelveLabs Pegasus as a video asset-analysis provider#220

Open
mohit-twelvelabs wants to merge 1 commit into
ATH-MaaS:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration
Open

feat: add TwelveLabs Pegasus as a video asset-analysis provider#220
mohit-twelvelabs wants to merge 1 commit into
ATH-MaaS:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration

Conversation

@mohit-twelvelabs

Copy link
Copy Markdown

Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).

中文摘要

本 PR 新增 TwelveLabs Pegasus 作为可选的视频素材分析(VLM)后端,与现有的 DashScope/Qwen 并列。Pegasus 是视频原生理解模型,会理解整段视频(运动、场景切换、动作),而非仅采样若干帧,因此非常适合在生成脚本/分镜前理解用户上传的视频素材。

  • 完全可选、不改变默认行为:未配置 twelvelabs API Key 时行为不变。
  • 在「素材分析服务」中作为仅视频选项出现(Pegasus 不分析图片)。
  • 已对 TwelveLabs 真实 API 做了端到端验证(上传 → 资产就绪轮询 → 分析返回文本),并附带单元测试。

可在 https://twelvelabs.io 免费获取 API Key(提供慷慨的免费额度)。

What this adds

This PR adds TwelveLabs Pegasus as an opt-in video asset-analysis (VLM) provider, alongside the existing DashScope/Qwen provider used by APIAssetAnalysisService. Pegasus is a video-native understanding model — it reasons over the whole clip (motion, scene changes, on-screen action) rather than a handful of sampled frames — which makes it a strong fit for understanding uploaded footage before script/storyboard generation.

Concretely:

  • PegasusClient (vlm_twelvelabs.py) mirrors the existing vlm_dashscope.py client style. It accepts public URLs and local files (local files are uploaded as a TwelveLabs asset; direct upload is capped at 200MB, larger footage should be passed as a public URL).
  • The unified VLM client routes pegasus* models to TwelveLabs and everything else to DashScope. Pegasus is video-only and cleanly rejects image input.
  • Wired into APIAssetAnalysisService.list_models() (exposed as a video-only option) and the api_providers config schema + config.example.yaml.
  • Added twelvelabs>=1.2.8 (the official SDK) to dependencies.

Why it helps this project

Pixelle-Video already supports swapping analysis backends (RunningHub / self-host ComfyUI / direct API VLM). For users who feed real video clips into the asset-based pipeline, a video-native model produces noticeably more grounded descriptions than frame-sampling VLMs, which improves downstream script and storyboard quality. It slots into the existing provider/registry pattern, so it's just one more option in the dropdown.

Opt-in & non-breaking

If no twelvelabs API key is configured, nothing changes — the provider is simply not listed. No defaults are altered.

How it was tested

  • Live-verified against the real TwelveLabs API in this PR's development: the full PegasusClient path (local file upload → asset-ready poll → analyze) returns a correct text description; I also confirmed the API contract directly.
  • Unit tests cover provider routing, the configured-only listing, the video-only image-rejection guard, and the "requires video" guard (no network). A live test (skipped unless TWELVELABS_API_KEY is set) exercises the real upload→analyze path. Note: the repo's .gitignore excludes test_*.py, so the test file is not committed — happy to add it under a different path/name if you'd like it tracked.
  • ruff check passes clean on the new files.

Possible follow-up

TwelveLabs also offers Marengo multimodal embeddings (512-dim), which would enable footage retrieval/selection (ranking uploaded clips against a query) — a natural next step if you'd like a selection backend; happy to follow up in a separate PR.

You can grab a free API key at https://twelvelabs.io — there's a generous free tier.


Note: TwelveLabs employee contribution. If AIDC-AI requires a CLA or maintainer sign-off for external contributions, just let me know and I'll complete it.

Add TwelveLabs Pegasus as an opt-in VLM provider for analyzing uploaded
video footage, alongside the existing DashScope/Qwen provider. Pegasus is
a video-native understanding model that reasons over the whole clip
(motion, scene changes, action) rather than a few sampled frames.

- New PegasusClient handles public URLs and local files (uploaded as a
  TwelveLabs asset; direct upload capped at 200MB, larger files via URL).
- VLM client routes pegasus* models to TwelveLabs, everything else to
  DashScope; Pegasus is video-only and rejects image input.
- Wired into APIAssetAnalysisService.list_models (video-only) and the
  api_providers config schema + config.example.yaml.
- Behavior is unchanged when no twelvelabs key is configured.
@CLAassistant

CLAassistant commented Jun 27, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants