feat: add TwelveLabs Pegasus as a video asset-analysis provider#220
Open
mohit-twelvelabs wants to merge 1 commit into
Open
feat: add TwelveLabs Pegasus as a video asset-analysis provider#220mohit-twelvelabs wants to merge 1 commit into
mohit-twelvelabs wants to merge 1 commit into
Conversation
Add TwelveLabs Pegasus as an opt-in VLM provider for analyzing uploaded video footage, alongside the existing DashScope/Qwen provider. Pegasus is a video-native understanding model that reasons over the whole clip (motion, scene changes, action) rather than a few sampled frames. - New PegasusClient handles public URLs and local files (uploaded as a TwelveLabs asset; direct upload capped at 200MB, larger files via URL). - VLM client routes pegasus* models to TwelveLabs, everything else to DashScope; Pegasus is video-only and rejects image input. - Wired into APIAssetAnalysisService.list_models (video-only) and the api_providers config schema + config.example.yaml. - Behavior is unchanged when no twelvelabs key is configured.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
中文摘要
本 PR 新增 TwelveLabs Pegasus 作为可选的视频素材分析(VLM)后端,与现有的 DashScope/Qwen 并列。Pegasus 是视频原生理解模型,会理解整段视频(运动、场景切换、动作),而非仅采样若干帧,因此非常适合在生成脚本/分镜前理解用户上传的视频素材。
twelvelabsAPI Key 时行为不变。可在 https://twelvelabs.io 免费获取 API Key(提供慷慨的免费额度)。
What this adds
This PR adds TwelveLabs Pegasus as an opt-in video asset-analysis (VLM) provider, alongside the existing DashScope/Qwen provider used by
APIAssetAnalysisService. Pegasus is a video-native understanding model — it reasons over the whole clip (motion, scene changes, on-screen action) rather than a handful of sampled frames — which makes it a strong fit for understanding uploaded footage before script/storyboard generation.Concretely:
PegasusClient(vlm_twelvelabs.py) mirrors the existingvlm_dashscope.pyclient style. It accepts public URLs and local files (local files are uploaded as a TwelveLabs asset; direct upload is capped at 200MB, larger footage should be passed as a public URL).VLMclient routespegasus*models to TwelveLabs and everything else to DashScope. Pegasus is video-only and cleanly rejects image input.APIAssetAnalysisService.list_models()(exposed as a video-only option) and theapi_providersconfig schema +config.example.yaml.twelvelabs>=1.2.8(the official SDK) to dependencies.Why it helps this project
Pixelle-Video already supports swapping analysis backends (RunningHub / self-host ComfyUI / direct API VLM). For users who feed real video clips into the asset-based pipeline, a video-native model produces noticeably more grounded descriptions than frame-sampling VLMs, which improves downstream script and storyboard quality. It slots into the existing provider/registry pattern, so it's just one more option in the dropdown.
Opt-in & non-breaking
If no
twelvelabsAPI key is configured, nothing changes — the provider is simply not listed. No defaults are altered.How it was tested
PegasusClientpath (local file upload → asset-ready poll →analyze) returns a correct text description; I also confirmed the API contract directly.TWELVELABS_API_KEYis set) exercises the real upload→analyze path. Note: the repo's.gitignoreexcludestest_*.py, so the test file is not committed — happy to add it under a different path/name if you'd like it tracked.ruff checkpasses clean on the new files.Possible follow-up
TwelveLabs also offers Marengo multimodal embeddings (512-dim), which would enable footage retrieval/selection (ranking uploaded clips against a query) — a natural next step if you'd like a selection backend; happy to follow up in a separate PR.
You can grab a free API key at https://twelvelabs.io — there's a generous free tier.
Note: TwelveLabs employee contribution. If AIDC-AI requires a CLA or maintainer sign-off for external contributions, just let me know and I'll complete it.