feat: 新增 Sonilo 供应商 — 按成片生成配乐 / 音效(v2m + video-to-SFX) - #91
Open
cindyxu1030 wants to merge 2 commits into
Open
Conversation
… from the finished cut
Adds Sonilo as an additional audio provider that composes from the rendered
cut instead of a text prompt. Music joins the music plugin as a third
provider (async job: multipart submit → poll /v1/tasks/{id} → presigned
audio URL); SFX joins /generate/sound behind a provider switch. Caps are
enforced locally before upload (music ≤6-min video, SFX ≤3-min). Every
track's license_id is surfaced on the generation result and archived as a
.license.json sidecar beside the audio file. Configured via SONILO_API_KEY
(+ optional SONILO_BASE_URL) with a read-only key probe.
…, music skill submit_music gains provider sonilo with mode v2m (project video sourceAssetId, single optional style prompt ≤500, one result); submit_sound gains provider sonilo (video sourceAssetId, promptless). Capability rows, generate workflow guidance, settings vendor page (music/sonilo) with route option and probe wiring, vendor icon, en dictionary entries, music skill reference, and the regenerated server tool catalog.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
这个 PR 做了什么
给音频生成加一个新的供应商选项:Sonilo,按成片出配乐和音效。
配乐(
submit_music·provider: "sonilo"·mode: "v2m")现在的音乐生成从文字描述出发。这条路反过来:把剪好的成片(工程里的视频素材)直接交给模型,配乐跟着画面的节奏、场景切换和情绪走,出来就是成片的长度,不用再对时长。想干预风格可以带一句提示(「暖一点的民谣,不要鼓」),不带也能出。视频最长 6 分钟,每个任务出一条
.m4a。音效(
submit_sound·provider: "sonilo")同一个 Key。成片交上去,出跟画面对位的音效(免版税),不用写提示词。视频最长 3 分钟。生成的音轨落进媒体池后,铺轨、闪避(ducking)这些还是走时间线现有操作。
两条都是可选项:不配
SONILO_API_KEY的话,设置面板、Agent 能力清单和现有生成路径完全没变化。版权
license_id,写进生成结果,同时在音频文件旁边落一个.license.json留档,商用审核时拿得出来。实现(照仓库现有模式来)
music-sonilo.ts+ 共享客户端sonilo-media.ts。异步任务全程接在generation-jobs上:multipart 提交拿task_id,先registerProviderTask再轮询,服务重启后从断点续上,不会重复提交;轮询GET /v1/tasks/{id},404 直接判失败不重试;结果是预签名 URL,下载请求不带鉴权头。/generate/sound的 provider 开关后面,同步返回,响应形状和现有音效一致({ path, durationSeconds },多一个licenseId)。SONILO_API_KEY,可选SONILO_BASE_URL。设置面板「生音乐」下新增 Sonilo 页,带只读连接探测(/v1/account/services),.env.local或设置面板都行,保存即生效。submit_music/submit_soundschema、capabilities 路由行、generate-workflow 提示、music skill 的references/sonilo.md。server tool catalog 已重新生成。User-Agent: OpenChatCut,方便服务端归因到本集成。行为变化说明(都很小,如实列一下)
GenerationResult加了一个可选字段licenseId,不影响现有序列化。provider字段,默认值保持原有供应商,现有调用不受影响。/generate/sound现在先校验请求体、再检查供应商 Key(原来顺序相反),报错信息会更准一点。submit_sound的 schema 里prompt从 required 改为按 provider 校验,服务端仍然强制。测试
server/plugins/music-sonilo.verify.ts(v2m 校验 + 任务音轨 / license 提取),已挂进test:serial。sound.verify.ts和generation-tool-input.verify.ts补了 sonilo 分支的断言。verify:affected(11/11 通过,含music.verify/key-probes.verify/keystore.verify/capabilities.verify)、verify:i18n、verify:server-tool-catalog、verify:mcp、verify:skills、npx tsc -b、npm run lint(只剩一条主分支上就存在的警告)。需要说明的
English summary: adds Sonilo as an additional audio provider — video-to-music (
submit_musicprovidersonilo, modev2m) composes music from the rendered cut itself (≤6-min video, one optional style prompt, works promptless), and video-to-SFX (submit_soundprovidersonilo, ≤3-min video, royalty-free). Music is licensed, safe for commercial use (terms apply); every track returns alicense_id, surfaced on the result and archived as a.license.jsonsidecar. Follows the existing vendor-plugin pattern (third music provider + sound provider switch, generation-jobs resume, keystore + settings page + key probe, tool catalog regenerated). No default behavior changes; nothing shows up untilSONILO_API_KEYis set. Disclosure: I work on growth at Sonilo.