Tolerate broken files per-item in video embedding

[JonasWurst]

Summary

Apply the per-item broken-file handling from the full-image (#1691) and crop (#1706) embedding paths to the video path.

• _load_video_frames returns None for an unreadable/undecodable video — catching (OSError, FFmpegError, IndexError, StopIteration) plus the no-usable-duration case — instead of raising, so a broken video skips itself rather than aborting the run.
• embed_videos records each video once as BROKEN/ADDED in a FileOutcomeReport, returns an EmbeddingResult carrying kept_indices (input positions of the embedded videos), and raises AllInputFilesFailedError when every video is broken.
• EmbeddingResult is propagated through the VideoEmbeddingGenerator.embed_videos protocol method and RandomEmbeddingGenerator.
• embedding_manager.embed_videos filters sample IDs by kept_indices before storing, keeping sample IDs and embeddings in sync.

kept_indices comes out already sorted (videos are processed in input order, unlike the crop path which groups by filepath), so no post-sort is needed.

Stacking

This PR is stacked on #1753 (the manual-batch-loop refactor), not main. Its base is jonas-lig-10035-refactor-video-embedding-manual-loop. Rebase onto main once #1753 merges. Review #1753 first — this PR's diff is only the tolerance change.

Testing

• make static-checks — ruff + format + mypy clean.
• pytest tests/dataset/{test_perception_encoder_embedding_generator,test_embedding_manager,test_embedding_generator}.py — 41 passed.
• Added video tests: mixed good/broken/missing with kept_indices assertions, all-broken raises AllInputFilesFailedError, and a manager stub-drop case asserting only kept sample IDs are stored; unwrapped .embeddings in the existing video tests.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Video embedding now returns a structured result that includes both the generated embeddings and a kept_indices mapping for successfully processed inputs.
  • Improved fault-tolerance: missing or broken videos are skipped while still returning results for valid inputs.
• Bug Fixes
  • Embeddings are now stored only for kept samples, ensuring embeddings remain aligned with the correct videos when some inputs fail.
• Tests
  • Added/updated unit tests to verify skipping behavior, correct kept_indices alignment, and raising an error when all inputs fail.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ce541b3c-1d97-4d0a-8db9-7e5a004ce777

📥 Commits

Reviewing files that changed from the base of the PR and between 89ddcee and f4ad705.

📒 Files selected for processing (2)

• CHANGELOG.md
• lightly_studio/tests/dataset/test_embedding_manager.py

🚧 Files skipped from review as they are similar to previous changes (1)

• lightly_studio/tests/dataset/test_embedding_manager.py

---

📝 Walkthrough

Walkthrough

Video embedding APIs now return EmbeddingResult objects with kept_indices. Perception Encoder processing skips missing or broken videos, reports failures, raises when all inputs fail, and keeps stored sample IDs aligned with returned embeddings.

Changes

Video embedding flow

Layer / File(s)	Summary
Embedding result contract and storage alignment lightly_studio/src/lightly_studio/dataset/embedding_generator.py, lightly_studio/src/lightly_studio/dataset/embedding_manager.py	Video generators return EmbeddingResult; EmbeddingManager filters sample IDs using kept_indices before storing embeddings.
Fault-tolerant perception encoding lightly_studio/src/lightly_studio/dataset/perception_encoder_embedding_generator.py	Perception Encoder skips missing or undecodable videos, records retained input indices, validates video metadata, wraps loading failures, and raises when all inputs fail.
Embedding flow validation lightly_studio/tests/dataset/test_embedding_manager.py, lightly_studio/tests/dataset/test_perception_encoder_embedding_generator.py, CHANGELOG.md	Tests cover sample-ID alignment, skipped files, retained-index mapping, all-failed input handling, and structured embedding results; the changelog documents skipped broken files.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant EmbeddingManager
  participant PerceptionEncoderEmbeddingGenerator
  participant FileOutcomeReport
  EmbeddingManager->>PerceptionEncoderEmbeddingGenerator: embed_videos(filepaths)
  PerceptionEncoderEmbeddingGenerator->>FileOutcomeReport: track video load outcomes
  FileOutcomeReport-->>PerceptionEncoderEmbeddingGenerator: successful indices and failures
  PerceptionEncoderEmbeddingGenerator-->>EmbeddingManager: EmbeddingResult
  EmbeddingManager->>EmbeddingManager: filter sample IDs using kept_indices

Loading

Possibly related PRs

• lightly-ai/lightly-studio#1710: Introduces the shared EmbeddingResult pattern for image embeddings.
• lightly-ai/lightly-studio#1753: Modifies the same Perception Encoder video embedding and frame-loading code.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the main change: per-item tolerance for broken files during video embedding.
Description check	✅ Passed	The description covers the change, motivation, testing, and changelog context, though it doesn't follow the template headings exactly.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch jonas-lig-10035-video-embedding-tolerate-broken-files-per-item-instead-of

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[lightly-fast-track-bot]

❌  Fast Track: checks did not pass

Guardrail	Result	Message
dummy	✅	Always passes.
frontend/complexity	✅	0 file(s) checked.
backend/complexity	✅	5 file(s) checked, no violations.
backend/coverage	❌	Guardrail threw: Command failed: uv run pytest tests/dataset/test_embedding_generator.py tests/dataset/test_embedding_manager.py tests/dataset/test_perception_encoder_embedding_generator.py --cov=src/lightly_studio/dataset --cov-report=json:coverage.json -q --no-header ImportError while loading conftest '/home/runner/work/lightly-studio/lightly-studio/lightly_studio/tests/conftest.py'. tests/conftest.py:18: in from lightly_studio.api import features src/lightly_studio/init.py:18: in from lightly_studio.core.start_gui import ( src/lightly_studio/core/start_gui.py:12: in from lightly_studio.api.server import Server src/lightly_studio/api/server.py:11: in from lightly_studio.api.app import app src/lightly_studio/api/app.py:16: in from lightly_studio.api.routes import ( src/lightly_studio/api/routes/webapp.py:19: in raise RuntimeError(f"Directory '{webapp_dir}' does not exist in '{project_root}'") E RuntimeError: Directory '/home/runner/work/lightly-studio/lightly-studio/lightly_studio/src/lightly_studio/dist_lightly_studio_view_app' does not exist in '/home/runner/work/lightly-studio/lightly-studio/lightly_studio/src/lightly_studio'
diff-size	✅	PR adds 165 line(s) (limit: 215).
frontend/coverage	✅	0 file(s) checked.

View the guardrail run

To run the guardrails locally, from fast_track/ run make install once, then make run-guardrails (or GUARDRAILS=<name1>,<name2> make run-guardrails for some guardrails).

_{Reflects 088a807.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0b9d976354

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[JonasWurst]

/review