feat: save file metadata as sidecar file#210
Conversation
|
hi, thank you for your contribution. Could you explain the purpose of implementing this feature? I also noticed that this feature requires changes to the existing storage interface definition, so I need some time to consider this idea and review the code. |
|
Added a configuration option to generate a .meta.json sidecar metadata file when saving files. If only the file is stored, it is difficult to trace its origin later, so the metadata is saved to identify the source and the associated message. The storage interface has been modified to return the final file path. Since the storage layer includes deduplication logic for duplicate names (e.g., file_1.ext, file_2.ext), the path provided by the caller may not be the actual saved path. This is my first time contributing to a Go project, and I’m not sure whether this approach aligns with common practices. Any feedback or suggestions would be appreciated. |
There was a problem hiding this comment.
Pull request overview
This PR introduces optional .meta.json sidecar files to persist Telegram message-derived metadata alongside saved files, and updates the storage layer to return the final saved path so callers can correctly attach sidecars (especially when storages auto-rename on collisions).
Changes:
- Add
save_metadataconfig flag and a newpkg/metadatamodule to build and serialize Telegram message metadata into.meta.jsonsidecars. - Change
storage.Storage.Saveto return(actualPath string, err error)and update all storage implementations/call sites accordingly. - Extend transfer logic to copy an existing metadata sidecar when transferring files between storages.
Reviewed changes
Copilot reviewed 32 out of 32 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| storage/webdav/webdav.go | Update Save to return actual saved path (supports collision renames). |
| storage/telegram/telegram.go | Update Save signature and propagate actual path result. |
| storage/storage.go | Storage interface Save now returns (string, error). |
| storage/s3/s3.go | Return actual saved key (incl. rename) from Save. |
| storage/s3/s3_test.go | Adjust tests for new Save return value; assert rename path. |
| storage/rclone/rclone.go | Update Save signature to return chosen candidate path. |
| storage/minio/client.go | Update Save signature and return actual saved key (incl. rename). |
| storage/minio/client_stub.go | Update stub Save signature for no_minio builds. |
| storage/local/local.go | Update Save to return actual saved path (incl. rename). |
| storage/alist/alist.go | Update Save signature and return actual path (incl. rename). |
| pkg/metadata/metadata.go | New metadata model + builder + JSON serializer + .meta.json suffix. |
| docs/content/zh/deployment/configuration/_index.md | Document save_metadata option (ZH). |
| docs/content/en/deployment/configuration/_index.md | Document save_metadata option (EN). |
| core/tasks/ytdlp/task_test.go | Update mock storage Save signature for tests. |
| core/tasks/ytdlp/execute.go | Update call sites for new Save return signature. |
| core/tasks/transfer/execute.go | Capture actual path and attempt to transfer metadata sidecar. |
| core/tasks/tfile/tftask.go | Pre-build metadata JSON (optional) and add metadata save helper. |
| core/tasks/tfile/stream.go | Capture actual saved path in stream mode; save metadata sidecar. |
| core/tasks/tfile/execute.go | Capture actual saved path; save metadata sidecar after upload. |
| core/tasks/telegraph/execute.go | Update Save call sites for new signature. |
| core/tasks/parsed/execute.go | Update Save call sites for new signature. |
| core/tasks/directlinks/execute.go | Update Save call sites for new signature. |
| core/tasks/batchtfile/task.go | Add per-element metadata JSON + save helper; update factory signature. |
| core/tasks/batchtfile/execute.go | Capture actual saved path per element; save metadata sidecar. |
| core/tasks/aria2dl/task_test.go | Update mock storage Save signature for tests. |
| core/tasks/aria2dl/execute.go | Update call sites for new Save return signature. |
| config/viper.go | Add SaveMetadata field and default config value. |
| config.example.toml | Add save_metadata example setting. |
| common/utils/tgutil/chatinfo.go | Add helper to derive chat title/username from ext/entities/API. |
| cmd/upload/cmd.go | Update CLI upload path to ignore returned Save value. |
| client/bot/handlers/utils/shortcut/tftask.go | Pass ctx into batch task element creation (for metadata build). |
| api/factory.go | Pass factory ctx into batch task element creation (for metadata build). |
Comments suppressed due to low confidence (1)
storage/telegram/telegram.go:146
- In the split-upload branch, Save() returns the value from splitUpload() (currently just
filename). That loses the originalstoragePathcontext (e.g., chat ID/dir segments) and makesactualPathinconsistent with the normal branch (which returnsoriginalPath). This breaks sidecar metadata saving/transfer because the.meta.jsonpath may be written to the wrong chat/path. Consider having the split-upload path return the same logical storage path as the non-split path (e.g.,originalPath) while still propagating any upload error.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| reader, _, err := readable.OpenFile(ctx, metaSourcePath) | ||
| if err != nil { | ||
| log.FromContext(ctx).Debugf("Failed to open metadata file %s, skipping: %s", metaSourcePath, err) | ||
| return nil | ||
| } | ||
| defer reader.Close() | ||
| _, err = elem.TargetStorage.Save(ctx, reader, actualPath+metadata.MetaSuffix) |
| if extCtx.Entities != nil { | ||
| if t, u, found := lookupEntities(extCtx.Entities, chatID); found { | ||
| return t, u | ||
| } | ||
| } | ||
|
|
||
| return fetchFromAPI(extCtx, peer, chatID) | ||
| } |
No description provided.