Skip to content

Fix inbound file handling and add cross-container file exchange - #28

Merged
dangoldbj merged 3 commits into
dangoldbj:mainfrom
lundog:file-exchange
Jul 19, 2026
Merged

Fix inbound file handling and add cross-container file exchange#28
dangoldbj merged 3 commits into
dangoldbj:mainfrom
lundog:file-exchange

Conversation

@lundog

@lundog lundog commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Why

Makes SimpleX file transfer reliable in external mode, including the common case where the simplex-chat runtime runs in a container (a filesystem OpenClaw doesn't share) while OpenClaw runs on the host or in its own container. It's transport-agnostic — no native-mode code — and everything is additive and backward compatible (unset config = today's behavior).

This is the file-exchange half of the earlier native-mode PR, split out per that PR's discussion so it can be reviewed and land on its own. A native-transport PR follows, stacked on top of this branch.

What

Inbound

  • Fix a file-receive race: queue the pending file and retry the accept on rcvFileDescrReady, cancelling on timeout — a not-yet-ready XFTP descriptor no longer drops the file.
  • Stage received files into OpenClaw's media/inbound store so media tools and sandboxed workspaces can read them.
  • Resolve relative inbound paths against a new connection.filesFolder (default ~/.simplex/files, matching the bundled runtime service). When the runtime runs with --files-folder it reports a bare file name, which is joined to filesFolder; without it, the runtime reports absolute paths and they're read directly.
  • Log the reported inbound path (absolute vs. relative) for diagnosability.

Outbound

  • Add connection.outboundFolder. When set, outbound media is staged there before sending, and the staged path — valid on the runtime's side — is what travels over the WebSocket. Staged files are reclaimed by a short-lived per-file timer after staging — not on the send path, since the runtime reads and uploads the file asynchronously after the send call returns, so deleting eagerly could race that read. Unset = the local path is passed as-is (single-filesystem default).
  • Add connection.outboundFolderOnClient for split-path deployments where the two sides mount the shared directory at different paths: the plugin stages into outboundFolder but rewrites the directory prefix to this before sending, so no verbatim/matching path is required. No effect without outboundFolder.
  • Resolve media://<subdir>/<id> store references to a physical path on send (e.g. re-sending a received file).

Housekeeping

  • Bump ws to ^8.21.0 (dependency-audit fix).
  • Consolidate the expandHome helper (was copy-pasted in four modules) into a single src/fs-paths.ts.

Config connection gains filesFolder, outboundFolder, and outboundFolderOnClient (types, Zod schema, regenerated manifest). Adds unit tests for inbound resolution, outbound staging + timer-based reaping (fake timers), the readable-path guard, and media reference handling; documents the settings in the config reference and the runtime-setup guide (symmetric inbound/outbound sections).

Compatibility

Additive and backward compatible: existing configs are unaffected, and with filesFolder/outboundFolder unset the plugin behaves exactly as before.

Testing

Validated end-to-end against a containerized simplex-chat runtime with OpenClaw on the host, sharing an inbound volume and a non-verbatim outbound mount (the two sides mount the shared directory at different paths, bridged by outboundFolderOnClient):

  • Inbound: received files from a mobile client — reported name-only, resolved against filesFolder, staged into media/inbound.
  • Outbound: the bot sent files sourced three ways — a local path, a media:// store reference, and an http(s) URL — each staged into the shared outbound dir, path-translated to the runtime's view, and delivered. Observed the runtime copy the source into its --temp-folder (an xftp-prefixed dir) within seconds and then remove it, confirming the source is safe to reclaim well before the TTL.
  • tsc, biome, and vitest all green.

lundog added 2 commits July 2, 2026 14:54
Makes SimpleX file transfer work reliably in external mode, including when
the simplex-chat runtime runs in a separate container sharing a volume rather
than a filesystem. Transport-agnostic.

Inbound:
- Fix a file-receive race: queue the pending file and retry the accept on
  rcvFileDescrReady, cancelling on timeout, so a not-yet-ready XFTP descriptor
  no longer drops the file.
- Stage received files into OpenClaw's media/inbound store so media tools and
  sandboxed workspaces can read them.
- Resolve relative inbound paths against connection.filesFolder (default
  ~/.simplex/files, matching the runtime service) instead of a hardcoded /tmp.
  When the runtime runs with --files-folder it reports a bare file name, which
  is joined to filesFolder; absolute paths (no --files-folder) are read directly.
- Log the reported inbound path (absolute vs relative) for diagnosability.

Outbound:
- Add connection.outboundFolder. When set, outbound media is staged into that
  directory before sending, and the staged path — valid inside the runtime's
  container when both sides mount the dir verbatim — is what travels over the
  WebSocket. Staged files are cleaned up after the send. Unset = legacy
  single-filesystem behavior (the local path is passed as-is).
- Resolve media://<subdir>/<id> store references to a physical path on send
  (e.g. re-sending a received file).

Config connection gains filesFolder and outboundFolder (types, schema, manifest).
Adds unit tests for inbound resolution, outbound staging/cleanup, the
readable-path guard, and media reference handling; documents both settings.

Co-Authored-By: Claude Opus 4.8

@dangoldbj dangoldbj left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lundog LGTM! 🚀 Thanks for opening this PR.

Heads up, these all depend on each other so they need to go in together, roughly in this order:

  • add the new src/fs-paths.ts file first
  • then apply the inline suggestions
  • then fold in the two existing expandHome copies.

New file: src/fs-paths.ts

Create a new file which is just a single home for expandHome, which right now is copy-pasted in four places:

import os from "node:os";
import path from "node:path";

/**
 * Expand a leading `~`, `~/`, or `~\` in a local path to the current user's home
 * directory. Only meaningful for OpenClaw-local paths, never apply it to a path
 * that is resolved on the runtime's side (e.g. `outboundFolderOnClient`).
 */
export function expandHome(value: string): string {
  if (value === "~") {
    return os.homedir();
  }
  if (value.startsWith("~/") || value.startsWith("~\\")) {
    return path.join(os.homedir(), value.slice(2));
  }
  return value;
}

Remove expandHome copies

  • src/simplex/runtime/db-path.ts: delete the local function expandHome and add import { expandHome } from "../../fs-paths.js";
  • src/cli/runtime-service.ts: delete the local function expandHome and add import { expandHome } from "../fs-paths.js";. The shared version also handles a bare ~ and ~\, which this copy didn't, so it's a small upgrade.

Comment thread src/channel/media/outbound-files.ts Outdated
Comment thread src/channel/media/outbound-files.ts Outdated
Comment thread src/channel/media/outbound-files.ts Outdated
Comment thread src/channel/media/outbound-files.ts Outdated
Comment thread src/channel/media/outbound-files.ts Outdated
Comment thread src/channel/messaging/simplex-send.ts Outdated
Comment thread src/actions/message-actions.ts Outdated
Comment thread src/actions/message-actions.ts Outdated
Comment thread src/channel/media/outbound-files.test.ts Outdated
Comment thread src/channel/media/outbound-files.test.ts Outdated
@lundog

lundog commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Thank you for all your help on this. I’m sorry I haven’t had time to follow up, I was away for a few days. I will get to work on the things you mentioned in the next few days.

Addresses review feedback on the file-exchange PR.

Outbound staging cleanup was racing the runtime. We deleted the staged file
the moment the send call returned, but that call only hands the path to the
runtime — the runtime reads/encrypts and uploads the file asynchronously
afterward, with no completion event back to the plugin, so deleting on
send-return could pull the file out from under it.

Replace delete-on-send with a per-file reaper:
- outbound-files.ts: STAGED_FILES becomes Map<sentPath, { onDisk, timeout }>;
  registerStagedFile() schedules an unref'd timer that unlinks the on-disk file
  after STAGED_FILE_TTL_MS (5 min). stageOutboundBuffer / stageOutboundLocalFile
  register instead of deleting. The timer runs regardless of how the send
  resolved, so it also reclaims leaks. cleanupStagedOutboundFiles is removed and
  the module comment updated.
- simplex-send.ts and actions/message-actions.ts: drop the cleanup import and
  call; a comment explains the reaper owns reclamation and why we must not
  delete on the send path.
- Tests rewritten with fake timers: file present up to the TTL and gone after,
  the client-dir case reaps the real on-disk file (not the translated sent
  path), and pre-existing files in the dir are left alone.

The 5-minute TTL only needs to outlast the runtime reading the source, not the
whole upload: simplex-chat copies the source into its --temp-folder (an
xftp-prefixed dir) within seconds and uploads from there, so the staged source
can be reclaimed well before the transfer finishes — verified with inbound and
outbound file transfers through OpenClaw.

Also de-duplicate expandHome, which was copy-pasted in four places, into a new
src/fs-paths.ts:
- new src/fs-paths.ts holds the single implementation (handles ~, ~/, ~\).
- db-path.ts, cli/runtime-service.ts, channel/media/outbound-files.ts, and
  channel/events/simplex-inbound-files.ts import it and drop their local copies
  (outbound-files and simplex-inbound-files also drop the now-unused os import).
  The runtime-service copy only handled ~/, so this is a small upgrade there.

@dangoldbj dangoldbj left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lundog LGTM! Thanks!

@dangoldbj
dangoldbj merged commit 2cf5f5e into dangoldbj:main Jul 19, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants