From 6581a9278fb8f4ec26fad4d27f64cf87b54e9249 Mon Sep 17 00:00:00 2001 From: duguwanglong Date: Wed, 8 Jul 2026 17:28:54 +0800 Subject: [PATCH 01/49] docs(onesig): align skill with strategy api v2.5.3 --- .flocks/plugins/skills/onesig-use/SKILL.md | 83 +- .../onesig-use/references/api-reference.md | 759 +++++------------- .../onesig-use/references/browser-workflow.md | 22 +- 3 files changed, 244 insertions(+), 620 deletions(-) diff --git a/.flocks/plugins/skills/onesig-use/SKILL.md b/.flocks/plugins/skills/onesig-use/SKILL.md index e1d39c563..9a4eafcaf 100644 --- a/.flocks/plugins/skills/onesig-use/SKILL.md +++ b/.flocks/plugins/skills/onesig-use/SKILL.md @@ -1,80 +1,79 @@ --- name: onesig-use -description: 用于处理 OneSIG(安全互联网网关 / Secure Internet Gateway)相关任务,适合通过 API 或者结合浏览器进行以下任务:威胁监控(仪表盘、防护大屏、失陷主机、入站/出站威胁事件、报告管理)、防护策略(全局白/黑名单、多维封锁、IPS、HTTP 黑名单、高危端口防护、API 联动、Syslog 自动封禁、FTP/SFTP 联动)、资产管理、平台管理(告警/审计/用户、HTTPS 解密、网口路由 DNS、HA、OneCC、设备升级与备份、license、MDR、诊断)、登录会话与改密、帮助文档。只要用户提到 OneSIG、SIG、安全互联网网关、微步互联网网关等相关操作时,必须先加载本 skill。本 skill 是 OneSIG 平台操作的唯一决策入口:在未阅读本 skill 并完成模式判断前,不要直接调用任何 `onesig_*` tool。 +description: 用于处理 OneSIG(安全互联网网关 / Secure Internet Gateway)相关任务,当前项目内优先适配 OneSIG Strategy API v2.5.3(`onesig_strategy_api_query` / `onesig_strategy_api_ops`):设备状态、资产、策略、全局白名单、全局黑名单、封禁白名单、HTTP 黑名单的查询与写操作。只要用户提到 OneSIG、SIG、安全互联网网关、微步互联网网关等相关操作时,必须先加载本 skill。本 skill 是 OneSIG 平台操作的唯一决策入口:在未阅读本 skill 并完成模式判断前,不要直接调用任何 `onesig_*` tool,也不要把 OneSEC 的调用约定套用到 OneSIG。 --- # OneSIG Use ## First -操作模式 API V.S Browser +先判断操作模式:API V.S Browser。 ### 何时使用 API -- 默认模式,默认使用 API -- !!! important: 如果已经进入了浏览器模式,就不要走 API 了 -- 查询类请求与处置 / 配置写入类请求要严格区分;用户没有明确要求执行写操作时,默认只使用只读查询能力 +- 默认先判断当前诉求是否被新版 Strategy API 覆盖;覆盖时优先使用 API +- 查询类请求与配置写入类请求必须严格区分;用户没有明确要求写操作时,默认只使用只读查询能力 +- 当前项目内最新工具包是 `.flocks/plugins/tools/device/onesig_v2_5_3`,只暴露两个 grouped tool: + - `onesig_strategy_api_query`:只读查询,`requires_confirmation=false` + - `onesig_strategy_api_ops`:写操作,`requires_confirmation=true` + +当前 Strategy API 覆盖的能力: + +- 设备状态:平台状态、系统状态、网络状态 +- 资产:资产组、资产列表、资产类型,以及资产 / 资产组的新增、更新、删除 +- 防护策略:策略列表、策略更新、策略删除 +- 全局白名单、全局黑名单、封禁白名单 +- HTTP 黑名单 ### 何时使用浏览器 -- 现有 API 没有覆盖目标能力 -- 未检测到对应 API 工具 -- API 当前不可用,例如未配置、未开通、无权限、认证失败、Cookie 过期、SSL 验证失败或服务不可达 +- 当前 Strategy API 没有覆盖目标能力,例如威胁监控、仪表盘、失陷主机、入站 / 出站事件、报告、IPS、Syslog 自动封禁、FTP/SFTP 联动、高危端口防护、用户管理、HTTPS 解密、网口 / 路由 / DNS、HA、OneCC、升级备份、license、MDR、诊断、帮助文档 +- API 工具未检测到、未配置、无权限、ApiKey / Secret 缺失、认证失败、SSL 验证失败或服务不可达 - 任务必须查看页面级详情、攻击链、威胁图、报表预览或人工确认弹窗 - 页面需要人工登录、图形验证码、TOTP、强制改密或页面级确认 - 用户明确要求使用浏览器,或者已经在浏览器操作过程中 ### 请求确认 -除非是用户要求使用浏览器,否则提示用户 API 不可用,请检查 API 配置或直接使用浏览器模式。 +除非用户要求使用浏览器,否则如果 API 不可用,应提示用户检查 OneSIG Strategy API 配置,或确认是否改用浏览器模式。 当确定操作模式后: -- API 模式:请阅读 API 模式使用指南 -- 浏览器模式:请阅读浏览器模式使用指南 +- API 模式:必须阅读 API 模式使用指南 +- 浏览器模式:必须阅读浏览器模式使用指南 ## API 模式使用指南 -OneSIG 一共 6 个 grouped tool,按用户语义分流: +必须阅读: -- 用户说"仪表盘""威胁防护大屏""失陷主机""入站/出站威胁事件""设备状态""报告""导出报表"时,优先走 `onesig_monitoring` -- 用户说"白名单""黑名单""多维封锁""API 联动密钥""Syslog 自动封禁""FTP/SFTP 联动""IPS 规则""HTTP 黑名单""端口防护组""一键 bypass""防护策略"时,优先走 `onesig_strategy` -- 用户说"资产""资产组""资产类型""资产导入/导出"时,优先走 `onesig_assets` -- 用户说"告警/通知配置""管理日志/审计""用户管理/登录策略""HTTPS 解密""网口/部署引导""路由 / DNS / 代理""HA 高可用""集中管控/OneCC""设备升级/重启/备份/恢复""日志外发""license""MDR""coredump""pcap""设备诊断"时,优先走 `onesig_device` -- 用户说"登录""退出""改密""账户信息""图形验证码""TOTP""恢复码""产品动态红点"时,优先走 `onesig_login` -- 用户说"帮助文档""产品版本""提交问题反馈"时,优先走 `onesig_helper` +各 grouped tool 与 action 的详细说明、最小调用示例、失败处理见 [references/api-reference.md](references/api-reference.md)。 -OneSIG 与 OneSEC 在调用约定上有几个**关键差异**,agent 不要混用: +核心调用约定: -- 时间字段是 `startTime` / `endTime`(不是 OneSEC 的 `time_from` / `time_to`),单位 **Unix 秒** -- 分页字段是 `pageNo` / `pageSize`(不是 `cur_page` / `page_size`,也不是 `page_items_num`) -- OneSIG 主要走 **Cookie 会话** + RSA-OAEP 加密密码登录,处理器会自动登录、会话过期会自动重登;只有当设备启用了图形验证码或 TOTP 时才需要显式调 `onesig_login` 的 `login` 动作,并把 `captcha` / `totp` 作为入参传入 -- 每个 grouped tool 的 action 除了 `action` 都按业务键平铺;POST 类 action 多数走 passthrough body,把所有筛选 / 分页字段直接放在请求体顶层即可 +- `onesig_strategy_api_query` 用于所有只读查询 +- `onesig_strategy_api_ops` 用于所有写操作;调用前必须确认影响范围 +- 入参优先使用 `{ "action": "...", "body": { ... } }` +- Strategy API 认证是 `ApiKey + Secret` 的 HMAC-SHA1 签名,handler 会自动生成 `apikey` / `timestamp` / `sign`,agent 不要手工拼签名 +- Strategy API 走 `/api/v3/...` 路径;不要混用旧控制台 Cookie 版 `/v3/...` 工具说明 +- 分页字段使用 `pageNo` / `pageSize`;不要传 OneSEC 的 `cur_page` / `page_size` / `page_items_num` +- 当前 Strategy API action 没有 OneSEC 风格的 `time_from` / `time_to`;不要把 OneSEC 时间参数套进 OneSIG +- 查询单条或写操作前,如果缺少业务 ID,应先调用对应 list action 获取主键,再执行下一步 高风险写操作要特别谨慎,例如: -- 设备级动作:`device_onekey_bypass` / `device_quick_bypass`(流量直通)、`device_reboot` / `device_shutdown` / `device_reinit`、`device_upgrade` / `device_upgrade_upload` / `system_upgrade` -- 用户与口令:`user_create` / `user_delete` / `user_update` / `user_secret_reset`、`change_password`、`regenerate_recovery_code` -- 黑白名单与封禁:`whitelist_*` / `blacklist_*` 系列写操作、`multiblock_rule_create` / `multiblock_rule_update` / `multiblock_rule_delete` / `multiblock_rule_active`、`auto_blacklist_create` / `auto_blacklist_update` / `auto_blacklist_delete` -- IPS / HTTP / 端口防护:`ips_rule_create` / `ips_rule_apply` / `ips_rule_all`、`ips_ruleset_create` / `ips_ruleset_update` / `ips_ruleset_delete`、`http_blacklist_*` 写操作、`port_protect_group_*` / `port_protect_port_*` 写操作、`protection_policy_update` / `protection_policy_delete` -- HTTPS 解密:`tls_decrypt_policy_create` / `_update` / `_enable` / `_delete` / `_batch`、`tls_cert_create` / `_update` / `_delete` / `_set_default`、`tls_detect_delete` / `set_decrypt_config` -- 网口 / 路由 / 网络:`interface_update`(涉及启停时需 password)、`interface_*_create/update/delete`(virtualLine / listen / bridge)、`route_static_*` / `ipv6_route_static_*` 写操作、`set_dns_config` / `set_proxy_config`、`hosts_create/update/delete` -- 高可用 / 集中管控:`set_ha_config` / `ha_switching` / `ha_sync_config`、`set_onecc_config` / `set_onecc_status` -- 备份与升级:`backup_create` / `backup_recover` / `backup_delete` / `backup_import` / `backup_update`、`device_download_package` -- 离线情报库 / 授权:`basic_information_import` / `basic_information_enable`、`basic_license_upload`、`mdr_service_enable` -- 诊断写操作:`device_pcap_set`、`device_coredump_delete` / `device_pcap_file_delete` -- 报表与日志外发:`report_form_create` / `report_form_delete`、`report_task_create` / `_update` / `_delete` / `_test`、`logaccess_create` / `_update` / `_delete`、`set_dnslog_config`、`set_clean_config` / `set_storage_config` -- API Key 与密钥:`apikey_create` / `apikey_update` / `apikey_delete`、`apikey_secret`(会回吐明文 secret,需当前用户密码二次校验) - -必须阅读: - -各 grouped tool 与 action 的详细说明、必填参数、最小调用示例见 [references/api-reference.md](references/api-reference.md)。 +- `asset_group_create` / `asset_group_update` / `asset_group_delete` +- `asset_create` / `asset_update` / `asset_delete` +- `protection_policy_update` / `protection_policy_delete` +- `whitelist_create` / `whitelist_update` / `whitelist_delete` / `whitelist_remove` +- `blacklist_create` / `blacklist_update` / `blacklist_delete` / `blacklist_remove` +- `banned_whitelist_create` / `banned_whitelist_update` / `banned_whitelist_delete` +- `http_blacklist_create` / `http_blacklist_update` / `http_blacklist_enable` / `http_blacklist_delete` ## 浏览器模式使用指南 -- ⚠️ 如果 OneSIG 设备的访问地址不清楚,请先询问用户,不要擅自填写域名。 -- ⚠️ 用 `--headed` 打开浏览器,人工完成登录(OneSIG 多数部署启用了图形验证码 / TOTP / 强制改密策略)。 -- ⚠️ OneSIG 控制台与 OneSEC / 青藤是不同产品;不要把 OneSEC 的页面路径或 OneSEC 的 API 套用到 OneSIG。 +- 如果 OneSIG 设备的访问地址不清楚,请先询问用户,不要擅自填写域名 +- 用 `--headed` 打开浏览器,人工完成登录(OneSIG 部署可能启用了图形验证码 / TOTP / 强制改密策略) +- OneSIG 控制台与 OneSEC / 青藤是不同产品;不要把 OneSEC 的页面路径或 OneSEC 的 API 套用到 OneSIG 只要进入浏览器模式,就请阅读并按照 browser-workflow 操作,不要直接跳过本 skill 去套用其他通用浏览器 skill。 diff --git a/.flocks/plugins/skills/onesig-use/references/api-reference.md b/.flocks/plugins/skills/onesig-use/references/api-reference.md index 80bb8316c..9fc3b6c82 100644 --- a/.flocks/plugins/skills/onesig-use/references/api-reference.md +++ b/.flocks/plugins/skills/onesig-use/references/api-reference.md @@ -1,695 +1,318 @@ -# OneSIG API 调用指南 +# OneSIG Strategy API 调用指南 -OneSIG 当前的 6 个 grouped tool(`onesig_login` / `onesig_assets` / `onesig_device` / `onesig_helper` / `onesig_monitoring` / `onesig_strategy`)都遵循"`action` + 业务键平铺"的调用模式。处理器会自动登录、自动续会话、自动按需做 RSA-OAEP 密码加密,所以业务调用通常**不需要**先单独调登录接口。 +当前项目内最新 OneSIG 工具包来自 `.flocks/plugins/tools/device/onesig_v2_5_3`,它不是旧控制台 Cookie 登录版的 6 个 grouped tool,而是 OneSIG v2.5.3 第三方 Strategy API 接入: + +- `onesig_strategy_api_query`:只读查询 +- `onesig_strategy_api_ops`:写操作 + +这两个工具使用 `ApiKey + Secret` 做 HMAC-SHA1 签名,调用 `/api/v3/...` 接口。不要把旧控制台插件的 `onesig_monitoring` / `onesig_strategy` / `onesig_assets` / `onesig_device` / `onesig_login` / `onesig_helper` 文档套到当前 Strategy API 工具上。 ## 先看这张路由表 -| 用户意图 | 推荐 tool | 推荐 action | 必备参数 | +| 用户意图 | 推荐 tool | 推荐 action | body 说明 | |---|---|---|---| -| 看仪表盘总览 / 出入站 / 零日 | `onesig_monitoring` | `dashboard_overview` / `dashboard_outbound` / `dashboard_inbound` / `dashboard_zeroday` | 通常空参,部分需 `startTime`/`endTime` | -| 看威胁防护大屏(事件/资产/趋势/占比) | `onesig_monitoring` | `overview_*` 系列 | 多数必填 `startTime`+`endTime`,部分要 `incIntervalSec` 或 `interval` | -| 看入站/出站威胁事件列表与详情 | `onesig_monitoring` | `event_inbound_*` / `event_outbound_*` | 列表与详情类必填 `startTime`+`endTime` | -| 看失陷主机统计 / 列表 / 详情 | `onesig_monitoring` | `alert_host_*` | 列表与详情类必填 `startTime`+`endTime`,详情类还要 `source` | -| 看 / 改报表 | `onesig_monitoring` | `report_form_*` / `report_task_*` | 下载需 `uniqueId`+`fileName`,删除需 `uniqueId` | -| 看设备状态(CPU/内存/网口/平台) | `onesig_monitoring` | `device_platform_status` / `device_system_status` / `device_network_status` / `common_interface_list` / `basic_cpu_attr` | `device_system_status` 必填 `time` | -| 改 / 查白黑名单 | `onesig_strategy` | `whitelist_*` / `blacklist_*` | 列表用 `*_list`,删除用 `uniqueIds` | -| 多维封锁规则 | `onesig_strategy` | `multiblock_rule_*` / `multiblock_executelog_*` | 单条规则用 `name` 做主键 | -| API 联动密钥 | `onesig_strategy` | `apikey_*` | `apikey_secret` 必填 `key`+`password` | -| Syslog 自动封禁 | `onesig_strategy` | `auto_blacklist_*` | `auto_blacklist_trend`/`_sample` 必填 `srcIp`(+组合键) | -| FTP/SFTP 联动 | `onesig_strategy` | `linkage_*` | `linkage_info` 必填 `uniqueId` | -| IPS 规则 / 规则集 | `onesig_strategy` | `ips_rule_*` / `ips_ruleset_*` / `ips_threat_types` | 单条规则集查询用 `name`,引用关系用 `ruleId`+`assetIp` | -| HTTP 黑名单 / 高级 / XFF | `onesig_strategy` | `http_blacklist_*` / `*_advanced_config` / `*_xff_config` | 列表 `_list`、写操作走 `_create` / `_update` / `_delete` | -| 高危端口防护 | `onesig_strategy` | `port_protect_group_*` / `port_protect_port_*` | 端口列表 `port_protect_port_list` 必填 `groupName`+`pageNo`+`pageSize` | -| 防护策略首页 | `onesig_strategy` | `protection_policy_*` / `device_onekey_bypass` | `protection_policy_get` 必填 `uniqueId` | -| 资产 / 资产组 / 资产类型 | `onesig_assets` | `asset_*` / `asset_group_*` / `asset_type_*` / `common_asset_group_tree` | 资产用 `uniqueId`(删除还需 `password`);资产组写操作用 `uid`(新增是 `pid`+`name`);导入用 `file_path` | -| 告警 / 通知策略 | `onesig_device` | `alert_policy_*` / `*_notice_config` / `test_email/syslog/webhook` | `alert_policy_find_by_config` 必填 `search`+`type` | -| 管理日志(审计) | `onesig_device` | `aclog_*` / `*_clean_config` | 列表 / 导出常需 `startTime`+`endTime`;删除需 `password` | -| 用户与登录策略 | `onesig_device` | `user_*` / `*_login_config` | 改密 / 删除 / 重置密码需 `password`(自动 RSA 加密) | -| HTTPS 解密 | `onesig_device` | `tls_decrypt_policy_*` / `tls_cert_*` / `tls_detect_*` | TLS 详情必填 `server`+`port`+`orderBy`+`sortBy` | -| 网口 / 部署引导 | `onesig_device` | `interface_*` | `interface_select_list` 必填 `workMode` | -| 路由(v4/v6) | `onesig_device` | `route_*` / `ipv6_route_*` | 写操作要带完整路由项 | -| DNS / 代理 / 网络测试 | `onesig_device` | `*_dns_config` / `hosts_*` / `*_proxy_config` / `test_network` / `test_proxy` | hosts 写操作走 create/update/delete | -| 高可用 HA | `onesig_device` | `ha_*` / `*_ha_config` | `ha_sync_status` 必填 `syncId`(前置 `ha_sync_config` 返回) | -| 集中管控 OneCC | `onesig_device` | `*_onecc_config` / `onecc_status` / `set_onecc_status` / `test_onecc` | 写操作前先看 status | -| 设备升级 / 重启 / 备份 / 恢复 | `onesig_device` | `device_upgrade*` / `device_reboot/shutdown/reinit` / `backup_*` / `system_upgrade` | 升级 / 上传备份用 `file_path`,敏感写操作需 `password` | -| 日志外发 | `onesig_device` | `logaccess_*` | 列表必填 `pageNo`+`pageSize`+`type`;样例必填 `srcIp`+`protocol`+`type` | -| 基本信息 / license / MDR | `onesig_device` | `basic_*` / `mdr_service_*` | license / 离线情报库导入用 `file_path` | -| 设备诊断 | `onesig_device` | `device_coredump_*` / `device_pcap_*` | coredump/pcap 删除走 DELETE | -| 帮助文档 / 产品反馈 | `onesig_helper` | `document_list` / `document_preview` / `product_*` | `document_preview` 必填 `id`(来自 `document_list`) | -| 登录 / 退出 / 改密 / 账户 | `onesig_login` | `login` / `logout` / `change_password` / `get_account` 等 | 改密必填 `new_password`,启用 captcha/TOTP 时 `login` 要传 `captcha`/`totp` | +| 查平台运行状态 | `onesig_strategy_api_query` | `platform_status` | 通常空对象 | +| 查系统状态 | `onesig_strategy_api_query` | `system_status` | 通常空对象 | +| 查网络状态 | `onesig_strategy_api_query` | `network_status` | 通常空对象 | +| 查资产组 | `onesig_strategy_api_query` | `asset_group_list` | GET 类接口,通常空对象 | +| 查资产列表 | `onesig_strategy_api_query` | `asset_list` | 可传 `pageNo`、`pageSize`、`search` 等厂商文档字段 | +| 查资产类型 | `onesig_strategy_api_query` | `asset_type_list` | GET 类接口,通常空对象 | +| 查防护策略 | `onesig_strategy_api_query` | `protection_policy_list` | 可传分页 / 筛选字段 | +| 查全局白名单 | `onesig_strategy_api_query` | `whitelist_list` | 可传分页 / 筛选字段 | +| 查全局黑名单 | `onesig_strategy_api_query` | `blacklist_list` | 可传分页 / 筛选字段 | +| 查封禁白名单 | `onesig_strategy_api_query` | `banned_whitelist_list` | 可传分页 / 筛选字段 | +| 查 HTTP 黑名单 | `onesig_strategy_api_query` | `http_blacklist_list` | 可传分页 / 筛选字段 | +| 新建 / 更新 / 删除资产组 | `onesig_strategy_api_ops` | `asset_group_create` / `asset_group_update` / `asset_group_delete` | 按 api-onesig-2.5.3 文档填写 | +| 新建 / 更新 / 删除资产 | `onesig_strategy_api_ops` | `asset_create` / `asset_update` / `asset_delete` | 按 api-onesig-2.5.3 文档填写 | +| 更新 / 删除防护策略 | `onesig_strategy_api_ops` | `protection_policy_update` / `protection_policy_delete` | 先查策略列表拿业务 ID | +| 新建 / 更新 / 删除 / 移除全局白名单 | `onesig_strategy_api_ops` | `whitelist_create` / `whitelist_update` / `whitelist_delete` / `whitelist_remove` | 写入前先确认方向、条件和影响范围 | +| 新建 / 更新 / 删除 / 移除全局黑名单 | `onesig_strategy_api_ops` | `blacklist_create` / `blacklist_update` / `blacklist_delete` / `blacklist_remove` | 写入前先确认对象和影响范围 | +| 新建 / 更新 / 删除封禁白名单 | `onesig_strategy_api_ops` | `banned_whitelist_create` / `banned_whitelist_update` / `banned_whitelist_delete` | 写入前先确认对象和影响范围 | +| 新建 / 更新 / 启停 / 删除 HTTP 黑名单 | `onesig_strategy_api_ops` | `http_blacklist_create` / `http_blacklist_update` / `http_blacklist_enable` / `http_blacklist_delete` | `enable` 也属于写操作 | ## 通用规则 -- OneSIG 全部 grouped tool 的入参形态:`{action: "...", ...业务字段}`(不像 TDP 把所有筛选放进统一 `body`,也不像 OneSEC 用 `time_from`/`time_to`) -- 时间字段统一是 **Unix 秒**:`startTime` / `endTime` / `time`,没有"recent"系列接口 -- 分页字段统一是 `pageNo`(默认 1)+ `pageSize`(默认 20);不要传 `cur_page` / `page_size` / `page_items_num` -- 排序统一是 `sortBy`(字段名)+ `orderBy`(`asc` / `desc`) -- POST 类 action 多数 passthrough body —— 任何过滤字段直接放在请求体顶层即可,与 Web 控制台抓包字段一一对应 -- 查询类 action 默认优先;写操作只有用户明确授权时才执行(特别是黑白名单批量写、设备升级、HTTPS 解密策略、HA / OneCC 配置) -- 业务键参考第二节的"业务 ID 字段";如果没有 ID,应先调对应列表 / namelist / tree 接口拿主键 - -## 业务 ID 字段对照(`required` 主键) - -OneSIG 不同模块用不同字段做"主键",没有 ID 是无法调单条接口的。如果 agent 拿不到主键,应该**先调对应的列表 / namelist / tree** action 取得主键,再调单条接口。 - -| 字段 | 用在哪些 action | 含义 | -|---|---|---| -| `id` | `document_preview` | 帮助文档主键(先 `document_list`) | -| `key` + `password` | `apikey_secret` | API Key 名称 + 当前用户登录密码(自动 RSA 加密) | -| `uniqueId` | `asset_update` / `asset_delete`(+`password`)/ `linkage_info` / `protection_policy_get` / `report_form_download` / 多数 `*_delete` 单条目 / 多数全局白黑名单 `*_update` | 资源唯一 ID,先调列表/树拿到 | -| `uniqueIds` | 批量删除(`whitelist_remove_batch` 等于 `globalWhitelist` DELETE、`blacklist_remove_batch` 等于 `globalBlacklist` DELETE、`*_delete` 批量) | 数组形式,多个资源一起删 | -| `uid` | `asset_group_update` / `asset_group_delete`(+`password`) | **资产组**主键(不是 `uniqueId`!) | -| `pid` + `name` | `asset_group_create` | 父级资产组 ID(根 = `0`) + 新组名 | -| `syncId` | `ha_sync_status` | HA 配置同步任务 ID(由前置 `ha_sync_config` 返回) | -| `name` | `multiblock_rule_get` / `multiblock_rule_preview` / `multiblock_executelog_list` / `ips_ruleset_info` / `ips_ruleset_delete` / `asset_type_delete` | OneSIG 用对象名当主键,没有独立数字 ID | -| `groupId` | `port_protect_group_update` / `_delete` / `_clone` 的 `fromGroupId`、`port_protect_port_*` 写操作、`port_protect_portinfo` | 端口防护**组**主键(写操作用) | -| `groupName` | `port_protect_port_list`(端口列表)/ `port_protect_port_export` | 端口防护组**名**(仅查列表 / 导出用,注意与 `groupId` 区分) | -| `ruleId` | `ips_rule_create`(**实际是查单条规则详情**,文档名为「新增 IPS 自定义规则」与实际语义不符)、`ips_ruleset_referred` 的一半 | IPS 规则 ID | -| `ruleId` + `assetIp` | `ips_ruleset_referred` | 单条 IPS 规则 + 涉事资产 IP,反查规则在哪台资产上命中 | -| `srcIp`(+`protocol`+`direction`) | `auto_blacklist_trend` / `auto_blacklist_sample` | 来自 syslog 自动黑名单的源 IP 复合主键 | -| `server` + `port` + `orderBy` + `sortBy` | `tls_detect_list_detail` | TLS 解密目标"主机:端口"复合键 | -| `workMode` | `interface_select_list` | 网口选择列表必须先选定工作模式(内联/旁路) | -| `search` + `type` | `alert_policy_find_by_config` | 按配置反查告警策略 | -| `time` | `device_system_status` | 系统资源采样时间窗(Unix 秒) | -| `incIntervalSec` / `interval` | `overview_asset_brief` / `overview_event_trend` / `overview_stat` | 聚合粒度(秒) | -| `key` (Body) + Query `type=physical` | `apikey_delete` / `_update`,新增 `apikey_create` 也要 `type=physical` | API 联动密钥的接口要求 Query `?type=physical`(厂商文档强制)—— 当前 handler 走 passthrough body,按目前实测把 `type:"physical"` 一起放进 body 即可,但若服务端严格要求 query,请改用浏览器流程 | - -## 时间窗口与时区 - -- 时间戳一律 Unix **秒**(不是毫秒) -- 文档示例按 `Asia/Shanghai`,实际调用时按业务时区计算 -- OneSIG 没有"最近 24 小时增量"专用接口;要"最近一段"统一用 `now - 24*3600`、`now - 7*86400` 之类自行算 -- 未传时间走服务端默认窗口,仅作兜底,不推荐依赖 - -## 高频场景 - -### 1. 看入站 / 出站威胁事件 - -推荐:`onesig_monitoring` + `event_inbound_list` 或 `event_outbound_list`。 - -最小示例(查最近 24 小时入站事件): +- 入参优先使用: ```json { - "action": "event_inbound_list", - "startTime": 1745683200, - "endTime": 1745769600, - "pageNo": 1, - "pageSize": 20 + "action": "asset_list", + "body": { + "pageNo": 1, + "pageSize": 20 + } } ``` -后续动作: +- handler 也兼容把业务字段平铺在 `action` 同级,但文档和新请求默认使用 `body`,避免和旧控制台工具混淆。 +- `GET` 类 action 只发送签名 query,不发送 JSON body;不要依赖 `body` 给 `asset_group_list` / `asset_type_list` 传筛选条件。 +- 分页字段使用 `pageNo` / `pageSize`。 +- 排序 / 筛选字段按厂商 `api-onesig-2.5.3` 文档填写;如果不确定字段名,先用最小参数查询,不要猜旧控制台字段。 +- 当前 Strategy API action 没有 `startTime` / `endTime`,也没有 OneSEC 的 `time_from` / `time_to`;不要混用时间字段。 +- 查询优先;写操作必须在用户明确授权后执行。 +- 写操作前应先通过 list action 获取当前对象和业务 ID,确认不会误改 / 误删。 +- 当 `delete` 与 `remove` 的语义不确定时,不要猜;先查厂商文档或只执行只读查询。 -- 拿到事件后调 `event_inbound_detail` / `_detail_trend` / `_detail_list` 看详情、趋势、关联记录 -- 导出文件用 `event_inbound_export` / `event_inbound_detail_export`(返回的是文件,不是 JSON) +## 认证与配置 -返回结果重点关注: +`onesig_v2_5_3` 的 provider 信息: -- 事件 `uniqueId`、源/目的 IP、威胁类型 `threatType`、威胁等级 `severity`、威胁标签 `threatLabel`、是否 TLS `isTls` +- `service_id`: `onesig_api` +- `version`: `2.5.3` +- `base_url`: OneSIG 设备地址;缺少协议时 handler 自动补 `https://` +- `api_key`: ApiKey,优先读服务配置,也支持 secret `onesig_v2_5_3_api_key` +- `secret`: Secret,优先读服务配置,也支持 secret `onesig_v2_5_3_secret` +- 环境变量兜底:`ONESIG_V2_5_3_BASE_URL`、`ONESIG_V2_5_3_API_KEY`、`ONESIG_V2_5_3_SECRET` +- `verify_ssl`: 默认 `false`,适合自签证书设备;需要严格校验证书时在服务配置中开启 -何时回退浏览器:需要事件原始报文、PCAP 取证、详情页攻击链。 +handler 会自动把以下签名参数追加到请求 query: -### 2. 失陷主机调查 +```text +apikey= +timestamp= +sign= +``` -推荐:`onesig_monitoring` + `alert_host_*`。 +agent 不需要、也不应该手工生成签名。 -```json -{ - "action": "alert_host_list", - "startTime": 1745683200, - "endTime": 1745769600, - "pageNo": 1, - "pageSize": 20 -} -``` +## 只读查询示例 -详情维度: +### 1. 查设备平台状态 ```json { - "action": "alert_host_detail", - "startTime": 1745683200, - "endTime": 1745769600, - "source": "10.0.0.5" + "action": "platform_status", + "body": {} } ``` -`source` 是失陷主机的 IP(必填)。`alert_host_detail_list` / `_export` 同步要求 `startTime` / `endTime` / `source`。 - -### 3. 看仪表盘 / 大屏 - -仪表盘:`dashboard_overview` / `dashboard_outbound` / `dashboard_inbound` / `dashboard_zeroday` / `dashboard_status` / `dashboard_ioc_type_sum` —— 通常空参或只需时间窗。 - -威胁防护大屏(`overview_*`)—— 多数需要时间窗 + 聚合粒度: +### 2. 查系统状态 ```json { - "action": "overview_asset_brief", - "startTime": 1745683200, - "endTime": 1745769600, - "incIntervalSec": 3600 + "action": "system_status", + "body": {} } ``` -```json -{ - "action": "overview_event_trend", - "startTime": 1745683200, - "endTime": 1745769600, - "interval": 3600 -} -``` +### 3. 查网络状态 ```json { - "action": "overview_event_inbound_agg", - "startTime": 1745683200, - "endTime": 1745769600, - "type": "threatType", - "pageNo": 1, - "pageSize": 20 + "action": "network_status", + "body": {} } ``` -注意:`overview_*_export` 系列返回文件,不要拿来在 chat 里展开。 - -### 4. 资产管理 - -推荐:`onesig_assets`。 - -查列表: +### 4. 查资产列表 ```json { "action": "asset_list", - "pageNo": 1, - "pageSize": 20, - "search": "10.0.0." + "body": { + "pageNo": 1, + "pageSize": 20, + "search": "10.0.0." + } } ``` -新增 / 更新 / 删除(参数严格按厂商 `assetsSegment.md`): +返回后重点关注资产唯一标识、资产名称、IP / 网段、资产组和资产类型字段。字段名以设备实际返回为准。 + +### 5. 查资产组与资产类型 ```json { - "action": "asset_create", - "name": "host-1", - "type": "服务器", - "groupId": 101, - "ip": ["10.0.0.1", "10.0.0.2"], - "remark": "生产环境" + "action": "asset_group_list", + "body": {} } ``` -`type` 是字符串字段(不是 `assetType`),`ip` 必须是**字符串数组**(支持单 IP / 网段 / 区间),`groupId` 是数字(先调 `asset_group_get` 拿)。 - ```json { - "action": "asset_update", - "uniqueId": "asset-id-1", - "name": "host-1-renamed", - "type": "服务器", - "groupId": 101, - "ip": ["10.0.0.1"] + "action": "asset_type_list", + "body": {} } ``` -更新接口必填 `uniqueId` + `name` + `type` + `groupId` + `ip`(厂商文档要求**全量**字段,不是部分更新)。 - -```json -{ "action": "asset_delete", "uniqueId": "asset-id-1", "password": "<当前登录用户密码>" } -``` - -⚠️ **删除资产是单条调用**(仅 `uniqueId`),并且**必须带 `password`**(当前登录用户密码),处理器会自动 RSA-OAEP 加密。要批量删,请按行循环调用。 - -资产组:`asset_group_get`(拿整树)/ `asset_group_create`(`pid`+`name`)/ `asset_group_update`(`uid`+`name`)/ `asset_group_delete`(`uid`+`password`)。注意资产组主键叫 `uid`,**不是** `uniqueId`。整树缓存用 `common_asset_group_tree`。 - -资产导入: - -```json -{ "action": "asset_import", "file_path": "/abs/path/to/assets.csv" } -``` - -`asset_template` / `asset_export` 返回文件(CSV)。 +资产写操作前,先查这两个接口确认可用分组和类型。 -### 5. 黑白名单 / 多维封锁 / API 联动 - -白名单(写)—— 注意 `condition` / `comments` 与 `whiteList` **平级**,不是嵌进数组元素: +### 6. 查防护策略列表 ```json { - "action": "whitelist_add", - "whiteList": [{ "direction": "inbound" }], - "condition": [{ "type": "srcIp", "cond": "equal", "value": "10.0.0.1" }], - "comments": "策略备注" + "action": "protection_policy_list", + "body": { + "pageNo": 1, + "pageSize": 20 + } } ``` -新增时 `whiteList` 数组每项**只含 `direction`**(值为 `inbound` / `outbound` / `both`),`condition` 与 `comments` 在顶层。 +策略更新 / 删除前,必须先用列表结果确认目标策略 ID、名称、作用范围和当前状态。 + +### 7. 查全局白名单 / 黑名单 / 封禁白名单 ```json { - "action": "whitelist_update", - "uniqueId": "id-1", - "direction": "inbound", - "condition": [{ "type": "srcIp", "cond": "equal", "value": "10.0.0.1" }], - "comments": "更新备注" + "action": "whitelist_list", + "body": { + "pageNo": 1, + "pageSize": 20 + } } ``` -更新时**没有** `whiteList` 字段,只有 `uniqueId` + 单值 `direction` + `condition` + `comments`。 - -```json -{ "action": "whitelist_remove_batch", "uniqueIds": ["id-1", "id-2"] } -``` - -黑名单(同 OneSIG `globalBlacklist`): - ```json { - "action": "blacklist_add", - "blackList": [ - { "object": "1.1.1.1", "direction": "inbound", "threatName": "", "sLifeCycle": "", "comments": "" } - ] + "action": "blacklist_list", + "body": { + "pageNo": 1, + "pageSize": 20 + } } ``` -新增时整批用 `blackList` 数组;写入前可先 `blacklist_check`(IP 类冲突校验)。`_update` / `_delete`(按 `uniqueIds` 批量)/ `_list` 用法与白名单类似。 - -多维封锁: - ```json { - "action": "multiblock_executelog_list", - "name": "rule-A", - "startTime": 1745683200, - "endTime": 1745769600, - "pageNo": 1, - "pageSize": 20 + "action": "banned_whitelist_list", + "body": { + "pageNo": 1, + "pageSize": 20 + } } ``` -`multiblock_rule_get` / `_preview` 必填 `name`;`_create` / `_update` 是写操作,要谨慎。 - -API 联动: - -```json -{ "action": "apikey_list", "pageNo": 1, "pageSize": 20 } -``` - -`apikey_list` 必填 `pageNo`+`pageSize`,否则服务端回 `responseCode=1004`。 - -```json -{ "action": "apikey_secret", "key": "my-key", "password": "<当前登录密码>" } -``` - -`apikey_secret` 会回吐明文 secret,处理器自动用 RSA-OAEP 加密 `password` 后发送(注意:`password` 在该接口走 Query 而非 Body)—— 不要替用户保存这个值,更不要回显在 chat 里。 - -写操作(创建 / 更新 / 删除):厂商接口要求带 Query `?type=physical`。当前 handler 走 passthrough body,请把 `type:"physical"` 一起放进 body 顶层(实测多数实例服务端会接受),如果遇到 `responseCode=1004` 提示 `type` 缺失则回退浏览器模式: - -```json -{ "action": "apikey_create", "type": "physical", "name": "soc-bridge" } -``` - -```json -{ "action": "apikey_update", "type": "physical", "name": "soc-bridge", "key": "k1", "status": 1 } -``` - -```json -{ "action": "apikey_delete", "type": "physical", "key": "k1" } -``` - -### 6. IPS 规则 / 规则集 - -```json -{ "action": "ips_ruleset_namelist" } -``` - -```json -{ "action": "ips_ruleset_info", "name": "default-ruleset" } -``` - -```json -{ "action": "ips_ruleset_referred", "ruleId": "rule-001", "assetIp": "10.0.0.5" } -``` - -⚠️ 注意 `ips_rule_create`(厂商 `POST /v3/ips/rule`)**实际语义是「查单条规则详情」**,body 必填 `ruleId`,不会真的"新增"自定义规则;这是 yaml/handler 命名遗留问题。需要新增/编辑规则集请走 `ips_ruleset_create` / `_update`。`ips_rule_apply` 才是"应用规则变更到目标规则集"的写操作(body:`rulesetsName`数组+`rules`数组),调用前确认变更已就绪。 - -### 7. HTTP 黑名单 / 端口防护 - -HTTP 黑:`http_blacklist_list` / `_create` / `_update` / `_delete` / `_enable` / `_export`。 - -端口防护组: - -```json -{ "action": "port_protect_group_list_full" } -``` +### 8. 查 HTTP 黑名单 ```json { - "action": "port_protect_port_list", - "groupName": "默认高危端口组", - "pageNo": 1, - "pageSize": 20, - "sortBy": "updateTime", - "orderBy": "desc" + "action": "http_blacklist_list", + "body": { + "pageNo": 1, + "pageSize": 20 + } } ``` -⚠️ action 名是 `port_protect_port_list`(**不是** `port_protect_group_port_list`),必填 `groupName`+`pageNo`+`pageSize`;响应里总数字段叫 `data.totalCount`(不是 `data.total`)。 - -写操作: +## 写操作示例 -- `port_protect_group_create` Body `groupName`;`_update` Body `groupId`+`groupName`;`_delete` Body `groupId`;`_clone` Body `fromGroupId`+`groupName` -- 端口规则:`port_protect_port_create` / `_update` Body `groupId`+`ports`(字符串)+`serviceName`(可选)+`comments`(可选);`_delete` Body `groupId`+`ports`(数组);`_onekey_import` Body `groupId`+`ports`(对象数组 `[{ports, serviceName, comments}]`) -- 端口占用预查:`port_protect_portinfo` Body `groupId`+`ports` +写操作统一调用 `onesig_strategy_api_ops`,且必须确认用户授权。下面示例只展示调用形态;业务字段必须按设备版本对应的 `api-onesig-2.5.3` 文档填写,不要从旧控制台插件文档里硬搬字段。 -### 8. Syslog 自动封禁 - -```json -{ "action": "auto_blacklist_list", "pageNo": 1, "pageSize": 20 } -``` - -```json -{ "action": "auto_blacklist_trend", "srcIp": "1.2.3.4", "startTime": 1745683200, "endTime": 1745769600 } -``` - -```json -{ "action": "auto_blacklist_sample", "srcIp": "1.2.3.4", "protocol": "tcp", "direction": "inbound" } -``` - -`auto_blacklist_check`(必填 `name` / `port` / `srcIp` / `protocol` / `direction`)是冲突校验,写入前调用。 - -### 9. 用户管理与改密 - -查询:`user_list` / `user_export`。 - -新增用户(写,敏感)—— 字段严格按 `deviceLoginManagement.md`: +### 1. 新建资产组 ```json { - "action": "user_create", - "username": "alice", - "role": 2, - "nickname": "", - "phone": "", - "expireTime": 0, - "loginLimit": [], - "password": "<明文新密码>", - "dupPassword": "<明文新密码>" + "action": "asset_group_create", + "body": { + "...": "按厂商文档填写资产组字段" + } } ``` -字段说明: - -- `username`(必填,**不是 `name`**):1~20 字符,符合 `userEditor` 用户名正则 -- `role`(必填):数字角色(如 `0`/`2`/`3`/`4`,`role=3` 跳审计页、`role=4` 跳大屏) -- `expireTime`(必填):到期时间 Unix 秒;选「长期」时填 `0` -- `loginLimit`(必填):可登录 IP 字符串数组,ALL 时填 `[]` -- `password` / `dupPassword`(必填):明文新密码,处理器自动 RSA-OAEP 加密。注意 **handler 只对 `password` 与 `dupPassword`(驼峰)做加密**,不要传 `dup_password`(蛇形不会被加密) - -更新用户用 `user_update`,必填 `username`(定位主键,**不可改**)+ `expireTime` + `loginLimit`,可选 `nickname` / `phone`;改密**不在** `user_update` 路径,要走 `Main/changePwdModalData`(即顶栏改密弹窗,agent 用 `change_password` 走 `onesig_login`)。 - -改密(当前用户)走 `onesig_login`: +### 2. 更新资产 ```json { - "action": "change_password", - "old_password": "<明文旧>", - "new_password": "<明文新>", - "dup_password": "<明文新>" + "action": "asset_update", + "body": { + "...": "先从 asset_list 获取目标资产 ID,再按厂商文档填写完整更新字段" + } } ``` -`user_secret_reset` / `user_delete` / `aclog_delete` / `interface_update`(启停场景)/ `device_upgrade*` 同样要传 `password`,规则一致。 - -如果 `POST /v3/login` 返回 responseCode `1009` / `1017` 但密码确实正确,先把 `oaep_hash` 切到 `sha256` 重试 —— OneSIG v2.5.x 多数走 SHA-1(JSEncrypt 默认),但有个别部署是 SHA-256。 - -### 10. HTTPS 解密 - -策略:`tls_decrypt_policy_list` / `_create` / `_update` / `_enable` / `_delete` / `_batch`。 - -证书:`tls_cert_list` / `_create`(multipart:`file_path` 指向 `.crt`/`.pem`)/ `_update` / `_delete` / `_set_default`。 - -检测对象: +### 3. 更新防护策略 ```json { - "action": "tls_detect_list_detail", - "server": "internal-svc.example.com", - "port": 443, - "orderBy": "desc", - "sortBy": "lastSeenTime" + "action": "protection_policy_update", + "body": { + "...": "先从 protection_policy_list 获取目标策略,再按厂商文档填写" + } } ``` -`server` + `port` + `orderBy` + `sortBy` 这 4 个是必填的复合主键。 - -### 11. 网口 / 部署引导 / 路由 / DNS - -网口:`interface_list` / `interface_select_list`(必填 `workMode`,"内联"/"旁路"二选一)/ `interface_update`(启停时 `password` 必填)/ `interface_check_loop` / `interface_relation_list`。 - -虚拟线 / 监听 / 桥:`interface_virtual_line_*` / `interface_listen_*` / `interface_bridge_*`。 - -路由:`route_outif_list` / `route_static_list` / `_create` / `_update` / `_delete` / `route_table_list`,IPv6 同名加 `ipv6_` 前缀。 - -DNS:`get_dns_config` / `set_dns_config` / `hosts_get` / `hosts_create` / `_update` / `_delete` / `test_network`。 - -### 12. HA 高可用 - -```json -{ "action": "ha_status" } -``` - -```json -{ "action": "ha_sync_config" } -``` - -```json -{ "action": "ha_sync_status", "syncId": "<上一步返回的 syncId>" } -``` - -`ha_switching` 是主备切换,**强破坏性写操作**,没有用户明确授权不要调。 - -### 13. 集中管控 OneCC - -```json -{ "action": "onecc_status" } -``` - -```json -{ "action": "test_onecc", "...": "..." } -``` - -`set_onecc_config` / `set_onecc_status` 都会改设备纳管状态,调用前先用 `get_onecc_config` / `onecc_status` 看清当前形态。 - -### 14. 设备配置 / 升级 / 备份 - -升级: - -- `basic_version` / `device_upgrade_info` —— 看版本与可升级状态 -- `get_upgrade_config` / `set_upgrade_config` —— 在线升级源配置 -- `device_download_package` —— 触发后台下载升级包(写) -- `device_upgrade` —— 已下载的包升级(写,敏感,`password` 必填) -- `device_upgrade_upload` —— 本地上传升级包(multipart:`file_path`,`password` 必填) -- `system_upgrade` —— 系统级升级(multipart:`file_path`) - -备份: - -```json -{ "action": "backup_list", "pageNo": 1, "pageSize": 20 } -``` - -```json -{ "action": "backup_create", "name": "manual-2026-04-27" } -``` - -```json -{ "action": "backup_recover", "uniqueId": "" } -``` - -```json -{ "action": "backup_import", "file_path": "/abs/path/to/backup.tar" } -``` - -`backup_recover` / `backup_import` / `backup_delete` 都强破坏性,确认后再调。 - -设备其它:`device_quick_bypass` / `device_onekey_bypass`(流量直通)、`device_reboot` / `device_shutdown` / `device_reinit`(重启 / 关机 / 出厂重置)—— 全是高风险,不要在不需要的时候触发。 - -### 15. 日志外发 +### 4. 新建全局黑名单 ```json { - "action": "logaccess_list", - "pageNo": 1, - "pageSize": 20, - "type": "syslog" + "action": "blacklist_create", + "body": { + "...": "按厂商文档填写黑名单对象、方向、备注等字段" + } } ``` +### 5. 启停 HTTP 黑名单 + ```json { - "action": "logaccess_sample", - "srcIp": "10.0.0.1", - "protocol": "udp", - "type": "syslog" + "action": "http_blacklist_enable", + "body": { + "...": "按厂商文档填写目标 ID 与启停状态" + } } ``` -`logaccess_check` 用来诊断外发链路是否正常。 - -### 16. 基本信息 / license / MDR - -- `basic_information` —— 设备基础信息 -- `basic_information_enable` —— 启用模块(写) -- `basic_information_import` —— 离线情报库更新(multipart:`file_path` + Query 需 `name`) -- `basic_license_get` —— 查 license -- `basic_license_upload` —— 授权 license 文件上传(multipart:`file_path`) -- `mdr_service_status` / `mdr_service_enable` —— MDR 服务 - -### 17. 设备诊断 - -- `device_coredump_list` / `_download` / `_delete` -- `device_pcap_get` / `_set` / `_file_list` / `_download` / `_file_delete` - -抓包写 (`device_pcap_set`) 与删除 (`*_delete`) 是写操作。 - -### 18. 帮助文档与产品反馈 - -```json -{ "action": "document_list", "pageNo": 1, "pageSize": 20, "search": "ips" } -``` - -```json -{ "action": "document_preview", "id": "<文档列表项里的 id>" } -``` - -⚠️ `document_preview` 必填 Query `id`(**不是 `fileName`**),其值来自 `document_list` 返回的 `data.list[].id`。返回的 `data` 是**路径字符串**(不是对象),前端会拼到 `window.location.origin` 后用 `window.open` 打开 —— agent 拿到这个路径后通常需要走浏览器模式才能查看 PDF/HTML 内容。 - -`product_news_get` / `product_news_mark_read` / `product_version` / `product_issue` 用于看红点 / 标已读 / 看版本 / 提反馈。其中 `product_news_mark_read` 的 body 应是先 `product_news_get` 返回对象的副本,再把要清除的标记位(如 `documentUpdate`)置 `false`,其它字段保持原值。 - -### 19. 登录 / 会话 - -绝大多数业务场景**不需要**显式登录 —— 处理器会按需自动登录、Cookie 过期会自动重登。下面这些场景才需要主动调 `onesig_login`: - -- 设备启用了图形验证码(`GET /v3/captcha` 返回 `enableCaptcha=true`): - - ```json - { "action": "login", "captcha": "xyzw" } - ``` - - 前端校验长度恰为 4。 - -- 设备启用了 TOTP / 双因素(`enableTotp=true` 或 `POST /v3/login` 返回 `responseCode=1012` 进入扫码页): - - ```json - { "action": "login", "totp": "123456" } - ``` - - 注意:handler 入参统一叫 `totp`(**不要传 `checksum`**),底层会按场景自动映射 —— inline 模式(`/v3/login` 同屏 TOTP)作为 `checksum` 字段拼进登录 body;扫码模式(先 `/v3/login` 拿到 `responseCode=1012`、再补一次 `/v3/login/totp`)作为 `{"checksum": "..."}` 体提交。也可以传恢复码(最长 12 字符)。 - -- 想立刻拿账户信息: - - ```json - { "action": "get_account" } - ``` - -- 改当前用户密码: - - ```json - { - "action": "change_password", - "old_password": "<明文旧>", - "new_password": "<明文新>", - "dup_password": "<明文新>" - } - ``` - -- 显式退出(调试 / 切换账号): - - ```json - { "action": "logout" } - ``` - -注意: - -- `logout` 会清空内存里的会话**和**已落盘的 Cookie 持久化(`~/.flocks/config/.secret.json` 中以 `onesig_session_cookie__` 命名的条目),下次调用会从 captcha → pubkey → /v3/login 重走完整链路 -- `regenerate_recovery_code` 一旦调用,旧恢复码立即失效,请在用户明确授权后再用 -- `get_pubkey` / `get_captcha` 通常不需要手动调 —— 处理器在登录前与每次发送 RSA 加密字段前都会自动拉 - -## 文件类返回 / 上传 - -下列 action 返回的是**二进制文件**(导出 / 模板 / 下载),不应该在 chat 里直接展开: - -- 导出:`asset_export` / `whitelist_export` / `blacklist_export` / `http_blacklist_export` / `port_protect_port_export` / `event_inbound_export` / `event_inbound_detail_export` / `event_outbound_export` / `event_outbound_detail_export` / `alert_host_export` / `alert_host_detail_export` / `tls_detect_group_export` / `tls_detect_list_export` / `aclog_export` / `user_export` / `alert_policy_export` / `multiblock_executelog_export` / `overview_export_*` -- 模板:`asset_template` / `whitelist_template` / `blacklist_template` / `linkage_template` -- 下载:`backup_download` / `report_form_download`(必填 `uniqueId`+`fileName`)/ `device_coredump_download` / `device_pcap_download` - -下列 action 是 **multipart 上传**,需要 `file_path` 指向本地绝对路径: - -- `asset_import`(CSV) -- `tls_cert_create` / `tls_cert_update`(`.crt`/`.pem`) -- `basic_information_import`(离线情报库包,Query 需 `name`) -- `basic_license_upload` / `system_upgrade` / `device_upgrade_upload`(升级 / 授权包) -- `backup_import`(备份包) +`http_blacklist_enable` 会改变防护行为,视为写操作。 ## 高风险写操作清单 -以下 action 默认视为高风险,agent 在执行前**必须**先确认用户授权: - -- `device_onekey_bypass` / `device_quick_bypass` -- `device_reboot` / `device_shutdown` / `device_reinit` -- `device_upgrade` / `device_upgrade_upload` / `system_upgrade` -- `ha_switching` / `ha_sync_config`(同步会改对端) -- `set_ha_config` / `set_onecc_config` / `set_onecc_status` -- `user_create` / `user_delete` / `user_update` / `user_secret_reset` / `change_password` -- `aclog_delete` -- `whitelist_add` / `whitelist_update` / `whitelist_delete` / `whitelist_remove_batch` / `whitelist_import` -- `blacklist_add` / `blacklist_update` / `blacklist_delete` / `blacklist_remove_batch` / `blacklist_import` -- `multiblock_rule_create` / `multiblock_rule_update` / `multiblock_rule_delete` / `multiblock_rule_active` -- `auto_blacklist_create` / `auto_blacklist_update` / `auto_blacklist_delete` -- `linkage_create` / `linkage_update` / `linkage_delete` / `linkage_enable` -- `ips_rule_create` / `ips_rule_apply` / `ips_rule_all` / `ips_ruleset_create` / `ips_ruleset_update` / `ips_ruleset_delete` -- `http_blacklist_create` / `http_blacklist_update` / `http_blacklist_delete` / `http_blacklist_enable` -- `port_protect_group_create` / `_update` / `_delete` / `_clone`、`port_protect_port_create` / `_update` / `_delete` / `_onekey_import` -- `protection_policy_update` / `protection_policy_delete` -- `tls_decrypt_policy_create` / `_update` / `_enable` / `_delete` / `_batch` -- `tls_cert_create` / `_update` / `_delete` / `_set_default`、`tls_detect_delete` -- `set_decrypt_config` / `set_detect_config` -- `interface_update`(启停)/ `interface_*_create/update/delete` -- `route_static_create` / `_update` / `_delete`、`ipv6_route_static_create` / `_update` / `_delete` -- `set_dns_config` / `hosts_create` / `_update` / `_delete`、`set_proxy_config` -- `backup_create` / `backup_recover` / `backup_delete` / `backup_update` / `backup_import` -- `set_storage_config` / `set_clean_config` / `set_dnslog_config` -- `logaccess_create` / `_update` / `_delete` -- `basic_information_enable` / `basic_information_import` / `basic_license_upload` -- `mdr_service_enable` -- `device_pcap_set` / `device_coredump_delete` / `device_pcap_file_delete` -- `apikey_create` / `apikey_update` / `apikey_delete` / `apikey_secret`(回吐明文 secret,需 `password`) -- `regenerate_recovery_code` -- `report_form_create` / `_delete`、`report_task_create` / `_update` / `_delete` / `_test` -- `set_advanced_config` / `set_xff_config` / `set_scan_config` / `set_login_config` / `set_upgrade_config` / `set_overview_config` / `set_custom_config` / `web_custom_column_set` - -## 常见失败原因 - -- 时间戳传成毫秒(要用秒) -- 错传 `cur_page` / `page_size` / `page_items_num`(应为 `pageNo` / `pageSize`) -- 漏传 OneSIG 必填的"业务 ID"主键 —— `name` / `uniqueId` / `srcIp` / `groupName` / `server`+`port` / `ruleId`+`assetIp` 任意一个缺失都会回 `responseCode=1004`(请求数据非法) -- 单条 GET 类接口忘了先调列表拿主键 -- 改密 / 删除 / 启停接口手动加密了 `password` —— 处理器会自己加密,要传**明文** -- `oaep_hash` 配置和设备实际不匹配(多数 v2.5.x 走 SHA-1,少数走 SHA-256),登录返回 `responseCode=1009`/`1017` 时优先怀疑这里 -- `api_prefix` 与设备反代部署形态不一致 —— 厂商前端代码里写的是 `/api/v3/...`,但接口规范文档里去掉了 `/api`。我们 v2.5.x 实测的实例多数是直连后端(`/v3/...`),所以 handler 的 `DEFAULT_API_PREFIX=""`;如果换到一个走 nginx 代理的实例,需要把 `api_prefix` 设成 `"/api"`。**现象**:登录前的 `/v3/pubkey` 直接 404,handler 会在错误信息里给出对应提示 -- `verify_ssl` 与设备证书不匹配(OneSIG 多用自签证书,默认应**关闭**)—— 现象是 SSL handshake 失败 -- 报错"Cookie 过期":调一下 `onesig_login` 的 `logout` 再让处理器自动重登 -- API Key 写操作返回 `responseCode=1004` 提示 `type` 缺失:服务端要求 Query `type=physical`;先尝试把 `type:"physical"` 放进 body 透传,如果仍然 1004 请回退浏览器模式 +以下 action 默认视为高风险,agent 在执行前必须先确认用户授权、对象、范围和回滚预案: + +- `asset_group_create` +- `asset_group_update` +- `asset_group_delete` +- `asset_create` +- `asset_update` +- `asset_delete` +- `protection_policy_update` +- `protection_policy_delete` +- `whitelist_create` +- `whitelist_update` +- `whitelist_delete` +- `whitelist_remove` +- `blacklist_create` +- `blacklist_update` +- `blacklist_delete` +- `blacklist_remove` +- `banned_whitelist_create` +- `banned_whitelist_update` +- `banned_whitelist_delete` +- `http_blacklist_create` +- `http_blacklist_update` +- `http_blacklist_enable` +- `http_blacklist_delete` + +## 返回与错误处理 + +- handler 会把成功响应中的 `data` 解包为 tool output;如果响应不是对象,会原样返回。 +- 如果响应包含 `response_code` 且不是 `0`,handler 会把结果标记为失败,并返回 `verbose_msg` 或原始响应。 +- HTTP `4xx` / `5xx` 会返回状态码和前 500 字符响应体。 +- `OneSIG Strategy API base_url is not configured.` 表示服务地址未配置。 +- `OneSIG Strategy API ApiKey and Secret are required.` 表示 ApiKey / Secret 未配置或 secret 引用无法解析。 +- `Request failed` 常见原因是设备网络不可达、证书验证配置不匹配或代理 / 防火墙拦截。 + +## 当前 Strategy API 不覆盖的场景 + +以下任务不要调用当前两个 Strategy API 工具硬凑,应改用浏览器模式,或在确认存在额外旧控制台工具并阅读对应文档后再处理: + +- 威胁监控、仪表盘、威胁防护大屏 +- 入站 / 出站威胁事件、失陷主机、报告管理、导出报表 +- 多维封锁、API 联动密钥管理、Syslog 自动封禁、FTP/SFTP 联动 +- IPS 规则 / 规则集、高危端口防护 +- 告警通知、审计日志、用户管理、登录 / 改密 +- HTTPS 解密、网口、部署引导、路由、DNS、代理 +- HA、OneCC、设备升级 / 重启 / 备份 / 恢复 +- 日志外发、license、MDR、coredump、pcap、帮助文档、产品反馈 +- 图形验证码、TOTP、强制改密等人工交互 ## 何时回退浏览器 以下情况优先回退浏览器(参考 [browser-workflow.md](browser-workflow.md)): -- 需要威胁防护大屏的可视化、IOC 关系图、攻击链 -- 需要看事件 / 失陷主机的页面级详情、报文 hex view、PCAP 在线播放 -- 需要点表格右侧抽屉、复杂筛选弹窗、一些深层报表预览 +- 当前 Strategy API 没有覆盖用户目标能力 +- 需要页面级图表、详情抽屉、攻击链、威胁图、报表预览或文件下载 - 需要图形验证码 / TOTP / 强制改密之类的人工交互 -- 需要看 Web 控制台原生导出文件并下载 +- API 未配置、认证失败、设备不可达或反复返回权限 / 签名错误 +- 用户明确要求使用浏览器 diff --git a/.flocks/plugins/skills/onesig-use/references/browser-workflow.md b/.flocks/plugins/skills/onesig-use/references/browser-workflow.md index 30ee77dfd..dbf3f1590 100644 --- a/.flocks/plugins/skills/onesig-use/references/browser-workflow.md +++ b/.flocks/plugins/skills/onesig-use/references/browser-workflow.md @@ -2,12 +2,12 @@ 只在以下情况进入浏览器模式: -- API 不可用(未配置 / 未开通 / 认证失败 / Cookie 持久反复过期 / SSL 校验失败 / 网络不通) +- API 不可用(未配置 / 未开通 / ApiKey 或 Secret 缺失 / 签名失败 / SSL 校验失败 / 网络不通) - 任务必须看页面级详情(攻击链、威胁图、报表预览、报文 hex view、PCAP 在线播放) - 需要图形验证码 / TOTP / 强制改密之类的人工交互 - 用户明确要求使用浏览器,或者已经在浏览器操作过程中 -如果走 API 能完成,请回到 [api-reference.md](api-reference.md) —— 浏览器操作不稳定、不可批量、字段不一定完整。 +如果当前 Strategy API 能完成,请回到 [api-reference.md](api-reference.md);浏览器操作不稳定、不可批量、字段不一定完整。 > ⚠️ OneSIG 的 Web 控制台与 OneSEC、青藤是不同产品;不要把 OneSEC / 青藤的页面路径或 API 套用到 OneSIG。 > 本文档统一按 `browser-use` 的 `cdp-direct` 流程执行:先 `flocks browser --doctor`,doctor 通过后只使用 `flocks browser`。 @@ -145,14 +145,17 @@ print(js("document.body.innerText.slice(0, 2000)")) ## 二、浏览器与 API 的互补建议 -进入浏览器模式后,对于"查询类"诉求,应该**优先回到 API**(除非 API 真的不可用): +进入浏览器模式后,只有当前 Strategy API 覆盖的诉求才优先回到 API;未覆盖的控制台能力继续走浏览器: | 任务 | 优先方案 | |---|---| -| 列威胁事件 / 失陷主机 / 趋势数据 | API(`onesig_monitoring`) —— 浏览器只在需要威胁图、报文 hex 时用 | -| 增删改黑白名单 / IPS 规则 / 多维封锁 | API(`onesig_strategy`),写操作前要二次确认 | -| 看资产清单、增改资产 | API(`onesig_assets`) —— 浏览器仅做导入文件预览 | -| 设备升级 / 重启 / HA 切换 | 浏览器 + API 并用:先在浏览器里走完确认弹窗、备份提示,再用 API 触发;或者全程在浏览器里完成(更稳) | +| 看设备平台 / 系统 / 网络状态 | API(`onesig_strategy_api_query`) | +| 看资产组、资产列表、资产类型 | API(`onesig_strategy_api_query`) | +| 增删改资产或资产组 | API(`onesig_strategy_api_ops`),写操作前要二次确认 | +| 看 / 改防护策略、全局白名单、全局黑名单、封禁白名单、HTTP 黑名单 | API(`onesig_strategy_api_query` / `onesig_strategy_api_ops`),写操作前要二次确认 | +| 列威胁事件 / 失陷主机 / 趋势数据 | 浏览器(当前 Strategy API 不覆盖) | +| IPS 规则、多维封锁、Syslog 自动封禁、FTP/SFTP 联动、高危端口防护 | 浏览器(当前 Strategy API 不覆盖) | +| 设备升级 / 重启 / HA 切换 | 浏览器;强破坏性动作必须显式确认 | | 看证书 / 解密策略详情 | 浏览器(API 返回的字段比页面少) | | 看页面级图表 / 攻击链 / IOC 关联 | 浏览器(API 没有) | | 处理图形验证码 / TOTP / 强制改密 | 浏览器(API 不能完成人工交互) | @@ -175,10 +178,9 @@ print(js("document.body.innerText.slice(0, 2000)")) ## 四、文件下载与导出 -OneSIG 控制台的导出按钮(资产 / 黑白名单 / 报表 / 审计 / coredump / pcap)多数会触发浏览器下载。优先顺序如下: +OneSIG 控制台的导出按钮(资产 / 黑白名单 / 报表 / 审计 / coredump / pcap)多数会触发浏览器下载。当前 Strategy API 工具没有导出 / 下载 action;需要导出文件时走浏览器页面。 -1. **优先用 API 的导出 / 下载 action**(参见 [api-reference.md](api-reference.md) "文件类返回 / 上传"小节)。 -2. 如果必须走页面下载,先用 `js(...)` 或稳定 selector 触发下载,再用浏览器系统的下载目录或页面提示确认下载已开始;不要在 skill 里承诺不存在的“等待下载完成”专有命令。 +先用 `js(...)` 或稳定 selector 触发下载,再用浏览器系统的下载目录或页面提示确认下载已开始;不要在 skill 里承诺不存在的“等待下载完成”专有命令。 页面下载最小模板: From 3489fb84dea029665875649bf96a9235e1ab82ab Mon Sep 17 00:00:00 2001 From: duguwanglong Date: Thu, 9 Jul 2026 11:12:32 +0800 Subject: [PATCH 02/49] fix(webui): serve static assets with stable MIME types --- flocks/server/static_webui.py | 8 +++++++- tests/server/test_static_webui.py | 3 +++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/flocks/server/static_webui.py b/flocks/server/static_webui.py index dede61320..20c70a0d6 100644 --- a/flocks/server/static_webui.py +++ b/flocks/server/static_webui.py @@ -14,6 +14,12 @@ _ASSET_CACHE_CONTROL = "public, max-age=31536000, immutable" _STATIC_CACHE_CONTROL = "no-cache" _FINGERPRINT_RE = re.compile(r"(?:^|[.-])[0-9a-f]{8,}(?:[.-]|$)", re.IGNORECASE) +_STATIC_MEDIA_TYPES = { + ".css": "text/css", + ".js": "application/javascript", + ".mjs": "application/javascript", + ".wasm": "application/wasm", +} _PROTECTED_PREFIXES = ( "/api", "/event", @@ -107,7 +113,7 @@ def _resolve_existing_static_file(dist_dir: Path, path: str) -> Path | None: def _file_response(path: Path, *, cache_control: str) -> FileResponse: headers = {"Cache-Control": cache_control} - return FileResponse(path, headers=headers) + return FileResponse(path, headers=headers, media_type=_STATIC_MEDIA_TYPES.get(path.suffix.lower())) def _cache_control_for_file(path: str, file_path: Path) -> str: diff --git a/tests/server/test_static_webui.py b/tests/server/test_static_webui.py index 8e478c5ac..4c918815f 100644 --- a/tests/server/test_static_webui.py +++ b/tests/server/test_static_webui.py @@ -1,3 +1,4 @@ +import mimetypes from pathlib import Path import pytest @@ -47,10 +48,12 @@ async def test_static_webui_serves_browser_root(monkeypatch, tmp_path: Path) -> @pytest.mark.asyncio async def test_static_webui_serves_assets_with_immutable_cache(monkeypatch, tmp_path: Path) -> None: monkeypatch.setenv("FLOCKS_WEBUI_DIST_DIR", str(_write_dist(tmp_path))) + monkeypatch.setattr(mimetypes, "guess_type", lambda *_args, **_kwargs: ("text/plain", None)) async with AsyncClient(transport=ASGITransport(app=_app()), base_url="http://test") as client: response = await client.get("/assets/app.12345678.js") assert response.status_code == 200 + assert response.headers["Content-Type"].startswith("application/javascript") assert "console.log" in response.text assert response.headers["Cache-Control"] == "public, max-age=31536000, immutable" From 37b96e77c0d868ae08a711796b5f44ba486bdbe9 Mon Sep 17 00:00:00 2001 From: duguwanglong Date: Thu, 9 Jul 2026 16:25:19 +0800 Subject: [PATCH 03/49] perf(webui): improve responsiveness under load Add shared frontend resources for common data hooks, consolidate SessionChat feature logic, lazy-load heavier UI paths, and bound SSE fan-out/backpressure so concurrent sessions and workflows create less browser and server load. --- flocks/server/routes/event.py | 66 +- tests/server/routes/test_event_routes.py | 44 + .../common/ChatPromptSelectors.test.tsx | 105 ++- .../components/common/ChatPromptSelectors.tsx | 63 +- .../components/common/EntitySheet.test.tsx | 4 +- webui/src/components/common/EntitySheet.tsx | 158 ++-- .../src/components/common/SessionChat.test.ts | 153 +++- webui/src/components/common/SessionChat.tsx | 856 +++++++++--------- .../components/common/sessionChatDisplay.ts | 6 + webui/src/components/layout/Layout.tsx | 20 +- .../features/session-chat/commands.test.ts | 78 ++ webui/src/features/session-chat/commands.ts | 50 + .../src/features/session-chat/display.test.ts | 16 + webui/src/features/session-chat/display.ts | 33 + webui/src/features/session-chat/index.ts | 31 + .../features/session-chat/sseActions.test.ts | 165 ++++ webui/src/features/session-chat/sseActions.ts | 160 ++++ .../features/session-chat/sseRouting.test.ts | 50 + webui/src/features/session-chat/sseRouting.ts | 26 + .../session-chat/usePendingQuestions.ts | 120 +++ .../useSessionContextUsage.test.ts | 100 ++ .../session-chat/useSessionContextUsage.ts | 118 +++ .../useSessionPromptQueue.test.ts | 195 ++++ .../session-chat/useSessionPromptQueue.ts | 148 +++ webui/src/hooks/useAgents.test.tsx | 31 +- webui/src/hooks/useAgents.ts | 106 ++- webui/src/hooks/useBackendStatus.test.tsx | 100 ++ webui/src/hooks/useBackendStatus.ts | 152 ++-- webui/src/hooks/useChatModelResources.ts | 67 ++ webui/src/hooks/useDefaultModelVision.test.ts | 76 +- webui/src/hooks/useDefaultModelVision.ts | 103 ++- webui/src/hooks/useHooks.test.tsx | 82 ++ webui/src/hooks/useHooks.ts | 37 +- webui/src/hooks/usePendingQuestions.ts | 129 +-- webui/src/hooks/useProviders.test.tsx | 122 +++ webui/src/hooks/useProviders.ts | 114 ++- webui/src/hooks/useSSE.test.tsx | 62 ++ webui/src/hooks/useSSE.ts | 434 ++++++--- webui/src/hooks/useSessions.test.ts | 30 + webui/src/hooks/useSessions.ts | 51 +- webui/src/hooks/useSharedResource.test.tsx | 72 ++ webui/src/hooks/useSharedResource.ts | 229 +++++ webui/src/hooks/useStats.test.tsx | 109 +++ webui/src/hooks/useStats.ts | 90 +- webui/src/hooks/useTasks.test.tsx | 175 ++++ webui/src/hooks/useTasks.ts | 510 ++++++----- webui/src/hooks/useTools.test.tsx | 32 +- webui/src/hooks/useTools.ts | 99 +- .../src/hooks/useWebUIContractPages.test.tsx | 37 +- webui/src/hooks/useWebUIContractPages.ts | 113 ++- webui/src/hooks/useWorkflow.test.tsx | 42 +- webui/src/hooks/useWorkflow.ts | 109 +-- webui/src/i18n.test.ts | 17 + webui/src/i18n.ts | 117 +-- webui/src/i18nResources.ts | 178 ++++ .../pages/DeviceIntegration/index.test.tsx | 2 +- webui/src/pages/DeviceIntegration/index.tsx | 336 +++---- webui/src/pages/Session/index.test.tsx | 100 +- webui/src/pages/Session/index.tsx | 523 ++++++----- webui/src/pages/Settings/index.tsx | 23 +- webui/src/pages/Skill/SkillSheet.tsx | 17 +- webui/src/pages/Task/QueuedSection.tsx | 38 +- webui/src/pages/Workspace/index.tsx | 41 +- webui/src/routes/index.tsx | 64 +- 64 files changed, 5545 insertions(+), 1989 deletions(-) create mode 100644 tests/server/routes/test_event_routes.py create mode 100644 webui/src/components/common/sessionChatDisplay.ts create mode 100644 webui/src/features/session-chat/commands.test.ts create mode 100644 webui/src/features/session-chat/commands.ts create mode 100644 webui/src/features/session-chat/display.test.ts create mode 100644 webui/src/features/session-chat/display.ts create mode 100644 webui/src/features/session-chat/index.ts create mode 100644 webui/src/features/session-chat/sseActions.test.ts create mode 100644 webui/src/features/session-chat/sseActions.ts create mode 100644 webui/src/features/session-chat/sseRouting.test.ts create mode 100644 webui/src/features/session-chat/sseRouting.ts create mode 100644 webui/src/features/session-chat/usePendingQuestions.ts create mode 100644 webui/src/features/session-chat/useSessionContextUsage.test.ts create mode 100644 webui/src/features/session-chat/useSessionContextUsage.ts create mode 100644 webui/src/features/session-chat/useSessionPromptQueue.test.ts create mode 100644 webui/src/features/session-chat/useSessionPromptQueue.ts create mode 100644 webui/src/hooks/useBackendStatus.test.tsx create mode 100644 webui/src/hooks/useChatModelResources.ts create mode 100644 webui/src/hooks/useHooks.test.tsx create mode 100644 webui/src/hooks/useProviders.test.tsx create mode 100644 webui/src/hooks/useSharedResource.test.tsx create mode 100644 webui/src/hooks/useSharedResource.ts create mode 100644 webui/src/hooks/useStats.test.tsx create mode 100644 webui/src/hooks/useTasks.test.tsx create mode 100644 webui/src/i18n.test.ts create mode 100644 webui/src/i18nResources.ts diff --git a/flocks/server/routes/event.py b/flocks/server/routes/event.py index b5b5b5350..8b0ca8433 100644 --- a/flocks/server/routes/event.py +++ b/flocks/server/routes/event.py @@ -33,6 +33,11 @@ "context.", "permission.", ) +EVENT_QUEUE_MAXSIZE = max(100, int(os.getenv("FLOCKS_EVENT_QUEUE_MAXSIZE", "1000"))) +EVENT_QUEUE_DROP_TO = max(0, min( + EVENT_QUEUE_MAXSIZE - 1, + int(os.getenv("FLOCKS_EVENT_QUEUE_DROP_TO", str(EVENT_QUEUE_MAXSIZE // 2))), +)) # Current directory context for SSE events @@ -56,9 +61,17 @@ class EventBroadcaster: _instance: Optional["EventBroadcaster"] = None - def __init__(self): + def __init__(self, queue_maxsize: int = EVENT_QUEUE_MAXSIZE, queue_drop_to: Optional[int] = None): self._clients: list[asyncio.Queue] = [] self._lock = asyncio.Lock() + self._queue_maxsize = max(1, queue_maxsize) + self._queue_drop_to = max( + 0, + min( + self._queue_maxsize - 1, + EVENT_QUEUE_DROP_TO if queue_drop_to is None else queue_drop_to, + ), + ) @classmethod def get(cls) -> "EventBroadcaster": @@ -69,7 +82,7 @@ def get(cls) -> "EventBroadcaster": async def subscribe(self) -> asyncio.Queue: """Subscribe a new client""" - queue: asyncio.Queue = asyncio.Queue() + queue: asyncio.Queue = asyncio.Queue(maxsize=self._queue_maxsize) async with self._lock: self._clients.append(queue) return queue @@ -83,11 +96,45 @@ async def unsubscribe(self, queue: asyncio.Queue): async def publish(self, event: dict): """Publish event to all clients""" async with self._lock: - for queue in self._clients: - try: - await queue.put(event) - except Exception: - pass # Ignore errors for disconnected clients + clients = list(self._clients) + for queue in clients: + self._publish_to_queue(queue, event) + + def _publish_to_queue(self, queue: asyncio.Queue, event: dict): + try: + queue.put_nowait(event) + return + except asyncio.QueueFull: + pass + + dropped = 0 + while queue.qsize() > self._queue_drop_to: + try: + queue.get_nowait() + dropped += 1 + except asyncio.QueueEmpty: + break + + try: + queue.put_nowait(create_event("server.events_dropped", { + "dropped": dropped, + "reason": "client_backpressure", + })) + except asyncio.QueueFull: + pass + + try: + queue.put_nowait(event) + except asyncio.QueueFull: + dropped += 1 + + if dropped > 0: + log.debug("event.queue.overflow", { + "dropped": dropped, + "queue_size": queue.qsize(), + "queue_maxsize": self._queue_maxsize, + "event_type": event.get("type"), + }) @property def client_count(self) -> int: @@ -99,10 +146,7 @@ async def shutdown(self): shutdown_event = create_event("server.shutting_down", {}) async with self._lock: for queue in self._clients: - try: - await queue.put(shutdown_event) - except Exception: - pass + self._publish_to_queue(queue, shutdown_event) self._clients.clear() log.info("event.broadcaster.shutdown", {"clients_notified": True}) diff --git a/tests/server/routes/test_event_routes.py b/tests/server/routes/test_event_routes.py new file mode 100644 index 000000000..547f2cad1 --- /dev/null +++ b/tests/server/routes/test_event_routes.py @@ -0,0 +1,44 @@ +import pytest + +from flocks.server.routes.event import EventBroadcaster + + +@pytest.mark.asyncio +async def test_event_broadcaster_compacts_overflowing_client_queue(): + broadcaster = EventBroadcaster(queue_maxsize=3, queue_drop_to=1) + queue = await broadcaster.subscribe() + + for index in range(5): + await broadcaster.publish({ + "type": "message.part.updated", + "properties": {"index": index}, + }) + + assert queue.qsize() <= 3 + + events = [] + while not queue.empty(): + events.append(queue.get_nowait()) + + assert any(event["type"] == "server.events_dropped" for event in events) + assert events[-1] == { + "type": "message.part.updated", + "properties": {"index": 4}, + } + + +@pytest.mark.asyncio +async def test_event_broadcaster_shutdown_does_not_block_on_full_queue(): + broadcaster = EventBroadcaster(queue_maxsize=2, queue_drop_to=0) + queue = await broadcaster.subscribe() + + await broadcaster.publish({"type": "event.one", "properties": {}}) + await broadcaster.publish({"type": "event.two", "properties": {}}) + await broadcaster.shutdown() + + events = [] + while not queue.empty(): + events.append(queue.get_nowait()) + + assert broadcaster.client_count == 0 + assert any(event["type"] == "server.shutting_down" for event in events) diff --git a/webui/src/components/common/ChatPromptSelectors.test.tsx b/webui/src/components/common/ChatPromptSelectors.test.tsx index 147ad560c..60ea93cbe 100644 --- a/webui/src/components/common/ChatPromptSelectors.test.tsx +++ b/webui/src/components/common/ChatPromptSelectors.test.tsx @@ -1,8 +1,32 @@ -import { render, screen } from '@testing-library/react'; +import { act, render, renderHook, screen, waitFor } from '@testing-library/react'; import userEvent from '@testing-library/user-event'; -import { describe, expect, it, vi } from 'vitest'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; -import { ChatModelPicker, type ChatModelProviderGroup } from './ChatPromptSelectors'; +import { + __resetChatModelOptionsResourcesForTesting, + ChatModelPicker, + useChatModelOptions, + type ChatModelProviderGroup, +} from './ChatPromptSelectors'; + +const { listDefinitionsMock, getResolvedMock, useProvidersMock } = vi.hoisted(() => ({ + listDefinitionsMock: vi.fn(), + getResolvedMock: vi.fn(), + useProvidersMock: vi.fn(), +})); + +vi.mock('@/api/provider', () => ({ + modelV2API: { + listDefinitions: listDefinitionsMock, + }, + defaultModelAPI: { + getResolved: getResolvedMock, + }, +})); + +vi.mock('@/hooks/useProviders', () => ({ + useProviders: useProvidersMock, +})); vi.mock('react-i18next', () => ({ useTranslation: () => ({ @@ -40,6 +64,45 @@ const groupedOptions: ChatModelProviderGroup[] = [ }, ]; +function makeProvider(id: string) { + return { + id, + name: id, + source: 'builtin', + env: [], + key: null, + options: {}, + models: {}, + configured: true, + modelCount: 1, + category: 'connected', + }; +} + +function makeModelDefinition(providerId = 'provider-1', modelId = 'model-1') { + return { + id: modelId, + name: modelId, + provider_id: providerId, + model_type: 'chat', + status: 'active', + capabilities: { supports_vision: false }, + limits: { context_window: 128000 }, + }; +} + +beforeEach(() => { + vi.clearAllMocks(); + __resetChatModelOptionsResourcesForTesting(); + useProvidersMock.mockReturnValue({ + providers: [makeProvider('provider-1')], + loading: false, + error: null, + connectedIds: ['provider-1'], + refetch: vi.fn(), + }); +}); + describe('ChatModelPicker', () => { it('opens the model menu toward the left edge of the trigger', async () => { const user = userEvent.setup(); @@ -62,3 +125,39 @@ describe('ChatModelPicker', () => { expect(menu).not.toHaveClass('left-0'); }); }); + +describe('useChatModelOptions', () => { + it('shares enabled model and default model requests across concurrent hook instances', async () => { + let resolveDefinitions: (value: { data: { models: any[] } }) => void = () => {}; + listDefinitionsMock.mockReturnValue(new Promise((resolve) => { + resolveDefinitions = resolve; + })); + getResolvedMock.mockResolvedValue({ + data: { provider_id: 'provider-1', model_id: 'model-1' }, + }); + + const first = renderHook(() => useChatModelOptions()); + const second = renderHook(() => useChatModelOptions()); + + expect(listDefinitionsMock).toHaveBeenCalledTimes(1); + + await act(async () => { + resolveDefinitions({ + data: { models: [makeModelDefinition()] }, + }); + }); + + await waitFor(() => { + expect(first.result.current.loading).toBe(false); + expect(second.result.current.loading).toBe(false); + expect(first.result.current.options).toHaveLength(1); + expect(second.result.current.options).toHaveLength(1); + }); + + await waitFor(() => { + expect(first.result.current.selectedModelKey).toBe('provider-1::model-1'); + expect(second.result.current.selectedModelKey).toBe('provider-1::model-1'); + }); + expect(getResolvedMock).toHaveBeenCalledTimes(1); + }); +}); diff --git a/webui/src/components/common/ChatPromptSelectors.tsx b/webui/src/components/common/ChatPromptSelectors.tsx index bcf576204..0df85d411 100644 --- a/webui/src/components/common/ChatPromptSelectors.tsx +++ b/webui/src/components/common/ChatPromptSelectors.tsx @@ -3,7 +3,11 @@ import { Bot, ChevronDown, Cpu, Info } from 'lucide-react'; import { useTranslation } from 'react-i18next'; import type { Agent } from '@/api/agent'; -import { defaultModelAPI, modelV2API } from '@/api/provider'; +import { + __resetChatModelResourcesForTesting, + useEnabledChatModelDefinitions, + useResolvedDefaultModel, +} from '@/hooks/useChatModelResources'; import { useAgents } from '@/hooks/useAgents'; import { useProviders } from '@/hooks/useProviders'; import { getAgentDisplayDescription, getAgentDisplayName, isAgentUsableInChat } from '@/utils/agentDisplay'; @@ -36,6 +40,10 @@ type SelectorTooltip = { y: number; }; +export function __resetChatModelOptionsResourcesForTesting(): void { + __resetChatModelResourcesForTesting(); +} + function formatAgentName(name: string): string { return name ? name.charAt(0).toUpperCase() + name.slice(1) : name; } @@ -68,28 +76,12 @@ export function useChatAgentOptions(options: { allowedAgentNames?: string[] } = export function useChatModelOptions() { const { t } = useTranslation('session'); const { providers, loading: loadingProviders } = useProviders(); - const [enabledModelDefinitions, setEnabledModelDefinitions] = useState([]); - const [loadingEnabledModels, setLoadingEnabledModels] = useState(true); + const { + data: enabledModelDefinitions, + loading: loadingEnabledModels, + } = useEnabledChatModelDefinitions(); const [selectedModelKey, setSelectedModelKey] = useState(null); - useEffect(() => { - let cancelled = false; - setLoadingEnabledModels(true); - Promise.resolve(modelV2API.listDefinitions({ enabled_only: true })) - .then((response) => { - if (!cancelled) setEnabledModelDefinitions(response?.data?.models ?? []); - }) - .catch(() => { - if (!cancelled) setEnabledModelDefinitions([]); - }) - .finally(() => { - if (!cancelled) setLoadingEnabledModels(false); - }); - return () => { - cancelled = true; - }; - }, []); - const options = useMemo(() => { const providerById = new Map( providers @@ -162,24 +154,19 @@ export function useChatModelOptions() { [options, selectedModelKey], ); + const { + data: resolvedDefaultModel, + initialized: resolvedDefaultModelInitialized, + } = useResolvedDefaultModel(options.length > 0); + useEffect(() => { - if (selectedModelKey || options.length === 0) return; - let cancelled = false; - Promise.resolve(defaultModelAPI.getResolved()) - .then((response) => { - if (cancelled) return; - const { provider_id: providerID, model_id: modelID } = response?.data ?? {}; - const defaultKey = `${providerID}::${modelID}`; - const fallbackKey = options[0]?.key ?? null; - setSelectedModelKey(options.some((option) => option.key === defaultKey) ? defaultKey : fallbackKey); - }) - .catch(() => { - if (!cancelled) setSelectedModelKey(options[0]?.key ?? null); - }); - return () => { - cancelled = true; - }; - }, [options, selectedModelKey]); + if (selectedModelKey || options.length === 0 || !resolvedDefaultModelInitialized) return; + const defaultKey = resolvedDefaultModel + ? `${resolvedDefaultModel.providerID}::${resolvedDefaultModel.modelID}` + : null; + const fallbackKey = options[0]?.key ?? null; + setSelectedModelKey(defaultKey && options.some((option) => option.key === defaultKey) ? defaultKey : fallbackKey); + }, [options, resolvedDefaultModel, resolvedDefaultModelInitialized, selectedModelKey]); useEffect(() => { if (loadingEnabledModels || options.length === 0 || !selectedModelKey) return; diff --git a/webui/src/components/common/EntitySheet.test.tsx b/webui/src/components/common/EntitySheet.test.tsx index 99a8ac78f..ffde9c541 100644 --- a/webui/src/components/common/EntitySheet.test.tsx +++ b/webui/src/components/common/EntitySheet.test.tsx @@ -178,14 +178,14 @@ describe('EntitySheet', () => { expect(screen.getByText('Form content')).toBeInTheDocument(); }); - it('defaults Rex workbench process details collapsed like workflow workbenches', () => { + it('defaults Rex workbench process details collapsed like workflow workbenches', async () => { render(
Form content
, ); - expect(screen.getByTestId('session-chat')).toHaveAttribute( + expect(await screen.findByTestId('session-chat')).toHaveAttribute( 'data-display', JSON.stringify({ collapseIntermediateSteps: true, processGroupsDefaultOpen: false }), ); diff --git a/webui/src/components/common/EntitySheet.tsx b/webui/src/components/common/EntitySheet.tsx index 802631221..37e39b752 100644 --- a/webui/src/components/common/EntitySheet.tsx +++ b/webui/src/components/common/EntitySheet.tsx @@ -12,7 +12,7 @@ * 3. 传入 onRunTest 启用「测试」Tab */ -import { useState, useEffect, useRef, useCallback, createContext, useContext } from 'react'; +import { lazy, Suspense, useState, useEffect, useRef, useCallback, createContext, useContext } from 'react'; import { X, FileText, @@ -27,7 +27,7 @@ import { } from 'lucide-react'; import { useTranslation } from 'react-i18next'; import client from '@/api/client'; -import SessionChat, { buildInstructionDisplayText, type SessionChatDisplay } from './SessionChat'; +import { buildInstructionDisplayText, type SessionChatDisplay } from '@/features/session-chat'; import { useSessionChat } from '@/hooks/useSessionChat'; import { useDefaultModelVision } from '@/hooks/useDefaultModelVision'; import ChatGuideDock, { type ChatGuideAction } from './ChatGuideDock'; @@ -40,6 +40,16 @@ import { } from './sidePanelSizing'; // ─── Context ────────────────────────────────────────────────────────────────── +const LazySessionChat = lazy(() => import('./SessionChat')); + +function ChatPanelFallback() { + return ( +
+ +
+ ); +} + interface EntitySheetCtx { /** Switch to the Rex tab, optionally sending an initial message */ openRex: (prefillMessage?: string) => void; @@ -582,13 +592,15 @@ export default function EntitySheet({ {activeTab === 'test' && (
{testSessionId ? ( - + }> + + ) : (
{testError ? ( @@ -674,69 +686,71 @@ export default function EntitySheet({
)} {!sessionError && rexSessionHydrated && ( - createAndSendRex({ - text, - imageParts, - agent: agentOverride || rexAgentName, - model: modelOverride === undefined ? rexModel : modelOverride, - displayText: options?.displayText, - }) : undefined} - welcomeContent={( - hasRexGuideActions ? ( - } - title={rexGuidePanelTitle ?? t('entity.rexAssist')} - description={rexGuidePanelDesc ?? t('entity.rexReady')} - groups={rexWelcomeGuideGroups} - onStartPrompt={startRexGuidePrompt} - /> - ) : ( -
- -

{t('entity.rexAssist')}

-

{t('entity.rexReady')}

-
- ) - )} - conversationBottomSlot={({ sendPrompt, sending, streaming, hasMessages }) => ( - hasRexGuideActions && hasMessages ? ( - { - if (prompt === EXTRACT_FROM_REX_GUIDE_PROMPT) { - void handleExtract(); - return; - } - - sendPrompt(prompt, { - displayText: buildInstructionDisplayText(label), - }); - }} - /> - ) : null - )} - /> + }> + createAndSendRex({ + text, + imageParts, + agent: agentOverride || rexAgentName, + model: modelOverride === undefined ? rexModel : modelOverride, + displayText: options?.displayText, + }) : undefined} + welcomeContent={( + hasRexGuideActions ? ( + } + title={rexGuidePanelTitle ?? t('entity.rexAssist')} + description={rexGuidePanelDesc ?? t('entity.rexReady')} + groups={rexWelcomeGuideGroups} + onStartPrompt={startRexGuidePrompt} + /> + ) : ( +
+ +

{t('entity.rexAssist')}

+

{t('entity.rexReady')}

+
+ ) + )} + conversationBottomSlot={({ sendPrompt, sending, streaming, hasMessages }) => ( + hasRexGuideActions && hasMessages ? ( + { + if (prompt === EXTRACT_FROM_REX_GUIDE_PROMPT) { + void handleExtract(); + return; + } + + sendPrompt(prompt, { + displayText: buildInstructionDisplayText(label), + }); + }} + /> + ) : null + )} + /> +
)}
)} diff --git a/webui/src/components/common/SessionChat.test.ts b/webui/src/components/common/SessionChat.test.ts index 934b5df8e..111a0ea24 100644 --- a/webui/src/components/common/SessionChat.test.ts +++ b/webui/src/components/common/SessionChat.test.ts @@ -7,7 +7,9 @@ import type { Message } from '@/types'; import { areChatMessagePartsRenderEqual, + areChatTimelineItemsRenderEqual, buildInstructionDisplayText, + buildChatTimelineItems, buildContextUsageBreakdown, buildTodoSummary, ChatMessageBubble, @@ -29,6 +31,7 @@ import { isActiveSessionStatus, listUploadedDocumentPaths, shouldRenderMessage, + shouldForwardSSEEventToParent, shouldRefetchFinishedMessage, truncateToolDisplayText, } from './SessionChat'; @@ -163,9 +166,13 @@ vi.mock('@/hooks/useReasoningToggle', () => ({ }), })); -vi.mock('@/hooks/usePendingQuestions', () => ({ - usePendingQuestions: () => pendingQuestionsHookMock, -})); +vi.mock('@/features/session-chat', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + usePendingQuestions: () => pendingQuestionsHookMock, + }; +}); vi.mock('./Toast', () => ({ useToast: () => toastMock, @@ -1028,6 +1035,146 @@ describe('getMessageErrorText', () => { }); }); +describe('shouldForwardSSEEventToParent', () => { + it('forwards global workflow, task, and session update events', () => { + expect(shouldForwardSSEEventToParent({ + type: 'workflow.updated', + properties: { id: 'workflow-1' }, + }, 'sess-1')).toBe(true); + expect(shouldForwardSSEEventToParent({ + type: 'task.updated', + properties: { executionID: 'task-1' }, + }, 'sess-1')).toBe(true); + expect(shouldForwardSSEEventToParent({ + type: 'session.updated', + properties: { id: 'other-session' }, + }, 'sess-1')).toBe(true); + }); + + it('forwards chat events only for the current session', () => { + expect(shouldForwardSSEEventToParent({ + type: 'message.part.updated', + properties: { part: { sessionID: 'sess-1' } }, + }, 'sess-1')).toBe(true); + expect(shouldForwardSSEEventToParent({ + type: 'message.part.updated', + properties: { part: { sessionID: 'other-session' } }, + }, 'sess-1')).toBe(false); + expect(shouldForwardSSEEventToParent({ + type: 'context.usage.updated', + properties: { sessionID: 'other-session' }, + }, 'sess-1')).toBe(false); + }); + + it('skips heartbeat-style events without payloads', () => { + expect(shouldForwardSSEEventToParent({ + type: 'server.heartbeat', + }, 'sess-1')).toBe(false); + }); +}); + +describe('buildChatTimelineItems', () => { + it('filters skipped and non-renderable messages while marking the active assistant', () => { + const messages = [ + makeMessage({ + id: 'user-1', + role: 'user', + parts: [{ id: 'user-part', type: 'text', text: 'hello' }] as Message['parts'], + }), + makeMessage({ + id: 'synthetic-1', + role: 'assistant', + parts: [{ id: 'synthetic-part', type: 'text', text: '', synthetic: true }] as Message['parts'], + }), + makeMessage({ + id: 'assistant-empty', + role: 'assistant', + parts: [], + finish: null, + }), + makeMessage({ + id: 'assistant-active', + role: 'assistant', + parts: [], + finish: null, + }), + ]; + + const items = buildChatTimelineItems({ + messages, + skipIndices: new Set([1]), + isStreaming: true, + }); + + expect(items.map((item) => item.message.id)).toEqual(['user-1', 'assistant-active']); + expect(items.map((item) => item.isActive)).toEqual([false, true]); + }); + + it('keeps the same visible set when not streaming', () => { + const messages = [ + makeMessage({ + id: 'assistant-empty', + role: 'assistant', + parts: [], + finish: null, + }), + makeMessage({ + id: 'assistant-text', + role: 'assistant', + parts: [{ id: 'text-part', type: 'text', text: 'done' }] as Message['parts'], + finish: 'stop', + }), + ]; + + const items = buildChatTimelineItems({ + messages, + skipIndices: new Set(), + isStreaming: false, + }); + + expect(items.map((item) => item.message.id)).toEqual(['assistant-text']); + expect(items[0].isActive).toBe(false); + }); +}); + +describe('areChatTimelineItemsRenderEqual', () => { + it('treats cloned assistant messages with identical visible parts as equal', () => { + const prevMessage = makeMessage({ + id: 'assistant-1', + role: 'assistant', + agent: 'rex', + parts: [{ id: 'text-1', type: 'text', text: 'hello' }] as Message['parts'], + finish: 'stop', + }); + const nextMessage = { + ...prevMessage, + parts: [{ id: 'text-1', type: 'text', text: 'hello' }] as Message['parts'], + }; + + expect(areChatTimelineItemsRenderEqual( + [{ message: prevMessage as any, isActive: false }], + [{ message: nextMessage as any, isActive: false }], + )).toBe(true); + }); + + it('detects visible text changes in otherwise stable timeline items', () => { + const prevMessage = makeMessage({ + id: 'assistant-1', + role: 'assistant', + parts: [{ id: 'text-1', type: 'text', text: 'hello' }] as Message['parts'], + }); + const nextMessage = { + ...prevMessage, + parts: [{ id: 'text-1', type: 'text', text: 'hello world' }] as Message['parts'], + }; + + expect(areChatTimelineItemsRenderEqual( + [{ message: prevMessage as any, isActive: false }], + [{ message: nextMessage as any, isActive: false }], + )).toBe(false); + }); +}); + describe('SessionChat error rendering', () => { it('renders empty assistant error messages instead of the thinking indicator', () => { useSessionMessagesMock.mockReturnValue({ diff --git a/webui/src/components/common/SessionChat.tsx b/webui/src/components/common/SessionChat.tsx index 8e2d4a98a..b3dd93096 100644 --- a/webui/src/components/common/SessionChat.tsx +++ b/webui/src/components/common/SessionChat.tsx @@ -28,10 +28,9 @@ import ImageLightbox from './ImageLightbox'; import { useSessionMessages } from '@/hooks/useSessions'; import { useSSE, type SSEConnectionStatus } from '@/hooks/useSSE'; import { useReasoningToggle } from '@/hooks/useReasoningToggle'; -import { usePendingQuestions, type PendingQuestion } from '@/hooks/usePendingQuestions'; import { sessionApi, type ContextUsageSnapshot, type QueuedPrompt } from '@/api/session'; import client, { getApiBase } from '@/api/client'; -import { commandAPI, type Command } from '@/api/skill'; +import type { Command } from '@/api/skill'; import type { Agent } from '@/api/agent'; import { useToast } from './Toast'; import { buildRunWorkflowHeaderSummary } from './toolStageSummary'; @@ -50,9 +49,33 @@ import { type ImagePartData, } from '@/utils/imageUpload'; import type { Message, MessagePart, SessionGoalState, ToolState } from '@/types'; +import { + buildInstructionDisplayText, + fetchSessionChatCommands, + getQueuedPromptText, + parseInstructionDisplayText, + resolveSessionChatSSEAction, + shouldForwardSSEEventToParent, + type CompactionStage, + usePendingQuestions, + useSessionContextUsage, + useSessionPromptQueue, + type PendingQuestion, + type PromptDisplayOptions, + type SSEChatEvent, + type SessionChatDisplay, +} from '@/features/session-chat'; export { formatSmartTime }; export type { SSEConnectionStatus }; +export { + buildInstructionDisplayText, + parseInstructionDisplayText, + shouldForwardSSEEventToParent, + type PromptDisplayOptions, + type SSEChatEvent, + type SessionChatDisplay, +} from '@/features/session-chat'; // ============================================================================ // Types @@ -60,11 +83,6 @@ export type { SSEConnectionStatus }; export type MergedMessage = Message & { _merged?: boolean }; -export interface SSEChatEvent { - type: string; - properties?: Record; -} - /** Node reference shown above the chat input as a dismissible chip */ export interface NodeRef { id: string; @@ -90,22 +108,6 @@ export interface ConversationBottomSlotActions { hasMessages: boolean; } -export interface PromptDisplayOptions { - displayText?: string; -} - -const INSTRUCTION_DISPLAY_PREFIX = '@@flocks-instruction:'; - -export function buildInstructionDisplayText(label: string): string { - return `${INSTRUCTION_DISPLAY_PREFIX}${label}`; -} - -export function parseInstructionDisplayText(text: string): string | null { - return text.startsWith(INSTRUCTION_DISPLAY_PREFIX) - ? text.slice(INSTRUCTION_DISPLAY_PREFIX.length).trim() || null - : null; -} - function getMessagePartDisplayText(part: MessagePart): string { const metadataDisplayText = part.metadata?.displayText ?? part.metadata?.display_text; return typeof metadataDisplayText === 'string' && metadataDisplayText @@ -113,24 +115,6 @@ function getMessagePartDisplayText(part: MessagePart): string { : part.text || ''; } -/** Display-related options grouped to reduce prop surface. */ -export interface SessionChatDisplay { - /** Compact mode for panels/dialogs (default: true). Set false for full-page. */ - compact?: boolean; - /** Let embedded chats use the full available message width. */ - fullWidth?: boolean; - /** Show copy action on assistant messages */ - showActions?: boolean; - /** Show timestamp below each message */ - showTimestamp?: boolean; - /** Default-collapse intermediate reasoning and tool-process details in embedded panels. */ - collapseIntermediateSteps?: boolean; - /** Initial open state for grouped reasoning/tool-process details. */ - processGroupsDefaultOpen?: boolean; - /** Keep grouped reasoning/tool-process details open while the assistant message is actively streaming. */ - processGroupsOpenWhileActive?: boolean; -} - export interface SessionChatProps { /** When null/undefined, only welcomeContent + input are rendered (lazy session). */ sessionId?: string | null; @@ -708,18 +692,6 @@ export function listUploadedDocumentPaths(items: UploadedDocumentAttachmentLike[ // don't share a draft, and namespaced to avoid colliding with other features. import { readChatDraft, writeChatDraft } from '@/utils/chatDraft'; -// Backend stages emitted by ``SessionCompaction.process`` / -// ``summarize_chunked`` via the ``session.compaction_progress`` SSE event. -// Keep in sync with ``flocks/session/lifecycle/compaction/{compaction,summary}.py``. -type CompactionStage = - | 'load' - | 'strategy' - | 'chunk_done' - | 'merge_started' - | 'merge_done' - | 'summarize_done' - | 'complete'; - interface CompactionStageEntry { stage: CompactionStage; data: Record; @@ -985,6 +957,91 @@ export function shouldRenderMessage( return true; } +export interface ChatTimelineItem { + message: MergedMessage; + isActive: boolean; +} + +export function buildChatTimelineItems({ + messages, + skipIndices, + isStreaming, +}: { + messages: MergedMessage[]; + skipIndices: Set; + isStreaming: boolean; +}): ChatTimelineItem[] { + const items: ChatTimelineItem[] = []; + for (let index = 0; index < messages.length; index++) { + if (skipIndices.has(index)) continue; + const message = messages[index]; + const isActive = + isStreaming && + index === messages.length - 1 && + message.role === 'assistant' && + !message.finish; + if (!shouldRenderMessage(message, { isActive })) continue; + items.push({ message, isActive }); + } + return items; +} + +export function areChatTimelineItemsRenderEqual( + prevItems: ChatTimelineItem[], + nextItems: ChatTimelineItem[], +): boolean { + if (prevItems.length !== nextItems.length) return false; + + for (let index = 0; index < prevItems.length; index++) { + const prev = prevItems[index]; + const next = nextItems[index]; + if (prev.isActive !== next.isActive) return false; + + const prevMessage = prev.message; + const nextMessage = next.message; + if (prevMessage === nextMessage) continue; + if (prevMessage.id !== nextMessage.id) return false; + if (prevMessage.role !== nextMessage.role) return false; + if (prevMessage.finish !== nextMessage.finish) return false; + if (prevMessage.error !== nextMessage.error) return false; + if (prevMessage.agent !== nextMessage.agent) return false; + if (prevMessage.timestamp !== nextMessage.timestamp) return false; + if (prevMessage.compacted !== nextMessage.compacted) return false; + if (!areChatMessagePartsRenderEqual(prevMessage.parts, nextMessage.parts)) return false; + } + + return true; +} + +function useStableChatTimelineSegments(items: ChatTimelineItem[]): { + historyItems: ChatTimelineItem[]; + tailItems: ChatTimelineItem[]; +} { + const previousRef = useRef<{ + historyItems: ChatTimelineItem[]; + tailItems: ChatTimelineItem[]; + } | null>(null); + + return useMemo(() => { + const tailStart = items.length > 0 && items[items.length - 1].isActive + ? items.length - 1 + : items.length; + const nextHistoryItems = tailStart === items.length ? items : items.slice(0, tailStart); + const nextTailItems = tailStart === items.length ? [] : items.slice(tailStart); + const previous = previousRef.current; + + const historyItems = previous && areChatTimelineItemsRenderEqual(previous.historyItems, nextHistoryItems) + ? previous.historyItems + : nextHistoryItems; + const tailItems = previous && areChatTimelineItemsRenderEqual(previous.tailItems, nextTailItems) + ? previous.tailItems + : nextTailItems; + const next = { historyItems, tailItems }; + previousRef.current = next; + return next; + }, [items]); +} + export function getMessageErrorText(message: Pick): string { const error = message.error as any; if (!error) return ''; @@ -1107,13 +1164,6 @@ function isAllowedUploadFile(file: File): boolean { return ALLOWED_UPLOAD_EXTENSIONS.has(getFileExtension(file.name)); } -function getQueuedPromptText(item: QueuedPrompt): string { - if (typeof item.displayText === 'string' && item.displayText) return item.displayText; - if (typeof item.display_text === 'string' && item.display_text) return item.display_text; - const textPart = item.parts.find((part) => part.type === 'text' && typeof part.text === 'string'); - return typeof textPart?.text === 'string' ? textPart.text : ''; -} - function getGoalBannerKey(goal: GoalBannerState | null): string { return goal ? `${goal.status}:${goal.objective}` : ''; } @@ -1502,11 +1552,23 @@ export default function SessionChat({ const [compactingMessage, setCompactingMessage] = useState(''); const [goalBanner, setGoalBanner] = useState(null); const [dismissedGoalKey, setDismissedGoalKey] = useState(() => readDismissedGoalKey(sessionId)); - const [queuedPrompts, setQueuedPrompts] = useState([]); - const [queueExpanded, setQueueExpanded] = useState(true); - const [editingQueueId, setEditingQueueId] = useState(null); - const [editingQueueText, setEditingQueueText] = useState(''); - const [queueActionId, setQueueActionId] = useState(null); + const { + items: queuedPrompts, + expanded: queueExpanded, + setExpanded: setQueueExpanded, + editingId: editingQueueId, + editingText: editingQueueText, + setEditingText: setEditingQueueText, + actionId: queueActionId, + refresh: fetchPromptQueue, + applyItems: applyPromptQueueItems, + enqueue: enqueuePrompt, + startEdit: startQueuedEdit, + cancelEdit: cancelQueuedEdit, + saveEdit: saveQueuedEdit, + remove: removeQueuedPrompt, + runNow: runQueuedPromptNow, + } = useSessionPromptQueue(sessionId); const [processGroupOpenState, setProcessGroupOpenState] = useState(() => ( readProcessGroupOpenState(sessionId) )); @@ -1569,12 +1631,14 @@ export default function SessionChat({ const [editingRole, setEditingRole] = useState(null); const [editingText, setEditingText] = useState(''); const [actionMessageId, setActionMessageId] = useState(null); - const [contextUsageSnapshot, setContextUsageSnapshot] = useState(null); - const [contextUsageRefreshing, setContextUsageRefreshing] = useState(false); - const [contextUsageWindowTokens, setContextUsageWindowTokens] = useState(0); - const contextUsageRequestRef = useRef<{ sessionId: string; promise: Promise } | null>(null); - const contextUsageRequestSeqRef = useRef(0); - const lastContextUsagePushAtRef = useRef(0); + const { + snapshot: contextUsageSnapshot, + refreshing: contextUsageRefreshing, + contextWindowTokens: contextUsageWindowTokens, + refresh: refreshContextUsage, + applyPushSnapshot: applyContextUsagePushSnapshot, + stopRefreshing: stopContextUsageRefreshing, + } = useSessionContextUsage(sessionId); const isCompactingRef = useRef(false); const prevStreamingRef = useRef(false); // Tracks "sessionId::message" key to prevent double-send in React StrictMode @@ -1599,6 +1663,7 @@ export default function SessionChat({ const messagesContentRef = useRef(null); const scrollContainerRef = useRef(null); const isAtBottomRef = useRef(true); + const scrollToBottomRafRef = useRef(null); const textareaRef = useRef(null); const fileInputRef = useRef(null); const isComposingRef = useRef(false); @@ -1638,11 +1703,20 @@ export default function SessionChat({ const scrollToBottom = useCallback(() => { if (!isAtBottomRef.current) return; - requestAnimationFrame(() => { + if (scrollToBottomRafRef.current !== null) return; + scrollToBottomRafRef.current = requestAnimationFrame(() => { + scrollToBottomRafRef.current = null; messagesEndRef.current?.scrollIntoView({ behavior: 'instant' }); }); }, []); + useEffect(() => () => { + if (scrollToBottomRafRef.current !== null) { + cancelAnimationFrame(scrollToBottomRafRef.current); + scrollToBottomRafRef.current = null; + } + }, []); + const loadOlderMessagesRef = useRef<(() => Promise) | null>(null); const hasMoreMessagesRef = useRef(false); const loadingOlderMessagesRef = useRef(false); @@ -1719,92 +1793,6 @@ export default function SessionChat({ const sseEnabled = Boolean(sessionId) && (live || isStreaming || !hideInput); - const fetchPromptQueue = useCallback(async () => { - if (!sessionId) { - setQueuedPrompts([]); - return; - } - try { - const response = await sessionApi.listPromptQueue(sessionId); - setQueuedPrompts(response.items ?? []); - } catch (err) { - console.warn('[SessionChat] Failed to fetch prompt queue:', err); - } - }, [sessionId]); - - const refreshContextUsage = useCallback((options?: { clear?: boolean; skipIfFreshMs?: number }) => { - if (!sessionId) { - setContextUsageSnapshot(null); - setContextUsageRefreshing(false); - setContextUsageWindowTokens(0); - contextUsageRequestSeqRef.current += 1; - contextUsageRequestRef.current = null; - lastContextUsagePushAtRef.current = 0; - return; - } - if (options?.clear) { - setContextUsageSnapshot(null); - setContextUsageRefreshing(true); - contextUsageRequestSeqRef.current += 1; - contextUsageRequestRef.current = null; - lastContextUsagePushAtRef.current = 0; - } else if ( - options?.skipIfFreshMs && - Date.now() - lastContextUsagePushAtRef.current < options.skipIfFreshMs - ) { - return; - } - - const existingRequest = contextUsageRequestRef.current; - if (existingRequest?.sessionId === sessionId) { - return existingRequest.promise; - } - - const requestSessionId = sessionId; - const requestSeq = contextUsageRequestSeqRef.current; - const request = sessionApi.getContextUsage(requestSessionId).then((snapshot) => { - if (requestSeq === contextUsageRequestSeqRef.current && snapshot.sessionID === sessionId) { - setContextUsageSnapshot(snapshot); - if (snapshot.contextWindow && snapshot.contextWindow > 0) { - setContextUsageWindowTokens(snapshot.contextWindow); - } - setContextUsageRefreshing(false); - } - }).catch((err) => { - setContextUsageRefreshing(false); - console.warn('[SessionChat] Failed to fetch context usage:', err); - }).finally(() => { - if (contextUsageRequestRef.current?.promise === request) { - contextUsageRequestRef.current = null; - } - }); - contextUsageRequestRef.current = { sessionId: requestSessionId, promise: request }; - return request; - }, [sessionId]); - - useEffect(() => { - if (!sessionId) { - void refreshContextUsage({ clear: true }); - return; - } - const requestIdle = (window as any).requestIdleCallback as - | ((cb: () => void, options?: { timeout?: number }) => number) - | undefined; - const cancelIdle = (window as any).cancelIdleCallback as - | ((id: number) => void) - | undefined; - if (requestIdle) { - const idleId = requestIdle(() => { - void refreshContextUsage({ clear: true }); - }, { timeout: 1500 }); - return () => cancelIdle?.(idleId); - } - const timer = window.setTimeout(() => { - void refreshContextUsage({ clear: true }); - }, 250); - return () => window.clearTimeout(timer); - }, [refreshContextUsage]); - useEffect(() => { goalHydrationVersionRef.current += 1; const hydrationVersion = goalHydrationVersionRef.current; @@ -1832,184 +1820,174 @@ export default function SessionChat({ const handleSSEEvent = useCallback( (event: SSEChatEvent) => { - const { type, properties } = event; - - // Forward events with payload to parent (e.g. session.updated, workflow.updated). - // Skip empty events like heartbeats to avoid noisy callbacks. - if (properties) onSSEEvent?.(event); - - if (!properties || !sessionId) return; - - if (type === 'session.cleared' && properties.sessionID === sessionId) { - abortingRef.current = false; - sessionBusyRef.current = false; - activeToolPartIdsRef.current.clear(); - abortedMessageIdRef.current = null; - suppressStreamingUntilIdleRef.current = false; - setContextUsageSnapshot(null); - setContextUsageRefreshing(true); - setContextUsageWindowTokens(0); - setIsStreaming(false); - setGoalBanner(null); - setDismissedGoalKey(''); - refetch(); - void refreshContextUsage({ clear: true }); - } else if ( - (type === 'session.status' && properties.sessionID === sessionId) - || (type === 'session.updated' && properties.id === sessionId && properties.status === 'idle') - ) { - const statusType = type === 'session.status' ? properties.status?.type : properties.status; - if (statusType === 'busy') { - sessionBusyRef.current = true; - if ( - !abortingRef.current && - !suppressStreamingUntilIdleRef.current - ) setIsStreaming(true); - setIsCompacting(false); - isCompactingRef.current = false; - } else if (statusType === 'compacting') { - sessionBusyRef.current = true; - if ( - !abortingRef.current && - !suppressStreamingUntilIdleRef.current - ) setIsStreaming(true); - setIsCompacting(true); - isCompactingRef.current = true; - setCompactingMessage(properties.status?.message || t('chat.compacting')); - // Reset progress state on each new compaction cycle so a stale - // run's stages do not leak into a fresh "Compacting..." panel. - setCompactionStages([]); - } else if (statusType === 'idle') { + // Forward only global events or events relevant to this chat. The global + // stream can be very noisy when multiple sessions run in parallel. + if (shouldForwardSSEEventToParent(event, sessionId)) onSSEEvent?.(event); + + const action = resolveSessionChatSSEAction(event, sessionId); + + switch (action.kind) { + case 'ignore': + return; + case 'session-cleared': + abortingRef.current = false; sessionBusyRef.current = false; - suppressStreamingUntilIdleRef.current = false; activeToolPartIdsRef.current.clear(); + abortedMessageIdRef.current = null; + suppressStreamingUntilIdleRef.current = false; setIsStreaming(false); - setIsCompacting(false); - isCompactingRef.current = false; - setCompactingMessage(''); - setCompactionStages([]); + setGoalBanner(null); + setDismissedGoalKey(''); refetch(); - void refreshContextUsage({ skipIfFreshMs: 500 }); - } - } else if (type === 'message.updated' && properties.info?.sessionID === sessionId) { - updateMessage(properties.info); - if ( - properties.info.role === 'assistant' && - (abortingRef.current || suppressStreamingUntilIdleRef.current) - ) { - abortedMessageIdRef.current = properties.info.id; - markMessageStopped(properties.info.id); - setIsStreaming(false); - setSending(false); - if (properties.info.finish || properties.info.time?.completed) { - void refreshContextUsage(); - } - } else if (properties.info.finish || properties.info.time?.completed) { - const shouldRefetch = shouldRefetchFinishedMessage({ - finishedMessageId: properties.info.id, - abortedMessageId: abortedMessageIdRef.current, - }); - // Preserve locally streamed partial text when the user aborts. The - // backend never persists in-flight text chunks, so refetching here - // would replace the visible partial response with an empty message. - if (shouldRefetch) { + void refreshContextUsage({ clear: true }); + return; + case 'session-status': + if (action.statusType === 'busy') { + sessionBusyRef.current = true; + if ( + !abortingRef.current && + !suppressStreamingUntilIdleRef.current + ) setIsStreaming(true); + setIsCompacting(false); + isCompactingRef.current = false; + } else if (action.statusType === 'compacting') { + sessionBusyRef.current = true; + if ( + !abortingRef.current && + !suppressStreamingUntilIdleRef.current + ) setIsStreaming(true); + setIsCompacting(true); + isCompactingRef.current = true; + setCompactingMessage(action.message || t('chat.compacting')); + // Reset progress state on each new compaction cycle so a stale + // run's stages do not leak into a fresh "Compacting..." panel. + setCompactionStages([]); + } else if (action.statusType === 'idle') { + sessionBusyRef.current = false; + suppressStreamingUntilIdleRef.current = false; + activeToolPartIdsRef.current.clear(); + setIsStreaming(false); + setIsCompacting(false); + isCompactingRef.current = false; + setCompactingMessage(''); + setCompactionStages([]); refetch(); - if (!sessionBusyRef.current && activeToolPartIdsRef.current.size === 0) { - setIsStreaming(false); + void refreshContextUsage({ skipIfFreshMs: 500 }); + } + return; + case 'message-updated': { + const { info } = action; + updateMessage(info); + if ( + info.role === 'assistant' && + (abortingRef.current || suppressStreamingUntilIdleRef.current) + ) { + if (info.id) { + abortedMessageIdRef.current = info.id; + markMessageStopped(info.id); + } + setIsStreaming(false); + setSending(false); + if (info.finish || info.time?.completed) { + void refreshContextUsage(); + } + } else if (info.finish || info.time?.completed) { + const shouldRefetch = shouldRefetchFinishedMessage({ + finishedMessageId: info.id, + abortedMessageId: abortedMessageIdRef.current, + }); + // Preserve locally streamed partial text when the user aborts. The + // backend never persists in-flight text chunks, so refetching here + // would replace the visible partial response with an empty message. + if (shouldRefetch) { + refetch(); + if (!sessionBusyRef.current && activeToolPartIdsRef.current.size === 0) { + setIsStreaming(false); + } } + void refreshContextUsage(); + abortingRef.current = false; + abortedMessageIdRef.current = null; + } else if ( + info.role === 'assistant' && + !info.finish && + !abortingRef.current + ) { + setIsStreaming(true); } - void refreshContextUsage(); - abortingRef.current = false; - abortedMessageIdRef.current = null; - } else if ( - properties.info.role === 'assistant' && - !properties.info.finish && - !abortingRef.current - ) { - setIsStreaming(true); + return; } - } else if (type === 'message.part.updated' && properties.part?.sessionID === sessionId) { - const part = properties.part as Pick; - if (part.id) { - if (isActiveToolPart(part)) { - activeToolPartIdsRef.current.add(part.id); - if (!abortingRef.current && !suppressStreamingUntilIdleRef.current) setIsStreaming(true); - } else { - activeToolPartIdsRef.current.delete(part.id); + case 'message-part-updated': { + const part = action.part as Pick; + if (part.id) { + if (isActiveToolPart(part)) { + activeToolPartIdsRef.current.add(part.id); + if (!abortingRef.current && !suppressStreamingUntilIdleRef.current) setIsStreaming(true); + } else { + activeToolPartIdsRef.current.delete(part.id); + } } - } - updateMessagePart(properties.part, properties.delta); - scrollToBottom(); - } else if (type === 'question.asked' && properties.sessionID === sessionId) { - const callID: string | undefined = properties.tool?.callID; - const requestId: string | undefined = properties.id; - if (callID && requestId) { - handleQuestionAsked(callID, requestId, properties.questions || []); + updateMessagePart(action.part, action.delta); scrollToBottom(); + return; } - } else if ( - (type === 'question.replied' || type === 'question.rejected') && - properties.sessionID === sessionId - ) { - const requestId: string | undefined = properties.requestID; - if (requestId) { - removeByRequestId(requestId); - } - } else if (type === 'session.compaction_progress' && properties.sessionID === sessionId) { - const stage = properties.stage as CompactionStage | undefined; - const data = (properties.data ?? {}) as Record; - if (!stage) return; - if (stage === 'complete' && data.result === 'continue') { - void refreshContextUsage({ skipIfFreshMs: 500 }); - } - // Single source of truth: append into ``compactionStages`` and let - // the progress bar derive ``done/total`` from it via useMemo. - // ``chunk_done`` arrives in non-deterministic order under - // ``asyncio.gather``; deduplicate by chunk index here so SSE - // reconnects / accidental re-deliveries are idempotent. - setCompactionStages((prev) => { - if (stage === 'chunk_done') { - const chunkIdx = typeof data.chunk === 'number' ? data.chunk : undefined; - if (chunkIdx !== undefined && prev.some( - (e) => e.stage === 'chunk_done' && (e.data as { chunk?: number }).chunk === chunkIdx, - )) { - return prev; + case 'question-asked': + handleQuestionAsked(action.callID, action.requestId, action.questions as QuestionItem[]); + scrollToBottom(); + return; + case 'question-resolved': + removeByRequestId(action.requestId); + return; + case 'compaction-progress': + if (action.stage === 'complete' && action.data.result === 'continue') { + void refreshContextUsage({ skipIfFreshMs: 500 }); + } + // Single source of truth: append into ``compactionStages`` and let + // the progress bar derive ``done/total`` from it via useMemo. + // ``chunk_done`` arrives in non-deterministic order under + // ``asyncio.gather``; deduplicate by chunk index here so SSE + // reconnects / accidental re-deliveries are idempotent. + setCompactionStages((prev) => { + if (action.stage === 'chunk_done') { + const chunkIdx = typeof action.data.chunk === 'number' ? action.data.chunk : undefined; + if (chunkIdx !== undefined && prev.some( + (e) => e.stage === 'chunk_done' && (e.data as { chunk?: number }).chunk === chunkIdx, + )) { + return prev; + } } + return [...prev, { stage: action.stage, data: action.data, ts: Date.now() }]; + }); + return; + case 'prompt-queue-updated': + applyPromptQueueItems(action.items); + return; + case 'goal-updated': { + const nextGoal = toGoalBannerState(action.goal); + if (nextGoal) { + goalHydrationVersionRef.current += 1; + setGoalBanner(nextGoal); + setDismissedGoalKey(readDismissedGoalKey(sessionId)); } - return [...prev, { stage, data, ts: Date.now() }]; - }); - } else if (type === 'session.prompt_queue.updated' && properties.sessionID === sessionId) { - const items = Array.isArray(properties.items) ? properties.items : []; - setQueuedPrompts(items as QueuedPrompt[]); - if (items.length > 0) setQueueExpanded(true); - } else if (type === 'session.goal.updated' && properties.sessionID === sessionId) { - const nextGoal = toGoalBannerState(properties as SessionGoalState); - if (nextGoal) { - goalHydrationVersionRef.current += 1; - setGoalBanner(nextGoal); - setDismissedGoalKey(readDismissedGoalKey(sessionId)); - } - } else if (type === 'context.compacted' && properties.sessionID === sessionId) { - void refreshContextUsage({ skipIfFreshMs: 500 }); - } else if (type === 'context.usage.updated' && properties.sessionID === sessionId) { - setContextUsageSnapshot(properties as ContextUsageSnapshot); - if (typeof properties.contextWindow === 'number' && properties.contextWindow > 0) { - setContextUsageWindowTokens(properties.contextWindow); + return; } - contextUsageRequestSeqRef.current += 1; - contextUsageRequestRef.current = null; - lastContextUsagePushAtRef.current = Date.now(); - setContextUsageRefreshing(false); - } else if (type === 'session.error' && properties.sessionID === sessionId) { - setIsStreaming(false); - setIsCompacting(false); - setCompactionStages([]); - setContextUsageRefreshing(false); - void refreshContextUsage({ skipIfFreshMs: 500 }); - abortingRef.current = false; - sessionBusyRef.current = false; - activeToolPartIdsRef.current.clear(); - onError?.(properties.error?.message || t('chat.placeholder')); + case 'context-compacted': + void refreshContextUsage({ skipIfFreshMs: 500 }); + return; + case 'context-usage-updated': + applyContextUsagePushSnapshot(action.snapshot); + return; + case 'session-error': + setIsStreaming(false); + setIsCompacting(false); + setCompactionStages([]); + stopContextUsageRefreshing(); + void refreshContextUsage({ skipIfFreshMs: 500 }); + abortingRef.current = false; + sessionBusyRef.current = false; + activeToolPartIdsRef.current.clear(); + onError?.(action.message || t('chat.placeholder')); + return; } }, [ @@ -2018,11 +1996,15 @@ export default function SessionChat({ updateMessagePart, refetch, refreshContextUsage, + applyContextUsagePushSnapshot, + stopContextUsageRefreshing, handleQuestionAsked, removeByRequestId, + applyPromptQueueItems, onSSEEvent, onError, scrollToBottom, + t, ], ); @@ -2125,11 +2107,6 @@ export default function SessionChat({ setCompactionStages([]); setGoalBanner(null); setDismissedGoalKey(''); - setQueuedPrompts([]); - setEditingQueueId(null); - setEditingQueueText(''); - setQueueActionId(null); - setContextUsageWindowTokens(0); setMentionRange(null); setMentionQuery(''); setSelectedMentionIndex(0); @@ -2242,25 +2219,7 @@ export default function SessionChat({ if (commandsLoadedRef.current) return; commandsLoadedRef.current = true; // Optimistic: prevent concurrent fetches try { - const res = await commandAPI.list(); - const serverCommands = res.data ?? []; - // Merge client-side /new command into the autocomplete list - setCommands([ - { - name: 'new', - canonical_name: 'new', - description: 'Create a new session', - template: '', - hidden: false, - aliases: [], - visible_surfaces: [], - execution_kind: 'session_control', - allow_attachments: false, - requires_existing_session: false, - channel_safe: false, - } satisfies Command, - ...serverCommands, - ]); + setCommands(await fetchSessionChatCommands()); } catch { commandsLoadedRef.current = false; // Allow retry on failure } @@ -2610,14 +2569,12 @@ export default function SessionChat({ if (!sessionId) return; const effectiveAgent = agentOverride || agentName; try { - await sessionApi.enqueuePrompt(sessionId, { + await enqueuePrompt({ parts: buildPromptParts(text, imageParts), ...(effectiveAgent ? { agent: effectiveAgent } : {}), ...(model ? { model } : {}), ...(options?.displayText ? { displayText: options.displayText } : {}), }); - await fetchPromptQueue(); - setQueueExpanded(true); } catch (err: any) { const statusCode = err?.response?.status; const detail = err?.response?.data?.detail; @@ -2904,62 +2861,41 @@ export default function SessionChat({ }, [isStreaming, markMessageStopped, sending, sessionId]); const handleQueuedEditStart = useCallback((item: QueuedPrompt) => { - setEditingQueueId(item.id); - setEditingQueueText(getQueuedPromptText(item)); - }, []); + startQueuedEdit(item); + }, [startQueuedEdit]); const handleQueuedEditCancel = useCallback(() => { - setEditingQueueId(null); - setEditingQueueText(''); - }, []); + cancelQueuedEdit(); + }, [cancelQueuedEdit]); const handleQueuedEditSave = useCallback(async (item: QueuedPrompt) => { - if (!sessionId) return; - const text = editingQueueText.trim(); - if (!text) return; - setQueueActionId(item.id); try { - await sessionApi.updateQueuedPrompt(sessionId, item.id, text); - handleQueuedEditCancel(); - await fetchPromptQueue(); + await saveQueuedEdit(item); } catch (err: any) { toast.error(err?.response?.data?.detail || err?.message || t('chat.queue.updateFailed')); - } finally { - setQueueActionId(null); } - }, [editingQueueText, fetchPromptQueue, handleQueuedEditCancel, sessionId, t, toast]); + }, [saveQueuedEdit, t, toast]); const handleQueuedRemove = useCallback(async (item: QueuedPrompt) => { - if (!sessionId) return; - setQueueActionId(item.id); try { - await sessionApi.removeQueuedPrompt(sessionId, item.id); - if (editingQueueId === item.id) handleQueuedEditCancel(); - await fetchPromptQueue(); + await removeQueuedPrompt(item); } catch (err: any) { toast.error(err?.response?.data?.detail || err?.message || t('chat.queue.removeFailed')); - } finally { - setQueueActionId(null); } - }, [editingQueueId, fetchPromptQueue, handleQueuedEditCancel, sessionId, t, toast]); + }, [removeQueuedPrompt, t, toast]); const handleQueuedRunNow = useCallback(async (item: QueuedPrompt) => { - if (!sessionId) return; - setQueueActionId(item.id); try { - await sessionApi.runQueuedPromptNow(sessionId, item.id); - if (editingQueueId === item.id) handleQueuedEditCancel(); - await fetchPromptQueue(); + const didRun = await runQueuedPromptNow(item); + if (!didRun) return; abortingRef.current = false; abortedMessageIdRef.current = null; suppressStreamingUntilIdleRef.current = false; setIsStreaming(true); } catch (err: any) { toast.error(err?.response?.data?.detail || err?.message || t('chat.queue.runNowFailed')); - } finally { - setQueueActionId(null); } - }, [editingQueueId, fetchPromptQueue, handleQueuedEditCancel, sessionId, t, toast]); + }, [runQueuedPromptNow, t, toast]); // Fire onStreamingDone when isStreaming transitions true → false useEffect(() => { @@ -3150,6 +3086,10 @@ export default function SessionChat({ return { merged, skipIndices }; }, [messages]); + const timelineItems = useMemo(() => ( + buildChatTimelineItems({ messages: merged, skipIndices, isStreaming }) + ), [isStreaming, merged, skipIndices]); + const { historyItems, tailItems } = useStableChatTimelineSegments(timelineItems); // ── Styling based on compact mode ── const msgAreaClass = compact @@ -3210,44 +3150,56 @@ export default function SessionChat({ )} - {merged.map((msg, i) => { - if (skipIndices.has(i)) return null; - const isActiveMessage = - isStreaming && - i === merged.length - 1 && - msg.role === 'assistant' && - !msg.finish; - if (!shouldRenderMessage(msg, { isActive: isActiveMessage })) return null; - return ( - - ); - })} + + {/* Compacting indicator with live progress stages */} {isCompacting && ( @@ -3795,6 +3747,72 @@ export interface ChatMessageBubbleProps { onRegenerate?: (messageId: string) => Promise; } +interface ChatMessageTimelineProps extends Omit { + items: ChatTimelineItem[]; +} + +function ChatMessageTimelineInner({ + items, + pendingQuestions, + onQuestionAnswer, + onQuestionReject, + showActions, + showTimestamp, + collapseIntermediateSteps, + processGroupsDefaultOpen, + processGroupsOpenWhileActive, + processGroupOpenState, + onProcessGroupOpenChange, + compact, + onCopy, + editingMessageId, + editingText, + actionsDisabled, + actionMessageId, + onEditStart, + onEditChange, + onEditCancel, + onEditSave, + onEditSend, + onRegenerate, +}: ChatMessageTimelineProps) { + return ( + <> + {items.map(({ message, isActive }) => ( + + ))} + + ); +} + +export const ChatMessageTimeline = memo(ChatMessageTimelineInner); + function ProcessGroupDetails({ defaultOpen, open, diff --git a/webui/src/components/common/sessionChatDisplay.ts b/webui/src/components/common/sessionChatDisplay.ts new file mode 100644 index 000000000..f54a3ac06 --- /dev/null +++ b/webui/src/components/common/sessionChatDisplay.ts @@ -0,0 +1,6 @@ +export { + buildInstructionDisplayText, + parseInstructionDisplayText, + type PromptDisplayOptions, + type SessionChatDisplay, +} from '@/features/session-chat'; diff --git a/webui/src/components/layout/Layout.tsx b/webui/src/components/layout/Layout.tsx index e199c1edb..dfe917045 100644 --- a/webui/src/components/layout/Layout.tsx +++ b/webui/src/components/layout/Layout.tsx @@ -26,6 +26,7 @@ import { type LucideIcon, } from 'lucide-react'; import { useState, useEffect, useLayoutEffect, useCallback, useMemo, useRef, lazy, Suspense } from 'react'; +import type { ComponentType } from 'react'; import { useTranslation } from 'react-i18next'; // Modals are only rendered after the user clicks/triggers them; pulling them // into the eager Layout chunk costs ~1.7k LOC + i18n keys + lucide icons that @@ -35,6 +36,18 @@ import { useTranslation } from 'react-i18next'; const ONBOARDING_DISMISSED_KEY = 'flocks_onboarding_dismissed'; const COLLAPSED_NAV_SECTIONS_KEY = 'flocks_layout_collapsed_nav_sections'; +type LazyLayoutModule = { default: ComponentType }; + +function lazyLayoutComponent( + loader: () => Promise, + namespaces: readonly string[] = [], +) { + return lazy(() => Promise.all([ + loader(), + preloadI18nNamespaces(namespaces), + ]).then(([module]) => module)); +} + function isOnboardingDismissed(): boolean { return localStorage.getItem(ONBOARDING_DISMISSED_KEY) === 'true'; } @@ -63,9 +76,9 @@ function saveCollapsedNavSectionIds(sectionIds: Set): void { } } -const OnboardingModal = lazy(() => import('@/components/common/OnboardingModal')); -const UpdateModal = lazy(() => import('@/components/common/UpdateModal')); -const NotificationModal = lazy(() => import('@/components/common/NotificationModal')); +const OnboardingModal = lazyLayoutComponent(() => import('@/components/common/OnboardingModal')); +const UpdateModal = lazyLayoutComponent(() => import('@/components/common/UpdateModal'), ['update']); +const NotificationModal = lazyLayoutComponent(() => import('@/components/common/NotificationModal'), ['notification']); import { checkUpdate, type VersionInfo } from '@/api/update'; import { consoleUpgradeApi } from '@/api/consoleUpgrade'; import { @@ -80,6 +93,7 @@ import { useProductName } from '@/contexts/ProductNameContext'; import { getLocalizedReleaseNotes } from '@/utils/releaseNotes'; import { UPDATE_DISMISSED_KEY, buildUpdateDismissalKey, isUpdateDismissed } from '@/utils/updateDismissal'; import { useWebUIContractPages } from '@/hooks/useWebUIContractPages'; +import { preloadI18nNamespaces } from '@/i18nResources'; import { resolveWebUIContractPageIcon } from '@/utils/webuiContractPageIcons'; import { buildWebUIContractWorkspaceSections, diff --git a/webui/src/features/session-chat/commands.test.ts b/webui/src/features/session-chat/commands.test.ts new file mode 100644 index 000000000..e9dffc62b --- /dev/null +++ b/webui/src/features/session-chat/commands.test.ts @@ -0,0 +1,78 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +import { + __resetSessionChatCommandsResourceForTesting, + fetchSessionChatCommands, +} from './commands'; + +const { listCommandsMock } = vi.hoisted(() => ({ + listCommandsMock: vi.fn(), +})); + +vi.mock('@/api/skill', () => ({ + commandAPI: { + list: listCommandsMock, + }, +})); + +function makeCommand(name: string) { + return { + name, + canonical_name: name, + description: `${name} command`, + template: '', + hidden: false, + aliases: [], + visible_surfaces: [], + execution_kind: 'direct', + allow_attachments: false, + requires_existing_session: true, + channel_safe: false, + }; +} + +describe('session chat commands resource', () => { + beforeEach(() => { + vi.clearAllMocks(); + __resetSessionChatCommandsResourceForTesting(); + }); + + it('shares concurrent command list requests and prepends the local /new command', async () => { + let resolveCommands: (value: { data: any[] }) => void = () => {}; + listCommandsMock.mockReturnValue(new Promise((resolve) => { + resolveCommands = resolve; + })); + + const first = fetchSessionChatCommands(); + const second = fetchSessionChatCommands(); + + expect(listCommandsMock).toHaveBeenCalledTimes(1); + + resolveCommands({ data: [makeCommand('summarize')] }); + + const [firstCommands, secondCommands] = await Promise.all([first, second]); + expect(firstCommands.map((command) => command.name)).toEqual(['new', 'summarize']); + expect(secondCommands.map((command) => command.name)).toEqual(['new', 'summarize']); + }); + + it('reuses fresh commands without another request', async () => { + listCommandsMock.mockResolvedValue({ data: [makeCommand('summarize')] }); + + await fetchSessionChatCommands(); + await fetchSessionChatCommands(); + + expect(listCommandsMock).toHaveBeenCalledTimes(1); + }); + + it('forces a retry after a failed command request', async () => { + listCommandsMock + .mockRejectedValueOnce(new Error('offline')) + .mockResolvedValueOnce({ data: [makeCommand('summarize')] }); + + await expect(fetchSessionChatCommands()).rejects.toThrow('offline'); + const commands = await fetchSessionChatCommands(); + + expect(commands.map((command) => command.name)).toEqual(['new', 'summarize']); + expect(listCommandsMock).toHaveBeenCalledTimes(2); + }); +}); diff --git a/webui/src/features/session-chat/commands.ts b/webui/src/features/session-chat/commands.ts new file mode 100644 index 000000000..980f14879 --- /dev/null +++ b/webui/src/features/session-chat/commands.ts @@ -0,0 +1,50 @@ +import { commandAPI, type Command } from '@/api/skill'; +import { + createSharedResource, + type SharedResourceFetchOptions, +} from '@/hooks/useSharedResource'; + +const SESSION_CHAT_COMMANDS_STALE_TIME_MS = 10 * 60 * 1000; + +const newSessionCommand = { + name: 'new', + canonical_name: 'new', + description: 'Create a new session', + template: '', + hidden: false, + aliases: [], + visible_surfaces: [], + execution_kind: 'session_control', + allow_attachments: false, + requires_existing_session: false, + channel_safe: false, +} satisfies Command; + +const sessionChatCommandsResource = createSharedResource({ + initialData: [], + staleTimeMs: SESSION_CHAT_COMMANDS_STALE_TIME_MS, + minFetchIntervalMs: 1_000, + fetcher: async () => { + const res = await commandAPI.list(); + return [newSessionCommand, ...(res.data ?? [])]; + }, + fallbackDataOnError: (previous) => previous, + getErrorMessage: (err) => (err instanceof Error && err.message ? err.message : 'Failed to fetch commands'), +}); + +export async function fetchSessionChatCommands( + options?: SharedResourceFetchOptions, +): Promise { + const current = sessionChatCommandsResource.getSnapshot(); + const fetchOptions = current.error && options?.force !== true + ? { ...options, force: true } + : options; + const commands = await sessionChatCommandsResource.fetch(fetchOptions); + const error = sessionChatCommandsResource.getSnapshot().error; + if (error) throw new Error(error); + return commands; +} + +export function __resetSessionChatCommandsResourceForTesting(): void { + sessionChatCommandsResource.resetForTesting(); +} diff --git a/webui/src/features/session-chat/display.test.ts b/webui/src/features/session-chat/display.test.ts new file mode 100644 index 000000000..2b97867eb --- /dev/null +++ b/webui/src/features/session-chat/display.test.ts @@ -0,0 +1,16 @@ +import { describe, expect, it } from 'vitest'; + +import { buildInstructionDisplayText, parseInstructionDisplayText } from './display'; + +describe('session chat display metadata', () => { + it('round-trips instruction display labels', () => { + const text = buildInstructionDisplayText('创建 SOC 自定义页面'); + + expect(text).toBe('@@flocks-instruction:创建 SOC 自定义页面'); + expect(parseInstructionDisplayText(text)).toBe('创建 SOC 自定义页面'); + }); + + it('ignores regular display text', () => { + expect(parseInstructionDisplayText('普通消息')).toBeNull(); + }); +}); diff --git a/webui/src/features/session-chat/display.ts b/webui/src/features/session-chat/display.ts new file mode 100644 index 000000000..3e1961110 --- /dev/null +++ b/webui/src/features/session-chat/display.ts @@ -0,0 +1,33 @@ +export interface PromptDisplayOptions { + displayText?: string; +} + +const INSTRUCTION_DISPLAY_PREFIX = '@@flocks-instruction:'; + +export function buildInstructionDisplayText(label: string): string { + return `${INSTRUCTION_DISPLAY_PREFIX}${label}`; +} + +export function parseInstructionDisplayText(text: string): string | null { + return text.startsWith(INSTRUCTION_DISPLAY_PREFIX) + ? text.slice(INSTRUCTION_DISPLAY_PREFIX.length).trim() || null + : null; +} + +/** Display-related options grouped to reduce prop surface. */ +export interface SessionChatDisplay { + /** Compact mode for panels/dialogs (default: true). Set false for full-page. */ + compact?: boolean; + /** Let embedded chats use the full available message width. */ + fullWidth?: boolean; + /** Show copy action on assistant messages */ + showActions?: boolean; + /** Show timestamp below each message */ + showTimestamp?: boolean; + /** Default-collapse intermediate reasoning and tool-process details in embedded panels. */ + collapseIntermediateSteps?: boolean; + /** Initial open state for grouped reasoning/tool-process details. */ + processGroupsDefaultOpen?: boolean; + /** Keep grouped reasoning/tool-process details open while the assistant message is actively streaming. */ + processGroupsOpenWhileActive?: boolean; +} diff --git a/webui/src/features/session-chat/index.ts b/webui/src/features/session-chat/index.ts new file mode 100644 index 000000000..c64a7155d --- /dev/null +++ b/webui/src/features/session-chat/index.ts @@ -0,0 +1,31 @@ +export { + buildInstructionDisplayText, + parseInstructionDisplayText, + type PromptDisplayOptions, + type SessionChatDisplay, +} from './display'; +export { + shouldForwardSSEEventToParent, + type SSEChatEvent, +} from './sseRouting'; +export { + resolveSessionChatSSEAction, + type CompactionStage, + type SessionChatSSEAction, +} from './sseActions'; +export { + useSessionContextUsage, + type RefreshContextUsageOptions, +} from './useSessionContextUsage'; +export { + getQueuedPromptText, + useSessionPromptQueue, + type EnqueuePromptPayload, +} from './useSessionPromptQueue'; +export { + usePendingQuestions, + type PendingQuestion, +} from './usePendingQuestions'; +export { + fetchSessionChatCommands, +} from './commands'; diff --git a/webui/src/features/session-chat/sseActions.test.ts b/webui/src/features/session-chat/sseActions.test.ts new file mode 100644 index 000000000..ca2874ac8 --- /dev/null +++ b/webui/src/features/session-chat/sseActions.test.ts @@ -0,0 +1,165 @@ +import { describe, expect, it } from 'vitest'; + +import { resolveSessionChatSSEAction } from './sseActions'; + +describe('resolveSessionChatSSEAction', () => { + it('ignores events without a current session or unrelated session payload', () => { + expect(resolveSessionChatSSEAction({ + type: 'message.updated', + properties: { info: { sessionID: 'sess-1' } }, + }, null)).toEqual({ kind: 'ignore' }); + + expect(resolveSessionChatSSEAction({ + type: 'message.updated', + properties: { info: { sessionID: 'other-session' } }, + }, 'sess-1')).toEqual({ kind: 'ignore' }); + + expect(resolveSessionChatSSEAction({ + type: 'server.heartbeat', + }, 'sess-1')).toEqual({ kind: 'ignore' }); + }); + + it('resolves status events from both status and updated payload shapes', () => { + expect(resolveSessionChatSSEAction({ + type: 'session.status', + properties: { + sessionID: 'sess-1', + status: { type: 'compacting', message: 'Compacting context' }, + }, + }, 'sess-1')).toEqual({ + kind: 'session-status', + statusType: 'compacting', + message: 'Compacting context', + }); + + expect(resolveSessionChatSSEAction({ + type: 'session.updated', + properties: { id: 'sess-1', status: 'idle' }, + }, 'sess-1')).toEqual({ + kind: 'session-status', + statusType: 'idle', + }); + }); + + it('resolves message and part update actions for the active session', () => { + expect(resolveSessionChatSSEAction({ + type: 'message.updated', + properties: { + info: { + id: 'msg-1', + sessionID: 'sess-1', + role: 'assistant', + parts: [], + timestamp: 1, + }, + }, + }, 'sess-1')).toMatchObject({ + kind: 'message-updated', + info: { id: 'msg-1' }, + }); + + expect(resolveSessionChatSSEAction({ + type: 'message.part.updated', + properties: { + part: { + id: 'part-1', + messageID: 'msg-1', + sessionID: 'sess-1', + type: 'text', + text: 'hello', + }, + delta: 'hello', + }, + }, 'sess-1')).toMatchObject({ + kind: 'message-part-updated', + part: { id: 'part-1' }, + delta: 'hello', + }); + }); + + it('resolves questions only when request and call ids are present', () => { + expect(resolveSessionChatSSEAction({ + type: 'question.asked', + properties: { + sessionID: 'sess-1', + id: 'request-1', + tool: { callID: 'call-1' }, + questions: [{ question: 'Continue?' }], + }, + }, 'sess-1')).toEqual({ + kind: 'question-asked', + callID: 'call-1', + requestId: 'request-1', + questions: [{ question: 'Continue?' }], + }); + + expect(resolveSessionChatSSEAction({ + type: 'question.asked', + properties: { sessionID: 'sess-1', id: 'request-1' }, + }, 'sess-1')).toEqual({ kind: 'ignore' }); + + expect(resolveSessionChatSSEAction({ + type: 'question.rejected', + properties: { sessionID: 'sess-1', requestID: 'request-1' }, + }, 'sess-1')).toEqual({ + kind: 'question-resolved', + requestId: 'request-1', + }); + }); + + it('resolves compaction, queue, goal, context, and error actions', () => { + expect(resolveSessionChatSSEAction({ + type: 'session.compaction_progress', + properties: { sessionID: 'sess-1', stage: 'chunk_done', data: { chunk: 2 } }, + }, 'sess-1')).toEqual({ + kind: 'compaction-progress', + stage: 'chunk_done', + data: { chunk: 2 }, + }); + + expect(resolveSessionChatSSEAction({ + type: 'session.prompt_queue.updated', + properties: { sessionID: 'sess-1', items: [{ id: 'queue-1' }] }, + }, 'sess-1')).toMatchObject({ + kind: 'prompt-queue-updated', + items: [{ id: 'queue-1' }], + }); + + expect(resolveSessionChatSSEAction({ + type: 'session.goal.updated', + properties: { sessionID: 'sess-1', status: 'active', objective: 'Ship it' }, + }, 'sess-1')).toEqual({ + kind: 'goal-updated', + goal: { sessionID: 'sess-1', status: 'active', objective: 'Ship it' }, + }); + + expect(resolveSessionChatSSEAction({ + type: 'context.usage.updated', + properties: { + sessionID: 'sess-1', + usedTokens: 100, + contextWindow: 1000, + percent: 10, + source: 'estimated', + estimatedTokens: 100, + compactedTokens: 0, + segments: [], + excludedSegments: [], + }, + }, 'sess-1')).toMatchObject({ + kind: 'context-usage-updated', + snapshot: { usedTokens: 100 }, + }); + + expect(resolveSessionChatSSEAction({ + type: 'session.error', + properties: { + sessionID: 'sess-1', + error: { message: 'provider unavailable' }, + }, + }, 'sess-1')).toEqual({ + kind: 'session-error', + message: 'provider unavailable', + }); + }); +}); diff --git a/webui/src/features/session-chat/sseActions.ts b/webui/src/features/session-chat/sseActions.ts new file mode 100644 index 000000000..58d5f3172 --- /dev/null +++ b/webui/src/features/session-chat/sseActions.ts @@ -0,0 +1,160 @@ +import type { ContextUsageSnapshot, QueuedPrompt } from '@/api/session'; +import type { Message, MessagePart, SessionGoalState } from '@/types'; + +import type { SSEChatEvent } from './sseRouting'; + +export type SessionMessageInfo = Partial & { + id?: string; + sessionID?: string; + role?: Message['role']; + time?: { completed?: number | boolean | null }; +}; + +export type CompactionStage = + | 'load' + | 'strategy' + | 'chunk_done' + | 'merge_started' + | 'merge_done' + | 'summarize_done' + | 'complete'; + +export type SessionChatSSEAction = + | { kind: 'ignore' } + | { kind: 'session-cleared' } + | { kind: 'session-status'; statusType?: string; message?: string } + | { kind: 'message-updated'; info: SessionMessageInfo } + | { kind: 'message-part-updated'; part: MessagePart; delta?: string } + | { kind: 'question-asked'; callID: string; requestId: string; questions: unknown[] } + | { kind: 'question-resolved'; requestId: string } + | { kind: 'compaction-progress'; stage: CompactionStage; data: Record } + | { kind: 'prompt-queue-updated'; items: QueuedPrompt[] } + | { kind: 'goal-updated'; goal: SessionGoalState } + | { kind: 'context-compacted' } + | { kind: 'context-usage-updated'; snapshot: ContextUsageSnapshot } + | { kind: 'session-error'; message?: string }; + +function isRecord(value: unknown): value is Record { + return Boolean(value) && typeof value === 'object'; +} + +function isCurrentSession(properties: Record, sessionId: string): boolean { + return properties.sessionID === sessionId; +} + +function readSessionStatusType(status: unknown): string | undefined { + if (typeof status === 'string') return status; + if (isRecord(status) && typeof status.type === 'string') return status.type; + return undefined; +} + +export function resolveSessionChatSSEAction( + event: SSEChatEvent, + sessionId?: string | null, +): SessionChatSSEAction { + if (!sessionId || !isRecord(event.properties)) return { kind: 'ignore' }; + + const { type } = event; + const properties = event.properties; + + if (type === 'session.cleared') { + return isCurrentSession(properties, sessionId) ? { kind: 'session-cleared' } : { kind: 'ignore' }; + } + + if (type === 'session.status') { + if (!isCurrentSession(properties, sessionId)) return { kind: 'ignore' }; + return { + kind: 'session-status', + statusType: readSessionStatusType(properties.status), + message: isRecord(properties.status) && typeof properties.status.message === 'string' + ? properties.status.message + : undefined, + }; + } + + if (type === 'session.updated') { + if (properties.id !== sessionId || properties.status !== 'idle') return { kind: 'ignore' }; + return { kind: 'session-status', statusType: 'idle' }; + } + + if (type === 'message.updated') { + if (!isRecord(properties.info) || properties.info.sessionID !== sessionId) return { kind: 'ignore' }; + return { kind: 'message-updated', info: properties.info as SessionMessageInfo }; + } + + if (type === 'message.part.updated') { + if (!isRecord(properties.part) || properties.part.sessionID !== sessionId) return { kind: 'ignore' }; + return { + kind: 'message-part-updated', + part: properties.part as MessagePart, + delta: typeof properties.delta === 'string' ? properties.delta : undefined, + }; + } + + if (type === 'question.asked') { + if (!isCurrentSession(properties, sessionId)) return { kind: 'ignore' }; + const callID = isRecord(properties.tool) && typeof properties.tool.callID === 'string' + ? properties.tool.callID + : undefined; + const requestId = typeof properties.id === 'string' ? properties.id : undefined; + if (!callID || !requestId) return { kind: 'ignore' }; + return { + kind: 'question-asked', + callID, + requestId, + questions: Array.isArray(properties.questions) ? properties.questions : [], + }; + } + + if (type === 'question.replied' || type === 'question.rejected') { + if (!isCurrentSession(properties, sessionId)) return { kind: 'ignore' }; + const requestId = typeof properties.requestID === 'string' ? properties.requestID : undefined; + return requestId ? { kind: 'question-resolved', requestId } : { kind: 'ignore' }; + } + + if (type === 'session.compaction_progress') { + if (!isCurrentSession(properties, sessionId) || typeof properties.stage !== 'string') { + return { kind: 'ignore' }; + } + return { + kind: 'compaction-progress', + stage: properties.stage as CompactionStage, + data: isRecord(properties.data) ? properties.data : {}, + }; + } + + if (type === 'session.prompt_queue.updated') { + if (!isCurrentSession(properties, sessionId)) return { kind: 'ignore' }; + return { + kind: 'prompt-queue-updated', + items: Array.isArray(properties.items) ? properties.items as QueuedPrompt[] : [], + }; + } + + if (type === 'session.goal.updated') { + if (!isCurrentSession(properties, sessionId)) return { kind: 'ignore' }; + return { kind: 'goal-updated', goal: properties as SessionGoalState }; + } + + if (type === 'context.compacted') { + return isCurrentSession(properties, sessionId) ? { kind: 'context-compacted' } : { kind: 'ignore' }; + } + + if (type === 'context.usage.updated') { + return isCurrentSession(properties, sessionId) + ? { kind: 'context-usage-updated', snapshot: properties as ContextUsageSnapshot } + : { kind: 'ignore' }; + } + + if (type === 'session.error') { + if (!isCurrentSession(properties, sessionId)) return { kind: 'ignore' }; + return { + kind: 'session-error', + message: isRecord(properties.error) && typeof properties.error.message === 'string' + ? properties.error.message + : undefined, + }; + } + + return { kind: 'ignore' }; +} diff --git a/webui/src/features/session-chat/sseRouting.test.ts b/webui/src/features/session-chat/sseRouting.test.ts new file mode 100644 index 000000000..a95681412 --- /dev/null +++ b/webui/src/features/session-chat/sseRouting.test.ts @@ -0,0 +1,50 @@ +import { describe, expect, it } from 'vitest'; + +import { shouldForwardSSEEventToParent } from './sseRouting'; + +describe('shouldForwardSSEEventToParent', () => { + it('forwards global list refresh events', () => { + expect(shouldForwardSSEEventToParent({ + type: 'session.updated', + properties: { id: 'other-session' }, + }, 'session-1')).toBe(true); + expect(shouldForwardSSEEventToParent({ + type: 'task.updated', + properties: { executionID: 'task-1' }, + }, 'session-1')).toBe(true); + expect(shouldForwardSSEEventToParent({ + type: 'workflow.updated', + properties: { workflowID: 'workflow-1' }, + }, 'session-1')).toBe(true); + }); + + it('forwards events that belong to the active session', () => { + expect(shouldForwardSSEEventToParent({ + type: 'message.updated', + properties: { info: { sessionID: 'session-1' } }, + }, 'session-1')).toBe(true); + expect(shouldForwardSSEEventToParent({ + type: 'message.part.updated', + properties: { part: { sessionID: 'session-1' } }, + }, 'session-1')).toBe(true); + expect(shouldForwardSSEEventToParent({ + type: 'session.status', + properties: { sessionID: 'session-1' }, + }, 'session-1')).toBe(true); + }); + + it('does not forward unrelated chat events', () => { + expect(shouldForwardSSEEventToParent({ + type: 'message.updated', + properties: { info: { sessionID: 'other-session' } }, + }, 'session-1')).toBe(false); + expect(shouldForwardSSEEventToParent({ + type: 'message.updated', + properties: { info: { sessionID: 'other-session' } }, + }, null)).toBe(false); + expect(shouldForwardSSEEventToParent({ + type: 'message.updated', + properties: undefined, + }, 'session-1')).toBe(false); + }); +}); diff --git a/webui/src/features/session-chat/sseRouting.ts b/webui/src/features/session-chat/sseRouting.ts new file mode 100644 index 000000000..e2cdaa08d --- /dev/null +++ b/webui/src/features/session-chat/sseRouting.ts @@ -0,0 +1,26 @@ +export interface SSEChatEvent { + type: string; + properties?: Record; +} + +export function shouldForwardSSEEventToParent(event: SSEChatEvent, sessionId?: string | null): boolean { + const { type, properties } = event; + if (!properties) return false; + + if ( + type === 'session.updated' || + type === 'task.updated' || + type.startsWith('workflow.') + ) { + return true; + } + + if (!sessionId) return false; + + return ( + properties.sessionID === sessionId || + properties.info?.sessionID === sessionId || + properties.part?.sessionID === sessionId || + (type.startsWith('session.') && properties.id === sessionId) + ); +} diff --git a/webui/src/features/session-chat/usePendingQuestions.ts b/webui/src/features/session-chat/usePendingQuestions.ts new file mode 100644 index 000000000..aba9323ea --- /dev/null +++ b/webui/src/features/session-chat/usePendingQuestions.ts @@ -0,0 +1,120 @@ +/** + * Hook to manage pending question state for the Question tool. + * + * Centralises SSE event handling, answer/reject API calls, and session-switch + * cleanup so SessionChat does not need to own question runtime state. + */ + +import { useCallback, useState } from 'react'; + +import client from '@/api/client'; +import type { QuestionItem } from '@/components/common/QuestionTool'; + +export interface PendingQuestion { + requestId: string; + questions: QuestionItem[]; +} + +interface PendingQuestionApiResponse { + id?: string; + questions?: QuestionItem[]; + tool?: { + callID?: string; + }; +} + +function isNotFoundError(error: unknown): boolean { + return !!( + error && + typeof error === 'object' && + 'response' in error && + (error as { response?: { status?: number } }).response?.status === 404 + ); +} + +export function usePendingQuestions() { + const [pendingQuestions, setPendingQuestions] = useState>({}); + + const removeByCallId = useCallback((callID: string) => { + setPendingQuestions(prev => { + const next = { ...prev }; + delete next[callID]; + return next; + }); + }, []); + + const handleQuestionAsked = useCallback( + (callID: string, requestId: string, questions: QuestionItem[]) => { + setPendingQuestions(prev => ({ + ...prev, + [callID]: { requestId, questions }, + })); + }, + [], + ); + + const submitAnswer = useCallback( + async (callID: string, requestId: string, answers: string[][]) => { + try { + await client.post(`/api/question/${requestId}/reply`, { answers }); + } catch (error) { + if (!isNotFoundError(error)) throw error; + } + removeByCallId(callID); + }, + [removeByCallId], + ); + + const submitReject = useCallback( + async (callID: string, requestId: string) => { + try { + await client.post(`/api/question/${requestId}/reject`, {}); + } catch (error) { + if (!isNotFoundError(error)) throw error; + } + removeByCallId(callID); + }, + [removeByCallId], + ); + + const removeByRequestId = useCallback((requestId: string) => { + setPendingQuestions(prev => { + const next = { ...prev }; + for (const [callID, pending] of Object.entries(prev)) { + if (pending.requestId === requestId) { + delete next[callID]; + } + } + return next; + }); + }, []); + + const fetchPendingQuestions = useCallback(async (sessionId: string) => { + const response = await client.get(`/api/question/session/${sessionId}/pending`); + const next: Record = {}; + for (const item of response.data || []) { + const callID = item.tool?.callID; + const requestId = item.id; + if (!callID || !requestId) continue; + next[callID] = { + requestId, + questions: item.questions || [], + }; + } + setPendingQuestions(next); + }, []); + + const clearAll = useCallback(() => { + setPendingQuestions({}); + }, []); + + return { + pendingQuestions, + handleQuestionAsked, + submitAnswer, + submitReject, + removeByRequestId, + fetchPendingQuestions, + clearAll, + } as const; +} diff --git a/webui/src/features/session-chat/useSessionContextUsage.test.ts b/webui/src/features/session-chat/useSessionContextUsage.test.ts new file mode 100644 index 000000000..076f7f2e8 --- /dev/null +++ b/webui/src/features/session-chat/useSessionContextUsage.test.ts @@ -0,0 +1,100 @@ +import { act, renderHook } from '@testing-library/react'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import type { ContextUsageSnapshot } from '@/api/session'; +import { useSessionContextUsage } from './useSessionContextUsage'; + +const getContextUsageMock = vi.fn(); + +vi.mock('@/api/session', () => ({ + sessionApi: { + getContextUsage: (...args: unknown[]) => getContextUsageMock(...args), + }, +})); + +function buildSnapshot(overrides: Partial = {}): ContextUsageSnapshot { + return { + sessionID: 'sess-1', + usedTokens: 100, + contextWindow: 1000, + percent: 10, + source: 'estimated', + estimatedTokens: 100, + compactedTokens: 0, + segments: [], + excludedSegments: [], + ...overrides, + }; +} + +function deferred() { + let resolve!: (value: T) => void; + let reject!: (reason?: unknown) => void; + const promise = new Promise((resolvePromise, rejectPromise) => { + resolve = resolvePromise; + reject = rejectPromise; + }); + return { promise, resolve, reject }; +} + +describe('useSessionContextUsage', () => { + beforeEach(() => { + vi.clearAllMocks(); + vi.useFakeTimers(); + }); + + afterEach(() => { + vi.useRealTimers(); + }); + + it('deduplicates in-flight refreshes for the same session', async () => { + const request = deferred(); + getContextUsageMock.mockReturnValue(request.promise); + const { result } = renderHook(() => useSessionContextUsage('sess-1')); + + act(() => { + result.current.refresh(); + result.current.refresh(); + }); + + expect(getContextUsageMock).toHaveBeenCalledTimes(1); + expect(getContextUsageMock).toHaveBeenCalledWith('sess-1'); + + await act(async () => { + request.resolve(buildSnapshot({ usedTokens: 120 })); + await request.promise; + }); + + expect(result.current.snapshot?.usedTokens).toBe(120); + }); + + it('skips a refresh immediately after a pushed snapshot', () => { + const { result } = renderHook(() => useSessionContextUsage('sess-1')); + + act(() => { + result.current.applyPushSnapshot(buildSnapshot({ usedTokens: 420 })); + result.current.refresh({ skipIfFreshMs: 500 }); + }); + + expect(getContextUsageMock).not.toHaveBeenCalled(); + expect(result.current.snapshot?.usedTokens).toBe(420); + }); + + it('ignores stale responses after a newer pushed snapshot', async () => { + const request = deferred(); + getContextUsageMock.mockReturnValue(request.promise); + const { result } = renderHook(() => useSessionContextUsage('sess-1')); + + act(() => { + result.current.refresh(); + result.current.applyPushSnapshot(buildSnapshot({ usedTokens: 420 })); + }); + + await act(async () => { + request.resolve(buildSnapshot({ usedTokens: 900 })); + await request.promise; + }); + + expect(result.current.snapshot?.usedTokens).toBe(420); + }); +}); diff --git a/webui/src/features/session-chat/useSessionContextUsage.ts b/webui/src/features/session-chat/useSessionContextUsage.ts new file mode 100644 index 000000000..1131cab52 --- /dev/null +++ b/webui/src/features/session-chat/useSessionContextUsage.ts @@ -0,0 +1,118 @@ +import { useCallback, useEffect, useRef, useState } from 'react'; + +import { sessionApi, type ContextUsageSnapshot } from '@/api/session'; + +export interface RefreshContextUsageOptions { + clear?: boolean; + skipIfFreshMs?: number; +} + +export function useSessionContextUsage(sessionId?: string | null) { + const [snapshot, setSnapshot] = useState(null); + const [refreshing, setRefreshing] = useState(false); + const [contextWindowTokens, setContextWindowTokens] = useState(0); + const requestRef = useRef<{ sessionId: string; promise: Promise } | null>(null); + const requestSeqRef = useRef(0); + const lastPushAtRef = useRef(0); + + const reset = useCallback((nextRefreshing = false) => { + setSnapshot(null); + setRefreshing(nextRefreshing); + setContextWindowTokens(0); + requestSeqRef.current += 1; + requestRef.current = null; + lastPushAtRef.current = 0; + }, []); + + const refresh = useCallback((options?: RefreshContextUsageOptions) => { + if (!sessionId) { + reset(false); + return; + } + + if (options?.clear) { + reset(true); + } else if ( + options?.skipIfFreshMs && + Date.now() - lastPushAtRef.current < options.skipIfFreshMs + ) { + return; + } + + const existingRequest = requestRef.current; + if (existingRequest?.sessionId === sessionId) { + return existingRequest.promise; + } + + const requestSessionId = sessionId; + const requestSeq = requestSeqRef.current; + const request = sessionApi.getContextUsage(requestSessionId).then((nextSnapshot) => { + if (requestSeq === requestSeqRef.current && nextSnapshot.sessionID === sessionId) { + setSnapshot(nextSnapshot); + if (nextSnapshot.contextWindow && nextSnapshot.contextWindow > 0) { + setContextWindowTokens(nextSnapshot.contextWindow); + } + setRefreshing(false); + } + }).catch((err) => { + setRefreshing(false); + console.warn('[SessionChat] Failed to fetch context usage:', err); + }).finally(() => { + if (requestRef.current?.promise === request) { + requestRef.current = null; + } + }); + + requestRef.current = { sessionId: requestSessionId, promise: request }; + return request; + }, [reset, sessionId]); + + const applyPushSnapshot = useCallback((nextSnapshot: ContextUsageSnapshot) => { + setSnapshot(nextSnapshot); + if (nextSnapshot.contextWindow && nextSnapshot.contextWindow > 0) { + setContextWindowTokens(nextSnapshot.contextWindow); + } + requestSeqRef.current += 1; + requestRef.current = null; + lastPushAtRef.current = Date.now(); + setRefreshing(false); + }, []); + + const stopRefreshing = useCallback(() => { + setRefreshing(false); + }, []); + + useEffect(() => { + if (!sessionId) { + reset(false); + return; + } + + const requestIdle = (window as any).requestIdleCallback as + | ((cb: () => void, options?: { timeout?: number }) => number) + | undefined; + const cancelIdle = (window as any).cancelIdleCallback as + | ((id: number) => void) + | undefined; + if (requestIdle) { + const idleId = requestIdle(() => { + void refresh({ clear: true }); + }, { timeout: 1500 }); + return () => cancelIdle?.(idleId); + } + const timer = window.setTimeout(() => { + void refresh({ clear: true }); + }, 250); + return () => window.clearTimeout(timer); + }, [refresh, reset, sessionId]); + + return { + snapshot, + refreshing, + contextWindowTokens, + refresh, + applyPushSnapshot, + reset, + stopRefreshing, + }; +} diff --git a/webui/src/features/session-chat/useSessionPromptQueue.test.ts b/webui/src/features/session-chat/useSessionPromptQueue.test.ts new file mode 100644 index 000000000..1dcfd5609 --- /dev/null +++ b/webui/src/features/session-chat/useSessionPromptQueue.test.ts @@ -0,0 +1,195 @@ +import { act, renderHook } from '@testing-library/react'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +import type { QueuedPrompt } from '@/api/session'; +import { getQueuedPromptText, useSessionPromptQueue } from './useSessionPromptQueue'; + +const listPromptQueueMock = vi.fn(); +const enqueuePromptMock = vi.fn(); +const updateQueuedPromptMock = vi.fn(); +const removeQueuedPromptMock = vi.fn(); +const runQueuedPromptNowMock = vi.fn(); + +vi.mock('@/api/session', () => ({ + sessionApi: { + listPromptQueue: (...args: unknown[]) => listPromptQueueMock(...args), + enqueuePrompt: (...args: unknown[]) => enqueuePromptMock(...args), + updateQueuedPrompt: (...args: unknown[]) => updateQueuedPromptMock(...args), + removeQueuedPrompt: (...args: unknown[]) => removeQueuedPromptMock(...args), + runQueuedPromptNow: (...args: unknown[]) => runQueuedPromptNowMock(...args), + }, +})); + +function buildQueuedPrompt(overrides: Partial = {}): QueuedPrompt { + return { + id: 'queue-1', + sessionID: 'sess-1', + parts: [{ type: 'text', text: 'queued text' }], + status: 'pending', + createdAt: 1, + updatedAt: 1, + ...overrides, + }; +} + +function deferred() { + let resolve!: (value: T) => void; + let reject!: (reason?: unknown) => void; + const promise = new Promise((resolvePromise, rejectPromise) => { + resolve = resolvePromise; + reject = rejectPromise; + }); + return { promise, resolve, reject }; +} + +describe('getQueuedPromptText', () => { + it('prefers display text and falls back to the first text part', () => { + expect(getQueuedPromptText(buildQueuedPrompt({ + displayText: 'Display text', + parts: [{ type: 'text', text: 'raw text' }], + }))).toBe('Display text'); + + expect(getQueuedPromptText(buildQueuedPrompt({ + displayText: '', + parts: [{ type: 'text', text: 'raw text' }], + }))).toBe('raw text'); + }); +}); + +describe('useSessionPromptQueue', () => { + beforeEach(() => { + vi.clearAllMocks(); + listPromptQueueMock.mockResolvedValue({ items: [] }); + enqueuePromptMock.mockResolvedValue({}); + updateQueuedPromptMock.mockResolvedValue({}); + removeQueuedPromptMock.mockResolvedValue({}); + runQueuedPromptNowMock.mockResolvedValue({}); + }); + + it('refreshes the queue for the active session', async () => { + const item = buildQueuedPrompt(); + listPromptQueueMock.mockResolvedValue({ items: [item] }); + const { result } = renderHook(() => useSessionPromptQueue('sess-1')); + + await act(async () => { + await result.current.refresh(); + }); + + expect(listPromptQueueMock).toHaveBeenCalledWith('sess-1'); + expect(result.current.items).toEqual([item]); + }); + + it('ignores stale refresh responses after a session switch', async () => { + const request = deferred<{ items: QueuedPrompt[] }>(); + listPromptQueueMock.mockReturnValue(request.promise); + const { result, rerender } = renderHook( + ({ sessionId }: { sessionId: string }) => useSessionPromptQueue(sessionId), + { initialProps: { sessionId: 'sess-1' } }, + ); + + act(() => { + void result.current.refresh(); + }); + + await act(async () => { + rerender({ sessionId: 'sess-2' }); + }); + + await act(async () => { + request.resolve({ items: [buildQueuedPrompt({ sessionID: 'sess-1' })] }); + await request.promise; + }); + + expect(result.current.items).toEqual([]); + }); + + it('lets pushed queue items supersede an older refresh response', async () => { + const request = deferred<{ items: QueuedPrompt[] }>(); + listPromptQueueMock.mockReturnValue(request.promise); + const pushed = buildQueuedPrompt({ id: 'queue-pushed', displayText: 'pushed' }); + const stale = buildQueuedPrompt({ id: 'queue-stale', displayText: 'stale' }); + const { result } = renderHook(() => useSessionPromptQueue('sess-1')); + + act(() => { + void result.current.refresh(); + result.current.applyItems([pushed]); + }); + + await act(async () => { + request.resolve({ items: [stale] }); + await request.promise; + }); + + expect(result.current.items).toEqual([pushed]); + expect(result.current.expanded).toBe(true); + }); + + it('enqueues a prompt and refreshes the queue', async () => { + const item = buildQueuedPrompt(); + listPromptQueueMock.mockResolvedValue({ items: [item] }); + const { result } = renderHook(() => useSessionPromptQueue('sess-1')); + + await act(async () => { + await result.current.enqueue({ + parts: [{ type: 'text', text: 'next' }], + agent: 'rex', + }); + }); + + expect(enqueuePromptMock).toHaveBeenCalledWith('sess-1', { + parts: [{ type: 'text', text: 'next' }], + agent: 'rex', + }); + expect(result.current.items).toEqual([item]); + }); + + it('saves edited queued text and clears edit state', async () => { + const item = buildQueuedPrompt(); + const updated = buildQueuedPrompt({ displayText: 'updated' }); + listPromptQueueMock.mockResolvedValue({ items: [updated] }); + const { result } = renderHook(() => useSessionPromptQueue('sess-1')); + + act(() => { + result.current.startEdit(item); + result.current.setEditingText(' updated '); + }); + + await act(async () => { + await result.current.saveEdit(item); + }); + + expect(updateQueuedPromptMock).toHaveBeenCalledWith('sess-1', 'queue-1', 'updated'); + expect(result.current.editingId).toBeNull(); + expect(result.current.editingText).toBe(''); + expect(result.current.actionId).toBeNull(); + expect(result.current.items).toEqual([updated]); + }); + + it('removes and runs queued prompts while clearing matching edit state', async () => { + const item = buildQueuedPrompt(); + const { result } = renderHook(() => useSessionPromptQueue('sess-1')); + + act(() => { + result.current.startEdit(item); + }); + + await act(async () => { + await result.current.remove(item); + }); + + expect(removeQueuedPromptMock).toHaveBeenCalledWith('sess-1', 'queue-1'); + expect(result.current.editingId).toBeNull(); + + act(() => { + result.current.startEdit(item); + }); + + await act(async () => { + await result.current.runNow(item); + }); + + expect(runQueuedPromptNowMock).toHaveBeenCalledWith('sess-1', 'queue-1'); + expect(result.current.editingId).toBeNull(); + expect(result.current.actionId).toBeNull(); + }); +}); diff --git a/webui/src/features/session-chat/useSessionPromptQueue.ts b/webui/src/features/session-chat/useSessionPromptQueue.ts new file mode 100644 index 000000000..4ecf0af6f --- /dev/null +++ b/webui/src/features/session-chat/useSessionPromptQueue.ts @@ -0,0 +1,148 @@ +import { useCallback, useLayoutEffect, useRef, useState } from 'react'; + +import { sessionApi, type QueuedPrompt } from '@/api/session'; + +export interface EnqueuePromptPayload { + parts: Array>; + agent?: string; + model?: Record; + variant?: string; + displayText?: string; +} + +export function useSessionPromptQueue(sessionId?: string | null) { + const [items, setItems] = useState([]); + const [expanded, setExpanded] = useState(true); + const [editingId, setEditingId] = useState(null); + const [editingText, setEditingText] = useState(''); + const [actionId, setActionId] = useState(null); + const requestSeqRef = useRef(0); + + const reset = useCallback(() => { + requestSeqRef.current += 1; + setItems([]); + setEditingId(null); + setEditingText(''); + setActionId(null); + }, []); + + useLayoutEffect(() => { + reset(); + }, [reset, sessionId]); + + const refresh = useCallback(async () => { + const requestSeq = ++requestSeqRef.current; + if (!sessionId) { + setItems([]); + return []; + } + + try { + const response = await sessionApi.listPromptQueue(sessionId); + const nextItems = response.items ?? []; + if (requestSeq === requestSeqRef.current) { + setItems(nextItems); + } + return nextItems; + } catch (err) { + if (requestSeq === requestSeqRef.current) { + console.warn('[SessionChat] Failed to fetch prompt queue:', err); + } + return []; + } + }, [sessionId]); + + const applyItems = useCallback((nextItems: QueuedPrompt[]) => { + requestSeqRef.current += 1; + setItems(nextItems); + if (nextItems.length > 0) setExpanded(true); + }, []); + + const enqueue = useCallback(async (payload: EnqueuePromptPayload) => { + if (!sessionId) return; + await sessionApi.enqueuePrompt(sessionId, payload); + await refresh(); + setExpanded(true); + }, [refresh, sessionId]); + + const startEdit = useCallback((item: QueuedPrompt) => { + setEditingId(item.id); + setEditingText(getQueuedPromptText(item)); + }, []); + + const cancelEdit = useCallback(() => { + setEditingId(null); + setEditingText(''); + }, []); + + const saveEdit = useCallback(async (item: QueuedPrompt) => { + if (!sessionId) return false; + const text = editingText.trim(); + if (!text) return false; + setActionId(item.id); + try { + await sessionApi.updateQueuedPrompt(sessionId, item.id, text); + cancelEdit(); + await refresh(); + return true; + } finally { + setActionId(null); + } + }, [cancelEdit, editingText, refresh, sessionId]); + + const remove = useCallback(async (item: QueuedPrompt) => { + if (!sessionId) return false; + setActionId(item.id); + try { + await sessionApi.removeQueuedPrompt(sessionId, item.id); + if (editingId === item.id) cancelEdit(); + await refresh(); + return true; + } finally { + setActionId(null); + } + }, [cancelEdit, editingId, refresh, sessionId]); + + const runNow = useCallback(async (item: QueuedPrompt) => { + if (!sessionId) return false; + setActionId(item.id); + try { + await sessionApi.runQueuedPromptNow(sessionId, item.id); + if (editingId === item.id) cancelEdit(); + await refresh(); + return true; + } finally { + setActionId(null); + } + }, [cancelEdit, editingId, refresh, sessionId]); + + return { + items, + expanded, + setExpanded, + editingId, + editingText, + setEditingText, + actionId, + refresh, + applyItems, + enqueue, + startEdit, + cancelEdit, + saveEdit, + remove, + runNow, + reset, + } as const; +} + +export function getQueuedPromptText(item: QueuedPrompt): string { + const displayText = item.displayText ?? item.display_text; + if (typeof displayText === 'string' && displayText.trim()) { + return displayText; + } + const firstText = item.parts + .map((part) => typeof part.text === 'string' ? part.text : '') + .find((text) => text.trim()); + return firstText || ''; +} diff --git a/webui/src/hooks/useAgents.test.tsx b/webui/src/hooks/useAgents.test.tsx index 4595be3f2..50274bf27 100644 --- a/webui/src/hooks/useAgents.test.tsx +++ b/webui/src/hooks/useAgents.test.tsx @@ -1,7 +1,7 @@ -import { renderHook, waitFor } from '@testing-library/react'; +import { act, renderHook, waitFor } from '@testing-library/react'; import { beforeEach, describe, expect, it, vi } from 'vitest'; -import { useAgents } from './useAgents'; +import { __resetAgentsResourceForTesting, useAgents } from './useAgents'; const { listMock, refreshMock } = vi.hoisted(() => ({ listMock: vi.fn(), @@ -18,6 +18,7 @@ vi.mock('@/api/agent', () => ({ describe('useAgents', () => { beforeEach(() => { vi.clearAllMocks(); + __resetAgentsResourceForTesting(); }); it('returns an empty array when the API payload is not an array', async () => { @@ -65,4 +66,30 @@ describe('useAgents', () => { expect(refreshMock).toHaveBeenCalledTimes(1); expect(listMock).toHaveBeenCalledTimes(2); }); + + it('shares the initial agent list request across concurrent hook instances', async () => { + let resolveList: (value: { data: any[] }) => void = () => {}; + listMock.mockReturnValue(new Promise((resolve) => { + resolveList = resolve; + })); + + const first = renderHook(() => useAgents()); + const second = renderHook(() => useAgents()); + + expect(listMock).toHaveBeenCalledTimes(1); + + await act(async () => { + resolveList({ + data: [{ name: 'rex', mode: 'primary' }], + }); + }); + + await waitFor(() => { + expect(first.result.current.loading).toBe(false); + expect(second.result.current.loading).toBe(false); + }); + + expect(first.result.current.agents).toHaveLength(1); + expect(second.result.current.agents).toHaveLength(1); + }); }); diff --git a/webui/src/hooks/useAgents.ts b/webui/src/hooks/useAgents.ts index 1bc42f022..1113a63ad 100644 --- a/webui/src/hooks/useAgents.ts +++ b/webui/src/hooks/useAgents.ts @@ -1,66 +1,76 @@ -import { useState, useEffect, useCallback, useRef } from 'react'; +import { useCallback } from 'react'; import { agentAPI, Agent } from '@/api/agent'; +import { createSharedResource, useRefreshOnResume, useSharedResource } from './useSharedResource'; -export function useAgents() { - const [agents, setAgents] = useState([]); - const [loading, setLoading] = useState(true); - const [error, setError] = useState(null); - const lastRefreshRef = useRef(0); +const AGENT_LIST_STALE_TIME_MS = 5000; +const AGENT_REFRESH_MIN_INTERVAL_MS = 1000; - const fetchAgents = useCallback(async (showLoading = true) => { - try { - if (showLoading) setLoading(true); - setError(null); - const response = await agentAPI.list(); - setAgents(Array.isArray(response.data) ? response.data : []); - } catch (err: any) { - setError(err.message || 'Failed to fetch agents'); - setAgents([]); - } finally { - if (showLoading) setLoading(false); - } - }, []); +const agentsResource = createSharedResource({ + initialData: [], + staleTimeMs: AGENT_LIST_STALE_TIME_MS, + minFetchIntervalMs: AGENT_REFRESH_MIN_INTERVAL_MS, + fetcher: async () => { + const response = await agentAPI.list(); + return Array.isArray(response.data) ? response.data : []; + }, + fallbackDataOnError: [], + getErrorMessage: (err) => (err instanceof Error && err.message ? err.message : 'Failed to fetch agents'), +}); - const refreshAndFetch = useCallback(async () => { - const now = Date.now(); - if (now - lastRefreshRef.current < 1000) return; - lastRefreshRef.current = now; +let agentRefreshInFlight: Promise | null = null; +let lastAgentRefreshAt = 0; - try { - await agentAPI.refresh(); - } catch { - // Best-effort: if refresh fails, still try to fetch the latest list. - } +function refreshAgentsResource(): Promise { + const now = Date.now(); + + if (agentRefreshInFlight) { + return agentRefreshInFlight; + } - await fetchAgents(false); - }, [fetchAgents]); + if (now - lastAgentRefreshAt < AGENT_REFRESH_MIN_INTERVAL_MS) { + return Promise.resolve(); + } + + lastAgentRefreshAt = now; + agentRefreshInFlight = agentAPI.refresh() + .catch(() => { + // Best-effort: if refresh fails, still try to fetch the latest list. + }) + .then(() => agentsResource.fetch({ force: true, silent: true })) + .then(() => undefined) + .finally(() => { + agentRefreshInFlight = null; + }); - useEffect(() => { - void fetchAgents(); + return agentRefreshInFlight; +} - const handleVisibilityChange = () => { - if (document.visibilityState === 'visible') { - void refreshAndFetch(); - } - }; +export function __resetAgentsResourceForTesting(): void { + agentsResource.resetForTesting(); + agentRefreshInFlight = null; + lastAgentRefreshAt = 0; +} - const handleWindowFocus = () => { - void refreshAndFetch(); - }; +export function useAgents() { + const { + data: agents, + loading, + error, + refetch: fetchAgents, + } = useSharedResource(agentsResource); - document.addEventListener('visibilitychange', handleVisibilityChange); - window.addEventListener('focus', handleWindowFocus); + const refreshAndFetch = useCallback(() => refreshAgentsResource(), []); + useRefreshOnResume(refreshAndFetch); - return () => { - document.removeEventListener('visibilitychange', handleVisibilityChange); - window.removeEventListener('focus', handleWindowFocus); - }; - }, [fetchAgents, refreshAndFetch]); + const refetch = useCallback( + (showLoading = true) => fetchAgents({ silent: !showLoading }), + [fetchAgents], + ); return { agents, loading, error, - refetch: (showLoading = true) => fetchAgents(showLoading), + refetch, }; } diff --git a/webui/src/hooks/useBackendStatus.test.tsx b/webui/src/hooks/useBackendStatus.test.tsx new file mode 100644 index 000000000..d26734492 --- /dev/null +++ b/webui/src/hooks/useBackendStatus.test.tsx @@ -0,0 +1,100 @@ +import { act, renderHook, waitFor } from '@testing-library/react'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { __resetBackendStatusResourceForTesting, useBackendStatus } from './useBackendStatus'; + +const { apiGetMock } = vi.hoisted(() => ({ + apiGetMock: vi.fn(), +})); + +vi.mock('@/api/client', () => ({ + apiClient: { + get: apiGetMock, + }, +})); + +describe('useBackendStatus', () => { + beforeEach(() => { + vi.clearAllMocks(); + __resetBackendStatusResourceForTesting(); + }); + + afterEach(() => { + vi.useRealTimers(); + __resetBackendStatusResourceForTesting(); + }); + + it('shares the initial health request across concurrent hook instances', async () => { + let resolveHealth: (value: { status: number }) => void = () => {}; + apiGetMock.mockReturnValue(new Promise((resolve) => { + resolveHealth = resolve; + })); + + const first = renderHook(() => useBackendStatus()); + const second = renderHook(() => useBackendStatus()); + + expect(apiGetMock).toHaveBeenCalledTimes(1); + expect(apiGetMock).toHaveBeenCalledWith('/api/health', { timeout: 5000 }); + + await act(async () => { + resolveHealth({ status: 200 }); + }); + + await waitFor(() => { + expect(first.result.current.status).toBe('connected'); + expect(second.result.current.status).toBe('connected'); + }); + }); + + it('keeps one health polling loop for multiple hook instances', async () => { + vi.useFakeTimers(); + apiGetMock.mockResolvedValue({ status: 200 }); + + const first = renderHook(() => useBackendStatus()); + const second = renderHook(() => useBackendStatus()); + + await act(async () => { + await Promise.resolve(); + await Promise.resolve(); + }); + expect(first.result.current.status).toBe('connected'); + expect(second.result.current.status).toBe('connected'); + expect(apiGetMock).toHaveBeenCalledTimes(1); + + await act(async () => { + await vi.advanceTimersByTimeAsync(3_600_000); + }); + + expect(apiGetMock).toHaveBeenCalledTimes(2); + + first.unmount(); + second.unmount(); + + await act(async () => { + await vi.advanceTimersByTimeAsync(3_600_000); + }); + + expect(apiGetMock).toHaveBeenCalledTimes(2); + }); + + it('keeps checkHealth compatible with the previous boolean return', async () => { + apiGetMock + .mockResolvedValueOnce({ status: 200 }) + .mockRejectedValueOnce({ response: { status: 503 } }); + + const { result } = renderHook(() => useBackendStatus()); + + await waitFor(() => { + expect(result.current.status).toBe('connected'); + }); + + let ok = true; + await act(async () => { + ok = await result.current.checkHealth(); + }); + + expect(ok).toBe(false); + expect(result.current.status).toBe('connecting'); + expect(result.current.message).toBe('后端服务暂时不可用'); + }); +}); diff --git a/webui/src/hooks/useBackendStatus.ts b/webui/src/hooks/useBackendStatus.ts index a8f734f41..2e735bb89 100644 --- a/webui/src/hooks/useBackendStatus.ts +++ b/webui/src/hooks/useBackendStatus.ts @@ -1,5 +1,6 @@ -import { useState, useEffect, useCallback } from 'react'; +import { useCallback, useEffect } from 'react'; import { apiClient } from '@/api/client'; +import { createSharedResource, useSharedResource } from './useSharedResource'; export type BackendStatus = 'connected' | 'connecting' | 'disconnected' | 'error'; @@ -9,73 +10,108 @@ interface BackendStatusState { lastChecked?: Date; } +const BACKEND_HEALTH_POLL_INTERVAL_MS = 3_600_000; +const BACKEND_HEALTH_STALE_TIME_MS = 5_000; +const BACKEND_HEALTH_MIN_FETCH_INTERVAL_MS = 1_000; + +async function fetchBackendHealth(): Promise { + try { + // 使用较短的超时时间快速检测连接问题 + const response = await apiClient.get('/api/health', { + timeout: 5000, + }); + + if (response.status === 200) { + return { + status: 'connected', + message: '后端服务正常', + lastChecked: new Date(), + }; + } + + return { + status: 'error', + message: '后端返回异常状态', + lastChecked: new Date(), + }; + } catch (error: any) { + // 根据错误类型提供不同的反馈 + let message = '无法连接到后端服务'; + let status: BackendStatus = 'disconnected'; + + if (error.code === 'ECONNABORTED') { + message = '连接超时'; + } else if (error.code === 'ERR_NETWORK') { + message = '后端服务可能正在重启'; + status = 'connecting'; + } else if (error.response?.status === 503) { + message = '后端服务暂时不可用'; + status = 'connecting'; + } + + return { + status, + message, + lastChecked: new Date(), + }; + } +} + +const backendStatusResource = createSharedResource({ + initialData: { + status: 'connecting', + }, + staleTimeMs: BACKEND_HEALTH_STALE_TIME_MS, + minFetchIntervalMs: BACKEND_HEALTH_MIN_FETCH_INTERVAL_MS, + fetcher: fetchBackendHealth, + fallbackDataOnError: (previous) => previous, +}); + +let backendHealthPollSubscriberCount = 0; +let backendHealthPollIntervalId: number | null = null; + +function subscribeBackendHealthPolling(): () => void { + backendHealthPollSubscriberCount += 1; + + if (backendHealthPollSubscriberCount === 1) { + backendHealthPollIntervalId = window.setInterval(() => { + void backendStatusResource.fetch({ silent: true }); + }, BACKEND_HEALTH_POLL_INTERVAL_MS); + } + + return () => { + backendHealthPollSubscriberCount = Math.max(0, backendHealthPollSubscriberCount - 1); + if (backendHealthPollSubscriberCount === 0 && backendHealthPollIntervalId !== null) { + window.clearInterval(backendHealthPollIntervalId); + backendHealthPollIntervalId = null; + } + }; +} + +export function __resetBackendStatusResourceForTesting(): void { + backendStatusResource.resetForTesting(); + backendHealthPollSubscriberCount = 0; + if (backendHealthPollIntervalId !== null) { + window.clearInterval(backendHealthPollIntervalId); + backendHealthPollIntervalId = null; + } +} + /** * 监控后端连接状态的 Hook * 定期检查后端健康状态,并在后端重启时提供友好的用户反馈 */ export function useBackendStatus() { - const [state, setState] = useState({ - status: 'connecting', - }); + const { data: state } = useSharedResource(backendStatusResource); const checkHealth = useCallback(async () => { - try { - // 使用较短的超时时间快速检测连接问题 - const response = await apiClient.get('/api/health', { - timeout: 5000, - }); - - if (response.status === 200) { - setState({ - status: 'connected', - message: '后端服务正常', - lastChecked: new Date(), - }); - return true; - } - - setState({ - status: 'error', - message: '后端返回异常状态', - lastChecked: new Date(), - }); - return false; - } catch (error: any) { - // 根据错误类型提供不同的反馈 - let message = '无法连接到后端服务'; - let status: BackendStatus = 'disconnected'; - - if (error.code === 'ECONNABORTED') { - message = '连接超时'; - } else if (error.code === 'ERR_NETWORK') { - message = '后端服务可能正在重启'; - status = 'connecting'; - } else if (error.response?.status === 503) { - message = '后端服务暂时不可用'; - status = 'connecting'; - } - - setState({ - status, - message, - lastChecked: new Date(), - }); - - return false; - } + const nextState = await backendStatusResource.fetch({ force: true }); + return nextState.status === 'connected'; }, []); useEffect(() => { - // 立即检查一次 - checkHealth(); - - // 每 1 小时检查一次(3600000 毫秒) - const interval = setInterval(() => { - checkHealth(); - }, 3600000); - - return () => clearInterval(interval); - }, [checkHealth]); + return subscribeBackendHealthPolling(); + }, []); return { ...state, diff --git a/webui/src/hooks/useChatModelResources.ts b/webui/src/hooks/useChatModelResources.ts new file mode 100644 index 000000000..cd93f25af --- /dev/null +++ b/webui/src/hooks/useChatModelResources.ts @@ -0,0 +1,67 @@ +import { defaultModelAPI, modelV2API } from '@/api/provider'; +import type { ModelDefinitionV2 } from '@/types'; +import { + createSharedResource, + useSharedResource, + type SharedResourceFetchOptions, +} from './useSharedResource'; + +export type ResolvedDefaultModel = { + providerID: string; + modelID: string; +} | null; + +const CHAT_MODEL_RESOURCES_STALE_TIME_MS = 10000; + +const enabledModelDefinitionsResource = createSharedResource({ + initialData: [], + staleTimeMs: CHAT_MODEL_RESOURCES_STALE_TIME_MS, + minFetchIntervalMs: 1000, + fetcher: async () => { + const response = await modelV2API.listDefinitions({ enabled_only: true }); + return response?.data?.models ?? []; + }, + fallbackDataOnError: [], +}); + +const resolvedDefaultModelResource = createSharedResource({ + initialData: null, + staleTimeMs: CHAT_MODEL_RESOURCES_STALE_TIME_MS, + minFetchIntervalMs: 1000, + fetcher: async () => { + const response = await defaultModelAPI.getResolved(); + const { provider_id: providerID, model_id: modelID } = response?.data ?? {}; + if (!providerID || !modelID) return null; + return { providerID, modelID }; + }, + fallbackDataOnError: null, +}); + +export function useEnabledChatModelDefinitions() { + return useSharedResource(enabledModelDefinitionsResource); +} + +export function useResolvedDefaultModel(enabled: boolean) { + return useSharedResource(resolvedDefaultModelResource, { + enabled, + loadOnMount: enabled, + silentInitialLoad: true, + }); +} + +export function fetchEnabledChatModelDefinitions( + options?: SharedResourceFetchOptions, +): Promise { + return enabledModelDefinitionsResource.fetch(options); +} + +export function fetchResolvedDefaultModel( + options?: SharedResourceFetchOptions, +): Promise { + return resolvedDefaultModelResource.fetch(options); +} + +export function __resetChatModelResourcesForTesting(): void { + enabledModelDefinitionsResource.resetForTesting(); + resolvedDefaultModelResource.resetForTesting(); +} diff --git a/webui/src/hooks/useDefaultModelVision.test.ts b/webui/src/hooks/useDefaultModelVision.test.ts index f1e00c7fe..a0c37fb80 100644 --- a/webui/src/hooks/useDefaultModelVision.test.ts +++ b/webui/src/hooks/useDefaultModelVision.test.ts @@ -4,7 +4,7 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; // We use dynamic imports so vi.mock() is hoisted correctly. vi.mock('@/api/provider', () => ({ defaultModelAPI: { getResolved: vi.fn() }, - modelV2API: { getDefinition: vi.fn() }, + modelV2API: { listDefinitions: vi.fn(), getDefinition: vi.fn() }, })); import { defaultModelAPI, modelV2API } from '@/api/provider'; @@ -12,6 +12,7 @@ import { __resetVisionCacheForTesting, MODEL_CHANGED_EVENT, useDefaultModelVisio import { renderHook, act, waitFor } from '@testing-library/react'; const mockResolved = defaultModelAPI.getResolved as ReturnType; +const mockDefinitions = modelV2API.listDefinitions as ReturnType; const mockDefinition = modelV2API.getDefinition as ReturnType; function makeResolvedResp(provider_id = 'openai', model_id = 'gpt-4o') { @@ -22,6 +23,27 @@ function makeDefResp(caps: Record, fetchFrom: 'predefined' | 'c return { data: { fetch_from: fetchFrom, capabilities: caps } }; } +function makeDefinitionsResp( + caps: Record, + fetchFrom: 'predefined' | 'customizable' = 'customizable', + providerId = 'openai', + modelId = 'gpt-4o', +) { + return { + data: { + models: [{ + id: modelId, + name: modelId, + provider_id: providerId, + model_type: 'chat', + status: 'active', + fetch_from: fetchFrom, + capabilities: caps, + }], + }, + }; +} + describe('useDefaultModelVision', () => { beforeEach(() => { __resetVisionCacheForTesting(); @@ -34,7 +56,7 @@ describe('useDefaultModelVision', () => { it('returns null initially then true for a vision model', async () => { mockResolved.mockResolvedValue(makeResolvedResp()); - mockDefinition.mockResolvedValue(makeDefResp({ supports_vision: true })); + mockDefinitions.mockResolvedValue(makeDefinitionsResp({ supports_vision: true })); const { result } = renderHook(() => useDefaultModelVision()); expect(result.current).toBeNull(); @@ -44,7 +66,7 @@ describe('useDefaultModelVision', () => { it('returns false for a non-vision customizable model', async () => { mockResolved.mockResolvedValue(makeResolvedResp()); - mockDefinition.mockResolvedValue(makeDefResp({ supports_vision: false })); + mockDefinitions.mockResolvedValue(makeDefinitionsResp({ supports_vision: false })); const { result } = renderHook(() => useDefaultModelVision()); await waitFor(() => expect(result.current).toBe(false)); @@ -52,7 +74,7 @@ describe('useDefaultModelVision', () => { it('returns false for a predefined model without the built-in allowlist even when it declares vision support', async () => { mockResolved.mockResolvedValue(makeResolvedResp('openai', 'gpt-4o')); - mockDefinition.mockResolvedValue(makeDefResp({ supports_vision: true }, 'predefined')); + mockDefinitions.mockResolvedValue(makeDefinitionsResp({ supports_vision: true }, 'predefined')); const { result } = renderHook(() => useDefaultModelVision()); await waitFor(() => expect(result.current).toBe(false)); @@ -60,7 +82,12 @@ describe('useDefaultModelVision', () => { it('returns true for an allowlisted predefined vision model', async () => { mockResolved.mockResolvedValue(makeResolvedResp('threatbook-cn-llm', 'qwen3.6-plus')); - mockDefinition.mockResolvedValue(makeDefResp({ supports_vision: true }, 'predefined')); + mockDefinitions.mockResolvedValue(makeDefinitionsResp( + { supports_vision: true }, + 'predefined', + 'threatbook-cn-llm', + 'qwen3.6-plus', + )); const { result } = renderHook(() => useDefaultModelVision()); await waitFor(() => expect(result.current).toBe(true)); @@ -68,7 +95,18 @@ describe('useDefaultModelVision', () => { it('returns null when capabilities are absent', async () => { mockResolved.mockResolvedValue(makeResolvedResp()); - mockDefinition.mockResolvedValue({ data: {} }); + mockDefinitions.mockResolvedValue({ + data: { + models: [{ + id: 'gpt-4o', + name: 'gpt-4o', + provider_id: 'openai', + model_type: 'chat', + status: 'active', + fetch_from: 'customizable', + }], + }, + }); const { result } = renderHook(() => useDefaultModelVision()); await waitFor(() => expect(result.current).toBeNull()); @@ -76,26 +114,27 @@ describe('useDefaultModelVision', () => { it('module-level cache: API called only once for multiple concurrent hooks', async () => { mockResolved.mockResolvedValue(makeResolvedResp()); - mockDefinition.mockResolvedValue(makeDefResp({ supports_vision: true })); + mockDefinitions.mockResolvedValue(makeDefinitionsResp({ supports_vision: true })); renderHook(() => useDefaultModelVision()); renderHook(() => useDefaultModelVision()); renderHook(() => useDefaultModelVision()); await waitFor(() => expect(mockResolved).toHaveBeenCalledTimes(1)); - expect(mockDefinition).toHaveBeenCalledTimes(1); + expect(mockDefinitions).toHaveBeenCalledTimes(1); + expect(mockDefinition).not.toHaveBeenCalled(); }); it('MODEL_CHANGED_EVENT invalidates cache and notifies subscribers', async () => { // First resolve: non-vision mockResolved.mockResolvedValue(makeResolvedResp()); - mockDefinition.mockResolvedValue(makeDefResp({ supports_vision: false })); + mockDefinitions.mockResolvedValueOnce(makeDefinitionsResp({ supports_vision: false })); const { result } = renderHook(() => useDefaultModelVision()); await waitFor(() => expect(result.current).toBe(false)); // Change to a vision model and dispatch the event - mockDefinition.mockResolvedValue(makeDefResp({ supports_vision: true })); + mockDefinitions.mockResolvedValueOnce(makeDefinitionsResp({ supports_vision: true })); act(() => { window.dispatchEvent(new CustomEvent(MODEL_CHANGED_EVENT)); @@ -103,12 +142,12 @@ describe('useDefaultModelVision', () => { await waitFor(() => expect(result.current).toBe(true)); // After invalidation, the API was called a second time - expect(mockDefinition).toHaveBeenCalledTimes(2); + expect(mockDefinitions).toHaveBeenCalledTimes(2); }); it('detects vision via modalities.input', async () => { mockResolved.mockResolvedValue(makeResolvedResp()); - mockDefinition.mockResolvedValue(makeDefResp({ modalities: { input: ['text', 'image'] } })); + mockDefinitions.mockResolvedValue(makeDefinitionsResp({ modalities: { input: ['text', 'image'] } })); const { result } = renderHook(() => useDefaultModelVision()); await waitFor(() => expect(result.current).toBe(true)); @@ -116,10 +155,21 @@ describe('useDefaultModelVision', () => { it('detects vision via features array', async () => { mockResolved.mockResolvedValue(makeResolvedResp()); - mockDefinition.mockResolvedValue(makeDefResp({ features: ['vision', 'tools'] })); + mockDefinitions.mockResolvedValue(makeDefinitionsResp({ features: ['vision', 'tools'] })); + + const { result } = renderHook(() => useDefaultModelVision()); + await waitFor(() => expect(result.current).toBe(true)); + }); + + it('falls back to getDefinition when the default model is missing from shared definitions', async () => { + mockResolved.mockResolvedValue(makeResolvedResp()); + mockDefinitions.mockResolvedValue({ data: { models: [] } }); + mockDefinition.mockResolvedValue(makeDefResp({ supports_vision: true })); const { result } = renderHook(() => useDefaultModelVision()); await waitFor(() => expect(result.current).toBe(true)); + + expect(mockDefinition).toHaveBeenCalledWith('openai', 'gpt-4o'); }); it('returns null on API error', async () => { diff --git a/webui/src/hooks/useDefaultModelVision.ts b/webui/src/hooks/useDefaultModelVision.ts index cabfcce23..90e980597 100644 --- a/webui/src/hooks/useDefaultModelVision.ts +++ b/webui/src/hooks/useDefaultModelVision.ts @@ -1,5 +1,12 @@ import { useEffect, useState } from 'react'; -import { defaultModelAPI, modelV2API } from '@/api/provider'; +import { modelV2API } from '@/api/provider'; +import type { ModelDefinitionV2 } from '@/types'; +import { + __resetChatModelResourcesForTesting, + fetchEnabledChatModelDefinitions, + fetchResolvedDefaultModel, +} from './useChatModelResources'; +import type { SharedResourceFetchOptions } from './useSharedResource'; /** * Detect whether the resolved default LLM model supports image (vision) input. @@ -21,10 +28,12 @@ import { defaultModelAPI, modelV2API } from '@/api/provider'; * Caching: * The resolved capability is cached at module scope so each newly mounted * composer (sidebar drawer, dialog, etc.) reuses the in-flight or - * completed lookup instead of firing a fresh `getResolved + getDefinition` - * pair. The cache is invalidated when ``MODEL_CHANGED_EVENT`` fires — - * pages that change the default model (see ``Model/index.tsx``) dispatch - * that event after a successful update so this hook re-resolves. + * completed lookup instead of firing a fresh request pair. The default + * model and enabled model definitions come from `useChatModelResources`, so + * composer chrome and model pickers reuse the same in-flight resource. The + * cache is invalidated when ``MODEL_CHANGED_EVENT`` fires — pages that + * change the default model (see ``Model/index.tsx``) dispatch that event + * after a successful update so this hook re-resolves. */ /** Window event other code can dispatch to invalidate the cached vision capability. */ @@ -40,34 +49,65 @@ function allowsBuiltInVision(modelId: string): boolean { return lowered.includes('qwen3.6-plus') || lowered.includes('kimi-k2.6'); } -async function resolveVisionSupport(): Promise { - try { - const resolvedResp = await defaultModelAPI.getResolved(); - const { provider_id, model_id } = resolvedResp.data; - if (!provider_id || !model_id) return null; - const defResp = await modelV2API.getDefinition(provider_id, model_id); - const def: any = defResp.data; - if (!def) return null; - const caps = def.capabilities; - if (!caps) return null; - const builtInVisionAllowed = allowsBuiltInVision(model_id); - if ( - caps.supports_vision === true || - caps.modalities?.input?.includes('image') || - (caps.features ?? []).includes('vision') - ) { - if (def.fetch_from !== 'customizable' && !builtInVisionAllowed) { - return false; - } - return true; - } - if (caps.supports_vision === false) { - return false; - } - if (def.fetch_from !== 'customizable') { +function visionSupportFromDefinition( + def: Pick | null | undefined, + modelId: string, +): VisionState { + const caps = def?.capabilities; + if (!caps) return null; + + const builtInVisionAllowed = allowsBuiltInVision(modelId); + if ( + caps.supports_vision === true || + caps.modalities?.input?.includes('image') || + (caps.features ?? []).includes('vision') + ) { + if (def.fetch_from !== 'customizable' && !builtInVisionAllowed) { return false; } + return true; + } + if (caps.supports_vision === false) { + return false; + } + if (def.fetch_from !== 'customizable') { + return false; + } + return null; +} + +async function getDefaultModelDefinition( + providerID: string, + modelID: string, + definitions: ModelDefinitionV2[], +): Promise | null> { + const sharedDefinition = definitions.find( + (definition) => definition.provider_id === providerID && definition.id === modelID, + ); + if (sharedDefinition) return sharedDefinition; + + try { + const defResp = await modelV2API.getDefinition(providerID, modelID); + return defResp.data ?? null; + } catch { return null; + } +} + +async function resolveVisionSupport(options?: SharedResourceFetchOptions): Promise { + try { + const [resolvedDefaultModel, definitions] = await Promise.all([ + fetchResolvedDefaultModel(options), + fetchEnabledChatModelDefinitions(options), + ]); + if (!resolvedDefaultModel) return null; + + const def = await getDefaultModelDefinition( + resolvedDefaultModel.providerID, + resolvedDefaultModel.modelID, + definitions, + ); + return visionSupportFromDefinition(def, resolvedDefaultModel.modelID); } catch { return null; } @@ -85,7 +125,7 @@ function invalidateAndRefetch(): void { // races ahead of this one cannot deliver a stale value to subscribers. // We only notify if our promise is still the current cached one by the // time it resolves. - const next = resolveVisionSupport(); + const next = resolveVisionSupport({ force: true, silent: true }); cachedPromise = next; next.then((value) => { if (cachedPromise === next) { @@ -107,6 +147,7 @@ if (typeof window !== 'undefined') { export function __resetVisionCacheForTesting(): void { cachedPromise = null; subscribers.clear(); + __resetChatModelResourcesForTesting(); } export function useDefaultModelVision(): VisionState { diff --git a/webui/src/hooks/useHooks.test.tsx b/webui/src/hooks/useHooks.test.tsx new file mode 100644 index 000000000..bf21b6e18 --- /dev/null +++ b/webui/src/hooks/useHooks.test.tsx @@ -0,0 +1,82 @@ +import { act, renderHook, waitFor } from '@testing-library/react'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +import { __resetHookStatusResourceForTesting, useHooks } from './useHooks'; + +const { getStatusMock } = vi.hoisted(() => ({ + getStatusMock: vi.fn(), +})); + +vi.mock('@/api/hooks', () => ({ + hooksApi: { + getStatus: getStatusMock, + }, +})); + +function makeHookStatus(overrides: Record = {}) { + return { + enabled: true, + session_memory: { + enabled: true, + message_count: 3, + use_llm_slug: false, + slug_timeout: 10, + }, + stats: { + total_event_keys: 2, + total_handlers: 4, + event_keys: {}, + }, + ...overrides, + }; +} + +describe('useHooks', () => { + beforeEach(() => { + vi.clearAllMocks(); + __resetHookStatusResourceForTesting(); + }); + + it('shares the initial hook status request across concurrent hook instances', async () => { + let resolveStatus: (value: ReturnType) => void = () => {}; + getStatusMock.mockReturnValue(new Promise((resolve) => { + resolveStatus = resolve; + })); + + const first = renderHook(() => useHooks()); + const second = renderHook(() => useHooks()); + + expect(getStatusMock).toHaveBeenCalledTimes(1); + + await act(async () => { + resolveStatus(makeHookStatus()); + }); + + await waitFor(() => { + expect(first.result.current.loading).toBe(false); + expect(second.result.current.loading).toBe(false); + }); + + expect(first.result.current.status?.enabled).toBe(true); + expect(second.result.current.status?.stats.total_handlers).toBe(4); + }); + + it('forces a new request when refetch is called', async () => { + getStatusMock + .mockResolvedValueOnce(makeHookStatus({ enabled: true })) + .mockResolvedValueOnce(makeHookStatus({ enabled: false })); + + const { result } = renderHook(() => useHooks()); + + await waitFor(() => { + expect(result.current.status?.enabled).toBe(true); + }); + + await act(async () => { + await result.current.refetch(); + }); + + expect(result.current.status?.enabled).toBe(false); + expect(getStatusMock).toHaveBeenCalledTimes(2); + }); +}); diff --git a/webui/src/hooks/useHooks.ts b/webui/src/hooks/useHooks.ts index d010c07ef..5d4b05b00 100644 --- a/webui/src/hooks/useHooks.ts +++ b/webui/src/hooks/useHooks.ts @@ -2,36 +2,29 @@ * React hook for managing hook system */ -import { useState, useEffect } from 'react'; import { hooksApi, type HookStatus } from '@/api/hooks'; +import { createSharedResource, useSharedResource } from './useSharedResource'; -export function useHooks() { - const [status, setStatus] = useState(null); - const [loading, setLoading] = useState(true); - const [error, setError] = useState(null); +const hookStatusResource = createSharedResource({ + initialData: null, + staleTimeMs: 5_000, + minFetchIntervalMs: 1_000, + fetcher: hooksApi.getStatus, + fallbackDataOnError: null, + getErrorMessage: (err) => (err instanceof Error && err.message ? err.message : 'Failed to fetch hook status'), +}); - const fetchStatus = async () => { - try { - setLoading(true); - setError(null); - const data = await hooksApi.getStatus(); - setStatus(data); - } catch (err: any) { - setError(err.message || 'Failed to fetch hook status'); - setStatus(null); - } finally { - setLoading(false); - } - }; +export function __resetHookStatusResourceForTesting(): void { + hookStatusResource.resetForTesting(); +} - useEffect(() => { - fetchStatus(); - }, []); +export function useHooks() { + const { data: status, loading, error, refetch } = useSharedResource(hookStatusResource); return { status, loading, error, - refetch: fetchStatus, + refetch, }; } diff --git a/webui/src/hooks/usePendingQuestions.ts b/webui/src/hooks/usePendingQuestions.ts index 044787a0b..a4d17334d 100644 --- a/webui/src/hooks/usePendingQuestions.ts +++ b/webui/src/hooks/usePendingQuestions.ts @@ -1,125 +1,4 @@ -/** - * Hook to manage pending question state for the Question tool. - * - * Centralises SSE event handling, answer/reject API calls, and session-switch - * cleanup so that SessionChat and Session/index don't duplicate this logic. - */ - -import { useState, useCallback } from 'react'; -import type { QuestionItem } from '@/components/common/QuestionTool'; -import client from '@/api/client'; - -export interface PendingQuestion { - requestId: string; - questions: QuestionItem[]; -} - -interface PendingQuestionApiResponse { - id?: string; - questions?: QuestionItem[]; - tool?: { - callID?: string; - }; -} - -function isNotFoundError(error: unknown): boolean { - return !!( - error && - typeof error === 'object' && - 'response' in error && - (error as { response?: { status?: number } }).response?.status === 404 - ); -} - -export function usePendingQuestions() { - const [pendingQuestions, setPendingQuestions] = useState>({}); - - const removeByCallId = useCallback((callID: string) => { - setPendingQuestions(prev => { - const next = { ...prev }; - delete next[callID]; - return next; - }); - }, []); - - /** Call from the SSE event handler when `question.asked` fires. */ - const handleQuestionAsked = useCallback( - (callID: string, requestId: string, questions: QuestionItem[]) => { - setPendingQuestions(prev => ({ - ...prev, - [callID]: { requestId, questions }, - })); - }, - [], - ); - - /** Submit answers for a pending question. */ - const submitAnswer = useCallback( - async (callID: string, requestId: string, answers: string[][]) => { - try { - await client.post(`/api/question/${requestId}/reply`, { answers }); - } catch (error) { - if (!isNotFoundError(error)) throw error; - } - removeByCallId(callID); - }, - [removeByCallId], - ); - - /** Reject / skip a pending question. */ - const submitReject = useCallback( - async (callID: string, requestId: string) => { - try { - await client.post(`/api/question/${requestId}/reject`, {}); - } catch (error) { - if (!isNotFoundError(error)) throw error; - } - removeByCallId(callID); - }, - [removeByCallId], - ); - - /** Remove a pending question by request ID (e.g. after SSE reply/reject). */ - const removeByRequestId = useCallback((requestId: string) => { - setPendingQuestions(prev => { - const next = { ...prev }; - for (const [callID, pending] of Object.entries(prev)) { - if (pending.requestId === requestId) { - delete next[callID]; - } - } - return next; - }); - }, []); - - /** Recover pending questions for the active session after session switch / refresh. */ - const fetchPendingQuestions = useCallback(async (sessionId: string) => { - const response = await client.get(`/api/question/session/${sessionId}/pending`); - const next: Record = {}; - for (const item of response.data || []) { - const callID = item.tool?.callID; - const requestId = item.id; - if (!callID || !requestId) continue; - next[callID] = { - requestId, - questions: item.questions || [], - }; - } - setPendingQuestions(next); - }, []); - - /** Clear all pending questions (e.g. on session switch). */ - const clearAll = useCallback(() => { - setPendingQuestions({}); - }, []); - - return { - pendingQuestions, - handleQuestionAsked, - submitAnswer, - submitReject, - removeByRequestId, - fetchPendingQuestions, - clearAll, - } as const; -} +export { + usePendingQuestions, + type PendingQuestion, +} from '@/features/session-chat/usePendingQuestions'; diff --git a/webui/src/hooks/useProviders.test.tsx b/webui/src/hooks/useProviders.test.tsx new file mode 100644 index 000000000..9a493efb0 --- /dev/null +++ b/webui/src/hooks/useProviders.test.tsx @@ -0,0 +1,122 @@ +import { act, renderHook, waitFor } from '@testing-library/react'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +import { __resetProvidersResourceForTesting, useProviders } from './useProviders'; + +const { listMock } = vi.hoisted(() => ({ + listMock: vi.fn(), +})); + +vi.mock('@/api/provider', async () => { + const actual = await vi.importActual('@/api/provider'); + return { + ...actual, + providerAPI: { + ...actual.providerAPI, + list: listMock, + }, + }; +}); + +function makeProvider(id: string, models: Record = {}) { + return { + id, + name: id, + source: 'builtin', + env: [], + key: null, + options: {}, + models, + }; +} + +describe('useProviders', () => { + beforeEach(() => { + vi.clearAllMocks(); + __resetProvidersResourceForTesting(); + }); + + it('enriches providers with connected state and model counts', async () => { + listMock.mockResolvedValue({ + data: { + all: [ + makeProvider('openai', { 'gpt-4o': {} }), + makeProvider('deepseek'), + ], + connected: ['openai'], + }, + }); + + const { result } = renderHook(() => useProviders()); + + await waitFor(() => { + expect(result.current.loading).toBe(false); + }); + + expect(result.current.connectedIds).toEqual(['openai']); + expect(result.current.providers).toMatchObject([ + { id: 'openai', configured: true, modelCount: 1, category: 'connected' }, + { id: 'deepseek', configured: false, modelCount: 0, category: 'chinese' }, + ]); + }); + + it('shares provider list requests across concurrent hook instances', async () => { + let resolveList: (value: { data: any }) => void = () => {}; + listMock.mockReturnValue(new Promise((resolve) => { + resolveList = resolve; + })); + + const first = renderHook(() => useProviders()); + const second = renderHook(() => useProviders()); + + expect(listMock).toHaveBeenCalledTimes(1); + + await act(async () => { + resolveList({ + data: { + all: [makeProvider('openai')], + connected: ['openai'], + }, + }); + }); + + await waitFor(() => { + expect(first.result.current.loading).toBe(false); + expect(second.result.current.loading).toBe(false); + }); + + expect(first.result.current.providers).toHaveLength(1); + expect(second.result.current.providers).toHaveLength(1); + }); + + it('forces a new request when refetch is called', async () => { + listMock + .mockResolvedValueOnce({ + data: { + all: [makeProvider('openai')], + connected: ['openai'], + }, + }) + .mockResolvedValueOnce({ + data: { + all: [makeProvider('openai'), makeProvider('deepseek')], + connected: ['openai', 'deepseek'], + }, + }); + + const { result } = renderHook(() => useProviders()); + + await waitFor(() => { + expect(result.current.providers).toHaveLength(1); + }); + + await act(async () => { + await result.current.refetch(); + }); + + await waitFor(() => { + expect(result.current.providers).toHaveLength(2); + }); + expect(listMock).toHaveBeenCalledTimes(2); + }); +}); diff --git a/webui/src/hooks/useProviders.ts b/webui/src/hooks/useProviders.ts index 268e69379..d671e5b4b 100644 --- a/webui/src/hooks/useProviders.ts +++ b/webui/src/hooks/useProviders.ts @@ -1,6 +1,7 @@ -import { useState, useEffect, useCallback } from 'react'; +import { useCallback } from 'react'; import { providerAPI, getProviderCategory } from '@/api/provider'; import type { ProviderInfoV2, ProviderCategory } from '@/types'; +import { createSharedResource, useSharedResource } from './useSharedResource'; export interface EnrichedProvider extends ProviderInfoV2 { configured: boolean; @@ -8,62 +9,81 @@ export interface EnrichedProvider extends ProviderInfoV2 { category: ProviderCategory; } -export function useProviders() { - const [providers, setProviders] = useState([]); - const [connectedIds, setConnectedIds] = useState([]); - const [loading, setLoading] = useState(true); - const [error, setError] = useState(null); +interface ProvidersResourceData { + providers: EnrichedProvider[]; + connectedIds: string[]; +} - const fetchProviders = useCallback(async () => { - try { - setLoading(true); - setError(null); - const response = await providerAPI.list(); - const data = response.data as any; +const PROVIDER_LIST_STALE_TIME_MS = 5000; +const PROVIDER_LIST_MIN_FETCH_INTERVAL_MS = 1000; - let rawProviders: ProviderInfoV2[] = []; - let connected: string[] = []; +function normalizeProvidersPayload(data: unknown): ProvidersResourceData { + let rawProviders: ProviderInfoV2[] = []; + let connectedIds: string[] = []; - if (data && Array.isArray(data.all)) { - rawProviders = data.all; - connected = data.connected || []; - } else if (Array.isArray(data)) { - rawProviders = data; - } + if (data && typeof data === 'object' && Array.isArray((data as any).all)) { + rawProviders = (data as any).all; + connectedIds = Array.isArray((data as any).connected) ? (data as any).connected : []; + } else if (Array.isArray(data)) { + rawProviders = data; + } - const connectedSet = new Set(connected); + const connectedSet = new Set(connectedIds); + const providers: EnrichedProvider[] = rawProviders.map((provider) => { + const modelCount = provider.models ? Object.keys(provider.models).length : 0; + const configured = connectedSet.has(provider.id); + const category: ProviderCategory = configured ? 'connected' : getProviderCategory(provider.id); + return { + ...provider, + configured, + modelCount, + category, + }; + }); - const enriched: EnrichedProvider[] = rawProviders.map((p) => { - const modelCount = p.models ? Object.keys(p.models).length : 0; - const configured = connectedSet.has(p.id); - const category = configured ? 'connected' : getProviderCategory(p.id); - return { - ...p, - configured, - modelCount, - category, - }; - }); + return { providers, connectedIds }; +} - setProviders(enriched); - setConnectedIds(connected); - } catch (err: any) { - setError(err.message || 'Failed to fetch providers'); - setProviders([]); - } finally { - setLoading(false); - } - }, []); +const providersResource = createSharedResource({ + initialData: { + providers: [], + connectedIds: [], + }, + staleTimeMs: PROVIDER_LIST_STALE_TIME_MS, + minFetchIntervalMs: PROVIDER_LIST_MIN_FETCH_INTERVAL_MS, + fetcher: async () => { + const response = await providerAPI.list(); + return normalizeProvidersPayload(response.data); + }, + fallbackDataOnError: { + providers: [], + connectedIds: [], + }, + getErrorMessage: (err) => (err instanceof Error && err.message ? err.message : 'Failed to fetch providers'), +}); - useEffect(() => { - fetchProviders(); - }, [fetchProviders]); +export function __resetProvidersResourceForTesting(): void { + providersResource.resetForTesting(); +} - return { - providers, - connectedIds, +export function useProviders() { + const { + data, loading, error, refetch: fetchProviders, + } = useSharedResource(providersResource); + + const refetch = useCallback( + () => fetchProviders({ force: true }), + [fetchProviders], + ); + + return { + providers: data.providers, + connectedIds: data.connectedIds, + loading, + error, + refetch, }; } diff --git a/webui/src/hooks/useSSE.test.tsx b/webui/src/hooks/useSSE.test.tsx index 20121cdc1..e606ba597 100644 --- a/webui/src/hooks/useSSE.test.tsx +++ b/webui/src/hooks/useSSE.test.tsx @@ -5,6 +5,7 @@ import { useSSE } from './useSSE'; describe('useSSE', () => { const eventSourceCtor = vi.fn(); + const eventSources: FakeEventSource[] = []; class FakeEventSource { url: string; @@ -21,6 +22,7 @@ describe('useSSE', () => { constructor(url: string, init?: EventSourceInit) { this.url = url; this.withCredentials = Boolean(init?.withCredentials); + eventSources.push(this); eventSourceCtor(url, init); } @@ -35,6 +37,7 @@ describe('useSSE', () => { beforeEach(() => { eventSourceCtor.mockClear(); + eventSources.length = 0; vi.stubGlobal('EventSource', FakeEventSource as unknown as typeof EventSource); }); @@ -70,4 +73,63 @@ describe('useSSE', () => { unmount(); }); + + it('shares one EventSource across subscribers with the same URL and credentials', () => { + const firstHandler = vi.fn(); + const secondHandler = vi.fn(); + + const first = renderHook(() => useSSE({ + url: '/api/event', + onEvent: firstHandler, + })); + const second = renderHook(() => useSSE({ + url: '/api/event', + onEvent: secondHandler, + })); + + expect(eventSourceCtor).toHaveBeenCalledTimes(1); + + eventSources[0].onmessage?.( + new MessageEvent('message', { + data: JSON.stringify({ type: 'ping', properties: { ok: true } }), + }), + ); + + expect(firstHandler).toHaveBeenCalledWith({ type: 'ping', properties: { ok: true } }); + expect(secondHandler).toHaveBeenCalledWith({ type: 'ping', properties: { ok: true } }); + + first.unmount(); + second.unmount(); + }); + + it('keeps the shared EventSource open while another subscriber is still mounted', () => { + const firstHandler = vi.fn(); + const secondHandler = vi.fn(); + + const first = renderHook(() => useSSE({ + url: '/api/event', + onEvent: firstHandler, + })); + const second = renderHook(() => useSSE({ + url: '/api/event', + onEvent: secondHandler, + })); + + const shared = eventSources[0]; + first.unmount(); + + expect(shared.readyState).not.toBe(FakeEventSource.CLOSED); + + shared.onmessage?.( + new MessageEvent('message', { + data: JSON.stringify({ type: 'pong', properties: {} }), + }), + ); + + expect(firstHandler).not.toHaveBeenCalled(); + expect(secondHandler).toHaveBeenCalledWith({ type: 'pong', properties: {} }); + + second.unmount(); + expect(shared.readyState).toBe(FakeEventSource.CLOSED); + }); }); diff --git a/webui/src/hooks/useSSE.ts b/webui/src/hooks/useSSE.ts index 58638ad21..2e223b104 100644 --- a/webui/src/hooks/useSSE.ts +++ b/webui/src/hooks/useSSE.ts @@ -1,4 +1,4 @@ -import { useEffect, useRef, useCallback, useState } from 'react'; +import { useEffect, useRef, useCallback, useState, type MutableRefObject } from 'react'; export interface SSEEvent { type: string; @@ -28,6 +28,236 @@ export interface UseSSEOptions { export type SSEConnectionStatus = 'connecting' | 'connected' | 'disconnected' | 'reconnecting' | 'failed'; +type SSEReconnectConfig = Required>; + +interface SharedSSESubscriber { + id: number; + paused: boolean; + onEventRef: MutableRefObject; + onErrorRef: MutableRefObject; + onReconnectRef: MutableRefObject; + reconnect: SSEReconnectConfig; + setStatus: (status: SSEConnectionStatus) => void; + setRetryCount: (count: number) => void; +} + +interface SharedSSEConnection { + key: string; + url: string; + withCredentials: boolean; + eventSource: EventSource | null; + reconnectTimeout: number | null; + retryCount: number; + status: SSEConnectionStatus; + subscribers: Map; +} + +const sharedConnections = new Map(); +let nextSubscriberId = 1; + +function connectionKey(url: string, withCredentials: boolean): string { + return `${withCredentials ? 'auth' : 'anon'}:${url}`; +} + +function hasActiveSubscribers(connection: SharedSSEConnection): boolean { + for (const subscriber of connection.subscribers.values()) { + if (!subscriber.paused) return true; + } + return false; +} + +function forEachActiveSubscriber( + connection: SharedSSEConnection, + callback: (subscriber: SharedSSESubscriber) => void, +) { + for (const subscriber of connection.subscribers.values()) { + if (!subscriber.paused) callback(subscriber); + } +} + +function notifyStatus(connection: SharedSSEConnection, status: SSEConnectionStatus) { + connection.status = status; + forEachActiveSubscriber(connection, (subscriber) => subscriber.setStatus(status)); +} + +function notifyRetryCount(connection: SharedSSEConnection) { + forEachActiveSubscriber(connection, (subscriber) => subscriber.setRetryCount(connection.retryCount)); +} + +function mergedReconnectConfig(connection: SharedSSEConnection): SSEReconnectConfig { + let reconnecting = 0; + let maxRetries = 0; + let initialDelay = Number.POSITIVE_INFINITY; + let maxDelay = Number.POSITIVE_INFINITY; + + forEachActiveSubscriber(connection, (subscriber) => { + if (!subscriber.reconnect.enabled) return; + reconnecting += 1; + maxRetries = Math.max(maxRetries, subscriber.reconnect.maxRetries); + initialDelay = Math.min(initialDelay, subscriber.reconnect.initialDelay); + maxDelay = Math.min(maxDelay, subscriber.reconnect.maxDelay); + }); + + if (reconnecting === 0) { + return { enabled: false, maxRetries: 0, initialDelay: 1000, maxDelay: 30000 }; + } + + return { + enabled: true, + maxRetries, + initialDelay, + maxDelay, + }; +} + +function clearReconnectTimeout(connection: SharedSSEConnection) { + if (connection.reconnectTimeout !== null) { + window.clearTimeout(connection.reconnectTimeout); + connection.reconnectTimeout = null; + } +} + +function getReconnectDelay(config: SSEReconnectConfig, retryCount: number) { + const delay = Math.min(config.initialDelay * Math.pow(2, retryCount), config.maxDelay); + return delay + Math.random() * 1000; +} + +function closeEventSource(connection: SharedSSEConnection) { + if (connection.eventSource) { + connection.eventSource.close(); + connection.eventSource = null; + } +} + +function cleanupInactiveConnection(connection: SharedSSEConnection) { + if (hasActiveSubscribers(connection)) return; + clearReconnectTimeout(connection); + closeEventSource(connection); + connection.status = 'disconnected'; + connection.retryCount = 0; + if (connection.subscribers.size === 0) { + sharedConnections.delete(connection.key); + } +} + +function connectShared(connection: SharedSSEConnection) { + if (!hasActiveSubscribers(connection)) { + cleanupInactiveConnection(connection); + return; + } + + clearReconnectTimeout(connection); + closeEventSource(connection); + + if (import.meta.env.DEV) { + console.log('[SSE] Creating shared EventSource connection to:', connection.url); + } + notifyStatus(connection, 'connecting'); + + const eventSource = new EventSource(connection.url, { withCredentials: connection.withCredentials }); + connection.eventSource = eventSource; + + eventSource.onopen = () => { + if (connection.eventSource !== eventSource) return; + if (import.meta.env.DEV) { + console.log('[SSE] Shared connection opened successfully'); + } + connection.retryCount = 0; + notifyRetryCount(connection); + notifyStatus(connection, 'connected'); + }; + + eventSource.onmessage = (event) => { + if (connection.eventSource !== eventSource) return; + try { + const data = JSON.parse(event.data); + forEachActiveSubscriber(connection, (subscriber) => { + subscriber.onEventRef.current(data); + }); + } catch (err) { + console.error('Failed to parse SSE event:', err); + } + }; + + eventSource.onerror = (error) => { + if (connection.eventSource !== eventSource) return; + + if (import.meta.env.DEV) { + console.warn('[SSE] Shared connection error, will attempt to reconnect'); + } + forEachActiveSubscriber(connection, (subscriber) => { + subscriber.onErrorRef.current?.(error); + }); + + eventSource.close(); + connection.eventSource = null; + + const reconnectConfig = mergedReconnectConfig(connection); + if (!reconnectConfig.enabled) { + notifyStatus(connection, 'failed'); + return; + } + + if (connection.retryCount < reconnectConfig.maxRetries) { + const delay = getReconnectDelay(reconnectConfig, connection.retryCount); + if (import.meta.env.DEV) { + console.log(`[SSE] Reconnecting shared connection in ${Math.round(delay)}ms (attempt ${connection.retryCount + 1}/${reconnectConfig.maxRetries})`); + } + notifyStatus(connection, 'reconnecting'); + + clearReconnectTimeout(connection); + connection.reconnectTimeout = window.setTimeout(() => { + if (!hasActiveSubscribers(connection)) { + cleanupInactiveConnection(connection); + return; + } + connection.retryCount++; + notifyRetryCount(connection); + forEachActiveSubscriber(connection, (subscriber) => { + subscriber.onReconnectRef.current?.(); + }); + connectShared(connection); + }, delay); + return; + } + + if (import.meta.env.DEV) { + console.log('[SSE] Max fast retries reached, switching shared connection to slow retry mode'); + } + notifyStatus(connection, 'reconnecting'); + + clearReconnectTimeout(connection); + connection.reconnectTimeout = window.setTimeout(() => { + if (!hasActiveSubscribers(connection)) { + cleanupInactiveConnection(connection); + return; + } + connection.retryCount = 0; + notifyRetryCount(connection); + connectShared(connection); + }, 30000); + }; +} + +function getOrCreateSharedConnection(url: string, withCredentials: boolean): SharedSSEConnection { + const key = connectionKey(url, withCredentials); + const existing = sharedConnections.get(key); + if (existing) return existing; + + const connection: SharedSSEConnection = { + key, + url, + withCredentials, + eventSource: null, + reconnectTimeout: null, + retryCount: 0, + status: 'disconnected', + subscribers: new Map(), + }; + sharedConnections.set(key, connection); + return connection; +} + export function useSSE({ url, onEvent, @@ -37,17 +267,16 @@ export function useSSE({ withCredentials = true, reconnect = {}, }: UseSSEOptions) { - const eventSourceRef = useRef(null); + const subscriberIdRef = useRef(null); + const subscriberRef = useRef(null); + const connectionRef = useRef(null); const onEventRef = useRef(onEvent); const onErrorRef = useRef(onError); const onReconnectRef = useRef(onReconnect); - const reconnectTimeoutRef = useRef(null); - const retryCountRef = useRef(0); - const mountedRef = useRef(true); const [status, setStatus] = useState('disconnected'); + const [retryCount, setRetryCount] = useState(0); - // 重连配置 const { enabled: reconnectEnabled = true, maxRetries = 10, @@ -62,156 +291,95 @@ export function useSSE({ onReconnectRef.current = onReconnect; }, [onEvent, onError, onReconnect]); - // 计算指数退避延迟 - const getReconnectDelay = useCallback((retryCount: number) => { - const delay = Math.min(initialDelay * Math.pow(2, retryCount), maxDelay); - // 添加一些随机抖动以避免雷暴效应 - return delay + Math.random() * 1000; - }, [initialDelay, maxDelay]); - - // 清理重连定时器 - const clearReconnectTimeout = useCallback(() => { - if (reconnectTimeoutRef.current) { - clearTimeout(reconnectTimeoutRef.current); - reconnectTimeoutRef.current = null; - } - }, []); - - // 创建连接 - const connect = useCallback(() => { - if (!mountedRef.current || !enabled) return; - - // 清理现有连接 - if (eventSourceRef.current) { - eventSourceRef.current.close(); - eventSourceRef.current = null; - } - - if (import.meta.env.DEV) { - console.log('[SSE] Creating EventSource connection to:', url); - } - setStatus('connecting'); - const eventSource = new EventSource(url, { withCredentials }); - eventSourceRef.current = eventSource; - - eventSource.onopen = () => { - if (!mountedRef.current) return; - if (import.meta.env.DEV) { - console.log('[SSE] Connection opened successfully'); - } - setStatus('connected'); - retryCountRef.current = 0; // 重置重试计数 - }; - - // Handle messages - eventSource.onmessage = (event) => { - if (!mountedRef.current) return; - try { - const data = JSON.parse(event.data); - // Debug: 打印收到的 SSE 事件 - if (import.meta.env.DEV && data.type === 'message.part.updated') { - console.log('[SSE] message.part.updated:', { - partId: data.properties?.part?.id, - textLength: data.properties?.part?.text?.length, - deltaLength: data.properties?.delta?.length, - delta: data.properties?.delta?.substring(0, 50), - }); - } - onEventRef.current(data); - } catch (err) { - console.error('Failed to parse SSE event:', err); - } - }; - - // Handle errors with auto-reconnect - eventSource.onerror = (error) => { - if (!mountedRef.current) return; - - if (import.meta.env.DEV) { - console.warn('[SSE] Connection error, will attempt to reconnect'); - } - onErrorRef.current?.(error); - - // 关闭当前连接 - eventSource.close(); - eventSourceRef.current = null; - - // 尝试重连(使用更宽松的策略) - if (reconnectEnabled && retryCountRef.current < maxRetries) { - const delay = getReconnectDelay(retryCountRef.current); - if (import.meta.env.DEV) { - console.log(`[SSE] Reconnecting in ${Math.round(delay)}ms (attempt ${retryCountRef.current + 1}/${maxRetries})`); - } - setStatus('reconnecting'); - - clearReconnectTimeout(); - reconnectTimeoutRef.current = window.setTimeout(() => { - if (!mountedRef.current) return; - retryCountRef.current++; - onReconnectRef.current?.(); - connect(); - }, delay); - } else { - // 即使达到最大重试次数,也不要完全放弃,而是使用较长的间隔继续尝试 - if (import.meta.env.DEV) { - console.log('[SSE] Max fast retries reached, switching to slow retry mode'); - } - setStatus('reconnecting'); - - clearReconnectTimeout(); - reconnectTimeoutRef.current = window.setTimeout(() => { - if (!mountedRef.current) return; - retryCountRef.current = 0; // 重置计数器,重新开始 - connect(); - }, 30000); // 30秒后重试 - } - }; - }, [url, enabled, withCredentials, reconnectEnabled, maxRetries, getReconnectDelay, clearReconnectTimeout]); - - // 主 effect useEffect(() => { - mountedRef.current = true; + if (subscriberIdRef.current === null) { + subscriberIdRef.current = nextSubscriberId++; + } if (!enabled) { if (import.meta.env.DEV) { console.log('[SSE] Not enabled, skipping connection'); } setStatus('disconnected'); + setRetryCount(0); return; } - connect(); + const connection = getOrCreateSharedConnection(url, withCredentials); + const subscriber: SharedSSESubscriber = { + id: subscriberIdRef.current, + paused: false, + onEventRef, + onErrorRef, + onReconnectRef, + reconnect: { + enabled: reconnectEnabled, + maxRetries, + initialDelay, + maxDelay, + }, + setStatus, + setRetryCount, + }; + + subscriberRef.current = subscriber; + connectionRef.current = connection; + connection.subscribers.set(subscriber.id, subscriber); + setStatus(connection.status); + setRetryCount(connection.retryCount); + + if (!connection.eventSource && connection.reconnectTimeout === null) { + connectShared(connection); + } - // Cleanup return () => { - mountedRef.current = false; - clearReconnectTimeout(); - if (eventSourceRef.current) { - eventSourceRef.current.close(); - eventSourceRef.current = null; + connection.subscribers.delete(subscriber.id); + if (subscriberRef.current === subscriber) { + subscriberRef.current = null; + } + if (connectionRef.current === connection) { + connectionRef.current = null; } + cleanupInactiveConnection(connection); }; - }, [url, enabled, connect, clearReconnectTimeout]); + }, [ + enabled, + initialDelay, + maxDelay, + maxRetries, + reconnectEnabled, + url, + withCredentials, + ]); - // 手动重连 const reconnectManually = useCallback(() => { - retryCountRef.current = 0; - connect(); - }, [connect]); + const connection = connectionRef.current; + const subscriber = subscriberRef.current; + if (!connection || !subscriber || !enabled) return; + subscriber.paused = false; + connection.retryCount = 0; + notifyRetryCount(connection); + connectShared(connection); + }, [enabled]); + + const close = useCallback(() => { + const connection = connectionRef.current; + const subscriber = subscriberRef.current; + if (!connection || !subscriber) return; + subscriber.paused = true; + setStatus('disconnected'); + setRetryCount(0); + cleanupInactiveConnection(connection); + }, []); return { /** 当前连接状态 */ status, /** 手动关闭连接 */ - close: () => { - clearReconnectTimeout(); - eventSourceRef.current?.close(); - eventSourceRef.current = null; - setStatus('disconnected'); - }, + close, /** 手动重连 */ reconnect: reconnectManually, /** 当前重试次数 */ - retryCount: retryCountRef.current, + retryCount, }; } diff --git a/webui/src/hooks/useSessions.test.ts b/webui/src/hooks/useSessions.test.ts index f5a6c503d..4dbc678d5 100644 --- a/webui/src/hooks/useSessions.test.ts +++ b/webui/src/hooks/useSessions.test.ts @@ -142,6 +142,7 @@ describe('applyMessagePartUpdate', () => { describe('updateMessagePart scheduling', () => { afterEach(() => { vi.clearAllMocks(); + vi.useRealTimers(); }); it('keeps parentID from fetched messages for regenerate truncation', async () => { @@ -213,6 +214,7 @@ describe('updateMessagePart scheduling', () => { }); it('second call with same part ID accumulates delta content correctly', async () => { + vi.useFakeTimers(); const { result } = renderHook(() => useSessionMessages('sess-1')); // Wait for initial fetch to settle await act(async () => {}); @@ -228,6 +230,7 @@ describe('updateMessagePart scheduling', () => { // Second call — content delta on the same part await act(async () => { result.current.updateMessagePart(delta, ' world'); + await vi.advanceTimersByTimeAsync(16); }); const msgs = result.current.messages; @@ -236,6 +239,33 @@ describe('updateMessagePart scheduling', () => { expect((msg!.parts as any[])[0].text).toBe('hello world'); }); + it('coalesces repeated known part updates into the next animation frame', async () => { + vi.useFakeTimers(); + const { result } = renderHook(() => useSessionMessages('sess-1')); + await act(async () => {}); + + const part = { id: 'part-batched', messageID: 'msg-batched', sessionID: 'sess-1', type: 'text', text: 'h' }; + + await act(async () => { + result.current.updateMessagePart(part); + }); + expect((result.current.messages[0].parts as any[])[0].text).toBe('h'); + + await act(async () => { + result.current.updateMessagePart({ ...part, text: 'he' }, 'e'); + result.current.updateMessagePart({ ...part, text: 'hel' }, 'l'); + result.current.updateMessagePart({ ...part, text: 'hello' }, 'lo'); + }); + + expect((result.current.messages[0].parts as any[])[0].text).toBe('h'); + + await act(async () => { + await vi.advanceTimersByTimeAsync(16); + }); + + expect((result.current.messages[0].parts as any[])[0].text).toBe('hello'); + }); + it('resets known part tracking when session changes', async () => { const { result, rerender } = renderHook( ({ id }: { id?: string }) => useSessionMessages(id), diff --git a/webui/src/hooks/useSessions.ts b/webui/src/hooks/useSessions.ts index 4f556780d..55c94ade5 100644 --- a/webui/src/hooks/useSessions.ts +++ b/webui/src/hooks/useSessions.ts @@ -8,6 +8,11 @@ const ABORTED_TOOL_ERROR = 'Tool execution was interrupted'; const SESSION_LIST_PAGE_SIZE = 100; const MESSAGE_PAGE_SIZE = 50; +interface PendingPartUpdate { + partInfo: any; + delta?: string; +} + function finalizeStoppedMessageParts(parts: Message['parts'], stoppedAt = Date.now()): Message['parts'] { return parts.map((part) => { if ( @@ -353,6 +358,8 @@ export function useSessionMessages(sessionId?: string) { // Tracks part IDs seen in this session to distinguish first-time creation // (structural change → immediate update) from content deltas (low-priority). const knownPartIdsRef = useRef>(new Set()); + const pendingKnownPartUpdatesRef = useRef>(new Map()); + const pendingKnownPartFrameRef = useRef(null); const fetchMessages = useCallback(async () => { if (!sessionId) return; @@ -405,9 +412,32 @@ export function useSessionMessages(sessionId?: string) { } }, [hasMore, loadingOlder, nextBefore, sessionId]); + const flushPendingKnownPartUpdates = useCallback(() => { + pendingKnownPartFrameRef.current = null; + const updates = Array.from(pendingKnownPartUpdatesRef.current.values()); + pendingKnownPartUpdatesRef.current.clear(); + if (updates.length === 0) return; + + startTransition(() => { + setMessages(prev => updates.reduce( + (next, update) => applyMessagePartUpdate(next, update.partInfo, update.delta), + prev, + )); + }); + }, []); + + const cancelPendingKnownPartUpdates = useCallback(() => { + if (pendingKnownPartFrameRef.current !== null) { + cancelAnimationFrame(pendingKnownPartFrameRef.current); + pendingKnownPartFrameRef.current = null; + } + pendingKnownPartUpdatesRef.current.clear(); + }, []); + // Reset state synchronously before paint when session changes // to prevent flash of welcome screen (useEffect runs AFTER paint) useLayoutEffect(() => { + cancelPendingKnownPartUpdates(); setMessages([]); setError(null); setHasMore(false); @@ -418,7 +448,9 @@ export function useSessionMessages(sessionId?: string) { } else { setLoading(false); } - }, [sessionId]); + }, [cancelPendingKnownPartUpdates, sessionId]); + + useEffect(() => cancelPendingKnownPartUpdates, [cancelPendingKnownPartUpdates]); useEffect(() => { fetchMessages(); @@ -522,21 +554,22 @@ export function useSessionMessages(sessionId?: string) { * @param delta - Optional text delta for this update. * * New parts are structural changes and update synchronously so thinking or - * streaming indicators appear immediately. Deltas for known parts are - * lowered with startTransition so React can batch high-frequency SSE chunks. + * streaming indicators appear immediately. Known parts are folded into the + * next animation frame so high-frequency text deltas trigger fewer renders. */ updateMessagePart: (partInfo: any, delta?: string) => { - const isNewPart = !knownPartIdsRef.current.has(partInfo.id); + const partId = partInfo?.id; + const isNewPart = !partId || !knownPartIdsRef.current.has(partId); if (isNewPart) { // Structural change: first appearance of this part — must render immediately // so that "thinking" / "streaming" indicators show without delay. - knownPartIdsRef.current.add(partInfo.id); + if (partId) knownPartIdsRef.current.add(partId); setMessages(prev => applyMessagePartUpdate(prev, partInfo, delta)); } else { - // Content delta on an existing part — low priority, allow React to batch. - startTransition(() => { - setMessages(prev => applyMessagePartUpdate(prev, partInfo, delta)); - }); + pendingKnownPartUpdatesRef.current.set(partId, { partInfo, delta }); + if (pendingKnownPartFrameRef.current === null) { + pendingKnownPartFrameRef.current = requestAnimationFrame(flushPendingKnownPartUpdates); + } } }, replaceMessageText: (messageId: string, partId: string, text: string) => { diff --git a/webui/src/hooks/useSharedResource.test.tsx b/webui/src/hooks/useSharedResource.test.tsx new file mode 100644 index 000000000..0843ae4c3 --- /dev/null +++ b/webui/src/hooks/useSharedResource.test.tsx @@ -0,0 +1,72 @@ +import { act, renderHook, waitFor } from '@testing-library/react'; +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import { createSharedResource, useSharedResource } from './useSharedResource'; + +describe('useSharedResource', () => { + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('deduplicates concurrent initial fetches for the same resource', async () => { + let resolveFetch: (value: string[]) => void = () => {}; + const fetcher = vi.fn(() => new Promise((resolve) => { + resolveFetch = resolve; + })); + const resource = createSharedResource({ + initialData: [], + fetcher, + }); + + const first = renderHook(() => useSharedResource(resource)); + const second = renderHook(() => useSharedResource(resource)); + + expect(fetcher).toHaveBeenCalledTimes(1); + + await act(async () => { + resolveFetch(['ready']); + }); + + await waitFor(() => { + expect(first.result.current.loading).toBe(false); + expect(second.result.current.loading).toBe(false); + }); + + expect(first.result.current.data).toEqual(['ready']); + expect(second.result.current.data).toEqual(['ready']); + }); + + it('reuses fresh data until the stale window expires', async () => { + let now = 1000; + vi.spyOn(Date, 'now').mockImplementation(() => now); + + const fetcher = vi.fn() + .mockResolvedValueOnce('first') + .mockResolvedValueOnce('second'); + const resource = createSharedResource({ + initialData: '', + staleTimeMs: 5000, + fetcher, + }); + + const { result } = renderHook(() => useSharedResource(resource)); + + await waitFor(() => { + expect(result.current.data).toBe('first'); + }); + expect(fetcher).toHaveBeenCalledTimes(1); + + await act(async () => { + await resource.fetch({ silent: true }); + }); + expect(fetcher).toHaveBeenCalledTimes(1); + + now += 6000; + await act(async () => { + await resource.fetch({ silent: true }); + }); + + expect(fetcher).toHaveBeenCalledTimes(2); + expect(result.current.data).toBe('second'); + }); +}); diff --git a/webui/src/hooks/useSharedResource.ts b/webui/src/hooks/useSharedResource.ts new file mode 100644 index 000000000..5c9fd3f97 --- /dev/null +++ b/webui/src/hooks/useSharedResource.ts @@ -0,0 +1,229 @@ +import { useCallback, useEffect, useSyncExternalStore } from 'react'; + +export interface SharedResourceSnapshot { + data: T; + loading: boolean; + error: string | null; + initialized: boolean; + updatedAt: number; +} + +export interface SharedResourceFetchOptions { + force?: boolean; + silent?: boolean; +} + +export interface CreateSharedResourceOptions { + initialData: T; + fetcher: () => Promise; + staleTimeMs?: number; + minFetchIntervalMs?: number; + getErrorMessage?: (error: unknown) => string; + fallbackDataOnError?: T | ((previous: T) => T); +} + +export interface SharedResource { + getSnapshot: () => SharedResourceSnapshot; + subscribe: (listener: () => void) => () => void; + fetch: (options?: SharedResourceFetchOptions) => Promise; + resetForTesting: () => void; +} + +export interface UseSharedResourceOptions { + enabled?: boolean; + loadOnMount?: boolean; + silentInitialLoad?: boolean; +} + +export interface UseRefreshOnResumeOptions { + enabled?: boolean; +} + +function defaultGetErrorMessage(error: unknown): string { + if (error instanceof Error && error.message) return error.message; + return 'Failed to fetch resource'; +} + +function resolveFallbackData( + fallback: CreateSharedResourceOptions['fallbackDataOnError'], + previous: T, +): T { + if (typeof fallback === 'function') { + return (fallback as (previous: T) => T)(previous); + } + if (fallback !== undefined) { + return fallback; + } + return previous; +} + +export function createSharedResource(options: CreateSharedResourceOptions): SharedResource { + const staleTimeMs = options.staleTimeMs ?? 0; + const minFetchIntervalMs = options.minFetchIntervalMs ?? 0; + const getErrorMessage = options.getErrorMessage ?? defaultGetErrorMessage; + const listeners = new Set<() => void>(); + + let inFlight: Promise | null = null; + let lastStartedAt = 0; + let snapshot: SharedResourceSnapshot = { + data: options.initialData, + loading: true, + error: null, + initialized: false, + updatedAt: 0, + }; + + const emit = () => { + listeners.forEach((listener) => listener()); + }; + + const updateSnapshot = (next: Partial>) => { + snapshot = { + ...snapshot, + ...next, + }; + emit(); + }; + + const fetch = (fetchOptions: SharedResourceFetchOptions = {}): Promise => { + const now = Date.now(); + const force = fetchOptions.force === true; + const silent = fetchOptions.silent === true; + const freshEnough = snapshot.initialized && now - snapshot.updatedAt < staleTimeMs; + const recentlyStarted = snapshot.initialized && now - lastStartedAt < minFetchIntervalMs; + + if (!force && freshEnough) { + return Promise.resolve(snapshot.data); + } + + if (inFlight) { + return inFlight; + } + + if (!force && recentlyStarted) { + return Promise.resolve(snapshot.data); + } + + lastStartedAt = now; + updateSnapshot({ + loading: silent ? snapshot.loading : true, + error: null, + }); + + const request = options.fetcher() + .then((data) => { + updateSnapshot({ + data, + loading: false, + error: null, + initialized: true, + updatedAt: Date.now(), + }); + return data; + }) + .catch((error: unknown) => { + const data = resolveFallbackData(options.fallbackDataOnError, snapshot.data); + updateSnapshot({ + data, + loading: false, + error: getErrorMessage(error), + initialized: true, + }); + return data; + }) + .finally(() => { + if (inFlight === request) { + inFlight = null; + } + }); + + inFlight = request; + return request; + }; + + return { + getSnapshot: () => snapshot, + subscribe: (listener: () => void) => { + listeners.add(listener); + return () => { + listeners.delete(listener); + }; + }, + fetch, + resetForTesting: () => { + inFlight = null; + lastStartedAt = 0; + snapshot = { + data: options.initialData, + loading: true, + error: null, + initialized: false, + updatedAt: 0, + }; + emit(); + }, + }; +} + +export function useSharedResource( + resource: SharedResource, + options: UseSharedResourceOptions = {}, +) { + const enabled = options.enabled ?? true; + const loadOnMount = options.loadOnMount ?? true; + const snapshot = useSyncExternalStore( + resource.subscribe, + resource.getSnapshot, + resource.getSnapshot, + ); + + useEffect(() => { + if (!enabled || !loadOnMount) return; + const current = resource.getSnapshot(); + void resource.fetch({ + silent: current.initialized || options.silentInitialLoad === true, + }); + }, [enabled, loadOnMount, options.silentInitialLoad, resource]); + + const refetch = useCallback( + (fetchOptions: SharedResourceFetchOptions = {}) => resource.fetch({ + force: true, + ...fetchOptions, + }), + [resource], + ); + + return { + ...snapshot, + refetch, + }; +} + +export function useRefreshOnResume( + refresh: () => void | Promise, + options: UseRefreshOnResumeOptions = {}, +) { + const enabled = options.enabled ?? true; + + useEffect(() => { + if (!enabled || typeof document === 'undefined' || typeof window === 'undefined') return; + + const handleVisibilityChange = () => { + if (document.visibilityState === 'visible') { + void refresh(); + } + }; + + const handleWindowFocus = () => { + void refresh(); + }; + + document.addEventListener('visibilitychange', handleVisibilityChange); + window.addEventListener('focus', handleWindowFocus); + + return () => { + document.removeEventListener('visibilitychange', handleVisibilityChange); + window.removeEventListener('focus', handleWindowFocus); + }; + }, [enabled, refresh]); +} diff --git a/webui/src/hooks/useStats.test.tsx b/webui/src/hooks/useStats.test.tsx new file mode 100644 index 000000000..969d99eea --- /dev/null +++ b/webui/src/hooks/useStats.test.tsx @@ -0,0 +1,109 @@ +import { act, renderHook, waitFor } from '@testing-library/react'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { __resetStatsResourceForTesting, useStats } from './useStats'; + +const { getSystemStatsMock } = vi.hoisted(() => ({ + getSystemStatsMock: vi.fn(), +})); + +vi.mock('@/api/stats', () => ({ + statsApi: { + getSystemStats: getSystemStatsMock, + }, +})); + +function makeStats(overrides: Record = {}) { + return { + tasks: { week: 1, scheduledActive: 1 }, + agents: { total: 2 }, + workflows: { total: 3 }, + skills: { total: 4 }, + tools: { total: 5 }, + models: { total: 6 }, + system: { status: 'healthy', message: 'ok' }, + ...overrides, + }; +} + +describe('useStats', () => { + beforeEach(() => { + vi.clearAllMocks(); + __resetStatsResourceForTesting(); + }); + + afterEach(() => { + vi.useRealTimers(); + __resetStatsResourceForTesting(); + }); + + it('shares the initial stats request across concurrent hook instances', async () => { + let resolveStats: (value: ReturnType) => void = () => {}; + getSystemStatsMock.mockReturnValue(new Promise((resolve) => { + resolveStats = resolve; + })); + + const first = renderHook(() => useStats()); + const second = renderHook(() => useStats()); + + expect(getSystemStatsMock).toHaveBeenCalledTimes(1); + + await act(async () => { + resolveStats(makeStats()); + }); + + await waitFor(() => { + expect(first.result.current.loading).toBe(false); + expect(second.result.current.loading).toBe(false); + }); + + expect(first.result.current.stats?.agents.total).toBe(2); + expect(second.result.current.stats?.models.total).toBe(6); + }); + + it('keeps one polling loop for multiple hook instances', async () => { + vi.useFakeTimers(); + getSystemStatsMock.mockResolvedValue(makeStats()); + + const first = renderHook(() => useStats()); + const second = renderHook(() => useStats()); + + await act(async () => { + await Promise.resolve(); + await Promise.resolve(); + }); + expect(first.result.current.loading).toBe(false); + expect(second.result.current.loading).toBe(false); + expect(getSystemStatsMock).toHaveBeenCalledTimes(1); + + await act(async () => { + await vi.advanceTimersByTimeAsync(30_000); + }); + + expect(getSystemStatsMock).toHaveBeenCalledTimes(2); + + first.unmount(); + second.unmount(); + + await act(async () => { + await vi.advanceTimersByTimeAsync(30_000); + }); + + expect(getSystemStatsMock).toHaveBeenCalledTimes(2); + }); + + it('maps unhealthy system stats to the existing Error return shape', async () => { + getSystemStatsMock.mockResolvedValue(makeStats({ + system: { status: 'error', message: 'backend down' }, + })); + + const { result } = renderHook(() => useStats()); + + await waitFor(() => { + expect(result.current.loading).toBe(false); + }); + + expect(result.current.error).toBeInstanceOf(Error); + expect(result.current.error?.message).toBe('backend down'); + }); +}); diff --git a/webui/src/hooks/useStats.ts b/webui/src/hooks/useStats.ts index 6f30db384..9e3d12403 100644 --- a/webui/src/hooks/useStats.ts +++ b/webui/src/hooks/useStats.ts @@ -1,44 +1,62 @@ -import { useState, useEffect } from 'react'; +import { useEffect, useMemo } from 'react'; import { statsApi, SystemStats } from '@/api/stats'; +import { createSharedResource, useSharedResource } from './useSharedResource'; + +const STATS_POLL_INTERVAL_MS = 30_000; +const STATS_STALE_TIME_MS = 5_000; +const STATS_MIN_FETCH_INTERVAL_MS = 1_000; + +const statsResource = createSharedResource({ + initialData: null, + staleTimeMs: STATS_STALE_TIME_MS, + minFetchIntervalMs: STATS_MIN_FETCH_INTERVAL_MS, + fetcher: statsApi.getSystemStats, + fallbackDataOnError: (previous) => previous, + getErrorMessage: (err) => (err instanceof Error && err.message ? err.message : 'Failed to fetch system stats'), +}); + +let statsPollSubscriberCount = 0; +let statsPollIntervalId: number | null = null; + +function subscribeStatsPolling(): () => void { + statsPollSubscriberCount += 1; + + if (statsPollSubscriberCount === 1) { + statsPollIntervalId = window.setInterval(() => { + void statsResource.fetch({ silent: true }); + }, STATS_POLL_INTERVAL_MS); + } + + return () => { + statsPollSubscriberCount = Math.max(0, statsPollSubscriberCount - 1); + if (statsPollSubscriberCount === 0 && statsPollIntervalId !== null) { + window.clearInterval(statsPollIntervalId); + statsPollIntervalId = null; + } + }; +} + +export function __resetStatsResourceForTesting(): void { + statsResource.resetForTesting(); + statsPollSubscriberCount = 0; + if (statsPollIntervalId !== null) { + window.clearInterval(statsPollIntervalId); + statsPollIntervalId = null; + } +} export function useStats() { - const [stats, setStats] = useState(null); - const [loading, setLoading] = useState(true); - const [error, setError] = useState(null); + const { data: stats, loading, error: fetchError, refetch } = useSharedResource(statsResource); useEffect(() => { - let mounted = true; - let intervalId: number | null = null; - - const fetchStats = async (isInitial = false) => { - if (isInitial) setLoading(true); - try { - const data = await statsApi.getSystemStats(); - if (mounted) { - setStats(data); - // 用系统状态判断是否需要显示错误(后端不可达时 getSystemStats 返回 status:'error') - if (data.system.status === 'error') { - setError(new Error(data.system.message)); - } else { - setError(null); - } - } - } finally { - if (mounted) setLoading(false); - } - }; - - fetchStats(true); - - intervalId = window.setInterval(() => { - if (mounted) fetchStats(false); - }, 30000); - - return () => { - mounted = false; - if (intervalId !== null) clearInterval(intervalId); - }; + return subscribeStatsPolling(); }, []); - return { stats, loading, error }; + const error = useMemo(() => { + if (fetchError) return new Error(fetchError); + if (stats?.system.status === 'error') return new Error(stats.system.message); + return null; + }, [fetchError, stats?.system.message, stats?.system.status]); + + return { stats, loading, error, refetch }; } diff --git a/webui/src/hooks/useTasks.test.tsx b/webui/src/hooks/useTasks.test.tsx new file mode 100644 index 000000000..814b3e4a4 --- /dev/null +++ b/webui/src/hooks/useTasks.test.tsx @@ -0,0 +1,175 @@ +import { act, renderHook, waitFor } from '@testing-library/react'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { + __resetTaskResourcesForTesting, + useQueueStatus, + useTaskDashboard, + useTaskExecutions, + useTaskExecutionsByScheduler, +} from './useTasks'; + +const { + dashboardMock, + getSystemNoticeMock, + listExecutionsMock, + listSchedulerExecutionsMock, + listSchedulersMock, + queueStatusMock, +} = vi.hoisted(() => ({ + dashboardMock: vi.fn(), + getSystemNoticeMock: vi.fn(), + listExecutionsMock: vi.fn(), + listSchedulerExecutionsMock: vi.fn(), + listSchedulersMock: vi.fn(), + queueStatusMock: vi.fn(), +})); + +vi.mock('@/api/task', () => ({ + taskAPI: { + dashboard: dashboardMock, + getSystemNotice: getSystemNoticeMock, + listExecutions: listExecutionsMock, + listSchedulerExecutions: listSchedulerExecutionsMock, + listSchedulers: listSchedulersMock, + queueStatus: queueStatusMock, + }, +})); + +function makeExecution(overrides: Record = {}) { + return { + id: 'exec-1', + schedulerID: 'scheduler-1', + title: 'Execution One', + description: '', + priority: 'normal', + source: { sourceType: 'user' }, + triggerType: 'run_once', + status: 'running', + deliveryStatus: 'unread', + executionInputSnapshot: {}, + retry: { maxRetries: 0, retryCount: 0, retryDelaySeconds: 0 }, + executionMode: 'workflow', + agentName: 'rex', + createdAt: '2026-07-09T00:00:00Z', + updatedAt: '2026-07-09T00:00:00Z', + ...overrides, + }; +} + +function makeExecutionPage(items = [makeExecution()]) { + return { + data: { + items, + total: items.length, + offset: 0, + limit: 20, + }, + }; +} + +describe('useTasks shared resources', () => { + beforeEach(() => { + vi.clearAllMocks(); + __resetTaskResourcesForTesting(); + dashboardMock.mockResolvedValue({ + data: { + running: 0, + queued: 0, + completed_week: 0, + completed_unviewed: 0, + failed_week: 0, + scheduled_active: 0, + queue_paused: false, + }, + }); + getSystemNoticeMock.mockResolvedValue({ data: null }); + listExecutionsMock.mockResolvedValue(makeExecutionPage()); + listSchedulerExecutionsMock.mockResolvedValue(makeExecutionPage()); + listSchedulersMock.mockResolvedValue({ + data: { items: [], total: 0, offset: 0, limit: 20 }, + }); + queueStatusMock.mockResolvedValue({ + data: { + paused: false, + max_concurrent: 2, + running: 0, + queued: 0, + }, + }); + }); + + afterEach(() => { + vi.useRealTimers(); + }); + + it('shares task execution list requests across concurrent hook instances with the same filters', async () => { + let resolveList: (value: ReturnType) => void = () => {}; + listExecutionsMock.mockReturnValue(new Promise((resolve) => { + resolveList = resolve; + })); + + const filters = { status: 'running' as const, offset: 0, limit: 20 }; + const first = renderHook(() => useTaskExecutions(filters)); + const second = renderHook(() => useTaskExecutions(filters)); + + expect(listExecutionsMock).toHaveBeenCalledTimes(1); + expect(listExecutionsMock).toHaveBeenCalledWith(filters); + + await act(async () => { + resolveList(makeExecutionPage([makeExecution()])); + }); + + await waitFor(() => { + expect(first.result.current.loading).toBe(false); + expect(second.result.current.loading).toBe(false); + }); + + expect(first.result.current.tasks).toHaveLength(1); + expect(second.result.current.tasks).toHaveLength(1); + }); + + it('deduplicates active execution polling across hook instances', async () => { + vi.useFakeTimers(); + listExecutionsMock.mockResolvedValue(makeExecutionPage([makeExecution({ status: 'running' })])); + + renderHook(() => useTaskExecutions({ offset: 0, limit: 20 }, { pollInterval: 5000 })); + renderHook(() => useTaskExecutions({ offset: 0, limit: 20 }, { pollInterval: 5000 })); + + await act(async () => { + await Promise.resolve(); + }); + expect(listExecutionsMock).toHaveBeenCalledTimes(1); + + await act(async () => { + await vi.advanceTimersByTimeAsync(5000); + }); + + expect(listExecutionsMock).toHaveBeenCalledTimes(2); + }); + + it('shares dashboard and queue status requests across concurrent hook instances', async () => { + const firstDashboard = renderHook(() => useTaskDashboard()); + const secondDashboard = renderHook(() => useTaskDashboard()); + const firstQueue = renderHook(() => useQueueStatus()); + const secondQueue = renderHook(() => useQueueStatus()); + + await waitFor(() => { + expect(firstDashboard.result.current.loading).toBe(false); + expect(secondDashboard.result.current.loading).toBe(false); + expect(firstQueue.result.current.loading).toBe(false); + expect(secondQueue.result.current.loading).toBe(false); + }); + + expect(dashboardMock).toHaveBeenCalledTimes(1); + expect(queueStatusMock).toHaveBeenCalledTimes(1); + }); + + it('does not fetch scheduler execution records when scheduler id is absent', async () => { + const { result } = renderHook(() => useTaskExecutionsByScheduler(undefined)); + + expect(result.current.loading).toBe(false); + expect(result.current.records).toEqual([]); + expect(listSchedulerExecutionsMock).not.toHaveBeenCalled(); + }); +}); diff --git a/webui/src/hooks/useTasks.ts b/webui/src/hooks/useTasks.ts index d8189ce4a..38f87434a 100644 --- a/webui/src/hooks/useTasks.ts +++ b/webui/src/hooks/useTasks.ts @@ -1,4 +1,4 @@ -import { useState, useEffect, useRef, useCallback } from 'react'; +import { useCallback, useEffect, useMemo, useRef, useState } from 'react'; import { taskAPI, TaskExecution, @@ -8,143 +8,270 @@ import { DashboardCounts, QueueStatus, TaskSystemNotice, + PaginatedResponse, } from '@/api/task'; +import { createSharedResource, useSharedResource, type SharedResource } from './useSharedResource'; const ACTIVE_EXECUTION_STATUSES = new Set(['pending', 'queued', 'running']); const ACTIVE_SCHEDULER_STATUSES = new Set(['active']); +const TASK_LIST_STALE_TIME_MS = 1000; +const TASK_LIST_MIN_FETCH_INTERVAL_MS = 1000; + +type TaskPageData = Pick, 'items' | 'total'>; + +const EMPTY_SCHEDULERS: TaskPageData = { items: [], total: 0 }; +const EMPTY_EXECUTIONS: TaskPageData = { items: [], total: 0 }; + +const schedulerListResources = new Map>>(); +const executionListResources = new Map>>(); +const schedulerExecutionResources = new Map>>(); + +const taskDashboardResource = createSharedResource({ + initialData: null, + staleTimeMs: TASK_LIST_STALE_TIME_MS, + minFetchIntervalMs: TASK_LIST_MIN_FETCH_INTERVAL_MS, + fetcher: async () => { + const response = await taskAPI.dashboard(); + return response.data; + }, + getErrorMessage: (err) => (err instanceof Error && err.message ? err.message : 'Failed to fetch dashboard'), +}); + +const queueStatusResource = createSharedResource({ + initialData: null, + staleTimeMs: TASK_LIST_STALE_TIME_MS, + minFetchIntervalMs: TASK_LIST_MIN_FETCH_INTERVAL_MS, + fetcher: async () => { + const response = await taskAPI.queueStatus(); + return response.data; + }, + getErrorMessage: (err) => (err instanceof Error && err.message ? err.message : 'Failed to fetch queue status'), +}); + +const taskSystemNoticeResource = createSharedResource({ + initialData: null, + staleTimeMs: 10000, + minFetchIntervalMs: TASK_LIST_MIN_FETCH_INTERVAL_MS, + fetcher: async () => { + const response = await taskAPI.getSystemNotice(); + return response.data ?? null; + }, + getErrorMessage: (err) => (err instanceof Error && err.message ? err.message : 'Failed to fetch system notice'), +}); + +function normalizeSchedulerFilters(filters?: SchedulerListParams): SchedulerListParams { + return { + status: filters?.status, + priority: filters?.priority, + scheduledOnly: filters?.scheduledOnly, + sortBy: filters?.sortBy, + sortOrder: filters?.sortOrder, + offset: filters?.offset, + limit: filters?.limit, + }; +} -export function useTaskSchedulers( - filters?: SchedulerListParams, - options?: { pollInterval?: number }, -) { - const [tasks, setTasks] = useState([]); - const [total, setTotal] = useState(0); - const [loading, setLoading] = useState(true); - const [error, setError] = useState(null); - const tasksRef = useRef([]); - const initializedRef = useRef(false); - - const fetchTasks = useCallback(async () => { - try { - if (!initializedRef.current) setLoading(true); - setError(null); - const response = await taskAPI.listSchedulers(filters); - const data = response.data; - const items = data.items ?? []; - setTasks(items); - setTotal(data.total ?? 0); - tasksRef.current = items; - } catch (err: any) { - setError(err.message || 'Failed to fetch tasks'); - setTasks([]); - setTotal(0); - } finally { - setLoading(false); - initializedRef.current = true; - } - }, [ - filters?.status, - filters?.priority, - filters?.scheduledOnly, - filters?.sortBy, - filters?.sortOrder, - filters?.offset, - filters?.limit, - ]); - - useEffect(() => { - fetchTasks(); - }, [fetchTasks]); - - // Auto-polling: use shorter interval when there are active tasks - useEffect(() => { - const baseInterval = options?.pollInterval; - if (!baseInterval) return; - - const schedule = () => { - const hasActive = tasksRef.current.some(t => ACTIVE_SCHEDULER_STATUSES.has(t.status)); - return hasActive ? Math.min(baseInterval, 4000) : baseInterval; - }; +function normalizeExecutionFilters(filters?: TaskListParams & { schedulerID?: string }): TaskListParams & { schedulerID?: string } { + return { + status: filters?.status, + priority: filters?.priority, + deliveryStatus: filters?.deliveryStatus, + schedulerID: filters?.schedulerID, + sortBy: filters?.sortBy, + sortOrder: filters?.sortOrder, + offset: filters?.offset, + limit: filters?.limit, + }; +} - let timerId: ReturnType; +function makeResourceKey(params: object): string { + return JSON.stringify(params); +} - const tick = async () => { - await fetchTasks(); - timerId = setTimeout(tick, schedule()); - }; +function getSchedulerListResource(filters?: SchedulerListParams): SharedResource> { + const params = normalizeSchedulerFilters(filters); + const key = makeResourceKey(params); + const existing = schedulerListResources.get(key); + if (existing) return existing; + + const resource = createSharedResource>({ + initialData: EMPTY_SCHEDULERS, + staleTimeMs: TASK_LIST_STALE_TIME_MS, + minFetchIntervalMs: TASK_LIST_MIN_FETCH_INTERVAL_MS, + fetcher: async () => { + const response = await taskAPI.listSchedulers(params); + return { + items: response.data.items ?? [], + total: response.data.total ?? 0, + }; + }, + fallbackDataOnError: EMPTY_SCHEDULERS, + getErrorMessage: (err) => (err instanceof Error && err.message ? err.message : 'Failed to fetch tasks'), + }); + + schedulerListResources.set(key, resource); + return resource; +} - timerId = setTimeout(tick, schedule()); - return () => clearTimeout(timerId); - }, [fetchTasks, options?.pollInterval]); +function getExecutionListResource(filters?: TaskListParams & { schedulerID?: string }): SharedResource> { + const params = normalizeExecutionFilters(filters); + const key = makeResourceKey(params); + const existing = executionListResources.get(key); + if (existing) return existing; + + const resource = createSharedResource>({ + initialData: EMPTY_EXECUTIONS, + staleTimeMs: TASK_LIST_STALE_TIME_MS, + minFetchIntervalMs: TASK_LIST_MIN_FETCH_INTERVAL_MS, + fetcher: async () => { + const response = await taskAPI.listExecutions(params); + return { + items: response.data.items ?? [], + total: response.data.total ?? 0, + }; + }, + fallbackDataOnError: EMPTY_EXECUTIONS, + getErrorMessage: (err) => (err instanceof Error && err.message ? err.message : 'Failed to fetch task executions'), + }); + + executionListResources.set(key, resource); + return resource; +} - return { tasks, total, loading, error, refetch: fetchTasks }; +function getSchedulerExecutionResource( + schedulerId: string, + params?: { offset?: number; limit?: number }, +): SharedResource> { + const normalizedParams = { + schedulerId, + offset: params?.offset, + limit: params?.limit, + }; + const key = makeResourceKey(normalizedParams); + const existing = schedulerExecutionResources.get(key); + if (existing) return existing; + + const resource = createSharedResource>({ + initialData: EMPTY_EXECUTIONS, + staleTimeMs: TASK_LIST_STALE_TIME_MS, + minFetchIntervalMs: TASK_LIST_MIN_FETCH_INTERVAL_MS, + fetcher: async () => { + const response = await taskAPI.listSchedulerExecutions(schedulerId, { + offset: params?.offset, + limit: params?.limit, + }); + return { + items: response.data.items ?? [], + total: response.data.total ?? 0, + }; + }, + fallbackDataOnError: EMPTY_EXECUTIONS, + getErrorMessage: (err) => (err instanceof Error && err.message ? err.message : 'Failed to fetch executions'), + }); + + schedulerExecutionResources.set(key, resource); + return resource; } -export function useTaskExecutions( - filters?: TaskListParams & { schedulerID?: string }, - options?: { pollInterval?: number }, +function usePollSharedResource( + resource: SharedResource, + pollInterval?: number, ) { - const [tasks, setTasks] = useState([]); - const [total, setTotal] = useState(0); - const [loading, setLoading] = useState(true); - const [error, setError] = useState(null); - const tasksRef = useRef([]); - const initializedRef = useRef(false); + useEffect(() => { + if (!pollInterval) return; + const id = window.setInterval(() => { + void resource.fetch({ silent: true }); + }, pollInterval); + return () => window.clearInterval(id); + }, [pollInterval, resource]); +} - const fetchTasks = useCallback(async () => { - try { - if (!initializedRef.current) setLoading(true); - setError(null); - const response = await taskAPI.listExecutions(filters); - const data = response.data; - const items = data.items ?? []; - setTasks(items); - setTotal(data.total ?? 0); - tasksRef.current = items; - } catch (err: any) { - setError(err.message || 'Failed to fetch task executions'); - setTasks([]); - setTotal(0); - } finally { - setLoading(false); - initializedRef.current = true; - } - }, [ - filters?.status, - filters?.priority, - filters?.deliveryStatus, - filters?.schedulerID, - filters?.sortBy, - filters?.sortOrder, - filters?.offset, - filters?.limit, - ]); +function useAdaptiveTaskPolling( + resource: SharedResource>, + items: T[], + activeStatuses: Set, + pollInterval?: number, +) { + const itemsRef = useRef(items); useEffect(() => { - fetchTasks(); - }, [fetchTasks]); + itemsRef.current = items; + }, [items]); useEffect(() => { - const baseInterval = options?.pollInterval; - if (!baseInterval) return; + if (!pollInterval) return; - const schedule = () => { - const hasActive = tasksRef.current.some(t => ACTIVE_EXECUTION_STATUSES.has(t.status)); - return hasActive ? Math.min(baseInterval, 4000) : baseInterval; + const scheduleDelay = () => { + const hasActive = itemsRef.current.some((task) => activeStatuses.has(task.status)); + return hasActive ? Math.min(pollInterval, 4000) : pollInterval; }; let timerId: ReturnType; - const tick = async () => { - await fetchTasks(); - timerId = setTimeout(tick, schedule()); + await resource.fetch({ silent: true }); + timerId = window.setTimeout(tick, scheduleDelay()); }; - timerId = setTimeout(tick, schedule()); - return () => clearTimeout(timerId); - }, [fetchTasks, options?.pollInterval]); + timerId = window.setTimeout(tick, scheduleDelay()); + return () => window.clearTimeout(timerId); + }, [activeStatuses, pollInterval, resource]); +} + +export function __resetTaskResourcesForTesting(): void { + schedulerListResources.forEach((resource) => resource.resetForTesting()); + executionListResources.forEach((resource) => resource.resetForTesting()); + schedulerExecutionResources.forEach((resource) => resource.resetForTesting()); + schedulerListResources.clear(); + executionListResources.clear(); + schedulerExecutionResources.clear(); + taskDashboardResource.resetForTesting(); + queueStatusResource.resetForTesting(); + taskSystemNoticeResource.resetForTesting(); +} - return { tasks, total, loading, error, refetch: fetchTasks }; +export function useTaskSchedulers( + filters?: SchedulerListParams, + options?: { pollInterval?: number }, +) { + const resource = useMemo( + () => getSchedulerListResource(filters), + [ + filters?.status, + filters?.priority, + filters?.scheduledOnly, + filters?.sortBy, + filters?.sortOrder, + filters?.offset, + filters?.limit, + ], + ); + const { data, loading, error, refetch } = useSharedResource(resource); + useAdaptiveTaskPolling(resource, data.items, ACTIVE_SCHEDULER_STATUSES, options?.pollInterval); + + return { tasks: data.items, total: data.total, loading, error, refetch }; +} + +export function useTaskExecutions( + filters?: TaskListParams & { schedulerID?: string }, + options?: { pollInterval?: number }, +) { + const resource = useMemo( + () => getExecutionListResource(filters), + [ + filters?.status, + filters?.priority, + filters?.deliveryStatus, + filters?.schedulerID, + filters?.sortBy, + filters?.sortOrder, + filters?.offset, + filters?.limit, + ], + ); + const { data, loading, error, refetch } = useSharedResource(resource); + useAdaptiveTaskPolling(resource, data.items, ACTIVE_EXECUTION_STATUSES, options?.pollInterval); + + return { tasks: data.items, total: data.total, loading, error, refetch }; } export function useTaskScheduler(schedulerId?: string) { @@ -174,123 +301,60 @@ export function useTaskScheduler(schedulerId?: string) { } export function useTaskDashboard(options?: { pollInterval?: number }) { - const [counts, setCounts] = useState(null); - const [loading, setLoading] = useState(true); - const [error, setError] = useState(null); - const initializedRef = useRef(false); - - const fetchDashboard = useCallback(async () => { - try { - if (!initializedRef.current) setLoading(true); - setError(null); - const response = await taskAPI.dashboard(); - setCounts(response.data); - } catch (err: any) { - setError(err.message || 'Failed to fetch dashboard'); - } finally { - setLoading(false); - initializedRef.current = true; - } - }, []); - - useEffect(() => { - fetchDashboard(); - }, [fetchDashboard]); - - useEffect(() => { - if (!options?.pollInterval) return; - const id = setInterval(fetchDashboard, options.pollInterval); - return () => clearInterval(id); - }, [fetchDashboard, options?.pollInterval]); - - return { counts, loading, error, refetch: fetchDashboard }; + const { + data: counts, + loading, + error, + refetch, + } = useSharedResource(taskDashboardResource); + usePollSharedResource(taskDashboardResource, options?.pollInterval); + + return { counts, loading, error, refetch }; } export function useTaskExecutionsByScheduler(schedulerId?: string, params?: { offset?: number; limit?: number }) { - const [records, setRecords] = useState([]); - const [total, setTotal] = useState(0); - const [loading, setLoading] = useState(false); - const [error, setError] = useState(null); - - const fetchRecords = useCallback(async () => { - if (!schedulerId) return; - try { - setLoading(true); - setError(null); - const response = await taskAPI.listSchedulerExecutions(schedulerId, params); - setRecords(response.data.items ?? []); - setTotal(response.data.total ?? 0); - } catch (err: any) { - setError(err.message || 'Failed to fetch executions'); - } finally { - setLoading(false); - } - }, [schedulerId, params?.offset, params?.limit]); - - useEffect(() => { - fetchRecords(); - }, [fetchRecords]); - - return { records, total, loading, error, refetch: fetchRecords }; + const resource = useMemo( + () => (schedulerId ? getSchedulerExecutionResource(schedulerId, params) : null), + [schedulerId, params?.offset, params?.limit], + ); + const { + data, + loading, + error, + refetch, + } = useSharedResource(resource ?? getSchedulerExecutionResource('__disabled__'), { + enabled: !!resource, + loadOnMount: !!resource, + }); + + return { + records: resource ? data.items : [], + total: resource ? data.total : 0, + loading: resource ? loading : false, + error: resource ? error : null, + refetch: resource ? refetch : async () => EMPTY_EXECUTIONS, + }; } export function useQueueStatus(options?: { pollInterval?: number }) { - const [queueStatus, setQueueStatus] = useState(null); - const [loading, setLoading] = useState(true); - const [error, setError] = useState(null); - const initializedRef = useRef(false); - - const fetchQueueStatus = useCallback(async () => { - try { - if (!initializedRef.current) setLoading(true); - setError(null); - const response = await taskAPI.queueStatus(); - setQueueStatus(response.data); - } catch (err: any) { - setError(err.message || 'Failed to fetch queue status'); - } finally { - setLoading(false); - initializedRef.current = true; - } - }, []); - - useEffect(() => { - fetchQueueStatus(); - }, [fetchQueueStatus]); - - useEffect(() => { - if (!options?.pollInterval) return; - const id = setInterval(fetchQueueStatus, options.pollInterval); - return () => clearInterval(id); - }, [fetchQueueStatus, options?.pollInterval]); - - return { queueStatus, loading, error, refetch: fetchQueueStatus }; + const { + data: queueStatus, + loading, + error, + refetch, + } = useSharedResource(queueStatusResource); + usePollSharedResource(queueStatusResource, options?.pollInterval); + + return { queueStatus, loading, error, refetch }; } export function useTaskSystemNotice() { - const [notice, setNotice] = useState(null); - const [loading, setLoading] = useState(false); - const [error, setError] = useState(null); - const initializedRef = useRef(false); - - const fetchNotice = useCallback(async () => { - try { - if (!initializedRef.current) setLoading(true); - setError(null); - const response = await taskAPI.getSystemNotice(); - setNotice(response.data ?? null); - } catch (err: any) { - setError(err.message || 'Failed to fetch system notice'); - } finally { - setLoading(false); - initializedRef.current = true; - } - }, []); - - useEffect(() => { - fetchNotice(); - }, [fetchNotice]); - - return { notice, loading, error, refetch: fetchNotice }; + const { + data: notice, + loading, + error, + refetch, + } = useSharedResource(taskSystemNoticeResource); + + return { notice, loading, error, refetch }; } - diff --git a/webui/src/hooks/useTools.test.tsx b/webui/src/hooks/useTools.test.tsx index 0f7711090..cff439668 100644 --- a/webui/src/hooks/useTools.test.tsx +++ b/webui/src/hooks/useTools.test.tsx @@ -1,7 +1,7 @@ import { act, renderHook, waitFor } from '@testing-library/react'; import { beforeEach, describe, expect, it, vi } from 'vitest'; -import { useTools } from './useTools'; +import { __resetToolsResourceForTesting, useTools } from './useTools'; const { listMock, refreshMock } = vi.hoisted(() => ({ listMock: vi.fn(), @@ -18,6 +18,7 @@ vi.mock('@/api/tool', () => ({ describe('useTools', () => { beforeEach(() => { vi.clearAllMocks(); + __resetToolsResourceForTesting(); }); it('renders the tool list without automatically refreshing plugins', async () => { @@ -66,11 +67,14 @@ describe('useTools', () => { expect(result.current.tools).toHaveLength(1); expect(refreshMock).not.toHaveBeenCalled(); + const futureNow = Date.now() + 6000; + const nowSpy = vi.spyOn(Date, 'now').mockReturnValue(futureNow); window.dispatchEvent(new Event('focus')); await waitFor(() => { expect(result.current.tools).toHaveLength(2); }); + nowSpy.mockRestore(); expect(refreshMock).not.toHaveBeenCalled(); expect(listMock).toHaveBeenCalledTimes(2); @@ -106,4 +110,30 @@ describe('useTools', () => { expect(refreshMock).toHaveBeenCalledTimes(1); expect(listMock).toHaveBeenCalledTimes(2); }); + + it('shares the initial tool list request across concurrent hook instances', async () => { + let resolveList: (value: { data: any[] }) => void = () => {}; + listMock.mockReturnValue(new Promise((resolve) => { + resolveList = resolve; + })); + + const first = renderHook(() => useTools()); + const second = renderHook(() => useTools()); + + expect(listMock).toHaveBeenCalledTimes(1); + + await act(async () => { + resolveList({ + data: [{ name: 'tool-alpha', description: 'alpha tool', category: 'custom', source: 'custom', enabled: true }], + }); + }); + + await waitFor(() => { + expect(first.result.current.loading).toBe(false); + expect(second.result.current.loading).toBe(false); + }); + + expect(first.result.current.tools).toHaveLength(1); + expect(second.result.current.tools).toHaveLength(1); + }); }); diff --git a/webui/src/hooks/useTools.ts b/webui/src/hooks/useTools.ts index 7a2c2dcba..7b9607204 100644 --- a/webui/src/hooks/useTools.ts +++ b/webui/src/hooks/useTools.ts @@ -1,64 +1,65 @@ -import { useState, useEffect, useCallback, useRef } from 'react'; +import { useCallback } from 'react'; import { toolAPI, Tool } from '@/api/tool'; +import { createSharedResource, useRefreshOnResume, useSharedResource } from './useSharedResource'; -export function useTools() { - const [tools, setTools] = useState([]); - const [loading, setLoading] = useState(true); - const [error, setError] = useState(null); - const initializedRef = useRef(false); +const TOOL_LIST_STALE_TIME_MS = 5000; +const TOOL_LIST_MIN_FETCH_INTERVAL_MS = 1000; + +const toolsResource = createSharedResource({ + initialData: [], + staleTimeMs: TOOL_LIST_STALE_TIME_MS, + minFetchIntervalMs: TOOL_LIST_MIN_FETCH_INTERVAL_MS, + fetcher: async () => { + const response = await toolAPI.list(); + return Array.isArray(response.data) ? response.data : []; + }, + getErrorMessage: (err) => (err instanceof Error && err.message ? err.message : 'Failed to fetch tools'), +}); - const fetchTools = useCallback(async (showLoading = false) => { - try { - if (showLoading && !initializedRef.current) setLoading(true); - setError(null); - const response = await toolAPI.list(); - setTools(Array.isArray(response.data) ? response.data : []); - } catch (err: any) { - setError(err.message || 'Failed to fetch tools'); - } finally { - if (showLoading && !initializedRef.current) setLoading(false); - initializedRef.current = true; - } - }, []); +let toolRefreshInFlight: Promise | null = null; - const refreshAndFetch = useCallback(async () => { - try { - await toolAPI.refresh(); - } catch { /* ignore */ } - await fetchTools(false); - }, [fetchTools]); +function refreshToolsResource(): Promise { + if (toolRefreshInFlight) { + return toolRefreshInFlight; + } - useEffect(() => { - let cancelled = false; + toolRefreshInFlight = toolAPI.refresh() + .catch(() => { + // Best-effort refresh; still update the visible list afterwards. + }) + .then(() => toolsResource.fetch({ force: true, silent: true })) + .then(() => undefined) + .finally(() => { + toolRefreshInFlight = null; + }); - const init = async () => { - await fetchTools(true); - if (cancelled) return; - }; + return toolRefreshInFlight; +} + +export function __resetToolsResourceForTesting(): void { + toolsResource.resetForTesting(); + toolRefreshInFlight = null; +} + +export function useTools() { + const { + data: tools, + loading, + error, + } = useSharedResource(toolsResource); - void init(); + const refreshVisibleList = useCallback( + () => toolsResource.fetch({ silent: true }), + [], + ); + useRefreshOnResume(refreshVisibleList); - const onVisible = () => { - if (document.visibilityState === 'visible') { - void fetchTools(false); - } - }; - const onFocus = () => { - void fetchTools(false); - }; - document.addEventListener('visibilitychange', onVisible); - window.addEventListener('focus', onFocus); - return () => { - cancelled = true; - document.removeEventListener('visibilitychange', onVisible); - window.removeEventListener('focus', onFocus); - }; - }, [fetchTools]); + const refetch = useCallback(() => refreshToolsResource(), []); return { tools, loading, error, - refetch: refreshAndFetch, + refetch, }; } diff --git a/webui/src/hooks/useWebUIContractPages.test.tsx b/webui/src/hooks/useWebUIContractPages.test.tsx index afbebdfb1..d38000e8e 100644 --- a/webui/src/hooks/useWebUIContractPages.test.tsx +++ b/webui/src/hooks/useWebUIContractPages.test.tsx @@ -1,7 +1,7 @@ -import { renderHook, waitFor } from '@testing-library/react'; +import { act, renderHook, waitFor } from '@testing-library/react'; import { beforeEach, describe, expect, it, vi } from 'vitest'; -import { useWebUIContractPages } from './useWebUIContractPages'; +import { __resetWebUIContractPagesResourceForTesting, useWebUIContractPages } from './useWebUIContractPages'; import { setupSSEMock } from '@/test/mocks/sse'; const { listMock, listWorkspacesMock } = vi.hoisted(() => ({ @@ -21,6 +21,7 @@ describe('useWebUIContractPages', () => { beforeEach(() => { vi.clearAllMocks(); + __resetWebUIContractPagesResourceForTesting(); listWorkspacesMock.mockResolvedValue({ data: [] }); }); @@ -54,6 +55,38 @@ describe('useWebUIContractPages', () => { expect(listWorkspacesMock).toHaveBeenCalledWith(true); }); + it('shares the initial navigation request across concurrent hook instances', async () => { + let resolvePages: (value: { data: any[] }) => void = () => {}; + let resolveWorkspaces: (value: { data: any[] }) => void = () => {}; + listMock.mockReturnValue(new Promise((resolve) => { + resolvePages = resolve; + })); + listWorkspacesMock.mockReturnValue(new Promise((resolve) => { + resolveWorkspaces = resolve; + })); + + const first = renderHook(() => useWebUIContractPages()); + const second = renderHook(() => useWebUIContractPages()); + + expect(listMock).toHaveBeenCalledTimes(1); + expect(listWorkspacesMock).toHaveBeenCalledTimes(1); + + await act(async () => { + resolvePages({ data: [{ id: 'dash-1', title: '仪表盘' }] }); + resolveWorkspaces({ data: [{ id: 'workspace-1', title: '工作区', pages: [] }] }); + }); + + await waitFor(() => { + expect(first.result.current.loading).toBe(false); + expect(second.result.current.loading).toBe(false); + }); + + expect(first.result.current.pages).toHaveLength(1); + expect(second.result.current.pages).toHaveLength(1); + expect(first.result.current.workspaces).toHaveLength(1); + expect(second.result.current.workspaces).toHaveLength(1); + }); + it('refetches when contracts.webui.pages.nav_changed SSE event arrives', async () => { listMock .mockResolvedValueOnce({ data: [] }) diff --git a/webui/src/hooks/useWebUIContractPages.ts b/webui/src/hooks/useWebUIContractPages.ts index 4457bd9c8..943075750 100644 --- a/webui/src/hooks/useWebUIContractPages.ts +++ b/webui/src/hooks/useWebUIContractPages.ts @@ -1,4 +1,4 @@ -import { useCallback, useEffect, useRef, useState } from 'react'; +import { useCallback } from 'react'; import i18n from '@/i18n'; import { webuiContractPagesAPI, @@ -6,80 +6,75 @@ import { type WebUIContractWorkspaceListItem, } from '@/api/webuiContractPages'; import { useSSE } from '@/hooks/useSSE'; +import { createSharedResource, useRefreshOnResume, useSharedResource } from './useSharedResource'; -export function useWebUIContractPages() { - const [pages, setPages] = useState([]); - const [workspaces, setWorkspaces] = useState([]); - const [loading, setLoading] = useState(true); - const [error, setError] = useState(null); - const loadingRef = useRef(false); - const lastRefreshRef = useRef(0); +interface WebUIContractNavResourceData { + pages: WebUIContractPageListItem[]; + workspaces: WebUIContractWorkspaceListItem[]; +} - const fetchPages = useCallback(async (silent = false) => { - if (loadingRef.current) return; - loadingRef.current = true; - if (!silent) setLoading(true); - setError(null); - try { - const [pagesResponse, workspacesResponse] = await Promise.all([ - webuiContractPagesAPI.list(true), - webuiContractPagesAPI.listWorkspaces(true), - ]); - setPages(Array.isArray(pagesResponse.data) ? pagesResponse.data : []); - setWorkspaces(Array.isArray(workspacesResponse.data) ? workspacesResponse.data : []); - } catch (err: unknown) { - setPages([]); - setWorkspaces([]); - setError(err instanceof Error ? err.message : i18n.t('nav.fetchFailed', { ns: 'webuiContractPage' })); - } finally { - loadingRef.current = false; - if (!silent) setLoading(false); - } - }, []); +const WEBUI_CONTRACT_NAV_STALE_TIME_MS = 1000; +const WEBUI_CONTRACT_NAV_MIN_FETCH_INTERVAL_MS = 1000; - useEffect(() => { - void fetchPages(); - }, [fetchPages]); +const webuiContractNavResource = createSharedResource({ + initialData: { + pages: [], + workspaces: [], + }, + staleTimeMs: WEBUI_CONTRACT_NAV_STALE_TIME_MS, + minFetchIntervalMs: WEBUI_CONTRACT_NAV_MIN_FETCH_INTERVAL_MS, + fetcher: async () => { + const [pagesResponse, workspacesResponse] = await Promise.all([ + webuiContractPagesAPI.list(true), + webuiContractPagesAPI.listWorkspaces(true), + ]); + return { + pages: Array.isArray(pagesResponse.data) ? pagesResponse.data : [], + workspaces: Array.isArray(workspacesResponse.data) ? workspacesResponse.data : [], + }; + }, + fallbackDataOnError: { + pages: [], + workspaces: [], + }, + getErrorMessage: (err) => ( + err instanceof Error ? err.message : i18n.t('nav.fetchFailed', { ns: 'webuiContractPage' }) + ), +}); - const refreshOnResume = useCallback((force = false) => { - const now = Date.now(); - if (!force && now - lastRefreshRef.current < 1000) return; - lastRefreshRef.current = now; - void fetchPages(true); - }, [fetchPages]); +export function __resetWebUIContractPagesResourceForTesting(): void { + webuiContractNavResource.resetForTesting(); +} - useEffect(() => { - const onVisible = () => { - if (document.visibilityState === 'visible') { - refreshOnResume(false); - } - }; - const onFocus = () => { - refreshOnResume(false); - }; - document.addEventListener('visibilitychange', onVisible); - window.addEventListener('focus', onFocus); - return () => { - document.removeEventListener('visibilitychange', onVisible); - window.removeEventListener('focus', onFocus); - }; - }, [refreshOnResume]); +export function useWebUIContractPages() { + const { + data, + loading, + error, + refetch, + } = useSharedResource(webuiContractNavResource); + + const refreshOnResume = useCallback( + () => webuiContractNavResource.fetch({ silent: true }), + [], + ); + useRefreshOnResume(refreshOnResume); useSSE({ url: '/api/event', onEvent: useCallback((evt) => { if (evt.type === 'contracts.webui.pages.nav_changed') { - void fetchPages(true); + void webuiContractNavResource.fetch({ force: true, silent: true }); } - }, [fetchPages]), + }, []), reconnect: { maxRetries: 5, initialDelay: 2000 }, }); return { - pages, - workspaces, + pages: data.pages, + workspaces: data.workspaces, loading, error, - refetch: () => fetchPages(), + refetch, }; } diff --git a/webui/src/hooks/useWorkflow.test.tsx b/webui/src/hooks/useWorkflow.test.tsx index c9566a604..671ac15ce 100644 --- a/webui/src/hooks/useWorkflow.test.tsx +++ b/webui/src/hooks/useWorkflow.test.tsx @@ -1,7 +1,8 @@ -import { renderHook, waitFor } from '@testing-library/react'; +import { act, renderHook, waitFor } from '@testing-library/react'; import { beforeEach, describe, expect, it, vi } from 'vitest'; -import { useWorkflows } from './useWorkflow'; +import type { Workflow } from '@/api/workflow'; +import { __resetWorkflowResourcesForTesting, useWorkflows } from './useWorkflow'; const { listMock, getMock } = vi.hoisted(() => ({ listMock: vi.fn(), @@ -15,7 +16,7 @@ vi.mock('@/api/workflow', () => ({ }, })); -function makeWorkflow() { +function makeWorkflow(overrides: Partial = {}): Workflow { return { id: 'wf-1', name: 'Workflow One', @@ -38,12 +39,14 @@ function makeWorkflow() { thumbsUp: 0, thumbsDown: 0, }, + ...overrides, }; } describe('useWorkflows', () => { beforeEach(() => { vi.clearAllMocks(); + __resetWorkflowResourcesForTesting(); }); it('clears workflows when a silent refetch fails', async () => { @@ -60,11 +63,14 @@ describe('useWorkflows', () => { expect(result.current.workflows).toHaveLength(1); + const futureNow = Date.now() + 6000; + const nowSpy = vi.spyOn(Date, 'now').mockReturnValue(futureNow); window.dispatchEvent(new Event('focus')); await waitFor(() => { expect(result.current.workflows).toEqual([]); }); + nowSpy.mockRestore(); expect(result.current.error).toBe('Session expired'); }); @@ -90,10 +96,40 @@ describe('useWorkflows', () => { configurable: true, value: 'visible', }); + const futureNow = Date.now() + 6000; + const nowSpy = vi.spyOn(Date, 'now').mockReturnValue(futureNow); document.dispatchEvent(new Event('visibilitychange')); await waitFor(() => { expect(result.current.workflows).toHaveLength(2); }); + nowSpy.mockRestore(); + }); + + it('shares workflow list requests across concurrent hook instances with the same filters', async () => { + let resolveList: (value: { data: any[] }) => void = () => {}; + listMock.mockReturnValue(new Promise((resolve) => { + resolveList = resolve; + })); + + const first = renderHook(() => useWorkflows('default', 'active')); + const second = renderHook(() => useWorkflows('default', 'active')); + + expect(listMock).toHaveBeenCalledTimes(1); + expect(listMock).toHaveBeenCalledWith({ category: 'default', status: 'active' }); + + await act(async () => { + resolveList({ + data: [makeWorkflow()], + }); + }); + + await waitFor(() => { + expect(first.result.current.loading).toBe(false); + expect(second.result.current.loading).toBe(false); + }); + + expect(first.result.current.workflows).toHaveLength(1); + expect(second.result.current.workflows).toHaveLength(1); }); }); diff --git a/webui/src/hooks/useWorkflow.ts b/webui/src/hooks/useWorkflow.ts index 466a76290..69ec589e4 100644 --- a/webui/src/hooks/useWorkflow.ts +++ b/webui/src/hooks/useWorkflow.ts @@ -1,62 +1,67 @@ -import { useState, useEffect, useCallback, useRef } from 'react'; +import { useCallback, useEffect, useMemo, useState } from 'react'; import { workflowAPI, Workflow } from '@/api/workflow'; +import { + createSharedResource, + useRefreshOnResume, + useSharedResource, + type SharedResource, +} from './useSharedResource'; -export function useWorkflows(category?: string, status?: string) { - const [workflows, setWorkflows] = useState([]); - const [loading, setLoading] = useState(true); - const [error, setError] = useState(null); - const loadingRef = useRef(false); - const lastRefreshRef = useRef(0); +const WORKFLOW_LIST_STALE_TIME_MS = 5000; +const WORKFLOW_LIST_MIN_FETCH_INTERVAL_MS = 1000; - const fetchWorkflows = useCallback(async (silent = false) => { - if (loadingRef.current) return; - loadingRef.current = true; - if (!silent) setLoading(true); - setError(null); - try { +const workflowListResources = new Map>(); + +function makeWorkflowListKey(category?: string, status?: string): string { + return JSON.stringify({ + category: category ?? null, + status: status ?? null, + }); +} + +function getWorkflowListResource(category?: string, status?: string): SharedResource { + const key = makeWorkflowListKey(category, status); + const existing = workflowListResources.get(key); + if (existing) return existing; + + const resource = createSharedResource({ + initialData: [], + staleTimeMs: WORKFLOW_LIST_STALE_TIME_MS, + minFetchIntervalMs: WORKFLOW_LIST_MIN_FETCH_INTERVAL_MS, + fetcher: async () => { const response = await workflowAPI.list({ category, status }); - if (Array.isArray(response.data)) { - setWorkflows(response.data); - } else { - setWorkflows([]); - } - } catch (err: any) { - setError(err.message || 'Failed to fetch workflows'); - setWorkflows([]); - } finally { - loadingRef.current = false; - if (!silent) setLoading(false); - } - }, [category, status]); + return Array.isArray(response.data) ? response.data : []; + }, + fallbackDataOnError: [], + getErrorMessage: (err) => (err instanceof Error && err.message ? err.message : 'Failed to fetch workflows'), + }); - // Initial fetch - useEffect(() => { - void fetchWorkflows(); - }, [fetchWorkflows]); + workflowListResources.set(key, resource); + return resource; +} - const refreshOnResume = useCallback((force = false) => { - const now = Date.now(); - if (!force && now - lastRefreshRef.current < 1000) return; - lastRefreshRef.current = now; - void fetchWorkflows(true); - }, [fetchWorkflows]); +export function __resetWorkflowResourcesForTesting(): void { + workflowListResources.forEach((resource) => resource.resetForTesting()); + workflowListResources.clear(); +} - useEffect(() => { - const onVisible = () => { - if (document.visibilityState === 'visible') { - refreshOnResume(false); - } - }; - const onFocus = () => { - refreshOnResume(false); - }; - document.addEventListener('visibilitychange', onVisible); - window.addEventListener('focus', onFocus); - return () => { - document.removeEventListener('visibilitychange', onVisible); - window.removeEventListener('focus', onFocus); - }; - }, [refreshOnResume]); +export function useWorkflows(category?: string, status?: string) { + const resource = useMemo( + () => getWorkflowListResource(category, status), + [category, status], + ); + const { + data: workflows, + loading, + error, + refetch: fetchWorkflows, + } = useSharedResource(resource); + + const refreshOnResume = useCallback( + () => resource.fetch({ silent: true }), + [resource], + ); + useRefreshOnResume(refreshOnResume); return { workflows, diff --git a/webui/src/i18n.test.ts b/webui/src/i18n.test.ts new file mode 100644 index 000000000..ed475f047 --- /dev/null +++ b/webui/src/i18n.test.ts @@ -0,0 +1,17 @@ +import { describe, expect, it } from 'vitest'; +import i18n, { preloadI18nNamespaces } from './i18n'; + +describe('i18n lazy namespaces', () => { + it('loads route namespaces on demand instead of bundling them up front', async () => { + await i18n.changeLanguage('en-US'); + i18n.removeResourceBundle('en-US', 'workflow'); + + expect(i18n.hasResourceBundle('en-US', 'home')).toBe(true); + expect(i18n.hasResourceBundle('en-US', 'workflow')).toBe(false); + + await preloadI18nNamespaces(['workflow']); + + expect(i18n.hasResourceBundle('en-US', 'workflow')).toBe(true); + expect(Object.keys(i18n.getResourceBundle('en-US', 'workflow'))).not.toHaveLength(0); + }); +}); diff --git a/webui/src/i18n.ts b/webui/src/i18n.ts index f362b884d..c2f5e6884 100644 --- a/webui/src/i18n.ts +++ b/webui/src/i18n.ts @@ -1,110 +1,30 @@ -import i18n from 'i18next'; +import i18n, { type BackendModule } from 'i18next'; import { initReactI18next } from 'react-i18next'; import LanguageDetector from 'i18next-browser-languagedetector'; +import { + INITIAL_NAMESPACES, + initialI18nResources, + preloadI18nNamespaces, + registerI18nInstance, + readLazyLocale, +} from './i18nResources'; -import enCommon from './locales/en-US/common.json'; -import enNav from './locales/en-US/nav.json'; -import enHome from './locales/en-US/home.json'; -import enSession from './locales/en-US/session.json'; -import enAgent from './locales/en-US/agent.json'; -import enTask from './locales/en-US/task.json'; -import enWorkflow from './locales/en-US/workflow.json'; -import enTool from './locales/en-US/tool.json'; -import enSkill from './locales/en-US/skill.json'; -import enModel from './locales/en-US/model.json'; -import enMcp from './locales/en-US/mcp.json'; -import enConfig from './locales/en-US/config.json'; -import enChannel from './locales/en-US/channel.json'; -import enPermission from './locales/en-US/permission.json'; -import enMonitoring from './locales/en-US/monitoring.json'; -import enUpdate from './locales/en-US/update.json'; -import enWorkspace from './locales/en-US/workspace.json'; -import enAuth from './locales/en-US/auth.json'; -import enNotification from './locales/en-US/notification.json'; -import enFlocksPro from './locales/en-US/flockspro.json'; -import enWebUIContractPage from './locales/en-US/webuiContractPage.json'; -import enDevice from './locales/en-US/device.json'; - -import zhCommon from './locales/zh-CN/common.json'; -import zhNav from './locales/zh-CN/nav.json'; -import zhHome from './locales/zh-CN/home.json'; -import zhSession from './locales/zh-CN/session.json'; -import zhAgent from './locales/zh-CN/agent.json'; -import zhTask from './locales/zh-CN/task.json'; -import zhWorkflow from './locales/zh-CN/workflow.json'; -import zhTool from './locales/zh-CN/tool.json'; -import zhSkill from './locales/zh-CN/skill.json'; -import zhModel from './locales/zh-CN/model.json'; -import zhMcp from './locales/zh-CN/mcp.json'; -import zhConfig from './locales/zh-CN/config.json'; -import zhChannel from './locales/zh-CN/channel.json'; -import zhPermission from './locales/zh-CN/permission.json'; -import zhMonitoring from './locales/zh-CN/monitoring.json'; -import zhUpdate from './locales/zh-CN/update.json'; -import zhWorkspace from './locales/zh-CN/workspace.json'; -import zhAuth from './locales/zh-CN/auth.json'; -import zhNotification from './locales/zh-CN/notification.json'; -import zhFlocksPro from './locales/zh-CN/flockspro.json'; -import zhWebUIContractPage from './locales/zh-CN/webuiContractPage.json'; -import zhDevice from './locales/zh-CN/device.json'; +const lazyLocaleBackend: BackendModule = { + type: 'backend', + init() {}, + read: readLazyLocale, +}; i18n + .use(lazyLocaleBackend) .use(LanguageDetector) .use(initReactI18next) .init({ - resources: { - 'en-US': { - common: enCommon, - nav: enNav, - home: enHome, - session: enSession, - agent: enAgent, - task: enTask, - workflow: enWorkflow, - tool: enTool, - skill: enSkill, - model: enModel, - mcp: enMcp, - config: enConfig, - channel: enChannel, - permission: enPermission, - monitoring: enMonitoring, - update: enUpdate, - workspace: enWorkspace, - auth: enAuth, - notification: enNotification, - flockspro: enFlocksPro, - webuiContractPage: enWebUIContractPage, - device: enDevice, - }, - 'zh-CN': { - common: zhCommon, - nav: zhNav, - home: zhHome, - session: zhSession, - agent: zhAgent, - task: zhTask, - workflow: zhWorkflow, - tool: zhTool, - skill: zhSkill, - model: zhModel, - mcp: zhMcp, - config: zhConfig, - channel: zhChannel, - permission: zhPermission, - monitoring: zhMonitoring, - update: zhUpdate, - workspace: zhWorkspace, - auth: zhAuth, - notification: zhNotification, - flockspro: zhFlocksPro, - webuiContractPage: zhWebUIContractPage, - device: zhDevice, - }, - }, + resources: initialI18nResources, fallbackLng: 'en-US', defaultNS: 'common', - ns: ['common', 'nav', 'home', 'session', 'agent', 'task', 'workflow', 'tool', 'skill', 'model', 'mcp', 'config', 'channel', 'permission', 'monitoring', 'update', 'workspace', 'auth', 'notification', 'flockspro', 'device', 'webuiContractPage'], + ns: INITIAL_NAMESPACES, + partialBundledLanguages: true, detection: { order: ['localStorage', 'navigator'], lookupLocalStorage: 'flocks-language', @@ -115,4 +35,7 @@ i18n }, }); +registerI18nInstance(i18n); + +export { preloadI18nNamespaces }; export default i18n; diff --git a/webui/src/i18nResources.ts b/webui/src/i18nResources.ts new file mode 100644 index 000000000..721c842b6 --- /dev/null +++ b/webui/src/i18nResources.ts @@ -0,0 +1,178 @@ +import type { ReadCallback, ResourceKey, i18n as I18nInstance } from 'i18next'; + +import enCommon from './locales/en-US/common.json'; +import enNav from './locales/en-US/nav.json'; +import enHome from './locales/en-US/home.json'; +import enAuth from './locales/en-US/auth.json'; +import enWebUIContractPage from './locales/en-US/webuiContractPage.json'; + +import zhCommon from './locales/zh-CN/common.json'; +import zhNav from './locales/zh-CN/nav.json'; +import zhHome from './locales/zh-CN/home.json'; +import zhAuth from './locales/zh-CN/auth.json'; +import zhWebUIContractPage from './locales/zh-CN/webuiContractPage.json'; + +export const INITIAL_NAMESPACES = ['common', 'nav', 'home', 'auth', 'webuiContractPage'] as const; +const LAZY_NAMESPACES = [ + 'agent', + 'channel', + 'config', + 'device', + 'flockspro', + 'mcp', + 'model', + 'monitoring', + 'notification', + 'permission', + 'session', + 'skill', + 'task', + 'tool', + 'update', + 'workflow', + 'workspace', +] as const; + +export const I18N_NAMESPACES = [...INITIAL_NAMESPACES, ...LAZY_NAMESPACES] as const; + +type SupportedLanguage = 'en-US' | 'zh-CN'; +type LazyNamespace = (typeof LAZY_NAMESPACES)[number]; +type LocaleModule = { default: ResourceKey }; +type LocaleLoader = () => Promise; + +export const initialI18nResources = { + 'en-US': { + common: enCommon, + nav: enNav, + home: enHome, + auth: enAuth, + webuiContractPage: enWebUIContractPage, + }, + 'zh-CN': { + common: zhCommon, + nav: zhNav, + home: zhHome, + auth: zhAuth, + webuiContractPage: zhWebUIContractPage, + }, +}; + +const lazyLocaleLoaders: Record> = { + 'en-US': { + agent: () => import('./locales/en-US/agent.json'), + channel: () => import('./locales/en-US/channel.json'), + config: () => import('./locales/en-US/config.json'), + device: () => import('./locales/en-US/device.json'), + flockspro: () => import('./locales/en-US/flockspro.json'), + mcp: () => import('./locales/en-US/mcp.json'), + model: () => import('./locales/en-US/model.json'), + monitoring: () => import('./locales/en-US/monitoring.json'), + notification: () => import('./locales/en-US/notification.json'), + permission: () => import('./locales/en-US/permission.json'), + session: () => import('./locales/en-US/session.json'), + skill: () => import('./locales/en-US/skill.json'), + task: () => import('./locales/en-US/task.json'), + tool: () => import('./locales/en-US/tool.json'), + update: () => import('./locales/en-US/update.json'), + workflow: () => import('./locales/en-US/workflow.json'), + workspace: () => import('./locales/en-US/workspace.json'), + }, + 'zh-CN': { + agent: () => import('./locales/zh-CN/agent.json'), + channel: () => import('./locales/zh-CN/channel.json'), + config: () => import('./locales/zh-CN/config.json'), + device: () => import('./locales/zh-CN/device.json'), + flockspro: () => import('./locales/zh-CN/flockspro.json'), + mcp: () => import('./locales/zh-CN/mcp.json'), + model: () => import('./locales/zh-CN/model.json'), + monitoring: () => import('./locales/zh-CN/monitoring.json'), + notification: () => import('./locales/zh-CN/notification.json'), + permission: () => import('./locales/zh-CN/permission.json'), + session: () => import('./locales/zh-CN/session.json'), + skill: () => import('./locales/zh-CN/skill.json'), + task: () => import('./locales/zh-CN/task.json'), + tool: () => import('./locales/zh-CN/tool.json'), + update: () => import('./locales/zh-CN/update.json'), + workflow: () => import('./locales/zh-CN/workflow.json'), + workspace: () => import('./locales/zh-CN/workspace.json'), + }, +}; + +const loadingLocaleBundles = new Map>(); +let registeredI18n: I18nInstance | null = null; + +export function registerI18nInstance(i18n: I18nInstance) { + registeredI18n = i18n; +} + +function normalizeLanguage(language: string): SupportedLanguage { + return language.toLowerCase().startsWith('zh') ? 'zh-CN' : 'en-US'; +} + +function getLazyLoader(language: string, namespace: string): LocaleLoader | undefined { + return lazyLocaleLoaders[normalizeLanguage(language)][namespace as LazyNamespace]; +} + +async function loadLazyLocaleBundle( + i18n: I18nInstance, + language: string, + namespace: string, +) { + const normalizedLanguage = normalizeLanguage(language); + if (i18n.hasResourceBundle(normalizedLanguage, namespace)) return; + + const loader = getLazyLoader(normalizedLanguage, namespace); + if (!loader) return; + + const bundleKey = `${normalizedLanguage}:${namespace}`; + const existingLoad = loadingLocaleBundles.get(bundleKey); + if (existingLoad) { + await existingLoad; + return; + } + + const loadPromise = loader() + .then((module) => { + i18n.addResourceBundle(normalizedLanguage, namespace, module.default, true, true); + }) + .finally(() => { + loadingLocaleBundles.delete(bundleKey); + }); + + loadingLocaleBundles.set(bundleKey, loadPromise); + await loadPromise; +} + +export function readLazyLocale(language: string, namespace: string, callback: ReadCallback) { + const loader = getLazyLoader(language, namespace); + if (!loader) { + callback(null, {}); + return; + } + + loader() + .then((module) => callback(null, module.default)) + .catch((error: unknown) => { + callback(error instanceof Error ? error : new Error(String(error)), false); + }); +} + +export async function preloadI18nNamespaces(namespaces: readonly string[]): Promise { + const uniqueNamespaces = [...new Set(namespaces)].filter(Boolean); + if (uniqueNamespaces.length === 0) return; + + const i18n = registeredI18n; + if (!i18n) { + if (import.meta.env.MODE === 'test') return; + throw new Error('i18n instance is unavailable'); + } + + const currentLanguage = normalizeLanguage(i18n.resolvedLanguage || i18n.language || 'en-US'); + const languages = currentLanguage === 'en-US' ? ['en-US'] : [currentLanguage, 'en-US']; + + await Promise.all( + languages.flatMap((language) => ( + uniqueNamespaces.map((namespace) => loadLazyLocaleBundle(i18n, language, namespace)) + )), + ); +} diff --git a/webui/src/pages/DeviceIntegration/index.test.tsx b/webui/src/pages/DeviceIntegration/index.test.tsx index 4d4a93474..68d70730c 100644 --- a/webui/src/pages/DeviceIntegration/index.test.tsx +++ b/webui/src/pages/DeviceIntegration/index.test.tsx @@ -415,7 +415,7 @@ describe('DeviceIntegrationPage', () => { await openManualAddWizard(user); - expect(screen.getByText('API 接入')).toBeInTheDocument(); + expect(await screen.findByText('API 接入')).toBeInTheDocument(); expect(screen.getByText('浏览器接入')).toBeInTheDocument(); expect(screen.getByText('TDP 接入')).toBeInTheDocument(); expect(screen.getByText('OneSEC 接入')).toBeInTheDocument(); diff --git a/webui/src/pages/DeviceIntegration/index.tsx b/webui/src/pages/DeviceIntegration/index.tsx index 1538bdc89..121c388ec 100644 --- a/webui/src/pages/DeviceIntegration/index.tsx +++ b/webui/src/pages/DeviceIntegration/index.tsx @@ -1,4 +1,4 @@ -import { useState, useEffect, useCallback, useMemo, useRef } from 'react'; +import { lazy, Suspense, useState, useEffect, useCallback, useMemo, useRef } from 'react'; import { useTranslation } from 'react-i18next'; import { Shield, CheckCircle, XCircle, AlertTriangle, RefreshCw, @@ -9,7 +9,6 @@ import { import PageHeader from '@/components/common/PageHeader'; import LoadingSpinner from '@/components/common/LoadingSpinner'; import { useToast } from '@/components/common/Toast'; -import SessionChat from '@/components/common/SessionChat'; import GuideInfoIcon from '@/components/common/GuideInfoIcon'; import { useRexComposerControls } from '@/components/common/useRexComposerControls'; import { useSessionChat, type CreateAndSendOptions } from '@/hooks/useSessionChat'; @@ -29,6 +28,15 @@ import { buildCustomDeviceModeRoutingPrompt } from './customDevice'; const DEFAULT_GROUP_ID = 'default-room'; const DEVICE_DRAWER_WIDTH = 560; const DEVICE_DRAWER_WIDTH_CSS = `${DEVICE_DRAWER_WIDTH}px`; +const LazySessionChat = lazy(() => import('@/components/common/SessionChat')); + +function SessionChatFallback() { + return ( +
+ +
+ ); +} /** Pull the backend's human-readable error detail (e.g. "机房名称已存在") * out of an axios error, falling back to a generic message. */ @@ -697,171 +705,172 @@ function DeviceAddRexPanel({ return (
- void detectLatestDraft(true)} - welcomeContent={ -
-
- {!showBuiltInTemplates ? ( - <> -
-
- + }> + void detectLatestDraft(true)} + welcomeContent={ +
+
+ {!showBuiltInTemplates ? ( + <> +
+
+ +
+

{t('wizard.guide.title')}

+

+ {t('wizard.guide.subtitle')} +

-

{t('wizard.guide.title')}

-

- {t('wizard.guide.subtitle')} -

-
-
- - startGuidedPrompt(t('wizard.guide.prompts.api'))} - /> - startGuidedPrompt(t('wizard.guide.prompts.browser'))} - /> - - - - handleCaseTemplate(['tdp'], t('wizard.guide.prompts.tdp'))} - /> - handleCaseTemplate(['onesec', 'one sec'], t('wizard.guide.prompts.onesec'))} - /> - setShowBuiltInTemplates(true)} - /> - -
- - ) : ( - <> -
- -

{t('wizard.supportedList.title')}

-

{t('wizard.supportedList.subtitle')}

-
+
+ + startGuidedPrompt(t('wizard.guide.prompts.api'))} + /> + startGuidedPrompt(t('wizard.guide.prompts.browser'))} + /> + -
- {vendorGroups.map(({ vendor, templates: vendorTemplates }) => { - const expanded = expandedVendors.has(vendor.id); - const vendorName = i18n.language.startsWith('zh') ? vendor.nameCn : vendor.nameEn; - const integratedCount = vendorTemplates.reduce( - (sum, template) => sum + (instanceCounts[template.storage_key] ?? 0), - 0, - ); - return ( -
- - {expanded && ( -
- {vendorTemplates.map((tpl) => { - const count = instanceCounts[tpl.storage_key] ?? 0; - const action = templateAction(tpl); - const installing = installingTemplateKey === tpl.storage_key; - const templateMeta = tpl.version ? formatTemplateVersion(tpl.version) : tpl.storage_key; - const stateBadge = tpl.installed - ? t('wizard.installState.installed') - : tpl.state === 'updateAvailable' - ? t('wizard.installState.updateAvailable') - : tpl.state === 'broken' - ? t('wizard.installState.brokenShort') - : t('wizard.installState.available'); - return ( -
+ + ) : ( + <> +
+ +

{t('wizard.supportedList.title')}

+

{t('wizard.supportedList.subtitle')}

+
+ +
+ {vendorGroups.map(({ vendor, templates: vendorTemplates }) => { + const expanded = expandedVendors.has(vendor.id); + const vendorName = i18n.language.startsWith('zh') ? vendor.nameCn : vendor.nameEn; + const integratedCount = vendorTemplates.reduce( + (sum, template) => sum + (instanceCounts[template.storage_key] ?? 0), + 0, + ); + return ( +
+ + {expanded && ( +
+ {vendorTemplates.map((tpl) => { + const count = instanceCounts[tpl.storage_key] ?? 0; + const action = templateAction(tpl); + const installing = installingTemplateKey === tpl.storage_key; + const templateMeta = tpl.version ? formatTemplateVersion(tpl.version) : tpl.storage_key; + const stateBadge = tpl.installed + ? t('wizard.installState.installed') + : tpl.state === 'updateAvailable' + ? t('wizard.installState.updateAvailable') + : tpl.state === 'broken' + ? t('wizard.installState.brokenShort') + : t('wizard.installState.available'); + return ( + - ); - })} -
- )} -
- ); - })} -
- - )} + + {count > 0 && {t('wizard.instanceCount', { count })}} + {installing ? t(action === 'update' ? 'wizard.installState.updating' : 'wizard.installState.installing') : stateBadge} + + {installing + ? + : ( + + )} + + ); + })} +
+ )} + + ); + })} +
+ + )}
} @@ -872,6 +881,7 @@ function DeviceAddRexPanel({ model: modelOverride === undefined ? rexComposerControls.rexModel : modelOverride, }) : undefined} /> + {detectedAction && (
diff --git a/webui/src/pages/Session/index.test.tsx b/webui/src/pages/Session/index.test.tsx index 5940ab6ce..321a0b315 100644 --- a/webui/src/pages/Session/index.test.tsx +++ b/webui/src/pages/Session/index.test.tsx @@ -1,9 +1,10 @@ import React from 'react'; import { beforeEach, describe, expect, it, vi } from 'vitest'; -import { render, screen, waitFor } from '@testing-library/react'; +import { act, render, screen, waitFor } from '@testing-library/react'; import userEvent from '@testing-library/user-event'; import { MemoryRouter, useNavigate } from 'react-router-dom'; -import SessionPage from './index'; +import { __resetChatModelResourcesForTesting } from '@/hooks/useChatModelResources'; +import SessionPage, { getVisibleSessionGroupItems, groupSessionsByDate } from './index'; const { client, @@ -107,6 +108,7 @@ vi.mock('@/components/common/SessionChat', () => ({ initialMessage, initialDisplayText, onCreateAndSend, + onSSEEvent, agentName, model, display, @@ -137,6 +139,7 @@ vi.mock('@/components/common/SessionChat', () => ({ modelOverride?: unknown, options?: { displayText?: string }, ) => Promise | unknown; + onSSEEvent?: (event: { type: string; properties?: Record }) => void; }) { const [input, setInput] = React.useState(''); return ( @@ -162,6 +165,15 @@ vi.mock('@/components/common/SessionChat', () => ({ +
); }, @@ -248,9 +260,65 @@ function renderSessionPage( ); } +describe('session sidebar grouping helpers', () => { + it('groups sessions by updated date and applies search filtering', () => { + const now = new Date(2026, 6, 9, 12, 0, 0); + const makeSession = (id: string, title: string, updated: number) => ({ + ...session, + id, + title, + time: { ...session.time, updated }, + }); + + const groups = groupSessionsByDate([ + makeSession('today', 'Today Investigation', new Date(2026, 6, 9, 9).getTime()), + makeSession('yesterday', 'Yesterday Work', new Date(2026, 6, 8, 9).getTime()), + makeSession('earlier', 'Old Investigation', new Date(2026, 5, 1, 9).getTime()), + ], 'investigation', now); + + expect(groups.map((group) => [group.key, group.items.map((item) => item.id)])).toEqual([ + ['today', ['today']], + ['earlier', ['earlier']], + ]); + }); + + it('limits collapsed older groups and reports hidden count', () => { + const group = { + key: 'thisWeek' as const, + labelKey: 'groupThisWeek', + items: Array.from({ length: 7 }, (_, index) => ({ + ...session, + id: `session-${index}`, + title: `Session ${index}`, + })), + }; + + expect(getVisibleSessionGroupItems({ + group, + expanded: false, + searching: false, + })).toMatchObject({ + visibleItems: group.items.slice(0, 5), + hiddenCount: 2, + limit: 5, + }); + + expect(getVisibleSessionGroupItems({ + group, + expanded: false, + searching: true, + })).toMatchObject({ + visibleItems: group.items, + hiddenCount: 0, + limit: Infinity, + }); + }); +}); + describe('SessionPage session actions menu', () => { beforeEach(() => { vi.clearAllMocks(); + __resetChatModelResourcesForTesting(); localStorage.clear(); sessionStorage.clear(); @@ -336,6 +404,34 @@ describe('SessionPage session actions menu', () => { expect(sessionApi.update).toHaveBeenCalledTimes(1); }); + it('coalesces bursty session.updated events into one sidebar refetch', () => { + vi.useFakeTimers(); + try { + renderSessionPage(); + + const emitSessionUpdated = screen.getByRole('button', { name: 'mock-session-updated' }); + act(() => { + emitSessionUpdated.click(); + emitSessionUpdated.click(); + }); + + expect(updateSessionTitle).toHaveBeenCalledTimes(2); + expect(refetchSessions).not.toHaveBeenCalled(); + + act(() => { + vi.advanceTimersByTime(499); + }); + expect(refetchSessions).not.toHaveBeenCalled(); + + act(() => { + vi.advanceTimersByTime(1); + }); + expect(refetchSessions).toHaveBeenCalledTimes(1); + } finally { + vi.useRealTimers(); + } + }); + it('downloads session data as CLI-compatible JSON', async () => { const user = userEvent.setup(); const OriginalBlob = Blob; diff --git a/webui/src/pages/Session/index.tsx b/webui/src/pages/Session/index.tsx index ee5eb702d..b801819f1 100644 --- a/webui/src/pages/Session/index.tsx +++ b/webui/src/pages/Session/index.tsx @@ -1,4 +1,4 @@ -import { useState, useEffect, useMemo, useCallback, useRef } from 'react'; +import { memo, useState, useEffect, useMemo, useCallback, useRef, type RefObject } from 'react'; import { MessageSquare, Plus, Trash2, ChevronDown, Sparkles, Shield, Search, AlertTriangle, @@ -24,8 +24,8 @@ import type { Agent } from '@/api/agent'; import { useSessions } from '@/hooks/useSessions'; import { useAgents } from '@/hooks/useAgents'; import { useProviders } from '@/hooks/useProviders'; +import { useEnabledChatModelDefinitions, useResolvedDefaultModel } from '@/hooks/useChatModelResources'; import client from '@/api/client'; -import { defaultModelAPI, modelV2API } from '@/api/provider'; import { useDefaultModelVision } from '@/hooks/useDefaultModelVision'; import { buildPromptParts, type ImagePartData } from '@/utils/imageUpload'; import { getAgentDisplayDescription, getAgentDisplayName, isAgentUsableInChat } from '@/utils/agentDisplay'; @@ -45,6 +45,7 @@ const LAST_SELECTED_SESSION_STORAGE_KEY = 'flocks:last-selected-session'; const SESSION_PAGE_VISITED_STORAGE_KEY = 'flocks:sessions:visited'; const SOC_WORKSPACE_COMPONENT_ID = 'soc-workspace'; const INSTALLED_HUB_STATES = new Set(['installed', 'localOnly', 'updateAvailable']); +const SESSION_UPDATE_REFETCH_DEBOUNCE_MS = 500; type AgentSourceFilter = 'all' | 'builtin' | 'custom'; type ChatModelOption = { key: string; @@ -122,6 +123,219 @@ function makeModelKey(providerID: string, modelID: string): string { return `${providerID}::${modelID}`; } +export type SessionGroupKey = 'today' | 'yesterday' | 'thisWeek' | 'lastWeek' | 'earlier'; + +export interface SessionGroup { + key: SessionGroupKey; + labelKey: string; + items: Session[]; +} + +const SESSION_GROUP_DEFAULT_LIMIT: Record = { + today: Infinity, + yesterday: Infinity, + thisWeek: 5, + lastWeek: 5, + earlier: 5, +}; + +const SESSION_GROUP_DEFS: Array> = [ + { key: 'today', labelKey: 'groupToday' }, + { key: 'yesterday', labelKey: 'groupYesterday' }, + { key: 'thisWeek', labelKey: 'groupThisWeek' }, + { key: 'lastWeek', labelKey: 'groupLastWeek' }, + { key: 'earlier', labelKey: 'groupEarlier' }, +]; + +export function groupSessionsByDate( + sessions: Session[], + searchQuery: string, + now = new Date(), +): SessionGroup[] { + const todayStart = new Date(now.getFullYear(), now.getMonth(), now.getDate()).getTime(); + const yesterdayStart = todayStart - 86400000; + const dayOfWeek = now.getDay() === 0 ? 7 : now.getDay(); + const thisWeekStart = todayStart - (dayOfWeek - 1) * 86400000; + const lastWeekStart = thisWeekStart - 7 * 86400000; + + const q = searchQuery.toLowerCase().trim(); + const filtered = q ? sessions.filter(s => s.title.toLowerCase().includes(q)) : sessions; + const buckets: SessionGroup[] = SESSION_GROUP_DEFS.map((group) => ({ + ...group, + items: [], + })); + + for (const session of filtered) { + const ts = session.time?.updated ?? 0; + if (ts >= todayStart) buckets[0].items.push(session); + else if (ts >= yesterdayStart) buckets[1].items.push(session); + else if (ts >= thisWeekStart) buckets[2].items.push(session); + else if (ts >= lastWeekStart) buckets[3].items.push(session); + else buckets[4].items.push(session); + } + + return buckets.filter(group => group.items.length > 0); +} + +export function getVisibleSessionGroupItems({ + group, + expanded, + searching, +}: { + group: SessionGroup; + expanded: boolean; + searching: boolean; +}): { visibleItems: Session[]; hiddenCount: number; limit: number } { + const limit = searching ? Infinity : SESSION_GROUP_DEFAULT_LIMIT[group.key]; + const visibleItems = (searching || expanded || group.items.length <= limit) + ? group.items + : group.items.slice(0, limit); + return { + visibleItems, + hiddenCount: group.items.length - visibleItems.length, + limit, + }; +} + +interface SessionSidebarItemProps { + session: Session; + selected: boolean; + selectMode: boolean; + checked: boolean; + menuOpen: boolean; + renaming: boolean; + renameValue: string; + renameSubmitting: boolean; + t: (key: string, options?: Record) => string; + renameInputRef: RefObject; + onSelect: (sessionId: string) => void; + onToggleCheck: (sessionId: string) => void; + onRenameValueChange: (value: string) => void; + onSubmitRename: (sessionId: string) => void | Promise; + onCancelRename: () => void; + onToggleMenu: (sessionId: string, trigger: HTMLElement) => void; +} + +function SessionSidebarItemInner({ + session, + selected, + selectMode, + checked, + menuOpen, + renaming, + renameValue, + renameSubmitting, + t, + renameInputRef, + onSelect, + onToggleCheck, + onRenameValueChange, + onSubmitRename, + onCancelRename, + onToggleMenu, +}: SessionSidebarItemProps) { + return ( +
onSelect(session.id)} + className={`group relative mx-2 mb-1 px-3 py-2.5 rounded-xl border cursor-pointer transition-all duration-150 ${ + !selectMode && selected + ? 'bg-gray-100 border-gray-300 shadow-sm dark:border-zinc-700 dark:bg-zinc-900 dark:shadow-none' + : selectMode && checked + ? 'bg-blue-50 border-blue-200 dark:border-blue-500/40 dark:bg-blue-950/30' + : 'border-gray-100 hover:border-gray-200 hover:bg-gray-50 hover:shadow-sm dark:border-transparent dark:hover:border-zinc-800 dark:hover:bg-zinc-900 dark:hover:shadow-none' + }`} + > +
+ {selectMode && ( + onToggleCheck(session.id)} + onClick={(e) => e.stopPropagation()} + className="flex-shrink-0 w-3.5 h-3.5 accent-blue-500 cursor-pointer rounded" + /> + )} + {session.category === 'workflow' && ( + + + + )} + {session.category === 'entity-config' && ( + + + + )} + {renaming ? ( + onRenameValueChange(e.target.value)} + onClick={(e) => e.stopPropagation()} + onBlur={() => void onSubmitRename(session.id)} + onKeyDown={(e) => { + if (e.key === 'Enter') { e.preventDefault(); void onSubmitRename(session.id); } + if (e.key === 'Escape') { e.preventDefault(); onCancelRename(); } + }} + placeholder={t('renamePlaceholder')} + disabled={renameSubmitting} + className="w-full min-w-0 rounded border border-blue-300 bg-white px-1.5 py-0.5 text-sm text-gray-900 outline-none focus:border-blue-400 dark:border-blue-500/50 dark:bg-zinc-950 dark:text-zinc-100" + aria-label={t('rename')} + data-session-rename-input + /> + ) : ( +

+ {session.title} + {session.isShared && ( + + {t('sharedTag')} + + )} +

+ )} +
+ {session.time?.updated && !renaming && ( +

+ {formatSessionDate(session.time.updated)} +

+ )} + {!selectMode && ( +
+ +
+ )} +
+ ); +} + +const SessionSidebarItem = memo(SessionSidebarItemInner, (prev, next) => ( + prev.session.id === next.session.id && + prev.session.title === next.session.title && + prev.session.category === next.session.category && + prev.session.isShared === next.session.isShared && + prev.session.time?.updated === next.session.time?.updated && + prev.selected === next.selected && + prev.selectMode === next.selectMode && + prev.checked === next.checked && + prev.menuOpen === next.menuOpen && + prev.renaming === next.renaming && + prev.renameValue === next.renameValue && + prev.renameSubmitting === next.renameSubmitting && + prev.t === next.t +)); + export default function SessionPage() { const { t, i18n } = useTranslation('session'); const location = useLocation(); @@ -133,8 +347,6 @@ export default function SessionPage() { const [showAgentOptions, setShowAgentOptions] = useState(false); const [selectedModelKey, setSelectedModelKey] = useState(null); const [showModelOptions, setShowModelOptions] = useState(false); - const [enabledModelDefinitions, setEnabledModelDefinitions] = useState([]); - const [loadingEnabledModels, setLoadingEnabledModels] = useState(true); const [sseStatus, setSseStatus] = useState('disconnected'); const [creating, setCreating] = useState(false); const [installingSocWorkspace, setInstallingSocWorkspace] = useState(false); @@ -157,6 +369,7 @@ export default function SessionPage() { const [selectorTooltip, setSelectorTooltip] = useState(null); const renameInputRef = useRef(null); const renameSubmitInFlightRef = useRef(false); + const sessionUpdateRefetchTimerRef = useRef(null); const toast = useToast(); const { @@ -173,6 +386,10 @@ export default function SessionPage() { } = useSessions(searchQuery); const { agents, loading: loadingAgents } = useAgents(); const { providers, loading: loadingProviders } = useProviders(); + const { + data: enabledModelDefinitions, + loading: loadingEnabledModels, + } = useEnabledChatModelDefinitions(); const primaryAgents = useMemo(() => agents.filter((a) => a.mode === 'primary' && isAgentUsableInChat(a)), [agents]); const subAgents = useMemo( () => agents.filter((a) => a.mode !== 'primary' && isAgentUsableInChat(a)), @@ -256,6 +473,16 @@ export default function SessionPage() { .filter((group) => group.models.length > 0) .sort((a, b) => a.providerName.localeCompare(b.providerName)); }, [chatModelOptions, providers]); + const listedSelectedSession = useMemo( + () => sessions.find(s => s.id === selectedSessionId) ?? null, + [sessions, selectedSessionId], + ); + const selectedSession = listedSelectedSession + ?? (selectedSessionFallback?.id === selectedSessionId ? selectedSessionFallback : null); + const pinnedModelKey = selectedSession?.model_pinned && selectedSession.provider && selectedSession.model + ? makeModelKey(selectedSession.provider, selectedSession.model) + : null; + const hasPinnedModelOption = !!pinnedModelKey && chatModelOptions.some((option) => option.key === pinnedModelKey); const selectedModelOption = useMemo( () => chatModelOptions.find((option) => option.key === selectedModelKey) ?? (selectedModelKey ? null : chatModelOptions[0] ?? null), [chatModelOptions, selectedModelKey], @@ -263,22 +490,11 @@ export default function SessionPage() { const selectedPromptModel = selectedModelOption ? { providerID: selectedModelOption.providerID, modelID: selectedModelOption.modelID } : null; + const { + data: resolvedDefaultModel, + initialized: resolvedDefaultModelInitialized, + } = useResolvedDefaultModel(chatModelOptions.length > 0 && !hasPinnedModelOption); const effectiveSupportsVision = selectedModelOption?.supportsVision ?? supportsVision; - const listedSelectedSession = useMemo( - () => sessions.find(s => s.id === selectedSessionId) ?? null, - [sessions, selectedSessionId], - ); - const selectedSession = listedSelectedSession - ?? (selectedSessionFallback?.id === selectedSessionId ? selectedSessionFallback : null); - - // 今天/昨天不限制;本周/上周/更早默认只显示 5 条 - const GROUP_DEFAULT_LIMIT: Record = { - today: Infinity, - yesterday: Infinity, - thisWeek: 5, - lastWeek: 5, - earlier: 5, - }; const [expandedGroups, setExpandedGroups] = useState>(new Set()); const toggleGroupExpand = useCallback((key: string) => { @@ -290,55 +506,54 @@ export default function SessionPage() { }, []); const groupedSessions = useMemo(() => { - const now = new Date(); - const todayStart = new Date(now.getFullYear(), now.getMonth(), now.getDate()).getTime(); - const yesterdayStart = todayStart - 86400000; - // Week starts on Monday - const dayOfWeek = now.getDay() === 0 ? 7 : now.getDay(); - const thisWeekStart = todayStart - (dayOfWeek - 1) * 86400000; - const lastWeekStart = thisWeekStart - 7 * 86400000; - - const q = searchQuery.toLowerCase().trim(); - const filtered = q ? sessions.filter(s => s.title.toLowerCase().includes(q)) : sessions; - - const buckets: { key: string; labelKey: string; items: typeof sessions }[] = [ - { key: 'today', labelKey: 'groupToday', items: [] }, - { key: 'yesterday', labelKey: 'groupYesterday', items: [] }, - { key: 'thisWeek', labelKey: 'groupThisWeek', items: [] }, - { key: 'lastWeek', labelKey: 'groupLastWeek', items: [] }, - { key: 'earlier', labelKey: 'groupEarlier', items: [] }, - ]; - - for (const s of filtered) { - const ts = s.time?.updated ?? 0; - if (ts >= todayStart) buckets[0].items.push(s); - else if (ts >= yesterdayStart) buckets[1].items.push(s); - else if (ts >= thisWeekStart) buckets[2].items.push(s); - else if (ts >= lastWeekStart) buckets[3].items.push(s); - else buckets[4].items.push(s); - } - - return buckets.filter(b => b.items.length > 0); + return groupSessionsByDate(sessions, searchQuery); }, [sessions, searchQuery]); + const visibleSessionGroups = useMemo(() => { + const searching = searchQuery.trim().length > 0; + return groupedSessions.map((group) => ({ + ...group, + ...getVisibleSessionGroupItems({ + group, + expanded: expandedGroups.has(group.key), + searching, + }), + expanded: expandedGroups.has(group.key), + searching, + })); + }, [expandedGroups, groupedSessions, searchQuery]); // Handle SSE events for session-level updates (title changes, etc.) const handleChatError = useCallback((msg: string) => { toast.error(t('chat.error', 'Error'), msg); }, [toast, t]); + const scheduleSessionListRefetch = useCallback(() => { + if (sessionUpdateRefetchTimerRef.current !== null) return; + sessionUpdateRefetchTimerRef.current = window.setTimeout(() => { + sessionUpdateRefetchTimerRef.current = null; + void refetchSessions(); + }, SESSION_UPDATE_REFETCH_DEBOUNCE_MS); + }, [refetchSessions]); + + useEffect(() => () => { + if (sessionUpdateRefetchTimerRef.current !== null) { + window.clearTimeout(sessionUpdateRefetchTimerRef.current); + sessionUpdateRefetchTimerRef.current = null; + } + }, []); + const handleSSEEvent = useCallback((event: SSEChatEvent) => { if (event.type === 'session.updated' && event.properties?.id) { if (event.properties?.title) { // Instant local title update so the sidebar reflects the change immediately. updateSessionTitle(event.properties.id, event.properties.title); } - // Always do a silent background sync: session.updated also changes - // time.updated (affects ordering) and potentially other metadata. - // refetchSessions() is safe here — it never shows a loading spinner - // after the initial load (see initializedRef in useSessions). - refetchSessions(); + // Session/title updates can arrive in bursts when several sessions or + // background tasks finish together. Coalesce the full sidebar refresh so + // those bursts don't turn into a request/re-render storm. + scheduleSessionListRefetch(); } - }, [updateSessionTitle, refetchSessions]); + }, [scheduleSessionListRefetch, updateSessionTitle]); // Keep the selected session in sync with URL query params (e.g. onboarding // or other in-app navigation to `/sessions?session=...`). Clear the params @@ -416,24 +631,6 @@ export default function SessionPage() { }; }, [listedSelectedSession, loadingSessions, selectedSessionId]); - useEffect(() => { - let cancelled = false; - setLoadingEnabledModels(true); - modelV2API.listDefinitions({ enabled_only: true }) - .then((response) => { - if (!cancelled) setEnabledModelDefinitions(response.data.models ?? []); - }) - .catch(() => { - if (!cancelled) setEnabledModelDefinitions([]); - }) - .finally(() => { - if (!cancelled) setLoadingEnabledModels(false); - }); - return () => { - cancelled = true; - }; - }, []); - // Close agent dropdown on outside click useEffect(() => { if (!showAgentOptions) return; @@ -461,35 +658,24 @@ export default function SessionPage() { return; } - const pinnedKey = selectedSession?.model_pinned && selectedSession.provider && selectedSession.model - ? makeModelKey(selectedSession.provider, selectedSession.model) - : null; - if (pinnedKey && chatModelOptions.some((option) => option.key === pinnedKey)) { - setSelectedModelKey(pinnedKey); + if (hasPinnedModelOption && pinnedModelKey) { + setSelectedModelKey(pinnedModelKey); return; } - let cancelled = false; setSelectedModelKey(null); - defaultModelAPI.getResolved() - .then((response) => { - if (cancelled) return; - const { provider_id: providerID, model_id: modelID } = response.data; - const defaultKey = makeModelKey(providerID, modelID); - const fallbackKey = chatModelOptions[0]?.key ?? null; - setSelectedModelKey(chatModelOptions.some((option) => option.key === defaultKey) ? defaultKey : fallbackKey); - }) - .catch(() => { - if (!cancelled) setSelectedModelKey(chatModelOptions[0]?.key ?? null); - }); - return () => { - cancelled = true; - }; + if (!resolvedDefaultModelInitialized) return; + const defaultKey = resolvedDefaultModel + ? makeModelKey(resolvedDefaultModel.providerID, resolvedDefaultModel.modelID) + : null; + const fallbackKey = chatModelOptions[0]?.key ?? null; + setSelectedModelKey(defaultKey && chatModelOptions.some((option) => option.key === defaultKey) ? defaultKey : fallbackKey); }, [ chatModelOptions, - selectedSession?.model, - selectedSession?.model_pinned, - selectedSession?.provider, + hasPinnedModelOption, + pinnedModelKey, + resolvedDefaultModel, + resolvedDefaultModelInitialized, selectedSessionId, ]); @@ -773,6 +959,24 @@ export default function SessionPage() { return next; }); }, []); + const handleSelectSessionRow = useCallback((sessionId: string) => { + if (selectMode) { + handleToggleCheck(sessionId); + } else { + setSelectedSessionId(sessionId); + } + }, [handleToggleCheck, selectMode]); + const handleToggleSessionMenu = useCallback((sessionId: string, trigger: HTMLElement) => { + setOpenMenuSessionId((current) => { + if (current === sessionId) { + setMenuAnchor(null); + return null; + } + const rect = trigger.getBoundingClientRect(); + setMenuAnchor({ top: rect.bottom + 4, right: window.innerWidth - rect.right }); + return sessionId; + }); + }, []); const handleSelectAll = useCallback(() => { if (checkedIds.size === sessions.length) { @@ -866,118 +1070,34 @@ export default function SessionPage() {

{t('noResults', 'No conversations found')}

) : ( - groupedSessions.map(({ key, labelKey, items }) => { - const isSearching = searchQuery.trim().length > 0; - const limit = isSearching ? Infinity : (GROUP_DEFAULT_LIMIT[key] ?? 5); - const isExpanded = expandedGroups.has(key); - const visibleItems = (isSearching || isExpanded || items.length <= limit) - ? items - : items.slice(0, limit); - const hiddenCount = items.length - visibleItems.length; - - return ( + visibleSessionGroups.map(({ key, labelKey, items, visibleItems, hiddenCount, limit, expanded, searching }) => (
{t(labelKey, labelKey)}
{visibleItems.map((session) => ( -
selectMode ? handleToggleCheck(session.id) : setSelectedSessionId(session.id)} - className={`group relative mx-2 mb-1 px-3 py-2.5 rounded-xl border cursor-pointer transition-all duration-150 ${ - !selectMode && selectedSessionId === session.id - ? 'bg-gray-100 border-gray-300 shadow-sm dark:border-zinc-700 dark:bg-zinc-900 dark:shadow-none' - : selectMode && checkedIds.has(session.id) - ? 'bg-blue-50 border-blue-200 dark:border-blue-500/40 dark:bg-blue-950/30' - : 'border-gray-100 hover:border-gray-200 hover:bg-gray-50 hover:shadow-sm dark:border-transparent dark:hover:border-zinc-800 dark:hover:bg-zinc-900 dark:hover:shadow-none' - }`} - > - {/* Title row */} -
- {selectMode && ( - handleToggleCheck(session.id)} - onClick={(e) => e.stopPropagation()} - className="flex-shrink-0 w-3.5 h-3.5 accent-blue-500 cursor-pointer rounded" - /> - )} - {session.category === 'workflow' && ( - - - - )} - {session.category === 'entity-config' && ( - - - - )} - {renamingSessionId === session.id ? ( - setRenameValue(e.target.value)} - onClick={(e) => e.stopPropagation()} - onBlur={() => void handleSubmitRename(session.id)} - onKeyDown={(e) => { - if (e.key === 'Enter') { e.preventDefault(); void handleSubmitRename(session.id); } - if (e.key === 'Escape') { e.preventDefault(); handleCancelRename(); } - }} - placeholder={t('renamePlaceholder')} - disabled={renameSubmitting} - className="w-full min-w-0 rounded border border-blue-300 bg-white px-1.5 py-0.5 text-sm text-gray-900 outline-none focus:border-blue-400 dark:border-blue-500/50 dark:bg-zinc-950 dark:text-zinc-100" - aria-label={t('rename')} - data-session-rename-input - /> - ) : ( -

- {session.title} - {session.isShared && ( - - {t('sharedTag')} - - )} -

- )} -
- {/* Timestamp row */} - {session.time?.updated && renamingSessionId !== session.id && ( -

- {formatSessionDate(session.time.updated)} -

- )} - - {/* Three-dot menu trigger */} - {!selectMode && ( -
- -
- )} -
+ session={session} + selected={selectedSessionId === session.id} + selectMode={selectMode} + checked={checkedIds.has(session.id)} + menuOpen={openMenuSessionId === session.id} + renaming={renamingSessionId === session.id} + renameValue={renameValue} + renameSubmitting={renameSubmitting} + t={t} + renameInputRef={renameInputRef} + onSelect={handleSelectSessionRow} + onToggleCheck={handleToggleCheck} + onRenameValueChange={setRenameValue} + onSubmitRename={handleSubmitRename} + onCancelRename={handleCancelRename} + onToggleMenu={handleToggleSessionMenu} + /> ))} {/* 展开/收起按钮 */} - {!isSearching && hiddenCount > 0 && ( + {!searching && hiddenCount > 0 && ( )} - {!isSearching && isExpanded && items.length > (GROUP_DEFAULT_LIMIT[key] ?? 5) && ( + {!searching && expanded && items.length > limit && ( )}
- ); - }) + )) )} {hasMoreSessions && (
diff --git a/webui/src/pages/Settings/index.tsx b/webui/src/pages/Settings/index.tsx index a8eca7d1a..3c55c998e 100644 --- a/webui/src/pages/Settings/index.tsx +++ b/webui/src/pages/Settings/index.tsx @@ -1,5 +1,5 @@ import { Suspense, lazy, useContext, useEffect, useMemo, useRef, useState } from 'react'; -import type { ChangeEvent, ReactNode } from 'react'; +import type { ChangeEvent, ComponentType, ReactNode } from 'react'; import { Link, Navigate, useLocation, useNavigate, useParams } from 'react-router-dom'; import { useTranslation } from 'react-i18next'; import { @@ -26,11 +26,24 @@ import { useAuth } from '@/contexts/AuthContext'; import { useProductName } from '@/contexts/ProductNameContext'; import { useToast } from '@/components/common/Toast'; import { flocksproUsersApi } from '@/api/flocksproUsers'; +import { preloadI18nNamespaces } from '@/i18nResources'; + +type LazySettingsModule = { default: ComponentType }; + +function lazySettingsPage( + loader: () => Promise, + namespaces: readonly string[] = [], +) { + return lazy(() => Promise.all([ + loader(), + preloadI18nNamespaces(namespaces), + ]).then(([module]) => module)); +} -const ConfigPage = lazy(() => import('@/pages/Config')); -const SystemLogPage = lazy(() => import('@/pages/SystemLog')); -const FlocksproUpgradePage = lazy(() => import('@/pages/FlocksproUpgrade')); -const AuditLogsPage = lazy(() => import('@/pages/AuditLogs')); +const ConfigPage = lazySettingsPage(() => import('@/pages/Config')); +const SystemLogPage = lazySettingsPage(() => import('@/pages/SystemLog')); +const FlocksproUpgradePage = lazySettingsPage(() => import('@/pages/FlocksproUpgrade'), ['flockspro']); +const AuditLogsPage = lazySettingsPage(() => import('@/pages/AuditLogs'), ['flockspro']); type SettingsSectionId = 'preferences' | 'account' | 'system-logs' | 'audit-logs' | 'flockspro'; diff --git a/webui/src/pages/Skill/SkillSheet.tsx b/webui/src/pages/Skill/SkillSheet.tsx index b11665c36..c1df862c5 100644 --- a/webui/src/pages/Skill/SkillSheet.tsx +++ b/webui/src/pages/Skill/SkillSheet.tsx @@ -7,16 +7,25 @@ * - Rex chat mode (natural language → extract config into form) */ -import { useMemo, useState } from 'react'; +import { lazy, Suspense, useMemo, useState } from 'react'; import { useTranslation } from 'react-i18next'; import { BookOpen, Lock, Pencil, Eye, Save, Loader2, Trash2 } from 'lucide-react'; -import ReactMarkdown from 'react-markdown'; import { skillAPI, Skill } from '@/api/skill'; import { useToast } from '@/components/common/Toast'; import EntitySheet from '@/components/common/EntitySheet'; import { buildGuidedCreateGroups } from '@/components/common/GuidedCreatePanel'; import { useRexComposerControls } from '@/components/common/useRexComposerControls'; +const ReactMarkdown = lazy(() => import('react-markdown')); + +function MarkdownFallback() { + return ( +
+ +
+ ); +} + interface SkillFormData { name: string; description: string; @@ -344,7 +353,9 @@ export default function SkillSheet({ skill, onClose, onSaved, onDeleted }: Skill className="px-4 py-3 bg-gray-50 border border-gray-200 rounded-lg text-sm prose prose-sm max-w-none overflow-y-auto" style={{ minHeight: '320px', maxHeight: '65vh' }} > - {contentForRender || t('sheet.noContent')} + }> + {contentForRender || t('sheet.noContent')} +
)}
diff --git a/webui/src/pages/Task/QueuedSection.tsx b/webui/src/pages/Task/QueuedSection.tsx index 68a33fc31..63ae86694 100644 --- a/webui/src/pages/Task/QueuedSection.tsx +++ b/webui/src/pages/Task/QueuedSection.tsx @@ -1,4 +1,4 @@ -import { useState, useEffect, useCallback, useMemo, useRef } from 'react'; +import { lazy, Suspense, useState, useEffect, useCallback, useMemo, useRef } from 'react'; import { useTranslation } from 'react-i18next'; import { ListTodo, Play, RotateCcw, XCircle, Trash2, @@ -6,7 +6,6 @@ import { } from 'lucide-react'; import LoadingSpinner from '@/components/common/LoadingSpinner'; import EmptyState from '@/components/common/EmptyState'; -import SessionChat from '@/components/common/SessionChat'; import { useToast } from '@/components/common/Toast'; import { useConfirm } from '@/components/common/ConfirmDialog'; import { useTaskExecutions } from '@/hooks/useTasks'; @@ -16,6 +15,15 @@ import { StatusBadge, PriorityBadge, SourceBadge, ModeBadge, ActionButton } from import { formatTime, formatDuration, PAGE_SIZE } from './helpers'; const DETAIL_POLL_INTERVAL_MS = 30000; +const LazySessionChat = lazy(() => import('@/components/common/SessionChat')); + +function SessionChatFallback() { + return ( +
+ +
+ ); +} export default function QueuedSection({ onRefreshGlobal }: { onRefreshGlobal: () => void }) { const { t } = useTranslation('task'); @@ -454,18 +462,20 @@ function QueuedDetailPanel({ task, onClose, onAction, onRefresh }: { {isWorkflowExecution ? ( ) : ( - { - if (event.type === 'task.updated' && event.properties?.executionID === task.id) { - onRefresh?.(); - } - }} - /> + }> + { + if (event.type === 'task.updated' && event.properties?.executionID === task.id) { + onRefresh?.(); + } + }} + /> + )}
diff --git a/webui/src/pages/Workspace/index.tsx b/webui/src/pages/Workspace/index.tsx index 2e8632ead..95d115dc7 100644 --- a/webui/src/pages/Workspace/index.tsx +++ b/webui/src/pages/Workspace/index.tsx @@ -1,18 +1,15 @@ -import { useState, useEffect, useCallback, useMemo, useRef, useReducer } from 'react'; +import { lazy, Suspense, useState, useEffect, useCallback, useMemo, useRef, useReducer } from 'react'; import { FolderOpen, Upload, Download, Trash2, Edit3, Save, X, ChevronRight, ChevronLeft, ChevronDown, ChevronUp, RefreshCw, FolderPlus, Brain, AlertTriangle, Search, ArrowLeft, Maximize2, Code2, Eye, ZoomIn, ZoomOut, } from 'lucide-react'; -import * as pdfjsLib from 'pdfjs-dist'; -import pdfWorkerUrl from 'pdfjs-dist/build/pdf.worker.min.mjs?url'; import { useTranslation } from 'react-i18next'; import PageHeader from '@/components/common/PageHeader'; import LoadingSpinner from '@/components/common/LoadingSpinner'; import { useToast } from '@/components/common/Toast'; import { useConfirm } from '@/components/common/ConfirmDialog'; -import { StreamingMarkdown } from '@/components/common/StreamingMarkdown'; import { workspaceAPI, WorkspaceNode, formatBytes, formatDate, fileIcon, } from '@/api/workspace'; @@ -52,7 +49,8 @@ const IMAGE_MIN_SCALE = 0.5; const IMAGE_MAX_SCALE = 3; const IMAGE_SCALE_STEP = 0.25; -pdfjsLib.GlobalWorkerOptions.workerSrc = pdfWorkerUrl; +const LazyStreamingMarkdown = lazy(() => import('@/components/common/StreamingMarkdown') + .then((module) => ({ default: module.StreamingMarkdown }))); function getViewportWidth(): number { return typeof window === 'undefined' ? PREVIEW_PANEL_MIN_WIDTH * 2 : window.innerWidth; @@ -361,6 +359,7 @@ function PdfPreview({ useEffect(() => { let cancelled = false; + let loadingTask: any = null; setPdfDoc(null); setPageNumber(1); setPageCount(0); @@ -370,26 +369,40 @@ function PdfPreview({ setLoading(true); setError(null); - const loadingTask = pdfjsLib.getDocument({ url: previewUrl, withCredentials: true }); - loadingTask.promise - .then((doc) => { + async function loadPdf() { + try { + const [pdfjsLib, pdfWorkerModule] = await Promise.all([ + import('pdfjs-dist'), + import('pdfjs-dist/build/pdf.worker.min.mjs?url'), + ]); + if (cancelled) { + return; + } + + pdfjsLib.GlobalWorkerOptions.workerSrc = pdfWorkerModule.default; + loadingTask = pdfjsLib.getDocument({ url: previewUrl, withCredentials: true }); + const doc = await loadingTask.promise; if (cancelled) { + doc?.destroy?.(); return; } + setPdfDoc(doc); setPageCount(doc.numPages); setPagesToRender(new Set(Array.from({ length: Math.min(doc.numPages, PDF_RENDER_WINDOW + 1) }, (_, index) => index + 1))); setLoading(false); - }) - .catch((e: any) => { + } catch (e: any) { if (cancelled) return; setError(e?.message ?? 'PDF preview failed'); setLoading(false); - }); + } + } + + loadPdf(); return () => { cancelled = true; - loadingTask.destroy(); + loadingTask?.destroy?.(); }; }, [previewUrl]); @@ -764,7 +777,9 @@ function RenderedPreview({ if (kind === 'markdown') { return (
- +
}> + +
); } diff --git a/webui/src/routes/index.tsx b/webui/src/routes/index.tsx index d26866ada..fbf2012e3 100644 --- a/webui/src/routes/index.tsx +++ b/webui/src/routes/index.tsx @@ -1,4 +1,5 @@ import { Suspense, lazy } from 'react'; +import type { ComponentType, ReactNode } from 'react'; import { Routes as RouterRoutes, Route, Navigate, useLocation, useParams } from 'react-router-dom'; import { useTranslation } from 'react-i18next'; import Layout from '@/components/layout/Layout'; @@ -6,37 +7,50 @@ import RoutePageSkeleton from '@/components/common/RoutePageSkeleton'; import AuthLayout from '@/components/layout/AuthLayout'; import Home from '@/pages/Home'; import { useAuth } from '@/contexts/AuthContext'; +import { preloadI18nNamespaces } from '@/i18nResources'; // All non-Home pages are code-split. Home stays eager because it's the very // first frame after auth and we don't want a Suspense flash on initial paint. // In particular, Session/Agent and the auth screens are kept lazy so heavy // transitive deps (SessionChat ~2.7k LOC + react-markdown + rehype/remark + // highlight.js) are not pulled into the main entry chunk. -const SessionPage = lazy(() => import('@/pages/Session')); -const AgentPage = lazy(() => import('@/pages/Agent')); -const LoginPage = lazy(() => import('@/pages/Login')); -const SetupAdminPage = lazy(() => import('@/pages/SetupAdmin')); -const ForceChangePasswordPage = lazy(() => import('@/pages/ForceChangePassword')); -const WorkflowListPage = lazy(() => import('@/pages/Workflow')); -const WorkflowCreate = lazy(() => import('@/pages/WorkflowCreate')); -const WorkflowEditor = lazy(() => import('@/pages/WorkflowEditor')); -const WorkflowDetail = lazy(() => import('@/pages/WorkflowDetail')); -const TaskPage = lazy(() => import('@/pages/Task')); -const ToolPage = lazy(() => import('@/pages/Tool')); -const HubPage = lazy(() => import('@/pages/Hub')); -const SkillPage = lazy(() => import('@/pages/Skill')); -const ModelPage = lazy(() => import('@/pages/Model')); -const ChannelPage = lazy(() => import('@/pages/Channel')); -const PermissionPage = lazy(() => import('@/pages/Permission')); -const MonitoringPage = lazy(() => import('@/pages/Monitoring')); -const WorkspacePage = lazy(() => import('@/pages/Workspace')); -const DeviceIntegrationPage = lazy(() => import('@/pages/DeviceIntegration')); -const FlocksproUpgradeCallbackPage = lazy(() => import('@/pages/FlocksproUpgrade/Callback')); -const SettingsPage = lazy(() => import('@/pages/Settings')); -const WebUIContractPageHost = lazy(() => import('@/pages/WebUIContractPageHost')); -const WebUIContractWorkspaceHost = lazy(() => import('@/pages/WebUIContractWorkspaceHost')); - -function LazyRoute({ children }: { children: React.ReactNode }) { +type LazyPageModule = { default: ComponentType }; + +function lazyPage( + loader: () => Promise, + namespaces: readonly string[] = [], +) { + return lazy(() => Promise.all([ + loader(), + preloadI18nNamespaces(namespaces), + ]).then(([module]) => module)); +} + +const SessionPage = lazyPage(() => import('@/pages/Session'), ['session']); +const AgentPage = lazyPage(() => import('@/pages/Agent'), ['agent']); +const LoginPage = lazyPage(() => import('@/pages/Login')); +const SetupAdminPage = lazyPage(() => import('@/pages/SetupAdmin')); +const ForceChangePasswordPage = lazyPage(() => import('@/pages/ForceChangePassword')); +const WorkflowListPage = lazyPage(() => import('@/pages/Workflow'), ['workflow']); +const WorkflowCreate = lazyPage(() => import('@/pages/WorkflowCreate'), ['workflow']); +const WorkflowEditor = lazyPage(() => import('@/pages/WorkflowEditor'), ['workflow']); +const WorkflowDetail = lazyPage(() => import('@/pages/WorkflowDetail'), ['workflow']); +const TaskPage = lazyPage(() => import('@/pages/Task'), ['task']); +const ToolPage = lazyPage(() => import('@/pages/Tool'), ['tool']); +const HubPage = lazyPage(() => import('@/pages/Hub')); +const SkillPage = lazyPage(() => import('@/pages/Skill'), ['skill']); +const ModelPage = lazyPage(() => import('@/pages/Model'), ['model']); +const ChannelPage = lazyPage(() => import('@/pages/Channel'), ['channel']); +const PermissionPage = lazyPage(() => import('@/pages/Permission'), ['permission']); +const MonitoringPage = lazyPage(() => import('@/pages/Monitoring'), ['monitoring']); +const WorkspacePage = lazyPage(() => import('@/pages/Workspace'), ['workspace']); +const DeviceIntegrationPage = lazyPage(() => import('@/pages/DeviceIntegration'), ['device']); +const FlocksproUpgradeCallbackPage = lazyPage(() => import('@/pages/FlocksproUpgrade/Callback'), ['flockspro']); +const SettingsPage = lazyPage(() => import('@/pages/Settings')); +const WebUIContractPageHost = lazyPage(() => import('@/pages/WebUIContractPageHost')); +const WebUIContractWorkspaceHost = lazyPage(() => import('@/pages/WebUIContractWorkspaceHost')); + +function LazyRoute({ children }: { children: ReactNode }) { return ( }> {children} From bcc9e72ce4c44a2cb0c78cca73c2eb9ae14d39ae Mon Sep 17 00:00:00 2001 From: duguwanglong Date: Thu, 9 Jul 2026 17:12:13 +0800 Subject: [PATCH 04/49] perf(webui): optimize hub and tool list loading --- flocks/hub/catalog.py | 113 +++++++--- flocks/server/routes/hub.py | 66 +++++- flocks/server/routes/tool.py | 208 +++++++++++++++++- tests/hub/test_hub_catalog.py | 36 +++ tests/server/routes/test_tool_routes.py | 65 +++++- webui/src/api/hub.ts | 32 ++- webui/src/api/tool.ts | 45 ++++ webui/src/hooks/useDebouncedValue.ts | 12 + webui/src/hooks/useTools.test.tsx | 133 ++++++++++- webui/src/hooks/useTools.ts | 139 +++++++++++- webui/src/pages/Hub/index.tsx | 176 ++++++--------- .../pages/Tool/components/LocalTabContent.tsx | 2 +- .../pages/Tool/components/MCPTabContent.tsx | 15 +- webui/src/pages/Tool/index.tsx | 178 +++++++-------- webui/src/types/index.ts | 1 + 15 files changed, 976 insertions(+), 245 deletions(-) create mode 100644 webui/src/hooks/useDebouncedValue.ts diff --git a/flocks/hub/catalog.py b/flocks/hub/catalog.py index bb08cb066..6fc39c8db 100644 --- a/flocks/hub/catalog.py +++ b/flocks/hub/catalog.py @@ -5,6 +5,7 @@ import json import platform import re +import threading from functools import lru_cache from pathlib import Path from typing import Any, Iterable, Optional @@ -31,6 +32,7 @@ _TOOL_TYPE_DIRS = frozenset({"api", "device", "python", "mcp", "generated"}) _SKIP_PLUGIN_DIRS = frozenset({"__pycache__"}) _PATH_SIGNATURE_MISSING = -1 +_CATALOG_ENTRIES_LOCK = threading.Lock() def _read_json(path: Path) -> dict: @@ -594,6 +596,86 @@ def _bundled_tool_roots() -> dict[tuple[PluginType, str], Path]: } +def _installed_plugins_cache_key() -> tuple[tuple[str, int, int], ...]: + signature: list[tuple[str, int, int]] = [_path_signature(local._record_path())] + + for plugin_type in ("skill", "agent", "workflow", "webui", "component"): + for scope in ("global", "project"): + base = local.install_root(plugin_type, scope) + signature.append(_path_signature(base)) + if not base.is_dir(): + continue + for child in sorted(base.iterdir(), key=lambda item: item.name): + if not child.is_dir(): + continue + signature.append(_path_signature(child)) + signature.extend(_plugin_manifest_signature(plugin_type, child)) + + for scope in ("global", "project"): + base = local.install_root("tool", scope) + signature.append(_path_signature(base)) + if not base.is_dir(): + continue + for directory in _iter_tool_plugin_dirs(base): + signature.append(_path_signature(directory)) + signature.extend(_plugin_manifest_signature("tool", directory)) + + return tuple(signature) + + +def _catalog_entries_cache_key() -> tuple[tuple[str, int, int], ...]: + root = get_bundled_hub_root() + return ( + _path_signature(root / "index.json"), + _path_signature(root / "taxonomy.json"), + *_installed_plugins_cache_key(), + *_system_plugin_roots_cache_key(), + *_bundled_tool_roots_cache_key(), + ) + + +@lru_cache(maxsize=8) +def _cached_catalog_entries( + _signature: tuple[tuple[str, int, int], ...], +) -> tuple[HubCatalogEntry, ...]: + records = local.load_installed_records() + inferred_installs = local.infer_local_installs() + entries = [ + _entry_from_index(item, records, inferred_installs) + for item in load_index().plugins + ] + seen: set[tuple[PluginType, str]] = {(entry.type, entry.id) for entry in entries} + for (system_type, system_id), root in _system_plugin_roots().items(): + if (system_type, system_id) in seen: + continue + manifest = _manifest_for_system_root(system_type, system_id, root) + if manifest: + entries.append(_entry_from_system_manifest(manifest, root)) + seen.add((system_type, system_id)) + + # FlocksHub-bundled tool plugins. The runtime ``ToolRegistry`` only + # discovers tools that have been installed into ``/tools``, + # so the catalog needs an explicit pass to surface bundled-but-not- + # yet-installed plugins like ``onesig_v2_5_3_D20250710``. These show + # up as ``state="available"`` until the user installs them, at which + # point the standard ``records``/``inferred_installs`` flow upgrades + # them to ``state="installed"`` (see :func:`_entry_from_bundled_tool`). + for (bundled_type, bundled_id), root in _bundled_tool_roots().items(): + if (bundled_type, bundled_id) in seen: + continue + manifest = _manifest_for_system_root(bundled_type, bundled_id, root) + if manifest: + entries.append(_entry_from_bundled_tool(manifest, root, records, inferred_installs)) + seen.add((bundled_type, bundled_id)) + return tuple(entries) + + +def _catalog_entries_snapshot() -> tuple[HubCatalogEntry, ...]: + signature = _catalog_entries_cache_key() + with _CATALOG_ENTRIES_LOCK: + return _cached_catalog_entries(signature) + + def clear_catalog_caches() -> None: """Clear Hub filesystem discovery caches after installs or explicit refreshes.""" load_index.cache_clear() @@ -601,6 +683,7 @@ def clear_catalog_caches() -> None: _manifest_path_lookup.cache_clear() _cached_system_plugin_roots.cache_clear() _cached_bundled_tool_roots.cache_clear() + _cached_catalog_entries.cache_clear() def system_plugin_root(plugin_type: PluginType, plugin_id: str) -> Optional[Path]: @@ -879,35 +962,7 @@ def list_catalog( risk: Optional[list[str]] = None, q: Optional[str] = None, ) -> list[HubCatalogEntry]: - records = local.load_installed_records() - inferred_installs = local.infer_local_installs() - entries = [ - _entry_from_index(item, records, inferred_installs) - for item in load_index().plugins - ] - seen: set[tuple[PluginType, str]] = {(entry.type, entry.id) for entry in entries} - for (system_type, system_id), root in _system_plugin_roots().items(): - if (system_type, system_id) in seen: - continue - manifest = _manifest_for_system_root(system_type, system_id, root) - if manifest: - entries.append(_entry_from_system_manifest(manifest, root)) - seen.add((system_type, system_id)) - - # FlocksHub-bundled tool plugins. The runtime ``ToolRegistry`` only - # discovers tools that have been installed into ``/tools``, - # so the catalog needs an explicit pass to surface bundled-but-not- - # yet-installed plugins like ``onesig_v2_5_3_D20250710``. These show - # up as ``state="available"`` until the user installs them, at which - # point the standard ``records``/``inferred_installs`` flow upgrades - # them to ``state="installed"`` (see :func:`_entry_from_bundled_tool`). - for (bundled_type, bundled_id), root in _bundled_tool_roots().items(): - if (bundled_type, bundled_id) in seen: - continue - manifest = _manifest_for_system_root(bundled_type, bundled_id, root) - if manifest: - entries.append(_entry_from_bundled_tool(manifest, root, records, inferred_installs)) - seen.add((bundled_type, bundled_id)) + entries = _catalog_entries_snapshot() query = (q or "").strip().lower() def keep(entry: HubCatalogEntry) -> bool: diff --git a/flocks/server/routes/hub.py b/flocks/server/routes/hub.py index ba37eef83..274fcf903 100644 --- a/flocks/server/routes/hub.py +++ b/flocks/server/routes/hub.py @@ -3,13 +3,20 @@ from __future__ import annotations import asyncio -from typing import Optional +from typing import Optional, Union from fastapi import APIRouter, HTTPException, Query from fastapi.responses import StreamingResponse from pydantic import BaseModel, Field -from flocks.hub.catalog import category_counts, clear_catalog_caches, legacy_removed_plugin_message, list_catalog, load_manifest +from flocks.hub.catalog import ( + category_counts, + clear_catalog_caches, + legacy_removed_plugin_message, + list_catalog, + load_manifest, + load_taxonomy, +) from flocks.hub.files import file_tree, read_file_content from flocks.hub.installer import install_plugin, uninstall_plugin, update_plugin from flocks.hub.models import ( @@ -32,6 +39,24 @@ class HubInstallRequest(BaseModel): scope: str = Field(default="global", description="'global' only") +class HubCatalogFacets(BaseModel): + type: dict[str, int] = Field(default_factory=dict) + category: dict[str, int] = Field(default_factory=dict) + tags: dict[str, int] = Field(default_factory=dict) + useCases: dict[str, int] = Field(default_factory=dict) + state: dict[str, int] = Field(default_factory=dict) + trust: dict[str, int] = Field(default_factory=dict) + riskLevel: dict[str, int] = Field(default_factory=dict) + + +class HubCatalogPageResponse(BaseModel): + items: list[HubCatalogEntry] = Field(default_factory=list) + total: int = Field(0) + offset: int = Field(0) + limit: int = Field(25) + facets: HubCatalogFacets = Field(default_factory=HubCatalogFacets) + + def _split_csv(value: Optional[str | list[str]]) -> Optional[list[str]]: if value is None: return None @@ -59,7 +84,22 @@ def _clear_hub_runtime_caches() -> None: pass -@router.get("/hub/catalog", response_model=list[HubCatalogEntry]) +def _build_hub_catalog_facets(items: list[HubCatalogEntry]) -> HubCatalogFacets: + facets = HubCatalogFacets() + for item in items: + facets.type[item.type] = facets.type.get(item.type, 0) + 1 + facets.category[item.category] = facets.category.get(item.category, 0) + 1 + facets.state[item.state] = facets.state.get(item.state, 0) + 1 + facets.trust[item.trust] = facets.trust.get(item.trust, 0) + 1 + facets.riskLevel[item.riskLevel] = facets.riskLevel.get(item.riskLevel, 0) + 1 + for tag in item.tags: + facets.tags[tag] = facets.tags.get(tag, 0) + 1 + for use_case in item.useCases: + facets.useCases[use_case] = facets.useCases.get(use_case, 0) + 1 + return facets + + +@router.get("/hub/catalog", response_model=Union[list[HubCatalogEntry], HubCatalogPageResponse]) async def hub_catalog( type: Optional[PluginType] = Query(default=None), # noqa: A002 - API field name category: Optional[str] = None, @@ -69,8 +109,10 @@ async def hub_catalog( trust: Optional[str] = None, risk: Optional[str] = None, q: Optional[str] = None, + offset: int = Query(0, ge=0), + limit: Optional[int] = Query(default=None, ge=1, le=200), ): - return await asyncio.to_thread( + entries = await asyncio.to_thread( list_catalog, plugin_type=type, category=_split_csv(category), @@ -81,10 +123,24 @@ async def hub_catalog( risk=_split_csv(risk), q=q, ) + if limit is None and offset == 0: + return entries + + page_limit = limit or 25 + total = len(entries) + return HubCatalogPageResponse( + items=entries[offset:offset + page_limit], + total=total, + offset=offset, + limit=page_limit, + facets=_build_hub_catalog_facets(entries), + ) @router.get("/hub/categories") -async def hub_categories(): +async def hub_categories(include_counts: bool = Query(True)): + if not include_counts: + return load_taxonomy().model_dump(mode="json") return await asyncio.to_thread(category_counts) diff --git a/flocks/server/routes/tool.py b/flocks/server/routes/tool.py index 658bc0897..f627b9f2c 100644 --- a/flocks/server/routes/tool.py +++ b/flocks/server/routes/tool.py @@ -4,7 +4,7 @@ import asyncio import time -from typing import Annotated, List, Optional, Dict, Any +from typing import Annotated, List, Optional, Dict, Any, Literal from fastapi import APIRouter, Depends, HTTPException, Query, status from pydantic import BaseModel, Field @@ -39,6 +39,7 @@ class ToolInfoResponse(BaseModel): source_name: Optional[str] = Field(None, description="Source detail, e.g. MCP server name or API module name") vendor: Optional[str] = Field(None, description="Manufacturer key for device tools (e.g. threatbook, qianxin, sangfor, qingteng)") parameters: List[Dict[str, Any]] = Field(default_factory=list, description="Tool parameters") + parameters_count: int = Field(0, description="Number of tool parameters") enabled: bool = Field(True, description="Effective enabled state (overlay applied, ANDed with API service flag)") enabled_default: bool = Field(True, description="Factory default from the YAML/registration source (no overlay)") enabled_customized: bool = Field(False, description="True if a user setting is recorded in flocks.json tool_settings") @@ -116,6 +117,23 @@ class BatchExecuteResponse(BaseModel): results: List[ToolExecuteResponse] = Field(..., description="Execution results") +class ToolListFacets(BaseModel): + """Facet counts for server-side tool list filtering.""" + category: Dict[str, int] = Field(default_factory=dict) + source: Dict[str, int] = Field(default_factory=dict) + source_name: Dict[str, int] = Field(default_factory=dict) + enabled: Dict[str, int] = Field(default_factory=dict) + + +class ToolListPageResponse(BaseModel): + """Paginated tool list response.""" + items: List[ToolInfoResponse] = Field(default_factory=list) + total: int = Field(0) + offset: int = Field(0) + limit: int = Field(20) + facets: ToolListFacets = Field(default_factory=ToolListFacets) + + # Helper: determine tool source _BUILTIN_CATEGORIES = { @@ -175,12 +193,13 @@ def _get_tool_source(tool_info: ToolInfo) -> tuple: return "custom", None -def _build_tool_response(t: ToolInfo) -> ToolInfoResponse: +def _build_tool_response(t: ToolInfo, *, include_parameters: bool = True) -> ToolInfoResponse: """Build ToolInfoResponse with source info and overlay metadata.""" source, source_name = _get_tool_source(t) setting = ConfigWriter.get_tool_setting(t.name) or {} customized = "enabled" in setting enabled_default = _get_default_enabled(t) + parameters = [p.model_dump() for p in t.parameters] if include_parameters else [] return ToolInfoResponse( name=t.name, description=t.description, @@ -189,7 +208,8 @@ def _build_tool_response(t: ToolInfo) -> ToolInfoResponse: source=source, source_name=source_name, vendor=t.vendor, - parameters=[p.model_dump() for p in t.parameters], + parameters=parameters, + parameters_count=len(t.parameters), enabled=_get_effective_tool_enabled(t), enabled_default=enabled_default, enabled_customized=customized, @@ -197,6 +217,84 @@ def _build_tool_response(t: ToolInfo) -> ToolInfoResponse: ) +def _split_csv_filter(value: Optional[str]) -> Optional[set[str]]: + if value is None: + return None + parts = {part.strip() for part in value.split(",") if part and part.strip()} + return parts or None + + +def _matches_tool_query(tool: ToolInfoResponse, query: str) -> bool: + if not query: + return True + haystack = " ".join([ + tool.name, + tool.description, + tool.description_cn or "", + tool.category, + tool.source, + tool.source_name or "", + tool.vendor or "", + ]).lower() + return query in haystack + + +def _build_tool_facets(items: List[ToolInfoResponse]) -> ToolListFacets: + facets = ToolListFacets() + for item in items: + facets.category[item.category] = facets.category.get(item.category, 0) + 1 + facets.source[item.source] = facets.source.get(item.source, 0) + 1 + source_name = item.source_name or "Flocks" + facets.source_name[source_name] = facets.source_name.get(source_name, 0) + 1 + enabled_key = str(item.enabled).lower() + facets.enabled[enabled_key] = facets.enabled.get(enabled_key, 0) + 1 + return facets + + +def _sort_tool_items( + items: List[ToolInfoResponse], + sort_by: Literal["category", "source", "source_name", "enabled", "name"], + sort_dir: Literal["asc", "desc"], +) -> List[ToolInfoResponse]: + reverse = sort_dir == "desc" + + def sort_key(item: ToolInfoResponse): + if sort_by == "enabled": + return 0 if item.enabled else 1 + if sort_by == "source_name": + return (item.source_name or "Flocks").lower() + return getattr(item, sort_by) + + return sorted(items, key=sort_key, reverse=reverse) + + +def _filter_tool_items( + items: List[ToolInfoResponse], + *, + category_filter: Optional[set[str]], + source_filter: Optional[set[str]], + source_name_filter: Optional[set[str]], + enabled_filter: Optional[set[str]], + query: str, + include_category: bool = True, + include_source: bool = True, + include_source_name: bool = True, + include_enabled: bool = True, +) -> List[ToolInfoResponse]: + result = items + if include_category and category_filter: + result = [tool for tool in result if tool.category in category_filter] + if include_source and source_filter: + result = [tool for tool in result if tool.source in source_filter] + if include_source_name and source_name_filter: + result = [tool for tool in result if (tool.source_name or "Flocks") in source_name_filter] + if include_enabled and enabled_filter: + result = [tool for tool in result if str(tool.enabled).lower() in enabled_filter] + if query: + result = [tool for tool in result if _matches_tool_query(tool, query)] + return result + + def _requires_session_backed_context(tool_info: ToolInfo) -> bool: """Return True when a tool must be anchored to a real session/message context.""" source, _ = _get_tool_source(tool_info) @@ -505,6 +603,110 @@ async def list_tools( return result +@router.get( + "/page", + response_model=ToolListPageResponse, + summary="List tools with server-side pagination", +) +async def list_tools_page( + category: Optional[str] = None, + source: Optional[str] = None, + source_name: Optional[str] = None, + enabled: Optional[str] = None, + q: Optional[str] = None, + sort_by: Literal["category", "source", "source_name", "enabled", "name"] = "source", + sort_dir: Literal["asc", "desc"] = "asc", + offset: int = Query(0, ge=0), + limit: int = Query(20, ge=1, le=200), +): + """ + List tools with server-side search/filter/sort/pagination. + + The paged list omits full parameter definitions and exposes + `parameters_count`; call `GET /api/tools/{tool_name}` for details. + """ + started_at = time.perf_counter() + await ToolRegistry.init_async() + + category_filter = _split_csv_filter(category) + source_filter = _split_csv_filter(source) + source_name_filter = _split_csv_filter(source_name) + enabled_filter = _split_csv_filter(enabled) + query = (q or "").strip().lower() + + all_items = [ + _build_tool_response(t, include_parameters=False) + for t in ToolRegistry.list_tools() + ] + + result = _filter_tool_items( + all_items, + category_filter=category_filter, + source_filter=source_filter, + source_name_filter=source_name_filter, + enabled_filter=enabled_filter, + query=query, + ) + facets = ToolListFacets( + category=_build_tool_facets(_filter_tool_items( + all_items, + category_filter=category_filter, + source_filter=source_filter, + source_name_filter=source_name_filter, + enabled_filter=enabled_filter, + query=query, + include_category=False, + )).category, + source=_build_tool_facets(_filter_tool_items( + all_items, + category_filter=category_filter, + source_filter=source_filter, + source_name_filter=source_name_filter, + enabled_filter=enabled_filter, + query=query, + include_source=False, + )).source, + source_name=_build_tool_facets(_filter_tool_items( + all_items, + category_filter=category_filter, + source_filter=source_filter, + source_name_filter=source_name_filter, + enabled_filter=enabled_filter, + query=query, + include_source_name=False, + )).source_name, + enabled=_build_tool_facets(_filter_tool_items( + all_items, + category_filter=category_filter, + source_filter=source_filter, + source_name_filter=source_name_filter, + enabled_filter=enabled_filter, + query=query, + include_enabled=False, + )).enabled, + ) + result = _sort_tool_items(result, sort_by, sort_dir) + total = len(result) + items = result[offset:offset + limit] + + log_route_timing(log, "tools.list_page.complete", started_at=started_at, extra={ + "count": len(items), + "total": total, + "category": category, + "source": source, + "q": q, + "offset": offset, + "limit": limit, + }) + return ToolListPageResponse( + items=items, + total=total, + offset=offset, + limit=limit, + facets=facets, + ) + + @router.get( "/{tool_name}", response_model=ToolInfoResponse, diff --git a/tests/hub/test_hub_catalog.py b/tests/hub/test_hub_catalog.py index 4921d8930..b1741cd22 100644 --- a/tests/hub/test_hub_catalog.py +++ b/tests/hub/test_hub_catalog.py @@ -75,6 +75,30 @@ def test_bundled_hub_catalog_loads(): assert {entry.type for entry in entries} >= {"skill", "agent", "tool", "device", "workflow", "webui", "component"} +def test_hub_catalog_snapshot_reuses_manifest_parse_for_counts(monkeypatch: pytest.MonkeyPatch): + from flocks.hub import catalog as catalog_module + + catalog_module.clear_catalog_caches() + original_read_yaml = catalog_module._read_yaml + calls = 0 + + def counted_read_yaml(path: Path): + nonlocal calls + calls += 1 + return original_read_yaml(path) + + monkeypatch.setattr(catalog_module, "_read_yaml", counted_read_yaml) + + assert catalog_module.list_catalog() + initial_calls = calls + assert initial_calls > 0 + + catalog_module.category_counts() + catalog_module.list_catalog(plugin_type="device") + + assert calls == initial_calls + + def test_workflow_catalog_exposes_chinese_names(): entries = {entry.id: entry for entry in list_catalog(plugin_type="workflow")} @@ -599,6 +623,18 @@ def test_hub_routes_cover_catalog_files_install_and_uninstall(isolated_hub_env): catalog = client.get("/api/hub/catalog").json() assert any(item["id"] == "ndr-alert-analysis" for item in catalog) + catalog_page = client.get("/api/hub/catalog", params={"limit": 1, "offset": 0}).json() + assert isinstance(catalog_page, dict) + assert len(catalog_page["items"]) == 1 + assert catalog_page["total"] == len(catalog) + assert catalog_page["limit"] == 1 + assert catalog_page["facets"]["type"] + assert catalog_page["facets"]["state"] + + taxonomy = client.get("/api/hub/categories", params={"include_counts": False}).json() + assert taxonomy["tags"] + assert "counts" not in taxonomy + detail = client.get("/api/hub/plugins/skill/ndr-alert-analysis").json() assert detail["id"] == "ndr-alert-analysis" diff --git a/tests/server/routes/test_tool_routes.py b/tests/server/routes/test_tool_routes.py index 898b03d63..5c3f958ef 100644 --- a/tests/server/routes/test_tool_routes.py +++ b/tests/server/routes/test_tool_routes.py @@ -11,7 +11,15 @@ from flocks.auth.context import AuthUser from flocks.session.message import Message, MessageRole from flocks.session.session import Session -from flocks.tool.registry import Tool, ToolCategory, ToolInfo, ToolRegistry, ToolResult +from flocks.tool.registry import ( + ParameterType, + Tool, + ToolCategory, + ToolInfo, + ToolParameter, + ToolRegistry, + ToolResult, +) @contextmanager @@ -386,3 +394,58 @@ async def handler(ctx) -> ToolResult: assert kwargs["session_id"] == session_id assert kwargs["metadata"]["messageID"] == message_id assert kwargs["tool"] == {"name": "http_batch_named_tool"} + + +class TestToolListPageRoute: + @pytest.mark.asyncio + async def test_list_page_searches_server_side_and_omits_parameter_payload(self, client: AsyncClient): + async def handler(ctx, text: str) -> ToolResult: + return ToolResult(success=True, output=f"{text}:{ctx.session_id}") + + tool = Tool( + info=ToolInfo( + name="page_unique_alpha_tool", + description="Needle Alpha paginated search tool", + category=ToolCategory.CUSTOM, + source="plugin_py", + parameters=[ + ToolParameter( + name="text", + type=ParameterType.STRING, + description="Text to echo", + ) + ], + ), + handler=handler, + ) + api_tool = Tool( + info=ToolInfo( + name="page_unique_alpha_api_tool", + description="Needle Alpha paginated API tool", + category=ToolCategory.CUSTOM, + source="api", + provider="page-api-provider", + ), + handler=handler, + ) + + with _temporary_tool(tool), _temporary_tool(api_tool): + response = await client.get( + "/api/tools/page", + params={"q": "needle alpha", "source": "plugin_py", "offset": 0, "limit": 20}, + ) + detail = await client.get("/api/tools/page_unique_alpha_tool") + + assert response.status_code == 200, response.text + payload = response.json() + assert payload["total"] == 1 + assert payload["offset"] == 0 + assert payload["limit"] == 20 + assert payload["facets"]["source"]["plugin_py"] == 1 + assert payload["facets"]["source"]["api"] == 1 + assert payload["items"][0]["name"] == "page_unique_alpha_tool" + assert payload["items"][0]["parameters"] == [] + assert payload["items"][0]["parameters_count"] == 1 + + assert detail.status_code == 200, detail.text + assert len(detail.json()["parameters"]) == 1 diff --git a/webui/src/api/hub.ts b/webui/src/api/hub.ts index 32c50a576..b65dce47e 100644 --- a/webui/src/api/hub.ts +++ b/webui/src/api/hub.ts @@ -112,12 +112,40 @@ export interface HubCatalogParams { q?: string; } +export interface HubCatalogPageParams extends HubCatalogParams { + offset?: number; + limit?: number; +} + +export interface HubCatalogFacets { + type: Record; + category: Record; + tags: Record; + useCases: Record; + state: Record; + trust: Record; + riskLevel: Record; +} + +export interface HubCatalogPageResponse { + items: HubCatalogEntry[]; + total: number; + offset: number; + limit: number; + facets: HubCatalogFacets; +} + export const hubAPI = { catalog: (params?: HubCatalogParams) => client.get('/api/hub/catalog', { params }), - categories: () => - client.get('/api/hub/categories'), + catalogPage: (params?: HubCatalogPageParams) => + client.get('/api/hub/catalog', { params }), + + categories: (params?: { includeCounts?: boolean }) => + client.get('/api/hub/categories', { + params: params ? { include_counts: params.includeCounts } : undefined, + }), get: (type: HubPluginType, id: string) => client.get(`/api/hub/plugins/${type}/${id}`), diff --git a/webui/src/api/tool.ts b/webui/src/api/tool.ts index b0299f6b8..93f0707b6 100644 --- a/webui/src/api/tool.ts +++ b/webui/src/api/tool.ts @@ -22,10 +22,55 @@ export interface ToolFixture { has_assertion: boolean; } +export type ToolListSortField = 'category' | 'source' | 'source_name' | 'enabled' | 'name'; +export type ToolListSortDir = 'asc' | 'desc'; + +export interface ToolListPageParams { + source?: string; + category?: string; + sourceName?: string; + enabled?: string; + q?: string; + sortBy?: ToolListSortField; + sortDir?: ToolListSortDir; + offset?: number; + limit?: number; +} + +export interface ToolListFacets { + category: Record; + source: Record; + source_name: Record; + enabled: Record; +} + +export interface ToolListPageResponse { + items: Tool[]; + total: number; + offset: number; + limit: number; + facets: ToolListFacets; +} + export const toolAPI = { list: (params?: { source?: ToolSource; category?: string }) => client.get('/api/tools', { params }), + listPage: (params?: ToolListPageParams) => + client.get('/api/tools/page', { + params: { + source: params?.source, + category: params?.category, + source_name: params?.sourceName, + enabled: params?.enabled, + q: params?.q, + sort_by: params?.sortBy, + sort_dir: params?.sortDir, + offset: params?.offset, + limit: params?.limit, + }, + }), + get: (name: string) => client.get(`/api/tools/${name}`), diff --git a/webui/src/hooks/useDebouncedValue.ts b/webui/src/hooks/useDebouncedValue.ts new file mode 100644 index 000000000..eaa5853d6 --- /dev/null +++ b/webui/src/hooks/useDebouncedValue.ts @@ -0,0 +1,12 @@ +import { useEffect, useState } from 'react'; + +export function useDebouncedValue(value: T, delayMs: number): T { + const [debouncedValue, setDebouncedValue] = useState(value); + + useEffect(() => { + const timer = window.setTimeout(() => setDebouncedValue(value), delayMs); + return () => window.clearTimeout(timer); + }, [value, delayMs]); + + return debouncedValue; +} diff --git a/webui/src/hooks/useTools.test.tsx b/webui/src/hooks/useTools.test.tsx index cff439668..ea51948db 100644 --- a/webui/src/hooks/useTools.test.tsx +++ b/webui/src/hooks/useTools.test.tsx @@ -1,20 +1,29 @@ import { act, renderHook, waitFor } from '@testing-library/react'; import { beforeEach, describe, expect, it, vi } from 'vitest'; -import { __resetToolsResourceForTesting, useTools } from './useTools'; +import { __resetToolsResourceForTesting, useToolPage, useTools } from './useTools'; -const { listMock, refreshMock } = vi.hoisted(() => ({ +const { listMock, listPageMock, refreshMock } = vi.hoisted(() => ({ listMock: vi.fn(), + listPageMock: vi.fn(), refreshMock: vi.fn(), })); vi.mock('@/api/tool', () => ({ toolAPI: { list: listMock, + listPage: listPageMock, refresh: refreshMock, }, })); +const emptyFacets = { + category: {}, + source: {}, + source_name: {}, + enabled: {}, +}; + describe('useTools', () => { beforeEach(() => { vi.clearAllMocks(); @@ -136,4 +145,124 @@ describe('useTools', () => { expect(first.result.current.tools).toHaveLength(1); expect(second.result.current.tools).toHaveLength(1); }); + + it('fetches a server-side page with normalized params', async () => { + listPageMock.mockResolvedValue({ + data: { + items: [{ name: 'tool-alpha', description: 'alpha tool', category: 'custom', source: 'mcp', enabled: true }], + total: 33, + offset: 20, + limit: 20, + facets: { + ...emptyFacets, + source: { mcp: 33 }, + }, + }, + }); + + const { result } = renderHook(() => useToolPage({ + source: 'mcp', + q: 'alpha', + offset: 20, + limit: 20, + })); + + await waitFor(() => { + expect(result.current.loading).toBe(false); + }); + + expect(result.current.tools).toHaveLength(1); + expect(result.current.total).toBe(33); + expect(result.current.facets.source.mcp).toBe(33); + expect(listPageMock).toHaveBeenCalledWith({ + source: 'mcp', + category: '', + sourceName: '', + enabled: '', + q: 'alpha', + sortBy: 'source', + sortDir: 'asc', + offset: 20, + limit: 20, + }); + }); + + it('shares the same paged request across concurrent hook instances', async () => { + let resolvePage: (value: { data: any }) => void = () => {}; + listPageMock.mockReturnValue(new Promise((resolve) => { + resolvePage = resolve; + })); + + const first = renderHook(() => useToolPage({ source: 'api', offset: 0, limit: 20 })); + const second = renderHook(() => useToolPage({ source: 'api', offset: 0, limit: 20 })); + + expect(listPageMock).toHaveBeenCalledTimes(1); + + await act(async () => { + resolvePage({ + data: { + items: [{ name: 'api-tool', description: 'api tool', category: 'custom', source: 'api', enabled: true }], + total: 1, + offset: 0, + limit: 20, + facets: { + ...emptyFacets, + source: { api: 1 }, + }, + }, + }); + }); + + await waitFor(() => { + expect(first.result.current.loading).toBe(false); + expect(second.result.current.loading).toBe(false); + }); + + expect(first.result.current.tools).toHaveLength(1); + expect(second.result.current.tools).toHaveLength(1); + }); + + it('refreshes only the visible paged list when refetch is called', async () => { + listPageMock + .mockResolvedValueOnce({ + data: { + items: [{ name: 'tool-alpha', description: 'alpha tool', category: 'custom', source: 'custom', enabled: true }], + total: 1, + offset: 0, + limit: 20, + facets: emptyFacets, + }, + }) + .mockResolvedValueOnce({ + data: { + items: [ + { name: 'tool-alpha', description: 'alpha tool', category: 'custom', source: 'custom', enabled: true }, + { name: 'tool-beta', description: 'beta tool', category: 'custom', source: 'custom', enabled: true }, + ], + total: 2, + offset: 0, + limit: 20, + facets: emptyFacets, + }, + }); + refreshMock.mockResolvedValue({ data: { status: 'success' } }); + + const { result } = renderHook(() => useToolPage({ offset: 0, limit: 20 })); + + await waitFor(() => { + expect(result.current.loading).toBe(false); + }); + + await act(async () => { + await result.current.refetch(); + }); + + await waitFor(() => { + expect(result.current.tools).toHaveLength(2); + }); + + expect(refreshMock).toHaveBeenCalledTimes(1); + expect(listMock).not.toHaveBeenCalled(); + expect(listPageMock).toHaveBeenCalledTimes(2); + }); }); diff --git a/webui/src/hooks/useTools.ts b/webui/src/hooks/useTools.ts index 7b9607204..724929284 100644 --- a/webui/src/hooks/useTools.ts +++ b/webui/src/hooks/useTools.ts @@ -1,9 +1,36 @@ -import { useCallback } from 'react'; -import { toolAPI, Tool } from '@/api/tool'; -import { createSharedResource, useRefreshOnResume, useSharedResource } from './useSharedResource'; +import { useCallback, useMemo } from 'react'; +import { + toolAPI, + Tool, + type ToolListFacets, + type ToolListPageParams, + type ToolListPageResponse, +} from '@/api/tool'; +import { + createSharedResource, + useRefreshOnResume, + useSharedResource, + type SharedResource, +} from './useSharedResource'; const TOOL_LIST_STALE_TIME_MS = 5000; const TOOL_LIST_MIN_FETCH_INTERVAL_MS = 1000; +const MAX_TOOL_PAGE_RESOURCES = 80; + +const EMPTY_TOOL_FACETS: ToolListFacets = { + category: {}, + source: {}, + source_name: {}, + enabled: {}, +}; + +const EMPTY_TOOL_PAGE: ToolListPageResponse = { + items: [], + total: 0, + offset: 0, + limit: 20, + facets: EMPTY_TOOL_FACETS, +}; const toolsResource = createSharedResource({ initialData: [], @@ -17,6 +44,54 @@ const toolsResource = createSharedResource({ }); let toolRefreshInFlight: Promise | null = null; +const toolPageResources = new Map>(); + +function normalizeToolPageParams(params: ToolListPageParams): Required { + return { + source: params.source ?? '', + category: params.category ?? '', + sourceName: params.sourceName ?? '', + enabled: params.enabled ?? '', + q: params.q ?? '', + sortBy: params.sortBy ?? 'source', + sortDir: params.sortDir ?? 'asc', + offset: params.offset ?? 0, + limit: params.limit ?? 20, + }; +} + +function makeToolPageResourceKey(params: ToolListPageParams): string { + return JSON.stringify(normalizeToolPageParams(params)); +} + +function getToolPageResource(params: ToolListPageParams): SharedResource { + const normalized = normalizeToolPageParams(params); + const key = makeToolPageResourceKey(normalized); + const existing = toolPageResources.get(key); + if (existing) return existing; + + const resource = createSharedResource({ + initialData: { + ...EMPTY_TOOL_PAGE, + offset: normalized.offset, + limit: normalized.limit, + }, + staleTimeMs: TOOL_LIST_STALE_TIME_MS, + minFetchIntervalMs: TOOL_LIST_MIN_FETCH_INTERVAL_MS, + fetcher: async () => { + const response = await toolAPI.listPage(normalized); + return response.data; + }, + fallbackDataOnError: (previous) => previous, + getErrorMessage: (err) => (err instanceof Error && err.message ? err.message : 'Failed to fetch tools'), + }); + toolPageResources.set(key, resource); + if (toolPageResources.size > MAX_TOOL_PAGE_RESOURCES) { + const oldestKey = toolPageResources.keys().next().value; + if (oldestKey) toolPageResources.delete(oldestKey); + } + return resource; +} function refreshToolsResource(): Promise { if (toolRefreshInFlight) { @@ -38,6 +113,8 @@ function refreshToolsResource(): Promise { export function __resetToolsResourceForTesting(): void { toolsResource.resetForTesting(); + toolPageResources.forEach((resource) => resource.resetForTesting()); + toolPageResources.clear(); toolRefreshInFlight = null; } @@ -63,3 +140,59 @@ export function useTools() { refetch, }; } + +export function useToolPage(params: ToolListPageParams) { + const resource = useMemo( + () => getToolPageResource(params), + [ + params.source, + params.category, + params.sourceName, + params.enabled, + params.q, + params.sortBy, + params.sortDir, + params.offset, + params.limit, + ], + ); + const { + data, + loading, + error, + } = useSharedResource(resource); + + const refreshVisiblePage = useCallback( + () => resource.fetch({ silent: true }), + [resource], + ); + useRefreshOnResume(refreshVisiblePage); + + const refetch = useCallback(async () => { + if (toolRefreshInFlight) { + await toolRefreshInFlight; + } else { + toolRefreshInFlight = toolAPI.refresh() + .catch(() => { + // Best-effort refresh; still update the visible page afterwards. + }) + .then(() => undefined) + .finally(() => { + toolRefreshInFlight = null; + }); + await toolRefreshInFlight; + } + await resource.fetch({ force: true, silent: true }); + }, [resource]); + + return { + tools: data.items, + total: data.total, + facets: data.facets, + offset: data.offset, + limit: data.limit, + loading, + error, + refetch, + }; +} diff --git a/webui/src/pages/Hub/index.tsx b/webui/src/pages/Hub/index.tsx index 0804f104e..aa2888a5a 100644 --- a/webui/src/pages/Hub/index.tsx +++ b/webui/src/pages/Hub/index.tsx @@ -1,4 +1,4 @@ -import { useEffect, useMemo, useState } from 'react'; +import { Suspense, lazy, useCallback, useEffect, useMemo, useState } from 'react'; import { useSearchParams } from 'react-router-dom'; import { Archive, @@ -28,20 +28,23 @@ import SuiteInstallProgressPanel, { import { hubAPI, HubCatalogEntry, + type HubCatalogFacets, HubFileContent, HubFileNode, HubInstallProgressEvent, HubManifest, HubPluginType, } from '@/api/hub'; -import FlowCanvas from '@/pages/WorkflowDetail/FlowCanvas'; import type { WorkflowJSON } from '@/api/workflow'; import { useTranslation } from 'react-i18next'; import { getCatalogDescription } from '@/utils/mcpCatalog'; +import { useDebouncedValue } from '@/hooks/useDebouncedValue'; import { useProductName } from '@/contexts/ProductNameContext'; type ViewMode = 'table' | 'tree'; +const FlowCanvas = lazy(() => import('@/pages/WorkflowDetail/FlowCanvas')); + interface HubTaxonomyCategory { id: string; name: string; @@ -243,66 +246,15 @@ function getHubName(entry: Pick, lang return language.toLowerCase().startsWith('zh') ? (nameCn || name || entry.id) : (name || nameCn || entry.id); } -interface HubFilterSnapshot { - query?: string; - type?: HubPluginType | ''; - useCase?: string; - tag?: string; - state?: string; -} - -interface HubFacetCounts { - type: Record; - useCases: Record; - tags: Record; - state: Record; -} - -function matchesCatalogEntry(entry: HubCatalogEntry, filters: HubFilterSnapshot) { - if (filters.type && entry.type !== filters.type) return false; - if (filters.useCase && !entry.useCases.includes(filters.useCase)) return false; - if (filters.tag && !entry.tags.includes(filters.tag)) return false; - if (filters.state && entry.state !== filters.state) return false; - const query = filters.query?.trim().toLowerCase(); - if (query) { - const haystack = [ - entry.id, - entry.name, - entry.description, - entry.descriptionCn ?? '', - entry.category, - ...entry.tags, - ...entry.useCases, - ].join(' ').toLowerCase(); - if (!haystack.includes(query)) return false; - } - return true; -} - -function addFacetCount(counts: Record, key: string) { - counts[key] = (counts[key] ?? 0) + 1; -} - -function buildFacetCounts(items: HubCatalogEntry[], filters: HubFilterSnapshot): HubFacetCounts { - const counts: HubFacetCounts = { type: {}, useCases: {}, tags: {}, state: {} }; - - items.forEach(item => { - if (matchesCatalogEntry(item, { query: filters.query })) { - addFacetCount(counts.type, item.type); - } - if (matchesCatalogEntry(item, { query: filters.query, type: filters.type })) { - item.useCases.forEach(useCase => addFacetCount(counts.useCases, useCase)); - } - if (matchesCatalogEntry(item, { query: filters.query, type: filters.type, useCase: filters.useCase })) { - item.tags.forEach(tag => addFacetCount(counts.tags, tag)); - } - if (matchesCatalogEntry(item, { query: filters.query, type: filters.type, useCase: filters.useCase, tag: filters.tag })) { - addFacetCount(counts.state, item.state); - } - }); - - return counts; -} +const EMPTY_HUB_FACETS: HubCatalogFacets = { + type: {}, + category: {}, + tags: {}, + useCases: {}, + state: {}, + trust: {}, + riskLevel: {}, +}; export default function HubPage() { const { i18n } = useTranslation(); @@ -331,13 +283,26 @@ export default function HubPage() { const [actionId, setActionId] = useState(null); const [suiteInstallProgress, setSuiteInstallProgress] = useState(null); const [taxonomy, setTaxonomy] = useState(null); + const [totalItems, setTotalItems] = useState(0); + const [facetCounts, setFacetCounts] = useState(EMPTY_HUB_FACETS); + const debouncedQuery = useDebouncedValue(query, 250); - const fetchCatalog = async (silent = false) => { + const fetchCatalog = useCallback(async (silent = false) => { try { if (!silent) setLoading(true); - const res = await hubAPI.catalog(); - const nextItems = Array.isArray(res.data) ? res.data : []; + const res = await hubAPI.catalogPage({ + q: debouncedQuery.trim() || undefined, + type: typeFilter || undefined, + useCases: useCaseFilter || undefined, + tags: tagFilter || undefined, + state: stateFilter || undefined, + offset: (page - 1) * pageSize, + limit: pageSize, + }); + const nextItems = Array.isArray(res.data.items) ? res.data.items : []; setCatalogItems(nextItems); + setTotalItems(res.data.total ?? nextItems.length); + setFacetCounts(res.data.facets ?? EMPTY_HUB_FACETS); setSelected(current => { if (!current) return current; return nextItems.find(item => item.type === current.type && item.id === current.id) ?? current; @@ -346,11 +311,14 @@ export default function HubPage() { } finally { if (!silent) setLoading(false); } - }; + }, [debouncedQuery, page, pageSize, stateFilter, tagFilter, typeFilter, useCaseFilter]); useEffect(() => { fetchCatalog(); - hubAPI.categories().then(res => setTaxonomy(res.data as HubTaxonomyResponse)).catch(() => setTaxonomy(null)); + }, [fetchCatalog]); + + useEffect(() => { + hubAPI.categories({ includeCounts: false }).then(res => setTaxonomy(res.data as HubTaxonomyResponse)).catch(() => setTaxonomy(null)); }, []); useEffect(() => { @@ -363,31 +331,7 @@ export default function HubPage() { } }, [catalogItems, urlPluginId, urlType]); - useEffect(() => { - setPage(1); - }, [query, typeFilter, stateFilter, tagFilter, useCaseFilter, viewMode, pageSize]); - - const items = useMemo( - () => catalogItems.filter(item => matchesCatalogEntry(item, { - query, - type: typeFilter, - useCase: useCaseFilter, - tag: tagFilter, - state: stateFilter, - })), - [catalogItems, query, typeFilter, useCaseFilter, tagFilter, stateFilter], - ); - - const facetCounts = useMemo( - () => buildFacetCounts(catalogItems, { - query, - type: typeFilter, - useCase: useCaseFilter, - tag: tagFilter, - state: stateFilter, - }), - [catalogItems, query, typeFilter, useCaseFilter, tagFilter, stateFilter], - ); + const items = catalogItems; const useCases = useMemo( () => taxonomy?.useCases ?? Array.from(new Set(catalogItems.flatMap(item => item.useCases))).sort(), @@ -398,12 +342,13 @@ export default function HubPage() { [catalogItems, taxonomy], ); const activeFilterCount = [typeFilter, useCaseFilter, tagFilter, stateFilter].filter(Boolean).length; - const totalPages = Math.max(1, Math.ceil(items.length / pageSize)); + const totalPages = Math.max(1, Math.ceil(totalItems / pageSize)); const currentPage = Math.min(page, totalPages); - const pagedItems = useMemo( - () => items.slice((currentPage - 1) * pageSize, currentPage * pageSize), - [items, currentPage, pageSize], - ); + const pagedItems = items; + + useEffect(() => { + if (page > totalPages) setPage(totalPages); + }, [page, totalPages]); const handleRefresh = async () => { setRefreshing(true); try { @@ -452,6 +397,7 @@ export default function HubPage() { setUseCaseFilter(''); setTagFilter(''); setStateFilter(''); + setPage(1); }; if (loading) { @@ -470,7 +416,10 @@ export default function HubPage() { setQuery(e.target.value)} + onChange={e => { + setQuery(e.target.value); + setPage(1); + }} placeholder={text.searchPlaceholder} className="w-full pl-9 pr-3 py-2 border border-gray-300 rounded-lg text-sm outline-none bg-white/90 focus:ring-2 focus:ring-slate-200 focus:border-slate-400" /> @@ -500,7 +449,10 @@ export default function HubPage() { setTypeFilter(value as HubPluginType | '')} + onChange={value => { + setTypeFilter(value as HubPluginType | ''); + setPage(1); + }} options={[ { value: '', label: text.all }, ...HUB_PLUGIN_TYPES.map(type => ({ @@ -517,7 +469,10 @@ export default function HubPage() { { + setUseCaseFilter(value); + setPage(1); + }} options={[ { value: '', label: text.all }, ...useCases.map(useCase => ({ @@ -531,7 +486,10 @@ export default function HubPage() { { + setTagFilter(value); + setPage(1); + }} options={[ { value: '', label: text.all }, ...tags.map(tag => ({ @@ -545,7 +503,10 @@ export default function HubPage() { { + setStateFilter(value); + setPage(1); + }} options={[ { value: '', label: text.all }, ...(['available', 'installed', 'updateAvailable', 'incompatible'] as const).map(state => ({ @@ -590,13 +551,16 @@ export default function HubPage() { {viewMode === 'table' && ( { + setPageSize(nextPageSize); + setPage(1); + }} /> )} @@ -1019,7 +983,9 @@ function PluginDetail({ entry, language, onClose, onAction, actionId, text }: { {text.workflowDiagram}
- +
}> + + ) : workflowError ? ( diff --git a/webui/src/pages/Tool/components/LocalTabContent.tsx b/webui/src/pages/Tool/components/LocalTabContent.tsx index ed0e64c3e..aa3e7b528 100644 --- a/webui/src/pages/Tool/components/LocalTabContent.tsx +++ b/webui/src/pages/Tool/components/LocalTabContent.tsx @@ -154,7 +154,7 @@ export default function LocalTabContent({ {/* Stats column */}
- {t('local.paramsCount', { count: tool.parameters?.length || 0 })} + {t('local.paramsCount', { count: tool.parameters_count ?? tool.parameters?.length ?? 0 })}
{/* Actions column */} diff --git a/webui/src/pages/Tool/components/MCPTabContent.tsx b/webui/src/pages/Tool/components/MCPTabContent.tsx index b7ede1274..ea6a2c3d5 100644 --- a/webui/src/pages/Tool/components/MCPTabContent.tsx +++ b/webui/src/pages/Tool/components/MCPTabContent.tsx @@ -170,6 +170,19 @@ export default function MCPTabContent({ setSelectedServerData(name ? (servers.find((server) => server.name === name) ?? null) : null); }, [servers]); + const selectMcpTool = useCallback((tool: Tool) => { + setSelectedToolFromMCP(tool); + toolAPI.get(tool.name) + .then((response) => { + setSelectedToolFromMCP((current) => ( + current?.name === tool.name ? response.data : current + )); + }) + .catch(() => { + // Keep the summary row visible; reopening retries the detail request. + }); + }, []); + const selectedServerObj = useMemo(() => { if (!selectedServer) return undefined; const fromServers = servers.find((server) => server.name === selectedServer); @@ -702,7 +715,7 @@ export default function MCPTabContent({ onRefresh={async () => { await mcpAPI.refresh(selectedServer); await fetchServers(); await onRefreshTools(); }} onStatusChange={async () => { await fetchServers(); await onRefreshTools(); }} onRemove={() => handleRemove(selectedServer)} - onSelectTool={setSelectedToolFromMCP} + onSelectTool={selectMcpTool} /> diff --git a/webui/src/pages/Tool/index.tsx b/webui/src/pages/Tool/index.tsx index e0f6eae94..3235ade6f 100644 --- a/webui/src/pages/Tool/index.tsx +++ b/webui/src/pages/Tool/index.tsx @@ -43,7 +43,8 @@ import { } from 'lucide-react'; import LoadingSpinner from '@/components/common/LoadingSpinner'; import EmptyState from '@/components/common/EmptyState'; -import { useTools } from '@/hooks/useTools'; +import { useToolPage } from '@/hooks/useTools'; +import { useDebouncedValue } from '@/hooks/useDebouncedValue'; import { canDirectlyTestTool, toolAPI, Tool, ToolFixture, ToolSource } from '@/api/tool'; import { mcpAPI, MCPServer } from '@/api/mcp'; import { providerAPI } from '@/api/provider'; @@ -54,7 +55,6 @@ import type { MCPFormData, ConnStatus as MCPConnStatus } from './ToolSheets'; import MCPTabContent from './components/MCPTabContent'; import APITabContent from './components/APITabContent'; import LocalTabContent from './components/LocalTabContent'; -import { getToolTabCounts } from './tabCounts'; import { getSourceLabel } from './constants'; import { getCatalogDescription, getMetadataDescription } from '@/utils/mcpCatalog'; import { getLocalizedToolDescription, getLocalizedFixtureLabel } from './toolDisplay'; @@ -138,6 +138,15 @@ const EMPTY_FILTERS: ColumnFilters = { enabled: new Set(), }; +function joinFilterValues(values: Set): string | undefined { + if (values.size === 0) return undefined; + return Array.from(values).sort().join(','); +} + +function mergeFacetKeys(facets: Record, activeValues: Set = new Set()): string[] { + return Array.from(new Set([...Object.keys(facets), ...Array.from(activeValues)])).sort(); +} + // ============================================================================ // Main Page // ============================================================================ @@ -171,8 +180,46 @@ export default function ToolPage() { const [sort, setSort] = useState({ field: 'source', dir: 'asc' }); const [filters, setFilters] = useState(EMPTY_FILTERS); - const { tools, loading, error, refetch } = useTools(); const [apiEnabledServicesCount, setApiEnabledServicesCount] = useState(0); + const debouncedSearchQuery = useDebouncedValue(searchQuery, 250); + const activeTabConfig = TABS.find((tab) => tab.key === activeTab); + const tabSourceFilter = activeTabConfig?.sourceFilter; + const sourceFilterParam = useMemo(() => { + const values = new Set(filters.source); + if (tabSourceFilter) { + const tabSources = Array.isArray(tabSourceFilter) ? tabSourceFilter : [tabSourceFilter]; + tabSources.forEach((source) => values.add(source)); + } + return joinFilterValues(values); + }, [filters.source, tabSourceFilter]); + const toolPageParams = useMemo(() => ({ + source: sourceFilterParam, + category: joinFilterValues(filters.category), + sourceName: joinFilterValues(filters.source_name), + enabled: joinFilterValues(filters.enabled), + q: debouncedSearchQuery.trim() || undefined, + sortBy: sort.field, + sortDir: sort.dir, + offset: (currentPage - 1) * PAGE_SIZE, + limit: PAGE_SIZE, + }), [ + sourceFilterParam, + filters.category, + filters.source_name, + filters.enabled, + debouncedSearchQuery, + sort.field, + sort.dir, + currentPage, + ]); + const { + tools, + total: totalTools, + facets: toolFacets, + loading, + error, + refetch, + } = useToolPage(toolPageParams); // Catalog data (fetched once at top level, shared with MCP & API tabs) const [catalogEntries, setCatalogEntries] = useState([]); @@ -239,100 +286,37 @@ export default function ToolPage() { // API catalog not yet implemented — keep this empty so entries are not duplicated across tabs. const apiCatalogEntries = useMemo(() => [] as MCPCatalogEntry[], []); - // Compute tab counts: all = active tools; mcp/api = unique active servers/modules - const tabCounts = useMemo( - () => getToolTabCounts(tools, apiEnabledServicesCount), - [tools, apiEnabledServicesCount], - ); + // Compute tab counts from server-side facets for the current query/filter set. + const tabCounts = useMemo(() => { + const allFromSourceFacets = Object.values(toolFacets.source).reduce((sum, count) => sum + count, 0); + return { + all: allFromSourceFacets || totalTools, + mcp: toolFacets.source.mcp ?? 0, + api: Math.max(toolFacets.source.api ?? 0, apiEnabledServicesCount), + local: toolFacets.source.plugin_py ?? 0, + }; + }, [totalTools, toolFacets.source, apiEnabledServicesCount]); - // Get unique values for filter options (active tools only) + // Filter dropdown options come from server-side facets, plus active values + // so a selected filter remains visible even when it narrows the result set. const filterOptions = useMemo(() => { - const cats = new Set(); - const sources = new Set(); - const sourceNames = new Set(); - tools.forEach((tool) => { - cats.add(tool.category); - sources.add(tool.source); - sourceNames.add(tool.source_name || 'Flocks'); - }); return { - category: Array.from(cats).sort(), - source: Array.from(sources).sort((a, b) => (SOURCE_SORT_ORDER[a] ?? 99) - (SOURCE_SORT_ORDER[b] ?? 99)), - source_name: Array.from(sourceNames).sort(), + category: mergeFacetKeys(toolFacets.category, filters.category), + source: mergeFacetKeys(toolFacets.source, filters.source) + .sort((a, b) => (SOURCE_SORT_ORDER[a] ?? 99) - (SOURCE_SORT_ORDER[b] ?? 99)), + source_name: mergeFacetKeys(toolFacets.source_name, filters.source_name), enabled: ['true', 'false'], }; - }, [tools]); - - // Apply tab filter + search + column filters + sort (All tab shows active tools only) - const processedTools = useMemo(() => { - let result = [...tools]; - - // Tab filter - const tabConfig = TABS.find((tab) => tab.key === activeTab); - if (tabConfig?.sourceFilter) { - const sf = tabConfig.sourceFilter; - const allowed = Array.isArray(sf) ? sf : [sf]; - result = result.filter((tool) => allowed.includes(tool.source)); - } - - // Search - if (searchQuery) { - const q = searchQuery.toLowerCase(); - result = result.filter( - (tool) => - tool.name.toLowerCase().includes(q) || - tool.description.toLowerCase().includes(q) || - (tool.description_cn || '').toLowerCase().includes(q) || - (tool.source_name || '').toLowerCase().includes(q) - ); - } - - // Column filters - if (filters.category.size > 0) { - result = result.filter((tool) => filters.category.has(tool.category)); - } - if (filters.source.size > 0) { - result = result.filter((tool) => filters.source.has(tool.source)); - } - if (filters.source_name.size > 0) { - result = result.filter((tool) => filters.source_name.has(tool.source_name || 'Flocks')); - } - if (filters.enabled.size > 0) { - result = result.filter((tool) => filters.enabled.has(String(tool.enabled))); - } - - // Sort - result.sort((a, b) => { - let cmp = 0; - switch (sort.field) { - case 'category': { - const la = a.category; - const lb = b.category; - cmp = la.localeCompare(lb); - break; - } - case 'source': - cmp = (SOURCE_SORT_ORDER[a.source] ?? 99) - (SOURCE_SORT_ORDER[b.source] ?? 99); - break; - case 'source_name': - cmp = (a.source_name || 'Flocks').localeCompare(b.source_name || 'Flocks', 'zh'); - break; - case 'enabled': - cmp = (a.enabled === b.enabled ? 0 : a.enabled ? -1 : 1); - break; - } - return sort.dir === 'desc' ? -cmp : cmp; - }); - - return result; - }, [tools, activeTab, searchQuery, filters, sort]); + }, [toolFacets, filters.category, filters.source, filters.source_name]); // Pagination - const totalPages = Math.ceil(processedTools.length / PAGE_SIZE); - const paginatedTools = useMemo(() => { - const start = (currentPage - 1) * PAGE_SIZE; - return processedTools.slice(start, start + PAGE_SIZE); - }, [processedTools, currentPage]); + const processedTools = tools; + const totalPages = Math.max(1, Math.ceil(totalTools / PAGE_SIZE)); + const paginatedTools = tools; + + useEffect(() => { + if (currentPage > totalPages) setCurrentPage(totalPages); + }, [currentPage, totalPages]); const handleTabChange = (tab: TabKey) => { setActiveTab(tab); @@ -384,6 +368,15 @@ export default function ToolPage() { setSelectedTool(tool); setTestParams('{}'); setTestResult(null); + toolAPI.get(tool.name) + .then((response) => { + setSelectedTool((current) => ( + current?.name === tool.name ? response.data : current + )); + }) + .catch(() => { + // Keep the lightweight row data visible; retry happens when reopened. + }); }; const toggleSort = (field: SortField) => { @@ -589,7 +582,7 @@ export default function ToolPage() { )} - {processedTools.length === 0 ? ( + {totalTools === 0 ? ( } title={t('empty.noTools')} @@ -603,7 +596,7 @@ export default function ToolPage() { filterOptions={filterOptions} currentPage={currentPage} totalPages={totalPages} - totalCount={processedTools.length} + totalCount={totalTools} pageSize={PAGE_SIZE} onSort={toggleSort} onToggleFilter={toggleFilter} @@ -3814,4 +3807,3 @@ const LANG_COLORS: Record = { // CatalogBrowser removed — catalog UI is now inline in MCPTabContent / APITabContent // (CatalogBrowser component removed — catalog UI is inline in MCPTabContent / APITabContent) - diff --git a/webui/src/types/index.ts b/webui/src/types/index.ts index 4b8995e03..3e0517bd2 100644 --- a/webui/src/types/index.ts +++ b/webui/src/types/index.ts @@ -203,6 +203,7 @@ export interface Tool { source: ToolSource; source_name?: string; parameters: ToolParameter[]; + parameters_count?: number; enabled: boolean; /** Factory default from the YAML/registration source (no overlay applied). */ enabled_default?: boolean; From 819e6bcad4a61e985f53d65214861d8bc83aa6a0 Mon Sep 17 00:00:00 2001 From: duguwanglong Date: Thu, 9 Jul 2026 17:22:17 +0800 Subject: [PATCH 05/49] perf(webui): cache tool list summaries Reuse lightweight tool summaries for paginated search/filter requests and invalidate the snapshot when tool or API service state changes. Add route coverage for cache reuse on the tools page. --- flocks/server/routes/provider.py | 10 ++++ flocks/server/routes/tool.py | 57 +++++++++++++++++++-- tests/server/routes/test_tool_routes.py | 68 +++++++++++++++++++++++++ 3 files changed, 131 insertions(+), 4 deletions(-) diff --git a/flocks/server/routes/provider.py b/flocks/server/routes/provider.py index 19c8e8397..62c713ec6 100644 --- a/flocks/server/routes/provider.py +++ b/flocks/server/routes/provider.py @@ -989,6 +989,16 @@ def _set_api_service_tools_enabled(provider_id: str, enabled: bool) -> int: matched_tools = _get_api_service_tool_infos(provider_id) for tool_info in matched_tools: tool_info.enabled = enabled + if matched_tools: + try: + from flocks.server.routes.tool import _invalidate_tool_summary_cache + + _invalidate_tool_summary_cache() + except Exception as e: + log.debug("tool.summary_cache.invalidate_failed", { + "provider_id": provider_id, + "error": str(e), + }) return len(matched_tools) diff --git a/flocks/server/routes/tool.py b/flocks/server/routes/tool.py index f627b9f2c..9fb1db308 100644 --- a/flocks/server/routes/tool.py +++ b/flocks/server/routes/tool.py @@ -3,6 +3,7 @@ """ import asyncio +import threading import time from typing import Annotated, List, Optional, Dict, Any, Literal from fastapi import APIRouter, Depends, HTTPException, Query, status @@ -149,6 +150,48 @@ class ToolListPageResponse(BaseModel): "Direct HTTP execution for local or permission-gated tools requires a verified " "session-backed request with both sessionID and messageID." ) +_TOOL_SUMMARY_CACHE_TTL_SECONDS = 5.0 +_tool_summary_cache_lock = threading.Lock() +_tool_summary_cache_key: tuple[int, tuple[str, ...]] | None = None +_tool_summary_cache_expires_at = 0.0 +_tool_summary_cache_items: tuple[ToolInfoResponse, ...] = () + + +def _invalidate_tool_summary_cache() -> None: + """Clear the lightweight list snapshot used by ``GET /api/tools/page``.""" + global _tool_summary_cache_key, _tool_summary_cache_expires_at, _tool_summary_cache_items + with _tool_summary_cache_lock: + _tool_summary_cache_key = None + _tool_summary_cache_expires_at = 0.0 + _tool_summary_cache_items = () + + +def _tool_summary_cache_current_key() -> tuple[int, tuple[str, ...]]: + return ToolRegistry.revision(), tuple(ToolRegistry.all_tool_ids()) + + +def _get_tool_summary_items() -> List[ToolInfoResponse]: + """Return cached, parameter-free tool summaries for list filtering.""" + global _tool_summary_cache_key, _tool_summary_cache_expires_at, _tool_summary_cache_items + + now = time.monotonic() + cache_key = _tool_summary_cache_current_key() + with _tool_summary_cache_lock: + if ( + _tool_summary_cache_items + and _tool_summary_cache_key == cache_key + and now < _tool_summary_cache_expires_at + ): + return list(_tool_summary_cache_items) + + items = tuple( + _build_tool_response(t, include_parameters=False) + for t in ToolRegistry.list_tools() + ) + _tool_summary_cache_key = cache_key + _tool_summary_cache_expires_at = now + _TOOL_SUMMARY_CACHE_TTL_SECONDS + _tool_summary_cache_items = items + return list(items) def _get_tool_source(tool_info: ToolInfo) -> tuple: @@ -634,10 +677,7 @@ async def list_tools_page( enabled_filter = _split_csv_filter(enabled) query = (q or "").strip().lower() - all_items = [ - _build_tool_response(t, include_parameters=False) - for t in ToolRegistry.list_tools() - ] + all_items = _get_tool_summary_items() result = _filter_tool_items( all_items, @@ -819,10 +859,12 @@ async def update_tool( # The in-memory ToolInfo.enabled reflects global state. Per-device # enabled=True is not a supported override; switch-on means clear the # per-device disable and follow the global tool setting. + _invalidate_tool_summary_cache() return _build_tool_response(tool.info) # --- Global mode (original behaviour) --- _set_global_tool_enabled(tool, desired) + _invalidate_tool_summary_cache() return _build_tool_response(tool.info) @@ -853,6 +895,7 @@ async def reset_tool_setting(tool_name: str, _admin: object = Depends(require_ad default = _get_default_enabled(tool.info) new_enabled = default and _service_allows_enable(tool.info) tool.info.enabled = new_enabled + _invalidate_tool_summary_cache() log.info("tool.setting.reset", { "name": tool_name, @@ -1054,6 +1097,7 @@ async def refresh_tools(_admin: object = Depends(require_admin)): log.error("tools.refresh.plugin_error", {"error": str(e)}) errors.append(f"plugin: {e}") + _invalidate_tool_summary_cache() tool_count = len(ToolRegistry.all_tool_ids()) log_route_timing(log, "tools.refresh.done", started_at=started_at, extra={ "tool_count": tool_count, @@ -1284,6 +1328,7 @@ async def create_tool(request: CreateToolRequest, _admin: object = Depends(requi ToolRegistry.register(tool) if tool.info.name not in ToolRegistry._plugin_tool_names: ToolRegistry._plugin_tool_names.append(tool.info.name) + _invalidate_tool_summary_cache() except Exception as e: log.error("tool.create.register_error", {"error": str(e), "name": request.name}) raise HTTPException( @@ -1357,12 +1402,14 @@ async def update_plugin_tool(name: str, request: UpdateToolRequest, _admin: obje if not tool.info.source: tool.info.source = "plugin_yaml" ToolRegistry.register(tool) + _invalidate_tool_summary_cache() return _build_tool_response(tool.info) except Exception as e: log.error("tool.update.reload_error", {"error": str(e), "name": name}) existing = ToolRegistry.get(name) if existing: + _invalidate_tool_summary_cache() return _build_tool_response(existing.info) raise HTTPException(status_code=500, detail="Tool updated but reload failed") @@ -1404,6 +1451,7 @@ async def delete_tool(name: str, _admin: object = Depends(require_admin)): # Refresh plugin tools so stale decorator-registered python tools are removed too. ToolRegistry.refresh_plugin_tools() + _invalidate_tool_summary_cache() from flocks.hub import local as hub_local @@ -1441,6 +1489,7 @@ async def reload_tool(name: str, _admin: object = Depends(require_admin)): if not tool.info.source: tool.info.source = "plugin_yaml" ToolRegistry.register(tool) + _invalidate_tool_summary_cache() log.info("tool.reloaded", {"name": name}) return _build_tool_response(tool.info) except Exception as e: diff --git a/tests/server/routes/test_tool_routes.py b/tests/server/routes/test_tool_routes.py index 5c3f958ef..b3f2c0232 100644 --- a/tests/server/routes/test_tool_routes.py +++ b/tests/server/routes/test_tool_routes.py @@ -27,6 +27,7 @@ def _temporary_tool(tool: Tool) -> Iterator[None]: ToolRegistry.init() existing = ToolRegistry._tools.get(tool.info.name) ToolRegistry.register(tool) + _clear_tool_summary_cache() try: yield finally: @@ -35,6 +36,13 @@ def _temporary_tool(tool: Tool) -> Iterator[None]: ToolRegistry._tools[tool.info.name] = existing else: ToolRegistry._tools.pop(tool.info.name, None) + _clear_tool_summary_cache() + + +def _clear_tool_summary_cache() -> None: + from flocks.server.routes import tool as tool_routes + + tool_routes._invalidate_tool_summary_cache() class _FakeSessionUser: @@ -449,3 +457,63 @@ async def handler(ctx, text: str) -> ToolResult: assert detail.status_code == 200, detail.text assert len(detail.json()["parameters"]) == 1 + + @pytest.mark.asyncio + async def test_list_page_reuses_lightweight_summary_cache( + self, + client: AsyncClient, + monkeypatch: pytest.MonkeyPatch, + ): + from flocks.server.routes import tool as tool_routes + + async def handler(ctx) -> ToolResult: + return ToolResult(success=True, output=ctx.session_id) + + tool = Tool( + info=ToolInfo( + name="page_summary_cache_unique_tool", + description="NeedlePageSummaryCacheUnique", + category=ToolCategory.CUSTOM, + source="plugin_py", + ), + handler=handler, + ) + + original_build_tool_response = tool_routes._build_tool_response + summary_builds = 0 + + def counted_build_tool_response(tool_info, *, include_parameters=True): + nonlocal summary_builds + if not include_parameters: + summary_builds += 1 + return original_build_tool_response(tool_info, include_parameters=include_parameters) + + monkeypatch.setattr(tool_routes, "_build_tool_response", counted_build_tool_response) + + with _temporary_tool(tool): + first = await client.get( + "/api/tools/page", + params={"q": "needlepagesummarycacheunique", "limit": 25}, + ) + first_builds = summary_builds + second = await client.get( + "/api/tools/page", + params={"q": "needlepagesummarycacheunique", "source": "plugin_py", "limit": 25}, + ) + second_builds = summary_builds + + tool_routes._invalidate_tool_summary_cache() + third = await client.get( + "/api/tools/page", + params={"q": "needlepagesummarycacheunique", "limit": 25}, + ) + + assert first.status_code == 200, first.text + assert second.status_code == 200, second.text + assert third.status_code == 200, third.text + assert first.json()["total"] == 1 + assert second.json()["total"] == 1 + assert third.json()["total"] == 1 + assert first_builds > 0 + assert second_builds == first_builds + assert summary_builds > second_builds From d6a9db9112f235e252e6ec510f89f7551c1b03a7 Mon Sep 17 00:00:00 2001 From: duguwanglong Date: Thu, 9 Jul 2026 17:56:38 +0800 Subject: [PATCH 06/49] perf(webui): optimize tool list loading --- flocks/server/routes/provider.py | 125 +++++++++++++++++- flocks/server/routes/tool.py | 105 ++++++++++++--- tests/provider/test_api_service_management.py | 83 +++++++++++- tests/server/routes/test_tool_routes.py | 21 ++- webui/src/api/provider.test.ts | 90 +++++++++++++ webui/src/api/provider.ts | 74 ++++++++++- webui/src/hooks/useTools.test.tsx | 31 +++++ webui/src/hooks/useTools.ts | 5 +- webui/src/pages/Tool/index.tsx | 61 ++++++--- webui/src/pages/Tool/tabLoading.test.ts | 29 ++++ webui/src/pages/Tool/tabLoading.ts | 17 +++ 11 files changed, 579 insertions(+), 62 deletions(-) create mode 100644 webui/src/api/provider.test.ts create mode 100644 webui/src/pages/Tool/tabLoading.test.ts create mode 100644 webui/src/pages/Tool/tabLoading.ts diff --git a/flocks/server/routes/provider.py b/flocks/server/routes/provider.py index 62c713ec6..31efde42e 100644 --- a/flocks/server/routes/provider.py +++ b/flocks/server/routes/provider.py @@ -6,10 +6,12 @@ temperature, tool_call, limit, etc. """ -import time +import json import re +import threading +import time from pathlib import Path -from typing import List, Optional, Dict, Any +from typing import Any, Dict, List, Optional from fastapi import APIRouter, Body, HTTPException, Query, status from pydantic import BaseModel, Field, ConfigDict @@ -42,6 +44,8 @@ router = APIRouter() log = Log.create(service="provider-routes") +_api_service_summary_metadata_cache_lock = threading.Lock() +_api_service_summary_metadata_cache: Dict[str, tuple[tuple[Any, ...], Optional[Dict[str, Any]]]] = {} def _load_provider_yaml_metadata(provider_id: str) -> Optional[Dict[str, Any]]: @@ -71,6 +75,118 @@ def _load_api_service_metadata_data(provider_id: str) -> Optional[Dict[str, Any] return merged or None +def _clear_api_service_summary_metadata_cache(provider_id: Optional[str] = None) -> None: + with _api_service_summary_metadata_cache_lock: + if provider_id is None: + _api_service_summary_metadata_cache.clear() + else: + _api_service_summary_metadata_cache.pop(provider_id, None) + + +def _provider_yaml_cache_key(provider_id: str) -> tuple[str, int]: + descriptor = _find_api_service_descriptor(provider_id) + if descriptor is None: + return "", 0 + try: + return str(descriptor.provider_yaml), descriptor.provider_yaml.stat().st_mtime_ns + except OSError: + return str(descriptor.provider_yaml), 0 + + +def _legacy_metadata_cache_key(provider_id: str) -> tuple[str, int]: + meta_file = api_service_schema_helpers._LEGACY_METADATA_DIR / f"{provider_id}.json" + if not meta_file.is_file(): + return "", 0 + try: + return str(meta_file), meta_file.stat().st_mtime_ns + except OSError: + return str(meta_file), 0 + + +def _api_service_summary_metadata_cache_key(provider_id: str) -> tuple[Any, ...]: + config_data = ConfigWriter.get_api_service_raw(provider_id) or {} + return ( + json.dumps(config_data, sort_keys=True, default=str), + _legacy_metadata_cache_key(provider_id), + _provider_yaml_cache_key(provider_id), + ) + + +def _find_api_service_descriptor(provider_id: str): + from flocks.config.api_versioning import discover_api_service_descriptors + + return next( + ( + d + for d in discover_api_service_descriptors() + if provider_id in (d.storage_key, d.service_id) + ), + None, + ) + + +def _load_provider_yaml_summary_metadata(provider_id: str) -> Optional[Dict[str, Any]]: + descriptor = _find_api_service_descriptor(provider_id) + if descriptor is None: + return None + try: + prov = api_service_schema_helpers.yaml.safe_load( + descriptor.provider_yaml.read_text(encoding="utf-8") + ) + except Exception as e: + log.debug("api_service.summary_metadata.provider_yaml_failed", { + "provider_id": provider_id, + "path": str(descriptor.provider_yaml), + "error": str(e), + }) + return None + if not isinstance(prov, dict): + return None + return { + "name": prov.get("name", provider_id), + "service_id": prov.get("service_id", provider_id), + "version": extract_provider_version(prov), + "description": prov.get("description"), + "description_cn": prov.get("description_cn"), + "defaults": prov.get("defaults", {}), + "integration_type": prov.get("integration_type"), + "vendor": prov.get("vendor"), + } + + +def _load_api_service_summary_metadata_data(provider_id: str) -> Optional[Dict[str, Any]]: + """Load only metadata needed by the API service list endpoint.""" + cache_key = _api_service_summary_metadata_cache_key(provider_id) + with _api_service_summary_metadata_cache_lock: + cached = _api_service_summary_metadata_cache.get(provider_id) + if cached and cached[0] == cache_key: + return dict(cached[1]) if isinstance(cached[1], dict) else cached[1] + + merged: Dict[str, Any] = {} + config_data = ConfigWriter.get_api_service_raw(provider_id) + if isinstance(config_data, dict): + merged.update(config_data) + + meta_file = api_service_schema_helpers._LEGACY_METADATA_DIR / f"{provider_id}.json" + if meta_file.is_file(): + with open(meta_file, "r", encoding="utf-8") as f: + metadata_data = api_service_schema_helpers.json.load(f) + if isinstance(metadata_data, dict): + merged = {**metadata_data, **merged} + + yaml_data = _load_provider_yaml_summary_metadata(provider_id) + if isinstance(yaml_data, dict): + merged = {**yaml_data, **merged} + + result = merged or None + with _api_service_summary_metadata_cache_lock: + _api_service_summary_metadata_cache[provider_id] = ( + cache_key, + dict(result) if isinstance(result, dict) else result, + ) + return result + + _EMAIL_KEY_PAIR_PATTERN = re.compile( r"([A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,})\s*:\s*([^\s]+)" ) @@ -1041,7 +1157,7 @@ def _build_api_service_summary( raw_statuses: Dict[str, Any], ) -> APIServiceSummary: """Build API service summary used by the Tool API page.""" - meta = _load_api_service_metadata_data(provider_id) or {} + meta = _load_api_service_summary_metadata_data(provider_id) or {} enabled = _get_api_service_enabled(provider_id) cached_status = raw_statuses.get(provider_id) or {} matched_tools = _get_api_service_tool_infos(provider_id) @@ -1187,6 +1303,7 @@ async def update_api_service(provider_id: str, request: APIServiceUpdateRequest) if request.verify_ssl is not None: existing["verify_ssl"] = request.verify_ssl ConfigWriter.set_api_service(provider_id, existing) + _clear_api_service_summary_metadata_cache(provider_id) matched_count = _set_api_service_tools_enabled(provider_id, request.enabled) @@ -1316,6 +1433,7 @@ async def delete_api_service(provider_id: str) -> Dict[str, Any]: backing_plugin = _find_user_installed_tool_plugin_for(provider_id) removed_config = ConfigWriter.remove_api_service(provider_id) + _clear_api_service_summary_metadata_cache(provider_id) secrets = get_secret_manager() deleted_secret = False @@ -1945,6 +2063,7 @@ async def set_service_credentials(provider_id: str, request: ProviderCredentialR if field.key == "base_url": existing.pop("baseUrl", None) ConfigWriter.set_api_service(provider_id, existing) + _clear_api_service_summary_metadata_cache(provider_id) log.info( "service.credentials.set", diff --git a/flocks/server/routes/tool.py b/flocks/server/routes/tool.py index 9fb1db308..2c9da905b 100644 --- a/flocks/server/routes/tool.py +++ b/flocks/server/routes/tool.py @@ -3,9 +3,10 @@ """ import asyncio +from dataclasses import dataclass import threading import time -from typing import Annotated, List, Optional, Dict, Any, Literal +from typing import Annotated, List, Optional, Dict, Any, Literal, Sequence from fastapi import APIRouter, Depends, HTTPException, Query, status from pydantic import BaseModel, Field @@ -131,7 +132,7 @@ class ToolListPageResponse(BaseModel): items: List[ToolInfoResponse] = Field(default_factory=list) total: int = Field(0) offset: int = Field(0) - limit: int = Field(20) + limit: int = Field(25) facets: ToolListFacets = Field(default_factory=ToolListFacets) @@ -150,11 +151,29 @@ class ToolListPageResponse(BaseModel): "Direct HTTP execution for local or permission-gated tools requires a verified " "session-backed request with both sessionID and messageID." ) + + +@dataclass(frozen=True) +class ToolListIndexItem: + name: str + description: str + description_cn: Optional[str] + category: str + source: str + source_name: Optional[str] + vendor: Optional[str] + parameters_count: int + enabled: bool + enabled_default: bool + enabled_customized: bool + requires_confirmation: bool + + _TOOL_SUMMARY_CACHE_TTL_SECONDS = 5.0 _tool_summary_cache_lock = threading.Lock() _tool_summary_cache_key: tuple[int, tuple[str, ...]] | None = None _tool_summary_cache_expires_at = 0.0 -_tool_summary_cache_items: tuple[ToolInfoResponse, ...] = () +_tool_summary_cache_items: tuple[ToolListIndexItem, ...] = () def _invalidate_tool_summary_cache() -> None: @@ -170,8 +189,8 @@ def _tool_summary_cache_current_key() -> tuple[int, tuple[str, ...]]: return ToolRegistry.revision(), tuple(ToolRegistry.all_tool_ids()) -def _get_tool_summary_items() -> List[ToolInfoResponse]: - """Return cached, parameter-free tool summaries for list filtering.""" +def _get_tool_summary_items() -> List[ToolListIndexItem]: + """Return cached, lightweight tool summaries for list filtering.""" global _tool_summary_cache_key, _tool_summary_cache_expires_at, _tool_summary_cache_items now = time.monotonic() @@ -184,10 +203,7 @@ def _get_tool_summary_items() -> List[ToolInfoResponse]: ): return list(_tool_summary_cache_items) - items = tuple( - _build_tool_response(t, include_parameters=False) - for t in ToolRegistry.list_tools() - ) + items = tuple(_build_tool_index_item(t) for t in ToolRegistry.list_tools()) _tool_summary_cache_key = cache_key _tool_summary_cache_expires_at = now + _TOOL_SUMMARY_CACHE_TTL_SECONDS _tool_summary_cache_items = items @@ -253,13 +269,51 @@ def _build_tool_response(t: ToolInfo, *, include_parameters: bool = True) -> Too vendor=t.vendor, parameters=parameters, parameters_count=len(t.parameters), - enabled=_get_effective_tool_enabled(t), + enabled=_get_effective_tool_enabled(t, source=source, source_name=source_name), enabled_default=enabled_default, enabled_customized=customized, requires_confirmation=t.requires_confirmation, ) +def _build_tool_index_item(t: ToolInfo) -> ToolListIndexItem: + """Build the lightweight index row used by the paged list endpoint.""" + source, source_name = _get_tool_source(t) + setting = ConfigWriter.get_tool_setting(t.name) or {} + return ToolListIndexItem( + name=t.name, + description=t.description, + description_cn=t.description_cn, + category=t.category.value, + source=source, + source_name=source_name, + vendor=t.vendor, + parameters_count=len(t.parameters), + enabled=_get_effective_tool_enabled(t, source=source, source_name=source_name), + enabled_default=_get_default_enabled(t), + enabled_customized="enabled" in setting, + requires_confirmation=t.requires_confirmation, + ) + + +def _tool_index_item_to_response(item: ToolListIndexItem) -> ToolInfoResponse: + return ToolInfoResponse( + name=item.name, + description=item.description, + description_cn=item.description_cn, + category=item.category, + source=item.source, + source_name=item.source_name, + vendor=item.vendor, + parameters=[], + parameters_count=item.parameters_count, + enabled=item.enabled, + enabled_default=item.enabled_default, + enabled_customized=item.enabled_customized, + requires_confirmation=item.requires_confirmation, + ) + + def _split_csv_filter(value: Optional[str]) -> Optional[set[str]]: if value is None: return None @@ -267,7 +321,7 @@ def _split_csv_filter(value: Optional[str]) -> Optional[set[str]]: return parts or None -def _matches_tool_query(tool: ToolInfoResponse, query: str) -> bool: +def _matches_tool_query(tool: ToolInfoResponse | ToolListIndexItem, query: str) -> bool: if not query: return True haystack = " ".join([ @@ -282,7 +336,7 @@ def _matches_tool_query(tool: ToolInfoResponse, query: str) -> bool: return query in haystack -def _build_tool_facets(items: List[ToolInfoResponse]) -> ToolListFacets: +def _build_tool_facets(items: Sequence[ToolInfoResponse | ToolListIndexItem]) -> ToolListFacets: facets = ToolListFacets() for item in items: facets.category[item.category] = facets.category.get(item.category, 0) + 1 @@ -295,10 +349,10 @@ def _build_tool_facets(items: List[ToolInfoResponse]) -> ToolListFacets: def _sort_tool_items( - items: List[ToolInfoResponse], + items: Sequence[ToolInfoResponse | ToolListIndexItem], sort_by: Literal["category", "source", "source_name", "enabled", "name"], sort_dir: Literal["asc", "desc"], -) -> List[ToolInfoResponse]: +) -> List[ToolInfoResponse | ToolListIndexItem]: reverse = sort_dir == "desc" def sort_key(item: ToolInfoResponse): @@ -312,7 +366,7 @@ def sort_key(item: ToolInfoResponse): def _filter_tool_items( - items: List[ToolInfoResponse], + items: Sequence[ToolInfoResponse | ToolListIndexItem], *, category_filter: Optional[set[str]], source_filter: Optional[set[str]], @@ -323,8 +377,8 @@ def _filter_tool_items( include_source: bool = True, include_source_name: bool = True, include_enabled: bool = True, -) -> List[ToolInfoResponse]: - result = items +) -> List[ToolInfoResponse | ToolListIndexItem]: + result = list(items) if include_category and category_filter: result = [tool for tool in result if tool.category in category_filter] if include_source and source_filter: @@ -554,9 +608,15 @@ def _service_allows_enable(t: ToolInfo) -> bool: return bool(svc.get("enabled", False)) -def _get_effective_tool_enabled(tool_info: ToolInfo) -> bool: +def _get_effective_tool_enabled( + tool_info: ToolInfo, + *, + source: Optional[str] = None, + source_name: Optional[str] = None, +) -> bool: """Compute tool enabled state without mutating the registry object.""" - source, source_name = _get_tool_source(tool_info) + if source is None: + source, source_name = _get_tool_source(tool_info) if source not in ("api", "device") or not source_name: return tool_info.enabled from flocks.server.routes.provider import _get_api_service_enabled @@ -660,7 +720,7 @@ async def list_tools_page( sort_by: Literal["category", "source", "source_name", "enabled", "name"] = "source", sort_dir: Literal["asc", "desc"] = "asc", offset: int = Query(0, ge=0), - limit: int = Query(20, ge=1, le=200), + limit: int = Query(25, ge=1, le=200), ): """ List tools with server-side search/filter/sort/pagination. @@ -727,7 +787,10 @@ async def list_tools_page( ) result = _sort_tool_items(result, sort_by, sort_dir) total = len(result) - items = result[offset:offset + limit] + items = [ + _tool_index_item_to_response(item) if isinstance(item, ToolListIndexItem) else item + for item in result[offset:offset + limit] + ] log_route_timing(log, "tools.list_page.complete", started_at=started_at, extra={ "count": len(items), diff --git a/tests/provider/test_api_service_management.py b/tests/provider/test_api_service_management.py index 1b90da15e..4647fb8df 100644 --- a/tests/provider/test_api_service_management.py +++ b/tests/provider/test_api_service_management.py @@ -33,7 +33,7 @@ async def test_list_api_services_returns_enabled_state_and_bilingual_description "threatbook_api": {"enabled": False}, }.get(service_id, {})), patch( - "flocks.server.routes.provider._load_api_service_metadata_data", + "flocks.server.routes.provider._load_api_service_summary_metadata_data", side_effect=lambda service_id: metadata_by_id[service_id], ), patch( @@ -66,6 +66,35 @@ async def test_list_api_services_returns_enabled_state_and_bilingual_description assert result[1].status == "disabled" assert result[1].tool_count == 2 + def test_api_service_summary_uses_lightweight_metadata_loader(self): + from flocks.server.routes import provider as provider_routes + + with ( + patch( + "flocks.server.routes.provider._load_api_service_summary_metadata_data", + return_value={ + "name": "Example API", + "description": "Summary metadata only", + "version": "1.2.3", + }, + ) as mock_summary_loader, + patch( + "flocks.server.routes.provider._load_api_service_metadata_data", + side_effect=AssertionError("list summaries should not load full API metadata"), + ), + patch("flocks.server.routes.provider._get_api_service_enabled", return_value=True), + patch("flocks.server.routes.provider._get_api_service_tool_infos", return_value=[]), + patch("flocks.server.routes.provider._is_api_service_builtin", return_value=False), + patch("flocks.config.config_writer.ConfigWriter.get_api_service_raw", return_value={}), + ): + summary = provider_routes._build_api_service_summary("example_api", {}) + + mock_summary_loader.assert_called_once_with("example_api") + assert summary.id == "example_api" + assert summary.name == "Example API" + assert summary.version == "1.2.3" + assert summary.tool_count == 0 + @pytest.mark.asyncio async def test_update_api_service_persists_enabled_flag_and_updates_status_cache(self): from flocks.server.routes.provider import ( @@ -394,6 +423,58 @@ def test_load_api_service_metadata_uses_config_writer_raw_data(self): assert metadata["base_url"] == "https://api.example.test" assert metadata["enabled"] is False + def test_load_api_service_summary_metadata_reuses_cached_yaml(self, tmp_path, monkeypatch): + from flocks.server.routes import provider as provider_routes + + provider_routes._clear_api_service_summary_metadata_cache() + monkeypatch.setattr(provider_routes.api_service_schema_helpers, "_LEGACY_METADATA_DIR", tmp_path) + try: + with ( + patch( + "flocks.config.config_writer.ConfigWriter.get_api_service_raw", + return_value={"enabled": True}, + ), + patch( + "flocks.server.routes.provider._provider_yaml_cache_key", + return_value=("/plugins/example/_provider.yaml", 123), + ), + patch( + "flocks.server.routes.provider._load_provider_yaml_summary_metadata", + return_value={ + "name": "Example Service", + "description": "Loaded from provider yaml", + }, + ) as mock_yaml_loader, + ): + first = provider_routes._load_api_service_summary_metadata_data("example_service") + second = provider_routes._load_api_service_summary_metadata_data("example_service") + + mock_yaml_loader.assert_called_once_with("example_service") + assert first == second + assert first["name"] == "Example Service" + assert first["enabled"] is True + finally: + provider_routes._clear_api_service_summary_metadata_cache() + + def test_summary_descriptor_lookup_uses_cached_discovery(self, monkeypatch): + from flocks.server.routes import provider as provider_routes + + calls = [] + + def discover_api_service_descriptors(*, refresh=False): + calls.append(refresh) + if refresh: + raise AssertionError("summary lookup should not refresh descriptor discovery") + return [] + + monkeypatch.setattr( + "flocks.config.api_versioning.discover_api_service_descriptors", + discover_api_service_descriptors, + ) + + assert provider_routes._find_api_service_descriptor("missing_service") is None + assert calls == [False] + def test_load_provider_yaml_metadata_from_project_plugins(self, tmp_path, monkeypatch): from flocks.server.routes.provider import _load_provider_yaml_metadata diff --git a/tests/server/routes/test_tool_routes.py b/tests/server/routes/test_tool_routes.py index b3f2c0232..3c0786495 100644 --- a/tests/server/routes/test_tool_routes.py +++ b/tests/server/routes/test_tool_routes.py @@ -479,28 +479,27 @@ async def handler(ctx) -> ToolResult: handler=handler, ) - original_build_tool_response = tool_routes._build_tool_response - summary_builds = 0 + original_build_tool_index_item = tool_routes._build_tool_index_item + index_builds = 0 - def counted_build_tool_response(tool_info, *, include_parameters=True): - nonlocal summary_builds - if not include_parameters: - summary_builds += 1 - return original_build_tool_response(tool_info, include_parameters=include_parameters) + def counted_build_tool_index_item(tool_info): + nonlocal index_builds + index_builds += 1 + return original_build_tool_index_item(tool_info) - monkeypatch.setattr(tool_routes, "_build_tool_response", counted_build_tool_response) + monkeypatch.setattr(tool_routes, "_build_tool_index_item", counted_build_tool_index_item) with _temporary_tool(tool): first = await client.get( "/api/tools/page", params={"q": "needlepagesummarycacheunique", "limit": 25}, ) - first_builds = summary_builds + first_builds = index_builds second = await client.get( "/api/tools/page", params={"q": "needlepagesummarycacheunique", "source": "plugin_py", "limit": 25}, ) - second_builds = summary_builds + second_builds = index_builds tool_routes._invalidate_tool_summary_cache() third = await client.get( @@ -516,4 +515,4 @@ def counted_build_tool_response(tool_info, *, include_parameters=True): assert third.json()["total"] == 1 assert first_builds > 0 assert second_builds == first_builds - assert summary_builds > second_builds + assert index_builds > second_builds diff --git a/webui/src/api/provider.test.ts b/webui/src/api/provider.test.ts new file mode 100644 index 000000000..45a9c8a9e --- /dev/null +++ b/webui/src/api/provider.test.ts @@ -0,0 +1,90 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; + +const mockGet = vi.fn(); +const mockPost = vi.fn(); +const mockPatch = vi.fn(); +const mockDelete = vi.fn(); +const mockPut = vi.fn(); + +vi.mock('./client', () => ({ + default: { + get: (...args: unknown[]) => mockGet(...args), + post: (...args: unknown[]) => mockPost(...args), + patch: (...args: unknown[]) => mockPatch(...args), + delete: (...args: unknown[]) => mockDelete(...args), + put: (...args: unknown[]) => mockPut(...args), + }, +})); + +function serviceSummary(id: string, name: string) { + return { + id, + name, + enabled: true, + status: 'unknown', + tool_count: 0, + verify_ssl: false, + }; +} + +describe('providerAPI.listApiServices', () => { + afterEach(() => { + vi.resetModules(); + vi.useRealTimers(); + mockGet.mockReset(); + mockPost.mockReset(); + mockPatch.mockReset(); + mockDelete.mockReset(); + mockPut.mockReset(); + }); + + it('shares concurrent API service list requests', async () => { + let resolveRequest: (value: { data: ReturnType[] }) => void = () => undefined; + mockGet.mockImplementation( + () => new Promise((resolve) => { + resolveRequest = resolve; + }), + ); + + const { providerAPI } = await import('./provider'); + const first = providerAPI.listApiServices(); + const second = providerAPI.listApiServices(); + + expect(mockGet).toHaveBeenCalledTimes(1); + expect(mockGet).toHaveBeenCalledWith('/api/provider/api-services'); + + resolveRequest({ data: [serviceSummary('svc-a', 'Service A')] }); + const [firstResponse, secondResponse] = await Promise.all([first, second]); + + firstResponse.data[0].name = 'Mutated'; + expect(secondResponse.data[0].name).toBe('Service A'); + }); + + it('serves a fresh cached response without another request', async () => { + mockGet.mockResolvedValue({ data: [serviceSummary('svc-a', 'Service A')] }); + + const { providerAPI } = await import('./provider'); + const firstResponse = await providerAPI.listApiServices(); + firstResponse.data[0].name = 'Mutated'; + const secondResponse = await providerAPI.listApiServices(); + + expect(mockGet).toHaveBeenCalledTimes(1); + expect(secondResponse.data[0].name).toBe('Service A'); + }); + + it('invalidates the cached list after an API service update', async () => { + mockGet + .mockResolvedValueOnce({ data: [serviceSummary('svc-a', 'Service A')] }) + .mockResolvedValueOnce({ data: [serviceSummary('svc-a', 'Service A Updated')] }); + mockPatch.mockResolvedValue({ data: serviceSummary('svc-a', 'Service A Updated') }); + + const { providerAPI } = await import('./provider'); + await providerAPI.listApiServices(); + await providerAPI.updateApiService('svc-a', { enabled: false }); + const response = await providerAPI.listApiServices(); + + expect(mockPatch).toHaveBeenCalledWith('/api/provider/api-services/svc-a', { enabled: false }); + expect(mockGet).toHaveBeenCalledTimes(2); + expect(response.data[0].name).toBe('Service A Updated'); + }); +}); diff --git a/webui/src/api/provider.ts b/webui/src/api/provider.ts index 5eb77ecb6..c4fd8cec1 100644 --- a/webui/src/api/provider.ts +++ b/webui/src/api/provider.ts @@ -16,6 +16,53 @@ import type { APIServiceMetadata, } from '@/types'; +const API_SERVICES_LIST_CACHE_TTL_MS = 5000; + +function requestApiServicesList() { + return client.get('/api/provider/api-services'); +} + +type APIServiceListResponse = Awaited>; + +let apiServicesListInFlight: Promise | null = null; +let apiServicesListCache: { response: APIServiceListResponse; updatedAt: number } | null = null; + +function cloneApiServicesResponse(response: APIServiceListResponse): APIServiceListResponse { + const services = Array.isArray(response.data) ? response.data : []; + return { + ...response, + data: services.map((service) => ({ ...service })), + }; +} + +function invalidateApiServicesListCache(): void { + apiServicesListCache = null; +} + +function listApiServicesCached(options: { force?: boolean } = {}) { + const useCache = !options.force; + if (useCache && apiServicesListCache && Date.now() - apiServicesListCache.updatedAt < API_SERVICES_LIST_CACHE_TTL_MS) { + return Promise.resolve(cloneApiServicesResponse(apiServicesListCache.response)); + } + if (useCache && apiServicesListInFlight) { + return apiServicesListInFlight.then(cloneApiServicesResponse); + } + + apiServicesListInFlight = requestApiServicesList() + .then((response) => { + apiServicesListCache = { + response: cloneApiServicesResponse(response), + updatedAt: Date.now(), + }; + return response; + }) + .finally(() => { + apiServicesListInFlight = null; + }); + + return apiServicesListInFlight.then(cloneApiServicesResponse); +} + // ==================== Provider API (Legacy + Enhanced) ==================== export const providerAPI = { @@ -54,7 +101,11 @@ export const providerAPI = { client.get(`/api/provider/${id}/service-credentials`), setServiceCredentials: (id: string, credentials: ProviderCredentialInput) => - client.post<{ success: boolean; message: string }>(`/api/provider/${id}/service-credentials`, credentials), + client.post<{ success: boolean; message: string }>(`/api/provider/${id}/service-credentials`, credentials) + .then((response) => { + invalidateApiServicesListCache(); + return response; + }), deleteCredentials: (id: string) => client.delete<{ success: boolean }>(`/api/provider/${id}/credentials`), @@ -70,17 +121,25 @@ export const providerAPI = { ), // API service status (connectivity) - listApiServices: () => - client.get('/api/provider/api-services'), + listApiServices: (options?: { force?: boolean }) => + listApiServicesCached(options), getServiceMetadata: (id: string) => client.get(`/api/provider/${id}/metadata`), updateApiService: (id: string, data: { enabled: boolean; verify_ssl?: boolean }) => - client.patch(`/api/provider/api-services/${id}`, data), + client.patch(`/api/provider/api-services/${id}`, data) + .then((response) => { + invalidateApiServicesListCache(); + return response; + }), deleteApiService: (id: string) => - client.delete<{ success: boolean }>(`/api/provider/api-services/${id}`), + client.delete<{ success: boolean }>(`/api/provider/api-services/${id}`) + .then((response) => { + invalidateApiServicesListCache(); + return response; + }), getApiServiceStatuses: () => client.get>( @@ -90,7 +149,10 @@ export const providerAPI = { refreshApiServiceStatuses: () => client.post<{ statuses: Record; refreshed_at: number }>( '/api/provider/api-services/refresh' - ), + ).then((response) => { + invalidateApiServicesListCache(); + return response; + }), getApiServiceStatus: (id: string) => client.get<{ status: string; message?: string; latency_ms?: number; tool_tested?: string; error?: string; checked_at?: number }>( diff --git a/webui/src/hooks/useTools.test.tsx b/webui/src/hooks/useTools.test.tsx index ea51948db..6698f0eed 100644 --- a/webui/src/hooks/useTools.test.tsx +++ b/webui/src/hooks/useTools.test.tsx @@ -187,6 +187,37 @@ describe('useTools', () => { }); }); + it('defaults paged requests to 25 items', async () => { + listPageMock.mockResolvedValue({ + data: { + items: [], + total: 0, + offset: 0, + limit: 25, + facets: emptyFacets, + }, + }); + + const { result } = renderHook(() => useToolPage({ source: 'api' })); + + await waitFor(() => { + expect(result.current.loading).toBe(false); + }); + + expect(result.current.limit).toBe(25); + expect(listPageMock).toHaveBeenCalledWith({ + source: 'api', + category: '', + sourceName: '', + enabled: '', + q: '', + sortBy: 'source', + sortDir: 'asc', + offset: 0, + limit: 25, + }); + }); + it('shares the same paged request across concurrent hook instances', async () => { let resolvePage: (value: { data: any }) => void = () => {}; listPageMock.mockReturnValue(new Promise((resolve) => { diff --git a/webui/src/hooks/useTools.ts b/webui/src/hooks/useTools.ts index 724929284..2691a0f07 100644 --- a/webui/src/hooks/useTools.ts +++ b/webui/src/hooks/useTools.ts @@ -16,6 +16,7 @@ import { const TOOL_LIST_STALE_TIME_MS = 5000; const TOOL_LIST_MIN_FETCH_INTERVAL_MS = 1000; const MAX_TOOL_PAGE_RESOURCES = 80; +const DEFAULT_TOOL_PAGE_LIMIT = 25; const EMPTY_TOOL_FACETS: ToolListFacets = { category: {}, @@ -28,7 +29,7 @@ const EMPTY_TOOL_PAGE: ToolListPageResponse = { items: [], total: 0, offset: 0, - limit: 20, + limit: DEFAULT_TOOL_PAGE_LIMIT, facets: EMPTY_TOOL_FACETS, }; @@ -56,7 +57,7 @@ function normalizeToolPageParams(params: ToolListPageParams): Required = { device: 6, }; -const PAGE_SIZE = 20; +const PAGE_SIZE = TOOL_PAGE_SIZE; /** MCP/API 服务器详情抽屉宽度(px) */ const DETAIL_DRAWER_WIDTH = 560; @@ -156,12 +160,12 @@ export default function ToolPage() { const TABS: TabConfig[] = [ { key: 'all', label: t('tabs.all'), icon: }, - { key: 'mcp', label: t('tabs.mcp'), icon: , sourceFilter: 'mcp' }, - { key: 'api', label: t('tabs.api'), icon: , sourceFilter: 'api' }, - { key: 'local', label: t('tabs.local'), icon: , sourceFilter: 'plugin_py' }, + { key: 'mcp', label: t('tabs.mcp'), icon: }, + { key: 'api', label: t('tabs.api'), icon: }, + { key: 'local', label: t('tabs.local'), icon: }, ]; - const [activeTab, setActiveTab] = useState('all'); + const [activeTab, setActiveTab] = useState(DEFAULT_TOOL_TAB); const [searchQuery, setSearchQuery] = useState(''); const [selectedTool, setSelectedTool] = useState(null); const [testParams, setTestParams] = useState('{}'); @@ -182,8 +186,7 @@ export default function ToolPage() { const [apiEnabledServicesCount, setApiEnabledServicesCount] = useState(0); const debouncedSearchQuery = useDebouncedValue(searchQuery, 250); - const activeTabConfig = TABS.find((tab) => tab.key === activeTab); - const tabSourceFilter = activeTabConfig?.sourceFilter; + const tabSourceFilter = getTabSourceFilter(activeTab); const sourceFilterParam = useMemo(() => { const values = new Set(filters.source); if (tabSourceFilter) { @@ -224,10 +227,14 @@ export default function ToolPage() { // Catalog data (fetched once at top level, shared with MCP & API tabs) const [catalogEntries, setCatalogEntries] = useState([]); const [catalogCategories, setCatalogCategories] = useState>({}); - const [catalogLoading, setCatalogLoading] = useState(true); + const [catalogLoading, setCatalogLoading] = useState(false); const [configuredIds, setConfiguredIds] = useState>(new Set()); + const catalogLoadedRef = useRef(false); useEffect(() => { + if (!shouldLoadMcpCatalog(activeTab) || catalogLoadedRef.current) return; + let cancelled = false; + const loadCatalog = async () => { try { setCatalogLoading(true); @@ -235,21 +242,35 @@ export default function ToolPage() { mcpAPI.catalogList(), mcpAPI.catalogCategories(), ]); + if (cancelled) return; setCatalogEntries(entriesRes.data); setCatalogCategories(catsRes.data); // Get currently configured IDs (no auto-setup to avoid re-adding removed entries) try { const confRes = await mcpAPI.catalogConfigured(); - setConfiguredIds(new Set(confRes.data)); + if (!cancelled) setConfiguredIds(new Set(confRes.data)); } catch { /* ignore */ } } catch (err) { console.error('Failed to load catalog:', err); } finally { - setCatalogLoading(false); + if (!cancelled) { + catalogLoadedRef.current = true; + setCatalogLoading(false); + } } }; - loadCatalog(); - }, []); + void loadCatalog(); + + return () => { + cancelled = true; + }; + }, [activeTab]); + + useEffect(() => { + if (!shouldLoadMcpCatalog(activeTab)) { + setCatalogLoading(false); + } + }, [activeTab]); const onConfiguredChange = useCallback((id: string) => { setConfiguredIds(prev => new Set(prev).add(id)); @@ -296,6 +317,10 @@ export default function ToolPage() { local: toolFacets.source.plugin_py ?? 0, }; }, [totalTools, toolFacets.source, apiEnabledServicesCount]); + const enabledSummary = useMemo(() => ({ + active: toolFacets.enabled['true'] ?? tools.filter((tool) => tool.enabled).length, + inactive: toolFacets.enabled['false'] ?? 0, + }), [toolFacets.enabled, tools]); // Filter dropdown options come from server-side facets, plus active values // so a selected filter remains visible even when it narrows the result set. @@ -435,9 +460,9 @@ export default function ToolPage() {

{t('pageTitle')}

- {tools.length} {t('statusBadge.active')} + {enabledSummary.active} {t('statusBadge.active')} · - {catalogEntries.length} {t('statusBadge.inactive')} + {enabledSummary.inactive} {t('statusBadge.inactive')}

diff --git a/webui/src/pages/Tool/tabLoading.test.ts b/webui/src/pages/Tool/tabLoading.test.ts new file mode 100644 index 000000000..8d30c0407 --- /dev/null +++ b/webui/src/pages/Tool/tabLoading.test.ts @@ -0,0 +1,29 @@ +import { describe, expect, it } from 'vitest'; +import { + DEFAULT_TOOL_TAB, + TOOL_PAGE_SIZE, + getTabSourceFilter, + shouldLoadMcpCatalog, +} from './tabLoading'; + +describe('tool tab loading strategy', () => { + it('starts from all tools with 25 visible rows', () => { + expect(DEFAULT_TOOL_TAB).toBe('all'); + expect(TOOL_PAGE_SIZE).toBe(25); + expect(getTabSourceFilter(DEFAULT_TOOL_TAB)).toBeUndefined(); + }); + + it('loads each source group only for its tab', () => { + expect(getTabSourceFilter('all')).toBeUndefined(); + expect(getTabSourceFilter('mcp')).toBe('mcp'); + expect(getTabSourceFilter('api')).toBe('api'); + expect(getTabSourceFilter('local')).toBe('plugin_py'); + }); + + it('defers MCP catalog requests until the MCP tab is opened', () => { + expect(shouldLoadMcpCatalog('all')).toBe(false); + expect(shouldLoadMcpCatalog('api')).toBe(false); + expect(shouldLoadMcpCatalog('local')).toBe(false); + expect(shouldLoadMcpCatalog('mcp')).toBe(true); + }); +}); diff --git a/webui/src/pages/Tool/tabLoading.ts b/webui/src/pages/Tool/tabLoading.ts new file mode 100644 index 000000000..ef3501115 --- /dev/null +++ b/webui/src/pages/Tool/tabLoading.ts @@ -0,0 +1,17 @@ +import type { ToolSource } from '@/api/tool'; + +export type TabKey = 'all' | 'mcp' | 'api' | 'local'; + +export const DEFAULT_TOOL_TAB: TabKey = 'all'; +export const TOOL_PAGE_SIZE = 25; + +export function getTabSourceFilter(tab: TabKey): ToolSource | ToolSource[] | undefined { + if (tab === 'mcp') return 'mcp'; + if (tab === 'api') return 'api'; + if (tab === 'local') return 'plugin_py'; + return undefined; +} + +export function shouldLoadMcpCatalog(tab: TabKey): boolean { + return tab === 'mcp'; +} From 72463e7d293f21e295d8f651f68eaccbc6bcf621 Mon Sep 17 00:00:00 2001 From: duguwanglong Date: Thu, 9 Jul 2026 18:21:15 +0800 Subject: [PATCH 07/49] perf(webui): reduce startup API load --- flocks/server/app.py | 5 + flocks/server/routes/monitoring.py | 156 ++++++++++++++++++ flocks/server/routes/session.py | 50 +++++- flocks/server/routes/stats.py | 140 ++++++++++++++++ flocks/server/routes/update.py | 67 +++++++- tests/server/routes/test_update_routes.py | 57 +++++++ tests/server/routes/test_webui_perf_routes.py | 103 ++++++++++++ webui/src/api/stats.test.ts | 22 +++ webui/src/api/stats.ts | 128 ++++++++------ webui/src/api/update.ts | 7 +- webui/src/components/layout/Layout.test.tsx | 8 +- webui/src/components/layout/Layout.tsx | 6 +- 12 files changed, 692 insertions(+), 57 deletions(-) create mode 100644 flocks/server/routes/monitoring.py create mode 100644 flocks/server/routes/stats.py create mode 100644 tests/server/routes/test_webui_perf_routes.py diff --git a/flocks/server/app.py b/flocks/server/app.py index f557c79e8..119faa919 100644 --- a/flocks/server/app.py +++ b/flocks/server/app.py @@ -1059,6 +1059,8 @@ async def general_exception_handler(request: Request, exc: Exception): from flocks.server.routes.update import router as update_router # Log viewing from flocks.server.routes.logs import router as logs_router +from flocks.server.routes.monitoring import router as monitoring_router +from flocks.server.routes.stats import router as stats_router from flocks.server.routes.auth import router as auth_router from flocks.server.routes.admin_users import router as admin_users_router from flocks.server.routes.notifications import router as notifications_router @@ -1120,6 +1122,8 @@ async def general_exception_handler(request: Request, exc: Exception): app.include_router(update_router, prefix="/api/update", tags=["Update"]) # Log viewing routes app.include_router(logs_router, prefix="/api/logs", tags=["Logs"]) +app.include_router(stats_router, prefix="/api/stats", tags=["Stats"]) +app.include_router(monitoring_router, prefix="/api/monitoring", tags=["Monitoring"]) app.include_router(auth_router, prefix="/api/auth", tags=["Auth"]) app.include_router(admin_users_router, prefix="/api/admin", tags=["Admin"]) app.include_router(notifications_router, prefix="/api/notifications", tags=["Notifications"]) @@ -1182,6 +1186,7 @@ async def general_exception_handler(request: Request, exc: Exception): # Permission routes (/permission) app.include_router(permission_router, prefix="/permission", tags=["Permission"]) +app.include_router(permission_router, prefix="/api/permission", tags=["Permission"]) # Question routes (/question) app.include_router(question_router, prefix="/question", tags=["Question"]) diff --git a/flocks/server/routes/monitoring.py b/flocks/server/routes/monitoring.py new file mode 100644 index 000000000..4b9e331cc --- /dev/null +++ b/flocks/server/routes/monitoring.py @@ -0,0 +1,156 @@ +""" +Lightweight WebUI monitoring routes. + +These endpoints back the existing Monitoring page without forcing the page to +poll heavier session/tool list APIs. Detailed historical sampling can be added +later without changing the frontend contract. +""" + +from __future__ import annotations + +import time +from typing import Any, Literal + +from fastapi import APIRouter, Query +from pydantic import BaseModel + +from flocks.agent import registry as agent_registry +from flocks.mcp import MCP +from flocks.session.session import Session +from flocks.utils.log import Log +from flocks.utils.monitor import get_monitor + +router = APIRouter() +log = Log.create(service="monitoring-routes") +_started_at = time.monotonic() + + +class SystemStatusResponse(BaseModel): + status: Literal["healthy", "degraded", "down"] + uptime: int + activeSessions: int + activeAgents: int + mcpServers: dict[str, str] + timestamp: int + + +class MetricsSnapshotResponse(BaseModel): + timestamp: int + messageRate: float + toolCallRate: float + errorRate: float + avgResponseTime: float + activeRequests: int + + +class PerformanceDataResponse(BaseModel): + category: Literal["llm", "tool", "api"] + name: str + avgDuration: float + p50: float | None = None + p95: float | None = None + p99: float | None = None + count: int + errors: int + + +def _timestamp_ms() -> int: + return int(time.time() * 1000) + + +async def _active_session_count() -> int: + cached_sessions = getattr(Session, "_all_sessions_cache", None) + if not cached_sessions: + return 0 + return sum(1 for session in cached_sessions if getattr(session, "status", None) == "active") + + +async def _active_agent_count() -> int: + agents = getattr(agent_registry, "_agents_ref", None) + if not agents: + return 0 + return sum(1 for agent in agents.values() if not getattr(agent, "hidden", False)) + + +async def _mcp_server_statuses() -> dict[str, str]: + try: + statuses = await MCP.status() + except Exception as exc: + log.warning("monitoring.mcp.status_failed", {"error": str(exc)}) + return {} + return { + name: str(info.status.value if hasattr(info.status, "value") else info.status) + for name, info in statuses.items() + } + + +@router.get("/status", response_model=SystemStatusResponse) +async def get_status() -> SystemStatusResponse: + return SystemStatusResponse( + status="healthy", + uptime=max(0, int(time.monotonic() - _started_at)), + activeSessions=await _active_session_count(), + activeAgents=await _active_agent_count(), + mcpServers=await _mcp_server_statuses(), + timestamp=_timestamp_ms(), + ) + + +@router.get("/metrics", response_model=MetricsSnapshotResponse) +async def get_metrics() -> MetricsSnapshotResponse: + metrics = get_monitor().get_metrics().get("global", {}) + total_calls = float(metrics.get("total_calls") or 0) + failed_calls = float(metrics.get("failed_parses") or 0) + return MetricsSnapshotResponse( + timestamp=_timestamp_ms(), + messageRate=0.0, + toolCallRate=total_calls, + errorRate=failed_calls, + avgResponseTime=0.0, + activeRequests=0, + ) + + +@router.get("/metrics/history") +async def get_metrics_history( + duration: int = Query(300, ge=1), + interval: int = Query(5, ge=1), +) -> dict[str, Any]: + return {"duration": duration, "interval": interval, "items": []} + + +@router.get("/performance", response_model=list[PerformanceDataResponse]) +async def get_performance( + category: Literal["llm", "tool", "api"] | None = None, +) -> list[PerformanceDataResponse]: + return [] + + +@router.get("/performance/llm", response_model=list[PerformanceDataResponse]) +async def get_llm_performance() -> list[PerformanceDataResponse]: + return [] + + +@router.get("/performance/tool", response_model=list[PerformanceDataResponse]) +async def get_tool_performance() -> list[PerformanceDataResponse]: + return [] + + +@router.get("/api-stats") +async def get_api_stats() -> dict[str, Any]: + return {"items": []} + + +@router.get("/api-stats/history") +async def get_api_stats_history(duration: int = Query(300, ge=1)) -> dict[str, Any]: + return {"duration": duration, "items": []} + + +@router.get("/events") +async def get_events( + level: Literal["info", "warn", "error"] | None = None, + service: str | None = None, + limit: int = Query(50, ge=1, le=200), + offset: int = Query(0, ge=0), +) -> list[dict[str, Any]]: + return [] diff --git a/flocks/server/routes/session.py b/flocks/server/routes/session.py index 7d7858b2d..f6152cd1d 100644 --- a/flocks/server/routes/session.py +++ b/flocks/server/routes/session.py @@ -37,6 +37,10 @@ DEFAULT_AGENT = "rex" DEFAULT_MESSAGE_PAGE_LIMIT = 50 _DESCENDANT_ABORT_SCAN_LIMIT = 3 +_CONTEXT_USAGE_CACHE_TTL_SECONDS = 5.0 +_context_usage_cache: Dict[Tuple[str, int], Tuple[float, ContextUsageSnapshot]] = {} +_context_usage_inflight: Dict[Tuple[str, int], asyncio.Task[ContextUsageSnapshot]] = {} +_context_usage_cache_lock = asyncio.Lock() # File extensions that are safe to persist when materialising data-URL uploads. # Intentionally narrow: any extension outside this set is rejected to prevent @@ -45,6 +49,50 @@ # ".exe"). _UPLOAD_SAFE_EXTS = frozenset({"png", "jpg", "jpeg", "gif", "webp", "bmp", "pdf"}) + +def _context_usage_cache_key(session_id: str, session: SessionModel) -> Tuple[str, int]: + return session_id, int(getattr(session.time, "updated", 0) or 0) + + +async def _cached_context_usage_snapshot( + session_id: str, + *, + session: SessionModel, +) -> ContextUsageSnapshot: + key = _context_usage_cache_key(session_id, session) + now = time.monotonic() + owner = False + + async with _context_usage_cache_lock: + cached = _context_usage_cache.get(key) + if cached and cached[0] > now: + return cached[1].model_copy(deep=True) + + task = _context_usage_inflight.get(key) + if task is None: + task = asyncio.create_task(build_context_usage_snapshot(session_id, session=session)) + _context_usage_inflight[key] = task + owner = True + + try: + snapshot = await task + finally: + if owner: + async with _context_usage_cache_lock: + if _context_usage_inflight.get(key) is task: + _context_usage_inflight.pop(key, None) + + if owner: + async with _context_usage_cache_lock: + for cache_key in [item for item in _context_usage_cache if item[0] == session_id and item != key]: + _context_usage_cache.pop(cache_key, None) + _context_usage_cache[key] = ( + time.monotonic() + _CONTEXT_USAGE_CACHE_TTL_SECONDS, + snapshot.model_copy(deep=True), + ) + + return snapshot.model_copy(deep=True) + # ============================================================================= # Request/Response Models - API Compatible (camelCase) # ============================================================================= @@ -597,7 +645,7 @@ async def get_session_context_usage(sessionID: str, request: Request) -> Context detail=f"Session {sessionID} not found", ) _require_session_read_access(session, current_user) - snapshot = await build_context_usage_snapshot(sessionID, session=session) + snapshot = await _cached_context_usage_snapshot(sessionID, session=session) log_route_timing(log, "session.context_usage.complete", started_at=started_at, extra={ "sessionID": sessionID, "source": snapshot.source, diff --git a/flocks/server/routes/stats.py b/flocks/server/routes/stats.py new file mode 100644 index 000000000..f2f3a4d57 --- /dev/null +++ b/flocks/server/routes/stats.py @@ -0,0 +1,140 @@ +""" +Lightweight WebUI summary stats. + +The home page needs counts, not the full agent/workflow/skill/tool payloads. +Keeping this aggregation on the server avoids serialising and parsing large +lists on every first paint. +""" + +from __future__ import annotations + +import asyncio +from typing import Any, Awaitable, Callable, Literal + +from fastapi import APIRouter +from pydantic import BaseModel + +from flocks.agent.registry import Agent +from flocks.server.routes.provider import list_providers +from flocks.server.routes.workflow import _list_workflows_from_fs, _migrate_storage_to_filesystem +from flocks.skill.skill import Skill +from flocks.task.manager import TaskManager +from flocks.tool.registry import ToolRegistry +from flocks.utils.log import Log + +router = APIRouter() +log = Log.create(service="stats-routes") + + +class CountResponse(BaseModel): + total: int + + +class TaskStatsResponse(BaseModel): + week: int + scheduledActive: int + + +class SystemHealthResponse(BaseModel): + status: Literal["healthy", "warning", "error"] + message: str + + +class SystemStatsResponse(BaseModel): + tasks: TaskStatsResponse + agents: CountResponse + workflows: CountResponse + skills: CountResponse + tools: CountResponse + models: CountResponse + system: SystemHealthResponse + + +async def _safe_count(name: str, loader: Callable[[], Awaitable[int]], failures: list[str]) -> int: + try: + return await loader() + except Exception as exc: + failures.append(name) + log.warning("stats.summary.source_failed", {"source": name, "error": str(exc)}) + return 0 + + +def _should_count_agent(agent: Any) -> bool: + if getattr(agent, "hidden", False): + return False + if getattr(agent, "mode", None) == "primary": + return True + tags = getattr(agent, "tags", None) + return not isinstance(tags, list) or "system" not in tags + + +async def _task_dashboard() -> dict[str, Any]: + return await TaskManager.dashboard() + + +async def _safe_dashboard(failures: list[str]) -> dict[str, Any]: + try: + return await _task_dashboard() + except Exception as exc: + failures.append("tasks") + log.warning("stats.summary.source_failed", {"source": "tasks", "error": str(exc)}) + return {} + + +async def _count_agents() -> int: + agents = await Agent.list() + return sum(1 for agent in agents if _should_count_agent(agent)) + + +async def _count_workflows() -> int: + await _migrate_storage_to_filesystem() + return len(_list_workflows_from_fs()) + + +async def _count_skills() -> int: + skills = await Skill.all() + return sum(1 for skill in skills if getattr(skill, "category", None) != "system") + + +async def _count_tools() -> int: + await ToolRegistry.init_async() + return len(ToolRegistry.all_tool_ids()) + + +async def _count_models() -> int: + providers = await list_providers() + connected = set(providers.connected or []) + return sum( + len(provider.models or {}) + for provider in providers.all + if provider.id in connected + ) + + +@router.get("/summary", response_model=SystemStatsResponse) +async def get_system_stats_summary() -> SystemStatsResponse: + failures: list[str] = [] + dashboard, agents, workflows, skills, tools, models = await asyncio.gather( + _safe_dashboard(failures), + _safe_count("agents", _count_agents, failures), + _safe_count("workflows", _count_workflows, failures), + _safe_count("skills", _count_skills, failures), + _safe_count("tools", _count_tools, failures), + _safe_count("models", _count_models, failures), + ) + + system_status: Literal["healthy", "warning"] = "warning" if failures else "healthy" + message = "部分统计加载失败" if failures else "所有服务运行正常" + + return SystemStatsResponse( + tasks=TaskStatsResponse( + week=int(dashboard.get("completed_week") or 0) + int(dashboard.get("failed_week") or 0), + scheduledActive=int(dashboard.get("scheduled_active") or 0), + ), + agents=CountResponse(total=agents), + workflows=CountResponse(total=workflows), + skills=CountResponse(total=skills), + tools=CountResponse(total=tools), + models=CountResponse(total=models), + system=SystemHealthResponse(status=system_status, message=message), + ) diff --git a/flocks/server/routes/update.py b/flocks/server/routes/update.py index 9836dfe0b..c0c5f0519 100644 --- a/flocks/server/routes/update.py +++ b/flocks/server/routes/update.py @@ -4,7 +4,8 @@ import asyncio import json -from typing import Literal +import time +from typing import Annotated, Literal from fastapi import APIRouter, Query, Request from fastapi.responses import StreamingResponse @@ -17,6 +18,65 @@ router = APIRouter() log = Log.create(service="update-routes") +_UPDATE_CHECK_CACHE_TTL_SECONDS = 600.0 +_UPDATE_CHECK_ERROR_CACHE_TTL_SECONDS = 60.0 +_update_check_cache: dict[tuple[str, str], tuple[float, VersionInfo]] = {} +_update_check_inflight: dict[tuple[str, str], asyncio.Task[VersionInfo]] = {} +_update_check_lock = asyncio.Lock() + + +def _update_cache_key(locale: str | None, edition: str) -> tuple[str, str]: + return (locale or "", edition) + + +def clear_update_check_cache() -> None: + _update_check_cache.clear() + _update_check_inflight.clear() + + +async def _check_update_cached( + *, + locale: str | None, + edition: Literal["flocks", "flockspro"], + force: bool, +) -> VersionInfo: + key = _update_cache_key(locale, edition) + if force: + info = await check_update(locale=locale, force_console_manifest=(edition == "flockspro")) + ttl = _UPDATE_CHECK_ERROR_CACHE_TTL_SECONDS if info.error else _UPDATE_CHECK_CACHE_TTL_SECONDS + _update_check_cache[key] = (time.monotonic() + ttl, info.model_copy(deep=True)) + return info + + owner = False + now = time.monotonic() + async with _update_check_lock: + cached = _update_check_cache.get(key) + if cached and cached[0] > now: + return cached[1].model_copy(deep=True) + + task = _update_check_inflight.get(key) + if task is None: + task = asyncio.create_task( + check_update(locale=locale, force_console_manifest=(edition == "flockspro")) + ) + _update_check_inflight[key] = task + owner = True + + try: + info = await task + finally: + if owner: + async with _update_check_lock: + if _update_check_inflight.get(key) is task: + _update_check_inflight.pop(key, None) + + if owner: + ttl = _UPDATE_CHECK_ERROR_CACHE_TTL_SECONDS if info.error else _UPDATE_CHECK_CACHE_TTL_SECONDS + async with _update_check_lock: + _update_check_cache[key] = (time.monotonic() + ttl, info.model_copy(deep=True)) + + return info.model_copy(deep=True) + @router.get( "/check", @@ -33,10 +93,13 @@ async def check_version( default="flocks", description="Version channel to check. flockspro checks the Console Pro bundle manifest.", ), + force: Annotated[bool, Query( + description="Bypass the short server-side cache for an explicit manual check.", + )] = False, ) -> VersionInfo: if edition == "flockspro": require_admin(request) - return await check_update(locale=locale, force_console_manifest=(edition == "flockspro")) + return await _check_update_cached(locale=locale, edition=edition, force=force) @router.post( diff --git a/tests/server/routes/test_update_routes.py b/tests/server/routes/test_update_routes.py index 8be953356..8b1d8544f 100644 --- a/tests/server/routes/test_update_routes.py +++ b/tests/server/routes/test_update_routes.py @@ -1,6 +1,7 @@ from __future__ import annotations import os +import asyncio from urllib.parse import quote, unquote, urlparse import pytest @@ -19,6 +20,15 @@ def _request() -> Request: return Request({"type": "http", "method": "GET", "path": "/api/update/check", "headers": []}) +@pytest.fixture(autouse=True) +def _clear_update_cache(): + from flocks.server.routes import update as update_routes + + update_routes.clear_update_check_cache() + yield + update_routes.clear_update_check_cache() + + def _manual_real_upgrade_branch() -> str: branch_input = ( os.environ.get(_MANUAL_REAL_UPGRADE_BRANCH_ENV, "").strip() @@ -126,6 +136,53 @@ async def _fake_check_update(**kwargs): assert info.current_version == "v2026.5.9" +async def test_check_version_reuses_cached_flocks_result(monkeypatch: pytest.MonkeyPatch): + from flocks.server.routes import update as update_routes + from flocks.updater.models import VersionInfo + + calls = 0 + + async def _fake_check_update(**kwargs): + nonlocal calls + calls += 1 + assert kwargs == {"locale": "zh-CN", "force_console_manifest": False} + return VersionInfo(current_version=f"v{calls}") + + monkeypatch.setattr(update_routes, "check_update", _fake_check_update) + + first = await update_routes.check_version(_request(), locale="zh-CN", edition="flocks") + second = await update_routes.check_version(_request(), locale="zh-CN", edition="flocks") + forced = await update_routes.check_version(_request(), locale="zh-CN", edition="flocks", force=True) + + assert calls == 2 + assert first.current_version == "v1" + assert second.current_version == "v1" + assert forced.current_version == "v2" + + +async def test_check_version_deduplicates_inflight_requests(monkeypatch: pytest.MonkeyPatch): + from flocks.server.routes import update as update_routes + from flocks.updater.models import VersionInfo + + calls = 0 + + async def _fake_check_update(**kwargs): + nonlocal calls + calls += 1 + await asyncio.sleep(0.01) + return VersionInfo(current_version=f"v{calls}") + + monkeypatch.setattr(update_routes, "check_update", _fake_check_update) + + results = await asyncio.gather(*[ + update_routes.check_version(_request(), locale="zh-CN", edition="flocks") + for _ in range(5) + ]) + + assert calls == 1 + assert [item.current_version for item in results] == ["v1"] * 5 + + @pytest.mark.skipif( os.environ.get(_MANUAL_REAL_UPGRADE_ENV) != "1", reason=f"manual real upgrade test; set {_MANUAL_REAL_UPGRADE_ENV}=1 to enable", diff --git a/tests/server/routes/test_webui_perf_routes.py b/tests/server/routes/test_webui_perf_routes.py new file mode 100644 index 000000000..32b700a2f --- /dev/null +++ b/tests/server/routes/test_webui_perf_routes.py @@ -0,0 +1,103 @@ +from __future__ import annotations + +from collections.abc import AsyncGenerator + +import pytest +from fastapi import FastAPI, status +from httpx import ASGITransport, AsyncClient + + +pytestmark = pytest.mark.asyncio + + +@pytest.fixture +async def webui_perf_client() -> AsyncGenerator[AsyncClient, None]: + from flocks.server.routes.monitoring import router as monitoring_router + from flocks.server.routes.permission import router as permission_router + from flocks.server.routes.stats import router as stats_router + + app = FastAPI() + app.include_router(stats_router, prefix="/api/stats") + app.include_router(monitoring_router, prefix="/api/monitoring") + app.include_router(permission_router, prefix="/api/permission") + + async with AsyncClient( + transport=ASGITransport(app=app), + base_url="http://test", + ) as client: + yield client + + +async def test_stats_summary_returns_lightweight_counts( + webui_perf_client: AsyncClient, + monkeypatch: pytest.MonkeyPatch, +): + from flocks.server.routes import stats as stats_routes + + async def _task_dashboard(): + return {"completed_week": 2, "failed_week": 1, "scheduled_active": 4} + + async def _count_agents(): + return 5 + + async def _count_workflows(): + return 6 + + async def _count_skills(): + return 7 + + async def _count_tools(): + return 8 + + async def _count_models(): + return 9 + + monkeypatch.setattr(stats_routes, "_task_dashboard", _task_dashboard) + monkeypatch.setattr(stats_routes, "_count_agents", _count_agents) + monkeypatch.setattr(stats_routes, "_count_workflows", _count_workflows) + monkeypatch.setattr(stats_routes, "_count_skills", _count_skills) + monkeypatch.setattr(stats_routes, "_count_tools", _count_tools) + monkeypatch.setattr(stats_routes, "_count_models", _count_models) + + response = await webui_perf_client.get("/api/stats/summary") + + assert response.status_code == status.HTTP_200_OK, response.text + assert response.json() == { + "tasks": {"week": 3, "scheduledActive": 4}, + "agents": {"total": 5}, + "workflows": {"total": 6}, + "skills": {"total": 7}, + "tools": {"total": 8}, + "models": {"total": 9}, + "system": {"status": "healthy", "message": "所有服务运行正常"}, + } + + +async def test_monitoring_routes_back_existing_webui_calls(webui_perf_client: AsyncClient): + status_response = await webui_perf_client.get("/api/monitoring/status") + metrics_response = await webui_perf_client.get("/api/monitoring/metrics") + llm_response = await webui_perf_client.get("/api/monitoring/performance/llm") + tool_response = await webui_perf_client.get("/api/monitoring/performance/tool") + + assert status_response.status_code == status.HTTP_200_OK, status_response.text + assert metrics_response.status_code == status.HTTP_200_OK, metrics_response.text + assert llm_response.status_code == status.HTTP_200_OK, llm_response.text + assert tool_response.status_code == status.HTTP_200_OK, tool_response.text + + status_payload = status_response.json() + assert status_payload["status"] == "healthy" + assert "uptime" in status_payload + assert "mcpServers" in status_payload + + metrics_payload = metrics_response.json() + assert "messageRate" in metrics_payload + assert "toolCallRate" in metrics_payload + assert llm_response.json() == [] + assert tool_response.json() == [] + + +async def test_api_permission_alias_matches_frontend_route(webui_perf_client: AsyncClient): + response = await webui_perf_client.get("/api/permission") + + assert response.status_code == status.HTTP_200_OK, response.text + assert response.json() == [] diff --git a/webui/src/api/stats.test.ts b/webui/src/api/stats.test.ts index 9f9dc98db..7407e93ba 100644 --- a/webui/src/api/stats.test.ts +++ b/webui/src/api/stats.test.ts @@ -9,6 +9,7 @@ vi.mock('./client', () => ({ // Helper: build a default mock for every endpoint except /api/skills and /api/agent. function defaultMock(skillsData: unknown[], agentsData: unknown[] = []) { mockGet.mockImplementation((url: string) => { + if (url === '/api/stats/summary') return Promise.reject(new Error('legacy backend')); if (url === '/api/skills') return Promise.resolve({ data: skillsData }); if (url === '/api/task-system/dashboard') return Promise.resolve({ data: {} }); if (url === '/api/agent') return Promise.resolve({ data: agentsData }); @@ -23,6 +24,26 @@ function defaultMock(skillsData: unknown[], agentsData: unknown[] = []) { describe('statsApi.getSystemStats', () => { beforeEach(() => vi.clearAllMocks()); + it('uses the lightweight summary endpoint when available', async () => { + const summary = { + tasks: { week: 3, scheduledActive: 1 }, + agents: { total: 2 }, + workflows: { total: 4 }, + skills: { total: 5 }, + tools: { total: 6 }, + models: { total: 7 }, + system: { status: 'healthy', message: 'ok' }, + }; + mockGet.mockResolvedValue({ data: summary }); + + const { statsApi } = await import('./stats'); + const result = await statsApi.getSystemStats(); + + expect(result).toEqual(summary); + expect(mockGet).toHaveBeenCalledTimes(1); + expect(mockGet).toHaveBeenCalledWith('/api/stats/summary'); + }); + it('counts only non-system skills', async () => { defaultMock([ { category: 'custom' }, @@ -62,6 +83,7 @@ describe('statsApi.getSystemStats', () => { it('handles skills API failure gracefully (returns 0)', async () => { mockGet.mockImplementation((url: string) => { + if (url === '/api/stats/summary') return Promise.reject(new Error('legacy backend')); if (url === '/api/skills') return Promise.reject(new Error('network')); if (url === '/api/task-system/dashboard') return Promise.resolve({ data: {} }); if (url === '/api/agent') return Promise.resolve({ data: [] }); diff --git a/webui/src/api/stats.ts b/webui/src/api/stats.ts index 363fb8095..89ad2130f 100644 --- a/webui/src/api/stats.ts +++ b/webui/src/api/stats.ts @@ -32,61 +32,87 @@ function shouldCountForAgentPage(agent: any): boolean { return !Array.isArray(agent.tags) || !agent.tags.includes('system'); } +function isSystemStats(value: any): value is SystemStats { + return Boolean( + value && + typeof value === 'object' && + value.tasks && + value.agents && + value.workflows && + value.skills && + value.tools && + value.models && + value.system + ); +} + +async function getSystemStatsLegacy(): Promise { + const [taskDash, agents, workflows, skills, tools, providers, health] = await Promise.all([ + apiClient.get('/api/task-system/dashboard').catch(() => ({ data: {} })), + apiClient.get('/api/agent').catch(() => ({ data: [] })), + apiClient.get('/api/workflow').catch(() => ({ data: [] })), + apiClient.get('/api/skills').catch(() => ({ data: [] })), + apiClient.get('/api/tools').catch(() => ({ data: [] })), + apiClient.get('/api/provider').catch(() => ({ data: { all: [] } })), + apiClient.get('/api/health').catch(() => ({ data: { status: 'error' } })), + ]); + + const dash = taskDash.data || {}; + const agentList = (Array.isArray(agents.data) ? agents.data : []).filter(shouldCountForAgentPage); + const workflowList = Array.isArray(workflows.data) ? workflows.data : []; + // Exclude `system` category skills so the count matches the Skills page, + // which hides system skills (e.g. onboarding) from the user. + const skillList = (Array.isArray(skills.data) ? skills.data : []).filter( + (s: any) => s?.category !== 'system' + ); + const toolList = Array.isArray(tools.data) ? tools.data : []; + const providerData = providers.data ?? {}; + const providerAll: any[] = providerData.all ?? (Array.isArray(providers.data) ? providers.data : []); + const connectedSet = new Set(providerData.connected ?? []); + const totalModels = providerAll + .filter((p: any) => connectedSet.has(p.id)) + .reduce((sum: number, p: any) => sum + Object.keys(p.models ?? {}).length, 0); + + return { + tasks: { + week: (dash.completed_week ?? 0) + (dash.failed_week ?? 0), + scheduledActive: dash.scheduled_active ?? 0, + }, + agents: { total: agentList.length }, + workflows: { total: workflowList.length }, + skills: { total: skillList.length }, + tools: { total: toolList.length }, + models: { total: totalModels }, + system: { + status: health.data.status === 'healthy' ? 'healthy' : 'error', + message: health.data.status === 'healthy' ? '所有服务运行正常' : '部分服务异常', + }, + }; +} + export const statsApi = { getSystemStats: async (): Promise => { try { - const [taskDash, agents, workflows, skills, tools, providers, health] = await Promise.all([ - apiClient.get('/api/task-system/dashboard').catch(() => ({ data: {} })), - apiClient.get('/api/agent').catch(() => ({ data: [] })), - apiClient.get('/api/workflow').catch(() => ({ data: [] })), - apiClient.get('/api/skills').catch(() => ({ data: [] })), - apiClient.get('/api/tools').catch(() => ({ data: [] })), - apiClient.get('/api/provider').catch(() => ({ data: { all: [] } })), - apiClient.get('/api/health').catch(() => ({ data: { status: 'error' } })), - ]); - - const dash = taskDash.data || {}; - const agentList = (Array.isArray(agents.data) ? agents.data : []).filter(shouldCountForAgentPage); - const workflowList = Array.isArray(workflows.data) ? workflows.data : []; - // Exclude `system` category skills so the count matches the Skills page, - // which hides system skills (e.g. onboarding) from the user. - const skillList = (Array.isArray(skills.data) ? skills.data : []).filter( - (s: any) => s?.category !== 'system' - ); - const toolList = Array.isArray(tools.data) ? tools.data : []; - const providerData = providers.data ?? {}; - const providerAll: any[] = providerData.all ?? (Array.isArray(providers.data) ? providers.data : []); - const connectedSet = new Set(providerData.connected ?? []); - const totalModels = providerAll - .filter((p: any) => connectedSet.has(p.id)) - .reduce((sum: number, p: any) => sum + Object.keys(p.models ?? {}).length, 0); - - return { - tasks: { - week: (dash.completed_week ?? 0) + (dash.failed_week ?? 0), - scheduledActive: dash.scheduled_active ?? 0, - }, - agents: { total: agentList.length }, - workflows: { total: workflowList.length }, - skills: { total: skillList.length }, - tools: { total: toolList.length }, - models: { total: totalModels }, - system: { - status: health.data.status === 'healthy' ? 'healthy' : 'error', - message: health.data.status === 'healthy' ? '所有服务运行正常' : '部分服务异常', - }, - }; + const response = await apiClient.get('/api/stats/summary'); + if (isSystemStats(response.data)) { + return response.data; + } + return await getSystemStatsLegacy(); } catch (error) { - console.error('Failed to fetch system stats:', error); - return { - tasks: { week: 0, scheduledActive: 0 }, - agents: { total: 0 }, - workflows: { total: 0 }, - skills: { total: 0 }, - tools: { total: 0 }, - models: { total: 0 }, - system: { status: 'error', message: '无法连接到后端服务' }, - }; + try { + return await getSystemStatsLegacy(); + } catch (fallbackError) { + console.error('Failed to fetch system stats:', fallbackError || error); + return { + tasks: { week: 0, scheduledActive: 0 }, + agents: { total: 0 }, + workflows: { total: 0 }, + skills: { total: 0 }, + tools: { total: 0 }, + models: { total: 0 }, + system: { status: 'error', message: '无法连接到后端服务' }, + }; + } } }, }; diff --git a/webui/src/api/update.ts b/webui/src/api/update.ts index 10de702f6..bd10d894f 100644 --- a/webui/src/api/update.ts +++ b/webui/src/api/update.ts @@ -52,11 +52,16 @@ export interface UpdateProgress { // API // ====================================================================== -export const checkUpdate = async (locale?: string, edition?: UpdateEdition): Promise => { +export const checkUpdate = async ( + locale?: string, + edition?: UpdateEdition, + force?: boolean, +): Promise => { const response = await client.get('/api/update/check', { params: { ...(locale ? { locale } : {}), ...(edition ? { edition } : {}), + ...(force ? { force: true } : {}), }, }); return response.data; diff --git a/webui/src/components/layout/Layout.test.tsx b/webui/src/components/layout/Layout.test.tsx index 21772c71e..8cb815302 100644 --- a/webui/src/components/layout/Layout.test.tsx +++ b/webui/src/components/layout/Layout.test.tsx @@ -370,10 +370,13 @@ describe('Layout onboarding entry', () => { renderHomeWithLayout(); await flushEffects(); + await act(async () => { + await vi.advanceTimersByTimeAsync(250); + }); expect(checkUpdate).toHaveBeenCalledTimes(1); await act(async () => { - await vi.advanceTimersByTimeAsync(3_599_999); + await vi.advanceTimersByTimeAsync(3_599_749); }); expect(checkUpdate).toHaveBeenCalledTimes(1); @@ -680,6 +683,9 @@ describe('Layout onboarding entry', () => { renderHomeWithLayout(); await flushEffects(); + await act(async () => { + await vi.advanceTimersByTimeAsync(250); + }); expect(checkUpdate).toHaveBeenCalledTimes(1); await act(async () => { diff --git a/webui/src/components/layout/Layout.tsx b/webui/src/components/layout/Layout.tsx index dfe917045..c279017ea 100644 --- a/webui/src/components/layout/Layout.tsx +++ b/webui/src/components/layout/Layout.tsx @@ -105,6 +105,7 @@ import type { WebUIContractWorkspaceListItem } from '@/api/webuiContractPages'; const UPDATE_CHECK_INTERVAL_MS = 3_600_000; const UPDATE_CHECK_MIN_GAP_MS = 600_000; +const UPDATE_CHECK_INITIAL_DELAY_MS = 250; interface LayoutNavItem { name: string; @@ -301,7 +302,9 @@ export default function Layout() { useEffect(() => { if (!flocksproStatusReady) return undefined; - refreshUpdateStatus(true); + const initialCheckTimerId = window.setTimeout(() => { + refreshUpdateStatus(true); + }, UPDATE_CHECK_INITIAL_DELAY_MS); const intervalId = window.setInterval(() => { if (document.visibilityState === 'visible') { @@ -323,6 +326,7 @@ export default function Layout() { window.addEventListener('focus', handleWindowFocus); return () => { + window.clearTimeout(initialCheckTimerId); window.clearInterval(intervalId); document.removeEventListener('visibilitychange', handleVisibilityChange); window.removeEventListener('focus', handleWindowFocus); From 53b9296a9914a196f6d52285fab47523757a5c23 Mon Sep 17 00:00:00 2001 From: duguwanglong Date: Thu, 9 Jul 2026 18:38:20 +0800 Subject: [PATCH 08/49] perf(webui): smooth loading and hub filters --- .../common/LoadingIndicators.test.tsx | 41 +++++++++++++++++++ .../src/components/common/LoadingSpinner.tsx | 9 +++- .../components/common/RoutePageSkeleton.tsx | 13 +++++- webui/src/hooks/useDelayedVisible.ts | 18 ++++++++ webui/src/pages/Agent/index.tsx | 2 +- webui/src/pages/Channel/index.tsx | 2 +- webui/src/pages/Home/index.tsx | 2 +- webui/src/pages/Hub/index.tsx | 30 ++++++++------ webui/src/pages/MCP/index.tsx | 4 +- webui/src/pages/Model/index.tsx | 2 +- webui/src/pages/Monitoring/index.tsx | 2 +- webui/src/pages/Permission/index.tsx | 2 +- webui/src/pages/Session/index.tsx | 2 +- webui/src/pages/Settings/index.tsx | 2 +- webui/src/pages/Skill/index.tsx | 2 +- webui/src/pages/Task/QueuedSection.tsx | 2 +- webui/src/pages/Task/ScheduledSection.tsx | 2 +- webui/src/pages/Tool/index.tsx | 2 +- .../WebUIContractPageHost/PageRuntimeHost.tsx | 3 ++ .../WebUIContractWorkspaceHost/index.tsx | 3 ++ webui/src/pages/Workflow/index.tsx | 2 +- webui/src/pages/WorkflowDetail/index.tsx | 2 +- webui/src/pages/WorkflowEditor/index.tsx | 3 +- webui/src/routes/index.tsx | 3 +- 24 files changed, 120 insertions(+), 35 deletions(-) create mode 100644 webui/src/components/common/LoadingIndicators.test.tsx create mode 100644 webui/src/hooks/useDelayedVisible.ts diff --git a/webui/src/components/common/LoadingIndicators.test.tsx b/webui/src/components/common/LoadingIndicators.test.tsx new file mode 100644 index 000000000..cea074360 --- /dev/null +++ b/webui/src/components/common/LoadingIndicators.test.tsx @@ -0,0 +1,41 @@ +import { act, render, screen } from '@testing-library/react'; +import { describe, expect, it, vi } from 'vitest'; +import LoadingSpinner from './LoadingSpinner'; +import RoutePageSkeleton from './RoutePageSkeleton'; + +describe('delayed loading indicators', () => { + it('hides short spinner flashes until the delay elapses', async () => { + vi.useFakeTimers(); + + render(); + + expect(screen.queryByRole('status')).not.toBeInTheDocument(); + + await act(async () => { + await vi.advanceTimersByTimeAsync(179); + }); + expect(screen.queryByRole('status')).not.toBeInTheDocument(); + + await act(async () => { + await vi.advanceTimersByTimeAsync(1); + }); + expect(screen.getByRole('status')).toBeInTheDocument(); + + vi.useRealTimers(); + }); + + it('hides short route skeleton flashes until the delay elapses', async () => { + vi.useFakeTimers(); + + render(); + + expect(screen.queryByTestId('route-page-skeleton')).not.toBeInTheDocument(); + + await act(async () => { + await vi.advanceTimersByTimeAsync(180); + }); + expect(screen.getByTestId('route-page-skeleton')).toBeInTheDocument(); + + vi.useRealTimers(); + }); +}); diff --git a/webui/src/components/common/LoadingSpinner.tsx b/webui/src/components/common/LoadingSpinner.tsx index 90dd7fd0c..3b25f59ac 100644 --- a/webui/src/components/common/LoadingSpinner.tsx +++ b/webui/src/components/common/LoadingSpinner.tsx @@ -1,19 +1,24 @@ import { Loader2 } from 'lucide-react'; +import { useDelayedVisible } from '@/hooks/useDelayedVisible'; interface LoadingSpinnerProps { size?: 'sm' | 'md' | 'lg'; className?: string; + delayMs?: number; } -export default function LoadingSpinner({ size = 'md', className = '' }: LoadingSpinnerProps) { +export default function LoadingSpinner({ size = 'md', className = '', delayMs = 0 }: LoadingSpinnerProps) { + const visible = useDelayedVisible(delayMs); const sizeClasses = { sm: 'w-4 h-4', md: 'w-8 h-8', lg: 'w-12 h-12', }; + if (!visible) return null; + return ( -
+
); diff --git a/webui/src/components/common/RoutePageSkeleton.tsx b/webui/src/components/common/RoutePageSkeleton.tsx index 82581c05f..223b7f780 100644 --- a/webui/src/components/common/RoutePageSkeleton.tsx +++ b/webui/src/components/common/RoutePageSkeleton.tsx @@ -1,6 +1,15 @@ -export default function RoutePageSkeleton() { +import { useDelayedVisible } from '@/hooks/useDelayedVisible'; + +interface RoutePageSkeletonProps { + delayMs?: number; +} + +export default function RoutePageSkeleton({ delayMs = 0 }: RoutePageSkeletonProps) { + const visible = useDelayedVisible(delayMs); + if (!visible) return null; + return ( -
+
diff --git a/webui/src/hooks/useDelayedVisible.ts b/webui/src/hooks/useDelayedVisible.ts new file mode 100644 index 000000000..2e7f80c00 --- /dev/null +++ b/webui/src/hooks/useDelayedVisible.ts @@ -0,0 +1,18 @@ +import { useEffect, useState } from 'react'; + +export function useDelayedVisible(delayMs = 0): boolean { + const [visible, setVisible] = useState(delayMs <= 0); + + useEffect(() => { + if (delayMs <= 0) { + setVisible(true); + return undefined; + } + + setVisible(false); + const timer = window.setTimeout(() => setVisible(true), delayMs); + return () => window.clearTimeout(timer); + }, [delayMs]); + + return visible; +} diff --git a/webui/src/pages/Agent/index.tsx b/webui/src/pages/Agent/index.tsx index 9114144c9..a9c3519b3 100644 --- a/webui/src/pages/Agent/index.tsx +++ b/webui/src/pages/Agent/index.tsx @@ -110,7 +110,7 @@ export default function AgentPage() { if (loading) { return (
- +
); } diff --git a/webui/src/pages/Channel/index.tsx b/webui/src/pages/Channel/index.tsx index 963a16693..958aa927e 100644 --- a/webui/src/pages/Channel/index.tsx +++ b/webui/src/pages/Channel/index.tsx @@ -2176,7 +2176,7 @@ export default function ChannelPage() { if (loading) { return (
- +
); } diff --git a/webui/src/pages/Home/index.tsx b/webui/src/pages/Home/index.tsx index b4d664cb0..27d4198ae 100644 --- a/webui/src/pages/Home/index.tsx +++ b/webui/src/pages/Home/index.tsx @@ -196,7 +196,7 @@ export default function Home() { {/* Stats */} {loading ? (
- +
) : ( <> diff --git a/webui/src/pages/Hub/index.tsx b/webui/src/pages/Hub/index.tsx index aa2888a5a..f5243d894 100644 --- a/webui/src/pages/Hub/index.tsx +++ b/webui/src/pages/Hub/index.tsx @@ -44,6 +44,7 @@ import { useProductName } from '@/contexts/ProductNameContext'; type ViewMode = 'table' | 'tree'; const FlowCanvas = lazy(() => import('@/pages/WorkflowDetail/FlowCanvas')); +const HUB_LOADING_DELAY_MS = 180; interface HubTaxonomyCategory { id: string; @@ -269,6 +270,7 @@ export default function HubPage() { const urlType = normalizePluginType(searchParams.get('type')); const [catalogItems, setCatalogItems] = useState([]); const [loading, setLoading] = useState(true); + const [hasLoadedCatalog, setHasLoadedCatalog] = useState(false); const [refreshing, setRefreshing] = useState(false); const [query, setQuery] = useState(searchParams.get('q') || urlPluginId); const [typeFilter, setTypeFilter] = useState(urlType); @@ -309,6 +311,7 @@ export default function HubPage() { }); return nextItems; } finally { + setHasLoadedCatalog(true); if (!silent) setLoading(false); } }, [debouncedQuery, page, pageSize, stateFilter, tagFilter, typeFilter, useCaseFilter]); @@ -345,6 +348,7 @@ export default function HubPage() { const totalPages = Math.max(1, Math.ceil(totalItems / pageSize)); const currentPage = Math.min(page, totalPages); const pagedItems = items; + const isInitialLoading = loading && !hasLoadedCatalog; useEffect(() => { if (page > totalPages) setPage(totalPages); @@ -400,12 +404,12 @@ export default function HubPage() { setPage(1); }; - if (loading) { - return
; + if (isInitialLoading) { + return
; } return ( -
+
-
+
-
+
{viewMode === 'table' ? ( ) : ( @@ -606,15 +610,15 @@ function FilterRow({ label, value, options, onChange }: { key={option.value || 'all'} title={option.title} onClick={() => onChange(option.value)} - className={`px-2 py-1 rounded-md transition-colors ${ + className={`inline-flex h-8 items-center whitespace-nowrap rounded-md px-2 py-1 font-medium tabular-nums transition-colors ${ active - ? 'bg-slate-800 text-white font-medium' + ? 'bg-slate-800 text-white' : 'text-gray-600 hover:bg-white hover:text-gray-900' }`} > {option.label} {option.count !== undefined && option.value && ( - {option.count} + {option.count} )} ); @@ -622,8 +626,8 @@ function FilterRow({ label, value, options, onChange }: { const [allOption, ...facetOptions] = options; return ( -
-
+
+
{label} @@ -650,7 +654,7 @@ function PaginationBar({ total, page, pageSize, totalPages, text, onPageChange, const start = total === 0 ? 0 : (page - 1) * pageSize + 1; const end = Math.min(total, page * pageSize); return ( -
+
{text.showing} {start}-{end} / {text.of} {total} {text.plugins}
@@ -693,7 +697,7 @@ function HubTable({ items, actionId, tagLabels, language, text, onSelect, onActi onAction: (entry: HubCatalogEntry, action: 'install' | 'update' | 'uninstall') => void; }) { return ( -
+
@@ -786,7 +790,7 @@ function HubTree({ items, actionId, text, onSelect, onAction }: { }, [items]); return ( -
+
); diff --git a/webui/src/pages/MCP/index.tsx b/webui/src/pages/MCP/index.tsx index bb7e27891..e5ca185da 100644 --- a/webui/src/pages/MCP/index.tsx +++ b/webui/src/pages/MCP/index.tsx @@ -119,7 +119,7 @@ function ServersTab() { setSelectedServer(server); }; - if (loading) return
; + if (loading) return
; if (error) return (
@@ -290,7 +290,7 @@ function CatalogTab() { } }; - if (loading) return
; + if (loading) return
; return (
diff --git a/webui/src/pages/Model/index.tsx b/webui/src/pages/Model/index.tsx index 8175777a6..48f617682 100644 --- a/webui/src/pages/Model/index.tsx +++ b/webui/src/pages/Model/index.tsx @@ -418,7 +418,7 @@ export default function ModelPage() { if (loading) { return (
- +
); } diff --git a/webui/src/pages/Monitoring/index.tsx b/webui/src/pages/Monitoring/index.tsx index 516760c7c..8a3ad77e9 100644 --- a/webui/src/pages/Monitoring/index.tsx +++ b/webui/src/pages/Monitoring/index.tsx @@ -48,7 +48,7 @@ export default function MonitoringPage() { if (loading && !status) { return (
- +
); } diff --git a/webui/src/pages/Permission/index.tsx b/webui/src/pages/Permission/index.tsx index ff6f1c77d..b6b56bc1d 100644 --- a/webui/src/pages/Permission/index.tsx +++ b/webui/src/pages/Permission/index.tsx @@ -51,7 +51,7 @@ export default function PermissionPage() { if (loading) { return (
- +
); } diff --git a/webui/src/pages/Session/index.tsx b/webui/src/pages/Session/index.tsx index b801819f1..73cc90684 100644 --- a/webui/src/pages/Session/index.tsx +++ b/webui/src/pages/Session/index.tsx @@ -1020,7 +1020,7 @@ export default function SessionPage() { if (loadingSessions) { return (
- +
); } diff --git a/webui/src/pages/Settings/index.tsx b/webui/src/pages/Settings/index.tsx index 3c55c998e..337cd9e92 100644 --- a/webui/src/pages/Settings/index.tsx +++ b/webui/src/pages/Settings/index.tsx @@ -380,7 +380,7 @@ function SettingsContent({ sectionId }: { sectionId: SettingsSectionId }) { if (sectionId === 'preferences') return ; return ( - }> + }> {sectionId === 'account' && } {sectionId === 'system-logs' && } {sectionId === 'audit-logs' && } diff --git a/webui/src/pages/Skill/index.tsx b/webui/src/pages/Skill/index.tsx index f90f38c48..50e2bab9a 100644 --- a/webui/src/pages/Skill/index.tsx +++ b/webui/src/pages/Skill/index.tsx @@ -275,7 +275,7 @@ export default function SkillPage() { if (loading) { return (
- +
); } diff --git a/webui/src/pages/Task/QueuedSection.tsx b/webui/src/pages/Task/QueuedSection.tsx index 63ae86694..2ebc15f5c 100644 --- a/webui/src/pages/Task/QueuedSection.tsx +++ b/webui/src/pages/Task/QueuedSection.tsx @@ -241,7 +241,7 @@ export default function QueuedSection({ onRefreshGlobal }: { onRefreshGlobal: () setSelectedTasks(new Set()); }; - if (loading && effectiveTasks.length === 0) return
; + if (loading && effectiveTasks.length === 0) return
; if (error) return
{error}
; return ( diff --git a/webui/src/pages/Task/ScheduledSection.tsx b/webui/src/pages/Task/ScheduledSection.tsx index e1ebc1b38..ce9c9645a 100644 --- a/webui/src/pages/Task/ScheduledSection.tsx +++ b/webui/src/pages/Task/ScheduledSection.tsx @@ -48,7 +48,7 @@ export default function ScheduledSection({ onRefreshGlobal }: { onRefreshGlobal: } }; - if (loading && tasks.length === 0) return
; + if (loading && tasks.length === 0) return
; if (error) return
{error}
; return ( diff --git a/webui/src/pages/Tool/index.tsx b/webui/src/pages/Tool/index.tsx index 99fe3d86e..48473a478 100644 --- a/webui/src/pages/Tool/index.tsx +++ b/webui/src/pages/Tool/index.tsx @@ -431,7 +431,7 @@ export default function ToolPage() { if (loading) { return (
- +
); } diff --git a/webui/src/pages/WebUIContractPageHost/PageRuntimeHost.tsx b/webui/src/pages/WebUIContractPageHost/PageRuntimeHost.tsx index e1cccee74..d1e87aa67 100644 --- a/webui/src/pages/WebUIContractPageHost/PageRuntimeHost.tsx +++ b/webui/src/pages/WebUIContractPageHost/PageRuntimeHost.tsx @@ -13,6 +13,7 @@ import { AlertCircle, Loader2 } from 'lucide-react'; import { getApiBase } from '@/api/client'; import { webuiContractPagesAPI } from '@/api/webuiContractPages'; import { useSSE } from '@/hooks/useSSE'; +import { useDelayedVisible } from '@/hooks/useDelayedVisible'; import { installWebUIContractPageRuntime, loadWebUIContractPageBundle } from './runtime'; interface WebUIContractPageErrorBoundaryProps { @@ -73,6 +74,7 @@ export default function PageRuntimeHost({ pageId }: PageRuntimeHostProps) { const [loading, setLoading] = useState(true); const [error, setError] = useState(null); const [buildHash, setBuildHash] = useState(''); + const showLoading = useDelayedVisible(loading ? 180 : 0); const loadBundle = useCallback(async (hash: string) => { if (!pageId || !hash) return; @@ -140,6 +142,7 @@ export default function PageRuntimeHost({ pageId }: PageRuntimeHostProps) { } if (loading) { + if (!showLoading) return null; return (
diff --git a/webui/src/pages/WebUIContractWorkspaceHost/index.tsx b/webui/src/pages/WebUIContractWorkspaceHost/index.tsx index df890edf8..cee5d4c5d 100644 --- a/webui/src/pages/WebUIContractWorkspaceHost/index.tsx +++ b/webui/src/pages/WebUIContractWorkspaceHost/index.tsx @@ -7,6 +7,7 @@ import { type WebUIContractWorkspaceListItem, } from '@/api/webuiContractPages'; import { useSSE } from '@/hooks/useSSE'; +import { useDelayedVisible } from '@/hooks/useDelayedVisible'; import { ThemeContext } from '@/contexts/ThemeContext'; import PageRuntimeHost from '@/pages/WebUIContractPageHost/PageRuntimeHost'; import { buildWebUIContractWorkspaceSections } from '@/utils/webuiContractWorkspaceSections'; @@ -17,6 +18,7 @@ export default function WebUIContractWorkspaceHost() { const [workspaces, setWorkspaces] = useState([]); const [loading, setLoading] = useState(true); const [error, setError] = useState(null); + const showLoading = useDelayedVisible(loading ? 180 : 0); const { theme, setTemporaryThemeOverride } = useContext(ThemeContext); const fetchWorkspaces = useCallback(async (silent = false) => { @@ -82,6 +84,7 @@ export default function WebUIContractWorkspaceHost() { } if (loading) { + if (!showLoading) return null; return (
diff --git a/webui/src/pages/Workflow/index.tsx b/webui/src/pages/Workflow/index.tsx index 330c183a1..ec2acd37b 100644 --- a/webui/src/pages/Workflow/index.tsx +++ b/webui/src/pages/Workflow/index.tsx @@ -103,7 +103,7 @@ export default function WorkflowPage() { if (loading) { return (
- +
); } diff --git a/webui/src/pages/WorkflowDetail/index.tsx b/webui/src/pages/WorkflowDetail/index.tsx index a3074d55f..7e1bbd8a4 100644 --- a/webui/src/pages/WorkflowDetail/index.tsx +++ b/webui/src/pages/WorkflowDetail/index.tsx @@ -596,7 +596,7 @@ export default function WorkflowDetail() { if (loading) { return (
- +
); } diff --git a/webui/src/pages/WorkflowEditor/index.tsx b/webui/src/pages/WorkflowEditor/index.tsx index 536e15a35..d65350068 100644 --- a/webui/src/pages/WorkflowEditor/index.tsx +++ b/webui/src/pages/WorkflowEditor/index.tsx @@ -32,6 +32,7 @@ import { Trash2, } from 'lucide-react'; import { workflowAPI, Workflow, WorkflowExecution, WorkflowJSON, WorkflowNode as APINode } from '@/api/workflow'; +import LoadingSpinner from '@/components/common/LoadingSpinner'; import { ThemeContext } from '@/contexts/ThemeContext'; import { extractErrorMessage } from '@/utils/error'; import { @@ -779,7 +780,7 @@ export default function WorkflowEditor() { if (loading) { return (
-
+
); } diff --git a/webui/src/routes/index.tsx b/webui/src/routes/index.tsx index fbf2012e3..df3834ba4 100644 --- a/webui/src/routes/index.tsx +++ b/webui/src/routes/index.tsx @@ -49,10 +49,11 @@ const FlocksproUpgradeCallbackPage = lazyPage(() => import('@/pages/FlocksproUpg const SettingsPage = lazyPage(() => import('@/pages/Settings')); const WebUIContractPageHost = lazyPage(() => import('@/pages/WebUIContractPageHost')); const WebUIContractWorkspaceHost = lazyPage(() => import('@/pages/WebUIContractWorkspaceHost')); +const ROUTE_FALLBACK_DELAY_MS = 180; function LazyRoute({ children }: { children: ReactNode }) { return ( - }> + }> {children} ); From 4f98fba5d082747fd9df376cb3ec644e0305908c Mon Sep 17 00:00:00 2001 From: duguwanglong Date: Thu, 9 Jul 2026 18:45:40 +0800 Subject: [PATCH 09/49] fix(webui): stabilize tool tab switching --- webui/src/hooks/useTools.ts | 2 + .../pages/Tool/components/APITabContent.tsx | 7 +- .../pages/Tool/components/MCPTabContent.tsx | 4 +- webui/src/pages/Tool/index.tsx | 209 ++++++++++-------- 4 files changed, 127 insertions(+), 95 deletions(-) diff --git a/webui/src/hooks/useTools.ts b/webui/src/hooks/useTools.ts index 2691a0f07..d1ba56aac 100644 --- a/webui/src/hooks/useTools.ts +++ b/webui/src/hooks/useTools.ts @@ -161,6 +161,7 @@ export function useToolPage(params: ToolListPageParams) { data, loading, error, + initialized, } = useSharedResource(resource); const refreshVisiblePage = useCallback( @@ -194,6 +195,7 @@ export function useToolPage(params: ToolListPageParams) { limit: data.limit, loading, error, + initialized, refetch, }; } diff --git a/webui/src/pages/Tool/components/APITabContent.tsx b/webui/src/pages/Tool/components/APITabContent.tsx index 3144ca050..6dafd3f22 100644 --- a/webui/src/pages/Tool/components/APITabContent.tsx +++ b/webui/src/pages/Tool/components/APITabContent.tsx @@ -8,6 +8,7 @@ import { providerAPI } from '@/api/provider'; import type { Tool } from '@/api/tool'; import type { APIServiceSummary, MCPCatalogCategory, MCPCatalogEntry } from '@/types'; import EmptyState from '@/components/common/EmptyState'; +import LoadingSpinner from '@/components/common/LoadingSpinner'; import { getCatalogDescription } from '@/utils/mcpCatalog'; import { APIServiceDetailPanel } from './ServiceDetailPanel'; import { SERVICE_TAB_GRID_COLS } from './gridLayout'; @@ -301,7 +302,11 @@ export default function APITabContent({ )}
- {services.length === 0 && filteredCatalog.length === 0 && !catalogLoading && !servicesLoading ? ( + {servicesLoading && services.length === 0 && filteredCatalog.length === 0 ? ( +
+ +
+ ) : services.length === 0 && filteredCatalog.length === 0 && !catalogLoading ? ( } title={t('api.noTools')} description={t('api.noToolsDesc')} /> ) : (
diff --git a/webui/src/pages/Tool/components/MCPTabContent.tsx b/webui/src/pages/Tool/components/MCPTabContent.tsx index ea6a2c3d5..9c30bab5d 100644 --- a/webui/src/pages/Tool/components/MCPTabContent.tsx +++ b/webui/src/pages/Tool/components/MCPTabContent.tsx @@ -437,7 +437,9 @@ export default function MCPTabContent({
{serversLoading && catalogLoading ? ( -
+
+ +
) : unifiedCards.length === 0 ? ( } title={t('mcp.noServers')} description={t('mcp.noServersDesc')} /> ) : ( diff --git a/webui/src/pages/Tool/index.tsx b/webui/src/pages/Tool/index.tsx index 48473a478..962160b1b 100644 --- a/webui/src/pages/Tool/index.tsx +++ b/webui/src/pages/Tool/index.tsx @@ -221,8 +221,10 @@ export default function ToolPage() { facets: toolFacets, loading, error, + initialized: toolPageInitialized, refetch, } = useToolPage(toolPageParams); + const [hasLoadedToolPage, setHasLoadedToolPage] = useState(false); // Catalog data (fetched once at top level, shared with MCP & API tabs) const [catalogEntries, setCatalogEntries] = useState([]); @@ -343,6 +345,12 @@ export default function ToolPage() { if (currentPage > totalPages) setCurrentPage(totalPages); }, [currentPage, totalPages]); + useEffect(() => { + if (toolPageInitialized && !error) { + setHasLoadedToolPage(true); + } + }, [toolPageInitialized, error]); + const handleTabChange = (tab: TabKey) => { setActiveTab(tab); setCurrentPage(1); @@ -428,7 +436,10 @@ export default function ToolPage() { setCurrentPage(1); }; - if (loading) { + const isInitialLoading = loading && !hasLoadedToolPage; + const isTabPageLoading = loading && hasLoadedToolPage && !toolPageInitialized; + + if (isInitialLoading) { return (
@@ -436,7 +447,7 @@ export default function ToolPage() { ); } - if (error) { + if (error && !hasLoadedToolPage) { return (
@@ -450,7 +461,7 @@ export default function ToolPage() { } return ( -
+
{/* Page Header */}
@@ -505,24 +516,24 @@ export default function ToolPage() { {/* Tabs row with search */}
-
+
- {/* Tab Content */} - {activeTab === 'mcp' ? ( - - ) : activeTab === 'api' ? ( - - ) : activeTab === 'local' ? ( - - ) : ( - /* All tab: active tools only */ -
- {/* Inactive services callout */} - {(mcpCatalogEntries.length > 0 || apiCatalogEntries.length > 0) && !searchQuery && ( -
- - {catalogEntries.length} {t('summary.inactiveServicesAvailable')} - -
- {mcpCatalogEntries.length > 0 && ( - - )} - {apiCatalogEntries.length > 0 && ( - - )} -
-
- )} - {totalTools === 0 ? ( - } - title={t('empty.noTools')} - description={searchQuery ? t('empty.tryOtherKeywords') : t('empty.noToolsInCategory')} - /> - ) : ( - - )} + {error && ( +
+ {error}
)} + {/* Tab Content */} +
+ {isTabPageLoading ? ( +
+ +
+ ) : activeTab === 'mcp' ? ( + + ) : activeTab === 'api' ? ( + + ) : activeTab === 'local' ? ( + + ) : ( + /* All tab: active tools only */ +
+ {/* Inactive services callout */} + {(mcpCatalogEntries.length > 0 || apiCatalogEntries.length > 0) && !searchQuery && ( +
+ + {catalogEntries.length} {t('summary.inactiveServicesAvailable')} + +
+ {mcpCatalogEntries.length > 0 && ( + + )} + {apiCatalogEntries.length > 0 && ( + + )} +
+
+ )} + {totalTools === 0 ? ( + } + title={t('empty.noTools')} + description={searchQuery ? t('empty.tryOtherKeywords') : t('empty.noToolsInCategory')} + /> + ) : ( + + )} +
+ )} +
+ {/* Tool Detail Drawer */} {selectedTool && ( Date: Thu, 9 Jul 2026 19:00:42 +0800 Subject: [PATCH 10/49] fix(session): validate selected session before chat mount --- webui/src/pages/Session/index.test.tsx | 26 +++++++++++++++++++++-- webui/src/pages/Session/index.tsx | 29 ++++++++++++++++++-------- 2 files changed, 44 insertions(+), 11 deletions(-) diff --git a/webui/src/pages/Session/index.test.tsx b/webui/src/pages/Session/index.test.tsx index 321a0b315..a468b3fdb 100644 --- a/webui/src/pages/Session/index.test.tsx +++ b/webui/src/pages/Session/index.test.tsx @@ -260,6 +260,16 @@ function renderSessionPage( ); } +function deferred() { + let resolve!: (value: T) => void; + let reject!: (reason?: unknown) => void; + const promise = new Promise((resolvePromise, rejectPromise) => { + resolve = resolvePromise; + reject = rejectPromise; + }); + return { promise, resolve, reject }; +} + describe('session sidebar grouping helpers', () => { it('groups sessions by updated date and applies search filtering', () => { const now = new Date(2026, 6, 9, 12, 0, 0); @@ -786,6 +796,7 @@ describe('SessionPage session actions menu', () => { }); it('keeps a selected session that is valid but missing from the current list', async () => { + const request = deferred(); useSessions.mockReturnValue({ sessions: [], loading: false, @@ -796,17 +807,28 @@ describe('SessionPage session actions menu', () => { removeSessions, addSession, }); - sessionApi.get.mockResolvedValue({ + sessionApi.get.mockReturnValue(request.promise); + const fetchedSession = { ...session, id: 'session-missing-from-list', title: 'Fetched Session', canWrite: false, - }); + }; renderSessionPage('/sessions?session=session-missing-from-list'); await waitFor(() => { expect(sessionApi.get).toHaveBeenCalledWith('session-missing-from-list'); + }); + expect(screen.queryByTestId('session-chat')).not.toBeInTheDocument(); + expect(screen.getByText('loading-spinner')).toBeInTheDocument(); + + await act(async () => { + request.resolve(fetchedSession); + await request.promise; + }); + + await waitFor(() => { expect(screen.getByTestId('session-chat')).toHaveTextContent('session-missing-from-list'); expect(screen.getByTestId('session-chat')).toHaveAttribute('data-hide-input', 'true'); }); diff --git a/webui/src/pages/Session/index.tsx b/webui/src/pages/Session/index.tsx index 73cc90684..15c85f1fb 100644 --- a/webui/src/pages/Session/index.tsx +++ b/webui/src/pages/Session/index.tsx @@ -479,6 +479,8 @@ export default function SessionPage() { ); const selectedSession = listedSelectedSession ?? (selectedSessionFallback?.id === selectedSessionId ? selectedSessionFallback : null); + const activeChatSessionId = selectedSession ? selectedSessionId : null; + const resolvingSelectedSession = Boolean(selectedSessionId && !selectedSession); const pinnedModelKey = selectedSession?.model_pinned && selectedSession.provider && selectedSession.model ? makeModelKey(selectedSession.provider, selectedSession.model) : null; @@ -596,9 +598,9 @@ export default function SessionPage() { }, [loadingSessions, location.state, searchParams, selectedSessionId]); useEffect(() => { - if (!selectedSessionId) return; + if (!selectedSessionId || selectedSession?.id !== selectedSessionId) return; writeLastSelectedSessionId(selectedSessionId); - }, [selectedSessionId]); + }, [selectedSession?.id, selectedSessionId]); useEffect(() => { if (!selectedSessionId) { @@ -609,6 +611,7 @@ export default function SessionPage() { setSelectedSessionFallback(null); return; } + if (selectedSessionFallback?.id === selectedSessionId) return; if (loadingSessions) return; let cancelled = false; @@ -629,7 +632,7 @@ export default function SessionPage() { return () => { cancelled = true; }; - }, [listedSelectedSession, loadingSessions, selectedSessionId]); + }, [listedSelectedSession, loadingSessions, selectedSessionFallback?.id, selectedSessionId]); // Close agent dropdown on outside click useEffect(() => { @@ -722,6 +725,7 @@ export default function SessionPage() { try { const response = await client.post('/api/session', { title: 'New Session' }); addSession(response.data); + setSelectedSessionFallback(response.data); setSelectedAgent('rex'); setSelectedModelKey(null); setSelectedSessionId(response.data.id); @@ -761,6 +765,7 @@ export default function SessionPage() { const newSessionId = response.data.id; addSession(response.data); + setSelectedSessionFallback(response.data); setSelectedModelKey(null); setSelectedSessionId(newSessionId); @@ -1194,10 +1199,15 @@ export default function SessionPage() {
{/* Chat — powered by unified SessionChat */} - + +
+ ) : ( + } - /> + /> + )}
{selectorTooltip && ( From 3d68fdf3196f90f99991e4f19c567eddfa5afb9a Mon Sep 17 00:00:00 2001 From: duguwanglong Date: Thu, 9 Jul 2026 19:27:54 +0800 Subject: [PATCH 11/49] fix(session): harden initial messages and cached tasks --- flocks/server/routes/session.py | 49 ++++---- flocks/server/routes/update.py | 38 +++--- tests/server/routes/test_session_routes.py | 129 +++++++++++++++++++++ tests/server/routes/test_update_routes.py | 42 +++++++ webui/src/pages/Session/index.test.tsx | 21 ++++ webui/src/pages/Session/index.tsx | 3 + 6 files changed, 246 insertions(+), 36 deletions(-) diff --git a/flocks/server/routes/session.py b/flocks/server/routes/session.py index f6152cd1d..c68ab65b8 100644 --- a/flocks/server/routes/session.py +++ b/flocks/server/routes/session.py @@ -54,6 +54,33 @@ def _context_usage_cache_key(session_id: str, session: SessionModel) -> Tuple[st return session_id, int(getattr(session.time, "updated", 0) or 0) +async def _build_context_usage_for_cache( + key: Tuple[str, int], + session_id: str, + *, + session: SessionModel, +) -> ContextUsageSnapshot: + current_task = asyncio.current_task() + try: + snapshot = await build_context_usage_snapshot(session_id, session=session) + async with _context_usage_cache_lock: + if _context_usage_inflight.get(key) is not current_task: + return snapshot + for cache_key in [item for item in _context_usage_cache if item[0] == session_id and item[1] < key[1]]: + _context_usage_cache.pop(cache_key, None) + has_newer_cache = any(item[0] == session_id and item[1] > key[1] for item in _context_usage_cache) + if not has_newer_cache: + _context_usage_cache[key] = ( + time.monotonic() + _CONTEXT_USAGE_CACHE_TTL_SECONDS, + snapshot.model_copy(deep=True), + ) + return snapshot + finally: + async with _context_usage_cache_lock: + if _context_usage_inflight.get(key) is current_task: + _context_usage_inflight.pop(key, None) + + async def _cached_context_usage_snapshot( session_id: str, *, @@ -61,7 +88,6 @@ async def _cached_context_usage_snapshot( ) -> ContextUsageSnapshot: key = _context_usage_cache_key(session_id, session) now = time.monotonic() - owner = False async with _context_usage_cache_lock: cached = _context_usage_cache.get(key) @@ -70,27 +96,10 @@ async def _cached_context_usage_snapshot( task = _context_usage_inflight.get(key) if task is None: - task = asyncio.create_task(build_context_usage_snapshot(session_id, session=session)) + task = asyncio.create_task(_build_context_usage_for_cache(key, session_id, session=session)) _context_usage_inflight[key] = task - owner = True - - try: - snapshot = await task - finally: - if owner: - async with _context_usage_cache_lock: - if _context_usage_inflight.get(key) is task: - _context_usage_inflight.pop(key, None) - - if owner: - async with _context_usage_cache_lock: - for cache_key in [item for item in _context_usage_cache if item[0] == session_id and item != key]: - _context_usage_cache.pop(cache_key, None) - _context_usage_cache[key] = ( - time.monotonic() + _CONTEXT_USAGE_CACHE_TTL_SECONDS, - snapshot.model_copy(deep=True), - ) + snapshot = await asyncio.shield(task) return snapshot.model_copy(deep=True) # ============================================================================= diff --git a/flocks/server/routes/update.py b/flocks/server/routes/update.py index c0c5f0519..ddf3b0af9 100644 --- a/flocks/server/routes/update.py +++ b/flocks/server/routes/update.py @@ -34,6 +34,26 @@ def clear_update_check_cache() -> None: _update_check_inflight.clear() +async def _run_update_check_for_cache( + key: tuple[str, str], + *, + locale: str | None, + edition: Literal["flocks", "flockspro"], +) -> VersionInfo: + current_task = asyncio.current_task() + try: + info = await check_update(locale=locale, force_console_manifest=(edition == "flockspro")) + ttl = _UPDATE_CHECK_ERROR_CACHE_TTL_SECONDS if info.error else _UPDATE_CHECK_CACHE_TTL_SECONDS + async with _update_check_lock: + if _update_check_inflight.get(key) is current_task: + _update_check_cache[key] = (time.monotonic() + ttl, info.model_copy(deep=True)) + return info + finally: + async with _update_check_lock: + if _update_check_inflight.get(key) is current_task: + _update_check_inflight.pop(key, None) + + async def _check_update_cached( *, locale: str | None, @@ -47,7 +67,6 @@ async def _check_update_cached( _update_check_cache[key] = (time.monotonic() + ttl, info.model_copy(deep=True)) return info - owner = False now = time.monotonic() async with _update_check_lock: cached = _update_check_cache.get(key) @@ -57,24 +76,11 @@ async def _check_update_cached( task = _update_check_inflight.get(key) if task is None: task = asyncio.create_task( - check_update(locale=locale, force_console_manifest=(edition == "flockspro")) + _run_update_check_for_cache(key, locale=locale, edition=edition) ) _update_check_inflight[key] = task - owner = True - - try: - info = await task - finally: - if owner: - async with _update_check_lock: - if _update_check_inflight.get(key) is task: - _update_check_inflight.pop(key, None) - - if owner: - ttl = _UPDATE_CHECK_ERROR_CACHE_TTL_SECONDS if info.error else _UPDATE_CHECK_CACHE_TTL_SECONDS - async with _update_check_lock: - _update_check_cache[key] = (time.monotonic() + ttl, info.model_copy(deep=True)) + info = await asyncio.shield(task) return info.model_copy(deep=True) diff --git a/tests/server/routes/test_session_routes.py b/tests/server/routes/test_session_routes.py index 5aa2e8994..4db5ba2a8 100644 --- a/tests/server/routes/test_session_routes.py +++ b/tests/server/routes/test_session_routes.py @@ -182,6 +182,135 @@ async def test_get_session(self, client: AsyncClient, session_id: str): assert resp.status_code == status.HTTP_200_OK assert resp.json()["id"] == session_id + @pytest.mark.asyncio + async def test_context_usage_keeps_inflight_task_after_cancelled_request( + self, + monkeypatch: pytest.MonkeyPatch, + ): + from flocks.server.routes import session as session_routes + from flocks.session.context_usage import ContextUsageSnapshot + + session_routes._context_usage_cache.clear() + session_routes._context_usage_inflight.clear() + + calls = 0 + started = asyncio.Event() + release = asyncio.Event() + session = SimpleNamespace(time=SimpleNamespace(updated=123), provider=None, model=None) + + async def fake_build_context_usage_snapshot(session_id: str, *, session=None): + nonlocal calls + calls += 1 + started.set() + await release.wait() + return ContextUsageSnapshot( + sessionID=session_id, + usedTokens=42, + contextWindow=100, + estimatedTokens=42, + ) + + monkeypatch.setattr( + session_routes, + "build_context_usage_snapshot", + fake_build_context_usage_snapshot, + ) + + first = asyncio.create_task( + session_routes._cached_context_usage_snapshot("ses_context_cancel", session=session) + ) + second = None + try: + await started.wait() + first.cancel() + with pytest.raises(asyncio.CancelledError): + await first + + second = asyncio.create_task( + session_routes._cached_context_usage_snapshot("ses_context_cancel", session=session) + ) + await asyncio.sleep(0) + assert calls == 1 + + release.set() + snapshot = await asyncio.wait_for(second, timeout=1) + assert snapshot.used_tokens == 42 + + cached = await session_routes._cached_context_usage_snapshot("ses_context_cancel", session=session) + assert cached.used_tokens == 42 + assert calls == 1 + finally: + release.set() + if second is not None and not second.done(): + await asyncio.wait_for(second, timeout=1) + session_routes._context_usage_cache.clear() + session_routes._context_usage_inflight.clear() + + @pytest.mark.asyncio + async def test_context_usage_keeps_newer_cache_when_older_task_finishes_late( + self, + monkeypatch: pytest.MonkeyPatch, + ): + from flocks.server.routes import session as session_routes + from flocks.session.context_usage import ContextUsageSnapshot + + session_routes._context_usage_cache.clear() + session_routes._context_usage_inflight.clear() + + old_started = asyncio.Event() + new_started = asyncio.Event() + old_release = asyncio.Event() + new_release = asyncio.Event() + old_session = SimpleNamespace(time=SimpleNamespace(updated=100), provider=None, model=None) + new_session = SimpleNamespace(time=SimpleNamespace(updated=200), provider=None, model=None) + + async def fake_build_context_usage_snapshot(session_id: str, *, session=None): + updated = session.time.updated + if updated == 100: + old_started.set() + await old_release.wait() + else: + new_started.set() + await new_release.wait() + return ContextUsageSnapshot( + sessionID=session_id, + usedTokens=updated, + contextWindow=1000, + estimatedTokens=updated, + ) + + monkeypatch.setattr( + session_routes, + "build_context_usage_snapshot", + fake_build_context_usage_snapshot, + ) + + old_task = asyncio.create_task( + session_routes._cached_context_usage_snapshot("ses_context_order", session=old_session) + ) + new_task = asyncio.create_task( + session_routes._cached_context_usage_snapshot("ses_context_order", session=new_session) + ) + try: + await old_started.wait() + await new_started.wait() + + new_release.set() + new_snapshot = await asyncio.wait_for(new_task, timeout=1) + assert new_snapshot.used_tokens == 200 + + old_release.set() + old_snapshot = await asyncio.wait_for(old_task, timeout=1) + assert old_snapshot.used_tokens == 100 + + assert ("ses_context_order", 200) in session_routes._context_usage_cache + assert ("ses_context_order", 100) not in session_routes._context_usage_cache + finally: + old_release.set() + new_release.set() + session_routes._context_usage_cache.clear() + session_routes._context_usage_inflight.clear() + @pytest.mark.asyncio async def test_get_session_not_found(self, client: AsyncClient): """GET for an unknown session ID returns 404.""" diff --git a/tests/server/routes/test_update_routes.py b/tests/server/routes/test_update_routes.py index 8b1d8544f..10ade912e 100644 --- a/tests/server/routes/test_update_routes.py +++ b/tests/server/routes/test_update_routes.py @@ -183,6 +183,48 @@ async def _fake_check_update(**kwargs): assert [item.current_version for item in results] == ["v1"] * 5 +async def test_check_version_keeps_inflight_task_after_cancelled_request(monkeypatch: pytest.MonkeyPatch): + from flocks.server.routes import update as update_routes + from flocks.updater.models import VersionInfo + + calls = 0 + started = asyncio.Event() + release = asyncio.Event() + + async def _fake_check_update(**kwargs): + nonlocal calls + calls += 1 + started.set() + await release.wait() + return VersionInfo(current_version=f"v{calls}") + + monkeypatch.setattr(update_routes, "check_update", _fake_check_update) + + first = asyncio.create_task(update_routes.check_version(_request(), locale="zh-CN", edition="flocks")) + second = None + try: + await started.wait() + first.cancel() + with pytest.raises(asyncio.CancelledError): + await first + + second = asyncio.create_task(update_routes.check_version(_request(), locale="zh-CN", edition="flocks")) + await asyncio.sleep(0) + assert calls == 1 + + release.set() + result = await asyncio.wait_for(second, timeout=1) + assert result.current_version == "v1" + + cached = await update_routes.check_version(_request(), locale="zh-CN", edition="flocks") + assert cached.current_version == "v1" + assert calls == 1 + finally: + release.set() + if second is not None and not second.done(): + await asyncio.wait_for(second, timeout=1) + + @pytest.mark.skipif( os.environ.get(_MANUAL_REAL_UPGRADE_ENV) != "1", reason=f"manual real upgrade test; set {_MANUAL_REAL_UPGRADE_ENV}=1 to enable", diff --git a/webui/src/pages/Session/index.test.tsx b/webui/src/pages/Session/index.test.tsx index a468b3fdb..0f44eb568 100644 --- a/webui/src/pages/Session/index.test.tsx +++ b/webui/src/pages/Session/index.test.tsx @@ -855,6 +855,27 @@ describe('SessionPage session actions menu', () => { }); }); + it('drops a URL initial message when the target session no longer exists', async () => { + const user = userEvent.setup(); + const message = 'Do not send this to another session'; + sessionApi.get.mockRejectedValue({ response: { status: 404 } }); + + renderSessionPage(`/sessions?session=session-deleted&message=${encodeURIComponent(message)}`); + + await waitFor(() => { + expect(sessionApi.get).toHaveBeenCalledWith('session-deleted'); + expect(screen.getByTestId('session-chat')).toHaveTextContent('no-session'); + expect(screen.getByTestId('session-chat')).toHaveAttribute('data-initial-message', ''); + }); + + await user.click(screen.getByText('Original Session')); + + await waitFor(() => { + expect(screen.getByTestId('session-chat')).toHaveTextContent('session-1'); + expect(screen.getByTestId('session-chat')).toHaveAttribute('data-initial-message', ''); + }); + }); + it('lists the same visible agents as the Agent page selector logic', async () => { const user = userEvent.setup(); useAgents.mockReturnValue({ diff --git a/webui/src/pages/Session/index.tsx b/webui/src/pages/Session/index.tsx index 15c85f1fb..bf3a4a124 100644 --- a/webui/src/pages/Session/index.tsx +++ b/webui/src/pages/Session/index.tsx @@ -574,6 +574,7 @@ export default function SessionPage() { setPendingInitialMessage(messageParam); setPendingInitialDisplayText(displayParam ? buildInstructionDisplayText(displayParam) : null); } else { + setPendingInitialMessage(null); setPendingInitialDisplayText(null); } setSearchParams({}, { replace: true }); @@ -626,6 +627,8 @@ export default function SessionPage() { if (statusCode === 403 || statusCode === 404) { setSelectedSessionId((current) => (current === selectedSessionId ? null : current)); setSelectedSessionFallback(null); + setPendingInitialMessage(null); + setPendingInitialDisplayText(null); writeLastSelectedSessionId(null); } }); From 9fc3a9b33b9be6b5bdf9fea959e10dd825e44e56 Mon Sep 17 00:00:00 2001 From: duguwanglong Date: Fri, 10 Jul 2026 11:12:07 +0800 Subject: [PATCH 12/49] fix(webui): harden optimized loading paths - enforce session ownership and admin boundaries across shared APIs - prevent stale caches, dropped events, and lazy chunk recovery failures - keep cold-start registries concurrent and add regression coverage --- flocks/agent/registry.py | 4 +- flocks/hub/catalog.py | 28 +- flocks/permission/next.py | 13 +- flocks/server/routes/event.py | 275 ++++++++++++++-- flocks/server/routes/global_.py | 4 +- flocks/server/routes/hub.py | 114 +++++-- flocks/server/routes/monitoring.py | 97 ++++-- flocks/server/routes/permission.py | 61 +++- flocks/server/routes/provider.py | 181 +++++++--- flocks/server/routes/question.py | 55 +++- flocks/server/routes/stats.py | 14 +- flocks/server/routes/tool.py | 32 +- flocks/server/routes/update.py | 31 +- flocks/skill/skill.py | 55 +++- tests/hub/test_hub_catalog.py | 80 +++++ tests/permission/test_permission_next.py | 46 +++ .../provider/test_api_service_credentials.py | 20 +- tests/server/routes/test_event_routes.py | 291 ++++++++++++++++ .../routes/test_global_mutation_auth.py | 162 +++++++++ .../routes/test_interaction_access_routes.py | 228 +++++++++++++ .../routes/test_provider_optional_api_key.py | 62 +++- tests/server/routes/test_remaining_routes.py | 12 + tests/server/routes/test_tool_routes.py | 1 + tests/server/routes/test_update_routes.py | 41 +++ tests/server/routes/test_webui_perf_routes.py | 143 +++++++- webui/src/api/monitoring.ts | 14 +- webui/src/api/provider.test.ts | 48 +++ webui/src/api/provider.ts | 34 +- webui/src/api/tool.test.ts | 81 +++++ webui/src/api/tool.ts | 21 ++ .../common/LazyLoadErrorBoundary.test.tsx | 67 ++++ .../common/LazyLoadErrorBoundary.tsx | 70 ++++ webui/src/components/common/SessionChat.tsx | 14 +- .../components/common/UpdateModal.test.tsx | 67 ++++ webui/src/components/common/UpdateModal.tsx | 20 +- webui/src/components/layout/Layout.test.tsx | 6 +- webui/src/components/layout/Layout.tsx | 69 ++-- .../features/session-chat/commands.test.ts | 16 + webui/src/features/session-chat/commands.ts | 2 +- webui/src/hooks/useSSE.test.tsx | 162 ++++++++- webui/src/hooks/useSSE.ts | 49 ++- webui/src/hooks/useSharedResource.test.tsx | 52 +++ webui/src/hooks/useSharedResource.ts | 30 ++ webui/src/hooks/useTasks.test.tsx | 10 + webui/src/hooks/useTasks.ts | 45 ++- webui/src/hooks/useTools.test.tsx | 44 +++ webui/src/hooks/useTools.ts | 9 +- webui/src/hooks/useWorkflow.test.tsx | 16 +- webui/src/hooks/useWorkflow.ts | 15 +- webui/src/locales/en-US/common.json | 2 + webui/src/locales/en-US/model.json | 1 + webui/src/locales/en-US/monitoring.json | 3 +- webui/src/locales/zh-CN/common.json | 2 + webui/src/locales/zh-CN/model.json | 1 + webui/src/locales/zh-CN/monitoring.json | 3 +- webui/src/pages/Hub/index.test.tsx | 311 ++++++++++++++++++ webui/src/pages/Hub/index.tsx | 173 ++++++++-- webui/src/pages/Model/index.test.tsx | 127 ++++++- webui/src/pages/Model/index.tsx | 41 ++- webui/src/pages/Monitoring/index.test.tsx | 62 ++++ webui/src/pages/Monitoring/index.tsx | 20 +- .../Tool/components/APITabContent.test.tsx | 115 +++++++ .../pages/Tool/components/APITabContent.tsx | 34 +- .../Tool/components/MCPTabContent.test.tsx | 73 +++- .../pages/Tool/components/MCPTabContent.tsx | 31 +- webui/src/pages/Tool/index.tsx | 63 ++-- webui/src/pages/Tool/tabCounts.test.ts | 20 +- webui/src/pages/Tool/tabCounts.ts | 15 +- webui/src/routes/index.tsx | 58 ++-- webui/src/types/index.ts | 11 +- webui/src/utils/chunkLoadRecovery.test.ts | 53 +++ webui/src/utils/chunkLoadRecovery.ts | 78 +++++ 72 files changed, 3901 insertions(+), 407 deletions(-) create mode 100644 tests/server/routes/test_global_mutation_auth.py create mode 100644 tests/server/routes/test_interaction_access_routes.py create mode 100644 webui/src/api/tool.test.ts create mode 100644 webui/src/components/common/LazyLoadErrorBoundary.test.tsx create mode 100644 webui/src/components/common/LazyLoadErrorBoundary.tsx create mode 100644 webui/src/components/common/UpdateModal.test.tsx create mode 100644 webui/src/pages/Hub/index.test.tsx create mode 100644 webui/src/pages/Monitoring/index.test.tsx create mode 100644 webui/src/pages/Tool/components/APITabContent.test.tsx create mode 100644 webui/src/utils/chunkLoadRecovery.test.ts create mode 100644 webui/src/utils/chunkLoadRecovery.ts diff --git a/flocks/agent/registry.py b/flocks/agent/registry.py index c2b001985..b5b004b46 100644 --- a/flocks/agent/registry.py +++ b/flocks/agent/registry.py @@ -252,7 +252,7 @@ async def _load_agents() -> Dict[str, AgentInfo]: from flocks.tool.registry import ToolRegistry cfg = await Config.get() - ToolRegistry.init() + await ToolRegistry.init_async() # ── ① Collect context ───────────────────────────────────────────── available_tools = [t.name for t in ToolRegistry.list_tools() if t.enabled] @@ -303,7 +303,7 @@ async def _load_agents() -> Dict[str, AgentInfo]: log.warn("agent.logic.deprecated", {"message": "non-rex agent logic is deprecated; using rex"}) # ── ② YAML agents ───────────────────────────────────────────────── - result = scan_and_load() + result = await asyncio.to_thread(scan_and_load) # Apply global base permissions on top of per-agent YAML permissions. # For agents with explicit tool lists, their deny-all baseline is kept; diff --git a/flocks/hub/catalog.py b/flocks/hub/catalog.py index 6fc39c8db..d82e001f5 100644 --- a/flocks/hub/catalog.py +++ b/flocks/hub/catalog.py @@ -951,7 +951,8 @@ def _contains_any(values: Iterable[str], selected: Optional[list[str]]) -> bool: return any(item.lower() in value_set for item in selected) -def list_catalog( +def filter_catalog_entries( + entries: Iterable[HubCatalogEntry], *, plugin_type: Optional[PluginType] = None, category: Optional[list[str]] = None, @@ -962,7 +963,6 @@ def list_catalog( risk: Optional[list[str]] = None, q: Optional[str] = None, ) -> list[HubCatalogEntry]: - entries = _catalog_entries_snapshot() query = (q or "").strip().lower() def keep(entry: HubCatalogEntry) -> bool: @@ -999,6 +999,30 @@ def keep(entry: HubCatalogEntry) -> bool: return [entry for entry in entries if keep(entry)] +def list_catalog( + *, + plugin_type: Optional[PluginType] = None, + category: Optional[list[str]] = None, + tags: Optional[list[str]] = None, + use_cases: Optional[list[str]] = None, + state: Optional[list[str]] = None, + trust: Optional[list[str]] = None, + risk: Optional[list[str]] = None, + q: Optional[str] = None, +) -> list[HubCatalogEntry]: + return filter_catalog_entries( + _catalog_entries_snapshot(), + plugin_type=plugin_type, + category=category, + tags=tags, + use_cases=use_cases, + state=state, + trust=trust, + risk=risk, + q=q, + ) + + def category_counts() -> dict: taxonomy = load_taxonomy().model_dump(mode="json") entries = list_catalog() diff --git a/flocks/permission/next.py b/flocks/permission/next.py index 5fc7c041f..18bfc51af 100644 --- a/flocks/permission/next.py +++ b/flocks/permission/next.py @@ -238,10 +238,12 @@ async def _apply_reply_without_future( cls._permanent_rules[permission] = "deny" await cls._persist_permanent_rule(permission, "deny") return - if reply == "allow_session": + if reply in {"allow_session", "deny_session"}: if resolved_session_id not in cls._session_permissions: cls._session_permissions[resolved_session_id] = {} - cls._session_permissions[resolved_session_id][permission] = "allow" + cls._session_permissions[resolved_session_id][permission] = ( + "allow" if reply == "allow_session" else "deny" + ) await cls._persist_session_rules(resolved_session_id) @classmethod @@ -376,11 +378,14 @@ async def ask( cls._permanent_rules[permission] = "deny" cls._schedule_persist(cls._persist_permanent_rule(permission, "deny")) raise DeniedError([]) - if reply == "allow_session": + if reply in {"allow_session", "deny_session"}: if session_id not in cls._session_permissions: cls._session_permissions[session_id] = {} - cls._session_permissions[session_id][permission] = "allow" + action = "allow" if reply == "allow_session" else "deny" + cls._session_permissions[session_id][permission] = action cls._schedule_persist(cls._persist_session_rules(session_id)) + if action == "deny": + raise DeniedError([]) return raise PermissionError(f"Unknown permission reply: {reply}") diff --git a/flocks/server/routes/event.py b/flocks/server/routes/event.py index 8b0ca8433..5c55b0d7c 100644 --- a/flocks/server/routes/event.py +++ b/flocks/server/routes/event.py @@ -14,15 +14,21 @@ import asyncio import json import os -from typing import AsyncGenerator, Optional +from collections import deque +from typing import AsyncGenerator, Optional, TYPE_CHECKING, cast from datetime import datetime from fastapi import APIRouter, Request from fastapi.responses import StreamingResponse +from flocks.server.auth import require_user from flocks.utils.log import Log from flocks.utils.id import Identifier +if TYPE_CHECKING: + from flocks.auth.context import AuthUser + from flocks.session.session import SessionInfo + router = APIRouter() log = Log.create(service="event-routes") @@ -44,6 +50,167 @@ _current_directory: str = os.getcwd() +def _event_session_id(event: dict) -> Optional[str]: + """Extract the session id from the event shapes emitted by the runtime.""" + properties = event.get("properties") + if not isinstance(properties, dict): + return None + + for key in ("sessionID", "session_id"): + value = properties.get(key) + if isinstance(value, str) and value: + return value + + for container_name in ("part", "info"): + container = properties.get(container_name) + if not isinstance(container, dict): + continue + for key in ("sessionID", "session_id"): + value = container.get(key) + if isinstance(value, str) and value: + return value + + event_type = event.get("type") + runtime_type = properties.get("runtimeType") + semantic_type = runtime_type if event_type == "runtime.event" else event_type + if isinstance(semantic_type, str) and semantic_type.startswith("session."): + value = properties.get("id") + if isinstance(value, str) and value: + return value + info = properties.get("info") + if isinstance(info, dict): + value = info.get("id") + if isinstance(value, str) and value: + return value + return None + + +async def _get_event_session(session_id: str) -> Optional["SessionInfo"]: + """Load a session without inheriting the publisher's auth context.""" + from flocks.auth.context import reset_current_auth_user, set_current_auth_user + from flocks.session.session import Session, SessionInfo + from flocks.storage.storage import Storage + + token = set_current_auth_user(None) + try: + session = await Session.get_by_id(session_id) + if session is not None: + return session + + cached_sessions = getattr(Session, "_all_sessions_cache", None) or [] + cached = next((item for item in cached_sessions if item.id == session_id), None) + if cached is not None: + return cached + + for key in await Storage.list_keys(prefix="session:"): + if key.endswith(f":{session_id}"): + return await Storage.get(key, SessionInfo) + return None + finally: + reset_current_auth_user(token) + + +def _snapshot_key(event: dict) -> Optional[tuple[str, str, str]]: + """Return the identity of a coalescible accumulated-text snapshot.""" + if event.get("type") != "message.part.updated": + return None + properties = event.get("properties") + part = properties.get("part") if isinstance(properties, dict) else None + if not isinstance(part, dict) or part.get("type") not in {"text", "reasoning", "thinking"}: + return None + session_id = part.get("sessionID") + message_id = part.get("messageID") + part_id = part.get("id") + if not isinstance(session_id, str) or not session_id: + return None + if not isinstance(message_id, str) or not message_id: + return None + if not isinstance(part_id, str) or not part_id: + return None + return session_id, message_id, part_id + + +def _merge_snapshots(previous: dict, current: dict) -> dict: + """Keep the latest full part while preserving all unconsumed deltas.""" + previous_properties = previous.get("properties") + current_properties = current.get("properties") + if not isinstance(previous_properties, dict) or not isinstance(current_properties, dict): + return current + + previous_delta = previous_properties.get("delta") + current_delta = current_properties.get("delta") + if not isinstance(previous_delta, str): + return current + + merged_properties = dict(current_properties) + merged_properties["delta"] = previous_delta + (current_delta if isinstance(current_delta, str) else "") + return {**current, "properties": merged_properties} + + +class EventQueue(asyncio.Queue[dict]): + """Queue with slow-client compaction for accumulated part snapshots.""" + + def __init__(self, maxsize: int = 0) -> None: + super().__init__(maxsize=maxsize) + self._pending_drop_count = 0 + + def _pending_events(self) -> deque[dict]: + return cast(deque[dict], self.__dict__["_queue"]) + + def coalesce_snapshot(self, event: dict) -> bool: + key = _snapshot_key(event) + pending = self._pending_events() + if key is None or not pending: + return False + for index in range(len(pending) - 1, -1, -1): + queued = pending[index] + if _snapshot_key(queued) != key: + continue + del pending[index] + pending.append(_merge_snapshots(queued, event)) + return True + return False + + def drop_snapshots_until(self, target_size: int) -> int: + pending = self._pending_events() + dropped = 0 + while len(pending) > target_size: + snapshot_index = next( + (index for index, queued in enumerate(pending) if _snapshot_key(queued) is not None), + None, + ) + if snapshot_index is None: + break + del pending[snapshot_index] + dropped += 1 + return dropped + + def report_dropped(self, count: int) -> None: + """Emit a recovery marker now or defer it until capacity is freed.""" + if count <= 0: + return + self._pending_drop_count += count + self._flush_drop_marker() + + def _flush_drop_marker(self) -> None: + if self._pending_drop_count <= 0 or self.full(): + return + dropped = self._pending_drop_count + self._pending_drop_count = 0 + super().put_nowait(create_event("server.events_dropped", { + "dropped": dropped, + "reason": "client_backpressure", + })) + + async def get(self) -> dict: + event = await super().get() + # A queue containing only control events has no safe item to evict for + # a marker. Once the client consumes one, append the deferred marker + # without sacrificing those control events. + self._flush_drop_marker() + return event + + def set_event_directory(directory: str): """Set the current directory for SSE events""" global _current_directory @@ -61,8 +228,8 @@ class EventBroadcaster: _instance: Optional["EventBroadcaster"] = None - def __init__(self, queue_maxsize: int = EVENT_QUEUE_MAXSIZE, queue_drop_to: Optional[int] = None): - self._clients: list[asyncio.Queue] = [] + def __init__(self, queue_maxsize: int = EVENT_QUEUE_MAXSIZE, queue_drop_to: Optional[int] = None) -> None: + self._clients: dict[EventQueue, Optional["AuthUser"]] = {} self._lock = asyncio.Lock() self._queue_maxsize = max(1, queue_maxsize) self._queue_drop_to = max( @@ -80,54 +247,101 @@ def get(cls) -> "EventBroadcaster": cls._instance = EventBroadcaster() return cls._instance - async def subscribe(self) -> asyncio.Queue: + async def subscribe(self, user: Optional["AuthUser"] = None) -> EventQueue: """Subscribe a new client""" - queue: asyncio.Queue = asyncio.Queue(maxsize=self._queue_maxsize) + queue = EventQueue(maxsize=self._queue_maxsize) async with self._lock: - self._clients.append(queue) + self._clients[queue] = user return queue - async def unsubscribe(self, queue: asyncio.Queue): + async def unsubscribe(self, queue: EventQueue) -> None: """Unsubscribe a client""" async with self._lock: - if queue in self._clients: - self._clients.remove(queue) + self._clients.pop(queue, None) - async def publish(self, event: dict): + async def publish(self, event: dict) -> None: """Publish event to all clients""" async with self._lock: - clients = list(self._clients) - for queue in clients: + clients = list(self._clients.items()) + + session = None + session_id = _event_session_id(event) + if session_id: + try: + session = await _get_event_session(session_id) + except Exception as exc: + log.warning("event.session_access_lookup_failed", { + "session_id": session_id, + "event_type": event.get("type"), + "error": str(exc), + }) + return + if session is None: + log.debug("event.session_access_missing", { + "session_id": session_id, + "event_type": event.get("type"), + }) + return + + if session is not None: + from flocks.session.policy import SessionPolicy + + for queue, user in clients: + if session is not None and user is not None and not SessionPolicy.can_read(session, user): + continue self._publish_to_queue(queue, event) - def _publish_to_queue(self, queue: asyncio.Queue, event: dict): + def _publish_to_queue(self, queue: EventQueue, event: dict) -> None: + if queue.coalesce_snapshot(event): + return + try: queue.put_nowait(event) return except asyncio.QueueFull: pass - dropped = 0 - while queue.qsize() > self._queue_drop_to: - try: - queue.get_nowait() - dropped += 1 - except asyncio.QueueEmpty: - break + is_snapshot = _snapshot_key(event) is not None + marker_target = min(self._queue_drop_to, max(0, self._queue_maxsize - 2)) + dropped = queue.drop_snapshots_until(marker_target) + + if is_snapshot and queue.qsize() > marker_target: + # A text snapshot must never evict permission/question/session + # control events. If removing older snapshots only leaves room for + # the recovery marker, keep that marker and discard this ordinary + # snapshot so the client knows it must reconcile over REST. + # Include the incoming snapshot itself. If the queue consists only + # of control events, defer the marker until the client frees one + # slot rather than evicting a permission/question/session event. + dropped += 1 + queue.report_dropped(dropped) + log.debug("event.queue.snapshot_dropped", { + "queue_size": queue.qsize(), + "queue_maxsize": self._queue_maxsize, + "event_type": event.get("type"), + }) + return + else: + while queue.qsize() > marker_target: + try: + queue.get_nowait() + dropped += 1 + except asyncio.QueueEmpty: + break - try: - queue.put_nowait(create_event("server.events_dropped", { - "dropped": dropped, - "reason": "client_backpressure", - })) - except asyncio.QueueFull: - pass + reported_drops = 0 + if dropped > 0 and queue.qsize() <= self._queue_maxsize - 2: + queue.report_dropped(dropped) + reported_drops = dropped try: queue.put_nowait(event) except asyncio.QueueFull: dropped += 1 + if dropped > reported_drops: + queue.report_dropped(dropped - reported_drops) + if dropped > 0: log.debug("event.queue.overflow", { "dropped": dropped, @@ -141,7 +355,7 @@ def client_count(self) -> int: """Number of connected clients""" return len(self._clients) - async def shutdown(self): + async def shutdown(self) -> None: """Notify all clients that the server is shutting down, then clear.""" shutdown_event = create_event("server.shutting_down", {}) async with self._lock: @@ -230,7 +444,7 @@ async def publish_runtime_event(runtime_type: str, properties: dict = None, dire async def sse_generator( - queue: asyncio.Queue, + queue: EventQueue, request: Request, directory: str = None, ) -> AsyncGenerator[str, None]: @@ -282,7 +496,8 @@ async def subscribe_events(request: Request): Returns: StreamingResponse with SSE events """ - queue = await EventBroadcaster.get().subscribe() + user = require_user(request) + queue = await EventBroadcaster.get().subscribe(user) log.info("event.subscribe", { "clients": EventBroadcaster.get().client_count, diff --git a/flocks/server/routes/global_.py b/flocks/server/routes/global_.py index 75dfc45a1..afe3fd518 100644 --- a/flocks/server/routes/global_.py +++ b/flocks/server/routes/global_.py @@ -18,6 +18,7 @@ from fastapi.responses import StreamingResponse from pydantic import BaseModel +from flocks.server.auth import require_user from flocks.utils.log import Log from flocks.server.routes.event import EventBroadcaster, sse_generator @@ -54,7 +55,8 @@ async def get_global_events(request: Request): This is the main event endpoint that Flocks TUI uses. """ - queue = await EventBroadcaster.get().subscribe() + user = require_user(request) + queue = await EventBroadcaster.get().subscribe(user) log.info("global.event.subscribe", { "clients": EventBroadcaster.get().client_count, diff --git a/flocks/server/routes/hub.py b/flocks/server/routes/hub.py index 274fcf903..92da6378d 100644 --- a/flocks/server/routes/hub.py +++ b/flocks/server/routes/hub.py @@ -5,13 +5,14 @@ import asyncio from typing import Optional, Union -from fastapi import APIRouter, HTTPException, Query +from fastapi import APIRouter, Depends, HTTPException, Query from fastapi.responses import StreamingResponse from pydantic import BaseModel, Field from flocks.hub.catalog import ( category_counts, clear_catalog_caches, + filter_catalog_entries, legacy_removed_plugin_message, list_catalog, load_manifest, @@ -28,6 +29,7 @@ InstalledPluginRecord, PluginType, ) +from flocks.server.auth import require_admin from flocks.utils.log import Log @@ -84,19 +86,43 @@ def _clear_hub_runtime_caches() -> None: pass -def _build_hub_catalog_facets(items: list[HubCatalogEntry]) -> HubCatalogFacets: - facets = HubCatalogFacets() +def _count_hub_catalog_facet( + items: list[HubCatalogEntry], + attribute: str, +) -> dict[str, int]: + counts: dict[str, int] = {} for item in items: - facets.type[item.type] = facets.type.get(item.type, 0) + 1 - facets.category[item.category] = facets.category.get(item.category, 0) + 1 - facets.state[item.state] = facets.state.get(item.state, 0) + 1 - facets.trust[item.trust] = facets.trust.get(item.trust, 0) + 1 - facets.riskLevel[item.riskLevel] = facets.riskLevel.get(item.riskLevel, 0) + 1 - for tag in item.tags: - facets.tags[tag] = facets.tags.get(tag, 0) + 1 - for use_case in item.useCases: - facets.useCases[use_case] = facets.useCases.get(use_case, 0) + 1 - return facets + values = getattr(item, attribute) + if isinstance(values, str): + values = [values] + for value in values: + counts[value] = counts.get(value, 0) + 1 + return counts + + +def _build_hub_catalog_facets_for_filters( + all_entries: list[HubCatalogEntry], + filters: dict[str, object], +) -> HubCatalogFacets: + """Build each facet with every active filter except that facet itself.""" + dimensions = { + "type": ("plugin_type", "type"), + "category": ("category", "category"), + "tags": ("tags", "tags"), + "useCases": ("use_cases", "useCases"), + "state": ("state", "state"), + "trust": ("trust", "trust"), + "riskLevel": ("risk", "riskLevel"), + } + counts: dict[str, dict[str, int]] = {} + for response_field, (query_field, item_attribute) in dimensions.items(): + facet_filters = dict(filters) + facet_filters[query_field] = None + counts[response_field] = _count_hub_catalog_facet( + filter_catalog_entries(all_entries, **facet_filters), + item_attribute, + ) + return HubCatalogFacets(**counts) @router.get("/hub/catalog", response_model=Union[list[HubCatalogEntry], HubCatalogPageResponse]) @@ -112,19 +138,26 @@ async def hub_catalog( offset: int = Query(0, ge=0), limit: Optional[int] = Query(default=None, ge=1, le=200), ): - entries = await asyncio.to_thread( - list_catalog, - plugin_type=type, - category=_split_csv(category), - tags=_split_csv(tags), - use_cases=_split_csv(useCases), - state=_split_csv(state), - trust=_split_csv(trust), - risk=_split_csv(risk), - q=q, - ) + filters: dict[str, object] = { + "plugin_type": type, + "category": _split_csv(category), + "tags": _split_csv(tags), + "use_cases": _split_csv(useCases), + "state": _split_csv(state), + "trust": _split_csv(trust), + "risk": _split_csv(risk), + "q": q, + } if limit is None and offset == 0: - return entries + return await asyncio.to_thread(list_catalog, **filters) + + def load_page() -> tuple[list[HubCatalogEntry], HubCatalogFacets]: + all_entries = list_catalog() + entries = filter_catalog_entries(all_entries, **filters) + facets = _build_hub_catalog_facets_for_filters(all_entries, filters) + return entries, facets + + entries, facets = await asyncio.to_thread(load_page) page_limit = limit or 25 total = len(entries) @@ -133,7 +166,7 @@ async def hub_catalog( total=total, offset=offset, limit=page_limit, - facets=_build_hub_catalog_facets(entries), + facets=facets, ) @@ -176,7 +209,12 @@ async def hub_plugin_file_content(plugin_type: PluginType, plugin_id: str, path: @router.post("/hub/plugins/{plugin_type}/{plugin_id}/install", response_model=InstalledPluginRecord) -async def hub_install_plugin(plugin_type: PluginType, plugin_id: str, req: HubInstallRequest = HubInstallRequest()): +async def hub_install_plugin( + plugin_type: PluginType, + plugin_id: str, + req: HubInstallRequest = HubInstallRequest(), + _admin: object = Depends(require_admin), +): _guard_legacy_removed_plugin(plugin_type, plugin_id) try: return await install_plugin(plugin_type, plugin_id, scope=req.scope) @@ -186,7 +224,12 @@ async def hub_install_plugin(plugin_type: PluginType, plugin_id: str, req: HubIn @router.post("/hub/plugins/{plugin_type}/{plugin_id}/install/stream") -async def hub_install_plugin_stream(plugin_type: PluginType, plugin_id: str, req: HubInstallRequest = HubInstallRequest()): +async def hub_install_plugin_stream( + plugin_type: PluginType, + plugin_id: str, + req: HubInstallRequest = HubInstallRequest(), + _admin: object = Depends(require_admin), +): _guard_legacy_removed_plugin(plugin_type, plugin_id) if plugin_type != "component": raise HTTPException(status_code=400, detail="Streaming install progress is only supported for components.") @@ -235,7 +278,12 @@ async def run_install() -> None: @router.post("/hub/plugins/{plugin_type}/{plugin_id}/update", response_model=InstalledPluginRecord) -async def hub_update_plugin(plugin_type: PluginType, plugin_id: str, req: HubInstallRequest = HubInstallRequest()): +async def hub_update_plugin( + plugin_type: PluginType, + plugin_id: str, + req: HubInstallRequest = HubInstallRequest(), + _admin: object = Depends(require_admin), +): _guard_legacy_removed_plugin(plugin_type, plugin_id) try: return await update_plugin(plugin_type, plugin_id, scope=req.scope) @@ -245,7 +293,11 @@ async def hub_update_plugin(plugin_type: PluginType, plugin_id: str, req: HubIns @router.delete("/hub/plugins/{plugin_type}/{plugin_id}") -async def hub_uninstall_plugin(plugin_type: PluginType, plugin_id: str): +async def hub_uninstall_plugin( + plugin_type: PluginType, + plugin_id: str, + _admin: object = Depends(require_admin), +): _guard_legacy_removed_plugin(plugin_type, plugin_id) try: removed = await uninstall_plugin(plugin_type, plugin_id) @@ -255,6 +307,6 @@ async def hub_uninstall_plugin(plugin_type: PluginType, plugin_id: str): @router.post("/hub/refresh") -async def hub_refresh(): +async def hub_refresh(_admin: object = Depends(require_admin)): _clear_hub_runtime_caches() return {"count": len(await asyncio.to_thread(list_catalog))} diff --git a/flocks/server/routes/monitoring.py b/flocks/server/routes/monitoring.py index 4b9e331cc..0d5b643be 100644 --- a/flocks/server/routes/monitoring.py +++ b/flocks/server/routes/monitoring.py @@ -9,6 +9,7 @@ from __future__ import annotations import time +import threading from typing import Any, Literal from fastapi import APIRouter, Query @@ -28,19 +29,19 @@ class SystemStatusResponse(BaseModel): status: Literal["healthy", "degraded", "down"] uptime: int - activeSessions: int - activeAgents: int + activeSessions: int | None + activeAgents: int | None mcpServers: dict[str, str] timestamp: int class MetricsSnapshotResponse(BaseModel): timestamp: int - messageRate: float - toolCallRate: float - errorRate: float - avgResponseTime: float - activeRequests: int + messageRate: float | None + toolCallRate: float | None + errorRate: float | None + avgResponseTime: float | None + activeRequests: int | None class PerformanceDataResponse(BaseModel): @@ -58,40 +59,87 @@ def _timestamp_ms() -> int: return int(time.time() * 1000) -async def _active_session_count() -> int: +async def _active_session_count() -> int | None: cached_sessions = getattr(Session, "_all_sessions_cache", None) - if not cached_sessions: - return 0 + if cached_sessions is None: + return None return sum(1 for session in cached_sessions if getattr(session, "status", None) == "active") -async def _active_agent_count() -> int: +async def _active_agent_count() -> int | None: agents = getattr(agent_registry, "_agents_ref", None) - if not agents: - return 0 + if agents is None: + return None return sum(1 for agent in agents.values() if not getattr(agent, "hidden", False)) -async def _mcp_server_statuses() -> dict[str, str]: +async def _mcp_server_statuses() -> tuple[dict[str, str], bool]: try: statuses = await MCP.status() except Exception as exc: log.warning("monitoring.mcp.status_failed", {"error": str(exc)}) - return {} - return { + return {}, False + return ({ name: str(info.status.value if hasattr(info.status, "value") else info.status) for name, info in statuses.items() - } + }, True) + + +_metrics_sample_lock = threading.Lock() +_METRICS_RATE_WINDOW_SECONDS = 60.0 +_metrics_rate_state: tuple[ + float, + float, + float, + float | None, + float | None, +] | None = None + + +def _metrics_clock() -> float: + return time.monotonic() + + +def _sample_tool_rates(total_calls: float, failed_calls: float) -> tuple[float | None, float | None]: + """Return a stable process-wide rate snapshot for the current time window.""" + global _metrics_rate_state + + now = _metrics_clock() + with _metrics_sample_lock: + state = _metrics_rate_state + if state is None: + _metrics_rate_state = (now, total_calls, failed_calls, None, None) + return None, None + + sampled_at, previous_calls, previous_failures, call_rate, failure_rate = state + elapsed_seconds = now - sampled_at + counters_reset = total_calls < previous_calls or failed_calls < previous_failures + if counters_reset: + _metrics_rate_state = (now, total_calls, failed_calls, None, None) + return None, None + if elapsed_seconds < _METRICS_RATE_WINDOW_SECONDS: + return call_rate, failure_rate + + elapsed_minutes = elapsed_seconds / 60 + call_rate = (total_calls - previous_calls) / elapsed_minutes + failure_rate = (failed_calls - previous_failures) / elapsed_minutes + _metrics_rate_state = (now, total_calls, failed_calls, call_rate, failure_rate) + return call_rate, failure_rate @router.get("/status", response_model=SystemStatusResponse) async def get_status() -> SystemStatusResponse: + mcp_servers, mcp_status_available = await _mcp_server_statuses() + unhealthy_mcp = any( + status.lower() in {"error", "failed", "down"} + for status in mcp_servers.values() + ) return SystemStatusResponse( - status="healthy", + status="degraded" if not mcp_status_available or unhealthy_mcp else "healthy", uptime=max(0, int(time.monotonic() - _started_at)), activeSessions=await _active_session_count(), activeAgents=await _active_agent_count(), - mcpServers=await _mcp_server_statuses(), + mcpServers=mcp_servers, timestamp=_timestamp_ms(), ) @@ -101,13 +149,14 @@ async def get_metrics() -> MetricsSnapshotResponse: metrics = get_monitor().get_metrics().get("global", {}) total_calls = float(metrics.get("total_calls") or 0) failed_calls = float(metrics.get("failed_parses") or 0) + tool_call_rate, error_rate = _sample_tool_rates(total_calls, failed_calls) return MetricsSnapshotResponse( timestamp=_timestamp_ms(), - messageRate=0.0, - toolCallRate=total_calls, - errorRate=failed_calls, - avgResponseTime=0.0, - activeRequests=0, + messageRate=None, + toolCallRate=tool_call_rate, + errorRate=error_rate, + avgResponseTime=None, + activeRequests=None, ) diff --git a/flocks/server/routes/permission.py b/flocks/server/routes/permission.py index 4162c7d9b..073652b27 100644 --- a/flocks/server/routes/permission.py +++ b/flocks/server/routes/permission.py @@ -11,9 +11,13 @@ from typing import Dict, List, Any, Optional from datetime import datetime -from fastapi import APIRouter, HTTPException +from fastapi import APIRouter, Depends, HTTPException, status from pydantic import BaseModel +from flocks.auth.context import AuthUser +from flocks.server.auth import require_user +from flocks.session.policy import SessionPolicy +from flocks.session.session import Session from flocks.utils.log import Log from flocks.utils.id import Identifier from flocks.permission.next import PermissionNext @@ -119,21 +123,49 @@ async def remove_permission(permission_id: str) -> bool: return True +async def _can_access_permission_session( + session_id: str, + user: AuthUser, + *, + write: bool, +) -> bool: + session = await Session.get_by_id(session_id) + if session is None: + return False + return SessionPolicy.can_write(session, user) if write else SessionPolicy.can_read(session, user) + + +async def _require_permission_session_access( + session_id: str, + user: AuthUser, + *, + write: bool, +) -> None: + if not await _can_access_permission_session(session_id, user, write=write): + detail = "Only the session owner can reply to permissions" if write else "Permission not found" + raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail=detail) + + @router.get( "", response_model=List[PermissionInfo], summary="List permissions", description="Get all pending permission requests" ) -async def list_permissions() -> List[PermissionInfo]: +async def list_permissions( + user: AuthUser = Depends(require_user), +) -> List[PermissionInfo]: """ List all pending permission requests. Flocks TUI uses this to display permission prompts. """ pending_infos = await PermissionNext.list_pending_infos() - return [ - PermissionInfo( + result: List[PermissionInfo] = [] + for info in pending_infos: + if not await _can_access_permission_session(info.session_id, user, write=False): + continue + result.append(PermissionInfo( id=info.id, sessionID=info.session_id, messageID=info.metadata.get("messageID", ""), @@ -143,9 +175,8 @@ async def list_permissions() -> List[PermissionInfo]: always=info.always, metadata=info.metadata, time=info.time, - ) - for info in pending_infos - ] + )) + return result @router.get( @@ -154,11 +185,15 @@ async def list_permissions() -> List[PermissionInfo]: summary="Get permission", description="Get a specific permission request" ) -async def get_permission_by_id(permission_id: str) -> PermissionInfo: +async def get_permission_by_id( + permission_id: str, + user: AuthUser = Depends(require_user), +) -> PermissionInfo: """Get a specific permission request""" info = await PermissionNext.get_pending_info(permission_id) if not info: raise HTTPException(status_code=404, detail="Permission not found") + await _require_permission_session_access(info.session_id, user, write=False) return PermissionInfo( id=info.id, sessionID=info.session_id, @@ -180,6 +215,7 @@ async def get_permission_by_id(permission_id: str) -> PermissionInfo: async def reply_permission( permission_id: str, request: PermissionReplyRequest, + user: AuthUser = Depends(require_user), ) -> Dict[str, bool]: """ Reply to a permission request. @@ -194,6 +230,7 @@ async def reply_permission( info = await PermissionNext.get_pending_info(permission_id) if not info: raise HTTPException(status_code=404, detail="Permission not found") + await _require_permission_session_access(info.session_id, user, write=True) log.info("permission.reply", { "id": permission_id, @@ -201,7 +238,13 @@ async def reply_permission( "always": request.always, }) - reply = "always" if request.allow and request.always else "allow" if request.allow else "never" if request.always else "deny" + if request.always: + if SessionPolicy.is_admin(user): + reply = "always" if request.allow else "never" + else: + reply = "allow_session" if request.allow else "deny_session" + else: + reply = "allow" if request.allow else "deny" await PermissionNext.reply(permission_id, reply, session_id=info.session_id) return {"success": True} diff --git a/flocks/server/routes/provider.py b/flocks/server/routes/provider.py index 31efde42e..8c8cf1866 100644 --- a/flocks/server/routes/provider.py +++ b/flocks/server/routes/provider.py @@ -12,12 +12,13 @@ import time from pathlib import Path from typing import Any, Dict, List, Optional -from fastapi import APIRouter, Body, HTTPException, Query, status +from fastapi import APIRouter, Body, Depends, HTTPException, Query, status from pydantic import BaseModel, Field, ConfigDict from flocks.utils.log import Log from flocks.provider.provider import Provider, ModelInfo as ProviderModelInfo from flocks.security.secrets import SecretManager +from flocks.server.auth import require_admin from flocks.config.config import Config from flocks.config.config_writer import ConfigWriter from flocks.storage.storage import Storage @@ -661,7 +662,8 @@ async def get_all_provider_auth() -> Dict[str, List[Any]]: ) async def oauth_authorize( provider_id: str, - method: int = 0 + method: int = 0, + _admin: object = Depends(require_admin), ) -> Optional[Dict[str, Any]]: """ Initiate OAuth authorization @@ -684,7 +686,8 @@ async def oauth_authorize( async def oauth_callback( provider_id: str, method: int, - code: Optional[str] = None + code: Optional[str] = None, + _admin: object = Depends(require_admin), ) -> bool: """ Handle OAuth callback @@ -714,6 +717,7 @@ async def list_api_services_route(): async def update_api_service_route( provider_id: str, request: Dict[str, Any] = Body(...), + _admin: object = Depends(require_admin), ): return await update_api_service(provider_id, APIServiceUpdateRequest.model_validate(request)) @@ -724,7 +728,10 @@ async def update_api_service_route( summary="Delete API service", description="Delete an API service configuration and its stored credential." ) -async def delete_api_service_route(provider_id: str): +async def delete_api_service_route( + provider_id: str, + _admin: object = Depends(require_admin), +): return await delete_api_service(provider_id) @@ -835,7 +842,11 @@ class ProviderConfigRequest(BaseModel): summary="Configure provider", description="Configure provider with API key and settings" ) -async def configure_provider(provider_id: str, config: ProviderConfigRequest) -> ProviderInfo: +async def configure_provider( + provider_id: str, + config: ProviderConfigRequest, + _admin: object = Depends(require_admin), +) -> ProviderInfo: """ Configure provider @@ -886,7 +897,10 @@ async def configure_provider(provider_id: str, config: ProviderConfigRequest) -> summary="Test provider", description="Test provider connection" ) -async def test_provider(provider_id: str) -> Dict[str, Any]: +async def test_provider( + provider_id: str, + _admin: object = Depends(require_admin), +) -> Dict[str, Any]: """ Test provider connection @@ -941,7 +955,11 @@ async def test_provider(provider_id: str) -> Dict[str, Any]: summary="Update provider", description="Update provider configuration" ) -async def update_provider(provider_id: str, config: ProviderConfigRequest) -> ProviderInfo: +async def update_provider( + provider_id: str, + config: ProviderConfigRequest, + _admin: object = Depends(require_admin), +) -> ProviderInfo: """ Update provider configuration @@ -973,7 +991,6 @@ async def update_provider(provider_id: str, config: ProviderConfigRequest) -> Pr config_key = f"provider_config/{provider_id}" await Storage.write(config_key, { "provider_id": provider_id, - "api_key": config.api_key, "base_url": config.base_url, "custom_settings": config.custom_settings, }) @@ -1589,7 +1606,10 @@ class ProviderCredentialResponse(BaseModel): summary="Get provider credentials (masked)", description="Get masked credential information for a registered LLM provider." ) -async def get_provider_credentials(provider_id: str): +async def get_provider_credentials( + provider_id: str, + _admin: object = Depends(require_admin), +): """Get credential info for an LLM provider (_llm_key convention). - api_key: from .secret.json (_llm_key first, legacy _api_key fallback) @@ -1631,10 +1651,10 @@ async def get_provider_credentials(provider_id: str): return ProviderCredentialResponse( secret_id=secret_id if api_key else None, - api_key=ui_api_key, + api_key=None, api_key_masked=( - SecretManager.mask(api_key) - if api_key and not is_placeholder + SecretManager.mask(ui_api_key) + if ui_api_key else None ), base_url=base_url, @@ -1651,7 +1671,11 @@ async def get_provider_credentials(provider_id: str): summary="Set provider credentials", description="Set authentication credentials for a provider or API service." ) -async def set_provider_credentials(provider_id: str, request: ProviderCredentialRequest): +async def set_provider_credentials( + provider_id: str, + request: ProviderCredentialRequest, + _admin: object = Depends(require_admin), +): """Set credentials for a provider. - api_key → .secret.json @@ -1665,37 +1689,66 @@ async def set_provider_credentials(provider_id: str, request: ProviderCredential try: # OpenAI-compatible endpoints (e.g. internal LLM gateways without auth) # and user-defined custom-* providers may legitimately have no API key. - # For other providers (OpenAI, Anthropic, etc.) the key is still required. + # For an already configured provider, an omitted/blank key means "keep + # the existing secret" so callers can safely update only the base URL + # or model selection without receiving the original key back first. api_key_optional = ( provider_id == "openai-compatible" or provider_id.startswith("custom-") ) + secrets = get_secret_manager() + secret_id = request.secret_id or f"{provider_id}_llm_key" effective_api_key = (request.api_key or "").strip() + preserve_existing_secret = False + if not effective_api_key: - if not api_key_optional: - raise HTTPException(status_code=400, detail="API key required") - # Persist a sentinel value so downstream OpenAI SDK clients (which - # require a non-empty key argument) keep working transparently. - # The GET endpoint masks this back to ``None`` for the UI. - effective_api_key = _NO_API_KEY_PLACEHOLDER + existing_secret_ids = [secret_id] + legacy_secret_id = f"{provider_id}_api_key" + if request.secret_id is None and legacy_secret_id not in existing_secret_ids: + existing_secret_ids.append(legacy_secret_id) + + for candidate in existing_secret_ids: + existing_value = secrets.get(candidate) + if isinstance(existing_value, str) and existing_value.strip(): + secret_id = candidate + effective_api_key = existing_value.strip() + preserve_existing_secret = True + break - secrets = get_secret_manager() + if not effective_api_key: + inline_api_key = _get_inline_provider_api_key(provider_id) + if inline_api_key: + effective_api_key = inline_api_key + preserve_existing_secret = True + + if not effective_api_key: + if not api_key_optional: + raise HTTPException(status_code=400, detail="API key required") + # Persist a sentinel value so downstream OpenAI SDK clients + # (which require a non-empty key argument) keep working. + effective_api_key = _NO_API_KEY_PLACEHOLDER # 1. Save API key to .secret.json using _llm_key convention for LLM providers - secret_id = request.secret_id or f"{provider_id}_llm_key" - secrets.set(secret_id, effective_api_key) - if _is_placeholder_api_key(effective_api_key): - masked = _NO_API_KEY_LOG_MARKER - elif len(effective_api_key) > 8: - masked = f"{effective_api_key[:4]}***{effective_api_key[-4:]}" + if preserve_existing_secret: + log.info("provider.credentials.preserved", { + "provider_id": provider_id, + "secret_id": secret_id, + "base_url": request.base_url, + }) else: - masked = "***" - log.info("provider.credentials.saving", { - "provider_id": provider_id, - "secret_id": secret_id, - "api_key_masked": masked, - "api_key_optional": api_key_optional, - "base_url": request.base_url, - }) + secrets.set(secret_id, effective_api_key) + if _is_placeholder_api_key(effective_api_key): + masked = _NO_API_KEY_LOG_MARKER + elif len(effective_api_key) > 8: + masked = f"{effective_api_key[:4]}***{effective_api_key[-4:]}" + else: + masked = "***" + log.info("provider.credentials.saving", { + "provider_id": provider_id, + "secret_id": secret_id, + "api_key_masked": masked, + "api_key_optional": api_key_optional, + "base_url": request.base_url, + }) # 2. Ensure provider entry exists in flocks.json and update base_url / name raw_provider = ConfigWriter.get_provider_raw(provider_id) @@ -1705,10 +1758,13 @@ async def set_provider_credentials(provider_id: str, request: ProviderCredential ConfigWriter.update_provider_field( provider_id, "options.baseURL", request.base_url ) - # Ensure apiKey reference is set - ConfigWriter.update_provider_field( - provider_id, "options.apiKey", f"{{secret:{provider_id}_llm_key}}" - ) + if not preserve_existing_secret: + # A newly supplied key uses the canonical secret reference. + # When preserving a stored or inline key, leave its existing + # config reference untouched. + ConfigWriter.update_provider_field( + provider_id, "options.apiKey", f"{{secret:{secret_id}}}" + ) if request.provider_name: ConfigWriter.update_provider_field( provider_id, "name", request.provider_name @@ -1811,7 +1867,10 @@ async def set_provider_credentials(provider_id: str, request: ProviderCredential summary="Delete provider credentials", description="Delete stored credentials for a provider or API service." ) -async def delete_provider_credentials(provider_id: str): +async def delete_provider_credentials( + provider_id: str, + _admin: object = Depends(require_admin), +): """Delete a provider: remove from flocks.json and .secret.json.""" from flocks.security import get_secret_manager @@ -1866,7 +1925,10 @@ async def delete_provider_credentials(provider_id: str): summary="Get API service credentials (masked)", description="Get masked credential information for an API service (tool integrations)." ) -async def get_service_credentials(provider_id: str): +async def get_service_credentials( + provider_id: str, + _admin: object = Depends(require_admin), +): """Get credential info for an API service (_api_key convention). Reads the secret_id from flocks.json api_services.{id}.secret first, @@ -1923,15 +1985,30 @@ async def get_service_credentials(provider_id: str): field_values["api_key"] = split_api_key field_values["secret"] = split_secret + sensitive_field_names = { + field.key + for field in schema + if field.sensitive or field.storage == "secret" + } + sensitive_field_names.update(secret_ids) + safe_field_values = { + field_name: ( + SecretManager.mask(value) + if value and field_name in sensitive_field_names + else value + ) + for field_name, value in field_values.items() + } + return ProviderCredentialResponse( secret_id=secret_ids.get("api_key"), - api_key=field_values.get("api_key"), + api_key=None, api_key_masked=SecretManager.mask(field_values["api_key"]) if field_values.get("api_key") else None, - secret=field_values.get("secret"), + secret=None, secret_masked=SecretManager.mask(field_values["secret"]) if field_values.get("secret") else None, base_url=field_values.get("base_url"), username=field_values.get("username"), - fields=field_values or None, + fields=safe_field_values or None, secret_ids=secret_ids or None, has_credential=bool(any(value for value in field_values.values())), ) @@ -1946,7 +2023,11 @@ async def get_service_credentials(provider_id: str): summary="Set API service credentials", description="Set authentication credentials for an API service." ) -async def set_service_credentials(provider_id: str, request: ProviderCredentialRequest): +async def set_service_credentials( + provider_id: str, + request: ProviderCredentialRequest, + _admin: object = Depends(require_admin), +): """Set credentials for an API service (_api_key convention). Saves to .secret.json AND writes the secret reference into flocks.json @@ -2100,7 +2181,11 @@ class TestCredentialRequest(BaseModel): summary="Test provider credentials", description="Test if the stored credentials are valid by making a real chat API call." ) -async def test_provider_credentials(provider_id: str, body: Optional[TestCredentialRequest] = None): +async def test_provider_credentials( + provider_id: str, + body: Optional[TestCredentialRequest] = None, + _admin: object = Depends(require_admin), +): """Test credentials for a provider or API service by making a real API call""" from flocks.security import get_secret_manager @@ -2717,7 +2802,9 @@ async def get_api_services_status(): summary="Refresh API service connectivity status", description="Manually trigger a refresh of all API service connectivity statuses." ) -async def refresh_api_services_status(): +async def refresh_api_services_status( + _admin: object = Depends(require_admin), +): """Refresh all API service connectivity statuses and cache them.""" try: await Storage.init() diff --git a/flocks/server/routes/question.py b/flocks/server/routes/question.py index 4696cb722..ef33b301a 100644 --- a/flocks/server/routes/question.py +++ b/flocks/server/routes/question.py @@ -9,9 +9,13 @@ from typing import Dict, List, Any, Optional -from fastapi import APIRouter, HTTPException +from fastapi import APIRouter, Depends, HTTPException, status from pydantic import BaseModel +from flocks.auth.context import AuthUser +from flocks.server.auth import require_user +from flocks.session.policy import SessionPolicy +from flocks.session.session import Session from flocks.utils.log import Log @@ -76,6 +80,21 @@ def clear_request_state(request_id: str) -> None: _request_rejected.discard(request_id) +async def _require_question_session_access( + session_id: str, + user: AuthUser, + *, + write: bool, +) -> None: + session = await Session.get_by_id(session_id) + if session is None: + raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Session not found") + allowed = SessionPolicy.can_write(session, user) if write else SessionPolicy.can_read(session, user) + if not allowed: + detail = "Only the session owner can answer questions" if write else "Session access denied" + raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail=detail) + + async def reject_session_questions(session_id: str) -> int: """Reject all pending question requests for a session. @@ -126,7 +145,11 @@ def has_pending_questions(session_id: str) -> bool: summary="List pending questions for a session", description="Return all unanswered Question requests for the given session", ) -async def get_pending_question_requests(session_id: str) -> List[Dict[str, Any]]: +async def get_pending_question_requests( + session_id: str, + user: AuthUser = Depends(require_user), +) -> List[Dict[str, Any]]: + await _require_question_session_access(session_id, user, write=False) return list_question_requests(session_id) @@ -138,9 +161,16 @@ async def get_pending_question_requests(session_id: str) -> List[Dict[str, Any]] async def reply_question_request( request_id: str, request: QuestionRequestReply, + user: AuthUser = Depends(require_user), ) -> Dict[str, Any]: - if request_id not in _question_requests: + question_request = _question_requests.get(request_id) + if question_request is None: raise HTTPException(status_code=404, detail="Question request not found") + await _require_question_session_access( + question_request.get("sessionID", ""), + user, + write=True, + ) log.info("question.request.reply", { "request_id": request_id, @@ -150,7 +180,6 @@ async def reply_question_request( try: from flocks.server.routes.event import publish_event - question_request = _question_requests[request_id] await publish_event("question.replied", { "sessionID": question_request.get("sessionID", ""), "requestID": request_id, @@ -159,7 +188,7 @@ async def reply_question_request( except Exception as e: log.error("question.replied.event.failed", {"error": str(e)}) - del _question_requests[request_id] + _question_requests.pop(request_id, None) return {"success": True} @@ -168,16 +197,24 @@ async def reply_question_request( summary="Reject question request", description="Reject a QuestionRequest (user doesn't want to answer)", ) -async def reject_question_request(request_id: str) -> Dict[str, bool]: - if request_id not in _question_requests: +async def reject_question_request( + request_id: str, + user: AuthUser = Depends(require_user), +) -> Dict[str, bool]: + question_request = _question_requests.get(request_id) + if question_request is None: raise HTTPException(status_code=404, detail="Question request not found") + await _require_question_session_access( + question_request.get("sessionID", ""), + user, + write=True, + ) log.info("question.request.reject", {"request_id": request_id}) _request_rejected.add(request_id) try: from flocks.server.routes.event import publish_event - question_request = _question_requests[request_id] await publish_event("question.rejected", { "sessionID": question_request.get("sessionID", ""), "requestID": request_id, @@ -185,7 +222,7 @@ async def reject_question_request(request_id: str) -> Dict[str, bool]: except Exception as e: log.error("question.rejected.event.failed", {"error": str(e)}) - del _question_requests[request_id] + _question_requests.pop(request_id, None) return {"success": True} diff --git a/flocks/server/routes/stats.py b/flocks/server/routes/stats.py index f2f3a4d57..af2e74766 100644 --- a/flocks/server/routes/stats.py +++ b/flocks/server/routes/stats.py @@ -15,6 +15,7 @@ from pydantic import BaseModel from flocks.agent.registry import Agent +from flocks.provider.provider import Provider from flocks.server.routes.provider import list_providers from flocks.server.routes.workflow import _list_workflows_from_fs, _migrate_storage_to_filesystem from flocks.skill.skill import Skill @@ -88,12 +89,16 @@ async def _count_agents() -> int: async def _count_workflows() -> int: await _migrate_storage_to_filesystem() - return len(_list_workflows_from_fs()) + return await asyncio.to_thread(lambda: len(_list_workflows_from_fs())) async def _count_skills() -> int: skills = await Skill.all() - return sum(1 for skill in skills if getattr(skill, "category", None) != "system") + return sum( + 1 + for skill in skills + if getattr(skill, "category", None) != "system" + ) async def _count_tools() -> int: @@ -102,6 +107,7 @@ async def _count_tools() -> int: async def _count_models() -> int: + await asyncio.to_thread(Provider._ensure_initialized) providers = await list_providers() connected = set(providers.connected or []) return sum( @@ -114,6 +120,10 @@ async def _count_models() -> int: @router.get("/summary", response_model=SystemStatsResponse) async def get_system_stats_summary() -> SystemStatsResponse: failures: list[str] = [] + + # Agent and Skill cold loading now wait on their registries without + # blocking the event loop, so all independent summary sources can stay + # concurrent instead of paying a separate registry warm-up phase. dashboard, agents, workflows, skills, tools, models = await asyncio.gather( _safe_dashboard(failures), _safe_count("agents", _count_agents, failures), diff --git a/flocks/server/routes/tool.py b/flocks/server/routes/tool.py index 2c9da905b..65421f7d3 100644 --- a/flocks/server/routes/tool.py +++ b/flocks/server/routes/tool.py @@ -123,6 +123,7 @@ class ToolListFacets(BaseModel): """Facet counts for server-side tool list filtering.""" category: Dict[str, int] = Field(default_factory=dict) source: Dict[str, int] = Field(default_factory=dict) + source_groups: Dict[str, int] = Field(default_factory=dict) source_name: Dict[str, int] = Field(default_factory=dict) enabled: Dict[str, int] = Field(default_factory=dict) @@ -348,6 +349,17 @@ def _build_tool_facets(items: Sequence[ToolInfoResponse | ToolListIndexItem]) -> return facets +def _build_source_group_counts( + items: Sequence[ToolInfoResponse | ToolListIndexItem], +) -> Dict[str, int]: + groups: Dict[str, set[str]] = {} + for item in items: + if not item.source_name: + continue + groups.setdefault(item.source, set()).add(item.source_name) + return {source: len(source_names) for source, source_names in groups.items()} + + def _sort_tool_items( items: Sequence[ToolInfoResponse | ToolListIndexItem], sort_by: Literal["category", "source", "source_name", "enabled", "name"], @@ -747,6 +759,15 @@ async def list_tools_page( enabled_filter=enabled_filter, query=query, ) + source_facet_items = _filter_tool_items( + all_items, + category_filter=category_filter, + source_filter=source_filter, + source_name_filter=source_name_filter, + enabled_filter=enabled_filter, + query=query, + include_source=False, + ) facets = ToolListFacets( category=_build_tool_facets(_filter_tool_items( all_items, @@ -757,15 +778,8 @@ async def list_tools_page( query=query, include_category=False, )).category, - source=_build_tool_facets(_filter_tool_items( - all_items, - category_filter=category_filter, - source_filter=source_filter, - source_name_filter=source_name_filter, - enabled_filter=enabled_filter, - query=query, - include_source=False, - )).source, + source=_build_tool_facets(source_facet_items).source, + source_groups=_build_source_group_counts(source_facet_items), source_name=_build_tool_facets(_filter_tool_items( all_items, category_filter=category_filter, diff --git a/flocks/server/routes/update.py b/flocks/server/routes/update.py index ddf3b0af9..fac08eb06 100644 --- a/flocks/server/routes/update.py +++ b/flocks/server/routes/update.py @@ -61,24 +61,31 @@ async def _check_update_cached( force: bool, ) -> VersionInfo: key = _update_cache_key(locale, edition) - if force: - info = await check_update(locale=locale, force_console_manifest=(edition == "flockspro")) - ttl = _UPDATE_CHECK_ERROR_CACHE_TTL_SECONDS if info.error else _UPDATE_CHECK_CACHE_TTL_SECONDS - _update_check_cache[key] = (time.monotonic() + ttl, info.model_copy(deep=True)) - return info - now = time.monotonic() + task: asyncio.Task[VersionInfo] async with _update_check_lock: - cached = _update_check_cache.get(key) - if cached and cached[0] > now: - return cached[1].model_copy(deep=True) - - task = _update_check_inflight.get(key) - if task is None: + if force: + # Make the forced check the authoritative in-flight request. An + # older non-forced task may still finish for its original caller, + # but its identity guard in _run_update_check_for_cache prevents + # it from overwriting this newer result. task = asyncio.create_task( _run_update_check_for_cache(key, locale=locale, edition=edition) ) _update_check_inflight[key] = task + else: + cached = _update_check_cache.get(key) + if cached and cached[0] > now: + return cached[1].model_copy(deep=True) + + existing_task = _update_check_inflight.get(key) + if existing_task is None: + task = asyncio.create_task( + _run_update_check_for_cache(key, locale=locale, edition=edition) + ) + _update_check_inflight[key] = task + else: + task = existing_task info = await asyncio.shield(task) return info.model_copy(deep=True) diff --git a/flocks/skill/skill.py b/flocks/skill/skill.py index 2bdb8f3cc..db5f2e1e5 100644 --- a/flocks/skill/skill.py +++ b/flocks/skill/skill.py @@ -6,6 +6,7 @@ """ import json +import asyncio import os import glob import re @@ -204,6 +205,45 @@ class Skill: """ _cache: Optional[Dict[str, SkillInfo]] = None + _cache_lock = threading.Lock() + _cache_generation = 0 + _cache_loads: Dict[int, threading.Event] = {} + + @classmethod + def _all_sync(cls) -> List[SkillInfo]: + """Discover skills once per cache generation without stale refills.""" + while True: + with cls._cache_lock: + if cls._cache is not None: + return list(cls._cache.values()) + generation = cls._cache_generation + completion = cls._cache_loads.get(generation) + should_discover = completion is None + if completion is None: + completion = threading.Event() + cls._cache_loads[generation] = completion + + if not should_discover: + completion.wait() + continue + + try: + discovered = cls._discover() + except BaseException: + with cls._cache_lock: + cls._cache_loads.pop(generation, None) + completion.set() + raise + + with cls._cache_lock: + if generation == cls._cache_generation and cls._cache is None: + cls._cache = discovered + cls._cache_loads.pop(generation, None) + completion.set() + if generation == cls._cache_generation and cls._cache is not None: + return list(cls._cache.values()) + # The cache was invalidated while discovery was running. Repeat + # against the new generation instead of publishing stale data. @classmethod def _parse_skill_md(cls, filepath: str, source: Optional[str] = None) -> Optional[SkillInfo]: @@ -460,10 +500,7 @@ async def all(cls) -> List[SkillInfo]: Returns: List of all discovered skills """ - if cls._cache is None: - cls._cache = cls._discover() - - return list(cls._cache.values()) + return await asyncio.to_thread(cls._all_sync) @classmethod async def list_enabled(cls) -> List[SkillInfo]: @@ -495,10 +532,8 @@ async def get(cls, name: str) -> Optional[SkillInfo]: Returns: SkillInfo or None if not found """ - if cls._cache is None: - cls._cache = cls._discover() - - skill = cls._cache.get(name) + skills = await cls.all() + skill = next((item for item in skills if item.name == name), None) if not skill: return None return skill @@ -506,7 +541,9 @@ async def get(cls, name: str) -> Optional[SkillInfo]: @classmethod def clear_cache(cls) -> None: """Clear the skill cache (for testing or forced refresh)""" - cls._cache = None + with cls._cache_lock: + cls._cache = None + cls._cache_generation += 1 log.info("skill.cache.cleared") @classmethod diff --git a/tests/hub/test_hub_catalog.py b/tests/hub/test_hub_catalog.py index b1741cd22..f26ec863e 100644 --- a/tests/hub/test_hub_catalog.py +++ b/tests/hub/test_hub_catalog.py @@ -614,9 +614,18 @@ async def test_catalog_clears_stale_skill_record_after_external_delete(isolated_ def test_hub_routes_cover_catalog_files_install_and_uninstall(isolated_hub_env): + from flocks.auth.context import AuthUser + from flocks.server.auth import require_admin from flocks.server.routes.hub import router app = FastAPI() + app.dependency_overrides[require_admin] = lambda: AuthUser( + id="hub-test-admin", + username="hub-test-admin", + role="admin", + status="active", + must_reset_password=False, + ) app.include_router(router, prefix="/api") client = TestClient(app, raise_server_exceptions=True) @@ -667,6 +676,68 @@ def test_hub_routes_cover_catalog_files_install_and_uninstall(isolated_hub_env): assert any(item["id"] == "ndr-alert-analysis" for item in available_catalog) +def test_hub_paginated_facets_exclude_their_own_filter(): + from flocks.hub.models import HubCatalogEntry + from flocks.server.routes import hub as hub_routes + + entries = [ + HubCatalogEntry( + id="skill-installed", + type="skill", + name="Installed skill", + category="security", + tags=["triage"], + useCases=["incident-response"], + trust="official", + riskLevel="low", + state="installed", + manifestPath="skills/installed.json", + ), + HubCatalogEntry( + id="agent-installed", + type="agent", + name="Installed agent", + category="security", + tags=["triage"], + useCases=["incident-response"], + trust="official", + riskLevel="low", + state="installed", + manifestPath="agents/installed.json", + ), + HubCatalogEntry( + id="skill-available", + type="skill", + name="Available skill", + category="productivity", + tags=["automation"], + useCases=["operations"], + trust="community", + riskLevel="medium", + state="available", + manifestPath="skills/available.json", + ), + ] + + facets = hub_routes._build_hub_catalog_facets_for_filters( + entries, + { + "plugin_type": "skill", + "category": None, + "tags": None, + "use_cases": None, + "state": ["installed"], + "trust": None, + "risk": None, + "q": None, + }, + ) + + assert facets.type == {"skill": 1, "agent": 1} + assert facets.state == {"installed": 1, "available": 1} + assert facets.category == {"security": 1} + + def test_hub_refresh_clears_catalog_and_device_template_caches(monkeypatch): from flocks.server.routes import hub as hub_routes @@ -683,6 +754,8 @@ def test_hub_refresh_clears_catalog_and_device_template_caches(monkeypatch): def test_hub_component_install_stream_reports_child_progress(isolated_hub_env, monkeypatch: pytest.MonkeyPatch): + from flocks.auth.context import AuthUser + from flocks.server.auth import require_admin from flocks.server.routes.hub import router async def noop_refresh(_plugin_type): @@ -692,6 +765,13 @@ async def noop_refresh(_plugin_type): _patch_webui_bundle_build(monkeypatch) app = FastAPI() + app.dependency_overrides[require_admin] = lambda: AuthUser( + id="hub-test-admin", + username="hub-test-admin", + role="admin", + status="active", + must_reset_password=False, + ) app.include_router(router, prefix="/api") client = TestClient(app, raise_server_exceptions=True) diff --git a/tests/permission/test_permission_next.py b/tests/permission/test_permission_next.py index 9e8c5d5f9..e0fba7b67 100644 --- a/tests/permission/test_permission_next.py +++ b/tests/permission/test_permission_next.py @@ -72,6 +72,26 @@ async def test_reply_persists_session_rule_without_in_memory_future(permission_s assert await Storage.get(f"{PermissionNext._SESSION_PREFIX}{request.session_id}") == {"write": "allow"} +@pytest.mark.asyncio +async def test_reply_persists_session_denial_without_in_memory_future(permission_storage) -> None: + request = PermissionRequestInfo( + id="per_testsessiondeny000000001", + sessionID="ses_testsessiondeny00000001", + permission="bash", + patterns=["*"], + ) + await Storage.set( + f"{PermissionNext._PENDING_PREFIX}{request.id}", + request.model_dump(by_alias=True), + "permission_pending", + ) + + await PermissionNext.reply(request.id, "deny_session", session_id=request.session_id) + + assert PermissionNext._session_permissions[request.session_id]["bash"] == "deny" + assert await Storage.get(f"{PermissionNext._SESSION_PREFIX}{request.session_id}") == {"bash": "deny"} + + @pytest.mark.asyncio @pytest.mark.parametrize("reply", ["allow", "once"]) async def test_reply_unblocks_waiting_request_via_persisted_reply_when_memory_future_missing( @@ -130,6 +150,32 @@ async def test_reply_denies_waiting_request_via_persisted_reply_when_memory_futu assert await Storage.get(f"{PermissionNext._REPLY_PREFIX}{request_id}") is None +@pytest.mark.asyncio +async def test_session_denial_applies_to_active_request_without_global_rule(permission_storage) -> None: + request_id = "per_waiting_session_deny" + session_id = "ses_waiting_session_deny" + ask_task = asyncio.create_task( + PermissionNext.ask( + session_id=session_id, + permission="bash", + patterns=["*"], + ruleset=[], + request_id=request_id, + ) + ) + + while request_id not in PermissionNext._pending: + await asyncio.sleep(0) + + await PermissionNext.reply(request_id, "deny_session", session_id=session_id) + + with pytest.raises(DeniedError): + await asyncio.wait_for(ask_task, timeout=2) + + assert PermissionNext._session_permissions[session_id]["bash"] == "deny" + assert "bash" not in PermissionNext._permanent_rules + + @pytest.mark.asyncio async def test_state_load_retries_after_transient_storage_failure(permission_storage, monkeypatch: pytest.MonkeyPatch) -> None: await Storage.set(f"{PermissionNext._PERMANENT_PREFIX}bash", "allow", "permission_rule") diff --git a/tests/provider/test_api_service_credentials.py b/tests/provider/test_api_service_credentials.py index d6013be8c..e42620b24 100644 --- a/tests/provider/test_api_service_credentials.py +++ b/tests/provider/test_api_service_credentials.py @@ -2,6 +2,8 @@ import pytest +from flocks.security.secrets import SecretManager + class TestAPIServiceCredentials: @pytest.mark.asyncio @@ -25,7 +27,9 @@ async def test_get_service_credentials_returns_base_url_and_username(self): result = await get_service_credentials("skyeye_api") assert result.secret_id == "skyeye_api_key" - assert result.api_key == "skyeye-login-key" + assert result.api_key is None + assert result.api_key_masked + assert result.api_key_masked != "skyeye-login-key" assert result.base_url == "https://skyeye-domain/skyeye" assert result.username == "skyeye" assert result.has_credential is True @@ -160,8 +164,10 @@ async def test_get_service_credentials_returns_tdp_secret(self): result = await get_service_credentials("tdp_api") assert result.secret_id == "tdp_api_key" - assert result.api_key == "tdp-api-key" - assert result.secret == "tdp-secret" + assert result.api_key is None + assert result.api_key_masked != "tdp-api-key" + assert result.secret is None + assert result.secret_masked != "tdp-secret" assert result.base_url == "https://tdp.example.com" assert result.has_credential is True @@ -231,8 +237,10 @@ async def test_get_service_credentials_splits_legacy_onesec_combined_secret(self result = await get_service_credentials("onesec_api") assert result.secret_id == "onesec_credentials" - assert result.api_key == "onesec-api-key" - assert result.secret == "onesec-secret" + assert result.api_key is None + assert result.api_key_masked != "onesec-api-key" + assert result.secret is None + assert result.secret_masked != "onesec-secret" assert result.base_url == "https://console.onesec.net" assert result.has_credential is True @@ -320,7 +328,7 @@ async def test_get_service_credentials_returns_dynamic_qingteng_fields(self): assert result.fields == { "base_url": "https://qt.example.com:8443/openapi", "username": "alice", - "password": "qt-secret", + "password": SecretManager.mask("qt-secret"), } assert result.secret_ids == {"password": "qingteng_password"} assert result.has_credential is True diff --git a/tests/server/routes/test_event_routes.py b/tests/server/routes/test_event_routes.py index 547f2cad1..bbffcfbc9 100644 --- a/tests/server/routes/test_event_routes.py +++ b/tests/server/routes/test_event_routes.py @@ -1,6 +1,13 @@ +import json +from unittest.mock import AsyncMock + import pytest +from starlette.requests import Request +from flocks.auth.context import AuthUser +from flocks.server.routes import event as event_routes from flocks.server.routes.event import EventBroadcaster +from flocks.session.session import SessionInfo @pytest.mark.asyncio @@ -42,3 +49,287 @@ async def test_event_broadcaster_shutdown_does_not_block_on_full_queue(): assert broadcaster.client_count == 0 assert any(event["type"] == "server.shutting_down" for event in events) + + +@pytest.mark.asyncio +async def test_event_broadcaster_keeps_latest_event_when_drop_target_is_max_minus_one(): + broadcaster = EventBroadcaster(queue_maxsize=3, queue_drop_to=2) + queue = await broadcaster.subscribe() + + for event_type in ("event.one", "event.two", "event.three", "event.latest"): + await broadcaster.publish({"type": event_type, "properties": {}}) + + events = [] + while not queue.empty(): + events.append(queue.get_nowait()) + + assert events[-1]["type"] == "event.latest" + assert any(event["type"] == "server.events_dropped" for event in events) + + +@pytest.mark.asyncio +async def test_event_broadcaster_coalesces_accumulated_text_and_preserves_delta(): + broadcaster = EventBroadcaster(queue_maxsize=3) + queue = await broadcaster.subscribe() + first = { + "type": "message.part.updated", + "properties": { + "part": { + "id": "part-1", + "messageID": "message-1", + "sessionID": "session-1", + "type": "text", + "text": "hello", + }, + "delta": "hello", + }, + } + latest = { + "type": "message.part.updated", + "properties": { + "part": { + "id": "part-1", + "messageID": "message-1", + "sessionID": "session-1", + "type": "text", + "text": "hello world", + }, + "delta": " world", + }, + } + + broadcaster._publish_to_queue(queue, first) + broadcaster._publish_to_queue(queue, latest) + + assert queue.qsize() == 1 + event = queue.get_nowait() + assert event["properties"]["part"]["text"] == "hello world" + assert event["properties"]["delta"] == "hello world" + + +@pytest.mark.asyncio +async def test_event_broadcaster_coalesces_snapshots_across_interleaved_events(): + broadcaster = EventBroadcaster(queue_maxsize=4) + queue = await broadcaster.subscribe() + first = { + "type": "message.part.updated", + "properties": { + "part": { + "id": "part-1", + "messageID": "message-1", + "sessionID": "session-1", + "type": "text", + "text": "hello", + }, + "delta": "hello", + }, + } + control = {"type": "question.asked", "properties": {"requestID": "question-1"}} + latest = { + "type": "message.part.updated", + "properties": { + "part": { + "id": "part-1", + "messageID": "message-1", + "sessionID": "session-1", + "type": "text", + "text": "hello world", + }, + "delta": " world", + }, + } + + broadcaster._publish_to_queue(queue, first) + broadcaster._publish_to_queue(queue, control) + broadcaster._publish_to_queue(queue, latest) + + assert queue.qsize() == 2 + assert queue.get_nowait() == control + snapshot = queue.get_nowait() + assert snapshot["properties"]["part"]["text"] == "hello world" + assert snapshot["properties"]["delta"] == "hello world" + + +@pytest.mark.asyncio +async def test_accumulated_text_queue_memory_tracks_latest_snapshot_not_full_history(): + broadcaster = EventBroadcaster(queue_maxsize=1000) + queue = await broadcaster.subscribe() + text = "" + + for _ in range(1000): + delta = "x" * 100 + text += delta + broadcaster._publish_to_queue(queue, { + "type": "message.part.updated", + "properties": { + "part": { + "id": "part-1", + "messageID": "message-1", + "sessionID": "session-1", + "type": "text", + "text": text, + }, + "delta": delta, + }, + }) + + assert queue.qsize() == 1 + payload = queue.get_nowait() + assert payload["properties"]["delta"] == text + assert len(json.dumps(payload)) < 250_000 + + +@pytest.mark.asyncio +async def test_text_snapshot_does_not_evict_control_events(): + broadcaster = EventBroadcaster(queue_maxsize=3, queue_drop_to=1) + queue = await broadcaster.subscribe() + controls = [ + {"type": "permission.request", "properties": {"requestID": "permission-1"}}, + {"type": "question.asked", "properties": {"id": "question-1"}}, + ] + for event in controls: + broadcaster._publish_to_queue(queue, event) + + def _snapshot(part_id: str) -> dict: + return { + "type": "message.part.updated", + "properties": { + "part": { + "id": part_id, + "messageID": "message-1", + "sessionID": "session-1", + "type": "text", + "text": part_id, + }, + "delta": part_id, + }, + } + + broadcaster._publish_to_queue(queue, _snapshot("part-old")) + broadcaster._publish_to_queue(queue, _snapshot("part-latest")) + + events = [] + while not queue.empty(): + events.append(queue.get_nowait()) + assert [event["type"] for event in events[:2]] == [ + "permission.request", + "question.asked", + ] + assert events[-1]["type"] == "server.events_dropped" + + +@pytest.mark.asyncio +async def test_snapshot_dropped_behind_full_control_queue_emits_deferred_marker(): + broadcaster = EventBroadcaster(queue_maxsize=3, queue_drop_to=1) + queue = await broadcaster.subscribe() + controls = [ + {"type": f"control.{index}", "properties": {"index": index}} + for index in range(3) + ] + for event in controls: + broadcaster._publish_to_queue(queue, event) + + broadcaster._publish_to_queue(queue, { + "type": "message.part.updated", + "properties": { + "part": { + "id": "part-dropped", + "messageID": "message-1", + "sessionID": "session-1", + "type": "text", + "text": "dropped", + }, + "delta": "dropped", + }, + }) + + assert queue.qsize() == 3 + assert await queue.get() == controls[0] + assert await queue.get() == controls[1] + assert await queue.get() == controls[2] + marker = await queue.get() + assert marker["type"] == "server.events_dropped" + assert marker["properties"]["dropped"] == 1 + + +@pytest.mark.asyncio +async def test_session_events_are_filtered_by_subscriber_user(monkeypatch: pytest.MonkeyPatch): + broadcaster = EventBroadcaster() + alice = AuthUser(id="user-alice", username="alice", role="member") + bob = AuthUser(id="user-bob", username="bob", role="member") + session = SessionInfo( + id="session-private", + projectID="project-1", + directory="/tmp/project-1", + ownerUserID=alice.id, + ownerUsername=alice.username, + ) + + async def _get_event_session(session_id: str): + assert session_id == session.id + return session + + monkeypatch.setattr(event_routes, "_get_event_session", _get_event_session) + alice_queue = await broadcaster.subscribe(alice) + bob_queue = await broadcaster.subscribe(bob) + + private_event = { + "type": "message.part.updated", + "properties": { + "part": { + "id": "part-1", + "messageID": "message-1", + "sessionID": session.id, + "type": "text", + "text": "private content", + } + }, + } + await broadcaster.publish(private_event) + + assert alice_queue.get_nowait() == private_event + assert bob_queue.empty() + + global_event = {"type": "workflow.updated", "properties": {"id": "workflow-1"}} + await broadcaster.publish(global_event) + assert alice_queue.get_nowait() == global_event + assert bob_queue.get_nowait() == global_event + + +@pytest.mark.asyncio +async def test_deleted_session_event_can_still_resolve_its_owner(monkeypatch: pytest.MonkeyPatch): + from flocks.session.session import Session + from flocks.storage.storage import Storage + + deleted = SessionInfo( + id="session-deleted", + projectID="project-1", + directory="/tmp/project-1", + status="deleted", + ownerUserID="user-owner", + ownerUsername="owner", + ) + monkeypatch.setattr(Session, "_all_sessions_cache", None) + monkeypatch.setattr(Session, "get_by_id", AsyncMock(return_value=None)) + monkeypatch.setattr(Storage, "list_keys", AsyncMock(return_value=["session:project-1:session-deleted"])) + monkeypatch.setattr(Storage, "get", AsyncMock(return_value=deleted)) + + assert await event_routes._get_event_session(deleted.id) == deleted + + +@pytest.mark.asyncio +async def test_global_event_subscription_keeps_authenticated_user(monkeypatch: pytest.MonkeyPatch): + from flocks.server.routes import global_ as global_routes + + user = AuthUser(id="user-alice", username="alice", role="member") + request = Request({"type": "http", "method": "GET", "path": "/global/event", "headers": []}) + request.state.auth_user = user + broadcaster = EventBroadcaster() + subscribe = AsyncMock(wraps=broadcaster.subscribe) + monkeypatch.setattr(broadcaster, "subscribe", subscribe) + monkeypatch.setattr(global_routes.EventBroadcaster, "get", classmethod(lambda cls: broadcaster)) + + response = await global_routes.get_global_events(request) + + subscribe.assert_awaited_once_with(user) + await response.body_iterator.aclose() diff --git a/tests/server/routes/test_global_mutation_auth.py b/tests/server/routes/test_global_mutation_auth.py new file mode 100644 index 000000000..2c2bd8d69 --- /dev/null +++ b/tests/server/routes/test_global_mutation_auth.py @@ -0,0 +1,162 @@ +"""Authorization boundaries for global provider and Hub mutations.""" + +from __future__ import annotations + +from unittest.mock import AsyncMock, MagicMock + +from fastapi import FastAPI +from fastapi.testclient import TestClient +import pytest + +from flocks.auth.context import AuthUser + + +def _user_client(router, *, prefix: str, role: str) -> TestClient: + app = FastAPI() + + @app.middleware("http") + async def user_auth(request, call_next): + request.state.auth_user = AuthUser( + id=f"{role}-test", + username=f"{role}-test", + role=role, + status="active", + must_reset_password=False, + ) + return await call_next(request) + + app.include_router(router, prefix=prefix) + return TestClient(app, raise_server_exceptions=True) + + +def test_provider_global_configuration_credentials_and_tests_require_admin(): + from flocks.server.routes.provider import router + + client = _user_client(router, prefix="/api/provider", role="member") + responses = [ + client.post("/api/provider/example/oauth/authorize?method=0"), + client.post("/api/provider/example/oauth/callback?method=0"), + client.post("/api/provider/example/configure", json={}), + client.post("/api/provider/example/test"), + client.put("/api/provider/example", json={}), + client.get("/api/provider/example/credentials"), + client.post("/api/provider/example/credentials", json={"api_key": "secret"}), + client.delete("/api/provider/example/credentials"), + client.get("/api/provider/example/service-credentials"), + client.post("/api/provider/example/service-credentials", json={"api_key": "secret"}), + client.post("/api/provider/example/test-credentials", json={}), + client.patch("/api/provider/api-services/example", json={"enabled": False}), + client.delete("/api/provider/api-services/example"), + client.post("/api/provider/api-services/refresh"), + ] + + assert [response.status_code for response in responses] == [403] * len(responses) + + +def test_admin_credential_routes_return_only_masked_secrets(monkeypatch: pytest.MonkeyPatch): + from flocks.server.routes import provider as provider_routes + + raw_llm_key = "sk-raw-llm-secret-1234" + raw_service_key = "raw-service-secret-5678" + secrets = MagicMock() + secrets.get.side_effect = lambda key: { + "llm-example_llm_key": raw_llm_key, + "service-example_api_key": raw_service_key, + }.get(key) + monkeypatch.setattr("flocks.security.get_secret_manager", lambda: secrets) + monkeypatch.setattr( + provider_routes.ConfigWriter, + "get_provider_raw", + lambda _provider_id: None, + ) + monkeypatch.setattr( + provider_routes.ConfigWriter, + "get_api_service_raw", + lambda _provider_id: { + "apiKey": "{secret:service-example_api_key}", + "base_url": "https://service.example.test", + }, + ) + monkeypatch.setattr( + provider_routes, + "_load_api_service_metadata_data", + lambda _provider_id: { + "credential_fields": [ + { + "key": "api_key", + "storage": "secret", + "sensitive": True, + "config_key": "apiKey", + "secret_id": "service-example_api_key", + }, + { + "key": "base_url", + "storage": "config", + "config_key": "base_url", + }, + ], + }, + ) + + client = _user_client(provider_routes.router, prefix="/api/provider", role="admin") + llm_response = client.get("/api/provider/llm-example/credentials") + service_response = client.get("/api/provider/service-example/service-credentials") + + assert llm_response.status_code == 200 + assert service_response.status_code == 200 + llm_payload = llm_response.json() + service_payload = service_response.json() + assert llm_payload["api_key"] is None + assert llm_payload["api_key_masked"] + assert service_payload["api_key"] is None + assert service_payload["api_key_masked"] + assert service_payload["fields"]["api_key"] != raw_service_key + assert service_payload["fields"]["base_url"] == "https://service.example.test" + assert raw_llm_key not in llm_response.text + assert raw_service_key not in service_response.text + + +def test_admin_provider_update_never_writes_raw_api_key_to_storage(monkeypatch: pytest.MonkeyPatch): + from flocks.server.routes import provider as provider_routes + + runtime_provider = MagicMock() + storage_write = AsyncMock() + monkeypatch.setattr(provider_routes.Provider, "_ensure_initialized", MagicMock()) + monkeypatch.setattr(provider_routes.Provider, "get", lambda _provider_id: runtime_provider) + monkeypatch.setattr(provider_routes.Storage, "write", storage_write) + monkeypatch.setattr( + provider_routes, + "get_provider", + AsyncMock(return_value=provider_routes.ProviderInfo(id="example", name="Example")), + ) + + client = _user_client(provider_routes.router, prefix="/api/provider", role="admin") + response = client.put( + "/api/provider/example", + json={ + "api_key": "raw-update-secret", + "base_url": "https://provider.example.test", + "custom_settings": {"timeout": 10}, + }, + ) + + assert response.status_code == 200 + storage_write.assert_awaited_once() + stored_payload = storage_write.await_args.args[1] + assert "api_key" not in stored_payload + assert "raw-update-secret" not in repr(stored_payload) + + +def test_hub_global_mutations_require_admin(): + from flocks.server.routes.hub import router + + client = _user_client(router, prefix="/api", role="member") + responses = [ + client.post("/api/hub/plugins/skill/example/install", json={"scope": "global"}), + client.post("/api/hub/plugins/component/example/install/stream", json={"scope": "global"}), + client.post("/api/hub/plugins/skill/example/update", json={"scope": "global"}), + client.delete("/api/hub/plugins/skill/example"), + client.post("/api/hub/refresh"), + ] + + assert [response.status_code for response in responses] == [403, 403, 403, 403, 403] diff --git a/tests/server/routes/test_interaction_access_routes.py b/tests/server/routes/test_interaction_access_routes.py new file mode 100644 index 000000000..00642bcf3 --- /dev/null +++ b/tests/server/routes/test_interaction_access_routes.py @@ -0,0 +1,228 @@ +"""Session ownership checks for question and permission interactions.""" + +from __future__ import annotations + +from unittest.mock import AsyncMock + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from flocks.auth.context import AuthUser +from flocks.server.auth import require_user +from flocks.session.session import SessionInfo + + +def _user(user_id: str, username: str, *, role: str = "member") -> AuthUser: + return AuthUser( + id=user_id, + username=username, + role=role, + status="active", + must_reset_password=False, + ) + + +def _client(router, *, prefix: str, user: AuthUser) -> TestClient: + app = FastAPI() + app.dependency_overrides[require_user] = lambda: user + app.include_router(router, prefix=prefix) + return TestClient(app, raise_server_exceptions=True) + + +@pytest.fixture() +def owned_session() -> SessionInfo: + return SessionInfo( + id="ses_owner", + project_id="project", + directory="/tmp", + title="Owned session", + owner_user_id="owner-id", + owner_username="owner", + ) + + +def test_question_routes_reject_non_owner(monkeypatch: pytest.MonkeyPatch, owned_session: SessionInfo): + from flocks.server.routes import question as question_routes + + request_id = "question-owner-only" + question_routes.store_question_request(request_id, { + "id": request_id, + "sessionID": owned_session.id, + "questions": [{"question": "Continue?"}], + }) + monkeypatch.setattr(question_routes.Session, "get_by_id", AsyncMock(return_value=owned_session)) + client = _client(question_routes.router, prefix="/question", user=_user("other-id", "other")) + + try: + assert client.get(f"/question/session/{owned_session.id}/pending").status_code == 403 + assert client.post( + f"/question/{request_id}/reply", + json={"answers": [["yes"]]}, + ).status_code == 403 + assert client.post(f"/question/{request_id}/reject").status_code == 403 + assert question_routes.get_question_request(request_id) is not None + finally: + question_routes.clear_request_state(request_id) + + +def test_question_owner_can_list_and_reply(monkeypatch: pytest.MonkeyPatch, owned_session: SessionInfo): + from flocks.server.routes import question as question_routes + + request_id = "question-owner-reply" + question_routes.store_question_request(request_id, { + "id": request_id, + "sessionID": owned_session.id, + "questions": [{"question": "Continue?"}], + }) + monkeypatch.setattr(question_routes.Session, "get_by_id", AsyncMock(return_value=owned_session)) + monkeypatch.setattr("flocks.server.routes.event.publish_event", AsyncMock()) + client = _client(question_routes.router, prefix="/question", user=_user("owner-id", "owner")) + + try: + pending = client.get(f"/question/session/{owned_session.id}/pending") + assert pending.status_code == 200 + assert [item["id"] for item in pending.json()] == [request_id] + reply = client.post( + f"/question/{request_id}/reply", + json={"answers": [["yes"]]}, + ) + assert reply.status_code == 200 + assert question_routes.get_request_answer(request_id) == [["yes"]] + finally: + question_routes.clear_request_state(request_id) + + +def test_permission_routes_filter_and_reject_non_owner( + monkeypatch: pytest.MonkeyPatch, + owned_session: SessionInfo, +): + from flocks.permission.next import PermissionRequestInfo + from flocks.server.routes import permission as permission_routes + + info = PermissionRequestInfo( + id="perm-owner-only", + sessionID=owned_session.id, + permission="bash", + patterns=["*"], + metadata={"messageID": "msg-1"}, + always=[], + tool={"name": "bash"}, + time={"created": 1}, + ) + monkeypatch.setattr(permission_routes.Session, "get_by_id", AsyncMock(return_value=owned_session)) + monkeypatch.setattr(permission_routes.PermissionNext, "list_pending_infos", AsyncMock(return_value=[info])) + monkeypatch.setattr(permission_routes.PermissionNext, "get_pending_info", AsyncMock(return_value=info)) + reply_mock = AsyncMock() + monkeypatch.setattr(permission_routes.PermissionNext, "reply", reply_mock) + client = _client(permission_routes.router, prefix="/permission", user=_user("other-id", "other")) + + assert client.get("/permission").json() == [] + assert client.get(f"/permission/{info.id}").status_code == 403 + assert client.post( + f"/permission/{info.id}/reply", + json={"allow": True, "always": False}, + ).status_code == 403 + reply_mock.assert_not_awaited() + + +def test_permission_owner_can_reply(monkeypatch: pytest.MonkeyPatch, owned_session: SessionInfo): + from flocks.permission.next import PermissionRequestInfo + from flocks.server.routes import permission as permission_routes + + info = PermissionRequestInfo( + id="perm-owner-reply", + sessionID=owned_session.id, + permission="bash", + patterns=["*"], + metadata={"messageID": "msg-1"}, + always=[], + tool={"name": "bash"}, + time={"created": 1}, + ) + monkeypatch.setattr(permission_routes.Session, "get_by_id", AsyncMock(return_value=owned_session)) + monkeypatch.setattr(permission_routes.PermissionNext, "get_pending_info", AsyncMock(return_value=info)) + reply_mock = AsyncMock() + monkeypatch.setattr(permission_routes.PermissionNext, "reply", reply_mock) + client = _client(permission_routes.router, prefix="/permission", user=_user("owner-id", "owner")) + + response = client.post( + f"/permission/{info.id}/reply", + json={"allow": True, "always": False}, + ) + + assert response.status_code == 200 + reply_mock.assert_awaited_once_with(info.id, "allow", session_id=owned_session.id) + + +@pytest.mark.parametrize( + ("allow", "expected_reply"), + [(True, "allow_session"), (False, "deny_session")], +) +def test_permission_member_remember_is_session_scoped( + monkeypatch: pytest.MonkeyPatch, + owned_session: SessionInfo, + allow: bool, + expected_reply: str, +): + from flocks.permission.next import PermissionRequestInfo + from flocks.server.routes import permission as permission_routes + + info = PermissionRequestInfo( + id=f"perm-member-remember-{allow}", + sessionID=owned_session.id, + permission="bash", + patterns=["*"], + ) + monkeypatch.setattr(permission_routes.Session, "get_by_id", AsyncMock(return_value=owned_session)) + monkeypatch.setattr(permission_routes.PermissionNext, "get_pending_info", AsyncMock(return_value=info)) + reply_mock = AsyncMock() + monkeypatch.setattr(permission_routes.PermissionNext, "reply", reply_mock) + client = _client(permission_routes.router, prefix="/permission", user=_user("owner-id", "owner")) + + response = client.post( + f"/permission/{info.id}/reply", + json={"allow": allow, "always": True}, + ) + + assert response.status_code == 200 + reply_mock.assert_awaited_once_with(info.id, expected_reply, session_id=owned_session.id) + + +@pytest.mark.parametrize( + ("allow", "expected_reply"), + [(True, "always"), (False, "never")], +) +def test_permission_admin_remember_can_be_global( + monkeypatch: pytest.MonkeyPatch, + owned_session: SessionInfo, + allow: bool, + expected_reply: str, +): + from flocks.permission.next import PermissionRequestInfo + from flocks.server.routes import permission as permission_routes + + admin = _user("admin-id", "admin", role="admin") + admin_session = owned_session.model_copy(update={ + "owner_user_id": admin.id, + "owner_username": admin.username, + }) + info = PermissionRequestInfo( + id=f"perm-admin-remember-{allow}", + sessionID=admin_session.id, + permission="bash", + patterns=["*"], + ) + monkeypatch.setattr(permission_routes.Session, "get_by_id", AsyncMock(return_value=admin_session)) + monkeypatch.setattr(permission_routes.PermissionNext, "get_pending_info", AsyncMock(return_value=info)) + reply_mock = AsyncMock() + monkeypatch.setattr(permission_routes.PermissionNext, "reply", reply_mock) + client = _client(permission_routes.router, prefix="/permission", user=admin) + + response = client.post( + f"/permission/{info.id}/reply", + json={"allow": allow, "always": True}, + ) + + assert response.status_code == 200 + reply_mock.assert_awaited_once_with(info.id, expected_reply, session_id=admin_session.id) diff --git a/tests/server/routes/test_provider_optional_api_key.py b/tests/server/routes/test_provider_optional_api_key.py index ef4309606..5aa342202 100644 --- a/tests/server/routes/test_provider_optional_api_key.py +++ b/tests/server/routes/test_provider_optional_api_key.py @@ -25,6 +25,7 @@ def patched_runtime(monkeypatch: pytest.MonkeyPatch): so credentials calls remain hermetic and never write to disk. """ fake_secrets = MagicMock() + fake_secrets.get.return_value = None runtime_provider = MagicMock() runtime_provider._client = None @@ -125,6 +126,63 @@ async def test_strict_provider_still_rejects_empty_api_key(self, patched_runtime assert excinfo.value.detail == "API key required" patched_runtime["secrets"].set.assert_not_called() + @pytest.mark.asyncio + async def test_strict_provider_blank_key_preserves_existing_stored_secret( + self, patched_runtime + ): + """Editing non-secret settings must not require or rewrite the saved key.""" + patched_runtime["secrets"].get.side_effect = lambda key: { + "openai_llm_key": "sk-existing-secret", + }.get(key) + + result = await provider_routes.set_provider_credentials( + "openai", + provider_routes.ProviderCredentialRequest( + base_url="https://new.example.com/v1", + ), + ) + + assert result["success"] is True + patched_runtime["secrets"].set.assert_not_called() + provider_routes.ConfigWriter.update_provider_field.assert_called_once_with( + "openai", "options.baseURL", "https://new.example.com/v1" + ) + configured = patched_runtime["provider"].configure.call_args.args[0] + assert configured.api_key == "sk-existing-secret" + assert configured.base_url == "https://new.example.com/v1" + + @pytest.mark.asyncio + async def test_strict_provider_blank_key_preserves_existing_inline_secret( + self, patched_runtime, monkeypatch: pytest.MonkeyPatch + ): + """Legacy inline keys are reused without copying them into the secret store.""" + monkeypatch.setattr( + provider_routes.ConfigWriter, + "get_provider_raw", + lambda _provider_id: { + "options": { + "apiKey": "inline-existing-secret", + "baseURL": "https://old.example.com/v1", + } + }, + ) + + result = await provider_routes.set_provider_credentials( + "openai", + provider_routes.ProviderCredentialRequest( + api_key="", + base_url="https://new.example.com/v1", + ), + ) + + assert result["success"] is True + patched_runtime["secrets"].set.assert_not_called() + provider_routes.ConfigWriter.update_provider_field.assert_called_once_with( + "openai", "options.baseURL", "https://new.example.com/v1" + ) + configured = patched_runtime["provider"].configure.call_args.args[0] + assert configured.api_key == "inline-existing-secret" + @pytest.mark.asyncio async def test_sentinel_logged_with_explicit_marker_not_naive_mask( self, patched_runtime, capfd @@ -334,7 +392,7 @@ async def test_placeholder_is_hidden_but_has_credential_true( assert response.base_url == "http://internal/v1" @pytest.mark.asyncio - async def test_real_api_key_still_returned_and_masked( + async def test_real_api_key_is_only_returned_masked( self, monkeypatch: pytest.MonkeyPatch ): fake_secrets = MagicMock() @@ -353,7 +411,7 @@ async def test_real_api_key_still_returned_and_masked( response = await provider_routes.get_provider_credentials("openai") - assert response.api_key == "sk-real-secret-1234567890" + assert response.api_key is None assert response.api_key_masked is not None assert response.api_key_masked != "sk-real-secret-1234567890" assert response.has_credential is True diff --git a/tests/server/routes/test_remaining_routes.py b/tests/server/routes/test_remaining_routes.py index feb8bf695..ce681b707 100644 --- a/tests/server/routes/test_remaining_routes.py +++ b/tests/server/routes/test_remaining_routes.py @@ -673,6 +673,7 @@ async def test_reply_missing_allow_field_returns_422(self, client: AsyncClient): async def test_permission_routes_preserve_request_created_time(self, client: AsyncClient, monkeypatch: pytest.MonkeyPatch): """List/detail routes should expose the stored permission request timestamp.""" from flocks.permission.next import PermissionRequestInfo + from flocks.session.session import SessionInfo info = PermissionRequestInfo( id="perm_time_test", @@ -693,6 +694,17 @@ async def test_permission_routes_preserve_request_created_time(self, client: Asy "flocks.server.routes.permission.PermissionNext.get_pending_info", AsyncMock(return_value=info), ) + monkeypatch.setattr( + "flocks.server.routes.permission.Session.get_by_id", + AsyncMock(return_value=SessionInfo( + id="ses_time_test", + project_id="test", + directory="/tmp", + title="Permission route test", + owner_user_id="api-token-service", + owner_username="api-token-service", + )), + ) list_resp = await client.get("/permission") assert list_resp.status_code == status.HTTP_200_OK diff --git a/tests/server/routes/test_tool_routes.py b/tests/server/routes/test_tool_routes.py index 3c0786495..faa087fa8 100644 --- a/tests/server/routes/test_tool_routes.py +++ b/tests/server/routes/test_tool_routes.py @@ -451,6 +451,7 @@ async def handler(ctx, text: str) -> ToolResult: assert payload["limit"] == 20 assert payload["facets"]["source"]["plugin_py"] == 1 assert payload["facets"]["source"]["api"] == 1 + assert payload["facets"]["source_groups"]["api"] == 1 assert payload["items"][0]["name"] == "page_unique_alpha_tool" assert payload["items"][0]["parameters"] == [] assert payload["items"][0]["parameters_count"] == 1 diff --git a/tests/server/routes/test_update_routes.py b/tests/server/routes/test_update_routes.py index 10ade912e..e4334c4c5 100644 --- a/tests/server/routes/test_update_routes.py +++ b/tests/server/routes/test_update_routes.py @@ -225,6 +225,47 @@ async def _fake_check_update(**kwargs): await asyncio.wait_for(second, timeout=1) +async def test_forced_check_cannot_be_overwritten_by_older_inflight_request( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.server.routes import update as update_routes + from flocks.updater.models import VersionInfo + + older_started = asyncio.Event() + release_older = asyncio.Event() + calls = 0 + + async def _fake_check_update(**kwargs): + nonlocal calls + calls += 1 + if calls == 1: + older_started.set() + await release_older.wait() + return VersionInfo(current_version="v1") + return VersionInfo(current_version="v2") + + monkeypatch.setattr(update_routes, "check_update", _fake_check_update) + + older = asyncio.create_task( + update_routes.check_version(_request(), locale="zh-CN", edition="flocks") + ) + await older_started.wait() + + forced = await update_routes.check_version( + _request(), locale="zh-CN", edition="flocks", force=True + ) + assert forced.current_version == "v2" + + release_older.set() + assert (await older).current_version == "v1" + + cached = await update_routes.check_version( + _request(), locale="zh-CN", edition="flocks" + ) + assert cached.current_version == "v2" + assert calls == 2 + + @pytest.mark.skipif( os.environ.get(_MANUAL_REAL_UPGRADE_ENV) != "1", reason=f"manual real upgrade test; set {_MANUAL_REAL_UPGRADE_ENV}=1 to enable", diff --git a/tests/server/routes/test_webui_perf_routes.py b/tests/server/routes/test_webui_perf_routes.py index 32b700a2f..51d33f3a2 100644 --- a/tests/server/routes/test_webui_perf_routes.py +++ b/tests/server/routes/test_webui_perf_routes.py @@ -1,11 +1,18 @@ from __future__ import annotations +import asyncio +import threading +import time from collections.abc import AsyncGenerator +from types import SimpleNamespace import pytest from fastapi import FastAPI, status from httpx import ASGITransport, AsyncClient +from flocks.auth.context import AuthUser +from flocks.server.auth import require_user + pytestmark = pytest.mark.asyncio @@ -17,6 +24,13 @@ async def webui_perf_client() -> AsyncGenerator[AsyncClient, None]: from flocks.server.routes.stats import router as stats_router app = FastAPI() + app.dependency_overrides[require_user] = lambda: AuthUser( + id="webui-perf-user", + username="webui-perf-user", + role="member", + status="active", + must_reset_password=False, + ) app.include_router(stats_router, prefix="/api/stats") app.include_router(monitoring_router, prefix="/api/monitoring") app.include_router(permission_router, prefix="/api/permission") @@ -73,6 +87,115 @@ async def _count_models(): } +async def test_stats_summary_starts_independent_sources_concurrently( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.server.routes import stats as stats_routes + + started: set[str] = set() + all_started = asyncio.Event() + release = asyncio.Event() + + async def _count(name: str, value): + started.add(name) + if len(started) == 6: + all_started.set() + await release.wait() + return value + + async def _task_dashboard(): + return await _count("dashboard", {}) + + async def _count_tools(): + return await _count("tools", 8) + + async def _count_skills(): + return await _count("skills", 7) + + async def _count_agents(): + return await _count("agents", 5) + + async def _count_workflows(): + return await _count("workflows", 6) + + async def _count_models(): + return await _count("models", 9) + + monkeypatch.setattr(stats_routes, "_task_dashboard", _task_dashboard) + monkeypatch.setattr(stats_routes, "_count_tools", _count_tools) + monkeypatch.setattr(stats_routes, "_count_skills", _count_skills) + monkeypatch.setattr(stats_routes, "_count_agents", _count_agents) + monkeypatch.setattr(stats_routes, "_count_workflows", _count_workflows) + monkeypatch.setattr(stats_routes, "_count_models", _count_models) + + summary_task = asyncio.create_task(stats_routes.get_system_stats_summary()) + await asyncio.wait_for(all_started.wait(), timeout=1) + release.set() + response = await summary_task + + assert started == {"dashboard", "agents", "workflows", "skills", "tools", "models"} + assert response.tools.total == 8 + assert response.skills.total == 7 + assert response.agents.total == 5 + + +async def test_skill_discovery_does_not_block_the_event_loop( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.server.routes import stats as stats_routes + from flocks.skill.skill import Skill + + def slow_discover(): + time.sleep(0.06) + return {"demo": SimpleNamespace(name="demo", category="user")} + + monkeypatch.setattr(Skill, "_discover", slow_discover) + Skill.clear_cache() + count_task = asyncio.create_task(stats_routes._count_skills()) + heartbeat_ticks = 0 + while not count_task.done(): + await asyncio.sleep(0.005) + heartbeat_ticks += 1 + + assert await count_task == 1 + assert heartbeat_ticks >= 3 + Skill.clear_cache() + + +async def test_skill_invalidation_during_discovery_cannot_restore_stale_cache( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.skill.skill import Skill + + discovery_started = threading.Event() + release_first_discovery = threading.Event() + discovery_count = 0 + + def discover(): + nonlocal discovery_count + discovery_count += 1 + if discovery_count == 1: + discovery_started.set() + assert release_first_discovery.wait(timeout=1) + name = "stale" + else: + name = "fresh" + return {name: SimpleNamespace(name=name, category="user")} + + monkeypatch.setattr(Skill, "_discover", discover) + Skill.clear_cache() + load_task = asyncio.create_task(Skill.all()) + assert await asyncio.to_thread(discovery_started.wait, 1) + Skill.clear_cache() + release_first_discovery.set() + + skills = await load_task + + assert [skill.name for skill in skills] == ["fresh"] + assert list((Skill._cache or {}).keys()) == ["fresh"] + Skill.clear_cache() + + async def test_monitoring_routes_back_existing_webui_calls(webui_perf_client: AsyncClient): status_response = await webui_perf_client.get("/api/monitoring/status") metrics_response = await webui_perf_client.get("/api/monitoring/metrics") @@ -90,12 +213,28 @@ async def test_monitoring_routes_back_existing_webui_calls(webui_perf_client: As assert "mcpServers" in status_payload metrics_payload = metrics_response.json() - assert "messageRate" in metrics_payload - assert "toolCallRate" in metrics_payload + assert metrics_payload["messageRate"] is None + assert metrics_payload["avgResponseTime"] is None + assert metrics_payload["activeRequests"] is None assert llm_response.json() == [] assert tool_response.json() == [] +async def test_monitoring_tool_rates_use_counter_deltas(monkeypatch: pytest.MonkeyPatch): + from flocks.server.routes import monitoring as monitoring_routes + + samples = iter([100.0, 101.0, 160.0, 161.0]) + monkeypatch.setattr(monitoring_routes, "_metrics_clock", lambda: next(samples)) + monkeypatch.setattr(monitoring_routes, "_metrics_rate_state", None) + + assert monitoring_routes._sample_tool_rates(10, 2) == (None, None) + assert monitoring_routes._sample_tool_rates(11, 2) == (None, None) + assert monitoring_routes._sample_tool_rates(15, 3) == (5.0, 1.0) + # Additional clients inside the same window see the same sample instead + # of advancing the global baseline and distorting each other's rates. + assert monitoring_routes._sample_tool_rates(16, 3) == (5.0, 1.0) + + async def test_api_permission_alias_matches_frontend_route(webui_perf_client: AsyncClient): response = await webui_perf_client.get("/api/permission") diff --git a/webui/src/api/monitoring.ts b/webui/src/api/monitoring.ts index e74a2c22d..ccf1d7682 100644 --- a/webui/src/api/monitoring.ts +++ b/webui/src/api/monitoring.ts @@ -3,19 +3,19 @@ import client from './client'; export interface SystemStatus { status: 'healthy' | 'degraded' | 'down'; uptime: number; - activeSessions: number; - activeAgents: number; + activeSessions: number | null; + activeAgents: number | null; mcpServers: Record; timestamp: number; } export interface MetricsSnapshot { timestamp: number; - messageRate: number; - toolCallRate: number; - errorRate: number; - avgResponseTime: number; - activeRequests: number; + messageRate: number | null; + toolCallRate: number | null; + errorRate: number | null; + avgResponseTime: number | null; + activeRequests: number | null; } export interface PerformanceData { diff --git a/webui/src/api/provider.test.ts b/webui/src/api/provider.test.ts index 45a9c8a9e..1e2cda5ce 100644 --- a/webui/src/api/provider.test.ts +++ b/webui/src/api/provider.test.ts @@ -87,4 +87,52 @@ describe('providerAPI.listApiServices', () => { expect(mockGet).toHaveBeenCalledTimes(2); expect(response.data[0].name).toBe('Service A Updated'); }); + + it('does not let an older list request repopulate the cache after a mutation', async () => { + let resolveOldList: (value: { data: ReturnType[] }) => void = () => undefined; + mockGet + .mockImplementationOnce(() => new Promise((resolve) => { + resolveOldList = resolve; + })) + .mockResolvedValueOnce({ data: [serviceSummary('svc-a', 'Fresh Service')] }); + mockPatch.mockResolvedValue({ data: serviceSummary('svc-a', 'Fresh Service') }); + + const { providerAPI } = await import('./provider'); + const oldRequest = providerAPI.listApiServices(); + await providerAPI.updateApiService('svc-a', { enabled: false }); + const freshRequest = providerAPI.listApiServices(); + resolveOldList({ data: [serviceSummary('svc-a', 'Stale Service')] }); + await Promise.all([oldRequest, freshRequest]); + + const response = await providerAPI.listApiServices(); + + expect(mockGet).toHaveBeenCalledTimes(2); + expect(response.data[0].name).toBe('Fresh Service'); + }); + + it('keeps a forced request authoritative when an older request finishes later', async () => { + let resolveOldList: (value: { data: ReturnType[] }) => void = () => undefined; + let resolveForcedList: (value: { data: ReturnType[] }) => void = () => undefined; + mockGet + .mockImplementationOnce(() => new Promise((resolve) => { + resolveOldList = resolve; + })) + .mockImplementationOnce(() => new Promise((resolve) => { + resolveForcedList = resolve; + })); + + const { providerAPI } = await import('./provider'); + const oldRequest = providerAPI.listApiServices(); + const forcedRequest = providerAPI.listApiServices({ force: true }); + + resolveForcedList({ data: [serviceSummary('svc-a', 'Forced Service')] }); + await forcedRequest; + resolveOldList({ data: [serviceSummary('svc-a', 'Stale Service')] }); + await oldRequest; + + const response = await providerAPI.listApiServices(); + + expect(mockGet).toHaveBeenCalledTimes(2); + expect(response.data[0].name).toBe('Forced Service'); + }); }); diff --git a/webui/src/api/provider.ts b/webui/src/api/provider.ts index c4fd8cec1..a35235f42 100644 --- a/webui/src/api/provider.ts +++ b/webui/src/api/provider.ts @@ -25,7 +25,9 @@ function requestApiServicesList() { type APIServiceListResponse = Awaited>; let apiServicesListInFlight: Promise | null = null; +let apiServicesListInFlightGeneration: number | null = null; let apiServicesListCache: { response: APIServiceListResponse; updatedAt: number } | null = null; +let apiServicesListGeneration = 0; function cloneApiServicesResponse(response: APIServiceListResponse): APIServiceListResponse { const services = Array.isArray(response.data) ? response.data : []; @@ -36,6 +38,7 @@ function cloneApiServicesResponse(response: APIServiceListResponse): APIServiceL } function invalidateApiServicesListCache(): void { + apiServicesListGeneration += 1; apiServicesListCache = null; } @@ -44,23 +47,38 @@ function listApiServicesCached(options: { force?: boolean } = {}) { if (useCache && apiServicesListCache && Date.now() - apiServicesListCache.updatedAt < API_SERVICES_LIST_CACHE_TTL_MS) { return Promise.resolve(cloneApiServicesResponse(apiServicesListCache.response)); } - if (useCache && apiServicesListInFlight) { + if ( + useCache + && apiServicesListInFlight + && apiServicesListInFlightGeneration === apiServicesListGeneration + ) { return apiServicesListInFlight.then(cloneApiServicesResponse); } - apiServicesListInFlight = requestApiServicesList() + if (options.force) { + invalidateApiServicesListCache(); + } + const requestGeneration = apiServicesListGeneration; + const request = requestApiServicesList() .then((response) => { - apiServicesListCache = { - response: cloneApiServicesResponse(response), - updatedAt: Date.now(), - }; + if (requestGeneration === apiServicesListGeneration) { + apiServicesListCache = { + response: cloneApiServicesResponse(response), + updatedAt: Date.now(), + }; + } return response; }) .finally(() => { - apiServicesListInFlight = null; + if (apiServicesListInFlight === request) { + apiServicesListInFlight = null; + apiServicesListInFlightGeneration = null; + } }); + apiServicesListInFlight = request; + apiServicesListInFlightGeneration = requestGeneration; - return apiServicesListInFlight.then(cloneApiServicesResponse); + return request.then(cloneApiServicesResponse); } // ==================== Provider API (Legacy + Enhanced) ==================== diff --git a/webui/src/api/tool.test.ts b/webui/src/api/tool.test.ts new file mode 100644 index 000000000..5d6d3f056 --- /dev/null +++ b/webui/src/api/tool.test.ts @@ -0,0 +1,81 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +const getMock = vi.fn(); + +vi.mock('./client', () => ({ + default: { + get: (...args: unknown[]) => getMock(...args), + post: vi.fn(), + patch: vi.fn(), + delete: vi.fn(), + }, +})); + +describe('listAllToolPages', () => { + beforeEach(() => { + getMock.mockReset(); + }); + + it('loads every page for one service without changing the list page size', async () => { + getMock + .mockResolvedValueOnce({ + data: { + items: [{ name: 'first' }], + total: 2, + offset: 0, + limit: 200, + facets: {}, + }, + }) + .mockResolvedValueOnce({ + data: { + items: [{ name: 'second' }], + total: 2, + offset: 1, + limit: 200, + facets: {}, + }, + }); + + const { listAllToolPages } = await import('./tool'); + const result = await listAllToolPages({ + source: 'api', + sourceName: 'service-a', + enabled: 'true', + q: 'indicator', + }); + + expect(result.map((tool) => tool.name)).toEqual(['first', 'second']); + expect(getMock).toHaveBeenNthCalledWith(1, '/api/tools/page', { + params: expect.objectContaining({ + source: 'api', + source_name: 'service-a', + enabled: 'true', + q: 'indicator', + offset: 0, + limit: 200, + }), + }); + expect(getMock).toHaveBeenNthCalledWith(2, '/api/tools/page', { + params: expect.objectContaining({ source: 'api', source_name: 'service-a', offset: 1, limit: 200 }), + }); + }); + + it('stops on an empty page even when the reported total is stale', async () => { + getMock.mockResolvedValue({ + data: { + items: [], + total: 500, + offset: 0, + limit: 200, + facets: {}, + }, + }); + + const { listAllToolPages } = await import('./tool'); + const result = await listAllToolPages({ source: 'mcp', sourceName: 'server-a' }); + + expect(result).toEqual([]); + expect(getMock).toHaveBeenCalledTimes(1); + }); +}); diff --git a/webui/src/api/tool.ts b/webui/src/api/tool.ts index 93f0707b6..1890a7204 100644 --- a/webui/src/api/tool.ts +++ b/webui/src/api/tool.ts @@ -40,6 +40,7 @@ export interface ToolListPageParams { export interface ToolListFacets { category: Record; source: Record; + source_groups: Record; source_name: Record; enabled: Record; } @@ -104,5 +105,25 @@ export const toolAPI = { client.delete<{ status: string; message: string }>(`/api/tools/${name}`), }; +export async function listAllToolPages(params: ToolListPageParams): Promise { + const pageSize = 200; + const items: Tool[] = []; + let offset = 0; + + while (true) { + const response = await toolAPI.listPage({ + ...params, + offset, + limit: pageSize, + }); + const pageItems = Array.isArray(response.data.items) ? response.data.items : []; + items.push(...pageItems); + offset += pageItems.length; + if (pageItems.length === 0 || offset >= response.data.total) break; + } + + return items; +} + export const canDirectlyTestTool = (tool: Pick) => tool.source !== 'builtin'; diff --git a/webui/src/components/common/LazyLoadErrorBoundary.test.tsx b/webui/src/components/common/LazyLoadErrorBoundary.test.tsx new file mode 100644 index 000000000..ab147219d --- /dev/null +++ b/webui/src/components/common/LazyLoadErrorBoundary.test.tsx @@ -0,0 +1,67 @@ +import { Component, type ReactNode } from 'react'; +import { render, screen } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import { describe, expect, it, vi } from 'vitest'; + +import LazyLoadErrorBoundary from './LazyLoadErrorBoundary'; + +vi.mock('react-i18next', () => ({ + useTranslation: () => ({ + t: (key: string) => key, + }), +})); + +function BrokenContent({ error }: { error: Error }): never { + throw error; +} + +class CapturingBoundary extends Component< + { children: ReactNode }, + { error: Error | null } +> { + state = { error: null as Error | null }; + + static getDerivedStateFromError(error: Error) { + return { error }; + } + + render() { + if (this.state.error) return
{this.state.error.message}
; + return this.props.children; + } +} + +describe('LazyLoadErrorBoundary', () => { + it('shows a recoverable fallback when lazy content fails', async () => { + vi.spyOn(console, 'error').mockImplementation(() => undefined); + const onRetry = vi.fn(); + const user = userEvent.setup(); + const error = new Error('Loading chunk route failed'); + error.name = 'ChunkLoadError'; + + render( + + + , + ); + + expect(screen.getByRole('alert')).toHaveTextContent('error.chunkLoadFailed'); + await user.click(screen.getByRole('button', { name: 'button.retry' })); + expect(onRetry).toHaveBeenCalledTimes(1); + }); + + it('propagates ordinary render errors to the surrounding error boundary', () => { + vi.spyOn(console, 'error').mockImplementation(() => undefined); + + render( + + + + + , + ); + + expect(screen.getByTestId('outer-error')).toHaveTextContent('ordinary render failure'); + expect(screen.queryByText('error.chunkLoadFailed')).not.toBeInTheDocument(); + }); +}); diff --git a/webui/src/components/common/LazyLoadErrorBoundary.tsx b/webui/src/components/common/LazyLoadErrorBoundary.tsx new file mode 100644 index 000000000..960581ccf --- /dev/null +++ b/webui/src/components/common/LazyLoadErrorBoundary.tsx @@ -0,0 +1,70 @@ +import { Component, type ErrorInfo, type ReactNode } from 'react'; +import { useTranslation } from 'react-i18next'; + +import { isChunkLoadError, retryChunkLoad } from '@/utils/chunkLoadRecovery'; + +interface LazyLoadErrorBoundaryProps { + children: ReactNode; + mode?: 'page' | 'overlay'; + onRetry?: () => void; +} + +interface LazyLoadErrorBoundaryState { + failed: boolean; +} + +function LazyLoadErrorFallback({ + mode, + onRetry, +}: Required>) { + const { t } = useTranslation('common'); + const containerClassName = mode === 'overlay' + ? 'fixed inset-0 z-[120] flex items-center justify-center bg-black/30 p-6' + : 'flex min-h-64 items-center justify-center p-6'; + + return ( +
+
+

+ {t('error.chunkLoadFailed')} +

+

+ {t('error.chunkLoadHint')} +

+ +
+
+ ); +} + +export default class LazyLoadErrorBoundary extends Component< + LazyLoadErrorBoundaryProps, + LazyLoadErrorBoundaryState +> { + state: LazyLoadErrorBoundaryState = { failed: false }; + + static getDerivedStateFromError(error: Error): LazyLoadErrorBoundaryState { + if (!isChunkLoadError(error)) throw error; + return { failed: true }; + } + + componentDidCatch(error: Error, info: ErrorInfo) { + console.error('[LazyLoadErrorBoundary] Failed to load lazy UI:', error, info.componentStack); + } + + render() { + if (!this.state.failed) return this.props.children; + return ( + + ); + } +} diff --git a/webui/src/components/common/SessionChat.tsx b/webui/src/components/common/SessionChat.tsx index b3dd93096..8f30fe300 100644 --- a/webui/src/components/common/SessionChat.tsx +++ b/webui/src/components/common/SessionChat.tsx @@ -1673,7 +1673,8 @@ export default function SessionChat({ const [showCommandDropdown, setShowCommandDropdown] = useState(false); const [commandQuery, setCommandQuery] = useState(''); const [selectedCommandIndex, setSelectedCommandIndex] = useState(0); - const commandsLoadedRef = useRef(false); + const commandsLoadedAtRef = useRef(0); + const commandsLoadingRef = useRef(false); const [mentionQuery, setMentionQuery] = useState(''); const [mentionRange, setMentionRange] = useState<{ start: number; end: number } | null>(null); const [selectedMentionIndex, setSelectedMentionIndex] = useState(0); @@ -2214,14 +2215,17 @@ export default function SessionChat({ return () => { if (timer) clearTimeout(timer); }; }, [isCompacting, sessionId, refetch]); - /** Lazily load slash commands on first use (for autocomplete dropdown). */ + /** Lazily load slash commands and periodically revalidate while autocomplete is used. */ const loadCommandsIfNeeded = useCallback(async (): Promise => { - if (commandsLoadedRef.current) return; - commandsLoadedRef.current = true; // Optimistic: prevent concurrent fetches + if (commandsLoadingRef.current || Date.now() - commandsLoadedAtRef.current < 5_000) return; + commandsLoadingRef.current = true; try { setCommands(await fetchSessionChatCommands()); + commandsLoadedAtRef.current = Date.now(); } catch { - commandsLoadedRef.current = false; // Allow retry on failure + commandsLoadedAtRef.current = 0; + } finally { + commandsLoadingRef.current = false; } }, []); diff --git a/webui/src/components/common/UpdateModal.test.tsx b/webui/src/components/common/UpdateModal.test.tsx new file mode 100644 index 000000000..67e4cda8e --- /dev/null +++ b/webui/src/components/common/UpdateModal.test.tsx @@ -0,0 +1,67 @@ +import { render, screen, waitFor } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +import UpdateModal from './UpdateModal'; + +const { applyUpdate, checkUpdate } = vi.hoisted(() => ({ + applyUpdate: vi.fn(), + checkUpdate: vi.fn(), +})); + +vi.mock('@/api/update', () => ({ + applyUpdate, + checkUpdate, +})); + +vi.mock('react-i18next', () => ({ + useTranslation: () => ({ + t: (key: string) => key, + i18n: { language: 'zh-CN' }, + }), +})); + +const currentVersion = { + current_version: '2026.07.10', + latest_version: '2026.07.10', + has_update: false, + release_notes: null, + release_url: null, + error: null, +}; + +describe('UpdateModal update checks', () => { + beforeEach(() => { + vi.clearAllMocks(); + checkUpdate.mockResolvedValue(currentVersion); + }); + + it('forces the initial request when Layout opens a manual check', async () => { + render( + , + ); + + await waitFor(() => { + expect(checkUpdate).toHaveBeenCalledWith('zh-CN', 'flocks', true); + }); + }); + + it('forces explicit refreshes from the modal', async () => { + const user = userEvent.setup(); + render( + , + ); + + await user.click(screen.getByRole('button', { name: 'checkUpdate' })); + + await waitFor(() => { + expect(checkUpdate).toHaveBeenCalledWith('zh-CN', 'flocks', true); + }); + }); +}); diff --git a/webui/src/components/common/UpdateModal.tsx b/webui/src/components/common/UpdateModal.tsx index ab312d31f..bef95e371 100644 --- a/webui/src/components/common/UpdateModal.tsx +++ b/webui/src/components/common/UpdateModal.tsx @@ -53,13 +53,21 @@ function formatBytes(value?: number | null): string { interface UpdateModalProps { initialInfo?: VersionInfo | null; + forceInitialCheck?: boolean; edition?: UpdateEdition; canUpgrade?: boolean; onClose: () => void; onDismiss?: () => void; } -export default function UpdateModal({ initialInfo, edition = 'flocks', canUpgrade = true, onClose, onDismiss }: UpdateModalProps) { +export default function UpdateModal({ + initialInfo, + forceInitialCheck = false, + edition = 'flocks', + canUpgrade = true, + onClose, + onDismiss, +}: UpdateModalProps) { const { t, i18n } = useTranslation('update'); const [info, setInfo] = useState(initialInfo ?? null); const [checking, setChecking] = useState(false); @@ -83,11 +91,11 @@ export default function UpdateModal({ initialInfo, edition = 'flocks', canUpgrad setRestarting(val); }; - const fetchVersion = useCallback(async () => { + const fetchVersion = useCallback(async (force = false) => { setChecking(true); setError(null); try { - const data = await checkUpdate(i18n.language, edition); + const data = await checkUpdate(i18n.language, edition, force); setInfo(data); if (data.error) setError(data.error); } catch (e: any) { @@ -99,9 +107,9 @@ export default function UpdateModal({ initialInfo, edition = 'flocks', canUpgrad useEffect(() => { if (!initialInfo) { - fetchVersion(); + void fetchVersion(forceInitialCheck); } - }, [fetchVersion, initialInfo]); + }, [fetchVersion, forceInitialCheck, initialInfo]); const handleUpgrade = useCallback(async () => { if (!info?.has_update) return; @@ -407,7 +415,7 @@ export default function UpdateModal({ initialInfo, edition = 'flocks', canUpgrad
+
+
+ ); + } + return (
: } {viewMode === 'table' ? text.treeView : text.tableView} - + {canManageHub && ( + + )}
} /> @@ -546,10 +658,19 @@ export default function HubPage() {
+ {visibleCatalogError && ( +
+ {visibleCatalogError} +
+ )} {viewMode === 'table' ? ( + ) : treeLoading && treeItems.length === 0 ? ( +
+ +
) : ( - + )}
@@ -1063,11 +1184,13 @@ function ActionButtons({ item, actionId, text, onAction, compact = false }: { compact?: boolean; onAction: (entry: HubCatalogEntry, action: 'install' | 'update' | 'uninstall') => void; }) { + const { user } = useAuth(); const busy = actionId?.startsWith(`${item.type}:${item.id}:`); const buttonClass = compact ? 'p-1.5 rounded-md border border-gray-200 hover:bg-gray-50 disabled:opacity-50' : 'inline-flex items-center gap-1 whitespace-nowrap px-2 py-1 rounded-md border border-gray-200 text-xs hover:bg-gray-50 disabled:opacity-50'; if (busy) return ; + if (user?.role !== 'admin') return null; if (item.native && item.state === 'installed') return null; if (item.state === 'available') return ; if (item.state === 'updateAvailable') return ; diff --git a/webui/src/pages/Model/index.test.tsx b/webui/src/pages/Model/index.test.tsx index 06e092dc7..1515819c6 100644 --- a/webui/src/pages/Model/index.test.tsx +++ b/webui/src/pages/Model/index.test.tsx @@ -20,6 +20,9 @@ const mocks = vi.hoisted(() => ({ listDefinitions: vi.fn(), catalogList: vi.fn(), createProvider: vi.fn(), + getCredentials: vi.fn(), + setCredentials: vi.fn(), + testCredentials: vi.fn(), })); vi.mock('react-i18next', () => ({ @@ -42,6 +45,7 @@ vi.mock('react-i18next', () => ({ 'form.baseUrlRequired': 'Please enter Base URL', 'form.apiKeyOptional': '(optional, leave empty for no-auth gateways)', 'form.apiKeyOptionalHint': 'Leave empty for no-auth gateway', + 'form.apiKeyKeepExisting': 'Leave blank to keep the existing API key', 'form.searchProvider': 'Search providers...', 'form.noResults': 'No results', 'form.alreadyAdded': 'Already added', @@ -108,9 +112,9 @@ vi.mock('@/components/common/EntitySheet', () => ({ vi.mock('@/api/provider', () => ({ providerAPI: { - getCredentials: vi.fn(), - setCredentials: vi.fn(), - testCredentials: vi.fn(), + getCredentials: mocks.getCredentials, + setCredentials: mocks.setCredentials, + testCredentials: mocks.testCredentials, }, modelV2API: { listDefinitions: mocks.listDefinitions, @@ -218,3 +222,120 @@ describe('ModelPage add provider dialog', () => { }); }); }); + +describe('ModelPage configure provider dialog', () => { + const provider = { + id: 'openai', + name: 'OpenAI', + source: 'config', + env: [], + key: null, + options: {}, + models: {}, + configured: true, + modelCount: 1, + category: 'connected', + }; + const model = { + id: 'gpt-4o', + name: 'GPT-4o', + provider_id: 'openai', + model_type: 'chat', + status: 'active', + capabilities: { + features: [], + supports_streaming: true, + supports_tools: true, + }, + }; + + beforeEach(() => { + vi.clearAllMocks(); + localStorage.clear(); + sessionStorage.clear(); + mocks.useProviders.mockReturnValue({ + providers: [provider], + connectedIds: ['openai'], + loading: false, + error: null, + refetch: mocks.refetch, + }); + mocks.getSummary.mockResolvedValue({ data: null }); + mocks.getResolved.mockResolvedValue({ data: null }); + mocks.listDefinitions.mockResolvedValue({ data: { models: [model], total: 1 } }); + mocks.catalogList.mockResolvedValue({ + data: { + providers: [{ id: 'openai', models: [] }], + }, + }); + mocks.getCredentials.mockResolvedValue({ + data: { + secret_id: 'openai_llm_key', + api_key: null, + api_key_masked: 'sk-***1234', + base_url: 'https://old.example.com/v1', + has_credential: true, + }, + }); + mocks.setCredentials.mockResolvedValue({ data: { success: true } }); + mocks.testCredentials.mockResolvedValue({ + data: { + success: true, + message: 'ok', + model_id: 'gpt-4o', + question: 'ping', + answer: 'pong', + latency_ms: 10, + }, + }); + }); + + async function openConfigureDialog(user: ReturnType) { + renderWithRouter(); + await user.click(await screen.findByTitle('Configure')); + expect(await screen.findByTestId('entity-sheet')).toBeInTheDocument(); + } + + it('preserves an existing secret when saving a new base URL with a blank key', async () => { + const user = userEvent.setup(); + await openConfigureDialog(user); + + const apiKeyInput = screen.getByPlaceholderText('Leave blank to keep the existing API key'); + expect(apiKeyInput).toHaveValue(''); + expect(apiKeyInput).not.toHaveValue('sk-***1234'); + + const baseUrlInput = screen.getByPlaceholderText('https://api.example.com/v1'); + await user.clear(baseUrlInput); + await user.type(baseUrlInput, 'https://new.example.com/v1'); + await user.click(screen.getByRole('button', { name: 'Save' })); + + await waitFor(() => expect(mocks.setCredentials).toHaveBeenCalled()); + const payload = mocks.setCredentials.mock.calls.at(-1)?.[1]; + expect(payload).toEqual(expect.objectContaining({ + base_url: 'https://new.example.com/v1', + })); + expect(payload).not.toHaveProperty('api_key'); + }); + + it('persists a base URL change without a key before testing the connection', async () => { + const user = userEvent.setup(); + await openConfigureDialog(user); + mocks.setCredentials.mockClear(); + mocks.testCredentials.mockClear(); + + const baseUrlInput = screen.getByPlaceholderText('https://api.example.com/v1'); + await user.clear(baseUrlInput); + await user.type(baseUrlInput, 'https://test.example.com/v1'); + await user.click(screen.getByRole('button', { name: 'form.testConnection2' })); + + await waitFor(() => expect(mocks.setCredentials).toHaveBeenCalled()); + const payload = mocks.setCredentials.mock.calls[0]?.[1]; + expect(payload).toEqual(expect.objectContaining({ + base_url: 'https://test.example.com/v1', + })); + expect(payload).not.toHaveProperty('api_key'); + await waitFor(() => { + expect(mocks.testCredentials).toHaveBeenCalledWith('openai', 'gpt-4o'); + }); + }); +}); diff --git a/webui/src/pages/Model/index.tsx b/webui/src/pages/Model/index.tsx index 48f617682..3a8e1b4c8 100644 --- a/webui/src/pages/Model/index.tsx +++ b/webui/src/pages/Model/index.tsx @@ -31,7 +31,7 @@ import { import type { ProviderCredentials, ModelDefinitionV2, UsageStats, CatalogProvider, CatalogModel, CatalogCredentialField, ModelSettingV2, - CustomModelCreate, + CustomModelCreate, ProviderCredentialInput, } from '@/types'; // ==================== Provider Auth Helpers ==================== @@ -2192,7 +2192,10 @@ function ConfigureProviderDialog({ provider, existingCredentials, models, onClos const toast = useToast(); const { t } = useTranslation('model'); const hasExisting = existingCredentials?.has_credential ?? false; - const existingKey = existingCredentials?.api_key ?? ''; + // Credential reads intentionally never return the raw key. Keep the input + // empty and use hasExisting to express "leave blank to preserve"; in + // particular, never seed this field with api_key_masked. + const existingKey = ''; const existingBaseUrl = existingCredentials?.base_url ?? ''; const [baseUrl, setBaseUrl] = useState(existingCredentials?.base_url ?? ''); @@ -2269,19 +2272,21 @@ function ConfigureProviderDialog({ provider, existingCredentials, models, onClos }; const handleSubmit = async () => { - if (!apiKey.trim() && !providerAllowsEmptyApiKey(provider.id)) { + const nextApiKey = apiKey.trim(); + if (!nextApiKey && !hasExisting && !providerAllowsEmptyApiKey(provider.id)) { toast.warning('Please enter API Key'); return; } try { setLoading(true); - await providerAPI.setCredentials(provider.id, { - api_key: apiKey.trim() || 'not-needed', + const payload: ProviderCredentialInput = { base_url: baseUrl.trim() || undefined, provider_name: (provider.id === 'openai-compatible' || provider.id.startsWith('custom-')) ? (providerName.trim() || undefined) : undefined, - }); + }; + if (nextApiKey) payload.api_key = nextApiKey; + await providerAPI.setCredentials(provider.id, payload); // Sync catalog model list: add newly selected, delete deselected if (catalogModels.length > 0) { @@ -2320,24 +2325,28 @@ function ConfigureProviderDialog({ provider, existingCredentials, models, onClos { apiKey, baseUrl }, ); + const nextApiKey = apiKey.trim(); + if (!nextApiKey && !hasExisting && !providerAllowsEmptyApiKey(provider.id)) { + toast.warning('Please enter API Key first'); + return; + } + // Persist pending credential changes before testing so the backend uses // the latest base URL even when the API key itself did not change. - if (apiKey.trim() && hasPendingChanges) { + if (hasPendingChanges) { try { - await providerAPI.setCredentials(provider.id, { - api_key: apiKey.trim(), + const payload: ProviderCredentialInput = { base_url: baseUrl.trim() || undefined, provider_name: (provider.id === 'openai-compatible' || provider.id.startsWith('custom-')) ? (providerName.trim() || undefined) : undefined, - }); + }; + if (nextApiKey) payload.api_key = nextApiKey; + await providerAPI.setCredentials(provider.id, payload); } catch (err: any) { toast.error(t('deleteFailed'), err.message); return; } - } else if (!apiKey.trim() && !hasExisting && !providerAllowsEmptyApiKey(provider.id)) { - toast.warning('Please enter API Key first'); - return; } try { setTesting(true); @@ -2451,7 +2460,7 @@ ${hasExisting ? '你已有凭证配置,可以更新或测试连接。' : '请
@@ -127,7 +131,7 @@ export default function MonitoringPage() {
- {status?.activeAgents || 0} + {status?.activeAgents ?? t('unavailable')}
{t('activeAgents')}
@@ -137,7 +141,7 @@ export default function MonitoringPage() {
- {metrics?.messageRate.toFixed(1) || '0'} + {metrics?.messageRate == null ? t('unavailable') : metrics.messageRate.toFixed(1)}
{t('messagesPerMin')}
@@ -151,31 +155,31 @@ export default function MonitoringPage() {
{t('metrics.messageRate')}
- {metrics.messageRate.toFixed(1)}/min + {formatRate(metrics.messageRate)}
{t('metrics.toolCallRate')}
- {metrics.toolCallRate.toFixed(1)}/min + {formatRate(metrics.toolCallRate)}
{t('metrics.errorRate')}
- {metrics.errorRate.toFixed(1)}/min + {formatRate(metrics.errorRate)}
{t('metrics.avgResponse')}
- {metrics.avgResponseTime.toFixed(0)}ms + {metrics.avgResponseTime == null ? t('unavailable') : `${metrics.avgResponseTime.toFixed(0)}ms`}
{t('metrics.activeRequests')}
- {metrics.activeRequests} + {metrics.activeRequests ?? t('unavailable')}
diff --git a/webui/src/pages/Tool/components/APITabContent.test.tsx b/webui/src/pages/Tool/components/APITabContent.test.tsx new file mode 100644 index 000000000..b6329a742 --- /dev/null +++ b/webui/src/pages/Tool/components/APITabContent.test.tsx @@ -0,0 +1,115 @@ +import { render, screen, waitFor } from '@testing-library/react'; +import userEvent from '@testing-library/user-event'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +import APITabContent from './APITabContent'; + +const { apiDetailProps, listAllToolPages, mcpAPI, providerAPI } = vi.hoisted(() => ({ + apiDetailProps: vi.fn(), + listAllToolPages: vi.fn(), + mcpAPI: { + catalogInstall: vi.fn(), + connect: vi.fn(), + }, + providerAPI: { + listApiServices: vi.fn(), + updateApiService: vi.fn(), + deleteApiService: vi.fn(), + }, +})); + +vi.mock('@/api/provider', () => ({ providerAPI })); +vi.mock('@/api/mcp', () => ({ mcpAPI })); +vi.mock('@/api/tool', () => ({ listAllToolPages })); + +vi.mock('@/components/common/LoadingSpinner', () => ({ + default: () =>
loading
, +})); + +vi.mock('@/components/common/EmptyState', () => ({ + default: ({ title }: { title: string }) =>
{title}
, +})); + +vi.mock('./ServiceDetailPanel', () => ({ + APIServiceDetailPanel: (props: { serviceTools: Array<{ name: string }> }) => { + apiDetailProps(props); + return ( +
+ detail-panel + {props.serviceTools.map((tool) => {tool.name})} +
+ ); + }, +})); + +vi.mock('react-i18next', () => ({ + useTranslation: () => ({ + t: (key: string) => key, + i18n: { language: 'zh-CN' }, + }), +})); + +describe('APITabContent', () => { + beforeEach(() => { + vi.clearAllMocks(); + providerAPI.listApiServices.mockResolvedValue({ data: [] }); + listAllToolPages.mockResolvedValue([]); + }); + + it('loads the complete tool list when a service detail drawer opens', async () => { + const user = userEvent.setup(); + providerAPI.listApiServices.mockResolvedValue({ + data: [ + { + id: 'service-a', + name: 'Service A', + description: 'Service A API', + enabled: true, + status: 'connected', + tool_count: 2, + verify_ssl: false, + }, + ], + }); + listAllToolPages.mockResolvedValue([ + { + name: 'complete-api-tool', + description: 'Loaded beyond the current page', + category: 'custom', + source: 'api', + source_name: 'service-a', + enabled: true, + }, + ]); + + render( + , + ); + + await user.click((await screen.findByText('Service A')).closest('button')!); + + await waitFor(() => { + expect(listAllToolPages).toHaveBeenCalledWith({ + source: 'api', + sourceName: 'service-a', + sortBy: 'name', + sortDir: 'asc', + }); + }); + expect(await screen.findByText('complete-api-tool')).toBeInTheDocument(); + expect(apiDetailProps).toHaveBeenLastCalledWith( + expect.objectContaining({ + serviceTools: [expect.objectContaining({ name: 'complete-api-tool' })], + }), + ); + }); +}); diff --git a/webui/src/pages/Tool/components/APITabContent.tsx b/webui/src/pages/Tool/components/APITabContent.tsx index 6dafd3f22..13f54e3c7 100644 --- a/webui/src/pages/Tool/components/APITabContent.tsx +++ b/webui/src/pages/Tool/components/APITabContent.tsx @@ -5,7 +5,7 @@ import { import { useTranslation } from 'react-i18next'; import { mcpAPI } from '@/api/mcp'; import { providerAPI } from '@/api/provider'; -import type { Tool } from '@/api/tool'; +import { listAllToolPages, type Tool } from '@/api/tool'; import type { APIServiceSummary, MCPCatalogCategory, MCPCatalogEntry } from '@/types'; import EmptyState from '@/components/common/EmptyState'; import LoadingSpinner from '@/components/common/LoadingSpinner'; @@ -65,6 +65,7 @@ export default function APITabContent({ const [installing, setInstalling] = useState(null); const [credModalEntry, setCredModalEntry] = useState(null); const [credValues, setCredValues] = useState>({}); + const [serviceToolCache, setServiceToolCache] = useState>({}); const fetchServices = useCallback(async () => { try { @@ -87,10 +88,36 @@ export default function APITabContent({ () => (selectedServiceId ? services.find((service) => service.id === selectedServiceId) ?? null : null), [selectedServiceId, services], ); - const selectedModuleTools = selectedServiceId ? (toolsByModule[selectedServiceId] || []) : []; + const selectedModuleTools = selectedServiceId + ? (serviceToolCache[selectedServiceId] ?? toolsByModule[selectedServiceId] ?? []) + : []; const isConfigured = useCallback((entryId: string) => configuredIds.has(entryId), [configuredIds]); const PRIORITY_IDS = useMemo(() => new Set(['virustotal_mcp', 'urlhaus']), []); + useEffect(() => { + setServiceToolCache({}); + }, [tools]); + + useEffect(() => { + if (!selectedServiceId || serviceToolCache[selectedServiceId]) return; + let cancelled = false; + void listAllToolPages({ + source: 'api', + sourceName: selectedServiceId, + sortBy: 'name', + sortDir: 'asc', + }).then((serviceTools) => { + if (!cancelled) { + setServiceToolCache((current) => ({ ...current, [selectedServiceId]: serviceTools })); + } + }).catch(() => { + // Keep the current page slice visible; reopening retries the complete service list. + }); + return () => { + cancelled = true; + }; + }, [selectedServiceId, serviceToolCache]); + const inactiveCatalogEntries = useMemo(() => { const activeModuleIds = new Set(services.map((service) => service.id)); return catalogEntries.filter((entry) => !activeModuleIds.has(entry.id)); @@ -311,7 +338,6 @@ export default function APITabContent({ ) : (
{services.map((service) => { - const serviceTools = toolsByModule[service.id] || []; const isSelected = selectedServiceId === service.id; const rowDescription = getServiceDescription(service) || `${service.name} API service`; @@ -361,7 +387,7 @@ export default function APITabContent({ {/* Stats column */}
- {serviceTools.length} + {service.tool_count} {service.latency_ms != null && {service.latency_ms}ms}
diff --git a/webui/src/pages/Tool/components/MCPTabContent.test.tsx b/webui/src/pages/Tool/components/MCPTabContent.test.tsx index b6895b306..f5948fdf0 100644 --- a/webui/src/pages/Tool/components/MCPTabContent.test.tsx +++ b/webui/src/pages/Tool/components/MCPTabContent.test.tsx @@ -4,12 +4,14 @@ import userEvent from '@testing-library/user-event'; import MCPTabContent from './MCPTabContent'; -const { mcpAPI } = vi.hoisted(() => ({ +const { listAllToolPages, mcpAPI, mcpDetailProps } = vi.hoisted(() => ({ + listAllToolPages: vi.fn(), mcpAPI: { list: vi.fn(), catalogInstall: vi.fn(), connect: vi.fn(), }, + mcpDetailProps: vi.fn(), })); vi.mock('@/api/mcp', () => ({ @@ -17,6 +19,7 @@ vi.mock('@/api/mcp', () => ({ })); vi.mock('@/api/tool', () => ({ + listAllToolPages, toolAPI: { test: vi.fn(), }, @@ -31,7 +34,15 @@ vi.mock('@/components/common/EmptyState', () => ({ })); vi.mock('./ServiceDetailPanel', () => ({ - MCPServerDetailPanel: () =>
detail-panel
, + MCPServerDetailPanel: (props: { serverTools: Array<{ name: string }> }) => { + mcpDetailProps(props); + return ( +
+ detail-panel + {props.serverTools.map((tool) => {tool.name})} +
+ ); + }, })); vi.mock('react-i18next', () => ({ @@ -69,6 +80,7 @@ describe('MCPTabContent', () => { }, }); mcpAPI.connect.mockResolvedValue({ data: true }); + listAllToolPages.mockResolvedValue([]); }); it('collects credentials and env vars before installing protected catalog entries', async () => { @@ -145,4 +157,61 @@ describe('MCPTabContent', () => { expect(onConfiguredChange).toHaveBeenCalledWith('panther'); expect(onRefreshTools).toHaveBeenCalled(); }); + + it('loads the complete tool list when a server detail drawer opens', async () => { + const user = userEvent.setup(); + mcpAPI.list.mockResolvedValue({ + data: [ + { + name: 'server-a', + status: 'connected', + tools: [], + resources: [], + tools_count: 2, + resources_count: 0, + }, + ], + }); + listAllToolPages.mockResolvedValue([ + { + name: 'complete-tool', + description: 'Loaded beyond the current page', + category: 'custom', + source: 'mcp', + source_name: 'server-a', + enabled: true, + }, + ]); + + render( + , + ); + + await user.click(await screen.findByText('server-a')); + + await waitFor(() => { + expect(listAllToolPages).toHaveBeenCalledWith({ + source: 'mcp', + sourceName: 'server-a', + sortBy: 'name', + sortDir: 'asc', + }); + }); + expect(await screen.findByText('complete-tool')).toBeInTheDocument(); + expect(mcpDetailProps).toHaveBeenLastCalledWith( + expect.objectContaining({ + serverTools: [expect.objectContaining({ name: 'complete-tool' })], + }), + ); + }); }); diff --git a/webui/src/pages/Tool/components/MCPTabContent.tsx b/webui/src/pages/Tool/components/MCPTabContent.tsx index 9c30bab5d..48fe0592c 100644 --- a/webui/src/pages/Tool/components/MCPTabContent.tsx +++ b/webui/src/pages/Tool/components/MCPTabContent.tsx @@ -5,7 +5,7 @@ import { Download, ExternalLink, Star, TestTube, Play, Info, CheckCircle, XCircle, AlertTriangle, Power, } from 'lucide-react'; import { mcpAPI } from '@/api/mcp'; -import { toolAPI } from '@/api/tool'; +import { listAllToolPages, toolAPI } from '@/api/tool'; import type { Tool } from '@/api/tool'; import type { MCPCatalogCategory, MCPCatalogEntry, MCPServer } from '@/types'; import LoadingSpinner from '@/components/common/LoadingSpinner'; @@ -73,6 +73,7 @@ export default function MCPTabContent({ const [setupEntry, setSetupEntry] = useState(null); const [setupCredentials, setSetupCredentials] = useState>({}); const [setupEnvOverrides, setSetupEnvOverrides] = useState>({}); + const [serverToolCache, setServerToolCache] = useState>({}); const fetchServers = useCallback(async () => { try { @@ -165,6 +166,30 @@ export default function MCPTabContent({ return map; }, [tools]); + useEffect(() => { + setServerToolCache({}); + }, [tools]); + + useEffect(() => { + if (!selectedServer || serverToolCache[selectedServer]) return; + let cancelled = false; + void listAllToolPages({ + source: 'mcp', + sourceName: selectedServer, + sortBy: 'name', + sortDir: 'asc', + }).then((serverTools) => { + if (!cancelled) { + setServerToolCache((current) => ({ ...current, [selectedServer]: serverTools })); + } + }).catch(() => { + // Keep the current page slice visible; reopening retries the complete server list. + }); + return () => { + cancelled = true; + }; + }, [selectedServer, serverToolCache]); + const setSelectedCard = useCallback((name: string | null) => { setSelectedServer(name); setSelectedServerData(name ? (servers.find((server) => server.name === name) ?? null) : null); @@ -512,7 +537,7 @@ export default function MCPTabContent({ {/* Stats column */}
- {serverTools.length} + {server.tools_count ?? serverTools.length} {server.connected_at ? ( {formatDuration(server.connected_at, t)} ) : entry ? ( @@ -711,7 +736,7 @@ export default function MCPTabContent({ )} handleConnect(selectedServer)} onDisconnect={() => handleDisconnect(selectedServer)} onRefresh={async () => { await mcpAPI.refresh(selectedServer); await fetchServers(); await onRefreshTools(); }} diff --git a/webui/src/pages/Tool/index.tsx b/webui/src/pages/Tool/index.tsx index 962160b1b..3c1e51c22 100644 --- a/webui/src/pages/Tool/index.tsx +++ b/webui/src/pages/Tool/index.tsx @@ -65,6 +65,7 @@ import { shouldLoadMcpCatalog, type TabKey, } from './tabLoading'; +import { getToolTabCounts } from './tabCounts'; // ============================================================================ // Constants & Config @@ -230,6 +231,8 @@ export default function ToolPage() { const [catalogEntries, setCatalogEntries] = useState([]); const [catalogCategories, setCatalogCategories] = useState>({}); const [catalogLoading, setCatalogLoading] = useState(false); + const [catalogError, setCatalogError] = useState(null); + const [catalogRetryKey, setCatalogRetryKey] = useState(0); const [configuredIds, setConfiguredIds] = useState>(new Set()); const catalogLoadedRef = useRef(false); @@ -238,8 +241,10 @@ export default function ToolPage() { let cancelled = false; const loadCatalog = async () => { + let loaded = false; try { setCatalogLoading(true); + setCatalogError(null); const [entriesRes, catsRes] = await Promise.all([ mcpAPI.catalogList(), mcpAPI.catalogCategories(), @@ -247,6 +252,7 @@ export default function ToolPage() { if (cancelled) return; setCatalogEntries(entriesRes.data); setCatalogCategories(catsRes.data); + loaded = true; // Get currently configured IDs (no auto-setup to avoid re-adding removed entries) try { const confRes = await mcpAPI.catalogConfigured(); @@ -254,9 +260,12 @@ export default function ToolPage() { } catch { /* ignore */ } } catch (err) { console.error('Failed to load catalog:', err); + if (!cancelled) { + setCatalogError(err instanceof Error ? err.message : 'Failed to load MCP catalog'); + } } finally { if (!cancelled) { - catalogLoadedRef.current = true; + catalogLoadedRef.current = loaded; setCatalogLoading(false); } } @@ -266,7 +275,7 @@ export default function ToolPage() { return () => { cancelled = true; }; - }, [activeTab]); + }, [activeTab, catalogRetryKey]); useEffect(() => { if (!shouldLoadMcpCatalog(activeTab)) { @@ -309,16 +318,10 @@ export default function ToolPage() { // API catalog not yet implemented — keep this empty so entries are not duplicated across tabs. const apiCatalogEntries = useMemo(() => [] as MCPCatalogEntry[], []); - // Compute tab counts from server-side facets for the current query/filter set. - const tabCounts = useMemo(() => { - const allFromSourceFacets = Object.values(toolFacets.source).reduce((sum, count) => sum + count, 0); - return { - all: allFromSourceFacets || totalTools, - mcp: toolFacets.source.mcp ?? 0, - api: Math.max(toolFacets.source.api ?? 0, apiEnabledServicesCount), - local: toolFacets.source.plugin_py ?? 0, - }; - }, [totalTools, toolFacets.source, apiEnabledServicesCount]); + const tabCounts = useMemo( + () => getToolTabCounts(totalTools, toolFacets, apiEnabledServicesCount), + [totalTools, toolFacets, apiEnabledServicesCount], + ); const enabledSummary = useMemo(() => ({ active: toolFacets.enabled['true'] ?? tools.filter((tool) => tool.enabled).length, inactive: toolFacets.enabled['false'] ?? 0, @@ -561,6 +564,19 @@ export default function ToolPage() {
)} + {activeTab === 'mcp' && catalogError && ( +
+ {catalogError} + +
+ )} + {/* Tab Content */}
{isTabPageLoading ? ( @@ -593,13 +609,22 @@ export default function ToolPage() { onConfiguredChange={onConfiguredChange} /> ) : activeTab === 'local' ? ( - +
+ + +
) : ( /* All tab: active tools only */
diff --git a/webui/src/pages/Tool/tabCounts.test.ts b/webui/src/pages/Tool/tabCounts.test.ts index a5b660397..2ca2cec9d 100644 --- a/webui/src/pages/Tool/tabCounts.test.ts +++ b/webui/src/pages/Tool/tabCounts.test.ts @@ -1,20 +1,18 @@ import { describe, expect, it } from 'vitest'; -import type { Tool } from '@/api/tool'; +import type { ToolListFacets } from '@/api/tool'; import { getToolTabCounts } from './tabCounts'; describe('getToolTabCounts', () => { it('uses enabled API service count for the API tab', () => { - const tools = [ - { source: 'api', source_name: 'f1' }, - { source: 'api', source_name: 'f2' }, - { source: 'api', source_name: 'f3' }, - { source: 'mcp', source_name: 'mcp-a' }, - { source: 'mcp', source_name: 'mcp-a' }, - { source: 'plugin_py', source_name: 'local' }, - { source: 'plugin_py', source_name: 'local' }, - ] as Tool[]; + const facets: ToolListFacets = { + category: {}, + source: { api: 3, mcp: 2, plugin_py: 2 }, + source_groups: { api: 3, mcp: 1, plugin_py: 1 }, + source_name: {}, + enabled: {}, + }; - expect(getToolTabCounts(tools, 3)).toEqual({ + expect(getToolTabCounts(2, facets, 3)).toEqual({ all: 7, mcp: 1, api: 3, diff --git a/webui/src/pages/Tool/tabCounts.ts b/webui/src/pages/Tool/tabCounts.ts index 76b12e62a..a81edc7b4 100644 --- a/webui/src/pages/Tool/tabCounts.ts +++ b/webui/src/pages/Tool/tabCounts.ts @@ -1,4 +1,4 @@ -import type { Tool } from '@/api/tool'; +import type { ToolListFacets } from '@/api/tool'; export interface ToolTabCounts { all: number; @@ -7,11 +7,16 @@ export interface ToolTabCounts { local: number; } -export function getToolTabCounts(tools: Tool[], apiEnabledServicesCount: number): ToolTabCounts { +export function getToolTabCounts( + totalTools: number, + facets: ToolListFacets, + apiEnabledServicesCount: number, +): ToolTabCounts { + const allTools = Object.values(facets.source).reduce((sum, count) => sum + count, 0); return { - all: tools.length, - mcp: new Set(tools.filter((tool) => tool.source === 'mcp').map((tool) => tool.source_name)).size, + all: allTools || totalTools, + mcp: facets.source_groups.mcp ?? 0, api: apiEnabledServicesCount, - local: tools.filter((tool) => tool.source === 'plugin_py').length, + local: facets.source.plugin_py ?? 0, }; } diff --git a/webui/src/routes/index.tsx b/webui/src/routes/index.tsx index df3834ba4..72e3113b1 100644 --- a/webui/src/routes/index.tsx +++ b/webui/src/routes/index.tsx @@ -3,11 +3,15 @@ import type { ComponentType, ReactNode } from 'react'; import { Routes as RouterRoutes, Route, Navigate, useLocation, useParams } from 'react-router-dom'; import { useTranslation } from 'react-i18next'; import Layout from '@/components/layout/Layout'; +import LazyLoadErrorBoundary from '@/components/common/LazyLoadErrorBoundary'; import RoutePageSkeleton from '@/components/common/RoutePageSkeleton'; import AuthLayout from '@/components/layout/AuthLayout'; import Home from '@/pages/Home'; import { useAuth } from '@/contexts/AuthContext'; import { preloadI18nNamespaces } from '@/i18nResources'; +import { installVitePreloadErrorRecovery, recoverLazyLoad } from '@/utils/chunkLoadRecovery'; + +installVitePreloadErrorRecovery(); // All non-Home pages are code-split. Home stays eager because it's the very // first frame after auth and we don't want a Suspense flash on initial paint. @@ -20,10 +24,12 @@ function lazyPage( loader: () => Promise, namespaces: readonly string[] = [], ) { - return lazy(() => Promise.all([ - loader(), - preloadI18nNamespaces(namespaces), - ]).then(([module]) => module)); + return lazy(() => recoverLazyLoad( + Promise.all([ + loader(), + preloadI18nNamespaces(namespaces), + ]).then(([module]) => module), + )); } const SessionPage = lazyPage(() => import('@/pages/Session'), ['session']); @@ -53,9 +59,11 @@ const ROUTE_FALLBACK_DELAY_MS = 180; function LazyRoute({ children }: { children: ReactNode }) { return ( - }> - {children} - + + }> + {children} + + ); } @@ -111,31 +119,37 @@ export function Routes() { if (!bootstrapped) { return ( - }> - - } /> - } /> - - + + }> + + } /> + } /> + + + ); } if (!user) { return ( - }> - - } /> - } /> - - + + }> + + } /> + } /> + + + ); } if (user.must_reset_password) { return ( - }> - - + + }> + + + ); } diff --git a/webui/src/types/index.ts b/webui/src/types/index.ts index 3e0517bd2..e78dbfd28 100644 --- a/webui/src/types/index.ts +++ b/webui/src/types/index.ts @@ -397,12 +397,10 @@ export interface MCPCatalogStats { export interface ProviderCredentials { secret_id?: string | null; /** - * The actual API key value, when one exists. - * Note: For openai-compatible / custom-* providers configured WITHOUT an - * API key (internal no-auth gateways) the backend stores a sentinel value - * but returns ``api_key: null`` while still setting ``has_credential: true`` - * and a real ``secret_id``. UI code should rely on ``has_credential`` (not - * ``api_key``) to decide whether a credential record exists. + * Write-only compatibility field. Credential reads never return the raw key; + * they return ``api_key: null`` plus ``api_key_masked`` and + * ``has_credential``. UI code must never submit the masked value as a new + * secret. Leave ``api_key`` out of an update to preserve the stored key. */ api_key?: string | null; api_key_masked?: string | null; @@ -410,6 +408,7 @@ export interface ProviderCredentials { secret_masked?: string | null; base_url?: string | null; username?: string | null; + /** Sensitive entries are masked on reads and must not be resubmitted unchanged. */ fields?: Record; secret_ids?: Record; has_credential: boolean; diff --git a/webui/src/utils/chunkLoadRecovery.test.ts b/webui/src/utils/chunkLoadRecovery.test.ts new file mode 100644 index 000000000..be99df7f2 --- /dev/null +++ b/webui/src/utils/chunkLoadRecovery.test.ts @@ -0,0 +1,53 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +import { recoverLazyLoad, reloadOnceForChunkLoadError } from './chunkLoadRecovery'; + +describe('chunkLoadRecovery', () => { + beforeEach(() => { + sessionStorage.clear(); + }); + + it('reloads only once for the same failed chunk and allows a new chunk failure', () => { + const reload = vi.fn(); + const staleChunk = new TypeError('Failed to fetch /assets/Session-old.js'); + + expect(reloadOnceForChunkLoadError(staleChunk, reload)).toBe(true); + expect(reloadOnceForChunkLoadError(staleChunk, reload)).toBe(false); + expect(reload).toHaveBeenCalledTimes(1); + + expect(reloadOnceForChunkLoadError( + new TypeError('Failed to fetch /assets/Session-new.js'), + reload, + )).toBe(true); + expect(reload).toHaveBeenCalledTimes(2); + }); + + it('does not reload when session storage cannot guard against a loop', () => { + const reload = vi.fn(); + const getItem = vi.spyOn(Storage.prototype, 'getItem').mockImplementation(() => { + throw new Error('storage unavailable'); + }); + + expect(reloadOnceForChunkLoadError(new Error('chunk failed'), reload)).toBe(false); + expect(reload).not.toHaveBeenCalled(); + getItem.mockRestore(); + }); + + it('reloads for an explicit dynamic-import failure', async () => { + const reload = vi.fn(); + const error = new TypeError('Failed to fetch dynamically imported module: /assets/Session-old.js'); + + await expect(recoverLazyLoad(Promise.reject(error), reload)).rejects.toBe(error); + + expect(reload).toHaveBeenCalledTimes(1); + }); + + it('does not reload for a non-chunk lazy dependency failure', async () => { + const reload = vi.fn(); + const error = new Error('i18n instance is unavailable'); + + await expect(recoverLazyLoad(Promise.reject(error), reload)).rejects.toBe(error); + + expect(reload).not.toHaveBeenCalled(); + }); +}); diff --git a/webui/src/utils/chunkLoadRecovery.ts b/webui/src/utils/chunkLoadRecovery.ts new file mode 100644 index 000000000..240b3541e --- /dev/null +++ b/webui/src/utils/chunkLoadRecovery.ts @@ -0,0 +1,78 @@ +const CHUNK_RELOAD_STORAGE_KEY = 'flocks:chunk-load-reload'; + +type ReloadPage = () => void; + +const CHUNK_LOAD_ERROR_PATTERNS = [ + /failed to fetch dynamically imported module/i, + /error loading dynamically imported module/i, + /importing a module script failed/i, + /loading (?:css )?chunk .+ failed/i, + /unable to preload css/i, +]; + +export function isChunkLoadError(error: unknown): boolean { + const candidate = error && typeof error === 'object' + ? error as { name?: unknown; message?: unknown } + : null; + const name = candidate && typeof candidate.name === 'string' ? candidate.name : ''; + const message = candidate && typeof candidate.message === 'string' + ? candidate.message + : typeof error === 'string' ? error : ''; + return name === 'ChunkLoadError' || CHUNK_LOAD_ERROR_PATTERNS.some((pattern) => pattern.test(message)); +} + +function errorFingerprint(error: unknown): string { + if (error instanceof Error) { + return `${error.name}:${error.message}`.slice(0, 1000); + } + if (typeof error === 'string') { + return error.slice(0, 1000); + } + return 'unknown-chunk-load-error'; +} + +export function reloadOnceForChunkLoadError( + error: unknown, + reload: ReloadPage = () => window.location.reload(), +): boolean { + const fingerprint = errorFingerprint(error); + try { + if (window.sessionStorage.getItem(CHUNK_RELOAD_STORAGE_KEY) === fingerprint) { + return false; + } + window.sessionStorage.setItem(CHUNK_RELOAD_STORAGE_KEY, fingerprint); + } catch { + // Without session storage we cannot safely prevent a reload loop. + return false; + } + + reload(); + return true; +} + +export function recoverLazyLoad(promise: Promise, reload?: ReloadPage): Promise { + return promise.catch((error: unknown) => { + if (isChunkLoadError(error)) { + reloadOnceForChunkLoadError(error, reload); + } + throw error; + }); +} + +export function retryChunkLoad(): void { + window.location.reload(); +} + +let preloadRecoveryInstalled = false; + +export function installVitePreloadErrorRecovery(): void { + if (preloadRecoveryInstalled || typeof window === 'undefined') return; + preloadRecoveryInstalled = true; + + window.addEventListener('vite:preloadError', (event) => { + const preloadEvent = event as Event & { payload?: unknown }; + if (reloadOnceForChunkLoadError(preloadEvent.payload)) { + event.preventDefault(); + } + }); +} From 13f89b908e32ca71c648eab2a91089450a7263ba Mon Sep 17 00:00:00 2001 From: duguwanglong Date: Fri, 10 Jul 2026 12:05:00 +0800 Subject: [PATCH 13/49] fix(webui): smooth streamed response rendering --- flocks/server/routes/event.py | 47 +++-- tests/server/routes/test_event_routes.py | 167 ++++++++-------- .../src/components/common/SessionChat.test.ts | 149 +++++++++++++++ webui/src/components/common/SessionChat.tsx | 35 +++- .../common/StreamingMarkdown.test.tsx | 180 ++++++++++++++++-- .../components/common/StreamingMarkdown.tsx | 179 +++++++++++++++-- webui/src/hooks/useSessions.test.ts | 46 ++++- webui/src/hooks/useSessions.ts | 68 +------ 8 files changed, 658 insertions(+), 213 deletions(-) diff --git a/flocks/server/routes/event.py b/flocks/server/routes/event.py index 5c55b0d7c..9f427cb67 100644 --- a/flocks/server/routes/event.py +++ b/flocks/server/routes/event.py @@ -44,6 +44,10 @@ EVENT_QUEUE_MAXSIZE - 1, int(os.getenv("FLOCKS_EVENT_QUEUE_DROP_TO", str(EVENT_QUEUE_MAXSIZE // 2))), )) +EVENT_QUEUE_COALESCE_AT = max(1, min( + EVENT_QUEUE_MAXSIZE, + int(os.getenv("FLOCKS_EVENT_QUEUE_COALESCE_AT", str(min(EVENT_QUEUE_MAXSIZE, 64)))), +)) # Current directory context for SSE events @@ -157,19 +161,21 @@ def __init__(self, maxsize: int = 0) -> None: def _pending_events(self) -> deque[dict]: return cast(deque[dict], self.__dict__["_queue"]) - def coalesce_snapshot(self, event: dict) -> bool: + def coalesce_tail_snapshot(self, event: dict) -> bool: + """Merge only with the immediately preceding snapshot. + + Control events form an ordering barrier: snapshots on opposite sides of + one must remain separate even while a client is backpressured. + """ key = _snapshot_key(event) pending = self._pending_events() if key is None or not pending: return False - for index in range(len(pending) - 1, -1, -1): - queued = pending[index] - if _snapshot_key(queued) != key: - continue - del pending[index] - pending.append(_merge_snapshots(queued, event)) - return True - return False + queued = pending[-1] + if _snapshot_key(queued) != key: + return False + pending[-1] = _merge_snapshots(queued, event) + return True def drop_snapshots_until(self, target_size: int) -> int: pending = self._pending_events() @@ -228,7 +234,12 @@ class EventBroadcaster: _instance: Optional["EventBroadcaster"] = None - def __init__(self, queue_maxsize: int = EVENT_QUEUE_MAXSIZE, queue_drop_to: Optional[int] = None) -> None: + def __init__( + self, + queue_maxsize: int = EVENT_QUEUE_MAXSIZE, + queue_drop_to: Optional[int] = None, + queue_coalesce_at: Optional[int] = None, + ) -> None: self._clients: dict[EventQueue, Optional["AuthUser"]] = {} self._lock = asyncio.Lock() self._queue_maxsize = max(1, queue_maxsize) @@ -239,6 +250,13 @@ def __init__(self, queue_maxsize: int = EVENT_QUEUE_MAXSIZE, queue_drop_to: Opti EVENT_QUEUE_DROP_TO if queue_drop_to is None else queue_drop_to, ), ) + self._queue_coalesce_at = max( + 1, + min( + self._queue_maxsize, + EVENT_QUEUE_COALESCE_AT if queue_coalesce_at is None else queue_coalesce_at, + ), + ) @classmethod def get(cls) -> "EventBroadcaster": @@ -292,7 +310,14 @@ async def publish(self, event: dict) -> None: self._publish_to_queue(queue, event) def _publish_to_queue(self, queue: EventQueue, event: dict) -> None: - if queue.coalesce_snapshot(event): + # Preserve the event cadence for healthy clients. Once a subscriber is + # materially behind, compact only adjacent snapshots for the same part; + # this bounds accumulated full-text payloads without crossing control + # events or changing their relative order. + if ( + queue.qsize() >= self._queue_coalesce_at + and queue.coalesce_tail_snapshot(event) + ): return try: diff --git a/tests/server/routes/test_event_routes.py b/tests/server/routes/test_event_routes.py index bbffcfbc9..35da794a2 100644 --- a/tests/server/routes/test_event_routes.py +++ b/tests/server/routes/test_event_routes.py @@ -10,6 +10,22 @@ from flocks.session.session import SessionInfo +def _snapshot(text: str, delta: str, part_type: str = "text") -> dict: + return { + "type": "message.part.updated", + "properties": { + "part": { + "id": "part-1", + "messageID": "message-1", + "sessionID": "session-1", + "type": part_type, + "text": text, + }, + "delta": delta, + }, + } + + @pytest.mark.asyncio async def test_event_broadcaster_compacts_overflowing_client_queue(): broadcaster = EventBroadcaster(queue_maxsize=3, queue_drop_to=1) @@ -68,115 +84,104 @@ async def test_event_broadcaster_keeps_latest_event_when_drop_target_is_max_minu @pytest.mark.asyncio -async def test_event_broadcaster_coalesces_accumulated_text_and_preserves_delta(): - broadcaster = EventBroadcaster(queue_maxsize=3) +async def test_event_broadcaster_preserves_low_backlog_snapshot_cadence(): + broadcaster = EventBroadcaster(queue_maxsize=10, queue_coalesce_at=3) queue = await broadcaster.subscribe() - first = { - "type": "message.part.updated", - "properties": { - "part": { - "id": "part-1", - "messageID": "message-1", - "sessionID": "session-1", - "type": "text", - "text": "hello", - }, - "delta": "hello", - }, - } - latest = { - "type": "message.part.updated", - "properties": { - "part": { - "id": "part-1", - "messageID": "message-1", - "sessionID": "session-1", - "type": "text", - "text": "hello world", - }, - "delta": " world", - }, - } + first = _snapshot("hello", "hello") + latest = _snapshot("hello world", " world") broadcaster._publish_to_queue(queue, first) broadcaster._publish_to_queue(queue, latest) - assert queue.qsize() == 1 - event = queue.get_nowait() - assert event["properties"]["part"]["text"] == "hello world" - assert event["properties"]["delta"] == "hello world" + assert queue.qsize() == 2 + assert queue.get_nowait() == first + assert queue.get_nowait() == latest +@pytest.mark.parametrize("part_type", ["text", "reasoning"]) @pytest.mark.asyncio -async def test_event_broadcaster_coalesces_snapshots_across_interleaved_events(): - broadcaster = EventBroadcaster(queue_maxsize=4) +async def test_event_broadcaster_coalesces_adjacent_snapshots_at_high_water(part_type: str): + broadcaster = EventBroadcaster(queue_maxsize=10, queue_coalesce_at=3) queue = await broadcaster.subscribe() - first = { - "type": "message.part.updated", - "properties": { - "part": { - "id": "part-1", - "messageID": "message-1", - "sessionID": "session-1", - "type": "text", - "text": "hello", - }, - "delta": "hello", - }, - } + snapshots = [ + _snapshot("h", "h", part_type), + _snapshot("he", "e", part_type), + _snapshot("hel", "l", part_type), + _snapshot("hello", "lo", part_type), + ] + + for snapshot in snapshots: + broadcaster._publish_to_queue(queue, snapshot) + + assert queue.qsize() == 3 + assert queue.get_nowait() == snapshots[0] + assert queue.get_nowait() == snapshots[1] + merged = queue.get_nowait() + assert merged["properties"]["part"]["text"] == "hello" + assert merged["properties"]["delta"] == "llo" + + +@pytest.mark.asyncio +async def test_event_broadcaster_does_not_coalesce_across_control_events(): + broadcaster = EventBroadcaster(queue_maxsize=10, queue_coalesce_at=2) + queue = await broadcaster.subscribe() + first = _snapshot("hello", "hello") control = {"type": "question.asked", "properties": {"requestID": "question-1"}} - latest = { - "type": "message.part.updated", + latest = _snapshot("hello world", " world") + + broadcaster._publish_to_queue(queue, first) + broadcaster._publish_to_queue(queue, control) + broadcaster._publish_to_queue(queue, latest) + + assert queue.qsize() == 3 + assert queue.get_nowait() == first + assert queue.get_nowait() == control + assert queue.get_nowait() == latest + + +@pytest.mark.asyncio +async def test_message_finish_is_an_ordering_barrier_at_high_water(): + broadcaster = EventBroadcaster(queue_maxsize=10, queue_coalesce_at=2) + queue = await broadcaster.subscribe() + first = _snapshot("hello", "hello") + finish = { + "type": "message.updated", "properties": { - "part": { - "id": "part-1", - "messageID": "message-1", + "info": { + "id": "message-1", "sessionID": "session-1", - "type": "text", - "text": "hello world", + "finish": "stop", }, - "delta": " world", }, } + later = _snapshot("hello world", " world") broadcaster._publish_to_queue(queue, first) - broadcaster._publish_to_queue(queue, control) - broadcaster._publish_to_queue(queue, latest) + broadcaster._publish_to_queue(queue, finish) + broadcaster._publish_to_queue(queue, later) - assert queue.qsize() == 2 - assert queue.get_nowait() == control - snapshot = queue.get_nowait() - assert snapshot["properties"]["part"]["text"] == "hello world" - assert snapshot["properties"]["delta"] == "hello world" + assert queue.qsize() == 3 + assert queue.get_nowait() == first + assert queue.get_nowait() == finish + assert queue.get_nowait() == later @pytest.mark.asyncio -async def test_accumulated_text_queue_memory_tracks_latest_snapshot_not_full_history(): - broadcaster = EventBroadcaster(queue_maxsize=1000) +async def test_high_backlog_snapshot_memory_remains_bounded(): + broadcaster = EventBroadcaster(queue_maxsize=1000, queue_coalesce_at=4) queue = await broadcaster.subscribe() text = "" for _ in range(1000): delta = "x" * 100 text += delta - broadcaster._publish_to_queue(queue, { - "type": "message.part.updated", - "properties": { - "part": { - "id": "part-1", - "messageID": "message-1", - "sessionID": "session-1", - "type": "text", - "text": text, - }, - "delta": delta, - }, - }) + broadcaster._publish_to_queue(queue, _snapshot(text, delta)) - assert queue.qsize() == 1 - payload = queue.get_nowait() - assert payload["properties"]["delta"] == text - assert len(json.dumps(payload)) < 250_000 + assert queue.qsize() == 4 + payloads = [queue.get_nowait() for _ in range(queue.qsize())] + assert "".join(payload["properties"]["delta"] for payload in payloads) == text + assert payloads[-1]["properties"]["part"]["text"] == text + assert len(json.dumps(payloads)) < 250_000 @pytest.mark.asyncio diff --git a/webui/src/components/common/SessionChat.test.ts b/webui/src/components/common/SessionChat.test.ts index 111a0ea24..0551d8ab4 100644 --- a/webui/src/components/common/SessionChat.test.ts +++ b/webui/src/components/common/SessionChat.test.ts @@ -1003,6 +1003,155 @@ describe('getRenderableThinkingText', () => { }); }); +describe('ChatMessageBubble reasoning streaming', () => { + it.each(['reasoning', 'thinking'] as const)( + 'paces an active %s part after a tool and flushes the completed text', + (partType) => { + type RafCallback = (time: number) => void; + const callbacks = new Map(); + let nextRafId = 0; + vi.stubGlobal('requestAnimationFrame', (callback: RafCallback) => { + const id = ++nextRafId; + callbacks.set(id, callback); + return id; + }); + vi.stubGlobal('cancelAnimationFrame', (id: number) => { + callbacks.delete(id); + }); + + const makeReasoningMessage = (text: string, finish?: Message['finish']) => makeMessage({ + id: 'assistant-reasoning-stream', + role: 'assistant', + finish, + parts: [ + { + id: 'tool-before-reasoning', + messageID: 'assistant-reasoning-stream', + sessionID: 'sess-1', + type: 'tool', + tool: 'read', + state: { status: 'completed', output: 'done' }, + } as any, + { + id: 'reasoning-stream', + messageID: 'assistant-reasoning-stream', + sessionID: 'sess-1', + type: partType, + text, + } as any, + ], + }); + + let unmount = () => {}; + try { + const rendered = render(React.createElement(ChatMessageBubble, { + message: makeReasoningMessage('思'), + isActive: true, + })); + unmount = rendered.unmount; + + rendered.rerender(React.createElement(ChatMessageBubble, { + message: makeReasoningMessage('思考过程'), + isActive: true, + })); + + expect(screen.getByText('思考中...')).toBeInTheDocument(); + expect(screen.getByText('思')).toBeInTheDocument(); + expect(screen.queryByText('思考过程')).not.toBeInTheDocument(); + + act(() => { + const pending = [...callbacks.values()]; + callbacks.clear(); + pending.forEach(callback => callback(1000 / 60)); + }); + expect(screen.getByText('思考')).toBeInTheDocument(); + + rendered.rerender(React.createElement(ChatMessageBubble, { + message: makeReasoningMessage('思考过程', 'stop'), + isActive: false, + })); + expect(screen.getByText('思考过程')).toBeInTheDocument(); + } finally { + unmount(); + vi.unstubAllGlobals(); + } + }, + ); + + it('does not animate reasoning while its process group is closed', () => { + let nextRafId = 0; + const requestAnimationFrameSpy = vi.fn(() => ++nextRafId); + vi.stubGlobal('requestAnimationFrame', requestAnimationFrameSpy); + vi.stubGlobal('cancelAnimationFrame', vi.fn()); + + const messageId = 'assistant-hidden-reasoning'; + const processGroupKey = `${messageId}:process:0`; + const makeHiddenReasoningMessage = (text: string) => makeMessage({ + id: messageId, + role: 'assistant', + parts: [ + { + id: 'tool-before-hidden-reasoning', + messageID: messageId, + sessionID: 'sess-1', + type: 'tool', + tool: 'read', + state: { status: 'completed', output: 'done' }, + } as any, + { + id: 'hidden-reasoning', + messageID: messageId, + sessionID: 'sess-1', + type: 'reasoning', + text, + } as any, + ], + }); + + let unmount = () => {}; + try { + const rendered = render(React.createElement(ChatMessageBubble, { + message: makeHiddenReasoningMessage('隐藏'), + isActive: true, + collapseIntermediateSteps: true, + processGroupsOpenWhileActive: true, + processGroupOpenState: { [processGroupKey]: false }, + })); + unmount = rendered.unmount; + + rendered.rerender(React.createElement(ChatMessageBubble, { + message: makeHiddenReasoningMessage('隐藏更新'), + isActive: true, + collapseIntermediateSteps: true, + processGroupsOpenWhileActive: true, + processGroupOpenState: { [processGroupKey]: false }, + })); + expect(requestAnimationFrameSpy).not.toHaveBeenCalled(); + + rendered.rerender(React.createElement(ChatMessageBubble, { + message: makeHiddenReasoningMessage('隐藏更新'), + isActive: true, + collapseIntermediateSteps: true, + processGroupsOpenWhileActive: true, + processGroupOpenState: { [processGroupKey]: true }, + })); + expect(requestAnimationFrameSpy).not.toHaveBeenCalled(); + + rendered.rerender(React.createElement(ChatMessageBubble, { + message: makeHiddenReasoningMessage('隐藏更新继续'), + isActive: true, + collapseIntermediateSteps: true, + processGroupsOpenWhileActive: true, + processGroupOpenState: { [processGroupKey]: true }, + })); + expect(requestAnimationFrameSpy).toHaveBeenCalledTimes(1); + } finally { + unmount(); + vi.unstubAllGlobals(); + } + }); +}); + describe('getMessageErrorText', () => { it('prefers user-facing display messages over raw provider errors', () => { expect(getMessageErrorText(makeMessage({ diff --git a/webui/src/components/common/SessionChat.tsx b/webui/src/components/common/SessionChat.tsx index 8f30fe300..c52d08b8a 100644 --- a/webui/src/components/common/SessionChat.tsx +++ b/webui/src/components/common/SessionChat.tsx @@ -18,7 +18,7 @@ import { useState, useCallback, useRef, useEffect, useMemo, memo } from 'react'; import { Send, Loader2, ChevronDown, Square, Copy, User, FileText, AlertCircle, X, RefreshCw, Pencil, Save, ImageIcon, Paperclip, ArrowUp, Clock, CheckCircle2, XCircle, Brain, Trash2, Bot, Check, ListTree } from 'lucide-react'; -import { StreamingMarkdown } from './StreamingMarkdown'; +import { StreamingMarkdown, useStreamingContent } from './StreamingMarkdown'; import { useTranslation } from 'react-i18next'; import LoadingSpinner from './LoadingSpinner'; import { QuestionTool, type QuestionItem } from './QuestionTool'; @@ -238,6 +238,17 @@ export function getRenderableThinkingText(part: Pick{displayContent}; +}); + function stringifyToolPayload(value: unknown): string { if (value == null) return ''; if (typeof value === 'string') return value; @@ -3896,7 +3907,7 @@ function ChatMessageBubbleInner({ const { t } = useTranslation('session'); const isUser = message.role === 'user'; const parts: MessagePart[] = Array.isArray(message.parts) ? message.parts : []; - const { getPartExpanded, togglePart, isReasoningDone } = useReasoningToggle(parts, message.finish); + const { getPartExpanded, togglePart } = useReasoningToggle(parts, message.finish); // Lightbox state for inline image previews. Browsers block top-level // navigation to ``data:`` URLs (the format we send for chat images), so a // ``window.open`` would land on a blank page. We open an in-app overlay @@ -4026,7 +4037,10 @@ function ChatMessageBubbleInner({ if (part.type === 'file') return !!part.url; return false; }; - const renderPart = (part: MessagePart, i: number) => ( + const activeTailPart = isActive + ? [...displayParts].reverse().find(isRenderableDisplayPart) + : undefined; + const renderPart = (part: MessagePart, i: number, isVisible = true) => ( // Spacing between consecutive parts is owned by this wrapper, // not by individual part components. Each part used to set its // own `mt-2 first:mt-0`, but since every part lives in its own @@ -4087,8 +4101,8 @@ function ChatMessageBubbleInner({ const thinkingText = getRenderableThinkingText(part); if (!thinkingText) return null; const partKey = part.id || `reasoning-${i}`; - const isExpanded = getPartExpanded(partKey); - const isThinking = !isReasoningDone; + const isThinking = part === activeTailPart; + const isExpanded = isThinking || getPartExpanded(partKey); return ( // Vertical spacing is provided by the parent part wrapper // (see `otherParts.map` above); keep this container neutral @@ -4121,9 +4135,12 @@ function ChatMessageBubbleInner({ )}
- {isExpanded && ( + {isExpanded && isVisible && (
- {thinkingText} +
)}
@@ -4168,14 +4185,14 @@ function ChatMessageBubbleInner({ )} >
- {group.map(({ part, index }) => renderPart(part, index))} + {group.map(({ part, index }) => renderPart(part, index, effectiveProcessGroupOpen))}
); }; const renderDisplayParts = () => { if (!collapseIntermediateSteps || isUser) { - return displayParts.map(renderPart); + return displayParts.map((part, index) => renderPart(part, index)); } const nodes: React.ReactNode[] = []; let processGroup: Array<{ part: MessagePart; index: number }> = []; diff --git a/webui/src/components/common/StreamingMarkdown.test.tsx b/webui/src/components/common/StreamingMarkdown.test.tsx index 6b85a6931..27af58ed8 100644 --- a/webui/src/components/common/StreamingMarkdown.test.tsx +++ b/webui/src/components/common/StreamingMarkdown.test.tsx @@ -1,39 +1,50 @@ import { describe, expect, it, vi, beforeEach, afterEach } from 'vitest'; import { renderHook, act, render } from '@testing-library/react'; -import { StreamingMarkdown, useStreamingContent } from './StreamingMarkdown'; +import { + StreamingMarkdown, + fallbackSplitStreamingGraphemes, + splitStreamingGraphemes, + useStreamingContent, +} from './StreamingMarkdown'; // ─── rAF fake setup ────────────────────────────────────────────────────────── type RafCallback = (time: number) => void; -let rafQueue: RafCallback[] = []; +let rafQueue = new Map(); let rafIdCounter = 0; +let rafTime = 0; function setupFakeRaf() { vi.stubGlobal('requestAnimationFrame', (cb: RafCallback) => { - rafIdCounter++; - rafQueue.push(cb); - return rafIdCounter; + const id = ++rafIdCounter; + rafQueue.set(id, cb); + return id; }); vi.stubGlobal('cancelAnimationFrame', (id: number) => { - // Mark cancelled by removing; simplified — good enough for these tests - rafQueue = rafQueue.filter((_, i) => i !== id - 1); + rafQueue.delete(id); }); } -function flushRaf() { - const pending = [...rafQueue]; - rafQueue = []; - pending.forEach(cb => cb(performance.now())); +function flushRafAt(time: number) { + rafTime = time; + const pending = [...rafQueue.values()]; + rafQueue.clear(); + pending.forEach(cb => cb(time)); +} + +function flushRaf(stepMs = 1000 / 60) { + flushRafAt(rafTime + stepMs); } // ─── Tests ─────────────────────────────────────────────────────────────────── describe('useStreamingContent', () => { beforeEach(() => { - rafQueue = []; + rafQueue = new Map(); rafIdCounter = 0; + rafTime = 0; setupFakeRaf(); }); @@ -86,8 +97,9 @@ describe('useStreamingContent', () => { it('streaming: multiple content updates in same frame only trigger one rAF', () => { const rafSpy = vi.fn().mockImplementation((cb: RafCallback) => { - rafQueue.push(cb); - return ++rafIdCounter; + const id = ++rafIdCounter; + rafQueue.set(id, cb); + return id; }); vi.stubGlobal('requestAnimationFrame', rafSpy); @@ -105,7 +117,7 @@ describe('useStreamingContent', () => { }); it('streaming→done: cancels pending rAF and applies final content immediately', () => { - const cancelSpy = vi.fn(); + const cancelSpy = vi.fn((id: number) => rafQueue.delete(id)); vi.stubGlobal('cancelAnimationFrame', cancelSpy); const { result, rerender } = renderHook( @@ -121,15 +133,19 @@ describe('useStreamingContent', () => { expect(cancelSpy).toHaveBeenCalled(); expect(result.current).toBe('chunk1 chunk2 final'); + + act(() => { flushRaf(); }); + expect(result.current).toBe('chunk1 chunk2 final'); }); - it('streaming: drains queued deltas progressively instead of jumping to latest content', () => { + it('streaming: types a small English backlog one character per frame', () => { const { result, rerender } = renderHook( ({ content, isStreaming }) => useStreamingContent(content, isStreaming), { initialProps: { content: 'a', isStreaming: true } }, ); - // Multiple updates before the frame fires + // Multiple updates before the frame fires still retain a typewriter-sized + // first step rather than jumping to the latest accumulated snapshot. act(() => { rerender({ content: 'ab', isStreaming: true }); }); act(() => { rerender({ content: 'abc', isStreaming: true }); }); act(() => { rerender({ content: 'abcd', isStreaming: true }); }); @@ -145,7 +161,27 @@ describe('useStreamingContent', () => { expect(result.current).toBe('abcd'); }); - it('streaming: catches up large backlogs within a bounded number of frames', () => { + it('streaming: types a small Chinese backlog one character per frame', () => { + const { result, rerender } = renderHook( + ({ content, isStreaming }) => useStreamingContent(content, isStreaming), + { initialProps: { content: '你', isStreaming: true } }, + ); + + act(() => { + rerender({ content: '你好世界', isStreaming: true }); + }); + + act(() => { flushRaf(); }); + expect(result.current).toBe('你好'); + + act(() => { flushRaf(); }); + expect(result.current).toBe('你好世'); + + act(() => { flushRaf(); }); + expect(result.current).toBe('你好世界'); + }); + + it('streaming: starts a large backlog with one character, then accelerates in bounded steps', () => { const fullContent = `a${'b'.repeat(120)}`; const { result, rerender } = renderHook( ({ content, isStreaming }) => useStreamingContent(content, isStreaming), @@ -159,10 +195,16 @@ describe('useStreamingContent', () => { act(() => { flushRaf(); }); - expect(result.current.length).toBeGreaterThan('a'.length); - expect(result.current.length).toBeLessThan(fullContent.length); + expect(result.current).toBe('ab'); + + act(() => { + flushRaf(); + }); + const acceleratedLength = result.current.length; + expect(acceleratedLength).toBeGreaterThan(2); + expect(acceleratedLength - 2).toBeLessThanOrEqual(8); - for (let i = 0; i < 12; i += 1) { + for (let i = 0; i < 90; i += 1) { act(() => { flushRaf(); }); @@ -170,6 +212,102 @@ describe('useStreamingContent', () => { expect(result.current).toBe(fullContent); }); + + it('streaming: advances at nearly the same rate on 60 Hz and 120 Hz displays', () => { + const progressAfter = (frameMs: number) => { + rafQueue.clear(); + rafTime = 0; + const fullContent = `a${'b'.repeat(300)}`; + const hook = renderHook( + ({ content, isStreaming }) => useStreamingContent(content, isStreaming), + { initialProps: { content: 'a', isStreaming: true } }, + ); + + act(() => { hook.rerender({ content: fullContent, isStreaming: true }); }); + const frameCount = Math.round(500 / frameMs); + for (let frame = 1; frame <= frameCount; frame += 1) { + act(() => { flushRafAt(frame * frameMs); }); + } + const length = hook.result.current.length; + hook.unmount(); + return length; + }; + + const progress60Hz = progressAfter(1000 / 60); + const progress120Hz = progressAfter(1000 / 120); + expect(Math.abs(progress60Hz - progress120Hz)).toBeLessThanOrEqual(4); + }); + + it('streaming: clamps a long stalled frame instead of dumping the backlog', () => { + const fullContent = `a${'b'.repeat(300)}`; + const { result, rerender } = renderHook( + ({ content, isStreaming }) => useStreamingContent(content, isStreaming), + { initialProps: { content: 'a', isStreaming: true } }, + ); + + act(() => { rerender({ content: fullContent, isStreaming: true }); }); + act(() => { flushRafAt(1000 / 60); }); + expect(result.current).toBe('ab'); + + act(() => { flushRafAt(5000); }); + expect(result.current.length - 2).toBeGreaterThan(0); + expect(result.current.length - 2).toBeLessThanOrEqual(8); + + const afterStallLength = result.current.length; + act(() => { flushRaf(); }); + expect(result.current.length - afterStallLength).toBeLessThanOrEqual(7); + }); + + it('streaming: never splits a grapheme cluster across frames', () => { + const graphemes = ['A', '👍🏽', '👨‍👩‍👧‍👦', '🇨🇳', 'é']; + const fullContent = graphemes.join(''); + expect(splitStreamingGraphemes(fullContent)).toEqual(graphemes); + + const { result, rerender } = renderHook( + ({ content, isStreaming }) => useStreamingContent(content, isStreaming), + { initialProps: { content: graphemes[0], isStreaming: true } }, + ); + act(() => { rerender({ content: fullContent, isStreaming: true }); }); + + for (let index = 2; index <= graphemes.length; index += 1) { + act(() => { flushRaf(); }); + expect(result.current).toBe(graphemes.slice(0, index).join('')); + } + }); + + it('streaming: re-segments an unpainted grapheme split across SSE updates', () => { + const { result, rerender } = renderHook( + ({ content, isStreaming }) => useStreamingContent(content, isStreaming), + { initialProps: { content: 'A', isStreaming: true } }, + ); + + act(() => { rerender({ content: 'A👍', isStreaming: true }); }); + act(() => { rerender({ content: 'A👍🏽', isStreaming: true }); }); + act(() => { flushRaf(); }); + + expect(result.current).toBe('A👍🏽'); + }); + + it('fallback: keeps common compound graphemes intact without Intl.Segmenter', () => { + const graphemes = ['A', '👍🏽', '👨‍👩‍👧‍👦', '🇨🇳', 'é', '1️⃣']; + expect(fallbackSplitStreamingGraphemes(graphemes.join(''))).toEqual(graphemes); + }); + + it('streaming: the final delta and finish settle synchronously with no stale frame', () => { + const { result, rerender } = renderHook( + ({ content, isStreaming }) => useStreamingContent(content, isStreaming), + { initialProps: { content: 'almost', isStreaming: true } }, + ); + + act(() => { + rerender({ content: 'almost done', isStreaming: true }); + rerender({ content: 'almost done', isStreaming: false }); + }); + expect(result.current).toBe('almost done'); + + act(() => { flushRafAt(5000); }); + expect(result.current).toBe('almost done'); + }); }); describe('StreamingMarkdown', () => { diff --git a/webui/src/components/common/StreamingMarkdown.tsx b/webui/src/components/common/StreamingMarkdown.tsx index 5e1a1330a..8f6e4d83d 100644 --- a/webui/src/components/common/StreamingMarkdown.tsx +++ b/webui/src/components/common/StreamingMarkdown.tsx @@ -1,4 +1,4 @@ -import { useState, useEffect, useRef, useCallback } from 'react'; +import { memo, useState, useEffect, useRef, useCallback } from 'react'; import ReactMarkdown from 'react-markdown'; import rehypeHighlight from 'rehype-highlight'; import rehypeRaw from 'rehype-raw'; @@ -12,6 +12,117 @@ const sanitizeSchema = { strip: [...(defaultSchema.strip || []), 'style'], }; +const BASE_STREAMING_GRAPHEMES_PER_SECOND = 60; +const BACKLOG_RATE_BOOST = 4; +const MAX_STREAMING_GRAPHEMES_PER_SECOND = 360; +const MAX_STREAMING_GRAPHEMES_PER_FRAME = 8; +const MAX_DRAIN_ELAPSED_MS = 50; + +interface SegmentData { + segment: string; +} + +interface GraphemeSegmenter { + segment(input: string): Iterable; +} + +type GraphemeSegmenterConstructor = new ( + locales?: string | string[], + options?: { granularity: 'grapheme' }, +) => GraphemeSegmenter; + +const Segmenter = (Intl as typeof Intl & { Segmenter?: GraphemeSegmenterConstructor }).Segmenter; +const graphemeSegmenter = Segmenter + ? new Segmenter(undefined, { granularity: 'grapheme' }) + : null; + +const MARK_PATTERN = /\p{Mark}/u; + +/** Narrow fallback for engines that predate Intl.Segmenter. */ +export function fallbackSplitStreamingGraphemes(value: string): string[] { + const clusters: string[] = []; + let cluster = ''; + let regionalIndicatorCount = 0; + + for (const codePoint of Array.from(value)) { + const code = codePoint.codePointAt(0) ?? 0; + const isRegionalIndicator = code >= 0x1f1e6 && code <= 0x1f1ff; + const isEmojiModifier = code >= 0x1f3fb && code <= 0x1f3ff; + const isVariationSelector = ( + (code >= 0xfe00 && code <= 0xfe0f) + || (code >= 0xe0100 && code <= 0xe01ef) + ); + const isEmojiTag = code >= 0xe0020 && code <= 0xe007f; + const isJoiner = code === 0x200d; + const joinsPrevious = ( + MARK_PATTERN.test(codePoint) + || isEmojiModifier + || isVariationSelector + || isEmojiTag + || isJoiner + || cluster.endsWith('\u200d') + || (isRegionalIndicator && regionalIndicatorCount === 1) + ); + + if (!cluster || joinsPrevious) { + cluster += codePoint; + } else { + clusters.push(cluster); + cluster = codePoint; + } + + if (isRegionalIndicator) { + regionalIndicatorCount = joinsPrevious ? regionalIndicatorCount + 1 : 1; + } else if (!isVariationSelector && !isEmojiModifier && !MARK_PATTERN.test(codePoint)) { + regionalIndicatorCount = 0; + } + } + + if (cluster) clusters.push(cluster); + return clusters; +} + +export function splitStreamingGraphemes(value: string): string[] { + if (!graphemeSegmenter) return fallbackSplitStreamingGraphemes(value); + return Array.from(graphemeSegmenter.segment(value), ({ segment }) => segment); +} + +interface DrainBudget { + count: number; + credit: number; +} + +/** + * Convert elapsed time into a bounded render budget. A small backlog keeps a + * natural typewriter cadence; a growing backlog raises the target rate so the + * UI catches up without dumping an entire model chunk in one paint. + */ +export function getStreamingDrainBudget( + backlogLength: number, + elapsedMs: number, + credit: number, + firstFrame: boolean, +): DrainBudget { + if (backlogLength <= 0) return { count: 0, credit: 0 }; + if (firstFrame) return { count: 1, credit: 0 }; + + const boundedElapsedMs = Math.max(0, Math.min(MAX_DRAIN_ELAPSED_MS, elapsedMs)); + const targetRate = Math.min( + MAX_STREAMING_GRAPHEMES_PER_SECOND, + BASE_STREAMING_GRAPHEMES_PER_SECOND + + Math.max(0, backlogLength - 1) * BACKLOG_RATE_BOOST, + ); + const availableCredit = credit + targetRate * boundedElapsedMs / 1000; + const count = Math.min( + backlogLength, + MAX_STREAMING_GRAPHEMES_PER_FRAME, + Math.floor(availableCredit), + ); + // Preserve only fractional credit. Integer work rejected by the per-frame + // cap must not become a multi-frame burst after a long main-thread stall. + return { count, credit: availableCredit - Math.floor(availableCredit) }; +} + /** * Smooths streamed content by queueing appended text and draining it across * animation frames. The previous implementation collapsed all updates that @@ -23,31 +134,46 @@ export function useStreamingContent(content: string, isStreaming: boolean): stri const pendingRafRef = useRef(null); const incomingContentRef = useRef(content); const displayedContentRef = useRef(content); - const queuedCharsRef = useRef([]); + const queuedGraphemesRef = useRef([]); const isStreamingRef = useRef(isStreaming); + const lastDrainTimeRef = useRef(null); + const drainCreditRef = useRef(0); const scheduleDrain = useCallback((drainQueue: (time: number) => void) => { if (pendingRafRef.current !== null) return; pendingRafRef.current = requestAnimationFrame(drainQueue); }, []); - const drainQueue = useCallback(() => { + const drainQueue = useCallback((time: number) => { pendingRafRef.current = null; - if (queuedCharsRef.current.length === 0) { + if (queuedGraphemesRef.current.length === 0) { + lastDrainTimeRef.current = null; + drainCreditRef.current = 0; return; } - // Drain progressively so a stalled frame does not dump the whole backlog - // in a single repaint. Larger backlogs should still catch up within a - // handful of frames instead of lagging behind for visibly too long. - const charsToRenderCount = Math.max(1, Math.ceil(queuedCharsRef.current.length / 3)); - const nextChunk = queuedCharsRef.current.splice(0, charsToRenderCount).join(''); - displayedContentRef.current += nextChunk; - setDisplayContent(displayedContentRef.current); + const previousDrainTime = lastDrainTimeRef.current; + const budget = getStreamingDrainBudget( + queuedGraphemesRef.current.length, + previousDrainTime === null ? 0 : time - previousDrainTime, + drainCreditRef.current, + previousDrainTime === null, + ); + lastDrainTimeRef.current = time; + drainCreditRef.current = budget.credit; - if (queuedCharsRef.current.length > 0 && isStreamingRef.current) { + if (budget.count > 0) { + const nextChunk = queuedGraphemesRef.current.splice(0, budget.count).join(''); + displayedContentRef.current += nextChunk; + setDisplayContent(displayedContentRef.current); + } + + if (queuedGraphemesRef.current.length > 0 && isStreamingRef.current) { scheduleDrain(drainQueue); + } else { + lastDrainTimeRef.current = null; + drainCreditRef.current = 0; } }, [scheduleDrain]); @@ -60,7 +186,9 @@ export function useStreamingContent(content: string, isStreaming: boolean): stri cancelAnimationFrame(pendingRafRef.current); pendingRafRef.current = null; } - queuedCharsRef.current = []; + queuedGraphemesRef.current = []; + lastDrainTimeRef.current = null; + drainCreditRef.current = 0; incomingContentRef.current = content; displayedContentRef.current = content; setDisplayContent(content); @@ -72,7 +200,9 @@ export function useStreamingContent(content: string, isStreaming: boolean): stri if (!content.startsWith(previousIncoming)) { // Content replaced or rewound: reset immediately to preserve correctness. - queuedCharsRef.current = []; + queuedGraphemesRef.current = []; + lastDrainTimeRef.current = null; + drainCreditRef.current = 0; displayedContentRef.current = content; setDisplayContent(content); return; @@ -81,7 +211,11 @@ export function useStreamingContent(content: string, isStreaming: boolean): stri const delta = content.slice(previousIncoming.length); if (!delta) return; - queuedCharsRef.current.push(...Array.from(delta)); + // The last queued grapheme may be completed by the next SSE delta (for + // example 👍 + 🏽 or an emoji ZWJ sequence). Re-segment the unpainted tail + // together with the new text before it reaches the screen. + const queuedTail = queuedGraphemesRef.current.pop() ?? ''; + queuedGraphemesRef.current.push(...splitStreamingGraphemes(queuedTail + delta)); scheduleDrain(drainQueue); }, [content, isStreaming, drainQueue, scheduleDrain]); @@ -95,7 +229,9 @@ export function useStreamingContent(content: string, isStreaming: boolean): stri [], ); - return displayContent; + // Completion must win in the same paint as the finish event. The effect + // above still clears queued work, but rendering does not wait for it. + return isStreaming ? displayContent : content; } export interface StreamingMarkdownProps { @@ -110,9 +246,7 @@ export interface StreamingMarkdownProps { * Content updates are throttled via requestAnimationFrame while streaming, * limiting ReactMarkdown re-parses to ~60fps instead of every SSE chunk. */ -export function StreamingMarkdown({ content, isStreaming }: StreamingMarkdownProps) { - const displayContent = useStreamingContent(content, isStreaming); - +const MarkdownContent = memo(function MarkdownContent({ content }: { content: string }) { return (
- {displayContent} + {content}
); +}); + +export function StreamingMarkdown({ content, isStreaming }: StreamingMarkdownProps) { + const displayContent = useStreamingContent(content, isStreaming); + return ; } diff --git a/webui/src/hooks/useSessions.test.ts b/webui/src/hooks/useSessions.test.ts index 4dbc678d5..b6b381ee5 100644 --- a/webui/src/hooks/useSessions.test.ts +++ b/webui/src/hooks/useSessions.test.ts @@ -214,7 +214,6 @@ describe('updateMessagePart scheduling', () => { }); it('second call with same part ID accumulates delta content correctly', async () => { - vi.useFakeTimers(); const { result } = renderHook(() => useSessionMessages('sess-1')); // Wait for initial fetch to settle await act(async () => {}); @@ -230,7 +229,6 @@ describe('updateMessagePart scheduling', () => { // Second call — content delta on the same part await act(async () => { result.current.updateMessagePart(delta, ' world'); - await vi.advanceTimersByTimeAsync(16); }); const msgs = result.current.messages; @@ -239,8 +237,7 @@ describe('updateMessagePart scheduling', () => { expect((msg!.parts as any[])[0].text).toBe('hello world'); }); - it('coalesces repeated known part updates into the next animation frame', async () => { - vi.useFakeTimers(); + it('applies every known part update without waiting for an animation frame', async () => { const { result } = renderHook(() => useSessionMessages('sess-1')); await act(async () => {}); @@ -253,20 +250,51 @@ describe('updateMessagePart scheduling', () => { await act(async () => { result.current.updateMessagePart({ ...part, text: 'he' }, 'e'); + }); + expect((result.current.messages[0].parts as any[])[0].text).toBe('he'); + + await act(async () => { result.current.updateMessagePart({ ...part, text: 'hel' }, 'l'); + }); + expect((result.current.messages[0].parts as any[])[0].text).toBe('hel'); + + await act(async () => { result.current.updateMessagePart({ ...part, text: 'hello' }, 'lo'); }); + expect((result.current.messages[0].parts as any[])[0].text).toBe('hello'); + }); - expect((result.current.messages[0].parts as any[])[0].text).toBe('h'); + it('commits the final delta before an immediately following finish update', async () => { + const { result } = renderHook(() => useSessionMessages('sess-1')); + await act(async () => {}); + const part = { + id: 'part-final', + messageID: 'msg-final', + sessionID: 'sess-1', + type: 'text', + text: 'almost', + }; await act(async () => { - await vi.advanceTimersByTimeAsync(16); + result.current.updateMessagePart(part); }); - expect((result.current.messages[0].parts as any[])[0].text).toBe('hello'); + await act(async () => { + result.current.updateMessagePart({ ...part, text: 'almost done' }, ' done'); + result.current.updateMessage({ + id: 'msg-final', + sessionID: 'sess-1', + role: 'assistant', + finish: 'stop', + }); + }); + + const message = result.current.messages.find((item) => item.id === 'msg-final'); + expect((message?.parts as any[])[0].text).toBe('almost done'); + expect(message?.finish).toBe('stop'); }); - it('resets known part tracking when session changes', async () => { + it('resets streamed messages when session changes', async () => { const { result, rerender } = renderHook( ({ id }: { id?: string }) => useSessionMessages(id), { initialProps: { id: 'sess-a' } }, @@ -280,7 +308,7 @@ describe('updateMessagePart scheduling', () => { result.current.updateMessagePart(part); }); - // Switch to a different session — messages and knownPartIds should reset + // Switching sessions must clear streamed state before the next paint. await act(async () => { rerender({ id: 'sess-b' }); }); diff --git a/webui/src/hooks/useSessions.ts b/webui/src/hooks/useSessions.ts index 55c94ade5..6064b1771 100644 --- a/webui/src/hooks/useSessions.ts +++ b/webui/src/hooks/useSessions.ts @@ -1,4 +1,4 @@ -import { useState, useEffect, useLayoutEffect, useCallback, useRef, startTransition } from 'react'; +import { useState, useEffect, useLayoutEffect, useCallback, useRef } from 'react'; import { sessionApi } from '@/api/session'; import client from '@/api/client'; import type { Session, Message } from '@/types'; @@ -8,11 +8,6 @@ const ABORTED_TOOL_ERROR = 'Tool execution was interrupted'; const SESSION_LIST_PAGE_SIZE = 100; const MESSAGE_PAGE_SIZE = 50; -interface PendingPartUpdate { - partInfo: any; - delta?: string; -} - function finalizeStoppedMessageParts(parts: Message['parts'], stoppedAt = Date.now()): Message['parts'] { return parts.map((part) => { if ( @@ -355,11 +350,6 @@ export function useSessionMessages(sessionId?: string) { const [hasMore, setHasMore] = useState(false); const [nextBefore, setNextBefore] = useState(null); const [error, setError] = useState(null); - // Tracks part IDs seen in this session to distinguish first-time creation - // (structural change → immediate update) from content deltas (low-priority). - const knownPartIdsRef = useRef>(new Set()); - const pendingKnownPartUpdatesRef = useRef>(new Map()); - const pendingKnownPartFrameRef = useRef(null); const fetchMessages = useCallback(async () => { if (!sessionId) return; @@ -412,45 +402,19 @@ export function useSessionMessages(sessionId?: string) { } }, [hasMore, loadingOlder, nextBefore, sessionId]); - const flushPendingKnownPartUpdates = useCallback(() => { - pendingKnownPartFrameRef.current = null; - const updates = Array.from(pendingKnownPartUpdatesRef.current.values()); - pendingKnownPartUpdatesRef.current.clear(); - if (updates.length === 0) return; - - startTransition(() => { - setMessages(prev => updates.reduce( - (next, update) => applyMessagePartUpdate(next, update.partInfo, update.delta), - prev, - )); - }); - }, []); - - const cancelPendingKnownPartUpdates = useCallback(() => { - if (pendingKnownPartFrameRef.current !== null) { - cancelAnimationFrame(pendingKnownPartFrameRef.current); - pendingKnownPartFrameRef.current = null; - } - pendingKnownPartUpdatesRef.current.clear(); - }, []); - // Reset state synchronously before paint when session changes // to prevent flash of welcome screen (useEffect runs AFTER paint) useLayoutEffect(() => { - cancelPendingKnownPartUpdates(); setMessages([]); setError(null); setHasMore(false); setNextBefore(null); - knownPartIdsRef.current.clear(); if (sessionId) { setLoading(true); } else { setLoading(false); } - }, [cancelPendingKnownPartUpdates, sessionId]); - - useEffect(() => cancelPendingKnownPartUpdates, [cancelPendingKnownPartUpdates]); + }, [sessionId]); useEffect(() => { fetchMessages(); @@ -488,14 +452,6 @@ export function useSessionMessages(sessionId?: string) { providerID: messageInfo.providerID ?? existing.providerID, cost: messageInfo.cost ?? existing.cost, }; - // When a message finishes streaming, evict its part IDs from the - // known-parts registry to reclaim memory. - if (messageInfo.finish) { - const parts = updated[existingIndex].parts as any[] | undefined; - parts?.forEach((p: any) => { - if (p?.id) knownPartIdsRef.current.delete(p.id); - }); - } return updated; } @@ -553,24 +509,12 @@ export function useSessionMessages(sessionId?: string) { * @param partInfo - Part object containing id, messageID, sessionID, type, text, etc. * @param delta - Optional text delta for this update. * - * New parts are structural changes and update synchronously so thinking or - * streaming indicators appear immediately. Known parts are folded into the - * next animation frame so high-frequency text deltas trigger fewer renders. + * Every SSE update enters message state immediately so a following finish + * event cannot overtake the final delta. Display-layer smoothing owns the + * frame-level typing cadence and Markdown parse budget. */ updateMessagePart: (partInfo: any, delta?: string) => { - const partId = partInfo?.id; - const isNewPart = !partId || !knownPartIdsRef.current.has(partId); - if (isNewPart) { - // Structural change: first appearance of this part — must render immediately - // so that "thinking" / "streaming" indicators show without delay. - if (partId) knownPartIdsRef.current.add(partId); - setMessages(prev => applyMessagePartUpdate(prev, partInfo, delta)); - } else { - pendingKnownPartUpdatesRef.current.set(partId, { partInfo, delta }); - if (pendingKnownPartFrameRef.current === null) { - pendingKnownPartFrameRef.current = requestAnimationFrame(flushPendingKnownPartUpdates); - } - } + setMessages(prev => applyMessagePartUpdate(prev, partInfo, delta)); }, replaceMessageText: (messageId: string, partId: string, text: string) => { setMessages(prev => prev.map((message) => { From 7e80bb635a6ddbe0f77a69cfc18de041b7e84095 Mon Sep 17 00:00:00 2001 From: John Yin <10972267+john-yin2333@user.noreply.gitee.com> Date: Fri, 10 Jul 2026 15:52:16 +0800 Subject: [PATCH 14/49] Add WhatsApp channel support --- flocks/channel/builtin/whatsapp/__init__.py | 6 + .../channel/builtin/whatsapp/bridge/bridge.js | 450 ++++ .../builtin/whatsapp/bridge/package-lock.json | 2184 +++++++++++++++++ .../builtin/whatsapp/bridge/package.json | 20 + .../builtin/whatsapp/bridge_runtime.py | 64 + flocks/channel/builtin/whatsapp/channel.py | 637 +++++ flocks/channel/builtin/whatsapp/config.py | 128 + flocks/channel/builtin/whatsapp/format.py | 23 + flocks/channel/builtin/whatsapp/inbound.py | 72 + flocks/channel/builtin/whatsapp/pairing.py | 184 ++ flocks/channel/registry.py | 2 + flocks/server/routes/channel.py | 63 + pyproject.toml | 5 + tests/channel/test_whatsapp.py | 252 ++ webui/src/locales/en-US/channel.json | 68 +- webui/src/locales/zh-CN/channel.json | 68 +- webui/src/pages/Channel/index.tsx | 414 +++- 17 files changed, 4637 insertions(+), 3 deletions(-) create mode 100644 flocks/channel/builtin/whatsapp/__init__.py create mode 100644 flocks/channel/builtin/whatsapp/bridge/bridge.js create mode 100644 flocks/channel/builtin/whatsapp/bridge/package-lock.json create mode 100644 flocks/channel/builtin/whatsapp/bridge/package.json create mode 100644 flocks/channel/builtin/whatsapp/bridge_runtime.py create mode 100644 flocks/channel/builtin/whatsapp/channel.py create mode 100644 flocks/channel/builtin/whatsapp/config.py create mode 100644 flocks/channel/builtin/whatsapp/format.py create mode 100644 flocks/channel/builtin/whatsapp/inbound.py create mode 100644 flocks/channel/builtin/whatsapp/pairing.py create mode 100644 tests/channel/test_whatsapp.py diff --git a/flocks/channel/builtin/whatsapp/__init__.py b/flocks/channel/builtin/whatsapp/__init__.py new file mode 100644 index 000000000..4b4679c2e --- /dev/null +++ b/flocks/channel/builtin/whatsapp/__init__.py @@ -0,0 +1,6 @@ +"""WhatsApp built-in channel.""" + +from .channel import WhatsAppChannel + +CHANNELS = [WhatsAppChannel()] + diff --git a/flocks/channel/builtin/whatsapp/bridge/bridge.js b/flocks/channel/builtin/whatsapp/bridge/bridge.js new file mode 100644 index 000000000..97281fc25 --- /dev/null +++ b/flocks/channel/builtin/whatsapp/bridge/bridge.js @@ -0,0 +1,450 @@ +#!/usr/bin/env node + +import { + DisconnectReason, + downloadMediaMessage, + fetchLatestBaileysVersion, + makeWASocket, + useMultiFileAuthState, +} from '@whiskeysockets/baileys'; +import { Boom } from '@hapi/boom'; +import express from 'express'; +import pino from 'pino'; +import path from 'path'; +import { createHash, randomBytes } from 'crypto'; +import { + existsSync, + mkdirSync, + readFileSync, + unlinkSync, + writeFileSync, +} from 'fs'; +import { execFileSync } from 'child_process'; +import { fileURLToPath } from 'url'; +import { tmpdir } from 'os'; + +const args = process.argv.slice(2); + +function getArg(name, defaultValue) { + const idx = args.indexOf(`--${name}`); + return idx >= 0 && args[idx + 1] ? args[idx + 1] : defaultValue; +} + +function envFlag(name, defaultValue = false) { + const raw = process.env[name]; + if (raw === undefined) return defaultValue; + return ['1', 'true', 'yes', 'on'].includes(String(raw).toLowerCase()); +} + +const PORT = Number.parseInt(getArg('port', '3100'), 10); +const SESSION_DIR = getArg('session', path.join(process.env.HOME || '.', '.flocks', 'workspace', 'channels', 'whatsapp', 'session')); +const MEDIA_DIR = process.env.FLOCKS_WHATSAPP_MEDIA_DIR + || path.join(process.env.HOME || '.', '.flocks', 'workspace', 'channels', 'whatsapp', 'media'); +const MODE = getArg('mode', process.env.FLOCKS_WHATSAPP_MODE || 'bot'); +const REPLY_PREFIX = process.env.FLOCKS_WHATSAPP_REPLY_PREFIX || ''; +const CHUNK_DELAY_MS = Number.parseInt(process.env.FLOCKS_WHATSAPP_CHUNK_DELAY_MS || '300', 10); +const SEND_TIMEOUT_MS = Number.parseInt(process.env.FLOCKS_WHATSAPP_SEND_TIMEOUT_MS || '60000', 10); +const BRIDGE_TOKEN = process.env.FLOCKS_WHATSAPP_BRIDGE_TOKEN || ''; +const CONFIG_HASH = process.env.FLOCKS_WHATSAPP_CONFIG_HASH || ''; +const ALLOWED_MEDIA_ROOTS = String(process.env.FLOCKS_WHATSAPP_ALLOWED_MEDIA_ROOTS || '') + .split(path.delimiter) + .map(item => item.trim()) + .filter(Boolean) + .map(item => path.resolve(item)); +const PAIR_ONLY = args.includes('--pair-only'); +const PAIR_JSON = args.includes('--pair-json'); +const PAIR_TIMEOUT_MS = Number.parseInt(process.env.FLOCKS_WHATSAPP_PAIR_TIMEOUT_MS || '120000', 10); +const DEBUG = envFlag('FLOCKS_WHATSAPP_DEBUG'); + +mkdirSync(SESSION_DIR, { recursive: true }); +mkdirSync(MEDIA_DIR, { recursive: true }); + +let scriptHash = ''; +let tokenHash = ''; +try { + scriptHash = createHash('sha256') + .update(readFileSync(fileURLToPath(import.meta.url))) + .digest('hex') + .slice(0, 16); +} catch {} +if (BRIDGE_TOKEN) { + tokenHash = createHash('sha256').update(BRIDGE_TOKEN).digest('hex').slice(0, 12); +} + +let sock = null; +let connectionState = 'disconnected'; +let messageQueue = []; +const messageStore = new Map(); +const recentlySent = new Map(); +let sendQueue = Promise.resolve(); + +const logger = pino({ level: DEBUG ? 'debug' : 'silent' }); + +function emitPairEvent(event) { + if (!PAIR_JSON) return; + try { + console.log(JSON.stringify({ ts: Date.now(), ...event })); + } catch {} +} + +function rememberSent(id) { + if (!id) return; + recentlySent.set(id, Date.now()); + if (recentlySent.size > 1000) { + const cutoff = Date.now() - 10 * 60 * 1000; + for (const [key, ts] of recentlySent.entries()) { + if (ts < cutoff) recentlySent.delete(key); + } + } +} + +function normalizeJid(value) { + if (!value) return ''; + return String(value).replace(':', '@'); +} + +function splitLongMessage(message, limit = 4096) { + const text = String(message || ''); + if (!text) return []; + if (text.length <= limit) return [text]; + const chunks = []; + let remaining = text; + while (remaining.length > limit) { + let splitAt = remaining.lastIndexOf('\n', limit); + if (splitAt < Math.floor(limit / 2)) splitAt = remaining.lastIndexOf(' ', limit); + if (splitAt < 1) splitAt = limit; + chunks.push(remaining.slice(0, splitAt).trimEnd()); + remaining = remaining.slice(splitAt).trimStart(); + } + if (remaining) chunks.push(remaining); + return chunks; +} + +function enqueueSend(fn) { + const task = sendQueue.then(() => fn(), () => fn()); + sendQueue = task.catch(() => {}); + return task; +} + +function sleep(ms) { + return new Promise(resolve => setTimeout(resolve, ms)); +} + +function sendWithTimeout(chatId, payload, options = {}) { + return enqueueSend(() => { + let timer; + const timeout = new Promise((_, reject) => { + timer = setTimeout(() => reject(new Error(`sendMessage timed out after ${SEND_TIMEOUT_MS}ms`)), SEND_TIMEOUT_MS); + }); + return Promise.race([ + sock.sendMessage(chatId, payload, options), + timeout, + ]).finally(() => clearTimeout(timer)); + }); +} + +function getMessageContent(msg) { + const message = msg?.message || {}; + return message.ephemeralMessage?.message + || message.viewOnceMessage?.message + || message.viewOnceMessageV2?.message + || message; +} + +function extractText(content) { + return content.conversation + || content.extendedTextMessage?.text + || content.imageMessage?.caption + || content.videoMessage?.caption + || content.documentMessage?.caption + || ''; +} + +function detectMediaBlock(content) { + if (content.imageMessage) return { block: content.imageMessage, type: 'image', ext: '.jpg' }; + if (content.videoMessage) return { block: content.videoMessage, type: 'video', ext: '.mp4' }; + if (content.audioMessage) return { block: content.audioMessage, type: content.audioMessage.ptt ? 'ptt' : 'audio', ext: '.ogg' }; + if (content.documentMessage) return { block: content.documentMessage, type: 'document', ext: path.extname(content.documentMessage.fileName || '') || '.bin' }; + if (content.stickerMessage) return { block: content.stickerMessage, type: 'sticker', ext: '.webp' }; + return null; +} + +function safeExt(ext) { + const cleaned = String(ext || '').toLowerCase().replace(/[^a-z0-9.]/g, ''); + return cleaned && cleaned.startsWith('.') ? cleaned : '.bin'; +} + +async function cacheInboundMedia(msg, mediaInfo) { + try { + const buffer = await downloadMediaMessage( + msg, + 'buffer', + {}, + { logger, reuploadRequest: sock.updateMediaMessage }, + ); + const fileName = `${mediaInfo.type}_${randomBytes(8).toString('hex')}${safeExt(mediaInfo.ext)}`; + const filePath = path.join(MEDIA_DIR, fileName); + writeFileSync(filePath, buffer); + return filePath; + } catch (err) { + console.warn('[whatsapp-bridge] failed to cache media:', err.message); + return ''; + } +} + +function buildBridgeEvent({ msg, chatId, senderId, isGroup, body, mediaInfo, mediaPath, fromOwner }) { + const content = getMessageContent(msg); + const contextInfo = content.extendedTextMessage?.contextInfo + || content.imageMessage?.contextInfo + || content.videoMessage?.contextInfo + || content.documentMessage?.contextInfo + || {}; + const botIds = new Set([ + normalizeJid(sock?.user?.id), + normalizeJid(sock?.user?.lid), + ].filter(Boolean)); + const mentionedJids = (contextInfo.mentionedJid || []).map(normalizeJid); + const mentioned = mentionedJids.some(jid => botIds.has(jid)); + const quotedParticipant = normalizeJid(contextInfo.participant || ''); + const isReplyToBot = Boolean(quotedParticipant && botIds.has(quotedParticipant)); + return { + messageId: msg.key.id || '', + chatId, + senderId, + senderName: msg.pushName || '', + isGroup, + body, + hasMedia: Boolean(mediaPath), + mediaUrls: mediaPath ? [mediaPath] : [], + mediaType: mediaInfo?.type || '', + mime: mediaInfo?.block?.mimetype || '', + quotedMessageId: contextInfo.stanzaId || '', + quotedParticipant, + mentioned, + isReplyToBot, + fromOwner: Boolean(fromOwner), + }; +} + +async function startSocket() { + const { state, saveCreds } = await useMultiFileAuthState(SESSION_DIR); + const { version } = await fetchLatestBaileysVersion(); + sock = makeWASocket({ + version, + auth: state, + logger, + printQRInTerminal: false, + browser: ['Flocks', 'Chrome', '120.0'], + syncFullHistory: false, + markOnlineOnConnect: false, + getMessage: async (key) => messageStore.get(key?.id) || { conversation: '' }, + }); + + sock.ev.on('creds.update', saveCreds); + + sock.ev.on('connection.update', (update) => { + const { connection, lastDisconnect, qr } = update; + if (qr) { + emitPairEvent({ event: 'qr', qr }); + if (!PAIR_JSON) console.log(qr); + } + if (connection === 'close') { + const reason = new Boom(lastDisconnect?.error)?.output?.statusCode; + connectionState = 'disconnected'; + if (reason === DisconnectReason.loggedOut) { + emitPairEvent({ event: 'error', error: 'logged_out', reason }); + process.exit(1); + } + emitPairEvent({ event: 'disconnected', reason }); + setTimeout(startSocket, reason === 515 ? 1000 : 3000); + } else if (connection === 'open') { + connectionState = 'connected'; + emitPairEvent({ + event: 'connected', + user: sock?.user ? { id: sock.user.id || null, name: sock.user.name || sock.user.verifiedName || null } : null, + }); + if (PAIR_ONLY) setTimeout(() => process.exit(0), 1500); + } + }); + + sock.ev.on('messages.upsert', async ({ messages, type }) => { + if (type !== 'notify' && type !== 'append') return; + for (const msg of messages || []) { + if (!msg?.message) continue; + const chatId = normalizeJid(msg.key.remoteJid); + const senderId = normalizeJid(msg.key.participant || chatId); + const isGroup = chatId.endsWith('@g.us'); + const fromMe = Boolean(msg.key.fromMe); + + if (fromMe) { + if (recentlySent.has(msg.key.id)) continue; + if (MODE === 'bot') continue; + const selfIds = new Set([ + normalizeJid(sock?.user?.id).replace(/:\d+@/, '@'), + normalizeJid(sock?.user?.lid).replace(/:\d+@/, '@'), + ].filter(Boolean)); + if (!selfIds.has(chatId)) continue; + } else if (MODE === 'self-chat') { + continue; + } + + const content = getMessageContent(msg); + const body = extractText(content); + const mediaInfo = detectMediaBlock(content); + const mediaPath = mediaInfo ? await cacheInboundMedia(msg, mediaInfo) : ''; + if (!body && !mediaPath) continue; + const event = buildBridgeEvent({ + msg, + chatId, + senderId, + isGroup, + body, + mediaInfo, + mediaPath, + fromOwner: fromMe, + }); + messageStore.set(msg.key.id, msg); + messageQueue.push(event); + if (messageQueue.length > 1000) messageQueue.shift(); + } + }); +} + +function validateHost(req, res, next) { + const raw = String(req.headers.host || '').trim(); + const host = raw.includes(':') ? raw.slice(0, raw.lastIndexOf(':')) : raw; + const normalized = host.replace(/^\[|\]$/g, '').toLowerCase(); + if (!['localhost', '127.0.0.1', '::1'].includes(normalized)) { + return res.status(400).json({ error: 'Invalid Host header' }); + } + return next(); +} + +function validateToken(req, res, next) { + const auth = String(req.headers.authorization || ''); + const bearer = auth.toLowerCase().startsWith('bearer ') ? auth.slice(7).trim() : ''; + const token = String(req.headers['x-flocks-bridge-token'] || bearer); + if (!BRIDGE_TOKEN || token !== BRIDGE_TOKEN) { + return res.status(401).json({ error: 'Unauthorized' }); + } + return next(); +} + +function isAllowedMediaPath(filePath) { + if (ALLOWED_MEDIA_ROOTS.length === 0) return false; + const resolved = path.resolve(filePath); + return ALLOWED_MEDIA_ROOTS.some(root => resolved === root || resolved.startsWith(`${root}${path.sep}`)); +} + +function inferMediaPayload(filePath, mediaType, caption) { + const buffer = readFileSync(filePath); + const ext = path.extname(filePath).slice(1).toLowerCase(); + const type = mediaType || ( + ['jpg', 'jpeg', 'png', 'webp', 'gif'].includes(ext) ? 'image' + : ['mp4', 'mov', 'mkv', 'webm'].includes(ext) ? 'video' + : ['ogg', 'opus', 'mp3', 'wav', 'm4a'].includes(ext) ? 'audio' + : 'document' + ); + if (type === 'image') return { image: buffer, caption: caption || undefined }; + if (type === 'video') return { video: buffer, caption: caption || undefined }; + if (type === 'audio') { + let audioBuffer = buffer; + let audioExt = ext; + let tmpPath = null; + if (!['ogg', 'opus'].includes(ext)) { + tmpPath = path.join(tmpdir(), `flocks_voice_${randomBytes(6).toString('hex')}.ogg`); + try { + execFileSync('ffmpeg', ['-y', '-i', filePath, '-ar', '48000', '-ac', '1', '-c:a', 'libopus', tmpPath], { timeout: 30000, stdio: 'pipe' }); + audioBuffer = readFileSync(tmpPath); + audioExt = 'ogg'; + } catch { + audioBuffer = buffer; + } finally { + try { if (tmpPath && existsSync(tmpPath)) unlinkSync(tmpPath); } catch {} + } + } + return { audio: audioBuffer, mimetype: audioExt === 'ogg' || audioExt === 'opus' ? 'audio/ogg; codecs=opus' : 'audio/mpeg', ptt: audioExt === 'ogg' || audioExt === 'opus' }; + } + return { + document: buffer, + fileName: path.basename(filePath), + mimetype: 'application/octet-stream', + caption: caption || undefined, + }; +} + +if (PAIR_ONLY) { + emitPairEvent({ event: 'started', session: SESSION_DIR }); + setTimeout(() => { + emitPairEvent({ event: 'error', error: 'pairing_timeout' }); + process.exit(2); + }, PAIR_TIMEOUT_MS); + await startSocket(); +} else { + await startSocket(); + const app = express(); + app.use(express.json({ limit: '2mb' })); + app.use(validateHost); + app.use(validateToken); + + app.get('/messages', (_req, res) => { + const messages = messageQueue.splice(0, messageQueue.length); + res.json(messages); + }); + + app.post('/send', async (req, res) => { + if (!sock || connectionState !== 'connected') return res.status(503).json({ error: 'Not connected to WhatsApp' }); + const { chatId, message, replyTo } = req.body || {}; + if (!chatId || !message) return res.status(400).json({ error: 'chatId and message are required' }); + try { + const text = REPLY_PREFIX ? `${REPLY_PREFIX}${message}` : String(message); + const chunks = splitLongMessage(text); + const messageIds = []; + for (let i = 0; i < chunks.length; i += 1) { + const quoted = replyTo && i === 0 ? messageStore.get(replyTo) : null; + const options = quoted ? { quoted } : {}; + const sent = await sendWithTimeout(chatId, { text: chunks[i] }, options); + rememberSent(sent?.key?.id); + if (sent?.key?.id) messageIds.push(sent.key.id); + if (i < chunks.length - 1) await sleep(CHUNK_DELAY_MS); + } + res.json({ success: true, messageId: messageIds[messageIds.length - 1] || '', messageIds }); + } catch (err) { + res.status(500).json({ error: err.message }); + } + }); + + app.post('/send-media', async (req, res) => { + if (!sock || connectionState !== 'connected') return res.status(503).json({ error: 'Not connected to WhatsApp' }); + const { chatId, filePath, mediaType, caption } = req.body || {}; + if (!chatId || !filePath) return res.status(400).json({ error: 'chatId and filePath are required' }); + if (!existsSync(filePath)) return res.status(404).json({ error: `File not found: ${filePath}` }); + if (!isAllowedMediaPath(filePath)) return res.status(403).json({ error: `File path is not allowed: ${filePath}` }); + try { + const payload = inferMediaPayload(filePath, mediaType, caption); + const sent = await sendWithTimeout(chatId, payload); + rememberSent(sent?.key?.id); + res.json({ success: true, messageId: sent?.key?.id || '' }); + } catch (err) { + res.status(500).json({ error: err.message }); + } + }); + + app.get('/health', (_req, res) => { + res.json({ + status: connectionState, + queueLength: messageQueue.length, + uptime: process.uptime(), + scriptHash, + sessionPath: path.resolve(SESSION_DIR), + mediaDir: path.resolve(MEDIA_DIR), + mode: MODE, + configHash: CONFIG_HASH, + tokenHash, + }); + }); + + app.listen(PORT, '127.0.0.1', () => { + console.log(`[whatsapp-bridge] listening on 127.0.0.1:${PORT}`); + }); +} diff --git a/flocks/channel/builtin/whatsapp/bridge/package-lock.json b/flocks/channel/builtin/whatsapp/bridge/package-lock.json new file mode 100644 index 000000000..b4fc7d4b6 --- /dev/null +++ b/flocks/channel/builtin/whatsapp/bridge/package-lock.json @@ -0,0 +1,2184 @@ +{ + "name": "flocks-whatsapp-bridge", + "version": "1.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "flocks-whatsapp-bridge", + "version": "1.0.0", + "dependencies": { + "@hapi/boom": "^10.0.1", + "@whiskeysockets/baileys": "7.0.0-rc13", + "express": "^4.21.2", + "pino": "^9.6.0" + } + }, + "node_modules/@borewit/text-codec": { + "version": "0.2.2", + "resolved": "https://registry.npmjs.org/@borewit/text-codec/-/text-codec-0.2.2.tgz", + "integrity": "sha512-DDaRehssg1aNrH4+2hnj1B7vnUGEjU6OIlyRdkMd0aUdIUvKXrJfXsy8LVtXAy7DRvYVluWbMspsRhz2lcW0mQ==", + "license": "MIT", + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Borewit" + } + }, + "node_modules/@cacheable/memory": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/@cacheable/memory/-/memory-2.2.0.tgz", + "integrity": "sha512-CTLKqLItRCEixEAewD3/j9DB3/o96gpTPD4eJ1v+DGOlxZRZncRQkGYqqnAGCscYd6RNeXfGeiuCphsPtqyIfQ==", + "license": "MIT", + "dependencies": { + "@cacheable/utils": "^2.5.0", + "@keyv/bigmap": "^1.3.1", + "hookified": "^1.15.1", + "keyv": "^5.6.0" + } + }, + "node_modules/@cacheable/node-cache": { + "version": "1.7.6", + "resolved": "https://registry.npmjs.org/@cacheable/node-cache/-/node-cache-1.7.6.tgz", + "integrity": "sha512-6Omk2SgNnjtxB5f/E6bTIWIt5xhdpx39fGNRQgU9lojvRxU68v+qY+SXXLsp3ZGukqoPjsK21wZ6XABFr/Ge3A==", + "license": "MIT", + "dependencies": { + "cacheable": "^2.3.1", + "hookified": "^1.14.0", + "keyv": "^5.5.5" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/@cacheable/utils": { + "version": "2.5.0", + "resolved": "https://registry.npmjs.org/@cacheable/utils/-/utils-2.5.0.tgz", + "integrity": "sha512-buipgOVDkkPXNR5+xBpDw7Zk2n1EvU7qBJCNUcL7rhQ//kfpOXPAvQ511Os0vpLYJ1pZnvudNytkQt2hst3wqA==", + "license": "MIT", + "dependencies": { + "hashery": "^1.5.1", + "keyv": "^5.6.0" + } + }, + "node_modules/@emnapi/runtime": { + "version": "1.11.2", + "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.2.tgz", + "integrity": "sha512-kyOl3X0DuTiT1h2ft8r2fYO8JYtU9a9Xis/zBSiGArNaagCOWx90N1k2wxp18czFDH+OgcWGb5ZP/XMt3dcyPA==", + "license": "MIT", + "optional": true, + "peer": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@hapi/boom": { + "version": "10.0.1", + "resolved": "https://registry.npmjs.org/@hapi/boom/-/boom-10.0.1.tgz", + "integrity": "sha512-ERcCZaEjdH3OgSJlyjVk8pHIFeus91CjKP3v+MpgBNp5IvGzP2l/bRiD78nqYcKPaZdbKkK5vDBVPd2ohHBlsA==", + "license": "BSD-3-Clause", + "dependencies": { + "@hapi/hoek": "^11.0.2" + } + }, + "node_modules/@hapi/hoek": { + "version": "11.0.7", + "resolved": "https://registry.npmjs.org/@hapi/hoek/-/hoek-11.0.7.tgz", + "integrity": "sha512-HV5undWkKzcB4RZUusqOpcgxOaq6VOAH7zhhIr2g3G8NF/MlFO75SjOr2NfuSx0Mh40+1FqCkagKLJRykUWoFQ==", + "license": "BSD-3-Clause" + }, + "node_modules/@img/colour": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@img/colour/-/colour-1.1.0.tgz", + "integrity": "sha512-Td76q7j57o/tLVdgS746cYARfSyxk8iEfRxewL9h4OMzYhbW4TAcppl0mT4eyqXddh6L/jwoM75mo7ixa/pCeQ==", + "license": "MIT", + "peer": true, + "engines": { + "node": ">=18" + } + }, + "node_modules/@img/sharp-darwin-arm64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-darwin-arm64/-/sharp-darwin-arm64-0.35.3.tgz", + "integrity": "sha512-RMnFX7YQsMoh7lWfcM4NEHHymBX/rLuKNPVM84XE9ONPcaSCDgE7CHIHpSgPcO2xcRthgBy1HfNO319mwhIAkg==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "peer": true, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-darwin-arm64": "1.3.2" + } + }, + "node_modules/@img/sharp-darwin-x64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-darwin-x64/-/sharp-darwin-x64-0.35.3.tgz", + "integrity": "sha512-Xo+5uFBtLN0BKqieTxiFzFPQAUlBbbH5iBKyRX/z1JrbnYsHTfKJnUfL8+p2TPXr1pXqao4eeL4Rl144uDpK9w==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "peer": true, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-darwin-x64": "1.3.2" + } + }, + "node_modules/@img/sharp-freebsd-wasm32": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-freebsd-wasm32/-/sharp-freebsd-wasm32-0.35.3.tgz", + "integrity": "sha512-lUxcqWIj2wMQ9BrwNjngcr1gWUr5xgaGThBRqPPalIC2n67Cqj1uPh8NnA/ZhAg8hUbKl+kVHKwgUIwe6ZYPrg==", + "license": "Apache-2.0", + "optional": true, + "os": [ + "freebsd" + ], + "peer": true, + "dependencies": { + "@img/sharp-wasm32": "0.35.3" + }, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-darwin-arm64": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-arm64/-/sharp-libvips-darwin-arm64-1.3.2.tgz", + "integrity": "sha512-9J6ypZFpQBj4YnePGoq/S38w6nz+vqg5WZLrLGY4YuSemdMq47GMLBPO42MzwdGwpg/agZ7xzZcFHa48xlywfg==", + "cpu": [ + "arm64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "darwin" + ], + "peer": true, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-darwin-x64": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-darwin-x64/-/sharp-libvips-darwin-x64-1.3.2.tgz", + "integrity": "sha512-m2pW1n6cns9VaubNwsZ+c3CRYjxNQWgJ5gPlnL1nbBcpkBvFm6SCFN5o0psFHI8w9n11NKhFkeEDns98tiqbEw==", + "cpu": [ + "x64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "darwin" + ], + "peer": true, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-arm": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm/-/sharp-libvips-linux-arm-1.3.2.tgz", + "integrity": "sha512-1eMLzy92I4J6rmi4mAT8yC3HxOtniyGELlzGbNMLLeqe052ahFQ0h6LFq+lh5DsDIdYViIDst08abvSbcEdLXQ==", + "cpu": [ + "arm" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-arm64": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-arm64/-/sharp-libvips-linux-arm64-1.3.2.tgz", + "integrity": "sha512-dqVSFynCox4C/J8kT16V7SIFAns0IjgLwkvYT7p8LQVmJ5OS5b6tI9IGflxTeuBS//zXeFIUbwt5dwxyZ17cnA==", + "cpu": [ + "arm64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-ppc64": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-ppc64/-/sharp-libvips-linux-ppc64-1.3.2.tgz", + "integrity": "sha512-3z0NHDxD6n5I9gc05U1eW1AyRm+Gznzq3naMrthPNqE6oYykcogW0l/jfpJdjYnuNl8R7yI9pNbE1XiUeyq0Aw==", + "cpu": [ + "ppc64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-riscv64": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-riscv64/-/sharp-libvips-linux-riscv64-1.3.2.tgz", + "integrity": "sha512-bsb4rI+NldGOsXuej2r8OdSS8+zXDVaCWxyWrcv6kneTOlgAHtZABRzBBCwdsPiD90J4myNJuHpg6kA20ImW/w==", + "cpu": [ + "riscv64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-s390x": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-s390x/-/sharp-libvips-linux-s390x-1.3.2.tgz", + "integrity": "sha512-/ABshyj8gCpyIrNXnHn4LorDJ0HHm1VhXPBlxZ8zAtfVPAaSafXPGn+sUSIRiwaSBy0mmFjSjiXI5mkcwdChKQ==", + "cpu": [ + "s390x" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linux-x64": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linux-x64/-/sharp-libvips-linux-x64-1.3.2.tgz", + "integrity": "sha512-ITPEtgffGJ0S6G9dRyw/366tJQqFRcHWPHhC+Stpg3Z8AEMrDrTr2lhdz4f/Y/HMbRh//7Z5mBzEpVdi62Oc3w==", + "cpu": [ + "x64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linuxmusl-arm64": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-arm64/-/sharp-libvips-linuxmusl-arm64-1.3.2.tgz", + "integrity": "sha512-zE9EdiUzUmg5mDT5a1rk5fYJ6GWPloTwWBYDS14naqHsL+EaMpDj1AWnpLgh3u0YCORv2Tt50wrcrpYqkP97Kw==", + "cpu": [ + "arm64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-libvips-linuxmusl-x64": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/@img/sharp-libvips-linuxmusl-x64/-/sharp-libvips-linuxmusl-x64-1.3.2.tgz", + "integrity": "sha512-m0lrLiUt+lBYnCFr8qV/65yMR4E/c7/wf78I5eKTdkEakFAlZ9QlzEM3QIhhAwVeUhLAHLcCq7a7Vszq/oFNZQ==", + "cpu": [ + "x64" + ], + "license": "LGPL-3.0-or-later", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-linux-arm": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-arm/-/sharp-linux-arm-0.35.3.tgz", + "integrity": "sha512-affVWCTLooy8TSxbDx2qkzuDeaWLNVBA+P//FNBirHsXpP2fuBhk5AuboYUnrDnzoXes8GFjpTx0SBFOCRg+FA==", + "cpu": [ + "arm" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-arm": "1.3.2" + } + }, + "node_modules/@img/sharp-linux-arm64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-arm64/-/sharp-linux-arm64-0.35.3.tgz", + "integrity": "sha512-QgKDspHPnrU+GQ55XPhGwyhC8acLVOOSyAvo1oVfFmrIXLkDNmGWzAfDZ4xK8oSA1qBQrALcHX0G5UZni/SuFQ==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-arm64": "1.3.2" + } + }, + "node_modules/@img/sharp-linux-ppc64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-ppc64/-/sharp-linux-ppc64-0.35.3.tgz", + "integrity": "sha512-sMd8rDxmpLOwv/7N44klFjOD5DUO7FLdjiXDI0hoxYaf7Ar262dQIEkosE98bps+5HPLtp/EvNqeqQtOycP/IA==", + "cpu": [ + "ppc64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-ppc64": "1.3.2" + } + }, + "node_modules/@img/sharp-linux-riscv64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-riscv64/-/sharp-linux-riscv64-0.35.3.tgz", + "integrity": "sha512-0Eob78yjlYPfL5vMNWAW55l3R9Y6BQS/gOfe0ZcP9mEz9ohhKSt4im1hayiknXgf8AWrFqMvJcKIdmLmEe7yeQ==", + "cpu": [ + "riscv64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-riscv64": "1.3.2" + } + }, + "node_modules/@img/sharp-linux-s390x": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-s390x/-/sharp-linux-s390x-0.35.3.tgz", + "integrity": "sha512-KgAxQ0DxpNOq1rG2t5cgTgShJFGSuU7XO45cqC+1NVOuZnP6tlgZRuSYOfNupGkHID0o3cJOsw4DVeJpMovcGw==", + "cpu": [ + "s390x" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-s390x": "1.3.2" + } + }, + "node_modules/@img/sharp-linux-x64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-linux-x64/-/sharp-linux-x64-0.35.3.tgz", + "integrity": "sha512-8pqvxubL2PGdhlPy6GLqzDYMUjyRmKAwKHYKixpdJYBUK7PJ0C029XdsnpFIdgRZG68fZiGdHVWcKPvtiPB4cA==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linux-x64": "1.3.2" + } + }, + "node_modules/@img/sharp-linuxmusl-arm64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-arm64/-/sharp-linuxmusl-arm64-0.35.3.tgz", + "integrity": "sha512-Vz0iQjzzcSX3HCbfwFfCSG/9SCIqyO0mH2sXyiHaAYfBk0cRsCWXRyQYX0ovCK/PAQBbTzQ0dsPQHh5MAFL59w==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linuxmusl-arm64": "1.3.2" + } + }, + "node_modules/@img/sharp-linuxmusl-x64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-linuxmusl-x64/-/sharp-linuxmusl-x64-0.35.3.tgz", + "integrity": "sha512-6O1NPKcDVj9QEdg7Hx549EX8U0rp6yXQERqru6yRN7fGBn32UvIRJUlWnk+8xDCiG76hXVBbX82NZ/ZKr0euIg==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "peer": true, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-libvips-linuxmusl-x64": "1.3.2" + } + }, + "node_modules/@img/sharp-wasm32": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-wasm32/-/sharp-wasm32-0.35.3.tgz", + "integrity": "sha512-cZ0XkcYGpHZkqW6iCkqTcmUC0CD9DhD5d/qeZlZkfRBn6GnHniZXLUo5+9xw8Iv76YE6LQFN9YNBlKREcCG76w==", + "license": "Apache-2.0 AND LGPL-3.0-or-later AND MIT", + "optional": true, + "peer": true, + "dependencies": { + "@emnapi/runtime": "^1.11.1" + }, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-webcontainers-wasm32": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-webcontainers-wasm32/-/sharp-webcontainers-wasm32-0.35.3.tgz", + "integrity": "sha512-2rnq7bX3NzeR2T4YWgz8qiG4h3TSdMe+vN1iQXpJleSJ3SM5zQ8Fy2SyyXAWlbxpEZ2Y+Z4u1BePgJEYbSy80Q==", + "cpu": [ + "wasm32" + ], + "license": "Apache-2.0", + "optional": true, + "peer": true, + "dependencies": { + "@img/sharp-wasm32": "0.35.3" + }, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-win32-arm64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-win32-arm64/-/sharp-win32-arm64-0.35.3.tgz", + "integrity": "sha512-4bPwFdMbeC4JQ8L8LOyWp6nsHcboP5fxkp6iPOXz2Vg49R42TuMs2whkJ5OAP4/Ul035qOzy0AecOF9VOscn4w==", + "cpu": [ + "arm64" + ], + "license": "Apache-2.0 AND LGPL-3.0-or-later", + "optional": true, + "os": [ + "win32" + ], + "peer": true, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-win32-ia32": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-win32-ia32/-/sharp-win32-ia32-0.35.3.tgz", + "integrity": "sha512-r53mXsBN6lFUDiST764SvgwUdHAqM4rPAiDzAmf4fLoB6X/rkfyTrLCg6+g17wJJiCmB3JYgHuUldCWUIRFSXw==", + "cpu": [ + "ia32" + ], + "license": "Apache-2.0 AND LGPL-3.0-or-later", + "optional": true, + "os": [ + "win32" + ], + "peer": true, + "engines": { + "node": "^20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@img/sharp-win32-x64": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/@img/sharp-win32-x64/-/sharp-win32-x64-0.35.3.tgz", + "integrity": "sha512-D4y1vNeZrIIJCN+uHaWVtH86B+aCrdMYYjicy9pXHvbGZeGYLLSd3wdVuC37FxVXlU1ARsk84eKWfWMXGYEqvA==", + "cpu": [ + "x64" + ], + "license": "Apache-2.0 AND LGPL-3.0-or-later", + "optional": true, + "os": [ + "win32" + ], + "peer": true, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + } + }, + "node_modules/@keyv/bigmap": { + "version": "1.3.1", + "resolved": "https://registry.npmjs.org/@keyv/bigmap/-/bigmap-1.3.1.tgz", + "integrity": "sha512-WbzE9sdmQtKy8vrNPa9BRnwZh5UF4s1KTmSK0KUVLo3eff5BlQNNWDnFOouNpKfPKDnms9xynJjsMYjMaT/aFQ==", + "license": "MIT", + "dependencies": { + "hashery": "^1.4.0", + "hookified": "^1.15.0" + }, + "engines": { + "node": ">= 18" + }, + "peerDependencies": { + "keyv": "^5.6.0" + } + }, + "node_modules/@keyv/serialize": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@keyv/serialize/-/serialize-1.1.1.tgz", + "integrity": "sha512-dXn3FZhPv0US+7dtJsIi2R+c7qWYiReoEh5zUntWCf4oSpMNib8FDhSoed6m3QyZdx5hK7iLFkYk3rNxwt8vTA==", + "license": "MIT" + }, + "node_modules/@pinojs/redact": { + "version": "0.4.0", + "resolved": "https://registry.npmjs.org/@pinojs/redact/-/redact-0.4.0.tgz", + "integrity": "sha512-k2ENnmBugE/rzQfEcdWHcCY+/FM3VLzH9cYEsbdsoqrvzAKRhUZeRNhAZvB8OitQJ1TBed3yqWtdjzS6wJKBwg==", + "license": "MIT" + }, + "node_modules/@protobufjs/aspromise": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz", + "integrity": "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/base64": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/base64/-/base64-1.1.2.tgz", + "integrity": "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/codegen": { + "version": "2.0.5", + "resolved": "https://registry.npmjs.org/@protobufjs/codegen/-/codegen-2.0.5.tgz", + "integrity": "sha512-zgXFLzW3Ap33e6d0Wlj4MGIm6Ce8O89n/apUaGNB/jx+hw+ruWEp7EwGUshdLKVRCxZW12fp9r40E1mQrf/34g==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/eventemitter": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@protobufjs/eventemitter/-/eventemitter-1.1.1.tgz", + "integrity": "sha512-vW1GmwMZNnL+gMRaovlh9yZX74kc+TTU3FObkkurpMaRtBfLP3ldjS9KQWlwZgraRE0+dheEEoAxdzcJQ8eXZg==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/fetch": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@protobufjs/fetch/-/fetch-1.1.1.tgz", + "integrity": "sha512-GpptLrs57adMSuHi3VNj0mAF8dwh36LMaYF6XyJ6JMWlVsc+t42tm1HSEDmOs3A8fC9yyeisgLhsTVQokOZ0zw==", + "license": "BSD-3-Clause", + "dependencies": { + "@protobufjs/aspromise": "^1.1.1" + } + }, + "node_modules/@protobufjs/float": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/@protobufjs/float/-/float-1.0.2.tgz", + "integrity": "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/path": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/path/-/path-1.1.2.tgz", + "integrity": "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/pool": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/@protobufjs/pool/-/pool-1.1.0.tgz", + "integrity": "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==", + "license": "BSD-3-Clause" + }, + "node_modules/@protobufjs/utf8": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/@protobufjs/utf8/-/utf8-1.1.2.tgz", + "integrity": "sha512-b1UQwcEZ4yCnMCD8DAL1VlbvBJE9/IX4FTIp7BG1xYpf29SLazLSrqUkj4w7Y5y7cCVP6E5tcqqcI0xemPkHug==", + "license": "BSD-3-Clause" + }, + "node_modules/@tokenizer/inflate": { + "version": "0.4.1", + "resolved": "https://registry.npmjs.org/@tokenizer/inflate/-/inflate-0.4.1.tgz", + "integrity": "sha512-2mAv+8pkG6GIZiF1kNg1jAjh27IDxEPKwdGul3snfztFerfPGI1LjDezZp3i7BElXompqEtPmoPx6c2wgtWsOA==", + "license": "MIT", + "dependencies": { + "debug": "^4.4.3", + "token-types": "^6.1.1" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Borewit" + } + }, + "node_modules/@tokenizer/inflate/node_modules/debug": { + "version": "4.4.3", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", + "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", + "license": "MIT", + "dependencies": { + "ms": "^2.1.3" + }, + "engines": { + "node": ">=6.0" + }, + "peerDependenciesMeta": { + "supports-color": { + "optional": true + } + } + }, + "node_modules/@tokenizer/inflate/node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "license": "MIT" + }, + "node_modules/@tokenizer/token": { + "version": "0.3.0", + "resolved": "https://registry.npmjs.org/@tokenizer/token/-/token-0.3.0.tgz", + "integrity": "sha512-OvjF+z51L3ov0OyAU0duzsYuvO01PH7x4t6DJx+guahgTnBHkhJdG7soQeTSFLWN3efnHyibZ4Z8l2EuWwJN3A==", + "license": "MIT" + }, + "node_modules/@types/node": { + "version": "26.1.1", + "resolved": "https://registry.npmjs.org/@types/node/-/node-26.1.1.tgz", + "integrity": "sha512-nxAkRSVkN1Y0JC1W8ky/fTfkGsMmcrRsbx+3XoZE+rMOX71kLYTV7fLXpqud1GpbpP5TuffXFqfX7fH2GgZREw==", + "license": "MIT", + "dependencies": { + "undici-types": "~8.3.0" + } + }, + "node_modules/@whiskeysockets/baileys": { + "version": "7.0.0-rc13", + "resolved": "https://registry.npmjs.org/@whiskeysockets/baileys/-/baileys-7.0.0-rc13.tgz", + "integrity": "sha512-8JPc8gaaCRykkjW2jxLGQ7/RZGrc7awO7WU+QJocf58eSUI9jAdcuYLynzhAbyU4UWvJJsHImZ+5E/JaZj5ypA==", + "hasInstallScript": true, + "license": "MIT", + "dependencies": { + "@cacheable/node-cache": "^1.4.0", + "@hapi/boom": "^9.1.3", + "async-mutex": "^0.5.0", + "libsignal": "^6.0.0", + "lru-cache": "^11.1.0", + "music-metadata": "^11.12.3", + "p-queue": "^9.0.0", + "pino": "^9.6", + "protobufjs": "^7.5.6", + "whatsapp-rust-bridge": "0.5.4", + "ws": "^8.13.0" + }, + "engines": { + "node": ">=20.0.0" + }, + "peerDependencies": { + "audio-decode": "^2.1.3", + "jimp": "^1.6.1", + "link-preview-js": "^3.0.0", + "sharp": "*" + }, + "peerDependenciesMeta": { + "audio-decode": { + "optional": true + }, + "jimp": { + "optional": true + }, + "link-preview-js": { + "optional": true + } + } + }, + "node_modules/@whiskeysockets/baileys/node_modules/@hapi/boom": { + "version": "9.1.4", + "resolved": "https://registry.npmjs.org/@hapi/boom/-/boom-9.1.4.tgz", + "integrity": "sha512-Ls1oH8jaN1vNsqcaHVYJrKmgMcKsC1wcp8bujvXrHaAqD2iDYq3HoOwsxwo09Cuda5R5nC0o0IxlrlTuvPuzSw==", + "license": "BSD-3-Clause", + "dependencies": { + "@hapi/hoek": "9.x.x" + } + }, + "node_modules/@whiskeysockets/baileys/node_modules/@hapi/hoek": { + "version": "9.3.0", + "resolved": "https://registry.npmjs.org/@hapi/hoek/-/hoek-9.3.0.tgz", + "integrity": "sha512-/c6rf4UJlmHlC9b5BaNvzAcFv7HZ2QHaV0D4/HNlBdvFnvQq8RI4kYdhyPCl7Xj+oWvTWQ8ujhqS53LIgAe6KQ==", + "license": "BSD-3-Clause" + }, + "node_modules/accepts": { + "version": "1.3.8", + "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.8.tgz", + "integrity": "sha512-PYAthTa2m2VKxuvSD3DPC/Gy+U+sOA1LAuT8mkmRuvw+NACSaeXEQ+NHcVF7rONl6qcaxV3Uuemwawk+7+SJLw==", + "license": "MIT", + "dependencies": { + "mime-types": "~2.1.34", + "negotiator": "0.6.3" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/array-flatten": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", + "integrity": "sha512-PCVAQswWemu6UdxsDFFX/+gVeYqKAod3D3UVm91jHwynguOwAvYPhx8nNlM++NqRcK6CxxpUafjmhIdKiHibqg==", + "license": "MIT" + }, + "node_modules/async-mutex": { + "version": "0.5.0", + "resolved": "https://registry.npmjs.org/async-mutex/-/async-mutex-0.5.0.tgz", + "integrity": "sha512-1A94B18jkJ3DYq284ohPxoXbfTA5HsQ7/Mf4DEhcyLx3Bz27Rh59iScbB6EPiP+B+joue6YCxcMXSbFC1tZKwA==", + "license": "MIT", + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/atomic-sleep": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/atomic-sleep/-/atomic-sleep-1.0.0.tgz", + "integrity": "sha512-kNOjDqAh7px0XWNI+4QbzoiR/nTkHAWNud2uvnJquD1/x5a7EQZMJT0AczqK0Qn67oY/TTQ1LbUKajZpp3I9tQ==", + "license": "MIT", + "engines": { + "node": ">=8.0.0" + } + }, + "node_modules/body-parser": { + "version": "1.20.6", + "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.6.tgz", + "integrity": "sha512-p5tAzS57i5MV9fZFDj9LeIiTZEufbSe2eDozP+ElheSUq1m74CRq1jI4mYNDdVs9vQztXFLuk/Gd6BWTdwRJ5g==", + "license": "MIT", + "dependencies": { + "bytes": "~3.1.2", + "content-type": "~1.0.5", + "debug": "2.6.9", + "depd": "2.0.0", + "destroy": "~1.2.0", + "http-errors": "~2.0.1", + "iconv-lite": "~0.4.24", + "on-finished": "~2.4.1", + "qs": "~6.15.1", + "raw-body": "~2.5.3", + "type-is": "~1.6.18", + "unpipe": "~1.0.0" + }, + "engines": { + "node": ">= 0.8", + "npm": "1.2.8000 || >= 1.4.16" + } + }, + "node_modules/bytes": { + "version": "3.1.2", + "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz", + "integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/cacheable": { + "version": "2.5.0", + "resolved": "https://registry.npmjs.org/cacheable/-/cacheable-2.5.0.tgz", + "integrity": "sha512-60cyAOytib/OzBw1JNSoSV/boK1AtHryDIjvVBk7XbN4ugfkM3+Sry7fEjNgPMGgOjuaZPAp8ruZ0Cxafwyq9g==", + "license": "MIT", + "dependencies": { + "@cacheable/memory": "^2.2.0", + "@cacheable/utils": "^2.5.0", + "hookified": "^1.15.0", + "keyv": "^5.6.0", + "qified": "^0.10.1" + } + }, + "node_modules/call-bind-apply-helpers": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz", + "integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "function-bind": "^1.1.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/call-bound": { + "version": "1.0.4", + "resolved": "https://registry.npmjs.org/call-bound/-/call-bound-1.0.4.tgz", + "integrity": "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.2", + "get-intrinsic": "^1.3.0" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/content-disposition": { + "version": "0.5.4", + "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.4.tgz", + "integrity": "sha512-FveZTNuGw04cxlAiWbzi6zTAL/lhehaWbTtgluJh4/E95DqMwTmha3KZN1aAWA8cFIhHzMZUvLevkw5Rqk+tSQ==", + "license": "MIT", + "dependencies": { + "safe-buffer": "5.2.1" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/content-type": { + "version": "1.0.5", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.5.tgz", + "integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/cookie": { + "version": "0.7.2", + "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz", + "integrity": "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/cookie-signature": { + "version": "1.0.7", + "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.0.7.tgz", + "integrity": "sha512-NXdYc3dLr47pBkpUCHtKSwIOQXLVn8dZEuywboCOJY/osA0wFSLlSawr3KN8qXJEyX66FcONTH8EIlVuK0yyFA==", + "license": "MIT" + }, + "node_modules/curve25519-js": { + "version": "0.0.4", + "resolved": "https://registry.npmjs.org/curve25519-js/-/curve25519-js-0.0.4.tgz", + "integrity": "sha512-axn2UMEnkhyDUPWOwVKBMVIzSQy2ejH2xRGy1wq81dqRwApXfIzfbE3hIX0ZRFBIihf/KDqK158DLwESu4AK1w==", + "license": "MIT" + }, + "node_modules/debug": { + "version": "2.6.9", + "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz", + "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==", + "license": "MIT", + "dependencies": { + "ms": "2.0.0" + } + }, + "node_modules/depd": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz", + "integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/destroy": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/destroy/-/destroy-1.2.0.tgz", + "integrity": "sha512-2sJGJTaXIIaR1w4iJSNoN0hnMY7Gpc/n8D4qSCJw8QqFWXf7cuAgnEHxBpweaVcPevC2l3KpjYCx3NypQQgaJg==", + "license": "MIT", + "engines": { + "node": ">= 0.8", + "npm": "1.2.8000 || >= 1.4.16" + } + }, + "node_modules/detect-libc": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz", + "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==", + "license": "Apache-2.0", + "peer": true, + "engines": { + "node": ">=8" + } + }, + "node_modules/dunder-proto": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz", + "integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.1", + "es-errors": "^1.3.0", + "gopd": "^1.2.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/ee-first": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz", + "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==", + "license": "MIT" + }, + "node_modules/encodeurl": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz", + "integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/es-define-property": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz", + "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-errors": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz", + "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/es-object-atoms": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.2.tgz", + "integrity": "sha512-HWcBoN6NileqtSydK2FqHbS/LoDd2pqrnQHLyJzBj4kOp/ky2MWMN694xOfkK8/SnUsW2DH7EfyVlydKCsm1Zw==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/escape-html": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz", + "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==", + "license": "MIT" + }, + "node_modules/etag": { + "version": "1.8.1", + "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz", + "integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/eventemitter3": { + "version": "5.0.4", + "resolved": "https://registry.npmjs.org/eventemitter3/-/eventemitter3-5.0.4.tgz", + "integrity": "sha512-mlsTRyGaPBjPedk6Bvw+aqbsXDtoAyAzm5MO7JgU+yVRyMQ5O8bD4Kcci7BS85f93veegeCPkL8R4GLClnjLFw==", + "license": "MIT" + }, + "node_modules/express": { + "version": "4.22.2", + "resolved": "https://registry.npmjs.org/express/-/express-4.22.2.tgz", + "integrity": "sha512-IuL+Elrou2ZvCFHs18/CIzy2Nzvo25nZ1/D2eIZlz7c+QUayAcYoiM2BthCjs+EBHVpjYjcuLDAiCWgeIX3X1Q==", + "license": "MIT", + "dependencies": { + "accepts": "~1.3.8", + "array-flatten": "1.1.1", + "body-parser": "~1.20.5", + "content-disposition": "~0.5.4", + "content-type": "~1.0.4", + "cookie": "~0.7.1", + "cookie-signature": "~1.0.6", + "debug": "2.6.9", + "depd": "2.0.0", + "encodeurl": "~2.0.0", + "escape-html": "~1.0.3", + "etag": "~1.8.1", + "finalhandler": "~1.3.1", + "fresh": "~0.5.2", + "http-errors": "~2.0.0", + "merge-descriptors": "1.0.3", + "methods": "~1.1.2", + "on-finished": "~2.4.1", + "parseurl": "~1.3.3", + "path-to-regexp": "~0.1.12", + "proxy-addr": "~2.0.7", + "qs": "~6.15.1", + "range-parser": "~1.2.1", + "safe-buffer": "5.2.1", + "send": "~0.19.0", + "serve-static": "~1.16.2", + "setprototypeof": "1.2.0", + "statuses": "~2.0.1", + "type-is": "~1.6.18", + "utils-merge": "1.0.1", + "vary": "~1.1.2" + }, + "engines": { + "node": ">= 0.10.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/file-type": { + "version": "21.3.4", + "resolved": "https://registry.npmjs.org/file-type/-/file-type-21.3.4.tgz", + "integrity": "sha512-Ievi/yy8DS3ygGvT47PjSfdFoX+2isQueoYP1cntFW1JLYAuS4GD7NUPGg4zv2iZfV52uDyk5w5Z0TdpRS6Q1g==", + "license": "MIT", + "dependencies": { + "@tokenizer/inflate": "^0.4.1", + "strtok3": "^10.3.4", + "token-types": "^6.1.1", + "uint8array-extras": "^1.4.0" + }, + "engines": { + "node": ">=20" + }, + "funding": { + "url": "https://github.com/sindresorhus/file-type?sponsor=1" + } + }, + "node_modules/finalhandler": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.3.2.tgz", + "integrity": "sha512-aA4RyPcd3badbdABGDuTXCMTtOneUCAYH/gxoYRTZlIJdF0YPWuGqiAsIrhNnnqdXGswYk6dGujem4w80UJFhg==", + "license": "MIT", + "dependencies": { + "debug": "2.6.9", + "encodeurl": "~2.0.0", + "escape-html": "~1.0.3", + "on-finished": "~2.4.1", + "parseurl": "~1.3.3", + "statuses": "~2.0.2", + "unpipe": "~1.0.0" + }, + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/forwarded": { + "version": "0.2.0", + "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz", + "integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/fresh": { + "version": "0.5.2", + "resolved": "https://registry.npmjs.org/fresh/-/fresh-0.5.2.tgz", + "integrity": "sha512-zJ2mQYM18rEFOudeV4GShTGIQ7RbzA7ozbU9I/XBpm7kqgMywgmylMwXHxZJmkVoYkna9d2pVXVXPdYTP9ej8Q==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/function-bind": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz", + "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/get-intrinsic": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz", + "integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==", + "license": "MIT", + "dependencies": { + "call-bind-apply-helpers": "^1.0.2", + "es-define-property": "^1.0.1", + "es-errors": "^1.3.0", + "es-object-atoms": "^1.1.1", + "function-bind": "^1.1.2", + "get-proto": "^1.0.1", + "gopd": "^1.2.0", + "has-symbols": "^1.1.0", + "hasown": "^2.0.2", + "math-intrinsics": "^1.1.0" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/get-proto": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz", + "integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==", + "license": "MIT", + "dependencies": { + "dunder-proto": "^1.0.1", + "es-object-atoms": "^1.0.0" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/gopd": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz", + "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/has-symbols": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz", + "integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/hashery": { + "version": "1.5.1", + "resolved": "https://registry.npmjs.org/hashery/-/hashery-1.5.1.tgz", + "integrity": "sha512-iZyKG96/JwPz1N55vj2Ie2vXbhu440zfUfJvSwEqEbeLluk7NnapfGqa7LH0mOsnDxTF85Mx8/dyR6HfqcbmbQ==", + "license": "MIT", + "dependencies": { + "hookified": "^1.15.0" + }, + "engines": { + "node": ">=20" + } + }, + "node_modules/hasown": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.4.tgz", + "integrity": "sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==", + "license": "MIT", + "dependencies": { + "function-bind": "^1.1.2" + }, + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/hookified": { + "version": "1.15.1", + "resolved": "https://registry.npmjs.org/hookified/-/hookified-1.15.1.tgz", + "integrity": "sha512-MvG/clsADq1GPM2KGo2nyfaWVyn9naPiXrqIe4jYjXNZQt238kWyOGrsyc/DmRAQ+Re6yeo6yX/yoNCG5KAEVg==", + "license": "MIT" + }, + "node_modules/http-errors": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz", + "integrity": "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==", + "license": "MIT", + "dependencies": { + "depd": "~2.0.0", + "inherits": "~2.0.4", + "setprototypeof": "~1.2.0", + "statuses": "~2.0.2", + "toidentifier": "~1.0.1" + }, + "engines": { + "node": ">= 0.8" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/iconv-lite": { + "version": "0.4.24", + "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz", + "integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==", + "license": "MIT", + "dependencies": { + "safer-buffer": ">= 2.1.2 < 3" + }, + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/ieee754": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz", + "integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/feross" + }, + { + "type": "patreon", + "url": "https://www.patreon.com/feross" + }, + { + "type": "consulting", + "url": "https://feross.org/support" + } + ], + "license": "BSD-3-Clause" + }, + "node_modules/inherits": { + "version": "2.0.4", + "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", + "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==", + "license": "ISC" + }, + "node_modules/ipaddr.js": { + "version": "1.9.1", + "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz", + "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==", + "license": "MIT", + "engines": { + "node": ">= 0.10" + } + }, + "node_modules/keyv": { + "version": "5.6.0", + "resolved": "https://registry.npmjs.org/keyv/-/keyv-5.6.0.tgz", + "integrity": "sha512-CYDD3SOtsHtyXeEORYRx2qBtpDJFjRTGXUtmNEMGyzYOKj1TE3tycdlho7kA1Ufx9OYWZzg52QFBGALTirzDSw==", + "license": "MIT", + "dependencies": { + "@keyv/serialize": "^1.1.1" + } + }, + "node_modules/libsignal": { + "version": "6.0.0", + "resolved": "https://registry.npmjs.org/libsignal/-/libsignal-6.0.0.tgz", + "integrity": "sha512-d/5V3YFtDljbFMufz4ncyUYGYhJl+vzAe+c2EFFBQ6bz1h8Q3IOMEGXYMzlibU60I+e8GagMMpji18iez3P1hA==", + "license": "GPL-3.0", + "dependencies": { + "curve25519-js": "^0.0.4", + "protobufjs": "^7.5.5" + } + }, + "node_modules/long": { + "version": "5.3.2", + "resolved": "https://registry.npmjs.org/long/-/long-5.3.2.tgz", + "integrity": "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA==", + "license": "Apache-2.0" + }, + "node_modules/lru-cache": { + "version": "11.5.2", + "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-11.5.2.tgz", + "integrity": "sha512-4pfM1Ff0x50o0tQwb5ucw/RzNyD0/YJME6IVcStalZuMWxdt3sR3huStTtxz4PUmvZfRguvDejasvQ2kifR11g==", + "license": "BlueOak-1.0.0", + "engines": { + "node": "20 || >=22" + } + }, + "node_modules/math-intrinsics": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz", + "integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, + "node_modules/media-typer": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-2.0.0.tgz", + "integrity": "sha512-kOy3OxT2HH39N70UnKgu4NWDZjLOz8W/mfyvniHjRH/DrL3f2pOfvWQ4p60offbbtDAnXWp0v9LfMIqMec269Q==", + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/merge-descriptors": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.3.tgz", + "integrity": "sha512-gaNvAS7TZ897/rVaZ0nMtAyxNyi/pdbjbAwUpFQpN70GqnVfOiXpeUUMKRBmzXaSQ8DdTX4/0ms62r2K+hE6mQ==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/methods": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/methods/-/methods-1.1.2.tgz", + "integrity": "sha512-iclAHeNqNm68zFtnZ0e+1L2yUIdvzNoauKU4WBA3VvH/vPFieF7qfRlwUZU+DA9P9bPXIS90ulxoUoCH23sV2w==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/mime": { + "version": "1.6.0", + "resolved": "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz", + "integrity": "sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg==", + "license": "MIT", + "bin": { + "mime": "cli.js" + }, + "engines": { + "node": ">=4" + } + }, + "node_modules/mime-db": { + "version": "1.52.0", + "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz", + "integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/mime-types": { + "version": "2.1.35", + "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz", + "integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==", + "license": "MIT", + "dependencies": { + "mime-db": "1.52.0" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/ms": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz", + "integrity": "sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A==", + "license": "MIT" + }, + "node_modules/music-metadata": { + "version": "11.13.0", + "resolved": "https://registry.npmjs.org/music-metadata/-/music-metadata-11.13.0.tgz", + "integrity": "sha512-uXRaov9dfjSpQufXIU7sMxVZnh+FilCQv2mXn+K5EJ/decP3dTWrgvPYa5r6MtRbieNSCE708Da4J0u1UGfQIw==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/Borewit" + }, + { + "type": "buymeacoffee", + "url": "https://buymeacoffee.com/borewit" + } + ], + "license": "MIT", + "dependencies": { + "@borewit/text-codec": "^0.2.2", + "@tokenizer/token": "^0.3.0", + "content-type": "^2.0.0", + "debug": "^4.4.3", + "file-type": "^21.3.4", + "media-typer": "^2.0.0", + "strtok3": "^10.3.5", + "token-types": "^6.1.2", + "uint8array-extras": "^1.5.0", + "win-guid": "^0.2.1" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/music-metadata/node_modules/content-type": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/content-type/-/content-type-2.0.0.tgz", + "integrity": "sha512-j/O/d7GcZCyNl7/hwZAb606rzqkyvaDctLmckbxLzHvFBzTJHuGEdodATcP3yIRoDrLHkIATJuvzbFlp/ki2cQ==", + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/express" + } + }, + "node_modules/music-metadata/node_modules/debug": { + "version": "4.4.3", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", + "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", + "license": "MIT", + "dependencies": { + "ms": "^2.1.3" + }, + "engines": { + "node": ">=6.0" + }, + "peerDependenciesMeta": { + "supports-color": { + "optional": true + } + } + }, + "node_modules/music-metadata/node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "license": "MIT" + }, + "node_modules/negotiator": { + "version": "0.6.3", + "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.3.tgz", + "integrity": "sha512-+EUsqGPLsM+j/zdChZjsnX51g4XrHFOIXwfnCVPGlQk/k5giakcKsuxCObBRu6DSm9opw/O6slWbJdghQM4bBg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/object-inspect": { + "version": "1.13.4", + "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz", + "integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==", + "license": "MIT", + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/on-exit-leak-free": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/on-exit-leak-free/-/on-exit-leak-free-2.1.2.tgz", + "integrity": "sha512-0eJJY6hXLGf1udHwfNftBqH+g73EU4B504nZeKpz1sYRKafAghwxEJunB2O7rDZkL4PGfsMVnTXZ2EjibbqcsA==", + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/on-finished": { + "version": "2.4.1", + "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz", + "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==", + "license": "MIT", + "dependencies": { + "ee-first": "1.1.1" + }, + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/p-queue": { + "version": "9.3.1", + "resolved": "https://registry.npmjs.org/p-queue/-/p-queue-9.3.1.tgz", + "integrity": "sha512-POWdiIPmsUPGwb4FeQ4OBg46aqmcInSWe45CKDsGHiOBiVQM9chqfQTuqhuTzcg2Vz9faTI65at0KkVyVEiCHw==", + "license": "MIT", + "dependencies": { + "eventemitter3": "^5.0.4", + "p-timeout": "^7.0.0" + }, + "engines": { + "node": ">=20" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/p-timeout": { + "version": "7.0.1", + "resolved": "https://registry.npmjs.org/p-timeout/-/p-timeout-7.0.1.tgz", + "integrity": "sha512-AxTM2wDGORHGEkPCt8yqxOTMgpfbEHqF51f/5fJCmwFC3C/zNcGT63SymH2ttOAaiIws2zVg4+izQCjrakcwHg==", + "license": "MIT", + "engines": { + "node": ">=20" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/parseurl": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz", + "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/path-to-regexp": { + "version": "0.1.13", + "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.13.tgz", + "integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==", + "license": "MIT" + }, + "node_modules/pino": { + "version": "9.14.0", + "resolved": "https://registry.npmjs.org/pino/-/pino-9.14.0.tgz", + "integrity": "sha512-8OEwKp5juEvb/MjpIc4hjqfgCNysrS94RIOMXYvpYCdm/jglrKEiAYmiumbmGhCvs+IcInsphYDFwqrjr7398w==", + "license": "MIT", + "dependencies": { + "@pinojs/redact": "^0.4.0", + "atomic-sleep": "^1.0.0", + "on-exit-leak-free": "^2.1.0", + "pino-abstract-transport": "^2.0.0", + "pino-std-serializers": "^7.0.0", + "process-warning": "^5.0.0", + "quick-format-unescaped": "^4.0.3", + "real-require": "^0.2.0", + "safe-stable-stringify": "^2.3.1", + "sonic-boom": "^4.0.1", + "thread-stream": "^3.0.0" + }, + "bin": { + "pino": "bin.js" + } + }, + "node_modules/pino-abstract-transport": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/pino-abstract-transport/-/pino-abstract-transport-2.0.0.tgz", + "integrity": "sha512-F63x5tizV6WCh4R6RHyi2Ml+M70DNRXt/+HANowMflpgGFMAym/VKm6G7ZOQRjqN7XbGxK1Lg9t6ZrtzOaivMw==", + "license": "MIT", + "dependencies": { + "split2": "^4.0.0" + } + }, + "node_modules/pino-std-serializers": { + "version": "7.1.0", + "resolved": "https://registry.npmjs.org/pino-std-serializers/-/pino-std-serializers-7.1.0.tgz", + "integrity": "sha512-BndPH67/JxGExRgiX1dX0w1FvZck5Wa4aal9198SrRhZjH3GxKQUKIBnYJTdj2HDN3UQAS06HlfcSbQj2OHmaw==", + "license": "MIT" + }, + "node_modules/process-warning": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/process-warning/-/process-warning-5.0.0.tgz", + "integrity": "sha512-a39t9ApHNx2L4+HBnQKqxxHNs1r7KF+Intd8Q/g1bUh6q0WIp9voPXJ/x0j+ZL45KF1pJd9+q2jLIRMfvEshkA==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/fastify" + }, + { + "type": "opencollective", + "url": "https://opencollective.com/fastify" + } + ], + "license": "MIT" + }, + "node_modules/protobufjs": { + "version": "7.6.5", + "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-7.6.5.tgz", + "integrity": "sha512-/FPD0nUc9jH6rfFjji9IBqOz4pcSE3CsT1m7Ep6Mdb0LxSUMj8hgl6GomOvZzpNpAqqGaXA0P3VSrZLFzIhQrw==", + "hasInstallScript": true, + "license": "BSD-3-Clause", + "dependencies": { + "@protobufjs/aspromise": "^1.1.2", + "@protobufjs/base64": "^1.1.2", + "@protobufjs/codegen": "^2.0.5", + "@protobufjs/eventemitter": "^1.1.1", + "@protobufjs/fetch": "^1.1.1", + "@protobufjs/float": "^1.0.2", + "@protobufjs/path": "^1.1.2", + "@protobufjs/pool": "^1.1.0", + "@protobufjs/utf8": "^1.1.1", + "@types/node": ">=13.7.0", + "long": "^5.3.2" + }, + "engines": { + "node": ">=12.0.0" + } + }, + "node_modules/proxy-addr": { + "version": "2.0.7", + "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz", + "integrity": "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==", + "license": "MIT", + "dependencies": { + "forwarded": "0.2.0", + "ipaddr.js": "1.9.1" + }, + "engines": { + "node": ">= 0.10" + } + }, + "node_modules/qified": { + "version": "0.10.1", + "resolved": "https://registry.npmjs.org/qified/-/qified-0.10.1.tgz", + "integrity": "sha512-+Owyggi9IxT1ePKGafcI87ubSmxol6smwJ+RAHDQlx9+9cPwFWDiKFFCPuWhr9ignlGpZ9vDQLw67N4dcTVFEA==", + "license": "MIT", + "dependencies": { + "hookified": "^2.1.1" + }, + "engines": { + "node": ">=20" + } + }, + "node_modules/qified/node_modules/hookified": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/hookified/-/hookified-2.2.0.tgz", + "integrity": "sha512-p/LgFzRN5FeoD3DLS6bkUapeye6E4SI6yJs6KetENd18S+FBthqYq2amJUWpt5z0EQwwHemidjY5OqJGEKm5uA==", + "license": "MIT" + }, + "node_modules/qs": { + "version": "6.15.3", + "resolved": "https://registry.npmjs.org/qs/-/qs-6.15.3.tgz", + "integrity": "sha512-O9gl3zCl5h5blw1KGUzQKhA5oUXSl8rwUIM5o0S3nCXMliSvy5Dzx7/DJcI+SwgICv+IneSZwhBh1oSyEHA71A==", + "license": "BSD-3-Clause", + "dependencies": { + "es-define-property": "^1.0.1", + "side-channel": "^1.1.1" + }, + "engines": { + "node": ">=0.6" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/quick-format-unescaped": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/quick-format-unescaped/-/quick-format-unescaped-4.0.4.tgz", + "integrity": "sha512-tYC1Q1hgyRuHgloV/YXs2w15unPVh8qfu/qCTfhTYamaw7fyhumKa2yGpdSo87vY32rIclj+4fWYQXUMs9EHvg==", + "license": "MIT" + }, + "node_modules/range-parser": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz", + "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/raw-body": { + "version": "2.5.3", + "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.5.3.tgz", + "integrity": "sha512-s4VSOf6yN0rvbRZGxs8Om5CWj6seneMwK3oDb4lWDH0UPhWcxwOWw5+qk24bxq87szX1ydrwylIOp2uG1ojUpA==", + "license": "MIT", + "dependencies": { + "bytes": "~3.1.2", + "http-errors": "~2.0.1", + "iconv-lite": "~0.4.24", + "unpipe": "~1.0.0" + }, + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/real-require": { + "version": "0.2.0", + "resolved": "https://registry.npmjs.org/real-require/-/real-require-0.2.0.tgz", + "integrity": "sha512-57frrGM/OCTLqLOAh0mhVA9VBMHd+9U7Zb2THMGdBUoZVOtGbJzjxsYGDJ3A9AYYCP4hn6y1TVbaOfzWtm5GFg==", + "license": "MIT", + "engines": { + "node": ">= 12.13.0" + } + }, + "node_modules/safe-buffer": { + "version": "5.2.1", + "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz", + "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/feross" + }, + { + "type": "patreon", + "url": "https://www.patreon.com/feross" + }, + { + "type": "consulting", + "url": "https://feross.org/support" + } + ], + "license": "MIT" + }, + "node_modules/safe-stable-stringify": { + "version": "2.5.0", + "resolved": "https://registry.npmjs.org/safe-stable-stringify/-/safe-stable-stringify-2.5.0.tgz", + "integrity": "sha512-b3rppTKm9T+PsVCBEOUR46GWI7fdOs00VKZ1+9c1EWDaDMvjQc6tUwuFyIprgGgTcWoVHSKrU8H31ZHA2e0RHA==", + "license": "MIT", + "engines": { + "node": ">=10" + } + }, + "node_modules/safer-buffer": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz", + "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==", + "license": "MIT" + }, + "node_modules/semver": { + "version": "7.8.5", + "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.5.tgz", + "integrity": "sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==", + "license": "ISC", + "peer": true, + "bin": { + "semver": "bin/semver.js" + }, + "engines": { + "node": ">=10" + } + }, + "node_modules/send": { + "version": "0.19.2", + "resolved": "https://registry.npmjs.org/send/-/send-0.19.2.tgz", + "integrity": "sha512-VMbMxbDeehAxpOtWJXlcUS5E8iXh6QmN+BkRX1GARS3wRaXEEgzCcB10gTQazO42tpNIya8xIyNx8fll1OFPrg==", + "license": "MIT", + "dependencies": { + "debug": "2.6.9", + "depd": "2.0.0", + "destroy": "1.2.0", + "encodeurl": "~2.0.0", + "escape-html": "~1.0.3", + "etag": "~1.8.1", + "fresh": "~0.5.2", + "http-errors": "~2.0.1", + "mime": "1.6.0", + "ms": "2.1.3", + "on-finished": "~2.4.1", + "range-parser": "~1.2.1", + "statuses": "~2.0.2" + }, + "engines": { + "node": ">= 0.8.0" + } + }, + "node_modules/send/node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "license": "MIT" + }, + "node_modules/serve-static": { + "version": "1.16.3", + "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.16.3.tgz", + "integrity": "sha512-x0RTqQel6g5SY7Lg6ZreMmsOzncHFU7nhnRWkKgWuMTu5NN0DR5oruckMqRvacAN9d5w6ARnRBXl9xhDCgfMeA==", + "license": "MIT", + "dependencies": { + "encodeurl": "~2.0.0", + "escape-html": "~1.0.3", + "parseurl": "~1.3.3", + "send": "~0.19.1" + }, + "engines": { + "node": ">= 0.8.0" + } + }, + "node_modules/setprototypeof": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz", + "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==", + "license": "ISC" + }, + "node_modules/sharp": { + "version": "0.35.3", + "resolved": "https://registry.npmjs.org/sharp/-/sharp-0.35.3.tgz", + "integrity": "sha512-ej0zVHuZGHCiABXcNxeYhpRnPNPAcvbG8RMdBAhDAxLKkCRVSpK3Iyu7qbqw3JMzoj0REeM6f3tJLtVwl0023Q==", + "license": "Apache-2.0", + "peer": true, + "dependencies": { + "@img/colour": "^1.1.0", + "detect-libc": "^2.1.2", + "semver": "^7.8.5" + }, + "engines": { + "node": ">=20.9.0" + }, + "funding": { + "url": "https://opencollective.com/libvips" + }, + "optionalDependencies": { + "@img/sharp-darwin-arm64": "0.35.3", + "@img/sharp-darwin-x64": "0.35.3", + "@img/sharp-freebsd-wasm32": "0.35.3", + "@img/sharp-libvips-darwin-arm64": "1.3.2", + "@img/sharp-libvips-darwin-x64": "1.3.2", + "@img/sharp-libvips-linux-arm": "1.3.2", + "@img/sharp-libvips-linux-arm64": "1.3.2", + "@img/sharp-libvips-linux-ppc64": "1.3.2", + "@img/sharp-libvips-linux-riscv64": "1.3.2", + "@img/sharp-libvips-linux-s390x": "1.3.2", + "@img/sharp-libvips-linux-x64": "1.3.2", + "@img/sharp-libvips-linuxmusl-arm64": "1.3.2", + "@img/sharp-libvips-linuxmusl-x64": "1.3.2", + "@img/sharp-linux-arm": "0.35.3", + "@img/sharp-linux-arm64": "0.35.3", + "@img/sharp-linux-ppc64": "0.35.3", + "@img/sharp-linux-riscv64": "0.35.3", + "@img/sharp-linux-s390x": "0.35.3", + "@img/sharp-linux-x64": "0.35.3", + "@img/sharp-linuxmusl-arm64": "0.35.3", + "@img/sharp-linuxmusl-x64": "0.35.3", + "@img/sharp-webcontainers-wasm32": "0.35.3", + "@img/sharp-win32-arm64": "0.35.3", + "@img/sharp-win32-ia32": "0.35.3", + "@img/sharp-win32-x64": "0.35.3" + }, + "peerDependenciesMeta": { + "@types/node": { + "optional": true + } + } + }, + "node_modules/side-channel": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.1.tgz", + "integrity": "sha512-6x6dK6zJdpTzF4sQeNYxwtvBzf6Eg4GtlesS94HOvTudUeyK2WXAaIfmDgsyslYrRBeFIlsi54AYsFGUuhmvrQ==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "object-inspect": "^1.13.4", + "side-channel-list": "^1.0.1", + "side-channel-map": "^1.0.1", + "side-channel-weakmap": "^1.0.2" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-list": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.1.tgz", + "integrity": "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==", + "license": "MIT", + "dependencies": { + "es-errors": "^1.3.0", + "object-inspect": "^1.13.4" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-map": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/side-channel-map/-/side-channel-map-1.0.1.tgz", + "integrity": "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==", + "license": "MIT", + "dependencies": { + "call-bound": "^1.0.2", + "es-errors": "^1.3.0", + "get-intrinsic": "^1.2.5", + "object-inspect": "^1.13.3" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/side-channel-weakmap": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz", + "integrity": "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==", + "license": "MIT", + "dependencies": { + "call-bound": "^1.0.2", + "es-errors": "^1.3.0", + "get-intrinsic": "^1.2.5", + "object-inspect": "^1.13.3", + "side-channel-map": "^1.0.1" + }, + "engines": { + "node": ">= 0.4" + }, + "funding": { + "url": "https://github.com/sponsors/ljharb" + } + }, + "node_modules/sonic-boom": { + "version": "4.2.1", + "resolved": "https://registry.npmjs.org/sonic-boom/-/sonic-boom-4.2.1.tgz", + "integrity": "sha512-w6AxtubXa2wTXAUsZMMWERrsIRAdrK0Sc+FUytWvYAhBJLyuI4llrMIC1DtlNSdI99EI86KZum2MMq3EAZlF9Q==", + "license": "MIT", + "dependencies": { + "atomic-sleep": "^1.0.0" + } + }, + "node_modules/split2": { + "version": "4.2.0", + "resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz", + "integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==", + "license": "ISC", + "engines": { + "node": ">= 10.x" + } + }, + "node_modules/statuses": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz", + "integrity": "sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/strtok3": { + "version": "10.3.5", + "resolved": "https://registry.npmjs.org/strtok3/-/strtok3-10.3.5.tgz", + "integrity": "sha512-ki4hZQfh5rX0QDLLkOCj+h+CVNkqmp/CMf8v8kZpkNVK6jGQooMytqzLZYUVYIZcFZ6yDB70EfD8POcFXiF5oA==", + "license": "MIT", + "dependencies": { + "@tokenizer/token": "^0.3.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Borewit" + } + }, + "node_modules/thread-stream": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/thread-stream/-/thread-stream-3.2.0.tgz", + "integrity": "sha512-zLBvqpwr4Esa0kRjcrzGU6zL25lePWaCLMx0RQFrmteozIfeNdaMLpG5U7PeHzvlFkAWaRKA9/KVW4F60iB+qw==", + "license": "MIT", + "dependencies": { + "real-require": "^0.2.0" + } + }, + "node_modules/toidentifier": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz", + "integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==", + "license": "MIT", + "engines": { + "node": ">=0.6" + } + }, + "node_modules/token-types": { + "version": "6.1.2", + "resolved": "https://registry.npmjs.org/token-types/-/token-types-6.1.2.tgz", + "integrity": "sha512-dRXchy+C0IgK8WPC6xvCHFRIWYUbqqdEIKPaKo/AcTUNzwLTK6AH7RjdLWsEZcAN/TBdtfUw3PYEgPr5VPr6ww==", + "license": "MIT", + "dependencies": { + "@borewit/text-codec": "^0.2.1", + "@tokenizer/token": "^0.3.0", + "ieee754": "^1.2.1" + }, + "engines": { + "node": ">=14.16" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Borewit" + } + }, + "node_modules/tslib": { + "version": "2.8.1", + "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", + "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", + "license": "0BSD" + }, + "node_modules/type-is": { + "version": "1.6.18", + "resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz", + "integrity": "sha512-TkRKr9sUTxEH8MdfuCSP7VizJyzRNMjj2J2do2Jr3Kym598JVdEksuzPQCnlFPW4ky9Q+iA+ma9BGm06XQBy8g==", + "license": "MIT", + "dependencies": { + "media-typer": "0.3.0", + "mime-types": "~2.1.24" + }, + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/type-is/node_modules/media-typer": { + "version": "0.3.0", + "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz", + "integrity": "sha512-dq+qelQ9akHpcOl/gUVRTxVIOkAJ1wR3QAvb4RsVjS8oVoFjDGTc679wJYmUmknUF5HwMLOgb5O+a3KxfWapPQ==", + "license": "MIT", + "engines": { + "node": ">= 0.6" + } + }, + "node_modules/uint8array-extras": { + "version": "1.5.0", + "resolved": "https://registry.npmjs.org/uint8array-extras/-/uint8array-extras-1.5.0.tgz", + "integrity": "sha512-rvKSBiC5zqCCiDZ9kAOszZcDvdAHwwIKJG33Ykj43OKcWsnmcBRL09YTU4nOeHZ8Y2a7l1MgTd08SBe9A8Qj6A==", + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/undici-types": { + "version": "8.3.0", + "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-8.3.0.tgz", + "integrity": "sha512-j375ScV60dom+YkPFIfTLcOiPxkN/buHz5GobjLhixFuANaNs3C9l4GmrWqejgXWJ7BbJcFYpTEUkS1Ge8bpZQ==", + "license": "MIT" + }, + "node_modules/unpipe": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz", + "integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/utils-merge": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/utils-merge/-/utils-merge-1.0.1.tgz", + "integrity": "sha512-pMZTvIkT1d+TFGvDOqodOclx0QWkkgi6Tdoa8gC8ffGAAqz9pzPTZWAybbsHHoED/ztMtkv/VoYTYyShUn81hA==", + "license": "MIT", + "engines": { + "node": ">= 0.4.0" + } + }, + "node_modules/vary": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz", + "integrity": "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg==", + "license": "MIT", + "engines": { + "node": ">= 0.8" + } + }, + "node_modules/whatsapp-rust-bridge": { + "version": "0.5.4", + "resolved": "https://registry.npmjs.org/whatsapp-rust-bridge/-/whatsapp-rust-bridge-0.5.4.tgz", + "integrity": "sha512-yYO1qSs0Fe7tGtnxOFHomocUD6IZtoAgmA4oDFyGIRZ67D3QZk3w7swA6XXFXNQngiyrg2k7tul6IrM3eUFh7A==", + "license": "MIT" + }, + "node_modules/win-guid": { + "version": "0.2.1", + "resolved": "https://registry.npmjs.org/win-guid/-/win-guid-0.2.1.tgz", + "integrity": "sha512-gEIQU4mkgl2OPeoNrWflcJFJ3Ae2BPd4eCsHHA/XikslkIVms/nHhvnvzIZV7VLmBvtFlDOzLt9rrZT+n6D67A==", + "license": "MIT" + }, + "node_modules/ws": { + "version": "8.21.0", + "resolved": "https://registry.npmjs.org/ws/-/ws-8.21.0.tgz", + "integrity": "sha512-Vsp28b7DRcimFQvrqu2Wek3z1iYxDCWqHYB8Qsnk/S4RfaCQzPGPyBNuVjJV3cd6UiKtUtp6sNM77gWvzcCH+g==", + "license": "MIT", + "engines": { + "node": ">=10.0.0" + }, + "peerDependencies": { + "bufferutil": "^4.0.1", + "utf-8-validate": ">=5.0.2" + }, + "peerDependenciesMeta": { + "bufferutil": { + "optional": true + }, + "utf-8-validate": { + "optional": true + } + } + } + } +} diff --git a/flocks/channel/builtin/whatsapp/bridge/package.json b/flocks/channel/builtin/whatsapp/bridge/package.json new file mode 100644 index 000000000..ffb826619 --- /dev/null +++ b/flocks/channel/builtin/whatsapp/bridge/package.json @@ -0,0 +1,20 @@ +{ + "name": "flocks-whatsapp-bridge", + "version": "1.0.0", + "private": true, + "type": "module", + "description": "Local WhatsApp bridge for Flocks using Baileys", + "scripts": { + "start": "node bridge.js" + }, + "dependencies": { + "@hapi/boom": "^10.0.1", + "@whiskeysockets/baileys": "7.0.0-rc13", + "express": "^4.21.2", + "pino": "^9.6.0" + }, + "overrides": { + "protobufjs": "^7.5.5" + } +} + diff --git a/flocks/channel/builtin/whatsapp/bridge_runtime.py b/flocks/channel/builtin/whatsapp/bridge_runtime.py new file mode 100644 index 000000000..4387beed4 --- /dev/null +++ b/flocks/channel/builtin/whatsapp/bridge_runtime.py @@ -0,0 +1,64 @@ +"""Runtime helpers for the bundled WhatsApp bridge.""" + +from __future__ import annotations + +import asyncio +import hashlib +import json +import os +import subprocess +from pathlib import Path +from typing import Any + +from .config import coerce_int + + +def file_hash(path: Path) -> str: + try: + return hashlib.sha256(path.read_bytes()).hexdigest()[:16] + except OSError: + return "" + + +def config_hash(values: dict[str, Any]) -> str: + payload = json.dumps(values, sort_keys=True, separators=(",", ":"), default=str) + return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16] + + +async def ensure_bridge_deps(bridge_dir: Path) -> None: + package_json = bridge_dir / "package.json" + if not package_json.exists(): + raise RuntimeError(f"WhatsApp bridge package.json not found: {package_json}") + node_modules = bridge_dir / "node_modules" + stamp = node_modules / ".flocks-pkg-hash" + pkg_hash = file_hash(package_json) + deps_fresh = False + if node_modules.exists(): + try: + deps_fresh = stamp.read_text(encoding="utf-8").strip() == pkg_hash and bool(pkg_hash) + except OSError: + deps_fresh = False + if deps_fresh: + return + + from .config import find_executable + + npm = find_executable("npm") + if not npm: + raise RuntimeError("npm is required to install WhatsApp bridge dependencies") + install_cmd = [npm, "ci", "--silent"] if (bridge_dir / "package-lock.json").exists() else [npm, "install", "--silent"] + result = await asyncio.to_thread( + subprocess.run, + install_cmd, + cwd=str(bridge_dir), + capture_output=True, + text=True, + timeout=coerce_int(os.getenv("FLOCKS_WHATSAPP_NPM_INSTALL_TIMEOUT"), 300), + ) + if result.returncode != 0: + raise RuntimeError(f"WhatsApp bridge npm install failed: {result.stderr or result.stdout}") + if pkg_hash: + try: + stamp.write_text(pkg_hash, encoding="utf-8") + except OSError: + pass diff --git a/flocks/channel/builtin/whatsapp/channel.py b/flocks/channel/builtin/whatsapp/channel.py new file mode 100644 index 000000000..aba0b8dcd --- /dev/null +++ b/flocks/channel/builtin/whatsapp/channel.py @@ -0,0 +1,637 @@ +"""WhatsApp ChannelPlugin implementation.""" + +from __future__ import annotations + +import asyncio +import json +import os +import signal +import secrets +import subprocess +import time +from pathlib import Path +from typing import Any, Awaitable, Callable, Optional +from urllib.parse import unquote, urlparse + +import aiohttp + +from flocks.channel.base import ( + ChannelCapabilities, + ChannelMeta, + ChannelPlugin, + ChatType, + DeliveryResult, + InboundMessage, + OutboundContext, +) +from flocks.utils.log import Log + +from .config import ( + DEFAULT_BRIDGE_PORT, + DEFAULT_MESSAGE_LIMIT, + DEFAULT_SEND_CHUNK_DELAY_MS, + DEFAULT_SEND_TIMEOUT_MS, + DEFAULT_TEXT_BATCH_DELAY_SECONDS, + VALID_DM_POLICIES, + VALID_GROUP_POLICIES, + VALID_GROUP_TRIGGERS, + VALID_MODES, + coerce_float, + coerce_int, + coerce_list, + coerce_str, + default_bridge_dir, + default_media_cache_dir, + default_session_path, + find_executable, + format_env_list, + matches_identifier, + parse_target, +) +from .bridge_runtime import config_hash, ensure_bridge_deps, file_hash +from .format import format_for_whatsapp +from .inbound import build_inbound_message + +log = Log.create(service="channel.whatsapp") + + +def _pid_exists(pid: int) -> bool: + try: + os.kill(pid, 0) + return True + except OSError: + return False + + +def _path_from_media_url(media_url: str) -> Optional[Path]: + parsed = urlparse(media_url) + if parsed.scheme == "file": + return Path(unquote(parsed.path)) + if parsed.scheme in {"", None}: + return Path(media_url) + return None + + +def _media_type_for_path(path: Path) -> str: + suffix = path.suffix.lower() + if suffix in {".jpg", ".jpeg", ".png", ".webp", ".gif"}: + return "image" + if suffix in {".mp4", ".mov", ".mkv", ".webm"}: + return "video" + if suffix in {".ogg", ".opus", ".mp3", ".wav", ".m4a"}: + return "audio" + return "document" + + +def _raw_reply_id(reply_to_id: Optional[str]) -> Optional[str]: + if not reply_to_id: + return None + value = coerce_str(reply_to_id) + if "@" in value and ":" in value: + return value.rsplit(":", 1)[-1] or value + return value + + +class WhatsAppChannel(ChannelPlugin): + """WhatsApp personal-account channel via a local Baileys bridge.""" + + def __init__(self) -> None: + super().__init__() + self._account_id = "default" + self._bridge_port = DEFAULT_BRIDGE_PORT + self._bridge_dir = default_bridge_dir() + self._session_path = default_session_path() + self._media_cache_dir = default_media_cache_dir() + self._mode = "bot" + self._dm_policy = "allowlist" + self._group_policy = "disabled" + self._group_trigger = "mention" + self._allow_from: list[str] = [] + self._group_allow_from: list[str] = [] + self._text_batch_delay = DEFAULT_TEXT_BATCH_DELAY_SECONDS + self._send_chunk_delay_ms = DEFAULT_SEND_CHUNK_DELAY_MS + self._send_timeout_ms = DEFAULT_SEND_TIMEOUT_MS + self._reply_prefix = "" + self._http: Optional[aiohttp.ClientSession] = None + self._bridge_token = secrets.token_urlsafe(32) + self._bridge_process: Optional[subprocess.Popen] = None + self._poll_task: Optional[asyncio.Task] = None + self._bridge_log_fh = None + self._bridge_log_path: Optional[Path] = None + self._pending_text: dict[str, InboundMessage] = {} + self._pending_tasks: dict[str, asyncio.Task] = {} + self._shutting_down = False + + def meta(self) -> ChannelMeta: + return ChannelMeta( + id="whatsapp", + label="WhatsApp", + aliases=["wa"], + order=45, + ) + + def capabilities(self) -> ChannelCapabilities: + return ChannelCapabilities( + chat_types=[ChatType.DIRECT, ChatType.GROUP], + media=True, + threads=False, + reactions=False, + edit=False, + rich_text=True, + ) + + def validate_config(self, config: dict) -> Optional[str]: + mode = coerce_str(config.get("mode") or "bot") + if mode not in VALID_MODES: + return "WhatsApp mode must be 'bot' or 'self-chat'" + dm_policy = coerce_str(config.get("dmPolicy") or "allowlist") + if dm_policy not in VALID_DM_POLICIES: + return "WhatsApp dmPolicy must be 'open', 'allowlist', or 'disabled'" + group_policy = coerce_str(config.get("groupPolicy") or "disabled") + if group_policy not in VALID_GROUP_POLICIES: + return "WhatsApp groupPolicy must be 'open', 'allowlist', or 'disabled'" + group_trigger = coerce_str(config.get("groupTrigger") or "mention") + if group_trigger not in VALID_GROUP_TRIGGERS: + return "WhatsApp groupTrigger must be 'mention' or 'all'" + bridge_port = coerce_int(config.get("bridgePort"), DEFAULT_BRIDGE_PORT) + if bridge_port <= 0 or bridge_port > 65535: + return "WhatsApp bridgePort must be between 1 and 65535" + if not find_executable("node"): + return "WhatsApp channel requires Node.js" + session_path = Path(coerce_str(config.get("sessionPath")) or default_session_path()).expanduser() + if not (session_path / "creds.json").exists(): + return "WhatsApp is not paired yet. Use QR pairing before enabling the channel." + return None + + @property + def text_chunk_limit(self) -> int: + return DEFAULT_MESSAGE_LIMIT + + @property + def rate_limit(self) -> tuple[float, int]: + return (3.0, 2) + + def format_message(self, text: str, format_hint: str = "markdown") -> str: + if format_hint == "plain": + return text + return format_for_whatsapp(text) + + def normalize_target(self, raw: str) -> Optional[str]: + target = parse_target(raw) + return target or None + + def target_hint(self) -> str: + return " / @s.whatsapp.net / @g.us" + + async def start( + self, + config: dict, + on_message: Callable[[InboundMessage], Awaitable[None]], + abort_event: Optional[asyncio.Event] = None, + ) -> None: + self._config = config + self._on_message = on_message + self._load_config(config) + self._shutting_down = False + + await self._ensure_bridge_deps() + await self._start_bridge() + self._http = aiohttp.ClientSession() + self.mark_connected() + log.info("whatsapp.connected", { + "bridge_port": self._bridge_port, + "session_path": str(self._session_path), + }) + + _abort = abort_event if abort_event is not None else asyncio.Event() + self._poll_task = asyncio.create_task(self._poll_messages(_abort)) + abort_task = asyncio.create_task(_abort.wait()) + try: + done, _pending = await asyncio.wait( + {abort_task, self._poll_task}, + return_when=asyncio.FIRST_COMPLETED, + ) + if self._poll_task in done: + exc = self._poll_task.exception() + if exc: + raise exc + finally: + abort_task.cancel() + await asyncio.gather(abort_task, return_exceptions=True) + await self.stop() + + async def stop(self) -> None: + self._shutting_down = True + for task in list(self._pending_tasks.values()): + task.cancel() + if self._pending_tasks: + await asyncio.gather(*self._pending_tasks.values(), return_exceptions=True) + self._pending_tasks.clear() + self._pending_text.clear() + + if self._poll_task and not self._poll_task.done(): + self._poll_task.cancel() + await asyncio.gather(self._poll_task, return_exceptions=True) + self._poll_task = None + + if self._http is not None: + await self._http.close() + self._http = None + + if self._bridge_process is not None: + proc = self._bridge_process + if proc.poll() is None: + try: + if os.name == "nt": + proc.terminate() + else: + os.killpg(proc.pid, signal.SIGTERM) + await asyncio.to_thread(proc.wait, 5) + except Exception: + try: + proc.kill() + except Exception: + pass + self._bridge_process = None + + if self._bridge_log_fh: + try: + self._bridge_log_fh.close() + except Exception: + pass + self._bridge_log_fh = None + self.mark_disconnected() + + async def send_text(self, ctx: OutboundContext) -> DeliveryResult: + if self._http is None: + return DeliveryResult( + channel_id="whatsapp", + message_id="", + success=False, + error="WhatsApp bridge is not connected", + retryable=True, + ) + chat_id = parse_target(ctx.to) + if not chat_id: + return DeliveryResult( + channel_id="whatsapp", + message_id="", + success=False, + error="Invalid WhatsApp target", + ) + + try: + async with self._http.post( + self._url("/send"), + json={ + "chatId": chat_id, + "message": ctx.text, + "replyTo": _raw_reply_id(ctx.reply_to_id), + }, + headers=self._bridge_headers(), + timeout=aiohttp.ClientTimeout(total=max(self._send_timeout_ms / 1000, 1)), + ) as resp: + data = await resp.json(content_type=None) + if resp.status >= 400 or data.get("success") is False: + return DeliveryResult( + channel_id="whatsapp", + message_id="", + success=False, + error=str(data.get("error") or f"HTTP {resp.status}"), + retryable=resp.status >= 500, + ) + except Exception as exc: + return DeliveryResult( + channel_id="whatsapp", + message_id="", + success=False, + error=f"WhatsApp send failed: {exc}", + retryable=True, + ) + + self.record_message() + return DeliveryResult( + channel_id="whatsapp", + message_id=coerce_str(data.get("messageId")), + chat_id=chat_id, + success=True, + ) + + async def send_media(self, ctx: OutboundContext) -> DeliveryResult: + if not ctx.media_url: + return await self.send_text(ctx) + if self._http is None: + return DeliveryResult( + channel_id="whatsapp", + message_id="", + success=False, + error="WhatsApp bridge is not connected", + retryable=True, + ) + chat_id = parse_target(ctx.to) + path = _path_from_media_url(ctx.media_url) + if not chat_id: + return DeliveryResult(channel_id="whatsapp", message_id="", success=False, error="Invalid WhatsApp target") + if path is None or not path.is_file(): + return DeliveryResult(channel_id="whatsapp", message_id="", success=False, error="WhatsApp media must be a local file") + + try: + async with self._http.post( + self._url("/send-media"), + json={ + "chatId": chat_id, + "filePath": str(path), + "mediaType": _media_type_for_path(path), + "caption": ctx.text or "", + "replyTo": _raw_reply_id(ctx.reply_to_id), + }, + headers=self._bridge_headers(), + timeout=aiohttp.ClientTimeout(total=max(self._send_timeout_ms / 1000, 1)), + ) as resp: + data = await resp.json(content_type=None) + if resp.status >= 400 or data.get("success") is False: + return DeliveryResult( + channel_id="whatsapp", + message_id="", + success=False, + error=str(data.get("error") or f"HTTP {resp.status}"), + retryable=resp.status >= 500, + ) + except Exception as exc: + return DeliveryResult( + channel_id="whatsapp", + message_id="", + success=False, + error=f"WhatsApp send_media failed: {exc}", + retryable=True, + ) + + self.record_message() + return DeliveryResult( + channel_id="whatsapp", + message_id=coerce_str(data.get("messageId")), + chat_id=chat_id, + success=True, + ) + + def _load_config(self, config: dict) -> None: + self._account_id = coerce_str(config.get("accountId")) or "default" + self._bridge_port = coerce_int(config.get("bridgePort"), DEFAULT_BRIDGE_PORT) + self._bridge_dir = Path(coerce_str(config.get("bridgeDir")) or default_bridge_dir()).expanduser() + self._session_path = Path(coerce_str(config.get("sessionPath")) or default_session_path()).expanduser() + self._media_cache_dir = Path(coerce_str(config.get("mediaCacheDir")) or default_media_cache_dir()).expanduser() + self._mode = coerce_str(config.get("mode") or "bot") + self._dm_policy = coerce_str(config.get("dmPolicy") or "allowlist") + self._group_policy = coerce_str(config.get("groupPolicy") or "disabled") + self._group_trigger = coerce_str(config.get("groupTrigger") or "mention") + self._allow_from = coerce_list(config.get("allowFrom")) + self._group_allow_from = coerce_list(config.get("groupAllowFrom")) + self._text_batch_delay = coerce_float(config.get("textBatchDelaySeconds"), DEFAULT_TEXT_BATCH_DELAY_SECONDS) + self._send_chunk_delay_ms = coerce_int(config.get("sendChunkDelayMs"), DEFAULT_SEND_CHUNK_DELAY_MS) + self._send_timeout_ms = coerce_int(config.get("sendTimeoutMs"), DEFAULT_SEND_TIMEOUT_MS) + self._reply_prefix = coerce_str(config.get("replyPrefix")) + + async def _ensure_bridge_deps(self) -> None: + await ensure_bridge_deps(self._bridge_dir) + + async def _start_bridge(self) -> None: + node = find_executable("node") + if not node: + raise RuntimeError("Node.js is required for WhatsApp channel") + script = self._bridge_dir / "bridge.js" + if not script.exists(): + raise RuntimeError(f"WhatsApp bridge script not found: {script}") + self._session_path.mkdir(parents=True, exist_ok=True) + self._media_cache_dir.mkdir(parents=True, exist_ok=True) + + if await self._reuse_or_stop_existing_bridge(script): + return + + self._bridge_log_path = self._session_path.parent / "bridge.log" + self._bridge_log_path.parent.mkdir(parents=True, exist_ok=True) + self._bridge_log_fh = open(self._bridge_log_path, "a", encoding="utf-8") + env = os.environ.copy() + env["FLOCKS_WHATSAPP_MEDIA_DIR"] = str(self._media_cache_dir) + env["FLOCKS_WHATSAPP_BRIDGE_TOKEN"] = self._bridge_token + env["FLOCKS_WHATSAPP_CONFIG_HASH"] = self._bridge_config_hash() + env["FLOCKS_WHATSAPP_ALLOWED_MEDIA_ROOTS"] = os.pathsep.join(self._allowed_media_roots()) + env["FLOCKS_WHATSAPP_MODE"] = self._mode + env["FLOCKS_WHATSAPP_DM_POLICY"] = self._dm_policy + env["FLOCKS_WHATSAPP_ALLOWED_USERS"] = format_env_list(self._allow_from) + env["FLOCKS_WHATSAPP_REPLY_PREFIX"] = self._reply_prefix + env["FLOCKS_WHATSAPP_CHUNK_DELAY_MS"] = str(self._send_chunk_delay_ms) + env["FLOCKS_WHATSAPP_SEND_TIMEOUT_MS"] = str(self._send_timeout_ms) + + self._bridge_process = subprocess.Popen( + [ + node, + str(script), + "--port", + str(self._bridge_port), + "--session", + str(self._session_path), + ], + cwd=str(self._bridge_dir), + stdout=self._bridge_log_fh, + stderr=self._bridge_log_fh, + start_new_session=(os.name != "nt"), + env=env, + ) + await self._wait_for_bridge(script) + + async def _reuse_or_stop_existing_bridge(self, script: Path) -> bool: + try: + async with aiohttp.ClientSession() as session: + async with session.get( + self._url("/health"), + headers=self._bridge_headers(), + timeout=aiohttp.ClientTimeout(total=2), + ) as resp: + data = await resp.json(content_type=None) + if resp.status == 200 and data.get("status") == "connected" and self._bridge_identity_matches(data, script): + self._bridge_process = None + return True + if resp.status == 200 and data.get("scriptHash") == file_hash(script): + if coerce_str(data.get("sessionPath")) != str(self._session_path): + raise RuntimeError( + f"WhatsApp bridge port {self._bridge_port} is already used by another session" + ) + except RuntimeError: + raise + except Exception: + pass + pid_file = self._session_path / "bridge.pid" + try: + pid = int(pid_file.read_text(encoding="utf-8").strip().splitlines()[0]) + if _pid_exists(pid): + os.kill(pid, signal.SIGTERM) + await asyncio.sleep(1) + except Exception: + pass + return False + + async def _wait_for_bridge(self, script: Path) -> None: + deadline = time.monotonic() + 30 + last_status = "unknown" + while time.monotonic() < deadline: + if self._bridge_process is not None and self._bridge_process.poll() is not None: + raise RuntimeError( + f"WhatsApp bridge exited early with code {self._bridge_process.returncode}; log: {self._bridge_log_path}" + ) + try: + async with aiohttp.ClientSession() as session: + async with session.get( + self._url("/health"), + headers=self._bridge_headers(), + timeout=aiohttp.ClientTimeout(total=2), + ) as resp: + data = await resp.json(content_type=None) + last_status = coerce_str(data.get("status")) or last_status + if resp.status == 200 and self._bridge_identity_matches(data, script): + if data.get("status") == "connected": + if self._bridge_process is not None: + try: + (self._session_path / "bridge.pid").write_text( + str(self._bridge_process.pid), + encoding="utf-8", + ) + except OSError: + pass + return + except Exception: + pass + await asyncio.sleep(1) + raise RuntimeError(f"WhatsApp bridge did not connect in time (last status: {last_status}); log: {self._bridge_log_path}") + + async def _poll_messages(self, abort_event: asyncio.Event) -> None: + consecutive_failures = 0 + while not abort_event.is_set() and not self._shutting_down: + if self._http is None: + return + if self._bridge_process is not None and self._bridge_process.poll() is not None: + raise RuntimeError(f"WhatsApp bridge exited with code {self._bridge_process.returncode}") + try: + async with self._http.get( + self._url("/messages"), + headers=self._bridge_headers(), + timeout=aiohttp.ClientTimeout(total=35), + ) as resp: + if resp.status != 200: + raise RuntimeError(f"WhatsApp bridge /messages returned HTTP {resp.status}") + events = await resp.json(content_type=None) + consecutive_failures = 0 + if isinstance(events, list): + for event in events: + await self._handle_bridge_event(event) + except asyncio.CancelledError: + break + except Exception as exc: + consecutive_failures += 1 + log.warning("whatsapp.poll.error", {"error": str(exc)}) + if consecutive_failures >= 3: + self.mark_disconnected(str(exc)) + raise RuntimeError(f"WhatsApp bridge polling failed {consecutive_failures} times: {exc}") from exc + try: + await asyncio.wait_for(abort_event.wait(), timeout=5) + except asyncio.TimeoutError: + pass + + async def _handle_bridge_event(self, event: Any) -> None: + if not isinstance(event, dict): + return + inbound = build_inbound_message(event, self._account_id) + if inbound is None or not self._is_allowed(inbound): + return + if inbound.chat_type == ChatType.GROUP and not self._passes_group_trigger(inbound): + return + await self._enqueue_or_dispatch(inbound) + + def _is_allowed(self, msg: InboundMessage) -> bool: + if msg.chat_type == ChatType.DIRECT: + if self._dm_policy == "disabled": + return False + if self._dm_policy == "open": + return True + return matches_identifier(msg.sender_id, self._allow_from) + if self._group_policy == "disabled": + return False + if self._group_policy == "open": + return True + return matches_identifier(msg.chat_id, self._group_allow_from) + + def _passes_group_trigger(self, msg: InboundMessage) -> bool: + if self._group_trigger == "all": + return True + return msg.mentioned + + async def _enqueue_or_dispatch(self, msg: InboundMessage) -> None: + if msg.media_url or self._text_batch_delay <= 0: + if self._on_message: + await self._on_message(msg) + return + key = f"{msg.account_id}:{msg.chat_id}" + existing = self._pending_text.get(key) + if existing is None: + self._pending_text[key] = msg + else: + existing.text = f"{existing.text}\n{msg.text}".strip() + existing.message_id = msg.message_id + existing.raw = msg.raw + prior = self._pending_tasks.get(key) + if prior and not prior.done(): + prior.cancel() + self._pending_tasks[key] = asyncio.create_task(self._flush_text_batch(key)) + + async def _flush_text_batch(self, key: str) -> None: + current = asyncio.current_task() + try: + await asyncio.sleep(self._text_batch_delay) + msg = self._pending_text.pop(key, None) + if msg and self._on_message: + await self._on_message(msg) + except asyncio.CancelledError: + pass + finally: + if self._pending_tasks.get(key) is current: + self._pending_tasks.pop(key, None) + + def _url(self, path: str) -> str: + return f"http://127.0.0.1:{self._bridge_port}{path}" + + def _bridge_headers(self) -> dict[str, str]: + return {"X-Flocks-Bridge-Token": self._bridge_token} + + def _bridge_config_hash(self) -> str: + return config_hash({ + "sessionPath": str(self._session_path), + "mediaDir": str(self._media_cache_dir), + "mode": self._mode, + "replyPrefix": self._reply_prefix, + "sendChunkDelayMs": self._send_chunk_delay_ms, + "sendTimeoutMs": self._send_timeout_ms, + }) + + def _bridge_identity_matches(self, data: dict[str, Any], script: Path) -> bool: + return ( + data.get("scriptHash") == file_hash(script) + and coerce_str(data.get("sessionPath")) == str(self._session_path) + and coerce_str(data.get("mediaDir")) == str(self._media_cache_dir) + and coerce_str(data.get("mode")) == self._mode + and coerce_str(data.get("configHash")) == self._bridge_config_hash() + ) + + def _allowed_media_roots(self) -> list[str]: + roots = [ + self._media_cache_dir, + Path.home() / ".flocks" / "workspace", + Path(os.getenv("TMPDIR") or "/tmp"), + ] + extra = coerce_list(self._config.get("mediaAllowedRoots") if hasattr(self, "_config") else None) + roots.extend(Path(item).expanduser() for item in extra) + resolved: list[str] = [] + for root in roots: + try: + resolved.append(str(root.resolve())) + except OSError: + resolved.append(str(root.expanduser())) + return list(dict.fromkeys(resolved)) diff --git a/flocks/channel/builtin/whatsapp/config.py b/flocks/channel/builtin/whatsapp/config.py new file mode 100644 index 000000000..25a2caf8f --- /dev/null +++ b/flocks/channel/builtin/whatsapp/config.py @@ -0,0 +1,128 @@ +"""Configuration helpers for the WhatsApp channel.""" + +from __future__ import annotations + +import os +import re +import shutil +from pathlib import Path +from typing import Any, Optional + +DEFAULT_BRIDGE_PORT = 3100 +DEFAULT_TEXT_BATCH_DELAY_SECONDS = 3.0 +DEFAULT_SEND_CHUNK_DELAY_MS = 300 +DEFAULT_MESSAGE_LIMIT = 4096 +DEFAULT_SEND_TIMEOUT_MS = 60000 + +VALID_MODES = {"bot", "self-chat"} +VALID_DM_POLICIES = {"open", "allowlist", "disabled"} +VALID_GROUP_POLICIES = {"open", "allowlist", "disabled"} +VALID_GROUP_TRIGGERS = {"mention", "all"} + +_PHONE_RE = re.compile(r"^\+?\d{6,20}$") +_JID_RE = re.compile(r"^[\w.+-]+@(?:s\.whatsapp\.net|g\.us|lid|broadcast|newsletter)$") + + +def coerce_str(value: Any) -> str: + if value is None: + return "" + return str(value).strip() + + +def coerce_int(value: Any, default: int) -> int: + if isinstance(value, bool): + return default + try: + parsed = int(value) + except (TypeError, ValueError): + return default + return parsed + + +def coerce_float(value: Any, default: float) -> float: + try: + parsed = float(value) + except (TypeError, ValueError): + return default + if parsed < 0: + return default + return parsed + + +def coerce_list(value: Any) -> list[str]: + if value is None: + return [] + if isinstance(value, list): + return [str(item).strip() for item in value if str(item).strip()] + return [part.strip() for part in str(value).split(",") if part.strip()] + + +def default_state_dir() -> Path: + raw = os.getenv("FLOCKS_WHATSAPP_DATA_DIR") + if raw: + return Path(raw).expanduser() + return Path.home() / ".flocks" / "workspace" / "channels" / "whatsapp" + + +def default_session_path() -> Path: + return default_state_dir() / "session" + + +def default_media_cache_dir() -> Path: + return default_state_dir() / "media" + + +def default_bridge_dir() -> Path: + return Path(__file__).resolve().parent / "bridge" + + +def find_executable(name: str) -> Optional[str]: + return shutil.which(name) + + +def normalize_jid(value: str) -> str: + raw = coerce_str(value) + if not raw: + return "" + if ":" in raw and "@" in raw: + raw = raw.replace(":", "@", 1) + if _JID_RE.fullmatch(raw): + return raw + phone = raw.lstrip("+").replace(" ", "") + if _PHONE_RE.fullmatch(phone): + return f"{phone}@s.whatsapp.net" + return raw + + +def strip_jid(value: str) -> str: + raw = normalize_jid(value) + if "@" in raw: + return raw.split("@", 1)[0].split(":", 1)[0] + return raw.lstrip("+") + + +def parse_target(raw: str) -> str: + value = coerce_str(raw) + for prefix in ("whatsapp:", "wa:", "user:", "group:"): + if value.lower().startswith(prefix): + value = value[len(prefix):].strip() + break + return normalize_jid(value) + + +def matches_identifier(candidate: str, allowed: list[str]) -> bool: + if not allowed: + return False + if "*" in allowed: + return True + aliases = {candidate, normalize_jid(candidate), strip_jid(candidate)} + for entry in allowed: + aliases_for_entry = {entry, normalize_jid(entry), strip_jid(entry)} + if aliases & aliases_for_entry: + return True + return False + + +def format_env_list(items: list[str]) -> str: + return ",".join(item for item in items if item) + diff --git a/flocks/channel/builtin/whatsapp/format.py b/flocks/channel/builtin/whatsapp/format.py new file mode 100644 index 000000000..8570d81b5 --- /dev/null +++ b/flocks/channel/builtin/whatsapp/format.py @@ -0,0 +1,23 @@ +"""WhatsApp text formatting helpers.""" + +from __future__ import annotations + +import re + + +_LINK_RE = re.compile(r"\[([^\]]+)\]\((https?://[^)\s]+)\)") + + +def format_for_whatsapp(text: str) -> str: + """Convert generic Markdown into WhatsApp-friendly plain markdown. + + WhatsApp supports lightweight emphasis but not Markdown links. Keeping + code fences intact gives readable monospace blocks on most clients. + """ + if not text: + return "" + out = _LINK_RE.sub(r"\1: \2", text) + out = re.sub(r"(? tuple[Optional[str], Optional[str]]: + urls = event.get("mediaUrls") + if isinstance(urls, list) and urls: + url = coerce_str(urls[0]) + if url: + return url, coerce_str(event.get("mime")) or None + url = coerce_str(event.get("mediaUrl")) + if url: + return url, coerce_str(event.get("mime")) or None + return None, None + + +def _is_group_jid(jid: str) -> bool: + return jid.endswith("@g.us") + + +def _resolve_mentioned(event: dict[str, Any], text: str) -> tuple[bool, str]: + if event.get("mentioned") is True: + mention_text = coerce_str(event.get("mentionText")) or text + return True, mention_text.strip() + if event.get("isGroup") and event.get("isReplyToBot"): + return True, text.strip() + return False, "" + + +def build_inbound_message(event: dict[str, Any], account_id: str = "default") -> Optional[InboundMessage]: + chat_id = normalize_jid(coerce_str(event.get("chatId"))) + sender_id = normalize_jid(coerce_str(event.get("senderId")) or chat_id) + message_id = coerce_str(event.get("messageId")) + if not chat_id or not message_id: + return None + + text = coerce_str(event.get("body") or event.get("text")) + media_url, media_mime = _first_media(event) + if not text and not media_url: + return None + + is_group = bool(event.get("isGroup")) or _is_group_jid(chat_id) + chat_type = ChatType.GROUP if is_group else ChatType.DIRECT + sender_name = coerce_str(event.get("senderName")) or None + + mentioned, mention_text = _resolve_mentioned(event, text) + + return InboundMessage( + channel_id="whatsapp", + account_id=account_id, + message_id=f"{chat_id}:{message_id}", + sender_id=strip_jid(sender_id) or sender_id, + sender_name=sender_name, + chat_id=chat_id, + chat_type=chat_type, + text=text, + media_url=media_url, + media_mime=media_mime, + reply_to_id=coerce_str(event.get("quotedMessageId")) or None, + thread_id=None, + mentioned=mentioned, + mention_text=mention_text, + raw=event, + ) + diff --git a/flocks/channel/builtin/whatsapp/pairing.py b/flocks/channel/builtin/whatsapp/pairing.py new file mode 100644 index 000000000..99d40128f --- /dev/null +++ b/flocks/channel/builtin/whatsapp/pairing.py @@ -0,0 +1,184 @@ +"""QR pairing helpers for the WhatsApp channel.""" + +from __future__ import annotations + +import asyncio +import json +import os +import time +import uuid +from dataclasses import dataclass, field +from pathlib import Path +from typing import Optional + +from flocks.utils.log import Log + +from .bridge_runtime import ensure_bridge_deps +from .config import default_bridge_dir, default_session_path, find_executable + +log = Log.create(service="channel.whatsapp.pairing") + + +@dataclass +class PairingSession: + id: str + session_path: Path + process: asyncio.subprocess.Process + qr: Optional[str] = None + status: str = "starting" + error: Optional[str] = None + user: Optional[dict] = None + created_at: float = field(default_factory=time.time) + finished_at: Optional[float] = None + _reader_task: Optional[asyncio.Task] = field(default=None, repr=False) + + +_sessions: dict[str, PairingSession] = {} +_active_session_paths: set[str] = set() + + +def _session_key(path: Path) -> str: + try: + return str(path.resolve()) + except OSError: + return str(path.expanduser()) + + +def _pid_exists(pid: int) -> bool: + try: + os.kill(pid, 0) + return True + except OSError: + return False + + +def _cleanup_pairings(ttl_seconds: int = 600) -> None: + now = time.time() + stale = [ + pairing_id + for pairing_id, pairing in _sessions.items() + if pairing.finished_at is not None and now - pairing.finished_at > ttl_seconds + ] + for pairing_id in stale: + _sessions.pop(pairing_id, None) + + +def _release_session(pairing: PairingSession) -> None: + _active_session_paths.discard(_session_key(pairing.session_path)) + pairing.finished_at = pairing.finished_at or time.time() + + +async def _read_pair_output(pairing: PairingSession) -> None: + assert pairing.process.stdout is not None + while True: + line = await pairing.process.stdout.readline() + if not line: + break + raw = line.decode("utf-8", errors="replace").strip() + if not raw: + continue + try: + event = json.loads(raw) + except json.JSONDecodeError: + continue + name = str(event.get("event") or "") + if name == "qr": + pairing.qr = str(event.get("qr") or "") + pairing.status = "qr" + elif name == "connected": + pairing.user = event.get("user") if isinstance(event.get("user"), dict) else None + pairing.status = "connected" + elif name == "error": + pairing.error = str(event.get("error") or "WhatsApp pairing failed") + pairing.status = "error" + elif name == "disconnected" and pairing.status not in {"qr", "connected"}: + pairing.status = "starting" + + code = await pairing.process.wait() + if pairing.status == "connected" and code == 0: + pairing.status = "complete" + elif pairing.status not in {"complete", "connected", "error", "cancelled"}: + pairing.status = "error" + pairing.error = f"WhatsApp pairing bridge exited with code {code}" + _release_session(pairing) + + +async def start_pairing( + *, + session_path: Optional[str] = None, + bridge_dir: Optional[str] = None, +) -> PairingSession: + _cleanup_pairings() + node = find_executable("node") + if not node: + raise RuntimeError("Node.js is required for WhatsApp QR pairing") + + bridge_root = Path(bridge_dir).expanduser() if bridge_dir else default_bridge_dir() + script = bridge_root / "bridge.js" + if not script.exists(): + raise RuntimeError(f"WhatsApp bridge script not found: {script}") + await ensure_bridge_deps(bridge_root) + + sess = Path(session_path).expanduser() if session_path else default_session_path() + sess.mkdir(parents=True, exist_ok=True) + key = _session_key(sess) + if key in _active_session_paths: + raise RuntimeError("WhatsApp pairing is already running for this session") + pid_file = sess / "bridge.pid" + try: + pid = int(pid_file.read_text(encoding="utf-8").strip().splitlines()[0]) + if _pid_exists(pid): + raise RuntimeError("WhatsApp channel is already running for this session; stop it before pairing") + except FileNotFoundError: + pass + except ValueError: + pass + + pairing_id = uuid.uuid4().hex + env = os.environ.copy() + env.setdefault("FLOCKS_WHATSAPP_PAIR_TIMEOUT_MS", "120000") + + proc = await asyncio.create_subprocess_exec( + node, + str(script), + "--pair-only", + "--pair-json", + "--session", + str(sess), + cwd=str(bridge_root), + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.STDOUT, + env=env, + ) + _active_session_paths.add(key) + pairing = PairingSession( + id=pairing_id, + session_path=sess, + process=proc, + ) + pairing._reader_task = asyncio.create_task(_read_pair_output(pairing)) + _sessions[pairing_id] = pairing + log.info("whatsapp.pairing.started", {"pairing_id": pairing_id}) + return pairing + + +def get_pairing(pairing_id: str) -> Optional[PairingSession]: + _cleanup_pairings() + return _sessions.get(pairing_id) + + +async def cancel_pairing(pairing_id: str) -> bool: + pairing = _sessions.pop(pairing_id, None) + if pairing is None: + return False + pairing.status = "cancelled" + _release_session(pairing) + if pairing.process.returncode is None: + pairing.process.terminate() + try: + await asyncio.wait_for(pairing.process.wait(), timeout=5) + except asyncio.TimeoutError: + pairing.process.kill() + if pairing._reader_task: + pairing._reader_task.cancel() + return True diff --git a/flocks/channel/registry.py b/flocks/channel/registry.py index c782f937a..e8ae74aa7 100644 --- a/flocks/channel/registry.py +++ b/flocks/channel/registry.py @@ -94,10 +94,12 @@ def _register_builtin_channels(self) -> None: from flocks.channel.builtin.feishu.channel import FeishuChannel from flocks.channel.builtin.telegram.channel import TelegramChannel from flocks.channel.builtin.wecom.channel import WeComChannel + from flocks.channel.builtin.whatsapp.channel import WhatsAppChannel from flocks.channel.builtin.weixin.channel import WeixinChannel self.register(FeishuChannel()) self.register(WeComChannel()) self.register(TelegramChannel()) + self.register(WhatsAppChannel()) self.register(DingTalkChannel()) self.register(WeixinChannel()) diff --git a/flocks/server/routes/channel.py b/flocks/server/routes/channel.py index 518e3fcbd..50568417b 100644 --- a/flocks/server/routes/channel.py +++ b/flocks/server/routes/channel.py @@ -380,6 +380,69 @@ async def weixin_qr_login_status(qrcode: str, baseUrl: Optional[str] = None): raise HTTPException(status_code=502, detail=str(exc)) +# --------------------------------------------------------------------------- +# WhatsApp QR pairing +# --------------------------------------------------------------------------- + +class WhatsAppPairStartRequest(BaseModel): + sessionPath: Optional[str] = None + bridgeDir: Optional[str] = None + + +@router.post("/whatsapp/pair/start") +async def whatsapp_pair_start(req: WhatsAppPairStartRequest): + """Start a WhatsApp Web QR pairing process. + + The helper starts the bundled Node bridge in pair-only mode and returns a + pairing_id. The frontend polls /whatsapp/pair/{pairing_id}/status until a + QR code is available or pairing completes. + """ + from flocks.channel.builtin.whatsapp.pairing import start_pairing + + try: + pairing = await start_pairing( + session_path=req.sessionPath, + bridge_dir=req.bridgeDir, + ) + return { + "ok": True, + "pairing_id": pairing.id, + "status": pairing.status, + "session_path": str(pairing.session_path), + } + except Exception as exc: + log.error("whatsapp.pair.start_failed", {"error": str(exc)}) + raise HTTPException(status_code=502, detail=str(exc)) + + +@router.get("/whatsapp/pair/{pairing_id}/status") +async def whatsapp_pair_status(pairing_id: str): + from flocks.channel.builtin.whatsapp.pairing import get_pairing + + pairing = get_pairing(pairing_id) + if pairing is None: + raise HTTPException(status_code=404, detail="WhatsApp pairing session not found") + return { + "ok": True, + "pairing_id": pairing.id, + "status": pairing.status, + "qr": pairing.qr, + "error": pairing.error, + "user": pairing.user, + "session_path": str(pairing.session_path), + } + + +@router.post("/whatsapp/pair/{pairing_id}/cancel") +async def whatsapp_pair_cancel(pairing_id: str): + from flocks.channel.builtin.whatsapp.pairing import cancel_pairing + + cancelled = await cancel_pairing(pairing_id) + if not cancelled: + raise HTTPException(status_code=404, detail="WhatsApp pairing session not found") + return {"ok": True, "pairing_id": pairing_id} + + # --------------------------------------------------------------------------- # Telegram pairing # --------------------------------------------------------------------------- diff --git a/pyproject.toml b/pyproject.toml index e68732a07..88e6ae933 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -109,6 +109,11 @@ build-backend = "hatchling.build" [tool.hatch.build.targets.wheel] packages = ["flocks"] +artifacts = [ + "flocks/channel/builtin/whatsapp/bridge/*.js", + "flocks/channel/builtin/whatsapp/bridge/package.json", + "flocks/channel/builtin/whatsapp/bridge/package-lock.json", +] [tool.ruff] line-length = 120 diff --git a/tests/channel/test_whatsapp.py b/tests/channel/test_whatsapp.py new file mode 100644 index 000000000..db752d984 --- /dev/null +++ b/tests/channel/test_whatsapp.py @@ -0,0 +1,252 @@ +from __future__ import annotations + +from pathlib import Path + +import pytest + +from flocks.channel.base import ChatType, OutboundContext +from flocks.channel.builtin.whatsapp.channel import WhatsAppChannel +from flocks.channel.builtin.whatsapp.config import ( + matches_identifier, + normalize_jid, + parse_target, + strip_jid, +) +from flocks.channel.builtin.whatsapp.inbound import build_inbound_message +from flocks.channel.builtin.whatsapp import pairing +from flocks.channel.registry import ChannelRegistry + + +def test_whatsapp_channel_metadata_and_capabilities(): + channel = WhatsAppChannel() + + assert channel.meta().id == "whatsapp" + assert "wa" in channel.meta().aliases + assert channel.capabilities().media is True + assert channel.capabilities().chat_types == [ChatType.DIRECT, ChatType.GROUP] + + +def test_registry_registers_builtin_whatsapp_channel(): + registry = ChannelRegistry() + registry.init() + + plugin = registry.get("whatsapp") + assert plugin is not None + assert plugin.meta().label == "WhatsApp" + + +@pytest.mark.parametrize( + ("raw", "expected"), + [ + ("+15551234567", "15551234567@s.whatsapp.net"), + ("15551234567", "15551234567@s.whatsapp.net"), + ("120363001234@g.us", "120363001234@g.us"), + ("abc123@lid", "abc123@lid"), + ("whatsapp:+15551234567", "15551234567@s.whatsapp.net"), + ], +) +def test_normalize_and_parse_targets(raw: str, expected: str): + assert parse_target(raw) == expected + assert normalize_jid(raw.replace("whatsapp:", "")) == expected + + +def test_strip_jid_and_allowlist_alias_matching(): + assert strip_jid("15551234567@s.whatsapp.net") == "15551234567" + assert matches_identifier("15551234567@s.whatsapp.net", ["+15551234567"]) + assert matches_identifier("abc123@lid", ["abc123"]) + assert matches_identifier("120363001234@g.us", ["120363001234@g.us"]) + assert not matches_identifier("15550000000@s.whatsapp.net", ["15551234567"]) + + +def test_build_inbound_dm_message(): + msg = build_inbound_message({ + "messageId": "m1", + "chatId": "15551234567@s.whatsapp.net", + "senderId": "15551234567@s.whatsapp.net", + "senderName": "Alice", + "body": "hello", + "isGroup": False, + }) + + assert msg is not None + assert msg.channel_id == "whatsapp" + assert msg.chat_type == ChatType.DIRECT + assert msg.sender_id == "15551234567" + assert msg.chat_id == "15551234567@s.whatsapp.net" + assert msg.text == "hello" + assert msg.message_id == "15551234567@s.whatsapp.net:m1" + + +def test_build_inbound_group_mention_message(): + msg = build_inbound_message({ + "messageId": "m2", + "chatId": "120363001234@g.us", + "senderId": "15551234567@s.whatsapp.net", + "body": "@bot investigate this", + "isGroup": True, + "mentioned": True, + "mentionText": "investigate this", + "quotedMessageId": "q1", + }) + + assert msg is not None + assert msg.chat_type == ChatType.GROUP + assert msg.mentioned is True + assert msg.mention_text == "investigate this" + assert msg.reply_to_id == "q1" + + +def test_build_inbound_media_message(tmp_path: Path): + media = tmp_path / "image.jpg" + media.write_bytes(b"jpg") + + msg = build_inbound_message({ + "messageId": "m3", + "chatId": "15551234567@s.whatsapp.net", + "senderId": "15551234567@s.whatsapp.net", + "body": "", + "mediaUrls": [str(media)], + "mime": "image/jpeg", + }) + + assert msg is not None + assert msg.media_url == str(media) + assert msg.media_mime == "image/jpeg" + + +def test_validate_config_requires_valid_mode_and_pairing(monkeypatch: pytest.MonkeyPatch, tmp_path: Path): + channel = WhatsAppChannel() + monkeypatch.setattr("flocks.channel.builtin.whatsapp.channel.find_executable", lambda name: "/usr/bin/node") + + assert channel.validate_config({"mode": "bad"}) == "WhatsApp mode must be 'bot' or 'self-chat'" + + session = tmp_path / "session" + session.mkdir() + assert "not paired" in (channel.validate_config({"sessionPath": str(session)}) or "") + + (session / "creds.json").write_text("{}", encoding="utf-8") + assert channel.validate_config({"sessionPath": str(session), "bridgePort": 3100}) is None + + +@pytest.mark.asyncio +async def test_send_text_posts_to_bridge(monkeypatch: pytest.MonkeyPatch): + channel = WhatsAppChannel() + channel._http = _FakeSession({"success": True, "messageId": "wa1"}) # type: ignore[attr-defined] + channel._send_timeout_ms = 1000 # type: ignore[attr-defined] + channel._bridge_port = 3100 # type: ignore[attr-defined] + + result = await channel.send_text( + OutboundContext(channel_id="whatsapp", to="+15551234567", text="hello", reply_to_id="q1") + ) + + assert result.success is True + assert result.message_id == "wa1" + assert result.chat_id == "15551234567@s.whatsapp.net" + assert channel._http.calls == [ # type: ignore[attr-defined] + ( + "http://127.0.0.1:3100/send", + { + "chatId": "15551234567@s.whatsapp.net", + "message": "hello", + "replyTo": "q1", + }, + {"X-Flocks-Bridge-Token": channel._bridge_token}, # type: ignore[attr-defined] + ) + ] + + +@pytest.mark.asyncio +async def test_send_text_strips_composite_reply_id(): + channel = WhatsAppChannel() + channel._http = _FakeSession({"success": True, "messageId": "wa1"}) # type: ignore[attr-defined] + channel._send_timeout_ms = 1000 # type: ignore[attr-defined] + channel._bridge_port = 3100 # type: ignore[attr-defined] + + result = await channel.send_text( + OutboundContext( + channel_id="whatsapp", + to="+15551234567", + text="hello", + reply_to_id="15551234567@s.whatsapp.net:q1", + ) + ) + + assert result.success is True + assert channel._http.calls[0][1]["replyTo"] == "q1" # type: ignore[attr-defined] + + +def test_bridge_identity_requires_matching_session_and_config(tmp_path: Path): + channel = WhatsAppChannel() + channel._session_path = tmp_path / "session" # type: ignore[attr-defined] + channel._media_cache_dir = tmp_path / "media" # type: ignore[attr-defined] + channel._mode = "bot" # type: ignore[attr-defined] + channel._reply_prefix = "" # type: ignore[attr-defined] + channel._send_chunk_delay_ms = 300 # type: ignore[attr-defined] + channel._send_timeout_ms = 60000 # type: ignore[attr-defined] + script = tmp_path / "bridge.js" + script.write_text("console.log('bridge')", encoding="utf-8") + + matching = { + "scriptHash": "placeholder", + "sessionPath": str(channel._session_path), # type: ignore[attr-defined] + "mediaDir": str(channel._media_cache_dir), # type: ignore[attr-defined] + "mode": "bot", + "configHash": channel._bridge_config_hash(), # type: ignore[attr-defined] + } + from flocks.channel.builtin.whatsapp.bridge_runtime import file_hash + + matching["scriptHash"] = file_hash(script) + assert channel._bridge_identity_matches(matching, script) is True # type: ignore[attr-defined] + + mismatched = {**matching, "sessionPath": str(tmp_path / "other")} + assert channel._bridge_identity_matches(mismatched, script) is False # type: ignore[attr-defined] + + +@pytest.mark.asyncio +async def test_pairing_rejects_running_session_after_dependency_bootstrap( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +): + bridge_dir = tmp_path / "bridge" + bridge_dir.mkdir() + (bridge_dir / "bridge.js").write_text("console.log('bridge')", encoding="utf-8") + session = tmp_path / "session" + session.mkdir() + (session / "bridge.pid").write_text(str(__import__("os").getpid()), encoding="utf-8") + calls: list[Path] = [] + + async def fake_ensure(path: Path) -> None: + calls.append(path) + + monkeypatch.setattr(pairing, "find_executable", lambda name: "/usr/bin/node") + monkeypatch.setattr(pairing, "ensure_bridge_deps", fake_ensure) + + with pytest.raises(RuntimeError, match="already running"): + await pairing.start_pairing(session_path=str(session), bridge_dir=str(bridge_dir)) + + assert calls == [bridge_dir] + + +class _FakeResponse: + def __init__(self, payload: dict, status: int = 200): + self._payload = payload + self.status = status + + async def __aenter__(self): + return self + + async def __aexit__(self, *args): + return False + + async def json(self, content_type=None): + return self._payload + + +class _FakeSession: + def __init__(self, payload: dict): + self.payload = payload + self.calls: list[tuple[str, dict, dict | None]] = [] + + def post(self, url: str, *, json: dict, headers=None, timeout=None): + self.calls.append((url, json, headers)) + return _FakeResponse(self.payload) diff --git a/webui/src/locales/en-US/channel.json b/webui/src/locales/en-US/channel.json index fa058f8f2..3036a8450 100644 --- a/webui/src/locales/en-US/channel.json +++ b/webui/src/locales/en-US/channel.json @@ -1,6 +1,6 @@ { "pageTitle": "Channels", - "pageDescription": "IM channel configuration (Feishu, WeCom, DingTalk)", + "pageDescription": "IM channel configuration (Feishu, WeCom, WeChat, DingTalk, Telegram, WhatsApp)", "cardTitle": "Channel Configuration", "cardDesc": "Configure IM bot integrations to allow AI Agents to receive and send messages on Feishu, WeCom, and other platforms.", "tabs": { @@ -12,6 +12,7 @@ "wecom": "WeCom", "dingtalk": "DingTalk", "telegram": "Telegram", + "whatsapp": "WhatsApp", "weixin": "WeChat" }, "status": { @@ -285,5 +286,70 @@ "qrError": "Failed to fetch QR code — check your network connection", "qrDone": "Done", "qrSuccess": "WeChat account connected — credentials auto-filled" + }, + "whatsapp": { + "credentials": "Account Connection", + "credentialsDesc": "Pair through WhatsApp Linked Devices. This channel uses a local Baileys bridge bound to 127.0.0.1 only.", + "optional": "Optional", + "qrLoginButton": "Connect WhatsApp", + "qrLoading": "Starting pairing...", + "sessionConfigured": "Session directory configured; scan again to replace it", + "qrModalTitle": "WhatsApp Device Pairing", + "qrHintLoading": "Starting the local WhatsApp bridge", + "qrHintScanning": "Open WhatsApp Linked Devices on your phone and scan this QR code", + "qrHintConnected": "Device connected; saving the session", + "qrHintComplete": "Pairing complete; configuration saved and channel restarting", + "qrConnected": "Device connected", + "qrComplete": "Pairing complete", + "qrRetry": "Retry", + "qrError": "WhatsApp pairing failed", + "qrDone": "Done", + "qrSuccess": "WhatsApp connected; channel configuration saved", + "sessionPath": "Session Directory", + "sessionPathHint": "Directory for WhatsApp login credentials (default ~/.flocks/workspace/channels/whatsapp/session)", + "mode": "Mode", + "modeHint": "Bot mode uses a dedicated WhatsApp account. Self-chat mode only handles messages sent to yourself.", + "modeBot": "Bot mode (dedicated account)", + "modeSelfChat": "Self-chat (your own chat only)", + "behavior": "Message Behavior", + "behaviorDesc": "Configure DM, group, and trigger policies.", + "defaultAgent": "Default Agent", + "defaultAgentHint": "Default Agent ID used when no Agent is specified", + "dmPolicy": "DM Policy", + "dmPolicyHint": "Controls which WhatsApp direct messages can trigger the Agent", + "dmPolicyOpen": "Open (everyone allowed)", + "dmPolicyAllowlist": "Allowlist (only listed users)", + "dmPolicyDisabled": "Disabled (no DMs accepted)", + "allowFromEnabled": "Enable User Allowlist", + "allowFromEnabledHint": "Recommended for production. Accepts phone numbers, full JIDs, or @lid IDs.", + "allowFrom": "Allowed Users", + "allowFromHint": "Supports phone numbers, 15551234567@s.whatsapp.net, or 123@lid", + "allowFromPlaceholder": "Enter phone number or WhatsApp JID and press Enter", + "groupPolicy": "Group Policy", + "groupPolicyHint": "Controls whether group chat messages can trigger the Agent", + "groupPolicyOpen": "Open (all groups)", + "groupPolicyAllowlist": "Allowlist (allowed groups only)", + "groupPolicyDisabled": "Disabled (ignore groups)", + "groupAllowFrom": "Allowed Group JIDs", + "groupAllowFromHint": "WhatsApp group IDs usually end with @g.us", + "groupAllowFromPlaceholder": "Enter group JID and press Enter", + "groupTrigger": "Group Trigger", + "groupTriggerHint": "When to trigger Agent responses in group chats", + "triggerMention": "Only @mention or reply to bot", + "triggerAll": "All messages", + "advanced": "Advanced Settings", + "advancedDesc": "Local bridge port, batching, and send timeout settings.", + "bridgePort": "Bridge Port", + "bridgePortHint": "Local bridge HTTP port, bound to 127.0.0.1 only (default 3100)", + "replyPrefix": "Reply Prefix", + "replyPrefixHint": "Optional text prefix added before outgoing WhatsApp replies", + "textBatchDelaySeconds": "Text Batch Delay (s)", + "textBatchDelaySecondsHint": "Quiet period for merging rapid text messages before invoking the Agent", + "sendChunkDelayMs": "Chunk Send Delay (ms)", + "sendChunkDelayMsHint": "Delay between chunks when a long reply is split", + "sendTimeoutMs": "Send Timeout (ms)", + "sendTimeoutMsHint": "Per-call timeout for bridge sendMessage calls", + "mediaCacheDir": "Media Cache Directory", + "mediaCacheDirHint": "Directory where inbound media is downloaded (default workspace channel directory)" } } diff --git a/webui/src/locales/zh-CN/channel.json b/webui/src/locales/zh-CN/channel.json index fbd4c2cbc..9ae6ff0f5 100644 --- a/webui/src/locales/zh-CN/channel.json +++ b/webui/src/locales/zh-CN/channel.json @@ -1,6 +1,6 @@ { "pageTitle": "通道", - "pageDescription": "IM 通道连接配置(飞书、企业微信、钉钉)", + "pageDescription": "IM 通道连接配置(飞书、企业微信、微信、钉钉、Telegram、WhatsApp)", "cardTitle": "通道配置", "cardDesc": "配置 IM 机器人接入,让 AI Agent 能够在飞书、企业微信等平台上接收和发送消息。", "tabs": { @@ -12,6 +12,7 @@ "wecom": "企业微信", "dingtalk": "钉钉", "telegram": "Telegram", + "whatsapp": "WhatsApp", "weixin": "微信" }, "status": { @@ -287,5 +288,70 @@ "qrError": "获取二维码失败,请检查网络连接", "qrDone": "完成", "qrSuccess": "微信账号连接成功,凭证已自动填入" + }, + "whatsapp": { + "credentials": "账号连接", + "credentialsDesc": "通过 WhatsApp Linked Devices 扫码连接。本通道使用本地 Baileys bridge,仅监听 127.0.0.1。", + "optional": "选填", + "qrLoginButton": "扫码连接 WhatsApp", + "qrLoading": "正在启动配对...", + "sessionConfigured": "已配置会话目录,可重新扫码替换", + "qrModalTitle": "WhatsApp 设备配对", + "qrHintLoading": "正在启动本地 WhatsApp bridge", + "qrHintScanning": "请在手机 WhatsApp 中打开 Linked Devices 并扫描此二维码", + "qrHintConnected": "设备已连接,正在保存会话", + "qrHintComplete": "连接完成,配置已保存并正在重启通道", + "qrConnected": "设备已连接", + "qrComplete": "连接完成", + "qrRetry": "重试", + "qrError": "WhatsApp 配对失败", + "qrDone": "完成", + "qrSuccess": "WhatsApp 已连接,通道配置已保存", + "sessionPath": "会话目录", + "sessionPathHint": "保存 WhatsApp 登录凭据的目录(留空使用 ~/.flocks/workspace/channels/whatsapp/session)", + "mode": "运行模式", + "modeHint": "Bot 模式用于独立 WhatsApp 账号对外服务;Self-chat 模式只处理发给自己的消息。", + "modeBot": "Bot 模式(独立账号)", + "modeSelfChat": "Self-chat(仅本人自聊)", + "behavior": "消息行为", + "behaviorDesc": "配置私信、群聊和触发策略。", + "defaultAgent": "默认 Agent", + "defaultAgentHint": "未指定 Agent 时使用的默认 Agent ID", + "dmPolicy": "私信策略", + "dmPolicyHint": "控制哪些 WhatsApp 私信可以触发 Agent", + "dmPolicyOpen": "开放(所有人均可)", + "dmPolicyAllowlist": "白名单(仅允许名单内用户)", + "dmPolicyDisabled": "关闭(不接受私信)", + "allowFromEnabled": "启用用户白名单", + "allowFromEnabledHint": "建议生产环境开启。可填写手机号、完整 JID 或 @lid。", + "allowFrom": "允许的用户", + "allowFromHint": "支持手机号、15551234567@s.whatsapp.net 或 123@lid", + "allowFromPlaceholder": "输入手机号或 WhatsApp JID 后按回车添加", + "groupPolicy": "群聊策略", + "groupPolicyHint": "控制群聊是否可触发 Agent", + "groupPolicyOpen": "开放(所有群均可)", + "groupPolicyAllowlist": "白名单(仅允许名单内群)", + "groupPolicyDisabled": "关闭(不处理群聊)", + "groupAllowFrom": "允许的群 JID", + "groupAllowFromHint": "WhatsApp 群 ID 通常以 @g.us 结尾", + "groupAllowFromPlaceholder": "输入群 JID 后按回车添加", + "groupTrigger": "群聊触发方式", + "groupTriggerHint": "群聊中什么情况下触发 Agent 响应", + "triggerMention": "仅 @ 或回复机器人时触发", + "triggerAll": "所有消息都触发", + "advanced": "高级设置", + "advancedDesc": "本地 bridge 端口、批处理和发送超时设置。", + "bridgePort": "Bridge 端口", + "bridgePortHint": "本地 bridge HTTP 端口,仅绑定 127.0.0.1(默认 3100)", + "replyPrefix": "回复前缀", + "replyPrefixHint": "发送到 WhatsApp 前追加的文本前缀(可留空)", + "textBatchDelaySeconds": "文本合并等待 (秒)", + "textBatchDelaySecondsHint": "连续多条文本消息在触发 Agent 前的合并等待时间", + "sendChunkDelayMs": "分块发送间隔 (ms)", + "sendChunkDelayMsHint": "长回复拆分后,各段消息之间的发送间隔", + "sendTimeoutMs": "发送超时 (ms)", + "sendTimeoutMsHint": "bridge 调用 WhatsApp sendMessage 的单次超时", + "mediaCacheDir": "媒体缓存目录", + "mediaCacheDirHint": "入站媒体下载到本地的目录(留空使用默认工作区目录)" } } diff --git a/webui/src/pages/Channel/index.tsx b/webui/src/pages/Channel/index.tsx index 963a16693..87cf26e79 100644 --- a/webui/src/pages/Channel/index.tsx +++ b/webui/src/pages/Channel/index.tsx @@ -144,7 +144,31 @@ interface WeixinChannelConfig { dataDir?: string; } -type ChannelConfig = FeishuChannelConfig | WeComChannelConfig | DingTalkChannelConfig | TelegramChannelConfig | WeixinChannelConfig; +interface WhatsAppChannelConfig { + enabled: boolean; + mode?: 'bot' | 'self-chat'; + sessionPath?: string; + bridgePort?: number; + defaultAgent?: string; + dmPolicy?: string; + allowFrom?: string[]; + groupPolicy?: string; + groupAllowFrom?: string[]; + groupTrigger?: string; + replyPrefix?: string; + textBatchDelaySeconds?: number; + sendChunkDelayMs?: number; + sendTimeoutMs?: number; + mediaCacheDir?: string; +} + +type ChannelConfig = + | FeishuChannelConfig + | WeComChannelConfig + | DingTalkChannelConfig + | TelegramChannelConfig + | WeixinChannelConfig + | WhatsAppChannelConfig; function defaultFeishuConfig(): FeishuChannelConfig { return { @@ -200,6 +224,21 @@ function defaultWeixinConfig(): WeixinChannelConfig { }; } +function defaultWhatsAppConfig(): WhatsAppChannelConfig { + return { + enabled: false, + mode: 'bot', + dmPolicy: 'allowlist', + allowFrom: [], + groupPolicy: 'disabled', + groupTrigger: 'mention', + bridgePort: 3100, + textBatchDelaySeconds: 3, + sendChunkDelayMs: 300, + sendTimeoutMs: 60000, + }; +} + // ============================================================================ // Form primitives // ============================================================================ @@ -1396,6 +1435,340 @@ function TelegramPanel({ config, onChange, onRefresh }: TelegramPanelProps) { ); } +// ============================================================================ +// WhatsApp Config Panel +// ============================================================================ + +interface WhatsAppPanelProps { + config: WhatsAppChannelConfig; + onChange: (c: WhatsAppChannelConfig) => void; + onPairSuccess?: (data: { sessionPath: string }) => Promise | void; +} + +type WhatsAppQrPhase = + | 'idle' + | 'loading' + | 'scanning' + | 'connected' + | 'complete' + | 'error'; + +function WhatsAppPanel({ config, onChange, onPairSuccess }: WhatsAppPanelProps) { + const { t } = useTranslation('channel'); + const toast = useToast(); + const set = useCallback( + (key: K, value: WhatsAppChannelConfig[K]) => + onChange({ ...config, [key]: value }), + [config, onChange] + ); + + const [qrPhase, setQrPhase] = useState('idle'); + const [qrValue, setQrValue] = useState(''); + const [qrError, setQrError] = useState(''); + const [pairingId, setPairingId] = useState(''); + const pollRef = useRef | null>(null); + const completedRef = useRef(false); + + const stopPolling = () => { + if (pollRef.current) { + clearInterval(pollRef.current); + pollRef.current = null; + } + }; + + useEffect(() => () => stopPolling(), []); + + const closeQrModal = async () => { + stopPolling(); + if (pairingId && !['complete', 'connected'].includes(qrPhase)) { + client.post(`/api/channel/whatsapp/pair/${pairingId}/cancel`, {}, { timeout: 5000 }).catch(() => {}); + } + setQrPhase('idle'); + setQrValue(''); + setQrError(''); + setPairingId(''); + }; + + const finishPairing = async (sessionPath: string) => { + if (completedRef.current) return; + completedRef.current = true; + stopPolling(); + setQrPhase('complete'); + const newConfig = { ...config, sessionPath, enabled: true }; + onChange(newConfig); + if (onPairSuccess) { + await onPairSuccess({ sessionPath }); + } + toast.success(t('whatsapp.qrSuccess')); + }; + + const startPairing = async () => { + stopPolling(); + completedRef.current = false; + setQrError(''); + setQrValue(''); + setQrPhase('loading'); + try { + const res = await client.post('/api/channel/whatsapp/pair/start', { + sessionPath: config.sessionPath || null, + }); + const id = String(res.data?.pairing_id ?? ''); + setPairingId(id); + pollRef.current = setInterval(async () => { + try { + const statusRes = await client.get(`/api/channel/whatsapp/pair/${id}/status`); + const status = String(statusRes.data?.status ?? ''); + const qr = String(statusRes.data?.qr ?? ''); + if (qr) { + setQrValue(qr); + setQrPhase('scanning'); + } + if (status === 'connected') { + setQrPhase('connected'); + } + if (status === 'complete') { + await finishPairing(String(statusRes.data?.session_path ?? config.sessionPath ?? '')); + } + if (status === 'error') { + stopPolling(); + setQrError(String(statusRes.data?.error ?? t('whatsapp.qrError'))); + setQrPhase('error'); + } + } catch { + // Pairing may still be starting; keep polling until the bridge reports a terminal state. + } + }, 1500); + } catch (err: any) { + const detail = err?.response?.data?.detail ?? err?.message ?? ''; + setQrError(detail); + setQrPhase('error'); + } + }; + + const showModal = qrPhase !== 'idle'; + const allowFromEnabled = config.allowFrom !== undefined; + + return ( + <> +
+
+ + {config.sessionPath && ( +

+ + {t('whatsapp.sessionConfigured')} +

+ )} +
+ + {showModal && ( +
+
+ +

{t('whatsapp.qrModalTitle')}

+ {qrPhase === 'loading' && ( +
+ +
+ )} + {qrPhase === 'scanning' && qrValue && ( + + )} + {(qrPhase === 'connected' || qrPhase === 'complete') && ( +
+ +

+ {qrPhase === 'complete' ? t('whatsapp.qrComplete') : t('whatsapp.qrConnected')} +

+
+ )} + {qrPhase === 'error' && ( +
+ +

{qrError || t('whatsapp.qrError')}

+ +
+ )} +

+ {qrPhase === 'loading' && t('whatsapp.qrHintLoading')} + {qrPhase === 'scanning' && t('whatsapp.qrHintScanning')} + {qrPhase === 'connected' && t('whatsapp.qrHintConnected')} + {qrPhase === 'complete' && t('whatsapp.qrHintComplete')} +

+ {qrPhase === 'complete' && ( + + )} +
+
+ )} + +
+ + + set('sessionPath', v || undefined)} + placeholder={t('whatsapp.optional')} + /> + + + set('dmPolicy', v)} + options={[ + { value: 'allowlist', label: t('whatsapp.dmPolicyAllowlist') }, + { value: 'open', label: t('whatsapp.dmPolicyOpen') }, + { value: 'disabled', label: t('whatsapp.dmPolicyDisabled') }, + ]} + /> + + + onChange({ ...config, allowFrom: enabled ? [] : undefined })} + /> + + {allowFromEnabled && ( + + set('allowFrom', v)} + placeholder={t('whatsapp.allowFromPlaceholder')} + /> + + )} + + set('groupTrigger', v)} + options={[ + { value: 'mention', label: t('whatsapp.triggerMention') }, + { value: 'all', label: t('whatsapp.triggerAll') }, + ]} + /> + + )} +
+ +
+ + set('bridgePort', v)} + min={1} + /> + + + set('replyPrefix', v || undefined)} + placeholder={t('whatsapp.optional')} + /> + + + set('textBatchDelaySeconds', v)} + min={0} + /> + + + set('sendChunkDelayMs', v)} + min={0} + /> + + + set('sendTimeoutMs', v)} + min={1000} + /> + + + set('mediaCacheDir', v || undefined)} + placeholder={t('whatsapp.optional')} + /> + +
+ + ); +} + // ============================================================================ // Weixin Config Panel // ============================================================================ @@ -1956,6 +2329,8 @@ export default function ChannelPage() { configs[ch.id] = { ...defaultDingTalkConfig(), ...saved }; } else if (ch.id === 'telegram') { configs[ch.id] = { ...defaultTelegramConfig(), ...saved }; + } else if (ch.id === 'whatsapp') { + configs[ch.id] = { ...defaultWhatsAppConfig(), ...saved }; } else if (ch.id === 'weixin') { configs[ch.id] = { ...defaultWeixinConfig(), ...saved }; } else { @@ -2090,6 +2465,36 @@ export default function ChannelPage() { setTimeout(() => { fetchAll(); fetchStatuses(true); }, 8000); }; + const handleWhatsAppPairSuccess = async ( + data: { sessionPath: string } + ) => { + const channelId = 'whatsapp'; + const savedChannelCfg = (fullConfig.channels?.[channelId] ?? {}) as Record; + const updatedChannelCfg: Record = { + ...savedChannelCfg, + ...stripEmpty(channelConfigs[channelId] ?? {}), + enabled: true, + }; + if (data.sessionPath) updatedChannelCfg.sessionPath = data.sessionPath; + + const updatedChannels = { ...(fullConfig.channels ?? {}), [channelId]: updatedChannelCfg }; + const updated = { ...fullConfig, channels: updatedChannels }; + + await client.patch('/api/config/', updated); + setFullConfig(updated); + setChannelConfigs((prev) => ({ + ...prev, + [channelId]: { ...prev[channelId], ...updatedChannelCfg } as ChannelConfig, + })); + originalConfigsRef.current = { + ...originalConfigsRef.current, + [channelId]: { ...originalConfigsRef.current[channelId], ...updatedChannelCfg }, + }; + client.post(`/api/channel/${channelId}/restart`, {}, { timeout: 5000 }).catch(() => {}); + setTimeout(() => { fetchAll(); fetchStatuses(true); }, 3000); + setTimeout(() => { fetchAll(); fetchStatuses(true); }, 8000); + }; + // Manual restart — useful when connection drops and user wants to reconnect const handleRestart = async (channelId?: string) => { const id = channelId ?? selectedId; @@ -2281,6 +2686,13 @@ export default function ChannelPage() { onRefresh={fetchAll} /> )} + {selectedId === 'whatsapp' && ( + handleChannelConfigChange('whatsapp', cfg)} + onPairSuccess={handleWhatsAppPairSuccess} + /> + )} {selectedId === 'weixin' && ( Date: Fri, 10 Jul 2026 16:21:41 +0800 Subject: [PATCH 15/49] feat(channel): add email gateway channel --- flocks/channel/builtin/email/__init__.py | 5 + flocks/channel/builtin/email/channel.py | 488 +++++++++++++++++++++++ flocks/channel/builtin/email/config.py | 103 +++++ flocks/channel/builtin/email/inbound.py | 245 ++++++++++++ flocks/channel/registry.py | 2 + tests/channel/test_email.py | 356 +++++++++++++++++ webui/src/locales/en-US/channel.json | 49 +++ webui/src/locales/zh-CN/channel.json | 49 +++ webui/src/pages/Channel/index.tsx | 199 ++++++++- 9 files changed, 1495 insertions(+), 1 deletion(-) create mode 100644 flocks/channel/builtin/email/__init__.py create mode 100644 flocks/channel/builtin/email/channel.py create mode 100644 flocks/channel/builtin/email/config.py create mode 100644 flocks/channel/builtin/email/inbound.py create mode 100644 tests/channel/test_email.py diff --git a/flocks/channel/builtin/email/__init__.py b/flocks/channel/builtin/email/__init__.py new file mode 100644 index 000000000..361b0601c --- /dev/null +++ b/flocks/channel/builtin/email/__init__.py @@ -0,0 +1,5 @@ +"""Built-in Email channel.""" + +from .channel import EmailChannel + +__all__ = ["EmailChannel"] diff --git a/flocks/channel/builtin/email/channel.py b/flocks/channel/builtin/email/channel.py new file mode 100644 index 000000000..f124fdefd --- /dev/null +++ b/flocks/channel/builtin/email/channel.py @@ -0,0 +1,488 @@ +"""Email ChannelPlugin using IMAP for inbound and SMTP for outbound.""" + +from __future__ import annotations + +import asyncio +import email as email_lib +import imaplib +import smtplib +import socket +import ssl +import uuid +from email import encoders +from email.mime.base import MIMEBase +from email.mime.multipart import MIMEMultipart +from email.mime.text import MIMEText +from email.utils import formatdate +from pathlib import Path +from typing import Any, Awaitable, Callable, Optional + +from flocks.channel.base import ( + ChannelCapabilities, + ChannelMeta, + ChannelPlugin, + ChatType, + DeliveryResult, + InboundMessage, + OutboundContext, +) +from flocks.utils.log import Log + +from .config import ( + is_valid_email, + normalize_email_address, + parse_allowed_senders, + resolved_config, +) +from .inbound import ( + build_inbound_message, + is_automated_sender, + verify_sender_authentication, +) + +log = Log.create(service="channel.email") + +SMTP_CONNECT_TIMEOUT = 30 + + +class EmailChannel(ChannelPlugin): + """Single-mailbox Email channel.""" + + def __init__(self) -> None: + super().__init__() + self._resolved: dict[str, Any] = {} + self._seen_uids: set[bytes] = set() + self._seen_uids_max = 5000 + self._thread_context: dict[str, dict[str, str]] = {} + + def meta(self) -> ChannelMeta: + return ChannelMeta( + id="email", + label="Email", + aliases=["mail", "imap", "smtp"], + order=60, + ) + + def capabilities(self) -> ChannelCapabilities: + return ChannelCapabilities( + chat_types=[ChatType.DIRECT], + media=True, + threads=True, + reactions=False, + edit=False, + rich_text=False, + ) + + @property + def text_chunk_limit(self) -> int: + return 50_000 + + @property + def rate_limit(self) -> tuple[float, int]: + return (0.2, 2) + + def validate_config(self, config: dict) -> Optional[str]: + cfg = resolved_config(config) + missing = [ + name + for name in ("address", "password", "imapHost", "smtpHost") + if not cfg.get(name) + ] + if missing: + return "Missing required config: " + ", ".join(missing) + if not is_valid_email(cfg["address"]): + return "Invalid email address" + if cfg["imapPort"] <= 0 or cfg["smtpPort"] <= 0: + return "IMAP/SMTP ports must be positive integers" + allowed = parse_allowed_senders(cfg) + if not cfg["allowAll"] and not allowed: + return "Email channel requires allowFrom or allowAll=true" + if ( + not cfg["allowAll"] + and allowed + and cfg["requireAuthenticatedSender"] + and not cfg["authservId"] + ): + return "Email channel requires authservId when requireAuthenticatedSender=true and allowFrom is configured" + return None + + def normalize_target(self, raw: str) -> Optional[str]: + target = normalize_email_address(raw) + return target if is_valid_email(target) else None + + def target_hint(self) -> str: + return "user@example.com" + + async def start( + self, + config: dict, + on_message: Callable[[InboundMessage], Awaitable[None]], + abort_event: Optional[asyncio.Event] = None, + ) -> None: + self._config = config + self._resolved = resolved_config(config) + self._on_message = on_message + + loop = asyncio.get_running_loop() + await loop.run_in_executor(None, self._test_connections) + + abort = abort_event or asyncio.Event() + while not abort.is_set(): + try: + messages = await loop.run_in_executor(None, self._fetch_new_messages) + for message in messages: + if abort.is_set(): + break + await on_message(message) + except asyncio.CancelledError: + raise + except Exception as exc: + self.mark_disconnected(str(exc)) + log.warning("email.poll.failed", {"error": str(exc)}) + + try: + await asyncio.wait_for( + abort.wait(), + timeout=float(self._resolved["pollIntervalSeconds"]), + ) + except asyncio.TimeoutError: + continue + + async def stop(self) -> None: + self.mark_disconnected() + + async def send_text(self, ctx: OutboundContext) -> DeliveryResult: + target = self.normalize_target(ctx.to) + if not target: + return DeliveryResult( + channel_id="email", + message_id="", + success=False, + error="Email send requires a valid recipient email address", + ) + + try: + loop = asyncio.get_running_loop() + message_id = await loop.run_in_executor( + None, + self._send_email, + target, + ctx.text, + ctx.reply_to_id, + ctx.thread_id, + None, + ) + self.record_message() + return DeliveryResult( + channel_id="email", + message_id=message_id, + chat_id=target, + success=True, + ) + except Exception as exc: + return DeliveryResult( + channel_id="email", + message_id="", + success=False, + error=f"Email send failed: {exc}", + retryable=self._is_retryable(exc), + ) + + async def send_media(self, ctx: OutboundContext) -> DeliveryResult: + target = self.normalize_target(ctx.to) + if not target: + return DeliveryResult( + channel_id="email", + message_id="", + success=False, + error="Email send_media requires a valid recipient email address", + ) + if not ctx.media_url: + return await self.send_text(ctx) + + path = self._local_media_path(ctx.media_url) + if path is None: + return await self.send_text( + OutboundContext( + **{ + **vars(ctx), + "text": f"{ctx.text}\n\nAttachment/link: {ctx.media_url}".strip(), + } + ) + ) + + try: + loop = asyncio.get_running_loop() + message_id = await loop.run_in_executor( + None, + self._send_email, + target, + ctx.text, + ctx.reply_to_id, + ctx.thread_id, + path, + ) + self.record_message() + return DeliveryResult( + channel_id="email", + message_id=message_id, + chat_id=target, + success=True, + ) + except Exception as exc: + return DeliveryResult( + channel_id="email", + message_id="", + success=False, + error=f"Email send_media failed: {exc}", + retryable=self._is_retryable(exc), + ) + + def _test_connections(self) -> None: + cfg = self._resolved + imap = imaplib.IMAP4_SSL(cfg["imapHost"], cfg["imapPort"], timeout=30) + try: + imap.login(cfg["address"], cfg["password"]) + imap.select("INBOX") + if cfg["skipExistingOnStart"]: + status, data = imap.uid("search", None, "ALL") + if status == "OK" and data and data[0]: + self._seen_uids.update(data[0].split()) + self._trim_seen_uids() + finally: + try: + imap.logout() + except Exception: + pass + + smtp = self._connect_smtp() + try: + smtp.login(cfg["address"], cfg["password"]) + finally: + try: + smtp.quit() + except Exception: + smtp.close() + + def _fetch_new_messages(self) -> list[InboundMessage]: + cfg = self._resolved + parsed_messages: list[InboundMessage] = [] + imap = imaplib.IMAP4_SSL(cfg["imapHost"], cfg["imapPort"], timeout=30) + try: + imap.login(cfg["address"], cfg["password"]) + imap.select("INBOX") + status, data = imap.uid("search", None, "UNSEEN") + if status != "OK" or not data or not data[0]: + return parsed_messages + + for uid in data[0].split(): + if uid in self._seen_uids: + continue + self._seen_uids.add(uid) + self._trim_seen_uids() + + status, msg_data = imap.uid("fetch", uid, "(RFC822)") + if status != "OK": + continue + try: + raw_email = msg_data[0][1] + except (IndexError, TypeError): + log.warning("email.imap.malformed_response", {"uid": uid.decode(errors="replace")}) + continue + if not isinstance(raw_email, (bytes, bytearray)): + log.warning("email.imap.non_bytes_payload", {"uid": uid.decode(errors="replace")}) + continue + + message = email_lib.message_from_bytes(raw_email) + parsed = self._parse_and_authorize(message, uid.decode(errors="replace")) + if parsed is not None: + parsed_messages.append(parsed) + finally: + try: + imap.logout() + except Exception: + pass + return parsed_messages + + def _parse_and_authorize( + self, + message: email_lib.message.Message, + uid: str, + ) -> Optional[InboundMessage]: + cfg = self._resolved + parsed = build_inbound_message( + message, + uid=uid, + account_id="default", + skip_attachments=bool(cfg["skipAttachments"]), + ) + if parsed is None: + return None + + inbound = parsed.inbound + sender = inbound.sender_id + if sender == cfg["address"]: + return None + if is_automated_sender(sender, dict(message.items())): + return None + + allowed = parse_allowed_senders(cfg) + if not cfg["allowAll"]: + if not allowed or sender not in allowed: + log.debug("email.sender.blocked", {"sender": sender}) + return None + + if cfg["requireAuthenticatedSender"] and allowed and not cfg["allowAll"]: + if not cfg["authservId"]: + log.warning("email.sender.authserv_id_missing", {"sender": sender}) + return None + authenticated, reason = verify_sender_authentication( + message, + sender, + authserv_id=str(cfg["authservId"]), + ) + if not authenticated: + log.warning("email.sender.unauthenticated", { + "sender": sender, + "reason": reason, + }) + return None + + context = { + "subject": parsed.subject, + "message_id": parsed.message_id, + "references": parsed.references, + } + self._thread_context[sender] = context + for key in {parsed.message_id, parsed.inbound.thread_id or "", parsed.inbound.reply_to_id or ""}: + if key: + self._thread_context[key] = context + return inbound + + def _connect_smtp(self) -> smtplib.SMTP: + cfg = self._resolved + context = ssl.create_default_context() + if int(cfg["smtpPort"]) == 465: + return smtplib.SMTP_SSL( + cfg["smtpHost"], + int(cfg["smtpPort"]), + timeout=SMTP_CONNECT_TIMEOUT, + context=context, + ) + smtp = smtplib.SMTP( + cfg["smtpHost"], + int(cfg["smtpPort"]), + timeout=SMTP_CONNECT_TIMEOUT, + ) + try: + smtp.starttls(context=context) + except Exception: + smtp.close() + raise + return smtp + + def _send_email( + self, + to_addr: str, + body: str, + reply_to_msg_id: Optional[str], + thread_id: Optional[str], + attachment_path: Optional[Path], + ) -> str: + cfg = self._resolved + msg = MIMEMultipart() + msg["From"] = cfg["address"] + msg["To"] = to_addr + + ctx = self._resolve_thread_context(to_addr, reply_to_msg_id, thread_id) + subject = ctx.get("subject") or cfg["defaultSubject"] + if not subject.lower().startswith("re:"): + subject = f"Re: {subject}" + msg["Subject"] = subject + + original_id = reply_to_msg_id or ctx.get("message_id") + references = thread_id or ctx.get("references") or original_id + if original_id: + msg["In-Reply-To"] = original_id + if references: + msg["References"] = references + + msg["Date"] = formatdate(localtime=True) + message_id = f"" + msg["Message-ID"] = message_id + msg.attach(MIMEText(body or "", "plain", "utf-8")) + + if attachment_path is not None: + with attachment_path.open("rb") as handle: + part = MIMEBase("application", "octet-stream") + part.set_payload(handle.read()) + encoders.encode_base64(part) + part.add_header( + "Content-Disposition", + f"attachment; filename={attachment_path.name}", + ) + msg.attach(part) + + smtp = self._connect_smtp() + try: + smtp.login(cfg["address"], cfg["password"]) + smtp.send_message(msg) + finally: + try: + smtp.quit() + except Exception: + smtp.close() + return message_id + + def _resolve_thread_context( + self, + to_addr: str, + reply_to_msg_id: Optional[str], + thread_id: Optional[str], + ) -> dict[str, str]: + for key in (thread_id, reply_to_msg_id, to_addr): + if key and key in self._thread_context: + return self._thread_context[key] + return {} + + def _message_id_domain(self) -> str: + address = str(self._resolved.get("address") or "") + if "@" not in address: + return "localhost" + return address.rsplit("@", 1)[-1] or "localhost" + + def _trim_seen_uids(self) -> None: + if len(self._seen_uids) <= self._seen_uids_max: + return + try: + sorted_uids = sorted(self._seen_uids, key=lambda item: int(item)) + self._seen_uids = set(sorted_uids[-self._seen_uids_max // 2:]) + except (TypeError, ValueError): + keep = list(self._seen_uids)[-self._seen_uids_max // 2:] + self._seen_uids = set(keep) + + @staticmethod + def _local_media_path(media_url: str) -> Optional[Path]: + value = str(media_url or "").strip() + if value.startswith("file://"): + from urllib.parse import unquote + value = unquote(value[len("file://"):]) + if value.startswith(("http://", "https://")): + return None + path = Path(value) + return path if path.is_file() else None + + @staticmethod + def _is_retryable(exc: Exception) -> bool: + return isinstance( + exc, + ( + TimeoutError, + ConnectionError, + OSError, + smtplib.SMTPServerDisconnected, + smtplib.SMTPConnectError, + smtplib.SMTPHeloError, + socket.timeout, + ), + ) diff --git a/flocks/channel/builtin/email/config.py b/flocks/channel/builtin/email/config.py new file mode 100644 index 000000000..8a5cde4b2 --- /dev/null +++ b/flocks/channel/builtin/email/config.py @@ -0,0 +1,103 @@ +"""Configuration helpers for the Email channel.""" + +from __future__ import annotations + +import re +from typing import Any + + +EMAIL_RE = re.compile(r"^[^@\s]+@[^@\s]+\.[^@\s]+$") + + +def coerce_str(value: Any) -> str: + if value is None: + return "" + return str(value).strip() + + +def coerce_int(value: Any, default: int) -> int: + if value is None or isinstance(value, bool): + return default + try: + return int(str(value).strip(), 10) + except (TypeError, ValueError): + return default + + +def coerce_bool(value: Any, default: bool = False) -> bool: + if value is None: + return default + if isinstance(value, bool): + return value + if isinstance(value, (int, float)): + return bool(value) + if isinstance(value, str): + normalized = value.strip().lower() + if normalized in {"1", "true", "yes", "y", "on"}: + return True + if normalized in {"0", "false", "no", "n", "off"}: + return False + return default + + +def normalize_email_address(raw: str) -> str: + value = coerce_str(raw).lower() + if value.startswith("mailto:"): + value = value[len("mailto:"):].strip() + if "<" in value and ">" in value: + value = value.split("<", 1)[1].split(">", 1)[0].strip() + return value + + +def parse_allowed_senders(config: dict[str, Any]) -> set[str]: + raw = config.get("allowFrom") + if raw is None: + raw = config.get("allowedSenders") + if raw is None: + raw = config.get("allowedUsers") + + if isinstance(raw, str): + values = raw.split(",") + elif isinstance(raw, list): + values = raw + else: + values = [] + + return { + normalize_email_address(str(item)) + for item in values + if normalize_email_address(str(item)) + } + + +def is_valid_email(raw: str) -> bool: + return bool(EMAIL_RE.fullmatch(normalize_email_address(raw))) + + +def resolved_config(config: dict[str, Any]) -> dict[str, Any]: + """Return normalized Email channel config with defaults applied.""" + return { + **config, + "address": normalize_email_address(coerce_str(config.get("address"))), + "password": coerce_str(config.get("password")), + "imapHost": coerce_str(config.get("imapHost") or config.get("imap_host")), + "imapPort": coerce_int(config.get("imapPort") or config.get("imap_port"), 993), + "smtpHost": coerce_str(config.get("smtpHost") or config.get("smtp_host")), + "smtpPort": coerce_int(config.get("smtpPort") or config.get("smtp_port"), 587), + "pollIntervalSeconds": max( + coerce_int( + config.get("pollIntervalSeconds") or config.get("pollInterval") or config.get("poll_interval"), + 15, + ), + 5, + ), + "skipExistingOnStart": coerce_bool(config.get("skipExistingOnStart"), True), + "skipAttachments": coerce_bool(config.get("skipAttachments"), False), + "allowAll": coerce_bool(config.get("allowAll"), False), + "requireAuthenticatedSender": coerce_bool( + config.get("requireAuthenticatedSender"), + True, + ), + "authservId": coerce_str(config.get("authservId") or config.get("authserv_id")), + "defaultSubject": coerce_str(config.get("defaultSubject")) or "Flocks Agent", + } diff --git a/flocks/channel/builtin/email/inbound.py b/flocks/channel/builtin/email/inbound.py new file mode 100644 index 000000000..ccfc87c5d --- /dev/null +++ b/flocks/channel/builtin/email/inbound.py @@ -0,0 +1,245 @@ +"""Inbound email parsing and sender safety helpers.""" + +from __future__ import annotations + +import email as email_lib +import re +from dataclasses import dataclass +from email.header import decode_header +from typing import Any, Optional + +from flocks.channel.base import ChatType, InboundMessage + +from .config import normalize_email_address + + +_NOREPLY_PATTERNS = ( + "noreply", "no-reply", "no_reply", "donotreply", "do-not-reply", + "mailer-daemon", "postmaster", "bounce", "notifications@", + "automated@", "auto-confirm", "auto-reply", "automailer", +) + +_AUTOMATED_HEADERS = { + "Auto-Submitted": lambda v: v.lower() != "no", + "Precedence": lambda v: v.lower() in {"bulk", "list", "junk"}, + "X-Auto-Response-Suppress": lambda v: bool(v), + "List-Unsubscribe": lambda v: bool(v), +} + +_AUTH_METHOD_RE = re.compile(r"\b(dmarc|dkim|spf)\s*=\s*([a-z]+)", re.IGNORECASE) +_AUTH_PROP_RE = re.compile( + r"\b(header\.from|header\.d|smtp\.mailfrom|smtp\.from|envelope-from)\s*=\s*([^\s;]+)", + re.IGNORECASE, +) + + +@dataclass +class ParsedEmail: + inbound: InboundMessage + subject: str + message_id: str + references: str + + +def decode_header_value(raw: str) -> str: + parts = decode_header(raw or "") + decoded: list[str] = [] + for part, charset in parts: + if isinstance(part, bytes): + decoded.append(part.decode(charset or "utf-8", errors="replace")) + else: + decoded.append(part) + return " ".join(decoded).strip() + + +def extract_email_address(raw: str) -> str: + match = re.search(r"<([^>]+)>", raw or "") + if match: + return normalize_email_address(match.group(1)) + return normalize_email_address(raw or "") + + +def extract_sender_name(raw: str, fallback: str) -> str: + decoded = decode_header_value(raw) + if "<" in decoded: + decoded = decoded.split("<", 1)[0].strip().strip('"') + return decoded or fallback + + +def strip_html(html: str) -> str: + text = re.sub(r"", "\n", html, flags=re.IGNORECASE) + text = re.sub(r"]*>", "\n", text, flags=re.IGNORECASE) + text = re.sub(r"

", "\n", text, flags=re.IGNORECASE) + text = re.sub(r"<[^>]+>", "", text) + replacements = { + " ": " ", + "&": "&", + "<": "<", + ">": ">", + """: '"', + "'": "'", + } + for old, new in replacements.items(): + text = text.replace(old, new) + return re.sub(r"\n{3,}", "\n\n", text).strip() + + +def extract_text_body(msg: email_lib.message.Message) -> str: + if msg.is_multipart(): + for preferred in ("text/plain", "text/html"): + for part in msg.walk(): + content_type = part.get_content_type() + disposition = str(part.get("Content-Disposition", "")) + if "attachment" in disposition or content_type != preferred: + continue + payload = part.get_payload(decode=True) + if not payload: + continue + charset = part.get_content_charset() or "utf-8" + text = payload.decode(charset, errors="replace") + return strip_html(text) if preferred == "text/html" else text + return "" + + payload = msg.get_payload(decode=True) + if not payload: + return "" + charset = msg.get_content_charset() or "utf-8" + text = payload.decode(charset, errors="replace") + return strip_html(text) if msg.get_content_type() == "text/html" else text + + +def attachment_summaries( + msg: email_lib.message.Message, + *, + skip_attachments: bool, +) -> list[str]: + if skip_attachments or not msg.is_multipart(): + return [] + + summaries: list[str] = [] + for part in msg.walk(): + disposition = str(part.get("Content-Disposition", "")) + if "attachment" not in disposition and "inline" not in disposition: + continue + content_type = part.get_content_type() + if content_type in {"text/plain", "text/html"} and "attachment" not in disposition: + continue + filename = part.get_filename() + filename = decode_header_value(filename) if filename else "attachment" + summaries.append(f"{filename} ({content_type})") + return summaries + + +def is_automated_sender(address: str, headers: dict[str, str]) -> bool: + addr = normalize_email_address(address) + if any(pattern in addr for pattern in _NOREPLY_PATTERNS): + return True + for header, check in _AUTOMATED_HEADERS.items(): + value = headers.get(header, "") + if value and check(value): + return True + return False + + +def _domain_of(address: str) -> str: + _, _, domain = normalize_email_address(address).rpartition("@") + return domain.strip().lower() + + +def _domains_aligned(a: str, b: str) -> bool: + a = (a or "").strip().lower().rstrip(".") + b = (b or "").strip().lower().rstrip(".") + if not a or not b: + return False + return a == b or a.endswith("." + b) or b.endswith("." + a) + + +def verify_sender_authentication( + msg: email_lib.message.Message, + from_addr: str, + *, + authserv_id: str = "", +) -> tuple[bool, str]: + from_domain = _domain_of(from_addr) + if not from_domain: + return False, "missing From domain" + + headers = msg.get_all("Authentication-Results") or [] + if not headers: + return False, "no Authentication-Results header" + + trusted = None + for raw in headers: + value = " ".join(str(raw).split()) + if authserv_id: + serv = value.split(";", 1)[0].strip().lower() + if not _domains_aligned(serv, authserv_id) and serv != authserv_id.lower(): + continue + trusted = value + break + if trusted is None: + return False, "no Authentication-Results from trusted authserv-id" + + methods = {m.lower(): r.lower() for m, r in _AUTH_METHOD_RE.findall(trusted)} + props = {p.lower(): v.strip().strip('"') for p, v in _AUTH_PROP_RE.findall(trusted)} + + if methods.get("dmarc") == "pass": + return True, "dmarc=pass" + + if methods.get("spf") == "pass": + spf_domain = _domain_of(props.get("smtp.mailfrom", "")) or props.get("smtp.from", "") or props.get("envelope-from", "") + spf_domain = _domain_of(spf_domain) if "@" in spf_domain else spf_domain + if _domains_aligned(spf_domain, from_domain): + return True, "spf=pass aligned" + + if methods.get("dkim") == "pass": + dkim_domain = props.get("header.d", "") or _domain_of(props.get("header.from", "")) + if _domains_aligned(dkim_domain, from_domain): + return True, "dkim=pass aligned" + + return False, f"authentication failed ({trusted[:120]})" + + +def build_inbound_message( + msg: email_lib.message.Message, + *, + uid: str, + account_id: str, + skip_attachments: bool, +) -> Optional[ParsedEmail]: + sender_raw = msg.get("From", "") + sender_addr = extract_email_address(sender_raw) + if not sender_addr: + return None + + subject = decode_header_value(msg.get("Subject", "(no subject)")) or "(no subject)" + message_id = str(msg.get("Message-ID", "")).strip() or f"" + in_reply_to = str(msg.get("In-Reply-To", "")).strip() + references = str(msg.get("References", "")).strip() + body = extract_text_body(msg).strip() + attachments = attachment_summaries(msg, skip_attachments=skip_attachments) + + text = body or "(empty email)" + if attachments: + text = f"{text}\n\n[Attachments]\n" + "\n".join(f"- {item}" for item in attachments) + if subject and not subject.lower().startswith("re:"): + text = f"[Subject: {subject}]\n\n{text}" + + return ParsedEmail( + inbound=InboundMessage( + channel_id="email", + account_id=account_id, + message_id=message_id, + sender_id=sender_addr, + sender_name=extract_sender_name(sender_raw, sender_addr), + chat_id=sender_addr, + chat_type=ChatType.DIRECT, + text=text, + reply_to_id=in_reply_to or None, + thread_id=references or in_reply_to or message_id, + raw={"uid": uid, "date": msg.get("Date", ""), "subject": subject}, + ), + subject=subject, + message_id=message_id, + references=references, + ) diff --git a/flocks/channel/registry.py b/flocks/channel/registry.py index c782f937a..f5cd3f530 100644 --- a/flocks/channel/registry.py +++ b/flocks/channel/registry.py @@ -91,6 +91,7 @@ def reset(self) -> None: def _register_builtin_channels(self) -> None: from flocks.channel.builtin.dingtalk.channel import DingTalkChannel + from flocks.channel.builtin.email.channel import EmailChannel from flocks.channel.builtin.feishu.channel import FeishuChannel from flocks.channel.builtin.telegram.channel import TelegramChannel from flocks.channel.builtin.wecom.channel import WeComChannel @@ -100,6 +101,7 @@ def _register_builtin_channels(self) -> None: self.register(TelegramChannel()) self.register(DingTalkChannel()) self.register(WeixinChannel()) + self.register(EmailChannel()) def _register_plugin_extension_point(self) -> None: from flocks.plugin import PluginLoader, ExtensionPoint diff --git a/tests/channel/test_email.py b/tests/channel/test_email.py new file mode 100644 index 000000000..f0f8ef044 --- /dev/null +++ b/tests/channel/test_email.py @@ -0,0 +1,356 @@ +from __future__ import annotations + +import uuid +from email.mime.multipart import MIMEMultipart +from email.mime.text import MIMEText +from unittest.mock import MagicMock + +import pytest + +from flocks.channel.base import ChatType, OutboundContext +from flocks.channel.builtin.email.channel import EmailChannel +from flocks.channel.builtin.email.config import parse_allowed_senders, resolved_config +from flocks.channel.builtin.email.inbound import ( + build_inbound_message, + decode_header_value, + extract_text_body, + strip_html, + verify_sender_authentication, +) + + +def _raw_email( + *, + sender: str = "User ", + subject: str = "Hello", + body: str = "Test body", + auth_results: str | None = None, +) -> bytes: + msg = MIMEText(body, "plain", "utf-8") + msg["From"] = sender + msg["Subject"] = subject + msg["Message-ID"] = f"<{uuid.uuid4().hex[:8]}@example.com>" + if auth_results: + msg["Authentication-Results"] = auth_results + return msg.as_bytes() + + +def test_email_channel_meta_and_validate_config() -> None: + plugin = EmailChannel() + + assert plugin.meta().id == "email" + assert "mail" in plugin.meta().aliases + assert ChatType.DIRECT in plugin.capabilities().chat_types + assert "address" in (plugin.validate_config({}) or "") + + error = plugin.validate_config({ + "address": "agent@example.com", + "password": "pw", + "imapHost": "imap.example.com", + "smtpHost": "smtp.example.com", + }) + assert error and "allowFrom" in error + + assert plugin.validate_config({ + "address": "agent@example.com", + "password": "pw", + "imapHost": "imap.example.com", + "smtpHost": "smtp.example.com", + "allowFrom": ["user@example.com"], + "authservId": "mx.example.com", + }) is None + + error = plugin.validate_config({ + "address": "agent@example.com", + "password": "pw", + "imapHost": "imap.example.com", + "smtpHost": "smtp.example.com", + "allowFrom": ["user@example.com"], + }) + assert error and "authservId" in error + + +def test_email_channel_registered_as_builtin() -> None: + from flocks.channel.registry import ChannelRegistry + + registry = ChannelRegistry() + registry._register_builtin_channels() + + assert registry.get("email") is not None + assert registry.get("mail") is registry.get("email") + + +def test_config_normalizes_allowed_senders() -> None: + cfg = resolved_config({ + "address": "Agent@Example.COM", + "password": "pw", + "imapHost": "imap.example.com", + "smtpHost": "smtp.example.com", + "allowFrom": "User , second@example.com", + }) + + assert cfg["address"] == "agent@example.com" + assert parse_allowed_senders(cfg) == {"user@example.com", "second@example.com"} + + +def test_email_parsing_helpers() -> None: + assert decode_header_value("=?utf-8?B?TWVyaGFiYQ==?=") == "Merhaba" + assert strip_html("

Hello
world & team

") == "Hello\nworld & team" + + msg = MIMEMultipart("alternative") + msg.attach(MIMEText("

HTML

", "html", "utf-8")) + msg.attach(MIMEText("Plain", "plain", "utf-8")) + assert extract_text_body(msg) == "Plain" + + +def test_build_inbound_message_includes_subject_and_attachment_names() -> None: + msg = MIMEMultipart() + msg["From"] = "User " + msg["Subject"] = "Investigate" + msg["Message-ID"] = "" + msg.attach(MIMEText("Please check this.", "plain", "utf-8")) + + attachment = MIMEText("payload", "plain", "utf-8") + attachment.add_header("Content-Disposition", "attachment", filename="report.txt") + msg.attach(attachment) + + parsed = build_inbound_message(msg, uid="1", account_id="default", skip_attachments=False) + + assert parsed is not None + inbound = parsed.inbound + assert inbound.channel_id == "email" + assert inbound.chat_type == ChatType.DIRECT + assert inbound.sender_id == "user@example.com" + assert inbound.chat_id == "user@example.com" + assert "[Subject: Investigate]" in inbound.text + assert "report.txt" in inbound.text + + +def test_sender_authentication_accepts_dmarc_pass() -> None: + msg = MIMEText("hi", "plain", "utf-8") + msg["Authentication-Results"] = "mx.example.com; dmarc=pass header.from=example.com" + + assert verify_sender_authentication(msg, "user@example.com")[0] is True + + +def test_sender_authentication_uses_only_trusted_authserv_id() -> None: + msg = MIMEText("hi", "plain", "utf-8") + msg["Authentication-Results"] = "attacker.example; dmarc=pass header.from=example.com" + msg["Authentication-Results"] = "mx.example.com; dmarc=fail header.from=example.com" + + assert verify_sender_authentication( + msg, + "user@example.com", + authserv_id="mx.example.com", + )[0] is False + + trusted_pass = MIMEText("hi", "plain", "utf-8") + trusted_pass["Authentication-Results"] = "attacker.example; dmarc=fail header.from=example.com" + trusted_pass["Authentication-Results"] = "mx.example.com; dmarc=pass header.from=example.com" + + assert verify_sender_authentication( + trusted_pass, + "user@example.com", + authserv_id="mx.example.com", + )[0] is True + + +def test_parse_and_authorize_blocks_unallowlisted_sender() -> None: + plugin = EmailChannel() + plugin._resolved = resolved_config({ + "address": "agent@example.com", + "password": "pw", + "imapHost": "imap.example.com", + "smtpHost": "smtp.example.com", + "allowFrom": ["allowed@example.com"], + }) + msg = MIMEText("hello", "plain", "utf-8") + msg["From"] = "user@example.com" + msg["Subject"] = "Hello" + msg["Message-ID"] = "" + + assert plugin._parse_and_authorize(msg, "1") is None + + +def test_parse_and_authorize_requires_authenticated_sender_for_allowlist() -> None: + plugin = EmailChannel() + plugin._resolved = resolved_config({ + "address": "agent@example.com", + "password": "pw", + "imapHost": "imap.example.com", + "smtpHost": "smtp.example.com", + "allowFrom": ["user@example.com"], + "authservId": "mx.example.com", + }) + msg = MIMEText("hello", "plain", "utf-8") + msg["From"] = "user@example.com" + msg["Subject"] = "Hello" + msg["Message-ID"] = "" + + assert plugin._parse_and_authorize(msg, "1") is None + + msg["Authentication-Results"] = "mx.example.com; dmarc=pass header.from=example.com" + inbound = plugin._parse_and_authorize(msg, "2") + assert inbound is not None + assert inbound.sender_id == "user@example.com" + + +def test_send_email_threads_reply(monkeypatch: pytest.MonkeyPatch) -> None: + plugin = EmailChannel() + plugin._resolved = resolved_config({ + "address": "agent@example.com", + "password": "pw", + "imapHost": "imap.example.com", + "smtpHost": "smtp.example.com", + "allowAll": True, + }) + plugin._thread_context["user@example.com"] = { + "subject": "Question", + "message_id": "", + "references": "", + } + + sent = {} + + class FakeSMTP: + def login(self, address, password): + sent["login"] = (address, password) + + def send_message(self, msg): + sent["msg"] = msg + + def quit(self): + sent["quit"] = True + + monkeypatch.setattr(plugin, "_connect_smtp", lambda: FakeSMTP()) + + message_id = plugin._send_email("user@example.com", "reply", None, None, None) + + assert message_id.startswith("" + assert msg["References"] == "" + + +def test_send_email_prefers_requested_thread_context(monkeypatch: pytest.MonkeyPatch) -> None: + plugin = EmailChannel() + plugin._resolved = resolved_config({ + "address": "agent@example.com", + "password": "pw", + "imapHost": "imap.example.com", + "smtpHost": "smtp.example.com", + "allowAll": True, + }) + plugin._thread_context["user@example.com"] = { + "subject": "Second question", + "message_id": "", + "references": "", + } + plugin._thread_context[""] = { + "subject": "First question", + "message_id": "", + "references": " ", + } + + sent = {} + + class FakeSMTP: + def login(self, address, password): + pass + + def send_message(self, msg): + sent["msg"] = msg + + def quit(self): + pass + + monkeypatch.setattr(plugin, "_connect_smtp", lambda: FakeSMTP()) + + plugin._send_email( + "user@example.com", + "reply", + "", + "", + None, + ) + + msg = sent["msg"] + assert msg["Subject"] == "Re: First question" + assert msg["In-Reply-To"] == "" + assert msg["References"] == "" + + +def test_email_password_is_extracted_to_secret(monkeypatch: pytest.MonkeyPatch) -> None: + from flocks.security.channel_secrets import extract_channel_secrets + + class FakeSecretManager: + def __init__(self) -> None: + self.values = {} + + def set(self, key: str, value: str) -> None: + self.values[key] = value + + fake = FakeSecretManager() + monkeypatch.setattr("flocks.security.secrets.get_secret_manager", lambda: fake) + + result = extract_channel_secrets({ + "email": { + "enabled": True, + "address": "agent@example.com", + "password": "plain-password", + } + }) + + assert result["email"]["password"] == "{secret:channel_email_password}" + assert fake.values["channel_email_password"] == "plain-password" + + +@pytest.mark.asyncio +async def test_send_text_rejects_invalid_target() -> None: + plugin = EmailChannel() + plugin._resolved = resolved_config({ + "address": "agent@example.com", + "password": "pw", + "imapHost": "imap.example.com", + "smtpHost": "smtp.example.com", + "allowAll": True, + }) + + result = await plugin.send_text( + OutboundContext(channel_id="email", to="not-an-email", text="hello") + ) + + assert result.success is False + assert "valid recipient" in (result.error or "") + + +def test_fetch_new_messages_skips_malformed_imap_response(monkeypatch: pytest.MonkeyPatch) -> None: + plugin = EmailChannel() + plugin._resolved = resolved_config({ + "address": "agent@example.com", + "password": "pw", + "imapHost": "imap.example.com", + "smtpHost": "smtp.example.com", + "allowAll": True, + }) + + fetch_calls = iter([ + ("OK", [None]), + ("OK", [(b"2 (RFC822 {123}", _raw_email(auth_results="mx.example.com; dmarc=pass header.from=example.com"))]), + ]) + + fake_imap = MagicMock() + fake_imap.uid.side_effect = lambda command, *args: ( + ("OK", [b"1 2"]) if command == "search" else next(fetch_calls) + ) + + monkeypatch.setattr( + "flocks.channel.builtin.email.channel.imaplib.IMAP4_SSL", + lambda *args, **kwargs: fake_imap, + ) + + messages = plugin._fetch_new_messages() + + assert len(messages) == 1 + assert messages[0].sender_id == "user@example.com" diff --git a/webui/src/locales/en-US/channel.json b/webui/src/locales/en-US/channel.json index fa058f8f2..5a4796da9 100644 --- a/webui/src/locales/en-US/channel.json +++ b/webui/src/locales/en-US/channel.json @@ -12,6 +12,7 @@ "wecom": "WeCom", "dingtalk": "DingTalk", "telegram": "Telegram", + "email": "Email", "weixin": "WeChat" }, "status": { @@ -234,6 +235,54 @@ "dedupTtlSecondsHint": "Retention time for message deduplication records (default 86400s = 24h)", "optional": "Optional" }, + "email": { + "credentials": "Mailbox Credentials", + "credentialsDesc": "Use a dedicated mailbox with IMAP enabled and an app password.", + "address": "Email Address", + "addressHint": "Address that receives user mail and sends agent replies.", + "password": "Password", + "passwordHint": "Mailbox password or app password. Stored in the Flocks secret store.", + "servers": "Mail Servers", + "serversDesc": "Configure the IMAP inbox and SMTP sender used by this channel.", + "imapHost": "IMAP Host", + "imapHostHint": "Inbox server, for example imap.gmail.com or outlook.office365.com.", + "imapPort": "IMAP Port", + "imapPortHint": "Default is 993 for IMAP over SSL.", + "smtpHost": "SMTP Host", + "smtpHostHint": "Outgoing mail server, for example smtp.gmail.com or smtp.office365.com.", + "smtpPort": "SMTP Port", + "smtpPortHint": "Default is 587 for STARTTLS, or 465 for implicit TLS.", + "accessControl": "Access Control", + "accessControlDesc": "Email is closed by default. Add allowed senders or explicitly allow everyone.", + "allowAll": "Allow All Senders", + "allowAllHint": "Accept mail from any sender. Use only with a dedicated public inbox.", + "allowAllLabel": "Allow all senders", + "allowFrom": "Allowed Senders", + "allowFromHint": "Only these email addresses can trigger the agent.", + "allowFromPlaceholder": "Enter email address and press Enter", + "requireAuthenticatedSender": "Sender Authentication", + "requireAuthenticatedSenderHint": "When an allowlist is used, require SPF/DKIM/DMARC evidence from Authentication-Results before trusting From.", + "requireAuthenticatedSenderLabel": "Require authenticated From domain", + "authservId": "Authserv ID", + "authservIdHint": "Trusted Authentication-Results authserv-id, usually your receiving mail server. Required when sender authentication is enabled.", + "behavior": "Message Behavior", + "behaviorDesc": "Configure routing defaults for email-originated sessions and replies.", + "defaultAgent": "Default Agent", + "defaultAgentHint": "Default Agent ID used when no Agent is specified.", + "defaultSubject": "Default Subject", + "defaultSubjectHint": "Subject used for proactive sends without an existing email thread.", + "advanced": "Advanced Settings", + "advancedDesc": "Polling, startup, and attachment handling.", + "pollIntervalSeconds": "Poll Interval (s)", + "pollIntervalSecondsHint": "Seconds between IMAP checks. Minimum is 5 seconds.", + "skipExistingOnStart": "Skip Existing Mail", + "skipExistingOnStartHint": "Ignore all mail already in the inbox when the channel starts, including unread mail received while the channel was offline.", + "skipExistingOnStartLabel": "Only process new mail after startup", + "skipAttachments": "Skip Attachments", + "skipAttachmentsHint": "Ignore attachment metadata in inbound mail.", + "skipAttachmentsLabel": "Skip inbound attachments", + "optional": "Optional" + }, "weixin": { "enableTitle": "Enable WeChat Channel", "enableDesc": "When enabled, the bot connects to WeChat personal accounts via the iLink Bot long-poll API.", diff --git a/webui/src/locales/zh-CN/channel.json b/webui/src/locales/zh-CN/channel.json index fbd4c2cbc..8c98d21c8 100644 --- a/webui/src/locales/zh-CN/channel.json +++ b/webui/src/locales/zh-CN/channel.json @@ -12,6 +12,7 @@ "wecom": "企业微信", "dingtalk": "钉钉", "telegram": "Telegram", + "email": "邮件", "weixin": "微信" }, "status": { @@ -236,6 +237,54 @@ "dedupTtlSecondsHint": "消息去重记录的保留时间(默认 86400 秒 = 24 小时)", "optional": "选填" }, + "email": { + "credentials": "邮箱凭证", + "credentialsDesc": "建议使用专用邮箱,并启用 IMAP 与应用专用密码。", + "address": "邮箱地址", + "addressHint": "用于接收用户邮件并发送 Agent 回复的邮箱地址。", + "password": "邮箱密码", + "passwordHint": "邮箱密码或应用专用密码。保存时会进入 Flocks 密钥存储。", + "servers": "邮件服务器", + "serversDesc": "配置此通道使用的 IMAP 收件箱和 SMTP 发件服务器。", + "imapHost": "IMAP 主机", + "imapHostHint": "收件服务器,例如 imap.gmail.com 或 outlook.office365.com。", + "imapPort": "IMAP 端口", + "imapPortHint": "默认 993,使用 IMAP SSL。", + "smtpHost": "SMTP 主机", + "smtpHostHint": "发件服务器,例如 smtp.gmail.com 或 smtp.office365.com。", + "smtpPort": "SMTP 端口", + "smtpPortHint": "默认 587 使用 STARTTLS;465 使用隐式 TLS。", + "accessControl": "访问控制", + "accessControlDesc": "邮件通道默认关闭开放访问。请添加允许发件人,或显式允许所有人。", + "allowAll": "允许所有发件人", + "allowAllHint": "接受任何发件人的邮件。仅建议用于专用公开邮箱。", + "allowAllLabel": "允许所有发件人", + "allowFrom": "允许的发件人", + "allowFromHint": "只有这些邮箱地址可以触发 Agent。", + "allowFromPlaceholder": "输入邮箱地址后按回车添加", + "requireAuthenticatedSender": "发件人认证", + "requireAuthenticatedSenderHint": "使用白名单时,要求 Authentication-Results 中存在 SPF/DKIM/DMARC 证据后才信任 From。", + "requireAuthenticatedSenderLabel": "要求 From 域通过认证", + "authservId": "Authserv ID", + "authservIdHint": "可信的 Authentication-Results authserv-id,通常是收件服务器;启用发件人认证时必填。", + "behavior": "消息行为", + "behaviorDesc": "配置邮件来源会话和回复的默认行为。", + "defaultAgent": "默认 Agent", + "defaultAgentHint": "未指定 Agent 时使用的默认 Agent ID。", + "defaultSubject": "默认主题", + "defaultSubjectHint": "主动发送且没有既有邮件线程时使用的主题。", + "advanced": "高级设置", + "advancedDesc": "轮询、启动和附件处理设置。", + "pollIntervalSeconds": "轮询间隔 (秒)", + "pollIntervalSecondsHint": "两次 IMAP 检查之间的秒数,最小 5 秒。", + "skipExistingOnStart": "跳过已有邮件", + "skipExistingOnStartHint": "通道启动时忽略收件箱中已有邮件,包括通道离线期间收到的未读邮件。", + "skipExistingOnStartLabel": "仅处理启动后的新邮件", + "skipAttachments": "跳过附件", + "skipAttachmentsHint": "入站邮件中忽略附件元数据。", + "skipAttachmentsLabel": "跳过入站附件", + "optional": "选填" + }, "weixin": { "enableTitle": "启用微信通道", "enableDesc": "启用后,机器人将通过 iLink Bot 长轮询接入微信,并开始接收和回复消息。", diff --git a/webui/src/pages/Channel/index.tsx b/webui/src/pages/Channel/index.tsx index 963a16693..5e1d6297e 100644 --- a/webui/src/pages/Channel/index.tsx +++ b/webui/src/pages/Channel/index.tsx @@ -129,6 +129,25 @@ interface TelegramChannelConfig { streamingCoalesceMs?: number; } +interface EmailChannelConfig { + enabled: boolean; + address?: string; + password?: string; + imapHost?: string; + imapPort?: number; + smtpHost?: string; + smtpPort?: number; + pollIntervalSeconds?: number; + allowFrom?: string[]; + allowAll?: boolean; + skipExistingOnStart?: boolean; + skipAttachments?: boolean; + requireAuthenticatedSender?: boolean; + authservId?: string; + defaultSubject?: string; + defaultAgent?: string; +} + interface WeixinChannelConfig { enabled: boolean; token?: string; @@ -144,7 +163,13 @@ interface WeixinChannelConfig { dataDir?: string; } -type ChannelConfig = FeishuChannelConfig | WeComChannelConfig | DingTalkChannelConfig | TelegramChannelConfig | WeixinChannelConfig; +type ChannelConfig = + | FeishuChannelConfig + | WeComChannelConfig + | DingTalkChannelConfig + | TelegramChannelConfig + | EmailChannelConfig + | WeixinChannelConfig; function defaultFeishuConfig(): FeishuChannelConfig { return { @@ -191,6 +216,21 @@ function defaultTelegramConfig(): TelegramChannelConfig { }; } +function defaultEmailConfig(): EmailChannelConfig { + return { + enabled: false, + imapPort: 993, + smtpPort: 587, + pollIntervalSeconds: 15, + allowFrom: [], + allowAll: false, + skipExistingOnStart: true, + skipAttachments: false, + requireAuthenticatedSender: true, + defaultSubject: 'Flocks Agent', + }; +} + function defaultWeixinConfig(): WeixinChannelConfig { return { enabled: false, @@ -644,6 +684,7 @@ function ConnectionStatusPanel({ status, config, channelId }: ConnectionStatusPa {channelId === 'dingtalk' && 'Stream'} {channelId === 'weixin' && 'Long-Poll'} {channelId === 'telegram' && ((config as TelegramChannelConfig).mode === 'webhook' ? 'Webhook' : 'Polling')} + {channelId === 'email' && 'IMAP Polling'}
@@ -1396,6 +1437,154 @@ function TelegramPanel({ config, onChange, onRefresh }: TelegramPanelProps) { ); } +// ============================================================================ +// Email Config Panel +// ============================================================================ + +interface EmailPanelProps { + config: EmailChannelConfig; + onChange: (c: EmailChannelConfig) => void; +} + +function EmailPanel({ config, onChange }: EmailPanelProps) { + const { t } = useTranslation('channel'); + const set = useCallback( + (key: K, value: EmailChannelConfig[K]) => + onChange({ ...config, [key]: value }), + [config, onChange] + ); + + return ( + <> +
+ + set('address', v || undefined)} + placeholder="agent@example.com" + /> + + + set('password', v || undefined)} + placeholder="app password" + /> + +
+ +
+ + set('imapHost', v || undefined)} + placeholder="imap.gmail.com" + /> + + + set('imapPort', v)} + min={1} + /> + + + set('smtpHost', v || undefined)} + placeholder="smtp.gmail.com" + /> + + + set('smtpPort', v)} + min={1} + /> + +
+ +
+ + set('allowAll', v)} + label={t('email.allowAllLabel')} + /> + + {!(config.allowAll ?? false) && ( + + set('allowFrom', v)} + placeholder={t('email.allowFromPlaceholder')} + /> + + )} + + set('requireAuthenticatedSender', v)} + label={t('email.requireAuthenticatedSenderLabel')} + /> + + + set('authservId', v || undefined)} + placeholder={t('email.optional')} + /> + +
+ +
+ + set('defaultAgent', v || undefined)} + placeholder={t('email.optional')} + /> + + + set('defaultSubject', v || undefined)} + placeholder="Flocks Agent" + /> + +
+ +
+ + set('pollIntervalSeconds', v)} + min={5} + /> + + + set('skipExistingOnStart', v)} + label={t('email.skipExistingOnStartLabel')} + /> + + + set('skipAttachments', v)} + label={t('email.skipAttachmentsLabel')} + /> + +
+ + ); +} + // ============================================================================ // Weixin Config Panel // ============================================================================ @@ -1956,6 +2145,8 @@ export default function ChannelPage() { configs[ch.id] = { ...defaultDingTalkConfig(), ...saved }; } else if (ch.id === 'telegram') { configs[ch.id] = { ...defaultTelegramConfig(), ...saved }; + } else if (ch.id === 'email') { + configs[ch.id] = { ...defaultEmailConfig(), ...saved }; } else if (ch.id === 'weixin') { configs[ch.id] = { ...defaultWeixinConfig(), ...saved }; } else { @@ -2281,6 +2472,12 @@ export default function ChannelPage() { onRefresh={fetchAll} /> )} + {selectedId === 'email' && ( + handleChannelConfigChange('email', cfg)} + /> + )} {selectedId === 'weixin' && ( Date: Fri, 10 Jul 2026 17:59:40 +0800 Subject: [PATCH 16/49] fix(storage): harden SQLite corruption recovery Fail closed when the live primary database is moved or replaced, and defer unsafe online recovery to startup. Preserve quarantined WAL evidence and make offline recovery atomic, non-destructive, and fully validated. --- flocks/channel/inbound/session_binding.py | 31 +- flocks/storage/storage.py | 322 ++++- scripts/recover_raw_flocks_db.py | 928 ++++++++++----- tests/scripts/test_recover_raw_flocks_db.py | 1187 +++++++++++++++++++ tests/storage/test_storage.py | 381 +++++- 5 files changed, 2447 insertions(+), 402 deletions(-) create mode 100644 tests/scripts/test_recover_raw_flocks_db.py diff --git a/flocks/channel/inbound/session_binding.py b/flocks/channel/inbound/session_binding.py index 906f913c1..c9e49cd2b 100644 --- a/flocks/channel/inbound/session_binding.py +++ b/flocks/channel/inbound/session_binding.py @@ -135,6 +135,9 @@ async def _get_db() -> aiosqlite.Connection: """ global _db_conn, _db_ready, _db_owner_pid + from flocks.storage.storage import Storage + + await Storage._ensure_init() current_pid = os.getpid() if ( _db_conn is not None @@ -169,7 +172,6 @@ async def _get_db() -> aiosqlite.Connection: _db_ready = False _db_owner_pid = None - from flocks.storage.storage import Storage db_path = Storage.get_db_path() db_path.parent.mkdir(parents=True, exist_ok=True) @@ -230,17 +232,28 @@ async def _migrate_legacy_binding_agent_ids(db: aiosqlite.Connection) -> None: }) -async def close_binding_db() -> None: - """Close the persistent connection (call during shutdown).""" +async def _close_binding_db_locked(*, suppress_errors: bool) -> None: + """Invalidate and close ``_db_conn`` while ``_init_lock`` is held.""" + global _db_conn, _db_ready, _db_owner_pid - if _db_conn is not None: + + conn = _db_conn + _db_conn = None + _db_ready = False + _db_owner_pid = None + if conn is not None: try: - await _db_conn.close() + await conn.close() except Exception: - pass - _db_conn = None - _db_ready = False - _db_owner_pid = None + if not suppress_errors: + raise + + +async def close_binding_db() -> None: + """Close the persistent connection (call during shutdown).""" + + async with _init_lock: + await _close_binding_db_locked(suppress_errors=True) class SessionBindingService: diff --git a/flocks/storage/storage.py b/flocks/storage/storage.py index 4a1b1ed41..4bf9124f6 100644 --- a/flocks/storage/storage.py +++ b/flocks/storage/storage.py @@ -8,6 +8,7 @@ import os import shutil import subprocess +import uuid from contextlib import asynccontextmanager from pathlib import Path @@ -71,6 +72,7 @@ class Storage: _log = Log.create(service="storage") _db_path: Optional[Path] = None _initialized = False + _db_identity: Optional[Tuple[int, int]] = None # PID of the process that called ``init()``. Used by ``_ensure_init`` to # detect ``fork()`` (uvicorn ``--reload`` / multiprocessing workers) and # re-initialise per-process state so the parent's open SQLite file @@ -149,6 +151,34 @@ def get_db_path(cls) -> Path: data_dir = Config.get_data_path() return data_dir / "flocks.db" + @staticmethod + def _file_identity(db_path: Path) -> Optional[Tuple[int, int]]: + """Return the filesystem identity used to detect an online DB replacement.""" + + try: + stat_result = db_path.stat() + except OSError: + return None + return stat_result.st_dev, stat_result.st_ino + + @classmethod + def _assert_active_db_identity(cls, db_path: Path) -> None: + """Reject direct connections after the active primary DB was moved or replaced.""" + + if ( + not cls._initialized + or cls._init_pid != os.getpid() + or cls._db_path is None + or cls._db_identity is None + or db_path.resolve() != cls._db_path.resolve() + ): + return + if cls._file_identity(db_path) != cls._db_identity: + raise StorageError( + f"The active SQLite database changed on disk: {db_path}. " + "Refusing to open a connection to a different file identity; restart Flocks." + ) + @classmethod def get_workflow_db_path(cls) -> Path: """Return the SQLite database path for workflow-domain KV data.""" @@ -264,9 +294,10 @@ def _quarantine_corrupt_db(cls, db_path: Path) -> Optional[Path]: Returns the new location of the main file so callers can surface it in logs or recovery instructions. Returns ``None`` when there was - nothing to quarantine (no main file present), or when the rename - failed — in which case the caller must propagate the original error - because we cannot safely recreate the file in place. + nothing to quarantine (no main file present), or when the main file + and its data-bearing WAL cannot be kept together. In that case the + caller must propagate the original error because recovery without the + matching WAL could silently omit committed transactions. """ db_path = Path(db_path) if not db_path.exists(): @@ -276,12 +307,18 @@ def _quarantine_corrupt_db(cls, db_path: Path) -> Optional[Path]: timestamp = datetime.now(UTC).strftime("%Y%m%dT%H%M%S") suffix = f".corrupt.{timestamp}" + sidecar_suffixes = ("-wal", "-shm") new_main = db_path.with_name(db_path.name + suffix) # Avoid collision if multiple corruptions happen within the same - # second — fall back to a counter suffix. + # second — fall back to a counter suffix. Reserve the paired WAL/SHM + # names as well so every quarantined main file keeps the exact sidecar + # names SQLite expects: ``-wal`` and ``-shm``. counter = 1 - while new_main.exists(): + while new_main.exists() or any( + new_main.with_name(new_main.name + sidecar_suffix).exists() + for sidecar_suffix in sidecar_suffixes + ): new_main = db_path.with_name(f"{db_path.name}{suffix}.{counter}") counter += 1 @@ -297,20 +334,34 @@ def _quarantine_corrupt_db(cls, db_path: Path) -> Optional[Path]: ) return None - for sidecar_name in (f"{db_path.name}-wal", f"{db_path.name}-shm"): - side_path = db_path.with_name(sidecar_name) + for sidecar_suffix in sidecar_suffixes: + side_path = db_path.with_name(db_path.name + sidecar_suffix) if not side_path.exists(): continue try: - side_path.rename(side_path.with_name(sidecar_name + suffix)) + side_path.rename(new_main.with_name(new_main.name + sidecar_suffix)) except OSError as exc: - cls._log.warn( + log_method = cls._log.error if sidecar_suffix == "-wal" else cls._log.warn + log_method( "storage.quarantine.sidecar_rename_failed", { "path": str(side_path), "error": str(exc), }, ) + if sidecar_suffix == "-wal": + try: + new_main.rename(db_path) + except OSError as rollback_exc: + cls._log.error( + "storage.quarantine.rollback_failed", + { + "original_path": str(db_path), + "quarantined_path": str(new_main), + "error": str(rollback_exc), + }, + ) + return None cls._log.error( "storage.corruption.quarantined", @@ -355,13 +406,8 @@ async def _assert_integrity_check_ok(cls, db_path: Path) -> None: @classmethod def _try_sqlite_recover_sync(cls, quarantined_path: Path, target_path: Path) -> Optional[Path]: - """Try SQLite's lightweight `.recover` and install the recovered DB. + """Recover from an isolated copy without allowing SQLite to mutate evidence.""" - This intentionally avoids the heavier raw-page/WAL reconstruction script. - It handles the common case where SQLite can still scan a malformed DB - enough to emit recoverable SQL. Failure is non-fatal; callers fall back - to bootstrapping an empty database. - """ sqlite_bin = shutil.which("sqlite3") if sqlite_bin is None: cls._log.warn( @@ -374,6 +420,67 @@ def _try_sqlite_recover_sync(cls, quarantined_path: Path, target_path: Path) -> ) return None + working_path = quarantined_path.with_name( + f".{quarantined_path.name}.recovery-source-{uuid.uuid4().hex}" + ) + working_files = [ + working_path, + working_path.with_name(working_path.name + "-wal"), + working_path.with_name(working_path.name + "-shm"), + working_path.with_name(working_path.name + "-journal"), + ] + try: + shutil.copy2(quarantined_path, working_path) + working_path.chmod(0o600) + for suffix in ("-wal", "-shm"): + evidence_sidecar = quarantined_path.with_name(quarantined_path.name + suffix) + if not evidence_sidecar.exists(): + continue + working_sidecar = working_path.with_name(working_path.name + suffix) + shutil.copy2(evidence_sidecar, working_sidecar) + working_sidecar.chmod(0o600) + return cls._try_sqlite_recover_working_copy_sync( + working_path, + quarantined_path, + target_path, + sqlite_bin, + ) + except OSError as exc: + cls._log.warn( + "storage.corruption.recovery.failed", + { + "db_path": str(target_path), + "quarantined_path": str(quarantined_path), + "stage": "copy_evidence", + "error": str(exc), + }, + ) + return None + finally: + for path in working_files: + try: + path.unlink(missing_ok=True) + except OSError as exc: + cls._log.warn( + "storage.corruption.recovery.temp_cleanup_failed", + {"path": str(path), "error": str(exc)}, + ) + + @classmethod + def _try_sqlite_recover_working_copy_sync( + cls, + working_path: Path, + quarantined_path: Path, + target_path: Path, + sqlite_bin: str, + ) -> Optional[Path]: + """Try SQLite's lightweight `.recover` and install the recovered DB. + + This intentionally avoids the heavier raw-page/WAL reconstruction script. + It handles the common case where SQLite can still scan a malformed DB + enough to emit recoverable SQL. Failure is non-fatal; callers fall back + to bootstrapping an empty database. + """ recovered_path = target_path.with_name(target_path.name + ".recovered") sql_path = target_path.with_name(target_path.name + ".recover.sql") for path in (recovered_path, sql_path): @@ -383,8 +490,13 @@ def _try_sqlite_recover_sync(cls, quarantined_path: Path, target_path: Path) -> pass try: + lost_and_found_table = f"flocks_recovery_lost_{uuid.uuid4().hex}" completed = subprocess.run( - [sqlite_bin, str(quarantined_path), ".recover"], + [ + sqlite_bin, + str(working_path), + f".recover --lost-and-found {lost_and_found_table}", + ], check=False, capture_output=True, text=True, @@ -493,6 +605,42 @@ def _try_sqlite_recover_sync(cls, quarantined_path: Path, target_path: Path) -> ) return target_path + @staticmethod + async def _wait_for_recovery_worker_on_cancel( + worker: "asyncio.Task[Optional[Path]]", + ) -> Optional[Path]: + """Do not let a cancelled coroutine leave a recovery thread mutating files.""" + + try: + return await asyncio.shield(worker) + except asyncio.CancelledError as cancelled: + while not worker.done(): + try: + await asyncio.shield(worker) + except asyncio.CancelledError: + continue + except BaseException: + break + if worker.done() and not worker.cancelled(): + try: + worker.result() + except BaseException: + pass + raise cancelled + + @classmethod + async def _run_sqlite_recover_async( + cls, + quarantined_path: Path, + target_path: Path, + ) -> Optional[Path]: + """Run blocking sqlite recovery without outliving a cancelled startup task.""" + + worker = asyncio.create_task( + asyncio.to_thread(cls._try_sqlite_recover_sync, quarantined_path, target_path) + ) + return await cls._wait_for_recovery_worker_on_cancel(worker) + @classmethod async def recover_corrupt_db( cls, @@ -515,6 +663,26 @@ async def recover_corrupt_db( if generation is not None and generation != cls._corruption_recovery_generation: return False + is_primary_db = cls._db_path == db_path or db_path.name == "flocks.db" + is_live_process = ( + cls._initialized + and cls._init_pid is not None + and cls._init_pid == os.getpid() + ) + if is_primary_db and is_live_process: + cls._log.error( + "storage.corruption.online_recovery_refused", + { + "db_path": str(db_path), + "action": action, + "hint": "Restart Flocks so recovery runs before request connections open.", + }, + ) + raise StorageError( + "Refusing to replace the primary SQLite database while Flocks is running; " + "restart Flocks to recover it safely." + ) from exc + cls._log.error( "storage.corruption.detected", { @@ -527,15 +695,16 @@ async def recover_corrupt_db( quarantined = cls._quarantine_corrupt_db(db_path) if quarantined is None: raise exc - await asyncio.to_thread(cls._try_sqlite_recover_sync, quarantined, db_path) + await cls._run_sqlite_recover_async(quarantined, db_path) if cls._db_path == db_path: cls._initialized = False cls._init_pid = None + cls._db_identity = None if reinitialize is not None: await reinitialize() - elif cls._db_path == db_path or db_path.name == "flocks.db": + elif is_primary_db: await cls.init(db_path) cls._corruption_recovery_generation += 1 @@ -557,19 +726,29 @@ async def _run_with_corruption_recovery( db_path: Path, action: str, ) -> R: - generation = cls._corruption_recovery_generation + """Run a request operation without replacing its SQLite file in-place. + + Runtime corruption is deliberately deferred to the next process start, + where no request handlers or persistent connections can still reference + the old inode. Primary-DB online recovery is explicitly refused. + """ + try: return await operation() except Exception as exc: if not cls._is_db_corruption_error(exc): raise - await cls.recover_corrupt_db( - db_path, - action=action, - exc=exc, - generation=generation, + cls._log.error( + "storage.corruption.deferred_to_restart", + { + "db_path": str(db_path), + "action": action, + "error": str(exc), + "error_type": type(exc).__name__, + "hint": "Restart Flocks to run startup database recovery safely.", + }, ) - return await operation() + raise @classmethod def _is_sqlite_busy_error(cls, exc: Exception) -> bool: @@ -666,9 +845,11 @@ async def _run_write_with_retry( async def connect(cls, db_path: Optional[Path] = None) -> AsyncIterator[aiosqlite.Connection]: """Open a configured async SQLite connection for the active storage DB.""" target = Path(db_path) if db_path is not None else cls.get_db_path() + cls._assert_active_db_identity(target) target.parent.mkdir(parents=True, exist_ok=True) conn = await aiosqlite.connect(target, timeout=cls._sqlite_timeout_s) try: + cls._assert_active_db_identity(target) await cls.configure_connection(conn) yield conn finally: @@ -678,10 +859,16 @@ async def connect(cls, db_path: Optional[Path] = None) -> AsyncIterator[aiosqlit def connect_sync(cls, db_path: Optional[Path] = None) -> sqlite3.Connection: """Open a configured sync SQLite connection for the active storage DB.""" target = Path(db_path) if db_path is not None else cls.get_db_path() + cls._assert_active_db_identity(target) target.parent.mkdir(parents=True, exist_ok=True) conn = sqlite3.connect(target, timeout=cls._sqlite_timeout_s) - conn.row_factory = sqlite3.Row - return cls.configure_sync_connection(conn) + try: + cls._assert_active_db_identity(target) + conn.row_factory = sqlite3.Row + return cls.configure_sync_connection(conn) + except BaseException: + conn.close() + raise @classmethod def register_ddl(cls, ddl: DDLScript) -> None: @@ -815,13 +1002,59 @@ async def init(cls, db_path: Optional[Path] = None) -> None: db_path = data_dir / "flocks.db" db_path = Path(db_path) + current_pid = os.getpid() + same_process = cls._init_pid is None or cls._init_pid == current_pid + if cls._initialized and same_process and cls._db_path is not None: + if cls._db_path != db_path: + # An explicit path switch is allowed for tests and embedded + # callers, but the process-owned persistent binding handle + # must not silently remain attached to the previous DB. + from flocks.channel.inbound.session_binding import close_binding_db + + await close_binding_db() + cls._initialized = False + cls._init_pid = None + cls._db_identity = None + elif not db_path.exists(): + from flocks.channel.inbound.session_binding import close_binding_db + + await close_binding_db() + cls._log.error( + "storage.live_path_disappeared", + { + "db_path": str(db_path), + "hint": "Restart Flocks after restoring or recovering the database file.", + }, + ) + raise StorageError( + f"The active SQLite database disappeared: {db_path}. " + "Refusing to create a second database while existing handles may reference " + "the old file; restart Flocks." + ) + elif ( + cls._db_identity is not None + and cls._file_identity(db_path) != cls._db_identity + ): + from flocks.channel.inbound.session_binding import close_binding_db + + await close_binding_db() + cls._log.error( + "storage.live_path_replaced", + { + "db_path": str(db_path), + "hint": "Restart Flocks; never replace SQLite files while it is running.", + }, + ) + raise StorageError( + f"The active SQLite database was replaced on disk: {db_path}. " + "Refusing to mix connections to different file identities; restart Flocks." + ) + # Tests and short-lived processes may initialize Storage against a # temporary database that later disappears. We also force a # reinit after ``fork()`` (detected via PID mismatch) to avoid the # child silently reusing the parent's open SQLite handle. - current_pid = os.getpid() same_path = cls._db_path == db_path and db_path.exists() - same_process = cls._init_pid is None or cls._init_pid == current_pid if cls._initialized and same_path and same_process: return @@ -879,7 +1112,7 @@ async def init(cls, db_path: Optional[Path] = None) -> None: quarantined = cls._quarantine_corrupt_db(cls._db_path) if quarantined is None: raise - await asyncio.to_thread(cls._try_sqlite_recover_sync, quarantined, cls._db_path) + await cls._run_sqlite_recover_async(quarantined, cls._db_path) await cls._bootstrap_schema() # Drain any residual WAL frames left by the previous process so the @@ -893,6 +1126,7 @@ async def init(cls, db_path: Optional[Path] = None) -> None: cls._log.warn("storage.startup_checkpoint.failed", {"error": str(exc)}) cls._init_pid = os.getpid() + cls._db_identity = cls._file_identity(cls._db_path) cls._initialized = True cls._log.info( "storage.initialized", @@ -979,6 +1213,19 @@ async def shutdown(cls) -> None: """ if not cls._initialized: return + if ( + cls._db_path is not None + and cls._db_identity is not None + and cls._file_identity(cls._db_path) != cls._db_identity + ): + cls._log.warn( + "storage.shutdown.checkpoint.skipped_replaced_path", + {"db_path": str(cls._db_path)}, + ) + cls._initialized = False + cls._init_pid = None + cls._db_identity = None + return attempts = max(int(cls._shutdown_checkpoint_attempts), 1) backoff = max(float(cls._shutdown_checkpoint_backoff_s), 0.0) @@ -1046,6 +1293,7 @@ async def shutdown(cls) -> None: finally: cls._initialized = False cls._init_pid = None + cls._db_identity = None @classmethod async def _bootstrap_schema(cls) -> None: @@ -1197,8 +1445,20 @@ async def _ensure_init(cls) -> None: ) cls._initialized = False cls._init_pid = None + cls._db_identity = None - if not cls._initialized or cls._db_path is None or not cls._db_path.exists(): + identity_changed = ( + cls._initialized + and cls._db_path is not None + and cls._db_identity is not None + and cls._file_identity(cls._db_path) != cls._db_identity + ) + if ( + not cls._initialized + or cls._db_path is None + or not cls._db_path.exists() + or identity_changed + ): await cls.init(cls._db_path) @classmethod diff --git a/scripts/recover_raw_flocks_db.py b/scripts/recover_raw_flocks_db.py index b77088240..846ef56b2 100644 --- a/scripts/recover_raw_flocks_db.py +++ b/scripts/recover_raw_flocks_db.py @@ -1,4 +1,8 @@ -"""One-click recovery for a damaged Flocks SQLite database.""" +"""Offline recovery for a damaged Flocks SQLite database. + +The helper writes recovery artifacts only. It never deletes SQLite sidecars or +installs a recovered database over the live Flocks store. +""" from __future__ import annotations @@ -9,13 +13,19 @@ import struct import subprocess import shutil +import uuid +from contextlib import contextmanager from dataclasses import dataclass from pathlib import Path from typing import Dict, Sequence +from dotenv import load_dotenv + WAL_MAGIC = 0x377F0682 +WAL_MAGIC_BIG_ENDIAN_CHECKSUM = 0x377F0683 +WAL_MAGIC_VALUES = {WAL_MAGIC, WAL_MAGIC_BIG_ENDIAN_CHECKSUM} WAL_VERSION = 3007000 -COMMON_SQLITE_PAGE_SIZES = (4096, 8192, 2048, 1024, 16384, 32768, 65536) +COMMON_SQLITE_PAGE_SIZES = (4096, 8192, 2048, 1024, 512, 16384, 32768, 65536) STORAGE_DDL = """ CREATE TABLE IF NOT EXISTS storage ( @@ -276,12 +286,76 @@ class RecoveryArtifacts: extracted_db: Path recovered_db: Path summary_path: Path + lost_and_found_table: str pagesize: int wal_frames: int wal_final_db_pages: int copied_rows: Dict[str, int] +def _cleanup_temporary_sqlite_files(path: Path) -> None: + """Remove only temporary files created by the current recovery operation.""" + + for candidate in ( + path, + path.with_name(f"{path.name}-journal"), + path.with_name(f"{path.name}-wal"), + path.with_name(f"{path.name}-shm"), + ): + candidate.unlink(missing_ok=True) + + +def _publish_file_exclusive(temporary_path: Path, destination: Path) -> None: + """Atomically publish a sibling temporary file without replacing a destination.""" + + destination.parent.mkdir(parents=True, exist_ok=True) + if os.name == "nt": + try: + os.rename(temporary_path, destination) + except FileExistsError: + raise FileExistsError(f"Refusing to overwrite existing recovery file: {destination}") + return + try: + os.link(temporary_path, destination, follow_symlinks=False) + except FileExistsError: + raise FileExistsError(f"Refusing to overwrite existing recovery file: {destination}") + temporary_path.unlink() + + +@contextmanager +def _atomic_output_path(destination: Path): + """Yield a sibling temporary path and publish it only after successful completion.""" + + destination.parent.mkdir(parents=True, exist_ok=True) + temporary_path = destination.with_name(f".{destination.name}.tmp-{uuid.uuid4().hex}") + try: + yield temporary_path + if not temporary_path.is_file(): + raise RuntimeError(f"Recovery output was not created: {temporary_path}") + unexpected_sidecars = [ + candidate + for candidate in ( + temporary_path.with_name(f"{temporary_path.name}-journal"), + temporary_path.with_name(f"{temporary_path.name}-wal"), + temporary_path.with_name(f"{temporary_path.name}-shm"), + ) + if candidate.exists() + ] + if unexpected_sidecars: + raise RuntimeError( + "Recovery staging file still has SQLite sidecars: " + + ", ".join(str(path) for path in unexpected_sidecars) + ) + # Windows maps fsync() to _commit(), which requires a writable file + # descriptor. ``r+b`` works on every supported platform and keeps the + # durability step consistent before the no-replace publish. + with temporary_path.open("r+b") as handle: + os.fsync(handle.fileno()) + _publish_file_exclusive(temporary_path, destination) + finally: + _cleanup_temporary_sqlite_files(temporary_path) + + def _quote_identifier(value: str) -> str: return '"' + value.replace('"', '""') + '"' @@ -294,44 +368,100 @@ def _table_exists(conn: sqlite3.Connection, table_name: str) -> bool: return row is not None -def _table_columns(conn: sqlite3.Connection, table_name: str) -> list[str]: - rows = conn.execute(f"PRAGMA table_info({_quote_identifier(table_name)})").fetchall() - return [str(row[1]) for row in rows] +def _wal_checksum( + data: bytes, + *, + byte_order: str, + state: tuple[int, int] = (0, 0), +) -> tuple[int, int]: + """Return SQLite's rolling WAL checksum for an 8-byte-aligned payload.""" + + if len(data) % 8 != 0: + raise ValueError("WAL checksum input must be aligned to 8 bytes.") + words = struct.unpack(f"{byte_order}{len(data) // 4}I", data) + checksum_1, checksum_2 = state + for index in range(0, len(words), 2): + checksum_1 = (checksum_1 + words[index] + checksum_2) & 0xFFFFFFFF + checksum_2 = (checksum_2 + words[index + 1] + checksum_1) & 0xFFFFFFFF + return checksum_1, checksum_2 def _parse_wal_frames(wal_bytes: bytes) -> tuple[int, int, Dict[int, bytes], int]: - """Return WAL page size, final db pages, latest page map, and frame count.""" + """Return committed, checksum-valid WAL pages through the last transaction.""" if len(wal_bytes) < 32: raise ValueError("WAL file is too small to contain a valid header.") - magic, version, pagesize, _, _, _, _, _ = struct.unpack(">8I", wal_bytes[:32]) - if magic != WAL_MAGIC: + magic, version, pagesize, _, salt_1, salt_2, stored_1, stored_2 = struct.unpack( + ">8I", wal_bytes[:32] + ) + if magic not in WAL_MAGIC_VALUES: raise ValueError(f"Unexpected WAL magic: 0x{magic:08x}") if version != WAL_VERSION: raise ValueError(f"Unexpected WAL version: {version}") - if pagesize <= 0: + if pagesize < 512 or pagesize > 65536 or pagesize & (pagesize - 1): raise ValueError(f"Invalid WAL pagesize: {pagesize}") + byte_order = "<" if magic == WAL_MAGIC else ">" + checksum = _wal_checksum(wal_bytes[:24], byte_order=byte_order) + if checksum != (stored_1, stored_2): + raise ValueError("WAL header checksum is invalid.") + frame_size = 24 + pagesize payload = len(wal_bytes) - 32 - if payload < 0 or payload % frame_size != 0: - raise ValueError("WAL payload is not aligned to complete frames.") - - frame_count = payload // frame_size - latest_pages: Dict[int, bytes] = {} + available_frames = payload // frame_size + valid_frames: list[tuple[int, bytes]] = [] + last_commit_frame_count = 0 final_db_pages = 0 - for frame_index in range(frame_count): + for frame_index in range(available_frames): offset = 32 + frame_index * frame_size - page_no, db_page_count, *_ = struct.unpack(">6I", wal_bytes[offset : offset + 24]) - latest_pages[page_no] = wal_bytes[offset + 24 : offset + 24 + pagesize] + frame_header = wal_bytes[offset : offset + 24] + page = wal_bytes[offset + 24 : offset + 24 + pagesize] + page_no, db_page_count, frame_salt_1, frame_salt_2, frame_sum_1, frame_sum_2 = ( + struct.unpack(">6I", frame_header) + ) + if page_no == 0 or (frame_salt_1, frame_salt_2) != (salt_1, salt_2): + break + + next_checksum = _wal_checksum( + frame_header[:8], + byte_order=byte_order, + state=checksum, + ) + next_checksum = _wal_checksum( + page, + byte_order=byte_order, + state=next_checksum, + ) + if next_checksum != (frame_sum_1, frame_sum_2): + break + + checksum = next_checksum + valid_frames.append((page_no, page)) if db_page_count: final_db_pages = db_page_count + last_commit_frame_count = len(valid_frames) - if final_db_pages <= 0: + if final_db_pages <= 0 or last_commit_frame_count == 0: raise ValueError("WAL does not contain any committed frames.") - return pagesize, final_db_pages, latest_pages, frame_count + latest_pages: Dict[int, bytes] = {} + for page_no, page in valid_frames[:last_commit_frame_count]: + latest_pages[page_no] = page + + return pagesize, final_db_pages, latest_pages, last_commit_frame_count + + +def _read_header_pagesize(raw_bytes: bytes) -> int | None: + """Return a valid SQLite page size from the database header, if present.""" + + if len(raw_bytes) < 100 or not raw_bytes.startswith(b"SQLite format 3\x00"): + return None + encoded = int.from_bytes(raw_bytes[16:18], "big") + pagesize = 65536 if encoded == 1 else encoded + if pagesize < 512 or pagesize > 65536 or pagesize & (pagesize - 1): + return None + return pagesize def _guess_raw_pagesize(raw_bytes: bytes) -> int: @@ -363,14 +493,14 @@ def _build_synthetic_page1(pagesize: int, total_pages: int) -> bytes: page1[40:44] = (1).to_bytes(4, "big") page1[44:48] = (4).to_bytes(4, "big") page1[56:60] = (1).to_bytes(4, "big") - page1[96] = 0x0D - page1[97] = 0x00 - page1[98:100] = (0).to_bytes(2, "big") + page1[92:96] = (1).to_bytes(4, "big") + # Page 1's b-tree header begins after the 100-byte database header. + page1[100] = 0x0D + page1[101:103] = (0).to_bytes(2, "big") + page1[103:105] = (0).to_bytes(2, "big") cell_content_area = 0 if pagesize == 65536 else pagesize - page1[100:102] = cell_content_area.to_bytes(2, "big") - page1[102] = 0 - page1[103] = 0 - page1[104:108] = (0).to_bytes(4, "big") + page1[105:107] = cell_content_area.to_bytes(2, "big") + page1[107] = 0 return bytes(page1) @@ -387,43 +517,49 @@ def reconstruct_sqlite_candidate( if wal_path is not None: wal_bytes = wal_path.read_bytes() pagesize, final_db_pages, latest_pages, frame_count = _parse_wal_frames(wal_bytes) + header_pagesize = _read_header_pagesize(raw_bytes) + if header_pagesize is not None and header_pagesize != pagesize: + raise ValueError( + "WAL page size does not match the SQLite database header: " + f"{pagesize} != {header_pagesize}." + ) if len(raw_bytes) % pagesize != 0: raise ValueError( f"Raw file size {len(raw_bytes)} is not aligned to WAL pagesize {pagesize}." ) else: - pagesize = _guess_raw_pagesize(raw_bytes) + pagesize = _read_header_pagesize(raw_bytes) or _guess_raw_pagesize(raw_bytes) final_db_pages = len(raw_bytes) // pagesize latest_pages = {} frame_count = 0 - output_path.parent.mkdir(parents=True, exist_ok=True) - with output_path.open("wb") as handle: - if wal_path is None and not raw_has_header: - handle.write(_build_synthetic_page1(pagesize, final_db_pages)) - start_page = 2 - else: - start_page = 1 - - if wal_path is None and raw_has_header: - handle.write(raw_bytes) - return { - "pagesize": pagesize, - "wal_frames": 0, - "wal_final_db_pages": final_db_pages, - "wal_pages_used": 0, - } - - for page_no in range(1, final_db_pages + 1): - if wal_path is None and page_no < start_page: - continue - if page_no in latest_pages: - handle.write(latest_pages[page_no]) - continue - - offset = (page_no - 1) * pagesize - page = raw_bytes[offset : offset + pagesize] - handle.write(page if len(page) == pagesize else (b"\x00" * pagesize)) + with _atomic_output_path(output_path) as temporary_path: + with temporary_path.open("xb") as handle: + if wal_path is None and not raw_has_header: + handle.write(_build_synthetic_page1(pagesize, final_db_pages)) + start_page = 2 + else: + start_page = 1 + + if wal_path is None and raw_has_header: + handle.write(raw_bytes) + return { + "pagesize": pagesize, + "wal_frames": 0, + "wal_final_db_pages": final_db_pages, + "wal_pages_used": 0, + } + + for page_no in range(1, final_db_pages + 1): + if wal_path is None and page_no < start_page: + continue + if page_no in latest_pages: + handle.write(latest_pages[page_no]) + continue + + offset = (page_no - 1) * pagesize + page = raw_bytes[offset : offset + pagesize] + handle.write(page if len(page) == pagesize else (b"\x00" * pagesize)) return { "pagesize": pagesize, @@ -433,254 +569,199 @@ def reconstruct_sqlite_candidate( } -def _run_sqlite_recover(candidate_db: Path, recover_sql_path: Path) -> None: +def _run_sqlite_recover( + candidate_db: Path, + recover_sql_path: Path, + *, + lost_and_found_table: str, +) -> None: """Write `sqlite3 .recover` output to a SQL file.""" completed = subprocess.run( - ["sqlite3", str(candidate_db), ".recover"], + [ + "sqlite3", + str(candidate_db), + f".recover --lost-and-found {lost_and_found_table}", + ], check=False, capture_output=True, text=True, encoding="utf-8", ) - recover_sql_path.write_text(completed.stdout or "", encoding="utf-8") if completed.returncode != 0 and not completed.stdout.strip(): stderr = completed.stderr.strip() or completed.stdout.strip() raise RuntimeError(f"sqlite3 .recover failed: {stderr or completed.returncode}") + with _atomic_output_path(recover_sql_path) as temporary_path: + temporary_path.write_text(completed.stdout or "", encoding="utf-8") def _materialize_recovered_sql(recover_sql_path: Path, extracted_db_path: Path) -> None: """Execute recovered SQL into a scratch SQLite database.""" - if extracted_db_path.exists(): - extracted_db_path.unlink() - - completed = subprocess.run( - ["sqlite3", str(extracted_db_path)], - input=recover_sql_path.read_text(encoding="utf-8"), - check=False, - capture_output=True, - text=True, - encoding="utf-8", - ) - if completed.returncode != 0: - stderr = completed.stderr.strip() or completed.stdout.strip() - raise RuntimeError(f"Failed to materialize recovered SQL: {stderr}") - - -def _ensure_recovered_schema(output_db_path: Path) -> None: - """Create the normalized target schema.""" - - if output_db_path.exists(): - output_db_path.unlink() - - output_db_path.parent.mkdir(parents=True, exist_ok=True) - conn = sqlite3.connect(output_db_path) - try: - conn.executescript(STORAGE_DDL) - conn.executescript(USAGE_RECORDS_DDL) - conn.executescript(TASKS_DDL) - conn.executescript(CHANNEL_BINDINGS_DDL) - for stmt in (*USAGE_INDEX_STMTS, *TASK_INDEX_STMTS): - conn.execute(stmt) - conn.commit() - finally: - conn.close() - + with _atomic_output_path(extracted_db_path) as temporary_path: + completed = subprocess.run( + ["sqlite3", str(temporary_path)], + input=recover_sql_path.read_text(encoding="utf-8"), + check=False, + capture_output=True, + text=True, + encoding="utf-8", + ) + if completed.returncode != 0: + stderr = completed.stderr.strip() or completed.stdout.strip() + raise RuntimeError(f"Failed to materialize recovered SQL: {stderr}") -def _insert_rows( - target_conn: sqlite3.Connection, - table_name: str, - columns: Sequence[str], - rows: Sequence[Sequence[object]], -) -> int: - """Insert rows into a table with INSERT OR REPLACE.""" - - if not rows: - return 0 - - quoted_columns = ", ".join(_quote_identifier(column) for column in columns) - placeholders = ", ".join("?" for _ in columns) - target_conn.executemany( - ( - f"INSERT OR REPLACE INTO {_quote_identifier(table_name)} " - f"({quoted_columns}) VALUES ({placeholders})" - ), - rows, - ) - return len(rows) +def _schema_manifest(conn: sqlite3.Connection) -> dict[tuple[str, str], tuple[str, str | None]]: + """Return logical schema objects, excluding SQLite-managed internal objects.""" -def _copy_table_rows( - source_conn: sqlite3.Connection, - target_conn: sqlite3.Connection, - table_name: str, -) -> int: - """Copy shared columns from one table to another.""" - - if not _table_exists(source_conn, table_name) or not _table_exists(target_conn, table_name): - return 0 - - source_columns = _table_columns(source_conn, table_name) - target_columns = set(_table_columns(target_conn, table_name)) - copy_columns = [column for column in source_columns if column in target_columns] - if not copy_columns: - return 0 - - quoted_columns = ", ".join(_quote_identifier(column) for column in copy_columns) - rows = source_conn.execute( - f"SELECT {quoted_columns} FROM {_quote_identifier(table_name)}" + rows = conn.execute( + """ + SELECT type, name, tbl_name, sql + FROM sqlite_schema + WHERE name NOT LIKE 'sqlite_%' + ORDER BY type, name + """ ).fetchall() - if not rows: - return 0 - - return _insert_rows(target_conn, table_name, copy_columns, rows) + return { + (str(row[0]), str(row[1])): (str(row[2]), None if row[3] is None else str(row[3])) + for row in rows + } -def _copy_lost_and_found_rows( +def _validate_recovered_db( source_conn: sqlite3.Connection, target_conn: sqlite3.Connection, - table_name: str, -) -> int: - """Recover business rows from `.recover` lost_and_found output.""" - - if not _table_exists(source_conn, "lost_and_found"): - return 0 - - if table_name == "storage": - rows = [ - tuple(row) - for row in source_conn.execute( - """ - SELECT c0, c1, c2, c3, c4 - FROM lost_and_found - WHERE nfield = 5 - AND c0 NOT IN ('table', 'index') - AND typeof(c0) = 'text' - AND typeof(c1) = 'text' - AND typeof(c2) = 'text' - AND typeof(c3) = 'text' - AND typeof(c4) = 'text' - """ - ).fetchall() - if ":" in str(row[0]) or "/" in str(row[0]) - ] - return _insert_rows( - target_conn, - "storage", - ("key", "value", "type", "created_at", "updated_at"), - rows, +) -> None: + """Fail closed if the published backup changed recovered business data.""" + + source_manifest = _schema_manifest(source_conn) + target_manifest = _schema_manifest(target_conn) + if target_manifest != source_manifest: + changed_keys = sorted( + key + for key in source_manifest.keys() | target_manifest.keys() + if source_manifest.get(key) != target_manifest.get(key) + ) + changed = changed_keys[0] if changed_keys else ("schema", "unknown") + raise sqlite3.DatabaseError( + f"Recovered DB schema object was not preserved: {changed[0]} {changed[1]}" ) - if table_name == "usage_records": - rows = [ - tuple(row) - for row in source_conn.execute( - """ - SELECT c0, c1, c2, c3, c4, c5, c6, c7, c8, c9, - c10, c11, c12, c13, c14, c15, c16, c17, c18, c19 - FROM lost_and_found - WHERE nfield = 20 - AND typeof(c0) = 'text' - AND typeof(c1) = 'text' - AND typeof(c2) = 'text' - """ - ).fetchall() - ] - return _insert_rows(target_conn, "usage_records", USAGE_RECORD_COLUMNS, rows) - - if table_name == "task_schedulers": - rows = [ - tuple(row) - for row in source_conn.execute( - """ - SELECT c0, c1, c2, c3, c4, c5, c6, c7, c8, c9, - c10, c11, c12, c13, c14, c15, c16, c17, c18, c19, c20 - FROM lost_and_found - WHERE nfield = 21 - AND typeof(c0) = 'text' - """ - ).fetchall() - if str(row[0]).startswith("tsk_") - ] - return _insert_rows(target_conn, "task_schedulers", TASK_SCHEDULER_COLUMNS, rows) - - if table_name == "task_executions": - rows = [ - tuple(row) - for row in source_conn.execute( - """ - SELECT c0, c1, c2, c3, c4, c5, c6, c7, c8, c9, - c10, c11, c12, c13, c14, c15, c16, c17, c18, c19, - c20, c21, c22, c23 - FROM lost_and_found - WHERE nfield = 24 - AND typeof(c0) = 'text' - """ - ).fetchall() - if str(row[0]).startswith("txe_") - ] - return _insert_rows(target_conn, "task_executions", TASK_EXECUTION_COLUMNS, rows) - - if table_name == "task_execution_queue_refs": - rows = [ - tuple(row) - for row in source_conn.execute( - """ - SELECT c0, c1, c2, c3, c4 - FROM lost_and_found - WHERE nfield = 5 - AND typeof(c0) = 'text' - AND typeof(c1) = 'text' - AND typeof(c2) = 'text' - AND typeof(c3) = 'text' - """ - ).fetchall() - if str(row[0]).startswith("tqref_") - ] - return _insert_rows(target_conn, "task_execution_queue_refs", QUEUE_REF_COLUMNS, rows) - - if table_name == "channel_bindings": - rows = [ - tuple(row) - for row in source_conn.execute( - """ - SELECT c0, c1, c2, c3, c4, c5, c6, c7, c8, c9 - FROM lost_and_found - WHERE nfield = 10 - AND typeof(c0) = 'text' - """ - ).fetchall() - if str(row[0]).startswith("chb_") - ] - return _insert_rows(target_conn, "channel_bindings", CHANNEL_BINDING_COLUMNS, rows) + source_tables = { + name for object_type, name in source_manifest if object_type == "table" + } + virtual_tables = { + name + for (object_type, name), (_, sql) in source_manifest.items() + if object_type == "table" + and sql is not None + and sql.lstrip().upper().startswith("CREATE VIRTUAL TABLE") + } + missing_row = object() + for table_name in sorted(source_tables - virtual_tables): + quoted_table = _quote_identifier(table_name) + table_sql = source_manifest[("table", table_name)][1] or "" + column_names = { + str(row[1]).casefold() + for row in source_conn.execute(f"PRAGMA table_info({quoted_table})").fetchall() + } + rowid_alias = next( + ( + alias + for alias in ("rowid", "_rowid_", "oid") + if alias.casefold() not in column_names + ), + None, + ) + include_rowid = rowid_alias is not None and "WITHOUT ROWID" not in table_sql.upper() + projection = f"{_quote_identifier(rowid_alias)}, *" if include_rowid else "*" + try: + source_rows = source_conn.execute(f"SELECT {projection} FROM {quoted_table}") + target_rows = target_conn.execute(f"SELECT {projection} FROM {quoted_table}") + row_number = 0 + while True: + source_row = source_rows.fetchone() + target_row = target_rows.fetchone() + if source_row is None and target_row is None: + break + row_number += 1 + source_value = missing_row if source_row is None else tuple(source_row) + target_value = missing_row if target_row is None else tuple(target_row) + if source_value != target_value: + raise sqlite3.DatabaseError( + f"Recovered DB row content changed for {table_name} at row {row_number}" + ) + except sqlite3.DatabaseError: + raise + except sqlite3.Error as exc: + raise sqlite3.DatabaseError( + f"Recovered DB table could not be compared: {table_name}: {exc}" + ) from exc + + for pragma_name in ("application_id", "user_version"): + source_value = source_conn.execute(f"PRAGMA {pragma_name}").fetchone()[0] + target_value = target_conn.execute(f"PRAGMA {pragma_name}").fetchone()[0] + if target_value != source_value: + raise sqlite3.DatabaseError( + f"Recovered DB PRAGMA {pragma_name} changed: {source_value} -> {target_value}" + ) - return 0 + integrity_rows = target_conn.execute("PRAGMA integrity_check").fetchall() + if integrity_rows != [("ok",)]: + detail = "; ".join(str(row[0]) for row in integrity_rows) + raise sqlite3.DatabaseError(f"Recovered DB integrity check failed: {detail}") + foreign_key_rows = target_conn.execute("PRAGMA foreign_key_check").fetchall() + if foreign_key_rows: + raise sqlite3.IntegrityError( + f"Recovered DB foreign key check failed: {foreign_key_rows[0]}" + ) def build_normalized_recovery_db( extracted_db_path: Path, output_db_path: Path, + *, + lost_and_found_table: str = "lost_and_found", ) -> Dict[str, int]: - """Copy supported recovered tables into a clean output database.""" + """Publish an exact recovered backup and retain unresolved rows for inspection. - _ensure_recovered_schema(output_db_path) + ``sqlite3 .recover`` stores rows it cannot attribute confidently in its + lost-and-found table. Guessing their destination from field count or value + prefixes can silently inject another table's data into Flocks, so this + function deliberately leaves those rows untouched. + """ copied_rows: Dict[str, int] = {} - source_conn = sqlite3.connect(extracted_db_path) - target_conn = sqlite3.connect(output_db_path) - try: - for table_name in SUPPORTED_COPY_TABLES: - direct_rows = _copy_table_rows(source_conn, target_conn, table_name) - if direct_rows == 0: - _copy_lost_and_found_rows(source_conn, target_conn, table_name) - target_conn.commit() - copied_rows[table_name] = target_conn.execute( - f"SELECT COUNT(*) FROM {_quote_identifier(table_name)}" - ).fetchone()[0] - finally: - source_conn.close() - target_conn.close() + with _atomic_output_path(output_db_path) as temporary_path: + source_conn = sqlite3.connect(extracted_db_path) + target_conn = sqlite3.connect(temporary_path) + try: + source_conn.backup(target_conn) + finally: + source_conn.close() + target_conn.close() + + source_conn = sqlite3.connect(extracted_db_path) + target_conn = sqlite3.connect(temporary_path) + try: + journal_mode = target_conn.execute("PRAGMA journal_mode=DELETE").fetchone()[0] + if str(journal_mode).lower() != "delete": + raise sqlite3.DatabaseError( + f"Could not switch recovered DB to DELETE journal mode: {journal_mode}" + ) + for table_name in SUPPORTED_COPY_TABLES: + copied_rows[table_name] = ( + target_conn.execute( + f"SELECT COUNT(*) FROM {_quote_identifier(table_name)}" + ).fetchone()[0] + if _table_exists(target_conn, table_name) + else 0 + ) + _validate_recovered_db(source_conn, target_conn) + finally: + source_conn.close() + target_conn.close() return copied_rows @@ -695,6 +776,7 @@ def _render_summary(artifacts: RecoveryArtifacts) -> str: f"extracted_db={artifacts.extracted_db}", f"recovered_db={artifacts.recovered_db}", f"summary_path={artifacts.summary_path}", + f"lost_and_found_table={artifacts.lost_and_found_table}", f"pagesize={artifacts.pagesize}", f"wal_frames={artifacts.wal_frames}", f"wal_final_db_pages={artifacts.wal_final_db_pages}", @@ -704,6 +786,34 @@ def _render_summary(artifacts: RecoveryArtifacts) -> str: return "\n".join(lines) + "\n" +def _recovery_artifact_paths( + recovery_dir: Path, + prefix: str, +) -> tuple[Path, Path, Path, Path, Path]: + """Return every file path written by one recovery run.""" + + return ( + recovery_dir / f"{prefix}.candidate.db", + recovery_dir / f"{prefix}.recover.sql", + recovery_dir / f"{prefix}.extracted.db", + recovery_dir / f"{prefix}.db", + recovery_dir / f"{prefix}.summary.txt", + ) + + +def _recovery_sqlite_sidecar_paths( + artifact_paths: tuple[Path, Path, Path, Path, Path], +) -> tuple[Path, ...]: + """Return sidecar paths SQLite may create while processing DB artifacts.""" + + database_paths = (artifact_paths[0], artifact_paths[2], artifact_paths[3]) + return tuple( + database_path.with_name(f"{database_path.name}{suffix}") + for database_path in database_paths + for suffix in ("-journal", "-wal", "-shm") + ) + + def recover_raw_storage_db( raw_path: Path, wal_path: Path | None, @@ -713,17 +823,32 @@ def recover_raw_storage_db( ) -> RecoveryArtifacts: """Recover a damaged raw SQLite file into a normalized database.""" + candidate_db, recover_sql, extracted_db, recovered_db, summary_path = _recovery_artifact_paths( + recovery_dir, + prefix, + ) + _validate_recovery_plan( + raw_path=raw_path, + wal_path=wal_path, + recovery_dir=recovery_dir, + output_db=recovered_db, + prefix=prefix, + ) recovery_dir.mkdir(parents=True, exist_ok=True) - candidate_db = recovery_dir / f"{prefix}.candidate.db" - recover_sql = recovery_dir / f"{prefix}.recover.sql" - extracted_db = recovery_dir / f"{prefix}.extracted.db" - recovered_db = recovery_dir / f"{prefix}.db" - summary_path = recovery_dir / f"{prefix}.summary.txt" + lost_and_found_table = f"flocks_recovery_lost_{uuid.uuid4().hex}" candidate_stats = reconstruct_sqlite_candidate(raw_path, wal_path, candidate_db) - _run_sqlite_recover(candidate_db, recover_sql) + _run_sqlite_recover( + candidate_db, + recover_sql, + lost_and_found_table=lost_and_found_table, + ) _materialize_recovered_sql(recover_sql, extracted_db) - copied_rows = build_normalized_recovery_db(extracted_db, recovered_db) + copied_rows = build_normalized_recovery_db( + extracted_db, + recovered_db, + lost_and_found_table=lost_and_found_table, + ) artifacts = RecoveryArtifacts( recovery_dir=recovery_dir, @@ -732,12 +857,14 @@ def recover_raw_storage_db( extracted_db=extracted_db, recovered_db=recovered_db, summary_path=summary_path, + lost_and_found_table=lost_and_found_table, pagesize=candidate_stats["pagesize"], wal_frames=candidate_stats["wal_frames"], wal_final_db_pages=candidate_stats["wal_final_db_pages"], copied_rows=copied_rows, ) - summary_path.write_text(_render_summary(artifacts), encoding="utf-8") + with _atomic_output_path(summary_path) as temporary_path: + temporary_path.write_text(_render_summary(artifacts), encoding="utf-8") return artifacts @@ -760,45 +887,196 @@ def _resolve_raw_path(args: argparse.Namespace) -> Path: def _detect_wal_path(raw_path: Path) -> Path | None: """Try to find a matching WAL file next to the damaged DB.""" - bases = {raw_path.name, raw_path.stem} - suffixes = {"", raw_path.suffix} + candidates: list[Path] = [] seen: set[Path] = set() - for base in bases: - for suffix in suffixes: + def add_candidate(name: str) -> None: + candidate = raw_path.with_name(name) + if candidate not in seen: + seen.add(candidate) + candidates.append(candidate) + + add_candidate(f"{raw_path.name}-wal") + add_candidate(f"{raw_path.name}.wal") + + marker = ".corrupt." + if marker in raw_path.name: + original_name, quarantine_suffix = raw_path.name.split(marker, 1) + add_candidate(f"{original_name}-wal{marker}{quarantine_suffix}") + suffix_base, separator, counter = quarantine_suffix.rpartition(".") + if separator and counter.isdigit(): + add_candidate(f"{original_name}-wal{marker}{suffix_base}") + + for base in (raw_path.stem, raw_path.name): + for suffix in ("", raw_path.suffix): for wal_suffix in ("-wal", ".wal"): - candidate = raw_path.with_name(f"{base}{wal_suffix}{suffix}") - if candidate in seen: - continue - seen.add(candidate) - if candidate.exists(): - return candidate.resolve() + add_candidate(f"{base}{wal_suffix}{suffix}") + + for candidate in candidates: + # A clean ``wal_checkpoint(TRUNCATE)`` commonly leaves a zero-length + # WAL beside a complete main database. Auto-detection should treat it + # as no pending WAL; an explicitly supplied ``--wal`` remains strict. + if candidate.is_file() and candidate.stat().st_size > 0: + return candidate.resolve() return None def _default_artifacts_dir(raw_path: Path) -> Path: - """Create the default workspace output directory for recovery artifacts.""" + """Return the default workspace output directory for recovery artifacts.""" - workspace_dir = Path(os.getenv("FLOCKS_WORKSPACE_DIR", Path.home() / ".flocks" / "workspace")) + workspace_value = _getenv_case_insensitive("FLOCKS_WORKSPACE_DIR") + workspace_dir = Path( + workspace_value + if workspace_value is not None + else Path.home() / ".flocks" / "workspace" + ) today = dt.date.today().isoformat() run_name = _sanitize_name(raw_path.name) - output_dir = workspace_dir / "outputs" / today / f"db-recovery-{run_name}" - output_dir.mkdir(parents=True, exist_ok=True) - return output_dir + run_id = uuid.uuid4().hex + return workspace_dir / "outputs" / today / f"db-recovery-{run_name}-{run_id}" + + +def _getenv_case_insensitive(name: str) -> str | None: + """Read a Flocks setting with the same case-insensitive key semantics as BaseSettings.""" + + result: str | None = None + for key, value in os.environ.items(): + if key.lower() == name.lower(): + result = value + return result + + +def _resolve_live_data_dir() -> Path: + """Resolve the live Flocks data directory without importing the application.""" + + data_dir = _getenv_case_insensitive("FLOCKS_DATA_DIR") + if data_dir is not None: + return Path(data_dir).resolve() + + xdg_data = _getenv_case_insensitive("XDG_DATA_HOME") + if xdg_data: + return (Path(xdg_data) / "flocks").resolve() + + flocks_root = _getenv_case_insensitive("FLOCKS_ROOT") + if flocks_root: + return (Path(flocks_root) / "data").resolve() + + return (Path.home() / ".flocks" / "data").resolve() + + +def _same_file_or_path(first: Path, second: Path) -> bool: + """Return whether two paths resolve to the same path or existing inode.""" + + if first.resolve() == second.resolve(): + return True + try: + return first.samefile(second) + except OSError: + return False + + +def _path_collision_key(path: Path) -> str: + """Return a conservative path key for case-insensitive filesystems.""" + + return os.path.normcase(str(path.resolve())).casefold() + +def _is_within(path: Path, directory: Path) -> bool: + """Return whether a resolved path is inside a resolved directory.""" -def _cleanup_live_sqlite_sidecars() -> list[Path]: - """Delete stale live WAL/SHM sidecar files under `~/.flocks/data/`.""" + path_key = _path_collision_key(path) + directory_key = _path_collision_key(directory).rstrip(os.sep) + return path_key == directory_key or path_key.startswith(f"{directory_key}{os.sep}") - data_dir = Path(os.getenv("FLOCKS_DATA_DIR", Path.home() / ".flocks" / "data")) - removed: list[Path] = [] - for name in ("flocks.db-shm", "flocks.db-wal"): - candidate = data_dir / name - if candidate.exists(): - candidate.unlink() - removed.append(candidate) - return removed + +def _validate_recovery_plan( + *, + raw_path: Path, + wal_path: Path | None, + recovery_dir: Path, + output_db: Path, + prefix: str, +) -> None: + """Reject recovery plans that could mutate live data or source evidence.""" + + live_data_dir = _resolve_live_data_dir() + live_db = live_data_dir / "flocks.db" + live_sidecars = (live_data_dir / "flocks.db-wal", live_data_dir / "flocks.db-shm") + artifact_paths = _recovery_artifact_paths(recovery_dir, prefix) + sqlite_sidecar_paths = _recovery_sqlite_sidecar_paths(artifact_paths) + recovered_db = artifact_paths[3] + + if _same_file_or_path(raw_path, live_db): + raise ValueError( + "Refusing to recover the live Flocks database directly. " + "Stop Flocks, copy the database and matching WAL to a separate path, " + "then recover that copy." + ) + + if wal_path is not None and any( + _same_file_or_path(wal_path, live_sidecar) for live_sidecar in live_sidecars + ): + raise ValueError( + "Refusing to read a live Flocks SQLite sidecar. " + "Stop Flocks and copy the database with its matching WAL before recovery." + ) + + output_key = _path_collision_key(output_db) + recovered_key = _path_collision_key(recovered_db) + if output_db.resolve() != recovered_db.resolve() and ( + output_key == recovered_key or _same_file_or_path(output_db, recovered_db) + ): + raise ValueError( + "Recovery output has an ambiguous case-insensitive collision with the recovered DB: " + f"{output_db}" + ) + + intermediate_paths = ( + *(path for path in artifact_paths if path != recovered_db), + *sqlite_sidecar_paths, + ) + colliding_artifact = next( + ( + artifact_path + for artifact_path in intermediate_paths + if ( + output_key == _path_collision_key(artifact_path) + or _same_file_or_path(output_db, artifact_path) + ) + ), + None, + ) + if colliding_artifact is not None: + raise ValueError( + f"Recovery output collides with an intermediate artifact: {colliding_artifact}" + ) + + write_paths = [*artifact_paths, *sqlite_sidecar_paths, output_db] + + source_paths = [raw_path] + if wal_path is not None: + source_paths.append(wal_path) + for write_path in write_paths: + if any(_same_file_or_path(write_path, source_path) for source_path in source_paths): + raise ValueError( + f"Refusing to overwrite recovery source evidence: {write_path}" + ) + + if any(_same_file_or_path(write_path, live_db) for write_path in write_paths): + raise ValueError( + "Refusing to write recovery output to the live Flocks database. " + "Generate a recovery artifact, stop Flocks, then install it explicitly." + ) + + for write_path in write_paths: + if _is_within(write_path, live_data_dir): + raise ValueError( + "Refusing to write recovery files inside the live Flocks data directory: " + f"{write_path}" + ) + if write_path.exists() or write_path.is_symlink(): + raise FileExistsError(f"Refusing to overwrite existing recovery file: {write_path}") def _resolve_output_paths(raw_path: Path, args: argparse.Namespace) -> tuple[Path, Path, str]: @@ -819,18 +1097,26 @@ def _resolve_output_paths(raw_path: Path, args: argparse.Namespace) -> tuple[Pat ) output_db = artifacts_dir / f"{_sanitize_name(raw_path.name)}.recovered.db" - prefix = args.prefix or output_db.stem + prefix = _sanitize_name(args.prefix or output_db.stem) return artifacts_dir, output_db, prefix +def _copy_file_exclusive(source: Path, destination: Path) -> None: + """Copy a file without ever replacing a destination created after validation.""" + + with _atomic_output_path(destination) as temporary_path: + with source.open("rb") as source_handle, temporary_path.open("xb") as destination_handle: + shutil.copyfileobj(source_handle, destination_handle) + + def build_parser() -> argparse.ArgumentParser: """Build the CLI parser.""" parser = argparse.ArgumentParser( description=( - "Recover a damaged Flocks SQLite DB in one command. The script can " + "Recover an offline copy of a damaged Flocks SQLite DB. The script can " "auto-detect a sibling WAL file and writes the repaired DB plus " - "intermediate artifacts." + "intermediate artifacts without modifying the live store." ) ) parser.add_argument( @@ -855,7 +1141,7 @@ def build_parser() -> argparse.ArgumentParser: "--output", type=Path, default=None, - help="Optional final output DB file path.", + help="Optional new output DB path. Existing files and the live Flocks DB are rejected.", ) parser.add_argument( "--artifacts-dir", @@ -881,14 +1167,31 @@ def build_parser() -> argparse.ArgumentParser: def main(argv: Sequence[str] | None = None) -> int: """Run recovery and print artifact locations.""" + # Match the normal ``flocks`` CLI: environment variables already exported + # by the operator win, and missing values are filled from the current + # project's .env before resolving the protected live data directory. + current_env = Path.cwd() / ".env" + if current_env.is_file(): + load_dotenv(current_env) + project_env = Path(__file__).resolve().parent.parent / ".env" + if project_env.is_file() and project_env != current_env: + load_dotenv(project_env) args = build_parser().parse_args(argv) - removed_sidecars = _cleanup_live_sqlite_sidecars() raw_path = _resolve_raw_path(args) if not raw_path.exists(): raise FileNotFoundError(f"Damaged DB file does not exist: {raw_path}") wal_path = args.wal.expanduser().resolve() if args.wal is not None else _detect_wal_path(raw_path) + if wal_path is not None and not wal_path.is_file(): + raise FileNotFoundError(f"WAL file does not exist or is not a file: {wal_path}") artifacts_dir, output_db, prefix = _resolve_output_paths(raw_path, args) + _validate_recovery_plan( + raw_path=raw_path, + wal_path=wal_path, + recovery_dir=artifacts_dir, + output_db=output_db, + prefix=prefix, + ) artifacts = recover_raw_storage_db( raw_path=raw_path, @@ -897,20 +1200,14 @@ def main(argv: Sequence[str] | None = None) -> int: prefix=prefix, ) - if artifacts.recovered_db != output_db: - output_db.parent.mkdir(parents=True, exist_ok=True) - shutil.copy2(artifacts.recovered_db, output_db) + if not _same_file_or_path(artifacts.recovered_db, output_db): + _copy_file_exclusive(artifacts.recovered_db, output_db) print(f"input_db={raw_path}") print(f"wal_path={wal_path if wal_path is not None else 'none'}") - print( - "removed_sidecars=" - + ( - ",".join(str(path) for path in removed_sidecars) - if removed_sidecars - else "none" - ) - ) + # Preserve the original machine-readable field for callers while making + # the non-destructive behavior explicit. + print("removed_sidecars=none") print(f"artifacts_dir={artifacts_dir}") print(f"recovered_db={output_db}") print(f"summary_path={artifacts.summary_path}") @@ -920,4 +1217,3 @@ def main(argv: Sequence[str] | None = None) -> int: if __name__ == "__main__": raise SystemExit(main()) - diff --git a/tests/scripts/test_recover_raw_flocks_db.py b/tests/scripts/test_recover_raw_flocks_db.py new file mode 100644 index 000000000..b53fdeaec --- /dev/null +++ b/tests/scripts/test_recover_raw_flocks_db.py @@ -0,0 +1,1187 @@ +"""Safety tests for the offline SQLite recovery helper.""" + +from __future__ import annotations + +import importlib.util +import os +import shutil +import sqlite3 +import struct +import sys +from pathlib import Path + +import pytest + + +def load_module(): + module_path = Path(__file__).resolve().parents[2] / "scripts" / "recover_raw_flocks_db.py" + module_name = "recover_raw_flocks_db_test_module" + spec = importlib.util.spec_from_file_location(module_name, module_path) + assert spec is not None + assert spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[module_name] = module + spec.loader.exec_module(module) + return module + + +recover_raw_flocks_db = load_module() + + +def _wal_checksum(data, *, byte_order, state=(0, 0)): + assert len(data) % 8 == 0 + words = struct.unpack(f"{byte_order}{len(data) // 4}I", data) + s0, s1 = state + for index in range(0, len(words), 2): + s0 = (s0 + words[index] + s1) & 0xFFFFFFFF + s1 = (s1 + words[index + 1] + s0) & 0xFFFFFFFF + return s0, s1 + + +def _build_wal(frames, *, page_size=1024, magic=None): + magic = magic or recover_raw_flocks_db.WAL_MAGIC + byte_order = "<" if magic == 0x377F0682 else ">" + salt1, salt2 = 0x12345678, 0x90ABCDEF + header_prefix = struct.pack( + ">6I", + magic, + recover_raw_flocks_db.WAL_VERSION, + page_size, + 0, + salt1, + salt2, + ) + checksum = _wal_checksum(header_prefix, byte_order=byte_order) + wal = bytearray(header_prefix + struct.pack(">2I", *checksum)) + for page_no, db_page_count, fill_byte in frames: + page = bytes([fill_byte]) * page_size + frame_prefix = struct.pack(">4I", page_no, db_page_count, salt1, salt2) + checksum = _wal_checksum(frame_prefix[:8], byte_order=byte_order, state=checksum) + checksum = _wal_checksum(page, byte_order=byte_order, state=checksum) + wal.extend(frame_prefix + struct.pack(">2I", *checksum) + page) + return bytes(wal) + + +def test_default_artifacts_dir_is_unique_and_not_created(tmp_path, monkeypatch): + workspace_dir = tmp_path / "workspace" + raw_path = tmp_path / "damaged.db" + monkeypatch.setenv("FLOCKS_WORKSPACE_DIR", str(workspace_dir)) + + first = recover_raw_flocks_db._default_artifacts_dir(raw_path) + second = recover_raw_flocks_db._default_artifacts_dir(raw_path) + + assert first != second + assert first.parent == second.parent + assert first.exists() is False + assert second.exists() is False + + +def test_empty_data_dir_env_matches_runtime_current_directory(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("FLOCKS_DATA_DIR", "") + + assert recover_raw_flocks_db._resolve_live_data_dir() == tmp_path.resolve() + + +def test_lowercase_data_dir_env_matches_runtime(tmp_path, monkeypatch): + data_dir = tmp_path / "lowercase-live-data" + monkeypatch.delenv("FLOCKS_DATA_DIR", raising=False) + monkeypatch.setenv("flocks_data_dir", str(data_dir)) + + assert recover_raw_flocks_db._resolve_live_data_dir() == data_dir.resolve() + + +@pytest.mark.skipif(os.name == "nt", reason="Windows environment keys are case-insensitive") +def test_data_dir_case_variant_priority_matches_runtime(tmp_path, monkeypatch): + from flocks.config.config import GlobalConfig + + uppercase_dir = tmp_path / "uppercase" + lowercase_dir = tmp_path / "lowercase" + monkeypatch.delenv("FLOCKS_DATA_DIR", raising=False) + monkeypatch.delenv("flocks_data_dir", raising=False) + monkeypatch.setenv("FLOCKS_DATA_DIR", str(uppercase_dir)) + monkeypatch.setenv("flocks_data_dir", str(lowercase_dir)) + + assert recover_raw_flocks_db._resolve_live_data_dir() == GlobalConfig().data_dir.resolve() + + +def test_cli_loads_dotenv_before_protecting_live_data_dir(tmp_path, monkeypatch): + live_data = tmp_path / "configured-live-data" + live_data.mkdir() + live_db = live_data / "flocks.db" + live_db.write_bytes(b"live") + (tmp_path / ".env").write_text( + f"FLOCKS_DATA_DIR={live_data}\n", + encoding="utf-8", + ) + monkeypatch.chdir(tmp_path) + monkeypatch.delenv("FLOCKS_DATA_DIR", raising=False) + + with pytest.raises(ValueError, match="live Flocks database"): + recover_raw_flocks_db.main([str(live_db)]) + + +def test_missing_input_does_not_touch_live_sidecars(tmp_path, monkeypatch): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + wal_path = data_dir / "flocks.db-wal" + shm_path = data_dir / "flocks.db-shm" + wal_path.write_bytes(b"live-wal") + shm_path.write_bytes(b"live-shm") + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + with pytest.raises(FileNotFoundError): + recover_raw_flocks_db.main([str(tmp_path / "missing.db")]) + + assert wal_path.read_bytes() == b"live-wal" + assert shm_path.read_bytes() == b"live-shm" + + +def test_recovery_cli_keeps_live_wal_usable(tmp_path, monkeypatch): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + db_path = data_dir / "flocks.db" + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + holder = sqlite3.connect(db_path) + try: + assert holder.execute("PRAGMA journal_mode=WAL").fetchone()[0] == "wal" + holder.execute("CREATE TABLE live_rows(value INTEGER)") + holder.execute("INSERT INTO live_rows VALUES (1)") + holder.commit() + assert db_path.with_name("flocks.db-wal").exists() + assert db_path.with_name("flocks.db-shm").exists() + + with pytest.raises(FileNotFoundError): + recover_raw_flocks_db.main([str(tmp_path / "missing.db")]) + + holder.execute("INSERT INTO live_rows VALUES (2)") + holder.commit() + with sqlite3.connect(db_path) as probe: + assert probe.execute("SELECT value FROM live_rows ORDER BY value").fetchall() == [ + (1,), + (2,), + ] + finally: + holder.close() + + +def test_rejects_live_database_as_output_before_recovery(tmp_path, monkeypatch): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + live_db = data_dir / "flocks.db" + live_db.write_bytes(b"live-db") + damaged_db = tmp_path / "damaged.db" + damaged_db.write_bytes(b"damaged-db") + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + recovery_called = False + + def fail_if_called(*_args, **_kwargs): + nonlocal recovery_called + recovery_called = True + raise AssertionError("recovery must not start for an unsafe output path") + + monkeypatch.setattr(recover_raw_flocks_db, "recover_raw_storage_db", fail_if_called) + + with pytest.raises(ValueError, match="live Flocks database"): + recover_raw_flocks_db.main([str(damaged_db), "--output", str(live_db)]) + + assert recovery_called is False + assert live_db.read_bytes() == b"live-db" + + +def test_rejects_live_database_as_input_before_writing_artifacts(tmp_path, monkeypatch): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + live_db = data_dir / "flocks.db" + live_db.write_bytes(b"live-db") + workspace_dir = tmp_path / "workspace" + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + monkeypatch.setenv("FLOCKS_WORKSPACE_DIR", str(workspace_dir)) + + with pytest.raises(ValueError, match="recover the live Flocks database"): + recover_raw_flocks_db.main([str(live_db)]) + + assert live_db.read_bytes() == b"live-db" + assert workspace_dir.exists() is False + + +def test_rejects_artifact_plan_that_writes_live_database(tmp_path, monkeypatch): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + live_db = data_dir / "flocks.db" + live_db.write_bytes(b"live-db") + damaged_db = tmp_path / "damaged.db" + damaged_db.write_bytes(b"damaged-db") + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + recovery_called = False + + def fail_if_called(*_args, **_kwargs): + nonlocal recovery_called + recovery_called = True + raise AssertionError("recovery must not start for an unsafe artifact path") + + monkeypatch.setattr(recover_raw_flocks_db, "recover_raw_storage_db", fail_if_called) + + with pytest.raises(ValueError, match="live Flocks database"): + recover_raw_flocks_db.main( + [ + str(damaged_db), + "--artifacts-dir", + str(data_dir), + "--prefix", + "flocks", + ] + ) + + assert recovery_called is False + assert live_db.read_bytes() == b"live-db" + + +@pytest.mark.parametrize("prefix, live_name", [("tasks", "tasks.db"), ("workflow", "workflow.db")]) +def test_rejects_artifacts_anywhere_in_live_data_dir( + tmp_path, + monkeypatch, + prefix, + live_name, +): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + live_db = data_dir / live_name + live_db.write_bytes(b"live-db") + damaged_db = tmp_path / "damaged.db" + damaged_db.write_bytes(b"damaged-db") + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + monkeypatch.setattr( + recover_raw_flocks_db, + "recover_raw_storage_db", + lambda *_args, **_kwargs: pytest.fail("unsafe recovery plan must be rejected"), + ) + + with pytest.raises(ValueError, match="live Flocks data directory"): + recover_raw_flocks_db.main( + [ + str(damaged_db), + "--artifacts-dir", + str(data_dir), + "--prefix", + prefix, + ] + ) + + assert live_db.read_bytes() == b"live-db" + + +def test_rejects_existing_artifact_before_recovery(tmp_path, monkeypatch): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + damaged_db = tmp_path / "damaged.db" + damaged_db.write_bytes(b"damaged-db") + artifacts_dir = tmp_path / "artifacts" + artifacts_dir.mkdir() + candidate_db = artifacts_dir / "safe.candidate.db" + candidate_db.write_bytes(b"existing-evidence") + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + with pytest.raises(FileExistsError, match="existing recovery file"): + recover_raw_flocks_db.main( + [ + str(damaged_db), + "--artifacts-dir", + str(artifacts_dir), + "--prefix", + "safe", + ] + ) + + assert candidate_db.read_bytes() == b"existing-evidence" + + +@pytest.mark.parametrize("suffix", ["-journal", "-wal", "-shm"]) +def test_rejects_existing_sqlite_sidecar_before_recovery(tmp_path, monkeypatch, suffix): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + damaged_db = tmp_path / "damaged.db" + damaged_db.write_bytes(b"damaged-db") + artifacts_dir = tmp_path / "artifacts" + artifacts_dir.mkdir() + sidecar_path = artifacts_dir / f"safe.db{suffix}" + sidecar_path.write_bytes(b"existing-sidecar") + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + with pytest.raises(FileExistsError, match="existing recovery file"): + recover_raw_flocks_db.main( + [ + str(damaged_db), + "--artifacts-dir", + str(artifacts_dir), + "--prefix", + "safe", + ] + ) + + assert sidecar_path.read_bytes() == b"existing-sidecar" + + +def test_rejects_hardlinked_artifact_to_live_database(tmp_path, monkeypatch): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + live_db = data_dir / "flocks.db" + live_db.write_bytes(b"live-db") + damaged_db = tmp_path / "damaged.db" + damaged_db.write_bytes(b"damaged-db") + artifacts_dir = tmp_path / "artifacts" + artifacts_dir.mkdir() + os.link(live_db, artifacts_dir / "safe.candidate.db") + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + with pytest.raises(ValueError, match="live Flocks database"): + recover_raw_flocks_db.main( + [ + str(damaged_db), + "--artifacts-dir", + str(artifacts_dir), + "--prefix", + "safe", + ] + ) + + assert live_db.read_bytes() == b"live-db" + + +def test_direct_recovery_call_rejects_live_data_directory(tmp_path, monkeypatch): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + live_db = data_dir / "flocks.db" + live_db.write_bytes(b"live-db") + damaged_db = tmp_path / "damaged.db" + damaged_db.write_bytes(b"damaged-db") + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + with pytest.raises(ValueError, match="live Flocks database"): + recover_raw_flocks_db.recover_raw_storage_db( + damaged_db, + None, + data_dir, + prefix="flocks", + ) + + assert live_db.read_bytes() == b"live-db" + + +def test_rejects_output_collision_with_intermediate_artifact(tmp_path, monkeypatch): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + damaged_db = tmp_path / "damaged.db" + damaged_db.write_bytes(b"damaged-db") + artifacts_dir = tmp_path / "artifacts" + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + with pytest.raises(ValueError, match="collides with an intermediate artifact"): + recover_raw_flocks_db.main( + [ + str(damaged_db), + "--output", + str(artifacts_dir / "safe.summary.txt"), + "--artifacts-dir", + str(artifacts_dir), + "--prefix", + "safe", + ] + ) + + assert artifacts_dir.exists() is False + + +def test_rejects_output_collision_with_sqlite_sidecar(tmp_path, monkeypatch): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + damaged_db = tmp_path / "damaged.db" + damaged_db.write_bytes(b"damaged-db") + artifacts_dir = tmp_path / "artifacts" + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + with pytest.raises(ValueError, match="collides with an intermediate artifact"): + recover_raw_flocks_db.main( + [ + str(damaged_db), + "--output", + str(artifacts_dir / "safe.db-journal"), + "--artifacts-dir", + str(artifacts_dir), + "--prefix", + "safe", + ] + ) + + assert artifacts_dir.exists() is False + + +def test_rejects_case_insensitive_output_collision(tmp_path, monkeypatch): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + damaged_db = tmp_path / "damaged.db" + damaged_db.write_bytes(b"damaged-db") + artifacts_dir = tmp_path / "artifacts" + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + with pytest.raises(ValueError, match="ambiguous case-insensitive collision"): + recover_raw_flocks_db.main( + [ + str(damaged_db), + "--output", + str(artifacts_dir / "SAFE.db"), + "--artifacts-dir", + str(artifacts_dir), + "--prefix", + "safe", + ] + ) + + assert artifacts_dir.exists() is False + + +def test_rejects_wal_path_that_collides_with_artifact(tmp_path, monkeypatch): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + damaged_db = tmp_path / "damaged.db" + damaged_db.write_bytes(b"damaged-db") + artifacts_dir = tmp_path / "artifacts" + artifacts_dir.mkdir() + wal_path = artifacts_dir / "safe.candidate.db" + wal_path.write_bytes(b"wal-evidence") + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + with pytest.raises(ValueError, match="source evidence"): + recover_raw_flocks_db.main( + [ + str(damaged_db), + "--wal", + str(wal_path), + "--artifacts-dir", + str(artifacts_dir), + "--prefix", + "safe", + ] + ) + + assert wal_path.read_bytes() == b"wal-evidence" + + +def test_rejects_live_wal_as_recovery_input(tmp_path, monkeypatch): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + live_wal = data_dir / "flocks.db-wal" + live_wal.write_bytes(b"live-wal") + damaged_db = tmp_path / "damaged.db" + damaged_db.write_bytes(b"damaged-db") + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + with pytest.raises(ValueError, match="live Flocks SQLite sidecar"): + recover_raw_flocks_db.main( + [ + str(damaged_db), + "--wal", + str(live_wal), + "--artifacts-dir", + str(tmp_path / "artifacts"), + ] + ) + + assert live_wal.read_bytes() == b"live-wal" + + +def test_detects_storage_quarantine_wal_name(tmp_path): + raw_path = tmp_path / "flocks.db.corrupt.20260710T083229" + wal_path = tmp_path / "flocks.db-wal.corrupt.20260710T083229" + raw_path.write_bytes(b"raw") + wal_path.write_bytes(b"wal") + + assert recover_raw_flocks_db._detect_wal_path(raw_path) == wal_path.resolve() + + +def test_auto_detect_ignores_empty_checkpointed_wal(tmp_path): + raw_path = tmp_path / "flocks.db" + raw_path.write_bytes(b"main") + raw_path.with_name("flocks.db-wal").touch() + + assert recover_raw_flocks_db._detect_wal_path(raw_path) is None + + +@pytest.mark.parametrize("magic", [0x377F0682, 0x377F0683]) +def test_wal_parser_discards_frames_after_last_commit(magic): + wal_bytes = _build_wal( + [ + (1, 1, ord("A")), + (1, 0, ord("B")), + ], + magic=magic, + ) + + page_size, final_pages, latest_pages, frame_count = recover_raw_flocks_db._parse_wal_frames( + wal_bytes + ) + + assert page_size == 1024 + assert final_pages == 1 + assert frame_count == 1 + assert latest_pages[1] == b"A" * 1024 + + +def test_wal_parser_rejects_invalid_header_checksum(): + wal_bytes = bytearray(_build_wal([(1, 1, ord("A"))])) + wal_bytes[24] ^= 0xFF + + with pytest.raises(ValueError, match="header checksum"): + recover_raw_flocks_db._parse_wal_frames(bytes(wal_bytes)) + + +def test_real_wal_snapshot_does_not_replay_uncommitted_spill(tmp_path): + live_db = tmp_path / "live.db" + source = sqlite3.connect(live_db) + try: + assert source.execute("PRAGMA journal_mode=WAL").fetchone()[0] == "wal" + source.execute("PRAGMA wal_autocheckpoint=0") + source.execute("PRAGMA cache_size=5") + source.execute("CREATE TABLE records(id INTEGER PRIMARY KEY, value BLOB)") + source.executemany( + "INSERT INTO records(value) VALUES (?)", + [(b"a" * 3500,) for _ in range(300)], + ) + source.commit() + + source.execute("BEGIN IMMEDIATE") + source.executemany( + "UPDATE records SET value=? WHERE id=?", + [(b"b" * 3500, row_id) for row_id in range(1, 301)], + ) + + raw_copy = tmp_path / "offline.db" + wal_copy = tmp_path / "offline.db-wal" + shutil.copy2(live_db, raw_copy) + shutil.copy2(live_db.with_name(f"{live_db.name}-wal"), wal_copy) + finally: + source.rollback() + source.close() + + wal_bytes = wal_copy.read_bytes() + page_size = int.from_bytes(wal_bytes[8:12], "big") + physical_frames = (len(wal_bytes) - 32) // (24 + page_size) + candidate_db = tmp_path / "candidate.db" + stats = recover_raw_flocks_db.reconstruct_sqlite_candidate( + raw_copy, + wal_copy, + candidate_db, + ) + + assert stats["wal_frames"] < physical_frames + with sqlite3.connect(candidate_db) as db: + assert db.execute("SELECT DISTINCT substr(value, 1, 1) FROM records").fetchall() == [ + (b"a",) + ] + + +def test_reconstruct_uses_valid_512_byte_header_page_size(tmp_path): + source_db = tmp_path / "page-512.db" + with sqlite3.connect(source_db) as db: + db.execute("PRAGMA page_size=512") + db.execute("VACUUM") + db.execute("CREATE TABLE probe(value TEXT)") + db.execute("INSERT INTO probe VALUES ('ok')") + db.commit() + assert db.execute("PRAGMA page_size").fetchone()[0] == 512 + + candidate_db = tmp_path / "candidate.db" + stats = recover_raw_flocks_db.reconstruct_sqlite_candidate(source_db, None, candidate_db) + + assert stats["pagesize"] == 512 + with sqlite3.connect(candidate_db) as db: + assert db.execute("SELECT value FROM probe").fetchone() == ("ok",) + + +def test_synthetic_page_one_is_a_valid_empty_sqlite_database(tmp_path): + candidate = tmp_path / "synthetic.db" + candidate.write_bytes(recover_raw_flocks_db._build_synthetic_page1(4096, 1)) + + with sqlite3.connect(candidate) as db: + assert db.execute("PRAGMA integrity_check").fetchone() == ("ok",) + assert db.execute("SELECT COUNT(*) FROM sqlite_schema").fetchone() == (0,) + + +def test_reconstruct_rejects_wal_with_different_header_page_size(tmp_path): + source_db = tmp_path / "page-4096.db" + with sqlite3.connect(source_db) as db: + db.execute("PRAGMA page_size=4096") + db.execute("VACUUM") + db.execute("CREATE TABLE probe(value TEXT)") + db.execute("INSERT INTO probe VALUES ('main-db')") + db.commit() + assert source_db.stat().st_size % 1024 == 0 + + unrelated_wal = tmp_path / "unrelated.db-wal" + unrelated_wal.write_bytes(_build_wal([(1, 1, ord("W"))], page_size=1024)) + + with pytest.raises(ValueError, match="does not match"): + recover_raw_flocks_db.reconstruct_sqlite_candidate( + source_db, + unrelated_wal, + tmp_path / "candidate.db", + ) + + +def test_successful_cli_path_preserves_sidecars_and_compatibility_output( + tmp_path, + monkeypatch, + capsys, +): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + wal_path = data_dir / "flocks.db-wal" + shm_path = data_dir / "flocks.db-shm" + wal_path.write_bytes(b"live-wal") + shm_path.write_bytes(b"live-shm") + damaged_db = tmp_path / "damaged.db" + damaged_db.write_bytes(b"damaged-db") + artifacts_dir = tmp_path / "artifacts" + output_db = tmp_path / "recovered.db" + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + def fake_recovery(raw_path, wal_path, recovery_dir, *, prefix): + assert raw_path == damaged_db + assert wal_path is None + recovery_dir.mkdir(parents=True) + artifact_paths = recover_raw_flocks_db._recovery_artifact_paths(recovery_dir, prefix) + candidate_db, recover_sql, extracted_db, recovered_db, summary_path = artifact_paths + recovered_db.write_bytes(b"recovered-db") + summary_path.write_text("recovery=ok\n", encoding="utf-8") + return recover_raw_flocks_db.RecoveryArtifacts( + recovery_dir=recovery_dir, + candidate_db=candidate_db, + recover_sql=recover_sql, + extracted_db=extracted_db, + recovered_db=recovered_db, + summary_path=summary_path, + lost_and_found_table="lost_and_found", + pagesize=4096, + wal_frames=0, + wal_final_db_pages=0, + copied_rows={}, + ) + + monkeypatch.setattr(recover_raw_flocks_db, "recover_raw_storage_db", fake_recovery) + + assert ( + recover_raw_flocks_db.main( + [ + str(damaged_db), + "--output", + str(output_db), + "--artifacts-dir", + str(artifacts_dir), + "--prefix", + "safe", + ] + ) + == 0 + ) + + assert output_db.read_bytes() == b"recovered-db" + assert wal_path.read_bytes() == b"live-wal" + assert shm_path.read_bytes() == b"live-shm" + assert "removed_sidecars=none\n" in capsys.readouterr().out + + +def test_real_recovery_preserves_auth_and_custom_tables(tmp_path, monkeypatch): + if shutil.which("sqlite3") is None: + pytest.skip("sqlite3 CLI is required for the recovery integration test") + + data_dir = tmp_path / "live-data" + data_dir.mkdir() + source_db = tmp_path / "source.db" + with sqlite3.connect(source_db) as db: + db.executescript( + """ + CREATE TABLE storage ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL, + type TEXT NOT NULL, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + ); + INSERT INTO storage VALUES ('probe', '{}', 'json', 'now', 'now'); + CREATE TABLE users ( + id TEXT PRIMARY KEY, + username TEXT NOT NULL UNIQUE, + password_hash TEXT NOT NULL, + role TEXT NOT NULL DEFAULT 'member', + status TEXT NOT NULL DEFAULT 'active', + must_reset_password INTEGER NOT NULL DEFAULT 0, + tenant_ids TEXT NOT NULL DEFAULT '[]', + asset_groups TEXT NOT NULL DEFAULT '[]', + temp_password_expires_at TEXT, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + last_login_at TEXT + ); + INSERT INTO users VALUES ( + 'u1', 'admin', 'hash', 'admin', 'active', 0, '[]', '[]', NULL, 'now', 'now', NULL + ); + CREATE TABLE user_sessions ( + session_id TEXT PRIMARY KEY, + user_id TEXT NOT NULL REFERENCES users(id) ON DELETE CASCADE, + expires_at TEXT NOT NULL, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + ); + INSERT INTO user_sessions VALUES ('session-1', 'u1', 'later', 'now', 'now'); + CREATE TABLE custom_records (id INTEGER PRIMARY KEY, value TEXT NOT NULL); + CREATE INDEX idx_custom_records_value ON custom_records(value); + CREATE VIEW custom_record_values AS SELECT value FROM custom_records; + CREATE TABLE custom_audit (record_id INTEGER NOT NULL, value TEXT NOT NULL); + CREATE TRIGGER custom_records_audit + AFTER INSERT ON custom_records + BEGIN + INSERT INTO custom_audit VALUES (NEW.id, NEW.value); + END; + INSERT INTO custom_records(value) VALUES ('preserved'); + CREATE TABLE lost_and_found (id INTEGER PRIMARY KEY, value TEXT NOT NULL); + INSERT INTO lost_and_found(value) VALUES ('business-table'); + PRAGMA user_version=7; + PRAGMA application_id=4242; + """ + ) + db.commit() + + artifacts_dir = tmp_path / "artifacts" + output_db = tmp_path / "recovered.db" + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + assert ( + recover_raw_flocks_db.main( + [ + str(source_db), + "--output", + str(output_db), + "--artifacts-dir", + str(artifacts_dir), + "--prefix", + "real", + ] + ) + == 0 + ) + + with sqlite3.connect(output_db) as db: + assert db.execute("PRAGMA integrity_check").fetchone() == ("ok",) + assert db.execute("SELECT username FROM users").fetchone() == ("admin",) + assert db.execute("SELECT session_id FROM user_sessions").fetchone() == ("session-1",) + assert db.execute("SELECT value FROM custom_records").fetchone() == ("preserved",) + assert db.execute("SELECT value FROM custom_record_values").fetchone() == ("preserved",) + assert db.execute("SELECT value FROM lost_and_found").fetchone() == ("business-table",) + assert db.execute( + "SELECT COUNT(*) FROM sqlite_master WHERE type='index' AND name='idx_custom_records_value'" + ).fetchone() == (1,) + db.execute("INSERT INTO custom_records(value) VALUES ('after-recovery')") + db.commit() + assert db.execute( + "SELECT value FROM custom_audit WHERE value='after-recovery'" + ).fetchone() == ("after-recovery",) + assert db.execute("PRAGMA user_version").fetchone() == (7,) + assert db.execute("PRAGMA application_id").fetchone() == (4242,) + + +def test_real_wal_recovery_preserves_committed_row(tmp_path, monkeypatch): + if shutil.which("sqlite3") is None: + pytest.skip("sqlite3 CLI is required for the recovery integration test") + + data_dir = tmp_path / "live-data" + data_dir.mkdir() + live_db = tmp_path / "snapshot-source.db" + source = sqlite3.connect(live_db) + try: + assert source.execute("PRAGMA journal_mode=WAL").fetchone()[0] == "wal" + source.execute("PRAGMA wal_autocheckpoint=0") + source.executescript( + """ + CREATE TABLE storage ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL, + type TEXT NOT NULL, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + ); + CREATE TABLE wal_business_records (id INTEGER PRIMARY KEY, value TEXT NOT NULL); + """ + ) + source.commit() + assert source.execute("PRAGMA wal_checkpoint(TRUNCATE)").fetchone()[0] == 0 + source.execute("INSERT INTO storage VALUES ('wal-row', '{}', 'json', 'now', 'now')") + source.execute("INSERT INTO wal_business_records(value) VALUES ('wal-only')") + source.commit() + + raw_copy = tmp_path / "offline-copy.db" + wal_copy = tmp_path / "offline-copy.db-wal" + shutil.copy2(live_db, raw_copy) + shutil.copy2(live_db.with_name(f"{live_db.name}-wal"), wal_copy) + main_only_copy = tmp_path / "main-only.db" + shutil.copy2(live_db, main_only_copy) + with sqlite3.connect(main_only_copy) as main_only: + assert main_only.execute("SELECT COUNT(*) FROM wal_business_records").fetchone() == ( + 0, + ) + + output_db = tmp_path / "wal-recovered.db" + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + assert ( + recover_raw_flocks_db.main( + [ + str(raw_copy), + "--wal", + str(wal_copy), + "--output", + str(output_db), + "--artifacts-dir", + str(tmp_path / "wal-artifacts"), + "--prefix", + "wal", + ] + ) + == 0 + ) + finally: + source.close() + + with sqlite3.connect(output_db) as db: + assert db.execute("PRAGMA integrity_check").fetchone() == ("ok",) + assert db.execute("SELECT key FROM storage WHERE key='wal-row'").fetchone() == ( + "wal-row", + ) + assert db.execute("SELECT value FROM wal_business_records").fetchone() == ("wal-only",) + + +def test_real_recovery_preserves_fts_virtual_table(tmp_path, monkeypatch): + if shutil.which("sqlite3") is None: + pytest.skip("sqlite3 CLI is required for the recovery integration test") + + source_db = tmp_path / "fts-source.db" + with sqlite3.connect(source_db) as db: + try: + db.execute("CREATE VIRTUAL TABLE memory_search USING fts5(content)") + except sqlite3.OperationalError: + pytest.skip("SQLite was built without FTS5") + db.execute("INSERT INTO memory_search(content) VALUES ('recoverable memory')") + db.commit() + + data_dir = tmp_path / "live-data" + data_dir.mkdir() + output_db = tmp_path / "fts-recovered.db" + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + assert ( + recover_raw_flocks_db.main( + [ + str(source_db), + "--output", + str(output_db), + "--artifacts-dir", + str(tmp_path / "fts-artifacts"), + "--prefix", + "fts", + ] + ) + == 0 + ) + + with sqlite3.connect(output_db) as db: + assert db.execute( + "SELECT content FROM memory_search WHERE memory_search MATCH 'recoverable'" + ).fetchone() == ("recoverable memory",) + + +def test_failed_normalized_build_does_not_publish_output(tmp_path, monkeypatch): + extracted_db = tmp_path / "extracted.db" + with sqlite3.connect(extracted_db) as db: + db.executescript(recover_raw_flocks_db.STORAGE_DDL) + db.commit() + + def fail_validation(*_args, **_kwargs): + raise RuntimeError("injected normalized build failure") + + monkeypatch.setattr( + recover_raw_flocks_db, + "_validate_recovered_db", + fail_validation, + ) + output_db = tmp_path / "recovered.db" + + with pytest.raises(RuntimeError, match="injected normalized build failure"): + recover_raw_flocks_db.build_normalized_recovery_db(extracted_db, output_db) + + assert output_db.exists() is False + + +def test_unattributed_lost_rows_are_retained_without_guessing_destination(tmp_path): + extracted_db = tmp_path / "extracted.db" + lost_table = "flocks_recovery_lost_test" + with sqlite3.connect(extracted_db) as db: + db.executescript(recover_raw_flocks_db.STORAGE_DDL) + db.execute( + "INSERT INTO storage VALUES (?, ?, ?, ?, ?)", + ("session:existing", '"direct"', "json", "now", "now"), + ) + db.executescript( + """ + CREATE TABLE storage_audit(key TEXT NOT NULL); + CREATE TRIGGER storage_insert_audit + AFTER INSERT ON storage + BEGIN + INSERT INTO storage_audit VALUES (NEW.key); + END; + """ + ) + lost_columns = ", ".join(f"c{index} TEXT" for index in range(24)) + db.execute(f"CREATE TABLE {lost_table} (nfield INTEGER, {lost_columns})") + placeholders = ", ".join("?" for _ in range(25)) + db.executemany( + f"INSERT INTO {lost_table} VALUES ({placeholders})", + [ + (6, "ignored", "x", "x", "x", "x", *(None for _ in range(19))), + ( + 5, + "session:existing", + '"lost"', + "json", + "old", + "old", + *(None for _ in range(19)), + ), + ( + 5, + "session:new", + '"recovered"', + "json", + "old", + "old", + *(None for _ in range(19)), + ), + ], + ) + db.commit() + + output_db = tmp_path / "recovered.db" + counts = recover_raw_flocks_db.build_normalized_recovery_db( + extracted_db, + output_db, + lost_and_found_table=lost_table, + ) + + assert counts["storage"] == 1 + with sqlite3.connect(output_db) as db: + assert db.execute( + "SELECT value FROM storage WHERE key='session:existing'" + ).fetchone() == ('"direct"',) + assert db.execute("SELECT value FROM storage WHERE key='session:new'").fetchone() is None + assert db.execute(f"SELECT COUNT(*) FROM {lost_table}").fetchone() == (3,) + assert db.execute("SELECT COUNT(*) FROM storage_audit").fetchone() == (0,) + db.execute( + "INSERT INTO storage VALUES (?, ?, ?, ?, ?)", + ("session:after", '"normal"', "json", "now", "now"), + ) + db.commit() + assert db.execute("SELECT key FROM storage_audit").fetchone() == ("session:after",) + + +def test_exact_backup_accepts_legacy_schema_and_narrow_lost_table(tmp_path): + """Current schema assumptions must not block an otherwise valid recovered DB.""" + extracted_db = tmp_path / "legacy-extracted.db" + lost_table = "flocks_recovery_lost_narrow" + orphan_table = "flocks_recovery_lost_orphan" + with sqlite3.connect(extracted_db) as db: + db.execute( + "CREATE TABLE usage_records(id TEXT PRIMARY KEY, legacy_value TEXT NOT NULL)" + ) + db.execute("INSERT INTO usage_records VALUES ('legacy-1', 'preserve')") + db.execute( + f"CREATE TABLE {lost_table} " + "(rootpgno INTEGER, pgno INTEGER, nfield INTEGER, c0 TEXT, c1 TEXT, c2 TEXT, c3 TEXT, c4 TEXT)" + ) + db.execute( + f"INSERT INTO {lost_table} VALUES (1, 2, 5, ?, ?, ?, ?, ?)", + ("https://example.test/path", "v", "t", "created", "updated"), + ) + orphan_columns = ", ".join(f"c{index} TEXT" for index in range(24)) + db.execute( + f"CREATE TABLE {orphan_table} " + f"(rootpgno INTEGER, pgno INTEGER, nfield INTEGER, {orphan_columns})" + ) + orphan_values = ( + 5, + 6, + 24, + "txe_orphan", + "missing_scheduler", + *(None for _ in range(22)), + ) + db.execute( + f"INSERT INTO {orphan_table} VALUES ({', '.join('?' for _ in orphan_values)})", + orphan_values, + ) + db.commit() + + output_db = tmp_path / "legacy-recovered.db" + counts = recover_raw_flocks_db.build_normalized_recovery_db( + extracted_db, + output_db, + lost_and_found_table=lost_table, + ) + + assert counts["usage_records"] == 1 + assert counts["storage"] == 0 + with sqlite3.connect(output_db) as db: + assert db.execute("PRAGMA table_info(usage_records)").fetchall() == [ + (0, "id", "TEXT", 0, None, 1), + (1, "legacy_value", "TEXT", 1, None, 0), + ] + assert db.execute("SELECT * FROM usage_records").fetchone() == ( + "legacy-1", + "preserve", + ) + assert db.execute(f"SELECT COUNT(*) FROM {lost_table}").fetchone() == (1,) + assert db.execute(f"SELECT COUNT(*) FROM {orphan_table}").fetchone() == (1,) + assert not db.execute( + "SELECT 1 FROM sqlite_schema WHERE name='task_executions'" + ).fetchone() + assert not db.execute( + "SELECT 1 FROM sqlite_schema WHERE name='idx_usage_provider'" + ).fetchone() + + +@pytest.mark.parametrize( + "mutation", + ["delete_row", "update_row", "rowid_change", "drop_view"], +) +def test_recovered_db_validator_fails_on_unknown_data_or_schema_loss(tmp_path, mutation): + source_db = tmp_path / "source.db" + target_db = tmp_path / "target.db" + with sqlite3.connect(source_db) as source: + source.executescript(recover_raw_flocks_db.STORAGE_DDL) + source.executescript( + """ + CREATE TABLE plugin_records(id INTEGER PRIMARY KEY, value TEXT NOT NULL); + INSERT INTO plugin_records(value) VALUES ('preserve-me'); + CREATE TABLE plugin_heap(value TEXT NOT NULL); + INSERT INTO plugin_heap(value) VALUES ('same-visible-value'); + CREATE VIEW plugin_record_values AS SELECT value FROM plugin_records; + """ + ) + source.commit() + with sqlite3.connect(target_db) as target: + source.backup(target) + + with sqlite3.connect(target_db) as target: + if mutation == "delete_row": + target.execute("DELETE FROM plugin_records") + elif mutation == "update_row": + target.execute("UPDATE plugin_records SET value='changed'") + elif mutation == "rowid_change": + target.execute("DELETE FROM plugin_heap") + target.execute( + "INSERT INTO plugin_heap(rowid, value) VALUES (99, 'same-visible-value')" + ) + else: + target.execute("DROP VIEW plugin_record_values") + target.commit() + + with sqlite3.connect(source_db) as source, sqlite3.connect(target_db) as target: + with pytest.raises(sqlite3.DatabaseError): + recover_raw_flocks_db._validate_recovered_db(source, target) + + +def test_validator_skips_unavailable_virtual_table_module(tmp_path): + source_db = tmp_path / "source.db" + target_db = tmp_path / "target.db" + with sqlite3.connect(source_db) as source: + source.execute("CREATE TABLE plugin_records(id INTEGER PRIMARY KEY, value TEXT)") + source.execute("INSERT INTO plugin_records(value) VALUES ('preserved')") + source.execute("PRAGMA writable_schema=ON") + source.execute( + "INSERT INTO sqlite_schema(type, name, tbl_name, rootpage, sql) " + "VALUES ('table', 'plugin_vtab', 'plugin_vtab', 0, " + "'CREATE VIRTUAL TABLE plugin_vtab USING unavailable_module(value)')" + ) + source.execute("PRAGMA writable_schema=OFF") + source.commit() + with sqlite3.connect(target_db) as target: + source.backup(target) + + with sqlite3.connect(source_db) as source, sqlite3.connect(target_db) as target: + recover_raw_flocks_db._validate_recovered_db(source, target) + + +def test_failed_final_copy_does_not_publish_output(tmp_path, monkeypatch): + source = tmp_path / "source.db" + source.write_bytes(b"recovered-content") + output = tmp_path / "output.db" + + def fail_copy(_source, destination): + destination.write(b"part") + raise OSError("injected copy failure") + + monkeypatch.setattr(recover_raw_flocks_db.shutil, "copyfileobj", fail_copy) + + with pytest.raises(OSError, match="injected copy failure"): + recover_raw_flocks_db._copy_file_exclusive(source, output) + + assert output.exists() is False + + +def test_output_created_after_validation_is_not_overwritten(tmp_path, monkeypatch): + data_dir = tmp_path / "live-data" + data_dir.mkdir() + damaged_db = tmp_path / "damaged.db" + damaged_db.write_bytes(b"damaged-db") + artifacts_dir = tmp_path / "artifacts" + output_db = tmp_path / "recovered.db" + monkeypatch.setenv("FLOCKS_DATA_DIR", str(data_dir)) + + def fake_recovery(raw_path, wal_path, recovery_dir, *, prefix): + assert raw_path == damaged_db + assert wal_path is None + recovery_dir.mkdir(parents=True) + artifact_paths = recover_raw_flocks_db._recovery_artifact_paths(recovery_dir, prefix) + candidate_db, recover_sql, extracted_db, recovered_db, summary_path = artifact_paths + recovered_db.write_bytes(b"recovered-db") + summary_path.write_text("recovery=ok\n", encoding="utf-8") + output_db.write_bytes(b"late-output") + return recover_raw_flocks_db.RecoveryArtifacts( + recovery_dir=recovery_dir, + candidate_db=candidate_db, + recover_sql=recover_sql, + extracted_db=extracted_db, + recovered_db=recovered_db, + summary_path=summary_path, + lost_and_found_table="lost_and_found", + pagesize=4096, + wal_frames=0, + wal_final_db_pages=0, + copied_rows={}, + ) + + monkeypatch.setattr(recover_raw_flocks_db, "recover_raw_storage_db", fake_recovery) + + with pytest.raises(FileExistsError): + recover_raw_flocks_db.main( + [ + str(damaged_db), + "--output", + str(output_db), + "--artifacts-dir", + str(artifacts_dir), + "--prefix", + "safe", + ] + ) + + assert output_db.read_bytes() == b"late-output" diff --git a/tests/storage/test_storage.py b/tests/storage/test_storage.py index 27c5fbdac..62ddb0c6e 100644 --- a/tests/storage/test_storage.py +++ b/tests/storage/test_storage.py @@ -7,6 +7,7 @@ import os import shutil import sqlite3 +import threading import pytest from pathlib import Path import tempfile @@ -252,17 +253,22 @@ async def test_storage_init_quarantines_invalid_header_and_boots(tmp_path): with patch.object(Storage, "_initialized", False), \ patch.object(Storage, "_db_path", None): await Storage.init(db_path) - - # Fresh DB is now usable - await Storage.set("hello", {"value": 1}) - assert await Storage.get("hello") == {"value": 1} + # Fresh DB is now usable before the patched global state is restored. + await Storage.set("hello", {"value": 1}) + assert await Storage.get("hello") == {"value": 1} siblings = sorted(p.name for p in tmp_path.iterdir()) assert "flocks.db" in siblings corrupt_files = [name for name in siblings if ".corrupt." in name] assert any(name.startswith("flocks.db.corrupt.") for name in corrupt_files), siblings - assert any(name.startswith("flocks.db-wal.corrupt.") for name in corrupt_files), siblings - assert any(name.startswith("flocks.db-shm.corrupt.") for name in corrupt_files), siblings + quarantined_main = next( + tmp_path / name + for name in corrupt_files + if name.startswith("flocks.db.corrupt.") + and not name.endswith(("-wal", "-shm")) + ) + assert quarantined_main.with_name(quarantined_main.name + "-wal").exists() + assert quarantined_main.with_name(quarantined_main.name + "-shm").exists() @pytest.mark.asyncio @@ -291,26 +297,31 @@ async def test_storage_init_recovers_when_pragma_reports_corruption(tmp_path): @pytest.mark.asyncio -async def test_storage_get_recovers_corrupt_db_after_init(tmp_path): - """A request-time storage read should quarantine corruption and retry once.""" +async def test_storage_get_defers_corruption_recovery_until_restart(tmp_path): + """A request-time read must not replace a DB while the server is running.""" db_path = tmp_path / "flocks.db" with patch.object(Storage, "_initialized", False), \ - patch.object(Storage, "_db_path", None): + patch.object(Storage, "_db_path", None), \ + patch.object( + Storage, + "_quarantine_corrupt_db", + side_effect=AssertionError("request-time recovery must not quarantine the live DB"), + ): await Storage.init(db_path) await Storage.set("hello", {"value": "before"}) - db_path.write_bytes(Storage._SQLITE_MAGIC + b"\xff" * 2048) - db_path.with_name("flocks.db-wal").write_bytes(b"fake wal payload") - db_path.with_name("flocks.db-shm").write_bytes(b"fake shm payload") + corrupt_payload = Storage._SQLITE_MAGIC + b"\xff" * 2048 + db_path.write_bytes(corrupt_payload) - assert await Storage.get("hello") is None - await Storage.set("hello", {"value": "after"}) - assert await Storage.get("hello") == {"value": "after"} + with pytest.raises(sqlite3.DatabaseError): + await Storage.get("hello") + + assert db_path.read_bytes() == corrupt_payload siblings = sorted(p.name for p in tmp_path.iterdir()) assert "flocks.db" in siblings - assert any(name.startswith("flocks.db.corrupt.") for name in siblings), siblings + assert not any(".corrupt." in name for name in siblings), siblings def test_try_sqlite_recover_installs_recovered_db(tmp_path): @@ -337,6 +348,8 @@ def test_try_sqlite_recover_installs_recovered_db(tmp_path): "INSERT INTO storage (key, value, type, created_at, updated_at) VALUES (?, ?, ?, ?, ?)", ("hello", '{"value": "recovered"}', "json", "old", "old"), ) + conn.execute("CREATE TABLE lost_and_found (value TEXT NOT NULL)") + conn.execute("INSERT INTO lost_and_found VALUES ('business-data')") conn.commit() finally: conn.close() @@ -349,10 +362,278 @@ def test_try_sqlite_recover_installs_recovered_db(tmp_path): ("hello",), ).fetchone()[0] == '{"value": "recovered"}' assert recovered.execute("PRAGMA integrity_check").fetchone()[0] == "ok" + assert recovered.execute("SELECT value FROM lost_and_found").fetchone() == ( + "business-data", + ) finally: recovered.close() +def test_sqlite_recover_reads_wal_paired_with_quarantined_main(tmp_path): + """A committed WAL must stay paired with the renamed main DB during recovery.""" + if shutil.which("sqlite3") is None: + pytest.skip("sqlite3 CLI is not available") + + source = tmp_path / "source.db" + source_conn = sqlite3.connect(source) + try: + assert source_conn.execute("PRAGMA journal_mode=WAL").fetchone()[0] == "wal" + source_conn.execute("PRAGMA wal_autocheckpoint=0") + source_conn.execute( + """ + CREATE TABLE storage ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL, + type TEXT NOT NULL, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + ) + """ + ) + source_conn.execute( + "INSERT INTO storage (key, value, type, created_at, updated_at) VALUES (?, ?, ?, ?, ?)", + ("wal-key", '{"value": "committed"}', "json", "old", "old"), + ) + source_conn.commit() + + db_path = tmp_path / "flocks.db" + shutil.copy2(source, db_path) + shutil.copy2(source.with_name("source.db-wal"), db_path.with_name("flocks.db-wal")) + + quarantined = Storage._quarantine_corrupt_db(db_path) + assert quarantined is not None + paired_wal = quarantined.with_name(quarantined.name + "-wal") + assert paired_wal.exists() + assert not db_path.with_name("flocks.db-wal").exists() + quarantined_bytes = quarantined.read_bytes() + wal_bytes = paired_wal.read_bytes() + + assert Storage._try_sqlite_recover_sync(quarantined, db_path) == db_path + assert quarantined.read_bytes() == quarantined_bytes + assert paired_wal.read_bytes() == wal_bytes + recovered = sqlite3.connect(db_path) + try: + assert recovered.execute( + "SELECT value FROM storage WHERE key = ?", + ("wal-key",), + ).fetchone()[0] == '{"value": "committed"}' + finally: + recovered.close() + finally: + source_conn.close() + + +def test_quarantine_rolls_back_main_when_wal_cannot_be_paired(tmp_path, monkeypatch): + """A WAL rename failure must abort instead of recovering an incomplete snapshot.""" + + db_path = tmp_path / "flocks.db" + wal_path = tmp_path / "flocks.db-wal" + main_payload = b"SQLite format 3\x00" + b"main" + wal_payload = b"committed transactions" + db_path.write_bytes(main_payload) + wal_path.write_bytes(wal_payload) + + real_rename = Path.rename + + def fail_wal_rename(path: Path, target: Path) -> Path: + if path == wal_path: + raise OSError("simulated WAL rename failure") + return real_rename(path, target) + + monkeypatch.setattr(Path, "rename", fail_wal_rename) + + assert Storage._quarantine_corrupt_db(db_path) is None + assert db_path.read_bytes() == main_payload + assert wal_path.read_bytes() == wal_payload + assert not list(tmp_path.glob("flocks.db.corrupt.*")) + + +@pytest.mark.asyncio +async def test_recover_corrupt_primary_refuses_online_replacement(tmp_path): + """No partial connection barrier may advertise unsafe online recovery.""" + db_path = tmp_path / "flocks.db" + + with patch.object(Storage, "_initialized", False), \ + patch.object(Storage, "_db_path", None), \ + patch.object(Storage, "_init_pid", None), \ + patch.object(Storage, "_corruption_recovery_generation", 0), \ + patch.object( + Storage, + "_quarantine_corrupt_db", + side_effect=AssertionError("online recovery must fail before quarantine"), + ): + try: + await Storage.init(db_path) + inode_before = db_path.stat().st_ino + async with Storage.connect(db_path) as held: + await held.execute("CREATE TABLE recovery_probe (id TEXT PRIMARY KEY)") + await held.execute("INSERT INTO recovery_probe VALUES ('before')") + await held.commit() + + with pytest.raises(Storage.StorageError, match="while Flocks is running"): + await Storage.recover_corrupt_db( + db_path, + action="test.explicit_recovery", + exc=sqlite3.DatabaseError("database disk image is malformed"), + ) + + await held.execute("INSERT INTO recovery_probe VALUES ('after')") + await held.commit() + + assert db_path.stat().st_ino == inode_before + with sqlite3.connect(db_path) as fresh: + rows = fresh.execute("SELECT id FROM recovery_probe ORDER BY id").fetchall() + assert rows == [("after",), ("before",)] + finally: + await Storage.shutdown() + + +@pytest.mark.asyncio +async def test_live_database_path_disappearance_fails_closed(tmp_path): + """A moved live DB must not cause _ensure_init to create a second inode.""" + from flocks.channel.inbound import session_binding + + await session_binding.close_binding_db() + db_path = tmp_path / "flocks.db" + archived_path = tmp_path / "flocks.db.moved" + + with patch.object(Storage, "_initialized", False), \ + patch.object(Storage, "_db_path", None), \ + patch.object(Storage, "_init_pid", None): + try: + await Storage.init(db_path) + held = await session_binding._get_db() + await held.execute("CREATE TABLE live_probe (value TEXT)") + await held.execute("INSERT INTO live_probe VALUES ('before')") + await held.commit() + + db_path.rename(archived_path) + for suffix in ("-wal", "-shm"): + sidecar = db_path.with_name(db_path.name + suffix) + if sidecar.exists(): + sidecar.rename(archived_path.with_name(archived_path.name + suffix)) + + with pytest.raises(Storage.StorageError, match="active SQLite database disappeared"): + await Storage.set("must-not-create", {"value": 1}) + + assert not db_path.exists() + with pytest.raises(ValueError, match="no active connection"): + await held.execute("SELECT 1") + finally: + await session_binding.close_binding_db() + await Storage.shutdown() + + +@pytest.mark.asyncio +async def test_live_database_inode_replacement_fails_closed(tmp_path): + """An atomic restore over the live path must not mix old and new DB handles.""" + from flocks.channel.inbound import session_binding + + await session_binding.close_binding_db() + db_path = tmp_path / "flocks.db" + archived_path = tmp_path / "flocks.db.before-restore" + + with patch.object(Storage, "_initialized", False), \ + patch.object(Storage, "_db_path", None), \ + patch.object(Storage, "_init_pid", None), \ + patch.object(Storage, "_db_identity", None): + try: + await Storage.init(db_path) + original_identity = Storage._db_identity + held = await session_binding._get_db() + await held.execute("CREATE TABLE live_probe (value TEXT)") + await held.execute("INSERT INTO live_probe VALUES ('old-inode')") + await held.commit() + + db_path.rename(archived_path) + for suffix in ("-wal", "-shm"): + sidecar = db_path.with_name(db_path.name + suffix) + if sidecar.exists(): + sidecar.rename(archived_path.with_name(archived_path.name + suffix)) + with sqlite3.connect(db_path) as replacement: + replacement.execute("CREATE TABLE restored_probe(value TEXT)") + replacement.execute("INSERT INTO restored_probe VALUES ('new-inode')") + replacement.commit() + + assert Storage._file_identity(db_path) != original_identity + with pytest.raises(Storage.StorageError, match="different file identity"): + async with Storage.connect(db_path): + pass + with pytest.raises(Storage.StorageError, match="different file identity"): + Storage.connect_sync(db_path) + with pytest.raises(Storage.StorageError, match="replaced on disk"): + await Storage.set("must-not-write", {"value": 1}) + with pytest.raises(ValueError, match="no active connection"): + await held.execute("SELECT 1") + + with sqlite3.connect(db_path) as replacement: + assert replacement.execute("SELECT value FROM restored_probe").fetchone() == ( + "new-inode", + ) + assert not replacement.execute( + "SELECT 1 FROM sqlite_schema WHERE name='storage'" + ).fetchone() + finally: + await session_binding.close_binding_db() + await Storage.shutdown() + + +@pytest.mark.asyncio +async def test_cancelled_recovery_waits_for_worker_to_finish(): + """Cancellation must not release callers while a recovery thread still mutates files.""" + started = threading.Event() + release = threading.Event() + + def blocked_worker() -> None: + started.set() + release.wait(timeout=5) + + worker = asyncio.create_task(asyncio.to_thread(blocked_worker)) + waiter = asyncio.create_task(Storage._wait_for_recovery_worker_on_cancel(worker)) + assert await asyncio.to_thread(started.wait, 2) + + waiter.cancel() + await asyncio.sleep(0) + assert not waiter.done() + + release.set() + with pytest.raises(asyncio.CancelledError): + await waiter + assert worker.done() + + +@pytest.mark.asyncio +async def test_cancelled_startup_waits_for_recovery_thread(tmp_path, monkeypatch): + """The real Storage.init recovery branch must not leave a detached installer thread.""" + db_path = tmp_path / "flocks.db" + db_path.write_bytes(Storage._SQLITE_MAGIC + b"\xff" * 4096) + started = threading.Event() + release = threading.Event() + finished = threading.Event() + + def blocked_recovery(_quarantined: Path, _target: Path) -> None: + started.set() + release.wait(timeout=5) + finished.set() + + monkeypatch.setattr(Storage, "_try_sqlite_recover_sync", staticmethod(blocked_recovery)) + with patch.object(Storage, "_initialized", False), \ + patch.object(Storage, "_db_path", None), \ + patch.object(Storage, "_init_pid", None), \ + patch.object(Storage, "_db_identity", None): + init_task = asyncio.create_task(Storage.init(db_path)) + assert await asyncio.to_thread(started.wait, 2) + + init_task.cancel() + await asyncio.sleep(0) + assert not init_task.done() + + release.set() + with pytest.raises(asyncio.CancelledError): + await init_task + assert finished.is_set() + + @pytest.mark.asyncio async def test_storage_init_recovers_real_malformed_sqlite_file(tmp_path): """Startup should recover a real DB that fails SQLite integrity checks.""" @@ -412,42 +693,50 @@ async def test_storage_init_recovers_real_malformed_sqlite_file(tmp_path): @pytest.mark.asyncio -async def test_storage_get_uses_recovered_db_before_empty_rebuild(tmp_path, monkeypatch): - """A recovered DB should be installed before retrying the failed read.""" - db_path = tmp_path / "flocks.db" - - def fake_recover(quarantined_path: Path, target_path: Path): - assert quarantined_path.name.startswith("flocks.db.corrupt.") - conn = sqlite3.connect(target_path) - try: - conn.execute( - """ - CREATE TABLE storage ( - key TEXT PRIMARY KEY, - value TEXT NOT NULL, - type TEXT NOT NULL, - created_at TEXT NOT NULL, - updated_at TEXT NOT NULL - ) - """ - ) - conn.execute( - "INSERT INTO storage (key, value, type, created_at, updated_at) VALUES (?, ?, ?, ?, ?)", - ("hello", '{"value": "recovered"}', "json", "old", "old"), +async def test_request_time_corruption_does_not_call_recover_corrupt_db(tmp_path): + """The generic request wrapper must surface corruption without replacing files.""" + + async def corrupt_operation(): + raise sqlite3.DatabaseError("database disk image is malformed") + + with patch.object(Storage, "recover_corrupt_db", new_callable=AsyncMock) as recover, \ + patch.object(Storage._log, "error") as error_log: + with pytest.raises(sqlite3.DatabaseError, match="malformed"): + await Storage._run_with_corruption_recovery( + corrupt_operation, + db_path=tmp_path / "flocks.db", + action="test.request_read", ) - conn.commit() - finally: - conn.close() - return target_path - monkeypatch.setattr(Storage, "_try_sqlite_recover_sync", staticmethod(fake_recover)) + recover.assert_not_awaited() + assert error_log.call_args.args[0] == "storage.corruption.deferred_to_restart" + + +@pytest.mark.asyncio +async def test_request_time_corruption_is_recovered_only_after_restart(tmp_path): + """Close the fail-closed loop: no online swap, then startup recovery succeeds.""" + db_path = tmp_path / "flocks.db" with patch.object(Storage, "_initialized", False), \ - patch.object(Storage, "_db_path", None): + patch.object(Storage, "_db_path", None), \ + patch.object(Storage, "_init_pid", None): + await Storage.init(db_path) + await Storage.set("before", {"value": 1}) + + corrupt_payload = Storage._SQLITE_MAGIC + b"\xff" * 4096 + db_path.write_bytes(corrupt_payload) + with pytest.raises(sqlite3.DatabaseError): + await Storage.get("before") + assert db_path.read_bytes() == corrupt_payload + assert not list(tmp_path.glob("flocks.db.corrupt.*")) + + await Storage.shutdown() await Storage.init(db_path) - db_path.write_bytes(Storage._SQLITE_MAGIC + b"\xff" * 2048) + await Storage.set("after", {"value": 2}) + assert await Storage.get("after") == {"value": 2} + await Storage.shutdown() - assert await Storage.get("hello") == {"value": "recovered"} + assert list(tmp_path.glob("flocks.db.corrupt.*")) @pytest.mark.asyncio From 7a992cb971b9392dc143864566375863e1b1821c Mon Sep 17 00:00:00 2001 From: xiami762 <> Date: Mon, 13 Jul 2026 09:49:36 +0800 Subject: [PATCH 17/49] fix(supervisor): remove HTTP health check, use pure liveness probe MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The health check previously required 5 steps (process alive → TCP port → /api/health → GET / SPA). HTTP-level checks (DNS, proxy, filesystem) could intermittently timeout and trigger a false-positive restart after only 2 failures (~10s). Now only checks: - process.poll() is None → process alive - socket.create_connection() succeeds → TCP port open Both must fail for a restart to trigger. HTTP-level timeouts no longer cause daemon to kill the backend. Closes #537 --- flocks/cli/service_process.py | 28 +---------------- tests/cli/test_service_manager.py | 51 ++++++------------------------- 2 files changed, 10 insertions(+), 69 deletions(-) diff --git a/flocks/cli/service_process.py b/flocks/cli/service_process.py index d6407417e..a64d82810 100644 --- a/flocks/cli/service_process.py +++ b/flocks/cli/service_process.py @@ -7,10 +7,6 @@ from dataclasses import dataclass from typing import Protocol -import httpx - -from flocks.cli.service_config import ServiceConfig - @dataclass(frozen=True) class ServiceProbeResult: @@ -59,29 +55,7 @@ def probe(self, process: subprocess.Popen | None, host: str, port: int) -> Servi ) if not tcp_port_accepts_connections(host, port): return ServiceProbeResult(healthy=False, reason=f"port {port} is not listening", restart=True) - - from flocks.cli.service_manager import _backend_health_url, _is_healthy_status_response, backend_access_base_url - - url = _backend_health_url(host, port) - try: - with httpx.Client(timeout=2.0, trust_env=False) as client: - response = client.get(url) - root_response = client.get( - backend_access_base_url(ServiceConfig(backend_host=host, backend_port=port)), - headers={"Accept": "text/html"}, - ) - healthy = _is_healthy_status_response(response) and _is_static_webui_response(root_response) - reason = f"health status={response.status_code}, root status={root_response.status_code}" - except Exception as exc: - healthy = False - reason = f"health failed: {exc}" - return ServiceProbeResult(healthy=healthy, reason=reason) - - -def _is_static_webui_response(response: httpx.Response) -> bool: - """Return True only when the unified service serves the SPA index.""" - content_type = response.headers.get("content-type", "").lower() - return response.status_code == 200 and "text/html" in content_type + return ServiceProbeResult(healthy=True, reason="liveness check passed") def tcp_port_accepts_connections(host: str, port: int) -> bool: diff --git a/tests/cli/test_service_manager.py b/tests/cli/test_service_manager.py index cef7885e4..d69f3f960 100644 --- a/tests/cli/test_service_manager.py +++ b/tests/cli/test_service_manager.py @@ -1522,70 +1522,37 @@ def test_supervisor_recovers_backend_when_port_disappears(monkeypatch, tmp_path: assert daemon.backend.pid == 333 -def test_supervisor_waits_for_second_backend_health_failure(monkeypatch, tmp_path: Path) -> None: +def test_supervisor_liveness_probe_keeps_backend_healthy(monkeypatch, tmp_path: Path) -> None: + """Liveness-only probe: process alive + TCP port open = healthy, no HTTP check needed.""" paths = _make_runtime_paths(tmp_path) calls: list[str] = [] monkeypatch.setattr(service_manager, "ensure_runtime_dirs", lambda: paths) daemon = service_supervisor.SupervisorDaemon( service_manager.ServiceConfig(backend_port=9995, frontend_port=9996), - failure_threshold=2, ) daemon.paths = paths daemon.backend.process = _fake_process(111, ["backend"]) - class FakeClient: - def __init__(self, *_args, **_kwargs) -> None: - pass - - def __enter__(self): - return self - - def __exit__(self, *_args) -> None: - return None - - def get(self, _url, **_kwargs): - return httpx.Response(503, json={"status": "unhealthy"}) - - monkeypatch.setattr(service_process.httpx, "Client", FakeClient) monkeypatch.setattr(service_process, "tcp_port_accepts_connections", lambda *_args: True) monkeypatch.setattr(service_manager, "_terminate_process", lambda _process, name, _console: calls.append(f"stop:{name}")) - monkeypatch.setattr( - service_manager, - "_start_backend_process", - lambda *_args, **_kwargs: calls.append("start:backend") or _fake_process(333, ["backend-new"]), - ) daemon.tick() assert calls == [] - assert daemon.backend.state == "degraded" + assert daemon.backend.state == "healthy" daemon.tick() - assert calls == ["stop:后端", "start:backend"] - - -def test_backend_probe_rejects_api_root_when_static_webui_missing(monkeypatch) -> None: - class FakeClient: - def __init__(self, *_args, **_kwargs) -> None: - pass - - def __enter__(self): - return self - - def __exit__(self, *_args) -> None: - return None + assert calls == [] + assert daemon.backend.state == "healthy" - def get(self, url, **_kwargs): - if str(url).endswith("/api/health"): - return httpx.Response(200, json={"status": "healthy"}) - return httpx.Response(200, json={"status": "running"}) +def test_backend_probe_liveness_only_accepts_alive_process(monkeypatch) -> None: + """Liveness-only probe: process alive + TCP port open = healthy regardless of HTTP response.""" monkeypatch.setattr(service_process, "tcp_port_accepts_connections", lambda *_args: True) - monkeypatch.setattr(service_process.httpx, "Client", FakeClient) result = service_process.BackendProcessAdapter().probe(_fake_process(111, ["backend"]), "127.0.0.1", 5173) - assert result.healthy is False - assert result.reason == "health status=200, root status=200" + assert result.healthy is True + assert result.reason == "liveness check passed" def test_supervisor_reports_webui_as_static_endpoint(monkeypatch, tmp_path: Path) -> None: From 222f9f14bb0d3dff5c7eec91c8fb7734c1e337d7 Mon Sep 17 00:00:00 2001 From: xiami762 <> Date: Mon, 13 Jul 2026 09:55:07 +0800 Subject: [PATCH 18/49] chore: remove dead code _is_healthy_status_response and _backend_health_url --- flocks/cli/service_manager.py | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/flocks/cli/service_manager.py b/flocks/cli/service_manager.py index cc4ce8d44..e75fb8d1a 100644 --- a/flocks/cli/service_manager.py +++ b/flocks/cli/service_manager.py @@ -1042,17 +1042,6 @@ def _is_running_status_response(response: httpx.Response) -> bool: return isinstance(payload, dict) and payload.get("status") == "running" -def _is_healthy_status_response(response: httpx.Response) -> bool: - """Return True when the backend health endpoint reports healthy.""" - if response.status_code != 200: - return False - try: - payload = response.json() - except ValueError: - return False - return isinstance(payload, dict) and payload.get("status") == "healthy" - - def wait_for_http( urls: Sequence[str], name: str, @@ -1086,10 +1075,6 @@ def print(self, *args, **_kwargs) -> None: sys.stdout.flush() -def _backend_health_url(host: str, port: int) -> str: - return f"http://{_format_host_for_url(access_host(host))}:{port}/api/health" - - def _terminate_process( process: subprocess.Popen | None, name: str, From 2fe5e64ddc5381ae7bca0e8a8554297fcc17e03f Mon Sep 17 00:00:00 2001 From: duguwanglong Date: Mon, 13 Jul 2026 11:45:10 +0800 Subject: [PATCH 19/49] fix(webui): harden async refresh and loading flows Preserve last-known-good tool state and expose structured partial refresh outcomes. Serialize concurrent loading paths and keep UI feedback aligned with active requests. --- flocks/mcp/__init__.py | 6 + flocks/mcp/server.py | 4 + flocks/plugin/loader.py | 123 +++++- flocks/server/routes/monitoring.py | 42 +- flocks/server/routes/provider.py | 78 +++- flocks/server/routes/tool.py | 90 +++-- flocks/server/routes/update.py | 28 +- flocks/tool/registry.py | 149 +++++-- tests/mcp/test_mcp_server.py | 13 + tests/plugin/test_plugin.py | 90 +++++ tests/server/routes/test_monitoring_routes.py | 109 +++++ .../test_provider_route_responsiveness.py | 213 ++++++++++ tests/server/routes/test_update_routes.py | 43 +- tests/tool/test_tool_refresh_outcomes.py | 382 ++++++++++++++++++ webui/src/api/monitoring.ts | 1 + webui/src/api/tool.ts | 18 +- webui/src/hooks/useSessions.test.ts | 316 +++++++++++++++ webui/src/hooks/useSessions.ts | 67 ++- webui/src/hooks/useSharedResource.test.tsx | 88 ++++ webui/src/hooks/useSharedResource.ts | 43 +- webui/src/hooks/useTasks.ts | 2 +- webui/src/hooks/useTools.test.tsx | 170 ++++++++ webui/src/hooks/useTools.ts | 86 ++-- webui/src/locales/en-US/monitoring.json | 3 +- webui/src/locales/en-US/tool.json | 4 + webui/src/locales/zh-CN/monitoring.json | 3 +- webui/src/locales/zh-CN/tool.json | 4 + webui/src/pages/Hub/index.test.tsx | 143 ++++++- webui/src/pages/Hub/index.tsx | 73 +++- webui/src/pages/Monitoring/index.test.tsx | 114 +++++- webui/src/pages/Monitoring/index.tsx | 114 ++++-- webui/src/pages/Tool/index.tsx | 64 ++- 32 files changed, 2448 insertions(+), 235 deletions(-) create mode 100644 tests/server/routes/test_monitoring_routes.py create mode 100644 tests/server/routes/test_provider_route_responsiveness.py create mode 100644 tests/tool/test_tool_refresh_outcomes.py diff --git a/flocks/mcp/__init__.py b/flocks/mcp/__init__.py index 3472e96f6..87e90c48c 100644 --- a/flocks/mcp/__init__.py +++ b/flocks/mcp/__init__.py @@ -87,6 +87,12 @@ async def status(cls) -> Dict[str, McpStatusInfo]: """ manager = get_manager() return await manager.status() + + @classmethod + def status_snapshot(cls) -> tuple[Dict[str, McpStatusInfo], bool]: + """Return current MCP status without starting uninitialized servers.""" + manager = get_manager() + return manager.status_snapshot() @classmethod async def get_server_info(cls, name: str) -> Optional[McpServerInfo]: diff --git a/flocks/mcp/server.py b/flocks/mcp/server.py index 6ef4eaa63..b7a20cd10 100644 --- a/flocks/mcp/server.py +++ b/flocks/mcp/server.py @@ -331,6 +331,10 @@ async def status(self) -> Dict[str, McpStatusInfo]: return self._status.copy() + def status_snapshot(self) -> tuple[Dict[str, McpStatusInfo], bool]: + """Return current status without triggering MCP initialization.""" + return self._status.copy(), self._initialized + async def get_server_info(self, name: str) -> Optional[McpServerInfo]: """ Get detailed server information diff --git a/flocks/plugin/loader.py b/flocks/plugin/loader.py index b0144a178..d0bbf59ea 100644 --- a/flocks/plugin/loader.py +++ b/flocks/plugin/loader.py @@ -147,8 +147,8 @@ class ExtensionPoint: subdir: str """Subdirectory under the plugin root, e.g. ``"agents"``.""" - consumer: Callable[[List[Any], str], None] - """Callback ``(items, source_path) -> None`` that receives validated items.""" + consumer: Callable[[List[Any], str], Optional[List[str]]] + """Callback receiving validated items and optionally returning errors.""" item_type: Optional[type] = None """If set, only items that are ``isinstance(item, item_type)`` are kept.""" @@ -255,7 +255,7 @@ def load_extension( project_dir: Optional[Path] = None, *, load_entry_points: bool = False, - ) -> None: + ) -> List[str]: """Load one registered extension point using normal plugin scan rules. This is the scoped counterpart to :meth:`load_all`. It scans the same @@ -269,16 +269,20 @@ def load_extension( ext = cls._extension_points.get(attr_name) if ext is None: log.warn("plugin.ext_point.not_found", {"attr": attr_name}) - return + return [f"extension point not found: {attr_name}"] + + errors: List[str] = [] cls._load_extension_point( ext, extra_sources=extra_sources, project_dir=project_dir or Path.cwd(), log_scope="load_extension", + errors=errors, ) if load_entry_points: - cls._load_entry_points() + cls._load_entry_points(errors=errors) + return errors @classmethod def load_for_extension( @@ -299,9 +303,12 @@ def load_for_extension( collected: List[Any] = [] original_consumer = ext.consumer - def _collecting_consumer(items: List[Any], source: str) -> None: + def _collecting_consumer( + items: List[Any], + source: str, + ) -> Optional[List[str]]: collected.extend(items) - original_consumer(items, source) + return original_consumer(items, source) ext.consumer = _collecting_consumer ext._seen_keys = set() @@ -352,6 +359,7 @@ def _load_extension_point( extra_sources: Optional[List[str]], project_dir: Path, log_scope: str, + errors: Optional[List[str]] = None, ) -> None: """Scan and load one registered extension point.""" if ext.load_once and ext._loaded: @@ -382,7 +390,7 @@ def _load_extension_point( "files": [Path(s).name for s in default_sources], }, ) - cls._load_sources_for_ext(ext, default_sources, subdir_path) + cls._load_sources_for_ext(ext, default_sources, subdir_path, errors=errors) # 2. Project-level plugin subdirectory (/.flocks/plugins/{subdir}/) project_subdir_path = project_plugin_root / ext.subdir @@ -402,17 +410,22 @@ def _load_extension_point( "files": [Path(s).name for s in project_sources], }, ) - cls._load_sources_for_ext(ext, project_sources, project_subdir_path) + cls._load_sources_for_ext( + ext, + project_sources, + project_subdir_path, + errors=errors, + ) # 3. Explicit sources from cfg.plugin if extra_sources: - cls._load_sources_for_ext(ext, extra_sources, project_dir) + cls._load_sources_for_ext(ext, extra_sources, project_dir, errors=errors) if ext.load_once: ext._loaded = True @classmethod - def _load_entry_points(cls) -> None: + def _load_entry_points(cls, errors: Optional[List[str]] = None) -> None: """ Load installed package entry-points under ``flocks.plugins``. @@ -425,6 +438,8 @@ def _load_entry_points(cls) -> None: eps = importlib.metadata.entry_points().select(group=group) except Exception as e: log.debug("plugin.entrypoints.scan_failed", {"group": group, "error": str(e)}) + if errors is not None: + errors.append(f"entry point scan: {type(e).__name__}: {e}") return for ep in eps: @@ -432,10 +447,14 @@ def _load_entry_points(cls) -> None: target = ep.load() except Exception as e: log.warning("plugin.entrypoint.load_failed", {"name": ep.name, "error": str(e)}) + if errors is not None: + errors.append(f"entry point {ep.name}: {e}") continue if not callable(target): log.warning("plugin.entrypoint.not_callable", {"name": ep.name}) + if errors is not None: + errors.append(f"entry point {ep.name}: target is not callable") continue try: @@ -447,6 +466,8 @@ def _load_entry_points(cls) -> None: log.info("plugin.entrypoint.loaded", {"name": ep.name, "group": group}) except Exception as e: log.warning("plugin.entrypoint.invoke_failed", {"name": ep.name, "error": str(e)}) + if errors is not None: + errors.append(f"entry point {ep.name}: {e}") @classmethod def _load_sources_for_ext( @@ -454,12 +475,13 @@ def _load_sources_for_ext( ext: ExtensionPoint, sources: List[str], base_dir: Path, + errors: Optional[List[str]] = None, ) -> None: """Load each source module and dispatch matching items to *ext.consumer*.""" for source in sources: source_path = Path(source) if source_path.suffix in (".yaml", ".yml"): - cls._load_yaml_source(ext, source_path) + cls._load_yaml_source(ext, source_path, errors=errors) continue try: @@ -473,6 +495,8 @@ def _load_sources_for_ext( "type": type(e).__name__, }, ) + if errors is not None: + errors.append(f"{source}: {type(e).__name__}: {e}") continue raw = getattr(module, ext.attr_name, None) @@ -487,11 +511,29 @@ def _load_sources_for_ext( "attr": ext.attr_name, }, ) + if errors is not None: + errors.append(f"{source}: {ext.attr_name} must be a list") continue - items = cls._validate_and_dedup(ext, list(raw), source) + items = cls._validate_and_dedup(ext, list(raw), source, errors=errors) if items: - ext.consumer(items, source) + try: + consumer_errors = ext.consumer(items, source) + except Exception as e: + log.error( + "plugin.consumer_failed", + { + "source": source, + "attr": ext.attr_name, + "error": str(e), + }, + ) + if errors is None: + raise + errors.append(f"{source}: consumer: {type(e).__name__}: {e}") + continue + if errors is not None and consumer_errors: + errors.extend(f"{source}: {error}" for error in consumer_errors) log.info( "plugin.dispatched", { @@ -502,7 +544,12 @@ def _load_sources_for_ext( ) @classmethod - def _load_yaml_source(cls, ext: ExtensionPoint, yaml_path: Path) -> None: + def _load_yaml_source( + cls, + ext: ExtensionPoint, + yaml_path: Path, + errors: Optional[List[str]] = None, + ) -> None: """Load a YAML config file and dispatch via the extension point's factory.""" if ext.yaml_item_factory is None: log.warn( @@ -513,27 +560,60 @@ def _load_yaml_source(cls, ext: ExtensionPoint, yaml_path: Path) -> None: "hint": f"Extension point '{ext.attr_name}' has no yaml_item_factory; skipping YAML file", }, ) + if errors is not None: + errors.append(f"{yaml_path}: YAML is not supported for {ext.attr_name}") return try: raw = yaml.safe_load(yaml_path.read_text(encoding="utf-8")) except Exception as e: log.error("plugin.yaml_parse_failed", {"path": str(yaml_path), "error": str(e)}) + if errors is not None: + errors.append(f"{yaml_path}: {e}") return if not isinstance(raw, dict): log.warn("plugin.yaml_invalid", {"path": str(yaml_path), "hint": "Expected a YAML mapping"}) + if errors is not None: + errors.append(f"{yaml_path}: expected a YAML mapping") return try: item = ext.yaml_item_factory(raw, yaml_path) except Exception as e: log.error("plugin.yaml_factory_failed", {"path": str(yaml_path), "error": str(e)}) + if errors is not None: + errors.append(f"{yaml_path}: {e}") return - items = cls._validate_and_dedup(ext, [item], str(yaml_path)) + items = cls._validate_and_dedup( + ext, + [item], + str(yaml_path), + errors=errors, + ) if items: - ext.consumer(items, str(yaml_path)) + try: + consumer_errors = ext.consumer(items, str(yaml_path)) + except Exception as e: + log.error( + "plugin.consumer_failed", + { + "source": str(yaml_path), + "attr": ext.attr_name, + "error": str(e), + }, + ) + if errors is None: + raise + errors.append( + f"{yaml_path}: consumer: {type(e).__name__}: {e}" + ) + return + if errors is not None and consumer_errors: + errors.extend( + f"{yaml_path}: {error}" for error in consumer_errors + ) log.debug( "plugin.yaml_dispatched", { @@ -549,20 +629,27 @@ def _validate_and_dedup( ext: ExtensionPoint, raw_items: List[Any], source: str, + errors: Optional[List[str]] = None, ) -> List[Any]: """Type-check and deduplicate items for an extension point.""" if ext.item_type is not None: valid = [it for it in raw_items if isinstance(it, ext.item_type)] if len(valid) < len(raw_items): + invalid_count = len(raw_items) - len(valid) log.warn( "plugin.invalid_entries", { "source": source, "attr": ext.attr_name, "expected_type": ext.item_type.__name__, - "invalid_count": len(raw_items) - len(valid), + "invalid_count": invalid_count, }, ) + if errors is not None: + errors.append( + f"{source}: {invalid_count} invalid {ext.attr_name} " + f"entries; expected {ext.item_type.__name__}" + ) else: valid = raw_items diff --git a/flocks/server/routes/monitoring.py b/flocks/server/routes/monitoring.py index 0d5b643be..9a43dcdfc 100644 --- a/flocks/server/routes/monitoring.py +++ b/flocks/server/routes/monitoring.py @@ -13,7 +13,7 @@ from typing import Any, Literal from fastapi import APIRouter, Query -from pydantic import BaseModel +from pydantic import BaseModel, Field from flocks.agent import registry as agent_registry from flocks.mcp import MCP @@ -37,11 +37,30 @@ class SystemStatusResponse(BaseModel): class MetricsSnapshotResponse(BaseModel): timestamp: int - messageRate: float | None - toolCallRate: float | None - errorRate: float | None - avgResponseTime: float | None - activeRequests: int | None + messageRate: float | None = Field( + default=None, + description="Messages per minute. None because message throughput is not collected yet.", + ) + toolCallRate: float | None = Field( + default=None, + description="Parsed tool calls per minute.", + ) + errorRate: float | None = Field( + default=None, + description="System-wide error rate. None until a global error metric is available.", + ) + toolParseFailureRate: float | None = Field( + default=None, + description="Failed tool-call parses per minute.", + ) + avgResponseTime: float | None = Field( + default=None, + description="Average response time in milliseconds. None until latency is collected.", + ) + activeRequests: int | None = Field( + default=None, + description="Active request count. None until request concurrency is collected.", + ) class PerformanceDataResponse(BaseModel): @@ -75,14 +94,14 @@ async def _active_agent_count() -> int | None: async def _mcp_server_statuses() -> tuple[dict[str, str], bool]: try: - statuses = await MCP.status() + statuses, initialized = MCP.status_snapshot() except Exception as exc: log.warning("monitoring.mcp.status_failed", {"error": str(exc)}) return {}, False return ({ name: str(info.status.value if hasattr(info.status, "value") else info.status) for name, info in statuses.items() - }, True) + }, initialized) _metrics_sample_lock = threading.Lock() @@ -131,7 +150,7 @@ def _sample_tool_rates(total_calls: float, failed_calls: float) -> tuple[float | async def get_status() -> SystemStatusResponse: mcp_servers, mcp_status_available = await _mcp_server_statuses() unhealthy_mcp = any( - status.lower() in {"error", "failed", "down"} + status.strip().lower() not in {"connected", "disabled"} for status in mcp_servers.values() ) return SystemStatusResponse( @@ -149,12 +168,13 @@ async def get_metrics() -> MetricsSnapshotResponse: metrics = get_monitor().get_metrics().get("global", {}) total_calls = float(metrics.get("total_calls") or 0) failed_calls = float(metrics.get("failed_parses") or 0) - tool_call_rate, error_rate = _sample_tool_rates(total_calls, failed_calls) + tool_call_rate, tool_parse_failure_rate = _sample_tool_rates(total_calls, failed_calls) return MetricsSnapshotResponse( timestamp=_timestamp_ms(), messageRate=None, toolCallRate=tool_call_rate, - errorRate=error_rate, + errorRate=None, + toolParseFailureRate=tool_parse_failure_rate, avgResponseTime=None, activeRequests=None, ) diff --git a/flocks/server/routes/provider.py b/flocks/server/routes/provider.py index 8c8cf1866..79e4d7e37 100644 --- a/flocks/server/routes/provider.py +++ b/flocks/server/routes/provider.py @@ -6,6 +6,7 @@ temperature, tool_call, limit, etc. """ +import asyncio import json import re import threading @@ -47,6 +48,60 @@ log = Log.create(service="provider-routes") _api_service_summary_metadata_cache_lock = threading.Lock() _api_service_summary_metadata_cache: Dict[str, tuple[tuple[Any, ...], Optional[Dict[str, Any]]]] = {} +_provider_initialization_lock = asyncio.Lock() +_provider_initialization_task: asyncio.Task[None] | None = None +_dynamic_provider_load_tasks: set[asyncio.Task[None]] = set() + + +async def _run_provider_initialization() -> None: + async with _provider_initialization_lock: + if Provider._initialized: + return + await asyncio.to_thread(Provider._ensure_initialized) + + +async def _ensure_provider_initialized() -> None: + """Run synchronous provider discovery without blocking the event loop.""" + global _provider_initialization_task + + if Provider._initialized: + return + + task = _provider_initialization_task + if task is None or task.done(): + task = asyncio.create_task(_run_provider_initialization()) + _provider_initialization_task = task + + def clear_completed(completed: asyncio.Task[None]) -> None: + global _provider_initialization_task + if _provider_initialization_task is completed: + _provider_initialization_task = None + if not completed.cancelled(): + completed.exception() + + task.add_done_callback(clear_completed) + + await asyncio.shield(task) + + +async def _run_dynamic_provider_load() -> None: + async with _provider_initialization_lock: + await asyncio.to_thread(Provider._load_dynamic_providers) + + +async def _load_dynamic_providers() -> None: + """Queue one serialized config scan per trigger without blocking the event loop.""" + task = asyncio.create_task(_run_dynamic_provider_load()) + _dynamic_provider_load_tasks.add(task) + + def clear_completed(completed: asyncio.Task[None]) -> None: + _dynamic_provider_load_tasks.discard(completed) + if not completed.cancelled(): + completed.exception() + + task.add_done_callback(clear_completed) + + await asyncio.shield(task) def _load_provider_yaml_metadata(provider_id: str) -> Optional[Dict[str, Any]]: @@ -470,8 +525,9 @@ async def list_providers() -> ProviderListResponse: List of providers with their models """ try: - # Ensure providers are initialized - Provider._ensure_initialized() + # Initial discovery imports provider modules and uses a threading lock. + # Keep that synchronous work off the server event loop. + await _ensure_provider_initialized() # Get config for enabled/disabled providers try: @@ -752,7 +808,7 @@ async def get_provider(provider_id: str) -> ProviderInfo: Provider information """ try: - Provider._ensure_initialized() + await _ensure_provider_initialized() provider = Provider.get(provider_id) if not provider: @@ -809,7 +865,7 @@ async def list_models(provider_id: str) -> List[Dict[str, Any]]: List of models in Flocks-compatible format """ try: - Provider._ensure_initialized() + await _ensure_provider_initialized() if provider_id not in ConfigWriter.list_provider_ids(): return [] config = await Config.get() @@ -858,7 +914,7 @@ async def configure_provider( Updated provider information """ try: - Provider._ensure_initialized() + await _ensure_provider_initialized() provider = Provider.get(provider_id) if not provider: @@ -911,7 +967,7 @@ async def test_provider( Test result """ try: - Provider._ensure_initialized() + await _ensure_provider_initialized() provider = Provider.get(provider_id) if not provider: @@ -966,7 +1022,7 @@ async def update_provider( Updates the provider's API key, base URL, and custom settings. """ try: - Provider._ensure_initialized() + await _ensure_provider_initialized() provider = Provider.get(provider_id) if not provider: @@ -1817,7 +1873,7 @@ async def set_provider_credentials( ConfigWriter.add_provider(provider_id, config_dict) # 3. Configure the provider runtime so is_configured() reflects the change - Provider._ensure_initialized() + await _ensure_provider_initialized() provider = Provider.get(provider_id) if provider: # Preserve the existing base_url from flocks.json when not explicitly provided, @@ -1893,7 +1949,7 @@ async def delete_provider_credentials( raise HTTPException(status_code=404, detail="No credentials found for this provider") # 4. Clear provider runtime config - Provider._ensure_initialized() + await _ensure_provider_initialized() provider = Provider.get(provider_id) if provider: provider._config = None @@ -2227,12 +2283,12 @@ async def test_provider_credentials( await _save_api_service_status_if_configured(provider_id, response) return response - Provider._ensure_initialized() + await _ensure_provider_initialized() # Re-run dynamic provider loading in case this provider was created after # the server started (e.g. user just added azure-openai via the UI). # _load_dynamic_providers skips already-registered providers, so it is safe # to call multiple times. - Provider._load_dynamic_providers() + await _load_dynamic_providers() # Apply config to ensure _config_models (user-defined models) are loaded config = await Config.get() await Provider.apply_config(config, provider_id=provider_id) diff --git a/flocks/server/routes/tool.py b/flocks/server/routes/tool.py index 65421f7d3..e217cbf3f 100644 --- a/flocks/server/routes/tool.py +++ b/flocks/server/routes/tool.py @@ -17,6 +17,7 @@ from flocks.permission.next import DeniedError, PermissionNext from flocks.tool.registry import ( ToolRegistry, + ToolRefreshError, ToolInfo, ToolSchema, ToolResult, @@ -187,7 +188,7 @@ def _invalidate_tool_summary_cache() -> None: def _tool_summary_cache_current_key() -> tuple[int, tuple[str, ...]]: - return ToolRegistry.revision(), tuple(ToolRegistry.all_tool_ids()) + return ToolRegistry.snapshot_identity() def _get_tool_summary_items() -> List[ToolListIndexItem]: @@ -1138,9 +1139,11 @@ async def execute_batch(request: BatchExecuteRequest): class RefreshResponse(BaseModel): """Tool refresh response""" - status: str = Field(..., description="Operation status") + status: Literal["success", "partial", "error"] = Field(..., description="Operation status") tool_count: int = Field(..., description="Total registered tool count after refresh") message: str = Field("", description="Human-readable summary") + stages: Dict[str, Literal["success", "error"]] = Field(default_factory=dict) + errors: List[str] = Field(default_factory=list) @router.post( @@ -1159,20 +1162,24 @@ async def refresh_tools(_admin: object = Depends(require_admin)): ToolRegistry.init() errors: list[str] = [] + stages: dict[str, Literal["success", "error"]] = {} - # 1. Reload generated tools (generated/) - try: - ToolRegistry.refresh_dynamic_tools() - except Exception as e: - log.error("tools.refresh.dynamic_error", {"error": str(e)}) - errors.append(f"dynamic: {e}") - - # 2. Reload plugin tools (api/, python/) — unregisters stale entries first - try: - ToolRegistry.refresh_plugin_tools() - except Exception as e: - log.error("tools.refresh.plugin_error", {"error": str(e)}) - errors.append(f"plugin: {e}") + for stage, refresh in ( + ("dynamic", ToolRegistry.refresh_dynamic_tools), + ("plugin", ToolRegistry.refresh_plugin_tools), + ): + try: + refresh() + stages[stage] = "success" + except ToolRefreshError as exc: + stages[stage] = "error" + stage_errors = [f"{stage}: {error}" for error in exc.errors] + errors.extend(stage_errors) + log.error(f"tools.refresh.{stage}_error", {"errors": exc.errors}) + except Exception as exc: + stages[stage] = "error" + errors.append(f"{stage}: {exc}") + log.error(f"tools.refresh.{stage}_error", {"error": str(exc)}) _invalidate_tool_summary_cache() tool_count = len(ToolRegistry.all_tool_ids()) @@ -1181,17 +1188,23 @@ async def refresh_tools(_admin: object = Depends(require_admin)): "errors": len(errors), }) - if errors: - return RefreshResponse( - status="partial", - tool_count=tool_count, - message=f"Refreshed with {len(errors)} error(s): {'; '.join(errors)}", - ) + failed_stages = sum(status == "error" for status in stages.values()) + if failed_stages == 0: + outcome: Literal["success", "partial", "error"] = "success" + message = f"All tools refreshed successfully ({tool_count} tools registered)" + elif failed_stages == len(stages): + outcome = "error" + message = f"Tool refresh failed: {'; '.join(errors)}" + else: + outcome = "partial" + message = f"Tool refresh completed with errors: {'; '.join(errors)}" return RefreshResponse( - status="success", + status=outcome, tool_count=tool_count, - message=f"All tools refreshed successfully ({tool_count} tools registered)", + message=message, + stages=stages, + errors=errors, ) @@ -1526,13 +1539,38 @@ async def delete_tool(name: str, _admin: object = Depends(require_admin)): detail=f"Plugin tool not found: {name}", ) - # Refresh plugin tools so stale decorator-registered python tools are removed too. - ToolRegistry.refresh_plugin_tools() + # The file deletion is already committed at this point. Keep the Hub + # installation record consistent even when the in-memory refresh fails. + cleanup_errors: List[str] = [] + try: + ToolRegistry.refresh_plugin_tools() + except Exception as e: + refresh_error = str(e) + cleanup_errors.append(f"registry refresh: {refresh_error}") + log.warning("tool.delete.refresh_failed", { + "error": refresh_error, + "name": name, + }) _invalidate_tool_summary_cache() from flocks.hub import local as hub_local - hub_local.remove_installed_record("tool", name) + try: + hub_local.remove_installed_record("tool", name) + except Exception as e: + hub_error = str(e) + cleanup_errors.append(f"Hub record cleanup: {hub_error}") + log.warning("tool.delete.hub_cleanup_failed", { + "error": hub_error, + "name": name, + }) + + if cleanup_errors: + return { + "status": "partial", + "message": f"Tool {name} deleted, but cleanup was incomplete", + "errors": cleanup_errors, + } return {"status": "success", "message": f"Tool {name} deleted"} diff --git a/flocks/server/routes/update.py b/flocks/server/routes/update.py index fac08eb06..d559fbe72 100644 --- a/flocks/server/routes/update.py +++ b/flocks/server/routes/update.py @@ -64,28 +64,22 @@ async def _check_update_cached( now = time.monotonic() task: asyncio.Task[VersionInfo] async with _update_check_lock: - if force: - # Make the forced check the authoritative in-flight request. An - # older non-forced task may still finish for its original caller, - # but its identity guard in _run_update_check_for_cache prevents - # it from overwriting this newer result. + if not force: + cached = _update_check_cache.get(key) + if cached and cached[0] > now: + return cached[1].model_copy(deep=True) + + # A forced check bypasses only the completed-result cache. Reuse any + # same-key request already reaching the upstream service so concurrent + # manual refreshes cannot fan out into an update-check storm. + existing_task = _update_check_inflight.get(key) + if existing_task is None: task = asyncio.create_task( _run_update_check_for_cache(key, locale=locale, edition=edition) ) _update_check_inflight[key] = task else: - cached = _update_check_cache.get(key) - if cached and cached[0] > now: - return cached[1].model_copy(deep=True) - - existing_task = _update_check_inflight.get(key) - if existing_task is None: - task = asyncio.create_task( - _run_update_check_for_cache(key, locale=locale, edition=edition) - ) - _update_check_inflight[key] = task - else: - task = existing_task + task = existing_task info = await asyncio.shield(task) return info.model_copy(deep=True) diff --git a/flocks/tool/registry.py b/flocks/tool/registry.py index 0f11995e7..d348a8387 100644 --- a/flocks/tool/registry.py +++ b/flocks/tool/registry.py @@ -25,6 +25,15 @@ log = Log.create(service="tool-registry") +class ToolRefreshError(RuntimeError): + """A refresh stage failed and the previous registry state was restored.""" + + def __init__(self, stage: str, errors: List[str]): + self.stage = stage + self.errors = list(errors) + super().__init__(f"{stage}: {'; '.join(self.errors)}") + + class ToolCategory(str, Enum): """Tool categories""" FILE = "file" @@ -602,6 +611,7 @@ class ToolRegistry: _failure_state: Dict[str, Dict[str, Any]] = {} _failure_disable_threshold: int = 3 _init_lock = threading.Lock() + _refresh_lock = threading.RLock() _initializing_thread_id: Optional[int] = None # Snapshot of every tool's factory-default ``enabled`` flag — captured @@ -646,8 +656,9 @@ def register(cls, tool: Tool) -> None: # ``refresh_plugin_tools`` cycle — must refresh the snapshot so # ``enabled_default`` / ``reset`` reflect the current source of # truth instead of the first value ever observed. - cls._enabled_defaults[tool.info.name] = bool(tool.info.enabled) - cls._tools[tool.info.name] = tool + with cls._refresh_lock: + cls._enabled_defaults[tool.info.name] = bool(tool.info.enabled) + cls._tools[tool.info.name] = tool log.debug("tool.registered", { "name": tool.info.name, "category": tool.info.category.value, @@ -660,7 +671,8 @@ def revision(cls) -> int: The revision is bumped when plugin or dynamic tools are reloaded so long-lived session caches can detect toolset changes. """ - return cls._revision + with cls._refresh_lock: + return cls._revision @classmethod def _bump_revision(cls, reason: str) -> None: @@ -728,7 +740,8 @@ def decorator(func: ToolHandler) -> ToolHandler: @classmethod def unregister(cls, name: str) -> bool: """Unregister a tool by name. Returns True if the tool was found and removed.""" - removed = cls._tools.pop(name, None) + with cls._refresh_lock: + removed = cls._tools.pop(name, None) if removed: log.debug("tool.unregistered", {"name": name}) return removed is not None @@ -743,18 +756,18 @@ def _ensure_initialized(cls) -> None: def get(cls, name: str) -> Optional[Tool]: """Get a tool by name""" cls._ensure_initialized() - return cls._tools.get(name) + with cls._refresh_lock: + return cls._tools.get(name) @classmethod def list_tools(cls, category: Optional[ToolCategory] = None) -> List[ToolInfo]: """List all registered tools, optionally filtered by category""" cls._ensure_initialized() - tools = list(cls._tools.values()) - - if category: - tools = [t for t in tools if t.info.category == category] - - return [t.info for t in tools] + with cls._refresh_lock: + tools = list(cls._tools.values()) + if category: + tools = [t for t in tools if t.info.category == category] + return [t.info for t in tools] @classmethod def get_schema(cls, name: str) -> Optional[ToolSchema]: @@ -983,13 +996,25 @@ async def execute_batch( def all_tool_ids(cls) -> List[str]: """Get all registered tool IDs""" cls._ensure_initialized() - return list(cls._tools.keys()) + with cls._refresh_lock: + return list(cls._tools.keys()) + + @classmethod + def snapshot_identity(cls) -> tuple[int, tuple[str, ...]]: + """Return a revision/tool-id identity from one consistent registry state.""" + cls._ensure_initialized() + with cls._refresh_lock: + return cls._revision, tuple(cls._tools.keys()) @classmethod def get_dynamic_tools_by_module(cls) -> Dict[str, List[str]]: """Return a copy of the dynamic-module → tool-names mapping.""" cls._ensure_initialized() - return dict(cls._dynamic_tools_by_module) + with cls._refresh_lock: + return { + module_name: list(tool_names) + for module_name, tool_names in cls._dynamic_tools_by_module.items() + } @classmethod def get_api_service_ids(cls) -> set: @@ -1039,7 +1064,7 @@ async def init_async(cls) -> None: await asyncio.to_thread(cls.init) @classmethod - def _load_plugin_tools(cls) -> None: + def _load_plugin_tools(cls, errors: Optional[List[str]] = None) -> None: """Load plugin tools from both user-level and project-level plugin dirs on init. Without this, YAML/Python plugin tools only appear after an explicit @@ -1057,9 +1082,13 @@ def _load_plugin_tools(cls) -> None: try: from flocks.plugin import PluginLoader - PluginLoader.load_extension("TOOLS", load_entry_points=True) + load_errors = PluginLoader.load_extension("TOOLS", load_entry_points=True) + if errors is not None: + errors.extend(load_errors) except Exception as e: log.warn("tool_registry.plugin_load_failed", {"error": str(e)}) + if errors is not None: + errors.append(f"plugin loader: {e}") after = set(cls._tools.keys()) new_plugin_tools = sorted(after - before) python_tool_sources: Dict[str, Path] = {} @@ -1102,6 +1131,8 @@ def _load_plugin_tools(cls) -> None: except ValueError: tool.info.native = True cls._plugin_tool_names = sorted(set(new_plugin_tools) | python_plugin_names) + if errors: + return cls._bootstrap_user_api_services() # Defence-in-depth: ``register()`` is the canonical writer for # ``_enabled_defaults`` but this catches any tool that landed in @@ -1265,7 +1296,8 @@ def get_default_enabled(cls, name: str) -> Optional[bool]: (e.g. dynamic tools registered after init). Callers should fall back to the live ``ToolInfo.enabled`` in that case. """ - return cls._enabled_defaults.get(name) + with cls._refresh_lock: + return cls._enabled_defaults.get(name) @classmethod def _apply_tool_settings(cls) -> None: @@ -1351,8 +1383,9 @@ def _is_native_source(source: str) -> bool: except ValueError: return True # Under /.flocks/plugins/ or elsewhere → project-level → native - def _consume_tools(items: list, source: str) -> None: + def _consume_tools(items: list, source: str) -> Optional[List[str]]: is_native = _is_native_source(source) + errors: List[str] = [] for spec in items: # YAML factory produces Tool instances directly if isinstance(spec, Tool): @@ -1380,6 +1413,7 @@ def _consume_tools(items: list, source: str) -> None: if not isinstance(spec, dict): log.warn("plugin.tool.invalid_spec", {"source": source}) + errors.append("tool definition must be a Tool or mapping") continue name = spec.get("name") handler = spec.get("handler") @@ -1388,6 +1422,7 @@ def _consume_tools(items: list, source: str) -> None: "source": source, "spec_keys": list(spec.keys()), }) + errors.append("tool definition requires name and handler") continue existing = cls._tools.get(name) if existing is not None: @@ -1411,8 +1446,19 @@ def _consume_tools(items: list, source: str) -> None: log.warn("plugin.tool.handler_not_found", { "source": source, "handler": spec.get("handler"), }) + errors.append( + f"tool {name}: handler {spec.get('handler')!r} not found" + ) continue + if not callable(handler): + log.warn("plugin.tool.handler_not_callable", { + "source": source, + "name": name, + }) + errors.append(f"tool {name}: handler must be callable") + continue + params = [ ToolParameter(**p) if isinstance(p, dict) else p for p in spec.get("parameters", []) @@ -1429,6 +1475,7 @@ def _consume_tools(items: list, source: str) -> None: native=is_native, ) cls.register(Tool(info=info, handler=handler)) + return errors def _dedup_key(item: Any) -> str: if isinstance(item, Tool): @@ -1572,18 +1619,68 @@ def refresh_plugin_tools(cls) -> List[str]: correctly removed from the registry. """ cls._ensure_initialized() - cls._unregister_plugin_tools() - cls._load_plugin_tools() - cls._bump_revision("plugin_refresh") - return cls.all_tool_ids() + with cls._refresh_lock: + tools_before = cls._tools.copy() + defaults_before = cls._enabled_defaults.copy() + plugin_names_before = list(cls._plugin_tool_names) + load_errors: List[str] = [] + + try: + cls._unregister_plugin_tools() + cls._load_plugin_tools(load_errors) + except Exception: + cls._tools.clear() + cls._tools.update(tools_before) + cls._enabled_defaults.clear() + cls._enabled_defaults.update(defaults_before) + cls._plugin_tool_names = plugin_names_before + raise + if load_errors: + cls._tools.clear() + cls._tools.update(tools_before) + cls._enabled_defaults.clear() + cls._enabled_defaults.update(defaults_before) + cls._plugin_tool_names = plugin_names_before + raise ToolRefreshError("plugin", load_errors) + + cls._bump_revision("plugin_refresh") + return cls.all_tool_ids() @classmethod def refresh_dynamic_tools(cls) -> List[str]: """Reload dynamically generated tools and return tool ids.""" cls._ensure_initialized() - cls._register_dynamic_tools() - cls._bump_revision("dynamic_refresh") - return cls.all_tool_ids() + with cls._refresh_lock: + tools_before = cls._tools.copy() + defaults_before = cls._enabled_defaults.copy() + dynamic_modules_before = cls._dynamic_modules.copy() + dynamic_tools_before = { + name: list(tool_names) + for name, tool_names in cls._dynamic_tools_by_module.items() + } + load_errors: List[str] = [] + + try: + cls._register_dynamic_tools(load_errors) + except Exception: + cls._tools.clear() + cls._tools.update(tools_before) + cls._enabled_defaults.clear() + cls._enabled_defaults.update(defaults_before) + cls._dynamic_modules = dynamic_modules_before + cls._dynamic_tools_by_module = dynamic_tools_before + raise + if load_errors: + cls._tools.clear() + cls._tools.update(tools_before) + cls._enabled_defaults.clear() + cls._enabled_defaults.update(defaults_before) + cls._dynamic_modules = dynamic_modules_before + cls._dynamic_tools_by_module = dynamic_tools_before + raise ToolRefreshError("dynamic", load_errors) + + cls._bump_revision("dynamic_refresh") + return cls.all_tool_ids() @classmethod def _reset_failure_state(cls, tool_name: str) -> None: @@ -1717,7 +1814,7 @@ def _unregister_dynamic_tools(cls, module_name: str) -> None: }) @classmethod - def _register_dynamic_tools(cls) -> None: + def _register_dynamic_tools(cls, errors: Optional[List[str]] = None) -> None: """Register dynamically generated tools by importing modules.""" modules = cls._discover_dynamic_modules() @@ -1743,6 +1840,8 @@ def _register_dynamic_tools(cls) -> None: "module": module_name, "error": str(e), }) + if errors is not None: + errors.append(f"{path}: {e}") continue after = set(cls._tools.keys()) diff --git a/tests/mcp/test_mcp_server.py b/tests/mcp/test_mcp_server.py index 1ea0264ea..b89a6b26d 100644 --- a/tests/mcp/test_mcp_server.py +++ b/tests/mcp/test_mcp_server.py @@ -43,6 +43,19 @@ async def list_resources(self) -> list: return [] +def test_status_snapshot_is_read_only_and_does_not_initialize(): + manager = McpServerManager() + status = SimpleNamespace(status=McpStatus.DISCONNECTED) + manager._status["demo"] = status + + snapshot, initialized = manager.status_snapshot() + + assert initialized is False + assert snapshot == {"demo": status} + snapshot.clear() + assert manager._status == {"demo": status} + + @pytest.mark.asyncio async def test_connect_and_register_accepts_legacy_env_alias(monkeypatch: pytest.MonkeyPatch): captured: dict[str, object] = {} diff --git a/tests/plugin/test_plugin.py b/tests/plugin/test_plugin.py index 1e75807f4..95758cfd7 100644 --- a/tests/plugin/test_plugin.py +++ b/tests/plugin/test_plugin.py @@ -259,6 +259,96 @@ def test_load_extension_scans_only_requested_extension(self, tmp_path: Path): ] assert tool_items == [] + def test_load_extension_reports_bad_sources_without_dropping_valid_ones(self, tmp_path: Path): + tools_dir = tmp_path / "tools" + _write_plugin(tools_dir, "good.py", 'TOOLS = [{"name": "working"}]\n') + _write_plugin(tools_dir, "broken.py", "TOOLS = [\n") + collected = [] + + PluginLoader._plugin_root = tmp_path + PluginLoader.register_extension_point(ExtensionPoint( + attr_name="TOOLS", + subdir="tools", + consumer=lambda items, _source: collected.extend(items), + item_type=dict, + dedup_key=lambda item: item["name"], + )) + + errors = PluginLoader.load_extension("TOOLS", project_dir=tmp_path) + + assert [item["name"] for item in collected] == ["working"] + assert len(errors) == 1 + assert "broken.py" in errors[0] + + def test_load_extension_reports_filtered_invalid_entries(self, tmp_path: Path): + tools_dir = tmp_path / "tools" + _write_plugin( + tools_dir, + "mixed.py", + 'TOOLS = [{"name": "working"}, "invalid"]\n', + ) + collected = [] + + PluginLoader._plugin_root = tmp_path + PluginLoader.register_extension_point(ExtensionPoint( + attr_name="TOOLS", + subdir="tools", + consumer=lambda items, _source: collected.extend(items), + item_type=dict, + dedup_key=lambda item: item["name"], + )) + + errors = PluginLoader.load_extension("TOOLS", project_dir=tmp_path) + + assert [item["name"] for item in collected] == ["working"] + assert len(errors) == 1 + assert "1 invalid TOOLS entries" in errors[0] + + def test_load_extension_propagates_consumer_errors(self, tmp_path: Path): + tools_dir = tmp_path / "tools" + _write_plugin(tools_dir, "invalid.py", 'TOOLS = [{"name": "bad"}]\n') + + PluginLoader._plugin_root = tmp_path + PluginLoader.register_extension_point(ExtensionPoint( + attr_name="TOOLS", + subdir="tools", + consumer=lambda _items, _source: ["tool definition rejected"], + item_type=dict, + )) + + errors = PluginLoader.load_extension("TOOLS", project_dir=tmp_path) + + assert len(errors) == 1 + assert "invalid.py: tool definition rejected" in errors[0] + + def test_load_extension_reports_entry_point_scan_failure( + self, + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + ): + PluginLoader._plugin_root = tmp_path + PluginLoader.register_extension_point(ExtensionPoint( + attr_name="TOOLS", + subdir="tools", + consumer=lambda _items, _source: None, + )) + + def _failed_scan(): + raise RuntimeError("metadata unavailable") + + monkeypatch.setattr( + "flocks.plugin.loader.importlib.metadata.entry_points", + _failed_scan, + ) + + errors = PluginLoader.load_extension( + "TOOLS", + project_dir=tmp_path, + load_entry_points=True, + ) + + assert errors == ["entry point scan: RuntimeError: metadata unavailable"] + def test_load_extension_can_load_legacy_entry_points(self, tmp_path: Path, monkeypatch): """Scoped loads opt in to the legacy flocks.plugins entry-point group.""" loaded = [] diff --git a/tests/server/routes/test_monitoring_routes.py b/tests/server/routes/test_monitoring_routes.py new file mode 100644 index 000000000..3886c9cf4 --- /dev/null +++ b/tests/server/routes/test_monitoring_routes.py @@ -0,0 +1,109 @@ +from __future__ import annotations + +from types import SimpleNamespace + +import pytest + + +pytestmark = pytest.mark.asyncio + + +@pytest.mark.parametrize("mcp_status", ["disconnected", "timeout", "mystery-state"]) +async def test_status_degrades_for_non_operational_mcp_states( + monkeypatch: pytest.MonkeyPatch, + mcp_status: str, +): + from flocks.server.routes import monitoring as monitoring_routes + + monkeypatch.setattr( + monitoring_routes.MCP, + "status_snapshot", + lambda: ({"demo-mcp": SimpleNamespace(status=mcp_status)}, True), + ) + + response = await monitoring_routes.get_status() + + assert response.status == "degraded" + assert response.mcpServers == {"demo-mcp": mcp_status} + + +async def test_status_degrades_when_mcp_status_check_fails( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.server.routes import monitoring as monitoring_routes + + def _status_snapshot(): + raise RuntimeError("MCP status failed") + + monkeypatch.setattr(monitoring_routes.MCP, "status_snapshot", _status_snapshot) + + response = await monitoring_routes.get_status() + + assert response.status == "degraded" + assert response.mcpServers == {} + + +async def test_status_snapshot_does_not_trigger_mcp_initialization( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.server.routes import monitoring as monitoring_routes + + async def _unexpected_status(): + raise AssertionError("monitoring must not initialize MCP while reading status") + + monkeypatch.setattr(monitoring_routes.MCP, "status", _unexpected_status) + monkeypatch.setattr( + monitoring_routes.MCP, + "status_snapshot", + lambda: ({}, False), + ) + + response = await monitoring_routes.get_status() + + assert response.status == "degraded" + assert response.mcpServers == {} + + +@pytest.mark.parametrize("mcp_status", ["connected", "disabled"]) +async def test_status_accepts_operational_or_intentionally_disabled_mcp_states( + monkeypatch: pytest.MonkeyPatch, + mcp_status: str, +): + from flocks.server.routes import monitoring as monitoring_routes + + monkeypatch.setattr( + monitoring_routes.MCP, + "status_snapshot", + lambda: ({"demo-mcp": SimpleNamespace(status=mcp_status)}, True), + ) + + response = await monitoring_routes.get_status() + + assert response.status == "healthy" + + +async def test_metrics_do_not_label_tool_parse_failures_as_system_errors( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.server.routes import monitoring as monitoring_routes + + monitor = SimpleNamespace( + get_metrics=lambda: { + "global": {"total_calls": 12, "failed_parses": 3}, + } + ) + monkeypatch.setattr(monitoring_routes, "get_monitor", lambda: monitor) + monkeypatch.setattr( + monitoring_routes, + "_sample_tool_rates", + lambda total, failed: (12.0, 3.0), + ) + + response = await monitoring_routes.get_metrics() + payload = response.model_dump() + + assert response.toolCallRate == 12.0 + assert response.toolParseFailureRate == 3.0 + assert response.errorRate is None + assert payload["errorRate"] is None + assert payload["toolParseFailureRate"] == 3.0 diff --git a/tests/server/routes/test_provider_route_responsiveness.py b/tests/server/routes/test_provider_route_responsiveness.py new file mode 100644 index 000000000..ba6967e37 --- /dev/null +++ b/tests/server/routes/test_provider_route_responsiveness.py @@ -0,0 +1,213 @@ +from __future__ import annotations + +import asyncio +import threading +import time + +import pytest + + +pytestmark = pytest.mark.asyncio + + +@pytest.fixture(autouse=True) +def _reset_provider_route_async_state(monkeypatch: pytest.MonkeyPatch): + from flocks.server.routes import provider as provider_routes + + monkeypatch.setattr(provider_routes, "_provider_initialization_lock", asyncio.Lock()) + monkeypatch.setattr(provider_routes, "_provider_initialization_task", None) + monkeypatch.setattr(provider_routes, "_dynamic_provider_load_tasks", set()) + + +async def test_initialized_provider_fast_path_does_not_schedule_worker_thread( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.server.routes import provider as provider_routes + + async def _unexpected_to_thread(*args, **kwargs): + raise AssertionError("initialized provider should stay on the fast path") + + monkeypatch.setattr(provider_routes.Provider, "_initialized", True) + monkeypatch.setattr(provider_routes.asyncio, "to_thread", _unexpected_to_thread) + + await provider_routes._ensure_provider_initialized() + + +async def test_concurrent_provider_initialization_uses_one_worker( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.server.routes import provider as provider_routes + + calls = 0 + + def _slow_initialize() -> None: + nonlocal calls + calls += 1 + time.sleep(0.05) + provider_routes.Provider._initialized = True + + monkeypatch.setattr(provider_routes.Provider, "_initialized", False) + monkeypatch.setattr(provider_routes.Provider, "_ensure_initialized", _slow_initialize) + + await asyncio.gather(*[ + provider_routes._ensure_provider_initialized() + for _ in range(10) + ]) + + assert calls == 1 + + +async def test_dynamic_provider_loading_does_not_block_event_loop( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.server.routes import provider as provider_routes + + monkeypatch.setattr( + provider_routes.Provider, + "_load_dynamic_providers", + lambda: time.sleep(0.05), + ) + + load_task = asyncio.create_task(provider_routes._load_dynamic_providers()) + heartbeat_ticks = 0 + while not load_task.done(): + await asyncio.sleep(0.005) + heartbeat_ticks += 1 + await load_task + + assert heartbeat_ticks >= 3 + + +async def test_concurrent_dynamic_load_triggers_are_serialized_but_not_merged( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.server.routes import provider as provider_routes + + first_started = threading.Event() + release_first = threading.Event() + second_started = threading.Event() + calls = 0 + config_version = "provider-a" + scanned_versions: list[str] = [] + active_workers = 0 + max_active_workers = 0 + + def _blocking_load() -> None: + nonlocal calls, active_workers, max_active_workers + calls += 1 + scanned_versions.append(config_version) + active_workers += 1 + max_active_workers = max(max_active_workers, active_workers) + try: + if calls == 1: + first_started.set() + release_first.wait(timeout=1) + else: + second_started.set() + finally: + active_workers -= 1 + + monkeypatch.setattr( + provider_routes.Provider, + "_load_dynamic_providers", + _blocking_load, + ) + + first = asyncio.create_task(provider_routes._load_dynamic_providers()) + while not first_started.is_set(): + await asyncio.sleep(0.001) + + config_version = "provider-b" + second = asyncio.create_task(provider_routes._load_dynamic_providers()) + try: + await asyncio.sleep(0.01) + assert calls == 1 + finally: + release_first.set() + + await asyncio.wait_for(asyncio.gather(first, second), timeout=0.2) + assert second_started.is_set() + assert calls == 2 + assert scanned_versions == ["provider-a", "provider-b"] + assert max_active_workers == 1 + + +async def test_cancelled_dynamic_load_waiter_keeps_its_scan_and_serial_order( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.server.routes import provider as provider_routes + + first_started = threading.Event() + release_first = threading.Event() + second_started = threading.Event() + calls = 0 + active_workers = 0 + max_active_workers = 0 + + def _blocking_load() -> None: + nonlocal calls, active_workers, max_active_workers + calls += 1 + active_workers += 1 + max_active_workers = max(max_active_workers, active_workers) + try: + if calls == 1: + first_started.set() + release_first.wait(timeout=1) + else: + second_started.set() + finally: + active_workers -= 1 + + monkeypatch.setattr( + provider_routes.Provider, + "_load_dynamic_providers", + _blocking_load, + ) + + cancelled_waiter = asyncio.create_task(provider_routes._load_dynamic_providers()) + while not first_started.is_set(): + await asyncio.sleep(0.001) + + cancelled_waiter.cancel() + with pytest.raises(asyncio.CancelledError): + await cancelled_waiter + + second = asyncio.create_task(provider_routes._load_dynamic_providers()) + try: + await asyncio.sleep(0.01) + assert calls == 1 + finally: + release_first.set() + + await asyncio.wait_for(second, timeout=0.2) + assert second_started.is_set() + assert calls == 2 + assert max_active_workers == 1 + + +async def test_list_providers_initialization_does_not_block_event_loop( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.server.routes import provider as provider_routes + + def _slow_initialize() -> None: + time.sleep(0.1) + + async def _config(): + raise RuntimeError("no config in focused route test") + + monkeypatch.setattr(provider_routes.Provider, "_initialized", False) + monkeypatch.setattr(provider_routes.Provider, "_ensure_initialized", _slow_initialize) + monkeypatch.setattr(provider_routes.Config, "get", _config) + monkeypatch.setattr(provider_routes.ConfigWriter, "list_provider_ids", lambda: []) + + route_task = asyncio.create_task(provider_routes.list_providers()) + heartbeat_ticks = 0 + while not route_task.done(): + await asyncio.sleep(0.005) + heartbeat_ticks += 1 + + response = await route_task + + assert heartbeat_ticks >= 3 + assert response.all == [] diff --git a/tests/server/routes/test_update_routes.py b/tests/server/routes/test_update_routes.py index e4334c4c5..060da1462 100644 --- a/tests/server/routes/test_update_routes.py +++ b/tests/server/routes/test_update_routes.py @@ -183,6 +183,33 @@ async def _fake_check_update(**kwargs): assert [item.current_version for item in results] == ["v1"] * 5 +async def test_check_version_deduplicates_concurrent_forced_requests( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.server.routes import update as update_routes + from flocks.updater.models import VersionInfo + + calls = 0 + + async def _fake_check_update(**kwargs): + nonlocal calls + calls += 1 + await asyncio.sleep(0.01) + return VersionInfo(current_version=f"v{calls}") + + monkeypatch.setattr(update_routes, "check_update", _fake_check_update) + + results = await asyncio.gather(*[ + update_routes.check_version( + _request(), locale="zh-CN", edition="flocks", force=True + ) + for _ in range(10) + ]) + + assert calls == 1 + assert [item.current_version for item in results] == ["v1"] * 10 + + async def test_check_version_keeps_inflight_task_after_cancelled_request(monkeypatch: pytest.MonkeyPatch): from flocks.server.routes import update as update_routes from flocks.updater.models import VersionInfo @@ -225,7 +252,7 @@ async def _fake_check_update(**kwargs): await asyncio.wait_for(second, timeout=1) -async def test_forced_check_cannot_be_overwritten_by_older_inflight_request( +async def test_forced_check_reuses_older_same_key_inflight_request( monkeypatch: pytest.MonkeyPatch, ): from flocks.server.routes import update as update_routes @@ -251,19 +278,23 @@ async def _fake_check_update(**kwargs): ) await older_started.wait() - forced = await update_routes.check_version( - _request(), locale="zh-CN", edition="flocks", force=True + forced = asyncio.create_task( + update_routes.check_version( + _request(), locale="zh-CN", edition="flocks", force=True + ) ) - assert forced.current_version == "v2" + await asyncio.sleep(0) + assert calls == 1 release_older.set() assert (await older).current_version == "v1" + assert (await forced).current_version == "v1" cached = await update_routes.check_version( _request(), locale="zh-CN", edition="flocks" ) - assert cached.current_version == "v2" - assert calls == 2 + assert cached.current_version == "v1" + assert calls == 1 @pytest.mark.skipif( diff --git a/tests/tool/test_tool_refresh_outcomes.py b/tests/tool/test_tool_refresh_outcomes.py new file mode 100644 index 000000000..12f826f47 --- /dev/null +++ b/tests/tool/test_tool_refresh_outcomes.py @@ -0,0 +1,382 @@ +from __future__ import annotations + +from collections.abc import Iterator +from contextlib import contextmanager +import threading + +import pytest + +from flocks.tool.registry import ( + Tool, + ToolCategory, + ToolInfo, + ToolRefreshError, + ToolRegistry, +) + + +def _tool(name: str, source: str) -> Tool: + return Tool( + info=ToolInfo( + name=name, + description=name, + category=ToolCategory.CUSTOM, + source=source, + ), + handler=lambda _ctx, **_kwargs: None, + ) + + +@contextmanager +def _isolated_registry() -> Iterator[None]: + state = { + "initialized": ToolRegistry._initialized, + "tools": ToolRegistry._tools, + "defaults": ToolRegistry._enabled_defaults, + "plugin_names": ToolRegistry._plugin_tool_names, + "dynamic_modules": ToolRegistry._dynamic_modules, + "dynamic_tools": ToolRegistry._dynamic_tools_by_module, + } + try: + ToolRegistry._initialized = True + ToolRegistry._tools = {} + ToolRegistry._enabled_defaults = {} + ToolRegistry._plugin_tool_names = [] + ToolRegistry._dynamic_modules = {} + ToolRegistry._dynamic_tools_by_module = {} + yield + finally: + ToolRegistry._initialized = state["initialized"] + ToolRegistry._tools = state["tools"] + ToolRegistry._enabled_defaults = state["defaults"] + ToolRegistry._plugin_tool_names = state["plugin_names"] + ToolRegistry._dynamic_modules = state["dynamic_modules"] + ToolRegistry._dynamic_tools_by_module = state["dynamic_tools"] + + +def test_plugin_refresh_restores_last_known_good_on_load_error( + monkeypatch: pytest.MonkeyPatch, +): + with _isolated_registry(): + previous = _tool("previous_plugin", "plugin_py") + ToolRegistry._tools[previous.info.name] = previous + ToolRegistry._plugin_tool_names = [previous.info.name] + + def _failed_load(cls, errors): + replacement = _tool("partial_replacement", "plugin_py") + cls._tools[replacement.info.name] = replacement + cls._plugin_tool_names = [replacement.info.name] + errors.append("broken plugin") + + monkeypatch.setattr(ToolRegistry, "_load_plugin_tools", classmethod(_failed_load)) + + with pytest.raises(ToolRefreshError, match="broken plugin"): + ToolRegistry.refresh_plugin_tools() + + assert ToolRegistry._tools == {previous.info.name: previous} + assert ToolRegistry._plugin_tool_names == [previous.info.name] + + +def test_dynamic_refresh_restores_last_known_good_on_load_error( + monkeypatch: pytest.MonkeyPatch, +): + with _isolated_registry(): + previous = _tool("previous_dynamic", "custom") + ToolRegistry._tools[previous.info.name] = previous + ToolRegistry._dynamic_modules = {"demo": "/old/demo.py"} + ToolRegistry._dynamic_tools_by_module = {"demo": [previous.info.name]} + + def _failed_load(cls, errors): + cls._tools.pop(previous.info.name) + replacement = _tool("partial_dynamic", "custom") + cls._tools[replacement.info.name] = replacement + cls._dynamic_modules = {"demo": "/new/demo.py"} + cls._dynamic_tools_by_module = {"demo": [replacement.info.name]} + errors.append("broken dynamic module") + + monkeypatch.setattr(ToolRegistry, "_register_dynamic_tools", classmethod(_failed_load)) + + with pytest.raises(ToolRefreshError, match="broken dynamic module"): + ToolRegistry.refresh_dynamic_tools() + + assert ToolRegistry._tools == {previous.info.name: previous} + assert ToolRegistry._dynamic_modules == {"demo": "/old/demo.py"} + assert ToolRegistry._dynamic_tools_by_module == {"demo": [previous.info.name]} + + +def test_refresh_transactions_are_serialized_across_stages( + monkeypatch: pytest.MonkeyPatch, +): + plugin_load_entered = threading.Event() + allow_plugin_failure = threading.Event() + dynamic_load_entered = threading.Event() + allow_dynamic_finish = threading.Event() + plugin_failures: list[Exception] = [] + dynamic_failures: list[Exception] = [] + + with _isolated_registry(): + base = _tool("base", "builtin") + ToolRegistry._tools[base.info.name] = base + + def _failed_plugin_load(cls, errors): + plugin_load_entered.set() + assert allow_plugin_failure.wait(timeout=2) + errors.append("broken plugin") + + def _successful_dynamic_load(cls, _errors): + dynamic_load_entered.set() + dynamic = _tool("new_dynamic", "custom") + cls._tools[dynamic.info.name] = dynamic + cls._dynamic_modules = {"demo": "/new/demo.py"} + cls._dynamic_tools_by_module = {"demo": [dynamic.info.name]} + assert allow_dynamic_finish.wait(timeout=2) + + monkeypatch.setattr( + ToolRegistry, + "_load_plugin_tools", + classmethod(_failed_plugin_load), + ) + monkeypatch.setattr( + ToolRegistry, + "_register_dynamic_tools", + classmethod(_successful_dynamic_load), + ) + monkeypatch.setattr(ToolRegistry, "_bump_revision", lambda _reason: None) + + def _refresh_plugin(): + try: + ToolRegistry.refresh_plugin_tools() + except Exception as exc: + plugin_failures.append(exc) + + def _refresh_dynamic(): + try: + ToolRegistry.refresh_dynamic_tools() + except Exception as exc: + dynamic_failures.append(exc) + + plugin_thread = threading.Thread(target=_refresh_plugin) + dynamic_thread = threading.Thread(target=_refresh_dynamic) + plugin_thread.start() + assert plugin_load_entered.wait(timeout=1) + dynamic_thread.start() + + # If transactions are not serialized, make the dynamic refresh commit + # before the plugin rollback to deterministically reproduce lost state. + dynamic_entered_while_plugin_blocked = dynamic_load_entered.wait(timeout=0.2) + if dynamic_entered_while_plugin_blocked: + allow_dynamic_finish.set() + dynamic_thread.join(timeout=1) + allow_plugin_failure.set() + else: + allow_plugin_failure.set() + allow_dynamic_finish.set() + + plugin_thread.join(timeout=1) + dynamic_thread.join(timeout=1) + + assert not plugin_thread.is_alive() + assert not dynamic_thread.is_alive() + assert len(plugin_failures) == 1 + assert isinstance(plugin_failures[0], ToolRefreshError) + assert dynamic_failures == [] + assert set(ToolRegistry._tools) == {"base", "new_dynamic"} + + +def test_registry_readers_do_not_observe_an_in_progress_refresh( + monkeypatch: pytest.MonkeyPatch, +): + load_entered = threading.Event() + allow_failure = threading.Event() + reader_finished = threading.Event() + reader_results: list[Tool | None] = [] + + with _isolated_registry(): + previous = _tool("previous_plugin", "plugin_py") + ToolRegistry._tools[previous.info.name] = previous + ToolRegistry._plugin_tool_names = [previous.info.name] + + def _failed_plugin_load(cls, errors): + load_entered.set() + assert allow_failure.wait(timeout=2) + errors.append("broken plugin") + + monkeypatch.setattr( + ToolRegistry, + "_load_plugin_tools", + classmethod(_failed_plugin_load), + ) + + refresh_thread = threading.Thread( + target=lambda: pytest.raises( + ToolRefreshError, + ToolRegistry.refresh_plugin_tools, + ), + ) + + def _read_plugin() -> None: + reader_results.append(ToolRegistry.get(previous.info.name)) + reader_finished.set() + + reader_thread = threading.Thread(target=_read_plugin) + refresh_thread.start() + assert load_entered.wait(timeout=1) + reader_thread.start() + + assert not reader_finished.wait(timeout=0.1) + allow_failure.set() + refresh_thread.join(timeout=1) + reader_thread.join(timeout=1) + + assert not refresh_thread.is_alive() + assert not reader_thread.is_alive() + assert reader_results == [previous] + + +def test_tool_extension_consumer_reports_invalid_specs(): + from flocks.plugin import PluginLoader + + extension_points_before = PluginLoader._extension_points.copy() + try: + PluginLoader.clear_extension_points() + ToolRegistry._register_plugin_extension_point() + extension = PluginLoader._extension_points["TOOLS"] + + errors = extension.consumer( + [ + {"name": "missing_handler"}, + {"name": "not_callable", "handler": 42}, + ], + "invalid_tools.py", + ) + + assert errors == [ + "tool definition requires name and handler", + "tool not_callable: handler must be callable", + ] + finally: + PluginLoader.clear_extension_points() + PluginLoader._extension_points.update(extension_points_before) + + +@pytest.mark.asyncio +async def test_refresh_route_reports_partial_and_total_stage_failures( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.server.routes import tool as tool_routes + + monkeypatch.setattr(tool_routes.ToolRegistry, "init", lambda: None) + monkeypatch.setattr(tool_routes.ToolRegistry, "all_tool_ids", lambda: ["working"]) + monkeypatch.setattr( + tool_routes.ToolRegistry, + "refresh_dynamic_tools", + lambda: (_ for _ in ()).throw(ToolRefreshError("dynamic", ["bad dynamic"])), + ) + monkeypatch.setattr(tool_routes.ToolRegistry, "refresh_plugin_tools", lambda: ["working"]) + + partial = await tool_routes.refresh_tools(_admin=object()) + + assert partial.status == "partial" + assert partial.stages == {"dynamic": "error", "plugin": "success"} + assert partial.errors == ["dynamic: bad dynamic"] + + monkeypatch.setattr( + tool_routes.ToolRegistry, + "refresh_plugin_tools", + lambda: (_ for _ in ()).throw(ToolRefreshError("plugin", ["bad plugin"])), + ) + + failed = await tool_routes.refresh_tools(_admin=object()) + + assert failed.status == "error" + assert failed.stages == {"dynamic": "error", "plugin": "error"} + assert failed.errors == ["dynamic: bad dynamic", "plugin: bad plugin"] + + +@pytest.mark.asyncio +async def test_delete_tool_cleans_hub_record_when_registry_refresh_fails( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.hub import local as hub_local + from flocks.server.routes import tool as tool_routes + from flocks.tool import tool_loader + + removed_records: list[tuple[str, str]] = [] + monkeypatch.setattr(tool_routes.ToolRegistry, "init", lambda: None) + monkeypatch.setattr(tool_loader, "find_yaml_tool", lambda _name: object()) + monkeypatch.setattr(tool_loader, "delete_yaml_tool", lambda _name: True) + monkeypatch.setattr( + tool_routes.ToolRegistry, + "refresh_plugin_tools", + lambda: (_ for _ in ()).throw(ToolRefreshError("plugin", ["broken"])), + ) + monkeypatch.setattr( + hub_local, + "remove_installed_record", + lambda kind, name: removed_records.append((kind, name)), + ) + + response = await tool_routes.delete_tool("demo", _admin=object()) + + assert response["status"] == "partial" + assert response["errors"] == ["registry refresh: plugin: broken"] + assert removed_records == [("tool", "demo")] + + +@pytest.mark.asyncio +async def test_delete_tool_reports_hub_cleanup_failure_as_partial( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.hub import local as hub_local + from flocks.server.routes import tool as tool_routes + from flocks.tool import tool_loader + + monkeypatch.setattr(tool_routes.ToolRegistry, "init", lambda: None) + monkeypatch.setattr(tool_loader, "find_yaml_tool", lambda _name: object()) + monkeypatch.setattr(tool_loader, "delete_yaml_tool", lambda _name: True) + monkeypatch.setattr( + tool_routes.ToolRegistry, + "refresh_plugin_tools", + lambda: [], + ) + monkeypatch.setattr( + hub_local, + "remove_installed_record", + lambda _kind, _name: (_ for _ in ()).throw(RuntimeError("index locked")), + ) + + response = await tool_routes.delete_tool("demo", _admin=object()) + + assert response["status"] == "partial" + assert response["errors"] == ["Hub record cleanup: index locked"] + + +@pytest.mark.asyncio +async def test_delete_tool_reports_all_post_delete_cleanup_failures( + monkeypatch: pytest.MonkeyPatch, +): + from flocks.hub import local as hub_local + from flocks.server.routes import tool as tool_routes + from flocks.tool import tool_loader + + monkeypatch.setattr(tool_routes.ToolRegistry, "init", lambda: None) + monkeypatch.setattr(tool_loader, "find_yaml_tool", lambda _name: object()) + monkeypatch.setattr(tool_loader, "delete_yaml_tool", lambda _name: True) + monkeypatch.setattr( + tool_routes.ToolRegistry, + "refresh_plugin_tools", + lambda: (_ for _ in ()).throw(ToolRefreshError("plugin", ["broken"])), + ) + monkeypatch.setattr( + hub_local, + "remove_installed_record", + lambda _kind, _name: (_ for _ in ()).throw(RuntimeError("index locked")), + ) + + response = await tool_routes.delete_tool("demo", _admin=object()) + + assert response["status"] == "partial" + assert response["errors"] == [ + "registry refresh: plugin: broken", + "Hub record cleanup: index locked", + ] diff --git a/webui/src/api/monitoring.ts b/webui/src/api/monitoring.ts index ccf1d7682..564169cf4 100644 --- a/webui/src/api/monitoring.ts +++ b/webui/src/api/monitoring.ts @@ -14,6 +14,7 @@ export interface MetricsSnapshot { messageRate: number | null; toolCallRate: number | null; errorRate: number | null; + toolParseFailureRate: number | null; avgResponseTime: number | null; activeRequests: number | null; } diff --git a/webui/src/api/tool.ts b/webui/src/api/tool.ts index 1890a7204..36729ddb2 100644 --- a/webui/src/api/tool.ts +++ b/webui/src/api/tool.ts @@ -53,6 +53,20 @@ export interface ToolListPageResponse { facets: ToolListFacets; } +export interface ToolRefreshResponse { + status: 'success' | 'partial' | 'error'; + tool_count: number; + message: string; + stages: Record; + errors: string[]; +} + +export interface ToolDeleteResponse { + status: 'success' | 'partial'; + message: string; + errors?: string[]; +} + export const toolAPI = { list: (params?: { source?: ToolSource; category?: string }) => client.get('/api/tools', { params }), @@ -76,7 +90,7 @@ export const toolAPI = { client.get(`/api/tools/${name}`), refresh: () => - client.post('/api/tools/refresh'), + client.post('/api/tools/refresh'), test: (name: string, params: Record) => client.post(`/api/tools/${name}/test`, { params }), @@ -102,7 +116,7 @@ export const toolAPI = { client.post(`/api/tools/${name}/reset`), delete: (name: string) => - client.delete<{ status: string; message: string }>(`/api/tools/${name}`), + client.delete(`/api/tools/${name}`), }; export async function listAllToolPages(params: ToolListPageParams): Promise { diff --git a/webui/src/hooks/useSessions.test.ts b/webui/src/hooks/useSessions.test.ts index b6b381ee5..2fe812538 100644 --- a/webui/src/hooks/useSessions.test.ts +++ b/webui/src/hooks/useSessions.test.ts @@ -142,6 +142,8 @@ describe('applyMessagePartUpdate', () => { describe('updateMessagePart scheduling', () => { afterEach(() => { vi.clearAllMocks(); + vi.mocked(client.get).mockReset(); + vi.mocked(client.get).mockResolvedValue({ data: [] }); vi.useRealTimers(); }); @@ -316,6 +318,275 @@ describe('updateMessagePart scheduling', () => { expect(result.current.messages).toHaveLength(0); }); + it('ignores a stale first-page response after switching sessions', async () => { + let resolveSessionA: (value: unknown) => void = () => {}; + let resolveSessionB: (value: unknown) => void = () => {}; + vi.mocked(client.get).mockImplementation((url: string) => new Promise((resolve) => { + if (url.includes('sess-a')) resolveSessionA = resolve; + if (url.includes('sess-b')) resolveSessionB = resolve; + }) as any); + + const { result, rerender } = renderHook( + ({ id }: { id: string }) => useSessionMessages(id), + { initialProps: { id: 'sess-a' } }, + ); + await act(async () => {}); + + rerender({ id: 'sess-b' }); + await act(async () => {}); + + await act(async () => { + resolveSessionB({ + data: [{ + info: { + id: 'msg-b', + sessionID: 'sess-b', + role: 'assistant', + time: { created: 200 }, + }, + parts: [], + }], + }); + }); + expect(result.current.messages.map((message) => message.id)).toEqual(['msg-b']); + + await act(async () => { + resolveSessionA({ + data: [{ + info: { + id: 'msg-a', + sessionID: 'sess-a', + role: 'assistant', + time: { created: 100 }, + }, + parts: [], + }], + }); + }); + + expect(result.current.messages.map((message) => message.id)).toEqual(['msg-b']); + expect(result.current.loading).toBe(false); + }); + + it('ignores an older-page response after switching sessions', async () => { + let resolveOlderSessionA: (value: unknown) => void = () => {}; + vi.mocked(client.get).mockImplementation((url: string, config?: any) => { + if (url.includes('sess-a') && config?.params?.before) { + return new Promise((resolve) => { + resolveOlderSessionA = resolve; + }) as any; + } + if (url.includes('sess-a')) { + return Promise.resolve({ + data: { + items: [{ + info: { + id: 'msg-a-new', + sessionID: 'sess-a', + role: 'assistant', + time: { created: 200 }, + }, + parts: [], + }], + hasMore: true, + nextBefore: 'msg-a-new', + }, + }) as any; + } + return Promise.resolve({ + data: [{ + info: { + id: 'msg-b', + sessionID: 'sess-b', + role: 'assistant', + time: { created: 300 }, + }, + parts: [], + }], + }) as any; + }); + + const { result, rerender } = renderHook( + ({ id }: { id: string }) => useSessionMessages(id), + { initialProps: { id: 'sess-a' } }, + ); + await act(async () => {}); + + let olderRequest: Promise = Promise.resolve(); + act(() => { + olderRequest = result.current.loadOlder(); + }); + + rerender({ id: 'sess-b' }); + await act(async () => {}); + expect(result.current.messages.map((message) => message.id)).toEqual(['msg-b']); + + await act(async () => { + resolveOlderSessionA({ + data: { + items: [{ + info: { + id: 'msg-a-old', + sessionID: 'sess-a', + role: 'user', + time: { created: 100 }, + }, + parts: [], + }], + hasMore: false, + nextBefore: null, + }, + }); + await olderRequest; + }); + + expect(result.current.messages.map((message) => message.id)).toEqual(['msg-b']); + expect(result.current.loadingOlder).toBe(false); + }); + + it('clears older-page loading when a first-page refetch invalidates it', async () => { + let resolveOlderPage: (value: unknown) => void = () => {}; + let firstPageRequestCount = 0; + vi.mocked(client.get).mockImplementation((_url: string, config?: any) => { + if (config?.params?.before) { + return new Promise((resolve) => { + resolveOlderPage = resolve; + }) as any; + } + + firstPageRequestCount += 1; + const messageId = firstPageRequestCount === 1 ? 'msg-current' : 'msg-refreshed'; + return Promise.resolve({ + data: { + items: [{ + info: { + id: messageId, + sessionID: 'sess-1', + role: 'assistant', + time: { created: 200 + firstPageRequestCount }, + }, + parts: [], + }], + hasMore: true, + nextBefore: messageId, + }, + }) as any; + }); + + const { result } = renderHook(() => useSessionMessages('sess-1')); + await act(async () => {}); + + let olderRequest: Promise = Promise.resolve(); + act(() => { + olderRequest = result.current.loadOlder(); + }); + expect(result.current.loadingOlder).toBe(true); + + await act(async () => { + await result.current.refetch(); + }); + expect(result.current.loadingOlder).toBe(false); + expect(result.current.messages.map((message) => message.id)).toEqual([ + 'msg-current', + 'msg-refreshed', + ]); + + await act(async () => { + resolveOlderPage({ + data: { + items: [{ + info: { + id: 'msg-stale-older', + sessionID: 'sess-1', + role: 'user', + time: { created: 100 }, + }, + parts: [], + }], + hasMore: false, + nextBefore: null, + }, + }); + await olderRequest; + }); + + expect(result.current.loadingOlder).toBe(false); + expect(result.current.messages.map((message) => message.id)).toEqual([ + 'msg-current', + 'msg-refreshed', + ]); + }); + + it('does not start an older-page request while a first-page refetch is pending', async () => { + let resolveRefetch: (value: unknown) => void = () => {}; + let firstPageRequests = 0; + let olderPageRequests = 0; + vi.mocked(client.get).mockImplementation((_url: string, config?: any) => { + if (config?.params?.before) { + olderPageRequests += 1; + return Promise.resolve({ data: { items: [], hasMore: false, nextBefore: null } }) as any; + } + + firstPageRequests += 1; + if (firstPageRequests === 1) { + return Promise.resolve({ + data: { + items: [{ + info: { + id: 'msg-current', + sessionID: 'sess-1', + role: 'assistant', + time: { created: 200 }, + }, + parts: [], + }], + hasMore: true, + nextBefore: 'old-cursor', + }, + }) as any; + } + + return new Promise((resolve) => { + resolveRefetch = resolve; + }) as any; + }); + + const { result } = renderHook(() => useSessionMessages('sess-1')); + await act(async () => {}); + + let refetchRequest: Promise = Promise.resolve(); + act(() => { + refetchRequest = result.current.refetch(); + }); + await act(async () => { + await result.current.loadOlder(); + }); + + expect(olderPageRequests).toBe(0); + + await act(async () => { + resolveRefetch({ + data: { + items: [{ + info: { + id: 'msg-refreshed', + sessionID: 'sess-1', + role: 'assistant', + time: { created: 300 }, + }, + parts: [], + }], + hasMore: true, + nextBefore: 'new-cursor', + }, + }); + await refetchRequest; + }); + + expect(result.current.loading).toBe(false); + expect(olderPageRequests).toBe(0); + }); + it('replaceMessageText updates the targeted text part by partId', async () => { const { result } = renderHook(() => useSessionMessages('sess-1')); await act(async () => {}); @@ -585,6 +856,51 @@ describe('updateMessagePart scheduling', () => { }); }); + it('starts only one older-page request when loadOlder is called twice before rerender', async () => { + let resolveOlderPage: (value: unknown) => void = () => {}; + let olderPageRequests = 0; + vi.mocked(client.get).mockImplementation((_url: string, config?: any) => { + if (config?.params?.before) { + olderPageRequests += 1; + return new Promise((resolve) => { + resolveOlderPage = resolve; + }) as any; + } + + return Promise.resolve({ + data: { + items: [], + hasMore: true, + nextBefore: 'older-cursor', + }, + }) as any; + }); + + const { result } = renderHook(() => useSessionMessages('sess-1')); + await act(async () => {}); + + let firstRequest: Promise = Promise.resolve(); + let secondRequest: Promise = Promise.resolve(); + act(() => { + firstRequest = result.current.loadOlder(); + secondRequest = result.current.loadOlder(); + }); + + expect(olderPageRequests).toBe(1); + + await act(async () => { + resolveOlderPage({ + data: { + items: [], + hasMore: false, + nextBefore: null, + }, + }); + await Promise.all([firstRequest, secondRequest]); + }); + expect(result.current.loadingOlder).toBe(false); + }); + }); describe('useSessions list loading', () => { diff --git a/webui/src/hooks/useSessions.ts b/webui/src/hooks/useSessions.ts index 6064b1771..03586f061 100644 --- a/webui/src/hooks/useSessions.ts +++ b/webui/src/hooks/useSessions.ts @@ -350,9 +350,26 @@ export function useSessionMessages(sessionId?: string) { const [hasMore, setHasMore] = useState(false); const [nextBefore, setNextBefore] = useState(null); const [error, setError] = useState(null); + const activeSessionIdRef = useRef(sessionId); + const firstPageRequestIdRef = useRef(0); + const firstPageInFlightRequestIdRef = useRef(null); + const olderPageRequestIdRef = useRef(0); + const olderPageInFlightRequestIdRef = useRef(null); const fetchMessages = useCallback(async () => { - if (!sessionId) return; + if (!sessionId || activeSessionIdRef.current !== sessionId) return; + + const requestSessionId = sessionId; + const requestId = ++firstPageRequestIdRef.current; + firstPageInFlightRequestIdRef.current = requestId; + // A fresh first page invalidates pagination based on an older snapshot. + olderPageRequestIdRef.current += 1; + olderPageInFlightRequestIdRef.current = null; + setLoadingOlder(false); + const isCurrentRequest = () => ( + activeSessionIdRef.current === requestSessionId + && firstPageRequestIdRef.current === requestId + ); try { setLoading(true); @@ -363,19 +380,44 @@ export function useSessionMessages(sessionId?: string) { params: { page: true, limit: MESSAGE_PAGE_SIZE, include_archived: true }, }); markMeasure('session:messages:first-page', startMark); + if (!isCurrentRequest()) return; const { messages: messagesData, hasMore, nextBefore } = transformMessageResponse(response.data); setMessages(prev => mergeLatestFetchedMessages(prev, messagesData)); setHasMore(hasMore); setNextBefore(nextBefore); } catch (err: any) { - setError(err.message || 'Failed to fetch messages'); + if (isCurrentRequest()) { + setError(err.message || 'Failed to fetch messages'); + } } finally { - setLoading(false); + if (firstPageInFlightRequestIdRef.current === requestId) { + firstPageInFlightRequestIdRef.current = null; + } + if (isCurrentRequest()) { + setLoading(false); + } } }, [sessionId]); const loadOlder = useCallback(async () => { - if (!sessionId || !hasMore || !nextBefore || loadingOlder) return; + if ( + !sessionId + || activeSessionIdRef.current !== sessionId + || firstPageInFlightRequestIdRef.current !== null + || olderPageInFlightRequestIdRef.current !== null + || !hasMore + || !nextBefore + ) return; + + const requestSessionId = sessionId; + const requestId = ++olderPageRequestIdRef.current; + olderPageInFlightRequestIdRef.current = requestId; + const firstPageRequestId = firstPageRequestIdRef.current; + const isCurrentRequest = () => ( + activeSessionIdRef.current === requestSessionId + && olderPageRequestIdRef.current === requestId + && firstPageRequestIdRef.current === firstPageRequestId + ); try { setLoadingOlder(true); @@ -391,24 +433,35 @@ export function useSessionMessages(sessionId?: string) { }, }); markMeasure('session:messages:older-page', startMark); + if (!isCurrentRequest()) return; const page = transformMessageResponse(response.data); setMessages(prev => prependOlderMessages(prev, page.messages)); setHasMore(page.hasMore); setNextBefore(page.nextBefore); } catch (err: any) { - setError(err.message || 'Failed to fetch older messages'); + if (isCurrentRequest()) { + setError(err.message || 'Failed to fetch older messages'); + } } finally { - setLoadingOlder(false); + if (olderPageInFlightRequestIdRef.current === requestId) { + olderPageInFlightRequestIdRef.current = null; + } + if (isCurrentRequest()) { + setLoadingOlder(false); + } } - }, [hasMore, loadingOlder, nextBefore, sessionId]); + }, [hasMore, nextBefore, sessionId]); // Reset state synchronously before paint when session changes // to prevent flash of welcome screen (useEffect runs AFTER paint) useLayoutEffect(() => { + activeSessionIdRef.current = sessionId; setMessages([]); setError(null); setHasMore(false); setNextBefore(null); + olderPageInFlightRequestIdRef.current = null; + setLoadingOlder(false); if (sessionId) { setLoading(true); } else { diff --git a/webui/src/hooks/useSharedResource.test.tsx b/webui/src/hooks/useSharedResource.test.tsx index 76acd1503..bc7e5cd4b 100644 --- a/webui/src/hooks/useSharedResource.test.tsx +++ b/webui/src/hooks/useSharedResource.test.tsx @@ -121,4 +121,92 @@ describe('useSharedResource', () => { expect(fetcher).toHaveBeenCalledTimes(2); expect(resource.getSnapshot().data).toBe('second'); }); + + it('preserves fallback data but rejects an explicit strict fetch', async () => { + const fetcher = vi.fn() + .mockResolvedValueOnce('ready') + .mockRejectedValueOnce(new Error('reload failed')); + const resource = createSharedResource({ + initialData: 'initial', + fetcher, + fallbackDataOnError: (previous) => previous, + }); + + await resource.fetch(); + resource.invalidate(); + + await expect(resource.fetch({ + force: true, + silent: true, + rejectOnError: true, + })).rejects.toThrow('reload failed'); + expect(resource.getSnapshot()).toMatchObject({ + data: 'ready', + error: 'reload failed', + loading: false, + }); + }); + + it.each(['lenient-first', 'strict-first'] as const)( + 'keeps per-caller error semantics when sharing an in-flight request (%s)', + async (order) => { + let rejectRequest: (error: Error) => void = () => {}; + const fetcher = vi.fn(() => new Promise((_resolve, reject) => { + rejectRequest = reject; + })); + const resource = createSharedResource({ + initialData: 'previous', + fetcher, + fallbackDataOnError: (previous) => previous, + }); + + const first = order === 'strict-first' + ? resource.fetch({ rejectOnError: true }) + : resource.fetch(); + const second = order === 'strict-first' + ? resource.fetch() + : resource.fetch({ rejectOnError: true }); + const strictRequest = order === 'strict-first' ? first : second; + const lenientRequest = order === 'strict-first' ? second : first; + + rejectRequest(new Error('shared request failed')); + + await expect(strictRequest).rejects.toThrow('shared request failed'); + await expect(lenientRequest).resolves.toBe('previous'); + expect(fetcher).toHaveBeenCalledTimes(1); + }, + ); + + it('runs a queued strict revalidation after an obsolete request fails', async () => { + const requests: Array<{ + resolve: (value: string) => void; + reject: (error: Error) => void; + }> = []; + const fetcher = vi.fn(() => new Promise((resolve, reject) => { + requests.push({ resolve, reject }); + })); + const resource = createSharedResource({ + initialData: 'initial', + fetcher, + }); + + const obsoleteRequest = resource.fetch({ rejectOnError: true }); + resource.invalidate(); + const revalidation = resource.fetch({ + force: true, + silent: true, + rejectOnError: true, + }); + + requests[0].reject(new Error('obsolete request failed')); + await expect(obsoleteRequest).rejects.toThrow('obsolete request failed'); + await waitFor(() => expect(fetcher).toHaveBeenCalledTimes(2)); + + requests[1].resolve('fresh'); + await expect(revalidation).resolves.toBe('fresh'); + expect(resource.getSnapshot()).toMatchObject({ + data: 'fresh', + error: null, + }); + }); }); diff --git a/webui/src/hooks/useSharedResource.ts b/webui/src/hooks/useSharedResource.ts index 827570ed4..2c0171a8d 100644 --- a/webui/src/hooks/useSharedResource.ts +++ b/webui/src/hooks/useSharedResource.ts @@ -11,6 +11,7 @@ export interface SharedResourceSnapshot { export interface SharedResourceFetchOptions { force?: boolean; silent?: boolean; + rejectOnError?: boolean; } export interface CreateSharedResourceOptions { @@ -64,9 +65,14 @@ export function createSharedResource(options: CreateSharedResourceOptions) const getErrorMessage = options.getErrorMessage ?? defaultGetErrorMessage; const listeners = new Set<() => void>(); - let inFlight: Promise | null = null; + type FetchOutcome = { + data: T; + failed: boolean; + error?: unknown; + }; + + let inFlight: Promise | null = null; let inFlightGeneration: number | null = null; - let queuedRevalidation: Promise | null = null; let generation = 0; let lastStartedAt = 0; let snapshot: SharedResourceSnapshot = { @@ -89,6 +95,16 @@ export function createSharedResource(options: CreateSharedResourceOptions) emit(); }; + const resolveOutcome = ( + request: Promise, + rejectOnError: boolean, + ): Promise => request.then((outcome) => { + if (rejectOnError && outcome.failed) { + throw outcome.error; + } + return outcome.data; + }); + const fetch = (fetchOptions: SharedResourceFetchOptions = {}): Promise => { const now = Date.now(); const force = fetchOptions.force === true; @@ -102,15 +118,9 @@ export function createSharedResource(options: CreateSharedResourceOptions) if (inFlight) { if (force && inFlightGeneration !== generation) { - if (!queuedRevalidation) { - queuedRevalidation = inFlight.then(() => { - queuedRevalidation = null; - return fetch(fetchOptions); - }); - } - return queuedRevalidation; + return inFlight.then(() => fetch(fetchOptions)); } - return inFlight; + return resolveOutcome(inFlight, fetchOptions.rejectOnError === true); } if (!force && recentlyStarted) { @@ -127,7 +137,7 @@ export function createSharedResource(options: CreateSharedResourceOptions) const request = options.fetcher() .then((data) => { if (requestGeneration !== generation) { - return data; + return { data, failed: false }; } updateSnapshot({ data, @@ -136,11 +146,13 @@ export function createSharedResource(options: CreateSharedResourceOptions) initialized: true, updatedAt: Date.now(), }); - return data; + return { data, failed: false }; }) .catch((error: unknown) => { if (requestGeneration !== generation) { - return snapshot.data; + // An obsolete request must not mutate the current snapshot, but its + // original strict caller still needs the real failure. + return { data: snapshot.data, failed: true, error }; } const data = resolveFallbackData(options.fallbackDataOnError, snapshot.data); updateSnapshot({ @@ -149,7 +161,7 @@ export function createSharedResource(options: CreateSharedResourceOptions) error: getErrorMessage(error), initialized: true, }); - return data; + return { data, failed: true, error }; }) .finally(() => { if (inFlight === request) { @@ -160,7 +172,7 @@ export function createSharedResource(options: CreateSharedResourceOptions) inFlight = request; inFlightGeneration = requestGeneration; - return request; + return resolveOutcome(request, fetchOptions.rejectOnError === true); }; return { @@ -180,7 +192,6 @@ export function createSharedResource(options: CreateSharedResourceOptions) resetForTesting: () => { inFlight = null; inFlightGeneration = null; - queuedRevalidation = null; generation = 0; lastStartedAt = 0; snapshot = { diff --git a/webui/src/hooks/useTasks.ts b/webui/src/hooks/useTasks.ts index 06cd97637..6d5cdbbb6 100644 --- a/webui/src/hooks/useTasks.ts +++ b/webui/src/hooks/useTasks.ts @@ -231,7 +231,7 @@ function useAdaptiveTaskPolling( return hasActive ? Math.min(pollInterval, 4000) : pollInterval; }; - let timerId: ReturnType; + let timerId: number; const tick = async () => { await resource.fetch({ silent: true }); timerId = window.setTimeout(tick, scheduleDelay()); diff --git a/webui/src/hooks/useTools.test.tsx b/webui/src/hooks/useTools.test.tsx index 87300447c..62fa7f2f8 100644 --- a/webui/src/hooks/useTools.test.tsx +++ b/webui/src/hooks/useTools.test.tsx @@ -121,6 +121,122 @@ describe('useTools', () => { expect(listMock).toHaveBeenCalledTimes(2); }); + it('reports a plugin refresh failure after reloading the visible tool list', async () => { + listMock + .mockResolvedValueOnce({ + data: [{ name: 'tool-alpha', description: 'alpha tool', category: 'custom', source: 'custom', enabled: true }], + }) + .mockResolvedValueOnce({ + data: [{ name: 'tool-alpha', description: 'alpha tool', category: 'custom', source: 'custom', enabled: true }], + }); + refreshMock.mockRejectedValue(new Error('plugin refresh failed')); + + const { result } = renderHook(() => useTools()); + await waitFor(() => expect(result.current.loading).toBe(false)); + + let caught: unknown; + await act(async () => { + try { + await result.current.refetch(); + } catch (error) { + caught = error; + } + }); + + expect(caught).toEqual(new Error('plugin refresh failed')); + expect(refreshMock).toHaveBeenCalledTimes(1); + expect(listMock).toHaveBeenCalledTimes(2); + expect(result.current.tools).toHaveLength(1); + }); + + it('returns a partial refresh outcome and keeps successfully loaded tools', async () => { + listMock + .mockResolvedValueOnce({ + data: [{ name: 'tool-alpha', description: 'alpha tool', category: 'custom', source: 'custom', enabled: true }], + }) + .mockResolvedValueOnce({ + data: [ + { name: 'tool-alpha', description: 'alpha tool', category: 'custom', source: 'custom', enabled: true }, + { name: 'tool-beta', description: 'beta tool', category: 'custom', source: 'custom', enabled: true }, + ], + }); + const partialResult = { + status: 'partial' as const, + tool_count: 2, + message: 'plugin: broken manifest', + }; + refreshMock.mockResolvedValue({ data: partialResult }); + + const { result } = renderHook(() => useTools()); + await waitFor(() => expect(result.current.loading).toBe(false)); + + let outcome; + await act(async () => { + outcome = await result.current.refetch(); + }); + + expect(outcome).toEqual(partialResult); + expect(listMock).toHaveBeenCalledTimes(2); + expect(result.current.tools.map((tool) => tool.name)).toEqual(['tool-alpha', 'tool-beta']); + }); + + it('returns an HTTP 200 error outcome without rejecting', async () => { + listMock.mockResolvedValue({ data: [] }); + const errorResult = { + status: 'error' as const, + tool_count: 0, + message: 'all refresh stages failed', + }; + refreshMock.mockResolvedValue({ data: errorResult }); + + const { result } = renderHook(() => useTools()); + await waitFor(() => expect(result.current.loading).toBe(false)); + + await expect(result.current.refetch()).resolves.toEqual(errorResult); + expect(listMock).toHaveBeenCalledTimes(2); + }); + + it('rejects when the visible list reload fails and preserves the previous tools', async () => { + listMock + .mockResolvedValueOnce({ + data: [{ name: 'tool-alpha', description: 'alpha tool', category: 'custom', source: 'custom', enabled: true }], + }) + .mockRejectedValueOnce(new Error('tool list unavailable')); + refreshMock.mockResolvedValue({ + data: { status: 'success', tool_count: 1, message: 'refreshed' }, + }); + + const { result } = renderHook(() => useTools()); + await waitFor(() => expect(result.current.loading).toBe(false)); + + await expect(result.current.refetch()).rejects.toThrow('tool list unavailable'); + expect(result.current.tools.map((tool) => tool.name)).toEqual(['tool-alpha']); + expect(result.current.error).toBe('tool list unavailable'); + }); + + it('preserves both backend details when refresh and visible list reload fail', async () => { + listMock + .mockResolvedValueOnce({ + data: [{ name: 'tool-alpha', description: 'alpha tool', category: 'custom', source: 'custom', enabled: true }], + }) + .mockRejectedValueOnce({ + response: { data: { detail: 'tool list storage unavailable' } }, + message: 'Request failed with status code 503', + }); + refreshMock.mockRejectedValue({ + response: { data: { detail: 'plugin registry unavailable' } }, + message: 'Request failed with status code 500', + }); + + const { result } = renderHook(() => useTools()); + await waitFor(() => expect(result.current.loading).toBe(false)); + + await expect(result.current.refetch()).rejects.toThrow( + 'Tool refresh failed: plugin registry unavailable; tool list reload failed: tool list storage unavailable', + ); + expect(result.current.tools.map((tool) => tool.name)).toEqual(['tool-alpha']); + }); + it('shares the initial tool list request across concurrent hook instances', async () => { let resolveList: (value: { data: any[] }) => void = () => {}; listMock.mockReturnValue(new Promise((resolve) => { @@ -298,6 +414,60 @@ describe('useTools', () => { expect(listPageMock).toHaveBeenCalledTimes(2); }); + it('reports a paged plugin refresh failure after reloading the visible page', async () => { + listPageMock.mockResolvedValue({ + data: { + items: [{ name: 'tool-alpha', description: 'alpha tool', category: 'custom', source: 'custom', enabled: true }], + total: 1, + offset: 0, + limit: 20, + facets: emptyFacets, + }, + }); + refreshMock.mockRejectedValue(new Error('paged refresh failed')); + + const { result } = renderHook(() => useToolPage({ offset: 0, limit: 20 })); + await waitFor(() => expect(result.current.loading).toBe(false)); + + let caught: unknown; + await act(async () => { + try { + await result.current.refetch(); + } catch (error) { + caught = error; + } + }); + + expect(caught).toEqual(new Error('paged refresh failed')); + expect(refreshMock).toHaveBeenCalledTimes(1); + expect(listPageMock).toHaveBeenCalledTimes(2); + expect(result.current.tools).toHaveLength(1); + }); + + it('rejects when the visible page reload fails and preserves the previous page', async () => { + listPageMock + .mockResolvedValueOnce({ + data: { + items: [{ name: 'tool-alpha', description: 'alpha tool', category: 'custom', source: 'custom', enabled: true }], + total: 1, + offset: 0, + limit: 20, + facets: emptyFacets, + }, + }) + .mockRejectedValueOnce(new Error('visible page unavailable')); + refreshMock.mockResolvedValue({ + data: { status: 'success', tool_count: 1, message: 'refreshed' }, + }); + + const { result } = renderHook(() => useToolPage({ offset: 0, limit: 20 })); + await waitFor(() => expect(result.current.loading).toBe(false)); + + await expect(result.current.refetch()).rejects.toThrow('visible page unavailable'); + expect(result.current.tools.map((tool) => tool.name)).toEqual(['tool-alpha']); + expect(result.current.error).toBe('visible page unavailable'); + }); + it('marks other cached queries stale and refreshes them when revisited', async () => { const callsBySource = new Map(); listPageMock.mockImplementation(async (params: { source?: string; offset?: number; limit?: number }) => { diff --git a/webui/src/hooks/useTools.ts b/webui/src/hooks/useTools.ts index 41abfac6a..28b09db34 100644 --- a/webui/src/hooks/useTools.ts +++ b/webui/src/hooks/useTools.ts @@ -5,6 +5,7 @@ import { type ToolListFacets, type ToolListPageParams, type ToolListPageResponse, + type ToolRefreshResponse, } from '@/api/tool'; import { createSharedResource, @@ -12,6 +13,7 @@ import { useSharedResource, type SharedResource, } from './useSharedResource'; +import { extractErrorMessage } from '@/utils/error'; const TOOL_LIST_STALE_TIME_MS = 5000; const TOOL_LIST_MIN_FETCH_INTERVAL_MS = 1000; @@ -45,7 +47,7 @@ const toolsResource = createSharedResource({ getErrorMessage: (err) => (err instanceof Error && err.message ? err.message : 'Failed to fetch tools'), }); -let toolRefreshInFlight: Promise | null = null; +let toolRefreshInFlight: Promise | null = null; const toolPageResources = new Map>(); function normalizeToolPageParams(params: ToolListPageParams): Required { @@ -95,21 +97,13 @@ function getToolPageResource(params: ToolListPageParams): SharedResource { +function refreshToolPlugins(): Promise { if (toolRefreshInFlight) { return toolRefreshInFlight; } toolRefreshInFlight = toolAPI.refresh() - .catch(() => { - // Best-effort refresh; still update the visible list afterwards. - }) - .then(() => { - toolsResource.invalidate(); - toolPageResources.forEach((resource) => resource.invalidate()); - return toolsResource.fetch({ force: true, silent: true }); - }) - .then(() => undefined) + .then((response) => response.data) .finally(() => { toolRefreshInFlight = null; }); @@ -117,6 +111,41 @@ function refreshToolsResource(): Promise { return toolRefreshInFlight; } +async function refreshToolsResource( + fetchVisible: () => Promise, +): Promise { + let refreshResult: ToolRefreshResponse | undefined; + let refreshError: unknown; + try { + refreshResult = await refreshToolPlugins(); + } catch (error) { + refreshError = error; + } + + toolsResource.invalidate(); + toolPageResources.forEach((resource) => resource.invalidate()); + let listError: unknown; + try { + await fetchVisible(); + } catch (error) { + listError = error; + } + + if (refreshError && listError) { + throw new Error( + `Tool refresh failed: ${extractErrorMessage(refreshError, 'unknown error')}; ` + + `tool list reload failed: ${extractErrorMessage(listError, 'unknown error')}`, + ); + } + if (refreshError) { + throw refreshError; + } + if (listError) { + throw listError; + } + return refreshResult!; +} + export function __resetToolsResourceForTesting(): void { toolsResource.resetForTesting(); toolPageResources.forEach((resource) => resource.resetForTesting()); @@ -137,7 +166,14 @@ export function useTools() { ); useRefreshOnResume(refreshVisibleList); - const refetch = useCallback(() => refreshToolsResource(), []); + const refetch = useCallback( + () => refreshToolsResource(() => toolsResource.fetch({ + force: true, + silent: true, + rejectOnError: true, + })), + [], + ); return { tools, @@ -175,24 +211,14 @@ export function useToolPage(params: ToolListPageParams) { ); useRefreshOnResume(refreshVisiblePage); - const refetch = useCallback(async () => { - if (toolRefreshInFlight) { - await toolRefreshInFlight; - } else { - toolRefreshInFlight = toolAPI.refresh() - .catch(() => { - // Best-effort refresh; still update the visible page afterwards. - }) - .then(() => undefined) - .finally(() => { - toolRefreshInFlight = null; - }); - await toolRefreshInFlight; - } - toolPageResources.forEach((pageResource) => pageResource.invalidate()); - toolsResource.invalidate(); - await resource.fetch({ force: true, silent: true }); - }, [resource]); + const refetch = useCallback( + () => refreshToolsResource(() => resource.fetch({ + force: true, + silent: true, + rejectOnError: true, + })), + [resource], + ); return { tools: data.items, diff --git a/webui/src/locales/en-US/monitoring.json b/webui/src/locales/en-US/monitoring.json index 9242abffb..d22f27961 100644 --- a/webui/src/locales/en-US/monitoring.json +++ b/webui/src/locales/en-US/monitoring.json @@ -15,7 +15,8 @@ "metrics": { "messageRate": "Message Rate", "toolCallRate": "Tool Calls", - "errorRate": "Tool Parse Failures", + "errorRate": "System Errors", + "toolParseFailureRate": "Tool Parse Failures", "avgResponse": "Avg Response", "activeRequests": "Active Requests" }, diff --git a/webui/src/locales/en-US/tool.json b/webui/src/locales/en-US/tool.json index 77169ccdf..90c78762c 100644 --- a/webui/src/locales/en-US/tool.json +++ b/webui/src/locales/en-US/tool.json @@ -562,6 +562,9 @@ "alert": { "refreshFailed": "Refresh failed: {{error}}", + "refreshFailedTitle": "Tool refresh failed", + "refreshPartialTitle": "Tool refresh completed with warnings", + "refreshPartialDefault": "Some refresh stages reported errors. Available tools were updated.", "mcpAddSuccess": "MCP service added successfully", "addFailed": "Add failed: {{error}}", "mcpNameRequired": "Please enter MCP service name", @@ -573,6 +576,7 @@ "confirmDeleteCred": "Are you sure you want to delete credentials?", "credDeleted": "Credentials deleted", "deleteFailed": "Delete failed: {{error}}", + "deletePartialTitle": "Tool deleted with cleanup warnings", "unknownError": "Unknown error", "fillAllRequired": "Please fill in all required fields", "configFailed": "Configuration failed: {{error}}", diff --git a/webui/src/locales/zh-CN/monitoring.json b/webui/src/locales/zh-CN/monitoring.json index 6be54d714..7f79aee9d 100644 --- a/webui/src/locales/zh-CN/monitoring.json +++ b/webui/src/locales/zh-CN/monitoring.json @@ -15,7 +15,8 @@ "metrics": { "messageRate": "消息速率", "toolCallRate": "工具调用", - "errorRate": "工具解析失败", + "errorRate": "系统错误", + "toolParseFailureRate": "工具解析失败", "avgResponse": "平均响应", "activeRequests": "活跃请求" }, diff --git a/webui/src/locales/zh-CN/tool.json b/webui/src/locales/zh-CN/tool.json index 7e21b2d42..59e98ef22 100644 --- a/webui/src/locales/zh-CN/tool.json +++ b/webui/src/locales/zh-CN/tool.json @@ -562,6 +562,9 @@ "alert": { "refreshFailed": "刷新失败: {{error}}", + "refreshFailedTitle": "工具刷新失败", + "refreshPartialTitle": "工具刷新完成,但存在警告", + "refreshPartialDefault": "部分刷新阶段出现错误,当前可用工具列表已更新。", "mcpAddSuccess": "MCP 服务添加成功", "addFailed": "添加失败: {{error}}", "mcpNameRequired": "请输入 MCP 服务名称", @@ -573,6 +576,7 @@ "confirmDeleteCred": "确定要删除凭证吗?", "credDeleted": "凭证已删除", "deleteFailed": "删除失败: {{error}}", + "deletePartialTitle": "工具已删除,但清理过程存在警告", "unknownError": "未知错误", "fillAllRequired": "请填写所有必填字段", "configFailed": "配置失败: {{error}}", diff --git a/webui/src/pages/Hub/index.test.tsx b/webui/src/pages/Hub/index.test.tsx index 780247fa9..152054b3a 100644 --- a/webui/src/pages/Hub/index.test.tsx +++ b/webui/src/pages/Hub/index.test.tsx @@ -5,7 +5,7 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'; import HubPage from './index'; -const { hubAPI } = vi.hoisted(() => ({ +const { hubAPI, toastError } = vi.hoisted(() => ({ hubAPI: { catalog: vi.fn(), catalogPage: vi.fn(), @@ -19,6 +19,7 @@ const { hubAPI } = vi.hoisted(() => ({ files: vi.fn(), fileContent: vi.fn(), }, + toastError: vi.fn(), })); vi.mock('@/api/hub', () => ({ hubAPI })); @@ -31,6 +32,9 @@ vi.mock('@/contexts/ProductNameContext', () => ({ vi.mock('@/contexts/AuthContext', () => ({ useAuth: () => ({ user: { id: 'admin-1', role: 'admin' } }), })); +vi.mock('@/components/common/Toast', () => ({ + useToast: () => ({ error: toastError }), +})); vi.mock('@/components/common/LoadingSpinner', () => ({ default: () =>
loading
, })); @@ -85,10 +89,12 @@ function catalogPage(items: ReturnType[], total = items.len function deferred() { let resolve!: (value: T) => void; - const promise = new Promise((resolvePromise) => { + let reject!: (reason?: unknown) => void; + const promise = new Promise((resolvePromise, rejectPromise) => { resolve = resolvePromise; + reject = rejectPromise; }); - return { promise, resolve }; + return { promise, resolve, reject }; } function renderHub() { @@ -266,6 +272,137 @@ describe('HubPage catalog loading', () => { expect(hubAPI.catalogPage).toHaveBeenLastCalledWith(expect.objectContaining({ q: 'latest' })); }); + it('shows a user-visible error when a manual refresh fails', async () => { + const user = userEvent.setup(); + hubAPI.catalogPage.mockResolvedValue(catalogPage([catalogEntry('initial-entry', 'Initial Entry')])); + hubAPI.refresh.mockRejectedValue(new Error('refresh unavailable')); + + renderHub(); + expect(await screen.findByText('Initial Entry')).toBeInTheDocument(); + await user.click(screen.getByRole('button', { name: 'Refresh' })); + + await waitFor(() => { + expect(toastError).toHaveBeenCalledWith('Hub refresh failed', 'refresh unavailable'); + }); + expect(screen.getByRole('button', { name: 'Refresh' })).toBeEnabled(); + expect(hubAPI.catalogPage).toHaveBeenCalledTimes(1); + }); + + it('distinguishes a completed manual refresh from a catalog reload failure', async () => { + const user = userEvent.setup(); + hubAPI.catalogPage + .mockResolvedValueOnce(catalogPage([catalogEntry('initial-entry', 'Initial Entry')])) + .mockRejectedValueOnce({ + response: { data: { detail: 'catalog temporarily unavailable' } }, + message: 'Request failed with status code 503', + }); + hubAPI.refresh.mockResolvedValue({ data: { status: 'success' } }); + + renderHub(); + expect(await screen.findByText('Initial Entry')).toBeInTheDocument(); + await user.click(screen.getByRole('button', { name: 'Refresh' })); + + await waitFor(() => { + expect(toastError).toHaveBeenCalledWith( + 'Hub refresh completed, but catalog reload failed', + 'catalog temporarily unavailable', + ); + }); + expect(toastError).not.toHaveBeenCalledWith('Hub refresh failed', expect.anything()); + expect(hubAPI.refresh).toHaveBeenCalledTimes(1); + expect(hubAPI.catalogPage).toHaveBeenCalledTimes(2); + expect(screen.getByRole('button', { name: 'Refresh' })).toBeEnabled(); + }); + + it('does not report a stale manual reload failure after the query changes', async () => { + const user = userEvent.setup(); + const staleReload = deferred>(); + hubAPI.catalogPage + .mockResolvedValueOnce(catalogPage([catalogEntry('initial-entry', 'Initial Entry')])) + .mockImplementationOnce(() => staleReload.promise) + .mockResolvedValueOnce(catalogPage([catalogEntry('latest-entry', 'Latest Entry')])); + hubAPI.refresh.mockResolvedValue({ data: { status: 'success' } }); + + renderHub(); + expect(await screen.findByText('Initial Entry')).toBeInTheDocument(); + await user.click(screen.getByRole('button', { name: 'Refresh' })); + await waitFor(() => expect(hubAPI.catalogPage).toHaveBeenCalledTimes(2)); + + fireEvent.change( + screen.getByPlaceholderText('Search plugin name, description, tag, use case'), + { target: { value: 'latest' } }, + ); + expect(await screen.findByText('Latest Entry')).toBeInTheDocument(); + + await act(async () => { + staleReload.reject(new Error('obsolete catalog failure')); + try { + await staleReload.promise; + } catch { + // The component owns this rejected request. + } + }); + + await waitFor(() => expect(screen.getByRole('button', { name: 'Refresh' })).toBeEnabled()); + expect(toastError).not.toHaveBeenCalledWith( + 'Hub refresh completed, but catalog reload failed', + expect.anything(), + ); + expect(screen.getByText('Latest Entry')).toBeInTheDocument(); + }); + + it.each([ + ['install', 'available', 'Install', 'Install failed'], + ['update', 'updateAvailable', 'Update', 'Update failed'], + ['uninstall', 'installed', 'Uninstall', 'Uninstall failed'], + ] as const)('shows a user-visible error when %s fails', async (action, state, buttonLabel, errorTitle) => { + const user = userEvent.setup(); + const entry = { ...catalogEntry('action-entry', 'Action Entry'), state }; + hubAPI.catalogPage.mockResolvedValue(catalogPage([entry])); + hubAPI[action].mockRejectedValue({ + response: { data: { detail: `${action} permission denied` } }, + message: `Request failed with status code 422`, + }); + + renderHub(); + await user.click(await screen.findByRole('button', { name: buttonLabel })); + + await waitFor(() => { + expect(toastError).toHaveBeenCalledWith( + `${errorTitle}: Action Entry`, + `${action} permission denied`, + ); + }); + expect(screen.getByRole('button', { name: buttonLabel })).toBeEnabled(); + expect(hubAPI.catalogPage).toHaveBeenCalledTimes(1); + }); + + it('does not report a completed action as failed when the follow-up reload fails', async () => { + const user = userEvent.setup(); + const entry = catalogEntry('action-entry', 'Action Entry'); + hubAPI.catalogPage.mockResolvedValue(catalogPage([entry])); + hubAPI.install.mockResolvedValue({ data: { ...entry, state: 'installed' } }); + hubAPI.catalog.mockRejectedValue({ + response: { data: { detail: 'catalog temporarily unavailable' } }, + message: 'Request failed with status code 503', + }); + + renderHub(); + await user.click(await screen.findByRole('button', { name: 'Install' })); + + await waitFor(() => { + expect(toastError).toHaveBeenCalledWith( + 'Action completed, but Hub reload failed: Action Entry', + 'catalog temporarily unavailable', + ); + }); + expect(toastError).not.toHaveBeenCalledWith( + 'Install failed: Action Entry', + expect.anything(), + ); + expect(hubAPI.install).toHaveBeenCalledTimes(1); + }); + it('does not let a tree action started under old filters overwrite the latest tree', async () => { const user = userEvent.setup(); const actionEntry = catalogEntry('tree-action', 'Tree Action'); diff --git a/webui/src/pages/Hub/index.tsx b/webui/src/pages/Hub/index.tsx index 3c90bc94e..4b0ed8f07 100644 --- a/webui/src/pages/Hub/index.tsx +++ b/webui/src/pages/Hub/index.tsx @@ -19,6 +19,8 @@ import { } from 'lucide-react'; import PageHeader from '@/components/common/PageHeader'; import LoadingSpinner from '@/components/common/LoadingSpinner'; +import { useToast } from '@/components/common/Toast'; +import { extractErrorMessage } from '@/utils/error'; import SuiteInstallProgressPanel, { applySuiteInstallProgressEvent, createSuiteInstallProgressState, @@ -143,6 +145,11 @@ const HUB_TEXT = { suiteInstallFailed: '安装失败', suiteInstallProgress: '安装进度', suiteInstallDismiss: '关闭', + refreshFailed: '刷新 Hub 失败', + refreshReloadFailed: 'Hub 刷新已完成,但列表重载失败', + actionRefreshFailed: '操作已完成,但刷新 Hub 列表失败', + actionFailed: { install: '安装失败', update: '更新失败', uninstall: '卸载失败' }, + unknownError: '未知错误', suiteItemStatuses: { pending: '等待中', installing: '安装中', @@ -199,6 +206,11 @@ const HUB_TEXT = { suiteInstallFailed: 'Install failed', suiteInstallProgress: 'Progress', suiteInstallDismiss: 'Close', + refreshFailed: 'Hub refresh failed', + refreshReloadFailed: 'Hub refresh completed, but catalog reload failed', + actionRefreshFailed: 'Action completed, but Hub reload failed', + actionFailed: { install: 'Install failed', update: 'Update failed', uninstall: 'Uninstall failed' }, + unknownError: 'Unknown error', suiteItemStatuses: { pending: 'Pending', installing: 'Installing', @@ -262,6 +274,7 @@ export default function HubPage() { const { i18n } = useTranslation(); const { user } = useAuth(); const { productName } = useProductName(); + const toast = useToast(); const [searchParams] = useSearchParams(); const text = i18n.language.toLowerCase().startsWith('zh') ? HUB_TEXT.zh : HUB_TEXT.en; const hubTitle = `${productName} Hub`; @@ -320,7 +333,7 @@ export default function HubPage() { currentTreeRequestKeyRef.current = treeRequestKey; const canManageHub = user?.role === 'admin'; - const fetchCatalog = useCallback(async (silent = false) => { + const fetchCatalog = useCallback(async (silent = false, propagateError = false) => { const requestKey = catalogRequestKey; if (requestKey !== currentCatalogRequestKeyRef.current) return null; const requestId = ++catalogRequestIdRef.current; @@ -347,9 +360,11 @@ export default function HubPage() { setFacetCounts(res.data.facets ?? EMPTY_HUB_FACETS); return nextItems; } catch (error) { - if (isCurrentRequest()) { + const currentRequest = isCurrentRequest(); + if (currentRequest) { setCatalogError(error instanceof Error ? error.message : 'Failed to load Hub catalog'); } + if (propagateError && currentRequest) throw error; return null; } finally { if (isCurrentRequest()) { @@ -359,7 +374,7 @@ export default function HubPage() { } }, [catalogRequestKey, debouncedQuery, page, pageSize, stateFilter, tagFilter, typeFilter, useCaseFilter]); - const fetchTreeCatalog = useCallback(async (silent = false) => { + const fetchTreeCatalog = useCallback(async (silent = false, propagateError = false) => { const requestKey = treeRequestKey; if (requestKey !== currentTreeRequestKeyRef.current) return; const requestId = ++treeRequestIdRef.current; @@ -380,9 +395,11 @@ export default function HubPage() { if (!isCurrentRequest()) return; setTreeItems(Array.isArray(res.data) ? res.data : []); } catch (error) { - if (isCurrentRequest()) { + const currentRequest = isCurrentRequest(); + if (currentRequest) { setTreeError(error instanceof Error ? error.message : 'Failed to load Hub tree'); } + if (propagateError && currentRequest) throw error; } finally { if (isCurrentRequest() && !silent) { setTreeLoading(false); @@ -439,10 +456,25 @@ export default function HubPage() { setRefreshing(true); try { await hubAPI.refresh(); + } catch (error) { + toast.error( + text.refreshFailed, + extractErrorMessage(error, text.unknownError), + ); + setRefreshing(false); + return; + } + + try { await Promise.all([ - fetchCatalog(true), - viewMode === 'tree' ? fetchTreeCatalog(true) : Promise.resolve(), + fetchCatalog(true, true), + viewMode === 'tree' ? fetchTreeCatalog(true, true) : Promise.resolve(), ]); + } catch (error) { + toast.error( + text.refreshReloadFailed, + extractErrorMessage(error, text.unknownError), + ); } finally { setRefreshing(false); } @@ -465,9 +497,24 @@ export default function HubPage() { } if (action === 'update') await hubAPI.update(entry.type, entry.id); if (action === 'uninstall') await hubAPI.uninstall(entry.type, entry.id); + } catch (error) { + if (action === 'install' && entry.type === 'component') { + const message = extractErrorMessage(error, text.suiteInstallFailed); + setSuiteInstallProgress(current => failSuiteInstallProgress(current, entry, message)); + } else { + toast.error( + `${text.actionFailed[action]}: ${getHubName(entry, i18n.language)}`, + extractErrorMessage(error, text.unknownError), + ); + } + setActionId(null); + return; + } + + try { const [, , refreshedEntry] = await Promise.all([ - fetchCatalog(true), - viewMode === 'tree' ? fetchTreeCatalog(true) : Promise.resolve(), + fetchCatalog(true, true), + viewMode === 'tree' ? fetchTreeCatalog(true, true) : Promise.resolve(), hubAPI.catalog({ q: entry.id, type: entry.type }).then(res => ( (Array.isArray(res.data) ? res.data : []).find(item => ( item.type === entry.type && item.id === entry.id @@ -478,12 +525,10 @@ export default function HubPage() { current?.type === entry.type && current?.id === entry.id ? refreshedEntry : current )); } catch (error) { - if (action === 'install' && entry.type === 'component') { - const message = error instanceof Error ? error.message : text.suiteInstallFailed; - setSuiteInstallProgress(current => failSuiteInstallProgress(current, entry, message)); - } else { - console.error(error); - } + toast.error( + `${text.actionRefreshFailed}: ${getHubName(entry, i18n.language)}`, + extractErrorMessage(error, text.unknownError), + ); } finally { setActionId(null); } diff --git a/webui/src/pages/Monitoring/index.test.tsx b/webui/src/pages/Monitoring/index.test.tsx index 9e643d598..ad3daf9e5 100644 --- a/webui/src/pages/Monitoring/index.test.tsx +++ b/webui/src/pages/Monitoring/index.test.tsx @@ -1,5 +1,6 @@ -import { render, screen, waitFor } from '@testing-library/react'; -import { beforeEach, describe, expect, it, vi } from 'vitest'; +import { StrictMode } from 'react'; +import { act, render, screen, waitFor } from '@testing-library/react'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; const monitoringAPI = vi.hoisted(() => ({ getStatus: vi.fn(), @@ -44,6 +45,7 @@ describe('MonitoringPage', () => { messageRate: null, toolCallRate: null, errorRate: null, + toolParseFailureRate: 2.5, avgResponseTime: null, activeRequests: null, }, @@ -52,11 +54,119 @@ describe('MonitoringPage', () => { monitoringAPI.getToolPerformance.mockResolvedValue({ data: [] }); }); + afterEach(() => { + vi.useRealTimers(); + }); + it('shows unavailable metrics instead of presenting fabricated zero rates', async () => { render(); await waitFor(() => expect(screen.getByText('Monitoring')).toBeInTheDocument()); expect(screen.getAllByText('No data').length).toBeGreaterThanOrEqual(5); + expect(screen.getByText('2.5/min')).toBeInTheDocument(); expect(screen.queryByText('0.0/min')).not.toBeInTheDocument(); }); + + it('waits for the current polling batch to finish before scheduling another', async () => { + vi.useFakeTimers(); + const never = new Promise(() => {}); + monitoringAPI.getStatus.mockReturnValue(never); + monitoringAPI.getMetrics.mockReturnValue(never); + monitoringAPI.getLLMPerformance.mockReturnValue(never); + monitoringAPI.getToolPerformance.mockReturnValue(never); + + const { unmount } = render(); + expect(monitoringAPI.getStatus).toHaveBeenCalledTimes(1); + + await act(async () => { + await vi.advanceTimersByTimeAsync(10_000); + }); + + expect(monitoringAPI.getStatus).toHaveBeenCalledTimes(1); + expect(monitoringAPI.getMetrics).toHaveBeenCalledTimes(1); + expect(monitoringAPI.getLLMPerformance).toHaveBeenCalledTimes(1); + expect(monitoringAPI.getToolPerformance).toHaveBeenCalledTimes(1); + unmount(); + }); + + it('does not start another polling batch while slow requests remain after a fast failure', async () => { + vi.useFakeTimers(); + let resolveMetrics: (value: unknown) => void = () => {}; + let resolveLLMPerformance: (value: unknown) => void = () => {}; + let resolveToolPerformance: (value: unknown) => void = () => {}; + monitoringAPI.getStatus.mockRejectedValueOnce(new Error('status failed')); + monitoringAPI.getMetrics.mockImplementationOnce(() => new Promise((resolve) => { + resolveMetrics = resolve; + })); + monitoringAPI.getLLMPerformance.mockImplementationOnce(() => new Promise((resolve) => { + resolveLLMPerformance = resolve; + })); + monitoringAPI.getToolPerformance.mockImplementationOnce(() => new Promise((resolve) => { + resolveToolPerformance = resolve; + })); + + const { unmount } = render(); + await act(async () => { + await vi.advanceTimersByTimeAsync(10_000); + }); + + expect(monitoringAPI.getStatus).toHaveBeenCalledTimes(1); + expect(monitoringAPI.getMetrics).toHaveBeenCalledTimes(1); + expect(monitoringAPI.getLLMPerformance).toHaveBeenCalledTimes(1); + expect(monitoringAPI.getToolPerformance).toHaveBeenCalledTimes(1); + + await act(async () => { + resolveMetrics({ data: { timestamp: 1 } }); + resolveLLMPerformance({ data: [] }); + resolveToolPerformance({ data: [] }); + }); + await act(async () => { + await vi.advanceTimersByTimeAsync(4_999); + }); + expect(monitoringAPI.getStatus).toHaveBeenCalledTimes(1); + + await act(async () => { + await vi.advanceTimersByTimeAsync(1); + }); + expect(monitoringAPI.getStatus).toHaveBeenCalledTimes(2); + unmount(); + }); + + it('reuses the in-flight polling batch during StrictMode effect replay', async () => { + const statusResolvers: Array<(value: unknown) => void> = []; + const metricsResolvers: Array<(value: unknown) => void> = []; + const llmResolvers: Array<(value: unknown) => void> = []; + const toolResolvers: Array<(value: unknown) => void> = []; + monitoringAPI.getStatus.mockImplementation(() => new Promise((resolve) => statusResolvers.push(resolve))); + monitoringAPI.getMetrics.mockImplementation(() => new Promise((resolve) => metricsResolvers.push(resolve))); + monitoringAPI.getLLMPerformance.mockImplementation(() => new Promise((resolve) => llmResolvers.push(resolve))); + monitoringAPI.getToolPerformance.mockImplementation(() => new Promise((resolve) => toolResolvers.push(resolve))); + + const { unmount } = render( + + + , + ); + + expect(monitoringAPI.getStatus).toHaveBeenCalledTimes(1); + + await act(async () => { + statusResolvers[0]({ + data: { + status: 'healthy', + uptime: 60, + activeSessions: 22, + activeAgents: null, + mcpServers: {}, + timestamp: 2, + }, + }); + metricsResolvers[0]({ data: { timestamp: 2 } }); + llmResolvers[0]({ data: [] }); + toolResolvers[0]({ data: [] }); + }); + expect(await screen.findByText('22')).toBeInTheDocument(); + expect(monitoringAPI.getStatus).toHaveBeenCalledTimes(1); + unmount(); + }); }); diff --git a/webui/src/pages/Monitoring/index.tsx b/webui/src/pages/Monitoring/index.tsx index 4188f544c..be480331b 100644 --- a/webui/src/pages/Monitoring/index.tsx +++ b/webui/src/pages/Monitoring/index.tsx @@ -1,4 +1,4 @@ -import { useState, useEffect } from 'react'; +import { useState, useEffect, useCallback, useRef } from 'react'; import { BarChart3, Activity, Clock, AlertCircle, TrendingUp, Zap } from 'lucide-react'; import { useTranslation } from 'react-i18next'; import PageHeader from '@/components/common/PageHeader'; @@ -13,37 +13,85 @@ export default function MonitoringPage() { const [toolPerformance, setToolPerformance] = useState([]); const [loading, setLoading] = useState(true); const [error, setError] = useState(null); + const latestRequestIdRef = useRef(0); + const inFlightRef = useRef | null>(null); + const inFlightRequestIdRef = useRef(null); + const mountedRef = useRef(false); - useEffect(() => { - fetchData(); - - // Auto refresh every 5 seconds - const interval = setInterval(fetchData, 5000); - return () => clearInterval(interval); + const fetchData = useCallback((): Promise => { + if (inFlightRef.current) return inFlightRef.current; + + const requestId = ++latestRequestIdRef.current; + const request = (async () => { + try { + setLoading(true); + setError(null); + + const [statusResult, metricsResult, llmPerfResult, toolPerfResult] = await Promise.allSettled([ + monitoringAPI.getStatus(), + monitoringAPI.getMetrics(), + monitoringAPI.getLLMPerformance(), + monitoringAPI.getToolPerformance(), + ]); + + const failedResult = [statusResult, metricsResult, llmPerfResult, toolPerfResult] + .find((result) => result.status === 'rejected'); + if (failedResult?.status === 'rejected') { + throw failedResult.reason; + } + if ( + statusResult.status !== 'fulfilled' + || metricsResult.status !== 'fulfilled' + || llmPerfResult.status !== 'fulfilled' + || toolPerfResult.status !== 'fulfilled' + ) return; + + if (!mountedRef.current || latestRequestIdRef.current !== requestId) return; + setStatus(statusResult.value.data); + setMetrics(metricsResult.value.data); + setLlmPerformance(llmPerfResult.value.data); + setToolPerformance(toolPerfResult.value.data); + } catch (err: any) { + if (mountedRef.current && latestRequestIdRef.current === requestId) { + setError(err.message); + } + } finally { + if (mountedRef.current && latestRequestIdRef.current === requestId) { + setLoading(false); + } + if (inFlightRequestIdRef.current === requestId) { + inFlightRef.current = null; + inFlightRequestIdRef.current = null; + } + } + })(); + + inFlightRef.current = request; + inFlightRequestIdRef.current = requestId; + return request; }, []); - const fetchData = async () => { - try { - setLoading(true); - setError(null); - - const [statusRes, metricsRes, llmPerfRes, toolPerfRes] = await Promise.all([ - monitoringAPI.getStatus(), - monitoringAPI.getMetrics(), - monitoringAPI.getLLMPerformance(), - monitoringAPI.getToolPerformance(), - ]); - - setStatus(statusRes.data); - setMetrics(metricsRes.data); - setLlmPerformance(llmPerfRes.data); - setToolPerformance(toolPerfRes.data); - } catch (err: any) { - setError(err.message); - } finally { - setLoading(false); - } - }; + useEffect(() => { + let cancelled = false; + let timerId: number | undefined; + mountedRef.current = true; + + const poll = async () => { + await fetchData(); + if (!cancelled) { + timerId = window.setTimeout(poll, 5000); + } + }; + + void poll(); + return () => { + cancelled = true; + mountedRef.current = false; + if (timerId !== undefined) { + window.clearTimeout(timerId); + } + }; + }, [fetchData]); if (loading && !status) { return ( @@ -151,7 +199,7 @@ export default function MonitoringPage() { {metrics && (

{t('realTimeMetrics')}

-
+
{t('metrics.messageRate')}
@@ -170,6 +218,12 @@ export default function MonitoringPage() { {formatRate(metrics.errorRate)}
+
+
{t('metrics.toolParseFailureRate')}
+
+ {formatRate(metrics.toolParseFailureRate)} +
+
{t('metrics.avgResponse')}
diff --git a/webui/src/pages/Tool/index.tsx b/webui/src/pages/Tool/index.tsx index 3c1e51c22..cd0c44400 100644 --- a/webui/src/pages/Tool/index.tsx +++ b/webui/src/pages/Tool/index.tsx @@ -43,6 +43,7 @@ import { } from 'lucide-react'; import LoadingSpinner from '@/components/common/LoadingSpinner'; import EmptyState from '@/components/common/EmptyState'; +import { useToast } from '@/components/common/Toast'; import { useToolPage } from '@/hooks/useTools'; import { useDebouncedValue } from '@/hooks/useDebouncedValue'; import { canDirectlyTestTool, toolAPI, Tool, ToolFixture } from '@/api/tool'; @@ -57,6 +58,7 @@ import APITabContent from './components/APITabContent'; import LocalTabContent from './components/LocalTabContent'; import { getSourceLabel } from './constants'; import { getCatalogDescription, getMetadataDescription } from '@/utils/mcpCatalog'; +import { extractErrorMessage } from '@/utils/error'; import { getLocalizedToolDescription, getLocalizedFixtureLabel } from './toolDisplay'; import { DEFAULT_TOOL_TAB, @@ -158,6 +160,7 @@ function mergeFacetKeys(facets: Record, activeValues: Set }, @@ -306,12 +309,31 @@ export default function ToolPage() { }, [fetchApiServicesCount]); const refreshToolData = useCallback(async () => { - await Promise.all([ + const [refreshResult] = await Promise.all([ refetch(), fetchApiServicesCount(), ]); + return refreshResult; }, [refetch, fetchApiServicesCount]); + const showRefreshOutcome = useCallback((status: 'success' | 'partial' | 'error', message: string) => { + if (status === 'partial') { + toast.warning(t('alert.refreshPartialTitle'), message || t('alert.refreshPartialDefault')); + } else if (status === 'error') { + toast.error(t('alert.refreshFailedTitle'), message || t('alert.unknownError')); + } + }, [t, toast]); + + const refreshToolDataAfterMutation = useCallback(async () => { + try { + const result = await refreshToolData(); + showRefreshOutcome(result.status, result.message); + } catch (err: unknown) { + const message = extractErrorMessage(err, t('alert.unknownError')); + toast.error(t('alert.refreshFailedTitle'), message); + } + }, [refreshToolData, showRefreshOutcome, t, toast]); + // The backend still marks some valid MCP catalog entries as "api" based on category. // Until API catalog has its own rendering path, show all catalog entries in the MCP tab. const mcpCatalogEntries = useMemo(() => catalogEntries, [catalogEntries]); @@ -371,14 +393,20 @@ export default function ToolPage() { if (refreshing) return; try { setRefreshing(true); - await Promise.all([ + setRefreshDone(false); + const [result] = await Promise.all([ refreshToolData(), new Promise((r) => setTimeout(r, 600)), ]); - setRefreshDone(true); - setTimeout(() => setRefreshDone(false), 2000); - } catch (err: any) { - alert(t('alert.refreshFailed', { error: err.message })); + if (result.status === 'success') { + setRefreshDone(true); + setTimeout(() => setRefreshDone(false), 2000); + } else { + showRefreshOutcome(result.status, result.message); + } + } catch (err: unknown) { + const message = extractErrorMessage(err, t('alert.unknownError')); + toast.error(t('alert.refreshFailedTitle'), message); } finally { setRefreshing(false); } @@ -455,7 +483,7 @@ export default function ToolPage() {

{error}

-
@@ -588,7 +616,7 @@ export default function ToolPage() { tools={processedTools} searchQuery={searchQuery} onSelectTool={openDetail} - onRefreshTools={refreshToolData} + onRefreshTools={refreshToolDataAfterMutation} catalogEntries={mcpCatalogEntries} catalogCategories={catalogCategories} catalogLoading={catalogLoading} @@ -601,7 +629,7 @@ export default function ToolPage() { { try { - await toolAPI.delete(name); + const { data: deleteResult } = await toolAPI.delete(name); setSelectedTool(null); setTestResult(null); + if (deleteResult.status === 'partial') { + toast.warning( + t('alert.deletePartialTitle'), + deleteResult.errors?.join('; ') || deleteResult.message, + ); + } await handleRefresh(); - } catch (err: any) { - alert(t('alert.deleteFailed', { error: err.response?.data?.detail || err.message })); + } catch (err: unknown) { + alert(t('alert.deleteFailed', { + error: extractErrorMessage(err, t('alert.unknownError')), + })); } }} onEnabledChange={(name, newEnabled) => { setSelectedTool((prev) => prev ? { ...prev, enabled: newEnabled } : prev); - void refreshToolData(); + void refreshToolDataAfterMutation(); }} /> )} From 90ce06aace499b895fd0f33a78263976dcae9b6f Mon Sep 17 00:00:00 2001 From: John Yin <10972267+john-yin2333@user.noreply.gitee.com> Date: Mon, 13 Jul 2026 17:44:03 +0800 Subject: [PATCH 20/49] fix: harden whatsapp channel pairing and ui --- .../channel/builtin/whatsapp/bridge/bridge.js | 2 +- flocks/channel/builtin/whatsapp/config.py | 3 +- flocks/channel/builtin/whatsapp/pairing.py | 19 ++++- flocks/server/routes/channel.py | 18 +++++ tests/channel/test_whatsapp.py | 55 ++++++++++++++ webui/public/channel-whatsapp.png | Bin 0 -> 32325 bytes webui/src/locales/en-US/channel.json | 5 +- webui/src/locales/zh-CN/channel.json | 5 +- webui/src/pages/Channel/index.tsx | 67 +++++++++++++----- 9 files changed, 149 insertions(+), 25 deletions(-) create mode 100644 webui/public/channel-whatsapp.png diff --git a/flocks/channel/builtin/whatsapp/bridge/bridge.js b/flocks/channel/builtin/whatsapp/bridge/bridge.js index 97281fc25..f215a3e57 100644 --- a/flocks/channel/builtin/whatsapp/bridge/bridge.js +++ b/flocks/channel/builtin/whatsapp/bridge/bridge.js @@ -100,7 +100,7 @@ function rememberSent(id) { function normalizeJid(value) { if (!value) return ''; - return String(value).replace(':', '@'); + return String(value).replace(/:\d+@/, '@'); } function splitLongMessage(message, limit = 4096) { diff --git a/flocks/channel/builtin/whatsapp/config.py b/flocks/channel/builtin/whatsapp/config.py index 25a2caf8f..f9af7441f 100644 --- a/flocks/channel/builtin/whatsapp/config.py +++ b/flocks/channel/builtin/whatsapp/config.py @@ -85,7 +85,7 @@ def normalize_jid(value: str) -> str: if not raw: return "" if ":" in raw and "@" in raw: - raw = raw.replace(":", "@", 1) + raw = re.sub(r":\d+@", "@", raw, count=1) if _JID_RE.fullmatch(raw): return raw phone = raw.lstrip("+").replace(" ", "") @@ -125,4 +125,3 @@ def matches_identifier(candidate: str, allowed: list[str]) -> bool: def format_env_list(items: list[str]) -> str: return ",".join(item for item in items if item) - diff --git a/flocks/channel/builtin/whatsapp/pairing.py b/flocks/channel/builtin/whatsapp/pairing.py index 99d40128f..9136212c3 100644 --- a/flocks/channel/builtin/whatsapp/pairing.py +++ b/flocks/channel/builtin/whatsapp/pairing.py @@ -5,6 +5,7 @@ import asyncio import json import os +import shutil import time import uuid from dataclasses import dataclass, field @@ -68,6 +69,18 @@ def _release_session(pairing: PairingSession) -> None: pairing.finished_at = pairing.finished_at or time.time() +def _backup_session_dir(session_path: Path) -> Optional[Path]: + if not session_path.exists() or not any(session_path.iterdir()): + return None + backup_path = session_path.with_name(f"{session_path.name}.backup.{int(time.time())}.{uuid.uuid4().hex[:8]}") + shutil.move(str(session_path), str(backup_path)) + log.info("whatsapp.pairing.session_backed_up", { + "session_path": str(session_path), + "backup_path": str(backup_path), + }) + return backup_path + + async def _read_pair_output(pairing: PairingSession) -> None: assert pairing.process.stdout is not None while True: @@ -107,6 +120,7 @@ async def start_pairing( *, session_path: Optional[str] = None, bridge_dir: Optional[str] = None, + reset_session: bool = False, ) -> PairingSession: _cleanup_pairings() node = find_executable("node") @@ -120,7 +134,6 @@ async def start_pairing( await ensure_bridge_deps(bridge_root) sess = Path(session_path).expanduser() if session_path else default_session_path() - sess.mkdir(parents=True, exist_ok=True) key = _session_key(sess) if key in _active_session_paths: raise RuntimeError("WhatsApp pairing is already running for this session") @@ -134,6 +147,10 @@ async def start_pairing( except ValueError: pass + if reset_session: + _backup_session_dir(sess) + sess.mkdir(parents=True, exist_ok=True) + pairing_id = uuid.uuid4().hex env = os.environ.copy() env.setdefault("FLOCKS_WHATSAPP_PAIR_TIMEOUT_MS", "120000") diff --git a/flocks/server/routes/channel.py b/flocks/server/routes/channel.py index 50568417b..833b34534 100644 --- a/flocks/server/routes/channel.py +++ b/flocks/server/routes/channel.py @@ -387,6 +387,7 @@ async def weixin_qr_login_status(qrcode: str, baseUrl: Optional[str] = None): class WhatsAppPairStartRequest(BaseModel): sessionPath: Optional[str] = None bridgeDir: Optional[str] = None + resetSession: bool = False @router.post("/whatsapp/pair/start") @@ -400,9 +401,12 @@ async def whatsapp_pair_start(req: WhatsAppPairStartRequest): from flocks.channel.builtin.whatsapp.pairing import start_pairing try: + if default_manager.is_channel_running("whatsapp"): + await default_manager.stop_channel("whatsapp") pairing = await start_pairing( session_path=req.sessionPath, bridge_dir=req.bridgeDir, + reset_session=req.resetSession, ) return { "ok": True, @@ -415,6 +419,20 @@ async def whatsapp_pair_start(req: WhatsAppPairStartRequest): raise HTTPException(status_code=502, detail=str(exc)) +@router.get("/whatsapp/session-status") +async def whatsapp_session_status(sessionPath: Optional[str] = None): + """Return whether the configured WhatsApp session contains credentials.""" + from pathlib import Path + + from flocks.channel.builtin.whatsapp.config import default_session_path + + path = Path(sessionPath).expanduser() if sessionPath else default_session_path() + return { + "session_path": str(path), + "paired": (path / "creds.json").exists(), + } + + @router.get("/whatsapp/pair/{pairing_id}/status") async def whatsapp_pair_status(pairing_id: str): from flocks.channel.builtin.whatsapp.pairing import get_pairing diff --git a/tests/channel/test_whatsapp.py b/tests/channel/test_whatsapp.py index db752d984..ac7ebb89f 100644 --- a/tests/channel/test_whatsapp.py +++ b/tests/channel/test_whatsapp.py @@ -43,6 +43,7 @@ def test_registry_registers_builtin_whatsapp_channel(): ("120363001234@g.us", "120363001234@g.us"), ("abc123@lid", "abc123@lid"), ("whatsapp:+15551234567", "15551234567@s.whatsapp.net"), + ("15551234567:12@s.whatsapp.net", "15551234567@s.whatsapp.net"), ], ) def test_normalize_and_parse_targets(raw: str, expected: str): @@ -227,6 +228,60 @@ async def fake_ensure(path: Path) -> None: assert calls == [bridge_dir] +@pytest.mark.asyncio +async def test_pairing_reset_session_backs_up_existing_credentials( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +): + bridge_dir = tmp_path / "bridge" + bridge_dir.mkdir() + (bridge_dir / "bridge.js").write_text("console.log('bridge')", encoding="utf-8") + session = tmp_path / "session" + session.mkdir() + (session / "creds.json").write_text("old", encoding="utf-8") + + class FakeStream: + async def readline(self) -> bytes: + return b"" + + class FakeProcess: + stdout = FakeStream() + returncode = 0 + + async def wait(self) -> int: + return 0 + + def terminate(self) -> None: + pass + + def kill(self) -> None: + pass + + async def fake_create_subprocess_exec(*args, **kwargs): + return FakeProcess() + + async def fake_ensure(path: Path) -> None: + pass + + monkeypatch.setattr(pairing, "find_executable", lambda name: "/usr/bin/node") + monkeypatch.setattr(pairing, "ensure_bridge_deps", fake_ensure) + monkeypatch.setattr(pairing.asyncio, "create_subprocess_exec", fake_create_subprocess_exec) + + session_info = await pairing.start_pairing( + session_path=str(session), + bridge_dir=str(bridge_dir), + reset_session=True, + ) + + backups = list(tmp_path.glob("session.backup.*")) + assert len(backups) == 1 + assert (backups[0] / "creds.json").read_text(encoding="utf-8") == "old" + assert session.exists() + assert not (session / "creds.json").exists() + + await pairing.cancel_pairing(session_info.id) + + class _FakeResponse: def __init__(self, payload: dict, status: int = 200): self._payload = payload diff --git a/webui/public/channel-whatsapp.png b/webui/public/channel-whatsapp.png new file mode 100644 index 0000000000000000000000000000000000000000..338198d43236b666190a870bc6104074e39d77da GIT binary patch literal 32325 zcmcG01ydYNwD#<>i@OG2+})iJ2=49@0>RyFv7jLYcZWc5Cqa|oPH+hBPH^Y)epO%H zUvQ_Yt7W>XYr0RL^Yl4SPn4R9ECwnGDgXc&@^aD|004R^f&gU1m(0ku)at*Qn})0u zP&G=n_fnC5uP1M%tPC)}l#u~Ygbe`suggm!c}W0(kOu+~UUJZXZF%7T{}u$~A^dOo zzmC@XB=P_t3CK&o*761&w#zzO&U;+@thEb-pDpWp>1NuV$nj?`743d3VodlmY(ZF` z5#D5A`xTj~=#vx`9x9V1W7LX&L4S!|UAx!Tk2cWYFTWG6=arGRs0e9r`t8|!>fk}c z%Tp6^x5s}wId)3yTfVSagzpsnmy%UTu43^A!A4k{|7T4#IleF_F>QX`*)TuYObfm5 zm|=g&bwpKR(7E7Dd+C^~M13QYg>tlgrCpUtK1C!2&$YPQqraiJ`vH5va<60`Nc)U z$UAd$s-XL;BbiKo2j*Fka9;^0Pfz~pnwoKNW!28dh={L_e_Oc3d3hW1L=U-yg>74O z#L=dwr_ZxQL_~b}XD24Os$I8LvxWJBju!ah5q*ScS4SJh{c=~nH+&RZrJp9zq0)8N z!qUsT`di=mnM^1F3lQMv-#)03#JH>~FK6*S{fVULUfqzNpRaWt4eh4`4_d%QOl_>J zO60zRs|LIr9obb+w6%0}`0-z!(Xp|`d(lZ2v^y~gE-tQusq!5{?{Hqi3#HuAk^O&U zg6~{iU3EIYhOw6P_4P3zAt5o0um~_2S}U*tEs@C#X|Iv*M+C1)S4KtVA% z4$REV-)8IIE;&xwVI-BUTncNWeLyoN&$|=FHFAqf&RYlS6wiIV41kp~QdlsT3u?O* zJA$yU%wsF`kbE6kxQ8$Hpq)6rXKrq;B7PvYJIcldadNiWH23!OHYh;Da36~gQy zv2Zo-4Kedm4r}k?j*)C!!2Q+M$#Q$ET9-+K^;ixy+z>Ekq*Y^^=_o5HDUnHPmAM9v z4M3ia@si1Ls9TyTM#R?~jMiaBl zN!+)I`y%6iM`Cx?9ABddxy$?SesFR+`7^;$1oUz8aB_0Gy$A}i3u3(#DvUJZmHYQF zp1d|Ag*RB<4DU$l#@_Gkfo5lM;1QKXc|;WA{w;ZJZEd9m1qFI_K0Q^&tt2uNnbH%P zVF7D(Aw|lGxP3g7DRZET!_kFQKBeUfFI7LImWm4~9XVOSC(l`$hw! zh`~CLv(R!7!Vz>k6mD@IbA7U6%4{ZUqf3Qp_O;=wUuSU zunTm*1~&ONP3!fQ>^C1Fvu5kQ^^>L6W<5z0>Z=YTV5QFIe7jxE2lPOpe#0hA0=>X6 zSHL>8CQ!_^{n~wRf^3G`9vmm`g7hebXuOW?h)QLYDDjIR3gQ)pj2Ql^q?3{koqJZdyI; ziF&Fv2v9@HdVBk2kj=ry!Pu|MTj2cUv*GBD(IPEw2Q!r-4h{}98e}R8`GPbq<~p;H zNE6h*dUs=ao2erJje}J>IEO<(Fur%Vs`#zt{NweCY5V)36hxcj%q?fPT)n3VnSYC(e-FDw=R~q1K z|A0x0h7M9UlqssyqlyawY7eX12f;{vp21;{~loO z3K2alpFEo6&)riqK1~$Lk`_&`=o=c^DSL5mkR#pn>~Dm7sWP2&i?Y-(DxnZJ42F5f zA%y2khw?LFw1z?6ZU&0dA<;x2uOiD~5{CsYZ3MXuew}6j04!`beB@QM;#HQXLER1w zkM-^-deAfblAol(W3}AObIROfs`TZTf9BV2{o2{_y|iV_SgE`8dpdEb|0q&IJQ92J zB>U5PR6g?IaaWJV7;+j^B=vr)SgCMBhF#4Gb$RW8L4gCQ9T&x}pny;CpwDF9Ie0_Q zG(wjT9SwI_Hq`t)RKMr9;1ZslB}Y~9j!e|4NtFy?K-^$o!Rrm~GOqMLAxt#wv0pSN zLdZTFk-NSt%IWtA$;6DYz6<-U5h!$mFWg)u))1N&tL>%(6;~ec>o2@4uB5d!ebP&olk#*A!{Mfg! zyVm1Hx|+M5Z#F&?4JPTbUUeK6r~AAD+zBR{IkT<&E3{eYD5@1ec=7# z%6inUHvS)!YkX>F)`Ruk6+c}}6k&Zz*_?u%+@KC495-89*`~Ioqlo_*$oq+&r}TYwBy6+bY_o zHhhU=t{GOz5tfWtMk9y*$aO+JLp^lDj%+EUTdC^7`jq^taXl0l!_)3iMNam#WdGm( zOr_zX`vr2;=SUtnsWW>o=FO51+qdiq?%BX3?uEba!@pk{uC6pW80K$2;0m25#UOn6 zvUvRX<2pdO;CF^n+Bp!|;j+Ru#F;1{34k4VIEBpa=`1H)8^^h5l9ZhJXqZc6E4dKD9MGT$xYxFXd^|(cEWLjp{=THDXjF?xH*?{Sfu^Z} zHOUZ(@*A6WCU(ZA$gY4)s5L9%4Ni^HZOR2uEr;)YknOenow>@)h%i+CJE3ZMj+qiB zc9&Q%q|So6yq+i9s8wiZ=u1Sh59=SM#5Pkp%vCC;>m2;n=hD(Xkn$4==sCmZ1hiYo zQ&{3x(C7Zi!*EcHo5!Npx)(aPfoLl(Bw#~t&Cpn9wNQ7JSg&biz$$KI>$osHJoe$O z-*&^!pN^z1N^B_`^D?-IYJdKC5@C{_8OKEWfd&;cCzU8A^Q^#CGr#aHa>4huc?uZH znckaYN$;*7#4Abjrxn;gkzQbnpDx;`A3Z3s9ajr>ng$t%qNwD!rxThe?p7&nt{l+- z&@OV8f2S&TsTK9ts~n3$v_)I%`L)fLT-?^1NO(~0&QdG}@sv<^!J82zHkp&mL!PCL zr4mJ?>$fGcY~WY91{{uH?7?t!X4oT7LQq03(qB*J$VKl-S#Y6KC<8QF|K4AJYQ#)%DhEezssVE2lxXu=!5#`#Q^~<>h zvCdg+ID-Eu6ied4IiGMpDSCZ2=I&%h+FLrt^GU64jh%ceos4kKt@QfhZ*1|V@rLpC z{;Iay`_+u^;#0PW-OFE-xCO)2*>AiaTFub;Gs z0c59!KBnw#bTtnor-OYIeurf#DJiK91}A6*=#9eM0(id*V;=&)Ki7R1b?z}DZFBTx z@D0bdVm-jT@4&%5`ao{W(=2~2xFZv$Nakbs# zBo&;L>OGC@qen3QS>TvV7>X#)YUQ7^bwpp`JbFh?xQj%OG%|h>U!2FJ3mdW_jPCum z&dh>2rRdWl|05)OT7-@I1JjBnj`UzaydEN&E7OZ|KK-ae7RXxtbHz4!O+Ov|ep5lz zG7WuY zM3sM@&LNG3U8dMg?}aV+7?S3e2N9gI7L=D5-LoehIz(#nTD^pZli;gyPhM!f&SNA zC&g@3D}_;PVsun19N6wTN&3AZVeW9DWg;V-)4AI8RFoO+i3Dv0JrOajN#Ctx?B; z6WzERGNm*%ODS~Ht3T~Dz<03Tuiqz>27W@sfqa<>Cd(n=-;q8FTE0R%-yd1YP%LA=9aHWB0nZRYTy$N?Bu;QiQdyTj)K+nm+DVFM|S@f9Z7zj zjwHK@@dL&oRZU^GL=*fe5@_o(g$XQcq+c@^T`ZpO7Jd$tAM)-*7^((UO&%6K+JFM# zPYdH7=juCs8r}hB;z!~(3m+ZzkjNVzgAtyEzdlW*fgM3{0FJpHG)mampg5DB^yWKk7{Ec`}iC?$^glng&*X85F;~rj5_(y1e=E7EFo(l7{CNw;D9+ zG9%?ip9Gxv{Q@E7PJ?T!Z6sBZZ}K045yUIRDSVpWIu2cU{Pdsk_>;9SP7@t8?;hu6 z&|@OXFIa|hRAQz<`Tlq-Zz1~9OjLdk0=|ZVNQT(*6 zTq`Qw*Fzrr;mbrtcbE?rZoko2M58}%r2xzu8_l&Cjt)I){Z0d|p!!i$g3}<)v%ec= zSMdAsyZOTDz9g@boeH1UN8|IHQt>M%-aDl~U6W$qhvh(o{8d=^uhWk$OxQet|P5<}h~+c5M6=tH6uq#jjN2g z+R7Wh%WL@^ze3|a*(NMPpTN3|D+-}SZiVbSQ=HwL0Vea7dH=WWyo7Bmx7bhm2i+)B z3f;rjV{e2KMifQUW}|mhMg(?Hodr*R82*>Ik4Ob(nGi)06iD$MCsM?$@)g9U_z+ppQbr%-M{g&Ckqv3W10V-Vvx)qe+gscMw zYOF->$l^leV4z~uCphk^9J}!Wx1N3S^Y%;)piA zegXBzslQI{|NQM>O>xNd<*ecVAtOw#v74Qw^DH1>!bfz^p}&*e5UYi`o4Bvj`&D)O z*Ww=xGk#WeNjz34Bk}|mBsmN+7xi7{2_!;i&=k`jI~eM$ffK!hAL(vP6};+8SGugs znu~mT;8AdqBEMaYce%pg_x#L4_^DJ(*_RM~A8Qr`I^}0^88mUFcy8V&n_-Dy1=tA&>Ey5_qr}K%ON?Ys#goncb@Y9CGqz{?h!qRv9mr#(_!pF~Ey$ zy}zOxz~*Pc#lez9ot9ZgwB$yYqm`sEo19^G0VPLsHo)GSwPOE@l@5bOA;jj1q}A)E z&ya#6)dJ@N)2}OLgwzwA{&B95`;#N@EmlH6B(?{4bv7Zwgh(pg}g38R=o4$+Hf+U35Mib%hb)U&@oE5i-C9heMfxH%qLYd z>sW@qD*PuU6$tGy0_L-NsWy|}6xXq|2z41l)b-Wmx{yIh1+S8ZP6_uA3_M~ZNz*hM zsG7gA*a-z+{sMUicdmvLnV7LuDaibrr`}q5W?Kz1v47@oXm~x6bv5C6UZVG8+VH4= z$&bHmhr5#I;3JK*)7HSg z0FQ9eqly=y0D&Z@g^>U17ZpB;tlW;O9F3pH9@ebjE_RxBQATFx4*VSCN!iOua(J$4 zfj5AHGcn6X+js+Vn>@dkbqrX>xMI6tKB|7PkSWt}~jynu2Q{Kis?Q z-mc6B*|J|4*98^KW&8s&qQ3q?`0LHYCE{*k(b%^jL zpWxg))7hVCJ_eVr7|0A6u_eokvLpvHN^Vrkh9q+SDmz_9CtX4#+<#oLUY#wkFW+arEaodA}2i9FLhEZIc|ghLG}8>Uv=dmzF+v; z9R(35&0?6)aQ{}LZ`2iw4N?ymE{0{vP`%PYt-3`Y;D@ZjbKO9$oko5`#kUMevplFv zk3sT#&Dve2hi+@YBNyis+!l!M_uM(FJkos6R?Y5(%gO34k{soC&B7yLkExn>VcPnS^ zo*ubx1KzMsC2HIi;|~tm+;jcIsojZYl%UV|o}w}z*E?S`YA*?Hbi1^uJGS}j>DN0# z4H81nCjy9TDv3piiVN_tiMq1b@$OA4H{y~)exuTI=VK`7%Of&Ko=s7^)8GZwL;uqq zz<1N_4%FHwHeu(h+mN}^?0xK&q}<>hyjV`-ljQ)@o%kk+8WoeoHO00omtfexikY5F zlt!;`uHVevCFR|OX7!W%3AUu3tOP2O-Q{{nDIgl!ak~8#8nLceRsF+(64he zJmGq%Ay~FBzKhFY=Jpx+#tSW`WV5O##}BgoureEHs~q_I52z?hCry%Rhg1y_1@v0- z{d-M*m&NR^mSD)+uYF3X;4d=+++8y}uuv^rx3ur9(&JS!Z@eH6g=vTk^z>x7a&&&C zD^_SotME=qz_fx>aZBRnCzjswJwmNw*eSc_fxl*u@5fL4&pUTxZ$P4!`$NS>SltJp z6u)|?hB$0aiY_A#Ei2U!2``Jf%gg6Kgs-36a@eS*V2LPjqTKZw0Sr{ zJ!M+eu5<8Q=@ejBUi!>)R`~Q$xLoIxQd!Pn=XLGFoy5`Iu_* z@#IKTtuq_FaBZtUb0MH8$q{i@FZ~*NPg0TU0fU~y+_L!)wy1424Z1xwJ7s^gwtiiM zxGQ4>Olg?jDlvgl#<<&a=P@TR_SH+{P3R0}O_@xe87#pMbq!Bg`{I9ttEID^s45Yj z)jA$?k#BH7xEYKr$te^0`enE*E2?DBFGpzX7DhH}?Uz>Jl8R zdm;%={Xh2cAU;`w*^XUi=wimc{*|#R^8k9a9_{sEs zVYY)>eKkfS7M7H z8~u%EKBgzV-HpsUFBYrky&RtH?ADWcz7-dSVT&MEllMjUL@|n2gSwdsY_u`Us-^A3 zBs_$%U-lLD57+bYlR`Mbjq8w)-6$71LT)9RVNNOvuXpIPt_UlkVCM=6z{LA-u2#e6 zZn}TtBn%%e5ueztqDQm%TmI?iBv$wc!&0uz*Bq?MOeZP4zU0KKV^8R<2*WgCkc{P) z?#qr_b9uJ}q=34I+Z+8VG{(P*OKOT?wVE|vfHYs+*E>aMS-re-D=R1Qz`KMdLMxIN zAV)iN7js86Fk|`;Dd-dl>H27~Poa}290wG5of9B5ApuBmFOW8`G;}-(x@~GyA^n*^ z(1Ree^p+HP6gM(CXvftVMw;oIj`MMHjt6WFCd=<3WJz(7Pk-Lr%N&W@jDLW>-&Xu5 z`tYn)p zA(ZAip3Wzy`r5+0vqg)utk>#!sA2Zeu(l~yKcml~xb7%gqmtr5NkqDqPCwd(CI*72 z5q{(&&#B(|;(FJ>krp)#hL(hi=zNPixli8&jA%Y+1vfm=psT)prhXtQ|Vn0xUR!oX0hl3t%s5z>c#TA{g?z1(3l}xVOkp z+SsF;&xL_v0DR6`xw@d_^}F2onV?RR6TcJsntw-MM7A#%ozkvqF@@}DB_Ah6qgYRO zh|;*B*|eBqXbLYfq?|Y`n=G@CC3QcI^D(-`JxdhO#eCUHn2QKYLKp5=T&X@+?4s=W*mBIod#0j;?7yqiK2# z!ly}Syv?duJQHNKv{MGKwXF5eSL}vPF?S1)`X0*t=o6)chnwHee5B5o8%G*6Crz#Z=tm4jTDsdA!IHq<}htxq`W zk@nfrNuXDG+mj08YTsAe>-P?fG28@MAB17@4M~q zM`P(U7!i#qGBMitweQ&MJ4N)&x5ZK-5eAXs5Q|c@bFWLmGDL z&vq&EFQhFX@N6d_Tr%2+3VR%^yG_-P^qZ#$kbs4VL(Z%z5^*f~CJO}lO8 zCxtf+7TtiJ@^BU@T@Tck$yDdMd=(FgVy3!(eZ+bZbw!Y-E?&=#CCmla1s_nE!eudVmEsQd zX<;G~3(wS*SBrGSJMs8InPLC@X{Z=CtWi&^zDSh0tmCTql=Bm@nI8qn%8 z3A)g~UM0|Tb>3=@?DyJx3*cJ@o9=*pLpRPBzxo8;pIf~|9sPSWmD8~P2{@I_SrX;snd()e4DLQv&LQ@) zhHf-ji1~|45_NgL+j*ftBsPrM*C21QB;9yMfyScDbpAUUdxKdpK{q45^Iqv`>PcTX z>P_?gpA?JcE3^yaz6IZ>PSgJ+?f+%JaO<+e(DZxHJM%AOaFzC#2U6lP;SsN&6!-Jm z-S3%>Tr*9b01P+-FW()I6zL53RDCKBOgF{?J!nq~+fOPSnHS*3IH<s>Zkkv$U z1Ezz9C9kPK?bQ`YxT@?A{uSbOyJjk{wVp@!7b?MbGt112_p)?|yo2_~f^f+%lEOHW zkeg``)L>r^RvA=0_kwY*o&HTS9g|Yu=S;_mC)(}OzS%&P&fn^?OAf^1llY(zI&VJ1 z$K(1IeNMgccQirR*Q;`)Xv?%#IX|qw1i0P@qavO^<5zz1juvYqNmN~O8#er8L=#b_YG)#fSFIEj8vI)i;fHl1x8D!_D#(i7G5IYTKh?UB{5@vWA0&g0n@ zM=ezW<_vjFoiMAPFdQ!c{BQ4i=1{+Zs~XgYm(J(Hu2X#H-}?3);nxKwu{cM!G3jcd zX*UD3k8d&Z_a10RigD8+lvC9C4y|4%RsjGeA`Hage>)(2ru+jNfa9n;6$k(aMN0q~KPryDt&pOPbDn_eu7sH`#Lp7AK<*Rx1aKktfn!qPxbfH|d9S znMO%39jz!SPvuFufRGwxw_m%o$^3x=nwTi#MMwR5-_?ukPl>gZ-EJc{EITvi8~C?*4q8S2lg(JDq|kex5hqXfLC=lLjPFX|4-skX8N<}J z+(u)F?A+@*i!y>fjPkQ>965htG3g%UDV<2GFsjJ=MT`Oh1zE|6RgA>rWuJ>eJ#j!5wA$^ku%zWQm-1VB!OuxkzO3 zA)LeCJlAM%NJfqv?3{#@cAjj6*dZSG?@M`Pk5WHNKc~S;o2u(DMZ2ocarUd+z78n8 z%wEJT$DW)-ZN1WyhS7==#e8i^6z*c*3Hx&n%NCkEmT zN~2IPNYZ9RLAwGJNFEB29RWW@(4u1Vc2fap)+k!$J-UCmLXC=1OH291`zI4t+7pbg zs`O$6vF4J)`OYd@Rk$uRvJ%!sF=>K7{z05nL*+@lg&@@#1~a+{jM9Sp{S^gNqFA^0 zNe-G~eo&i~%kyI$>+gYNMwyerWGx;Mq0O{z=jQ(AB7*hwyUTs*Nm_%g3*fff@DX z?Qdc)e}mWj1SInL?+;g(2Ta^_2w6LE%4xyaJT)&wmHT&}#dHc`1(+b&x!bHf9>ODAc9~&R+Ot3oq zb0J6xPPf)Y+0jp@u7eykEiyq_s`U2vMzNk2wxNGn6FXyyQ^!iz=YNCB7eZBiB6gm7 z#t^E$3JxjMqjFko*I6S;&W5(hSv zp^NTZld3iInhHSmAn4cZL49$|8B*_?vK>8p);@QTZSB>ZFD#R?!|-h8!%kJajVC1^ zp1Jz+V-1JGRDl9yw6N8;6%<|8;W|*?vx}?HG48sf`W6S3H{nztQjw}63oj!8h%7YW z(NJ2Cbj!e$YHZ;HB;LI;R2sm?)Y`P0HU%vNg$pdxN^7&=!1g%HYLqlzjJJ$TQnLG3 z?H7Uv%(8Q&{-=ZCNBc$mUm4661o zmEF9R?RzhSD_N7QOK%i}XTxFCmH#o`+AKc?Z`F>3R>R;p8|DiXd+z9^QXAZUe#cbh z%fkh>va%ur199XJblM|%*l?Dthck7x5C2p@=_MIkB%5m{sC-c7u&al5dQM8C#>(RS zJ(n}le~XiUYtQuYordz4q=c%#EM55qyXnm{{mbIZOHRj3o;fmqFtYWDfPCi+HM`&W z++13?z}CjU2ijpw!Vuh7e zR2N3x=&|iV@AyvYji5S*pri%+Cz6FThn`s-|2z*+vhaRGf0a>zXa0wy3^xD<7^2iN zKwrWquAlG;yq|HHXqaN9%Vh~@mQgytaUi?oI(owP`JCMV> z+B#Dd4YWZQ?Ht}`F`Szg3WA=IC8sEP%$iEXduR_SO`6*BARid^#8-6H?Ng@9O)0wpT#zgo>Qn(`2^w@J_(aA$1w_3T={6jc680@c}FV zGjTpr@{F?_y0^DQAlegX%WktPwqp5>3!rlZ|Fm34V@6sXbS3DY_3V9%oTorA%n0t^ z3^Q|(8uPB!hFEyw%Tb0z(goEIIg;DVL;P(I9f!x}VRa7Haot|%+YrtE)P2jKPYSw_XnFhY#n+=jK z^g~tPw2+k~hm0%ND~E@&F{lE0VD^{tsQn_({Md3Gw^;9T9YU)ovc5H|(eC$E>PK2^ zcxliQDj7VquSNgsf77nUcuP>Kku0S|upBX9R7ll>1v|iH)A}WX10CIZDks+e3V?xm zqy09^K)-Xop0`?Xc@Y<_{M~%aSuU-~zbp1fAZelDr9d3G(jmCsBI_tx51+OVX2!8M zfdvM&F!d9zLZbmK=6zF~E0El`XY$OLkGv+kMX_p-**a$y!mWL1YtgyFGXa}HhC=I^ z!v~8wquq`VCat$0zn9;f3DU4>iwe-fCPw{qqM$J(dJMMn;)Q+~}Ka zIfU=eL}qCcMGes&BGY4@b3umL&Q#^NeM9atCo?P+O+cmaKYd*ql)j(Z5b_0-o0h58 z7xveousml1WzB%FN*Yr7YYFXsM%eo|`T!vozp5-MRk!}+!iuft@-`rdOHF#W@9}^3eUps8UROd|)cJIB z!+O3X02+9S%CpsrC?1oO{XrEVU2kF}fUI(RK|_JFt#qv_qz-IAQW_h(b6LGg%0{HT zI1X?8j8w$?tB7i5PVLlHpkY~u_w~}NHk(j6J?mUK)q|K*j~t82kUabLmQ=-q5+uDg znZIbA9f0kJobDq9nbls>@lSylKGRQf1fZ5*j1YlUa1=;8Pzb1@M(vspS9^H2FGj{( zjbEFqEDb37tu(4swJ;G#QD=belS>eJ{!vDK-w~c23LCoU*g?(*IaYLb>s8<5M{Ic= zxjDS|U!$dQ{CgbUJ$4IzLOuGqc8R!o5XSKOWmyTV(30MPhv4eK-8!&~jSqK^JSghE zr@j$*2TnRL$o{f+`rC=ro@a?hOo#YY5u;jsuG9XYt&)sMo2YCVVdRZ7D}9|3817N{5I@~g|Dx8eklnhMw)wl*RT%mxa} zR$-XGysGoD)<*Rzhw`ctE#8f)rn3a-jx;{m1;?--inP>wew{aH?3qfMdF3yD7AadQ z{1TuG^y^*n%hrC}Ekh4zKGsaYm-x{h`Gv9l(G|!*@G&49?#-9+Uj2CPwX?f!@uUI; zwV*SNQzXciszvQ@u^Xwe)BvpQiW zT(o9S+^yoaR0eQ{uGdf_!%FL1Z!J5JULv<&0+)g>eR3kfB8WoX^*jmpYj3_zYkosE zlMl{FyFCL-)9f)f`0lBznn#tq*BA=(GS-@?4f6fL1-|i})G%G_%O9IN+UgxwcKfkW zq4#5;>jBId5pU{uaGhTGN%oKe=n3yp3{ChJ*W|cX-x{}+*Vs-PgJH5rQmp1nhDiUV ziF+mgX{O?!nwb$4^9#9)#}IFfRU1-ML@Nla!KKKx==Y>^abdw-I{KMLLx5nCi}3GS z?7d6N9r;NS5(64+%f|zP=b+B`uE&u+rSK3qp&U^SEjO13h-$A>EQD1wug#S;=r9f) z)Y*AUkrCtr73Mi(_o-fLCm4INU;;;H(%;X+ru%uy6j*22!vyLbpfON(<~@V0(Z8tI zk1ss7rJh~i6_0~@EKe+E_1>BN%I)vjW9cQ)R+NH!a6hc1(4)ZHXuJi%w zQRh1cmUCK_8I~s16|mbam)Ypg&FGaMNH4>srUoO#2f3Cx-e`e863Q2u5q+AXJ4Hjf z5H_qqxB&v64$z4tVTsL+Zz&f8#;JY}q74nl?H_HIRjQ`3C1W?kq$NM z;JX>oN$iJPM^x0D_r`-Ci`1U)Gvi)}Ym@x%dt$P>>Y90i5G0+SxlMh$=z6K7s@|IZ%S7cAEHqxKMv`E`$#(Jp%n7|9<)~2>jLzT2N^D16 zlN!CM)(4SgJaTIpE%3p0*V@Cy;iv7#l7>=VTJ|D6x)^7z)!N>|nRjQS{SK2~vlAJ% zVrphAG{3>)EoLF-LL2FlsCAXMQv0XQ;*-A)4=YxKMw^d>kTGEze%fKE%4J*~qCWkl zaw=iBPZg2k_nhHlz!O~-AW+nYE!i|Ea(YSXl-@DX(WQ%4M?RPTF?5@ywbTAvBM)aQ z0Y@!r(Rk6{ujvOMWE2Y{Iom7Us#kCuoFzUPK=G7*4!Nn@)J@9MsimMoal!*U^P9j} zlw&4CY?Z^-?mUs)fR%S&;^EvPE=Cvd3$SSynsFW$ZQFZ2X|zoC7Cez`^gvIpt0r~L zL4Iq^J0{38)0pTMABCgnF$Hn!D90~FhV6I4EO}{euEW3VBA3tA`AV-iC@e;p+2g(z z)OQ@R1a98!#CITPb>aUkthFn6@pp)GHNJ1eMPXL<&}URCc=L*%CpT!0#9y()c6jKEhS>7QE zFwg}^I+d<%D*Oq>kpB+uuC(C3Nvvf`oBf&eH-Y%Dv|;)Qow>|0>a^Zv)AmwG6Ef1K z|4-8N7wnSaPIK7F%yUEvLM9?CMCjMRgl&HSZsG&z4#44T?SW-{nNflhv!b3gGR71PAeVLPo7~)o*O~ZA$e=L73)y!369#e(4K)%$csI-J>^b&ug!tBi8yx!( z=PL488<;?ACmQ~G#DG!q6Q4}NLT8#8dQgo2_ws#xrETWl@X#$@M)x8Y1X%@j!8O%JhlaoB(d~d&D zONI@VuVBj0(tbh#0HH)w!C8sM`J7t^EQ&LUhh;cfnD~j9QuK6oO;xKHW zt1}W`5_`jd|6x13SxMWbxB^UI<)*BdM^J1pSZduQUhI~Bd-+MXF4};Q%G~t!b!Mh(_U+oCw+PqxjaXNju0Rbs0>RG1AXVrWSEQ9U66rXY> z+_r~X9zgU(NNMT~2}K6In>`KM-w@clph%a5Nl^zaiSLRBw$?VuUDXmuWs?X7k@j)U+=aC` zVCH#3GfrXG4#|ciSA$17PCp6t?IA<1ww}iHcmI<7Nvs;B;(Y0~tCM@_z8%YN5nno} zNu*OUKC8{$Ndsxm);%s*Xdp@FLPN?&5v%F{d@?P72(?{TDRx&2wxsf~hHi@uQyKu} z0th{&T~2wDaWn8)+?hfLu5t2oi)lJvqxxGDq3>#R&TDNZcoHzPFgCm1r)02|RJ5>i zs=&n&P^N~iS6Nz3cpK>&w-X;!hY2dx&1g`wl3qOPrAwA+79k638|sBWMC~PF8`I3rWjAiB!LaMSvAO^|33v@V)Mh+RLf|I{$A>H;X3&%3C3i2(b#ZI4q z^r=k7?(XiZ*|iG5#n;@YQbPv7-$nJ9y#bq{m<62_VZ+N!eALtRHixMJT~C-b=sOY9 z+)?YmoExRWwf?pIq|t;W#hBleef6Rv@-=}a_!@Bk+$2ivwMIWVcN=wEbN+@$l&|6v zrJi`B1_%`VF<8KS9_}zUe`ilXBx!LbkSt&z?OpQ(_p#kaaeFNS%jaz8cV0pqvD`ox zWRMtCGL$2HLZSLivLen=@U#%=XDit~!}2_-ffRwsuImu+4H@=B(y#>|j|LaPKhbH9 zI!>v$Wr`}e${lbD7Cof5KfoG~DX*0`>X=5^rIX72dDY={{aADTE`MI9nQwfnohlpt zWts|K33An{G7&CuApG7DbER(ZHJruqUh@_59n`jOA^bN+#Nk4CN^}aHFGV~88QMB! z7G*S-bl6o1CJTEbX5;))+I)oihap#trcsnwG;LMu`v*@9he{?y(9JnI6x0vq#J&PI zIDh@I8j@>P9cG*JLU$(IWHuX<0^UqcuDb=43{u++cogx_@iMis#p)6ROA8Ob@Zwg? z5Vmk?5#uA;32=$VU4Y2LduxWy6pUHjDLX7cCvvX`z(e*()jMXsv~vlJDvM9^FouT< zf^p>{N3PT!__O7*XpbtaSOnYS%yw{)g41<@hDixz9b;(>6fH z`oq0oNZ*+gG~8TjgR~r%_(ud9rkF49Xt%c~|$t*HzKpgM=M30+@b zV?~J(E0iK~nOGzgkSps(iwJ%Al)D+M->TQoj)_c#IM{$p^$=GXI%tk-I&~~nWLw*h zW$)LOfnxqC^etFNh;h}|b8)gn9!Lq=aiGe}w32(%V?o_;`Wn%)rvhzuLSwlgIYw6I zky2q!2xU^wLq7R?^=6uop4a#b@qK*3=k+4!IMLg}B`B4M(IuljW;0t}ihn{otL2bW~P}IZ2S{=frel?Tt$l}2PK&o2Utqk!VCVN=c>TP+0_95u9W!C2qGdd{b@(K;1+^6> z$W2|$yf{d^@Ai+HkIScN`mHPGy&JZ|jOq4~Q2KXDjy66^{7+?P;n3v!wegJxrAv?w zX%LVQgdx%)p_H_e(o#~}01*KxX_S(d?q(yDlvO=`dvPc5Py0lC00|rW+!rLR77B!2Kz+PdpL;$4iuH2XsEEhnvkWiYsKO(k&Q!^g z-oVrV&=9)l!zv8(b3*(JR>b%{rAxQ7(2utHgVyxTA$R4fu`Zecz zO*gWax?f!hn;N_`sJW&C)_2-GkmC$>h$7(Me@wrUItnl*Tl!MUsJ^6lf0hu?+dx0L zfmsNVwgTD{tSzZ#-Ixf?GdUKs8Z9=sEf_C;e)ytkIBQ(q`1OO-j{u2?q&yGWiXgJ( zjH%-Kyr24D4tw3sX09&!#3nV+&1O>E%8C`B7uX)&^XVW-kQh$4?a2b&!dqBaVCcl6 zk;u8u5ELzQ{aey_<*H^b1RLjnqJysJgU)%bno ztiA@5rPa+`*)XP8)`lLxc!(Km`8crO4*qD&{1a8G=KyLO*nh)!TGikm;o=i4PS(ij zHcK9Y*;*f2o!vD$7TkbbJJ>d+Ip=rI36p76>e?LA=swF&S^GAQ=xFZ$av+iT)ZX0x z^AqEyZN*BkF?vItn1kRkH{|;$_BtMWZDKG5a;D&LA^>nzzlAr!&_{0OP3{UmL-5{r zTPPX6P)dlA=k|B2j^VJIC?KP}wWpM*ZiXV(f~a%1o3;h`?EZ1i??;_XzFx2s+!Xi{ zue7`mFA|_cG@tM|{3o3P*XAzYK=;hz^tA8oDE4j?GfLzdA-6@3`&N_ z8^PA$cJ*9EUFm_uzSgpKLyXru=X|5<<=KzJf(#u24yT!|-b&6jY*4BPw=VdzGp-aL z7vpnY#L!iNC$FzR#>+xQLp*Fnf-k#mx+N9(36#wFr;MDA2UCQ?Wjt0~lceAxuSrHd ztzvV2{f<7S7mC(?JR(>Iyna9Gs!8gveiwKX_(J4LkWfUqdYON4)dy0Q- zI4YZW*`UtV(_l=avFw`$<6k>1lD5T~`ueW!zprxKWa{_T{>>1Re#gMS2$^kgnPtKA z^xSc1k>^Vep(25X@lUZbkhY(V$iIlBWbeE=9TI!@bJfc5mS}U}gRNKaANArsiPpy& zzPb>a+(kD0RZ>g{D5u}=$P%INkROL~ZYGxZ$o9;dR`<+N&)o<0Aza!Kd+33?ictAAR%@jbnpn|~Jrmn@?r2BoMz`s41 znV9y1aUc`^_$|}S7Rh_FTmNv%a@22Z%5NZtw>|S8E4v_1i*Kjv~a#_$37Y=p>__4O8{f04UBi+%~U?^p_Q-8b+q zxyf}dangucJkhe;{zISutU0Xin@yWGQy{7wu3k=c^puNDkr+G)VTqDm`r-A{^mCal zKJ<7p`11=qCZ6vF=$y&*oacM9f%bTI6}ki~K7U>P)w|7zLje|fD`)LbQoQl;@oDAU zQ&LinG#G$mcuxk2>!}~nVjErCMO#9BU};&Un$nNd173rm0}VY<#KtVu7u$rQ=Px^( zp#g5sesO4b>=Mw~el)&xGhKN1Juc((V4wg(3on*0L1Xs$*1ZDHp@N@ciGjZZ)uHcX zg+J~_6q3lyFz&UO@bm`Y+@u#xC$GX(D$Je&mQ&p;syaHdSetP(U!CUDIFO&05|c|) zsr=(P;MIqF?zA(f4WfWlZ~qI?SA3V`evD8rCE7aX&Nfa5cjvaf%?*+Uy6VNC>^k?k zphVO!atGnyRnXq`RJ+%iu6Gf-e2?u|n)Z#12j9nq_wAjMkNC-iNV(6{&+MiHl0R=t z%rx7um&@DyrUt1xeRU+bndr$0m z?M8?OydFwFKT;q*v{ifyRV9%V*K#lLp+sHU6Vlz}xPT;)%Xjdl{WiBpMfwe!hUIJ= z>tpHX8ez9m+5nJ2<_Xr3H#die!bXi6y1?^Bo?tZr8V>S*DH|kHL4&bLVH9|Vch{Po zL~pad1HuLfz0vOeRB`Gytga7&0{h~QB%W8Ke{;W#erkUR?EeewXe;W7TVL$64%Ik zT7l=1J#nD;PK?L~ynebhWTJSx%N+@T%Dh<{iHUuAB1pmt6&my{E*op#OvympZOfoP zym9hybv+ak7GB!6dbzE$TVg3C??#ZtTmF!@VwONBC^_sUD0*eBLrz z5=dnDRBYmR>MZ!wVf{R!d34H&vYRG?^{TGFm!Y-`!y!$sOQHIsGW|t{F+r`=68bd$ zL>mnR3rxv_uy! zYxIG0#_Chtg6T+i60*Z{Z+iebV{D_!={4$Q5tpu#;Fc*z2q93e<-~5(9`b3%OXn)c zh?1tNn5bus!otN7BpgoE(CCkefZ4hU{CYu+esY}B5Wyk8jQE)2aex2RjM0%}*;^QQ z4+^CvJ>(T*j$nK!?TAv80kzy=GiJ>ZPtxS8GD8g*=*3NrmkAQCgq}nk8n_hUbCA(s zXw&u3kA7k;T7CAsIiX%#YgACila`lqgnOFn8MWHR#0(l3>yM`cd@>&MwOh6#_wn{V zIk?eG6KZda`{s$&D?g6XtGhH(59R8hK7OvA=lZ>e^2O2Yy-QYuU`{-^G38e?_NhM!y|4Iso1CpgL1SZ zN*ZJzg%CH&j=Gv&&zAXk1N2H)z7>A;uNtNyVzZ!2?YJppV)K_PjSYBEaki?6;^ z@anDeRr5YrgPG~hfmS>yq*x{75B&PQN)&5TfmHd zTx0qDv!LS5t2LRmyZi`Wf5>T4j!!=+2?`c2f)lG`RUIY~^%7|1DZdsDBf#Ms%m`W| z2u4VSCZPqihqR(urE>;JqEA*S0s2>B4l`6-RL^(g70i{k*j{VpM=VxfS#8SX1oAJQ z3|Et9Tnsf@f0yxSw^#H4Im!{qSH!_9H@V}o`FZ-(bn5S+u)kM^iagp!D=miYo<4<1eHhAwf4lLsglWgOZYDgx&VIqL`qh*)E>{~UCC=1Mk+wZ>ztC2lg^eHk9&4ycN`MGC@jL3Hus6E-13W!so%ew zmX`iU8Tfd{31u#5K0NCO zD@cKS4~bBDCBOaqS$zK`R(=4R)7aR!J`6aKpGOQ}HFZSaw5Lym0{Ru6`7$O#m%?<7 zjEuS_^*|lk&y_Rp%E@hVNnVEvcxs%AycYTX%20`YIpOiDywcT6L;29W%Y8CftUOX7)ItbNBO+*UVgzyMWkX)-hQXk&R7TR83 zL`L8ZIrk1K^r*%lVgNrx0R#!YEC7Js&lr)D z0}Bpahiuu<@ieQzy^?0=@>}^3P7b+brzcH3WM2n?kIWMRtW+P#^cMOzx5BV&9KKNP z`AfO|r$|~yFN;Sn)RO;>*3i!*dp|8~gns9!AE! z-C6NS&N+m;e+Ag?rrs>J)DXm`FXbw&kBzJ+9Zd7@w0Y5ba|k1>$HH!L&W<|-7L72W zw;ig%zQhm%AM1C9<_zBADJkl3l$K?@rxxe1CKuHI0lh(juiE5)m6e%mX%&p0CBmz( z395N6TBV~jq#bhOxoB^BGHH0u4Z_6cHC?+hn1-=q8KcVt-bo8t0jiWo0(XQmhmg+j z_dj`h%cI*|XHA=yf0dg!)YX(4co9Xbz&TgB#&`0BZ@w!ye+V2r|0)>mGpl;mYOI+( zd-|_)K(TaahbtNMRwVDn)bQF7Y zy{$4S1RNT(D3yRBUt#Q}a62QxK9V9qx05+ltT^{n_q(*z%#2{LD}T#X4`?GtzqVSF zfP$_3^36eiSDEgY>k7>i^1y)+6BM6B)XjqKlY7U{;yej1!kS|}q_&Rt| zgYlQi&LS3t4ZZ~0D?S`!I_Ub^OJI8+ZGFG4BQ^R(->Ke9z09`Si=5fFC6vD}>wF)C zYR{`oHPYHpet)?7m@nrYZdQ$OZCRcqkqyM=OHJNcRcOqq}+NeX!Kx?Sbio z-1Rv}%!c2p*J1Us$y*2?Z$N!az>X>&;RljTIq==btY`S~E;MUjH@6=l`I4(^YRg=! z9(R!I3m9-rIu}MITm}MGeV977D_iUgBY`ZdS`Z*Q8dg)juAU;K0$ zJlM|X0ctewljv=pYc$Vb-^w|-w|K3u&{8m7FCUuRZ@N*|t@wG=EokP@gySKMbPHy& zGm7p3-Sy~gsmvBq*)(J2jzDB^+*Dm?^WkwoT4!0Qd2(feXE+{-O3EeqoW+|r3lG%r z+H+oq;nl<(l1G#|2-O5hw`j5`E=I8)-=_-o{}u_xr;6#=3PGSm*Lf2oTloSHUv@Lz zilw)4A35MeFFK=6Z10&zKBciI5w|fH%LDK7lW?$~l0J(AHz^N9a9{8g#=y>qXG`0V zlpCO;3XY@(kc95o!mTIVQ9KPGeJlcW!rI1dD46(d^LkP=1KR_-Ndg6!k9d>WRdZeF zX<~{S_SnP)`+;S6Gs@r$YU0NgA%ESERyt4$oM9oYCO>;5c`t-QRgyW{%SqVdB+9&> zc{_Be@GIl#o?yvn<(KUB)#NkwoR^#E^cfrD+9PAcmGbHef37&^tNVQ_Aw_&H6>ie} zVqAC1WD{BB74;_FMjS@Nlzx4=NblQJ_#w%;($%;eMbx$$p+Nq0$Bs{=p~os|32gkI=5`rP9yuaC7BtqaehMVk{E=vYx#vi~o-d5G zxC=rX@ZF%Tp9NhW2B=~lef}(Y^Yj;z4zgmPQ!g;If}1H%dY@V{@-G}fx`z_P*#WDe z#0ydy_NVvw&5he+W%P-{oGE2kYLg_ENjMUtsEP-j16=vq0v}rYe;V55SFl+<|IBK_Z)0}>NEMl_Hb~rfVYZ#>!L_8cq(a|tcU%79} zghk-yGuE>wKj>&h2OAS5(is0P^9sz@xHE7D`b?5OJee$>k8h+o_f`?YhziF2F)|`q zz2X9h(U=-q!$1>J0E?`22MSVI^6g<9M@)SMj5?CXAr@vqh>jrt3M#m9M;d&r*_ClL zat?$u5_eo)O%~nP@l(d>AYPd4xA4Y9z4hhoct{lZo={0e7VkJHJ2HXDBSNtTVW75~ z3h)`pw5g(q%`AH{TNp1;z3A0NO%;<9tMX+--^|c?FlOk`3g;W&O?DeuUE+d27>TD< zipMH8-jMf`++XmGwUL=9=7Gr~Y zm+TV-9`!;{xDxSaZEU7DzOdsMIzOBaNG`itG&K9d%DOPJtsWmbcA5VGTHAvrwl^o1}@F2KB8u_6wX2v)r>S1}#7xXIS8ni;~^ z!fS}_m$$K=v9O-Tc<8`iC*_bPzRw5+Q$m}s@fd#fz ziR|tl_^^S|6_t26t|kKL``R3)p)}%2uxCG z@%~O8%%^!!u^yTwNc%n@RxCGcXuy4DMhwE4{m@JPE6cH48* zU)13d99ioHRAC&h&bv98et}m6;T3B942e6rMI^ zJ;L>bk1Qiy4}K{qF>bWr4^y4|v%3D`UQJAr&#np%LXhrk077aMIaiyMJP!IiI))*r z+TvgjaH6td+gyIE3pPFkA+bo0q_j)W`ZkYjEx}}9FJPEik;@5vt2p$|{bAYbymXus ztwwZVrCtu{Vmp~NyKvhgEo!IL z%wpMBFF4+oJ#cn;@VLAr*To;v;b^_hx;3Q0_h{~IGqFGX#pszQP|0@p1**2gMU^sW zYmj_7?q+s|^DbLeuDuXJ(aa5}I9Vjl-f8bs_$L3fAlJMy>#`*tGN$&Nwvq9o6t_&f19=Ae|Z_9MN0F@TZt9DGx+s#lj0T7WL zL%XQ_Zi(INY~B>pKhdN`^|@1-88kQPQL7lm#DjZADt#VJb)y6+*$o3}ykJoI&(S z0AU+<%y)JIt2v45$V{DeN8CXgi!M6#Oh$A?;Dh@JuLT^YJtY1D^_e0GEH<{IV`83e z(+fCJu;yk#%Z!6=Ubtw{tfZ$M4*0g)0XbK+=P}Q~l5Y2mUCW>mcEHc~PA3&s;+HlQ zugZ1UZ#k52E|RWXAqVFEM|WSl*}sLajEe!^mL&Y99pw^84!JJ=;0^29(x;S>Ib<^6 zIsLY?aHg11F^AFUin64O8Co*0oBgjWe{hjDXCl-oRV;hIhf&nxX+U!s4!NV9TXuo;F^nHBrF=0bGHJ7OXp(?5IASnMz0sB5Aps!?9f=xx8pE3a66+wLKT+)-r>?ez_Z!QRAO7<0JcEa zgZ+x2YdqN3@RaQpj2EtosT|9#hvWHLs|Fbxs*A^GQdjF?3 zewGX;-sG?6DN5S= zp?%u?u%H}CfOb}g<8hy6T(`QFw=XCAHx%n_!?pa~vxLjGc3-tgh_WkoLnL1FxJr9q zN~dw|j(+dz@V~_dEVu}hnTW>+^RdbEtd{Clr@6!d{0`?55lc%JCHXi^^maluOpHKD zKcVd%2UHbpJ4VYnSE(n_Bn!{EY*fR~V&qiu!UvS7l5uK~OT9+Q_fl6?9xEtb6{Y`p zeoYwpqa9fssH>oLPkWg5JNqm%)^QjoDiW|4=&o|FF%Qm3#b4r^xVz$iF$}J;!`9=2 zVl^|TxAOtt9xgn?ivVL$2*lFZ7(DKI!#W^%+L?}kBgg9}Vta!59Qx3}#HSoFeznv# z?znQ+`S8I1T*mpD`~);}TJm(|4odzn3#SK#xne4cUuFd&|GUtQ@`F5**^6(8Pgqa- zo})@?y|u1U}x;SkWCt0XlMARw`Y!=TJhaxE!V#PN_6%u(?VHjSTIt~WoqKe zrF&$Q>hMwC(dIn6117ZP?szS1sMB%gQ&PzGO;+6Htdkd$K}cscI-sql7{S|FIU;fM zH~IpJKCBzHy4ezg1Xu}SPOiO^5kt2<-4dsyah*b4h^M$swufkSv5o;t#U`vLPjoac z2VKTw<^ssx2+Wjyyn6TMNv*Bnk2Sr%0=-h?lb_nq)LY3xt;$O6>_|4=bD2|1S&&CI zseQtxw$Z7eLT&{vO|S48GWJJ7Ln3B(mXYpWow@;7(fBJY9Yz z7!GeP5kb&qO_Dnhe@dw<>2*Ze>0s~?iMI*4ivan zA?_&eZKx}HOSSW|%6GcKaT9zLHD4N`$o}OB#6*0}y4sgPsvWIyqh3S}EyVot?SytJ zWnv{n0Ik-d@L2Ghus=Cj9#Qu3OrFby1d(#fNuWiP$np_i^WOO9{B42XXMDAKg0>wN z)`Oes!7y95-J8zF3%Sz5PXF`a8uZJXox-~~w7;3%<_{bbf*+>f0)i?(MJJO0r~2=I zxyq?x4RfI5ijIA*k5}17M|?f0uvgaslNiVSOV^IuIH8h0N%Ncut{md9eTKHGoSEw; z%p?dsDOPaZ7RRQPAwPr5CP}=%jB&Rq!Wq*T2@Z3!&Jm*EVU;qANlAkd`Ye9C-=*@nYJ%mED#c_wk^xQOCiE2~95W?Ka720aDy7CHAB*f&lGYrmZwrf%3-0-orMY;OMW z5=?w;H3)m81>41p65yTGVgapRz_y_lkni@GAf@B1&hAnw{@Rdo%b>XnTwOYMJgaTf zpJi3=s=(SXP53S`>hfJ2kLR#x#EF9@a?(7f{<%VUG}rH2u*hkP0AG$j;a!eXUcC2( zF9}EL3_G>WU&BXKfLL!04#;S>LO-_f8}_?B*)#r2P-#D?vUm=DYKYx3GQqa^&mOYi zB$FKy;y0FN^B+f4P*Dpn&a*3`6|scQxAProUpk-j)}SRPuCE6EUKs(gk=Ro)><8@t z{OjW{e%Cci%KBTVzkIJqptok1)}LUzIBxhOPPo4yz{&R-JaNH3<~L(P7Y=?C4YwG) zP-X9W?vwO_(o?osmC9t)_gQ7%jW@oq|=EisvtK*{zRQ@0KKi0BO2`iW`l#dyR_aShz_%UtXKg^LwQ@x8iVpr3+C1DACl z;@#FSr;ntp7ci=IRen{a0;N&E4BmqhZyMl^`f-Zv=<{L_dN~KPmy>N0X9d5)gUufH z=|oAI8N5(VwAa$GxZVqCAs1!VxZ$f425KUS4w?Oep4_8`mRTIPZ!0Jb zV~PvbAbtmve(cIkx&aCFNyCWJ4iUt7k#yDFu|F)F`RO1sLFvf>5&&|6=3q2%`H)P= z5M~2~n42>#YKC0J+{%=?QoS%FML_(LZTVKSNri3<`P8YbPRb*4HA#pjiAPmzy{`@W?nZ?yW$t zL7At;`tJi3b!G6Xr>u>{#y%jF=g~vFL@bPWCr2oRzvc0>Cz@m{C54YwIpcTxVK zT<^>~qi9lV6GNBiPk=f22c@8yaDp#tPl{q6KRx?SniHkT9$XeaBu;xoTy56Xr<}bk z)m9&dOb@8uYc3Tlqa3X1Tjad=VJ9m!2uXfqcQJy)E!SQB7Yj5Qh07lczVKGmmv+C};4}{UkYxgKCpPmC3pL;`dQfN6X zN#L>Jsssx+O($Dv*6ztDB&wh|d5}0GN_5_>N)kvF z>eDMXKtRx;OLCK^OPRUTsOYkBL0l8&QMg@Ko(5$*J-}wL1iyUm23qv53R*4h=mdBb z#+@!TeSm?v@k|%7q4>m^_4p^~+t}0i9Zg=qSM;_=?2_Rftk({iLS}I zShURJ1y__P3M(?dYkXi%6Co`qnf;W3PI~G)+e|MHqE8z1{-u9Tjd|*o-lf3AJ7XH4 z$8-oT90o!QsDEDyz8OvY+~d%6x#KdEy7YZ(K2oD`!Su#GIm|oyRL>dN8K1H;erZ{Z zD}m={Cni!K-o0FjcR@P z;-y_`g~~SM`4H^kfvJQ$QRLtDs9~P#Kp+cE+6>RCo+Hh4+R)&60>kAG#zkx^KUTmC zuq;SBNI51?0BuB+o!|rQq!z1gRSo5+02%T;@CgK0#jVh*DFj0>7R?0>Ed_cT9D=l= z8@!Jwlql^*89H%m2}1*d3%cZT`K2S$o|vgX@_FKk0>tNxbm-R#TZ>4|>Z>8@b1L|2 zvL0v7u=&la=emv!v?KDCN_vE2HSofeGVGgz$QC`(cGo`v^=lg47us8@Kx{Hga9~?& ztLWW-*F(L6BagPg3&ChUu~qd~U^?&t+4Ww&4=3QXkpa+-?Y!zh9wl>+xeH*w)FnSb z={bZO-mRfP+zLwV1X7XvG~*W21Lrf`i6Kz}E;QQh1Qo3b?GUdF?+K?+i@+3m{|C z01Q9?VQHt~fEjY!q-IcP0N~u)AY90@$~T(-gdd-TnD}+?XcE5KOSvC=h4|_N94DQp z3ZU`>w_SR=2)k8FugbY?e2OB``sSQ_HAC$V=PqBJSB;#Ns+B{@rM z-q7CCgEE`3Dh|KR_kUj5=K=Ayk8w^VnRbf*{PipA^Jl%M@lI(g@p?Im=z%KmMVU7U zdozmEB3=csr?T&F#Q9R*=~UOOt>O*`vxbQEo_yB7#$O0XOr$SXtgR3$0h})T_xK)(cR*nZJ zMH8a7+wg1|jcAl_tZw{{P&GWUEhA2lRt|G{%B9LUgWsEN>*oJ?p4M(mZtDHw7r zg#9{K^jr_#1l3>j+nW6dCwct~#o?2Hr@=Kzx*E9!L0`;TCwY&~<2t?+fbQSLE&JDY z;s=YJytvV%ApHccsDnM{0;r9#N*}|70rs_6P(m~kl#J9sQ_Fkhrq1mFlR0L+9f#7l zv`yfjlM9P_TiGV9<~Fm82n&XWe_7X!ENe%`v_#_M(4dG^g; z53o0RD*LBgOg?P~lyWOR^{T@aqHo#x=!PH`z%>XvIYs<;dC()tuBRlIpo|x$!^sNy{nxu2#9h*_7g3{I9{o%vAcOO~d#r68?>r#Nc z{UWSrSD~1`5;bXkx6XL9P&3^eH<6Vu0IuUF=aH zjx%dFpRTAowV-T5TE4_z4x6kAxvx0X7-jh&cmPm=3QK)V5&$f%8yce=uSBht>1inil6=MRt7rw;n_` zi@rplml2keNT+D_d)J9qdyoC~6Ul*hPk3!LGysZC}QJ0Zu7r#h`h(0mDsVJj5;gf*;#wS@FFab9vYHjvB{35^BKiQS}rQz)s zcS^K3aN6YKvEJ6$S*mFW9T>kgQoaiRqg}9nxyl9Wj84JX`V@Z<8*v0~=&Op%5D-<2 zb1Dky1%#9yi;UzRItNWikuFB7KfPB&uie7*1M4}_V0H_f%76g!Q$<8VK2l=7Jp&c; zcXTwlFUxU@b)J2@sCp0SG$%ty4NLlJJ~@$3WsgQChX51+Na#wIugkjv4u@#m%5?ML zS|EWiS3?o-_XE~G#t5;1Z8&}llymr5GcYh+jIPz}mZVknZSUTs>~g9lHuG{zzOVu1=5#$RtG?CYNPJx;a4XHS+7?Rj zenO5%ABu#TVn1}L$53@y0Y6tYV=Kclp1ELz0cUa!y~=t+&+U}Se){Pa9Gp_L2{N26 zEPB8cIP4yeo^G?>trDx}1n!se749x^su#(75@O+kWht=Gq*Hao?gCMLibu>-wK^P;DkGf0_3{yHR-cV`X)P4goA=^&-ak&-8r|og!71a z;Op?&IUIVW34fKDwRGSjPV^tyc?Ygj-$OnYxhLY{|0p=efj{6BKcEK$Jq052^6T=h z^wpHe1dI5VuJr?>qRZmzo#m>Vt`V6#+R-QE4LLLKt{Xh0&U;bfx2qMW%Q+hIYz{;T z2Gu>;=)P-z@a4Rt$3LP~#`)BRseJ1PcdW?{UpY1SvBtAu5E_%d=oR6z-5L2=RLxO|vER_pk z@33tp2pe)zr@Qi&;R#2?1O%%9zPpwwZ?kx$@;x$bGRQZ1mU=F2Z}QybwwgKQ_O$fL z-50bi3-oG?s?`ke3T%zKU_oJt6;SbE=9j&I7oBbv{ah*P` zu+^p?0cIWDVS8oe^zCYMI9Q?cE7nJD2p{+=B%hM;_+l-w%f9Uu2+(=W9B{){enUIc@$ z?1x=*y-Dg19zmE}G)@sZYIEXq{?{jKY5!F>XUzmZlD8Z=j{AIV2n&%7S14F^7ciVf zVU6<4ad1YM%};g(m+`DRxO286FP6o2fldviJ&erDI=7X&RUE}6H5pDWZX-wjI>rgyN=f$jaczzOtxw>d(<8+ts3>M zEEzXQi!19jdFO;drj7jrrjHdA@J;jvL9gaC=U1xeC61f!C=bmZcJTGu3b%S(I~h1; zzKbBT#EP$gS@P>!hsXEX#5G@8U)MeTT;BKb&$={@D3jC@IxhD3Cu&q~=f-PE+Zu64S3osKXT1%bSJ=cK=)TnqYSEEa_0hM> zJo%sgwL7-8DFnXl7CP(-n#&hdyI;M|uUV}o!Wtf*NUUx&zPKo8s4W%Rvh2#ONct+n?(-~USGyCl~8NvA44%xxfb$8_m0|#@I7tu)l zCAGvM?~%i}Z*n~?sQgfdP*!M7bm*CL=;l8hy2^tBA2ysh%zJwx;V2UP_mbs+(5IeCEwR zV)X90T9~yy1LXGwZH&L;JZ?mvjHhj0(5w7TWCYWd#^>{Ni7)VL(P=z2;+25W6vWw2 zcoEPq62&TTD&wjG@;xvU|MYbKbh9q03D=KrS$g{W*dgsX`0DQtAARNYR_XMo9I;8R zT^8;==DJ(RdlCoDP(Wx=TaSm83e8S&b;kg@+Ty3|&Kw`?`bAz(GCp zOgn-M>3M(0F9T+tJ^LnK)#d&3R02{Yu8pv7GkSW3D%zgi@3Z)>c)u}3)AAMC>BNTg z{yL!jpqAF=$-;<_ugoX52BgOS&M(rxuHt_>MND^ zlE&5-!`CmW+J7nK<+IS2@oWpFGW^05@tV0ZtGXhy-;kuC(HM%MrRo!Stxg5rT2_B& z@j}?uid*;MSmqNgTSj{N+yLIjgB|fd>Ed74ANbL{Y|oVhWEVpG*7+}Y?>B>W;mry@ zbzitUqsN6_vBEvhC#z1jN!zwP$IfJTh0bTc61_|jG~dgd5Gz}os@OUmE-qg`PkY9R zu1JV#2B0==k@HH`&6}!6TYm%Yf)1QyKs#60d;>kWhY9z?wXi2PwI17X+I7V!FmzF8p zSRCG?gGAt2B%sg6Tz)mP?dzex7oLB2pFqEN z*~k&&)h_a6N&YJ?htO|UemKb}m4n$oGi-ubUN`E#`Qc~G>=z!ZWs^^#WUtA*?z5Oc z*8Am*C~h?tSHg6{AN2XYhknN0NB9*?gP)^ju)K+mptfV!fc0v1zEY31A}smbIibP6 z6;(@6RU~X=Ac&4^R8&^Wrp2dSRIK^jF5DLs5TMEIy#MJqjW-8HZj0mO`HVCKsQq4Q z4f6XvLGRk8?^RgFIT*jxQ+1o~Aiee#ZFwA@`^guc$0hh5KO@i{-oTTwycf7Nhkkt+W_y0`&l%gHO+7Zl z2sL@9vB4vLkntaf?(^Q;qG<8k6t&w2f$U|EPk-_o<^8Jdr|d^)CHO5AJ;?rwH+VA_Wut!J{NRVe32krpG5yW66X*k1^{R%DLIpKFv_&UhztyH;Ym zkFLReuVlV%O|w&Tro|a0r>z73=pQ1KA~Veq?>B}1?MvmQRoB+Rd0trO_45(-&9vpwd20?cwM}QzJLZ!f3q5xDhufM7p%wnD68wuu-7&?j=Sot z_nd8)>Tvjh_lcI8M@F>mFP~&ySu)iIRPy*;4)w2EZI|k9E!jUTKLQ6#mc%5*Ow~!O z2A8q(9!^!dnYFMVhP6wZ=Ts)sCf@TS~LND6{0~{8?FhD*ssRJqTb|&-*FdwbImuRI3zz z54zmTM9x~qv!Fgy-He6&Kz+^$?lIT-ul?CX-dl3x7YxJSe=HmT@Jdaxly;MigtJ~F zsaw|&ytzN{QoyF+IDm1_h1tyBj!9ePzoy*(=I@mL$y@pgIY5B#c~j*ebvR`zBY1b! zg4_LZIzh611h_L*>Q*2Px11IbD+c?9s^-QZV6}1ZPN8LfFhsmRn5rSqW9eZReprDvUb&W)fr)u+ = { dingtalk: '/channel-dingtalk.png', telegram: '/channel-telegram.png', weixin: '/channel-weixin.png', + whatsapp: '/channel-whatsapp.png', }; const FEISHU_GUIDE_PDF_URL = '/feishu-bot-guide.pdf'; @@ -1494,7 +1496,7 @@ function WhatsAppPanel({ config, onChange, onPairSuccess }: WhatsAppPanelProps) completedRef.current = true; stopPolling(); setQrPhase('complete'); - const newConfig = { ...config, sessionPath, enabled: true }; + const newConfig = { ...config, sessionPath, enabled: true, _paired: true }; onChange(newConfig); if (onPairSuccess) { await onPairSuccess({ sessionPath }); @@ -1502,15 +1504,20 @@ function WhatsAppPanel({ config, onChange, onPairSuccess }: WhatsAppPanelProps) toast.success(t('whatsapp.qrSuccess')); }; - const startPairing = async () => { + const startPairing = async (replaceExisting = false) => { stopPolling(); completedRef.current = false; setQrError(''); setQrValue(''); setQrPhase('loading'); try { + const pairingSessionPath = + replaceExisting && config.sessionPath + ? `${config.sessionPath}.relink.${Date.now()}` + : (config.sessionPath || null); const res = await client.post('/api/channel/whatsapp/pair/start', { - sessionPath: config.sessionPath || null, + sessionPath: pairingSessionPath, + resetSession: false, }); const id = String(res.data?.pairing_id ?? ''); setPairingId(id); @@ -1546,7 +1553,8 @@ function WhatsAppPanel({ config, onChange, onPairSuccess }: WhatsAppPanelProps) }; const showModal = qrPhase !== 'idle'; - const allowFromEnabled = config.allowFrom !== undefined; + const isPaired = Boolean(config._paired); + const allowFromEnabled = (config.dmPolicy ?? 'allowlist') === 'allowlist'; return ( <> @@ -1554,7 +1562,7 @@ function WhatsAppPanel({ config, onChange, onPairSuccess }: WhatsAppPanelProps)
- {config.sessionPath && ( -

+ {isPaired && ( +

{t('whatsapp.sessionConfigured')}

@@ -1606,7 +1618,7 @@ function WhatsAppPanel({ config, onChange, onPairSuccess }: WhatsAppPanelProps)

{qrError || t('whatsapp.qrError')}

+
+ + {open && !disabled && ( +
+ {visibleOptions.map((option) => ( + + ))} +
+ )} +
+ ); +} + function SecretInput({ value, onChange, @@ -441,37 +573,6 @@ function NumberInput({ ); } -function DatalistInput({ - value, - onChange, - options, - placeholder, - listId, -}: { - value: string; - onChange: (v: string) => void; - options: string[]; - placeholder?: string; - listId: string; -}) { - return ( - <> - onChange(e.target.value)} - placeholder={placeholder} - className="w-full px-3 py-1.5 text-sm border border-gray-300 rounded-md focus:outline-none focus:ring-2 focus:ring-red-500" - /> - - {options.map((item) => ( - - - ); -} - function TagsInput({ value, onChange, @@ -567,6 +668,7 @@ const CHANNEL_ICON_SRC: Record = { wecom: '/channel-wecom.png', dingtalk: '/channel-dingtalk.png', telegram: '/channel-telegram.png', + email: '/channel-email.png', weixin: '/channel-weixin.png', whatsapp: '/channel-whatsapp.png', }; @@ -1548,18 +1650,11 @@ function EmailPanel({ config, onChange }: EmailPanelProps) {
- set('imapHost', v || undefined)} placeholder="imap.gmail.com" - options={[ - 'imap.gmail.com', - 'imap.mail.yahoo.com', - 'imap.mail.qq.com', - 'imap.163.com', - 'outlook.office365.com', - ]} - listId="email-imap-host-list" + options={EMAIL_IMAP_HOST_OPTIONS} /> @@ -1581,18 +1676,11 @@ function EmailPanel({ config, onChange }: EmailPanelProps) { /> - set('smtpHost', v || undefined)} placeholder="smtp.gmail.com" - options={[ - 'smtp.gmail.com', - 'smtp.office365.com', - 'smtp.163.com', - 'smtp.mail.qq.com', - 'smtp-mail.outlook.com', - ]} - listId="email-smtp-host-list" + options={EMAIL_SMTP_HOST_OPTIONS} /> From 14604cd813a95198c2dd86ffe5de30a78db9a80b Mon Sep 17 00:00:00 2001 From: John Yin <10972267+john-yin2333@user.noreply.gitee.com> Date: Wed, 15 Jul 2026 10:45:20 +0800 Subject: [PATCH 43/49] fix(channel): clear stale errors on reconnect --- flocks/channel/base.py | 1 + tests/channel/test_channel.py | 3 +++ 2 files changed, 4 insertions(+) diff --git a/flocks/channel/base.py b/flocks/channel/base.py index 5d58315d9..67e96148c 100644 --- a/flocks/channel/base.py +++ b/flocks/channel/base.py @@ -278,6 +278,7 @@ def reset_status(self, channel_id: str, attempt: int = 0) -> None: def mark_connected(self) -> None: self._status.connected = True + self._status.last_error = None def mark_disconnected(self, error: Optional[str] = None) -> None: self._status.connected = False diff --git a/tests/channel/test_channel.py b/tests/channel/test_channel.py index b1a7539eb..ca43b75fb 100644 --- a/tests/channel/test_channel.py +++ b/tests/channel/test_channel.py @@ -130,8 +130,11 @@ def test_reset_status(self): def test_mark_connected(self): ch = _StubChannel() ch.reset_status("feishu") + ch.mark_disconnected(error="timeout") ch.mark_connected() assert ch.status.connected is True + assert ch.status.last_error is None + assert ch.status.error_count == 1 def test_mark_disconnected_without_error(self): ch = _StubChannel() From 93e058742ec1ec449084f63525dff33f42e65eb5 Mon Sep 17 00:00:00 2001 From: xiami762 <> Date: Wed, 15 Jul 2026 11:24:38 +0800 Subject: [PATCH 44/49] feat(tdp): add realtime threat and custom rule APIs --- .flocks/plugins/skills/tdp-use/SKILL.md | 3 +- .../tdp-use/references/api-reference.md | 21 +- .../tools/device/tdp_v3_3_10/_provider.yaml | 9 +- .../tools/device/tdp_v3_3_10/_test.yaml | 19 ++ .../tools/device/tdp_v3_3_10/tdp.handler.py | 143 ++++++++++++++ .../tdp_v3_3_10/tdp_platform_config.yaml | 64 ++++++- .../tdp_v3_3_10/tdp_threat_monitor_list.yaml | 57 ++++++ tests/tool/test_tdp_api_tools.py | 180 ++++++++++++++++++ tests/tool/test_tdp_skyeye_api_plugins.py | 87 +++++++++ 9 files changed, 573 insertions(+), 10 deletions(-) create mode 100644 .flocks/plugins/tools/device/tdp_v3_3_10/tdp_threat_monitor_list.yaml diff --git a/.flocks/plugins/skills/tdp-use/SKILL.md b/.flocks/plugins/skills/tdp-use/SKILL.md index fb5cfa16b..bcf4bab74 100644 --- a/.flocks/plugins/skills/tdp-use/SKILL.md +++ b/.flocks/plugins/skills/tdp-use/SKILL.md @@ -39,13 +39,14 @@ API 参数和适用场景见 [references/api-reference.md](references/api-refere - 用户说“告警”“告警记录”“告警日志”“明细记录”“查某 IP 的告警”时,默认走 `tdp_log_search` - 用户说“看板”“概览”“趋势”“统计”时,先用 `tdp_dashboard_status` - 用户说“威胁事件”“外部攻击”“攻击事件”“事件总览”“事件趋势”时,优先用 `tdp_incident_list` 或 `tdp_threat_inbound_attack` +- 用户说“实时监控”“威胁监控列表”时,优先用正式 API 工具 `tdp_threat_monitor_list`;不要用 Cookie 认证的旧 Web CLI 代替 - 用户说“告警主机”“受害主机”“主机下的事件”时,优先用 `tdp_host_threat_list` - 用户说“系统状态”“核心服务状态”“数据库状态”时,优先用 `tdp_system_status` - 用户说“MDR”“研判结果”“研判统计”时,优先用 `tdp_mdr_alert_list` - 用户说“脆弱性”“弱口令”“登录入口”“上传接口”“API 风险”“隐私数据”时,走对应资产或风险类工具 - 用户说“云服务”“云实例”“访问云服务的源主机”时,优先用 `tdp_cloud_facilities` - 用户说“下载 PCAP”“下载恶意文件”时,优先走下载类 API;下载前先确认用户确实要下载 -- 用户说“白名单”“资产配置”“策略配置”“联动阻断”“处置状态修改”时,先判断是否涉及写操作;只有用户明确授权后才调用配置类工具 +- 用户说“白名单”“资产配置”“自定义规则”“策略配置”“联动阻断”“处置状态修改”时,先判断是否涉及写操作;只有用户明确授权后才调用配置类工具 ## 浏览器模式使用指南 diff --git a/.flocks/plugins/skills/tdp-use/references/api-reference.md b/.flocks/plugins/skills/tdp-use/references/api-reference.md index 4c4f66bf9..3bd3bef99 100644 --- a/.flocks/plugins/skills/tdp-use/references/api-reference.md +++ b/.flocks/plugins/skills/tdp-use/references/api-reference.md @@ -10,6 +10,7 @@ | 查原始告警日志 | `tdp_log_search` | `search` | `time_from`、`time_to`、`sql` | | 查字段聚合统计 | `tdp_log_search` | `terms` | `time_from`、`time_to`、`term` | | 查威胁事件列表 | `tdp_incident_list` | `search` | `time_from`、`time_to`;可补 `severity`、`phase`、`result`、`keyword`、分页参数 | +| 查威胁实时监控列表 | `tdp_threat_monitor_list` | 默认 | 时间范围不超过 24 小时;可补 `sql`、`net_data_type`、`assets_group`、分页参数 | | 看事件时间线 / 结果分布 / 攻击者明细 | `tdp_incident_list` | `timeline` / `result_distribution` / `attacker_ip_detail` | 通常先要 `incident_id` | | 查外部攻击严重性分布 | `tdp_threat_inbound_attack` | 默认 | `time_from`、`time_to`;可补 `severity`、`result_list`、`keyword` | | 查告警主机汇总 / 主机下事件 | `tdp_host_threat_list` | `summary` / `events` | `summary` 可补 `severity`、`direction`、`threat_type`、`keyword`;`events` 至少要 `asset_machine` | @@ -24,7 +25,7 @@ | 查 MDR 研判列表 / 指标 | `tdp_mdr_alert_list` | `list` / `indicator` | 常见补时间范围、`section_list`、`threat_severity`、`judge_result_status`、`keyword` | | 查系统状态 | `tdp_system_status` | `all` / `core` / `database` 等 | 通常空参 | | 下载 PCAP / 恶意文件 | `tdp_pcap_download` / `tdp_file_download` | 默认 | `alert_id + occ_time` 或 `hash` | -| 管理平台配置 / 策略 | `tdp_platform_config` / `tdp_policy_settings` | 多 action | 写操作,必须先获用户确认 | +| 管理平台配置 / 自定义规则 / 策略 | `tdp_platform_config` / `tdp_policy_settings` | 多 action | 写操作,必须先获用户确认 | ## 时间参数注意事项(重点) @@ -109,6 +110,7 @@ tdp_log_search(time_from=today_start, time_to=today_end, sql="...") | 用户实际要查什么 | 推荐 tool | 说明 | |---|---|---| | 威胁事件 / 攻击事件 / 事件总览 | `tdp_incident_list` / `tdp_threat_inbound_attack` | 事件维度,平台已聚合 | +| 实时监控 / 威胁监控列表 | `tdp_threat_monitor_list` | 实时威胁监控列表,单次时间范围不得超过 24 小时 | | 告警 / 告警日志 / 原始检测记录 | `tdp_log_search` | 告警维度,一条就是一条原始记录 | | 告警主机 / 受害主机 / 主机下事件 | `tdp_host_threat_list` | 主机维度,按主机聚合 | | 漏洞、弱口令、登录入口、API 风险等 | 对应资产或风险类 tool | 资产/风险维度,不要混进日志查询 | @@ -117,6 +119,7 @@ tdp_log_search(time_from=today_start, time_to=today_end, sql="...") - 提到“告警”“最近一小时告警”“查某 IP 的告警”时,默认优先 `tdp_log_search` - 提到“威胁事件”“攻击事件”“看下最近有什么事件”时,优先 `tdp_incident_list` +- 提到“实时监控”“威胁监控列表”时,优先 `tdp_threat_monitor_list` - 提到“哪些主机被打了”“告警主机”“受害主机”时,优先 `tdp_host_threat_list` - 用户没说清时,默认把“明细”理解为告警日志,把“总览/聚合”理解为事件 @@ -648,6 +651,22 @@ PCAP 下载: 处置日志可直接走 `tdp_platform_config(action="disposal_log_list")`;handler 会补默认时间范围、分页和 `cts` 倒序。 +查询自定义规则: + +```json +{ + "action": "custom_rule_list", + "custom_rule_status": [1], + "custom_rule_keyword": "SQL", + "cur_page": 1, + "page_size": 20 +} +``` + +新增、编辑、删除自定义规则分别使用 `custom_rule_add`、`custom_rule_update`、 +`custom_rule_delete`。这些动作会修改检测策略,必须先获得用户明确授权;新增和编辑通过 +`custom_rule` 传完整规则对象,删除通过 `custom_rule_ids` 传规则 `suuid` 列表。 + ## 高风险与低风险 TDP 这里大多数调查类 tool 是读操作,但以下情况仍要谨慎: diff --git a/.flocks/plugins/tools/device/tdp_v3_3_10/_provider.yaml b/.flocks/plugins/tools/device/tdp_v3_3_10/_provider.yaml index 8f7fc28e9..07b38cd26 100644 --- a/.flocks/plugins/tools/device/tdp_v3_3_10/_provider.yaml +++ b/.flocks/plugins/tools/device/tdp_v3_3_10/_provider.yaml @@ -4,11 +4,12 @@ service_id: tdp_api version: "3.3.10" integration_type: device description: > - TDP monitoring platform API for dashboard, threat investigation, asset risk, - log search, MDR analysis, and system status queries. + TDP monitoring platform API for dashboard, real-time threat monitoring, + threat investigation, asset risk, log search, MDR analysis, system status, + and custom rule management. description_cn: > - TDP 监测平台接口,提供监控看板、威胁调查、资产风险、日志检索、MDR、告警调查 - 研判和系统运行状态查询能力。 + TDP 监测平台接口,提供监控看板、威胁实时监控、威胁调查、资产风险、日志检索、 + MDR 告警研判、系统运行状态查询和自定义规则管理能力。 docs_url: "https://agentflocks.github.io/flocks-docs/md/modules/devices/tdp-integration" auth: type: custom diff --git a/.flocks/plugins/tools/device/tdp_v3_3_10/_test.yaml b/.flocks/plugins/tools/device/tdp_v3_3_10/_test.yaml index d6ff6eed9..1eae486e1 100644 --- a/.flocks/plugins/tools/device/tdp_v3_3_10/_test.yaml +++ b/.flocks/plugins/tools/device/tdp_v3_3_10/_test.yaml @@ -39,6 +39,15 @@ fixtures: params: {} assert: success: true + - label: "List custom rules" + label_cn: "查询自定义规则列表" + tags: [configuration, rules] + params: + action: custom_rule_list + cur_page: 1 + page_size: 20 + assert: + success: true tdp_incident_list: - label: "Search incidents (page 1)" @@ -144,6 +153,16 @@ fixtures: tags: [smoke, threats] params: {} + tdp_threat_monitor_list: + - label: "List real-time threats (last 24 hours)" + label_cn: "查询威胁实时监控列表(最近 24 小时)" + tags: [smoke, threats] + params: + cur_page: 1 + page_size: 20 + assert: + success: true + tdp_login_weakpwd_list: - label: "List weak-password login risks (page 1)" label_cn: "查询弱密码登录风险(第 1 页)" diff --git a/.flocks/plugins/tools/device/tdp_v3_3_10/tdp.handler.py b/.flocks/plugins/tools/device/tdp_v3_3_10/tdp.handler.py index 4ae242d42..1e39d02b2 100644 --- a/.flocks/plugins/tools/device/tdp_v3_3_10/tdp.handler.py +++ b/.flocks/plugins/tools/device/tdp_v3_3_10/tdp.handler.py @@ -66,6 +66,7 @@ CLOUD_ACCESS_FUZZY_FIELDS = ["machine", "cloud_instance"] CLOUD_INSTANCE_ACCESS_FUZZY_FIELDS = ["cloud_instance", "external_ip", "machine"] MDR_FUZZY_FIELDS = ["task_id", "machine", "asset_info", "threat_name"] +CUSTOM_RULE_FUZZY_FIELDS = ["threat_name", "threat_msg"] SYSTEM_STATUS_ENDPOINTS = { "core": "/api/v1/core-status", "ioc_update": "/api/v1/ioc-update-status", @@ -550,6 +551,21 @@ def _inbound_attack_body(body: dict[str, Any]) -> dict[str, Any]: ) +def _threat_monitor_body(body: dict[str, Any]) -> dict[str, Any]: + time_from, time_to = _default_time_range(days=1) + defaults = { + "condition": { + "time_from": time_from, + "time_to": time_to, + "sql": "", + "size": 1000, + "refresh_rate": -1, + }, + "page": {"cur_page": 1, "page_size": 20}, + } + return _deep_merge(defaults, body) + + def _login_entry_body(body: dict[str, Any]) -> dict[str, Any]: defaults = _body_with_condition_time( { @@ -721,6 +737,10 @@ def _custom_intel_delete_body(body: Any) -> dict[str, Any]: return _deep_merge({"action": "delete", "data": {}}, _dict_body(body)) +def _custom_rule_list_body(body: Any) -> dict[str, Any]: + return _paged_body(body, sort_by="updated_time") + + def _ip_reputation_list_body(body: Any) -> dict[str, Any]: return _paged_body(body, sort_by="updated_at") @@ -903,6 +923,10 @@ def _passthrough_body(body: Any) -> Any: _cascade_children_body, "device_cascade_platform_children", ), + "custom_rule_list": ("/api/v1/customRule/list", _custom_rule_list_body, "custom_rule_list"), + "custom_rule_add": ("/api/v1/customRule/add", _passthrough_body, "custom_rule_add"), + "custom_rule_update": ("/api/v1/customRule/update", _passthrough_body, "custom_rule_update"), + "custom_rule_delete": ("/api/v1/customRule/delete", _passthrough_body, "custom_rule_delete"), "disposal_log_list": ("/api/v1/disposal/log/list", _disposal_log_list_body, "disposal_log_list"), } @@ -1062,6 +1086,27 @@ def _validate_log_sql_expression(action: str, sql: str | None) -> ToolResult | N return None +def _validate_threat_monitor_time_range(time_from: int | None, time_to: int | None) -> ToolResult | None: + if (time_from is None) != (time_to is None): + return ToolResult( + success=False, + error="TDP threat monitor requires time_from and time_to together.", + ) + if time_from is None or time_to is None: + return None + if time_from > time_to: + return ToolResult( + success=False, + error="TDP threat monitor requires time_from to be less than or equal to time_to.", + ) + if time_to - time_from > 24 * 60 * 60: + return ToolResult( + success=False, + error="TDP threat monitor time range cannot exceed 24 hours.", + ) + return None + + def _dict_copy(value: Any) -> dict[str, Any]: if isinstance(value, dict): return dict(value) @@ -1585,6 +1630,46 @@ async def inbound_attack( ) +async def threat_monitor_list( + context: ToolContext, + condition: dict[str, Any] | None = None, + page: dict[str, Any] | None = None, + time_from: int | None = None, + time_to: int | None = None, + sql: str | None = None, + size: int | None = None, + assets_group: list[Any] | None = None, + net_data_type: list[Any] | None = None, + refresh_rate: int | None = None, + cur_page: int | None = None, + page_size: int | None = None, +) -> ToolResult: + del context + validation_error = _validate_threat_monitor_time_range(time_from, time_to) + if validation_error: + return validation_error + validation_error = _validate_log_sql_expression("threat_monitor_list", sql) + if validation_error: + return validation_error + + body = _compose_payload(condition=condition, page=page) + condition_dict = body.setdefault("condition", {}) + _set_if_present(condition_dict, "time_from", time_from) + _set_if_present(condition_dict, "time_to", time_to) + _set_if_present(condition_dict, "sql", sql) + _set_if_present(condition_dict, "size", size) + _set_list_if_present(condition_dict, "assets_group", assets_group) + _set_list_if_present(condition_dict, "net_data_type", net_data_type) + _set_if_present(condition_dict, "refresh_rate", refresh_rate) + _set_page_overrides(body, cur_page=cur_page, page_size=page_size) + return await _run_json_tool( + "threat_monitor_list", + "/api/v1/monitor/threat/list", + _threat_monitor_body, + body, + ) + + async def login_entry_list( context: ToolContext, action: str = "list", @@ -1891,6 +1976,17 @@ async def platform_config( rule: dict[str, Any] | None = None, ips: list[Any] | None = None, keyword: str | None = None, + custom_rule: dict[str, Any] | None = None, + custom_rule_ids: list[Any] | None = None, + delete_alert_data: int | None = None, + custom_rule_status: list[Any] | None = None, + custom_rule_severity: list[Any] | None = None, + custom_rule_result: list[Any] | None = None, + custom_rule_keyword: str | None = None, + cur_page: int | None = None, + page_size: int | None = None, + sort_by: str | None = None, + sort_order: str | None = None, ) -> ToolResult: del context selected_action = _normalize_action(action, "asset_list") @@ -1906,6 +2002,27 @@ async def platform_config( body["lock"] = dict(lock) elif selected_action.startswith("white_rule_") and selected_action != "white_rule_search": body = _dict_copy(rule) + elif selected_action == "custom_rule_list": + body = _compose_payload(condition=condition, page=page) + condition_dict = body.setdefault("condition", {}) + _set_list_if_present(condition_dict, "status", custom_rule_status) + _set_list_if_present(condition_dict, "threat_severity", custom_rule_severity) + _set_list_if_present(condition_dict, "threat_result", custom_rule_result) + _set_keyword_fuzzy(condition_dict, custom_rule_keyword, CUSTOM_RULE_FUZZY_FIELDS) + _set_page_overrides( + body, + cur_page=cur_page, + page_size=page_size, + sort_by=sort_by, + sort_order=sort_order, + ) + elif selected_action in {"custom_rule_add", "custom_rule_update"}: + body = _dict_copy(custom_rule) + elif selected_action == "custom_rule_delete": + body = {} + if custom_rule_ids is not None: + body["suuid"] = list(custom_rule_ids) + _set_if_present(body, "delete_alert_data", delete_alert_data) elif selected_action == "cascade_children": body = {} _set_if_present(body, "keyword", keyword) @@ -1924,6 +2041,32 @@ async def platform_config( validation_error = _validate_required_body_fields(selected_action, body, "id") if validation_error: return validation_error + elif selected_action in {"custom_rule_add", "custom_rule_update"}: + required_fields = [ + "threat_name", + "threat_msg", + "threat_severity", + "threat_type", + "threat_result", + "directions", + "body", + "attacker", + "status", + ] + if selected_action == "custom_rule_update": + required_fields.insert(0, "suuid") + validation_error = _validate_required_body_fields(selected_action, body, *required_fields) + if validation_error: + return validation_error + elif selected_action == "custom_rule_delete": + validation_error = _validate_non_empty_list_body( + selected_action, + body, + fallback_keys=("suuid",), + label="custom rule suuid list", + ) + if validation_error: + return validation_error return await _run_action_json_tool( PLATFORM_CONFIG_ACTIONS, default_action="asset_list", diff --git a/.flocks/plugins/tools/device/tdp_v3_3_10/tdp_platform_config.yaml b/.flocks/plugins/tools/device/tdp_v3_3_10/tdp_platform_config.yaml index dae0f2cab..dfd7c12f9 100644 --- a/.flocks/plugins/tools/device/tdp_v3_3_10/tdp_platform_config.yaml +++ b/.flocks/plugins/tools/device/tdp_v3_3_10/tdp_platform_config.yaml @@ -1,9 +1,10 @@ name: tdp_platform_config description: > - 管理 TDP 平台配置相关接口,覆盖资产配置、白名单过滤和级联平台管理。 - 通过 action 可切换到资产列表、资产增删改、白名单搜索与维护、子节点列表和处置日志等子接口; + 管理 TDP 平台配置相关接口,覆盖资产配置、白名单过滤、级联平台管理和自定义规则。 + 通过 action 可切换到资产列表、资产增删改、白名单搜索与维护、子节点列表、自定义规则和处置日志等子接口; 其中 `asset_add`、`asset_update`、`asset_delete`、`white_rule_add`、 - `white_rule_update`、`white_rule_delete` 属于写操作,调用前应确认影响范围。 + `white_rule_update`、`white_rule_delete`、`custom_rule_add`、`custom_rule_update`、 + `custom_rule_delete` 属于写操作,调用前应确认影响范围。 category: custom enabled: true requires_confirmation: true @@ -17,7 +18,9 @@ inputSchema: description: > 平台配置子接口动作。`asset_list` / `white_rule_search` 会补默认分页; `asset_delete` 需要非空资产 IP 列表;`cascade_children` - 使用根级 `keyword` 做子节点模糊查询;其余动作使用对应的显式对象参数。 + 使用根级 `keyword` 做子节点模糊查询;`custom_rule_list` 使用独立的 + `custom_rule_*` 筛选参数,新增与编辑通过 `custom_rule` 传完整规则对象, + 删除通过 `custom_rule_ids` 传规则标识列表。 enum: - asset_list - asset_add @@ -28,6 +31,10 @@ inputSchema: - white_rule_update - white_rule_delete - cascade_children + - custom_rule_list + - custom_rule_add + - custom_rule_update + - custom_rule_delete - disposal_log_list default: asset_list condition: @@ -53,6 +60,55 @@ inputSchema: keyword: type: string description: 级联平台子节点搜索关键词,适用于 `cascade_children`。 + custom_rule: + type: object + description: > + 自定义规则对象,适用于 `custom_rule_add`、`custom_rule_update`。新增必填字段为 + `threat_name`、`threat_msg`、`threat_severity`、`threat_type`、`threat_result`、 + `directions`、`body`、`attacker`、`status`;编辑还必须包含 `suuid`。 + custom_rule_ids: + type: array + items: + type: string + description: 自定义规则 `suuid` 列表,适用于 `custom_rule_delete`。 + delete_alert_data: + type: integer + enum: [0, 1] + description: 删除自定义规则时是否同时删除关联告警数据;0=否,1=是。 + custom_rule_status: + type: array + items: + type: integer + enum: [0, 1] + description: 自定义规则状态筛选;0=禁用,1=启用,适用于 `custom_rule_list`。 + custom_rule_severity: + type: array + items: + type: integer + enum: [1, 2, 3, 4] + description: 自定义规则严重级别筛选;1=低危,2=中危,3=高危,4=严重。 + custom_rule_result: + type: array + items: + type: string + enum: [success, failed, unknown] + description: 自定义规则检出结果筛选。 + custom_rule_keyword: + type: string + description: 自定义规则关键词,handler 会在 `threat_name`、`threat_msg` 字段中模糊查询。 + cur_page: + type: integer + description: 当前页码,适用于 `custom_rule_list`。 + page_size: + type: integer + description: 每页条数,适用于 `custom_rule_list`。 + sort_by: + type: string + description: 排序字段,适用于 `custom_rule_list`,默认 `updated_time`。 + sort_order: + type: string + enum: [asc, desc] + description: 排序方向,适用于 `custom_rule_list`,默认 `desc`。 required: [] handler: type: script diff --git a/.flocks/plugins/tools/device/tdp_v3_3_10/tdp_threat_monitor_list.yaml b/.flocks/plugins/tools/device/tdp_v3_3_10/tdp_threat_monitor_list.yaml new file mode 100644 index 000000000..2830e3a1f --- /dev/null +++ b/.flocks/plugins/tools/device/tdp_v3_3_10/tdp_threat_monitor_list.yaml @@ -0,0 +1,57 @@ +name: tdp_threat_monitor_list +description: > + 查询 TDP 威胁-实时监控列表,对应正式 API `/api/v1/monitor/threat/list`。 + 查询时间范围不得超过 24 小时;未传时间时 handler 动态使用最近 24 小时。 + `sql` 是 TDP 过滤表达式,不是包含 SELECT/FROM 的完整 SQL。 +category: custom +enabled: true +requires_confirmation: false +provider: tdp_api + +inputSchema: + type: object + properties: + time_from: + type: integer + description: 开始时间戳,Unix 秒级时间戳;与 `time_to` 同时传入,范围不得超过 24 小时。 + time_to: + type: integer + description: 截止时间戳,Unix 秒级时间戳;与 `time_from` 同时传入,范围不得超过 24 小时。 + sql: + type: string + description: > + 告警过滤表达式,例如 `threat.type = 'c2'`。不得使用 SELECT、FROM 等完整 SQL 语法。 + size: + type: integer + description: 返回的最大记录条数;未传时使用文档默认值 1000。 + assets_group: + type: array + items: + type: integer + description: 业务组 ID 列表。 + net_data_type: + type: array + items: + type: string + enum: [attack, risk, action] + description: 日志类型列表;attack=攻击、risk=风险、action=敏感行为。 + refresh_rate: + type: integer + description: 自动刷新间隔;`-1` 表示关闭自动刷新,其他值可能使响应返回 `new_alert_count`。 + cur_page: + type: integer + description: 当前页码,默认 1。 + page_size: + type: integer + description: 每页条数,默认 20。 + condition: + type: object + description: 高级兼容参数,仅在顶层语义化参数未覆盖目标字段时使用。 + page: + type: object + description: 高级兼容分页参数,仅在需要自定义底层分页结构时使用。 + required: [] +handler: + type: script + script_file: tdp.handler.py + function: threat_monitor_list diff --git a/tests/tool/test_tdp_api_tools.py b/tests/tool/test_tdp_api_tools.py index ba7073b5f..232ae407e 100644 --- a/tests/tool/test_tdp_api_tools.py +++ b/tests/tool/test_tdp_api_tools.py @@ -283,6 +283,61 @@ async def test_tdp_threat_inbound_attack_uses_severity_distribution_endpoint(): assert request_kwargs["json"]["condition"]["time_from"] < request_kwargs["json"]["condition"]["time_to"] +@pytest.mark.asyncio +async def test_tdp_threat_monitor_list_uses_public_api_endpoint(): + tool = _load_tool("tdp_threat_monitor_list.yaml") + fake_session = _FakeSession( + [_FakeResponse(json_payload={"response_code": 0, "data": {"items": [{"machine": "10.0.0.1"}]}})] + ) + + with ( + patch( + "flocks.config.config_writer.ConfigWriter.get_api_service_raw", + return_value={ + "apiKey": "{secret:tdp_api_key}", + "secret": "{secret:tdp_secret}", + "base_url": "https://tdp.local", + }, + ), + patch( + "flocks.security.get_secret_manager", + return_value=MagicMock( + get=MagicMock(side_effect=lambda key: {"tdp_api_key": "demo-api", "tdp_secret": "demo-secret"}.get(key)) + ), + ), + patch("aiohttp.ClientSession", return_value=fake_session), + ): + result = await tool.handler( + ToolContext(session_id="test", message_id="test"), + time_from=1700000000, + time_to=1700003600, + sql="threat.type = 'c2'", + size=100, + assets_group=[1, 3], + net_data_type=["attack", "risk"], + cur_page=2, + page_size=50, + ) + + assert result.success is True + assert result.metadata["api"] == "threat_monitor_list" + method, request_url, request_kwargs = fake_session.calls[0] + assert method == "POST" + assert request_url == "https://tdp.local/api/v1/monitor/threat/list" + assert request_kwargs["json"] == { + "condition": { + "time_from": 1700000000, + "time_to": 1700003600, + "sql": "threat.type = 'c2'", + "size": 100, + "refresh_rate": -1, + "assets_group": [1, 3], + "net_data_type": ["attack", "risk"], + }, + "page": {"cur_page": 2, "page_size": 50}, + } + + @pytest.mark.asyncio async def test_tdp_login_api_list_can_switch_to_summary_action(): tool = _load_tool("tdp_login_api_list.yaml") @@ -861,3 +916,128 @@ async def test_tdp_policy_settings_ip_reputation_delete_supports_ids_wrapper(): assert result.metadata["api"] == "ip_reputation_delete" assert fake_session.calls[0][1] == "https://tdp.local/api/v1/ipReputation/delete" assert fake_session.calls[0][2]["json"] == [302773, 302774] + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("action", "path", "handler_kwargs", "expected_body"), + [ + ( + "custom_rule_list", + "/api/v1/customRule/list", + { + "custom_rule_status": [1], + "custom_rule_severity": [3, 4], + "custom_rule_result": ["success"], + "custom_rule_keyword": "SQL", + }, + { + "condition": { + "status": [1], + "threat_severity": [3, 4], + "threat_result": ["success"], + "fuzzy": {"keyword": "SQL", "fieldlist": ["threat_name", "threat_msg"]}, + }, + "page": { + "cur_page": 1, + "page_size": 20, + "sort": [{"sort_by": "updated_time", "sort_order": "desc"}], + }, + }, + ), + ( + "custom_rule_add", + "/api/v1/customRule/add", + { + "custom_rule": { + "threat_name": "Custom SQL injection", + "threat_msg": "Detected SQL injection", + "threat_severity": 3, + "threat_type": "exploit", + "threat_result": "success", + "directions": ["in"], + "body": [{"field": "http.uri", "method": "GET", "content": "/api"}], + "attacker": "src", + "status": 1, + } + }, + { + "threat_name": "Custom SQL injection", + "threat_msg": "Detected SQL injection", + "threat_severity": 3, + "threat_type": "exploit", + "threat_result": "success", + "directions": ["in"], + "body": [{"field": "http.uri", "method": "GET", "content": "/api"}], + "attacker": "src", + "status": 1, + }, + ), + ( + "custom_rule_update", + "/api/v1/customRule/update", + { + "custom_rule": { + "suuid": "rule-1", + "threat_name": "Updated SQL injection rule", + "threat_msg": "Detected SQL injection", + "threat_severity": 4, + "threat_type": "exploit", + "threat_result": "success", + "directions": ["in"], + "body": [{"field": "http.uri", "method": "GET", "content": "/api"}], + "attacker": "src", + "status": 1, + } + }, + { + "suuid": "rule-1", + "threat_name": "Updated SQL injection rule", + "threat_msg": "Detected SQL injection", + "threat_severity": 4, + "threat_type": "exploit", + "threat_result": "success", + "directions": ["in"], + "body": [{"field": "http.uri", "method": "GET", "content": "/api"}], + "attacker": "src", + "status": 1, + }, + ), + ( + "custom_rule_delete", + "/api/v1/customRule/delete", + {"custom_rule_ids": ["rule-1"], "delete_alert_data": 0}, + {"suuid": ["rule-1"], "delete_alert_data": 0}, + ), + ], +) +async def test_tdp_platform_config_supports_custom_rule_endpoints(action, path, handler_kwargs, expected_body): + tool = _load_tool("tdp_platform_config.yaml") + fake_session = _FakeSession([_FakeResponse(json_payload={"response_code": 0, "verbose_msg": "OK"})]) + + with ( + patch( + "flocks.config.config_writer.ConfigWriter.get_api_service_raw", + return_value={ + "apiKey": "{secret:tdp_api_key}", + "secret": "{secret:tdp_secret}", + "base_url": "https://tdp.local", + }, + ), + patch( + "flocks.security.get_secret_manager", + return_value=MagicMock( + get=MagicMock(side_effect=lambda key: {"tdp_api_key": "demo-api", "tdp_secret": "demo-secret"}.get(key)) + ), + ), + patch("aiohttp.ClientSession", return_value=fake_session), + ): + result = await tool.handler( + ToolContext(session_id="test", message_id="test"), + action=action, + **handler_kwargs, + ) + + assert result.success is True + assert fake_session.calls[0][1] == f"https://tdp.local{path}" + assert fake_session.calls[0][2]["json"] == expected_body diff --git a/tests/tool/test_tdp_skyeye_api_plugins.py b/tests/tool/test_tdp_skyeye_api_plugins.py index 0d6a60521..d3cf573f6 100644 --- a/tests/tool/test_tdp_skyeye_api_plugins.py +++ b/tests/tool/test_tdp_skyeye_api_plugins.py @@ -139,6 +139,89 @@ async def test_tdp_platform_config_exposes_disposal_log_list(): assert mock_run.await_args.kwargs["action"] == "disposal_log_list" +async def test_tdp_threat_monitor_rejects_partial_or_oversized_time_ranges(): + module = _load_module("test_tdp_handler_threat_monitor_time_range", _TDP_HANDLER) + + partial = await module.threat_monitor_list(_ctx(), time_from=1700000000) + oversized = await module.threat_monitor_list( + _ctx(), + time_from=1700000000, + time_to=1700000000 + 24 * 60 * 60 + 1, + ) + + assert partial.success is False + assert "together" in partial.error + assert oversized.success is False + assert "24 hours" in oversized.error + + +async def test_tdp_threat_monitor_maps_semantic_filters(): + module = _load_module("test_tdp_handler_threat_monitor_mapping", _TDP_HANDLER) + mock_result = ToolResult(success=True, output={"status": 200}) + + with patch.object(module, "_run_json_tool", AsyncMock(return_value=mock_result)) as mock_run: + result = await module.threat_monitor_list( + _ctx(), + time_from=1700000000, + time_to=1700003600, + sql="threat.type = 'c2'", + size=50, + assets_group=[1], + net_data_type=["attack"], + refresh_rate=10, + cur_page=3, + page_size=25, + ) + + assert result.success is True + api_name, path, payload = _built_json_payload(mock_run) + assert api_name == "threat_monitor_list" + assert path == "/api/v1/monitor/threat/list" + assert payload["condition"] == { + "time_from": 1700000000, + "time_to": 1700003600, + "sql": "threat.type = 'c2'", + "size": 50, + "assets_group": [1], + "net_data_type": ["attack"], + "refresh_rate": 10, + } + assert payload["page"] == {"cur_page": 3, "page_size": 25} + + +async def test_tdp_platform_custom_rule_write_validation(): + module = _load_module("test_tdp_handler_platform_custom_rule_validation", _TDP_HANDLER) + + missing_add_fields = await module.platform_config( + _ctx(), + action="custom_rule_add", + custom_rule={"threat_name": "Incomplete rule"}, + ) + missing_update_suuid = await module.platform_config( + _ctx(), + action="custom_rule_update", + custom_rule={ + "threat_name": "Rule", + "threat_msg": "Message", + "threat_severity": 3, + "threat_type": "exploit", + "threat_result": "success", + "directions": ["in"], + "body": [{"field": "http.uri", "method": "GET", "content": "/api"}], + "attacker": "src", + "status": 1, + }, + ) + missing_delete_ids = await module.platform_config(_ctx(), action="custom_rule_delete") + + assert missing_add_fields.success is False + assert "threat_msg" in missing_add_fields.error + assert missing_update_suuid.success is False + assert "suuid" in missing_update_suuid.error + assert missing_delete_ids.success is False + assert "suuid list" in missing_delete_ids.error + + async def test_tdp_policy_ip_reputation_delete_requires_non_empty_ids(): module = _load_module("test_tdp_handler_policy", _TDP_HANDLER) @@ -762,6 +845,7 @@ def test_tdp_query_yaml_promotes_semantic_top_level_fields(): "tdp_assets_domain_list.yaml": {"domain_name_or_ip", "has_login_api", "second_level_domain"}, "tdp_privacy_diagram.yaml": {"itag", "methods", "fuzzy_url_host"}, "tdp_threat_inbound_attack.yaml": {"severity", "result_list", "keyword"}, + "tdp_threat_monitor_list.yaml": {"time_from", "time_to", "sql", "assets_group", "cur_page"}, "tdp_machine_asset_list.yaml": {"time_from", "time_to", "service", "service_class", "application", "keyword"}, "tdp_mdr_alert_list.yaml": {"section_list", "threat_severity", "keyword"}, "tdp_cloud_facilities.yaml": {"cloud_vendor", "cloud_instance", "keyword"}, @@ -789,6 +873,9 @@ def test_tdp_platform_yaml_uses_keyword_and_requires_confirmation(): assert "keyword" in raw["inputSchema"]["properties"] assert "device_id" not in raw["inputSchema"]["properties"] assert "disposal_log_list" in raw["inputSchema"]["properties"]["action"]["enum"] + assert "custom_rule_list" in raw["inputSchema"]["properties"]["action"]["enum"] + assert "custom_rule" in raw["inputSchema"]["properties"] + assert "custom_rule_ids" in raw["inputSchema"]["properties"] def test_tdp_policy_yaml_requires_confirmation_and_uses_object_ioc_list(): From b4677939c85313ebbe4e85d4860df1ec870e53c6 Mon Sep 17 00:00:00 2001 From: John Yin <10972267+john-yin2333@user.noreply.gitee.com> Date: Wed, 15 Jul 2026 12:51:35 +0800 Subject: [PATCH 45/49] fix: expose extended messaging channels to Rex --- flocks/agent/agents/rex/prompt_builder.py | 4 +-- flocks/tool/channel/channel_message.py | 32 ++++++++++++++++++----- flocks/tool/channel/im_send_message.py | 15 +++++++---- flocks/tool/task/schedule_task_center.py | 18 ++++++++----- tests/tool/test_channel_message.py | 31 ++++++++++++++++++---- tests/tool/test_im_send_message.py | 22 ++++++++++++++++ tests/tool/test_task_center_compat.py | 14 ++++++++++ 7 files changed, 112 insertions(+), 24 deletions(-) diff --git a/flocks/agent/agents/rex/prompt_builder.py b/flocks/agent/agents/rex/prompt_builder.py index 4435432c0..281234bbc 100644 --- a/flocks/agent/agents/rex/prompt_builder.py +++ b/flocks/agent/agents/rex/prompt_builder.py @@ -432,5 +432,5 @@ def _build_security_priority_section(available_agents: List["AvailableAgent"]) - def _build_im_send_pointer_section() -> str: return """### IM Messaging -When the user wants to send a message to an IM platform, call `im_send_message`. -When creating a scheduled task that sends to an IM platform later, resolve the target `session_id` with `im_send_message(resolve_only=true)` before calling `schedule_task_create`.""" +When the user wants to send a message to a connected messaging channel (including IM platforms and email), call `im_send_message`. +When creating a scheduled task that sends to a connected messaging channel later, resolve the target `session_id` with `im_send_message(resolve_only=true)` before calling `schedule_task_create`.""" diff --git a/flocks/tool/channel/channel_message.py b/flocks/tool/channel/channel_message.py index acc5e1c20..6a908e8e1 100644 --- a/flocks/tool/channel/channel_message.py +++ b/flocks/tool/channel/channel_message.py @@ -1,8 +1,9 @@ """ -channel_message tool — sends a message to the IM channel bound to a given session. +channel_message tool — sends a message to the messaging channel bound to a given session. Looks up the SessionBinding for the given session_id to automatically resolve -the target channel (WeCom / Feishu / DingTalk) and chat_id, so the caller +the target channel (WeCom / Weixin / Feishu / DingTalk / Telegram / WhatsApp / Email) +and chat_id, so the caller does not need to specify them manually. The optional channel_type parameter selects a specific channel when a session @@ -25,6 +26,9 @@ "weixin": ["weixin", "微信", "wechat", "wx"], "feishu": ["feishu", "飞书", "lark"], "dingtalk": ["dingtalk", "钉钉", "dingding", "dingtalk-connector"], + "telegram": ["telegram", "tg", "tele"], + "whatsapp": ["whatsapp", "wa"], + "email": ["email", "mail", "邮件", "imap", "smtp"], } @@ -130,8 +134,9 @@ async def _http_session_send( @ToolRegistry.register_function( name="channel_message", description=( - "Send a message to the IM channel bound to a session. " - "Channel types: WeCom/企业微信=wecom, Weixin/微信=weixin, Feishu=feishu, DingTalk=dingtalk. " + "Send a message to the messaging channel bound to a session. " + "Channel types: WeCom/企业微信=wecom, Weixin/微信=weixin, Feishu=feishu, DingTalk=dingtalk, " + "Telegram=telegram, WhatsApp=whatsapp, Email/邮件=email. " "Resolves the target channel and chat automatically from session_id. " "Use channel_type to target a specific channel when the session has multiple bindings." ), @@ -153,9 +158,24 @@ async def _http_session_send( name="channel_type", type=ParameterType.STRING, required=False, - enum=["wecom", "weixin", "feishu", "dingtalk", "企微", "企业微信", "微信", "飞书", "钉钉"], + enum=[ + "wecom", + "weixin", + "feishu", + "dingtalk", + "telegram", + "whatsapp", + "email", + "企微", + "企业微信", + "微信", + "飞书", + "钉钉", + "邮件", + ], description=( - "Target channel: wecom=企业微信, weixin=微信, feishu=飞书, or dingtalk=钉钉. " + "Target channel: wecom=企业微信, weixin=微信, feishu=飞书, dingtalk=钉钉, " + "telegram=Telegram, whatsapp=WhatsApp, or email=邮件. " "Chinese aliases are accepted. " "If omitted and the session has only one binding, that channel is used automatically. " "If omitted and the session has multiple bindings, the message is sent to all of them." diff --git a/flocks/tool/channel/im_send_message.py b/flocks/tool/channel/im_send_message.py index 68fbc1078..834218ada 100644 --- a/flocks/tool/channel/im_send_message.py +++ b/flocks/tool/channel/im_send_message.py @@ -20,6 +20,9 @@ "weixin": ["weixin", "微信", "wechat", "wx"], "feishu": ["feishu", "飞书", "lark"], "dingtalk": ["dingtalk", "钉钉", "dingding", "dingtalk-connector"], + "telegram": ["telegram", "tg", "tele"], + "whatsapp": ["whatsapp", "wa"], + "email": ["email", "mail", "邮件", "imap", "smtp"], } @@ -200,10 +203,12 @@ async def _resolve_target( @ToolRegistry.register_function( name="im_send_message", description=( - "Resolve an IM target session and optionally send a message. " - "Use this for WeCom/企业微信, Weixin/微信, Feishu, DingTalk, or custom channel sessions when the user asks to send an IM message. " - "Use channel_type=wecom for 企业微信 and channel_type=weixin for 微信. " - "If session_id is omitted, it uses the current IM session when available, otherwise asks the user to pick one." + "Resolve a messaging channel target session and optionally send a message. " + "Use this for WeCom/企业微信, Weixin/微信, Feishu, DingTalk, Telegram, WhatsApp, Email/邮件, " + "or custom channel sessions when the user asks to send a message through a connected channel. " + "Use channel_type=wecom for 企业微信, channel_type=weixin for 微信, channel_type=telegram for Telegram, " + "channel_type=whatsapp for WhatsApp, and channel_type=email for 邮件. " + "If session_id is omitted, it uses the current channel session when available, otherwise asks the user to pick one." ), category=ToolCategory.SYSTEM, parameters=[ @@ -225,7 +230,7 @@ async def _resolve_target( required=False, description=( "Optional channel filter, such as wecom=企业微信, weixin=微信, " - "feishu, dingtalk, telegram, or a custom channel id." + "feishu, dingtalk, telegram, whatsapp, email=邮件, or a custom channel id." ), ), ToolParameter( diff --git a/flocks/tool/task/schedule_task_center.py b/flocks/tool/task/schedule_task_center.py index ec2a3ab14..aa1c6775e 100644 --- a/flocks/tool/task/schedule_task_center.py +++ b/flocks/tool/task/schedule_task_center.py @@ -128,12 +128,15 @@ def _normalize_schedule_task_create_inputs( "'一次', '这次', specific date like '明天下午3点', '下周五晚上', '2024-01-15 18:00'\n" "Queue-only (use type=queued, no schedule): " "'等会', '稍后', '待会', '有空时', '不着急'\n\n" - "IMPORTANT — IM session resolution before creating:\n" - "If the task involves sending a message to an IM platform " - "(企业微信/WeCom、微信/Weixin/WeChat、飞书/Feishu、钉钉/DingTalk), " + "IMPORTANT — Messaging channel session resolution before creating:\n" + "If the task involves sending a message to a connected messaging channel " + "(企业微信/WeCom、微信/Weixin/WeChat、飞书/Feishu、钉钉/DingTalk、" + "Telegram、WhatsApp、邮件/Email), " "you MUST resolve the target session_id and channel_type with im_send_message(resolve_only=true) " "BEFORE calling this tool. " - "Use channel_type=wecom for 企业微信 and channel_type=weixin for 微信. " + "Use channel_type=wecom for 企业微信, channel_type=weixin for 微信, " + "channel_type=telegram for Telegram, channel_type=whatsapp for WhatsApp, " + "and channel_type=email for 邮件. " "Embed both into description and user_prompt. " "If the user cannot provide a session_id, do NOT create the task." ), @@ -150,9 +153,12 @@ def _normalize_schedule_task_create_inputs( type=ParameterType.STRING, description=( "Detailed task description. " - "If the task involves sending a message to an IM platform (WeCom/Weixin/Feishu/DingTalk), " + "If the task involves sending a message to a connected messaging channel " + "(WeCom/Weixin/Feishu/DingTalk/Telegram/WhatsApp/Email), " "MUST include the resolved channel_type and session_id here. " - "Use channel_type=wecom for 企业微信 and channel_type=weixin for 微信. " + "Use channel_type=wecom for 企业微信, channel_type=weixin for 微信, " + "channel_type=telegram for Telegram, channel_type=whatsapp for WhatsApp, " + "and channel_type=email for 邮件. " "Example: '每天早上8点向飞书群发送日报 channel_type=feishu session_id=ses_abc123'" ), required=True, diff --git a/tests/tool/test_channel_message.py b/tests/tool/test_channel_message.py index add4aca2a..ed815ef57 100644 --- a/tests/tool/test_channel_message.py +++ b/tests/tool/test_channel_message.py @@ -26,14 +26,35 @@ def test_channel_message_normalizes_wecom_aliases() -> None: assert _normalize_channel_type("wxwork") == "wecom" -def test_channel_message_schema_includes_weixin() -> None: +def test_channel_message_normalizes_telegram_whatsapp_email_aliases() -> None: + assert _normalize_channel_type("telegram") == "telegram" + assert _normalize_channel_type("tg") == "telegram" + assert _normalize_channel_type("tele") == "telegram" + assert _normalize_channel_type("whatsapp") == "whatsapp" + assert _normalize_channel_type("wa") == "whatsapp" + assert _normalize_channel_type("email") == "email" + assert _normalize_channel_type("mail") == "email" + assert _normalize_channel_type("邮件") == "email" + + +def test_channel_message_schema_includes_builtin_channels() -> None: schema = ToolRegistry.get_schema("channel_message") assert schema is not None - assert "wecom" in schema.properties["channel_type"]["enum"] - assert "企业微信" in schema.properties["channel_type"]["enum"] - assert "weixin" in schema.properties["channel_type"]["enum"] - assert "微信" in schema.properties["channel_type"]["enum"] + channel_enum = schema.properties["channel_type"]["enum"] + for value in ( + "wecom", + "企业微信", + "weixin", + "微信", + "feishu", + "dingtalk", + "telegram", + "whatsapp", + "email", + "邮件", + ): + assert value in channel_enum @pytest.mark.asyncio diff --git a/tests/tool/test_im_send_message.py b/tests/tool/test_im_send_message.py index 72df25a42..42e41a674 100644 --- a/tests/tool/test_im_send_message.py +++ b/tests/tool/test_im_send_message.py @@ -51,6 +51,28 @@ def test_im_send_message_normalizes_wecom_aliases() -> None: assert _normalize_channel_type("wxwork") == "wecom" +def test_im_send_message_normalizes_telegram_whatsapp_email_aliases() -> None: + assert _normalize_channel_type("telegram") == "telegram" + assert _normalize_channel_type("tg") == "telegram" + assert _normalize_channel_type("tele") == "telegram" + assert _normalize_channel_type("whatsapp") == "whatsapp" + assert _normalize_channel_type("wa") == "whatsapp" + assert _normalize_channel_type("email") == "email" + assert _normalize_channel_type("mail") == "email" + assert _normalize_channel_type("邮件") == "email" + + +def test_im_send_message_schema_mentions_extended_builtin_channels() -> None: + schema = ToolRegistry.get_schema("im_send_message") + + assert schema is not None + channel_description = schema.properties["channel_type"]["description"] + tool_description = ToolRegistry.get("im_send_message").info.description + for channel in ("telegram", "whatsapp", "email"): + assert channel in channel_description.lower() + assert channel in tool_description.lower() + + @pytest.mark.asyncio async def test_im_send_message_requires_message_unless_resolve_only() -> None: result = await im_send_message(_ctx(), session_id="ses_target") diff --git a/tests/tool/test_task_center_compat.py b/tests/tool/test_task_center_compat.py index 654cd28f5..afd9dd9ea 100644 --- a/tests/tool/test_task_center_compat.py +++ b/tests/tool/test_task_center_compat.py @@ -60,6 +60,20 @@ def test_task_create_schema_allows_legacy_schedule_type(self): assert "action" in schema.properties assert "type" not in schema.required + def test_task_create_schema_mentions_extended_message_channels(self): + tool = ToolRegistry.get("schedule_task_create") + schema = ToolRegistry.get_schema("schedule_task_create") + + assert tool is not None + assert schema is not None + text = ( + tool.info.description + + " " + + schema.properties["description"]["description"] + ).lower() + for value in ("telegram", "whatsapp", "email", "channel_type=telegram", "channel_type=whatsapp", "channel_type=email"): + assert value in text + def test_task_update_schema_makes_action_optional_and_exposes_trigger_fields(self): schema = ToolRegistry.get_schema("schedule_task_update") From 45027ad445ee797404152d76b915e5c5db7c0947 Mon Sep 17 00:00:00 2001 From: xiami762 <> Date: Wed, 15 Jul 2026 14:37:46 +0800 Subject: [PATCH 46/49] fix(browser): reuse connected daemon for state save --- flocks/browser/admin.py | 6 ++++++ flocks/browser/run.py | 5 ++++- tests/browser/test_run.py | 18 ++++++++++++++++-- 3 files changed, 26 insertions(+), 3 deletions(-) diff --git a/flocks/browser/admin.py b/flocks/browser/admin.py index 23dbd6417..0ed418e08 100644 --- a/flocks/browser/admin.py +++ b/flocks/browser/admin.py @@ -151,6 +151,12 @@ def _daemon_browser_connection(name: str): return None +def daemon_browser_connected(name: str | None = None) -> bool: + """Return whether the selected daemon still has a live browser connection.""" + effective_name = ipc.runtime_paths(name or NAME).name + return _daemon_browser_connection(effective_name) is not None + + def browser_connections() -> list[dict]: """Return live daemons with healthy browser connections.""" output = [] diff --git a/flocks/browser/run.py b/flocks/browser/run.py index 05a21f34b..746c0dfc0 100644 --- a/flocks/browser/run.py +++ b/flocks/browser/run.py @@ -8,6 +8,7 @@ from .admin import ( _version, + daemon_browser_connected, ensure_daemon, print_update_banner, restart_daemon, @@ -100,10 +101,12 @@ def _run_state_command(args: list[str]) -> None: url, no_reload = _parse_state_options(rest, allow_no_reload=action == "load") print_update_banner() - ensure_daemon() if action == "save": + if not daemon_browser_connected(): + ensure_daemon() result = save_state(path, url=url) else: + ensure_daemon() result = load_state(path, url=url, reload=not no_reload) print(json.dumps(result, ensure_ascii=False, indent=2)) diff --git a/tests/browser/test_run.py b/tests/browser/test_run.py index d9003ef8f..2912c5834 100644 --- a/tests/browser/test_run.py +++ b/tests/browser/test_run.py @@ -46,10 +46,11 @@ def test_state_show_prints_summary_without_daemon() -> None: mock_daemon.assert_not_called() -def test_state_save_ensures_daemon_and_prints_result() -> None: +def test_state_save_reuses_connected_daemon_and_prints_result() -> None: stdout = StringIO() with ( patch("flocks.browser.run.save_state", return_value={"path": "/tmp/auth-state.json", "cookies": 2}), + patch("flocks.browser.run.daemon_browser_connected", return_value=True), patch("flocks.browser.run.ensure_daemon") as mock_daemon, patch("flocks.browser.run.print_update_banner") as mock_banner, patch("sys.stdout", stdout), @@ -58,6 +59,18 @@ def test_state_save_ensures_daemon_and_prints_result() -> None: assert '"cookies": 2' in stdout.getvalue() mock_banner.assert_called_once() + mock_daemon.assert_not_called() + + +def test_state_save_ensures_daemon_when_browser_is_disconnected() -> None: + with ( + patch("flocks.browser.run.save_state", return_value={"path": "/tmp/auth-state.json"}), + patch("flocks.browser.run.daemon_browser_connected", return_value=False), + patch("flocks.browser.run.ensure_daemon") as mock_daemon, + patch("flocks.browser.run.print_update_banner"), + ): + run.main(["state", "save", "/tmp/auth-state.json"]) + mock_daemon.assert_called_once() @@ -65,7 +78,7 @@ def test_state_load_passes_optional_flags() -> None: stdout = StringIO() with ( patch("flocks.browser.run.load_state", return_value={"finalUrl": "https://example.com"}) as mock_load, - patch("flocks.browser.run.ensure_daemon"), + patch("flocks.browser.run.ensure_daemon") as mock_daemon, patch("flocks.browser.run.print_update_banner"), patch("sys.stdout", stdout), ): @@ -85,6 +98,7 @@ def test_state_load_passes_optional_flags() -> None: url="https://example.com/dashboard", reload=False, ) + mock_daemon.assert_called_once() assert '"finalUrl": "https://example.com"' in stdout.getvalue() From e757507be99f6834d0870294910790161f58bb42 Mon Sep 17 00:00:00 2001 From: John Yin <10972267+john-yin2333@user.noreply.gitee.com> Date: Wed, 15 Jul 2026 16:23:38 +0800 Subject: [PATCH 47/49] Improve email host mismatch guidance --- webui/src/locales/en-US/channel.json | 2 + webui/src/locales/zh-CN/channel.json | 2 + webui/src/pages/Channel/index.tsx | 105 +++++++++++++++++++++++++-- 3 files changed, 101 insertions(+), 8 deletions(-) diff --git a/webui/src/locales/en-US/channel.json b/webui/src/locales/en-US/channel.json index 14651a29b..8f8ad1c40 100644 --- a/webui/src/locales/en-US/channel.json +++ b/webui/src/locales/en-US/channel.json @@ -247,6 +247,7 @@ "serversDesc": "Configure the IMAP inbox and SMTP sender used by this channel.", "imapHost": "IMAP Host", "imapHostHint": "Inbox server, for example imap.gmail.com or outlook.office365.com.", + "imapHostMismatchWarning": "This mailbox usually uses {{expectedHost}} as the IMAP host. Check whether the host address is correct.", "imapPort": "IMAP Port", "imapPortHint": "Default is 993 for IMAP over SSL.", "imapSecurity": "IMAP Security", @@ -256,6 +257,7 @@ "securityInsecure": "Insecure (not recommended)", "smtpHost": "SMTP Host", "smtpHostHint": "Outgoing mail server, for example smtp.gmail.com or smtp.office365.com.", + "smtpHostMismatchWarning": "This mailbox usually uses {{expectedHost}} as the SMTP host. Check whether the host address is correct.", "smtpPort": "SMTP Port", "smtpPortHint": "Default is 587 for STARTTLS, or 465 for implicit TLS.", "smtpSecurity": "SMTP Security", diff --git a/webui/src/locales/zh-CN/channel.json b/webui/src/locales/zh-CN/channel.json index 7c45822ea..7c2940f8a 100644 --- a/webui/src/locales/zh-CN/channel.json +++ b/webui/src/locales/zh-CN/channel.json @@ -249,6 +249,7 @@ "serversDesc": "配置此通道使用的 IMAP 收件箱和 SMTP 发件服务器。", "imapHost": "IMAP 主机", "imapHostHint": "收件服务器,例如 imap.gmail.com 或 outlook.office365.com。", + "imapHostMismatchWarning": "当前邮箱通常使用 {{expectedHost}} 作为 IMAP 主机,请检查主机地址是否填错。", "imapPort": "IMAP 端口", "imapPortHint": "默认 993,使用 IMAP SSL。", "imapSecurity": "IMAP 安全模式", @@ -258,6 +259,7 @@ "securityInsecure": "不加密(高风险,不推荐)", "smtpHost": "SMTP 主机", "smtpHostHint": "发件服务器,例如 smtp.gmail.com 或 smtp.office365.com。", + "smtpHostMismatchWarning": "当前邮箱通常使用 {{expectedHost}} 作为 SMTP 主机,请检查主机地址是否填错。", "smtpPort": "SMTP 端口", "smtpPortHint": "默认 587 使用 STARTTLS;465 使用隐式 TLS。", "smtpSecurity": "SMTP 安全模式", diff --git a/webui/src/pages/Channel/index.tsx b/webui/src/pages/Channel/index.tsx index 5bacdfdd2..101dc5dd8 100644 --- a/webui/src/pages/Channel/index.tsx +++ b/webui/src/pages/Channel/index.tsx @@ -152,14 +152,56 @@ interface EmailChannelConfig { } const EMAIL_HOST_PRESETS = [ - { id: 'gmail', label: 'Gmail', imapHost: 'imap.gmail.com', smtpHost: 'smtp.gmail.com' }, - { id: 'outlook', label: 'Outlook / Microsoft 365', imapHost: 'outlook.office365.com', smtpHost: 'smtp.office365.com' }, - { id: 'qq', label: 'QQ Mail', imapHost: 'imap.qq.com', smtpHost: 'smtp.qq.com' }, - { id: 'netease-163', label: 'NetEase 163', imapHost: 'imap.163.com', smtpHost: 'smtp.163.com' }, - { id: 'netease-126', label: 'NetEase 126', imapHost: 'imap.126.com', smtpHost: 'smtp.126.com' }, - { id: 'tencent-exmail', label: 'Tencent Exmail', imapHost: 'imap.exmail.qq.com', smtpHost: 'smtp.exmail.qq.com' }, - { id: 'aliyun', label: 'Alibaba Mail', imapHost: 'imap.aliyun.com', smtpHost: 'smtp.aliyun.com' }, - { id: 'yahoo', label: 'Yahoo Mail', imapHost: 'imap.mail.yahoo.com', smtpHost: 'smtp.mail.yahoo.com' }, + { + id: 'gmail', + label: 'Gmail', + domains: ['gmail.com', 'googlemail.com'], + imapHost: 'imap.gmail.com', + smtpHost: 'smtp.gmail.com', + }, + { + id: 'outlook', + label: 'Outlook / Microsoft 365', + domains: ['outlook.com', 'hotmail.com', 'live.com', 'msn.com'], + imapHost: 'outlook.office365.com', + smtpHost: 'smtp.office365.com', + }, + { id: 'qq', label: 'QQ Mail', domains: ['qq.com'], imapHost: 'imap.qq.com', smtpHost: 'smtp.qq.com' }, + { + id: 'netease-163', + label: 'NetEase 163', + domains: ['163.com'], + imapHost: 'imap.163.com', + smtpHost: 'smtp.163.com', + }, + { + id: 'netease-126', + label: 'NetEase 126', + domains: ['126.com'], + imapHost: 'imap.126.com', + smtpHost: 'smtp.126.com', + }, + { + id: 'tencent-exmail', + label: 'Tencent Exmail', + domains: ['exmail.qq.com'], + imapHost: 'imap.exmail.qq.com', + smtpHost: 'smtp.exmail.qq.com', + }, + { + id: 'aliyun', + label: 'Alibaba Mail', + domains: ['aliyun.com'], + imapHost: 'imap.aliyun.com', + smtpHost: 'smtp.aliyun.com', + }, + { + id: 'yahoo', + label: 'Yahoo Mail', + domains: ['yahoo.com', 'ymail.com'], + imapHost: 'imap.mail.yahoo.com', + smtpHost: 'smtp.mail.yahoo.com', + }, ]; const EMAIL_IMAP_HOST_OPTIONS = EMAIL_HOST_PRESETS.map((entry) => ({ @@ -172,6 +214,40 @@ const EMAIL_SMTP_HOST_OPTIONS = EMAIL_HOST_PRESETS.map((entry) => ({ label: entry.label, })); +function normalizeEmailHost(raw: string | undefined): string { + return (raw ?? '') + .trim() + .toLowerCase() + .replace(/^[a-z][a-z0-9+.-]*:\/\//, '') + .split('/')[0] + .split(':')[0]; +} + +function getEmailDomain(raw: string | undefined): string { + const normalized = (raw ?? '').trim().toLowerCase(); + const atIndex = normalized.lastIndexOf('@'); + if (atIndex < 0) return ''; + return normalized.slice(atIndex + 1); +} + +function getEmailHostPreset(address: string | undefined) { + const domain = getEmailDomain(address); + if (!domain) return undefined; + return EMAIL_HOST_PRESETS.find((entry) => entry.domains.includes(domain)); +} + +function isEmailHostMismatch( + address: string | undefined, + host: string | undefined, + protocol: 'imap' | 'smtp' +): boolean { + const preset = getEmailHostPreset(address); + const normalizedHost = normalizeEmailHost(host); + if (!preset || !normalizedHost) return false; + const expectedHost = protocol === 'imap' ? preset.imapHost : preset.smtpHost; + return normalizedHost !== expectedHost; +} + interface WeixinChannelConfig { enabled: boolean; token?: string; @@ -1628,6 +1704,9 @@ function EmailPanel({ config, onChange }: EmailPanelProps) { onChange({ ...config, [key]: value }), [config, onChange] ); + const emailHostPreset = getEmailHostPreset(config.address); + const showImapHostWarning = isEmailHostMismatch(config.address, config.imapHost, 'imap'); + const showSmtpHostWarning = isEmailHostMismatch(config.address, config.smtpHost, 'smtp'); return ( <> @@ -1656,6 +1735,11 @@ function EmailPanel({ config, onChange }: EmailPanelProps) { placeholder="imap.gmail.com" options={EMAIL_IMAP_HOST_OPTIONS} /> + {showImapHostWarning && emailHostPreset && ( +

+ {t('email.imapHostMismatchWarning', { expectedHost: emailHostPreset.imapHost })} +

+ )}
Date: Wed, 15 Jul 2026 16:52:04 +0800 Subject: [PATCH 48/49] chore/update-version-2026-7-15 --- pyproject.toml | 2 +- uv.lock | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index 88e6ae933..783712fde 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "flocks" -version = "v2026.7.8" +version = "v2026.7.15" description = "AI-Native SecOps platform with multi-agent collaboration" authors = [ {name = "Flocks Team", email = "team@example.com"} diff --git a/uv.lock b/uv.lock index f1c30af20..a09013247 100644 --- a/uv.lock +++ b/uv.lock @@ -553,7 +553,7 @@ wheels = [ [[package]] name = "flocks" -version = "2026.7.8" +version = "2026.7.15" source = { editable = "." } dependencies = [ { name = "aiofiles" }, From 09cd975c7f70495ca6428935dfffff295df6cadf Mon Sep 17 00:00:00 2001 From: xiami762 <> Date: Wed, 15 Jul 2026 17:01:47 +0800 Subject: [PATCH 49/49] fix(cli): handle missing Windows netstat output --- flocks/cli/service_manager.py | 4 +++- tests/cli/test_service_manager.py | 22 ++++++++++++++++++++++ 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/flocks/cli/service_manager.py b/flocks/cli/service_manager.py index e75fb8d1a..46134d52f 100644 --- a/flocks/cli/service_manager.py +++ b/flocks/cli/service_manager.py @@ -838,6 +838,8 @@ def port_is_in_use(port: int, listeners: Sequence[int] | None = None) -> bool: current_listeners = list(listeners) if listeners is not None else port_owner_pids(port) if current_listeners: return True + if sys.platform == "win32": + return not _bind_port_available(port) if _port_owner_lookup_available(): return False return not _bind_port_available(port) @@ -2169,7 +2171,7 @@ def _run_windows_netstat(port: int) -> str: return "" target = f":{port}" lines = [] - for line in completed.stdout.splitlines(): + for line in (completed.stdout or "").splitlines(): if "LISTENING" not in line.upper(): continue if target not in line: diff --git a/tests/cli/test_service_manager.py b/tests/cli/test_service_manager.py index d69f3f960..d75c1cac0 100644 --- a/tests/cli/test_service_manager.py +++ b/tests/cli/test_service_manager.py @@ -512,6 +512,16 @@ def test_parse_windows_netstat_output_extracts_unique_pids() -> None: assert service_manager._parse_windows_netstat_output(output) == [1234, 5678] +def test_run_windows_netstat_handles_missing_stdout(monkeypatch) -> None: + monkeypatch.setattr( + service_manager.subprocess, + "run", + lambda *_args, **_kwargs: SimpleNamespace(returncode=0, stdout=None), + ) + + assert service_manager._run_windows_netstat(5173) == "" + + def test_port_owner_pids_warns_when_no_tool_found(monkeypatch) -> None: monkeypatch.setattr(service_manager.sys, "platform", "linux") monkeypatch.setattr(service_manager, "which", lambda _name: None) @@ -529,6 +539,18 @@ def test_port_is_in_use_falls_back_to_bind_when_pid_lookup_unavailable(monkeypat assert service_manager.port_is_in_use(5173) is True +@pytest.mark.parametrize(("port_available", "expected"), [(True, False), (False, True)]) +def test_windows_port_is_in_use_confirms_empty_pid_lookup_with_bind( + monkeypatch, + port_available: bool, + expected: bool, +) -> None: + monkeypatch.setattr(service_manager.sys, "platform", "win32") + monkeypatch.setattr(service_manager, "_bind_port_available", lambda _port: port_available) + + assert service_manager.port_is_in_use(5173, listeners=[]) is expected + + def test_bind_port_available_checks_all_ipv4_interfaces(monkeypatch) -> None: binds: list[tuple[str, int]] = [] sockopts: list[tuple[object, ...]] = []