Feat : Merge 3D frontend to main branch#22
Open
tmacychen wants to merge 13 commits into
Open
Conversation
Change transports from polling-only to websocket-first with automatic polling fallback, and enable upgrade to reduce latency and bandwidth overhead in real-time control scenarios.
- Add Three.js 3D simulator with robotic arm (src/sim/) - Add React frontend with Vite (src/App.tsx, src/components/) - Add cloud training service integration (src/services/) - Update main.html with UI improvements and collapsible sections - Add demo.html and demo2.html standalone demos - Add arm toggle feature persisted to localStorage
- Rename frontend/ -> frontend-old/ (2D Canvas simulator preserved) - Move 3D frontend (src/, package.json, vite.config.ts, etc.) into frontend/ - Update vite.config.ts: base='/', proxy /api + /socket.io to backend, build to static/ - Fix WebSocket proxy to use ws:true for real-time communication
- Rename old frontend/ to frontend-old/ (2D Canvas simulator) - Move 3D frontend files from root src/ into frontend/ - frontend/ builds to static-3d/ with base=/sim3d/ - frontend-old/ builds to static/ (default Flask serve) - Update frontend.py routing: - / serves templates/index.html (2D simulator, via static/) - /sim3d serves static-3d/index.html (3D Three.js + robotic arm) - /sim3d/<path> serves 3D static assets - Remove setup.sh anonymous volume for frontend/node_modules - Add npm install step before frontend build in setup.sh
…-detect model dims - api: add temporal ensembling buffer for action chunk caching in infer/step - api: replace discrete action mapping with continuous [velocity, angularVelocity] - websocket: auto-detect state_dim/action_dim from checkpoint metadata - websocket: fallback to env vars when checkpoint metadata unavailable
- Uncomment training mode switch (frontend/cloud) in sidebar - Add obstacle placement system: click-to-place box/cylinder on ground - Support drag-to-move, R to rotate, Delete to remove obstacles - Obstacles participate in collision detection automatically - Add scene preset save/load with localStorage persistence - Support export/import scene presets as JSON files - Fix placementMode stale closure via useRef pattern
问题1: 浏览器端点击训练后页面冻结无响应
- 根因: TF.js WebGL 后端首次 model.fit() 时同步编译 WebGL shader,
阻塞主线程 10-30 秒,导致浏览器完全冻结
- 修复: 训练时切换到 CPU 后端(tf.setBackend('cpu')),训练完成后恢复
WebGL 用于推理;模型从 conv2d 改为纯 dense 层,避免 shader 编译;
prepareTrainingData 改为 async 并在循环中 yield 防止阻塞
问题2: 后端 POST /api/dataset 500 错误 (TypeError: must be real number, not NoneType)
- 根因: sim.current.speed/turnSpeed 未赋值(undefined),recordFrame 中
v += undefined 导致 v=NaN,JSON.stringify 将 NaN 序列化为 null,
后端 torch.tensor() 收到 None 报错
- 修复: 在 sim ref 初始化时添加 speed/turnSpeed 默认值,通过 useEffect
同步 React state 到 sim.current;后端添加 _sanitize_float_list()
防御性处理,将 None/NaN/Inf 替换为 0.0
其他: 移除 run.py 中多余的 HTTPS 线程启动
- static-3d/assets/*.js - static-3d/assets/*.css - static-3d/index.html 这些文件是 vite build 自动生成的,每次构建都会变化,不应提交到仓库
这些文件已被 .gitignore 忽略,从仓库中移除跟踪
- 从 frontend-old 迁移控制台页面(BaseControlPage)和按钮组件(ControlButton) - 新增 control.html 和 control-main.tsx 作为控制台页面的独立构建入口 - 新增 vite.config.control.ts 配置,控制台构建输出到 static/,base 路径为 / - 添加 build:control 和 build:all 构建脚本 - 重构后端路由:/ 服务控制台页面,/assets/ 服务其资源,移除旧的 render_template 逻辑 - 统一 _send_static 辅助函数,自动设置正确的 MIME 类型 - 路由:/ → AKA-00 控制台,/sim3d → 3D 模拟器
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
主要工作汇报
我在cnb的平台下测试本项目,依靠ai编码完成。工作内容汇报如下:
1. 创建项目构建脚本
创建dockerfile与docker compose配置文件,制作容器镜像用于项目开发和调试。创建setup脚本,自动编译前端代码,一键启动项目,。
2. 将3d-font分支合并入main
测试发现main分支的3d界面与3d-font分支的有较大不同,将3d-font分支的前端页面合并到main分支,支持前端与后端的训练和推理。旧的3d界面保存在frontend-old目录。
3. 修复前端训练页面卡住等几个问题
主要修改为cpu来训练,模型从 conv2d 改为纯 dense 层,避免 shader 编译等。prepareTrainingData 改为 async 并在循环中 yield 防止阻塞。Commit 22e315c
App.tsx:833-834 中 A 键让 w 减小(左转),但 updateRobotMovement:912 中 A 键让 w 增大。训练数据中的转向和实际运动完全相反。
_apply_continuous_action(api.py:254-272)在返回 action 后还累加速度、移小车、发 Socket.IO 事件——前端完全不用。后端应只返回 action。
runCloudInference 每 200ms 设置一次 velocity = v,但 updatePhysics 每帧执行
velocity *= 0.9。200ms ≈ 12 帧,衰减到 v * 0.9^12 ≈ 0.28v,机器人几乎无法移动。需要在推理模式下跳过摩擦衰减。目前的效果
主要问题