Web demo: ASE humanoid runs in the browser via PhysX WASM + ONNX Runtime#95
Draft
zalo wants to merge 94 commits intoxbpeng:mainfrom
Draft
Web demo: ASE humanoid runs in the browser via PhysX WASM + ONNX Runtime#95zalo wants to merge 94 commits intoxbpeng:mainfrom
zalo wants to merge 94 commits intoxbpeng:mainfrom
Conversation
- tools/mjcf_to_physx_usd: Converts MuJoCo MJCF XML to USDA with PhysX physics schemas (ArticulationRootAPI, PhysicsJoint, drives, limits). Flat USD hierarchy with world-space body positions for PhysX compatibility. - mimickit/engines/ovphysx_engine.py: Standalone PhysX 5 engine using the ovphysx pip package for physics and Newton ViewerGL for visualization. Includes body ordering mapping between ovphysx link indices and Newton DFS body order, ground grid rendering, and DOF target clamping. - mimickit/engines/warp_engine.py: Newton/Warp engine using SolverXPBD with explicit PD torque computation matching Isaac Lab's force-based PD. - data/engines/ovphysx_engine.yaml, warp_engine.yaml: Engine configs. - Modified engine_builder.py to register both new backends. - Modified isaac_lab_engine.py to suppress carb performance warnings and disable expensive RTX rendering features for better framerate. - CLAUDE.md: Project guidance for Claude Code. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Key changes to match isaacsim.asset.importer.mjcf behavior: - Joint frame rotation: compute quaternion aligning MJCF axis with PhysX X-axis via GetRotationQuat, matching computeJointFrame() exactly - Single hinge joints use PhysicsRevoluteJoint with axis="X" + rotated localRot0/localRot1 (not identity rotation with mapped axis names) - Multi-hinge D6 joints use axisMap with dot-product Y/Z comparison for proper axis ordering, matching Isaac Lab's 2/3-joint logic - Added maxForce from actuatorfrcrange on all drives - Added PhysxSchemaPhysxLimitAPI stiffness/damping on joints - Added PhysxSchemaPhysxJointAPI armature values - Translation axes locked (low=1, high=-1) on D6 joints - Drive type = "force" (not "acceleration"), matching Isaac Lab Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- tools/export_onnx.py: Exports the ASE actor (obs+latent → action) to ONNX with baked-in observation normalizer and action unnormalizer. The exported graph takes raw obs (158) + latent (64) and outputs action (31). Verified PyTorch vs ONNX Runtime: rel_diff < 1e-6. - data/models/ase_humanoid_sword_shield_actor.onnx: 3.2KB exported model ready for onnxruntime-web / browser inference. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Complete in-browser demo running the ASE sword-shield character with: - physx-js-webidl (PhysX 5 compiled to WASM) for articulation physics - three.js for 3D rendering with shadows and orbit controls - onnxruntime-web for neural network policy inference - All running client-side with no server required tools/export_humanoid_json.py: Generalized MJCF-to-JSON exporter that works with any MimicKit character. Accepts --mjcf, --model, --output, --pelvis_z, --fixed_bodies args. Auto-detects fixed-attachment bodies when not specified. web/index.html: Single-page demo (863 lines) with PhysX articulation setup, three.js visualization, ONNX inference loop, ground grid, orbit camera. web/humanoid_data.json: Exported humanoid description (17 bodies, 31 DOFs) web/ase_humanoid_sword_shield_actor.onnx: Exported policy model (7MB) Deployed to: https://mimickit-demo.pages.dev Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Disable self-collision between articulation links via eDISABLE_SELF_COLLISION flag + collision group filtering (articulation shapes only collide with ground, not each other) - Add debug panel with: Active Drives toggle (go limp), NN Policy toggle, Self-Collision toggle, Gravity toggle, Drive Stiffness/Damping scale sliders, Sim Substeps slider, Time Scale slider - Live debug readout: root_z, root_speed, max_dof_vel, link/DOF count - Wrap all PhysX joint queries in try-catch to prevent WASM memory access crashes from locked/fixed joint axis queries Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three critical bugs causing policy-driven spasming: 1. Double observation normalization: ONNX model already contains the normalizer (baked in by export_onnx.py). JS was normalizing AGAIN before passing to ONNX, producing nonsensical outputs (~1000x scale). Fix: removed JS-side normalization. 2. Wrong observation format: JS was building ~88-dim obs stuffed into wrong slots of 158-dim vector. Python uses heading-removed quaternion as 6D tan_norm (not raw 4D quat), joint rotations as 96-dim tan_norm (not 31-dim raw DOF positions), 6 key bodies (not 5), and all values in heading-relative frame. Fix: rewrote buildObservation() with full quaternion math pipeline matching char_env.py exactly. 3. Action clamping too restrictive: [-1.5, 1.5] was clamping all 31 DOFs when legitimate actions reach ±6.7. Fix: per-DOF bounds from a_mean ± 3*a_std using normalizer stats from humanoid_data.json. Also verified: ONNX model is purely feedforward (no recurrent state). ASE latent vector is the only inter-frame state (already handled). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Python kinematic model treats 3-hinge joints as SPHERICAL and uses exp_map_to_quat() to convert DOF positions to quaternions. The JS was composing individual PhysX axis rotations instead. - Added kinematicJoints array to humanoid_data.json with exact joint types (SPHERICAL/HINGE/FIXED), DOF indices, and axes from MJCFCharModel - Rewrote joint rotation section of buildObservation() to: - SPHERICAL: read 3 DOFs as exp_map, convert via expMapToQuat - HINGE: read 1 DOF, convert via axisAngleToQuat with Z-up axis - FIXED: use identity quaternion - Iterate in kinematic model body order (not PhysX DOF order) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
quatYupToZup was [-qx, -qz, -qy, qw] which negates all rotation components, causing the policy to see every rotation as reversed. The character tried to "stand up sideways" because it perceived left as right, forward as backward, etc. Correct formula is [qx, qz, qy, qw] — swap Y↔Z without negation. Verified against all test cases: identity, heading rotation, forward tilt, and sideways lean all produce correct Z-up quaternions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Eliminated all Y-up↔Z-up coordinate conversions that were causing rotation bugs (character trying to stand sideways/upside-down). - PhysX gravity now (0,0,-9.81) matching MJCF Z-up convention - Body positions, joint frames, geom poses all used directly in Z-up - Observation reads PhysX state as Z-up natively (no conversion) - Deleted zup(), yupToZup3(), quatYupToZup(), quatZupToYup() - Single Three.js worldGroup with rotation.x=-PI/2 for rendering - Camera tracking converts Z-up root pos to Y-up for OrbitControls Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PxBoxGeometry(50, 0.5, 50) had the thin dimension on Y, creating a 100x100 wall in the Z direction that the character spawned inside of. Changed to PxBoxGeometry(50, 50, 0.5) so Z is the thin axis — correct for Z-up where gravity is (0, 0, -9.81). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The ONNX model was using clamp(-5,5) and min_std=1e-5, but the actual MimicKit agent uses clip=10.0 (from base_agent.py) and min_std=1e-4 (from Normalizer). This caused 34 observation dims to be clamped differently, producing actions off by up to 1700x. Fixed export_onnx.py to use clamp(-10,10) and min_std=1e-4. PyTorch vs ONNX now match within 2.4e-6 (verified via WS server). Also added: - tools/ws_inference_server.py: WebSocket server for side-by-side PyTorch vs ONNX inference comparison - WS Inference checkbox in web debug panel for live testing Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
onnxruntime-web cannot load external .data files via MountedFiles. Re-saved the model with all tensor data embedded in the single .onnx file (7MB, no external .data companion file needed). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Verified: JS ONNX, Python ONNX, and Python PyTorch all produce identical actions for the same first-frame observation (diff=0). The flailing occurs on subsequent frames as the character moves and observations diverge from the training distribution. - Test Pose button: raises both arms with hardcoded drive targets (verifies drives and coordinate system without the policy) - Full observation JSON dump on first frame for Python comparison - WS server min_std fixed to 1e-4 matching Normalizer Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The env resets the character to init_pose (a combat-ready stance from the env config), not T-pose. This was causing 85/158 observation dims to mismatch on the first frame. - Added init_root_pos, init_root_rot_quat, init_dof_pos to JSON - applyInitPose() sets root pose via setRootGlobalPose and joint positions via setJointPosition, then drive targets to match - Step physics once after init to propagate joint→body FK - pelvis_z corrected from 0.903 to 0.703 (matching env config) - First frame obs now matches Newton within 0.94 max diff (only key_body_pos differs due to FK propagation timing) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The last 18 observation dims (key body positions) were wrong because PhysX link positions don't update from setJointPosition until a full sim step. Instead of relying on PhysX, compute FK in JS matching the Python kinematic model exactly: - Exported fk_parent_indices, fk_local_translations, fk_local_rotations from MJCFCharModel to humanoid_data.json - JS buildObservation() runs FK using these, same as Python's forward_kinematics(): body_pos = parent_pos + quat_rotate(parent_rot, local_trans), body_rot = quat_mul(parent_rot, quat_mul(local_rot, j_rot)) - When policy is OFF, drives target init_pose (not T-pose zeros) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
updateMassAndInertia was called with density=1000 for ALL links, ignoring MJCF per-geom densities (250-2226). This gave every body the wrong mass, making drives push with incorrect force-to-acceleration ratios. Now computes volume-weighted average density per body from its geom densities, matching how Isaac Lab/MuJoCo compute mass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaced flat density=1000 and volume-based density approximation with exact per-body masses from the Newton solver (total 53.001 kg). Mass ranges from 0.5 kg (hand) to 12.0 kg (torso), matching the MJCF geom densities (250-2226) applied to the actual geometry volumes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
articulation.setSolverIterationCounts(32, 1) was using 8x more position solver iterations than Isaac Lab's max_position_iteration_count=4. This made constraint resolution too stiff, amplifying force mismatches and causing instability. Changed to (4, 0) matching Isaac Lab exactly. Also explains why "decreasing substeps makes it calmer" — fewer substeps meant fewer total solver iterations per frame. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Diagnostic sequence: 1. Reset to init_pose 2. Read raw PhysX DOF positions, compare with init_dof_pos 3. Build observation, log it 4. Run ONNX with zero latent, log action 5. Apply action to drives 6. Step physics ONCE (1/120s, not 4 substeps) 7. Read resulting state: root pos/vel, DOF positions, DOF deltas This isolates each pipeline stage to find where divergence occurs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
setMassAndUpdateInertia computes inertia from collision shapes which don't exactly match the MJCF geometry. Now directly setting: - mass from Newton reference - diagonal inertia tensor (Ixx, Iyy, Izz) from Newton reference - center of mass from Newton reference This eliminates inertia mismatch as a possible cause of the 9x faster fall rate observed in the Newton trace replay test. Also added Newton trace replay button and newton_trace.json for direct step-by-step physics comparison. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Set armature per joint (0.01-0.02) matching MJCF values - Set angularDamping, maxDepenetrationVelocity, maxAngularVelocity - Enable TGS solver matching Isaac Lab GPU PhysX - Set bounceThresholdVelocity=0.2 matching Isaac Lab - Add per-DOF action bounds from training env (joint limits * 1.2/1.4) - Clip actions in applyActions to prevent extreme drive targets - Set exact initial state from Isaac Lab trace in replay diagnostic - Add detailed per-DOF comparison logging in replay - Add Isaac Lab trace for comparison (10 steps with zero latent) - Add ovphysx CPU PhysX replay script for debugging Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…imit bounds The ONNX model outputs unbounded actions that the training env clips to joint-limit-based bounds. Without clipping, extreme actions (±11 rad) caused violent joint oscillations and character explosions. Key fixes: - Add cache buster to humanoid_data.json fetch (was serving stale cached version) - Compute per-DOF action bounds: 1.2x joint limits for spherical, 1.4x for revolute - Clip in applyActions to these bounds (matching Python env._apply_action) - Remove old a_mean±3*a_std clamp (too loose, allowed 3x beyond training bounds) - Increase substeps to 8 (240Hz physics) and solver iterations to 32 for CPU stability - Log raw vs clipped action ranges for debugging Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add exponential moving average smoothing (alpha=0.5) on policy actions to reduce oscillations from CPU/GPU PhysX dynamics mismatch - Revert solver iterations to 4,1 (PhysX default, matching Isaac Lab) - Revert substeps to 4 at 120Hz (matching training config) - Reset prevAction on humanoid reset - Character now stabilizes instead of exploding, but falls and stays down Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Explicitly set driveVelocity to 0 for all axes (was unset/default) - Reset prevAction on humanoid reset for clean smoothing state - Tune smoothing alpha to 0.8 (less aggressive than 0.5) - Use 32 solver iterations for better CPU PhysX drive convergence Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The v1 ONNX export manually reconstructed the actor network from checkpoint keys, which introduced subtle numerical differences (GPU vs CPU float32 arithmetic in normalization with very small std values). This caused the ONNX model to produce actions differing by up to 3.0 from PyTorch, making the character fall in Isaac Lab. The v2 export wraps the actual agent._model._actor_layers and _mean_net directly, guaranteeing bit-exact equivalence with PyTorch inference. Verified: v2 ONNX model maintains balance in Isaac Lab (root_z 0.56-0.86) while v1 model immediately fell (root_z → 0.30 by step 30). Added test_onnx_in_isaaclab.py for verifying ONNX models in the actual training environment. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The v2 ONNX export (from export_onnx_v2.py) uses the actual agent model layers rather than manual reconstruction. This fixes the 3.0 action error that caused the character to fall even in Isaac Lab. With the correct ONNX model: - Character stabilizes in a crouched pose (root_z ~0.42) in web PhysX - Actions stay moderate (±1.7) instead of exploding (±15) - Policy actively maintains equilibrium (constant obs/action each frame) - Remaining gap to standing is CPU vs GPU PhysX dynamics difference Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Logs the actual up-vector direction from PhysX root quaternion every 60 frames for diagnosing rotation observation issues. Verified: rotation observation is correct - the character at 50° tilt sees obs norm[2]=0.641 (correctly reflecting the tilt). The policy settles in a fallen state because it was trained with early termination on fall, so it never learned get-up behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Isaac Lab CPU PhysX also runs the policy successfully (root_z 0.57-0.83), confirming the ONNX model is correct. The CPU vs WASM PhysX replay shows 8cm divergence over 10 steps - both are CPU PhysX but likely different SDK versions (Isaac Lab uses Omniverse PhysX, web uses physx-js-webidl). The character survives 3 frames with correct height in web demo but eventually falls, suggesting the remaining issue is in scene/articulation configuration differences between Omniverse's MJCF importer and our manual PhysX WASM setup. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Isaac Lab USD inspection revealed key parameters: - Joint friction: 0 (PhysX default was 0.05 in our web demo) - Articulation solver: 32 pos, 1 vel (already set) - Rigid body solver: 16 pos, 1 vel (per-body, set by Omniverse) - cfmScale: 0.025 (we don't set this) - enableGyroscopicForces: true (we don't set this) - No contact/rest offset overrides (using defaults) - Scene: TGS solver, bounceThreshold=0.2, PCM collision system Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scene-level: - Friction type: patch - frictionCorrelationDistance: 0.025 - frictionOffsetThreshold: 0.04 Per-body (rigid body): - Per-body solver iterations: 16 pos, 1 vel - Sleep threshold: 5e-5 - Stabilization threshold: 1e-5 - CFM scale: 0.025 (constraint force mixing for stability) - Enable gyroscopic forces Articulation-level: - Sleep threshold: 5e-5 - Stabilization threshold: 1e-5 Per-joint: - Joint friction: 0 (was 0.05 PhysX default) - Max joint velocity: 1000000 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ASE latent space has no named skills — it's a continuous unsupervised manifold on a 64-dim unit hypersphere. Instead of static random latents: - Auto-cycle: sample new target every 30-150 steps (matching ASE training) - SLERP interpolation: smooth blend between current and target latent - Skills folder in GUI: toggle auto-cycle, adjust blend speed, manual trigger Set default substeps to 2 for better stability. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Prints copy-pasteable JSON array to console each time the latent cycles to a new target. Users can watch the character, find poses they like, and save the latent from the console for later use. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The policy was trained at 30Hz (4 substeps × 1/120s = 1/30s per step). Previously, the web ran one policy step per render frame regardless of frame rate, causing the simulation to run at different speeds depending on FPS and substep count. Now: accumulate real time and step policy+physics at exactly 30Hz (one step per 1/30s of real time). At most one step per render frame to prevent spiral-of-death on slow machines. Rendering happens every frame but physics only advances when enough real time has passed. substeps=4 (fixed, matching training). Substep slider still available in GUI for experimentation but 4 is the correct value. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Physics: substeps now run at 120Hz real-time (one per 1/120s of real time accumulated), with policy stepping every 4th substep (30Hz). At 60fps this gives ~2 physics steps per render frame for smooth visuals. At 30fps all 4 substeps run in one frame. Physics never runs faster than real-time thanks to the time accumulator. Latent paste: text field in Skills folder accepts a JSON array from the console log. Pasting a latent auto-disables cycling and holds that skill. Workflow: watch character → copy LATENT from console → paste to restore later. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Exports 4 high-level controllers from the ASE repo: - heading (walk in direction): 258-dim obs → 64-dim latent - location (walk to point): 255-dim obs → 64-dim latent - reach (hand to target): 256-dim obs → 64-dim latent - strike (sword attack): 268-dim obs → 64-dim latent NOTE: These HLCs expect the original ASE 253-dim LLC obs format, but MimicKit uses 158-dim obs. They need to be retrained or the obs gap (95 dims) needs to be bridged before they work in the web demo. For now they serve as reference for the architecture. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Skills folder now has a Mode dropdown: - Free: current behavior (random/manual latent control) - Heading, Location, Strike: stubbed out (Coming Soon) The HLC modes require retraining the high-level controllers for MimicKit's 158-dim obs format (the pretrained ASE HLCs expect 253-dim obs from the original ASE codebase). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
tools/encode_motions_to_latents.py runs each motion clip through the ASE encoder to get the corresponding latent vector, labeled by clip name (Atk_SlashDown, Shield_BlockLeft, WalkForward, Idle_Battle, etc.) web/latent_presets.json contains 79 named presets in categories: Attacks (26), Counters (5), Blocks (6), Parries (9), Locomotion (10), Idles (3), Taunts (3), Dodges (3), Turns (4) Skills dropdown in lil-gui loads presets by name. Selecting a preset sets the latent directly and disables auto-cycling. "Random" resamples a random unit vector. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Instead of averaging latents across the clip (which blurs distinct phases), take the element-wise signed maximum: for each of the 64 latent dimensions, pick the value with the largest absolute magnitude across all frames. This captures the peak/most characteristic activation in each dimension, producing a latent that emphasizes the clip's most extreme features. Normalized to unit sphere after. Dense sliding window: one encoder sample per 1/30s across the full clip duration (e.g., 297 frames for a 10s clip). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Skip L2 normalization after element-wise peak encoding to see if raw magnitudes produce stronger/more distinct skill behaviors. Norms are ~1.9-2.4 instead of 1.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…e vectors Each preset is now a timed sequence of latent keyframes (~10fps) sampled across the entire motion clip via dense sliding window. The web demo plays them back in a loop, interpolating smoothly between keyframes. Each keyframe is unit-normalized (from encoder). Example: Atk_2xCombo01 = 32 keyframes over 3.2s, capturing the full windup → slash → recovery trajectory through latent space. 79 presets, ~1.9MB total. Added tqdm progress bar to encoder script. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Move activeSequence declaration to module scope (was local to init, causing ReferenceError in mainLoop) - Set latent keyframes directly without interpolation — play at native animation speed (accumulator advances by DT per substep) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The sequence accumulator was adding DT (1/120s) but only ran once per policy step (every 4 substeps). Now adds numSubsteps*DT = 1/30s per policy step, so keyframes advance at real-time speed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
physx-js-webidl may not expose getShape(index) on articulation links. Gracefully skip filter data update if the API is unavailable. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Ports the ASE hierarchical RL architecture to MimicKit:
learning/hrl_agent.py (165 lines):
- HRLAgent extends PPOAgent
- Overrides action space to output 64-dim latent vectors
- Overrides _step_env to run frozen LLC for llc_steps per HLC step
- Combines task reward (from env) + discriminator reward (from LLC)
- Loads pre-trained ASE model as frozen LLC
Training:
python mimickit/run.py \
--arg_file args/hrl_steering_humanoid_sword_shield_args.txt \
--mode train
The HLC learns to select latent skills for task goals (steering,
location, strike) while the frozen LLC executes them as motor actions.
The trained HLC can be exported to ONNX for the web demo.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Patch env action space to latent_dim (64) for model and normalizer build - Wrap env with LLCEnvWrapper to hide task obs during LLC construction - Use _curr_obs from rollout loop for first LLC step - Add missing action_reg_weight to config - Handle gym/gymnasium import Training running: 4096 envs, RTX 4090, ~6GB VRAM, 131K samples/iter. Estimated ~3 hours for 50M samples. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Added timing instrumentation to _step_env (prints avg ms per call) - Added tqdm progress bar to _rollout_train for per-step visibility - Set normalizer_samples=0, test_episodes=0 to skip warmup/eval - Set llc_steps=5 (matching original ASE) Known issue: training is very slow (~4 min per iteration with 64 envs) due to serial LLC env.step() calls through Isaac Lab. Each HLC step does 5 serial env.step() calls. Needs optimization: - Use more envs (4096) to amortize per-call overhead - Or batch LLC steps into a single vectorized call - Or use Isaac Gym backend which has lower per-call overhead Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MimicKit's serial Python env.step() loop makes HRL training impractical (~5 hours per iteration with 4096 envs × 5 LLC steps). Removed: hrl_agent.py, HRL agent config, args file. Reverted: agent_builder.py, base_agent.py to main. Next approach: use original ASE LLC+HLC models directly by reconstructing the 253-dim ASE obs format in JavaScript (all body pos/rot/vel in heading frame), avoiding retraining. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
git add -A swept up untracked files that shouldn't be in the PR. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The ASE HLC models use a different obs format (253-dim) and PD gains (kp=5) than MimicKit's LLC (158-dim, kp=300+). Removed: - tools/export_hlc_onnx.py - tools/encode_motions_to_latents.py Latent presets (from the encoder) remain in web/latent_presets.json. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bake 17 hand-derived skill presets directly into index.html and remove the external JSON file and trajectory playback infrastructure. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reduces web demo to a single HTML file + ONNX model. Removes the separate parseMJCF.js and drops "v2" from the ONNX export tool name. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…s, trajectory comparison - Force PGS solver instead of TGS to match Isaac Lab PhysX 5.4 default - Set articulation solver iterations to 4/1 matching Isaac Lab scene caps (was 16/4) - Remove redundant per-link solver iteration overrides - Use PhysX link world positions for key body obs instead of FK (matches training) - Add trajectory playback/comparison system with ghost visualization and error metrics - Add tools/record_trajectory.py for recording Isaac Lab reference trajectories Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use getters for links, humanoidData, and articulation so external tooling (Playwright, devtools) always sees the current module-scoped variables, even after resetHumanoid() reassigns them. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PhysX has no analytic cylinder primitive. The previous capsule approximation was terrible for thin discs like the shield (radius=0.3, halfHeight=0.015 → effectively a sphere r=0.3). Now uses PxConvexMeshGeometry with a 16-sided polygon disc, giving proper flat collision geometry. Falls back to capsule if convex mesh cooking fails. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Strip trajectory comparison GUI, ghost visualization, recording hooks, buildObservationFromState, window._mk export, and tools/record_trajectory.py. These remain on the physx-parity-audit branch for future use. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PhysX WASM accepts 0 velocity iterations — no minimum clamping needed. Isaac Lab sets maxVelocityIterationCount=0 at the scene level, so the web demo should use 0 as well. Previously set to 1 under the incorrect assumption that PhysX WASM enforced a minimum of 1. Also removes unused driveScale variable from joint configuration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Reset button at top level (no folder wrapper) - Remove Auto-Cycle and Blend Speed controls - Latent field auto-populates on preset select and random - Remove Stats folder (internal only) - Add Lock Y Axis checkbox: uses a PhysX D6 joint between a static anchor and the root link, locking Y translation while freeing all other DOFs. Proper physics constraint, no teleportation hacks. - Post-reset hooks re-apply simulation settings (self-collision, gravity, drives, Y constraint) and preserve current latent/preset Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Live Demo
https://zalo.github.io/MimicKit/
ASE humanoid sword-shield character performing combat motions in real-time, running entirely in the browser — no server required.
Summary
Single-file ONNX character kit
The ONNX model file is a complete, self-contained character package:
The web demo extracts this metadata by scanning the ONNX protobuf binary for a sentinel key (`mimickit_config`), since onnxruntime-web doesn't expose `metadata_props` via its JS API. The baked MJCF is parsed at runtime by `parseMJCF.js` (DOMParser), which computes body mass, inertia tensors, and center-of-mass analytically from geom shapes.
To add a new character: train in Isaac Lab, run `export_onnx_v2.py`, drop the ONNX file into `web/`.
Key Technical Finding
PhysX WASM's spherical joints require DOF values mapped to consecutive axis order `(TWIST=0, SWING1=1, SWING2=2)` matching the expmap input. Isaac Lab's DOF ordering can have non-consecutive assignments (e.g. hips use `[0,2,1]`). Fixed by per-joint axis remapping at load time.
Files
Local Development
TODO
🤖 Generated with Claude Code