Training stability requires gradient clipping, but there's no clamp/clip gate
in the autograd engine. The forward pass is min(max(x, lo), hi); gradient is
zero outside the bounds, 1 inside.
Task
Add autograd/gates_basic.v or new gates_clip.v with forward and backward.
Wire into nn/optimizers/ as a clipper option (clip_grad_norm or clip_grad_value).
Files
autograd/gates_basic.v or new autograd/gates_clip.v
nn/optimizers/ — add clip option
Training stability requires gradient clipping, but there's no
clamp/clipgatein the autograd engine. The forward pass is
min(max(x, lo), hi); gradient iszero outside the bounds, 1 inside.
Task
Add
autograd/gates_basic.vor newgates_clip.vwith forward and backward.Wire into
nn/optimizers/as a clipper option (clip_grad_normorclip_grad_value).Files
autograd/gates_basic.vor newautograd/gates_clip.vnn/optimizers/— add clip option