Only Linear and Conv2D have CUDA backward. All other layers lack GPU backward:
LSTM, Attention, BatchNorm, LayerNorm, Embedding, MaxPool, Dropout (training mode).
This blocks full GPU training for models using these layers.
Task
For each layer type, add CUDA backward kernels:
nn/internal/lstm_backward_cuda_d_cuda.v
nn/internal/attention_backward_cuda_d_cuda.v
nn/internal/batchnorm_backward_cuda_d_cuda.v
nn/internal/layer_norm_backward_cuda_d_cuda.v
nn/internal/embedding_backward_cuda_d_cuda.v
nn/internal/pool_backward_cuda_d_cuda.v
Verification
- Each backward matches CPU reference within
1e-4 relative error
- Gradient check utility confirms correctness
Only Linear and Conv2D have CUDA backward. All other layers lack GPU backward:
LSTM, Attention, BatchNorm, LayerNorm, Embedding, MaxPool, Dropout (training mode).
This blocks full GPU training for models using these layers.
Task
For each layer type, add CUDA backward kernels:
nn/internal/lstm_backward_cuda_d_cuda.vnn/internal/attention_backward_cuda_d_cuda.vnn/internal/batchnorm_backward_cuda_d_cuda.vnn/internal/layer_norm_backward_cuda_d_cuda.vnn/internal/embedding_backward_cuda_d_cuda.vnn/internal/pool_backward_cuda_d_cuda.vVerification
1e-4relative error