Skip to content

Add Adaptive Computation Time (ACT) layer#3111

Merged
davisking merged 35 commits into
davisking:masterfrom
Cydral:master
Sep 28, 2025
Merged

Add Adaptive Computation Time (ACT) layer#3111
davisking merged 35 commits into
davisking:masterfrom
Cydral:master

Conversation

@Cydral

@Cydral Cydral commented Aug 27, 2025

Copy link
Copy Markdown
Contributor

Implement adaptive_computation_time_ layer following Graves (2016) "Adaptive Computation Time for Recurrent Neural Networks".

This layer enables neural networks to adaptively determine the number of computation steps to perform for each sequence position, spending more computation on difficult parts while quickly processing easier parts.

Key features:

  • Full CPU and CUDA implementations
  • Configurable halting threshold and ponder penalty for regularization
  • Support for depth-dependent gradient scaling
  • Early termination optimization when all positions converge
  • Template-based max_steps configuration (act4, act8, act16 aliases)

Note: This ACT implementation can serve as a building block for hierarchical reasoning mechanisms such as those described in "Hierarchical Reasoning Model" (Wang et al., 2025, arXiv:2506.21734) for advanced sequential reasoning tasks.

Cydral and others added 30 commits April 28, 2025 22:10
…des an optimized linear transformation for multi-dimensional inputs.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Comment thread dlib/cuda/cuda_dlib.cu Outdated
Comment thread dlib/cuda/tensor_tools.h
@Cydral Cydral requested a review from davisking September 14, 2025 13:53
Repository owner deleted a comment Sep 18, 2025
@davisking

Copy link
Copy Markdown
Owner

Nice, this is great. Thanks for the PR :D

@davisking davisking merged commit d890f7d into davisking:master Sep 28, 2025
9 checks passed
@Cydral

Cydral commented Sep 29, 2025

Copy link
Copy Markdown
Contributor Author

No problem. My next plan is to work on a native Transformer implementation to make them easier to use within Dlib.
In fact, I think I've brought almost everything needed: the existing template structure used in samples works well to maintain the library's style while allowing for flexible configuration (number of layers, vocab size, etc.). I'll also use this opportunity to add an image preprocessing layer, creating a unified framework for Transformers in Dlib.
After that, training a respectable-sized model will be the next challenge, of course... Perhaps, the place for another example to add ;-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants