Understanding meta-learning

[embarrassing-noob-questions]

Hi, thanks for this repo, and please excuse the noob question: I am trying to understand the meta-learning side better.

If it's ok for you I will focus first on Titans/ATLAS, because you seem to understand it well:

Suppose I want to train a model to do well on unseen long context problems, or in other words - I want to teach a model to use its memory effectively to do well on long context (= contagious sequences that are longer than its context window).

So I would think that at training I will give it long texts, many times longer than its context window, and after each text like that I will somehow re-init the memory - the same way I would do at inference.

I don't see any of this in the code - means I am missing something

[weipeilun]

We re-init fast weights to (persistant) weight, which is the model's parameter learned by the outer optimization.
This happens when state is None, which is at the start of each sequence, at both training and test time.
The inner loop optimize the fast weights even at test time, while the outer loop optimize the (persistent) weight only in pretraining.

[embarrassing-noob-questions]

Is the sequence split into smaller segments each processed with full causal attention like in the Titans MAC implementation? Or is it more of a sliding window like in the Titans MAG / ATLAS implementations?

[weipeilun]

Maybe we can start from modern RNNs such as RWKV and RetNet, or State-Space Models like Mamba. In contrast, I would not say Titans as either full causal attention or a sliding window approach (e.g., Swin Transformers or CNNs). Instead, Titans (including MAC and MAG) and ATLAS are all just memories, where information is selectively added to and removed from memory based on surprise. lucidrains implementation looks like a full causal attention but it's actually a performance optimization for the sequential process (from O(n) to O(log n), see the assoc_scan part)

[embarrassing-noob-questions]

In Titans it seems that the idea is that long context is split to segments, each processed with full attention, and context is carried between segments via memory, and this is why it can scale to much longer contexts not seen in training.
What I am not sure about is if each segment is processed through all the layers, with output potentially generated auto-regressively, and then the next segment is processed, and so on?

[weipeilun]

Yes indeed.