Add transformer example with RoPE and MoE-like mechanisms

[Cydral]

This PR introduces a new example demonstrating:

1. Rotary Positional Embeddings (RoPE) implementation
2. Experimental Mixture-of-Experts (MoE) layer showing how to:
   • Extend Dlib's capabilities without modifying core library
   • Achieve better results than basic feed-forward layers
   • Note: current MoE implementation is simplified (2 experts) but provides a working template for expansion

Key features:

• Complete text processing pipeline (train/generate/verify modes)
• Reconstruction capability
• Memory-efficient sliding window training
• Custom BPE tokenizer integration

The MoE layer serves as both:

• A practical performance improvement over standard FFN layers
• An educational example of extending Dlib's neural network components

Implementation references:

• RoPE: https://arxiv.org/abs/2104.09864
• MoE: https://arxiv.org/abs/1701.06538

[pfeatherstone]

Have you looked at https://arxiv.org/pdf/2202.08906 and https://arxiv.org/pdf/2308.00951

[Cydral]

Have you looked at https://arxiv.org/pdf/2202.08906 and https://arxiv.org/pdf/2308.00951

Yes, I'm familiar with these mechanisms, although strict implementation would be greatly facilitated by support for dynamic network building—which is not typically the approach or philosophy behind Dlib. The example I provided is actually closer to a Soft MoE mechanism rather than a standard MoE.

That said, thanks to the most recent additions to Dlib, we now have quite a few tools available to build more "modern" networks, at least in line with recent publications. For example, for those interested in integrating causal attention in image processing, I've also published a fully functional and performant ViT-like architecture using Dlib (and a pre-computed model to test). I'm still experimenting with a few specific architectural patterns, and I hope to include a ViT example in the coming weeks.

Back to the MoE topic, I have the intuition that we might get close to dynamic networks via a sort of "network-in-a-network" approach—or more precisely, networks within a layer. I'm currently evaluating this possibility, and if results are promising, I’ll be sure to share an example as well.

[Cydral]

Of course, the shared example is just that—an example. For a more robust MoE implementation, we would likely need to add things like a Gaussian noise layer to improve the distribution across experts (ideally deactivated or made transparent during inference), implement a top-n ranking mechanism, and so on. But again, embedding such logic directly within a layer would significantly simplify the whole process.

[pfeatherstone]

I did think a couple years ago to create a new api which was dynamic but then gave up and now do everything in PyTorch and onnxruntime. I think the template based network building is no longer fit for purpose. I found it could take up to 20 minutes to compile a yolo model.

[davisking]

I did think a couple years ago to create a new api which was dynamic but then gave up and now do everything in PyTorch and onnxruntime. I think the template based network building is no longer fit for purpose. I found it could take up to 20 minutes to compile a yolo model.

Yeah I regret not making it all runtime classes instead of the template thing :| Oh well. I too use pytorch for DNN stuff as well. Still use dlib for many other things though.

[Cydral]

Despite these constraints, with the recent modifications, it is now possible to build networks in Dlib following the most "modern" architectures, whether for text sequence processing (LM) or image processing (ViT). I remember several people doubting that we could even implement causal attention in Dlib, yet I managed to produce - even with a templated structure - a functional and relatively performant building block, thus paving the way for bringing in all new, up-to-date models. I will also share a ViT example soon (a model is already available as an example). The key is simply allowing ourselves to not necessarily put all layers directly in Dlib to somewhat simplify future developments.

[pfeatherstone]

What would be super awesome is the ability to provide an attention predicate like the new flex_attention() API in pytorch. We could add a class template parameter in the attention layer which would be used when computing the attention score. For example:

struct causal_block_mask
{
    bool operator()(size_t b, size_t h, size_t q_idx, size_t kv_idx)
    {
        return q_idx >= kv_idx;
    }
};

Then use causal_block_mask in the attention layer template parameters.
Another example:

template<size_t window_size>
struct sliding_window_block_mask
{
    bool operator()(size_t b, size_t h, size_t q_idx, size_t kv_idx)
    {
         constexpr size_t hlen  = window_size // 2;
         return std::abs(q_idx - kv_idx) <= hlen;
    }
};

Hopefully this could be use to improve performance as well by not wasting compute.

[Cydral]

For now, I’ll stay focused on wrapping up the current work: finalizing the "Static MoE" example, then moving straight to a "Dynamic MoE" example (a draft is available here for first reference). As a result, I've completely refactored the implementation of the bpe_tokenizer class to provide a much more powerful version, capable of handling very large input texts for vocabulary learning.

[davisking]

Nice, this is cool. Sorry for the late merge. Forgot about this for a bit 😓

[Cydral]

No worries at all about the late merge, Davis!
Since we're here, I wanted to ask you about something I've been encountering with the example. While it works quite well overall (I can actually observe some interesting generalization phenomena when using a sufficient corpus - the model produces sentences that appear syntactically and even semantically at least partially correct), I'm running into a specific issue during inference.
I'm forced to inference with a 2-entry vector for things to work properly. When I use just a single entry (one sequence), the output comes out incorrect. I'm wondering if this might be related to my GPU configuration - I've noticed that using an NVIDIA A4000 stack doesn't allow me to validate test programs for instance.
Have you noticed anything particular on your end that might explain this behavior? Could there be a bug somewhere, or is this potentially hardware-specific? I've been searching around quite a bit without pinpointing the root cause. My current "workaround" solution isn't bad, but it obviously slows down inference.
Any insights would be greatly appreciated!

[davisking]

Yeah no idea. Sometimes there are GPU specific bugs, maybe that's what you are hitting. I haven't hit any in a while though so 🤷