Are you going to try encoder-decoder transformer?

[dinhanhx]

For example, you can compare with BART.

[dinhanhx]

I notice that there are some asserts to make sure the model doesn't accept encoder hidden state and cross attention.

simplified_transformers/simplified_transformers/model_utils.py

Line 197 in 1d8eb6d

assert is_cross_attention == False

simplified_transformers/simplified_transformers/model_utils.py

Line 408 in 1d8eb6d

assert encoder_hidden_states is None

Let say, I want to implement encoder-decoder-ish transformer. How can I use with your simplified transformer blocks?

[bobby-he]

Hi, thanks for the question. To be honest we haven't thought much about the encoder-decoder attention. I think it would be compatible with our simplified transformer block, with some additional modifications to shaped attention to account for the encoder representations, though I haven't tried it out. If there was interest in implementing it to test if it works I would be happy to be involved.

[dinhanhx]

Yesterday I tried to read your code. I'm still not confident. Maybe I haven't understood well enough. I also think it's doable with a bit modification to your layer. Obviously, your code is based on GPT2Attention

https://github.com/huggingface/transformers/blob/v4.35.1/src/transformers/models/gpt2/modeling_gpt2.py#L123

https://github.com/bobby-he/simplified_transformers/blob/main/simplified_transformers/model_utils.py#L191

If there was interest in implementing it to test if it works I would be happy to be involved.

I may want to implement such things. But I think I will have lots of questions about shaped attention.

[dinhanhx]

So, compared to the OG Transformer block, eq 13rd doesn't have weight for V and does have some extra alpha, beta, gamma weight, right?