Fast Feedforward is slower

[superFilicos]

Hi!
I'm also focusing on the project Fast feedforward, and when I use it in my model I found that it was much slower than Linear. I also use your code and get the same result.

So, did you encounter the same problem with me?

[issamemariold]

Indeed I also get similar results. I noticed however that you can get a speedup in extreme cases. For example under the following configuration:

in_features = 2000
leaf_features = 5
out_features = 2000
depth = 4
batch_size = 500
max_iter = 1000

I get ~1.7x speedup:

FastFeedForward avg batch inference duration: 0.027593517475409817
Linear avg batch inference duration: 0.04902171762295078
Speedup: 1.7765664586487342
FastFeedForward avg batch inference duration: 0.027575688951612876
Linear avg batch inference duration: 0.04907984716129028
Speedup: 1.7798230625320297
FastFeedForward avg batch inference duration: 0.027542462952380928
Linear avg batch inference duration: 0.04911540638095234
Speedup: 1.7832612306992872

[superFilicos]

Hi!
Yes, but you need to set the hidden_size to more than 2000... I wish maybe someday I can have such powerful gpus...