feat: support server side response cache

[kemingy]

Describe the feature

refer to:

• https://github.com/triton-inference-server/server/blob/main/docs/user_guide/response_cache.md

Some ML models might benefit from the cache.

As for the storage part, I think ideally we should support both local and remote cache.

Why do you need this feature?

No response

Additional context

No response

[AlexXi19]

Hey Keming, interested in taking a look at this issue, I briefly looked into some rust crates for this feature and found this crate. This crate seems to have support for redis cache, sized cache and timed cache (although i dont believe they have timed + sized cache). My first thought would be to add an axum middleware to handling the caching logic. What are your thoughts on this?

[kemingy]

Hey Keming, interested in taking a look at this issue, I briefly looked into some rust crates for this feature and found this crate. This crate seems to have support for redis cache, sized cache and timed cache (although i dont believe they have timed + sized cache). My first thought would be to add an axum middleware to handling the caching logic. What are your thoughts on this?

I think this PR should come with a benchmark. I don't know if this lib fits our requirements.

• multi routes
• local & remote cache
• cache TTL
• cache size limit

I don't know how it handles the cache key. Since the key/value could be a huge image (like 3 x 1000 x 1000 f32). The benchmark should include different key/value types like a simple string, an image, an embedding, etc.

[AlexXi19]

Good point. Do you think the cache should be aware of the exact content type?

[kemingy]

Good point. Do you think the cache should be aware of the exact content type?

No. Because we don't really parse the HTTP request body on the Rust side. I list different types of data just because their sizes are different.

[kemingy]

For the benchmark, you can check https://github.com/tensorchord/inference-benchmark/tree/main/benchmark

[kemingy]

A possible solution: sqlite.

• pros
  • easy to use
  • high performance
  • support both RAM & disk
  • work well even when the cache miss rate is high
• cons:
  • doesn't support hybrid storage (tier 1 with RAM and tier 2 with disk)
    • can increase the cache_size
  • TTL and size limit require some SQL to implement