Current Claude Opus 4.5 degradation is not being reflected

[wheelerz]

There are a few possibilities -- among them the degradation is unevenly distributed (some accounts/keys and not others), or they are gaming your tests while degrading elsewhere.

What is the solution to this? A random distributed test generated from first principles rather than tests made statically available open-source? I haven't looked to see how you are currently handling this adversarial possibility, if at all.

[wheelerz]

https://www.reddit.com/r/ClaudeAI/comments/1qb4j7u/claude_opus_output_quality_degradation_and/
https://www.reddit.com/r/ClaudeCode/comments/1q8z0rh/opus_45_has_gone_dumb_again/
https://www.reddit.com/r/GithubCopilot/comments/1q4uc50/opus_45_degredation/

[StudioPlatforms]

Hey, thanks for bringing this up, you've identified an important limitation in what we currently measure.

Right now we only test the official Anthropic API endpoints (api.anthropic.com), not the claude.ai web interface. These are actually separate systems, and they can behave quite differently. Anthropic might deploy different model versions to each platform, apply different safety filters, or run A/B tests on subsets of web users. It's entirely possible for the API to perform fine while the web interface degrades, or vice versa.

The reports you linked from Reddit about claude.ai degradation are legitimate user experiences, but they're about the web chat interface which we don't currently monitor. Our benchmarks focus on the developer API because that's what most people building production applications actually use. If you're integrating Claude into your product, API performance is what matters.

We do plan to expand our monitoring to include web interfaces in the future, but for now we're focused on the official API endpoints that developers rely on. In the meantime, we'll add a clearer disclaimer on the site explaining this distinction.

As for your point about anti-gaming, you're thinking along the right lines. We do have measures in place like hidden test cases, randomized task selection, and prompt versioning to prevent providers from optimizing specifically for our benchmarks. But you're right that there's always a theoretical risk, which is why transparency about our methodology is so important.