AI Interview DropAdvanced TierScenario+50 XP on read
Your LLM feature costs $40k/month and is growing 30% monthly. Leadership wants it cut by 70% without hurting quality. What do you do?
#LLM#Cost#Inference#Caching
Core Summary
Most LLM spend is wasted on tokens that did not need to be generated, requests that did not need a frontier model, and identical prefixes reprocessed on every call. Measure where the money goes first, then attack caching, routing and prompt size before anything as drastic as self-hosting.
Hints
Hint 1: Cost is tokens times price — attack both factors separately
Hint 2: Not every request needs the largest model
Hint 3: Identical prompt prefixes are being paid for repeatedly
Reported in interviews at OpenAI, Anthropic, Notion