Your LLM bill doubled last month because a single user ran a script that hammered your chat endpoint 847 times in three hours. You add rate limiting. Now legitimate users are seeing “Too many requests” errors during normal workflow, and your support queue is filling up with complaints.
This is the rate limiting paradox for AI-native products: you need limits to control costs and prevent abuse, but naive implementations destroy the user experience that made your product valuable in the first place. After engineering LLM integrations for dozens of products, we’ve learned that effective rate limiting requires rethinking both the technical implementation and the product design around it.
Why LLM Rate Limiting Is Different
Traditional API rate limiting assumes relatively uniform request costs. A database query costs roughly the same whether it returns one row or fifty. LLM calls are wildly heterogeneous in cost, latency, and user value.
A simple autocomplete suggestion might cost $0.0001 and complete in 200ms. A document analysis could cost $0.50 and take 12 seconds. Both hit the same endpoint. If you limit to “10 requests per minute,” you’re either leaving money on the table or creating terrible UX.
The fundamental mistake is counting requests instead of measuring what actually matters: token consumption, cost, and user intent.
Token-Based Budgets Over Request Counts
Instead of limiting requests, limit token budgets. Give each user a replenishing pool of tokens they can spend however they want. This aligns your rate limiting with your actual costs and gives users flexibility in how they use your product.
// Token-based rate limiter implementation
class TokenBudgetLimiter {
constructor(redis, budgetConfig) {
this.redis = redis;
this.budgetConfig = budgetConfig;
}
async checkAndConsume(userId, estimatedTokens) {
const key = `token_budget:${userId}`;
const budget = this.budgetConfig[getUserTier(userId)];
// Get current usage
const current = await this.redis.get(key);
const used = parseInt(current || '0');
// Check if budget allows this request
if (used + estimatedTokens > budget.maxTokens) {
const ttl = await this.redis.ttl(key);
return {
allowed: false,
resetIn: ttl > 0 ? ttl : budget.windowSeconds,
remaining: Math.max(0, budget.maxTokens - used)
};
}
// Consume tokens
const newUsed = await this.redis.incrby(key, estimatedTokens);
// Set expiry on first use in window
if (used === 0) {
await this.redis.expire(key, budget.windowSeconds);
}
return {
allowed: true,
remaining: budget.maxTokens - newUsed,
resetIn: await this.redis.ttl(key)
};
}
}This approach requires estimating tokens before the LLM call. For chat completions, use the input length plus your max_tokens setting. For embeddings, count the input. The estimates don’t need to be perfect—you’re optimizing for cost control, not accounting precision.
Tiered Limits Based on Operation Value
Not all LLM operations deliver equal user value. A user generating their tenth variation of the same marketing email is different from someone using AI to analyze a critical document for the first time.
Implement operation-specific quotas that reflect both cost and value:
- High-value, high-cost operations (document analysis, code generation): Lower limits, higher per-operation cost
- High-frequency, low-cost operations (autocomplete, suggestions): Higher limits, lower per-operation cost
- Retry/regenerate operations: Increasingly expensive, discouraging low-effort iteration
The key is making the cost structure visible and predictable. Users will self-regulate if they understand the trade-offs.
Progressive Degradation, Not Hard Walls
When a user approaches their limit, don’t just shut them down. Engineer a graceful degradation path:
- Warning phase (80% of budget): Show usage indicator, suggest they prioritize important requests
- Soft limit (100% of budget): Switch to cheaper models (GPT-4 → GPT-3.5, Claude Opus → Sonnet), reduce max_tokens, add slight delays
- Hard limit (120% of budget): Only then block requests, with clear messaging about when limits reset
This approach keeps users productive while protecting your margins. In our implementations, progressive degradation reduces hard-limit collisions by 60-70% while keeping cost overruns under 5%.
Smart Queuing for Burst Traffic
Users don’t work at steady rates. They upload a document and trigger five LLM operations simultaneously. They paste a long email thread and want it analyzed. Burst traffic is normal user behavior, not abuse.
Instead of rejecting burst requests, queue them and process them at your preferred rate. Show users a real-time progress indicator. Most users will wait 30 seconds for processing if they can see it happening.
async function queueLLMRequest(userId, operation, priority = 'normal') {
const queueKey = `llm_queue:${userId}`;
const request = {
id: generateId(),
operation,
priority,
timestamp: Date.now()
};
await redis.zadd(queueKey,
priority === 'high' ? Date.now() - 10000 : Date.now(),
JSON.stringify(request)
);
// Return immediately with request ID for status polling
return request.id;
}This transforms rate limiting from a frustration into a predictable, manageable wait. Users understand queues. They don’t understand arbitrary request denials.
Cache Aggressively, Especially Near Limits
When users approach their rate limits, they become more likely to retry identical or near-identical requests. This is exactly when caching delivers maximum value.
Implement semantic caching that catches similar requests, not just exact matches. Hash the user intent, not the raw input. If someone asks “summarize this document” and then “give me a summary of this,” that’s the same request.
Near the rate limit threshold, increase cache TTLs and reduce similarity thresholds. A cached response that’s 90% relevant is better than no response.
Expose Limits as a Product Feature
The best rate limiting systems make limits visible, understandable, and controllable. Add a usage dashboard showing token consumption by operation type. Let users see which features are expensive. Give them tools to manage their own usage.
When limits become a product feature rather than a hidden constraint, user behavior changes. Power users upgrade to higher tiers. Casual users learn to use expensive features more deliberately. Everyone builds better mental models of what your product actually costs to run.
Monitor What Actually Predicts Problems
Track metrics that predict rate limit issues before they impact users:
- Token budget exhaustion rate (percentage of users hitting 100% in a window)
- Retry frequency after limit warnings
- Average tokens per operation by feature
- Ratio of cached to uncached responses
- Time between limit resets and next request (shows pent-up demand)
These metrics tell you where your limits are too tight, where users are hitting unexpected constraints, and where you need better UX around the limits themselves.
The Real Goal Is Cost Predictability
Rate limiting exists to make your LLM costs predictable and sustainable. But predictability matters for users too. They need to predict whether their workflow will complete, whether they should save a complex task for later, whether they’re about to burn through their quota on something trivial.
Engineer your rate limiting to create shared predictability. Make the system’s constraints clear. Give users agency in how they spend their budget. Build degradation paths that keep the product functional even under constraint.
The teams that get this right don’t treat rate limiting as a necessary evil. They treat it as a core product design challenge that shapes how users interact with AI. When you ship limits that users understand and work with rather than fight against, you’ve built something sustainable for both your infrastructure and your user experience.
