Rate limits next to paid usage

Some limits make money: 1,000 requests a day on the free plan. Others protect you: no customer gets more than a couple of requests a second, however many credits they bought.

When the limit you sell is the limit you need, one entitlement can do both. When they differ, as they do once a rate limit or an abuse ceiling comes in, give each its own entitlement, so raising a paid allowance never raises a safety limit.

import { Limitr } from '@formata/limitr';

const limitr = await Limitr.new(`
policy: {
    credits: {
        request: { label: 'Request' }
    }
    plans: {
        free: {
            default: true
            entitlements: {
                // Control: bursts of 5, then 2 requests a second. Credit packs never lift it.
                api_rate: {
                    limit: {
                        credit: 'request', value: 1e12, grants_apply: false
                        governor_enabled: true
                        governor_capacity: 5, governor_refill_rate: 0.002
                    }
                }
                // Monetize: 1,000 requests a day.
                api: {
                    limit: {
                        credit: 'request', value: 1000
                        resets: true, reset_inc: 1day
                    }
                }
            }
        }
    }
}`);
await limitr.ensureCustomer('ada');

const REFILL_PER_MS = 0.002;

// Middleware: returns an HTTP status, and when to retry if rate limited.
async function gate(userId: string): Promise<string> {
    if (!(await limitr.check(userId, 'api_rate', 1))) {
        const tokens = (await limitr.allowance(userId, 'api_rate')) ?? 0;
        // Seconds until the next token, for a Retry-After header.
const retryAfter = Math.ceil((1 - tokens) / REFILL_PER_MS / 1000);
        return `429 (retry after ${retryAfter}s)`;
    }
    if (!(await limitr.check(userId, 'api', 1))) return '402 (daily quota used)';
    await limitr.allow(userId, 'api_rate', 1);
    await limitr.allow(userId, 'api', 1);
    return '200';
}

const burst = [];
for (let i = 0; i < 7; i++) burst.push(await gate('ada'));
console.log(burst.join(', '));

await new Promise((resolve) => setTimeout(resolve, 1000));
console.log('a second later:', await gate('ada'), await gate('ada'), await gate('ada'));

It prints:

200, 200, 200, 200, 200, 429 (retry after 1s), 429 (retry after 1s)
a second later: 200 200 429 (retry after 1s)

How it works

• api_rate is control. Its huge value means the limit itself never matters; the governor does the work. A bucket of 5 tokens (governor_capacity) refills at 0.002 tokens per millisecond (2 a second), and each request spends one. A burst of 5 goes through, then the customer has to slow down. Denials send meter-governed.
• grants_apply: false means purchased credits never lift it.
• api is monetize. The quota the customer's plan includes.
• Check control first, so a denied request says why: 429 (slow down) is different from 402 (out of quota).
• allowance is what can run right now, counting the governor. Here it gives the wait for a Retry-After header.

Picking the numbers

Limit	Governor
10 a second, bursts of 20	governor_capacity: 20, governor_refill_rate: 0.01
60 a minute	governor_capacity: 60, governor_refill_rate: 0.001
1,000 an hour, no bursts	governor_capacity: 1, governor_refill_rate: 0.000278

refill_rate is tokens per millisecond: requests per second divided by 1,000.

Variations

• Per plan. Each plan has its own api_rate, so Pro customers get a faster bucket.
• By cost, not count. Meter tokens instead of requests: each call spends its value in tokens, so a governor on AI tokens limits tokens per second.
• A daily abuse ceiling. A hard limit with a daily reset and grants_apply: false, next to the paid allowance.
• One customer gets more. Give them a faster plan's rate limit: giveEntitlement(id, 'api_rate', 'pro').
• Concurrent requests. Each call to the engine is atomic. Under heavy concurrency, check then allow can let an extra request through between them; when that matters, allow the rate limit first and treat a later denial of the quota as the answer.