# Rate limits next to paid usage

> Protect your infrastructure with rate limits (governors) that sit next to the usage customers pay for, never lifted by credit packs, with a Retry-After for your 429s.

# Rate limits next to paid usage

Some limits make money: 1,000 requests a day on the free plan. Others protect you: no customer gets more than a
couple of requests a second, however many credits they bought.

When the limit you sell is the limit you need, one entitlement can do both. When they differ, as they do once a rate
limit or an abuse ceiling comes in, give each its own entitlement, so raising a paid allowance never raises a safety
limit.

<Tabs groupId="lang" queryString>
<TabItem value="typescript" label="TypeScript">

```ts
const limitr = await Limitr.new(`
policy: {
    credits: {
        request: { label: 'Request' }
    }
    plans: {
        free: {
            default: true
            entitlements: {
                // Control: bursts of 5, then 2 requests a second. Credit packs never lift it.
                api_rate: {
                    limit: {
                        credit: 'request', value: 1e12, grants_apply: false
                        governor_enabled: true
                        governor_capacity: 5, governor_refill_rate: 0.002
                    }
                }
                // Monetize: 1,000 requests a day.
                api: {
                    limit: {
                        credit: 'request', value: 1000
                        resets: true, reset_inc: 1day
                    }
                }
            }
        }
    }
}`);
await limitr.ensureCustomer('ada');

const REFILL_PER_MS = 0.002;

// Middleware: returns an HTTP status, and when to retry if rate limited.
async function gate(userId: string): Promise<string> {
    if (!(await limitr.check(userId, 'api_rate', 1))) {
        const tokens = (await limitr.allowance(userId, 'api_rate')) ?? 0;
        // Seconds until the next token, for a Retry-After header.
const retryAfter = Math.ceil((1 - tokens) / REFILL_PER_MS / 1000);
        return `429 (retry after ${retryAfter}s)`;
    }
    if (!(await limitr.check(userId, 'api', 1))) return '402 (daily quota used)';
    await limitr.allow(userId, 'api_rate', 1);
    await limitr.allow(userId, 'api', 1);
    return '200';
}

const burst = [];
for (let i = 0; i < 7; i++) burst.push(await gate('ada'));
console.log(burst.join(', '));

await new Promise((resolve) => setTimeout(resolve, 1000));
console.log('a second later:', await gate('ada'), await gate('ada'), await gate('ada'));
```

It prints:

```text
200, 200, 200, 200, 200, 429 (retry after 1s), 429 (retry after 1s)
a second later: 200 200 429 (retry after 1s)
```

</TabItem>
<TabItem value="python" label="Python">

```python
from limitr import Limitr

limitr = Limitr("""
policy: {
    credits: {
        request: { label: 'Request' }
    }
    plans: {
        free: {
            default: true
            entitlements: {
                // Control: bursts of 5, then 2 requests a second. Credit packs never lift it.
                api_rate: {
                    limit: {
                        credit: 'request', value: 1e12, grants_apply: false
                        governor_enabled: true
                        governor_capacity: 5, governor_refill_rate: 0.002
                    }
                }
                // Monetize: 1,000 requests a day.
                api: {
                    limit: {
                        credit: 'request', value: 1000
                        resets: true, reset_inc: 1day
                    }
                }
            }
        }
    }
}""")
limitr.ensure_customer('ada')

REFILL_PER_MS = 0.002

# Middleware: returns an HTTP status, and when to retry if rate limited.
def gate(user_id):
    if not limitr.check(user_id, 'api_rate', 1):
        tokens = limitr.allowance(user_id, 'api_rate') or 0
        # Seconds until the next token, for a Retry-After header.
        retry_after = math.ceil((1 - tokens) / REFILL_PER_MS / 1000)
        return f'429 (retry after {retry_after}s)'
    if not limitr.check(user_id, 'api', 1):
        return '402 (daily quota used)'
    limitr.allow(user_id, 'api_rate', 1)
    limitr.allow(user_id, 'api', 1)
    return '200'

print(', '.join(gate('ada') for _ in range(7)))

time.sleep(1)
print('a second later:', gate('ada'), gate('ada'), gate('ada'))
```

It prints:

```text
200, 200, 200, 200, 200, 429 (retry after 1s), 429 (retry after 1s)
a second later: 200 200 429 (retry after 1s)
```

</TabItem>
</Tabs>

## How it works

- **`api_rate` is control.** Its huge `value` means the limit itself never matters; the
  [governor](../reference/entitlements#governors) does the work. A bucket of 5 tokens (`governor_capacity`) refills
  at 0.002 tokens per millisecond (2 a second), and each request spends one. A burst of 5 goes through, then the
  customer has to slow down. Denials send `meter-governed`.
- **`grants_apply: false`** means purchased credits never lift it.
- **`api` is monetize.** The quota the customer's plan includes.
- **Check control first**, so a denied request says why: 429 (slow down) is different from 402 (out of quota).
- **`allowance`** is what can run right now, counting the governor. Here it gives the wait for a `Retry-After`
  header.

## Picking the numbers

| Limit | Governor |
|---|---|
| 10 a second, bursts of 20 | `governor_capacity: 20, governor_refill_rate: 0.01` |
| 60 a minute | `governor_capacity: 60, governor_refill_rate: 0.001` |
| 1,000 an hour, no bursts | `governor_capacity: 1, governor_refill_rate: 0.000278` |

`refill_rate` is tokens per millisecond: requests per second divided by 1,000.

## Variations

- **Per plan.** Each plan has its own `api_rate`, so Pro customers get a faster bucket.
- **By cost, not count.** Meter tokens instead of requests: each call spends its value in tokens, so a governor on
  AI tokens limits tokens per second.
- **A daily abuse ceiling.** A hard limit with a daily reset and `grants_apply: false`, next to the paid allowance.
- **One customer gets more.** Give them a faster plan's rate limit: `giveEntitlement(id, 'api_rate', 'pro')`.
- **Concurrent requests.** Each call to the engine is atomic. Under heavy concurrency, `check` then `allow` can let
  an extra request through between them; when that matters, `allow` the rate limit first and treat a later denial of
  the quota as the answer.
