Skip to main content

Reserve and settle LLM calls

An LLM call's size is only known when it's done, and calls run at the same time. check then allow can't handle that: ten calls can each pass a check that together they blow through. Wrap every model call in reserve, call, settle:

import { Limitr } from '@formata/limitr';

const limitr = await Limitr.new(`
policy: {
    credits: {
        ai_token: { label: 'AI token', price: { amount: 0.00002 }, overhead_cost: 0.000008 }
    }
    plans: {
        free: {
            default: true
            entitlements: {
                chat: {
                    // A call's size depends on its input tokens
                    estimate_basis: 'input'
                    // Each model learns its own estimate
                    estimate_segment: 'model'
                    limit: {
                        credit: 'ai_token', value: 10_000
                        resets: true, reset_sch: 'monthly:1'
                    }
                }
            }
        }
    }
}`);
await limitr.ensureCustomer('ada');

// Stands in for your model provider: answers are about 400 tokens.
async function callModel(model: string, input: number) {
    await new Promise((resolve) => setTimeout(resolve, 20));
    return { model, input, output: 380 + (input % 50) };
}

// Every model call goes through here.
async function chat(userId: string, model: string, input: number): Promise<string> {
    const context = { model, input };
    const hold = await limitr.reserve(userId, 'chat', { context });
    if (!hold) return 'denied';
    try {
        const usage = await callModel(model, input);
        await limitr.settle(userId, 'chat', hold, usage.input + usage.output, usage);
        return `used ${usage.input + usage.output}`;
    } catch (error) {
        // The call failed: nothing to record
        await limitr.release(userId, 'chat', hold);
        throw error;
    }
}

// Teach the estimate with a few calls, one at a time.
for (const input of [200, 900, 400, 1200, 600]) {
    await chat('ada', 'large', input);
}
console.log('used so far:', await limitr.value('ada', 'chat'));

const next = await limitr.estimate('ada', 'chat', { context: { model: 'large', input: 800 } });
console.log('an 800-token prompt should use about', Math.round(next?.value ?? 0));

// Then eight calls at once. Holds count each other, so they can't overspend together.
const inputs = [700, 800, 900, 700, 800, 900, 700, 800];
const results = await Promise.all(inputs.map((input) => chat('ada', 'large', input)));
console.log(results.join(', '));
console.log('used:', await limitr.value('ada', 'chat'), 'of 10000');

It prints:

used so far: 5200
an 800-token prompt should use about 1180
used 1080, used 1180, used 1280, used 1080, denied, denied, denied, denied
used: 9820 of 10000

The pattern​

  1. reserve before the call. It decides like check (limits, caps, grants, rules), counting every other open hold, and holds room for this call. With no value, it holds what it predicts the call will use. It returns a hold ID, or null when the call shouldn't run: return a 402 or 429, or offer an upgrade.
  2. Call the model.
  3. settle with what the provider says the call used. It records the real usage (even past a hard limit, since it already happened), lets go of the hold, and teaches the estimate. Pass the same event data, with the output tokens, so a cost function prices the call.
  4. release instead if the call failed. Nothing is recorded.

Holds expire after 10 minutes (the policy's hold_ttl), so a crashed request can't hold room forever.

What the hold should be​

  • Let the engine predict it. estimate_basis: 'input' tells it a call's size depends on its input tokens, and estimate_segment: 'model' keeps one estimate per model. It learns a fixed amount per call plus an amount per input token, and holds at the 90th percentile, so most calls come in under their hold.
  • Before it has learned anything, it holds the limit's standard increment. With only a few calls, holds are cautious. After 5 calls with different input sizes, they track real usage closely.
  • Hold a fixed amount instead with { value: 4000 }, Ex. your max_tokens.
  • Each customer's own estimate takes over after 5 of their calls; before that, every customer's calls together.

Things to get right​

  • Always settle or release. Use try/catch (or finally) so a thrown error releases the hold.
  • Pass the basis every time. On an entitlement with estimate_basis, every reserve and settle needs that field in its event data. Without it, the call is metered but not learned from, and the engine logs it.
  • Keep segments few: a model name, never a user or request ID.
  • Bill from events: settle sends the usual meter events, meter-overage included. See Bill from events.
  • Pass the provider's cost to settle as overhead when it reports one.

Streaming​

For a streamed response, reserve before the stream starts and settle when it ends, with the final usage (most providers report it in the last chunk). If the client disconnects, settle what was generated so far, or release the hold when nothing was.

Next​

Last updated on