Predict output tokens

Before a call runs you know its input, but not how long the answer will be. Limitr can learn that from the calls you already make, and predict it for the next one: to set max_tokens, quote a price before the call, size a progress bar, or route a request to a cheaper model.

Add an entitlement with no limit. It's never metered, so it only learns:

import { Limitr } from '@formata/limitr';

const limitr = await Limitr.new(`
policy: {
    credits: {
        ai_token: { label: 'AI token', price: { amount: 0.00002 } }
    }
    plans: {
        pro: {
            default: true
            entitlements: {
                // No limit, so it's never metered. It only learns:
                // output tokens from input tokens, per kind of call.
                answer_length: { estimate_basis: 'input', estimate_segment: 'kind' }
            }
        }
    }
}`);
await limitr.ensureCustomer('ada');

// Past calls: [kind, input tokens, output tokens].
// Chat answers are about 300 tokens whatever the question.
// Summaries are about a quarter of the text.
const history: [string, number, number][] = [
    ['chat', 40, 310], ['chat', 2500, 290], ['chat', 300, 305],
    ['chat', 1200, 300], ['chat', 80, 295], ['chat', 4000, 300],
    ['summary', 4000, 1010], ['summary', 800, 195], ['summary', 2000, 505],
    ['summary', 12000, 2990], ['summary', 6000, 1500],
];
for (const [kind, input, output] of history) {
    await limitr.observe('ada', 'answer_length', output, { context: { kind, input } });
}

// Ask before new calls: a typical answer (quantile 0.5), and a cautious one (0.9).
const upcoming = [
    ['chat', 100], ['chat', 8000],
    ['summary', 1000], ['summary', 8000],
] as const;
for (const [kind, input] of upcoming) {
    const context = { kind, input };
    const typical = await limitr.estimate('ada', 'answer_length', { context, quantile: 0.5 });
    const cautious = await limitr.estimate('ada', 'answer_length', { context, quantile: 0.9 });

    const expect = Math.round(typical!.value);
    const maxTokens = Math.round(cautious!.value);
    console.log(`${kind}, ${input} tokens: expect ${expect}, max_tokens ${maxTokens}`);
}

It prints:

chat, 100 tokens: expect 302, max_tokens 310
chat, 8000 tokens: expect 288, max_tokens 295
summary, 1000 tokens: expect 254, max_tokens 261
summary, 8000 tokens: expect 1997, max_tokens 2005

In your app

After each call, record the output with the same event data (next to settle, if you also reserve):

const context = { kind, input: usage.input_tokens };
await limitr.observe(userId, 'answer_length', usage.output_tokens, { context });

Before the next call, ask:

const context = { kind, input: inputTokens };
const predicted = await limitr.estimate(userId, 'answer_length', { context, quantile: 0.9 });

// Null until something has been learned for this kind of call.
const maxTokens = predicted ? Math.ceil(predicted.value) : 1024;

What it learns

A fixed amount per call plus an amount per input token: a line through your traffic.

• Chat learned about 300 tokens per call and almost nothing per input token, so a short question and a long one both predict about 300.
• Summaries learned almost nothing fixed and about a quarter of every input token, so the prediction grows with the text.
• Until there are 5 calls with different input sizes, it scales the average call by the input instead.

Choosing the numbers

• quantile: 0.5 is the typical answer: quotes, progress bars, picking a model.
• quantile: 0.9 (the default) is cautious: max_tokens, budget checks.
• Segments keep estimates apart. Use the field that changes the shape of the answer: the model, the kind of call, or both (kind: 'large:summary'). Keep the values few, never a user or request ID.
• Scope. Each customer's own estimate takes over after 5 of their calls. Before that, the prediction comes from every customer's calls together (scope: 'policy' in the result).

Already reserving?

reserve on a metered entitlement with estimate_basis already predicts each call's total tokens the same way. The separate entitlement is for when you need the output on its own.

See Reservations and estimates for every option.