Reservations and estimates

Some calls only tell you their size when they're done: an LLM call's output tokens, a job's compute time. Two problems follow:

Concurrent calls overspend. Ten calls start at once, each passes check, and together they blow through the limit.
You don't know what to check for. How many tokens should a call be allowed before it runs?

Reservations fix the first: reserve holds room before the call, and settle records what it really used. Estimates fix the second: the engine learns how big calls are and predicts the next one.

Reserve, settle, release

import { Limitr } from '@formata/limitr';

const limitr = await Limitr.new(`
policy: {
    credits: { ai_token: { label: 'AI token' } }
    plans: {
        pro: {
            default: true
            entitlements: {
                chat: { limit: { credit: 'ai_token', value: 10_000 } }
            }
        }
    }
}`);
await limitr.ensureCustomer('ada');

async function report() {
    const held = await limitr.held('ada', 'chat');
    const used = await limitr.value('ada', 'chat');
    console.log('held:', held, 'used:', used);
}

// Two calls start at once, each expecting about 6,000 tokens.
const first = await limitr.reserve('ada', 'chat', { value: 6_000 });
const second = await limitr.reserve('ada', 'chat', { value: 6_000 });
console.log('first:', first ? 'held' : 'denied', '| second:', second ? 'held' : 'denied');
await report();

// The first call finishes having used 4,500.
await limitr.settle('ada', 'chat', first, 4_500);
await report();

// Now there's room for another.
const third = await limitr.reserve('ada', 'chat', { value: 5_000 });
console.log('third:', third ? 'held' : 'denied');
await limitr.release('ada', 'chat', third!);   // the call failed: nothing is recorded
await report();

It prints:

first: held | second: denied
held: 6000 used: 0
held: 0 used: 4500
third: held
held: 0 used: 4500

Call	Returns	Does
`reserve(id, entitlement, { value?, context?, basis?, segment?, ttl?, quantile?, overhead? })`	hold ID or null	Holds `value`, or the predicted size of the call when `value` is left out. Null when the call wouldn't be allowed right now.
`settle(id, entitlement, holdId, value, event?, overhead?, basis?, segment?)`	bool	Drops the hold, records what the call used, and learns from it.
`release(id, entitlement, holdId)`	bool	Drops the hold without recording anything (the call never ran or failed).
`held(id, entitlement)`	number	Room held right now by unexpired holds.

In Stof: reserve(id, ent, value?, context?, basis?, segment?, ttl?, quantile = 0.9, overhead?) and the same settle, release, and held.

reserve decides like check, counting every other hold: limits, caps, grants, the governor, and the entitlement's enabled rule. Holds count against the meter's limit and the caps checked on it.
Holds are never billed. Another call's hold can deny a call, but it never turns an in-limit call into overage.
settle always records. It uses force, so the usage is metered even past a hard limit or cap: it already happened. An unknown or expired hold still records the usage. It sends the usual meter events, with event as the event data.
Holds expire after the policy's hold_ttl (10 minutes by default), or a ttl you pass (ms), so a crashed request can't hold room forever.
Pass the actual cost to settle as overhead when the provider reports it.

Estimates

Leave value out of reserve and it holds what it predicts the call will use. Ask for a prediction without holding anything with estimate:

const context = { model: 'large', input: 2000 };
const predicted = await limitr.estimate('ada', 'chat', { context });
// { value, overhead, samples, source, scope } or null

Field	Meaning
`value`	Predicted size of the call, in the credit's units, at the requested quantile.
`overhead`	Predicted cost, in runes.
`samples`	How many calls the estimate learned from.
`scope`	`'customer'` (this customer's own estimate) or `'policy'` (everyone's calls together).
`source`	`'local'`, or `'cloud'` when Limitr Cloud seeded it.

What it learns from. Every settle, and every observe(id, entitlement, value, { context?, overhead?, basis?, segment? }), which records a call without a hold.
Customer, then policy. Each customer's own estimate is used once it has estimate_min_samples calls (5 by default). Before that, the policy-wide estimate learned from every customer's calls.
Quantile. quantile: 0.9 (the default) is a cautious prediction most calls come in under: good for holds, max_tokens, and budget checks. 0.5 is the typical call: good for quotes and progress bars.
Before anything is learned, estimate returns null and reserve holds the limit's standard increment.
Constant memory. An estimate is a handful of numbers (recency-weighted means, variances, and covariances), not a history of calls.

Basis: what a call's size depends on

estimate_basis on the entitlement names the event data field a call's size depends on, Ex. a prompt's input tokens:

chat: {
    estimate_basis: 'input'         // a call's size depends on its input tokens
    estimate_segment: 'model'       // and each model learns its own estimate
    limit: { credit: 'ai_token', mode: 'soft', value: 2_000_000, resets: true }
}

With a basis, the estimate learns a line: a fixed amount per call, plus an amount per unit of the basis. That fits a chat answer that's about the same length whatever the question, a summary that grows with the text, and anything in between.

Until there are 5 calls whose bases differ (a spread of at least 10%), the estimate scales the average call by the basis instead, which holds cautiously.
Without a basis, estimates are per call (the average call).
Every reserve, settle, estimate, and observe on an entitlement with a basis needs that field in its event data. A call without it is still metered, but isn't learned from, and the engine logs it.
basis and segment arguments override what the event data says. A basis on an entitlement with no estimate_basis is ignored (and logged).
Changing estimate_basis starts its estimates over.

Segment: what keeps estimates apart

estimate_segment names the event data field that keeps separate estimates, Ex. per model, since a small model and a large one answer differently. Keep its values few: a model or a feature, never a user or request ID. Each value keeps an estimate in memory for as long as the engine runs.

Segment values are stored with . and spaces replaced by _ ('claude-sonnet-4.5' becomes claude-sonnet-4_5). With no segment, the key is all.

Predicting part of a call

An entitlement with no limit is never metered, so it can learn anything about a call on its own, Ex. output tokens from input tokens, to set max_tokens or quote a price before the call:

chat_output: { estimate_basis: 'input', estimate_segment: 'model' }

observe the output after each call and estimate it before the next. See Predict output tokens.

Where estimates live

Policy-wide estimates are in policy.estimates.<entitlement>.<segment>, and each customer's in its meter (meters.<entitlement>.estimates.<segment>), so they're saved with the customer. They're plain data: Cloud can seed them or replace them with numbers pooled across your traffic (source: 'cloud').

Estimate field	Meaning
`samples`	Calls learned from.
`alpha`	Smoothing (0.1): a plain average until 10 samples, then recent calls weigh more.
`per_basis`	Learned against a basis.
`value_mean`, `value_var`	The call's size.
`overhead_mean`, `overhead_var`	The call's cost.
`basis_mean`, `basis_var`, `value_cov`, `overhead_cov`	The line, with a basis.
`source`, `updated`	`'local'` or `'cloud'`, and the last update.

With a basis, the prediction is value_mean + (value_cov / basis_var) * (basis - basis_mean), plus the variance the line doesn't explain at the requested quantile.

Reserve, settle, release​

Estimates​

Basis: what a call's size depends on​

Segment: what keeps estimates apart​

Predicting part of a call​

Where estimates live​