Limitr 0.7: Real Costs, Predicted Usage, and Feature Flags in the Open-Source Engine

Limitr 0.7 is out (0.7.3 as of this post). It's the biggest release of the open-source engine so far, and most of it came straight from conversations with teams putting AI features in front of paying customers.

Three questions kept coming up:

• What did this call actually cost us? Not an average, this call, on this model, at today's provider prices.
• How much room should a call get before we know its size? An LLM call's usage is only known after it finishes.
• Who gets this feature right now? Plans decide most of it, but launches, betas, and custom deals don't fit neatly into plans.

0.7 answers all three inside the same policy, running in your process, with no network call on the hot path.

npm i @formata/limitr

Costs that change on every call

A fixed cost per token was never true for AI features. Input and output tokens are priced differently, every model has its own rates, and providers change their prices.

In 0.7, a credit can carry a cost function: a small function in the policy that prices one call from the call's own data.

credits: {
    ai_token: {
        price: { amount: 0.00002 }                 // what you charge per token
        overhead_cost: 0.000005                    // fallback cost per token
        rates: {                                   // what each model costs you
            large: { input: 0.000003, output: 0.000015 }
            small: { input: 0.000001, output: 0.000005 }
        }
        fn overhead(units: float, context?: obj) -> float {
            const rate = self.rates.get(context?.model ?? '');
            if (rate == null) return units * self.overhead_cost;
            const input = context.input ?? 0;
            const output = context.output ?? Num::max(units - input, 0);
            input * rate.input + output * rate.output
        }
    }
}

context is the event data you already pass with the call ({ model, input, output }). Because the rates live in the policy, you change them when a provider changes its prices, without a deploy. If your provider reports the exact cost of a call, pass it to allow and the engine records that instead.

Around it:

• Cost is recorded per period, call by call. A price change mid-month never re-prices the usage that came before it.
• Every meter event carries the call's cost and its list price (meter.overhead_diff and meter.price_diff), so every event has the call's margin. meter-overage adds overage_price: exactly what to bill, after credits.
• Cost with no usage. A failed call that still cost you can be recorded as cost alone.
• Spend caps say when they stop a call. A call denied by a cap now sends a cap-limit event. Before, cap denials were silent.
• Several credits can share one cost function, written once in a Stof type.

Reserve before the call, settle after

An LLM call's size is only known when it's done, and ten calls running at once can each pass a limit check that together they blow through. Reservations fix both:

const hold = await limitr.reserve(userId, 'chat', { context: { model, input: inputTokens } });
if (!hold) return paymentRequired();
try {
    const res = await callModel(model, messages);
    await limitr.settle(userId, 'chat', hold, res.usage.input_tokens + res.usage.output_tokens,
        { model, input: res.usage.input_tokens, output: res.usage.output_tokens });
} catch (e) {
    await limitr.release(userId, 'chat', hold);
    throw e;
}

reserve checks the call and holds room for it, so concurrent calls count each other. settle records what really happened, even past a hard limit, because it already happened. Holds that are never settled expire on their own, and another call's hold never turns an in-limit call into overage.

Predicting the size of a call

Leave out the amount and reserve holds what it predicts the call will use. The engine learns from every settled call, in constant memory, per customer once a customer has enough history and policy-wide before that.

The entitlement says what a call's size depends on and what keeps estimates apart:

chat: {
    estimate_basis: 'input'      // a call's size depends on its input tokens
    estimate_segment: 'model'    // each model learns its own estimate
    limit: { credit: 'ai_token', mode: 'soft', value: 2_000_000, resets: true }
}

The estimate learns a fixed amount per call plus an amount per input token. That matters more than it sounds. Chat answers are about the same length whatever the question. Trained on chat traffic like that, a simple ratio of output to input predicts a 5-token answer to a 20-token question and a 986-token answer to a 4,000-token one. Learning both parts, the engine predicts about 300 for each, which is what actually happens. A summarizer learns the opposite shape: almost nothing fixed, and a share of every input token.

estimate() asks for a prediction without holding anything. It answers at the 90th percentile by default, so most calls come in under it, or at any quantile you ask for.

Predicting the answer before you ask

The same machinery can learn part of a call on its own. An entitlement with no limit is never metered, so it can just learn:

chat_output: { estimate_basis: 'input', estimate_segment: 'model' }

Record each call's output tokens with observe, and ask before the next call:

const answer = await limitr.estimate(userId, 'chat_output', { context: { model, input }, quantile: 0.5 });

That's a typical answer length for this prompt size on this model: useful for a quote, a progress bar, max_tokens, or picking a model.

Feature flags and custom deals

An entitlement with no limit has always been a feature flag: on for every plan that lists it. Real launches need more than that, and so do sales teams.

Rollouts and kill switches

Every entitlement can now have an optional enabled rule:

#[type]
Rollouts: {
    #[static]
    fn beta_or_quarter(customer: Customer, entitlement: str, context?: obj) -> bool {
        (customer.metadata?.beta ?? false) || customer.bucket(entitlement) < 25
    }
}

policy: {
    plans: {
        pro: {
            entitlements: {
                ai_assistant: { enabled: <Rollouts>.beta_or_quarter, track: true }
                new_checkout: {
                    fn enabled(customer: Customer, entitlement: str, context?: obj) -> bool {
                        context?.country == 'US'
                    }
                }
                old_editor: { enabled: false }       // switched off for everyone
                sso: {}
            }
        }
    }
}

• A rule is a function, written inline or shared across many features. It gets the customer, the feature's name, and the call's event data, so it can read a beta list from customer metadata, a country, an app version, or anything else you pass.
• customer.bucket(name) gives each customer a number from 0 to 99 that never changes for that feature. bucket < 25 is the same quarter of customers on every call and every server, and raising it to 50 keeps them and adds more. Each feature splits customers differently, so the same people aren't first in line for everything.
• enabled: false is a kill switch, live the moment the policy updates.
• Rules only narrow. They run after the plan, so a bug in a rollout rule can turn a feature off but can never hand out a paid feature. A rule that fails counts as off, and the error is logged. A call that already happened always settles, even if its feature was switched off mid-call.

features(customer, context) lists everything a customer can use right now, ready to send to your front end.

track: true sends a feature-checked event each time a feature is checked, with whether it was on. Join that with the usage events and you know who saw a feature and what they did next, which is all an A/B test needs.

Giving and taking features

Overrides have always changed how much of a feature a customer gets. Now you can change whether they have it:

await limitr.giveEntitlement('acme', 'sso');                   // SSO on their plan, from the closest plan that has it
await limitr.giveEntitlement('acme', 'api', 'enterprise');     // Enterprise's API limit for this one customer
await limitr.takeEntitlement('acme', 'exports');               // pause one feature
await limitr.restoreEntitlement('acme', 'exports');            // back to what the plan says

• Nothing can be invented. A feature is given from a plan that defines it, so it comes with that plan's limits, and overrides still work on top.
• Gives and takes are stored on the customer, so they last through plan changes. Keeping SSO after a downgrade is one give.
• They can expire, which makes a two-week trial of a higher plan's feature one call.
• An org's gives and takes reach its users, the same way the org's plan does.
• Each change sends its own event, so your records (and Limitr Cloud) see every custom deal.

Small and fast for as long as it runs

Limitr runs inside your process, often for weeks at a time, so this release spent real effort on staying lean:

• No growth per call. Every temporary object a call creates is dropped on every return path. The engine's memory now grows only with customers, their meters, open holds, and estimate segments, and the test suite can check for leaks (stof test --leaks).
• Faster as customers are added. Dropping a call's temporary objects no longer searches the whole document.
• A time limit on policy calls. A custom function stuck in a loop fails after 5 seconds instead of blocking every call after it.
• A sandbox by default. A policy has no network access unless you allow it, and a Cloud policy can only reach Limitr Cloud.
• Errors you can see. When a cost function or a rollout rule fails, the reason reaches your console.

Built on Stof 0.10.6

Limitr policies are Stof documents, and the engine itself is written in Stof. 0.7 moves to Stof 0.10.6, which brings using declarations (temporary objects that clean themselves up), Lib::func library calls, destructuring, explicit null handling, and a long list of performance and correctness fixes. Read more in the Stof 0.10 release notes.

Learn it

• The tutorial has 15 short chapters, each one a small program you can run, from your first policy to AI calls, org budgets, feature flags, and the Stof underneath. Each chapter shows exactly what its program prints.
• Skills for AI coding agents. The npm package ships two skills, limitr (the policy reference, every engine function, and every event) and limitr-ts (the TypeScript SDK), so an agent working in your codebase can write and review policies correctly.

Both are in the Limitr repo, and the docs cover every piece of the engine with programs you can run.

New in 0.7.3

• Override any engine function and keep the original. Extend Limitr with your own type, and a function with an engine function's name runs in its place. It can still call the engine's version with self.allow<Limitr>(...), so you add a rule before or after the engine instead of rewriting it. See Extend Limitr.
• Exchange rates at runtime. setExchangePair adds or replaces a rate from your app, so a scheduled job can keep currencies and model prices current. A rate that can't convert is refused and the last one stays. See Live rates.
• Event handlers can't fail a call. A handler that throws is logged and skipped, and the call returns normally. Before, the error reached the caller after the usage was already recorded.
• A reset schedule means it resets. A limit, cap, or credit pack with reset_sch resets without also needing resets: true.
• Capability results come back as data. runCapability returns an object result as a plain record.
• The Stof skill ships in the npm package, next to the Limitr skills, so coding agents have the language reference too.

New in 0.7.2

• Credit packs follow plan changes. A plan's included credit pack moves with the customer when they change plans. By default an upgrade tops the balance up to the new plan's amount, and a downgrade keeps the larger balance, so a plan change never takes away credits already given. A pack can choose replace (start over at the new amount) or keep (leave the balance as it is) instead, and purchased packs always stay with the customer. See Change plans.
• Clear errors for IDs with a period. Customer IDs can't contain a period, and one used to fail silently. Now it's refused with an error in your logs. To key customers by email, replace the periods with a placeholder of your own, or keep the email in the customer's metadata.
• Fixes. Credit packs on a calendar schedule no longer refill on their first read, results returned to Claude from capabilities are always text, and ensureCustomer returns false when it couldn't create the customer.

Upgrading from 0.6

Most apps upgrade by updating the package. Check these if they apply to you:

• The policy lives at root.policy, and <Limitr>.api is gone. If you call engine functions directly, call them on the policy ('policy.allow' from the SDK). Limitr.new and Limitr.cloud handle this for you.
• Stof 0.10 is strict about null. Arithmetic with a missing value is now an error. If you wrote your own functions in a policy, give optional fields a default with ??.
• The compiled engine export is limitrEngine (it was limitrApi), if you imported it directly.
• Policy network access is off by default for Limitr.new. Allow the hosts you need with limitr.doc.allowHttp([...]).
• Event data passed as an empty string now gives a cost function no context, instead of an empty object.

The full list is in the changelog.

What's next

Limitr Cloud runs the same engine with versioned policies, shared customer state, invoicing, and margin analytics. Cloud moves to 0.7 next, with a refreshed app built around what this release makes possible: costs and margins per call, predictions, rollouts, and custom deals you can see and change in one place. Until then, nothing changes for Cloud customers. We'll write about that release when it ships.

If you're building usage-based pricing for an AI product and want to talk it through, book time with the founders.