Tests
The Tests tab on the Policy page holds tests for your policy: small checks that say what it should do ("Free stops at 3 chats", "Pro gets 2M tokens"), so a change that breaks one shows before your customers see it. Tests are on every plan, Free included, so you can try a policy before your app is connected.
Write tests
A test is a Stof function marked #[test]. It runs against your policy, so self is the
policy and the engine's functions are there: make a customer with create_customer, then call allow, check,
limit, remaining, and the rest of the engine functions, and check the answers
with assert, assert_eq, or assert_neq.
#[test]
/// Free gets 3 chats, then it's refused.
fn free_stops_at_three() {
self.create_customer('t_free', 'free');
for (let i = 0; i < 3; i += 1) assert(self.allow('t_free', 'chat', 1));
assert(!self.allow('t_free', 'chat', 1));
pln('remaining: ', self.remaining('t_free', 'chat'));
}
#[test]
#[errors]
/// Passes only when it throws: here, the assert on a fourth chat.
fn free_refuses_a_fourth() {
self.create_customer('t_free4', 'free');
for (let i = 0; i < 3; i += 1) self.allow('t_free4', 'chat', 1);
assert(self.allow('t_free4', 'chat', 1));
}
Pro: {
#[test]
fn has_100_chats() {
super.create_customer('t_pro', 'pro'); // in a group, the policy is `super`
assert_eq(super.limit('t_pro', 'chat'), 100);
}
}
- Use a new customer ID in each test. Tests share one engine for the run, in no set order.
- One behavior per test, named so a failure reads as a sentence. A
///comment says why it matters. #[errors]marks a test that passes only when it throws.- Group tests in an object (
Pro: { ... }). Inside a group, the policy issuper. plnprints into the test's output, which helps when one fails.- The engine doesn't throw on a name it doesn't know:
allowon an entitlement the plan doesn't have isfalse, andlimitfor an unknown customer isnull. Assert on those values.
With no tests yet, Start from this policy writes a first few from your plans' limits, and Ask the Analyst has Limitr Analyst write a suite for what you care about. Edits save to the draft (Ctrl S, Cmd S on a Mac), like the Code tab. A suite that doesn't parse isn't saved, and the line with the problem is marked.
Run them
Run runs the saved tests against the draft, and each result shows as it finishes. With unsaved edits, the button is Run edits: it runs the text in the editor without saving it. Click a test to see why it failed and what it printed. Stof output shows the whole run as Stof writes it:
running 4 tests
test free_stops_at_three ... ok
test free_refuses_a_fourth ... ok
test Pro.has_100_chats ... ok
test Pro.has_200_chats ... FAILED
output and failures:
---- free_stops_at_three ----
remaining: 0
---- Pro.has_200_chats ----
assertion failed: '100' does not equal '200'
at root.policy.Pro.has_200_chats (30:9)
test result: FAILED. 3 passed; 1 failed; finished in 0.41s
When a test fails, either the policy or the test is wrong: fix whichever doesn't say what you mean.
Tests run in Limitr Cloud, where the Analyst's simulations run: a fresh engine for each run, with no network and no access to your customers or data. Each call has a few seconds and a run has 30. Anyone in the workspace can run them.
Versions and publishing
- Saved with the draft. Tests are part of the draft, so teammates and the Analyst see the same suite. They're not sent to your engines.
- Changes and the publish dialog show what changed in the tests under the policy's changes.
- Publishing runs them. The publish dialog runs the tests first. If any fail, it names them, and you can still publish: the version keeps the result. History shows how each version's tests did when it was published.
- Require passing. The switch under the results makes publishing wait until every test passes, in test and live. It needs the Publish live policies permission (see Team).
- Versioned with the policy. Restoring a version brings its tests, and Copy to live (or Copy to test) brings the tests with the policy. Each mode has its own suite.
A policy and its tests can be up to 400 KB together, and a run up to 200 tests.
With Limitr Analyst
Ask the Analyst to write tests ("write tests for the Pro plan's limits"), run them, or explain a failure. It saves tests to the draft like a policy change, shows what changed in the conversation, and shows each run's output. After it changes the policy, it runs the tests and says what passed.