Skip to main content

Feature flags, rollouts, and experiments

Feature flags live in the policy, next to the plans, so every server, your front end, and Limitr Cloud agree on who has what. Your code stays the same check and allow. This walkthrough rolls out a new editor to 10% of customers, widens it to 50%, and measures it.

import { Limitr } from '@formata/limitr';

const policy = (percent: number) => `
#[type]
Rollouts: {
    #[static]
    /// Staff and beta customers first, then a share of everyone else.
    fn new_editor(customer: Customer, entitlement: str, context?: obj) -> bool {
        (customer.metadata?.staff ?? false) || (customer.metadata?.beta ?? false) ||
            customer.bucket(entitlement) < ${percent}
    }
}

policy: {
    plans: {
        pro: {
            default: true
            entitlements: {
                new_editor: { enabled: <Rollouts>.new_editor, track: true }
                // Switched off for everyone
                legacy_export: { enabled: false }
                desktop_sync: {
                    // Only for app versions that support it, from the call's event data.
                    fn enabled(customer: Customer, entitlement: str, context?: obj) -> bool {
                        (context?.app_version ?? 0) >= 3
                    }
                }
            }
        }
    }
}`;

// Start the rollout at 10%.
let limitr = await Limitr.new(policy(10));
const customers = Array.from({ length: 200 }, (_, i) => `user_${i}`);
for (const id of customers) await limitr.ensureCustomer(id);
await limitr.ensureCustomer('grace', 'pro', 'user', 'Grace', null, null, { staff: true });

// The customers who have the new editor right now.
async function withNewEditor() {
    const ids = [];
    for (const id of customers) {
        if (await limitr.check(id, 'new_editor')) ids.push(id);
    }
    return ids;
}
const first = await withNewEditor();
const grace = await limitr.check('grace', 'new_editor');
console.log(`10%: ${first.length} of 200, and grace: ${grace}`);

// Widen it to 50%. Everyone who had it keeps it.
const saved = await limitr.customers();
limitr = await Limitr.new(policy(50));
await limitr.loadCustomers(saved!);
const second = await withNewEditor();
const keptFirst = first.every((id) => second.includes(id));
console.log(`50%: ${second.length} of 200, kept all of the first group: ${keptFirst}`);

// What the front end shows, per request.
console.log('grace, app v3:', await limitr.features('grace', { app_version: 3 }));
console.log('grace, app v2:', await limitr.features('grace', { app_version: 2 }));

// Measuring: tracked features send feature-checked with whether they were on.
const seen = { on: 0, off: 0 };
limitr.addHandler('experiment', (name, value) => {
    if (name !== 'feature-checked') return;
    const { enabled } = JSON.parse(value as string);
    if (enabled) seen.on++;
    else seen.off++;
});
for (const id of customers) await limitr.check(id, 'new_editor');
console.log('checks:', seen);

It prints:

10%: 18 of 200, and grace: true
50%: 112 of 200, kept all of the first group: true
grace, app v3: [ 'new_editor', 'desktop_sync' ]
grace, app v2: [ 'new_editor' ]
checks: { on: 112, off: 88 }

The pieces​

  • A shared rule in a #[type]: Rollouts.new_editor turns the feature on for staff, then beta customers, then customer.bucket(entitlement) < percent. One rule can serve many features, because it's told which one it's deciding.
  • customer.bucket(name) is a stable number from 0 to 99 per customer and feature, so 10% is always the same customers, on every call and every server. The split is a hash, so 10% of 200 is about 20, not exactly 20.
  • Widening keeps everyone who already had it: raising 10 to 50 only adds customers.
  • Staff and beta lists are customer metadata ({ staff: true }), read by the rule.
  • A kill switch is enabled: false: off for everyone the moment the policy changes.
  • Per-request rules read the call's event data: desktop_sync is on for app version 3 and up.
  • features(customer, context) lists what a customer can use right now, ready for the front end.
  • track: true sends feature-checked each time the feature is checked, with whether it was on.

Rolling out in production​

  1. Ship the code behind check(userId, 'new_editor'), with the rule at 0% (or enabled: false).
  2. Turn it on for staff, then a beta list, through metadata.
  3. Raise the percent in the policy: 10, 25, 50, 100. Each step keeps everyone from the last.
  4. Something wrong? Set enabled: false. It's off on the next call.
  5. Done? Remove the rule. The plan decides again.

With Limitr Cloud, each change is a policy edit that reaches every running engine in seconds, with no deploy. Self-hosted, it's a change to your policy file.

Measuring​

feature-checked carries the customer, whether the feature was on, and the call's event data. Forward it to your analytics and join it with the meter events that follow: who saw the feature, and what they did next. An A/B test is a rollout at 50 with track: true.

Track only what you're measuring: an event per check adds up.

Rules only narrow​

A rule runs after the plan, gives, and takes, so it can turn a feature off but can't give one a customer's plan doesn't have. A bug in a rollout rule can't give away a paid feature. A rule that throws counts as off, and the error is logged. To give a feature outside a plan, give it.

See Feature rules for every detail, including computing a customer's bucket on your own servers.

Last updated on