Feature Flags & Progressive Rollouts

The Problem

Shipping code and releasing features should be independent activities. Feature flags decouple deployment from release — code ships to production behind a flag and is activated for specific users, percentages, or segments without redeploying. This enables progressive rollouts (1% → 10% → 50% → 100%), instant kill switches when things go wrong, A/B experimentation, and per-customer feature gating for different plan tiers. The challenge is building a system that evaluates flags with sub-millisecond latency, scales to millions of evaluations per second, and maintains consistency so users don’t see features flicker on and off.

Architectural Approaches

Architecture

Loading diagram...

Key Components

Data Model

Loading diagram...

Consistent Percentage Bucketing

When rolling out to 10% of users, you need the same 10% to see the feature consistently. Random selection would cause the feature to flicker on/off across requests.

Algorithm:

1. Concatenate the flag key and user ID: "new-checkout" + "user_123"
2. Hash with MurmurHash3 to get a 32-bit integer
3. Normalize to 0-100: hash % 100
4. If the result < rollout_percentage, enable the flag

Properties:

• Deterministic: Same user + flag always produces the same bucket
• Uniform distribution: MurmurHash3 provides excellent distribution
• Independent per flag: A user in the 10% for flag A is not necessarily in the 10% for flag B (different hash inputs)
• Monotonic rollout: Users in the 10% cohort remain in the 25% cohort when you increase the percentage

To ensure monotonic rollout, sort users by their hash value and always expand the inclusion from the same end. This prevents users from losing access when the percentage increases.

Targeting Rules Engine

Flags support complex targeting beyond simple percentages:

Rule types:

• User list: Enable for specific user IDs (internal testing, beta users)
• Segment match: Enable for users matching attributes (plan=enterprise, country=US)
• Percentage rollout: Enable for N% of users (consistent bucketing)
• Schedule: Enable/disable at specific times (launch events)
• Dependency: Enable only if another flag is also enabled

Evaluation order: Rules are evaluated in priority order. First matching rule wins. If no rule matches, the default variation is returned.

Context attributes: The calling code provides a user context object with attributes like userId, email, plan, country, deviceType. The evaluation engine matches these against rule conditions using operators: equals, contains, startsWith, regex, semver comparison, and set membership.

Performance: With cached rules, evaluation of even complex targeting rules takes <0.1ms. The bottleneck is context enrichment (looking up user attributes), which should be cached or pre-computed.

Kill Switch and Incident Response

Feature flags are the fastest way to mitigate incidents caused by new code.

Kill switch pattern: Every significant feature ships behind a flag. If the feature causes errors, latency spikes, or customer complaints, an on-call engineer can disable the flag instantly through the admin dashboard — no deployment needed.

Best practices:

• One-click disable: The dashboard should have a prominent “kill” button per flag with no confirmation dialogs
• Propagation speed: Flag changes should reach all instances within 5-10 seconds (SSE streaming is ideal)
• Safe defaults: When the flag service is unreachable, default to the “off” variation for new features (fail closed)
• Automatic rollback: Monitor error rates per flag variation. If the “on” variation’s error rate exceeds a threshold, automatically disable the flag and alert the team
• Flag hygiene: Remove flags after full rollout. Stale flags accumulate tech debt. Set expiration dates and alert when flags are past their expected lifetime

Blast radius control: Combine flags with progressive rollout. Start at 1%, monitor metrics for 30 minutes, expand to 10%, monitor, expand to 50%, then 100%. Each step is a checkpoint where you can pull back.

Real-World Examples

Architectural Tradeoffs

# Learn the System Design of: Feature Flags & Progressive Rollouts

You are an expert software architect and teacher. I want to understand feature flags & progressive rollouts as a practical system design capability. (Gradual feature deployment with audience targeting, kill switches, and experimentation).

Use the following architectural guide as a learning reference. Avoid one-size-fits-all recommendations. Keep assumptions, costs, and tradeoffs explicit.

---

import { Mermaid, ComponentList, ComponentCard, ApproachList, Approach, Example, Tradeoff } from '@/components/mdx';

## The Problem

Shipping code and releasing features should be independent activities. Feature flags decouple deployment from release — code ships to production behind a flag and is activated for specific users, percentages, or segments without redeploying. This enables progressive rollouts (1% → 10% → 50% → 100%), instant kill switches when things go wrong, A/B experimentation, and per-customer feature gating for different plan tiers. The challenge is building a system that evaluates flags with sub-millisecond latency, scales to millions of evaluations per second, and maintains consistency so users don't see features flicker on and off.

## Architectural Approaches

<ApproachList>
<Approach
  name="Client-Side SDK with Cached Rules"
  pros={[
    'Sub-millisecond evaluation — no network round-trip',
    'Works even if the flag service is down (cached rules)',
    'Scales infinitely — evaluation is local to each instance',
    'No added latency to request processing',
  ]}
  cons={[
    'Stale data window between poll intervals',
    'SDK must be maintained for each language/runtime',
    'Large rule sets increase memory usage per instance',
    'Streaming updates add connection management complexity',
  ]}
>
The application embeds an SDK that fetches flag configurations from the flag service on startup and caches them locally. Flag evaluation happens entirely in-process with no network call ...

[Full prompt copied to clipboard]