Feed & Timeline Generation

The Problem

Social platforms, news apps, and marketplace dashboards need to generate personalized feeds showing relevant content from followed users, topics, or sources. The core challenge: when a user opens their feed, you must quickly assemble a sorted, deduplicated, ranked list of items from potentially thousands of sources. At scale, this involves billions of content items and trillions of follow relationships. The feed must load in under 200ms, support infinite scroll pagination, and feel fresh — showing new content within seconds of creation.

Architectural Approaches

Architecture

Loading diagram...

Key Components

Data Model

Loading diagram...

Fan-Out Worker Design

The fan-out worker is the workhorse of the push model. When a user with 100K followers posts, the worker must write to 100K inboxes quickly.

Architecture:

1. Content creation event arrives in a queue (Kafka/SQS)
2. Worker fetches the author’s follower list (paginated, 1000 at a time)
3. For each batch of followers, pipeline ZADD commands to Redis: ZADD user:{follower_id}:inbox {timestamp} {content_id}
4. Trim each inbox to the max size (e.g., keep latest 1000 items): ZREMRANGEBYRANK user:{follower_id}:inbox 0 -1001

Performance: Redis ZADD is O(log N). With pipelining, a single worker can write ~50K inbox entries per second. For a 100K follower fan-out, this takes ~2 seconds with a single worker, or ~200ms with 10 parallel workers.

Backpressure: During viral moments (celebrity posts), the fan-out queue can grow rapidly. Use priority queues — fan out to active users first (users who opened the app in the last 24 hours), then backfill inactive users in lower priority.

Failure handling: If a fan-out partially fails, some followers will miss the content. Use idempotent writes (ZADD is naturally idempotent) and retry failed batches. A periodic reconciliation job can catch any drift.

Feed Ranking Algorithms

Chronological feeds are simple but don’t maximize engagement. Most platforms use ranked feeds.

Ranking signals:

• Recency: Newer content scores higher. Exponential decay function: score = base_score * e^(-λ * age_hours)
• Engagement: Content with more likes/comments/shares is boosted. Normalize by impression count to avoid bias toward older content.
• Affinity: How closely connected is the viewer to the author? Signals: message frequency, profile visits, reaction history. “Interest graph” vs “social graph.”
• Content type: Users may prefer photos over text posts. Learn per-user preferences from interaction history.
• Diversity: Penalize consecutive items from the same author or topic. Inject content from less-seen sources to avoid filter bubbles.

Simple scoring formula:

score = w1 * recency_score + w2 * engagement_score + w3 * affinity_score + w4 * content_type_score - w5 * repetition_penalty

ML-based ranking: At scale, train a model (gradient boosted trees, neural networks) to predict engagement probability. Features include all the above signals plus user-level features (activity level, account age, device type). Instagram and TikTok use deep learning models that consider hundreds of signals.

Cursor-Based Pagination

Feeds use cursor-based pagination instead of offset-based for consistency during infinite scroll.

Why not offset?: With offset pagination (LIMIT 20 OFFSET 40), new content inserted at the top shifts all items down, causing duplicates or missed items as the user scrolls.

Cursor approach: Each page response includes a cursor (the score/timestamp of the last item). The next request uses this cursor: “give me 20 items with score less than 1707100000.” New items at the top don’t affect the cursor position.

Implementation with Redis sorted sets:

• First page: ZREVRANGEBYSCORE user:inbox +inf -inf LIMIT 0 20
• Next page: ZREVRANGEBYSCORE user:inbox ({last_score} -inf LIMIT 0 20

Opaque cursors: Encode the cursor as a base64 string so clients can’t manipulate it. Include the score, last item ID (for tie-breaking), and an HMAC signature to prevent tampering.

Gap detection: If there’s a large time gap between feed sessions, show a “catch-up” summary instead of scrolling through hundreds of items. “While you were away: 42 new posts from 15 people.”

Real-World Examples

Architectural Tradeoffs

# Learn the System Design of: Feed & Timeline Generation

You are an expert software architect and teacher. I want to understand feed & timeline generation as a practical system design capability. (Fan-out-on-write vs fan-out-on-read strategies for personalized content feeds).

Use the following architectural guide as a learning reference. Avoid one-size-fits-all recommendations. Keep assumptions, costs, and tradeoffs explicit.

---

import { Mermaid, ComponentList, ComponentCard, ApproachList, Approach, Example, Tradeoff } from '@/components/mdx';

## The Problem

Social platforms, news apps, and marketplace dashboards need to generate personalized feeds showing relevant content from followed users, topics, or sources. The core challenge: when a user opens their feed, you must quickly assemble a sorted, deduplicated, ranked list of items from potentially thousands of sources. At scale, this involves billions of content items and trillions of follow relationships. The feed must load in under 200ms, support infinite scroll pagination, and feel fresh — showing new content within seconds of creation.

## Architectural Approaches

<ApproachList>
<Approach
  name="Fan-Out on Write (Push Model)"
  pros={[
    'Ultra-fast feed reads — just fetch from a pre-computed list',
    'Consistent read latency regardless of how many people you follow',
    'Simple pagination — cursor through the sorted set',
    'Feed is always ready when the user opens the app',
  ]}
  cons={[
    'Write amplification — a celebrity with 10M followers requires 10M inbox writes',
    'High storage cost — content duplicated across millions of inboxes',
    'Delete/edit propagation is expensive (must update all inboxes)',
    'Wasted work for inactive users who may never read their feed',
  ]}
>
When a user creates content, immediately "push" it to the feed inbox of every follower. Each user has a pre-computed feed stored in a sorted set (Redis) or timeline table. Reading the feed is a simple ordered r...

[Full prompt copied to clipboard]