System Design: News Feed — How Twitter and Instagram Scale to Billions

Published data: 18 April 2026

Twitter serves 500M+ users
a personalised feed in
under 500 ms. The
engineering behind it
took a decade to perfect.

Scroll your Twitter timeline. Tap Instagram Stories. The feed appears in under a second — effortlessly. What’s invisible is one of the most difficult distributed-systems challenges in production engineering: delivering the right posts, from the right people, to 500 million users, faster than they can blink.

This post walks the news feed problem from naïve SQL all the way to the hybrid architecture that Twitter, Instagram, and Meta actually run. Each level exposes a real failure mode from the previous one — the same progression you need to demonstrate in a system-design interview.

1. The Core Problem

When Alice loads her feed she wants to see posts from everyone she follows, sorted newest-first, paginated. That requirement looks almost trivial — it isn’t.

Here is what “trivial” actually means at scale:

• Alice follows 200 people on average
• Those 200 people post at rates from once a week to once a minute
• Her feed must load in under 500 ms — or she closes the app
• Meanwhile, 500 million other Alices are doing the exact same thing simultaneously
• Beyoncé just posted, which means 200 million feeds need updating right now

The fundamental tension is between read latency (how fast can Alice load her feed?) and write amplification (how much work does one post create across the system?). Every architectural decision you make is a negotiation between these two forces.

2. Requirements

Before designing anything, nail down the constraints. Interviewers reward explicit requirement scoping.

Functional requirements:

• Users follow / unfollow other users
• Users see a feed of posts from followed accounts, sorted reverse-chronologically
• Feed supports infinite scroll (cursor-based pagination)
• Posts include: text (up to 280 chars), optional media reference, timestamp, author

Non-functional requirements:

• Feed load time: < 500 ms at p99
• Scale: 500 M daily active users
• Post ingestion: 1 B posts / day ≈ 11,500 posts / sec
• Read / write ratio: ~100:1 (heavily read-dominant)
• Average follows per user: 200
• 95th-percentile celebrity follower count: 50 M+
• Availability: 99.99 % (under 1 hour downtime per year)
• Eventual consistency acceptable — a few seconds of feed staleness is fine

Back-of-envelope to keep in mind:

3. Level 1 — Naive Pull (Fan-out on Read)

The first instinct is to compute the feed dynamically at read time. When Alice requests her feed:

1. Fetch Alice’s list of 200 followed user-IDs
2. Query the posts table for those IDs in the last 24 hours
3. Sort by timestamp, paginate to 20, return

-- Compute the feed on every single read request
SELECT
    p.id,
    p.author_id,
    p.content,
    p.created_at,
    p.media_url
FROM  posts p
WHERE p.author_id IN (
    SELECT followee_id
    FROM  follows
    WHERE follower_id = :alice_user_id
)
AND   p.created_at > NOW() - INTERVAL '24 hours'
ORDER BY p.created_at DESC
LIMIT 20;

Why this works (for a while): Simple, consistent, no pre-computation. Every read reflects the absolute latest data instantly.

Why this breaks at scale: The IN clause with 200 IDs can still be fast with indexes at low traffic. But the real problem is thundering herd: 115 000 feed reads per second all hitting the posts table with complex range scans. Even with read replicas, each query is expensive. When a celebrity posts to 50 M followers and everyone refreshes simultaneously, your database falls over.

Measured latency: 50–200 ms on a warm, lightly loaded database. 2–10 seconds under real production load. Unacceptable at scale.

4. Level 2 — Fan-out on Write (Push Model)

The insight: pre-compute the feed at write time. When a user posts, immediately push that post into the feed cache of every follower. Reads become a trivial cache lookup.

Write path: User Bob creates a post → post saved to DB → fan-out worker fetches Bob’s follower list → writes the post ID into a Redis sorted set for each follower.

Read path: Alice requests feed → read feed:{alice_id} from Redis → done.

# Write path: fan-out to each follower's sorted set
def fanout_post(post_id, author_id, timestamp):
    followers = get_followers(author_id)         # fetch follower IDs
    score     = timestamp                          # Unix ms as sort key

    for follower_id in followers:
        key = "feed:" + str(follower_id)
        redis.zadd(key, {post_id: score})        # O(log N) per write
        redis.zremrangebyrank(key, 0, -1001)    # keep latest 1000 only

# Read path: single Redis call, sub-millisecond
def get_feed(user_id, cursor=None, limit=20):
    key      = "feed:" + str(user_id)
    post_ids = redis.zrevrangebyscore(
                   key, cursor or "+inf", "-inf",
                   start=0, num=limit)
    return batch_fetch_posts(post_ids)           # mget from post cache

Why reads are blazing fast: Sub-millisecond sorted set range queries. No SQL joins, no table scans. A feed page loads in 1–5 ms including network.

The celebrity problem: Bob has 200 followers → 200 Redis writes. Fine. Beyoncé has 200 million followers. One tweet = 200 million Redis sorted-set writes. At 100 000 writes / second that takes 33 minutes. Early followers see the post; late followers wait half an hour. Completely unacceptable.

The celebrity problem is why the hybrid approach is necessary. The math simply does not work: 200 M writes × ~1 μs each = 200 s of pure single-threaded Redis time, ignoring network and coordination overhead.

This is the write amplification problem, and solving it is the central engineering challenge of every major news feed system.

5. Level 3 — Fan-out Visualizer

Watch write amplification in action. Click each user to simulate a post and observe how differently the system behaves based on follower count.

6. Level 4 — The Hybrid Approach

The solution Twitter, Instagram, and Facebook converged on is a hybrid routing model: fan-out on write for regular users, fan-out on read for celebrities.

The routing rule:

# Threshold separating regular users from celebrities
CELEBRITY_THRESHOLD = 1_000_000

def handle_new_post(post, author):
    if author.follower_count < CELEBRITY_THRESHOLD:
        # Regular user: fan-out on write → push to Redis
        fanout_to_followers(post, author.follower_ids)
    else:
        # Celebrity: store in celebrity cache; merge at read time
        celebrity_cache.add_post(post)

def get_feed(user_id):
    # Step 1: Pre-computed feed from Redis (regular follows)
    regular_posts = redis.zrevrange("feed:" + user_id, 0, 49)

    # Step 2: Live pull for celebrity follows
    celeb_ids   = get_celebrity_followees(user_id)
    celeb_posts = celebrity_cache.get_recent(celeb_ids, limit=20)

    # Step 3: Merge, sort by score/time, paginate
    return merge_sorted(regular_posts, celeb_posts)[:20]

Trade-off matrix:

The celebrity threshold (1 M) is configurable. Some systems use 100 K; Instagram reportedly adjusts it dynamically based on current system load. Users near the threshold may flip between paths as follower counts change — the system handles this gracefully because it queries both paths on every read regardless, deduplicating by post ID.

7. Level 5 — Feed Service Architecture

The complete system consists of cooperating services with clearly separated concerns. Click any node to learn its role, or highlight a data flow path using the legend.

8. Level 6 — Feed Ranking

Pure reverse-chronological order was Instagram’s original approach. After switching to a ranked feed, engagement increased by roughly 40 %. The tradeoff: ranked feeds are stickier but reduce serendipity and can create filter bubbles.

Why ranking beats chronological:

• A celebrity’s low-quality retweet shouldn’t displace a close friend’s life milestone
• Viral content from yesterday is often more interesting than a routine post from 30 seconds ago
• Different users have different engagement patterns — personalisation increases time-in-app

Scoring model:

import math, time

def score_post(post, user):
    # Recency: exponential decay, half-life = 6 hours
    age_hrs  = (time.time() - post.created_at) / 3600
    recency  = math.exp(-0.693 * age_hrs / 6.0)   # [0.0, 1.0]

    # Engagement: log-scaled, comments/shares weighted higher
    raw_eng  = post.likes + post.comments * 3 + post.shares * 5
    engage   = math.log1p(raw_eng) / 20.0              # normalised [0.0, ~1.0]

    # Relationship strength: how often user has interacted with author
    rel      = user.interaction_score(post.author_id)   # [0.0, 1.0]

    # Content-type affinity: user preference for photo / video / text
    affinity = user.content_pref(post.media_type)       # [0.0, 1.0]

    # Weighted combination (weights learned by gradient boosting)
    score = (0.35 * recency
           + 0.25 * engage
           + 0.25 * rel
           + 0.15 * affinity)
    return score

In production, the scoring function is a trained ML model — typically a gradient-boosted tree or a two-tower neural network. The features above are inputs; the model weights are learned from implicit feedback signals: did the user like it, comment, share, or scroll past in under 0.3 seconds (a strong negative signal)?

Where ranking runs: Ranking scores are computed in the Feed Read Service after candidate retrieval, before the final sort-and-paginate step. Ranking the entire Redis sorted set would be too slow; in practice only the top ~200 candidates from the cache are scored, then the top 20 are returned.

9. Level 7 — Pagination

You cannot paginate a live feed with page numbers. Here is why.

Alice opens her feed and sees posts 1–20. She scrolls. While she reads, 5 new posts arrive at the top. She requests “page 2.” With offset-based pagination:

-- Page 2 with offset pagination (WRONG for live feeds)
SELECT * FROM feed_posts
WHERE  user_id = :uid
ORDER BY score DESC
LIMIT 20 OFFSET 20;

-- The 5 new posts at the top shifted everything down by 5.
-- Alice sees posts 16–20 again (duplicates).
-- Posts 21–25 are silently skipped (missing content).
-- Page 2 of a live feed is a DIFFERENT page 2 every time.

Cursor-based pagination solves this. The cursor encodes the position of the last seen item — new posts at the top do not affect it.

def get_feed_page(user_id, cursor=None, limit=20):
    if cursor is None:
        # First page: get most recent posts
        posts = redis.zrevrangebyscore(
            "feed:" + user_id, "+inf", "-inf",
            start=0, num=limit
        )
    else:
        # Subsequent page: posts with score STRICTLY LESS THAN cursor
        # "(" prefix in Redis means exclusive bound
        posts = redis.zrevrangebyscore(
            "feed:" + user_id,
            "(" + str(cursor),   # exclusive lower-than-cursor
            "-inf",
            start=0, num=limit
        )

    next_cursor = posts[-1].score if posts else None
    return {"posts": posts, "next_cursor": next_cursor}

Cursor-based pagination isn’t just better for feeds — it’s essential. Page 2 of a real-time feed is a different page 2 every time you load it. Offset-based pagination is a correctness bug, not just a performance issue.

The cursor is opaque to the client — it sends back whatever the server gave it. Internally it can be a Unix millisecond timestamp, a post ID, or a composite (timestamp + post ID to handle ties). For ranked feeds, the cursor must encode the ranking score, not just timestamp, to prevent items re-appearing or disappearing as scores change during a scroll session.

10. The Feed Simulator

Follow users, generate posts, and toggle between push and pull models to see how the system behaves differently. Notice that with the push model, only followed-user posts pre-populate your feed — unfollowing removes them on the next refresh (not immediately, a known trade-off).

11. Storage Estimation

No system design answer is complete without storage math.

Post row schema:

CREATE TABLE posts (
    post_id        BIGINT       NOT NULL,  -- 8 B  (Snowflake ID encodes time + node)
    author_id      BIGINT       NOT NULL,  -- 8 B
    content        VARCHAR(280) NOT NULL,  -- avg ~140 B UTF-8
    media_url      VARCHAR(256),            -- 256 B nullable
    created_at     BIGINT       NOT NULL,  -- 8 B  Unix ms
    likes_count    INT          DEFAULT 0, -- 4 B
    comments_count INT          DEFAULT 0, -- 4 B
    metadata       JSONB,                   -- ~200 B avg (hashtags, mentions, geo)
    PRIMARY KEY (author_id, post_id)        -- Cassandra: partition by author
);
-- Estimated row size: ~850 B raw ≈ 1 KB with Cassandra overhead + secondary indexes

Volume calculations:

Ingestion bandwidth:

• Posts DB write: 11 500 / sec × 1 KB = 11.5 MB / sec — comfortable for a 20-node Cassandra cluster
• Fan-out writes: 2.3 M Redis ops / sec — requires a Redis cluster of ~15–20 shards
• Media ingest: 100 TB / day = ~1.2 GB / sec — handled by S3 multipart upload with transfer acceleration

Feed cache sizing: 4 TB is the theoretical max. In practice, LRU eviction is applied to users inactive for more than 30 days. Instagram reported ~60 % of users are DAU, so the working set fits in roughly 2.4 TB of Redis RAM — about 20 nodes at 128 GB each, leaving headroom for replication.

12. Key Takeaways

Walking all seven levels, the core principles are clear.

Read / write tension is the whole game. Every news-feed decision is a negotiation: faster reads require more write work. The hybrid model is the industry consensus because it optimises for the common case (regular users, cheap fan-out) while isolating the edge case (celebrities, skip fan-out).

Architecture evolves with load. The naïve SQL approach works at 10 K users. Fan-out on write works at 1 M users. The hybrid is necessary at 100 M+. There is no universally correct architecture — only architectures appropriate for a given load.

Cursor pagination is non-negotiable. On any feed with live updates, offset-based pagination produces incorrect results. This is a correctness requirement, not a performance optimisation.

Decouple writes from side-effects. The Kafka layer between Post Service and Fan-out Worker makes the system resilient to worker outages without affecting write availability. Post creation returns in < 5 ms; fan-out completes asynchronously.

Store IDs in caches, not documents. The Redis feed cache holds post IDs, not full post content. This means posts can be edited or deleted without invalidating feed caches — the downstream batch fetch always retrieves the current version. Cache invalidation becomes trivial.

Twitter’s original architecture used fan-out on write exclusively — until Justin Bieber joined. His first tweet triggered fan-out to millions of followers simultaneously, saturating all fan-out workers. The “Bieber problem” forced Twitter to build the hybrid architecture now used industry-wide.

Interview checklist:

1. Scope requirements: DAU, posts/sec, read/write ratio, follower distribution
2. Naive pull → identify thundering-herd DB bottleneck
3. Fan-out on write → identify celebrity write-amplification problem
4. Hybrid routing → explain the threshold and merge strategy
5. Draw full architecture: Post Service → Kafka → Fan-out Worker → Redis / Celebrity Cache → Feed Read Service
6. Cursor-based pagination — explain why OFFSET is wrong
7. Ranking as an enhancement — recency decay + engagement + relationship strength
8. Back-of-envelope: storage, bandwidth, Redis cluster sizing

Demonstrate that progression coherently and you have answered one of the most common distributed-systems questions at senior-level engineering interviews.

Next in the series: #6 — Design a Distributed Cache — from a single Redis node to a globally consistent multi-region caching layer.