System Design: A Mental Model for Trade-off Thinking

System design is not about memorizing technologies. It’s about recognizing that every decision buys you something and costs you something else. The skill is knowing what you’re buying, what you’re paying, and whether that’s the right price for this problem.

1. The Core Idea

“Use Redis” is not an answer. “Why Redis, and what does it cost me?” is the actual question.

Example — Cache:

✅ Lower latency, lower DB load
❌ Consistency gets harder, cache invalidation problems, extra infrastructure to operate

Example — Replication:

✅ Higher availability, higher durability
❌ Consistency complexity, storage cost, replication lag

Example — Queue:

✅ Decoupling, throughput, resilience
❌ Eventual consistency, operational complexity, harder debugging

This pattern — gain X, pay Y — repeats at every layer of a system. Learning to spot it automatically is the actual skill.

2. The Mental Model (Order of Thinking)

When you see a system design question, walk through this in your head, top to bottom:

flowchart TD
    Req["REQUIREMENTS"] --> Traffic["How much traffic?"]

    Traffic --> RW["Read / Write"]
    Traffic --> DS["Data size"]

    RW --> TP["Throughput"]
    DS --> ST["Storage"]

    TP --> Latency["LATENCY"]
    ST --> Latency

    Latency --> Arch["ARCHITECTURE"]

    Arch --> Compute["Compute"]
    Arch --> Storage["Storage"]
    Arch --> Messaging["Messaging"]

    Compute --> Servers["Servers"]
    Storage --> DB["DB"]
    Messaging --> Queue["Queue"]

    Servers --> Scale["SCALE / PARTITION"]
    DB --> Scale
    Queue --> Scale

    Scale --> Repl["REPLICATION"]
    Repl --> Fail["FAILURE MODES"]

    Fail --> Avail["Availability"]
    Fail --> Dur["Durability"]
    Fail --> Rel["Reliability"]

    Avail --> Obs["OBSERVABILITY"]
    Dur --> Obs
    Rel --> Obs

    Obs --> Trade["TRADE-OFFS"]

Rendering diagram...

Algorithms are not separate from this diagram — they live inside it, specifically at:

Load balancing · Caching · Partitioning · Sharding · Scheduling · Rate limiting · ID generation · Indexing · Searching · Deduplication · Consensus · Replication

3. Core Concepts Cheat Sheet

Concept	The Problem It Solves	Typical Solutions
Scalability	How does the system grow as load increases?	Horizontal scaling, load balancing, sharding, partitioning, caching, stateless services
Throughput	How do we do more work per second?	Parallelism, batching, async processing, queues, partitioning, caching
High Throughput	How do we process massive volumes of work?	Kafka/streaming, partitioning, batching, consumer groups, async I/O, distributed workers
Latency	How do we speed up a single request?	Caching, CDN, indexing, DB optimization, connection pooling, data locality, async processing
Performance	How do we speed up the system overall?	Profiling, caching, indexing, query optimization, concurrency, compression
Durability	How does accepted data never get lost?	Replication, WAL, fsync, quorum writes, persistent storage, backups, cross-AZ/region replication
Resilience	How does the system keep working when a component breaks?	Retry, timeout, circuit breaker, failover, replication, bulkhead, graceful degradation
Availability	How does the user always reach the system?	Replication, load balancer, failover, multi-AZ, multi-region, health checks
Reliability	How does the system keep behaving correctly and consistently?	Idempotency, transactions, consistency models, validation, retries, reconciliation, monitoring

4. 50+ Real-World Problems, Trade-offs & Solutions

#	Problem / Scenario	Trade-off (what you gain / what you pay)	Typical Solution
1	Database is overloaded by repeated reads of the same data	+ Lower latency, lower DB load — Stale data, invalidation complexity	In-memory cache (Redis/Memcached) with TTL
2	Cached data goes stale after a write	+ Freshness — Extra latency on write path, complexity	Cache invalidation / write-through / write-behind cache
3	Single DB can’t handle write volume	+ Higher write throughput — Cross-shard queries get hard, rebalancing cost	Sharding / horizontal partitioning
4	One server can’t handle traffic	+ Scale out, no single bottleneck — Statefulness becomes a problem	Load balancer + stateless horizontal scaling
5	A user’s session tied to one server (sticky session)	+ Simplicity — Uneven load, failover breaks sessions	Externalize session state (Redis/DB-backed sessions)
6	Slow user-facing pages due to distant data center	+ Lower latency for users — Cache coherency across edges	CDN / edge caching
7	Write to primary DB is slow due to fsync on every commit	+ Durability — Latency per write	Group commit / async fsync with WAL, tunable durability
8	Primary DB dies, data unreachable	+ Durability & availability — Storage cost, lag	Leader-follower replication with failover
9	Replica lag causes stale reads after write	+ Availability, read scaling — Consistency (read-your-writes issue)	Read-after-write consistency / sticky reads to primary
10	Network partition splits the cluster	+ System keeps running — Risk of split-brain, inconsistency	Quorum-based consensus (Raft/Paxos), fencing tokens
11	Leader election needed after crash	+ Automatic recovery — Complexity, brief unavailability during election	Consensus protocol (Raft, ZooKeeper, etcd)
12	Synchronous call to a slow downstream service blocks everything	+ Simplicity of sync calls — Cascading latency/failures	Timeout + circuit breaker + async/queue-based decoupling
13	One failing dependency takes down the whole service	+ Simplicity — Cascading failure	Bulkhead isolation (separate thread/connection pools per dependency)
14	Repeated retries overload an already struggling service	+ Resilience for transient errors — Can worsen an outage (retry storm)	Exponential backoff + jitter + circuit breaker
15	Duplicate requests processed twice (double charge, double email)	+ Simpler client retry logic — Risk of duplicate side effects	Idempotency keys / idempotent operations
16	Payment must not be lost or double-applied	+ Correctness — Latency, coordination overhead	Distributed transaction / saga pattern with compensating actions
17	Two services need to agree on multi-step business process	+ Consistency across services — Complex failure handling, eventual consistency	Saga pattern (orchestration or choreography)
18	Producer is faster than consumer, causing overload	+ Decoupling, absorbs bursts — Added latency, operational complexity	Message queue (Kafka, SQS, RabbitMQ) as a buffer
19	Need to process huge volume of events in near real-time	+ High throughput, replay capability — Partition management, consumer lag	Kafka/streaming platform with partitioning and consumer groups
20	Traffic spikes overwhelm backend (flash sale, viral post)	+ Protects backend — Some users get rejected/delayed	Rate limiting / throttling (token bucket, leaky bucket)
21	A noisy user/tenant starves others of resources	+ Fairness — Extra bookkeeping	Per-tenant rate limiting / quota system
22	Need globally unique IDs across distributed nodes	+ No central bottleneck — Clock skew issues, ID size	Snowflake-style ID generation (timestamp + node ID + sequence)
23	Auto-increment IDs don’t scale across shards	+ Simplicity per shard — Global uniqueness lost	UUID or Snowflake ID instead of auto-increment
24	Searching text in a relational DB is slow	+ Fast full-text search — Extra infra, indexing lag	Dedicated search engine (Elasticsearch/OpenSearch)
25	Range/lookup queries are slow on large tables	+ Fast reads — Slower writes, extra storage	Database indexing (B-tree, hash index)
26	Too many indexes slow down writes	+ Read performance — Write performance, storage cost	Selective indexing based on query patterns
27	Analytics queries slow down the transactional DB	+ Isolation of workloads — Data freshness lag, extra infra	Read replicas or separate OLAP warehouse (ETL/CDC pipeline)
28	Need strong consistency across regions	+ Correctness guarantees — High latency (cross-region round trips)	Synchronous cross-region replication / consensus (costly)
29	Cross-region sync replication too slow for user experience	+ Low latency for users — Risk of data loss on region failure	Async replication with defined RPO (accepted data loss window)
30	Choosing between SQL and NoSQL for a new service	+ SQL: strong consistency, joins, transactions. NoSQL: horizontal scale, flexible schema — SQL: harder to scale writes. NoSQL: weaker consistency, no joins	Pick based on access patterns (relational integrity vs. massive scale/flexibility)
31	Monolith becomes hard to deploy/scale as team grows	+ Independent deployability, scaling per service — Network calls, distributed debugging, data consistency across services	Microservices architecture
32	Microservices introduce network calls everywhere	+ Modularity — Latency, partial failure complexity	Service mesh (retries, timeouts, observability built-in)
33	Services need to discover each other’s location	+ Dynamic scaling — Extra moving part, potential single point of failure	Service discovery (Consul, Eureka, DNS-based)
34	Client overwhelmed by talking to many microservices directly	+ Simplifies client — Extra hop, potential bottleneck	API Gateway (routing, auth, rate limiting in one place)
35	Need to know if the system is healthy in production	+ Faster incident detection — Overhead of collecting/storing telemetry	Observability: metrics, logs, distributed tracing
36	Debugging a request across 10 microservices	+ Root cause found faster — Instrumentation overhead everywhere	Distributed tracing (OpenTelemetry, Jaeger) with correlation IDs
37	Deploying new code risks breaking production for everyone	+ Safety — Slower rollout, more infra for routing	Canary deployment / blue-green deployment
38	New feature might have bugs affecting all users at once	+ Controlled risk — Extra complexity in flag management	Feature flags / gradual rollout
39	Data schema change breaks running services mid-deploy	+ Zero downtime — Requires careful multi-step migration	Backward-compatible schema migration (expand/contract pattern)
40	Storing huge binary files (images/videos) in the main DB	+ Simplicity of one datastore — DB bloat, slow backups, cost	Object storage (S3/GCS) + DB stores only metadata/URL
41	Large file uploads time out or fail	+ Reliability for big files — More complex client/server logic	Chunked/multipart upload with resumability
42	Popular content (celebrity post) creates a “hot key”	+ N/A for the hot key itself — Single shard/cache node overwhelmed	Key splitting / local + shared cache layers / request coalescing
43	Thousands of clients hit the DB directly on cache miss at once	+ N/A — Cache stampede, DB overload	Request coalescing / locking / stale-while-revalidate
44	Need to know which server is healthy behind a load balancer	+ Traffic only to healthy nodes — Extra network chatter	Health checks (active/passive)
45	A single slow node degrades overall latency (tail latency)	+ Better P99 latency — Wasted extra work/resources	Request hedging / “send to 2, take first response”
46	Choosing consistency model for a distributed store	+ Strong: correctness. Eventual: availability & low latency — Strong: latency/availability cost. Eventual: stale reads, app-level complexity	CAP-aware choice: CP vs AP depending on business need
47	Users in different regions need low-latency writes	+ Fast local writes — Conflict resolution needed	Multi-leader replication with conflict resolution (CRDTs, last-write-wins)
48	Concurrent edits to the same record conflict	+ Allows concurrency — Merge complexity	Optimistic concurrency control / versioning / CRDTs
49	Long-running batch job blocks user-facing request path	+ Responsiveness for users — Eventual consistency of batch result	Move to async background job + notify on completion (webhook/polling)
50	Need exactly-once processing semantics in a stream	+ Correctness — Throughput cost, complexity	Idempotent consumers + dedup table, or transactional outbox pattern
51	Service crashes mid-transaction, leaving inconsistent state	+ Simplicity of direct writes — Risk of partial failure	Transactional outbox pattern (DB write + event in one transaction)
52	Third-party API is flaky and slow	+ Keep system responsive — Added indirection	Timeout + retry + circuit breaker + fallback/cached response
53	Search index falls out of sync with source-of-truth DB	+ Fast search — Eventual consistency, extra pipeline	Change Data Capture (CDC) to keep index updated asynchronously
54	Need to roll back a bad deployment fast	+ Fast recovery — Requires versioned, backward-compatible artifacts	Immutable deployments + one-click rollback
55	Config changes require redeploying the whole service	+ Simplicity — Slow iteration, risk with every deploy	Externalized/dynamic configuration service (feature flags, config server)
56	Storage cost grows unbounded with data retention	+ Keep all data — High storage cost	Data lifecycle policies (archive to cold storage, TTL deletion)
57	Global users get slow static asset loads	+ Fast global delivery — Cache invalidation across edge nodes	CDN for static assets with cache-busting via versioned URLs

5. The One-Sentence Summary

Every “solution” in system design is really a trade: pick the pain you can afford, for the property you actually need.

There is no universally “correct” architecture — only the one that fits your traffic, your consistency needs, your team’s operational maturity, and your failure tolerance.