System Design Interview Question: How Does Load Balancing Work?

Short answer: a load balancer spreads incoming requests across multiple servers so no single server is overloaded, and routes around servers that fail health checks.

Layer 4 vs Layer 7

• L4 (transport): routes TCP/UDP connections by IP and port. Very fast, protocol-agnostic, can't see HTTP paths or headers.
• L7 (application): understands HTTP. Can route /api and /images to different services, terminate TLS, add headers, retry and rate-limit.

Balancing algorithms

• Round robin (optionally weighted): simple and even when requests are similar.
• Least connections / least outstanding requests: better when request durations vary.
• IP or consistent hashing: the same client or key goes to the same server; useful for caches.
• Power of two choices: pick two servers at random, use the less loaded one; simple and effective at scale.

Health checks

The balancer regularly calls something like /healthz on each server and stops sending traffic to failures. Combined with connection draining, this enables rolling deploys without errors.

Sticky sessions

Pinning a user to one server works around in-memory sessions, but hurts even distribution and failover. Better: keep servers stateless and store sessions in Redis or a database, so any server can handle any request.

Avoiding a single point of failure

• Run load balancers in pairs or use a managed cloud load balancer that's highly available by design.
• Use DNS-based or anycast global load balancing to send users to the nearest healthy region.

Bonus

Inside microservice architectures, load balancing often happens client-side or in a service mesh sidecar, so each service call is balanced without a central hop.