Load Balancer

Load balancer

A load balancer sits in front of a group of web servers and evenly distributes incoming traffic among them. The group it distributes to is called the load-balanced set.

How the traffic flows

Clients no longer connect to a web server directly. Instead:

1. Clients connect to the public IP of the load balancer.
2. The load balancer picks a web server from the pool and forwards the request to it over a private IP.
3. The web server responds back through the load balancer to the client.

Because the web servers are now only reached through the load balancer, they no longer need public IPs at all. A private IP is an address that is only routable between servers inside the same network and cannot be reached from the internet. Moving the web servers onto private IPs is better for security: an attacker on the internet cannot address them directly, only the load balancer is exposed.

How does it pick a server?

Step 2 above says the load balancer "picks a web server from the pool." The rule it uses to pick is called a load balancing algorithm. A few common ones:

• Round robin. Hand requests to servers in a fixed rotation: server 1, server 2, server 3, server 1 again, and so on. Simple and predictable. It assumes every request costs about the same and every server is equally powerful, which is not always true.
• Weighted round robin. Same rotation, but you give stronger servers a higher weight so they receive proportionally more requests. Useful when the pool has a mix of big and small machines.
• Least connections. Send the next request to whichever server currently has the fewest open connections. This adapts to reality: a server stuck on a few slow requests stops receiving new ones until it frees up. Good when requests vary a lot in how long they take.
• Least response time. Like least connections, but it also factors in which server is answering fastest right now, so it leans toward the server that is both least busy and quickest.
• IP hash. Run the client's IP address through a hash function and use the result to pick a server. The same client keeps landing on the same server, which matters if that server holds session state for them in local memory. The downside is that if that server dies, those clients lose their session.

Round robin and least connections cover most cases. The book does not go deeper than "evenly distributes," so treat this as background you will reach for later when session state or uneven request cost forces the choice.

What this buys the web tier

Adding a load balancer and a second web server fixes the single point of failure in the web tier and improves its availability.

• Failover. If server 1 goes offline, the load balancer notices and routes all traffic to server 2, so the site stays up. A fresh healthy server can then be added back into the pool to carry its share of the load.
• Easy scaling. If traffic grows and two servers are no longer enough, you just add more servers to the pool. The load balancer automatically starts sending requests to the new servers, with no change on the client side.

What is still fragile

The web tier now looks solid, but the data tier does not. The design still has a single database, so it has no failover and no redundancy: if that one database fails, the whole application loses its data layer. The common fix is database replication, which is the next post.

But what if the load balancer itself fails?

A sharp reader will notice we just moved the single point of failure rather than removing it. All traffic now flows through the load balancer, so if it dies, the whole site is unreachable even though every web server is healthy.

In practice a load balancer is not run as one box. You deploy at least two, a primary and a secondary, that watch each other with a heartbeat. If the primary stops responding, the secondary takes over its IP address and keeps serving, usually within a few seconds. Managed load balancers from cloud providers (AWS ELB, GCP Cloud Load Balancing, and similar) do this for you: what looks like a single endpoint is a redundant, self-healing fleet behind the scenes. So the load balancer is a single point in the diagram, but not a single point of failure in a real deployment.