A load balancer distributes incoming requests across a pool of servers so that no single one is overwhelmed, and it removes servers that fail. It is a central component of automated infrastructure and one of the first things added when a service must scale. Its reputation as a safety net is partly deserved and partly dangerous, because it encourages the belief that adding a load balancer settles questions of capacity and availability. It does not. A balancer works with simple information, such as the number of connections or the answers to health checks, and it makes decisions based on assumptions about the workload. When those assumptions fail, traffic can be evenly distributed and the system can still fall over.
How effective is it with variable workloads?
Basic algorithms such as round robin treat every request as equal. If some requests cost a thousand times more than others, an equal number of requests per server produces very unequal load, and one server can be saturated while its neighbours are idle. Least-connections and latency-aware strategies do better, because they respond to current conditions. Even they struggle with sudden changes, since information is always slightly out of date. Newly started servers that have not warmed their caches may be given a full share of traffic immediately and perform badly, which is why slow-start features exist. Some workloads need to stick to one server, for example to use cached session data, and session affinity then works against even balancing.
How does it handle unexpected failures?
A balancer only learns that a server has failed through health checks or failed requests, and that takes time. During the detection interval, traffic continues to flow to a broken server and users see errors. Health checks that are too shallow report healthy for a server that is up but unable to reach its database, while checks that are too strict can remove every server at once when a shared dependency is slow, turning a partial problem into a total outage. When a failing server is removed, its share of traffic shifts to the others, which may push them over their capacity and cause cascading failure. Retries by clients and by the balancer itself can multiply the load at exactly the wrong moment.
Not all protocols and connections are equal
Balancers operate at different layers. A layer 4 balancer distributes connections, and a layer 7 balancer understands requests and can route on paths or headers. Long-lived connections complicate both. With protocols such as WebSockets or HTTP/2 and gRPC, a single connection carries many requests for a long time, so balancing connections at the moment they are made does not balance the work, and a new server added later may receive almost nothing. Some stateful or streaming workloads cannot be moved between servers at all. Addressing these cases takes measures such as request-level balancing, periodic connection recycling, connection limits and explicit draining when servers are removed.
Why it matters
Load balancing remains essential, and it works best as one layer of several. Autoscaling adds capacity when demand grows, backpressure and rate limits protect servers from overload, timeouts and circuit breakers contain failures, and good health checks reflect whether a server can actually serve users. Testing the system under uneven load and during failures reveals where the balancer's assumptions do not hold, before real users do.
A load balancer shares the load. It does not decide whether the load is something the servers can bear.
