HAProxy Load Balancing: Health Checks and High Availability

HAProxy can distribute requests across several application nodes and stop sending new traffic to a backend that fails a health check. That is useful, but it is not high availability by itself. If the load balancer is a single machine, its failure can still take the service offline. Application state, databases, DNS, TLS, and the network path must also be considered.

This guide uses a conceptual HTTP backend and an illustrative HAProxy configuration. Test every directive against the HAProxy version installed on your host and the real health endpoint of your application. The two backend IPs below are documentation examples; do not copy them into a real configuration.

1. Define a health endpoint that means something

An endpoint that always replies 200 OK even when the application cannot serve customers will keep a bad node in rotation. A useful health check should be fast, stable, and limited to dependencies necessary for the service. Decide whether a database outage should make a node unhealthy; if every node shares that database, removing all nodes may not improve recovery.

Document the expected status code, timeout, and whether the check needs a Host header. Protect diagnostic detail from anonymous visitors. The health endpoint should not expose credentials or internal topology.

2. Start with active HTTP checks

The following example declares a frontend and two backends. Adapt TLS termination and logging to your environment. The check parameter enables active checks on each server; option httpchk changes the probe from a TCP connect test to an HTTP request.

frontend public_http
    bind :80
    default_backend app_nodes

backend app_nodes
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    server app1 192.0.2.11:8080 check inter 3s fall 3 rise 2
    server app2 192.0.2.12:8080 check inter 3s fall 3 rise 2

192.0.2.0/24 is a documentation address range. fall 3 requires repeated failures before removal; rise 2 requires repeated successful checks before readmission. Tune these against expected response time and the cost of flapping. A probe interval that is too aggressive can create load, while a slow one delays detection.

3. Validate before any reload

Keep a copy of the last known-good configuration and use the installed HAProxy binary to validate syntax. The service name and path below are common Linux examples; verify them first.

sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl status haproxy --no-pager
sudo journalctl -u haproxy --since '15 minutes ago' --no-pager

Only reload after the syntax check passes and the target nodes are ready. A valid file can still route traffic to the wrong backend, so validate the live application with representative requests, not just the HAProxy process state.

4. Test failure and recovery as an exercise

In staging, take one backend out of rotation using a controlled test method, then observe whether traffic reaches the other node. Verify how long the failure takes to register, how sessions behave, and what the customer sees. Restore the backend and verify it is admitted after successful checks. Never run a deliberate failure drill on production without approval, a maintenance window, and a rollback plan.

Watch HAProxy metrics and logs for backend status transitions, queue depth, connection errors, and response times. Compare these with node metrics. If both backends fail together, investigate shared dependencies rather than simply changing the health-check threshold.

5. Close the availability gap around the balancer

For true service-level availability, review redundant HAProxy nodes or a managed load-balancing layer, failover address/DNS behavior, certificate distribution, configuration synchronization, and persistent application state. A traffic switch cannot repair a shared database outage or a broken deployment replicated to both nodes.

Troubleshooting and common errors

• Backend is marked down while /health works in a browser: inspect the probe method, host header, TLS settings, path, expected status, and source address.
• Requests still fail after one node is removed: examine the surviving node's capacity and the shared database or cache.
• Backend flaps between up and down: correlate latency, inter, fall, rise, application warmup, and resource saturation.
• Sticky sessions break after failover: move essential session state to a shared, resilient store or design for stateless requests where practical.
• All traffic depends on one HAProxy host: plan balancer redundancy; two application nodes alone do not remove that single point of failure.

Security considerations

Protect HAProxy administrative endpoints and stats with authentication and network restrictions. Use TLS for external traffic and consider encryption to backends where the network boundary requires it. Preserve real client IP only through trusted headers or the PROXY protocol when both sides are configured. Never trust a header supplied directly by arbitrary clients.

FAQ

Is a TCP connect check enough?

It proves a port accepted a connection, not that the application can serve a useful request. HTTP checks often provide better service-level evidence.

Why use rise and fall counters?

They reduce unnecessary state changes caused by a single transient result. Choose values based on expected failure and recovery behavior.

Does HAProxy replace application monitoring?

No. It can route around unhealthy backends, while monitoring should explain why they failed and whether the remaining capacity is adequate.

Continue reading

Understand the surrounding Linux health monitoring and backup recovery design. Talk to East4Serv about a reviewed high-availability architecture.

Reference: HAProxy health-check documentation.