Configuring better load balancing and health checks for Plone with HAProxy

In the previous post of this blog I enumerated some problems we face when using normal load balancer health checks on Plone instances and I described a possible solution using five.z2monitor. In this post I'm going to show you how to configure HAProxy and other components of the stack in order to get the most of it.

As mentioned previously, we can do load balancing using nginx, Varnish or HAProxy. Both nginx and Varnish provide more features on their commercial versions, but I'm going to focus only on the community versions. In my opinion, if you ever need more features in your load balancer you should try HAProxy before buying one of those subscriptions.

On nginx the load balancer is handled by the ngx_http_upstream_module, it's very easy to enable and configure and it has some nice features (like backup servers and the least connections method), but also many limitations (like basic health checks and no queue control). This works very well on small sites.

On Varnish the load balancer is handled by the directors module, and I have to admit that I'm not impressed by its features as it lacks support for least connections which is the chief method with Plone instances. On the other side, backend definitions and health checks are more configurable, but support HTTP requests only. You may consider Varnish as a load balancer if you already use it on high traffic sites and is going to do a good and honest work.

(Previously, I was arguing in favor of the hash method, but later tests showed that the standard round robin is less prone to instance overloads.)

As you can see both nginx and Varnish can only handle HTTP requests for health checks and we have very limited control of request queues, something that in my experience has prove to be an issue.

(I never care about the ngnix queue, but in Varnish I always configure a max_connections directive to avoid flooding instances with requests that could make them unreachable for a long time.)

Let's go back on track: HAProxy is a very complex piece of software; it's so complex and its documentation so huge that I preferred just to removed it from the equation when I first had to take care of the infrastructure in our company some years ago.

That choice proved to be right at the moment, but I always like to investigate and act as my own devil's advocate from time to time. So, when one of our main sites started receiving some very nasty DoS attacks I started playing with it again.

It took me a whole week to prepare, test and review the configuration I'm going to share with you, but first I'm going to talk a little bit about our infrastructure and rationale.

The sites I'm talking about use (at least) 2 servers with a full stack: nginx, Varnish, HAProxy and Zope/Plone; both servers are configured using ZODB Replicated Storage (ZRS), one acting as the master ZEO server and the other as the slave. All Plone instances point to both ZEO servers. When the master ZEO server fails, all instances connect automatically to the slave in read-only mode. When all instances in one server fail, Varnish connects to the HAProxy in the other server as a backup. When Varnish fails, nginx tries to use the Varnish on the other server. That's more or less how it works.

This is our working haproxy.cfg and I going to explain some choices I made:


First the standard listen section: we use option tcp-check (layer 4) to make the health checks as is way faster that doing a HTTP (layer 7) check: we ask Zope on the alternate binded port for the ok command and expect the OK string as a result.

In my tests layer 4 checks took just a couple of milliseconds to finish, 2 orders of magnitude less than layer 7 checks. Faster, non-blocking responses mean you can check more frequently and you're going to detect sooner any failure on the instance.

Health checks are done every 2 seconds and, in case of failure, the slowstart parameter avoids flooding an instance during it's defined warming up period of one minute.

We inform the status of the backend upstream to Varnish using the monitor-uri directive: if we have no backends alive, HAProxy's monitor will also fail.

In Varnish we have have configured the backend like this:


Varnish will detect any backend failure in 2 seconds, at worst; Varnish will then try to use the HAProxy backend on the other server as a backup.

Back into HAProxy's configuration, we set maxconn 4 on backend servers to avoid sending more that 4 request to any instance; similarly, we set timeout queue 1m to keep the queue size under control.

Finally, the global section: you'll see we're using maxconn 256; in my opinion there is no reason to use a bigger value: if you have more that 256 requests on queue, you obviously have a problem and you need more instances to serve the traffic.

The typical stats screen of HAProxy with this configuration looks like this:

As you can see, over a period of almost 25 days we have no backend failures in this server, even with a large number of individual instance failures.

Last, but not least: if you want to cut the start up time of your instances you have to include the following directives on your buildout configuration (more about that on this Plone forum thread):

[instance]
…
# play with this value depending on the number of objects in your ZODB
zeo-client-cache-size = 256MB
zeo-client-client = zeoclient

Share and enjoy!