Wishlist for a new load balancer (HLB v2)

[PeterNic]

Folks,

We have collected quite a few suggestions from customers on the load balancer appliance (HLB). I am starting this thread in order to share them with everyone and ask for feedback and additional ideas/suggestions.

Load Balancer Software

Consider using HAProxy instead of pound.

Statistics

Add load-balancer-specific counters/stats, especially state of the outputs. Consider at least the following:

number of connected outputs (i.e., the max number of outputs that can be active)
number of active outputs (up)
number of inactive outputs (down)


For completeness, we can also provide the state of each output in a separate counter, with the following values: 0-unconnected, 1-up, 2-down.

In addition, stats for the performance will be helpful (e.g., avg response time for each output and all; # connections; # requests/sec; avg # up outputs, etc.)

Health check options

Add more flexibility to how back-end servers health is being checked
We can keep the default as it is now (http ping, essentially connecting to port 80 over TCP and disconnecting). Add the following options:

how frequently to check (separate values for up-check and down-check?)
check method: { tcp_ping | http_get }
if http_get:
-- name of document to retrieve (e.g., /health_check.php)
-- check value: string to look for in the retrieved document (e.g., "This service is UP". No check if empty)


Note: when using http_get as a check method, use a special user agent when requesting, so that the servers can exclude it from the logs/log analysis (e.g., "AppLogic-healthcheck/1.0 HLB/0.1.2 Wget/1.9")


Programmatic Control

Allow programmatic control of which outputs are currently considered enabled -- allows for disabling an output, e.g., to restart and/or upgrade the server behind it, or to disable the output before shutting down a server (e.g., due to low load). The control will likely be provided in the form of a separate input terminal, "ctl", using REST-like web service interface. The same terminal will also provide some basic status -- e.g., which outputs are currently considered up.

Non-http load balancing
HAProxy can load balance other services. It may be useful to consider being able to load balance other TCP/UDP traffic (without session awareness).

Please feel free to post responses to this thread with comments and ideas. We appreciate your input!

Regards,
-- Peter

[PeterNic]

And the first question was: what should we name it?

Here are a few ideas:

HALB: HAProxy-based Load Balancer
HLB2: Http session aware load balancer, version 2
HLBP: HLB "plus"


So far bets are on HALB -- not many offensive associations and it almost matches a Bulgarian word for "beer mug" (халба / halba)

-- Peter

[tmart]

I like the idea of the beer mug association, but it fails to flow naturally from my non-Bulgarian brain.

Regarding the health check options, I think if http_get is allowed, then it would be very useful to:

1) optionally allow the string to be a REx (including negation of patterns)--- maybe a checkbox to indicate that the string is actually a REx

2) permit basic checking for user-specified good and/or bad HTTP "error" codes (eg. Good list might be: 2xx, 3xx, ; Bad List might be: 4xx, 5xx, etc.)

[digerata]

HLB2 fits in with the rest of the naming...

I am really excited about what you propose. Any ideas on a release date at this point?

[PeterNic]

digerata,

I expect we'll have HALB available late August / early September. A prototype is available for the brave of heart.

Regards,
-- Peter

[digerata]

That time frame would be awesome. I'd be happy to help with any beta testing...

[PeterNic]

Let me know when you're ready. We have the prototype (not beta) available for AppLogic 2.3; if you are ready to put it through its paces, we can cook a 2.1 version as well. PM me for details.

Regards,
-- Peter

[digerata]

Peter, we are ready to test it. We are running 2.3 now. How do we go about obtaining it?

Thanks!

-Mike

[digerata]

Peter, we are ready to test it. We are running 2.3 now. How do we go about obtaining it?

Thanks!

-Mike

[Karl]

Hi Peter,

Wouldn't mind getting my hands on this, as we've got some other items we'd like to load balance other than just HTTP - SMTP being the main one, whilst it's not crucial, it does save on IPs and appliances. We're running 2.3 :)

Thanks,

[PeterNic]

Hi Peter,

Wouldn't mind getting my hands on this, as we've got some other items we'd like to load balance other than just HTTP - SMTP being the main one, whilst it's not crucial, it does save on IPs and appliances. We're running 2.3 :)

Thanks,

Karl, please contact your VPDC support -- they can get you the appliance. If you are a direct 3Tera customer, please use https://helpdesk.3tera.com/ or email helpdesk at 3tera-dot-com.

Best regards,
-- Peter

[digerata]

Jessie just set us up with the HALB appliance. Thanks for that! It is looking great except for an issue I have with the health check.

I'm not sure if this is the right place to post this issue, so apologies if it should have been elsewhere.

We should be able to tell HALB the a specific URL to check for a terminal being up or down. I would assume that this URL would need to be relative so that it could be used on each of the out terminals. So in our case, /monitor/status.jsp would report the status as 200 or 500 depending on state. Unfortunately, starting up the appliance with that URL for the healthcheck_url property results in an error:

Error: the healthcheck_url must begin with 'http://' and contain a full path to the document (ex: http://host.name.domain/file/to/check/for.html).

But how can HALB determine the status of a single connected host when the URL is not specific to any one host? It is it supposed to be something like http://out/monitor/status.jsp?

Thanks,

-Mike

[digerata]

Reading further in the documentation on HALB, it appears that the 3tera's idea for the health check was to tell if the overall service is up or down and not to verify if a host on a connected terminal is up or down.

Is this the case?

[Karl]

Got a copy of it (not installed it yet) but reading the PDF, it looks like it's still HTTP only, which is a shame, as we really were hoping that it'd make use of the full range of HAproxy and do balancing on other TCP ports as well.

[PeterNic]

Got a copy of it (not installed it yet) but reading the PDF, it looks like it's still HTTP only, which is a shame, as we really were hoping that it'd make use of the full range of HAproxy and do balancing on other TCP ports as well.

Karl,

This is a limitation of the beta.

I have a query out to the HALB developer to see what it would take to rev HALB to provide non-http load balancing and whether we can do it in a short timeframe so that you can get to use it in the mode you need.

I am sure we won't ever the full extent of HAProxy's capabilities in a single appliance (just as WEBx does not even begin to expose all apache configuration capabilities). It may be that we'll end up having a separate L3LB, built from the same codebase, so that you first choose the appropriate function appliance, then configure it (rather than making a more complex-to-configure appliance).

Cheers,
-- Peter

[PeterNic]

Mike,

Jessie just set us up with the HALB appliance. Thanks for that! It is looking great except for an issue I have with the health check.

I'm not sure if this is the right place to post this issue, so apologies if it should have been elsewhere.

This is as good a place as any. No worries; once we release it, we'll open a separate forum thread for issues and questions related to HALB.


We should be able to tell HALB the a specific URL to check for a terminal being up or down. I would assume that this URL would need to be relative so that it could be used on each of the out terminals. So in our case, /monitor/status.jsp would report the status as 200 or 500 depending on state. Unfortunately, starting up the appliance with that URL for the healthcheck_url property results in an error:

Error: the healthcheck_url must begin with 'http://' and contain a full path to the document (ex: http://host.name.domain/file/to/check/for.html).

But how can HALB determine the status of a single connected host when the URL is not specific to any one host? It is it supposed to be something like http://out/monitor/status.jsp?

:confused: Looks like we had a brain freeze, hopefully temporary

This should've been exactly as you stated it, a path relative to the host. For example, /monitor/status.jsp. Note that this will not set the hostname apache env var -- if that is required, we'll have to add a separate property; it is possible to list the hostname as "localhost" although I am not sure this will help. I'll follow up on this; in any case, something is wrong -- either the doc or the implementation.

Reading further in the documentation on HALB, it appears that the 3tera's idea for the health check was to tell if the overall service is up or down and not to verify if a host on a connected terminal is up or down.

Is this the case?

No, you were right the first time; see above.


Thanks,
-- Peter

[Karl]

Hi Peter, fully appreciate that HAproxy isn't exactly a simple to configure appliance if you don't want to overload it with 100s of settings. The way we've built our own appliances is to put really key properties in the appliance preferences pane, then we've allowed an option to drop on a config template file, to over-ride the default config template, which gives us the option to tweak all settings for a piece of software.

[digerata]

:confused: Looks like we had a brain freeze, hopefully temporary

This should've been exactly as you stated it, a path relative to the host. For example, /monitor/status.jsp. Note that this will not set the hostname apache env var -- if that is required, we'll have to add a separate property; it is possible to list the hostname as "localhost" although I am not sure this will help. I'll follow up on this; in any case, something is wrong -- either the doc or the implementation.


Great! No worries, here. As an aside, we found if we put in the IP address of the INSSL connected to the HALB, the failover of the terminals *appears* to work. (value was: http://X.X.X.X/monitor/status.jsp) We were going to do further testing to confirm that it wasn't luck and really did work but hadn't gotten to it yet. I wonder if HAProxy was just taking everything after the hostname for the health check? Regardless, it is working for us right now.

-Mike

[PeterNic]

@digerata: I confirmed the brain freeze was only partial -- the behavior is as intended, the hostname is ignored by HALB but passed as-is to the backend servers in case you use virtual hosts. Good idea (I think) but not the best implementation, as the virtual hosts are the exception for most apps. What we decided to do is keep the current behavior as an option but make the default simply a root-relative url, like /monitor/status.jsp (so that if you don't know about virtual hosts, you don't have to learn).

@Karl: we now have a prototype of a new load balancer appliance, L3LB (layer 3 load balancer), currently supporting only TCP (with UDP coming down the pipe as well). Let our support know if you want to try it out. The offer is open to all who are willing to take a (working) prototype :)

Best regards,
-- Peter

[Karl]

Peter - Sounds good, will give you a mail during the week, want to try 2.4.5 as well, but someone has stolen parts from my distribution server, lol (we're only testing/developing our appliances at the moment)

[PeterNic]

The discussion for layer 3 functionality -- load balancing TCP and UDP -- has been moved to a new thread: http://forum.3tera.com/showthread.php?p=1462#poststop

Check it out, too, as some of the suggestions there apply to HALB as well (e.g., Karl's suggestion to add a "server of last resort" output terminal).

The discussion on HALB prior to its final production release can continue in this thread.

Regards,
-- Peter

[PeterNic]

Karl, digerata, and all who tried the prototype/beta version:

Thank you for your suggestions and for helping out with the early testing. We really appreciate your input and participation!

HALB is now available in the standard catalog of AppLogic 2.4.7, superseding HLB. Please continue posting HALB-related issues and suggestions at http://forum.3tera.com/showthread.php?t=378 or open a new thread in this forum (Feature/App Requests (http://forum.3tera.com/forumdisplay.php?f=12)).

Regards,
-- Peter