Load distribution to multiple external hosts

[kapow]

In my application, I have several instances of the same appliance that grab data from websites. To more evenly distribute the request load, I would like for this custom appliance to submit it's requests through the HLB class which would then by connected to several NET appliances:


DataGrabber ----- | | ----- NET
DataGrabber ----- | | ----- NET
DataGrabber ----- | HLB | ----- NET
DataGrabber ----- | | ----- NET
DataGrabber ----- | | ----- NET


The existing configuration is


DataGrabber ----- |
DataGrabber ----- |
DataGrabber ----- | NET
DataGrabber ----- |
DataGrabber ----- |


However, in the HLB configuration, the DataGrabber appliances have no gateway through to the NET appliances and no DNS capability. Is there someway to branch HLB and get it to act as a gateway as well?

[PeterNic]

Kapow,

To answer your question directly -- yes, you will have to branch HLB.

Before we explore this further, let's consider whether this is the right structure: remember that HLB is a single appliance; if all traffic passes through it, then the max. amount of traffic you can pass is constrained by HLB. HLB has more processing to do on the data path than NET -- NET just forwards traffic at L3, while HLB is a L7 switch.

What is the bottleneck that you are trying to remove? DNS? Multiple connections to the same web site from the same IP? Bandwidth through the external network? (is your external network 100Mbps or 1Gbps?)

Depending on the answers to these questions, you may have better results by either connecting the NETs to the DataGrabbers directly (one or more DataGrabber to a NET), or by separating the DNS traffic and load balancing that, while leaving NET as a single gateway; or a combination.

Regards,
-- Peter

[kapow]

Ok, I've been wrestling with this for a couple of weeks and progress has hit a roadblock.

To redefine what I'm trying to do, is setup a standard application that acts as a web proxy instead of bundling it into my main application. The requirements are:

Proxy standard and SSL requests through one port - 80 - this is required given the application utilizing the proxy
Distribute the requests through multiple IP addresses and forward on to the target urlI have been successful in setting up the following application configuration:

| APACHE_PROXY_CUSTOM APPLIANCE -> NET
IN ->HLB->| APACHE_PROXY_CUSTOM APPLIANCE -> NET
| APACHE_PROXY_CUSTOM APPLIANCE -> NET


This configuration works perfectly for standard HTTP requests. It does not for SSL requests, hence the issue. I get a 501 - Not Implemented error. This is due to the standard pound configuration not listening for HTTPS requests.

In attempt to rectify this issue, I branched the HLB appliance and attempted to modify the /usr/lib/pound/pcfg.tmpl to listen for HTTPS requests on port 80. Since pound is already listening for HTTP requests on 80, this isn't allowed. Additionally, it appears pound will take the HTTPS request and turn into an HTTP request which, when forwarded, won't, of course, work.

So, I attempted to at least have pound configured to listen for HTTPS requests only simply by changing the LISTEN directive to HTTPS. While requests were received by the HLB appliance, they were never forwarded. I'm not completely clear as to why. Several other changes based on extensive web searches yielding no changes in these results.

Overall, it seems using pound as load balancer doesn't make sense when trying to meet my requirements.

My future attempts will be to replace the HLB appliance with a custom appliance and attempt to use the Apache proxy_balancer to distribute to the secondary proxies.

Any guidance here would be greatly appreciated.

What's frustrating is that, surely, this is something almost all large IT organization configure and setup - take all web traffic at one ip and distribute it to many for both http and https requests.

[PeterNic]

Can you use the INSSL appliance instead of IN? And then have the proxy appliance re-encrypt?

Another alternative is a prototype appliance we had at some point, LB4; it is a layer 3 round-robin load balancer that uses iptables. I will check with our developers if they can pull that from the archive. One caveat: this was a prototype; even though it generally works, some additional work on it may be needed.

Regards,
- Peter

[kapow]

Thanks Peter.

If I use the INSSL appliance I have to setup a key which would be a potential man-in-the-middle, correct? I started with that but removed it and went with standard in.

If the LB4 appliance might work, I'm willing to try it.

[PeterNic]

Kapow,

On a second thought, LB4 is not going to help you here -- it will not work because the proxy will not be able to extract the target host from the SSL-encrypted html header.

I think there are two simple options:
- use a custom version of the NET appliance, with multiple outgoing IP addresses (see private e-mail, aka the IP-hopping NET gateway)
- put the INSSL gateway and a key; tell the clients to accept or ignore the server SSL certificate (in this case, your proxy SHOULD enforce the SSL certificate match to host name, when forwarding https requests -- this way there will be no degradation in security)

Does the above help?

-- Peter
-

[kapow]

Worth a try. When you say see private-email, IP-hopping NET gateway, where do I find that?

[PeterNic]

It was in an e-mail I sent you directly. I will re-send it.

Regards,
-- Peter

[enovikoff]

I would like the IP-hopping NET gateway as well. I have a customer application that sends emails from mailing lists. To make sure mailing lists don't get blocked by actions from other mailing lists, he wants one IP per mailing list. Creating one sendmail appliance for each IP address is too expensive - I'd rather just have the outgoing mails be mapped to a particular IP address. If you have an IP-hopping NET gateway, then that might do the trick. Do you mind copying me on the private email you sent kapow? (Yes, the customer only sends opted-in emails :)

Thanks,
-Eric