500 Internal Server Error when retrieving XML

[kuletog12]

hi,

we are testing a webservice to pull-out XML data. Sometimes it works but most of the time we encounter the 500 Internal server error. We encountered the same error at our development server when we used pound. We increased the timeout and it worked just fine.

Now that we uploaded it at the LT environment we have the HLB load balancer connected to 2 web appliances. We encountered the same error but even if we increased the timeout of the HLB, we still encounter the same error.

I checked the monitoring tool and the CPU does not even reach 50%, and the received bytes is only around 32-35k.

:confused:

[PeterNic]

We are currently working on some problems with pound in HLB and INSSL. The key problem we found is that the kernel adjusts socket buffer sizes whenever you transfer larger files -- and this exceeds the memory usage calculations we made and based HLB/INSSL's resource ranges on.

For that issue, the workaround is to provide HLB/INSSL with 256MB RAM (you may need to branch the appliance and modify its max. memory; you can put it back in a catalog).

Also, check the disk space (df -h); there is a log that doesn't rotate properly and may lead to disk full (unlikely to be causing the problem you are seeing but worth a shot). Also check the pound config file as created by HLB on startup -- and compare it to what you had on your development server.

How long does it take to pull out a single document (1, 5, 10, 15 seconds or longer)? Which version of the grid do you use ("grid info" in the shell)?

Let us know if anything suggested here helped... if not, we'll get someone to look at your system.

Regards,
-- Peter

[kuletog12]

Hi Peter, Thanks for the reply


HLB and INSSL memory is already 256M

HLB disk space:
Filesystem Size Used Avail Use% Mounted on
/dev/hda1 78M 66M 8.3M 89% /
/dev/hda2 194M 151M 34M 83% /usr
none 129M 0 129M 0% /dev/shm


Filesystem Size Used Avail Use% Mounted on
/dev/hda1 78M 65M 8.9M 88% /
/dev/hda2 194M 152M 33M 83% /usr
/dev/hda3 9.7M 1.2M 8.1M 13% /mnt/key
none 129M 0 129M 0% /dev/shm

To pull-out a single document without error it takes 15secs
Version of the Grid is 2.1.1

I'll still have to check the pound config....

[kuletog12]

Hi Again,

How can I set the client directive of pound in the HLB?

Thanks

[PeterNic]

Thanks for the info.


To pull-out a single document without error it takes 15secs

On the 15 seconds -- one customer has reported that HLB appears to drop requests that don't start sending data 15 seconds from the request (possible problem with the TimeOut directive). Strangely, INSSL, which uses the same pound, doesn't do that. I guess that there is a config setting which we do not configure in HLB's pound config. If find the differences from your development setup of pound, this may lead to the answer -- please let us know.


How can I set the client directive of pound in the HLB?

The only way to modify any aspect of any appliance in a way that properties don't allow is to simply branch the appliance and make the change. If you want the change to be configurable -- almost always a good idea -- then add it as a property instead of hard-coding it. You can see the scripts in the /appliance directory on how the appliance uses its existing properties to prepare pound's config file.
Once you are happy with the appliance's operation, it may be best to make it a catalog appliance -- just move it to the 'user' catalog.

If you think the change could be beneficial to others, please post at least a description of the change in the Appliance Workshop, under the appliance's thread (each appliance has its own thread on the forum, for defects, questions and suggestions). HLB and INSSL seem to be the most active appliances.

Regards,
-- Peter

[bkonia]

Peter,

I'm experiencing this same problem. Error 500 on the load balancer when the web server takes more than 15 seconds to respond. I did some digging around and discovered the following:

1. If no TimeOut value is specified in the Pound config file, it defaults to 15 seconds.

2. The HLB appliance has a Timeout property with a default value of zero.

3. In /usr/lib/pound/pcfg.pl, the $timeout variable is set to 2147483647 unless the user has specified a non-zero timeout value in the HLB appliance Property settings.

4. In /usr/lib/pound/pcfg.tmpl, there is no line anywhere in the template that sets the TimeOut parameter. Thus, even though the $timeout variable is set in the Perl script, it is not used in the template.

I therefore decided that the obvious solution was to branch the appliance and add the following line to pcfg.tmpl:

TimeOut $timeout

I did exactly that, but when I restarted the Grid, it failed to start because the HLB failed to start. Therefore, something about the line that I added is causing Pound to not start.

Can you help me resolve this please? I'm using the 2.3.9 hf2425 beta.

[PeterNic]

bkonia,

The timeout property of the HLB appliance controls expiration of idle sessions. From its datasheet: "Optional timeout (in seconds) to expire inactive sessions. If this is set to a non-zero value, a session that has been inactive for more than the value of this property in seconds is 'forgotten'. Further requests bearing the 'forgotten' cookie are treated as if they have no cookie at all and are directed to to a random server, using the usual round-robin method.". That explains why it doesn't affect the request completion time.

I'll check with the folks here whether someone has successfully gotten long requests to work with HLB reconfiguration and how.

Also, we have a prototype available of our next load balancer, HALB, which is based on HAProxy. Early tests showed that is about twice as fast, and does not exhibit some of the problems we've seen with pound in HLB. If you want to join the test program, please open a ticket at our helpdesk.

Best regards,
-- Peter

[bkonia]

The timeout property of the HLB appliance controls expiration of idle sessions. From its datasheet: "Optional timeout (in seconds) to expire inactive sessions. If this is set to a non-zero value, a session that has been inactive for more than the value of this property in seconds is 'forgotten'. Further requests bearing the 'forgotten' cookie are treated as if they have no cookie at all and are directed to to a random server, using the usual round-robin method.". That explains why it doesn't affect the request completion time.


Well, like I said, that $timeout variable doesn't appear anywhere in the pcfg.tmpl file, so it would seem that this property would have no effect on anything at this point. Also, according to the pound man page:

TimeOut value - How long should Pound wait for a response from the back-end (in seconds). Default: 15 seconds. This value can be overridden for specific back-ends.

The fact that my requests are timing out at exactly 15 seconds leads me to believe that this is the setting that needs to be changed.


Also, we have a prototype available of our next load balancer, HALB, which is based on HAProxy. Early tests showed that is about twice as fast, and does not exhibit some of the problems we've seen with pound in HLB. If you want to join the test program, please open a ticket at our helpdesk.


OK, I just opened a support ticket to request the new load balancer.

[PeterNic]

bkonia,

To set the request timeout value, you can branch the HLB appliance and perform the following change to the script, as described by one of our engineers below. Once you get it to work, you can move the HLB into the user or proto catalogs.

===
in the /usr/lib/pound/pcfg.pl file, there's this string constant, change it as needed
(the timeout line, specifically):


$be_tmplt = '"
BackEnd
Address $out
Port 80
Timeout 15
#HAport
End
"';

===

Best regards,
-- Peter