Running load tests from within 3Tera?

[donorschoose]

Hi, we're wanting to do some simple ab-based load tests, i.e., stress test our server via ab from a server outside of our application. We do have a way of stress testing the server from the outside, but we want to run some informal ab tests so as to try different configurations to improve server performance. One option is to rent some space on, say, Amazon EC2, to run ab from, but another might be just to use a different appliance in 3Tera. However, I'm a bit unclear on how resources are allocated within 3Tera. If we wanted to run a load test from a VM on 3Tera we obviously wouldn't want that load test to be run on a machine that shared physical CPU and memory, etc., with one of the VMs in our application. Is there a way to provision a VM or VMs that are guaranteed to be physically on a separate machines from our main app, so we can get a reasonably clean load test?

[PeterNic]

Donorschoose,

Yes -- there are several ways to do it.

1. First and foremost, depending on the load you intend to put and the number of load generating appliances, the normal separation between appliances on the grid should be enough to give you sufficient test capability, without having to account for possible interference between appliances on the same server.

2. If you want to be sure that there is no interference, there are three ways to run the load test on AppLogic:

2.1. The simplest and fastest: on the same grid, on a dedicated server.

On most grids, the last server (highest number) is not loaded since it is kept for N+1 redundancy (and the scheduler generally fills up the lower-numbered servers first). You can check the appliance-to-server mapping with the "server list --map" shell command and see that there are no appliances on the last server.

For the test itself, you can pin the test appliance to the last server (using the "server override" attribute of the appliance instance). If you go this route, please check that there are no active volumes in use on that server as well ("volume list server=srvN", where srvN is the last server -- e.g., srv4 on a 4-server grid; if any volumes show as "in-use", then you may want to move them out first -- really only if they are heavily loaded, like the database or the content volumes).

(The reason why it may be OK to do the test using the hot standby server is that if you are doing the test interactively, you will be there and can quickly stop the test to free up resources in case another server picks exactly that time to fail (in my experience, servers most often fail at 3am, never while you are looking at them :))).

2.2. You can temporarily order a separate small grid, ideally in the same DC/backbone, and run the test load from it. This makes more sense if you need a lot of load generators.

2.3. If you want to embed the load test appliance in the application being tested (so it is easy to bring up and run tests periodically), you can use the "failover group" method of asking AppLogic to never place the load generator and the input gateway of your application on the same physical server. I don't think it will give you much different results from #1 above, but it will achieve the separation you are looking for.

Note: if you use an external load test (outside of the same grid/backbone), please be aware of the traffic amount to avoid (or be prepared for) any traffic overages. Grids have sufficient included traffic so most grids never go above it -- but doing long heavy tests may exceed the included limits. You can also pre-pay for a larger traffic (at a lower price than the overage).

3. A good approach is to first run the load test on the same grid until you are happy you have resolved most of the problems (if any). This also gives you the fastest connection -- essentially a gigabit link that is always only one hop away from your app. Once you no longer see problems at your target loads, you can try an external test to verify the whole chain.

Also, we will be interested to help you with the tests -- please contact your account rep or me directly (PM me). It may also be interesting to publish the results once you are done -- I am guessing they will be of interest to others.

Let me know if the above helps and which method you think will work best for you.

Best regards,
-- Peter