Is RAID 10 or 5 possible in AppLogic?

[jonesy]

Hi all,

I'm hitting very large brick walls in terms of IO performance on database components (MYSQL5). My cpu and memory utilization is fine, but running 'iowait' on the node during times when there is decent load, or running especially hairy queries results in the cpu hitting %iowait of up to 98% -- sustained!

Is there a way to implement software RAID within an applogic component to help distribute the load across more spindles?

[PeterNic]

Brian,

Yes, absolutely you can do RAID within the appliance. You can set up spanning or striping -- or any other raid level -- from within an appliance. AppLogic's volume limit per appliance is currently 12 (leaving, let say, 8 for a RAID-type user data volume, plus boot, system (R/O), swap and an extra/logs volume).

You can use LVM or mdadm, or, in fact, anything else that you could use on a physical server. If you want to do it for performance, I would recommend RAID 0, so that with AppLogic's built-in RAID 1 you will essentially get RAID 1+0 (which, without getting into religious discussions, IMHO is the best raid configuration). In addition to better performance, it will also give you greater capacity, as you will now be able to get more servers involved.

I don't know if it will give the performance benefits in your particular usage, but it is so easy to try that it is probably easier and faster to try and measure than plan it too much. If you decide to go this way, you will need to keep track that these are N separate volumes that have to be used together; they will appear as RAW filesystem to AppLogic - although there are some easy ways to still have the ability to resize/manipulate the volumes outside of the production appliance (borrowing a page from the upcoming release of AppLogic, you can create a filer appliance that mounts one or two such volume sets - we can go in more detail if you decide to use this mechanism).

Let me know if this helps,

-- Peter

[jonesy]

I would very much like to take advantage of this capability, and would greatly appreciate more information about how to prepare the volumes needed in such a way that they will be used together, or otherwise how to manage things such that they are always used together, assuming there is some applogic-specific stuff involved. Also, by "RAW filesystems", do you mean that they will not have a filesystem? This would seem to imply that these are not typical volumes you would create in the AppLogic GUI. More details please!

[PeterNic]

Brian,

The good and the bad:

Good: you already know everything you need in order to make it work

Bad: there is no hidden AppLogic feature that will automagically make it work

In short, here are the steps:

you branch the appliance (assuming MYSQL or MYSQL5)
you edit it, adding, let say 3 more data volumes, for a total of 4 placeholder volumes (take care to assign the volume device names correctly)
you delete the existing data volume (I assume you have the data elsewhere)
create 4 brand new volumes with fs type none (raw); I suggest naming them data_stripe1, data_stripe2, etc. Use --prefill - this will take long time but it will ensure that your volumes have all their blocks preallocated and most likely sequential on the physical disk
configure the 4 volumes into the slots in the appliance
start the appliance, add lvm or mdadmin, set up RAID 0 as needed inside a regular Linux OS to see the 4 devices as a single volume; format it with mkfs (I suggest the largefiles option for total fs size >0.5TB) [choose your favorite fstype: e.g., ext3 or reiserfs]
edit fstab to add the new file system, mount it
restart the appliance; MYSQL should create the initial database


Done! (stop, rename the class and move it to the user catalog once it works well, so that you can instantiate it easily)

Limitations to be aware of:

AppLogic does not know that the 4 volumes form a single logical volume; as a result, operations like 'vol resize' and file-level copy will not work; the filesystem on the 4 volumes will be shown as "raw".
you may need to manually lay out the volumes across multiple servers; if using 4 volumes, you probably should have at least 8 physical servers (nodes), as each stripe is mirrored. You may get a limited gain on read-intensive databases from 4 nodes with 4 volumes, but to get full I/O benefits, without getting killed by the seeks. I recommend creating no more than N/2 volumes, where N is the number of physical servers in the grid.
Use 'srv disable' and 'srv enable' to lay out the logical volumes across the physical servers in your desired fashion (disable all but srv1/srv2, create the first volume; disable all but srv3/4, create the second, etc. -- there is a chance AppLogic will do this automatically, but if you are fine-tuning the performance, use the manual allocation)


If this approach gets more traction, we may consider supporting volume pools (i.e., a volume that consists of multiple volumes) -- or provide volume spanning and/or raid10 in the base OS, so that your appliances remain pure.

The easiest way to manipulate the new composite volume's file system is from the appliance itself. Alternatively, you can create a simple filer appliance -- e.g., by branching LUX and doing the same lvm/mdraid instrumentation -- so that you can manage the volumes from there. The best approach is to leave two multi-volume slots -- src1,src2,src3,src4 and dst1,dst2,dst3,dst4 -- this will allow you to use the filer appliance to copy files between such composite volumes (and, as a result, be able to resize by creating a new volume set and copying the files from the old). For some ideas, see http://doc.3tera.net/AppLogic23/CatFilerLinux.html -- you don't really need anything nearly as functional -- just the src/dst slots and ssh access. Think of it as a physical server with two arrays of removable disks.


Let me know if any questions and please do post your results here. You can PM me or Jessie if you want to set up a webex and work out the details.

Regards,
-- Peter

[enovikoff]

The other alternative to getting more bandwidth to storage is finding a way to use less of it :) We often find that our customers have configured either their applications or their databases to execute more queries than they actually need, or run inefficient queries. Even if you double your I/O performance using the technique above, it won't solve issues where you're using 10 or 100 times as much bandwidth as you may actually need to run your app. My experience is that MySQL performance is also very susceptible to adjustments in its parameters. This is why we tend to use PostGreSQL whenever possible, which requires less configuration to run efficiently.

Another possibility which we have succeeded with is to supplement in-grid storage with an external NAS. By assigning your database to a unique server (separate from your web server) you can get the full frontside network bandwidth available for file operations.