ZFS or AppLogic's RAID?

[digerata]

I'm trying to get a feel for how well ZFS capabilities are utilized in AppLogic. If we were to use ZFS for its well known features like:

* Storage Pools
* Disk scrubbing
* RAID-Z

It seems like we would not want to use the RAID mirroring that AppLogic provides. Additionally, it seems that a performance and capacity penalty would be paid for if we are using ZFS with AppLogic's mirroring turned on.

Is it possible to not mirror the storage using AppLogic RAID and instead use RAID-Z? Additionally, is it possible to allow ZFS to directly take advantage of all of the drives in the grid? (Or at least, create a storage pool with drives from multiple servers in the grid?)

With my current knowledge of "grid new wave", my perfect world (for our purpose) would be a large zpool that uses, for example, 3 out of the 4 drives on each server. When we add a new server, we would simply run "zpool add" to add the drives from the new server to the pool. (The zpool would be presented as an NFSv4 share to other appliances on the grid)

How far off the mark am I?

[PeterNic]

Yeah, it is a bit off as far as the grid concept goes.

AppLogic provides a virtual volume store, allowing each application to have its own, private storage volumes -- thus, in many cases, avoiding the need and complexity of large filesystems. It also allows you to set up any filesystem you want -- ext2, ext3, reiserfs, zfs, ntfs, even fat.

That is not to say you can't do that yourself on the grid. For example, you can add more placeholder volumes to the OSOL appliance, and create pool. Not all placeholders need to be filled -- you can start with one volume and keep adding volumes; each volume may be on separate disks. You can create the volumes not mirrored - but that has some side effects, I wouldn't recommend it -- still doable, though.

I am seeing raised interest in clustered file systems -- like GFS, lustre and similar -- primarily as a way to increase storage capacity to unhinged levels; usually, performance requirements follow closely, which is easy to arrange with additional appliances.

Regards,
-- Peter

[digerata]

What are the side effects of creating the volumes not mirrored?

Can one appliance have virtual volumes from multiple servers?

However, an appliance is still limited to about 8 volumes, correct?

So really, as we add more storage (via more servers) over time, we can't easily add more than 8 new volumes to a single appliance. Instead we would have to add twice current capacity and then grow the existing volume to use up all of the new storage? (If a single volume can exist on more than one server?) From my observations, it appears that volume resizing makes a new volume at the new size and then copies the data, correct?

[PeterNic]

What are the side effects of creating the volumes not mirrored?
If the volume is unavailable (e.g., its server is down), the appliance won't start until you manually remove the volume. However, if the volume dies while the appliance is running, the appliance will continue running without it (in your case, leaving it up to ZFS to figure out what happened).

Can one appliance have virtual volumes from multiple servers?
Yes

However, an appliance is still limited to about 8 volumes, correct?
Paravirtualized appliances are limited to 12 volumes/appliance in the current releases (2.1/2.2/2.3), counting the boot volume. See the release notes for other dimensions.

So really, as we add more storage (via more servers) over time, we can't easily add more than 8 new volumes to a single appliance.
Well, more than 8 (see above), but yes, that is the case.

Instead we would have to add twice current capacity and then grow the existing volume to use up all of the new storage?
(I didn't get that)

(If a single volume can exist on more than one server?)
Other than the mirroring, currently no. If (or when) we add the capability for a single volume to span multiple servers, there still may be a reasonably low number of disks that it can span (may).

From my observations, it appears that volume resizing makes a new volume at the new size and then copies the data, correct?
Yes, that is correct. Many filesystems don't support resizing of the physical disk; as we get more info on the ones that do, we may be able to do in-place resize for them. At this time, the grid errs on the side of safety.

---

It looks to me like a clustered filesystem may achieve the result you are looking for.

---

Another alternative: while AppLogic does limit the number of locally attached disks to 12 per appliance, I expect you can use network-attached disks to make up the ZFS pool -- nbd, iscsi, etc. You will then have a number of "disk-server" appliances (with one or more volumes attached to each) that expose block-level devices; and a ZFS appliance that connects to all of the disk servers and creates a pool from the disks exposed by them. (Of course, if this runs in your own datacenter, you can use a locally-attached IP SAN, too.) Let me know if you have any questions on that.

Regards,
-- Peter

[digerata]

Okay, I think this is good for now. Thanks for your help Peter. This is all very informative. There is quite a bit to consider.