Nas-r

[Karl]

Hi,

We've been discussing this one here as an idea to develop as an in-house appliance, but it occured to us that it'd probably be useful to a lot of other people as well.

How about a NAS-R appliance? i.e. Replicated NAS. Similar to how you setup mysqlr, so that you can have anything written to a NAS appliance replicated to another NAS appliance (potentially through a gateway). We were looking to see if we could use DRBD for it.

Thanks,

[PeterNic]

Karl,

DRBD does block-level replication. Most file systems will choke up and die if you modify the volume from under them (ext3, NTFS, you name it). There are cluster file systems that are designed for this, but they are not really mainstream (e.g., GFS).

There are two approaches:
- NASR - using a combination of file system change monitoring and synchronization -- more or less like MYSQL replication but for files. That's one we're researching and prototyping.
- NAS/RSYNC - a much simpler to set up with rsync on a cron job -- not realtime sync but may be good enough for many uses

Regards,
-- Peter

[digerata]

We used GFS for essentially this purpose in our previous installation. We left it simply because of the expensive hardware and expensive support contract with Redhat. (You need fiber channel between nodes and storage.) Times have changed I'm sure, but during our use, we ran into two different bugs on two different occasions that required actual developers at Redhat to issue patches. (Not to mention hours of downtime)

If there was a true NASR appliance (must be > NFSv3) that would really catapult AppLogic into a whole new level shared by no one. That is the single bigest issue we are tangling with. If you could do that between two NASR on two grids... WOW.

[PeterNic]

Mike,


We used GFS for essentially this purpose in our previous installation.


Nothing like having first-hand experience!


If you could do that between two NASR on two grids... WOW.


When we get NASR, it will definitely work between grids (even between datacenters) -- just as we do with MYSQLR now. We're also building a VPN tunnel gateway, which will provide an out-of-the-box secure VPN channel for inter-application and inter-datacenter traffic, for including database/file system replication. (Let me know if you are interested to review the preliminary data sheet -- I'll post a message to the new appliance's forum.)

Doing this type of NAS-R, with a clustered file system, is something we will be looking into -- we know what it will take to build. Even though many engineers in our team have significant storage background (or precisely because of it), we have not yet undertaken this. We're looking to partner with someone who is a specialist in clustered file systems and does that for a living. We're also not beyond contracting out such a NAS-R appliance if we find the right person/team.

Have you looked at:

lustre (www.lustre.org) -- picked up by Sun some time ago
mogileFS (http://www.danga.com/mogilefs/) -- a non-Posix API distributed filesystem (reportedly used by Digg; may be good enough for content distribution; I won't be surprised if a FUSE-based interface shows up to provide a Posix-style API to it)
PVFS (http://www.pvfs.org/) -- I just found this one, seems like a good approach; I'll bookmark it and read more about it



(there's also a somewhat dated list of clustered filesystems at http://www.yolinux.com/TUTORIALS/LinuxClustersAndFileSystems.html)

Regards,
-- Peter

[digerata]

Thanks for the info, Peter. It is good to hear this.

I spent some time setting up an OpenVPN appliance for VPNs on the grid. However, we hit a snag. Not a single partner or client of ours has a firewall that will support SSL VPN connections. We had to switch to a low tech solution, SSH tunnels! Not optimal but got the job done. Admittedly, this is a different goal... to connect non grid users to the grid, versus the one you propose. I'm curious, is the VPN tunnel gateway you guys are working based on OpenVPN? I am definitely interested in the data sheet.

Back on the filesystem topic, I'll add a few. We have been watching Caringo: http://www.caringo.com/. Their concept is very attractive and our long term goal (before this thread) was to look at getting a Caringo system sitting next to our grid. Of course, CAS is not POSIX... The other solution (that would be POSIX) we had been looking at was ZFS and a Sun X4500 sitting next to our grid(s).

Of course, a NAS-R may change our plans :)

-Mike

[PeterNic]

Thanks for the info, Peter. It is good to hear this.

I spent some time setting up an OpenVPN appliance for VPNs on the grid. However, we hit a snag. Not a single partner or client of ours has a firewall that will support SSL VPN connections. We had to switch to a low tech solution, SSH tunnels! Not optimal but got the job done. Admittedly, this is a different goal... to connect non grid users to the grid, versus the one you propose. I'm curious, is the VPN tunnel gateway you guys are working based on OpenVPN? I am definitely interested in the data sheet.


Yes, it will be based on OpenVPN, targeting to support both shared secret and certificate mode. I will also consider supporting ssh tunnel -- that will be a really cool addition (Thanks!) -- in most of our enterprise accounts, our customers are having hard time getting ssh ports open (not unreasonably, due to ssh's tunneling capabilities), so SSL is preferred. We're revving the datasheet at the moment, will post it as soon as I have the updated version.


Back on the filesystem topic, I'll add a few. We have been watching Caringo: http://www.caringo.com/. Their concept is very attractive and our long term goal (before this thread) was to look at getting a Caringo system sitting next to our grid. Of course, CAS is not POSIX... The other solution (that would be POSIX) we had been looking at was ZFS and a Sun X4500 sitting next to our grid(s).


Have you considered using OpenSolaris with ZFS support on AppLogic 2.4?



Of course, a NAS-R may change our plans :)


You will be the first to know.;)


Best regards,
-- Peter

[Karl]

Hi,

Sorry for the delayed reply, been a bit busy and forgot about this post.

VPN - Will IPSec be making an appearance? All our VPNs tend to be IPSec and for the sake of consistancy it'd be nice to keep it that way.

Also, as you're developing a VPN appliance for G-2-G communication, will you be also making that available for Public-2-Grid connections as well? Would make sense to have a VPN Gateway to allow customers to securely connect in to management backends etc.

Thanks,

[PeterNic]

Karl,

The first VPN gateway appliance is only SSL (actually, it supports cleartext, shared/pki and ssh mode). It doesn't support IPSec, however. It can be used between grid apps as well as between grid apps and non-grid apps (both client machines and servers).

I am sure an IPSec gateway will eventually appear. Unlike SSL, IPSec is a bit harder to configure right in a way that it properly interoperates with hardware-based VPNs, so we will need to find a good IPSec VPN specialist.

Also, we have completed early proof-of-concept prototypes of NASR, and a data sheet is being prepared. I'll post it here for review as soon as it ready for this.

Regards,
-- Peter

[Karl]

Hi Peter,

Fire it over and I'll take a look :) Was 3rd on my to-do list of appliances to take a look at going something like Nas-r. Just got to finish tidying up some others and the new Django specific appliance.

Racoon isn't too bad for IPSec - not perfect but it does work. We've got a fair few m0n0walls running it and our hardware based VPNs don't have any problems with it - well, once they are up and running, the odd device can be a pain to get talking to them to start with - but we've found that between hardware devices as well.

Cheers,