AppLogic - TGL

[khfinity]

I'm considering a VPS from TGL. Since their sales is far too slow, I'll seek some answers here.

1. I understood that upgrading AppLogic, requires taking the whole grid down. How often and long is this?

2. In case of a node failure, the automatic failover takes from 5-10 minutes. I recall that you were aiming at lowering this down to 30 seconds. Where is this now?

3. AppLogic mirrors data on 2 nodes. Are the physical locations chosen with some redundancy in mind? Meaning 2 nodes on different power strips, switches...etc.

[EricT]

Hello,

1. Upgrading your grid to a new AppLogic release typically takes about 1 hour. A new AppLogic release comes out about once every few months.
2. That is correct, right now the recovery time upon a node failure is about 5-10 minutes. We are aiming at lowering this downtime in a future release; we are currently tying to see where this fits on the AppLogic roadmap.
3. No, AppLogic cannot take power strips or any other hardware/infrastructure into account while mirroring volumes (at least not yet). AppLogic relies that all of the servers in the grid are similar (i.e., each server has the same amount of CPU, memory, etc).

Let us know if you need anything else.

- Eric

[khfinity]

Thank you for your prompt and thorough answers Eric.

1. Maybe with a private grid this is tolerable, where you can choose the time to upgrade. But in a shared environment, I think this cancels HA. For the end user, taking an entire grid down for 1 hour every couple of months, is similar to having scheduled hardware failures. What options are available to minimize this?

3. I read that in your own setup, you have servers in different DCs, and different parts of the country. Is that doable with a single grid? Where would the AppLogic OS be installed in that case?

[EricT]

AppLogic upgrades - correct, for VPSes, this is not tolerable. This is dealt with in TGL by first installing a new grid with the latest version of AppLogic. Afterwards, they migrate VPSes over to the new grid one by one (let me find out how long this process takes, but I believe this is much shorter than the 1 hour upgrade time). This migration is scheduled at the user's request.

Any single AppLogic grid must consist of servers that are located within the same DC (i.e., any single grid cannot be made up of servers that are contained in different DCs). We have many AppLogic grids running in different DCs in different countries.

Does this make sense?

- Eric

[khfinity]

AppLogic upgrades - correct, for VPSes, this is not tolerable. This is dealt with in TGL by first installing a new grid with the latest version of AppLogic. Afterwards, they migrate VPSes over to the new grid one by one (let me find out how long this process takes, but I believe this is much shorter than the 1 hour upgrade time). This migration is scheduled at the user's request.

Great news! I didn't know that (how would I?). LT does not mention that, though IMO it's a critical piece of information.

Awaiting the time estimates.

Any single AppLogic grid must consist of servers that are located within the same DC (i.e., any single grid cannot be made up of servers that are contained in different DCs). We have many AppLogic grids running in different DCs in different countries.

What got me asking is a post by Vlad (http://support.3tera.net/archive/index.php/t-52.html). Where he says:

...For example, on an average day, at 3Tera we are running around 200 servers in 4-5 datacenters spread around North America and Europe. They are usually configured into half a dozen to a dozen grids that are used for things like development, testing, tuning large-scale installations, hardware certifications, evaluations, even our own internal infrastructure (we eat our own dog food). All of this is operated by a single guy - our support manager - on a part-time basis.

I was under the impression they were all under the same AppLogic roof.

Does this make sense?

Perfect sense!

[EricT]

ah OK. Our 200 servers are partitioned into many different grids where each grid is running in a specific data center. Most of them are test and demo grids we use every day.

[khfinity]

It's sure refreshing getting some answers. The guys at LT need to work on their response time, and educate their sales better. I'm sure this is not the first time you get this, but I hope you convey it to LT. As it affects your Matrix (we're heading there) OS, and the whole process of adopting AppLogic by mainstream.

Now to round 2 :)

4. When I purchase a 4 nodes private grid, do I get 4 dedicated physical servers? Or 4 servers "worth" of CPU, RAM, and storage?

5. What reliable providers offer AppLogic grid hosting?

[EricT]

Thank you for letting us know about your expierence with LT, we will talk to them about this. Any time you have a question, feel free to post it here on our forum.

When you purchase a private grid (known as a "dedicated grid"), it is *not* shared with any other users - you are the sole owner of the grid. One important point about dedicated grids is that you do not get direct access to the physical servers; there is no need for that. AppLogic aggregates the resources of all of the servers in your grid: CPU, memory and disk. As you build and start AppLogic applications on your grid, AppLogic takes care of the resource scheduling for you.

You still have some control over the physical servers in your grid. For example, you can tell AppLogic to reboot a specific server using the "srv reboot" command. You can also enable/disable servers in your grid using the "srv enable" and "srv disable" commands. When a server is disabled for example, AppLogic will not use resources on that server. You can also pin AppLogic appliances to run on specific servers if desired.

We have several partners who offer AppLogic grid hosting (in addition to 3tera), there are:
- Layered (LT)
- 1-800-Hosting
- Fortress ITX

- Eric

[khfinity]

This is dealt with in TGL by first installing a new grid with the latest version of AppLogic. Afterwards, they migrate VPSes over to the new grid one by one (let me find out how long this process takes, but I believe this is much shorter than the 1 hour upgrade time). This migration is scheduled at the user's request.

I specifically asked LT about this, they said that they upgrade public grid at THEIR schedule. When I mentioned your quote above, they said they'll check with the TGL guys, and get back to me.

I'll keep you posted.

[khfinity]

snip

We have several partners who offer AppLogic grid hosting (in addition to 3tera), there are:

snip

- Eric

Where can I find more on 3Tera's hosting offerings?

[EricT]

We have links to our partners on our web site: http://www.3tera.com/partners.html

Also, regarding your question as to VPS downtime: when TGL migrates a VPS to a new grid, it actually can cause several hours of downtime depending on the size of the volumes in your VPS (worse than doing an in-place AppLogic upgrade). They do it this way because since the VPSes run on shared grids, if they upgrade the grid itself all VPSes will experience downtime. To avoid this, they migrate the VPSes one by one at times that are scheduled by the users. One other point to keep in mind is that in the 1+ years that TGL has been operational, this has only occured one time (and it is completely up to the user, they will only migrate your VPS if you tell them to do it, if your VPS is running fine w/o any problems, you can just leave it alone). Most of the time whenever a new AppLogic release comes out, it is not critical that the shared grids are upgraded right away, espcially for shared grids that are only running VPSes.

- Eric

[khfinity]

We have links to our partners on our web site: http://www.3tera.com/partners.html


I meant when hosting with 3Tera directly; not your partners.

Also, regarding your question as to VPS downtime: when TGL migrates a VPS to a new grid, it actually can cause several hours of downtime depending on the size of the volumes in your VPS (worse than doing an in-place AppLogic upgrade). They do it this way because since the VPSes run on shared grids, if they upgrade the grid itself all VPSes will experience downtime. To avoid this, they migrate the VPSes one by one at times that are scheduled by the users. One other point to keep in mind is that in the 1+ years that TGL has been operational, this has only occured one time (and it is completely up to the user, they will only migrate your VPS if you tell them to do it, if your VPS is running fine w/o any problems, you can just leave it alone). Most of the time whenever a new AppLogic release comes out, it is not critical that the shared grids are upgraded right away, espcially for shared grids that are only running VPSes.

Well that's what I hope, and what I understood from you. It's just that their sales said otherwise, at least for now. I'm still waiting their update. (as I mentioned in this post (http://support.3tera.net/showpost.php?p=591&postcount=9))

[EricT]

Hmm...let's see what LT says about this. If you are concerned with downtime, I am sure they will accommodate you.

For 3tera hosting, see http://www.3tera.com/utility-computing-plans.html
Also, you may contact Stacey Kane at (949)-468-7121 for more information (or just shoot her an email).

- Eric

[khfinity]

Hmm...let's see what LT says about this. If you are concerned with downtime, I am sure they will accommodate you.


Again I repeated what you told me, and they'll get back to me.


For 3tera hosting, see http://www.3tera.com/utility-computing-plans.html
Also, you may contact Stacey Kane at (949)-468-7121 for more information (or just shoot her an email).

I'll definitely check out those. Do you run your own Data Center?

[EricT]

I did not realize that they did that with the VPSes...my understanding is that they scehduled this downtime with the user. I will talk to our support team and find exactly how this is suppose to work.

Nope, we don't run our own data center. We use servers from other data centers for our directly hosted AppLogic grids.

- Eric

[PeterNic]

I specifically asked LT about this, they said that they upgrade public grid at THEIR schedule.

khfinity,

I am sure that TGL needs to be able to schedule upgrades and critical hotfixes, which is why that's probably the only reply they could give you. VPSes are by nature running on a shared system, so if maintenance is required it is practically impossible to agree on a maintenance schedule with all customers.

At the same time, they have been extremely protective of the VPS customers' uptime, and have followed the procedure Eric described for most VPSes I know of. Ultimately, of course, whatever TGL tells you is what they do.

Price permitting, you can also get a VPD (virtual private datacenter). Each VPD runs on its own full dedicated physical servers and so you would be in control of the timing for maintenance and upgrades (yes, if all you need is a small VPS, then such dedicated VPD is probably beyond what you would be willing to pay). FYI, the recent AppLogic releases were in Feb '07, Sep '07, Dec '07 - so we provide major releases every 3-6 months. You are not required to upgrade but as each new version provides improved stability, in addition to new features, you will probably want to.

Regards,
-- Peter

[PeterNic]

For questions on TGL's offerings and procedures, as well as to discuss with other TGL users their experience, you can also use TGL's forums at http://layer0.layeredtech.com/ (scroll down to the 3 TGL forums section).

Best regards,
-- Peter

[khfinity]

khfinity,

I am sure that TGL needs to be able to schedule upgrades and critical hotfixes, which is why that's probably the only reply they could give you. VPSes are by nature running on a shared system, so if maintenance is required it is practically impossible to agree on a maintenance schedule with all customers.

Understood. But the procedure described above by Eric, overcomes this.

At the same time, they have been extremely protective of the VPS customers' uptime, and have followed the procedure Eric described for most VPSes I know of. Ultimately, of course, whatever TGL tells you is what they do.

Well, that's not what they say. Maybe with VPDs they do.


Price permitting, you can also get a VPD (virtual private datacenter). Each VPD runs on its own full dedicated physical servers and so you would be in control of the timing for maintenance and upgrades (yes, if all you need is a small VPS, then such dedicated VPD is probably beyond what you would be willing to pay). FYI, the recent AppLogic releases were in Feb '07, Sep '07, Dec '07 - so we provide major releases every 3-6 months. You are not required to upgrade but as each new version provides improved stability, in addition to new features, you will probably want to.

Price permitting, I can get a physical DC :). But one the beautiful things about Grid hosting, is that it allows you to get what you need. No more no less.

[PeterNic]

Price permitting, I can get a physical DC :).


A grid is better :)


But one the beautiful things about Grid hosting, is that it allows you to get what you need. No more no less.

Agree; utility computing is now possible and can be had at commodity prices. Based what I understand about your requirements, I think a VPS on TGL is the best solution for you.

Regards,
-- Peter

[khfinity]

For questions on TGL's offerings and procedures, as well as to discuss with other TGL users their experience, you can also use TGL's forums at http://layer0.layeredtech.com/ (scroll down to the 3 TGL forums section).

I did, I also contacted their sales. The only time you'd get some thirst quenched, is when Jeremy (LTADMIN) steps in. It seems this guy has lot on his shoulders.

Actually, as described at the beginning of the thread, I only resorted to you guys, after frustration with LT. I'm sorry if I'm bothering you, yet I feel I came to the right place.

Thank you all for your support and concern.

[Jsmart]

--Jessie@3tera

[khfinity]

A grid is better :)

From what I'm learning, yes it is.

Agree; utility computing is now possible and can be had at commodity prices. Based what I understand about your requirements, I think a VPS on TGL is the best solution for you.

That's what I think too. But this single detail cancels HA. For the end user, this is guaranteed random down time: The very thing I'm trying to avoid in the first place.

[khfinity]

you can fill in the web form on our site (http://www.3tera.com) and someone will contact you.

3tera does not offer VPS/shared type hosting; we host full production grids only.

--Jessie@3tera

I already did. And Stacey is kindly tolerating my inquiries.

Thanks Jessie.

[lewis.george]

Hello,
I am Lewis with Layered Technologies. I am the team lead for the GridLayer department. I just wanted to clarify and update a few bits of our policy, so as to clear a few things up. Things may have been different in the past, but this is the way we operate now. For shared grids, we setup the grids on the absolute most recent stable version of AppLogic, applying any and all required and recommended hotfixes and patches before we put it into production. After that point, we only apply hotfixes and patches that 3tera has released as high priority and needed. If the hotfix or patch requires a grid reboot, we will contact all customers on the affected grid letting them know of the downtime. Now, for unscheduled downtimes, such as hardware failures or network issues, we really cannot forcast that.

With the VPDC's (private grids), when the grid is ordered, we will install the most recent stable version of AppLogic, unless otherwise requested, and apply all applicable hotfixes and patches. At that time, the grid is yours. If you request a specific hotfix or patch, we will apply it when you request it. If you request an upgrade, we can perform that, in a time frame you specify. The only time we will do anything else is with critical hotfixes or patches from 3tera, in which case, if it does not require stopping any applications or rebooting the grid, we will apply it, if it does, we will inform all affected customer to schedule a downtime.

That is our policy as it stands, though in the past policies may have been different. If you have any questions, you can let us know anytime. As for our sales department, if you would like, you can try to speak with either Geoff Pennington or Karen Rhodes. They are both very knowledgeable about the GridLayer and our offerings and procedures. I know you can also request a live demo which is done via phone and screen sharing.

Lewis George
Support Engineer - Team Lead
lgeorge@layeredtechnologies.com
Layered Technologies Inc.

[khfinity]

Thank you Lewis for taking the time to clarify.

For shared grids, we setup the grids on the absolute most recent stable version of AppLogic, applying any and all required and recommended hotfixes and patches before we put it into production. After that point, we only apply hotfixes and patches that 3tera has released as high priority and needed. If the hotfix or patch requires a grid reboot, we will contact all customers on the affected grid letting them know of the downtime.

What frequency and estimated down time are we talking about?

Now, for unscheduled downtimes, such as hardware failures or network issues, we really cannot forcast that.

No one should ask for such a forecast. The only case in question was the AppLogic OS upgrade. Actually, hardware failures or network issues should be where AppLogic shines.

With the VPDC's (private grids), when the grid is ordered, we will install the most recent stable version of AppLogic, unless otherwise requested, and apply all applicable hotfixes and patches. At that time, the grid is yours. If you request a specific hotfix or patch, we will apply it when you request it. If you request an upgrade, we can perform that, in a time frame you specify. The only time we will do anything else is with critical hotfixes or patches from 3tera, in which case, if it does not require stopping any applications or rebooting the grid, we will apply it, if it does, we will inform all affected customer to schedule a downtime.

This is a clear case.

[lewis.george]

We do not have any pre-scheduled times that we apply hotfixes and patches. If 3tera releases a critical hotfix or patch, and we find that is applicable, we will apply it. Again, if it does not affect applications or the grid being up, then we will apply it immediately. If, on the other hand, it does affect uptime, we will inform all affected parties, and schedule it for a period of time that is generally better for a downtime (i.e. 2am sunday nights).

Now the estimated downtime for such a patch or hotfix should be only about 10 minutes, but can sometimes be a full 30 minutes to reboot the grid, all the nodes, and get all of the VPS accounts restarted.

We do not particularly enjoy having to apply large hotfixes, for this reason. That is why we always build new share grids with the most recent versions and apply any existing hotfixes at that time. If there is anything critical that comes up down the road, we may need to at that time.

As for the VPDC's, if the hotfix requires a restart, it is at the discretion of the customer if and when it will be applied. If it does not affect grid usage, or uptime, we will usually apply it. The GridLayer services are, after all, semi-managed.

As for the hardware or network issues, it is true the AppLogic does provide wonderful fail-over ability in those cases, though the customers do need to keep an active eye on their grid status and their volumes. If a server ever goes down, the volumes mirrored on it will become degraded. If they are not repaired, and another server goes down, it can put things into a bad situation where sometimes there is data loss, volumes 'going back in time' and other issue. As for network side though, admittedly, if it is something higher-level than that specific nodes, such as the switch or core router the grid is on, that can cause network problems overall, which AppLogic cannot help.

Have a good day.

Lewis George
Support Engineer - Team Lead
lgeorge@layeredtechnologies.com
Layered Technologies Inc.

[khfinity]

We do not have any pre-scheduled times that we apply hotfixes and patches. If 3tera releases a critical hotfix or patch, and we find that is applicable, we will apply it. Again, if it does not affect applications or the grid being up, then we will apply it immediately. If, on the other hand, it does affect uptime, we will inform all affected parties, and schedule it for a period of time that is generally better for a downtime (i.e. 2am sunday nights).

Now the estimated downtime for such a patch or hotfix should be only about 10 minutes, but can sometimes be a full 30 minutes to reboot the grid, all the nodes, and get all of the VPS accounts restarted.

We do not particularly enjoy having to apply large hotfixes, for this reason. That is why we always build new share grids with the most recent versions and apply any existing hotfixes at that time. If there is anything critical that comes up down the road, we may need to at that time.

I understand your need to perform updates in a batch practical way. You are definitely better to judge, though the procedure described earlier by Eric, sounded solid.

I'm merely describing it from the end user view (not even me as a provider). This is extended uncontrolled guaranteed down time. The very thing we want to avoid in the first place.

I might be asking for too much from a VPS, but my question to Lewis and anyone reading: Am I being too drastic?

[khfinity]

As for the hardware or network issues, it is true the AppLogic does provide wonderful fail-over ability in those cases, though the customers do need to keep an active eye on their grid status and their volumes. If a server ever goes down, the volumes mirrored on it will become degraded. If they are not repaired, and another server goes down, it can put things into a bad situation where sometimes there is data loss, volumes 'going back in time' and other issue. As for network side though, admittedly, if it is something higher-level than that specific nodes, such as the switch or core router the grid is on, that can cause network problems overall, which AppLogic cannot help.

Nothing cancels the need for off site backup. I think your DynaVol (http://www.thegridlayer.com/products/dynavol.php) is a rather beautiful solution for this.

[khfinity]

As for the VPDC's, if the hotfix requires a restart, it is at the discretion of the customer if and when it will be applied. If it does not affect grid usage, or uptime, we will usually apply it. The GridLayer services are, after all, semi-managed.

One of the numerous wonders AppLogic offers, is making HA affordable. The cost of a VPDC (around $1000) is comparable to an HA cluster with load balancing. Which is, again, something we want to avoid.

With Applogic/TGL, one could get something pretty descent for a fraction of the price.

[EricT]

Regarding AppLogic hotfixes, for as long as AppLogic has been in production -- i.e., for more than a year -- we have had only 1 critical hotfix that needed to be applied to all grids and required a grid reboot (thus causing downtime). All other mandatory hotfixes we have had could be applied on a grid w/o affecting the uptime of the grids or their running applications (no grid reboot was required for these other hotfixes).

In other words, the forced scheduled downtime that Lewis mentioned in his post has only occurred once for critical AppLogic hotfixes.