Independent component reconfiguration

[tmart]

It would be very helpful and I believe would also increase overall application availability to allow component/class instances to be able to be changed within the GUI (and saved), component stopped, component reconfigured (perhaps verified too) and component restarted -- all at the component level without requiring the whole application to be cycled.

... or am I thinking about this whole thing in the wrong way? We wouldn't want a minor change that could be performed and verified while the app was live to require an app outage.

Thanks again for your feedback on this topic.

[PeterNic]

tmart,

This is a cool idea and has been discussed many times. We even have a design of exactly how it would be implemented (think of it as an incremental build/start). This is on the roadmap.

You also pointed out the main drawback -- what if the incremental start fails? It may actually not be that bad: you may end up having to restart the whole app... which is exactly what you need to do today. The incremental build will still do all the checks the full build does -- so it will not even try to affect the running application if the changes are not up to snuff.

Regards,
-- Peter

[Dmitry@Rivermine]

I'd be also interested in this functionality

[Karl]

I was only asking about this today, as it makes no sense to restart a whole app for a change in a web server that's load balanced for example.

[Chris]

Would definitely be nice to see this feature sometime soon.

[digerata]

I'd also like to express how cool this would be. Now that we have some real applications running in our grid, we are seeing that it takes 15 - 30 minutes to restart an app. Of course, we accept this because of how B.A. AppLogic is. This feature would just make it that much more B.A. :)

[PeterNic]

Gee, I wonder if anyone needs this feature... :)

Guys, thanks for the feedback -- and the B.A. comment. We do get how important it is to have it. I would want to focus first on eliminating the controller SPOF, so that both you and we can sleep better at night. The dynamic changes will likely follow closely.

Here's one approach to how they might work:

1. We will add a new command, "app update" (or add an --incremental option to restart).

2. When you modify the app and save it, if the app is running, you use "app update" instead of "app restart".

3. When you do that, AppLogic performs a full build, including all the design checks as it does today. Then it creates a "diff" file between what's currently running and the new set. This will involve stopping appliances that have been put on standby or removed; starting appliances that have been added.

4. Appliances whose configuration has changes -- be it by changing the amount of resources (e.g., adding more RAM), or property/connection changes -- will be restarted so that the new values take effect. Appliances that were not affected at all will be left alone.

5. The "app update" completes when all this is carried out.

6. Option: for appliances that agree to listen for configuration changes live (via the VMA agent and hook scripts), in many cases we may be able to avoid restarting the appliance. Instead, we'll just feed it an event that its configuration has changed, and the appliance will have the option of accepting the change live, or refusing it, in which case it will be restarted. This way, certain appliances, like load balancers, web servers, etc., which can accept live reconfig, will experience minimal disruption (if any).

7. Another nice option may be to show the appliance state in the editor (likely a special "view" mode), where you can see which appliance is in what state (stopped/starting/running/stopping/restarting) -- similar to what we already do for apps on the dashboard. With this, you can edit the app, hit "update" and see the changes take effect on the visual display.

BTW, this feature is also referred to as "incremental build" (change request SCR591).

Any comments and suggestions will be appreciated.

-- Peter

[Karl]

Hi Peter, all sounds good.

I'd like to see #6, as our own custom appliances have a script they run at boot (or any other time) which sets configs etc. and restarts daemons with the new configs, so that'd be absolutely perfect for us if we could just right click and say, "Update config".

7 would be good as well, just a little icon at the top right of an appliance, showing if it is stopped (in standy) or playing (running).

[digerata]

#6 is even better than I imagined it would be. :) I think you guys are right on with this this feature... and in handling the controller SPOF first.

Thanks for the update!

-Mike

[tmart]

Peter,
Thank you for taking this on. Your suggestions are a very good start.

But honestly, I have some fear about your #3, #4 and #6 as follows: if the resulting reconfig produces a broken configuration, we wind up with a running app and no way to get back to a good state if for some reason the app must go down now for a restart. This is a general issue that I have with AppLogic... I can save a change (or many changes) in the GUI and go away for 2 months before the effects of a typo might raise its head - at the moment that I actually try to reconfig or restart the app. I will always remember the days of Evi Nemmeth's UNIX Admin 101: always force a reboot after a change that is critical to the bootup process. Otherwise, by the time that you do get a chance/forced to reboot, if something went wrong, your mind will have lost the context for what you were working on.

What I would like to see is the ability to produce a sort of overlay to the running (good) configuration that would allow new components to be added or updated and will get integrated into the next full reconfiguration at the user's request. For simplicity there probably should not be more than one "layer" of overlay, but it might be possible to merge all deltas into a single overlay that remains pending integration.

Using the overlay, we could retain the last good configuration that was used to get the app running. Then we can make some changes that can be made to that running config (using your suggestions above). Then, if something goes wrong or the app/grid crashes, we have the option of starting the app back up with either the good config or the good config + overlay.

I would make one additional request, if this is easy to do... please filter nonfunctional changes from the "diff" so that comments, etc. do not trigger a rebuild of that component.

-- T

[PeterNic]

Tim, all good stuff! See my notes below


I will always remember the days of Evi Nemmeth's UNIX Admin 101: always force a reboot after a change that is critical to the bootup process. Otherwise, by the time that you do get a chance/forced to reboot, if something went wrong, your mind will have lost the context for what you were working on.

Always a good advice (even if not always practical to follow). Even more true for Windows (which reboots all by itself, so that's less of a concern:)


But honestly, I have some fear about your #3, #4 and #6 as follows: if the resulting reconfig produces a broken configuration, we wind up with a running app and no way to get back to a good state if for some reason the app must go down now for a restart. This is a general issue that I have with AppLogic... I can save a change (or many changes) in the GUI and go away for 2 months before the effects of a typo might raise its head - at the moment that I actually try to reconfig or restart the app.

Agree. That would become a bigger problem with the proposed dynamic support if not addressed.


What I would like to see is the ability to produce a sort of overlay to the running (good) configuration that would allow new components to be added or updated and will get integrated into the next full reconfiguration at the user's request. For simplicity there probably should not be more than one "layer" of overlay, but it might be possible to merge all deltas into a single overlay that remains pending integration.

Using the overlay, we could retain the last good configuration that was used to get the app running. Then we can make some changes that can be made to that running config (using your suggestions above). Then, if something goes wrong or the app/grid crashes, we have the option of starting the app back up with either the good config or the good config + overlay.

I am sure that the only way to get this to work predictably is to save the "last known good" (thanks, Windows!) as of end-of-start. No matter what you do to the application, you should be able to restore the app design back to that version -- to what is running. (that obviously excludes some destructive volume ops, but I think this would be OK, we'll look into it).

The idea of being able to choose the last-known-good as a view (as opposed to the current view) is great -- thank you for it. A visual diff in the editor will be an interesting challenge for our UI/usability folks I am sure they'll enjoy (we haven't given them anything wild since the original editor).


I would make one additional request, if this is easy to do... please filter nonfunctional changes from the "diff" so that comments, etc. do not trigger a rebuild of that component.

OK. Such changes today don't trigger a rebuild and that will remain. The rebuild/incremental update/restart will be based on the compiled version, not the text. Diffs will be tracked on the source form, so you can see changes in comments.

Regards,
-- Peter

[Karl]

Presumably SVN/CVS would be an idea for all config files generated within AppLogic - not only from a rollback perspective, but also from a debug perspective - if something breaks, it would make it much easier for ourselves and AppLogic support to see what had changed and when.