appliances can't talk through newly added terminals

[antoro]

Hi,

We have an application where we house multiple GSC's that need to
talk to each other. However, for whatever reason, we can't communicate
with them on the private IP address / backplane (though we can login individually
from the public IP.

My understanding is that this is a known issue and has something to do
with a default ARP configuration in Linux or the routing tables.

Can you please tell us what we need to do to make sure that GSC's
are able to communicate appropriately on the Grid, and any other
best practices for this sort of setup?

Thanks,


Wisesa

[PeterNic]

Wisesa,

Currently AppLogic does not have provisions to allow different applications to communicate over the private network; the latter is reserved as an interconnect. This is something we will add as a capability in an upcoming release.

Here are two options:
- you can put the GSCs that need to talk to each other in the same application and have them talk to each other through terminals
- contact your hosting provider and see if they can arrange for an unmetered GigE front-side connection

If none of these work for you, please PM me and we can discuss what other options may be available.

Regards,
-- Peter

[PeterNic]

We have an application where we house multiple GSC's that need to
talk to each other. However, for whatever reason, we can't communicate
with them on the private IP address / backplane

My understanding is that this is a known issue and has something to do
with a default ARP configuration in Linux or the routing tables.


Sorry, I didn't read your post correctly the first time. Appliances of the same application should always be able to communicate using the private network.

Yes, the problem you are seeing is a known issue in GSC and in appliances that are created from scratch (e.g., different Linux distro).

By default, Linux networking will reply to ARP request arriving at a network interface, even though the request is for an IP address that the host has on a different interface. This may be useful for multi-homed hosts that have different interfaces on different networks but it wreaks havoc when the host has multiple interfaces on the same network and needs to distinguish between them.

To cut the long story short, this is easily fixed by adding a line in the /etc/sysctl.conf file of the appliance:

net.ipv4.conf.all.arp_ignore = 1

This turns off the default ARP behavior and lets interfaces reply only when spoken to. Here is how a typical sysctl.conf file in an appliance should look like (the new lines in bold):


# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1


#ignore arp
net.ipv4.conf.all.arp_ignore = 1



Once you put this inside, the appliance will start working and the phantom networking problems will disappear.

Since this problem, until fixed, may prevent login via the AppLogic shell, it may be hard to actually make the change (except in GSC, where you can login from the external interface). A possible approach is to mark all other appliances of the application as standby and start the app with --debug option; this way, the only appliance that will start will be the one you want to fix and you will be able to ssh into it.

This fix is needed when you add a first terminal to an appliance that didn't have any terminals. Since the effect is intermittent, you may not see it right away. All catalog appliances have this fixed, so if you branch an appliance from the catalog, you don't need to apply this fix.


Best regards,

-- Peter

[antoro]

Hi Peter..

Sorry I wasnt reply you promply.. I just have a chance to reply now..

About the problem : one application has been changed.. but there is another problem.. I have 4 component on one application and the real problem is I cant get inside the component since I forgot the password of the 3 other components.. is there any way to change the root password.. coz to implement your solution we need to get inside and since ssh command at TGL doesnt respond correctly the only way is to get inside from outside the net using ssh.. and this where I forgot the password.. and my question.. again :) is there any way to restore or reset the password so I can get inside ?

Regards,


Wisesa

[Jsmart]

Hello,

1st.. if you are on a private grid you can always ssh into the appliance from your grid controller without a password.

2nd.. if you are not on a private grid your provider can do this for you and change the password to allow you to have access..

if this does not answer your question please let me know and ill give it another shot :)

--Jessie

[antoro]

Hello,

1st.. if you are on a private grid you can always ssh into the appliance from your grid controller without a password.

2nd.. if you are not on a private grid your provider can do this for you and change the password to allow you to have access..

if this does not answer your question please let me know and ill give it another shot :)

--Jessie

Hi Jessie,

The problem is I cant either login inside grid controller..
for example, I run this command inside TGL controller :
TGL> ssh APP-TUYU:main.WWW

usually with only one component inside the application its run perfectly.. but APP-TUYU have 4 component inside of it and its all connected each other.. and when I push enter after that line of command... nothing happen and after several minutes.. I get message :

ssh: connect to host 10.136.60.6 port 22: Connection timed out

this why I cant get inside the component from grid controller..

Regards,


Wisesa

[PeterNic]

Wisesa,

The reason you are unable to login is likely the ARP problem that this config option resolves.

Two options:

1. Put all but one appliance of the application on standby. This way, when you start the app, the very first traffic it will receive will be on the default interface -- the one that the grid controller uses to ssh into the appliance. If this works, then you can add the config option to the appliance; then stop the appliance, start another one, rinse and repeat.

If this doesn't work, use option 2, which allows access to the appliance volume in all cases (even if something else is screwed up -- e.g., sshd in the appliance is disabled or similar):

2. Direct volume access
- stop the application
- through the shell, mount the boot volume of the first appliance. If that appliance is GSC, use the following command: vol mount myapp:GSC.boot
(you can see the full set of volumes in your application, together with their exact names using the following command: vol list myapp --all)
- copy into the clipboard the path returned by the mount command
- use sftp or scp to access the config file on the volume. You can get the config file on your workstation, modify it and copy it back to the volume. Putty's scp client is 'pscp'; in Windows you can also use FileZilla GUI app (http://filezilla.sourceforge.net/). In Linux, you can use scp or sftp command line clients or Nautilus GUI for sftp.
- once you have updated the config file, unmount the volume, using the 'vol unmount' command with the volume name. For example: vol unmount myapp:GSC.boot
- repeat for each appliance that needs the change
- start the application, verify you can ssh into each appliance, and that they now can talk to each other via terminals.


For extra bonus, there is of course option 3: in the editor, remove the terminals from the appliance, save the app and start it. Since there will be one network interface on the private network now, the ARP "feature" no longer causes the problem; you can modify the config file, stop the app. Now that you have prepared the appliance for terminals, you can re-add the terminals and start the app.

BTW, in AppLogic 2.0 beta, I believe the GSC has the ARP option, so it is terminal-ready

Let me know if this helps,
-- Peter

[antoro]

Many thanks!.. :) I was using option 3 and it works.. :) I am be able to change the sysctl.conf and now all running perfectly.. :) Once again thanks..

[PeterNic]

When adding terminals to GSC-type appliances (appliances with external interface that has a publicly accessibly network interface), in addition setting the arp_ignore=1 option described above, also don't forget to open the firewall ports for the terminal's interfaces (or simply configure the firewall to restrict only the external interface)

See Jeremy's post at:
http://support.3tera.net/showthread.php?p=290#post290