Application Provisioning - Failed to copy filesystem

[sapoblue]

Hi there,

I'm trying to provision an app on our development grid following the instructions found here:
http://doc.3tera.com/AppLogic24/RefAppsVdsCentos51.html

When running the app provision command as described in the above document I get the following "Failed to copy filesystem" error:

EFXDevGrid1> app provision VDS_CentOS51_r5 TestCentOS user=myuser user_pw=****** root_pw=******** primary_ip=172.xx.xxx.120 netmask=255.255.255.128 gateway=172.xx.xxx.1 hostname=mydomain.com.au dns1=172.xx.xxx.110 dns2=172.xx.xxx.1 cpu.dflt=0.50 mem.dflt=1G VDS_CENTOS51.boot.size=10G
Provisioning application TestCentOS...
Copying descriptors for application 'TestCentOS'...
Copying volume VDS_CENTOS51.boot ...
Failed to copy filesystem - Failed to start filer application 'Sys_Filer_TestCentOS-class-VDS_CENTOS51-boot'.

Failed to copy volume: VDS_CENTOS51.boot

EFXDevGrid1>Where do I go from here?
Are there any log files apart from the /var/log/ messages applogic & 3tshell to look at? (those don't give any detailed debug info btw).

Any help is much appreciated.

Edwin Groenescheij - Principal Consultant
Sapo Blue Pty Ltd (http://sapoblue.com.au/)

[PeterNic]

sapoblue,

The first step would be to figure out why the filer failed to start -- is there anything interesting in the "log list" output? Are there enough free resources to run the filer?

(The filer is a system application that performs the volume's filesystem copy -- so it needs some amount of resources to run; if you have the 0.5 CPU and 1G RAM that you are provisioning the app for, it should be plenty for the filer.)

The second step, just as a test, would be to provision the application without resizing the volume (removing the ...size=10G parameter) to see the provisioning succeed and then resize the volume. (You can also provision without starting)

Beyond this, I suggest contacting our support.

Best Regards,
-- Peter

[sapoblue]

Hi Peter,

Thanks for your quick reply.

The log list command shows:

EFXDevGrid1> log list
Oct 14 06:46:26 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.05.18.zip.
Oct 14 06:46:28 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.04.11.zip.
Oct 14 06:46:29 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.04.24.zip.
Oct 14 06:46:29 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.03.29.zip.
Oct 14 06:46:30 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.05.07.zip.
Oct 14 06:46:30 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.06.01.zip.
Oct 14 06:46:30 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.10.08.zip.
Oct 14 06:46:31 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.05.19.zip.
Oct 14 06:46:33 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.04.12.zip.
Oct 14 06:46:34 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.04.25.zip.
Oct 14 06:46:34 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.03.30.zip.
Oct 14 06:46:34 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.05.08.zip.
Oct 14 06:46:35 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.10.09.zip.
Oct 14 06:46:35 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.05.20.zip.
Oct 14 06:46:35 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.06.02.zip.
Oct 14 06:46:37 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.04.13.zip.
Oct 14 06:46:39 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.04.26.zip.
Oct 14 06:46:39 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.03.31.zip.
Oct 14 06:46:39 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.05.09.zip.
Oct 14 06:46:39 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.10.10.zip.
Oct 14 06:46:40 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.06.03.zip.
Oct 14 06:46:40 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.05.21.zip.
Oct 14 06:46:41 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.04.14.zip.
Oct 14 06:46:44 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.05.10.zip.
Oct 14 06:46:44 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.04.27.zip.
Oct 14 06:46:44 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.10.11.zip.
Oct 14 06:46:44 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.04.01.zip.
Oct 14 06:46:45 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.06.04.zip.
Oct 14 06:46:45 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.05.22.zip.
Oct 14 06:46:46 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.04.15.zip.I believe this is a tail of /var/log/messages , correct? There are lot's more of these messages in the /var/log/messages file.

When I provision the app without resizing the volume there is no error upon copying the volume, however the application fails to start:

EFXDevGrid1> app provision VDS_CentOS51_r5 TestCentOS user=myuser user_pw=****** root_pw=******** primary_ip=172.xx.xxx.120 netmask=255.255.255.128 gateway=172.xx.xxx.1 hostname=mydomain.com.au dns1=172.xx.xxx.110 dns2=172.xx.xxx.1 cpu.dflt=0.50 mem.dflt=1G
Provisioning application TestCentOS...
Copying descriptors for application 'TestCentOS'...
Copying volume VDS_CENTOS51.boot ...Done
Configuring application TestCentOS...
Modifying package descriptor for application 'TestCentOS'...
Building application...
Configuring application...Done

Loading application...
Scheduling application...

Starting application TestCentOS
main.vds_centos51 failed to start; cleaning up...

Application TestCentOS failed to start; cleaning up...

Application TestCentOS stopped

Freeing application resources...
Unloading application...
The provisioned application 'TestCentOS' is left created.

EFXDevGrid1> Trying to start the app with debug on and related log messages:

EFXDevGrid1> app start --debug TestCentOS
Building application...
Configuring application...Done

Loading application...
Scheduling application...

Starting application TestCentOS
main.vds_centos51 failed to start

Application TestCentOS failed to start: remains in error state


EFXDevGrid1> log list
Oct 14 08:11:27 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.04.23.zip.
====CUT ====
Oct 14 08:11:35 EFXDevGrid1 applogic: [233:error s:81]: srv1: VM 'vm.srv1.TestCentOS.main.vds_centos51' failed to boot within boot timeout period.
Oct 14 08:11:35 EFXDevGrid1 applogic: [66572:error s:0]: TestCentOS:main.vds_centos51: failed to start component
====CUT ====
Oct 14 08:11:45 EFXDevGrid1 BMC: Metering report failure - Failed to transfer log file for date 2009.04.15.zip.

EFXDevGrid1>

Edwin Groenescheij - Principal Consultant
Sapo Blue Pty Ltd (http://sapoblue.com.au/)

[sapoblue]

Also when I do an app info, the CPU & memory details are different to the configured settings when provisioning the app:

EFXDevGrid1> app info TestCentOS --verbose
Name : TestCentOS
Description : Virtual Dedicated Server - Based on CentOS 5.1 (v1.0.4-1)
ID : 21
Documentation URL : http://doc.3tera.net/AppLogic24/RefAppsVdsCentos51.html
Template : no
Locked : no
User1 :
User2 :
Destroy on Stop : no
State : stopped
CPU : 0.25
Memory : 256.00 MB
Bandwidth : 250 Mbps
Default Login Appliance :

EFXDevGrid1>

Edwin Groenescheij - Principal Consultant
Sapo Blue Pty Ltd (http://sapoblue.com.au/)

[PeterNic]

sapoblue,

Someone will be in touch with you.

Please heed the alert on your dashboard regarding metering data -- from the log it appears your grid is not properly connected to the internet. I will send you a PM on this issue.

Best regards,
-- Peter

[PeterNic]

sapoblue,

Can you provide the output of 'grid info' (more specifically, the full version string)?

I am guessing that you may not have a particular hotfix that resolves a rare problem that occurs in AppLogic 2.4.8 when metering data has not been sent for a prolonged period (http://wiki.3tera.net/bin/view/AppLogic24/HotFixhf2794). If this is the problem, you will also experience slow performance, which is likely the reason your appliance failed to start within the timeout period.

If that is the problem, the correct solution is to apply the hotfix and/or resolve the issue with the metering data send. Let me know if you're currently running any production apps on that grid prior to fixing the metering data issue. The next best approach is to try to start the app with the --debug option -- it will still take longer than usual but it will most likely start.

I'll have someone contact you to help clear up the metering data issue -- and your grid should return to normal operation.


Best regards,
-- Peter

[sapoblue]

Hi Peter,

The output of grid info is:
EFXDevGrid1> grid info
Grid Name : EFXDevGrid1
Grid Description : EFX Dev Grid 1
AppLogic Version : 2.4.8
System Uptime : 4 days, 2 hours
HA State : ok
Total Servers : 2
Servers Running : 2
Servers Enabled : 2
Applications Running : 0
Total CPUs : 16.00
Free CPUs : 15.90
CPU Load : 0.00
Allocated CPUs : 0.00
Total Memory : 31.99 GB
Reserved Memory : 1.97 GB
Free Memory : 30.03 GB
Allocated Memory : 0.00 MB
Total Bandwidth : 4000.00 Mbps
Free Bandwidth : 4000.00 Mbps
Allocated Bandwidth : 0.00 Mbps
Total Disks : 2
Total Storage : 261.58 GB
Free Storage : 97.46 GB

EFXDevGrid1>I've started the app with the --debug option with the same negative results.

We're not running any production apps at the moment, so I'm quite happy to apply the hot fix or upgrade to the latest version.

Thanks again for your help :)

Edwin Groenescheij - Principal Consultant
Sapo Blue Pty Ltd (http://sapoblue.com.au/)

[Jsmart]

Hello,

I think here you may be missing either the "filer applications" or the "filer catalog"

Please send us an email to helpdesk@3tera.com ( this will also open a ticket in my support portal for tracking) with your direct contact info. We can setup a interactive session and look at both of your issues at the same time.

Regards,

Jessie