AppScale sample fails to boot on Docker / macOS when following getting started guide

[robinwassen]

AppScale fails to boot when following the steps for Docker in the Try AppScale in Three Steps guide. It simply gets stuck at Granting admin privileges to a@a.com

Docker: CE 2.0.0.3 (31259), 4GB RAM
OS: macOS Mojave 10.14.4
Computer specs: Macbook Pro 15" Late 2018, 16GB RAM

Steps to reproduce:

• Follow the Try AppScale in Three Steps guide for Docker on macOS

Expected results: AppScale should boot

Actual results: AppScale gets stuck booting

Workarounds tested:

• Start docker with --privileged flag as mentioned in this thread - > Made no difference
• Exit the script -> Seems like it leaves AppScale in a broken semi-started state

Logs:

robin@rmbp15 ~ $ docker run -i --privileged -t appscale/appscale:latest /bin/bash
root@02c3c1205847:/# cd root/
root@02c3c1205847:~# bash appscale/scripts/fast-start.sh --no-demo-app
Detected enviroment: Docker
Private IP found: 172.17.0.2
Public IP found:  172.17.0.2
Creating AppScalefile...done.
Downloading sample app.
/root/guestbook.tar.gz: OK
Executing ssh-copy-id for host: 172.17.0.2
Generated a new SSH key for this deployment at /root/.appscale/appscale6abea85b4ae84ed4b97c900102793b69
Starting AppScale 3.7
Log in to your head node: ssh -i /root/.appscale/appscale6abea85b4ae84ed4b97c900102793b69.key root@172.17.0.2
Starting AppController at 172.17.0.2
Please wait for the AppController to finish pre-processing tasks.
Head node successfully initialized at 172.17.0.2.
Waiting for head node to initialize...
Waiting for head node to initialize...
Waiting for head node to initialize...
Waiting for head node to initialize...
Using the provided admin username/password
Creating new user account a@a.com
Creating new user account a@172.17.0.2
Your XMPP username is a@172.17.0.2
Granting admin privileges to a@a.com

[robinwassen]

I left it running for a while and got a timeout with some more information:

Port 172.17.0.2:1080 did not open in time. Aborting...

A log with more information is available at
/root/.appscale/log-c8d8b983-fa1d-4baa-8976-0e338edb3253.
root@02c3c1205847:~#
root@02c3c1205847:~#
root@02c3c1205847:~# cat /root/.appscale/log-c8d8b983-fa1d-4baa-8976-0e338edb3253
stacktrace : Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/appscale/tools/scripts/appscale.py", line 45, in main
    appscale.up()
  File "/usr/local/lib/python2.7/dist-packages/appscale/tools/appscale.py", line 303, in up
    AppScaleTools.run_instances(run_instances_opts)
  File "/usr/local/lib/python2.7/dist-packages/appscale/tools/appscale_tools.py", line 889, in run_instances
    options.keyname), RemoteHelper.APP_DASHBOARD_PORT, options.verbose)
  File "/usr/local/lib/python2.7/dist-packages/appscale/tools/remote_helper.py", line 378, in sleep_until_port_is_open
    "Aborting...".format(host, port))
TimeoutException: Port 172.17.0.2:1080 did not open in time. Aborting...

exception : TimeoutException

tools_version : 3.7.0

platform : Linux-4.9.125-linuxkit-x86_64-with-Ubuntu-16.04-xenial

message : Port 172.17.0.2:1080 did not open in time. Aborting...

runtime : CPython

[scragraham]

Hi Robin,

Thanks for the issue report, I believe we were able to reproduce the issue. The cause looks like an IPv6 entry for localhost in /etc/hosts.

Before running the fast-start script, please change the following line in /etc/hosts on the container:

::1	localhost ip6-localhost ip6-loopback

To:

::1	ip6-localhost ip6-loopback

With that change we were able to successfully run the faststart script, Let us know if that works for you.

[vanilla111]

I had same problem. I modified the file /etc/hosts as recommended by the contributor, and it still took a long time to run the fast-start script(pause at Granting admin privileges to a@a.com)，although there were the same errors.

[obino]

Hello vanilla111, you may need to start docker with the --privileged options: it's needed to be able to start some of our services which we put in cgroups. We will need to modify documentation for it, thanks for reminding it us. With that option we suggest to use docker as dev-test environment. cheers graziano

…

-----Original Message----- From: vanilla111 <notifications@github.com> Reply-To: AppScale/appscale < reply@reply.github.com> To: AppScale/appscale <appscale@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: Re: [AppScale/appscale] AppScale sample fails to boot on Docker / macOS when following getting started guide (#3061) Date: Thu, 18 Jul 2019 06:22:40 -0700 I had same problem. I modified the file /etc/hosts as recommended by the contributor, and it still took a long time to run the fast-start script(pause at Granting admin privileges to a@a.com)，although there were the same errors. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread. -- Graziano Obertelli | VP Operations 805.570.1647 graziano@appscale.com www.appscale.com

[scragraham]

Also to avoid having to modify the hosts file you can run the image with the following option:

--sysctl net.ipv6.conf.lo.disable_ipv6=0

Docker does some things under the hood with the /etc/hosts file, which is why you have to enable ipv6. It's not really a good idea to modify it.

This is related to: moby/moby#35954

[lucadvp]

Hi, I still have that problem also on a fresh Ubuntu 16.04.5 LTS on amd64 machine following docker try-appscale instruction at step 2/3, even if I add the --privileged and the --sysctl options suggested in the answers above.
I attach the terminal commands, outputs and the appscale log.

Do you have any suggestion? Thanks in advance.

20190920_try-appscale_docker_with_options.txt

[obino]

Hello, unfortunately that log is not telling us much, just that the dashboard didn't come up on time. Can you get onto the container and give us the /var/log/appscale/controller.log file? Actually if you could make us available a tgz of /var/log/appscale I could take a look at what service failed to start in your environment. cheers graziano

…

-----Original Message----- From: lucadvp <notifications@github.com> Reply-To: AppScale/appscale < reply@reply.github.com> To: AppScale/appscale <appscale@noreply.github.com> Cc: graziano obertelli <graziano@appscale.com>, Comment < comment@noreply.github.com> Subject: Re: [AppScale/appscale] AppScale sample fails to boot on Docker / macOS when following getting started guide (#3061) Date: Fri, 20 Sep 2019 12:49:00 -0700 Hi, I still have that problem also on a fresh Ubuntu 16.04.5 LTS on amd64 machine following docker try-appscale instruction at step 2/3, even if I add the --privileged and the --sysctl options suggested in the answers above. I attach the terminal commands, outputs and the appscale log. Do you have any suggestion? Thanks in advance. 20190920_try-appscale_docker_with_options.txt — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread. -- Graziano Obertelli | VP Operations 805.570.1647 graziano@appscale.com www.appscale.com

[lucadvp]

Hi, Graziano, thank you for the answer. I attach the archive with the whole log folder you requested.
Best regards,
Luca
20190921_var_log_appscale.tar.gz

[cdonati]

The log line "No compute node is available to scale..." usually indicates that there is not enough free memory available to start application instances. It could also indicate that the machine's CPUs are too busy, but that is less common.

If your machine already has enough memory (>4G of free memory is usually enough), you could try adding the line "verbose: True" in your AppScalefile to get more detailed controller logs.

[lucadvp]

Hi cdonati, thank you for the suggestion. The issue was actually due to the memory shortage and when around 4GB were available appscale started as expected.
Thanks again,
Luca