Ferm restart and systemd

[alternico]

Hi,

it looks that, while running the basic DebOps playbooks on an Ubuntu 16.04.3 host, when reaching ferm role, everything it is stopping.

Seems to be related systemd.
To you have the same?

Thanks a lot

Jan 19 17:20:11 inf-c-cons-001 systemd[1]: Reloading.
Jan 19 17:20:12 inf-c-cons-001 systemd[1]: message repeated 3 times: [ Reloading.]
Jan 19 17:20:13 inf-c-cons-001 systemd[1]: Starting ferm firewall configuration...
Jan 19 17:20:13 inf-c-cons-001 ferm[15037]:  * Starting Firewall ferm
Jan 19 17:20:13 inf-c-cons-001 kernel: [  470.461108] ip_tables: (C) 2000-2006 Netfilter Core Team
Jan 19 17:20:13 inf-c-cons-001 kernel: [  470.490840] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
Jan 19 17:20:13 inf-c-cons-001 ferm[15037]:    ...done.
Jan 19 17:20:13 inf-c-cons-001 systemd[1]: Started ferm firewall configuration.
Jan 19 17:20:13 inf-c-cons-001 systemd[1]: Reloading.
Jan 19 17:20:14 inf-c-cons-001 systemd[1]: Reexecuting.
Jan 19 17:20:14 inf-c-cons-001 systemd[1]: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)
Jan 19 17:20:14 inf-c-cons-001 systemd[1]: Detected virtualization vmware.
Jan 19 17:20:14 inf-c-cons-001 systemd[1]: Detected architecture x86-64.
Jan 19 17:20:14 inf-c-cons-001 systemd[1]: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)Jan 19 17:20:39 inf-c-cons-001 systemd[1]: Failed to subscribe to NameOwnerChanged signal for 'org.freedesktop.login1': Connection timed out
Jan 19 17:20:39 inf-c-cons-001 systemd[1]: Failed to subscribe to activation signal: Connection timed out
Jan 19 17:20:39 inf-c-cons-001 systemd[1]: Failed to register name: Connection timed out
Jan 19 17:20:39 inf-c-cons-001 systemd[1]: Failed to set up API bus: Connection timed out
Jan 19 17:20:40 inf-c-cons-001 systemd[1]: Looping too fast. Throttling execution a little.
Jan 19 17:21:03 inf-c-cons-001 systemd[1]: message repeated 18 times: [ Looping too fast. Throttling execution a little.]
Jan 19 17:21:05 inf-c-cons-001 systemd[1]: Looping too fast. Throttling execution a little.

[drybjed]

What roles did you enable for that host? There were some issues with debops.libvirtd and debops.docker that prevented the host to be rebooted properly, basically ferm tried to start libvirtd or docker services at the wrong time. These issues should be resolved already in the DebOps monorepo, you could try using the roles from there.

The error messages you provided unfortunately don't give me any clues as to what it might be. Is this log from the time when you run the DebOps playbooks? Or from a reboot? How about when you try and restart the ferm service manually via systemctl restart ferm?

[alternico]

The only playbook that I am trying to run manually is the common.yml one.

I am already running from DebOps monorepo.
Yes, the log is an extract when ansible is configuring the ferm service.
Due to that error, I guess, the following ansible task never get completed:

TASK [debops.ferm : Restart ferm]

Same when I try to restart manually. Only a reboot seems helping.

I really think the issue is on systemd.

[drybjed]

The debops.ferm task list doesn't reload systemd. I'm not sure what could have caused systemd to reload its configuration at that time.

Check the /etc/ferm/ directory, are there any custom hooks? This is a VMWare host, how did you deploy it? Did you use an Ubuntu image with some custom software?

I don't really see anything that could have caused this, at least from the logs you provided. The debops.ferm role is applied multiple times in this GitLab CI pipeline with no errors, all tests on Debian Stretch though.

Any chance that you could test this using a Debian Stretch host, for comparsion?

[alternico]

No custom hooks.
The thing is that, after I am able to refresh/restart systemd - with a reboot - everything is running fine together with other roles/playbooks.

Is there a way you could test on a vanilla Ubuntu 16.04?
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial

[drybjed]

Sure, I'll try to do that a bit later.

[alternico]

URL: http://releases.ubuntu.com/16.04/ubuntu-16.04.3-server-amd64.iso

The only thing that I do, before running ansible, is a full system upgrade.
Thanks

[alternico]

Additional notes to reproduce the issue:

• debops bootstrap
• debops

this will stop to the ferm configuration, as I have explained before.

INSTEAD, if I do the following:

• debops bootstrap
• (reboot the server)
• debops

all is working fine.

At this point, I think the issue is related to systemd not working correctly.. I have notice that, during the execution of the bootstrap role, I have different lines saying:
systemd[1]: message repeated 6 times: [ Reloading.]
Thanks

[drybjed]

How about a test then. Create a new host, and run on it:

debops bootstrap
debops common --skip-tags role::ferm

This should avoid any ferm configuration. If systemd still causes problems, it's definitely something else than ferm.

[alternico]

Hi,

I can confirm. It has failed with the same reason also skipping ferm.
This time, it has failed on debops.ntp..

The error message is always the same.
If I reboot and re-run the common.yml, all is going fine..

Any ideas on where I can look at?
thanks a lot

[drybjed]

This Ubuntu bug might be perhaps related? It was created 4 hours ago.

[alternico]

Hi, Was me opening this bug.. Thanks

…

On 19. Jan 2018, at 20:34, Maciej Delmanowski ***@***.***> wrote: This Ubuntu bug might be perhaps related? It was created 4 hours ago. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.