Add Ansible Script to Allow Nodes to Keep Their Own External Data Up-to-date

[cailafinn]

Adds a new role (that is, by default, set to run never in the playbook) to the agent playbook that downloads and then sets up a crontab job to keep the external data store up-to-date. This stops nodes having to wait until a build runs on them to download the data, improving parallelization.

The changes also adjust the way the data is stored on the host machine, moving it from a docker volume (which is inaccessible from the host machine) to a mount in the root directory. This allows viewing and manipulation of the contents of the volume to be performed without having to enter into a docker container.

This change was implemented primarily to get around a bug in a packaging script where new data could not be downloaded.
Having the data present on the machine ahead of time was an easier solution.

To Test

• Take a linux node offline.
• SSH into the node. Remove some data from the /{agent_name}_external_data/MD5 directory.
• Wait a while.
• Check that the crontab job downloaded the missing files by checking update_log.txt
• Remove the crontab job (sudo crontab -e -u root)
• Run the ansible script
  ansible-playbook -i inventory.txt jenkins-agent-production.yml -t "mirror, agent" -u {fedID}
• SSH back into the machine and check the crontab job is present and the data has been downloaded.

[idigs]

--whole-file, -W
              This option disables rsync's delta-transfer algorithm, which causes all transferred files to be sent whole.  The transfer may be faster if this option  is  used  when
              the  bandwidth between the source and destination machines is higher than the bandwidth to disk (especially when the "disk" is actually a networked filesystem).  This
              is the default when both the source and destination are specified as local paths, but only if no batch-writing option is in effect.

[cailafinn]

Including the -W flag seemed to reduce some of the flakiness that was happening when trying to rsync data though the load balancer.

[warunawickramasingha]

When running the ansbile script, for remote_user, I had to use my fedid with with --ask-become-pass option instead, as ubuntu user as it gave below error

fatal: [172.16.111.215 -> 172.16.114.127]: UNREACHABLE! => {"changed": false, "msg": "Task failed: Failed to connect to the host via ssh: ubuntu@172.16.114.127: Permission denied (publickey,password).", "unreachable": true}

[warunawickramasingha]

The ssh connection was closed due to long running process. It would be better to run this asynchronously as follows.

Suggested change

	ansible.builtin.command: "rsync -azvW --perms -o -g {{ ansible_user_id }}@{{ data_server_hostname }}:/external-data/MD5/ /{{ agent_name }}_external_data/MD5 -v"
	ansible.builtin.command: >
	rsync -azvW --perms -o -g
	{{ ansible_user_id }}@{{ data_server_hostname }}:/external-data/MD5/
	/{{ agent_name }}_external_data/MD5/
	async: 10800 # allow up to 3h
	poll: 30 # check every 30s

[warunawickramasingha]

In addition, after deleting some of the files from the mount path of the docker volume in the host machine, I cannot see the missing files are made available even after several cron runs.

Shown below are all the current cron jobs,
crontab -l >>

#Ansible: update_admin_keys
0 7 * * * /usr/local/sbin/update_keys.sh
#Ansible: update_cloud_keys
0 6 * * * /usr/local/sbin/update_cloud_users.sh
#Ansible: Update external data
*/5 * * * * /isis-cloud-linux-b-9_external_data/update-external-data.sh 172.16.114.127 isis-cloud-linux-b-9 kli94267 >> /isis-cloud-linux-b-9_external_data/update-log.txt 2>&1