fix: AWS deployment system

[cherriechang]

by migrating to AWS SDK

[changeset-bot]

🦋 Changeset detected

Latest commit: 5aa0200

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
pushkin-cli	Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[becky-gilbert]

Here's a list of the changes I made on this PR. Happy to revert any commits that fix issues that have already been resolved on other branches.

• b504862: fix incorrect use of .iam on useIAM in aws/index.js. The useIAM variable in this file is already the profile name string, so useIAM.iam was resulting in undefined. I think this was working for others because, when the profile string is not found, the AWS commands will fall back to the default profile (if there is one). In my case, the default was a different (non-pushkin) AWS profile, which is how I discovered the issue.
• 7a8cbe0: remove the credentials parameter in the CreateDistributionWithTagsCommand because that command does not support it (the auth is handled by cloudFrontClient)
• 2297138: await checkIAMUser because it's async - this allows us to catch any async errors.
• bbf3b3c: add retries in case PutBucketPolicyCommand fails (e.g. because new S3 bucket and/or cloud ARNs do not resolve yet)
• 4a0f18f: I had to change DB_HOST and TRANS_HOST back to host to get this to work. In pushkin.yaml, host is the Docker service name, which is what is needed here for within Docker communication (test_db, test_transaction_db). Whereas url is localhost, which is just for host machine access.
• 682b9e7: Improve DB connection error handling by adding catch block to DB test queries, which use promise.then(). The try/catch only catches synchronous errors. (We can't change the promise.then(...).catch(...) to try/catch with await because DefaultHandler constructor cannot be async.)
• 840dfe4: Automated yarn.lock changes (removes file protocol entries due to the use of yalc add/yarn add in local dev, plus other yarn normalization changes).
• abe577a: The experiment worker connection needs SSL on AWS RDS but not locally. So I created a DB_SSL flag and set it to true on AWS deployment. Then, in the worker index.js files, I set the ssl_config parameter in the connection settings based on this flag. When DB_SSL is true (i.e. on AWS deployments), the worker's DB connection parameters will contain ssl: { rejectUnauthorized: false }. When run locally in Docker, the connection parameters will contain ssl: false.
• b7db1fc: This fixes a bug with AWS deployment and multiple experiments, where one experiment would log under another experiment prefix and was running the wrong image. The task.services info was being mutated in place, which was causing the current task to overwrite all of the others on each iteration of .map. This fixes the problem by deep-cloning the template object.

These changes were made to fix failing CI checks:

• 2475a74: Skip the failing e2e tests. The issues with these tests are all unrelated to AWS changes - I will open a separate PR to fix them.
• fb5ca46: CodeQL was flagging the regex for handling the user input into a shell command, as it can be an unreliable way of sanitizing user input. So this switches from constructing shell string from user input to using execFile and a list of arguments, which gets around the problems of passing user input directly into shell commands.
• 96662a0: This just adds some basic unit tests for buildDockerArgs, to make sure that the switch from exec to execFile is still handling spaces in file paths and has not changed the commands in any other ways. (Note that I couldn't test this with the source code directly because it isn't exported, so instead I had to replicate the logic in the tests. This is not ideal, but we'd have to refactor the source code to make this piece importable/testable.)