Add tooling to enable quick release testing by amcp · Pull Request #129 · JanusGraph/janusgraph

amcp · 2017-02-20T18:46:31Z

This change adds a command line tool that ideally will allow any combination of tests to be run. All you have to do is define the command line arguments to maven for each test run and execute the CLI as per the README.md. Currently, the maximum number of parallel actions (build steps) in a stage is 5, so I split up the various test targets into groups of up to four builds.

Status of builds in j1:

bdb - Passed in 15 min 10 secs
hadoop-lucene-solr - Passed in 21 minutes 12 seconds - artifact assembly needs to be fixed
test - Passed in 21 minutes 37 seconds
cassandra - Passed in 110 minutes

Status of builds in j2:

hbase098 - Passed in 65 minutes
hbase10 - Passed in 39 minutes 44 seconds
es - Failed in 15 min 12 sec - cant be run as root
cql - Failed in 67 minutes (cql tests will be fixed elsewhere)

Status of builds in jtp1:

es-hadoop-lucene-solr - Passed in 6 minutes 49 seconds - artifact assembly needs to be fixed
bdb - Passed in 118 minutes

Status of builds in jtp2:

test - Passed in 32 minutes 4 seconds
cassandra - Failed in 131 minutes (known failures)
hbase098 - Failed in 409 minutes (known failures)
hbase10 - Passed in 467 minutes (very close to timeout limit of 8 hours)
cql - Timed out in 8 hours (currently 14-15 hours. the timeout is a limitation of CodeBuild. will not run to completion until docker orchestration drops the run time for cql)

amcp · 2017-02-21T17:52:01Z

@sjudeng @jerryjch I am running off amcp/janusgraph master now. I will report back how long it took for all pipelines to finish testing. But, the PR is good to start reviewing.

amcp · 2017-02-22T03:18:45Z

@amcp regroup tests into those that pass and those that fail. Want to see the kind of integration that can be had on passing pipelines.

janusgraph-bot · 2017-02-22T06:03:03Z

Committer of one or more commits is not listed as a CLA signer, either individual or as a member of an organization.

mbrukman · 2017-02-24T22:44:36Z

@amcp — you used two different addresses here, one as "author", a different one as "committer", so your CLA is not recognized. You can see this because there are two photos next to each of your commits. Could you please fix this?

amcp · 2017-02-25T11:18:19Z

@mbrukman I will fix my commit.

sjudeng · 2017-03-23T02:12:49Z

 /log/
 /output/
 /scripts/
+/janusgraph-codepipelines-ci/dependency-reduced-pom.xml


Can you update Maven shade plugin config in janusgraph-codepipelines-ci/pom.xml to include createDependencyReducedPom=false and then remove this exclusion?

sjudeng · 2017-03-23T02:13:19Z

+/codepipelines-ci/.classpath
+/codepipelines-ci/.project
+/codepipelines-ci/.settings/
 /janusgraph-*/.classpath


These exclusions aren't necessary since they're covered under the below janusgraph-* patterns

sjudeng · 2017-03-23T02:13:35Z

+artifacts:
+  base-directory: module
+  files:
+    - target/**/*


Can you move buildspec.yml to the janusgraph-codepipelines-ci module?

The standard location is at the root of the repository. We could add a step in the manual workflow to copy the file to the repo root, zip up the repo, copy it to S3, and use that as the source for Code Pipelines. Sounds like a lot of mechanism. Can we revisit this later, perhaps adding a TODO?

sjudeng · 2017-03-23T02:14:19Z

@@ -0,0 +1 @@
+org.apache.tinkerpop.gremlin.process.TraversalPerformanceTest


Where is this test resource used? Same question for bdb2 and bdb3 below.

Now that I have CI working for the DynamoDB Storage Backend here, I want to build on this experience to inform the mechanism for dividing up tests here.

Basically, considering the large number of tests we have, parameterization and the length of concatenated canonical test class names, using these kinds of files is unwieldy and potentially may cause length exceeded errors on the command line used to start tests. I would rather define test categories and use those to drive the tests.

Of course, depending on the granularity of division, we may need a tool to make sure that TP updates and janusgraph-test updates did not introduce new tests that are not labelled with the new test categories.

sjudeng · 2017-03-23T02:15:18Z

+--github-repo janusgraph \
+--github-branch <branch you want to test> \
+--profile code-pipelines \
+--environments=./environments.json \


Should --environment be --pipelines and can you document the expected JSON content format?

renamed and moved to the bottom

sjudeng · 2017-03-23T02:32:22Z

        <compiler.target>1.8</compiler.target>
+        <test.pipelines.excludes></test.pipelines.excludes>
+        <test.pipelines.includes></test.pipelines.includes>
+        <test.skip.pipelines>true</test.skip.pipelines>


Where are these used?

I guess nowhere so i removed them.

sjudeng · 2017-03-23T02:32:32Z

                            <exclude>**/*ProcessPerformanceTest.java</exclude>
                            <exclude>**/*StructureTest.java</exclude>
                        </excludes>
+


Remove extra newline

sjudeng · 2017-03-23T02:32:55Z

+                                    <log4j.configuration>file:${project.build.directory}/test-classes/log4j.properties</log4j.configuration>
+                                </systemPropertyVariables>
+                            </configuration>
+                        </execution>


Do the janusgraph-codepipeline-ci tests warrant this custom test configuration? Would default surefire configuration suffice?

sjudeng · 2017-03-23T02:33:51Z

+        <relativePath>../pom.xml</relativePath>
+    </parent>
+    <artifactId>janusgraph-codepipelines-ci</artifactId>
+    <name>JanusGraph-CodePipeline CI: Distributed Graph Database</name>


Update description from "Distributed Graph Database" to something describing module (e.g. "AWS Codepipelines" or something)

sjudeng · 2017-03-23T02:37:12Z

+```bash
+mvn clean && mvn package && java -jar target/janusgraph-codepipelines-ci-0.1.0-SNAPSHOT.jar --region <region close to you> \
+--bucket <bucket> \
+--name <name of ci stack> \


Are github-owner and github-token really required? I'm not familiar with AWS CodePipelines but the below docs suggest personal access tokens aren't required when using Git pull. If those docs are applicable it looks like only OutputBucketName and AllowedIps (GitHub 192.30.252.0/22?) would be required. I'm not positive but it doesn't look like even the ApiSecret would be required.

https://aws.amazon.com/blogs/devops/integrating-git-with-aws-codepipeline/

Thank you for referring me to the blog article. My read on the article indicates that OutputBucketName plus one of AllowedIps or ApiSecret would be required. But, I like your inclination. AllowedIps requires less effort on the user's part and should continue to be viable for the foreseeable future.

On second thought, the existing configuration sets up a hook where new commits on the selected branch automatically get built in code pipeline/codebuild. Scenario: set up the pipeline for a release branch, tests get run once and some fail, you fix the failing tests, push a new commit to the release branch and then codepipeline/codebuild automatically builds anew.

amcp · 2017-06-08T08:02:52Z

+ */
+@Slf4j
+public class AwsCodePipelinesCi {
+    private static final String AWS_CODEBUILD_JAVA_OPENJDK_8 = "aws/codebuild/java:openjdk-8";


pull into a parameter

amcp · 2017-06-08T08:03:18Z

+                        .withPackaging(ArtifactPackaging.NONE)
+                        .withType(ArtifactsType.CODEPIPELINE)
+                        .withName(artifactsName))
+                .withTimeoutInMinutes(480) //8 hours max - TODO externalize


pull into a parameter

amcp · 2017-06-09T04:56:41Z

@sjudeng I'm working on this again. Any help you could give to push this to completion would be greatly appreciated.

amcp · 2017-06-23T17:23:11Z

@sjudeng getting the cql tests passing is also out of scope of this cr, so I am marking those complete as well.

sjudeng

Some feedback on the documentation

sjudeng · 2017-06-24T18:57:39Z

@@ -0,0 +1,57 @@
+#JanusGraph CodePipelines CI


Need space after # for markdown section heading

sjudeng · 2017-06-24T18:58:12Z

+         --batch-mode \
+         --file ${SRC_DIR}/pom.xml \
+         -DskipTests=true -Dmaven.javadoc.skip=true -Dmaven.compiler.showWarnings=false \
+    && rm -rf ${SRC_DIR}


How is this Dockerfile used?

The purpose of this Dockerfile is to create a reusable cache of maven dependencies.

on second thought, it seems like I am not using this Dockerfile, so I will remove it.

sjudeng · 2017-06-24T18:58:46Z

+FROM maven:3-jdk-8
+ENV DEBIAN_FRONTEND noninteractive
+COPY settings-docker.xml /usr/share/maven/ref/
+ENV COMMIT 9165ee8407b3607f39031cdaeff1d15ba0fd4e9e


What's the significance of this commit?

It is a recent commit I can bake into a Docker repository so that building code does not take a long time - I use it to pull all the maven dependencies. Any better ideas?

sjudeng · 2017-06-24T19:06:49Z

+It also requires you to create two service roles in IAM: one for CodePipeline and
+one for CodeBuild.
+
+1. Get a personal access token from [GitHub](https://github.com/settings/tokens) with repo and admin:repo_hook scopes.


Is write:repo_hook required or just read:repo_hook?

I've confirmed in a use case and The documentation only mentions the repo scope but it assumes the user will create the pipeline using the console. To create the pipeline programmatically, the admin:repo_hook scope is required.

sjudeng · 2017-06-24T19:13:46Z

+CodePipelines CI is a mechanism JanusGraph can use to do release testing in massively
+parallel fashion (to the extent of AWS CodePipelines and CodeBuild service limits).
+
+## Steps to kick off a build


Consider splitting this into two sections, one for one-time "Setup" (with steps 1-6 below) and one for "Testing".

sjudeng · 2017-06-24T19:59:11Z

+--bucket jgamcp \
+--github-owner amcp \
+--github-repo janusgraph \
+--github-branch codepipelines \


Set region, bucket, github owner, and branch as part of the environment above.

sjudeng · 2017-06-24T20:00:00Z

+mvn clean package
+#define variables
+export AWS_PROFILE_NAME='profile_to_use'
+export AWS_ACCOUNT_NUMBER='000000000000'


Is this the root account number?

I can use STS to get the root account number, I will update

sjudeng · 2017-06-24T20:14:46Z

+--codebuild-role-arn arn:aws:iam::${AWS_ACCOUNT_NUMBER}:role/${AWS_CODEBUILD_ROLE} \
+--codepipeline-role-arn arn:aws:iam::${AWS_ACCOUNT_NUMBER}:role/${AWS_CODEPIPELINE_ROLE} \
+--github-token ${GITHUB_TOKEN} \
+--pipelines ${PWD}/pipe.json


Consider moving pipelines file to the environment as well.

I provided two pipelines files so I prefer that users be able to copy paste the execution command after setting up their environment.

second thought, i like your comment below so I will change according to your suggestion

sjudeng · 2017-06-24T20:16:13Z

+--github-token ${GITHUB_TOKEN} \
+--pipelines ${PWD}/pipe.json
+
+#kick off tinkerpop tests


Consider removing the second call here. It's a complicated call and the only difference is the pipeline used. Instead create another section or discussion (or link) on how to create the pipelines files and indicate that examples are included to run default (pipe.json) and TinkerPop (tp-pipe.json) tests.

sjudeng · 2017-06-24T20:17:03Z

+```
+8. Navigate to the [AWS Console](https://console.aws.amazon.com) and check on the status of your pipeline to see status.
+
+## Future ideas


What about tracking this under a separate issue instead of here?

moved to an issue #367

sjudeng

Can you include a section on cleaning up resources (e.g. CodeBuild, CodePipeline, S3 bucket, IAM roles/policies/user, CloudWatch logs, etc.)?

sjudeng · 2017-06-26T22:31:30Z

+          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
+                      https://maven.apache.org/xsd/settings-1.0.0.xsd">
+    <localRepository>/usr/share/maven/ref/repository</localRepository>
+</settings>


Can this be removed since you've removed Dockerfile?

sjudeng · 2017-06-26T22:31:54Z

+2. Second, configure the parameters below to customize your test stack.
+
+```bash
+#define variables


Remove comment

sjudeng · 2017-06-26T22:32:11Z

+and `export PIPELINE_CONFIGURATION=${PWD}/tp-pipe.json` for TinkerPop tests.
+
+```bash
+#kick off regular tests


Remove comment

sjudeng · 2017-06-26T22:42:04Z

+#Name of bucket you want to use or create and use for storing your build artifacts.
+export AWS_S3_BUCKET_NAME=''
+#Name of pipeline configuration file to use
+export PIPELINE_CONFIGURATION=''


Do you want to generalize "Create test pipeline" into it's own list item/section and describe the pipeline.json format and provide the two examples?

Created a "Create test pipeline" section and described one example

sjudeng · 2017-06-27T00:01:43Z

+mvn clean package
+#constants
+export AWS_PROFILE_NAME='code-pipelines'
+export AWS_ACCOUNT_NUMBER=`aws sts get-caller-identity --output text | cut -f1`


Add --profile code-pipelines

sjudeng · 2017-06-27T00:54:28Z

+--pipelines ${PIPELINE_CONFIGURATION}
+```
+
+4. Navigate to the [AWS Console](https://console.aws.amazon.com) and check on the status of your pipeline to see status.


Consider changing link to http://console.aws.amazon.com/codepipeline

sjudeng · 2017-06-27T01:49:47Z

+and a default [output format](http://docs.aws.amazon.com/cli/latest/userguide/controlling-output.html#controlling-output-format)
+of your choice (`json`, `text`, and `table` are available).
+4. Create a service role for CodeBuild as described
+[here](http://docs.aws.amazon.com/codebuild/latest/userguide/setting-up.html#setting-up-service-role).


"Create a CodeBuild policy and associate it to a new service role as described ..."

sjudeng · 2017-06-27T02:12:30Z

+[here](http://docs.aws.amazon.com/codebuild/latest/userguide/setting-up.html#setting-up-service-role).
+5. Create an IAM policy for a CodePipelines service role as described
+[here](http://docs.aws.amazon.com/codepipeline/latest/userguide/iam-identity-based-access-control.html#view-default-service-role-policy).
+6. Create a service role for CodePipelines by using the policy you created in step 5 and then set the STS trust to


Combine 5 and 6 (makes it consistent with step 4): "Create a CodePipeline policy and associate it to a new service role." Indicate (maybe as a sub-bullet) that "EC2" (or whichever) role be selected when creating the role since AWS CodePipeline doesn't appear to be an available option. Be more specific (maybe also as a sub-bullet) on how to set STS trust to codepipeline.amazonaws.com.

sjudeng · 2017-06-27T02:16:36Z

+#define variables
+#GitHub personal access token you created in step 1
+export GITHUB_TOKEN=''
+#GitHub organization or user name


Add "of the repository to build". Need to make it clear that it's the user/group of the repository to test not (necessarily) the GitHub handle.

sjudeng · 2017-06-27T02:18:24Z

+```
+
+3. Kick off the regular and TinkerPop tests. Use `export PIPELINE_CONFIGURATION=${PWD}/pipe.json` for regular tests
+and `export PIPELINE_CONFIGURATION=${PWD}/tp-pipe.json` for TinkerPop tests.


I think you can remove the ${PWD}/ prefixes

amcp · 2017-06-27T04:58:16Z

I forgot to add a cleanup section. I can do that next.

amcp · 2017-06-27T05:18:22Z

@sjudeng I added the cleanup section.

sjudeng

Looking good. I think it's just down to getting the es tests passing and fixing the issue (what is it?) with "artifact assembly" in hadoop-lucene-solr test group.

sjudeng · 2017-06-27T11:00:12Z

+export AWS_REGION_NAME=''
+#Name of bucket you want to use or create and use for storing your build artifacts.
+export AWS_S3_BUCKET_NAME=''
+#Name of pipeline configuration file to use


"Name of pipeline JSON configuration file to use"

sjudeng · 2017-06-27T11:02:37Z

+
+To kick off the regular and TinkerPop tests. Use `export PIPELINE_CONFIGURATION=pipe.json` for regular tests
+and `export PIPELINE_CONFIGURATION=tp-pipe.json` for TinkerPop tests.
+


Add a section break here for ## Run Test Pipelines. I'd keep the "export PIPELINE_CONF..." examples in the previous section.

sjudeng

More minor feedback. Do you want to create a separate issue to get ES tests running through pipelines? That would just leave the issue you mention of fixing artifact assembly for hadoop-lucene-solr test group.

sjudeng · 2017-06-27T17:54:00Z

+4. Create an IAM policy for CodeBuild and associate it to a new service role as described
+[here](http://docs.aws.amazon.com/codebuild/latest/userguide/setting-up.html#setting-up-service-role). Select the
+__Amazon EC2__ role type as CodeBuild is not yet available on this page. Edit the new role's trust relationship
+and replace it with the text in step 16 of the CodeBuild user guide.


There is now an AWS CodeBuild role so I don't think you need the second or third sentences (default role trust relationship includes codebuild.amazonaws.com principal).

sjudeng · 2017-06-27T17:55:19Z

+--pipelines ${PIPELINE_CONFIGURATION}
+```
+
+Navigate to the [AWS Console](https://console.aws.amazon.com/codepipeline) and check on the status of your pipeline to see status.


"...status of your pipeline."

sjudeng · 2017-06-27T18:25:47Z

+```
+The first pipeline is called `j1` and the second pipeline is called `j2`. Each of these pipelines have four
+parallel build actions defined. Each build action is keyed at the action name and includes the environment
+variables to be passed CodeBuild and used in the buildspec.yml file. Use `export PIPELINE_CONFIGURATION=pipe.json` for regular tests


Instead of "Use 'export PIPELINE..." how about something like below?

Define the PIPELINE_CONFIGURATION environment variable to point to the custom pipeline file or use existing pipe.json/tp-pipe.json to run all default/TinkerPop tests.

export PIPELINE_CONFIGURATION=pipe.json

amcp · 2017-06-27T18:37:52Z

lets create a separate issue for fixing ES build #372

amcp · 2017-06-30T00:48:32Z

@sjudeng I fixed assembly of composite builds like hadoop-lucene-solr so I am ready to close this PR out.

sjudeng

Looks good. Nice work!

Signed-off-by: Alexander Patrikalakis <amcp@amazon.co.jp>

janusgraph-bot added the cla: yes This PR is compliant with the CLA label Feb 20, 2017

amcp mentioned this pull request Feb 21, 2017

Reduce amount of time to run tinkerpop tests from 40 hours to 2.5 hours #111

Closed

14 tasks

amcp force-pushed the codepipelines branch from 80a32c2 to 918a553 Compare February 21, 2017 17:48

amcp changed the title ~~[WIP] add tooling to enable quick release testing~~ add tooling to enable quick release testing Feb 21, 2017

amcp requested a review from jerryjch February 21, 2017 17:50

amcp force-pushed the codepipelines branch 3 times, most recently from af06612 to f22c9d3 Compare February 21, 2017 18:47

amcp force-pushed the codepipelines branch from 3292be3 to 6e5f59d Compare February 22, 2017 06:02

janusgraph-bot added cla: no This PR is not compliant with the CLA and removed cla: yes This PR is compliant with the CLA labels Feb 22, 2017

amcp force-pushed the codepipelines branch from 6e5f59d to 0976c5f Compare March 6, 2017 00:29

janusgraph-bot added cla: yes This PR is compliant with the CLA and removed cla: no This PR is not compliant with the CLA labels Mar 6, 2017

amcp force-pushed the codepipelines branch from 0976c5f to 50dd37d Compare March 6, 2017 01:13

sjudeng requested changes Mar 23, 2017

View reviewed changes

amcp changed the title ~~add tooling to enable quick release testing~~ [WIP] add tooling to enable quick release testing Apr 29, 2017

amcp force-pushed the codepipelines branch from 29cfdda to b21c139 Compare April 29, 2017 04:07

amcp force-pushed the codepipelines branch 3 times, most recently from 3aa60ec to 13c50dc Compare June 8, 2017 07:32

amcp commented Jun 8, 2017

View reviewed changes

amcp force-pushed the codepipelines branch from 13c50dc to ccd391c Compare June 8, 2017 09:33

amcp changed the title ~~[WIP] add tooling to enable quick release testing~~ add tooling to enable quick release testing Jun 23, 2017

amcp changed the title ~~add tooling to enable quick release testing~~ Add tooling to enable quick release testing Jun 23, 2017

sjudeng reviewed Jun 24, 2017

View reviewed changes

amcp force-pushed the codepipelines branch 2 times, most recently from 9e4f2a0 to 1231adb Compare June 26, 2017 16:33

sjudeng reviewed Jun 27, 2017

View reviewed changes

amcp force-pushed the codepipelines branch from 7c52bcf to 6215ae1 Compare June 27, 2017 04:57

amcp requested review from analytically and removed request for jerryjch June 27, 2017 04:58

amcp force-pushed the codepipelines branch from 6215ae1 to 287665b Compare June 27, 2017 05:18

sjudeng reviewed Jun 27, 2017

View reviewed changes

amcp force-pushed the codepipelines branch from 287665b to e92551e Compare June 27, 2017 17:36

sjudeng reviewed Jun 27, 2017

View reviewed changes

amcp force-pushed the codepipelines branch 2 times, most recently from 4f6d207 to a2a9360 Compare June 27, 2017 18:35

amcp force-pushed the codepipelines branch 3 times, most recently from 37d1b87 to b08d6fc Compare June 30, 2017 00:44

sjudeng approved these changes Jun 30, 2017

View reviewed changes

amcp force-pushed the codepipelines branch from b08d6fc to b811141 Compare June 30, 2017 01:32

Add tooling to enable quick release testing [skip ci]

956d130

Signed-off-by: Alexander Patrikalakis <amcp@amazon.co.jp>

amcp force-pushed the codepipelines branch from b811141 to 956d130 Compare June 30, 2017 03:41

amcp merged commit 2a4d48f into JanusGraph:master Jun 30, 2017

amcp deleted the codepipelines branch June 30, 2017 10:39

sjudeng added this to the Release v0.2.0 milestone Jul 6, 2017

		@@ -0,0 +1 @@
		org.apache.tinkerpop.gremlin.process.TraversalPerformanceTest


		To kick off the regular and TinkerPop tests. Use `export PIPELINE_CONFIGURATION=pipe.json` for regular tests
		and `export PIPELINE_CONFIGURATION=tp-pipe.json` for TinkerPop tests.

Uh oh!

Conversation

amcp commented Feb 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amcp commented Feb 21, 2017

Uh oh!

amcp commented Feb 22, 2017

Uh oh!

janusgraph-bot commented Feb 22, 2017

Uh oh!

mbrukman commented Feb 24, 2017

Uh oh!

amcp commented Feb 25, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sjudeng Mar 23, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amcp commented Jun 9, 2017

Uh oh!

amcp commented Jun 23, 2017

Uh oh!

sjudeng left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amcp commented Feb 20, 2017 •

edited

Loading

sjudeng Mar 23, 2017 •

edited

Loading