Skip to content

added gitlab support #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
empath-nirvana wants to merge 16 commits into
base: master
Choose a base branch
from
Open

added gitlab support #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
ad39f4b
added gitlab support
empath-nirvana Jun 30, 2021
dbe2c61
Update pkg/services/scm_provider/gitlab.go
empath-nirvana Jun 30, 2021
3b81c83
fixed path search
empath-nirvana Jul 1, 2021
fb6557b
Merge branch 'gitlab-provider' of https://github.com/empath-nirvana/a…
empath-nirvana Jul 1, 2021
331f38d
fixed merge
empath-nirvana Jul 1, 2021
1b4bf58
fixed tests
empath-nirvana Jul 2, 2021
08ec55a
removed rate limit check
empath-nirvana Jul 2, 2021
6a35972
tidy
empath-nirvana Jul 2, 2021
1c898b1
fixed github references in comments
empath-nirvana Jul 6, 2021
b45c317
added includeSubgroup option, fixed lints
empath-nirvana Jul 7, 2021
cd73253
removed print debugging
empath-nirvana Jul 7, 2021
3cf2d26
updated docs
empath-nirvana Jul 7, 2021
cc9803c
Update applicationset_types.go
empath-nirvana Jul 7, 2021
e5446dd
updated manifests
empath-nirvana Jul 7, 2021
4649889
Merge branch 'gitlab-provider' of https://github.com/empath-nirvana/a…
empath-nirvana Jul 7, 2021
74016d5
fixed manifests
empath-nirvana Jul 7, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 15 additions & 1 deletion api/v1alpha1/applicationset_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ type GitFileGeneratorItem struct {
type SCMProviderGenerator struct {
// Which provider to use and config for it.
Github *SCMProviderGeneratorGithub `json:"github,omitempty"`
// TODO other providers.
Gitlab *SCMProviderGeneratorGitlab `json:"gitlab,omitempty"`
// Filters for which repos should be considered.
Filters []SCMProviderGeneratorFilter `json:"filters,omitempty"`
// Which protocol to use for the SCM URL. Default is provider-specific but ssh if possible. Not all providers
Expand All @@ -181,6 +181,20 @@ type SCMProviderGeneratorGithub struct {
AllBranches bool `json:"allBranches,omitempty"`
}

// SCMProviderGeneratorGitlab defines a connection info specific to Gitlab.
type SCMProviderGeneratorGitlab struct {
// Gitlab group to scan. Required. You can use either the project id (recommended) or the full namespaced path.
Group string `json:"group"`
// Recurse through subgroups (true) or scan only the base group (false). Defaults to "false"
IncludeSubgroups bool `json:"includeSubgroups,omitempty"`
// The Gitlab API URL to talk to.
API string `json:"api,omitempty"`
// Authentication token reference.
TokenRef *SecretRef `json:"tokenRef,omitempty"`
// Scan all branches instead of just the default branch.
AllBranches bool `json:"allBranches,omitempty"`
}

// SCMProviderGeneratorFilter is a single repository filter.
// If multiple filter types are set on a single struct, they will be AND'd together. All filters must
// pass for a repo to be included.
Expand Down
25 changes: 25 additions & 0 deletions api/v1alpha1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

40 changes: 39 additions & 1 deletion docs/Generators-SCM-Provider.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@

The SCM Provider generator uses the API of an SCMaaS provider (eg GitHub) to automatically discover repositories within an organization. This fits well with GitOps layout patterns that split microservices across many repositories.

Support is currently limited to GitHub, but PRs are welcome to add more SCM providers.

```yaml
apiVersion: argoproj.io/v1alpha1
Expand Down Expand Up @@ -57,6 +56,45 @@ For label filtering, the repository topics are used.

Available clone protocols are `ssh` and `https`.

## Gitlab

The Gitlab mode uses the Gitlab API to scan and organization in either gitlab.com or self-hosted gitlab.

```yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: myapps
spec:
generators:
- scmProvider:
gitlab:
# The base Gitlab group to scan. You can either use the group id or the full namespaced path.
group: "8675309"
# For GitHub Enterprise:
api: https://gitlab.example.com/
# If true, scan every branch of every repository. If false, scan only the default branch. Defaults to false.
allBranches: true
# If true, recurses through subgroups. If false, it searches only in the base group. Defaults to false.
includeSubgroups: true
# Reference to a Secret containing an access token. (optional)
tokenRef:
secretName: gitlab-token
key: token
template:
# ...
```

* `group`: Required name of the base Gitlab group to scan. If you have multiple base groups, use multiple generators.
* `api`: If using GitHub Enterprise, the URL to access it.
* `allBranches`: By default (false) the template will only be evaluated for the default branch of each repo. If this is true, every branch of every repository will be passed to the filters. If using this flag, you likely want to use a `branchMatch` filter.
* `allBranches`: By default (false) the controller will only search for repos directly in the base group. If this is true, it will recurse through all the subgroups searching for repos to scan.
* `tokenRef`: A `Secret` name and key containing the Gitlab access token to use for requests. If not specified, will make anonymous requests which have a lower rate limit and can only see public repositories.

For label filtering, the repository tags are used.

Available clone protocols are `ssh` and `https`.

## Filters

Filters allow selecting which repositories to generate for. Each filter can declare one or more conditions, all of which must pass. If multiple filters are present, any can match for a repository to be included. If no filters are specified, all repositories will be processed.
Expand Down
1 change: 1 addition & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ require (
github.com/sirupsen/logrus v1.6.0
github.com/stretchr/testify v1.6.1
github.com/valyala/fasttemplate v1.2.1
github.com/xanzy/go-gitlab v0.50.0
golang.org/x/oauth2 v0.0.0-20210413134643-5e61552d6c78
k8s.io/api v0.19.2
k8s.io/apimachinery v0.19.2
Expand Down
11 changes: 11 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -377,6 +377,12 @@ github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
github.com/grpc-ecosystem/grpc-gateway v1.12.2 h1:D0EVSTwQoQOyfY35QNSuPJA4jpZRtkoGYWQMB7XNg5o=
github.com/grpc-ecosystem/grpc-gateway v1.12.2/go.mod h1:8XEsbTttt/W+VvjtQhLACqCisSPWTxCZ7sBRjU6iH9c=
github.com/hashicorp/go-cleanhttp v0.5.1 h1:dH3aiDG9Jvb5r5+bYHsikaOUIpcM0xvgMXVoDkXMzJM=
github.com/hashicorp/go-cleanhttp v0.5.1/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80=
github.com/hashicorp/go-hclog v0.9.2 h1:CG6TE5H9/JXsFWJCfoIVpKFIkFe6ysEuHirp4DxCsHI=
github.com/hashicorp/go-hclog v0.9.2/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ=
github.com/hashicorp/go-retryablehttp v0.6.8 h1:92lWxgpa+fF3FozM4B3UZtHZMJX8T5XT+TFdCxsPyWs=
github.com/hashicorp/go-retryablehttp v0.6.8/go.mod h1:vAew36LZh98gCBJNLH42IQ1ER/9wtLZZ8meHqQvEYWY=
github.com/hashicorp/go-syslog v1.0.0/go.mod h1:qPfqrKkXGihmCqbJM2mZgkZGvKG1dFdvsLplgctolz4=
github.com/hashicorp/golang-lru v0.0.0-20180201235237-0fb14efe8c47/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
Expand Down Expand Up @@ -654,6 +660,8 @@ github.com/vmihailenco/msgpack/v5 v5.0.0-beta.5/go.mod h1:MPECSZPg8yittBek5Gq2Mh
github.com/vmihailenco/msgpack/v5 v5.1.0/go.mod h1:C5gboKD0TJPqWDTVTtrQNfRbiBwHZGo8UTqP/9/XvLI=
github.com/vmihailenco/tagparser v0.1.2/go.mod h1:OeAg3pn3UbLjkWt+rN9oFYB6u/cQgqMEUPoW2WPyhdI=
github.com/vmware/govmomi v0.20.3/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU=
github.com/xanzy/go-gitlab v0.50.0 h1:t7IoYTrnLSbdEZN7d8X/5zcr+ZM4TZQ2mXa8MqWlAZQ=
github.com/xanzy/go-gitlab v0.50.0/go.mod h1:Q+hQhV508bDPoBijv7YjK/Lvlb4PhVhJdKqXVQrUoAE=
github.com/xanzy/ssh-agent v0.2.1 h1:TCbipTQL2JiiCprBWx9frJ2eJlCYT00NmctrHxVAr70=
github.com/xanzy/ssh-agent v0.2.1/go.mod h1:mLlQY/MoOhWBj+gOGMQkOeiEvkx+8pJSI+0Bx9h2kr4=
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
Expand Down Expand Up @@ -786,10 +794,12 @@ golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81R
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20201006153459-a7d1128ccaa0/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201022231255-08b38378de70/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201024042810-be3efd7ff127 h1:pZPp9+iYUqwYKLjht0SDBbRCRK/9gAXDy7pz5fRDpjo=
golang.org/x/net v0.0.0-20201024042810-be3efd7ff127/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20181106182150-f42d05182288/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
Expand Down Expand Up @@ -964,6 +974,7 @@ google.golang.org/api v0.28.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0M
google.golang.org/api v0.29.0/go.mod h1:Lcubydp8VUV7KeIHD9z2Bys/sm/vGKnG1UHuDBSrHWM=
google.golang.org/api v0.30.0/go.mod h1:QGmEvQ87FHZNiUVJkT14jQNYJ4ZJjdRF23ZXz5138Fc=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.3.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
Expand Down
76 changes: 72 additions & 4 deletions manifests/crds/argoproj.io_applicationsets.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3748,8 +3748,8 @@ spec:
all protocols.
type: string
filters:
description: TODO other providers. Filters for
which repos should be considered.
description: Filters for which repos should be
considered.
items:
description: SCMProviderGeneratorFilter is a
single repository filter. If multiple filter
Expand Down Expand Up @@ -3805,6 +3805,41 @@ spec:
required:
- organization
type: object
gitlab:
description: SCMProviderGeneratorGitlab defines
a connection info specific to Gitlab.
properties:
allBranches:
description: Scan all branches instead of
just the default branch.
type: boolean
api:
description: The Gitlab API URL to talk to.
type: string
group:
description: Gitlab group to scan. Required. You
can use either the project id (recommended)
or the full namespaced path.
type: string
includeSubgroups:
description: Recurse through subgroups (true)
or scan only the base group (false). Defaults
to "false"
type: boolean
tokenRef:
description: Authentication token reference.
properties:
key:
type: string
secretName:
type: string
required:
- key
- secretName
type: object
required:
- group
type: object
requeueAfterSeconds:
description: Standard parameters.
format: int64
Expand Down Expand Up @@ -4646,8 +4681,7 @@ spec:
necessarily support all protocols.
type: string
filters:
description: TODO other providers. Filters for which repos
should be considered.
description: Filters for which repos should be considered.
items:
description: SCMProviderGeneratorFilter is a single repository
filter. If multiple filter types are set on a single
Expand Down Expand Up @@ -4700,6 +4734,40 @@ spec:
required:
- organization
type: object
gitlab:
description: SCMProviderGeneratorGitlab defines a connection
info specific to Gitlab.
properties:
allBranches:
description: Scan all branches instead of just the default
branch.
type: boolean
api:
description: The Gitlab API URL to talk to.
type: string
group:
description: Gitlab group to scan. Required. You can
use either the project id (recommended) or the full
namespaced path.
type: string
includeSubgroups:
description: Recurse through subgroups (true) or scan
only the base group (false). Defaults to "false"
type: boolean
tokenRef:
description: Authentication token reference.
properties:
key:
type: string
secretName:
type: string
required:
- key
- secretName
type: object
required:
- group
type: object
requeueAfterSeconds:
description: Standard parameters.
format: int64
Expand Down
Loading