Skip to content

Collect cloud volume metrics from AWS #620

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
samuelvl wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Collect cloud volume metrics from AWS #620

samuelvl wants to merge 2 commits into from

Conversation

@samuelvl
Copy link
Contributor

@samuelvl samuelvl commented Jan 20, 2026 •
edited
Loading

Collect and send metrics about AWS volumes. The controller needs to be configured with the following values:

controller:
  enabled: true
  extraArgs:
    log-level: debug
    cloud-provider: aws
    cloud-provider-aws-region: eu-central-1
    cloud-provider-storage-sync-enabled: true
    cloud-provider-storage-sync-interval: 30s
agent:
  enabled: true
  extraArgs:
    log-level: debug
    storage-stats-enabled: true

Example output for node ip-192-168-91-132.eu-central-1.compute.internal:

time=2026-01-21T09:54:46.375Z level=info source=storage_pipeline.go:137 msg="collected 3 cloud volume metrics" component=ctrl
  [0] Node: ip-192-168-91-132.eu-central-1.compute.internal, Provider: aws, VolumeID: vol-0df38487238ae02dc, Type: gp3, State: in-use, Size: 21474836480 bytes, IOPS: 3000, Throughput: 131072000 bytes/s, Encrypted: false
  [1] Node: ip-192-168-91-132.eu-central-1.compute.internal, Provider: aws, VolumeID: vol-0f1d36a1c66c32e3f, Type: gp2, State: in-use, Size: 10737418240 bytes, IOPS: 100, Throughput: 0 bytes/s, Encrypted: false
  [2] Node: ip-192-168-91-132.eu-central-1.compute.internal, Provider: aws, VolumeID: vol-0ea980516ca5c632d, Type: gp2, State: in-use, Size: 10737418240 bytes, IOPS: 100, Throughput: 0 bytes/s, Encrypted: false

@samuelvl samuelvl force-pushed the collect-aws-cloud-volumes branch 2 times, most recently from b824b16 to 3cdad7c Compare January 21, 2026 09:57
@samuelvl samuelvl marked this pull request as ready for review January 21, 2026 09:57
@samuelvl samuelvl force-pushed the collect-aws-cloud-volumes branch from 3cdad7c to d937fd7 Compare January 21, 2026 10:19
samuelvl
samuelvl commented Jan 21, 2026
View reviewed changes
@samuelvl samuelvl force-pushed the collect-aws-cloud-volumes branch 4 times, most recently from fc08852 to dfa4e9c Compare January 21, 2026 13:42
patrickpichler
patrickpichler reviewed Jan 26, 2026
View reviewed changes
cmd/controller/controllers/volume_state_controller.go Outdated
instanceIDToNodeName[instanceID] = node.Name
}

err := c.cloudProvider.RefreshStorageState(ctx, instanceIDs...)
Copy link
Contributor

@patrickpichler patrickpichler Jan 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I see RefreshStorageState and GetStorageState are always called together. Do we really need two method for this?

Copy link
Contributor Author

@samuelvl samuelvl Jan 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplified to only use GetStorageState

patrickpichler
patrickpichler reviewed Jan 26, 2026
View reviewed changes
api/v1/kube/kube_api.proto

message CloudVolumeInfo {
string cloud_provider = 1;
string zone = 2;
Copy link
Contributor

@patrickpichler patrickpichler Jan 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we also care about the region?

Copy link
Contributor Author

@samuelvl samuelvl Jan 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The region is not returned in the API response and we would need to derive it from the zone so I would leave the zone only for now

patrickpichler
patrickpichler reviewed Jan 26, 2026
View reviewed changes
pkg/cloudprovider/aws/test/integration_test.go Outdated
}

// TestRefreshStorageState calls RefreshStorageState and prints the results.
func TestRefreshStorageState(t *testing.T) {
Copy link
Contributor

@patrickpichler patrickpichler Jan 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this test always run?

Copy link
Contributor Author

@samuelvl samuelvl Jan 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally we should have e2e tests covering CSP functionality but we don't have those yet. These integration test are meant to be used only for local testing for now

patrickpichler
patrickpichler reviewed Jan 26, 2026
View reviewed changes
@samuelvl samuelvl force-pushed the collect-aws-cloud-volumes branch from dfa4e9c to e848e4f Compare January 26, 2026 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@patrickpichler patrickpichler patrickpichler left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@samuelvl @patrickpichler