Value of 0 is invalid

[thesmith]

I'm using the 0.3.3 through the dropwizard - cloudwatch integration, but it looks like the code to filter out empty data is the same across versions.

I'm recieving the following in my logs:

ERROR [2015-10-22 04:06:24,815] com.blacklocus.metrics.CloudWatchReporter: Exception reporting metrics to CloudWatch. The data sent in this CloudWatch API request may have been discarded.
! com.amazonaws.services.cloudwatch.model.InvalidParameterValueException: The value 0 for parameter MetricData.member.9.Value is invalid.
! at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
! at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
! at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
! at com.amazonaws.http.AmazonHttp...

I'm assuming that the filter should also be checking that input.getValue() > 0 as well as sample count?

[thesmith]

Clearly I need to write a way to replicate this.

According to Netflix, this could be because of a really small value: https://github.com/Netflix/servo/blob/master/servo-aws/src/test/java/com/netflix/servo/publish/cloudwatch/CloudWatchValueTest.java

[dirkraft]

0 values are allowed by CloudWatch since recording a 0 (versus recording no data point at all) can be significant.

I tried to tease out what this could be with the aws cli (aws cloudwatch put-metric-data ...), but since 0 values are allowed, I'm not sure what the actual problem is.

Having said all that this made me realize that if you wanted to record 0s (for counting behavior), you currently cannot, because the CloudWatchReporter omits 0 values to save money on API requests. So I'm opening an issue to make that configurable. #14

[dirkraft]

For reference, 0 values are allowed (via aws cli)

as a single data point

aws cloudwatch put-metric-data --namespace test --metric-data '[{"MetricName":"small_value","Value": 0}]'

as a statistic set

aws cloudwatch put-metric-data --namespace test --metric-data '[{"MetricName":"small_value","StatisticValues": {"SampleCount": 1, "Sum": 0, "Minimum": 0, "Maximum": 0}}]'

Data samples of 0 is not allowed, which is expected. That case is already handled by current CloudWatchReporter

aws cloudwatch put-metric-data --namespace test --metric-data '[{"MetricName":"small_value","StatisticValues": {"SampleCount": 0, "Sum": 1, "Minimum": 1, "Maximum": 1}}]'

[thesmith]

Ok, using the Netflix link above, I was able to replicate the error once the positive small numbers got close to 0:

aws cloudwatch put-metric-data --namespace test --metric-data '[{"MetricName":"small_value","Value": 2.1289799200040754E-109}]'

You get that value by multiplying 1 by 0.5, keep multiplying current result by 0.5 another 360 times.

Evidently some of my metrics emit close enough to 0 numbers that AWS doesn't count them as valid.

I guess what we should do in this situation is just force the number to 0 if the number is sufficiently close to 0?

[ben-manes]

I'm seeing this as well.

InvalidParameterValueException: The value 0 for parameter MetricData.member.16.Value is invalid.

ERROR [2017-07-12 18:18:36,502] [com.blacklocus.metrics.CloudWatchReporter] Exception reporting metrics to CloudWatch. The data in this CloudWatch API request may have been discarded, did not make it to CloudWatch.
com.amazonaws.services.cloudwatch.model.InvalidParameterValueException: The value 0 for parameter MetricData.member.16.Value is invalid. (Service: AmazonCloudWatch; Status Code: 400; Error Code: InvalidParameterValue; Request ID: 858dd20f-672e-11e7-a6a7-33f4295acab5)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
	at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.doInvoke(AmazonCloudWatchClient.java:1073)
	at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.invoke(AmazonCloudWatchClient.java:1049)
	at com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.executePutMetricData(AmazonCloudWatchClient.java:957)
	at com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient$10.call(AmazonCloudWatchAsyncClient.java:642)
	at com.amazonaws.services.cloudwatch.AmazonCloudWatchAsyncClient$10.call(AmazonCloudWatchAsyncClient.java:636)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	... 3 common frames omitted