Skip to content

BigTable: Invalid start_time for gauge metrics in Cloud Monitoring export #2757

@josefalshorji

Description

@josefalshorji

Environment details

  1. API: BigTable Java Client
  2. OS type and version: Linux (GKE environment)
  3. Java version: 25
  4. bigtable version: 2.66.0

Steps to reproduce

  1. Deploy a Java application using google-cloud-bigtable 2.66.0 in GKE
  2. Grant the service account roles/monitoring.metricWriter IAM permission
  3. Application attempts to export Bigtable client metrics automatically
  4. Error occurs due to invalid timestamp intervals for gauge metrics

Code example

// Standard Bigtable client initialization
// No special configuration required to reproduce

Stack trace

com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: Field points[0].interval.start_time had an invalid value: The start time must be equal to the end time for the gauge metric 'bigtable.googleapis.com/internal/client/grpc/subchannel/open_connections'.
	at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:92)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98)
	at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
	at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1125)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
	at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1004)
	at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:767)
	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:516)
	at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:668)
	at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:638)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:569)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at com.google.api.gax.grpc.GrpcLoggingInterceptor$1$1.onClose(GrpcLoggingInterceptor.java:98)
	at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:487)
	at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:451)
	at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:484)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:565)
	at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:733)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:714)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
	at java.base/java.lang.Thread.run(Thread.java:1474)
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written
	at io.grpc.Status.asRuntimeException(Status.java:532)
	... 21 common frames omitted

Any additional information below

Issue: The Bigtable Java client (v2.66.0) incorrectly sets different start_time and end_time values for gauge metrics when exporting to Google Cloud Monitoring. According to the Cloud Monitoring API, gauge metrics must have start_time equal to end_time.

Expected: For gauge metrics like bigtable.googleapis.com/internal/client/grpc/subchannel/open_connections, start_time should equal end_time.

Actual: The client sets different timestamps, causing INVALID_ARGUMENT errors.

Questions:

  1. Has this been fixed in versions newer than 2.66.0?
  2. What is the recommended approach to export Bigtable metrics correctly?

Metadata

Metadata

Assignees

No one assigned

    Labels

    api: bigtableIssues related to the googleapis/java-bigtable API.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions