RSDK-13144: Bidirectional grpc arbitrary metadata passing

[aldenh-viam]

Refresher on metadata passing in both directions: https://github.com/grpc/grpc-go/blob/master/Documentation/grpc-metadata.md#sending-and-receiving-metadata---client-side

Server-to-client (pass arbitrary metadata via headers):

• Rename existing ContextWithMetadataUnaryClientInterceptor to ContextWithMetadataServerToClientUnaryClientInterceptor
• Extend to allow passing metadata from from server to client across modules.

Client-to-server (pass arbitrary metadata via context):

• add contextutils.AppendToOutgoingContext helper that records the arbitrary keys to avoid mixing with internal (grpc or viam) keys.
• Add interceptor that forwards incoming context containing the arbitrary metadata to outgoing context, which grpc does not do by default for security reasons.

I will add these for streaming interceptors next. It should look very similar.

Example

I have a robot with 2 modules and a RobotClient that calls DoCommand on module 1, which calls GetImages on module 2.

The RobotClient sets arbitrary metadata:

k1: [v1, v2]
k2: [v3]
k3: [v4, v5]
k4: [v6}
k4-bin: [[]byte{0x03, 0x04}]  // this gets encoded to base64 and back as needed

module 1 sets:

outgoing:
  module1-k: [module1-v]
  k3: [v7, v8]
  k4: [v9]

return (metadata to send back to caller):
  m1: [m1]

module 2 sets

outgoing:
  none

return:
  m2: [m2, m3]

---

Module 1 sees the arbitrary metadata from RobotClient & module 2

from RobotClient to fwd to module2
  k1: [v1, v2]
  k2: [v3]
  k3: [v4, v5]
  k4: [v6]
  k4-bin: [[]byte{0x03, 0x04}] 

from module2 to fwd to RC
  m2: [m2, m3]

Module 2 sees the arbitrary metadata from RobotClient & module1

from RobotClient, module1
  k1: [v1, v2]
  k2: [v3]
  k3: [v4, v5, v7, v8]
  k4: [v6, v9]
  k4-bin: [[]byte{0x03, 0x04}] 
  module1-k: [module1-v]

RobotClient sees the arbitrary metadata from modules 1 & 2

  m1: [m1]
  m2: [m2, m3]

Keys will be merged as needed.

The above will also work for viam-server built-ins, such as if I updated the fake sensor's GetReadings to return some arbitrary metadata.

These interceptors will run on every GRPC hop, so it will come with a non-zero performance hit, but it should be barely noticeable with 0 or very few arbitrary metadata.

cc @viamrobotics/netcode

[aldenh-viam]

This upgrades every context to the ContextWithMetadata, which is akin to the current behavior: server writes its arbitrary metadata (if any) to headers regardless and client only decides whether to read it. The client choosing to not read it does not help save bandwidth.

To make this forwarding optional depending on whether the client wants the metadata or not, we'd need another metadata key, something like VIAM_FORWARD_METADATA=T/F

[cheukt]

gonna untag myself and tag @danielbotros

[aldenh-viam]

This is technically a breaking change, in that the original ContextWithMetadataUnaryClientInterceptor would forward all headers (including opid and SafetyMonitoredResourceMetadataKey), while the updated one only forwards the user-provided ones (registered automatically if you use contextutils.SetHeader instead of grpc.SetHeader). However, given that the original only worked inside viam-server (not across modules), and the only usages are in utils/contextutils/context_test.go and components/camera/client_test.go I expect this change to be safe.