diff --git a/docs/rip-2-least-privilege.md b/docs/rip-2-least-privilege.md new file mode 100644 index 00000000000..057065bc957 --- /dev/null +++ b/docs/rip-2-least-privilege.md @@ -0,0 +1,121 @@ +# RIP-2 Proxy Admin — Least Privilege Configuration Guide + +The RIP-2 admin surface authorizes every RPC against dedicated `proxy.admin.*` +ACL 2.0 resources (see `docs/rip-2-proxy-admin.md`, decision D2). This guide +shows the minimum-permission policy for each operational role. + +## 1. Resource & Action Model + +Resources (ACL 2.0 keys; modeled as cluster-typed literals with reserved names): + +| Resource key | Protects | +|---|---| +| `cluster:proxy.admin.client` | online client query & client diagnostics | +| `cluster:proxy.admin.config` | runtime config query & hot update | +| `cluster:proxy.admin.connection` | kick/disconnect clients, telemetry commands (HIGH) | +| `cluster:proxy.admin.quota` | quota query & adjustment (adjust = HIGH) | +| `cluster:proxy.admin.route` | route topology & route event stream | +| `cluster:proxy.admin.ops` | broker-facing ops: stats/topic status/message query (read) and reset offset / delete subscription / admin send (HIGH) | + +Action classes: + +- Read-only: `Get`, `List` +- High privilege (mutating / disruptive): `Update`, `Delete`, `Pub` + +The server maps every RPC to exactly one (resource, action) pair; granting a +read-only action can never authorize a high-privilege RPC. + +## 2. Role Templates + +All commands run against any broker/namesrv of the cluster (ACL 2.0 storage). +Create users first: + +```bash +sh mqadmin createUser -n -u -p +``` + +### Role A — Read-only observer (dashboard service account) + +Online clients, subscriptions, accumulation, diagnostics, config/route views. + +```bash +sh mqadmin updateAcl -n \ + -s user:rip2-ro \ + -r cluster:proxy.admin.client,cluster:proxy.admin.config,cluster:proxy.admin.quota,cluster:proxy.admin.route,cluster:proxy.admin.ops \ + -a Get,List \ + -d Allow +``` + +Note: `Get,List` on `proxy.admin.ops` covers the read-only broker-facing RPCs; +the mutating ops RPCs require `Update`/`Delete`/`Pub` and stay denied. + +### Role B — On-call operator (observer + connection control) + +Role A plus the ability to kick misbehaving clients. + +```bash +sh mqadmin updateAcl -n \ + -s user:rip2-oncall \ + -r cluster:proxy.admin.connection \ + -a Update \ + -d Allow +# plus the Role A grant above +``` + +### Role C — Admin (full control, break-glass) + +Config hot update, quota adjustment, offset reset, subscription deletion, +admin send. + +```bash +sh mqadmin updateAcl -n \ + -s user:rip2-admin \ + -r cluster:proxy.admin.client,cluster:proxy.admin.config,cluster:proxy.admin.connection,cluster:proxy.admin.quota,cluster:proxy.admin.route,cluster:proxy.admin.ops \ + -a Get,List,Update,Delete,Pub \ + -d Allow +``` + +(Keep Role C accounts to a minimum; every use is recorded in the auth audit +log with the `[PROXY-ADMIN-AUDIT]` prefix.) + +### Environment restriction (recommended) + +Restrict admin access to the operations network via the `-i` sourceIp option: + +```bash +sh mqadmin updateAcl -n \ + -s user:rip2-ro \ + -r cluster:proxy.admin.client \ + -a Get,List \ + -d Allow \ + -i 10.0.0.0/8 +``` + +## 3. Fail-Closed Mode + +By default the admin server follows the cluster-wide authentication switch +(same behavior as the data plane). To require credentials unconditionally: + +```properties +# proxy.json / -D proxyAdminRequireAuth=true +proxyAdminRequireAuth: true +``` + +With `proxyAdminRequireAuth=true` and the cluster authentication disabled, all +admin requests are rejected (fail-closed) — use this when the admin port cannot +be network-isolated. + +## 4. Disabling the Surface + +```properties +proxyAdminEnabled: false # admin gRPC server is not started at all +``` + +or set `adminGrpcPort` to 0 / negative. + +## 5. Audit + +Every served admin RPC logs subject (Console login user / AK), method, resource, +action and source IP to the auth audit logger; denied requests are logged by the +ACL 2.0 engine itself. This satisfies the four-tuple audit requirement +(Console user + AK + resource + operation). diff --git a/docs/rip-2-proxy-admin.md b/docs/rip-2-proxy-admin.md new file mode 100644 index 00000000000..01f9654674f --- /dev/null +++ b/docs/rip-2-proxy-admin.md @@ -0,0 +1,197 @@ +# RIP-2: Proxy Admin Standardized Management Interface + +## 1. Motivation + +RocketMQ 5.0 moved client access behind the stateless Proxy, but operations still +observe clients through broker-side structures (`ConsumerManager` on the broker, +Remoting-era admin commands). gRPC clients attached to a Proxy are invisible to +those tools: the control plane cannot answer "which SDK clients are online, what +do they subscribe to, are they healthy" without indirect metrics heuristics. + +RIP-1 (Control Plane 5.0 dashboard) requirement CLIENT-01 explicitly depends on a +standard server-side interface to read complete gRPC client data. RIP-2 defines +and implements that interface on the Proxy itself. + +## 2. Goals + +1. A dedicated, independent gRPC Admin service on the Proxy, isolated from the + data-plane `MessagingService`. +2. A stable, backward-compatible proto contract (`ProxyAdminService` in + `apache/rocketmq/v2/admin.proto`, rocketmq-apis). +3. First-class authorization under dedicated `proxy.admin.*` ACL 2.0 resources + with read-only / high-privilege action separation. +4. The service exposes its own call RT and error-rate metrics. +5. Multi-Proxy semantics: a documented, predictable story for cluster-wide views. + +Non-goals (this iteration): broker-side quota storage, Remoting client kick +(Remoting clients remain observable via existing broker channels; the service +reports gRPC clients, proto carries a `protocol` field for future D5 coverage). + +## 3. Design Decisions + +### D1 — Service placement + +Dedicated `ProxyAdminService` (Option B), separate from both the data-plane +`MessagingService` and the broker-facing `Admin` service. Served on its own gRPC +server/port (`adminGrpcPort`, default 8083) with its own interceptor chain. A +global kill switch `proxyAdminEnabled` disables the whole surface. + +Rationale: control-plane traffic must never contend with the data plane, and the +admin port can be firewalled to the operations network only. The admin server +intentionally does NOT expose channelz or proto reflection. + +### D2 — Authorization: dedicated `proxy.admin.*` resources + +Credentials arrive in the standard gRPC `Authorization` metadata (same scheme as +the data plane, ACL 2.0 signature). Every RPC maps to one resource + one action: + +| Resource | RPCs | Actions | +|---|---|---| +| `proxy.admin.client` | ListClients / ListClientsByGroup / ListClientsByTopic | List | +| `proxy.admin.client` | DescribeClient / DescribePopReceiptHandles / DescribeBatchConsumeDiagnostics / ListSubscription / DescribeSubscription / ListConsumerConnection / DescribeGroupAccumulation / GetConsumerRunningInfo / QueryTimeSpan | Get | +| `proxy.admin.config` | DescribeProxyConfig | Get | +| `proxy.admin.config` | UpdateProxyConfig / ChangeLogLevel | Update | +| `proxy.admin.connection` | KickClient / DisconnectChannel / PrintThreadStackTrace / VerifyMessage | Update (high privilege) | +| `proxy.admin.quota` | DescribeQuota | Get | +| `proxy.admin.quota` | UpdateQuota | Update (high privilege) | +| `proxy.admin.route` | DescribeRouteTopology / GetTopicRoute | Get | +| `proxy.admin.route` | SubscribeRouteEvents | List | +| `proxy.admin.ops` | GetProxyRuntimeStats / DescribeTopicStatus / QueryMessage | Get | +| `proxy.admin.ops` | ResetGroupOffset | Update (high privilege) | +| `proxy.admin.ops` | DeleteSubscription | Delete (high privilege) | +| `proxy.admin.ops` | AdminSendMessage | Pub (high privilege) | + +Modeling note: ACL 2.0 resource types are cluster/namespace/topic/group. The +admin resources are modeled as CLUSTER-typed literals with reserved names +(resource keys `cluster:proxy.admin.`), which yields exact least- +privilege matching without colliding with real cluster names and without +changing the auth core. + +Modes: +- cluster auth disabled, `proxyAdminRequireAuth=false` → open surface (same + semantics as the data plane); +- cluster auth enabled → standard authenticate + authorize pipeline; +- `proxyAdminRequireAuth=true` → fail-closed: requests without verifiable + credentials are rejected even if the cluster-wide switch is off. + +Audit: every served RPC writes `[PROXY-ADMIN-AUDIT] subject/method/resource/ +action/sourceIp` to the auth audit logger (satisfies the Console-user + AK + +resource + operation audit tuple requirement together with the ACL 2.0 engine's +own audit log). + +### D3 — Multi-proxy semantics + +Each Proxy returns its LOCAL view, tagged with `proxy_endpoint` + a monotonic +`epoch`, so a consumer can always attribute and deduplicate results. + +Cluster-wide view: `ListClientsRequest.scope = PROXY_SCOPE_ALL_PROXIES`. When +`proxyAdminPeerEndpoints` (host:port of peer admin servers) is configured, the +serving proxy fans the query out to all peers in parallel, merges the local +views and dedups by `client_id` (a client is attached to exactly one proxy at a +time; local view wins on duplicates). Peer failures degrade gracefully: an +unreachable peer is skipped with a warning, the merged view of the remaining +nodes is returned. When no peers are configured, the request is served from the +local view — the scheme therefore satisfies the spec's "local view OR cluster +aggregation" requirement with both options available, chosen per request. + +Rationale for not building a central registry: proxies are stateless and have no +membership service; peer-list configuration is explicit, auditable, and matches +how operators already firewall the admin port. + +### D4 — Pagination + +Cursor-based `next_token` for client listings. The cursor is the clientId-sorted +position (base64-opaque) of the last returned element: page boundaries stay +stable while clients connect/disconnect between calls — offset pagination would +shift or duplicate entries under churn and does not scale to very large +connection counts. Diagnostic snapshots (pop handles, batch diagnostics) use +offset pagination (`page_num`/`page_size`, max 100) because they are bounded, +point-in-time views. + +### D5 — Protocol coverage + +`ClientInstance.protocol` distinguishes GRPC vs REMOTING. This iteration tracks +gRPC clients (the proxy's `GrpcChannelManager` is the authority); the field +keeps the contract forward-compatible for Remoting coverage. + +## 4. Proto Contract (rocketmq-apis, `apache/rocketmq/v2/admin.proto`) + +`service ProxyAdminService` — 14 RPCs: + +M1 (required by RIP-1 CLIENT-01): +- `ListClients(ClientFilter, page_size, next_token, ProxyScope)` +- `DescribeClient(client_id)` → ClientDetail (instance, settings, subscriptions, + recent_heartbeats, auth_status, consume_progress, network_info) +- `ListClientsByGroup(group, ...)` +- `ListClientsByTopic(topic, ...)` + +M2: +- `DescribeProxyConfig()` / `UpdateProxyConfig(ProxyRuntimeConfig)` → changed_fields +- `KickClient(client_id, reason)` / `DisconnectChannel(channel_id, reason)` +- `DescribeQuota(...)` / `UpdateQuota(QuotaPolicy)` +- `DescribePopReceiptHandles(group[, topic], page)` → summary + handles + (renew/renewRetry counts, nextVisibleTime, invisibleTime, expired flag, lock owner) +- `DescribeBatchConsumeDiagnostics(group[, topic][, client_id], page)` → + per-client unacked/renew/expired aggregates + group summary +- `SubscribeRouteEvents(topics, event_types)` → server-streaming + (ROUTE_SNAPSHOT / TOPIC_CREATE / TOPIC_DELETE / QUEUE_SCALE / BROKER_ONLINE / + BROKER_OFFLINE), replaying current snapshots on subscribe +- `DescribeRouteTopology([topic])` → proxy→broker links + per-broker queue load + +Compatibility rules: additive-only field evolution, all new fields optional, +`ProxyScope` defaults to local, no field number reuse. + +Building the proto artifact: `org.apache.rocketmq:rocketmq-proto:2.3.0` is +generated from the `rocketmq-apis` repository (branch +`feature/rip-2-proxy-admin-grpc`), which carries the `ProxyAdminService` +contract and a self-contained Maven build (`mvn clean install` in the +rocketmq-apis checkout; protoc and grpc-java plugins come from Maven Central). +The apis repository is consumed as a local development dependency and is +intentionally NOT vendored or submoduled into this repository; CI/developers +install the artifact into their local repository once, then build this repo +normally. + +## 5. Observability + +The admin server exports (OpenTelemetry, honoring the proxy's metrics exporter +configuration): + +- `rocketmq_proxy_admin_rpc_total{rpc_method, status=success|error, error_type?}` + — error rate = rate(status="error") +- `rocketmq_proxy_admin_rpc_latency{rpc_method, status}` (ms histogram) — RT P50/P99 + +Transport-level failures (auth rejections, permission denials) and business +failures are both counted as errors; successful RPCs are counted once. + +## 6. Configuration Reference + +| Key | Default | Meaning | +|---|---|---| +| `proxyAdminEnabled` | true | D2 kill switch; false = admin server not started | +| `adminGrpcPort` | 8083 | dedicated admin gRPC port (<=0 disables) | +| `proxyAdminRequireAuth` | false | fail-closed credential enforcement | +| `proxyAdminPeerEndpoints` | [] | peer admin endpoints for D3 ALL_PROXIES fan-out | +| `proxyAdminPeerTimeoutMillis` | 3000 | per-peer fan-out timeout | +| `proxyAdminHeartbeatHistorySize` | 16 | heartbeat records kept per client | + +## 7. Milestones + +- M1 (this delivery): online client query — ListClients / DescribeClient / + ByGroup / ByTopic with stable cursor pagination and cluster fan-out. +- M2 (this delivery): runtime config hot update, connection control (kick / + disconnect), quota visualization & adjustment, route observation + (SubscribeRouteEvents streaming + DescribeRouteTopology). +- M3/M4 (this delivery): POP receipt-handle diagnostics and batch consumption + diagnostics from the proxy's own receipt-handle tracking. +- Future: Remoting client coverage under the same contract (D5), broker-side + quota storage integration. + +## 8. Acceptance Criteria Mapping + +| Criterion | Status | +|---|---| +| RIP document + stable backward-compatible proto contract | this document + rocketmq-apis `admin.proto` | +| Client query RPCs merged; pagination scales with connection churn | D4 stable cursor; page cost O(pageSize) after sort | +| Independent ACL control, read-only/high-risk separation, least-privilege doc | D2 resources/actions + `docs/rip-2-least-privilege.md` | +| RPC RT & error-rate metrics | §5 instruments | +| E2E with RIP-1 dashboard | contract frozen for dashboard CLIENT-01 integration (cross-repo) | diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/ProxyStartup.java b/proxy/src/main/java/org/apache/rocketmq/proxy/ProxyStartup.java index 1b38a19ae6a..eb8bb75303e 100644 --- a/proxy/src/main/java/org/apache/rocketmq/proxy/ProxyStartup.java +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/ProxyStartup.java @@ -39,7 +39,17 @@ import org.apache.rocketmq.proxy.config.ProxyConfig; import org.apache.rocketmq.proxy.grpc.GrpcServer; import org.apache.rocketmq.proxy.grpc.GrpcServerBuilder; +import org.apache.rocketmq.proxy.grpc.admin.ProxyAdminAuthInterceptor; +import org.apache.rocketmq.proxy.grpc.admin.ProxyAdminGrpcService; +import org.apache.rocketmq.proxy.grpc.admin.ProxyAdminMetricsInterceptor; +import org.apache.rocketmq.proxy.grpc.admin.ProxyAdminMetricsManager; +import org.apache.rocketmq.proxy.grpc.admin.ProxyAdminPeerClient; +import org.apache.rocketmq.proxy.grpc.admin.ProxyAdminServiceGrpcService; +import org.apache.rocketmq.proxy.grpc.admin.RouteChangeNotifier; +import org.apache.rocketmq.proxy.grpc.v2.DefaultGrpcMessagingActivity; import org.apache.rocketmq.proxy.grpc.v2.GrpcMessagingApplication; +import org.apache.rocketmq.proxy.grpc.v2.channel.GrpcChannelManager; +import org.apache.rocketmq.proxy.grpc.v2.common.GrpcClientSettingsManager; import org.apache.rocketmq.proxy.metrics.ProxyMetricsManager; import org.apache.rocketmq.proxy.processor.DefaultMessagingProcessor; import org.apache.rocketmq.proxy.processor.MessagingProcessor; @@ -81,10 +91,13 @@ public static void main(String[] args) { TlsCertificateManager tlsCertificateManager = new TlsCertificateManager(); PROXY_START_AND_SHUTDOWN.appendStartAndShutdown(tlsCertificateManager); - // create grpcServer + // create grpcServer (data plane). Capture the application reference so the + // RIP-2 admin server can reuse the SAME GrpcChannelManager / GrpcClientSettingsManager + // that the data plane uses to track online clients. + GrpcMessagingApplication dataPlaneApplication = createServiceProcessor(messagingProcessor); GrpcServer grpcServer = GrpcServerBuilder.newBuilder(executor, ConfigurationManager.getProxyConfig().getGrpcServerPort(), tlsCertificateManager) - .addService(createServiceProcessor(messagingProcessor)) + .addService(dataPlaneApplication) .addService(ChannelzService.newInstance(100)) .addService(ProtoReflectionService.newInstance()) .configInterceptor() @@ -92,6 +105,58 @@ public static void main(String[] args) { .build(); PROXY_START_AND_SHUTDOWN.appendStartAndShutdown(grpcServer); + // RIP-2: dedicated admin gRPC server (control plane). It MUST reuse the data plane's + // shared GrpcChannelManager, otherwise online clients connected to the data plane would + // never be visible to the admin queries (listClients / describeClient / ... would return + // an always-empty, isolated manager). The whole surface is gated by the D2 kill switch + // proxyAdminEnabled; the admin port intentionally does NOT expose channelz/proto + // reflection (control-plane attack surface is kept minimal). + Integer adminPort = ConfigurationManager.getProxyConfig().getAdminGrpcPort(); + if (ConfigurationManager.getProxyConfig().isProxyAdminEnabled() + && adminPort != null && adminPort > 0) { + DefaultGrpcMessagingActivity dataPlaneActivity = + (DefaultGrpcMessagingActivity) dataPlaneApplication.getGrpcMessagingActivity(); + GrpcChannelManager sharedChannelManager = dataPlaneActivity.getGrpcChannelManager(); + GrpcClientSettingsManager sharedSettingsManager = dataPlaneActivity.getGrpcClientSettingsManager(); + DefaultMessagingProcessor defaultProcessor = (DefaultMessagingProcessor) messagingProcessor; + + // D3 cluster aggregation (peer fan-out) and route observation, both participate + // in the proxy start/shutdown lifecycle. + ProxyAdminPeerClient peerClient = new ProxyAdminPeerClient(ConfigurationManager.getAuthConfig()); + PROXY_START_AND_SHUTDOWN.appendStartAndShutdown(peerClient); + RouteChangeNotifier routeChangeNotifier = new RouteChangeNotifier(); + defaultProcessor.getServiceManager().getTopicRouteService().addRouteRefreshListener(routeChangeNotifier); + PROXY_START_AND_SHUTDOWN.appendStartAndShutdown(routeChangeNotifier); + + // Acceptance criteria #4: the admin surface reports its own RT & error rate. + ProxyAdminMetricsManager.init(ConfigurationManager.getProxyConfig()); + + ProxyAdminGrpcService adminService = new ProxyAdminGrpcService( + defaultProcessor.getServiceManager(), + messagingProcessor, + sharedChannelManager, + sharedSettingsManager); + ProxyAdminServiceGrpcService proxyAdminService = new ProxyAdminServiceGrpcService( + defaultProcessor.getServiceManager(), + defaultProcessor, + sharedChannelManager, + sharedSettingsManager, + peerClient, + routeChangeNotifier); + GrpcServer adminGrpcServer = GrpcServerBuilder.newBuilder(executor, adminPort, tlsCertificateManager) + .addService(adminService) + .addService(proxyAdminService) + .configInterceptor() + // interceptor execution order: metrics (outermost) -> auth -> standard pipeline + .appendInterceptor(new ProxyAdminAuthInterceptor( + ConfigurationManager.getAuthConfig(), messagingProcessor)) + .appendInterceptor(new ProxyAdminMetricsInterceptor()) + .shutdownTime(ConfigurationManager.getProxyConfig().getGrpcShutdownTimeSeconds(), TimeUnit.SECONDS) + .build(); + PROXY_START_AND_SHUTDOWN.appendStartAndShutdown(adminGrpcServer); + log.info("RIP-2 admin gRPC server will start on port {}", adminPort); + } + RemotingProtocolServer remotingServer = new RemotingProtocolServer(messagingProcessor, tlsCertificateManager); PROXY_START_AND_SHUTDOWN.appendStartAndShutdown(remotingServer); diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/config/ProxyConfig.java b/proxy/src/main/java/org/apache/rocketmq/proxy/config/ProxyConfig.java index a7896c11e07..a82277bc882 100644 --- a/proxy/src/main/java/org/apache/rocketmq/proxy/config/ProxyConfig.java +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/config/ProxyConfig.java @@ -20,6 +20,7 @@ import java.net.InetAddress; import java.net.UnknownHostException; import java.time.Duration; +import java.util.ArrayList; import java.util.Comparator; import java.util.HashMap; import java.util.List; @@ -89,6 +90,38 @@ public class ProxyConfig implements ConfigFile { */ private String proxyMode = ProxyMode.CLUSTER.name(); private Integer grpcServerPort = 8081; + /** + * Dedicated gRPC port for the RIP-2 Proxy Admin service. When > 0 the proxy + * starts an independent admin gRPC server (separate ACL scope, isolated + * traffic) in addition to the data-plane gRPC server. + */ + private Integer adminGrpcPort = 8083; + /** + * RIP-2 D2 global kill switch for the Proxy Admin surface. When false the + * admin gRPC server is not started at all, regardless of {@link #adminGrpcPort}. + */ + private boolean proxyAdminEnabled = true; + /** + * When true the admin server enforces credential checks even if the cluster-wide + * authentication switch is off; requests without verifiable credentials are rejected + * (fail-closed mode). When false the admin server follows the cluster-wide + * authenticationEnabled/authorizationEnabled switches (same behavior as the data plane). + */ + private boolean proxyAdminRequireAuth = false; + /** + * Peer proxy admin endpoints (host:port of the peer's admin gRPC server) used for the + * PROXY_SCOPE_ALL_PROXIES cluster-wide aggregation (RIP-2 D3). Empty means the proxy only + * serves its local view. + */ + private List proxyAdminPeerEndpoints = new ArrayList<>(); + /** + * Timeout in milliseconds for fan-out queries to peer proxy admin endpoints. + */ + private long proxyAdminPeerTimeoutMillis = 3000L; + /** + * Number of recent heartbeat records kept per client channel for DescribeClient. + */ + private int proxyAdminHeartbeatHistorySize = 16; private long grpcShutdownTimeSeconds = 30; private int grpcBossLoopNum = 1; private int grpcWorkerLoopNum = PROCESSOR_NUMBER * 2; @@ -483,6 +516,54 @@ public void setGrpcServerPort(Integer grpcServerPort) { this.grpcServerPort = grpcServerPort; } + public Integer getAdminGrpcPort() { + return adminGrpcPort; + } + + public void setAdminGrpcPort(Integer adminGrpcPort) { + this.adminGrpcPort = adminGrpcPort; + } + + public boolean isProxyAdminEnabled() { + return proxyAdminEnabled; + } + + public void setProxyAdminEnabled(boolean proxyAdminEnabled) { + this.proxyAdminEnabled = proxyAdminEnabled; + } + + public boolean isProxyAdminRequireAuth() { + return proxyAdminRequireAuth; + } + + public void setProxyAdminRequireAuth(boolean proxyAdminRequireAuth) { + this.proxyAdminRequireAuth = proxyAdminRequireAuth; + } + + public List getProxyAdminPeerEndpoints() { + return proxyAdminPeerEndpoints; + } + + public void setProxyAdminPeerEndpoints(List proxyAdminPeerEndpoints) { + this.proxyAdminPeerEndpoints = proxyAdminPeerEndpoints; + } + + public long getProxyAdminPeerTimeoutMillis() { + return proxyAdminPeerTimeoutMillis; + } + + public void setProxyAdminPeerTimeoutMillis(long proxyAdminPeerTimeoutMillis) { + this.proxyAdminPeerTimeoutMillis = proxyAdminPeerTimeoutMillis; + } + + public int getProxyAdminHeartbeatHistorySize() { + return proxyAdminHeartbeatHistorySize; + } + + public void setProxyAdminHeartbeatHistorySize(int proxyAdminHeartbeatHistorySize) { + this.proxyAdminHeartbeatHistorySize = proxyAdminHeartbeatHistorySize; + } + public long getGrpcShutdownTimeSeconds() { return grpcShutdownTimeSeconds; } diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/AdminModelConverter.java b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/AdminModelConverter.java new file mode 100644 index 00000000000..4af5b028bf7 --- /dev/null +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/AdminModelConverter.java @@ -0,0 +1,332 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.rocketmq.proxy.grpc.admin; + +import apache.rocketmq.v2.Broker; +import apache.rocketmq.v2.Code; +import apache.rocketmq.v2.DescribeGroupAccumulationResponse; +import apache.rocketmq.v2.DescribeTopicStatusResponse; +import apache.rocketmq.v2.GetTopicRouteResponse; +import apache.rocketmq.v2.Message; +import apache.rocketmq.v2.MessageType; +import apache.rocketmq.v2.QueryTimeSpanResponse; +import apache.rocketmq.v2.Resource; +import apache.rocketmq.v2.Status; +import apache.rocketmq.v2.SystemProperties; +import com.alibaba.fastjson.JSON; +import com.google.protobuf.ByteString; +import java.util.ArrayList; +import java.util.List; +import org.apache.rocketmq.common.message.MessageExt; +import org.apache.rocketmq.proxy.service.admin.AdminService; +import org.apache.rocketmq.proxy.service.route.AddressableMessageQueue; +import org.apache.rocketmq.proxy.service.route.MessageQueueView; +import org.apache.rocketmq.remoting.protocol.admin.ConsumeStats; +import org.apache.rocketmq.remoting.protocol.admin.OffsetWrapper; +import org.apache.rocketmq.remoting.protocol.route.TopicRouteData; +import org.apache.rocketmq.remoting.protocol.statictopic.TopicConfigAndQueueMapping; + +/** + * Bridge that translates between the broker's internal wire types + * ({@code org.apache.rocketmq.remoting.*}) and the RIP-2 gRPC protocol + * ({@code apache.rocketmq.v2.*}, generated from rocketmq-apis). + * + *

This is the ONLY class that imports both worlds. The gRPC admin service + * ({@link ProxyAdminGrpcService}) stays protocol-pure (v2 only); the broker + * gateway ({@code DefaultAdminService}) stays remoting-pure. Keeping the + * conversion here ensures neither layer leaks into the other. + */ +final class AdminModelConverter { + + private AdminModelConverter() { + } + + private static Status ok() { + return Status.newBuilder().setCode(Code.OK).build(); + } + + static DescribeGroupAccumulationResponse.GroupAccumulation toGroupAccumulation( + AdminService adminService, String brokerAddr, String group, String topic, long timeoutMillis) throws Exception { + ConsumeStats consumeStats = adminService.fetchConsumeStats(brokerAddr, group, topic, timeoutMillis); + long total = 0; + if (consumeStats != null && consumeStats.getOffsetTable() != null) { + for (OffsetWrapper wrapper : consumeStats.getOffsetTable().values()) { + long diff = wrapper.getBrokerOffset() - wrapper.getConsumerOffset(); + if (diff > 0) { + total += diff; + } + } + } + return DescribeGroupAccumulationResponse.GroupAccumulation.newBuilder() + .setAccumulation(total) + .setReadyMessages(total) + .build(); + } + + /** + * RIP-2 fix: aggregate accumulation across ALL brokers hosting the topic, deduplicating + * by message queue, instead of only querying the first broker of the route. + */ + static DescribeGroupAccumulationResponse.GroupAccumulation toGroupAccumulationMultiBroker( + AdminService adminService, java.util.Collection brokerAddrs, String group, String topic, + long timeoutMillis) throws Exception { + long total = 0; + java.util.Set counted = new java.util.HashSet<>(); + for (String brokerAddr : brokerAddrs) { + ConsumeStats consumeStats = adminService.fetchConsumeStats(brokerAddr, group, topic, timeoutMillis); + if (consumeStats == null || consumeStats.getOffsetTable() == null) { + continue; + } + for (java.util.Map.Entry entry : + consumeStats.getOffsetTable().entrySet()) { + if (!counted.add(entry.getKey())) { + continue; + } + long diff = entry.getValue().getBrokerOffset() - entry.getValue().getConsumerOffset(); + if (diff > 0) { + total += diff; + } + } + } + return DescribeGroupAccumulationResponse.GroupAccumulation.newBuilder() + .setAccumulation(total) + .setReadyMessages(total) + .build(); + } + + static QueryTimeSpanResponse toQueryTimeSpan( + AdminService adminService, String brokerAddr, String group, String topic, + MessageQueueView mqv, long timeoutMillis) throws Exception { + ConsumeStats consumeStats = adminService.fetchConsumeStats(brokerAddr, group, topic, timeoutMillis); + QueryTimeSpanResponse.Builder builder = QueryTimeSpanResponse.newBuilder().setStatus(ok()); + if (mqv != null && mqv.getReadSelector() != null) { + for (AddressableMessageQueue mq : mqv.getReadSelector().getQueues()) { + String queueBrokerAddr = mq.getBrokerAddr(); + if (queueBrokerAddr == null || queueBrokerAddr.isEmpty()) { + queueBrokerAddr = brokerAddr; + } + long minStoretime = adminService.getEarliestMsgStoretime(queueBrokerAddr, mq, timeoutMillis); + QueryTimeSpanResponse.QueueTimeSpan.Builder span = QueryTimeSpanResponse.QueueTimeSpan.newBuilder() + .setMessageQueue(toMessageQueue(mq)) + .setMinTimestamp(minStoretime); + // RIP-2 fix: report the real last consume timestamp from the offset table + // (fallback to minStoretime when the queue has never been consumed). + OffsetWrapper wrapper = consumeStats != null && consumeStats.getOffsetTable() != null + ? consumeStats.getOffsetTable().get(mq) : null; + if (wrapper != null && wrapper.getLastTimestamp() > 0) { + span.setConsumeTimestamp(wrapper.getLastTimestamp()); + } else { + span.setConsumeTimestamp(minStoretime); + } + builder.addQueueTimeSpanList(span.build()); + } + } + return builder.build(); + } + + static GetTopicRouteResponse toTopicRoute(AdminService adminService, String topic) throws Exception { + TopicRouteData topicRouteData = adminService.getTopicRouteData(topic); + String json = topicRouteData == null ? "{}" : JSON.toJSONString(topicRouteData); + return GetTopicRouteResponse.newBuilder() + .setStatus(ok()) + .setTopicRouteData(json) + .build(); + } + + static DescribeTopicStatusResponse toTopicStatus(AdminService adminService, String brokerAddr, String topic, + long timeoutMillis) throws Exception { + TopicConfigAndQueueMapping topicConfig = adminService.getTopicConfig(brokerAddr, topic, timeoutMillis); + DescribeTopicStatusResponse.Builder builder = DescribeTopicStatusResponse.newBuilder().setStatus(ok()); + if (topicConfig != null) { + builder.setTopicMessageType(MessageType.MESSAGE_TYPE_UNSPECIFIED); + builder.setDescription("topic=" + topicConfig.getTopicName() + + " readQueues=" + topicConfig.getReadQueueNums() + + " writeQueues=" + topicConfig.getWriteQueueNums() + + " perm=" + topicConfig.getPerm()); + } + return builder.build(); + } + + static Message toMessage(MessageExt ext) { + if (ext == null) { + return null; + } + List keys = new ArrayList<>(); + if (ext.getKeys() != null && !ext.getKeys().isEmpty()) { + for (String k : ext.getKeys().split("\\s+")) { + if (!k.isEmpty()) { + keys.add(k); + } + } + } + SystemProperties.Builder sp = SystemProperties.newBuilder() + .setMessageId(ext.getMsgId() == null ? "" : ext.getMsgId()) + .setTag(ext.getTags() == null ? "" : ext.getTags()); + if (!keys.isEmpty()) { + sp.addAllKeys(keys); + } + return Message.newBuilder() + .setTopic(Resource.newBuilder().setName(ext.getTopic()).build()) + .setSystemProperties(sp.build()) + .setBody(ByteString.copyFrom(ext.getBody() == null ? new byte[0] : ext.getBody())) + .build(); + } + + static apache.rocketmq.v2.MessageQueue toMessageQueue(org.apache.rocketmq.common.message.MessageQueue mq) { + return apache.rocketmq.v2.MessageQueue.newBuilder() + .setTopic(Resource.newBuilder().setName(mq.getTopic()).build()) + .setBroker(Broker.newBuilder().setName(mq.getBrokerName()).build()) + .setId(mq.getQueueId()) + .build(); + } + + /** + * RIP-2 route observation: translate the broker-internal route data into the v2 proto + * snapshot consumed by SubscribeRouteEvents / DescribeRouteTopology. Returns null when + * the route is empty (topic absent on the NameServer). + */ + static apache.rocketmq.v2.TopicRouteSnapshot toTopicRouteSnapshot(String topic, TopicRouteData data) { + if (data == null || data.getBrokerDatas() == null || data.getBrokerDatas().isEmpty()) { + return null; + } + apache.rocketmq.v2.TopicRouteSnapshot.Builder builder = + apache.rocketmq.v2.TopicRouteSnapshot.newBuilder().setTopic(topic); + for (org.apache.rocketmq.remoting.protocol.route.BrokerData brokerData : data.getBrokerDatas()) { + apache.rocketmq.v2.BrokerInfo.Builder brokerInfo = apache.rocketmq.v2.BrokerInfo.newBuilder() + .setCluster(brokerData.getCluster() == null ? "" : brokerData.getCluster()) + .setBrokerName(brokerData.getBrokerName() == null ? "" : brokerData.getBrokerName()); + if (brokerData.getBrokerAddrs() != null) { + brokerData.getBrokerAddrs().forEach((id, addr) -> { + if (id != null && addr != null) { + brokerInfo.putBrokerAddrs(id, addr); + } + }); + } + builder.addBrokers(brokerInfo); + } + if (data.getQueueDatas() != null) { + for (org.apache.rocketmq.remoting.protocol.route.QueueData queueData : data.getQueueDatas()) { + builder.addQueues(apache.rocketmq.v2.QueueInfo.newBuilder() + .setBrokerName(queueData.getBrokerName() == null ? "" : queueData.getBrokerName()) + .setReadQueueNums(queueData.getReadQueueNums()) + .setWriteQueueNums(queueData.getWriteQueueNums()) + .setPerm(toPermission(queueData.getPerm())) + .build()); + } + } + return builder.build(); + } + + private static apache.rocketmq.v2.Permission toPermission(int perm) { + // org.apache.rocketmq.common.constant.PermName bit layout: read=4, write=2, inherit=1 + boolean readable = (perm & 4) == 4; + boolean writable = (perm & 2) == 2; + if (readable && writable) { + return apache.rocketmq.v2.Permission.READ_WRITE; + } + if (readable) { + return apache.rocketmq.v2.Permission.READ; + } + if (writable) { + return apache.rocketmq.v2.Permission.WRITE; + } + return apache.rocketmq.v2.Permission.NONE; + } + + /** + * RIP-2 DescribeClient consume progress: compute the aggregated lag of one subscribed + * topic across ALL brokers hosting it, deduplicating by message queue. Returns -1 when + * the route or the consume stats are unavailable. + */ + static long computeTopicLag(AdminService adminService, MessageQueueView view, String group, String topic, + long timeoutMillis) { + if (view == null || view.getReadSelector() == null || view.getReadSelector().getQueues().isEmpty()) { + return -1; + } + java.util.Set brokerAddrs = new java.util.LinkedHashSet<>(); + for (AddressableMessageQueue queue : view.getReadSelector().getQueues()) { + if (queue.getBrokerAddr() != null && !queue.getBrokerAddr().isEmpty()) { + brokerAddrs.add(queue.getBrokerAddr()); + } + } + long lag = 0; + boolean anyStats = false; + java.util.Set counted = new java.util.HashSet<>(); + for (String brokerAddr : brokerAddrs) { + ConsumeStats stats; + try { + stats = adminService.fetchConsumeStats(brokerAddr, group, topic, timeoutMillis); + } catch (Throwable t) { + continue; + } + if (stats == null || stats.getOffsetTable() == null) { + continue; + } + anyStats = true; + for (java.util.Map.Entry entry : + stats.getOffsetTable().entrySet()) { + if (!counted.add(entry.getKey())) { + continue; + } + long brokerOffset = entry.getValue().getBrokerOffset(); + long consumerOffset = entry.getValue().getConsumerOffset(); + if (brokerOffset >= consumerOffset) { + lag += brokerOffset - consumerOffset; + } + } + } + return anyStats ? lag : -1; + } + + /** + * RIP-2 DescribeRouteTopology: append proxy-to-broker links and per-broker load rows for + * one cached topic route. + */ + static void addRouteTopology(apache.rocketmq.v2.DescribeRouteTopologyResponse.Builder builder, String topic, + TopicRouteData data, String proxyEndpoint, int activeConnections) { + if (data == null) { + return; + } + if (data.getBrokerDatas() != null) { + for (org.apache.rocketmq.remoting.protocol.route.BrokerData brokerData : data.getBrokerDatas()) { + if (brokerData.getBrokerAddrs() == null) { + continue; + } + for (String brokerAddr : brokerData.getBrokerAddrs().values()) { + builder.addLinks(apache.rocketmq.v2.ProxyBrokerLink.newBuilder() + .setProxyEndpoint(proxyEndpoint) + .setBrokerName(brokerData.getBrokerName() == null ? "" : brokerData.getBrokerName()) + .setBrokerAddress(brokerAddr == null ? "" : brokerAddr) + .setHealthy(true) + .setActiveConnections(activeConnections) + .build()); + } + } + } + if (data.getQueueDatas() != null) { + for (org.apache.rocketmq.remoting.protocol.route.QueueData queueData : data.getQueueDatas()) { + builder.addLoad(apache.rocketmq.v2.LoadBalanceInfo.newBuilder() + .setBrokerName(queueData.getBrokerName() == null ? "" : queueData.getBrokerName()) + .setReadQueueNums(queueData.getReadQueueNums()) + .setWriteQueueNums(queueData.getWriteQueueNums()) + .setCurrentLoad(0) + .setRegionAffinity(false) + .build()); + } + } + } +} diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminAuthInterceptor.java b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminAuthInterceptor.java new file mode 100644 index 00000000000..655ccf8e2a8 --- /dev/null +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminAuthInterceptor.java @@ -0,0 +1,280 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.rocketmq.proxy.grpc.admin; + +import io.grpc.Grpc; +import io.grpc.Metadata; +import io.grpc.ServerCall; +import io.grpc.ServerCallHandler; +import io.grpc.ServerInterceptor; +import io.grpc.Status; +import java.net.InetSocketAddress; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import org.apache.commons.lang3.StringUtils; +import org.apache.rocketmq.auth.authentication.context.DefaultAuthenticationContext; +import org.apache.rocketmq.auth.authentication.exception.AuthenticationException; +import org.apache.rocketmq.auth.authentication.factory.AuthenticationFactory; +import org.apache.rocketmq.auth.authentication.model.User; +import org.apache.rocketmq.auth.authorization.AuthorizationEvaluator; +import org.apache.rocketmq.auth.authentication.AuthenticationEvaluator; +import org.apache.rocketmq.auth.authorization.context.DefaultAuthorizationContext; +import org.apache.rocketmq.auth.authorization.exception.AuthorizationException; +import org.apache.rocketmq.auth.authorization.factory.AuthorizationFactory; +import org.apache.rocketmq.auth.authorization.model.Resource; +import org.apache.rocketmq.auth.config.AuthConfig; +import org.apache.rocketmq.common.action.Action; +import org.apache.rocketmq.common.constant.LoggerName; +import org.apache.rocketmq.common.resource.ResourcePattern; +import org.apache.rocketmq.common.resource.ResourceType; +import org.apache.rocketmq.logging.org.slf4j.Logger; +import org.apache.rocketmq.logging.org.slf4j.LoggerFactory; +import org.apache.rocketmq.proxy.config.ConfigurationManager; +import org.apache.rocketmq.proxy.config.ProxyConfig; +import org.apache.rocketmq.proxy.processor.MessagingProcessor; + +/** + * RIP-2 D2 authorization interceptor for the dedicated Proxy Admin gRPC server. + * + *

Every admin RPC is bound to a dedicated ACL 2.0 resource under the + * {@code proxy.admin.*} namespace with a distinct action, giving true + * read-only / high-privilege isolation on top of the standard ACL 2.0 policy + * engine: + * + * 

+ *   proxy.admin.client      ListClients / DescribeClient / ByGroup / ByTopic / diagnostics (GET/LIST)
+ *   proxy.admin.config      DescribeProxyConfig (GET) / UpdateProxyConfig (UPDATE)
+ *   proxy.admin.connection  KickClient / DisconnectChannel (UPDATE, high privilege)
+ *   proxy.admin.quota       DescribeQuota (GET) / UpdateQuota (UPDATE, high privilege)
+ *   proxy.admin.route       DescribeRouteTopology (GET) / SubscribeRouteEvents (LIST)
+ *   proxy.admin.ops         broker-facing operations served by the Admin service
+ *                           (GET/LIST for queries; UPDATE/DELETE/PUB for mutations)
+ *
+ * + *

Resources are modeled as {@code CLUSTER}-typed literal resources + * (resource key {@code cluster:proxy.admin.}) because the ACL 2.0 + * resource model only defines cluster/namespace/topic/group types; a + * cluster-typed literal gives exact least-privilege matching without + * colliding with real cluster names. + * + *

Behavior modes: + *

    + *
  • Cluster auth disabled and {@code proxyAdminRequireAuth=false}: the + * admin surface is open (same semantics as the data plane).
    • + *
  • Cluster auth enabled: requests are authenticated from the standard + * {@code Authorization} gRPC metadata and authorized against the + * per-method {@code proxy.admin.*} resource, exactly like the data + * plane does for topic/group resources.
    • + *
  • {@code proxyAdminRequireAuth=true}: fail-closed mode. Requests + * without verifiable credentials are rejected even if the cluster-wide + * authentication switch is off.
    • + *
+ */ +public class ProxyAdminAuthInterceptor implements ServerInterceptor { + + private static final Logger log = LoggerFactory.getLogger(LoggerName.PROXY_LOGGER_NAME); + private static final Logger logAudit = LoggerFactory.getLogger(LoggerName.ROCKETMQ_AUTH_AUDIT_LOGGER_NAME); + + public static final String RESOURCE_CLIENT = "proxy.admin.client"; + public static final String RESOURCE_CONFIG = "proxy.admin.config"; + public static final String RESOURCE_CONNECTION = "proxy.admin.connection"; + public static final String RESOURCE_QUOTA = "proxy.admin.quota"; + public static final String RESOURCE_ROUTE = "proxy.admin.route"; + public static final String RESOURCE_OPS = "proxy.admin.ops"; + + private static final Map METHOD_PERMISSIONS = new HashMap<>(); + + static { + // ProxyAdminService: M1 online client query. + METHOD_PERMISSIONS.put("ListClients", new ResourceAction(RESOURCE_CLIENT, Action.LIST)); + METHOD_PERMISSIONS.put("ListClientsByGroup", new ResourceAction(RESOURCE_CLIENT, Action.LIST)); + METHOD_PERMISSIONS.put("ListClientsByTopic", new ResourceAction(RESOURCE_CLIENT, Action.LIST)); + METHOD_PERMISSIONS.put("DescribeClient", new ResourceAction(RESOURCE_CLIENT, Action.GET)); + // ProxyAdminService: M2 runtime config & connection control. + METHOD_PERMISSIONS.put("DescribeProxyConfig", new ResourceAction(RESOURCE_CONFIG, Action.GET)); + METHOD_PERMISSIONS.put("UpdateProxyConfig", new ResourceAction(RESOURCE_CONFIG, Action.UPDATE)); + METHOD_PERMISSIONS.put("KickClient", new ResourceAction(RESOURCE_CONNECTION, Action.UPDATE)); + METHOD_PERMISSIONS.put("DisconnectChannel", new ResourceAction(RESOURCE_CONNECTION, Action.UPDATE)); + // ProxyAdminService: M2 quota visualization & controlled adjustment. + METHOD_PERMISSIONS.put("DescribeQuota", new ResourceAction(RESOURCE_QUOTA, Action.GET)); + METHOD_PERMISSIONS.put("UpdateQuota", new ResourceAction(RESOURCE_QUOTA, Action.UPDATE)); + // ProxyAdminService: M3/M4 POP & batch consume diagnostics. + METHOD_PERMISSIONS.put("DescribePopReceiptHandles", new ResourceAction(RESOURCE_CLIENT, Action.GET)); + METHOD_PERMISSIONS.put("DescribeBatchConsumeDiagnostics", new ResourceAction(RESOURCE_CLIENT, Action.GET)); + // ProxyAdminService: route observation. + METHOD_PERMISSIONS.put("SubscribeRouteEvents", new ResourceAction(RESOURCE_ROUTE, Action.LIST)); + METHOD_PERMISSIONS.put("DescribeRouteTopology", new ResourceAction(RESOURCE_ROUTE, Action.GET)); + + // Admin service (broker-facing operations, also served on the admin server). + METHOD_PERMISSIONS.put("GetProxyRuntimeStats", new ResourceAction(RESOURCE_OPS, Action.GET)); + METHOD_PERMISSIONS.put("GetTopicRoute", new ResourceAction(RESOURCE_ROUTE, Action.GET)); + METHOD_PERMISSIONS.put("DescribeTopicStatus", new ResourceAction(RESOURCE_OPS, Action.GET)); + METHOD_PERMISSIONS.put("ListSubscription", new ResourceAction(RESOURCE_CLIENT, Action.LIST)); + METHOD_PERMISSIONS.put("DescribeSubscription", new ResourceAction(RESOURCE_CLIENT, Action.GET)); + METHOD_PERMISSIONS.put("ListConsumerConnection", new ResourceAction(RESOURCE_CLIENT, Action.LIST)); + METHOD_PERMISSIONS.put("DescribeGroupAccumulation", new ResourceAction(RESOURCE_CLIENT, Action.GET)); + METHOD_PERMISSIONS.put("GetConsumerRunningInfo", new ResourceAction(RESOURCE_CLIENT, Action.GET)); + METHOD_PERMISSIONS.put("QueryTimeSpan", new ResourceAction(RESOURCE_CLIENT, Action.GET)); + METHOD_PERMISSIONS.put("QueryMessage", new ResourceAction(RESOURCE_OPS, Action.GET)); + METHOD_PERMISSIONS.put("ChangeLogLevel", new ResourceAction(RESOURCE_CONFIG, Action.UPDATE)); + // High-privilege mutations: strictly separated from the read-only actions above. + METHOD_PERMISSIONS.put("DeleteSubscription", new ResourceAction(RESOURCE_OPS, Action.DELETE)); + METHOD_PERMISSIONS.put("ResetGroupOffset", new ResourceAction(RESOURCE_OPS, Action.UPDATE)); + METHOD_PERMISSIONS.put("AdminSendMessage", new ResourceAction(RESOURCE_OPS, Action.PUB)); + METHOD_PERMISSIONS.put("PrintThreadStackTrace", new ResourceAction(RESOURCE_CONNECTION, Action.UPDATE)); + METHOD_PERMISSIONS.put("VerifyMessage", new ResourceAction(RESOURCE_CONNECTION, Action.UPDATE)); + } + + private final AuthConfig authConfig; + private final AuthenticationEvaluator authenticationEvaluator; + private final AuthorizationEvaluator authorizationEvaluator; + + public ProxyAdminAuthInterceptor(AuthConfig authConfig, MessagingProcessor messagingProcessor) { + this.authConfig = authConfig; + this.authenticationEvaluator = AuthenticationFactory.getEvaluator(authConfig, + messagingProcessor::getMetadataService); + this.authorizationEvaluator = AuthorizationFactory.getEvaluator(authConfig, + messagingProcessor::getMetadataService); + } + + @Override + public ServerCall.Listener interceptCall(ServerCall call, Metadata headers, + ServerCallHandler next) { + String method = call.getMethodDescriptor().getBareMethodName(); + long startNanos = System.nanoTime(); + try { + ProxyConfig proxyConfig = ConfigurationManager.getProxyConfig(); + boolean requireAuth = proxyConfig != null && proxyConfig.isProxyAdminRequireAuth(); + boolean authnEnabled = authConfig != null && authConfig.isAuthenticationEnabled(); + boolean authzEnabled = authConfig != null && authConfig.isAuthorizationEnabled(); + + // Open mode: identical semantics to the data plane when cluster auth is off. + if (!requireAuth && !authnEnabled && !authzEnabled) { + return next.startCall(call, headers); + } + + // Fail-closed: credentials are demanded but cannot be verified at all. + if (requireAuth && !authnEnabled) { + call.close(Status.UNAUTHENTICATED.withDescription( + "proxyAdminRequireAuth is on but cluster authenticationEnabled is off; " + + "enable authentication before using the admin surface in fail-closed mode"), new Metadata()); + return noopListener(); + } + + String username = null; + if (authnEnabled || requireAuth) { + DefaultAuthenticationContext authenticationContext = buildAuthenticationContext(call, headers); + username = authenticationContext.getUsername(); + if (StringUtils.isBlank(username)) { + if (requireAuth) { + call.close(Status.UNAUTHENTICATED.withDescription("missing credentials for proxy admin"), + new Metadata()); + return noopListener(); + } + authenticationEvaluator.evaluate(authenticationContext); + } else { + authenticationEvaluator.evaluate(authenticationContext); + } + } + + ResourceAction resourceAction = METHOD_PERMISSIONS.get(method); + if (resourceAction != null && (authzEnabled || requireAuth)) { + if (StringUtils.isBlank(username)) { + call.close(Status.UNAUTHENTICATED.withDescription("missing credentials for proxy admin"), + new Metadata()); + return noopListener(); + } + DefaultAuthorizationContext authorizationContext = DefaultAuthorizationContext.of( + User.of(username), + Resource.of(ResourceType.CLUSTER, resourceAction.resource, ResourcePattern.LITERAL), + resourceAction.action, + resolveSourceIp(call)); + authorizationContext.setRpcCode(call.getMethodDescriptor().getFullMethodName()); + authorizationEvaluator.evaluate(Collections.singletonList(authorizationContext)); + } + + logAudit.info("[PROXY-ADMIN-AUDIT] subject = {} method = {} resource = {} action = {} sourceIp = {}", + StringUtils.isBlank(username) ? "anonymous" : username, method, + resourceAction == null ? "unmapped" : resourceAction.resource, + resourceAction == null ? "unknown" : resourceAction.action.getName(), + resolveSourceIp(call)); + return next.startCall(call, headers); + } catch (AuthenticationException e) { + ProxyAdminMetricsManager.recordError(method, (System.nanoTime() - startNanos) / 1_000_000L, e); + log.warn("RIP-2 admin authentication failed. method:{}, cause:{}", method, e.getMessage()); + call.close(Status.UNAUTHENTICATED.withDescription(e.getMessage()), new Metadata()); + return noopListener(); + } catch (AuthorizationException e) { + ProxyAdminMetricsManager.recordError(method, (System.nanoTime() - startNanos) / 1_000_000L, e); + log.warn("RIP-2 admin authorization denied. method:{}, cause:{}", method, e.getMessage()); + call.close(Status.PERMISSION_DENIED.withDescription(e.getMessage()), new Metadata()); + return noopListener(); + } catch (Throwable t) { + ProxyAdminMetricsManager.recordError(method, (System.nanoTime() - startNanos) / 1_000_000L, t); + log.error("RIP-2 admin auth interceptor error. method:{}", method, t); + call.close(Status.INTERNAL.withDescription(t.getMessage()), new Metadata()); + return noopListener(); + } + } + + private DefaultAuthenticationContext buildAuthenticationContext(ServerCall call, + Metadata headers) { + // The builder only uses the message for its descriptor name; pass the shared + // Status default instance and overwrite rpcCode with the real admin method. + Object context = AuthenticationFactory.newContext(authConfig, headers, + apache.rocketmq.v2.Status.getDefaultInstance()); + if (!(context instanceof DefaultAuthenticationContext)) { + throw new AuthenticationException("unsupported authentication context type for proxy admin"); + } + DefaultAuthenticationContext authenticationContext = (DefaultAuthenticationContext) context; + authenticationContext.setRpcCode(call.getMethodDescriptor().getFullMethodName()); + return authenticationContext; + } + + private static String resolveSourceIp(ServerCall call) { + try { + InetSocketAddress remoteAddress = (InetSocketAddress) call.getAttributes() + .get(Grpc.TRANSPORT_ATTR_REMOTE_ADDR); + if (remoteAddress != null && remoteAddress.getAddress() != null) { + return remoteAddress.getAddress().getHostAddress(); + } + } catch (Throwable ignore) { + // best-effort only + } + return ""; + } + + private static ServerCall.Listener noopListener() { + return new ServerCall.Listener() { + }; + } + + static ResourceAction resolveResourceAction(String method) { + return METHOD_PERMISSIONS.get(method); + } + + static final class ResourceAction { + final String resource; + final Action action; + + ResourceAction(String resource, Action action) { + this.resource = resource; + this.action = action; + } + } +} diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminConfigSupport.java b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminConfigSupport.java new file mode 100644 index 00000000000..75fad7a8beb --- /dev/null +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminConfigSupport.java @@ -0,0 +1,344 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.rocketmq.proxy.grpc.admin; + +import apache.rocketmq.v2.DescribeProxyConfigRequest; +import apache.rocketmq.v2.DescribeProxyConfigResponse; +import apache.rocketmq.v2.DescribeQuotaRequest; +import apache.rocketmq.v2.DescribeQuotaResponse; +import apache.rocketmq.v2.ProxyRuntimeConfig; +import apache.rocketmq.v2.QuotaDimension; +import apache.rocketmq.v2.QuotaPolicy; +import apache.rocketmq.v2.Resource; +import apache.rocketmq.v2.UpdateProxyConfigRequest; +import apache.rocketmq.v2.UpdateProxyConfigResponse; +import apache.rocketmq.v2.UpdateQuotaRequest; +import apache.rocketmq.v2.UpdateQuotaResponse; +import com.google.protobuf.Duration; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; +import org.apache.commons.lang3.StringUtils; +import org.apache.rocketmq.common.metrics.MetricsExporterType; +import org.apache.rocketmq.proxy.config.ConfigurationManager; +import org.apache.rocketmq.proxy.config.ProxyConfig; + +/** + * RIP-2 M2 runtime config query & hot update, plus quota visualization & + * controlled adjustment, served by {@link org.apache.rocketmq.proxy.grpc.admin.ProxyAdminServiceGrpcService}. + * + *

Hot update semantics: every field carried by {@code ProxyRuntimeConfig} is mapped onto + * the live {@link ProxyConfig} instance; the response reports the snake_case field names that + * actually changed. Fields whose effect needs a process restart (ports, TLS material) are + * still applied to the config object and reported, and additionally listed in the + * {@code restart_required} documentation of the RIP-2 proposal. + * + *

proto3 scalar default caveat: an UpdateProxyConfig request cannot express "set this + * numeric/string field to zero/empty" because default values are indistinguishable from + * absent fields. Callers should send partial configs containing only the fields to change. + */ +public class ProxyAdminConfigSupport { + + public static final String METRIC_MAX_MESSAGE_SIZE = "MAX_MESSAGE_SIZE"; + public static final String METRIC_GRPC_MAX_CONCURRENT_CALLS = "GRPC_MAX_CONCURRENT_CALLS_PER_CONNECTION"; + public static final String METRIC_GRPC_THREAD_POOL = "GRPC_THREAD_POOL_NUMS"; + + /** + * Quota policies managed by the proxy admin surface, keyed by + * {@code dimension:resource:metric}. Seeded from the live proxy config on first access; + * UpdateQuota mutates this registry and the backing ProxyConfig when a direct mapping + * exists. + */ + private final ConcurrentMap quotaRegistry = new ConcurrentHashMap<>(); + private volatile boolean seeded; + + // --------------------------------------------------------------------- + // runtime config + // --------------------------------------------------------------------- + + public DescribeProxyConfigResponse describeProxyConfig(DescribeProxyConfigRequest request, + apache.rocketmq.v2.Status ok) { + ProxyConfig config = ConfigurationManager.getProxyConfig(); + return DescribeProxyConfigResponse.newBuilder() + .setStatus(ok) + .setConfig(toProto(config)) + .build(); + } + + public UpdateProxyConfigResponse updateProxyConfig(UpdateProxyConfigRequest request, + apache.rocketmq.v2.Status ok) { + ProxyConfig config = ConfigurationManager.getProxyConfig(); + ProxyRuntimeConfig incoming = request.getConfig(); + List changedFields = new ArrayList<>(); + + if (!incoming.getProxyMode().isEmpty() && !incoming.getProxyMode().equals(config.getProxyMode())) { + config.setProxyMode(incoming.getProxyMode()); + changedFields.add("proxy_mode"); + } + if (!incoming.getRocketmqClusterName().isEmpty() + && !incoming.getRocketmqClusterName().equals(config.getRocketMQClusterName())) { + config.setRocketMQClusterName(incoming.getRocketmqClusterName()); + changedFields.add("rocketmq_cluster_name"); + } + if (!incoming.getProxyClusterName().isEmpty() + && !incoming.getProxyClusterName().equals(config.getProxyClusterName())) { + config.setProxyClusterName(incoming.getProxyClusterName()); + changedFields.add("proxy_cluster_name"); + } + if (!incoming.getProxyName().isEmpty() && !incoming.getProxyName().equals(config.getProxyName())) { + config.setProxyName(incoming.getProxyName()); + changedFields.add("proxy_name"); + } + if (!incoming.getLocalServeAddr().isEmpty() + && !incoming.getLocalServeAddr().equals(config.getLocalServeAddr())) { + config.setLocalServeAddr(incoming.getLocalServeAddr()); + changedFields.add("local_serve_addr"); + } + if (!incoming.getNamesrvAddr().isEmpty() && !incoming.getNamesrvAddr().equals(config.getNamesrvAddr())) { + config.setNamesrvAddr(incoming.getNamesrvAddr()); + changedFields.add("namesrv_addr"); + } + if (incoming.getGrpcServerPort() > 0 && incoming.getGrpcServerPort() != config.getGrpcServerPort()) { + config.setGrpcServerPort(incoming.getGrpcServerPort()); + changedFields.add("grpc_server_port"); + } + if (incoming.getGrpcThreadPoolNums() > 0 + && incoming.getGrpcThreadPoolNums() != config.getGrpcThreadPoolNums()) { + config.setGrpcThreadPoolNums(incoming.getGrpcThreadPoolNums()); + changedFields.add("grpc_thread_pool_nums"); + } + if (incoming.hasDefaultInvisibleTime()) { + long millis = toMillis(incoming.getDefaultInvisibleTime()); + if (millis > 0 && millis != config.getDefaultInvisibleTimeMills()) { + config.setDefaultInvisibleTimeMills(millis); + changedFields.add("default_invisible_time"); + } + } + if (incoming.hasMaxInvisibleTime()) { + long millis = toMillis(incoming.getMaxInvisibleTime()); + if (millis > 0 && millis != config.getMaxInvisibleTimeMills()) { + config.setMaxInvisibleTimeMills(millis); + changedFields.add("max_invisible_time"); + } + } + if (incoming.hasMinInvisibleTimeForRecv()) { + long millis = toMillis(incoming.getMinInvisibleTimeForRecv()); + if (millis > 0 && millis != config.getMinInvisibleTimeMillsForRecv()) { + config.setMinInvisibleTimeMillsForRecv(millis); + changedFields.add("min_invisible_time_for_recv"); + } + } + if (incoming.hasMaxDelayTime()) { + long millis = toMillis(incoming.getMaxDelayTime()); + if (millis > 0 && millis != config.getMaxDelayTimeMills()) { + config.setMaxDelayTimeMills(millis); + changedFields.add("max_delay_time"); + } + } + if (incoming.getMaxMessageSize() > 0 && incoming.getMaxMessageSize() != config.getMaxMessageSize()) { + config.setMaxMessageSize(incoming.getMaxMessageSize()); + changedFields.add("max_message_size"); + } + if (incoming.getTlsTestModeEnable() != config.isTlsTestModeEnable()) { + config.setTlsTestModeEnable(incoming.getTlsTestModeEnable()); + changedFields.add("tls_test_mode_enable"); + } + if (!incoming.getTlsKeyPath().isEmpty() && !incoming.getTlsKeyPath().equals(config.getTlsKeyPath())) { + config.setTlsKeyPath(incoming.getTlsKeyPath()); + changedFields.add("tls_key_path"); + } + if (!incoming.getTlsCertPath().isEmpty() && !incoming.getTlsCertPath().equals(config.getTlsCertPath())) { + config.setTlsCertPath(incoming.getTlsCertPath()); + changedFields.add("tls_cert_path"); + } + if (!incoming.getMetricsExporterType().isEmpty() + && !incoming.getMetricsExporterType().equalsIgnoreCase(String.valueOf(config.getMetricsExporterType()))) { + try { + config.setMetricsExporterType(MetricsExporterType.valueOf( + StringUtils.upperCase(incoming.getMetricsExporterType()))); + changedFields.add("metrics_exporter_type"); + } catch (IllegalArgumentException ignore) { + // unknown exporter type: keep current value + } + } + if (incoming.getMetricsPromExporterPort() > 0 + && incoming.getMetricsPromExporterPort() != config.getMetricsPromExporterPort()) { + config.setMetricsPromExporterPort(incoming.getMetricsPromExporterPort()); + changedFields.add("metrics_prom_exporter_port"); + } + if (incoming.getTraceOn() != config.isTraceOn()) { + config.setTraceOn(incoming.getTraceOn()); + changedFields.add("trace_on"); + } + if (incoming.getProxyAdminEnabled() != config.isProxyAdminEnabled()) { + config.setProxyAdminEnabled(incoming.getProxyAdminEnabled()); + changedFields.add("proxy_admin_enabled"); + } + if (incoming.getProxyAdminServerPort() > 0 + && incoming.getProxyAdminServerPort() != safeInt(config.getAdminGrpcPort())) { + config.setAdminGrpcPort(incoming.getProxyAdminServerPort()); + changedFields.add("proxy_admin_server_port"); + } + + return UpdateProxyConfigResponse.newBuilder() + .setStatus(ok) + .setConfig(toProto(ConfigurationManager.getProxyConfig())) + .addAllChangedFields(changedFields) + .build(); + } + + public ProxyRuntimeConfig toProto(ProxyConfig config) { + ProxyRuntimeConfig.Builder builder = ProxyRuntimeConfig.newBuilder() + .setProxyMode(StringUtils.defaultString(config.getProxyMode())) + .setRocketmqClusterName(StringUtils.defaultString(config.getRocketMQClusterName())) + .setProxyClusterName(StringUtils.defaultString(config.getProxyClusterName())) + .setProxyName(StringUtils.defaultString(config.getProxyName())) + .setLocalServeAddr(StringUtils.defaultString(config.getLocalServeAddr())) + .setNamesrvAddr(StringUtils.defaultString(config.getNamesrvAddr())) + .setGrpcServerPort(safeInt(config.getGrpcServerPort())) + .setGrpcThreadPoolNums(config.getGrpcThreadPoolNums()) + .setProxyAdminEnabled(config.isProxyAdminEnabled()) + .setProxyAdminServerPort(safeInt(config.getAdminGrpcPort())) + .setProxyAdminThreadPoolNums(0) + .setMaxMessageSize(config.getMaxMessageSize()) + .setDefaultInvisibleTime(durationMillis(config.getDefaultInvisibleTimeMills())) + .setMaxInvisibleTime(durationMillis(config.getMaxInvisibleTimeMills())) + .setMinInvisibleTimeForRecv(durationMillis(config.getMinInvisibleTimeMillsForRecv())) + .setMaxDelayTime(durationMillis(config.getMaxDelayTimeMills())) + .setTlsTestModeEnable(config.isTlsTestModeEnable()) + .setTlsKeyPath(StringUtils.defaultString(config.getTlsKeyPath())) + .setTlsCertPath(StringUtils.defaultString(config.getTlsCertPath())) + .setMetricsExporterType(String.valueOf(config.getMetricsExporterType())) + .setMetricsPromExporterPort(config.getMetricsPromExporterPort()) + .setTraceOn(config.isTraceOn()); + return builder.build(); + } + + // --------------------------------------------------------------------- + // quota visualization & controlled adjustment + // --------------------------------------------------------------------- + + public DescribeQuotaResponse describeQuota(DescribeQuotaRequest request, apache.rocketmq.v2.Status ok) { + seedRegistry(); + DescribeQuotaResponse.Builder builder = DescribeQuotaResponse.newBuilder().setStatus(ok); + for (QuotaPolicy policy : quotaRegistry.values()) { + if (request.getDimension() != QuotaDimension.QUOTA_DIMENSION_UNSPECIFIED + && policy.getDimension() != request.getDimension()) { + continue; + } + if (!request.getResource().getName().isEmpty() + && !request.getResource().getName().equals(policy.getResource().getName())) { + continue; + } + builder.addPolicies(policy); + } + return builder.build(); + } + + public UpdateQuotaResponse updateQuota(UpdateQuotaRequest request, apache.rocketmq.v2.Status ok, + apache.rocketmq.v2.Status badRequest) { + seedRegistry(); + QuotaPolicy policy = request.getPolicy(); + if (policy == null || policy.getLimit() <= 0 || StringUtils.isBlank(policy.getMetric())) { + return UpdateQuotaResponse.newBuilder() + .setStatus(badRequest) + .build(); + } + String key = quotaKey(policy.getDimension(), policy.getResource().getName(), policy.getMetric()); + QuotaPolicy.Builder updated = policy.toBuilder(); + // Apply to the live proxy config when the metric maps onto a real proxy control knob, + // so the adjustment takes effect immediately instead of being only bookkeeping. + ProxyConfig config = ConfigurationManager.getProxyConfig(); + if (METRIC_MAX_MESSAGE_SIZE.equals(policy.getMetric())) { + config.setMaxMessageSize((int) policy.getLimit()); + } else if (METRIC_GRPC_MAX_CONCURRENT_CALLS.equals(policy.getMetric())) { + config.setGrpcMaxConcurrentCallsPerConnection((int) policy.getLimit()); + } else if (METRIC_GRPC_THREAD_POOL.equals(policy.getMetric())) { + config.setGrpcThreadPoolNums((int) policy.getLimit()); + } + if (!updated.hasWindow()) { + updated.setWindow(Duration.newBuilder().setSeconds(60).build()); + } + quotaRegistry.put(key, updated.build()); + return UpdateQuotaResponse.newBuilder() + .setStatus(ok) + .setPolicy(updated.build()) + .build(); + } + + private void seedRegistry() { + if (seeded) { + return; + } + synchronized (this) { + if (seeded) { + return; + } + ProxyConfig config = ConfigurationManager.getProxyConfig(); + Duration window = Duration.newBuilder().setSeconds(60).build(); + putSeed(QuotaDimension.QUOTA_DIMENSION_TOPIC, "*", METRIC_MAX_MESSAGE_SIZE, config.getMaxMessageSize(), window); + putSeed(QuotaDimension.QUOTA_DIMENSION_GROUP, "*", METRIC_GRPC_MAX_CONCURRENT_CALLS, + config.getGrpcMaxConcurrentCallsPerConnection(), window); + putSeed(QuotaDimension.QUOTA_DIMENSION_GROUP, "*", METRIC_GRPC_THREAD_POOL, config.getGrpcThreadPoolNums(), window); + seeded = true; + } + } + + private void putSeed(QuotaDimension dimension, String resourceName, String metric, long limit, Duration window) { + QuotaPolicy policy = QuotaPolicy.newBuilder() + .setDimension(dimension) + .setResource(Resource.newBuilder().setName(resourceName).build()) + .setMetric(metric) + .setLimit(limit) + .setCurrentUsage(0) + .setRecentTriggerCount(0) + .setWindow(window) + .build(); + quotaRegistry.put(quotaKey(dimension, resourceName, metric), policy); + } + + private static String quotaKey(QuotaDimension dimension, String resourceName, String metric) { + return dimension.name() + ":" + StringUtils.defaultString(resourceName) + ":" + metric; + } + + private static long toMillis(Duration duration) { + return duration.getSeconds() * 1000L + duration.getNanos() / 1_000_000L; + } + + private static Duration durationMillis(long millis) { + return Duration.newBuilder() + .setSeconds(millis / 1000) + .setNanos((int) ((millis % 1000) * 1_000_000)) + .build(); + } + + private static int safeInt(Integer value) { + return value == null ? 0 : value; + } + + Map getQuotaRegistryView() { + seedRegistry(); + return new java.util.HashMap<>(quotaRegistry); + } + + List allQuotaPolicies() { + seedRegistry(); + return new ArrayList<>(quotaRegistry.values()); + } +} diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminDiagnosticsSupport.java b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminDiagnosticsSupport.java new file mode 100644 index 00000000000..f5be5a0c9c0 --- /dev/null +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminDiagnosticsSupport.java @@ -0,0 +1,296 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.rocketmq.proxy.grpc.admin; + +import apache.rocketmq.v2.BatchConsumeClientDiagnostics; +import apache.rocketmq.v2.BatchConsumeGroupSummary; +import apache.rocketmq.v2.DescribeBatchConsumeDiagnosticsRequest; +import apache.rocketmq.v2.DescribeBatchConsumeDiagnosticsResponse; +import apache.rocketmq.v2.DescribePopReceiptHandlesRequest; +import apache.rocketmq.v2.DescribePopReceiptHandlesResponse; +import apache.rocketmq.v2.MessageModel; +import apache.rocketmq.v2.PopLockView; +import apache.rocketmq.v2.PopReceiptHandleGroupSummary; +import apache.rocketmq.v2.PopReceiptHandleInfo; +import com.google.protobuf.Duration; +import com.google.protobuf.Timestamp; +import io.netty.channel.Channel; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import org.apache.rocketmq.common.consumer.ReceiptHandle; +import org.apache.rocketmq.proxy.common.MessageReceiptHandle; +import org.apache.rocketmq.proxy.common.ReceiptHandleGroupKey; +import org.apache.rocketmq.proxy.grpc.v2.channel.GrpcClientChannel; +import org.apache.rocketmq.proxy.processor.DefaultMessagingProcessor; +import org.apache.rocketmq.proxy.processor.ReceiptHandleProcessor; +import org.apache.rocketmq.proxy.service.receipt.DefaultReceiptHandleManager; + +/** + * RIP-2 M3/M4 diagnostics: POP receipt handle inspection and batch consumption + * diagnostics, computed from the proxy's own receipt handle tracking (the same + * state driving invisible-time renewal), so the data is always consistent with + * what the proxy actually holds for connected consumers. + */ +public class ProxyAdminDiagnosticsSupport { + + private static final int MAX_DIAG_PAGE_SIZE = 100; + + private final DefaultMessagingProcessor messagingProcessor; + + public ProxyAdminDiagnosticsSupport(DefaultMessagingProcessor messagingProcessor) { + this.messagingProcessor = messagingProcessor; + } + + // --------------------------------------------------------------------- + // M3: POP receipt handle diagnostics + // --------------------------------------------------------------------- + + public DescribePopReceiptHandlesResponse describePopReceiptHandles(DescribePopReceiptHandlesRequest request, + apache.rocketmq.v2.Status ok, apache.rocketmq.v2.Status badRequest) { + String group = request.getGroup(); + if (group == null || group.isEmpty()) { + return DescribePopReceiptHandlesResponse.newBuilder().setStatus(badRequest).build(); + } + String topicFilter = request.getTopic(); + int pageSize = request.getPageSize() <= 0 ? 20 : Math.min(request.getPageSize(), MAX_DIAG_PAGE_SIZE); + int pageNum = Math.max(request.getPageNum(), 1); + + List all = new ArrayList<>(); + PopReceiptHandleGroupSummary.Builder summary = PopReceiptHandleGroupSummary.newBuilder().setGroup(group); + long totalRenew = 0; + long totalRenewRetry = 0; + int expired = 0; + long now = System.currentTimeMillis(); + Map lockViews = new LinkedHashMap<>(); + + DefaultReceiptHandleManager manager = receiptHandleManager(); + if (manager != null) { + List collected = new ArrayList<>(); + manager.scanReceiptHandles((groupKey, handle) -> { + if (!group.equals(handle.getGroup())) { + return; + } + if (!topicFilter.isEmpty() && !topicFilter.equals(handle.getTopic())) { + return; + } + collected.add(new Object[] {groupKey, handle}); + }); + for (Object[] pair : collected) { + ReceiptHandleGroupKey groupKey = (ReceiptHandleGroupKey) pair[0]; + MessageReceiptHandle handle = (MessageReceiptHandle) pair[1]; + PopReceiptHandleInfo info = toHandleInfo(groupKey, handle, now); + all.add(info); + totalRenew += handle.getRenewTimes(); + totalRenewRetry += handle.getRenewRetryTimes(); + if (info.getIsExpired()) { + expired++; + } + String lockKey = handle.getTopic() + ":" + handle.getQueueId(); + lockViews.computeIfAbsent(lockKey, k -> PopLockView.newBuilder() + .setGroup(group) + .setTopic(handle.getTopic()) + .setQueueId(handle.getQueueId()) + .setLockOwner(info.getLockOwner()) + .setLocked(true)); + } + } + + summary.setTotalHandles(all.size()) + .setTotalMessages(all.size()) + .setTotalRenewTimes(totalRenew) + .setTotalRenewRetryTimes(totalRenewRetry) + .setExpiredHandles(expired) + .setTotalAckCount(0) + .setTotalNackCount(0) + .addAllLockView(buildLockViews(lockViews)); + + int fromIndex = Math.min((pageNum - 1) * pageSize, all.size()); + int toIndex = Math.min(fromIndex + pageSize, all.size()); + return DescribePopReceiptHandlesResponse.newBuilder() + .setStatus(ok) + .setSummary(summary) + .addAllHandles(all.subList(fromIndex, toIndex)) + .setTotal(all.size()) + .setPageNum(pageNum) + .setPageSize(pageSize) + .build(); + } + + private PopReceiptHandleInfo toHandleInfo(ReceiptHandleGroupKey groupKey, MessageReceiptHandle handle, long now) { + PopReceiptHandleInfo.Builder builder = PopReceiptHandleInfo.newBuilder() + .setGroup(handle.getGroup()) + .setTopic(handle.getTopic()) + .setQueueId(handle.getQueueId()) + .setMessageId(handle.getMessageId()) + .setQueueOffset(handle.getQueueOffset()) + .setReconsumeTimes(handle.getReconsumeTimes()) + .setRenewTimes(handle.getRenewTimes()) + .setRenewRetryTimes(handle.getRenewRetryTimes()) + .setConsumeTimestamp(timestamp(handle.getConsumeTimestamp())) + .setReceiptHandle(handle.getReceiptHandleStr()) + .setLockOwner(clientIdOf(groupKey.getChannel())); + try { + ReceiptHandle decoded = ReceiptHandle.decode(handle.getReceiptHandleStr()); + if (decoded != null) { + builder.setNextVisibleTime(timestamp(decoded.getNextVisibleTime())) + .setInvisibleTime(Duration.newBuilder() + .setSeconds(decoded.getInvisibleTime() / 1000) + .setNanos((int) ((decoded.getInvisibleTime() % 1000) * 1_000_000)) + .build()) + .setBrokerName(decoded.getBrokerName() == null ? "" : decoded.getBrokerName()) + .setIsExpired(decoded.getNextVisibleTime() < now); + } + } catch (Throwable t) { + builder.setIsExpired(false); + } + return builder.build(); + } + + private static List buildLockViews(Map lockViews) { + List result = new ArrayList<>(); + for (PopLockView.Builder builder : lockViews.values()) { + result.add(builder.build()); + } + return result; + } + + // --------------------------------------------------------------------- + // M4: batch consumption diagnostics + // --------------------------------------------------------------------- + + public DescribeBatchConsumeDiagnosticsResponse describeBatchConsumeDiagnostics( + DescribeBatchConsumeDiagnosticsRequest request, apache.rocketmq.v2.Status ok, + apache.rocketmq.v2.Status badRequest) { + String group = request.getGroup(); + if (group == null || group.isEmpty()) { + return DescribeBatchConsumeDiagnosticsResponse.newBuilder().setStatus(badRequest).build(); + } + String topicFilter = request.getTopic(); + String clientFilter = request.getClientId(); + int pageSize = request.getPageSize() <= 0 ? 20 : Math.min(request.getPageSize(), MAX_DIAG_PAGE_SIZE); + int pageNum = Math.max(request.getPageNum(), 1); + long now = System.currentTimeMillis(); + + Map perClient = new LinkedHashMap<>(); + DefaultReceiptHandleManager manager = receiptHandleManager(); + if (manager != null) { + manager.scanReceiptHandles((groupKey, handle) -> { + if (!group.equals(handle.getGroup())) { + return; + } + if (!topicFilter.isEmpty() && !topicFilter.equals(handle.getTopic())) { + return; + } + String clientId = clientIdOf(groupKey.getChannel()); + if (!clientFilter.isEmpty() && !clientFilter.equals(clientId)) { + return; + } + BatchConsumeClientDiagnostics.Builder builder = perClient.computeIfAbsent(clientId, key -> { + BatchConsumeClientDiagnostics.Builder newBuilder = BatchConsumeClientDiagnostics.newBuilder() + .setClientId(key) + .setChannelId(groupKey.getChannel() == null ? "" : groupKey.getChannel().id().asShortText()) + .setConsumeType("PUSH") + .setMessageModel(MessageModel.CLUSTERING); + if (groupKey.getChannel() instanceof GrpcClientChannel) { + newBuilder.setConnectTime(timestamp(((GrpcClientChannel) groupKey.getChannel()).getConnectTimeMillis())); + } + return newBuilder; + }); + builder.setUnackedMessageCount(builder.getUnackedMessageCount() + 1); + builder.setUnackedHandleCount(builder.getUnackedHandleCount() + 1); + builder.setTotalRenewTimes(builder.getTotalRenewTimes() + handle.getRenewTimes()); + builder.setTotalRenewRetryTimes(builder.getTotalRenewRetryTimes() + handle.getRenewRetryTimes()); + boolean expiredHandle = false; + try { + ReceiptHandle decoded = ReceiptHandle.decode(handle.getReceiptHandleStr()); + expiredHandle = decoded != null && decoded.getNextVisibleTime() < now; + } catch (Throwable ignore) { + // undecodable handle: count it as unexpired + } + if (expiredHandle) { + builder.setExpiredHandleCount(builder.getExpiredHandleCount() + 1); + } + builder.putTopicDistribution(handle.getTopic(), + builder.getTopicDistributionMap().getOrDefault(handle.getTopic(), 0) + 1); + }); + } + + BatchConsumeGroupSummary.Builder summary = BatchConsumeGroupSummary.newBuilder() + .setGroup(group) + .setTotalClients(perClient.size()); + List all = new ArrayList<>(); + for (BatchConsumeClientDiagnostics.Builder builder : perClient.values()) { + BatchConsumeClientDiagnostics diagnostics = builder.build(); + all.add(diagnostics); + summary.setTotalUnackedMessages(summary.getTotalUnackedMessages() + diagnostics.getUnackedMessageCount()); + summary.setTotalUnackedHandles(summary.getTotalUnackedHandles() + diagnostics.getUnackedHandleCount()); + summary.setTotalExpiredHandles(summary.getTotalExpiredHandles() + diagnostics.getExpiredHandleCount()); + summary.setTotalRenewTimes(summary.getTotalRenewTimes() + diagnostics.getTotalRenewTimes()); + summary.setTotalRenewRetryTimes(summary.getTotalRenewRetryTimes() + diagnostics.getTotalRenewRetryTimes()); + } + + int fromIndex = Math.min((pageNum - 1) * pageSize, all.size()); + int toIndex = Math.min(fromIndex + pageSize, all.size()); + return DescribeBatchConsumeDiagnosticsResponse.newBuilder() + .setStatus(ok) + .setSummary(summary) + .addAllDiagnostics(all.subList(fromIndex, toIndex)) + .setTotal(all.size()) + .setPageNum(pageNum) + .setPageSize(pageSize) + .build(); + } + + // --------------------------------------------------------------------- + // helpers + // --------------------------------------------------------------------- + + private DefaultReceiptHandleManager receiptHandleManager() { + if (messagingProcessor == null) { + return null; + } + ReceiptHandleProcessor processor = messagingProcessor.getReceiptHandleProcessor(); + return processor == null ? null : processor.getReceiptHandleManager(); + } + + private static String clientIdOf(Channel channel) { + if (channel instanceof GrpcClientChannel) { + return ((GrpcClientChannel) channel).getClientId(); + } + return channel == null ? "" : channel.id().asShortText(); + } + + private static Timestamp timestamp(long millis) { + return Timestamp.newBuilder() + .setSeconds(millis / 1000) + .setNanos((int) ((millis % 1000) * 1_000_000)) + .build(); + } + + Map countHandlesByGroup() { + Map counts = new HashMap<>(); + DefaultReceiptHandleManager manager = receiptHandleManager(); + if (manager != null) { + manager.scanReceiptHandles((groupKey, handle) -> + counts.merge(handle.getGroup(), 1, Integer::sum)); + } + return counts; + } +} diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminGrpcService.java b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminGrpcService.java new file mode 100644 index 00000000000..726065408e3 --- /dev/null +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminGrpcService.java @@ -0,0 +1,716 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.rocketmq.proxy.grpc.admin; + +import apache.rocketmq.v2.AdminGrpc; +import apache.rocketmq.v2.ClientInfo; +import apache.rocketmq.v2.Code; +import apache.rocketmq.v2.ConsumerRunningInfo; +import apache.rocketmq.v2.DescribeGroupAccumulationRequest; +import apache.rocketmq.v2.DescribeGroupAccumulationResponse; +import apache.rocketmq.v2.DescribeSubscriptionRequest; +import apache.rocketmq.v2.DescribeSubscriptionResponse; +import apache.rocketmq.v2.DescribeTopicStatusRequest; +import apache.rocketmq.v2.DescribeTopicStatusResponse; +import apache.rocketmq.v2.DeleteSubscriptionRequest; +import apache.rocketmq.v2.DeleteSubscriptionResponse; +import apache.rocketmq.v2.FilterExpression; +import apache.rocketmq.v2.GetConsumerRunningInfoRequest; +import apache.rocketmq.v2.GetConsumerRunningInfoResponse; +import apache.rocketmq.v2.GetProxyRuntimeStatsRequest; +import apache.rocketmq.v2.GetProxyRuntimeStatsResponse; +import apache.rocketmq.v2.GetTopicRouteRequest; +import apache.rocketmq.v2.GetTopicRouteResponse; +import apache.rocketmq.v2.ListConsumerConnectionRequest; +import apache.rocketmq.v2.ListConsumerConnectionResponse; +import apache.rocketmq.v2.ListMessageRequest; +import apache.rocketmq.v2.ListMessageResponse; +import apache.rocketmq.v2.ListSubscriptionRequest; +import apache.rocketmq.v2.ListSubscriptionResponse; +import apache.rocketmq.v2.PrintThreadStackTraceRequest; +import apache.rocketmq.v2.PrintThreadStackTraceResponse; +import apache.rocketmq.v2.QueryTimeSpanRequest; +import apache.rocketmq.v2.QueryTimeSpanResponse; +import apache.rocketmq.v2.ChangeLogLevelRequest; +import apache.rocketmq.v2.ChangeLogLevelResponse; +import apache.rocketmq.v2.ResetGroupOffsetRequest; +import apache.rocketmq.v2.ResetGroupOffsetResponse; +import apache.rocketmq.v2.Resource; +import apache.rocketmq.v2.VerifyMessageRequest; +import apache.rocketmq.v2.VerifyMessageResponse; +import apache.rocketmq.v2.AdminSendMessageRequest; +import apache.rocketmq.v2.AdminSendMessageResponse; +import apache.rocketmq.v2.Status; +import apache.rocketmq.v2.SubscriptionInfo; +import apache.rocketmq.v2.UA; +import io.grpc.stub.StreamObserver; +import java.util.ArrayList; +import java.util.Collection; +import java.util.List; +import java.util.concurrent.TimeUnit; +import org.apache.rocketmq.common.message.MessageExt; +import org.apache.rocketmq.logging.org.slf4j.Logger; +import org.apache.rocketmq.logging.org.slf4j.LoggerFactory; +import org.apache.rocketmq.proxy.common.ProxyContext; +import org.apache.rocketmq.proxy.grpc.v2.channel.GrpcChannelManager; +import org.apache.rocketmq.proxy.grpc.v2.channel.GrpcClientChannel; +import org.apache.rocketmq.proxy.grpc.v2.common.GrpcClientSettingsManager; +import org.apache.rocketmq.proxy.processor.MessagingProcessor; +import org.apache.rocketmq.proxy.service.ServiceManager; +import org.apache.rocketmq.proxy.service.admin.AdminService; + +/** + * RIP-2 Proxy Admin gRPC service. + * + *

Every capability is served from the proxy process itself (via the gRPC proxy): + *

    + *
  • Online clients / connections are read from the proxy's own + * {@link GrpcChannelManager} (the authority on gRPC clients connected to the proxy).
    • + *
  • Runtime stats, log level and topic route are proxy-owned state.
    • + *
  • Subscriptions are derived from the client settings tracked by the proxy.
    • + *
  • Client diagnostics (thread stack / verify / running info) are dispatched through the + * proxy's own telemetry channel to the target client.
    • + *
  • Broker-internal data (offsets, accumulation, reset, message query) is reached ONLY through + * the proxy's own {@link AdminService} gateway — i.e. the proxy's managed broker client. + * The admin code never opens a direct link to the broker.
    • + *
+ * + *

This class is protocol-pure: it only depends on the RIP-2 gRPC contract + * ({@code apache.rocketmq.v2.*}, generated from rocketmq-apis). The translation between the + * broker's internal wire types and the v2 protocol lives in {@link AdminModelConverter}. + */ +public class ProxyAdminGrpcService extends AdminGrpc.AdminImplBase { + + private static final Logger log = LoggerFactory.getLogger(ProxyAdminGrpcService.class); + private static final long DEFAULT_TIMEOUT_MILLIS = 3000L; + private static final String PROXY_NAME = "rocketmq-proxy"; + private static final String PROXY_VERSION = "5.5.0"; + + private final ServiceManager serviceManager; + private final MessagingProcessor messagingProcessor; + private final GrpcChannelManager grpcChannelManager; + private final GrpcClientSettingsManager grpcClientSettingsManager; + + public ProxyAdminGrpcService(ServiceManager serviceManager, MessagingProcessor messagingProcessor, + GrpcChannelManager grpcChannelManager, GrpcClientSettingsManager grpcClientSettingsManager) { + this.serviceManager = serviceManager; + this.messagingProcessor = messagingProcessor; + this.grpcChannelManager = grpcChannelManager; + this.grpcClientSettingsManager = grpcClientSettingsManager; + } + + // ------------------------------------------------------------------------- + // helpers + // ------------------------------------------------------------------------- + + private Status ok() { + return Status.newBuilder().setCode(Code.OK).build(); + } + + private Status fail(Code code, String message) { + return Status.newBuilder().setCode(code).setMessage(message).build(); + } + + private ProxyContext ctx() { + return ProxyContext.create(); + } + + private String resolveBrokerAddr(String topic) throws Exception { + org.apache.rocketmq.proxy.service.route.MessageQueueView mqv = + serviceManager.getTopicRouteService().getAllMessageQueueView(ctx(), topic); + if (mqv == null || mqv.getReadSelector() == null || mqv.getReadSelector().getQueues().isEmpty()) { + throw new RuntimeException("topic route not found for " + topic); + } + org.apache.rocketmq.proxy.service.route.AddressableMessageQueue mq = + mqv.getReadSelector().getQueues().get(0); + String brokerAddr = mq.getBrokerAddr(); + if (brokerAddr == null || brokerAddr.isEmpty()) { + throw new RuntimeException("broker address not found for topic " + topic); + } + return brokerAddr; + } + + /** + * RIP-2 fix: resolve ALL distinct broker addresses hosting the topic, so multi-broker + * clusters get complete data for accumulation/reset/query/delete operations. + */ + private List resolveBrokerAddrs(String topic) throws Exception { + org.apache.rocketmq.proxy.service.route.MessageQueueView mqv = + serviceManager.getTopicRouteService().getAllMessageQueueView(ctx(), topic); + if (mqv == null || mqv.getReadSelector() == null || mqv.getReadSelector().getQueues().isEmpty()) { + throw new RuntimeException("topic route not found for " + topic); + } + java.util.LinkedHashSet addrs = new java.util.LinkedHashSet<>(); + for (org.apache.rocketmq.proxy.service.route.AddressableMessageQueue mq : mqv.getReadSelector().getQueues()) { + if (mq.getBrokerAddr() != null && !mq.getBrokerAddr().isEmpty()) { + addrs.add(mq.getBrokerAddr()); + } + } + if (addrs.isEmpty()) { + throw new RuntimeException("broker address not found for topic " + topic); + } + return new ArrayList<>(addrs); + } + + + private ClientInfo buildClientInfo(GrpcClientChannel channel) { + String clientId = channel.getClientId(); + ClientInfo.Builder builder = ClientInfo.newBuilder().setClientId(clientId); + apache.rocketmq.v2.Settings settings = grpcClientSettingsManager.getRawClientSettings(clientId); + if (settings != null) { + UA ua = settings.getUserAgent(); + if (ua != null) { + builder.setVersion(ua.getVersion()); + if (ua.getLanguage() != null) { + builder.setLanguage(ua.getLanguage().name()); + } + builder.setHostname(ua.getHostname()); + } + } + String remoteAddress = channel.getRemoteAddress(); + if (remoteAddress != null && !remoteAddress.isEmpty()) { + builder.setEgressIp(remoteAddress); + } + return builder.build(); + } + + private boolean matchGroup(apache.rocketmq.v2.Settings settings, Resource group) { + if (group.getName().isEmpty()) { + return true; + } + if (settings == null || !settings.hasSubscription()) { + return false; + } + return group.getName().equals(settings.getSubscription().getGroup().getName()); + } + + private List onlineConsumers(Resource group) { + List result = new ArrayList<>(); + Collection channels = grpcChannelManager.getClientChannels(); + for (GrpcClientChannel channel : channels) { + apache.rocketmq.v2.Settings settings = grpcClientSettingsManager.getRawClientSettings(channel.getClientId()); + if (settings == null || settings.getClientType() == apache.rocketmq.v2.ClientType.CLIENT_TYPE_UNSPECIFIED) { + continue; + } + if (!matchGroup(settings, group)) { + continue; + } + result.add(buildClientInfo(channel)); + } + return result; + } + + // ------------------------------------------------------------------------- + // RIP-2 RPCs + // ------------------------------------------------------------------------- + + @Override + public void changeLogLevel(ChangeLogLevelRequest request, StreamObserver responseObserver) { + String remark; + try { + // The proxy is wired to logback (rocketmq-logback-classic); change the root logger level + // through the relocated logback API. No broker link is involved. + org.apache.rocketmq.logging.org.slf4j.ILoggerFactory factory = + org.apache.rocketmq.logging.org.slf4j.LoggerFactory.getILoggerFactory(); + if (factory instanceof org.apache.rocketmq.logging.ch.qos.logback.classic.LoggerContext) { + org.apache.rocketmq.logging.ch.qos.logback.classic.LoggerContext loggerContext = + (org.apache.rocketmq.logging.ch.qos.logback.classic.LoggerContext) factory; + org.apache.rocketmq.logging.ch.qos.logback.classic.Level level = + org.apache.rocketmq.logging.ch.qos.logback.classic.Level.toLevel(request.getLevel().name()); + loggerContext.getLogger( + org.apache.rocketmq.logging.ch.qos.logback.classic.Logger.ROOT_LOGGER_NAME) + .setLevel(level); + remark = "log level changed to " + level; + } else { + remark = "unsupported logging backend, cannot change log level at runtime"; + } + } catch (Throwable t) { + remark = "failed to change log level: " + t.getMessage(); + log.warn("changeLogLevel failed", t); + } + responseObserver.onNext(ChangeLogLevelResponse.newBuilder().setRemark(remark).build()); + responseObserver.onCompleted(); + } + + @Override + public void getProxyRuntimeStats(GetProxyRuntimeStatsRequest request, + StreamObserver responseObserver) { + int producers = 0; + int consumers = 0; + for (GrpcClientChannel channel : grpcChannelManager.getClientChannels()) { + apache.rocketmq.v2.Settings settings = grpcClientSettingsManager.getRawClientSettings(channel.getClientId()); + if (settings == null) { + continue; + } + switch (settings.getClientType()) { + case PRODUCER: + case LITE_PUSH_CONSUMER: + producers++; + break; + case PUSH_CONSUMER: + case SIMPLE_CONSUMER: + case PULL_CONSUMER: + case LITE_SIMPLE_CONSUMER: + consumers++; + break; + default: + break; + } + } + GetProxyRuntimeStatsResponse.Builder builder = GetProxyRuntimeStatsResponse.newBuilder() + .setProxyName(PROXY_NAME) + .setVersion(PROXY_VERSION) + .setConnections(grpcChannelManager.getClientChannels().size()) + .setProducers(producers) + .setConsumers(consumers); + responseObserver.onNext(builder.build()); + responseObserver.onCompleted(); + } + + @Override + public void getTopicRoute(GetTopicRouteRequest request, StreamObserver responseObserver) { + try { + GetTopicRouteResponse response = AdminModelConverter.toTopicRoute( + serviceManager.getAdminService(), request.getTopic().getName()); + responseObserver.onNext(response); + } catch (Throwable t) { + log.warn("getTopicRoute failed", t); + responseObserver.onNext(GetTopicRouteResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())) + .build()); + } + responseObserver.onCompleted(); + } + + @Override + public void describeTopicStatus(DescribeTopicStatusRequest request, + StreamObserver responseObserver) { + try { + String topic = request.getTopic().getName(); + String brokerAddr = resolveBrokerAddr(topic); + DescribeTopicStatusResponse response = AdminModelConverter.toTopicStatus( + serviceManager.getAdminService(), brokerAddr, topic, DEFAULT_TIMEOUT_MILLIS); + responseObserver.onNext(response); + } catch (Throwable t) { + log.warn("describeTopicStatus failed", t); + responseObserver.onNext(DescribeTopicStatusResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + } + responseObserver.onCompleted(); + } + + @Override + public void listSubscription(ListSubscriptionRequest request, StreamObserver responseObserver) { + try { + ListSubscriptionResponse.Builder builder = ListSubscriptionResponse.newBuilder().setStatus(ok()); + Resource topicFilter = request.getTopic(); + Resource groupFilter = request.getGroup(); + for (GrpcClientChannel channel : grpcChannelManager.getClientChannels()) { + apache.rocketmq.v2.Settings settings = grpcClientSettingsManager.getRawClientSettings(channel.getClientId()); + if (settings == null || !settings.hasSubscription()) { + continue; + } + apache.rocketmq.v2.Subscription subscription = settings.getSubscription(); + String group = subscription.getGroup().getName(); + if (!groupFilter.getName().isEmpty() && !group.equals(groupFilter.getName())) { + continue; + } + for (apache.rocketmq.v2.SubscriptionEntry entry : subscription.getSubscriptionsList()) { + String entryTopic = entry.hasTopic() ? entry.getTopic().getName() : ""; + if (!topicFilter.getName().isEmpty() && !entryTopic.equals(topicFilter.getName())) { + continue; + } + SubscriptionInfo.Builder info = SubscriptionInfo.newBuilder() + .setGroup(resource(group)) + .setTopic(resource(entryTopic)); + if (entry.hasExpression()) { + info.setExpression(entry.getExpression()); + } + info.setOnline(true); + builder.addSubscriptionInfo(info); + } + } + responseObserver.onNext(builder.build()); + } catch (Throwable t) { + log.warn("listSubscription failed", t); + responseObserver.onNext(ListSubscriptionResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + } + responseObserver.onCompleted(); + } + + @Override + public void describeSubscription(DescribeSubscriptionRequest request, + StreamObserver responseObserver) { + try { + DescribeSubscriptionResponse.Builder builder = DescribeSubscriptionResponse.newBuilder().setStatus(ok()); + Resource topicFilter = request.getTopic(); + Resource groupFilter = request.getGroup(); + for (GrpcClientChannel channel : grpcChannelManager.getClientChannels()) { + apache.rocketmq.v2.Settings settings = grpcClientSettingsManager.getRawClientSettings(channel.getClientId()); + if (settings == null || !settings.hasSubscription()) { + continue; + } + apache.rocketmq.v2.Subscription subscription = settings.getSubscription(); + String group = subscription.getGroup().getName(); + if (!groupFilter.getName().isEmpty() && !group.equals(groupFilter.getName())) { + continue; + } + for (apache.rocketmq.v2.SubscriptionEntry entry : subscription.getSubscriptionsList()) { + String entryTopic = entry.hasTopic() ? entry.getTopic().getName() : ""; + if (!topicFilter.getName().isEmpty() && !entryTopic.equals(topicFilter.getName())) { + continue; + } + SubscriptionInfo.Builder info = SubscriptionInfo.newBuilder() + .setGroup(resource(group)) + .setTopic(resource(entryTopic)); + if (entry.hasExpression()) { + info.setExpression(entry.getExpression()); + } + info.setOnline(true); + DescribeSubscriptionResponse.ClientSubscriptionInfo.Builder csi = + DescribeSubscriptionResponse.ClientSubscriptionInfo.newBuilder() + .setClientInfo(buildClientInfo(channel)) + .setSubscriptionInfo(info); + builder.addClientSubscriptionInfo(csi); + } + } + responseObserver.onNext(builder.build()); + } catch (Throwable t) { + log.warn("describeSubscription failed", t); + responseObserver.onNext(DescribeSubscriptionResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + } + responseObserver.onCompleted(); + } + + @Override + public void deleteSubscription(DeleteSubscriptionRequest request, + StreamObserver responseObserver) { + try { + String topic = request.getTopic().getName(); + String group = request.hasGroup() ? request.getGroup().getName() : ""; + if (topic == null || topic.isEmpty() || group.isEmpty()) { + responseObserver.onNext(DeleteSubscriptionResponse.newBuilder() + .setStatus(fail(Code.BAD_REQUEST, "topic and group are required")).build()); + responseObserver.onCompleted(); + return; + } + log.info("deleteSubscription requested for group={}, topic={}", group, topic); + List brokerAddrs = resolveBrokerAddrs(topic); + StringBuilder errors = new StringBuilder(); + for (String brokerAddr : brokerAddrs) { + try { + serviceManager.getAdminService().deleteSubscriptionGroup(brokerAddr, group, false, + DEFAULT_TIMEOUT_MILLIS); + } catch (Throwable t) { + log.warn("deleteSubscription failed on broker {}", brokerAddr, t); + if (errors.length() > 0) { + errors.append("; "); + } + errors.append(brokerAddr).append(": ").append(t.getMessage()); + } + } + if (errors.length() > 0) { + responseObserver.onNext(DeleteSubscriptionResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, errors.toString())).build()); + } else { + responseObserver.onNext(DeleteSubscriptionResponse.newBuilder().setStatus(ok()).build()); + } + } catch (Throwable t) { + log.warn("deleteSubscription failed", t); + responseObserver.onNext(DeleteSubscriptionResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + } + responseObserver.onCompleted(); + } + + @Override + public void listConsumerConnection(ListConsumerConnectionRequest request, + StreamObserver responseObserver) { + try { + List clients = onlineConsumers(request.getGroup()); + responseObserver.onNext(ListConsumerConnectionResponse.newBuilder() + .setStatus(ok()) + .addAllClientInfo(clients) + .build()); + } catch (Throwable t) { + log.warn("listConsumerConnection failed", t); + responseObserver.onNext(ListConsumerConnectionResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + } + responseObserver.onCompleted(); + } + + @Override + public void describeGroupAccumulation(DescribeGroupAccumulationRequest request, + StreamObserver responseObserver) { + try { + String group = request.getGroup().getName(); + String topic = request.getTopicsCount() > 0 ? request.getTopics(0).getName() : null; + List brokerAddrs = resolveBrokerAddrs(topic == null ? group : topic); + DescribeGroupAccumulationResponse.GroupAccumulation accumulation = + AdminModelConverter.toGroupAccumulationMultiBroker(serviceManager.getAdminService(), brokerAddrs, + group, topic, DEFAULT_TIMEOUT_MILLIS); + responseObserver.onNext(DescribeGroupAccumulationResponse.newBuilder() + .setStatus(ok()) + .setAccumulation(accumulation) + .build()); + } catch (Throwable t) { + log.warn("describeGroupAccumulation failed", t); + responseObserver.onNext(DescribeGroupAccumulationResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + } + responseObserver.onCompleted(); + } + + @Override + public void resetGroupOffset(ResetGroupOffsetRequest request, StreamObserver responseObserver) { + try { + String group = request.getGroup().getName(); + String topic = request.getTopic().getName(); + long resetTimestamp = request.getResetTimestamp().getSeconds() * 1000L; + // RIP-2 fix: reset on EVERY broker hosting the topic, not just the first one. + List brokerAddrs = resolveBrokerAddrs(topic); + StringBuilder errors = new StringBuilder(); + for (String brokerAddr : brokerAddrs) { + try { + serviceManager.getAdminService().resetOffset(brokerAddr, topic, group, resetTimestamp, true, + DEFAULT_TIMEOUT_MILLIS); + } catch (Throwable t) { + log.warn("resetGroupOffset failed on broker {}", brokerAddr, t); + if (errors.length() > 0) { + errors.append("; "); + } + errors.append(brokerAddr).append(": ").append(t.getMessage()); + } + } + if (errors.length() > 0) { + responseObserver.onNext(ResetGroupOffsetResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, errors.toString())).build()); + } else { + responseObserver.onNext(ResetGroupOffsetResponse.newBuilder().setStatus(ok()).build()); + } + } catch (Throwable t) { + log.warn("resetGroupOffset failed", t); + responseObserver.onNext(ResetGroupOffsetResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + } + responseObserver.onCompleted(); + } + + @Override + public void queryMessage(ListMessageRequest request, StreamObserver responseObserver) { + try { + if (!request.hasMessageId() && !request.hasMessageKey()) { + responseObserver.onNext(ListMessageResponse.newBuilder() + .setStatus(fail(Code.BAD_REQUEST, "one of message_id or message_key is required")) + .build()); + responseObserver.onCompleted(); + return; + } + String topic = request.getTopic().getName(); + List messageExtList = new ArrayList<>(); + if (request.hasMessageId()) { + String brokerAddr = resolveBrokerAddr(topic); + messageExtList.add(serviceManager.getAdminService().viewMessage(brokerAddr, topic, + decodeOffset(request.getMessageId()), DEFAULT_TIMEOUT_MILLIS)); + } else { + long begin = request.hasBeginTimestamp() ? request.getBeginTimestamp().getSeconds() * 1000L : 0L; + long end = request.hasEndTimestamp() ? request.getEndTimestamp().getSeconds() * 1000L : + System.currentTimeMillis(); + int maxNums = request.getMaxMessageNums() > 0 ? request.getMaxMessageNums() : 32; + // RIP-2 fix: a message key may live on any broker hosting the topic; search + // all of them until the requested number of messages is collected. + for (String brokerAddr : resolveBrokerAddrs(topic)) { + if (messageExtList.size() >= maxNums) { + break; + } + try { + messageExtList.addAll(serviceManager.getAdminService().queryMessage(brokerAddr, topic, + request.getMessageKey(), maxNums, begin, end, + DEFAULT_TIMEOUT_MILLIS)); + } catch (Throwable t) { + log.warn("queryMessage failed on broker {}", brokerAddr, t); + } + } + } + ListMessageResponse.Builder builder = ListMessageResponse.newBuilder().setStatus(ok()); + for (MessageExt ext : messageExtList) { + if (ext == null) { + continue; + } + builder.addMessages(AdminModelConverter.toMessage(ext)); + } + responseObserver.onNext(builder.build()); + } catch (Throwable t) { + log.warn("queryMessage failed", t); + responseObserver.onNext(ListMessageResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + } + responseObserver.onCompleted(); + } + + @Override + public void printThreadStackTrace(PrintThreadStackTraceRequest request, + StreamObserver responseObserver) { + try { + GrpcClientChannel channel = grpcChannelManager.getChannel(request.getClientId()); + if (channel == null) { + responseObserver.onNext(PrintThreadStackTraceResponse.newBuilder() + .setStatus(fail(Code.NOT_FOUND, "client not connected to this proxy")).build()); + } else { + channel.writeTelemetryCommand(apache.rocketmq.v2.TelemetryCommand.newBuilder() + .setPrintThreadStackTraceCommand(apache.rocketmq.v2.PrintThreadStackTraceCommand.newBuilder().build()) + .build()); + responseObserver.onNext(PrintThreadStackTraceResponse.newBuilder().setStatus(ok()).build()); + } + } catch (Throwable t) { + log.warn("printThreadStackTrace failed", t); + responseObserver.onNext(PrintThreadStackTraceResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + } + responseObserver.onCompleted(); + } + + @Override + public void verifyMessage(VerifyMessageRequest request, StreamObserver responseObserver) { + try { + GrpcClientChannel channel = grpcChannelManager.getChannel(request.getClientId()); + if (channel == null) { + responseObserver.onNext(VerifyMessageResponse.newBuilder() + .setStatus(fail(Code.NOT_FOUND, "client not connected to this proxy")).build()); + } else { + MessageExt ext = new MessageExt(); + ext.setTopic(request.getTopic().getName()); + ext.setMsgId(request.getMessageId()); + ext.setBody(new byte[0]); + channel.writeTelemetryCommand(apache.rocketmq.v2.TelemetryCommand.newBuilder() + .setVerifyMessageCommand(apache.rocketmq.v2.VerifyMessageCommand.newBuilder() + .setMessage(org.apache.rocketmq.proxy.grpc.v2.common.GrpcConverter.getInstance().buildMessage(ext)) + .build()) + .build()); + responseObserver.onNext(VerifyMessageResponse.newBuilder().setStatus(ok()).build()); + } + } catch (Throwable t) { + log.warn("verifyMessage failed", t); + responseObserver.onNext(VerifyMessageResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + } + responseObserver.onCompleted(); + } + + @Override + public void adminSendMessage(AdminSendMessageRequest request, StreamObserver responseObserver) { + try { + String topic = request.getTopic().getName(); + org.apache.rocketmq.common.message.Message msg = new org.apache.rocketmq.common.message.Message( + topic, request.getBody().toByteArray()); + if (request.hasTag()) { + msg.setTags(request.getTag()); + } + if (request.hasKey()) { + msg.setKeys(request.getKey()); + } + if (request.getUserPropertiesMap() != null) { + request.getUserPropertiesMap().forEach(msg::putUserProperty); + } + List list = new ArrayList<>(); + list.add(msg); + List sendResults = + messagingProcessor.sendMessage(ctx(), null, "ADMIN_SEND_PRODUCER_GROUP", 0, list, + DEFAULT_TIMEOUT_MILLIS).get(DEFAULT_TIMEOUT_MILLIS, TimeUnit.MILLISECONDS); + String messageId = sendResults != null && !sendResults.isEmpty() ? sendResults.get(0).getMsgId() : ""; + responseObserver.onNext(AdminSendMessageResponse.newBuilder() + .setStatus(ok()) + .setMessageId(messageId) + .build()); + } catch (Throwable t) { + log.warn("adminSendMessage failed", t); + responseObserver.onNext(AdminSendMessageResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + } + responseObserver.onCompleted(); + } + + @Override + public void getConsumerRunningInfo(GetConsumerRunningInfoRequest request, + StreamObserver responseObserver) { + try { + GrpcClientChannel channel = grpcChannelManager.getChannel(request.getClientId()); + if (channel == null) { + responseObserver.onNext(GetConsumerRunningInfoResponse.newBuilder() + .setStatus(fail(Code.NOT_FOUND, "client not connected to this proxy")).build()); + responseObserver.onCompleted(); + return; + } + apache.rocketmq.v2.Settings settings = grpcClientSettingsManager.getRawClientSettings(request.getClientId()); + ConsumerRunningInfo.Builder cri = ConsumerRunningInfo.newBuilder(); + if (settings != null && settings.hasSubscription()) { + for (apache.rocketmq.v2.SubscriptionEntry entry : settings.getSubscription().getSubscriptionsList()) { + FilterExpression fe = entry.hasExpression() ? entry.getExpression() : + FilterExpression.newBuilder().setType(apache.rocketmq.v2.FilterType.TAG).setExpression("*").build(); + cri.putSubscriptions(entry.hasTopic() ? entry.getTopic().getName() : "", fe); + } + } + responseObserver.onNext(GetConsumerRunningInfoResponse.newBuilder() + .setStatus(ok()) + .setConsumerRunningInfo(cri) + .build()); + } catch (Throwable t) { + log.warn("getConsumerRunningInfo failed", t); + responseObserver.onNext(GetConsumerRunningInfoResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + } + responseObserver.onCompleted(); + } + + @Override + public void queryTimeSpan(QueryTimeSpanRequest request, StreamObserver responseObserver) { + try { + String group = request.getGroup().getName(); + String topic = request.getTopicsCount() > 0 ? request.getTopics(0).getName() : group; + String brokerAddr = resolveBrokerAddr(topic); + org.apache.rocketmq.proxy.service.route.MessageQueueView mqv = + serviceManager.getTopicRouteService().getAllMessageQueueView(ctx(), topic); + QueryTimeSpanResponse response = AdminModelConverter.toQueryTimeSpan( + serviceManager.getAdminService(), brokerAddr, group, topic, mqv, DEFAULT_TIMEOUT_MILLIS); + responseObserver.onNext(response); + } catch (Throwable t) { + log.warn("queryTimeSpan failed", t); + responseObserver.onNext(QueryTimeSpanResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + } + responseObserver.onCompleted(); + } + + // ------------------------------------------------------------------------- + // converters + // ------------------------------------------------------------------------- + + private Resource resource(String name) { + return Resource.newBuilder().setName(name).build(); + } + + private long decodeOffset(String messageId) { + try { + return org.apache.rocketmq.common.message.MessageDecoder.decodeMessageId(messageId).getOffset(); + } catch (Throwable t) { + return 0L; + } + } +} diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminMetricsInterceptor.java b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminMetricsInterceptor.java new file mode 100644 index 00000000000..b5c6b9d4396 --- /dev/null +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminMetricsInterceptor.java @@ -0,0 +1,52 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.rocketmq.proxy.grpc.admin; + +import io.grpc.ForwardingServerCall.SimpleForwardingServerCall; +import io.grpc.Metadata; +import io.grpc.ServerCall; +import io.grpc.ServerCallHandler; +import io.grpc.ServerInterceptor; +import io.grpc.Status; + +/** + * RIP-2 acceptance criteria #4: measures transport-level failures of admin RPCs + * (authentication rejections, permission denials, framework errors) and feeds + * {@link ProxyAdminMetricsManager}. Business-level success/error outcomes are + * recorded by the admin services themselves, because they complete the gRPC + * call with an OK status and carry the error inside the response payload. + */ +public class ProxyAdminMetricsInterceptor implements ServerInterceptor { + + @Override + public ServerCall.Listener interceptCall(ServerCall call, Metadata headers, + ServerCallHandler next) { + final String method = call.getMethodDescriptor().getBareMethodName(); + final long startNanos = System.nanoTime(); + ServerCall observedCall = new SimpleForwardingServerCall(call) { + @Override + public void close(Status status, Metadata trailers) { + if (!status.isOk()) { + long latencyMillis = (System.nanoTime() - startNanos) / 1_000_000L; + ProxyAdminMetricsManager.recordError(method, latencyMillis, status.asRuntimeException()); + } + super.close(status, trailers); + } + }; + return next.startCall(observedCall, headers); + } +} diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminMetricsManager.java b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminMetricsManager.java new file mode 100644 index 00000000000..50e9ad4762f --- /dev/null +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminMetricsManager.java @@ -0,0 +1,252 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.rocketmq.proxy.grpc.admin; + +import io.opentelemetry.api.common.AttributeKey; +import io.opentelemetry.api.common.Attributes; +import io.opentelemetry.api.metrics.LongCounter; +import io.opentelemetry.api.metrics.DoubleHistogram; +import io.opentelemetry.api.metrics.Meter; +import io.opentelemetry.exporter.logging.otlp.OtlpJsonLoggingMetricExporter; +import io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter; +import io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporterBuilder; +import io.opentelemetry.exporter.prometheus.PrometheusHttpServer; +import io.opentelemetry.sdk.OpenTelemetrySdk; +import io.opentelemetry.sdk.metrics.InstrumentType; +import io.opentelemetry.sdk.metrics.SdkMeterProvider; +import io.opentelemetry.sdk.metrics.SdkMeterProviderBuilder; +import io.opentelemetry.sdk.metrics.data.AggregationTemporality; +import io.opentelemetry.sdk.metrics.export.MetricExporter; +import io.opentelemetry.sdk.metrics.export.PeriodicMetricReader; +import io.opentelemetry.sdk.resources.Resource; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.TimeUnit; +import org.apache.commons.lang3.StringUtils; +import org.apache.rocketmq.common.constant.LoggerName; +import org.apache.rocketmq.common.metrics.MetricsExporterType; +import org.apache.rocketmq.logging.org.slf4j.Logger; +import org.apache.rocketmq.logging.org.slf4j.LoggerFactory; +import org.apache.rocketmq.proxy.config.ProxyConfig; +import com.google.common.base.Splitter; + +/** + * RIP-2 acceptance criteria #4: the admin interface exposes its own call RT and + * error-rate metrics. + * + *

Exported instruments (OpenTelemetry, same exporter configuration as + * {@code ProxyMetricsManager}): + *

    + *
  • {@code rocketmq_proxy_admin_rpc_total} — LongCounter labeled by + * {@code rpc_method} and {@code status} (success/error, with + * {@code error_type} on failure). Error rate = rate of status=error.
    • + *
  • {@code rocketmq_proxy_admin_rpc_latency} — DoubleHistogram in + * milliseconds labeled by {@code rpc_method} and {@code status}; + * quantiles give P50/P99 RT per method.
    • + *
+ */ +public class ProxyAdminMetricsManager { + + private static final Logger log = LoggerFactory.getLogger(LoggerName.PROXY_LOGGER_NAME); + + public static final String METRIC_RPC_TOTAL = "rocketmq_proxy_admin_rpc_total"; + public static final String METRIC_RPC_LATENCY = "rocketmq_proxy_admin_rpc_latency"; + + public static final AttributeKey LABEL_METHOD = AttributeKey.stringKey("rpc_method"); + public static final AttributeKey LABEL_STATUS = AttributeKey.stringKey("status"); + public static final AttributeKey LABEL_ERROR_TYPE = AttributeKey.stringKey("error_type"); + public static final AttributeKey LABEL_CLUSTER_NAME = AttributeKey.stringKey("cluster_name"); + public static final AttributeKey LABEL_NODE_ID = AttributeKey.stringKey("node_id"); + + public static final String STATUS_SUCCESS = "success"; + public static final String STATUS_ERROR = "error"; + + private static final String OPEN_TELEMETRY_METER_NAME = "org.apache.rocketmq.proxy.admin"; + + private static volatile boolean initialized; + private static LongCounter rpcTotal; + private static DoubleHistogram rpcLatency; + private static String clusterName = ""; + private static String nodeName = ""; + + // exporter handles for shutdown + private static OtlpGrpcMetricExporter otlpExporter; + private static PeriodicMetricReader periodicReader; + private static PrometheusHttpServer prometheusHttpServer; + private static MetricExporter loggingExporter; + + private ProxyAdminMetricsManager() { + } + + /** + * Initialize the admin metrics pipeline from the same exporter settings used by the + * proxy data-plane metrics. Safe to call multiple times; first call wins. When the + * exporter is disabled the record methods become no-ops. + */ + public static synchronized void init(ProxyConfig proxyConfig) { + if (initialized) { + return; + } + try { + MetricsExporterType exporterType = proxyConfig.getMetricsExporterType(); + if (exporterType == null || !exporterType.isEnable()) { + log.info("RIP-2 admin metrics disabled, metricsExporterType:{}", exporterType); + return; + } + clusterName = StringUtils.defaultString(proxyConfig.getProxyClusterName()); + nodeName = StringUtils.defaultString(proxyConfig.getProxyName()); + + SdkMeterProviderBuilder providerBuilder = SdkMeterProvider.builder().setResource(Resource.empty()); + if (exporterType == MetricsExporterType.OTLP_GRPC) { + String endpoint = proxyConfig.getMetricsGrpcExporterTarget(); + if (StringUtils.isBlank(endpoint)) { + log.warn("RIP-2 admin metrics: OTLP exporter enabled but no target configured"); + return; + } + if (!endpoint.startsWith("http")) { + endpoint = "https://" + endpoint; + } + OtlpGrpcMetricExporterBuilder exporterBuilder = OtlpGrpcMetricExporter.builder() + .setEndpoint(endpoint) + .setTimeout(proxyConfig.getMetricGrpcExporterTimeOutInMills(), TimeUnit.MILLISECONDS) + .setAggregationTemporalitySelector(type -> { + if (proxyConfig.isMetricsInDelta() + && (type == InstrumentType.COUNTER || type == InstrumentType.OBSERVABLE_COUNTER + || type == InstrumentType.HISTOGRAM)) { + return AggregationTemporality.DELTA; + } + return AggregationTemporality.CUMULATIVE; + }); + String headers = proxyConfig.getMetricsGrpcExporterHeader(); + if (StringUtils.isNotBlank(headers)) { + Map headerMap = new HashMap<>(); + List kvPairs = Splitter.on(',').omitEmptyStrings().splitToList(headers); + for (String item : kvPairs) { + String[] split = item.split(":"); + if (split.length != 2) { + continue; + } + headerMap.put(split[0], split[1]); + } + headerMap.forEach(exporterBuilder::addHeader); + } + otlpExporter = exporterBuilder.build(); + periodicReader = PeriodicMetricReader.builder(otlpExporter) + .setInterval(proxyConfig.getMetricGrpcExporterIntervalInMills(), TimeUnit.MILLISECONDS) + .build(); + providerBuilder.registerMetricReader(periodicReader); + } else if (exporterType == MetricsExporterType.PROM) { + String host = proxyConfig.getMetricsPromExporterHost(); + if (StringUtils.isBlank(host)) { + host = "0.0.0.0"; + } + prometheusHttpServer = PrometheusHttpServer.builder() + .setHost(host) + // +1 avoids binding the same port as the data-plane PrometheusHttpServer + // when both run in one process; operators may override via config. + .setPort(proxyConfig.getMetricsPromExporterPort() + 1) + .build(); + providerBuilder.registerMetricReader(prometheusHttpServer); + } else if (exporterType == MetricsExporterType.LOG) { + loggingExporter = OtlpJsonLoggingMetricExporter.create(proxyConfig.isMetricsInDelta() + ? AggregationTemporality.DELTA : AggregationTemporality.CUMULATIVE); + periodicReader = PeriodicMetricReader.builder(loggingExporter) + .setInterval(proxyConfig.getMetricLoggingExporterIntervalInMills(), TimeUnit.MILLISECONDS) + .build(); + providerBuilder.registerMetricReader(periodicReader); + } + + Meter meter = OpenTelemetrySdk.builder() + .setMeterProvider(providerBuilder.build()) + .build() + .getMeter(OPEN_TELEMETRY_METER_NAME); + + rpcTotal = meter.counterBuilder(METRIC_RPC_TOTAL) + .setDescription("total number of RIP-2 proxy admin RPC calls") + .build(); + rpcLatency = meter.histogramBuilder(METRIC_RPC_LATENCY) + .setDescription("latency of RIP-2 proxy admin RPC calls") + .setUnit("ms") + .build(); + initialized = true; + log.info("RIP-2 admin metrics initialized, exporterType:{}", exporterType); + } catch (Throwable t) { + log.error("RIP-2 admin metrics init failed, metrics disabled", t); + } + } + + public static void recordSuccess(String rpcMethod, long latencyMillis) { + if (!initialized) { + return; + } + Attributes attributes = Attributes.builder() + .put(LABEL_METHOD, rpcMethod) + .put(LABEL_STATUS, STATUS_SUCCESS) + .put(LABEL_CLUSTER_NAME, clusterName) + .put(LABEL_NODE_ID, nodeName) + .build(); + rpcTotal.add(1, attributes); + rpcLatency.record(latencyMillis, attributes); + } + + public static void recordError(String rpcMethod, long latencyMillis, Throwable error) { + if (!initialized) { + return; + } + Attributes attributes = Attributes.builder() + .put(LABEL_METHOD, rpcMethod) + .put(LABEL_STATUS, STATUS_ERROR) + .put(LABEL_ERROR_TYPE, error == null ? "unknown" : StringUtils.defaultIfBlank( + error.getClass().getSimpleName(), "unknown")) + .put(LABEL_CLUSTER_NAME, clusterName) + .put(LABEL_NODE_ID, nodeName) + .build(); + rpcTotal.add(1, attributes); + rpcLatency.record(latencyMillis, attributes); + } + + public static synchronized void shutdown() { + if (!initialized) { + return; + } + try { + if (periodicReader != null) { + periodicReader.forceFlush(); + periodicReader.shutdown(); + } + if (otlpExporter != null) { + otlpExporter.shutdown(); + } + if (prometheusHttpServer != null) { + prometheusHttpServer.forceFlush(); + prometheusHttpServer.shutdown(); + } + if (loggingExporter != null) { + loggingExporter.shutdown(); + } + } catch (Throwable t) { + log.warn("RIP-2 admin metrics shutdown failed", t); + } finally { + initialized = false; + } + } + + public static boolean isInitialized() { + return initialized; + } +} diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminPeerClient.java b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminPeerClient.java new file mode 100644 index 00000000000..cc5c83a3124 --- /dev/null +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminPeerClient.java @@ -0,0 +1,264 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.rocketmq.proxy.grpc.admin; + +import apache.rocketmq.v2.ClientInstance; +import apache.rocketmq.v2.Code; +import apache.rocketmq.v2.ListClientsByGroupRequest; +import apache.rocketmq.v2.ListClientsByGroupResponse; +import apache.rocketmq.v2.ListClientsByTopicRequest; +import apache.rocketmq.v2.ListClientsByTopicResponse; +import apache.rocketmq.v2.ListClientsRequest; +import apache.rocketmq.v2.ListClientsResponse; +import apache.rocketmq.v2.ProxyAdminServiceGrpc; +import apache.rocketmq.v2.ProxyScope; +import io.grpc.ManagedChannel; +import io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; +import java.util.concurrent.TimeUnit; +import org.apache.rocketmq.common.constant.LoggerName; +import org.apache.rocketmq.common.utils.StartAndShutdown; +import org.apache.rocketmq.logging.org.slf4j.Logger; +import org.apache.rocketmq.logging.org.slf4j.LoggerFactory; + +/** + * RIP-2 D3 cluster-wide aggregation ("PROXY_SCOPE_ALL_PROXIES"). + * + *

Each proxy always serves its own local view; when the caller requests the + * ALL_PROXIES scope, this client fans the query out to the configured peer + * proxy admin endpoints ({@code proxyAdminPeerEndpoints}) in parallel and + * merges the per-node local views. Client instances are deduplicated by + * {@code client_id} (a client is attached to exactly one proxy at a time); + * the local node's view wins on duplicates. Every merged instance keeps the + * {@code proxy_endpoint}/{@code epoch} tag of the node that owns it, so the + * result is auditable. + * + *

Peer failures never fail the aggregated call: an unreachable peer is + * skipped (with a warning log) and the merged view of the remaining nodes is + * returned. + */ +public class ProxyAdminPeerClient implements StartAndShutdown { + + private static final Logger log = LoggerFactory.getLogger(LoggerName.PROXY_LOGGER_NAME); + + /** + * Fan-out requests always use LOCAL scope towards peers to avoid recursion. + */ + private static final int MAX_PEER_PAGE_SIZE = 1000; + + private final ConcurrentMap channels = new ConcurrentHashMap<>(); + private final org.apache.rocketmq.auth.config.AuthConfig authConfig; + + public ProxyAdminPeerClient() { + this(null); + } + + public ProxyAdminPeerClient(org.apache.rocketmq.auth.config.AuthConfig authConfig) { + this.authConfig = authConfig; + } + + /** + * Aggregate ListClients across all proxies. + * + * @param localView this node's already-filtered local instances + * @param request original request (filter is forwarded to peers, scope forced to LOCAL) + * @param peerEndpoints peer admin endpoints (host:port) + * @param timeoutMillis per-peer timeout + * @return merged view, local instances first + */ + public List listClientsAllProxies(List localView, ListClientsRequest request, + List peerEndpoints, long timeoutMillis) { + ListClientsRequest peerRequest = request.toBuilder() + .setScope(ProxyScope.PROXY_SCOPE_LOCAL_PROXY) + .setPageSize(MAX_PEER_PAGE_SIZE) + .clearNextToken() + .build(); + List>> futures = new ArrayList<>(); + for (String endpoint : peerEndpoints) { + futures.add(CompletableFuture.supplyAsync(() -> { + try { + ProxyAdminServiceGrpc.ProxyAdminServiceFutureStub stub = futureStub(endpoint, timeoutMillis); + ListClientsResponse response = stub.listClients(peerRequest) + .get(timeoutMillis, TimeUnit.MILLISECONDS); + if (response.getStatus().getCode() != Code.OK) { + log.warn("RIP-2 peer listClients returned non-OK. peer:{}, status:{}", endpoint, + response.getStatus()); + return new ArrayList(); + } + return response.getClientsList(); + } catch (Throwable t) { + log.warn("RIP-2 peer listClients failed, skip peer. peer:{}", endpoint, t); + return new ArrayList(); + } + })); + } + return merge(localView, futures, timeoutMillis); + } + + /** + * Aggregate ListClientsByGroup across all proxies. + */ + public List listClientsByGroupAllProxies(List localView, + ListClientsByGroupRequest request, List peerEndpoints, long timeoutMillis) { + ListClientsByGroupRequest peerRequest = request.toBuilder() + .setScope(ProxyScope.PROXY_SCOPE_LOCAL_PROXY) + .setPageSize(MAX_PEER_PAGE_SIZE) + .clearNextToken() + .build(); + List>> futures = new ArrayList<>(); + for (String endpoint : peerEndpoints) { + futures.add(CompletableFuture.supplyAsync(() -> { + try { + ProxyAdminServiceGrpc.ProxyAdminServiceFutureStub stub = futureStub(endpoint, timeoutMillis); + ListClientsByGroupResponse response = stub.listClientsByGroup(peerRequest) + .get(timeoutMillis, TimeUnit.MILLISECONDS); + if (response.getStatus().getCode() != Code.OK) { + log.warn("RIP-2 peer listClientsByGroup returned non-OK. peer:{}, status:{}", endpoint, + response.getStatus()); + return new ArrayList(); + } + return response.getClientsList(); + } catch (Throwable t) { + log.warn("RIP-2 peer listClientsByGroup failed, skip peer. peer:{}", endpoint, t); + return new ArrayList(); + } + })); + } + return merge(localView, futures, timeoutMillis); + } + + /** + * Aggregate ListClientsByTopic across all proxies. + */ + public List listClientsByTopicAllProxies(List localView, + ListClientsByTopicRequest request, List peerEndpoints, long timeoutMillis) { + ListClientsByTopicRequest peerRequest = request.toBuilder() + .setScope(ProxyScope.PROXY_SCOPE_LOCAL_PROXY) + .setPageSize(MAX_PEER_PAGE_SIZE) + .clearNextToken() + .build(); + List>> futures = new ArrayList<>(); + for (String endpoint : peerEndpoints) { + futures.add(CompletableFuture.supplyAsync(() -> { + try { + ProxyAdminServiceGrpc.ProxyAdminServiceFutureStub stub = futureStub(endpoint, timeoutMillis); + ListClientsByTopicResponse response = stub.listClientsByTopic(peerRequest) + .get(timeoutMillis, TimeUnit.MILLISECONDS); + if (response.getStatus().getCode() != Code.OK) { + log.warn("RIP-2 peer listClientsByTopic returned non-OK. peer:{}, status:{}", endpoint, + response.getStatus()); + return new ArrayList(); + } + return response.getClientsList(); + } catch (Throwable t) { + log.warn("RIP-2 peer listClientsByTopic failed, skip peer. peer:{}", endpoint, t); + return new ArrayList(); + } + })); + } + return merge(localView, futures, timeoutMillis); + } + + private List merge(List localView, + List>> peerFutures, long timeoutMillis) { + Map merged = new LinkedHashMap<>(); + for (ClientInstance instance : localView) { + merged.putIfAbsent(instance.getClientId(), instance); + } + long deadline = System.currentTimeMillis() + timeoutMillis; + for (CompletableFuture> future : peerFutures) { + long remaining = Math.max(1, deadline - System.currentTimeMillis()); + try { + List peerView = future.get(remaining, TimeUnit.MILLISECONDS); + for (ClientInstance instance : peerView) { + merged.putIfAbsent(instance.getClientId(), instance); + } + } catch (Throwable t) { + future.cancel(true); + log.warn("RIP-2 peer aggregation timed out or failed, skip peer result", t); + } + } + return new ArrayList<>(merged.values()); + } + + private ProxyAdminServiceGrpc.ProxyAdminServiceFutureStub futureStub(String endpoint, long timeoutMillis) { + ManagedChannel channel = channels.computeIfAbsent(endpoint, key -> + NettyChannelBuilder.forTarget(key) + .usePlaintext() + .build()); + ProxyAdminServiceGrpc.ProxyAdminServiceFutureStub stub = + ProxyAdminServiceGrpc.newFutureStub(channel) + .withDeadlineAfter(timeoutMillis, TimeUnit.MILLISECONDS); + ProxyAdminClientAuthInterceptor authInterceptor = buildAuthInterceptor(); + if (authInterceptor != null) { + stub = stub.withInterceptors(authInterceptor); + } + return stub; + } + + /** + * Fan-out requests authenticate as the proxy's inner client (SUPER user seeded via + * innerClientAuthenticationCredentials) when cluster authentication is enabled. + */ + private ProxyAdminClientAuthInterceptor buildAuthInterceptor() { + if (authConfig == null || !authConfig.isAuthenticationEnabled()) { + return null; + } + String credentialsJson = authConfig.getInnerClientAuthenticationCredentials(); + if (credentialsJson == null || credentialsJson.isEmpty()) { + return null; + } + try { + org.apache.rocketmq.acl.common.SessionCredentials credentials = + com.alibaba.fastjson.JSON.parseObject(credentialsJson, + org.apache.rocketmq.acl.common.SessionCredentials.class); + if (credentials == null || credentials.getAccessKey() == null + || credentials.getSecretKey() == null) { + return null; + } + return new ProxyAdminClientAuthInterceptor(credentials.getAccessKey(), credentials.getSecretKey()); + } catch (Throwable t) { + log.warn("RIP-2 peer auth credentials are invalid, fan-out will be unauthenticated", t); + return null; + } + } + + @Override + public void start() throws Exception { + } + + @Override + public void shutdown() throws Exception { + for (ManagedChannel channel : channels.values()) { + try { + channel.shutdown(); + if (!channel.awaitTermination(3, TimeUnit.SECONDS)) { + channel.shutdownNow(); + } + } catch (Throwable t) { + log.warn("RIP-2 peer channel shutdown failed", t); + } + } + channels.clear(); + } +} diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminServiceGrpcService.java b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminServiceGrpcService.java new file mode 100644 index 00000000000..9f0c0eccef1 --- /dev/null +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminServiceGrpcService.java @@ -0,0 +1,862 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.rocketmq.proxy.grpc.admin; + +import apache.rocketmq.v2.AuthStatus; +import apache.rocketmq.v2.ClientConsumeProgress; +import apache.rocketmq.v2.ClientDetail; +import apache.rocketmq.v2.ClientFilter; +import apache.rocketmq.v2.ClientInstance; +import apache.rocketmq.v2.ClientProtocol; +import apache.rocketmq.v2.ClientRole; +import apache.rocketmq.v2.ClientTopicProgress; +import apache.rocketmq.v2.ClientType; +import apache.rocketmq.v2.Code; +import apache.rocketmq.v2.DescribeBatchConsumeDiagnosticsRequest; +import apache.rocketmq.v2.DescribeBatchConsumeDiagnosticsResponse; +import apache.rocketmq.v2.DescribeClientRequest; +import apache.rocketmq.v2.DescribeClientResponse; +import apache.rocketmq.v2.DescribePopReceiptHandlesRequest; +import apache.rocketmq.v2.DescribePopReceiptHandlesResponse; +import apache.rocketmq.v2.DescribeProxyConfigRequest; +import apache.rocketmq.v2.DescribeProxyConfigResponse; +import apache.rocketmq.v2.DescribeQuotaRequest; +import apache.rocketmq.v2.DescribeQuotaResponse; +import apache.rocketmq.v2.DescribeRouteTopologyRequest; +import apache.rocketmq.v2.DescribeRouteTopologyResponse; +import apache.rocketmq.v2.DisconnectChannelRequest; +import apache.rocketmq.v2.DisconnectChannelResponse; +import apache.rocketmq.v2.KickClientRequest; +import apache.rocketmq.v2.KickClientResponse; +import apache.rocketmq.v2.Language; +import apache.rocketmq.v2.ListClientsByGroupRequest; +import apache.rocketmq.v2.ListClientsByGroupResponse; +import apache.rocketmq.v2.ListClientsByTopicRequest; +import apache.rocketmq.v2.ListClientsByTopicResponse; +import apache.rocketmq.v2.ListClientsRequest; +import apache.rocketmq.v2.ListClientsResponse; +import apache.rocketmq.v2.NetworkInfo; +import apache.rocketmq.v2.ProxyAdminServiceGrpc; +import apache.rocketmq.v2.ProxyScope; +import apache.rocketmq.v2.PublishSettings; +import apache.rocketmq.v2.Resource; +import apache.rocketmq.v2.Settings; +import apache.rocketmq.v2.Status; +import apache.rocketmq.v2.SubscribeRouteEventsRequest; +import apache.rocketmq.v2.SubscribeRouteEventsResponse; +import apache.rocketmq.v2.UA; +import apache.rocketmq.v2.UpdateProxyConfigRequest; +import apache.rocketmq.v2.UpdateProxyConfigResponse; +import apache.rocketmq.v2.UpdateQuotaRequest; +import apache.rocketmq.v2.UpdateQuotaResponse; +import com.google.protobuf.Timestamp; +import io.grpc.Context; +import io.grpc.stub.StreamObserver; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Base64; +import java.util.Collection; +import java.util.Comparator; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import org.apache.commons.lang3.StringUtils; +import org.apache.rocketmq.logging.org.slf4j.Logger; +import org.apache.rocketmq.logging.org.slf4j.LoggerFactory; +import org.apache.rocketmq.proxy.common.ProxyContext; +import org.apache.rocketmq.proxy.config.ConfigurationManager; +import org.apache.rocketmq.proxy.config.ProxyConfig; +import org.apache.rocketmq.proxy.grpc.v2.channel.GrpcChannelManager; +import org.apache.rocketmq.proxy.grpc.v2.channel.GrpcClientChannel; +import org.apache.rocketmq.proxy.grpc.v2.common.GrpcClientSettingsManager; +import org.apache.rocketmq.proxy.processor.DefaultMessagingProcessor; +import org.apache.rocketmq.proxy.service.ServiceManager; +import org.apache.rocketmq.proxy.service.route.MessageQueueView; + +/** + * RIP-2 Proxy Admin gRPC service — the complete {@code ProxyAdminService} surface: + * + *

    + *
  • M1 online client query: ListClients / DescribeClient / ListClientsByGroup / + * ListClientsByTopic;
    • + *
  • M2 runtime config & connection control: DescribeProxyConfig / UpdateProxyConfig / + * KickClient / DisconnectChannel;
    • + *
  • M2 quota visualization: DescribeQuota / UpdateQuota;
    • + *
  • M3/M4 diagnostics: DescribePopReceiptHandles / DescribeBatchConsumeDiagnostics;
    • + *
  • route observation: SubscribeRouteEvents (server streaming) / DescribeRouteTopology.
    • + *
+ * + *

Design notes (RIP-2): + *

    + *
  • D3 multi-proxy semantics: every reply is tagged with {@code proxy_endpoint} + + * {@code epoch}; when the caller asks for {@code PROXY_SCOPE_ALL_PROXIES} and peer + * endpoints are configured, the query fans out via {@link ProxyAdminPeerClient} and + * returns the deduplicated cluster-wide view.
    • + *
  • D4 pagination: stable cursor-based {@code next_token}. The cursor is the + * clientId-sorted position of the last returned element, so pages stay consistent + * even while clients connect/disconnect between calls.
    • + *
  • Every capability is served from the proxy itself; broker-internal data is only + * reached through the proxy's managed broker client (AdminService).
    • + *
+ */ +public class ProxyAdminServiceGrpcService extends ProxyAdminServiceGrpc.ProxyAdminServiceImplBase { + + private static final Logger log = LoggerFactory.getLogger(ProxyAdminServiceGrpcService.class); + + // Server-enforced maximum page size for cursor pagination (D4). + private static final int MAX_PAGE_SIZE = 1000; + private static final int DEFAULT_PAGE_SIZE = 100; + private static final String CURSOR_PREFIX = "c1:"; + private static final long CONSUME_PROGRESS_TIMEOUT_MILLIS = 3000L; + + private final ServiceManager serviceManager; + private final DefaultMessagingProcessor messagingProcessor; + private final GrpcChannelManager grpcChannelManager; + private final GrpcClientSettingsManager grpcClientSettingsManager; + private final ProxyAdminPeerClient peerClient; + private final RouteChangeNotifier routeChangeNotifier; + private final ProxyAdminConfigSupport configSupport; + private final ProxyAdminDiagnosticsSupport diagnosticsSupport; + + private final String proxyEndpoint; + private final long epoch; + + public ProxyAdminServiceGrpcService(ServiceManager serviceManager, DefaultMessagingProcessor messagingProcessor, + GrpcChannelManager grpcChannelManager, GrpcClientSettingsManager grpcClientSettingsManager, + ProxyAdminPeerClient peerClient, RouteChangeNotifier routeChangeNotifier) { + this.serviceManager = serviceManager; + this.messagingProcessor = messagingProcessor; + this.grpcChannelManager = grpcChannelManager; + this.grpcClientSettingsManager = grpcClientSettingsManager; + this.peerClient = peerClient; + this.routeChangeNotifier = routeChangeNotifier; + this.configSupport = new ProxyAdminConfigSupport(); + this.diagnosticsSupport = new ProxyAdminDiagnosticsSupport(messagingProcessor); + this.proxyEndpoint = resolveProxyEndpoint(); + this.epoch = System.currentTimeMillis(); + } + + // ------------------------------------------------------------------------- + // M1: online client query + // ------------------------------------------------------------------------- + + @Override + public void listClients(ListClientsRequest request, StreamObserver responseObserver) { + long start = System.currentTimeMillis(); + try { + ClientFilter filter = request.hasFilter() ? request.getFilter() : null; + List local = filterInstances(allClientInstances(), filter); + List view = applyScope(local, request.getScope(), + peers -> peerClient.listClientsAllProxies(local, request, peers, peerTimeoutMillis())); + Page page = page(view, request.getPageSize(), request.getNextToken()); + ListClientsResponse.Builder builder = ListClientsResponse.newBuilder() + .setStatus(success()) + .setProxyEndpoint(proxyEndpoint) + .setEpoch(epoch) + .addAllClients(page.items); + if (!page.nextToken.isEmpty()) { + builder.setNextToken(page.nextToken); + } + responseObserver.onNext(builder.build()); + responseObserver.onCompleted(); + ProxyAdminMetricsManager.recordSuccess("ListClients", System.currentTimeMillis() - start); + } catch (Throwable t) { + ProxyAdminMetricsManager.recordError("ListClients", System.currentTimeMillis() - start, t); + log.error("listClients failed", t); + responseObserver.onNext(ListClientsResponse.newBuilder().setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + responseObserver.onCompleted(); + } + } + + @Override + public void describeClient(DescribeClientRequest request, StreamObserver responseObserver) { + long start = System.currentTimeMillis(); + try { + String clientId = request.getClientId(); + GrpcClientChannel channel = grpcChannelManager.getChannel(clientId); + if (channel == null) { + responseObserver.onNext(DescribeClientResponse.newBuilder() + .setStatus(fail(Code.NOT_FOUND, "client not connected to this proxy")) + .build()); + responseObserver.onCompleted(); + return; + } + Settings settings = grpcClientSettingsManager.getRawClientSettings(clientId); + ClientDetail.Builder detail = ClientDetail.newBuilder().setInstance(buildClientInstance(channel)); + + detail.addAllRecentHeartbeats(channel.getRecentHeartbeats()); + detail.setAuthStatus(buildAuthStatus(channel)); + + if (settings != null) { + detail.setSettings(settings); + if (settings.hasSubscription()) { + detail.addAllSubscriptions(settings.getSubscription().getSubscriptionsList()); + detail.setConsumeProgress(buildConsumeProgress(channel, settings)); + } + if (settings.hasPublishing()) { + detail.setPublishSettings(PublishSettings.newBuilder() + .addAllTopics(settings.getPublishing().getTopicsList())); + } + } + detail.setNetworkInfo(NetworkInfo.newBuilder() + .setLocalAddress(str(channel.getLocalAddress())) + .setRemoteAddress(str(channel.getRemoteAddress())) + .build()); + responseObserver.onNext(DescribeClientResponse.newBuilder() + .setStatus(success()) + .setClientDetail(detail) + .build()); + responseObserver.onCompleted(); + ProxyAdminMetricsManager.recordSuccess("DescribeClient", System.currentTimeMillis() - start); + } catch (Throwable t) { + ProxyAdminMetricsManager.recordError("DescribeClient", System.currentTimeMillis() - start, t); + log.error("describeClient failed", t); + responseObserver.onNext(DescribeClientResponse.newBuilder().setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + responseObserver.onCompleted(); + } + } + + @Override + public void listClientsByGroup(ListClientsByGroupRequest request, StreamObserver responseObserver) { + long start = System.currentTimeMillis(); + try { + String group = request.hasGroup() ? request.getGroup().getName() : ""; + List local = new ArrayList<>(); + for (ClientInstance instance : allClientInstances()) { + if (group.isEmpty() || instance.getGroupsList().contains(group)) { + local.add(instance); + } + } + List view = applyScope(local, request.getScope(), + peers -> peerClient.listClientsByGroupAllProxies(local, request, peers, peerTimeoutMillis())); + Page page = page(view, request.getPageSize(), request.getNextToken()); + ListClientsByGroupResponse.Builder builder = ListClientsByGroupResponse.newBuilder() + .setStatus(success()) + .setProxyEndpoint(proxyEndpoint) + .setEpoch(epoch) + .addAllClients(page.items); + if (!page.nextToken.isEmpty()) { + builder.setNextToken(page.nextToken); + } + responseObserver.onNext(builder.build()); + responseObserver.onCompleted(); + ProxyAdminMetricsManager.recordSuccess("ListClientsByGroup", System.currentTimeMillis() - start); + } catch (Throwable t) { + ProxyAdminMetricsManager.recordError("ListClientsByGroup", System.currentTimeMillis() - start, t); + log.error("listClientsByGroup failed", t); + responseObserver.onNext(ListClientsByGroupResponse.newBuilder().setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + responseObserver.onCompleted(); + } + } + + @Override + public void listClientsByTopic(ListClientsByTopicRequest request, StreamObserver responseObserver) { + long start = System.currentTimeMillis(); + try { + String topic = request.hasTopic() ? request.getTopic().getName() : ""; + List local = new ArrayList<>(); + for (ClientInstance instance : allClientInstances()) { + if (topic.isEmpty() || instance.getTopicsList().contains(topic)) { + local.add(instance); + } + } + List view = applyScope(local, request.getScope(), + peers -> peerClient.listClientsByTopicAllProxies(local, request, peers, peerTimeoutMillis())); + Page page = page(view, request.getPageSize(), request.getNextToken()); + ListClientsByTopicResponse.Builder builder = ListClientsByTopicResponse.newBuilder() + .setStatus(success()) + .setProxyEndpoint(proxyEndpoint) + .setEpoch(epoch) + .addAllClients(page.items); + if (!page.nextToken.isEmpty()) { + builder.setNextToken(page.nextToken); + } + responseObserver.onNext(builder.build()); + responseObserver.onCompleted(); + ProxyAdminMetricsManager.recordSuccess("ListClientsByTopic", System.currentTimeMillis() - start); + } catch (Throwable t) { + ProxyAdminMetricsManager.recordError("ListClientsByTopic", System.currentTimeMillis() - start, t); + log.error("listClientsByTopic failed", t); + responseObserver.onNext(ListClientsByTopicResponse.newBuilder().setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + responseObserver.onCompleted(); + } + } + + // ------------------------------------------------------------------------- + // M2: runtime config & connection control + // ------------------------------------------------------------------------- + + @Override + public void describeProxyConfig(DescribeProxyConfigRequest request, + StreamObserver responseObserver) { + long start = System.currentTimeMillis(); + try { + responseObserver.onNext(configSupport.describeProxyConfig(request, success())); + responseObserver.onCompleted(); + ProxyAdminMetricsManager.recordSuccess("DescribeProxyConfig", System.currentTimeMillis() - start); + } catch (Throwable t) { + ProxyAdminMetricsManager.recordError("DescribeProxyConfig", System.currentTimeMillis() - start, t); + log.error("describeProxyConfig failed", t); + responseObserver.onNext(DescribeProxyConfigResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + responseObserver.onCompleted(); + } + } + + @Override + public void updateProxyConfig(UpdateProxyConfigRequest request, + StreamObserver responseObserver) { + long start = System.currentTimeMillis(); + try { + responseObserver.onNext(configSupport.updateProxyConfig(request, success())); + responseObserver.onCompleted(); + ProxyAdminMetricsManager.recordSuccess("UpdateProxyConfig", System.currentTimeMillis() - start); + } catch (Throwable t) { + ProxyAdminMetricsManager.recordError("UpdateProxyConfig", System.currentTimeMillis() - start, t); + log.error("updateProxyConfig failed", t); + responseObserver.onNext(UpdateProxyConfigResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + responseObserver.onCompleted(); + } + } + + @Override + public void kickClient(KickClientRequest request, StreamObserver responseObserver) { + long start = System.currentTimeMillis(); + try { + String clientId = request.getClientId(); + if (StringUtils.isBlank(clientId)) { + responseObserver.onNext(KickClientResponse.newBuilder() + .setStatus(fail(Code.BAD_REQUEST, "client_id is required")).build()); + responseObserver.onCompleted(); + return; + } + if (StringUtils.isBlank(request.getReason())) { + responseObserver.onNext(KickClientResponse.newBuilder() + .setStatus(fail(Code.BAD_REQUEST, "reason is required for audit")).build()); + responseObserver.onCompleted(); + return; + } + boolean disconnected = disconnectClient(clientId, "kick by admin, reason: " + request.getReason()); + responseObserver.onNext(KickClientResponse.newBuilder() + .setStatus(success()) + .setDisconnected(disconnected) + .build()); + responseObserver.onCompleted(); + ProxyAdminMetricsManager.recordSuccess("KickClient", System.currentTimeMillis() - start); + } catch (Throwable t) { + ProxyAdminMetricsManager.recordError("KickClient", System.currentTimeMillis() - start, t); + log.error("kickClient failed", t); + responseObserver.onNext(KickClientResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + responseObserver.onCompleted(); + } + } + + @Override + public void disconnectChannel(DisconnectChannelRequest request, + StreamObserver responseObserver) { + long start = System.currentTimeMillis(); + try { + String channelId = request.getChannelId(); + if (StringUtils.isBlank(channelId)) { + responseObserver.onNext(DisconnectChannelResponse.newBuilder() + .setStatus(fail(Code.BAD_REQUEST, "channel_id is required")).build()); + responseObserver.onCompleted(); + return; + } + if (StringUtils.isBlank(request.getReason())) { + responseObserver.onNext(DisconnectChannelResponse.newBuilder() + .setStatus(fail(Code.BAD_REQUEST, "reason is required for audit")).build()); + responseObserver.onCompleted(); + return; + } + boolean disconnected = false; + for (GrpcClientChannel channel : grpcChannelManager.getClientChannels()) { + if (channelId.equals(channel.id().asShortText()) || channelId.equals(channel.id().asLongText()) + || channelId.equals(channel.getClientId())) { + disconnected = disconnectClient(channel.getClientId(), + "channel disconnected by admin, reason: " + request.getReason()); + break; + } + } + responseObserver.onNext(DisconnectChannelResponse.newBuilder() + .setStatus(success()) + .setDisconnected(disconnected) + .build()); + responseObserver.onCompleted(); + ProxyAdminMetricsManager.recordSuccess("DisconnectChannel", System.currentTimeMillis() - start); + } catch (Throwable t) { + ProxyAdminMetricsManager.recordError("DisconnectChannel", System.currentTimeMillis() - start, t); + log.error("disconnectChannel failed", t); + responseObserver.onNext(DisconnectChannelResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + responseObserver.onCompleted(); + } + } + + // ------------------------------------------------------------------------- + // M2: quota visualization & controlled adjustment + // ------------------------------------------------------------------------- + + @Override + public void describeQuota(DescribeQuotaRequest request, StreamObserver responseObserver) { + long start = System.currentTimeMillis(); + try { + responseObserver.onNext(configSupport.describeQuota(request, success())); + responseObserver.onCompleted(); + ProxyAdminMetricsManager.recordSuccess("DescribeQuota", System.currentTimeMillis() - start); + } catch (Throwable t) { + ProxyAdminMetricsManager.recordError("DescribeQuota", System.currentTimeMillis() - start, t); + log.error("describeQuota failed", t); + responseObserver.onNext(DescribeQuotaResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + responseObserver.onCompleted(); + } + } + + @Override + public void updateQuota(UpdateQuotaRequest request, StreamObserver responseObserver) { + long start = System.currentTimeMillis(); + try { + responseObserver.onNext(configSupport.updateQuota(request, success(), fail(Code.BAD_REQUEST, + "policy with positive limit and metric is required"))); + responseObserver.onCompleted(); + ProxyAdminMetricsManager.recordSuccess("UpdateQuota", System.currentTimeMillis() - start); + } catch (Throwable t) { + ProxyAdminMetricsManager.recordError("UpdateQuota", System.currentTimeMillis() - start, t); + log.error("updateQuota failed", t); + responseObserver.onNext(UpdateQuotaResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + responseObserver.onCompleted(); + } + } + + // ------------------------------------------------------------------------- + // M3/M4: POP & batch consume diagnostics + // ------------------------------------------------------------------------- + + @Override + public void describePopReceiptHandles(DescribePopReceiptHandlesRequest request, + StreamObserver responseObserver) { + long start = System.currentTimeMillis(); + try { + responseObserver.onNext(diagnosticsSupport.describePopReceiptHandles(request, success(), + fail(Code.BAD_REQUEST, "group is required"))); + responseObserver.onCompleted(); + ProxyAdminMetricsManager.recordSuccess("DescribePopReceiptHandles", System.currentTimeMillis() - start); + } catch (Throwable t) { + ProxyAdminMetricsManager.recordError("DescribePopReceiptHandles", System.currentTimeMillis() - start, t); + log.error("describePopReceiptHandles failed", t); + responseObserver.onNext(DescribePopReceiptHandlesResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + responseObserver.onCompleted(); + } + } + + @Override + public void describeBatchConsumeDiagnostics(DescribeBatchConsumeDiagnosticsRequest request, + StreamObserver responseObserver) { + long start = System.currentTimeMillis(); + try { + responseObserver.onNext(diagnosticsSupport.describeBatchConsumeDiagnostics(request, success(), + fail(Code.BAD_REQUEST, "group is required"))); + responseObserver.onCompleted(); + ProxyAdminMetricsManager.recordSuccess("DescribeBatchConsumeDiagnostics", + System.currentTimeMillis() - start); + } catch (Throwable t) { + ProxyAdminMetricsManager.recordError("DescribeBatchConsumeDiagnostics", + System.currentTimeMillis() - start, t); + log.error("describeBatchConsumeDiagnostics failed", t); + responseObserver.onNext(DescribeBatchConsumeDiagnosticsResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + responseObserver.onCompleted(); + } + } + + // ------------------------------------------------------------------------- + // route observation + // ------------------------------------------------------------------------- + + @Override + public void subscribeRouteEvents(SubscribeRouteEventsRequest request, + StreamObserver responseObserver) { + long start = System.currentTimeMillis(); + try { + RouteChangeNotifier.Subscription subscription = routeChangeNotifier.subscribe(request, responseObserver, + serviceManager.getTopicRouteService().snapshotTopicRouteCache()); + Context.current().addListener(cancelledContext -> { + routeChangeNotifier.unsubscribe(subscription); + }, command -> command.run()); + ProxyAdminMetricsManager.recordSuccess("SubscribeRouteEvents", System.currentTimeMillis() - start); + } catch (Throwable t) { + ProxyAdminMetricsManager.recordError("SubscribeRouteEvents", System.currentTimeMillis() - start, t); + log.error("subscribeRouteEvents failed", t); + responseObserver.onNext(SubscribeRouteEventsResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + responseObserver.onCompleted(); + } + } + + @Override + public void describeRouteTopology(DescribeRouteTopologyRequest request, + StreamObserver responseObserver) { + long start = System.currentTimeMillis(); + try { + String topicFilter = request.hasTopic() ? request.getTopic().getName() : ""; + DescribeRouteTopologyResponse.Builder builder = DescribeRouteTopologyResponse.newBuilder() + .setStatus(success()); + Map routes = serviceManager.getTopicRouteService().snapshotTopicRouteCache(); + int activeConnections = grpcChannelManager.getClientChannels().size(); + for (Map.Entry entry : routes.entrySet()) { + String topic = entry.getKey(); + MessageQueueView view = entry.getValue(); + if (view == null || view.isEmptyCachedQueue()) { + continue; + } + if (!topicFilter.isEmpty() && !topicFilter.equals(topic)) { + continue; + } + AdminModelConverter.addRouteTopology(builder, topic, view.getTopicRouteData(), proxyEndpoint, + activeConnections); + } + responseObserver.onNext(builder.build()); + responseObserver.onCompleted(); + ProxyAdminMetricsManager.recordSuccess("DescribeRouteTopology", System.currentTimeMillis() - start); + } catch (Throwable t) { + ProxyAdminMetricsManager.recordError("DescribeRouteTopology", System.currentTimeMillis() - start, t); + log.error("describeRouteTopology failed", t); + responseObserver.onNext(DescribeRouteTopologyResponse.newBuilder() + .setStatus(fail(Code.INTERNAL_ERROR, t.getMessage())).build()); + responseObserver.onCompleted(); + } + } + + // ------------------------------------------------------------------------- + // helpers + // ------------------------------------------------------------------------- + + /** + * Forcefully disconnect a client: detach its channel from the manager and close the + * underlying transport. Returns true when the client was found and disconnected. + */ + private boolean disconnectClient(String clientId, String reason) { + GrpcClientChannel channel = grpcChannelManager.removeChannel(clientId); + if (channel == null) { + return false; + } + log.info("RIP-2 admin disconnect. clientId:{}, {}", clientId, reason); + try { + channel.close(); + } catch (Throwable t) { + log.warn("RIP-2 admin disconnect close failed. clientId:{}", clientId, t); + } + return true; + } + + private List filterInstances(List instances, ClientFilter filter) { + if (filter == null) { + return instances; + } + List filtered = new ArrayList<>(); + for (ClientInstance instance : instances) { + if (matchFilter(instance, filter)) { + filtered.add(instance); + } + } + return filtered; + } + + /** + * D3: resolve the requested scope. For ALL_PROXIES with configured peers, replace the + * local view with the merged cluster-wide view; otherwise keep the local view. + */ + private List applyScope(List localView, ProxyScope scope, + java.util.function.Function, List> aggregator) { + if (scope != ProxyScope.PROXY_SCOPE_ALL_PROXIES) { + return localView; + } + List peers = ConfigurationManager.getProxyConfig().getProxyAdminPeerEndpoints(); + if (peers == null || peers.isEmpty()) { + return localView; + } + return aggregator.apply(peers); + } + + private long peerTimeoutMillis() { + return ConfigurationManager.getProxyConfig().getProxyAdminPeerTimeoutMillis(); + } + + private List allClientInstances() { + List list = new ArrayList<>(); + Collection channels = grpcChannelManager.getClientChannels(); + for (GrpcClientChannel channel : channels) { + list.add(buildClientInstance(channel)); + } + return list; + } + + private ClientInstance buildClientInstance(GrpcClientChannel channel) { + String clientId = channel.getClientId(); + ClientInstance.Builder builder = ClientInstance.newBuilder().setClientId(clientId); + + ClientRole role = ClientRole.CLIENT_ROLE_UNSPECIFIED; + List groups = new ArrayList<>(); + List topics = new ArrayList<>(); + String clientVersion = ""; + Language language = Language.LANGUAGE_UNSPECIFIED; + + Settings settings = grpcClientSettingsManager.getRawClientSettings(clientId); + if (settings != null) { + UA ua = settings.getUserAgent(); + if (ua != null) { + clientVersion = ua.getVersion(); + language = ua.getLanguage(); + } + role = toClientRole(settings.getClientType()); + if (settings.hasSubscription()) { + Resource group = settings.getSubscription().getGroup(); + if (group != null && !group.getName().isEmpty()) { + groups.add(group.getName()); + } + for (apache.rocketmq.v2.SubscriptionEntry entry : settings.getSubscription().getSubscriptionsList()) { + if (entry.getTopic() != null && !entry.getTopic().getName().isEmpty()) { + topics.add(entry.getTopic().getName()); + } + } + } + if (settings.hasPublishing()) { + for (Resource topic : settings.getPublishing().getTopicsList()) { + if (topic != null && !topic.getName().isEmpty()) { + topics.add(topic.getName()); + } + } + } + } + + builder.setClientVersion(clientVersion); + builder.setLanguage(language); + builder.setProtocol(ClientProtocol.CLIENT_PROTOCOL_GRPC); + builder.setRole(role); + builder.addAllGroups(groups); + builder.addAllTopics(topics); + builder.setAccessPoint(str(channel.getRemoteAddress())); + builder.setConnectTime(toTimestamp(channel.getConnectTimeMillis())); + // RIP-2: real liveness — last heartbeat / telemetry observed on this channel. + builder.setLastActiveTime(toTimestamp(channel.getLastActiveTimeMillis())); + String authUsername = channel.getAuthUsername(); + if (StringUtils.isNotBlank(authUsername)) { + builder.setAuthSubject(authUsername); + } + builder.setProxyEndpoint(proxyEndpoint); + builder.setEpoch(epoch); + return builder.build(); + } + + private AuthStatus buildAuthStatus(GrpcClientChannel channel) { + String authUsername = channel.getAuthUsername(); + AuthStatus.Builder builder = AuthStatus.newBuilder(); + if (StringUtils.isNotBlank(authUsername)) { + builder.setAuthenticated(true) + .setUsername(authUsername) + .setLastAuthTime(toTimestamp(channel.getLastAuthTimeMillis())); + } else { + builder.setAuthenticated(false) + .setFailureReason("no credentials observed on this connection"); + } + return builder.build(); + } + + /** + * Best-effort consume progress: for every topic the client's group subscribes to, query + * broker-side consume stats through the proxy's own admin gateway and aggregate the lag. + * Latency is not tracked at the broker offset layer and stays unset. + */ + private ClientConsumeProgress buildConsumeProgress(GrpcClientChannel channel, Settings settings) { + ClientConsumeProgress.Builder progress = ClientConsumeProgress.newBuilder(); + try { + if (!settings.hasSubscription()) { + return progress.build(); + } + String group = settings.getSubscription().getGroup().getName(); + if (StringUtils.isBlank(group)) { + return progress.build(); + } + long totalLag = 0; + Set queriedTopics = new HashSet<>(); + for (apache.rocketmq.v2.SubscriptionEntry entry : settings.getSubscription().getSubscriptionsList()) { + String topic = entry.hasTopic() ? entry.getTopic().getName() : ""; + if (StringUtils.isBlank(topic) || !queriedTopics.add(topic)) { + continue; + } + long topicLag = queryTopicLag(group, topic); + if (topicLag >= 0) { + totalLag += topicLag; + progress.addTopicProgress(ClientTopicProgress.newBuilder() + .setTopic(topic) + .setLag(topicLag) + .build()); + } + } + progress.setLag(totalLag); + } catch (Throwable t) { + log.warn("buildConsumeProgress failed. clientId:{}", channel.getClientId(), t); + } + return progress.build(); + } + + private long queryTopicLag(String group, String topic) { + try { + MessageQueueView view = serviceManager.getTopicRouteService() + .getAllMessageQueueView(ProxyContext.create(), topic); + return AdminModelConverter.computeTopicLag(serviceManager.getAdminService(), view, group, topic, + CONSUME_PROGRESS_TIMEOUT_MILLIS); + } catch (Throwable t) { + log.warn("queryTopicLag failed. group:{}, topic:{}", group, topic, t); + return -1; + } + } + + private static ClientRole toClientRole(ClientType clientType) { + if (clientType == null) { + return ClientRole.CLIENT_ROLE_UNSPECIFIED; + } + switch (clientType) { + case PRODUCER: + return ClientRole.CLIENT_ROLE_PRODUCER; + case PUSH_CONSUMER: + case PULL_CONSUMER: + case LITE_PUSH_CONSUMER: + return ClientRole.CLIENT_ROLE_PUSH_CONSUMER; + case SIMPLE_CONSUMER: + case LITE_SIMPLE_CONSUMER: + return ClientRole.CLIENT_ROLE_SIMPLE_CONSUMER; + default: + return ClientRole.CLIENT_ROLE_UNSPECIFIED; + } + } + + private boolean matchFilter(ClientInstance instance, ClientFilter filter) { + if (filter.hasGroup() && !instance.getGroupsList().contains(filter.getGroup().getName())) { + return false; + } + if (filter.hasTopic() && !instance.getTopicsList().contains(filter.getTopic().getName())) { + return false; + } + if (filter.hasClientIdPrefix() && !instance.getClientId().startsWith(filter.getClientIdPrefix())) { + return false; + } + if (filter.hasLanguage() && instance.getLanguage() != filter.getLanguage()) { + return false; + } + if (filter.hasRole() && instance.getRole() != filter.getRole()) { + return false; + } + if (filter.hasConnectedAfter() && instance.hasConnectTime() + && instance.getConnectTime().getSeconds() < filter.getConnectedAfter().getSeconds()) { + return false; + } + if (filter.hasConnectedBefore() && instance.hasConnectTime() + && instance.getConnectTime().getSeconds() > filter.getConnectedBefore().getSeconds()) { + return false; + } + return true; + } + + /** + * D4 stable cursor pagination: instances are sorted by clientId and the cursor is the + * (opaque, base64-encoded) clientId of the last returned element. Membership churn + * between calls therefore cannot shift the window. + */ + private Page page(List all, int pageSize, String nextToken) { + int size = pageSize > 0 ? Math.min(pageSize, MAX_PAGE_SIZE) : DEFAULT_PAGE_SIZE; + all.sort(Comparator.comparing(ClientInstance::getClientId)); + String afterClientId = decodeCursor(nextToken); + int start = 0; + if (afterClientId != null) { + for (int i = 0; i < all.size(); i++) { + if (all.get(i).getClientId().compareTo(afterClientId) > 0) { + start = i; + break; + } + start = i + 1; + } + } + int end = Math.min(start + size, all.size()); + Page page = new Page(); + page.items = new ArrayList<>(all.subList(start, end)); + page.nextToken = end < all.size() && !page.items.isEmpty() + ? encodeCursor(page.items.get(page.items.size() - 1).getClientId()) : ""; + return page; + } + + private static String encodeCursor(String clientId) { + return CURSOR_PREFIX + Base64.getEncoder().encodeToString(clientId.getBytes(StandardCharsets.UTF_8)); + } + + private static String decodeCursor(String token) { + if (token == null || !token.startsWith(CURSOR_PREFIX)) { + return null; + } + try { + return new String(Base64.getDecoder().decode(token.substring(CURSOR_PREFIX.length())), + StandardCharsets.UTF_8); + } catch (Throwable t) { + return null; + } + } + + private static final class Page { + List items; + String nextToken; + } + + private static Timestamp toTimestamp(long millis) { + return Timestamp.newBuilder() + .setSeconds(millis / 1000) + .setNanos((int) ((millis % 1000) * 1_000_000)) + .build(); + } + + private static String str(Object address) { + return address == null ? "" : address.toString(); + } + + private static String resolveProxyEndpoint() { + try { + ProxyConfig config = ConfigurationManager.getProxyConfig(); + String addr = config.getLocalServeAddr(); + Integer port = config.getGrpcServerPort(); + String endpoint = (addr == null ? "" : addr) + (port == null ? "" : ":" + port); + if (endpoint.isEmpty()) { + endpoint = config.getProxyName(); + } + return endpoint; + } catch (Throwable t) { + return "rocketmq-proxy"; + } + } + + private Status success() { + return Status.newBuilder().setCode(Code.OK).build(); + } + + private Status fail(Code code, String message) { + return Status.newBuilder().setCode(code).setMessage(message == null ? "" : message).build(); + } +} diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/RouteChangeNotifier.java b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/RouteChangeNotifier.java new file mode 100644 index 00000000000..b15c1cfee27 --- /dev/null +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/admin/RouteChangeNotifier.java @@ -0,0 +1,332 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.rocketmq.proxy.grpc.admin; + +import apache.rocketmq.v2.BrokerInfo; +import apache.rocketmq.v2.Code; +import apache.rocketmq.v2.QueueInfo; +import apache.rocketmq.v2.RouteChangeEvent; +import apache.rocketmq.v2.RouteChangeEventType; +import apache.rocketmq.v2.Status; +import apache.rocketmq.v2.SubscribeRouteEventsRequest; +import apache.rocketmq.v2.SubscribeRouteEventsResponse; +import apache.rocketmq.v2.TopicRouteSnapshot; +import com.google.protobuf.Timestamp; +import io.grpc.stub.StreamObserver; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; +import java.util.concurrent.CopyOnWriteArrayList; +import org.apache.rocketmq.common.constant.LoggerName; +import org.apache.rocketmq.common.utils.StartAndShutdown; +import org.apache.rocketmq.logging.org.slf4j.Logger; +import org.apache.rocketmq.logging.org.slf4j.LoggerFactory; +import org.apache.rocketmq.proxy.service.route.MessageQueueView; +import org.apache.rocketmq.proxy.service.route.TopicRouteService; + +/** + * RIP-2 route observation: detects route changes from the proxy's topic route + * cache refreshes and streams them to admin subscribers (SubscribeRouteEvents). + * + *

This class is protocol-pure: it only works on the RIP-2 gRPC contract + * ({@code apache.rocketmq.v2.*} generated from rocketmq-apis). The translation + * from the broker-internal route representation into the v2 proto snapshot is + * delegated to {@link AdminModelConverter}. + * + *

Detected event types: + *

    + *
  • {@code ROUTE_SNAPSHOT} — full route snapshot emitted on first observation + * of a topic and immediately on subscribe (replay of current state);
    • + *
  • {@code TOPIC_CREATE} / {@code TOPIC_DELETE} — route appears / disappears;
    • + *
  • {@code QUEUE_SCALE} — read/write queue nums changed on a broker;
    • + *
  • {@code BROKER_ONLINE} / {@code BROKER_OFFLINE} — broker added/removed + * from the route (registration change on the NameServer).
    • + *
+ */ +public class RouteChangeNotifier implements TopicRouteService.RouteRefreshListener, StartAndShutdown { + + private static final Logger log = LoggerFactory.getLogger(LoggerName.PROXY_LOGGER_NAME); + + /** + * A live SubscribeRouteEvents stream with its filters. + */ + public final class Subscription { + private final Set topics; + private final Set eventTypes; + private final StreamObserver observer; + private volatile boolean cancelled; + + private Subscription(Set topics, Set eventTypes, + StreamObserver observer) { + this.topics = topics; + this.eventTypes = eventTypes; + this.observer = observer; + } + + public boolean isCancelled() { + return cancelled; + } + + void cancel() { + this.cancelled = true; + } + + private boolean matches(RouteChangeEvent event) { + if (!topics.isEmpty() && !topics.contains(event.getTopic())) { + return false; + } + return eventTypes.isEmpty() || eventTypes.contains(event.getEventType()); + } + + synchronized void deliver(RouteChangeEvent event) { + if (cancelled) { + return; + } + try { + observer.onNext(SubscribeRouteEventsResponse.newBuilder() + .setStatus(Status.newBuilder().setCode(Code.OK).build()) + .setEvent(event) + .build()); + } catch (Throwable t) { + cancelled = true; + subscriptions.remove(this); + log.info("RIP-2 route event subscriber dropped. cause:{}", t.getMessage()); + } + } + } + + private final List subscriptions = new CopyOnWriteArrayList<>(); + /** + * Baseline proto snapshots per topic used to diff refreshes into change events. + */ + private final ConcurrentMap baseline = new ConcurrentHashMap<>(); + + /** + * Subscribe to route events. Immediately replays ROUTE_SNAPSHOT events for the topics + * currently known (matching the request filter), then streams live changes. + */ + public Subscription subscribe(SubscribeRouteEventsRequest request, + StreamObserver observer, Map currentRoutes) { + Set topics = new HashSet<>(request.getTopicsList()); + Set types = new HashSet<>(request.getEventTypesList()); + Subscription subscription = new Subscription(topics, types, observer); + subscriptions.add(subscription); + // replay current state as ROUTE_SNAPSHOT so the consumer starts from a consistent view + for (Map.Entry entry : currentRoutes.entrySet()) { + TopicRouteSnapshot snapshot = snapshotOf(entry.getKey(), entry.getValue()); + if (snapshot == null) { + continue; + } + RouteChangeEvent snapshotEvent = baseEvent(RouteChangeEventType.ROUTE_SNAPSHOT, snapshot) + .setRouteSnapshot(snapshot) + .build(); + if (subscription.matches(snapshotEvent)) { + subscription.deliver(snapshotEvent); + } + } + return subscription; + } + + public void unsubscribe(Subscription subscription) { + if (subscription != null) { + subscription.cancel(); + subscriptions.remove(subscription); + } + } + + @Override + public void onRouteLoaded(String topic, MessageQueueView view) { + TopicRouteSnapshot snapshot = snapshotOf(topic, view); + TopicRouteSnapshot previous = snapshot == null ? baseline.remove(topic) : baseline.put(topic, snapshot); + if (snapshot == null) { + if (previous != null) { + publish(baseEvent(RouteChangeEventType.TOPIC_DELETE, previous).build()); + } + return; + } + if (previous == null) { + // first observation: emit a snapshot so subscribers learn the current state + publish(baseEvent(RouteChangeEventType.ROUTE_SNAPSHOT, snapshot) + .setRouteSnapshot(snapshot) + .build()); + } else { + diffAndPublish(previous, snapshot); + } + } + + @Override + public void onRouteRefreshed(String topic, MessageQueueView oldView, MessageQueueView newView) { + TopicRouteSnapshot oldSnapshot = snapshotOf(topic, oldView); + if (oldSnapshot == null) { + oldSnapshot = baseline.get(topic); + } + TopicRouteSnapshot newSnapshot = snapshotOf(topic, newView); + if (newSnapshot == null) { + baseline.remove(topic); + if (oldSnapshot != null) { + publish(baseEvent(RouteChangeEventType.TOPIC_DELETE, oldSnapshot).build()); + } + return; + } + baseline.put(topic, newSnapshot); + if (oldSnapshot == null) { + publish(baseEvent(RouteChangeEventType.TOPIC_CREATE, newSnapshot) + .setRouteSnapshot(newSnapshot) + .build()); + return; + } + diffAndPublish(oldSnapshot, newSnapshot); + } + + private void diffAndPublish(TopicRouteSnapshot oldSnapshot, TopicRouteSnapshot newSnapshot) { + String topic = newSnapshot.getTopic(); + + Map oldBrokers = brokersByName(oldSnapshot); + Map newBrokers = brokersByName(newSnapshot); + for (Map.Entry entry : newBrokers.entrySet()) { + if (!oldBrokers.containsKey(entry.getKey())) { + publish(baseEvent(RouteChangeEventType.BROKER_ONLINE, newSnapshot) + .setBrokerName(entry.getKey()) + .setBrokerAddress(masterAddr(entry.getValue())) + .build()); + } + } + for (Map.Entry entry : oldBrokers.entrySet()) { + if (!newBrokers.containsKey(entry.getKey())) { + publish(baseEvent(RouteChangeEventType.BROKER_OFFLINE, newSnapshot) + .setBrokerName(entry.getKey()) + .setBrokerAddress(masterAddr(entry.getValue())) + .build()); + } + } + + Map oldQueues = queuesByBroker(oldSnapshot); + Map newQueues = queuesByBroker(newSnapshot); + for (Map.Entry entry : newQueues.entrySet()) { + QueueInfo oldQueue = oldQueues.get(entry.getKey()); + QueueInfo newQueue = entry.getValue(); + if (oldQueue == null) { + continue; + } + if (oldQueue.getReadQueueNums() != newQueue.getReadQueueNums() + || oldQueue.getWriteQueueNums() != newQueue.getWriteQueueNums()) { + publish(baseEvent(RouteChangeEventType.QUEUE_SCALE, newSnapshot) + .setBrokerName(entry.getKey()) + .setPreviousReadQueueNums(oldQueue.getReadQueueNums()) + .setCurrentReadQueueNums(newQueue.getReadQueueNums()) + .setPreviousWriteQueueNums(oldQueue.getWriteQueueNums()) + .setCurrentWriteQueueNums(newQueue.getWriteQueueNums()) + .build()); + } + } + } + + /** + * Convert the proxy-internal route view into the v2 proto snapshot through the shared + * converter layer (the only place that touches broker-internal route types). + */ + private static TopicRouteSnapshot snapshotOf(String topic, MessageQueueView view) { + if (view == null || view.isEmptyCachedQueue()) { + return null; + } + return AdminModelConverter.toTopicRouteSnapshot(topic, view.getTopicRouteData()); + } + + private RouteChangeEvent.Builder baseEvent(RouteChangeEventType type, TopicRouteSnapshot snapshot) { + long now = System.currentTimeMillis(); + RouteChangeEvent.Builder builder = RouteChangeEvent.newBuilder() + .setEventType(type) + .setTimestamp(Timestamp.newBuilder().setSeconds(now / 1000).setNanos((int) ((now % 1000) * 1_000_000)).build()) + .setTopic(snapshot.getTopic()); + if (snapshot.getBrokersCount() > 0 && !snapshot.getBrokers(0).getCluster().isEmpty()) { + builder.setCluster(snapshot.getBrokers(0).getCluster()); + } + return builder; + } + + private void publish(RouteChangeEvent event) { + for (Subscription subscription : subscriptions) { + if (subscription.isCancelled()) { + subscriptions.remove(subscription); + continue; + } + if (subscription.matches(event)) { + subscription.deliver(event); + } + } + } + + private static Map brokersByName(TopicRouteSnapshot snapshot) { + Map map = new ConcurrentHashMap<>(); + for (BrokerInfo broker : snapshot.getBrokersList()) { + if (!broker.getBrokerName().isEmpty()) { + map.put(broker.getBrokerName(), broker); + } + } + return map; + } + + private static Map queuesByBroker(TopicRouteSnapshot snapshot) { + Map map = new ConcurrentHashMap<>(); + for (QueueInfo queue : snapshot.getQueuesList()) { + if (!queue.getBrokerName().isEmpty()) { + map.put(queue.getBrokerName(), queue); + } + } + return map; + } + + /** + * MixAll.MASTER_ID == 0; kept inline to avoid pulling a broker constant into the + * protocol-pure layer. + */ + private static String masterAddr(BrokerInfo broker) { + if (broker.getBrokerAddrsMap().containsKey(0L)) { + return broker.getBrokerAddrsMap().get(0L); + } + if (broker.getBrokerAddrsCount() > 0) { + return broker.getBrokerAddrsMap().values().iterator().next(); + } + return ""; + } + + public int getSubscriptionCount() { + return subscriptions.size(); + } + + @Override + public void start() throws Exception { + } + + @Override + public void shutdown() throws Exception { + List current = new ArrayList<>(subscriptions); + subscriptions.clear(); + for (Subscription subscription : current) { + subscription.cancel(); + try { + subscription.observer.onCompleted(); + } catch (Throwable ignore) { + // subscriber may already be gone + } + } + } +} diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/DefaultGrpcMessagingActivity.java b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/DefaultGrpcMessagingActivity.java index 88099207b93..fc602c99eaa 100644 --- a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/DefaultGrpcMessagingActivity.java +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/DefaultGrpcMessagingActivity.java @@ -97,6 +97,14 @@ protected void init(MessagingProcessor messagingProcessor) { this.appendStartAndShutdown(this.grpcClientSettingsManager); } + public GrpcChannelManager getGrpcChannelManager() { + return this.grpcChannelManager; + } + + public GrpcClientSettingsManager getGrpcClientSettingsManager() { + return this.grpcClientSettingsManager; + } + @Override public CompletableFuture queryRoute(ProxyContext ctx, QueryRouteRequest request) { return this.routeActivity.queryRoute(ctx, request); diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/GrpcMessagingApplication.java b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/GrpcMessagingApplication.java index 3429ad54e27..a2b1150bf2d 100644 --- a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/GrpcMessagingApplication.java +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/GrpcMessagingApplication.java @@ -78,6 +78,10 @@ public class GrpcMessagingApplication extends MessagingServiceGrpc.MessagingServ private final GrpcMessagingActivity grpcMessagingActivity; + public GrpcMessagingActivity getGrpcMessagingActivity() { + return this.grpcMessagingActivity; + } + protected final RequestPipeline requestPipeline; protected ThreadPoolExecutor routeThreadPoolExecutor; diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/channel/GrpcChannelManager.java b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/channel/GrpcChannelManager.java index a18cf7600c1..bb6a59a657c 100644 --- a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/channel/GrpcChannelManager.java +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/channel/GrpcChannelManager.java @@ -69,6 +69,10 @@ public GrpcClientChannel getChannel(String clientId) { return clientIdChannelMap.get(clientId); } + public java.util.Collection getClientChannels() { + return clientIdChannelMap.values(); + } + public GrpcClientChannel removeChannel(String clientId) { return this.clientIdChannelMap.remove(clientId); } diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/channel/GrpcClientChannel.java b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/channel/GrpcClientChannel.java index 0135818fb3b..0d20d2b3d7f 100644 --- a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/channel/GrpcClientChannel.java +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/channel/GrpcClientChannel.java @@ -16,6 +16,7 @@ */ package org.apache.rocketmq.proxy.grpc.v2.channel; +import apache.rocketmq.v2.HeartbeatRecord; import apache.rocketmq.v2.NotifyUnsubscribeLiteCommand; import apache.rocketmq.v2.PrintThreadStackTraceCommand; import apache.rocketmq.v2.RecoverOrphanedTransactionCommand; @@ -26,13 +27,18 @@ import com.google.common.collect.ComparisonChain; import com.google.protobuf.InvalidProtocolBufferException; import com.google.protobuf.TextFormat; +import com.google.protobuf.Timestamp; import com.google.protobuf.util.JsonFormat; import io.grpc.StatusRuntimeException; import io.grpc.stub.StreamObserver; import io.netty.channel.Channel; import io.netty.channel.ChannelId; +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.List; import java.util.Objects; import java.util.concurrent.CompletableFuture; +import java.util.concurrent.atomic.AtomicLong; import java.util.concurrent.atomic.AtomicReference; import org.apache.rocketmq.common.constant.LoggerName; import org.apache.rocketmq.common.message.MessageExt; @@ -69,6 +75,23 @@ public class GrpcClientChannel extends ProxyChannel implements ChannelExtendAttr private final Object telemetryWriteLock = new Object(); private final String clientId; + // RIP-2: capture the moment this client channel was created, surfaced in + // ProxyAdminService client listings (ClientInstance.connect_time). + private final long connectTimeMillis = System.currentTimeMillis(); + + // RIP-2: last activity observed on this channel (heartbeat / telemetry SETTINGS), + // surfaced in ClientInstance.last_active_time. + private final AtomicLong lastActiveTimeMillis = new AtomicLong(System.currentTimeMillis()); + + // RIP-2: bounded ring of recent heartbeat observations, surfaced in + // ClientDetail.recent_heartbeats. + private final ArrayDeque heartbeatHistory = new ArrayDeque<>(); + + // RIP-2: last observed authentication state of this connection, captured from the + // data-plane authentication pipeline; surfaced in ClientDetail.auth_status. + private volatile String authUsername; + private volatile long lastAuthTimeMillis; + public GrpcClientChannel(ProxyRelayService proxyRelayService, GrpcClientSettingsManager grpcClientSettingsManager, GrpcChannelManager grpcChannelManager, ProxyContext ctx, String clientId) { super(proxyRelayService, null, new GrpcChannelId(clientId), @@ -260,6 +283,72 @@ public String getClientId() { return clientId; } + public long getConnectTimeMillis() { + return connectTimeMillis; + } + + /** + * RIP-2: mark activity on this channel (heartbeat / telemetry). Updates + * {@code last_active_time} exposed by the admin service. + */ + public void touch() { + this.lastActiveTimeMillis.set(System.currentTimeMillis()); + } + + public long getLastActiveTimeMillis() { + return lastActiveTimeMillis.get(); + } + + /** + * RIP-2: record a heartbeat observation into the bounded history exposed via + * ClientDetail.recent_heartbeats. + */ + public void recordHeartbeat(boolean success, String remark) { + long now = System.currentTimeMillis(); + this.lastActiveTimeMillis.set(now); + HeartbeatRecord record = HeartbeatRecord.newBuilder() + .setTimestamp(Timestamp.newBuilder().setSeconds(now / 1000).setNanos((int) ((now % 1000) * 1_000_000)).build()) + .setSuccess(success) + .setRemark(remark == null ? "" : remark) + .build(); + int maxSize = ConfigurationManager.getProxyConfig().getProxyAdminHeartbeatHistorySize(); + synchronized (heartbeatHistory) { + heartbeatHistory.addLast(record); + while (maxSize <= 0 || heartbeatHistory.size() > maxSize) { + heartbeatHistory.pollFirst(); + } + } + } + + /** + * RIP-2: snapshot of recent heartbeat records, oldest first. + */ + public List getRecentHeartbeats() { + synchronized (heartbeatHistory) { + return new ArrayList<>(heartbeatHistory); + } + } + + /** + * RIP-2: record the authenticated username observed on this connection (captured after + * the data-plane authentication pipeline succeeded for one of its requests). + */ + public void recordAuthUsername(String username) { + if (username == null || username.isEmpty()) { + return; + } + this.authUsername = username; + this.lastAuthTimeMillis = System.currentTimeMillis(); + } + + public String getAuthUsername() { + return authUsername; + } + + public long getLastAuthTimeMillis() { + return lastAuthTimeMillis; + } + public void writeTelemetryCommand(TelemetryCommand command) { StreamObserver observer = this.telemetryCommandRef.get(); if (observer == null) { diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/client/ClientActivity.java b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/client/ClientActivity.java index abc23a53a3e..57426a48fb0 100644 --- a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/client/ClientActivity.java +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/client/ClientActivity.java @@ -33,6 +33,8 @@ import apache.rocketmq.v2.ThreadStackTrace; import apache.rocketmq.v2.VerifyMessageResult; import com.google.common.collect.ImmutableSet; +import io.grpc.Context; +import io.grpc.Metadata; import io.grpc.StatusRuntimeException; import io.grpc.stub.StreamObserver; import io.netty.channel.Channel; @@ -50,6 +52,7 @@ import org.apache.rocketmq.broker.client.ProducerGroupEvent; import org.apache.rocketmq.common.MQVersion; import org.apache.rocketmq.common.attribute.TopicMessageType; +import org.apache.rocketmq.common.constant.GrpcConstants; import org.apache.rocketmq.common.constant.LoggerName; import org.apache.rocketmq.common.consumer.ConsumeFromWhere; import org.apache.rocketmq.common.lite.LiteSubscriptionAction; @@ -99,6 +102,8 @@ public CompletableFuture heartbeat(ProxyContext ctx, Heartbea CompletableFuture future = new CompletableFuture<>(); try { + // RIP-2: heartbeat is the authoritative liveness signal for admin client views. + GrpcClientChannel adminChannel = touchAdminClientChannel(ctx); Settings clientSettings = grpcClientSettingsManager.getClientSettings(ctx); if (clientSettings == null) { future.complete(HeartbeatResponse.newBuilder() @@ -130,11 +135,18 @@ public CompletableFuture heartbeat(ProxyContext ctx, Heartbea return future; } } + if (adminChannel != null) { + adminChannel.recordHeartbeat(true, Code.OK.name()); + } future.complete(HeartbeatResponse.newBuilder() .setStatus(ResponseBuilder.getInstance().buildStatus(Code.OK, Code.OK.name())) .build()); return future; } catch (Throwable t) { + GrpcClientChannel failedChannel = touchAdminClientChannel(ctx); + if (failedChannel != null) { + failedChannel.recordHeartbeat(false, t.getMessage()); + } future.completeExceptionally(t); } return future; @@ -285,6 +297,9 @@ public void onNext(ProxyContext ctx, TelemetryCommand request) { try { switch (request.getCommandCase()) { case SETTINGS: { + // RIP-2: telemetry SETTINGS also proves liveness (initial handshake and + // every settings refresh), refresh the admin activity timestamp. + touchAdminClientChannel(ctx); processAndWriteClientSettings(ctx, request, responseObserver); break; } @@ -315,6 +330,36 @@ public void onCompleted() { }; } + /** + * RIP-2: mark the client channel active for the admin view and capture the authenticated + * username (written into the request metadata by the data-plane authentication pipeline). + * Never throws: admin tracking must not affect the data path. + */ + protected GrpcClientChannel touchAdminClientChannel(ProxyContext ctx) { + try { + String clientId = ctx.getClientID(); + if (StringUtils.isBlank(clientId)) { + return null; + } + GrpcClientChannel channel = this.grpcChannelManager.getChannel(clientId); + if (channel == null) { + return null; + } + channel.touch(); + Metadata metadata = GrpcConstants.METADATA.get(Context.current()); + if (metadata != null) { + String username = metadata.get(GrpcConstants.AUTHORIZATION_AK); + if (StringUtils.isNotBlank(username)) { + channel.recordAuthUsername(username); + } + } + return channel; + } catch (Throwable t) { + log.debug("RIP-2 admin channel tracking failed", t); + return null; + } + } + private static LiteSubscriptionAction toLiteAction(apache.rocketmq.v2.LiteSubscriptionAction gRpcAction) { switch (gRpcAction) { case PARTIAL_ADD: diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/common/GrpcConverter.java b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/common/GrpcConverter.java index 87d20ebca1b..695ac137d9c 100644 --- a/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/common/GrpcConverter.java +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/grpc/v2/common/GrpcConverter.java @@ -109,10 +109,11 @@ public Message buildMessage(MessageExt messageExt) { protected Map buildUserAttributes(MessageExt messageExt) { Map userAttributes = new HashMap<>(); Map properties = messageExt.getProperties(); - - for (Map.Entry property : properties.entrySet()) { - if (!MessageConst.STRING_HASH_SET.contains(property.getKey())) { - userAttributes.put(property.getKey(), property.getValue()); + if (properties != null) { + for (Map.Entry property : properties.entrySet()) { + if (!MessageConst.STRING_HASH_SET.contains(property.getKey())) { + userAttributes.put(property.getKey(), property.getValue()); + } } } @@ -161,7 +162,9 @@ protected SystemProperties buildSystemProperties(MessageExt messageExt) { } // message_type - TopicMessageType topicMessageType = TopicMessageType.parseFromMessageProperty(messageExt.getProperties()); + Map properties = messageExt.getProperties(); + TopicMessageType topicMessageType = TopicMessageType.parseFromMessageProperty( + properties != null ? properties : new HashMap()); systemPropertiesBuilder.setMessageType(convertToGrpcMessageType(topicMessageType)); // born_timestamp (millis) diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/processor/DefaultMessagingProcessor.java b/proxy/src/main/java/org/apache/rocketmq/proxy/processor/DefaultMessagingProcessor.java index a56bc42596b..5cec246f9c0 100644 --- a/proxy/src/main/java/org/apache/rocketmq/proxy/processor/DefaultMessagingProcessor.java +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/processor/DefaultMessagingProcessor.java @@ -385,6 +385,17 @@ public MetadataService getMetadataService() { return this.serviceManager.getMetadataService(); } + public ServiceManager getServiceManager() { + return this.serviceManager; + } + + /** + * RIP-2 M3: access to the receipt handle processor for admin pop-handle diagnostics. + */ + public ReceiptHandleProcessor getReceiptHandleProcessor() { + return this.receiptHandleProcessor; + } + @Override public void addReceiptHandle(ProxyContext ctx, Channel channel, String group, String msgID, MessageReceiptHandle messageReceiptHandle) { diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/processor/ReceiptHandleProcessor.java b/proxy/src/main/java/org/apache/rocketmq/proxy/processor/ReceiptHandleProcessor.java index bc3730aed9a..97491f1d1bf 100644 --- a/proxy/src/main/java/org/apache/rocketmq/proxy/processor/ReceiptHandleProcessor.java +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/processor/ReceiptHandleProcessor.java @@ -33,6 +33,13 @@ public class ReceiptHandleProcessor extends AbstractProcessor { protected final static Logger log = LoggerFactory.getLogger(LoggerName.PROXY_LOGGER_NAME); protected DefaultReceiptHandleManager receiptHandleManager; + /** + * RIP-2 M3: read-only access to the receipt handle manager for admin diagnostics. + */ + public DefaultReceiptHandleManager getReceiptHandleManager() { + return receiptHandleManager; + } + public ReceiptHandleProcessor(MessagingProcessor messagingProcessor, ServiceManager serviceManager) { super(messagingProcessor, serviceManager); StateEventListener eventListener = event -> { diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/service/admin/AdminService.java b/proxy/src/main/java/org/apache/rocketmq/proxy/service/admin/AdminService.java index a9e6686b438..532b05c41cc 100644 --- a/proxy/src/main/java/org/apache/rocketmq/proxy/service/admin/AdminService.java +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/service/admin/AdminService.java @@ -18,6 +18,10 @@ package org.apache.rocketmq.proxy.service.admin; import java.util.List; +import java.util.Map; +import org.apache.rocketmq.common.message.MessageExt; +import org.apache.rocketmq.common.message.MessageQueue; +import org.apache.rocketmq.remoting.protocol.admin.ConsumeStats; import org.apache.rocketmq.remoting.protocol.route.BrokerData; public interface AdminService { @@ -29,4 +33,40 @@ boolean createTopicOnTopicBrokerIfNotExist(String createTopic, String sampleTopi boolean createTopicOnBroker(String topic, int wQueueNum, int rQueueNum, List curBrokerDataList, List sampleBrokerDataList, boolean examineTopic, int retryCheckCount) throws Exception; + + // ========================================================================= + // RIP-2 Admin: broker-facing gateway methods. + // + // IMPORTANT: every implementation delegates to the proxy's OWN managed + // broker client (rocketmq-proxy's MQClientAPIFactory). The proxy is the + // single entry point (via the gRPC proxy); these calls never open a direct + // link to the broker from the admin code itself. + // ========================================================================= + + long getMaxOffset(String brokerAddr, MessageQueue messageQueue, long timeoutMillis) throws Exception; + + long getMinOffset(String brokerAddr, MessageQueue messageQueue, long timeoutMillis) throws Exception; + + long getEarliestMsgStoretime(String brokerAddr, MessageQueue messageQueue, long timeoutMillis) throws Exception; + + ConsumeStats fetchConsumeStats(String brokerAddr, String consumerGroup, String topic, long timeoutMillis) throws Exception; + + Map resetOffset(String brokerAddr, String topic, String group, long timestamp, + boolean isForce, long timeoutMillis) throws Exception; + + /** + * RIP-2: delete a subscription group on the target broker (optionally cleaning its + * offsets), going through the proxy's own managed broker client. + */ + void deleteSubscriptionGroup(String brokerAddr, String group, boolean removeOffset, + long timeoutMillis) throws Exception; + + List queryMessage(String brokerAddr, String topic, String key, int maxNum, + long beginTimestamp, long endTimestamp, long timeoutMillis) throws Exception; + + MessageExt viewMessage(String brokerAddr, String topic, long phyoffset, long timeoutMillis) throws Exception; + + org.apache.rocketmq.remoting.protocol.statictopic.TopicConfigAndQueueMapping getTopicConfig(String brokerAddr, String topic, long timeoutMillis) throws Exception; + + org.apache.rocketmq.remoting.protocol.route.TopicRouteData getTopicRouteData(String topic) throws Exception; } diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/service/admin/DefaultAdminService.java b/proxy/src/main/java/org/apache/rocketmq/proxy/service/admin/DefaultAdminService.java index f3c68eab5c4..7430bccd7ec 100644 --- a/proxy/src/main/java/org/apache/rocketmq/proxy/service/admin/DefaultAdminService.java +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/service/admin/DefaultAdminService.java @@ -18,13 +18,24 @@ package org.apache.rocketmq.proxy.service.admin; import java.time.Duration; +import java.util.ArrayList; import java.util.HashSet; import java.util.List; +import java.util.Map; import java.util.Set; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.TimeUnit; import org.apache.rocketmq.common.MixAll; import org.apache.rocketmq.common.TopicConfig; import org.apache.rocketmq.common.constant.LoggerName; import org.apache.rocketmq.common.constant.PermName; +import org.apache.rocketmq.common.message.MessageDecoder; +import org.apache.rocketmq.common.message.MessageExt; +import org.apache.rocketmq.common.message.MessageQueue; +import org.apache.rocketmq.remoting.protocol.ResponseCode; +import org.apache.rocketmq.remoting.netty.ResponseFuture; +import org.apache.rocketmq.remoting.protocol.RemotingCommand; +import org.apache.rocketmq.remoting.protocol.admin.ConsumeStats; import org.apache.rocketmq.remoting.protocol.route.BrokerData; import org.apache.rocketmq.remoting.protocol.route.TopicRouteData; import org.apache.rocketmq.common.topic.TopicValidator; @@ -33,6 +44,8 @@ import org.apache.rocketmq.client.impl.mqclient.MQClientAPIExt; import org.apache.rocketmq.client.impl.mqclient.MQClientAPIFactory; import org.apache.rocketmq.proxy.service.route.TopicRouteHelper; +import org.apache.rocketmq.remoting.InvokeCallback; +import org.apache.rocketmq.remoting.protocol.header.QueryMessageRequestHeader; public class DefaultAdminService implements AdminService { private static final Logger log = LoggerFactory.getLogger(LoggerName.PROXY_LOGGER_NAME); @@ -143,4 +156,92 @@ protected TopicRouteData getTopicRouteDataDirectlyFromNameServer(String topic) t protected MQClientAPIExt getClient() { return this.mqClientAPIFactory.getClient(); } + + // ========================================================================= + // RIP-2 Admin: broker-facing gateway methods. + // Every call goes through the proxy's OWN managed broker client. + // ========================================================================= + + @Override + public long getMaxOffset(String brokerAddr, MessageQueue messageQueue, long timeoutMillis) throws Exception { + return this.getClient().getMaxOffset(brokerAddr, messageQueue, timeoutMillis); + } + + @Override + public long getMinOffset(String brokerAddr, MessageQueue messageQueue, long timeoutMillis) throws Exception { + return this.getClient().getMinOffset(brokerAddr, messageQueue, timeoutMillis); + } + + @Override + public long getEarliestMsgStoretime(String brokerAddr, MessageQueue messageQueue, long timeoutMillis) throws Exception { + return this.getClient().getEarliestMsgStoretime(brokerAddr, messageQueue, timeoutMillis); + } + + @Override + public ConsumeStats fetchConsumeStats(String brokerAddr, String consumerGroup, String topic, long timeoutMillis) throws Exception { + return this.getClient().getConsumeStats(brokerAddr, consumerGroup, topic, timeoutMillis); + } + + @Override + public Map resetOffset(String brokerAddr, String topic, String group, long timestamp, + boolean isForce, long timeoutMillis) throws Exception { + return this.getClient().invokeBrokerToResetOffset(brokerAddr, topic, group, timestamp, isForce, timeoutMillis); + } + + @Override + public void deleteSubscriptionGroup(String brokerAddr, String group, boolean removeOffset, + long timeoutMillis) throws Exception { + this.getClient().deleteSubscriptionGroup(brokerAddr, group, removeOffset, timeoutMillis); + } + + @Override + public MessageExt viewMessage(String brokerAddr, String topic, long phyoffset, long timeoutMillis) throws Exception { + return this.getClient().viewMessage(brokerAddr, topic, phyoffset, timeoutMillis); + } + + @Override + public org.apache.rocketmq.remoting.protocol.statictopic.TopicConfigAndQueueMapping getTopicConfig(String brokerAddr, String topic, long timeoutMillis) throws Exception { + return this.getClient().getTopicConfig(brokerAddr, topic, timeoutMillis); + } + + @Override + public org.apache.rocketmq.remoting.protocol.route.TopicRouteData getTopicRouteData(String topic) throws Exception { + return this.getTopicRouteDataDirectlyFromNameServer(topic); + } + + @Override + public List queryMessage(String brokerAddr, String topic, String key, int maxNum, + long beginTimestamp, long endTimestamp, long timeoutMillis) throws Exception { + QueryMessageRequestHeader requestHeader = new QueryMessageRequestHeader(); + requestHeader.setTopic(topic); + requestHeader.setKey(key); + requestHeader.setMaxNum(maxNum); + requestHeader.setBeginTimestamp(beginTimestamp); + requestHeader.setEndTimestamp(endTimestamp); + + CompletableFuture> future = new CompletableFuture<>(); + this.getClient().queryMessage(brokerAddr, requestHeader, timeoutMillis, new InvokeCallback() { + @Override + public void operationComplete(ResponseFuture responseFuture) { + try { + RemotingCommand response = responseFuture.getResponseCommand(); + if (response != null && response.getCode() == ResponseCode.SUCCESS && response.getBody() != null) { + List messageList = MessageDecoder.decodes( + java.nio.ByteBuffer.wrap(response.getBody()), true); + future.complete(messageList); + } else { + future.complete(new ArrayList<>()); + } + } catch (Throwable t) { + future.completeExceptionally(t); + } + } + + @Override + public void operationFail(Throwable e) { + future.completeExceptionally(e); + } + }, false); + return future.get(timeoutMillis, TimeUnit.MILLISECONDS); + } } diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/service/receipt/DefaultReceiptHandleManager.java b/proxy/src/main/java/org/apache/rocketmq/proxy/service/receipt/DefaultReceiptHandleManager.java index f9dfd825337..ccf41a37604 100644 --- a/proxy/src/main/java/org/apache/rocketmq/proxy/service/receipt/DefaultReceiptHandleManager.java +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/service/receipt/DefaultReceiptHandleManager.java @@ -150,6 +150,21 @@ public int getUnackedMessageCount(ProxyContext context, Channel channel, String return handleGroup == null ? 0 : handleGroup.getMsgCount(); } + @Override + public void scanReceiptHandles(ReceiptHandleScanVisitor visitor) { + if (visitor == null) { + return; + } + for (Map.Entry entry : receiptHandleGroupMap.entrySet()) { + ReceiptHandleGroupKey groupKey = entry.getKey(); + try { + entry.getValue().scan((msgID, handleStr, handle) -> visitor.onHandle(groupKey, handle)); + } catch (Throwable t) { + log.warn("RIP-2 receipt handle scan failed for group key:{}", groupKey, t); + } + } + } + protected boolean clientIsOffline(ReceiptHandleGroupKey groupKey) { return this.consumerManager.findChannel(groupKey.getGroup(), groupKey.getChannel()) == null; } diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/service/receipt/ReceiptHandleManager.java b/proxy/src/main/java/org/apache/rocketmq/proxy/service/receipt/ReceiptHandleManager.java index 16ad57b07d9..5527ff10bb5 100644 --- a/proxy/src/main/java/org/apache/rocketmq/proxy/service/receipt/ReceiptHandleManager.java +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/service/receipt/ReceiptHandleManager.java @@ -20,6 +20,7 @@ import io.netty.channel.Channel; import org.apache.rocketmq.proxy.common.MessageReceiptHandle; import org.apache.rocketmq.proxy.common.ProxyContext; +import org.apache.rocketmq.proxy.common.ReceiptHandleGroupKey; public interface ReceiptHandleManager { void addReceiptHandle(ProxyContext context, Channel channel, String group, String msgID, MessageReceiptHandle messageReceiptHandle); @@ -27,4 +28,19 @@ public interface ReceiptHandleManager { MessageReceiptHandle removeReceiptHandle(ProxyContext context, Channel channel, String group, String msgID, String receiptHandle); int getUnackedMessageCount(ProxyContext context, Channel channel, String group); + + /** + * RIP-2 M3: read-only scan over all tracked receipt handles for diagnostics. The visitor + * receives the owning channel-group key and each handle; implementations must not block + * the renew pipeline while scanning. + */ + default void scanReceiptHandles(ReceiptHandleScanVisitor visitor) { + } + + /** + * Visitor for {@link #scanReceiptHandles(ReceiptHandleScanVisitor)}. + */ + interface ReceiptHandleScanVisitor { + void onHandle(ReceiptHandleGroupKey groupKey, MessageReceiptHandle handle); + } } diff --git a/proxy/src/main/java/org/apache/rocketmq/proxy/service/route/TopicRouteService.java b/proxy/src/main/java/org/apache/rocketmq/proxy/service/route/TopicRouteService.java index dae30057461..ddecd706ff2 100644 --- a/proxy/src/main/java/org/apache/rocketmq/proxy/service/route/TopicRouteService.java +++ b/proxy/src/main/java/org/apache/rocketmq/proxy/service/route/TopicRouteService.java @@ -22,7 +22,9 @@ import com.google.common.annotations.VisibleForTesting; import java.time.Duration; import java.util.ArrayList; +import java.util.HashMap; import java.util.List; +import java.util.Map; import java.util.Optional; import java.util.concurrent.ThreadPoolExecutor; import java.util.concurrent.TimeUnit; @@ -51,11 +53,56 @@ public abstract class TopicRouteService extends AbstractStartAndShutdown { private static final Logger log = LoggerFactory.getLogger(LoggerName.PROXY_LOGGER_NAME); + /** + * RIP-2: observer hook invoked whenever a topic route is loaded or refreshed from the + * NameServer, powering the admin SubscribeRouteEvents / DescribeRouteTopology surface. + */ + public interface RouteRefreshListener { + /** + * Invoked when a topic route is loaded for the first time (or re-loaded after expiry). + */ + default void onRouteLoaded(String topic, MessageQueueView view) { + } + + /** + * Invoked when a cached topic route is refreshed; {@code oldView} is the previous value. + */ + default void onRouteRefreshed(String topic, MessageQueueView oldView, MessageQueueView newView) { + } + } + private final MQFaultStrategy mqFaultStrategy; protected final LoadingCache topicCache; protected final ThreadPoolExecutor cacheRefreshExecutor; protected final List> penalizers = new ArrayList<>(); protected MessageQueuePriorityProvider priorityProvider = new DefaultMessageQueuePriorityProvider(); + private final List routeRefreshListeners = new java.util.concurrent.CopyOnWriteArrayList<>(); + + public void addRouteRefreshListener(RouteRefreshListener listener) { + if (listener != null) { + this.routeRefreshListeners.add(listener); + } + } + + private void notifyRouteLoaded(String topic, MessageQueueView view) { + for (RouteRefreshListener listener : routeRefreshListeners) { + try { + listener.onRouteLoaded(topic, view); + } catch (Throwable t) { + log.warn("RIP-2 route refresh listener failed. topic:{}", topic, t); + } + } + } + + private void notifyRouteRefreshed(String topic, MessageQueueView oldView, MessageQueueView newView) { + for (RouteRefreshListener listener : routeRefreshListeners) { + try { + listener.onRouteRefreshed(topic, oldView, newView); + } catch (Throwable t) { + log.warn("RIP-2 route refresh listener failed. topic:{}", topic, t); + } + } + } public TopicRouteService(MQClientAPIFactory mqClientAPIFactory) { ProxyConfig config = ConfigurationManager.getProxyConfig(); @@ -78,9 +125,12 @@ public TopicRouteService(MQClientAPIFactory mqClientAPIFactory) { public @Nullable MessageQueueView load(String topic) throws Exception { try { TopicRouteData topicRouteData = mqClientAPIFactory.getClient().getTopicRouteInfoFromNameServer(topic, Duration.ofSeconds(3).toMillis()); - return buildMessageQueueView(topic, topicRouteData); + MessageQueueView view = buildMessageQueueView(topic, topicRouteData); + notifyRouteLoaded(topic, view); + return view; } catch (Exception e) { if (TopicRouteHelper.isTopicNotExistError(e)) { + notifyRouteLoaded(topic, MessageQueueView.WRAPPED_EMPTY_QUEUE); return MessageQueueView.WRAPPED_EMPTY_QUEUE; } throw e; @@ -91,8 +141,16 @@ public TopicRouteService(MQClientAPIFactory mqClientAPIFactory) { public @Nullable MessageQueueView reload(@NonNull String key, @NonNull MessageQueueView oldValue) throws Exception { try { - return load(key); + TopicRouteData topicRouteData = mqClientAPIFactory.getClient() + .getTopicRouteInfoFromNameServer(key, Duration.ofSeconds(3).toMillis()); + MessageQueueView newValue = buildMessageQueueView(key, topicRouteData); + notifyRouteRefreshed(key, oldValue, newValue); + return newValue; } catch (Exception e) { + if (TopicRouteHelper.isTopicNotExistError(e)) { + notifyRouteRefreshed(key, oldValue, MessageQueueView.WRAPPED_EMPTY_QUEUE); + return MessageQueueView.WRAPPED_EMPTY_QUEUE; + } log.warn(String.format("reload topic route from namesrv. topic: %s", key), e); return oldValue; } @@ -174,6 +232,15 @@ public MessageQueueView getAllMessageQueueView(ProxyContext ctx, String topicNam return getCacheMessageQueueWrapper(this.topicCache, topicName); } + /** + * RIP-2: read-only snapshot of the cached topic route views, used by the admin + * DescribeRouteTopology surface. The returned map is a detached copy. + */ + public Map snapshotTopicRouteCache() { + return new HashMap<>(this.topicCache.asMap()); + } + + public abstract MessageQueueView getCurrentMessageQueueView(ProxyContext ctx, String topicName) throws Exception; public abstract ProxyTopicRouteData getTopicRouteForProxy(ProxyContext ctx, List
requestHostAndPortList, diff --git a/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/AdminModelConverterTest.java b/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/AdminModelConverterTest.java new file mode 100644 index 00000000000..0cafdee5d1b --- /dev/null +++ b/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/AdminModelConverterTest.java @@ -0,0 +1,211 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.rocketmq.proxy.grpc.admin; + +import apache.rocketmq.v2.Code; +import apache.rocketmq.v2.DescribeGroupAccumulationResponse; +import apache.rocketmq.v2.DescribeTopicStatusResponse; +import apache.rocketmq.v2.GetTopicRouteResponse; +import apache.rocketmq.v2.Message; +import apache.rocketmq.v2.MessageQueue; +import apache.rocketmq.v2.QueryTimeSpanResponse; +import com.alibaba.fastjson.JSON; +import java.util.Arrays; +import java.util.HashMap; +import java.util.Map; +import org.apache.rocketmq.common.message.MessageExt; +import org.apache.rocketmq.proxy.service.admin.AdminService; +import org.apache.rocketmq.proxy.service.route.AddressableMessageQueue; +import org.apache.rocketmq.proxy.service.route.MessageQueueSelector; +import org.apache.rocketmq.proxy.service.route.MessageQueueView; +import org.apache.rocketmq.remoting.protocol.admin.ConsumeStats; +import org.apache.rocketmq.remoting.protocol.admin.OffsetWrapper; +import org.apache.rocketmq.remoting.protocol.route.TopicRouteData; +import org.apache.rocketmq.remoting.protocol.statictopic.TopicConfigAndQueueMapping; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.Mock; +import org.mockito.junit.MockitoJUnitRunner; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyLong; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.when; + +@RunWith(MockitoJUnitRunner.class) +public class AdminModelConverterTest { + + @Mock + private AdminService adminService; + @Mock + private MessageQueueView messageQueueView; + @Mock + private MessageQueueSelector messageQueueSelector; + + private static final String TOPIC = "topicA"; + private static final String GROUP = "groupA"; + private static final String BROKER_ADDR = "127.0.0.1:10911"; + + @Before + public void setUp() { + } + + @Test + public void toGroupAccumulationSumsDiff() throws Exception { + ConsumeStats consumeStats = new ConsumeStats(); + Map table = new HashMap<>(); + OffsetWrapper wrapper = new OffsetWrapper(); + wrapper.setBrokerOffset(100); + wrapper.setConsumerOffset(60); + table.put(new org.apache.rocketmq.common.message.MessageQueue(TOPIC, "broker-a", 0), wrapper); + consumeStats.setOffsetTable(table); + + when(adminService.fetchConsumeStats(eq(BROKER_ADDR), eq(GROUP), eq(TOPIC), anyLong())) + .thenReturn(consumeStats); + + DescribeGroupAccumulationResponse.GroupAccumulation accumulation = + AdminModelConverter.toGroupAccumulation(adminService, BROKER_ADDR, GROUP, TOPIC, 3000L); + + assertNotNull(accumulation); + assertEquals(40L, accumulation.getAccumulation()); + assertEquals(40L, accumulation.getReadyMessages()); + } + + @Test + public void toGroupAccumulationNullConsumeStats() throws Exception { + when(adminService.fetchConsumeStats(eq(BROKER_ADDR), eq(GROUP), eq(TOPIC), anyLong())) + .thenReturn(null); + + DescribeGroupAccumulationResponse.GroupAccumulation accumulation = + AdminModelConverter.toGroupAccumulation(adminService, BROKER_ADDR, GROUP, TOPIC, 3000L); + + assertNotNull(accumulation); + assertEquals(0L, accumulation.getAccumulation()); + assertEquals(0L, accumulation.getReadyMessages()); + } + + @Test + public void toTopicRouteReturnsJson() throws Exception { + TopicRouteData topicRouteData = new TopicRouteData(); + topicRouteData.setOrderTopicConf("orderConf"); + when(adminService.getTopicRouteData(eq(TOPIC))).thenReturn(topicRouteData); + + GetTopicRouteResponse response = AdminModelConverter.toTopicRoute(adminService, TOPIC); + + assertEquals(Code.OK, response.getStatus().getCode()); + assertEquals(JSON.toJSONString(topicRouteData), response.getTopicRouteData()); + } + + @Test + public void toTopicStatusReturnsConfig() throws Exception { + TopicConfigAndQueueMapping topicConfig = new TopicConfigAndQueueMapping(); + topicConfig.setTopicName(TOPIC); + topicConfig.setReadQueueNums(8); + topicConfig.setWriteQueueNums(16); + topicConfig.setPerm(6); + when(adminService.getTopicConfig(eq(BROKER_ADDR), eq(TOPIC), anyLong())).thenReturn(topicConfig); + + DescribeTopicStatusResponse response = + AdminModelConverter.toTopicStatus(adminService, BROKER_ADDR, TOPIC, 3000L); + + assertEquals(Code.OK, response.getStatus().getCode()); + assertTrue(response.getDescription().contains(TOPIC)); + assertTrue(response.getDescription().contains("readQueues=8")); + assertTrue(response.getDescription().contains("writeQueues=16")); + } + + @Test + public void toTopicStatusNullConfig() throws Exception { + when(adminService.getTopicConfig(eq(BROKER_ADDR), eq(TOPIC), anyLong())).thenReturn(null); + + DescribeTopicStatusResponse response = + AdminModelConverter.toTopicStatus(adminService, BROKER_ADDR, TOPIC, 3000L); + + assertEquals(Code.OK, response.getStatus().getCode()); + assertTrue(response.getDescription().isEmpty()); + } + + @Test + public void toQueryTimeSpanReturnsPerQueueSpan() throws Exception { + AddressableMessageQueue amq = new AddressableMessageQueue( + new org.apache.rocketmq.common.message.MessageQueue(TOPIC, "broker-a", 0), BROKER_ADDR); + when(messageQueueView.getReadSelector()).thenReturn(messageQueueSelector); + when(messageQueueSelector.getQueues()).thenReturn(Arrays.asList(amq)); + when(adminService.getEarliestMsgStoretime(eq(BROKER_ADDR), any(), anyLong())).thenReturn(12345L); + + QueryTimeSpanResponse response = AdminModelConverter.toQueryTimeSpan( + adminService, BROKER_ADDR, GROUP, TOPIC, messageQueueView, 3000L); + + assertEquals(Code.OK, response.getStatus().getCode()); + assertEquals(1, response.getQueueTimeSpanListCount()); + QueryTimeSpanResponse.QueueTimeSpan span = response.getQueueTimeSpanList(0); + assertEquals(12345L, span.getMinTimestamp()); + assertEquals("broker-a", span.getMessageQueue().getBroker().getName()); + } + + @Test + public void toMessageConvertsFields() { + MessageExt ext = new MessageExt(); + ext.setTopic(TOPIC); + ext.setMsgId("msg-1"); + ext.setTags("tagA"); + ext.setKeys("k1 k2"); + ext.setBody("hello".getBytes()); + + Message message = AdminModelConverter.toMessage(ext); + + assertNotNull(message); + assertEquals(TOPIC, message.getTopic().getName()); + assertEquals("msg-1", message.getSystemProperties().getMessageId()); + assertEquals("tagA", message.getSystemProperties().getTag()); + assertEquals(Arrays.asList("k1", "k2"), message.getSystemProperties().getKeysList()); + assertEquals("hello", new String(message.getBody().toByteArray())); + } + + @Test + public void toMessageNullReturnsNull() { + assertNull(AdminModelConverter.toMessage(null)); + } + + @Test + public void toMessageQueueSetsBrokerAndId() { + org.apache.rocketmq.common.message.MessageQueue mq = + new org.apache.rocketmq.common.message.MessageQueue(TOPIC, "broker-a", 3); + + MessageQueue v2 = AdminModelConverter.toMessageQueue(mq); + + assertEquals("broker-a", v2.getBroker().getName()); + assertEquals(3, v2.getId()); + assertEquals(TOPIC, v2.getTopic().getName()); + } + + @Test + public void toMessageQueueEmptyBrokerName() { + org.apache.rocketmq.common.message.MessageQueue mq = + new org.apache.rocketmq.common.message.MessageQueue(TOPIC, "", 0); + + MessageQueue v2 = AdminModelConverter.toMessageQueue(mq); + + assertEquals("", v2.getBroker().getName()); + assertEquals(0, v2.getId()); + } +} diff --git a/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminAuthInterceptorTest.java b/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminAuthInterceptorTest.java new file mode 100644 index 00000000000..bd4e2bb6a21 --- /dev/null +++ b/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminAuthInterceptorTest.java @@ -0,0 +1,234 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.rocketmq.proxy.grpc.admin; + +import io.grpc.Metadata; +import io.grpc.MethodDescriptor; +import io.grpc.ServerCall; +import io.grpc.ServerCallHandler; +import io.grpc.Status; +import java.io.ByteArrayInputStream; +import java.io.InputStream; +import java.util.HashSet; +import java.util.Set; +import org.apache.rocketmq.auth.config.AuthConfig; +import org.apache.rocketmq.common.action.Action; +import org.apache.rocketmq.proxy.config.ConfigurationManager; +import org.apache.rocketmq.proxy.config.InitConfigTest; +import org.apache.rocketmq.proxy.processor.MessagingProcessor; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.Mock; +import org.mockito.junit.MockitoJUnitRunner; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@RunWith(MockitoJUnitRunner.class) +public class ProxyAdminAuthInterceptorTest extends InitConfigTest { + + @Mock + private MessagingProcessor messagingProcessor; + + private static final MethodDescriptor.Marshaller BYTE_MARSHALLER = + new MethodDescriptor.Marshaller() { + @Override + public InputStream stream(byte[] value) { + return new ByteArrayInputStream(value == null ? new byte[0] : value); + } + + @Override + public byte[] parse(InputStream stream) { + return new byte[0]; + } + }; + + private static MethodDescriptor method(String name) { + return MethodDescriptor.newBuilder() + .setType(MethodDescriptor.MethodType.UNARY) + .setFullMethodName("apache.rocketmq.v2.ProxyAdminService/" + name) + .setRequestMarshaller(BYTE_MARSHALLER) + .setResponseMarshaller(BYTE_MARSHALLER) + .build(); + } + + @SuppressWarnings("unchecked") + private static ServerCall serverCall(String methodName) { + ServerCall call = mock(ServerCall.class); + when(call.getMethodDescriptor()).thenReturn(method(methodName)); + return call; + } + + // --------------------------------------------------------------------- + // per-method resource / action mapping (read-only vs high-privilege isolation) + // --------------------------------------------------------------------- + + @Test + public void allAdminMethodsAreMapped() { + String[] methods = { + "ListClients", "ListClientsByGroup", "ListClientsByTopic", "DescribeClient", + "DescribeProxyConfig", "UpdateProxyConfig", "KickClient", "DisconnectChannel", + "DescribeQuota", "UpdateQuota", "DescribePopReceiptHandles", "DescribeBatchConsumeDiagnostics", + "SubscribeRouteEvents", "DescribeRouteTopology", + "GetProxyRuntimeStats", "GetTopicRoute", "DescribeTopicStatus", "ListSubscription", + "DescribeSubscription", "ListConsumerConnection", "DescribeGroupAccumulation", + "GetConsumerRunningInfo", "QueryTimeSpan", "QueryMessage", "ChangeLogLevel", + "DeleteSubscription", "ResetGroupOffset", "AdminSendMessage", "PrintThreadStackTrace", + "VerifyMessage"}; + for (String m : methods) { + assertNotNull("missing permission mapping for " + m, + ProxyAdminAuthInterceptor.resolveResourceAction(m)); + } + } + + @Test + public void highPrivilegeOperationsNeverMapToReadActions() { + assertEquals(Action.UPDATE, ProxyAdminAuthInterceptor.resolveResourceAction("KickClient").action); + assertEquals(Action.UPDATE, ProxyAdminAuthInterceptor.resolveResourceAction("DisconnectChannel").action); + assertEquals(Action.UPDATE, ProxyAdminAuthInterceptor.resolveResourceAction("UpdateProxyConfig").action); + assertEquals(Action.UPDATE, ProxyAdminAuthInterceptor.resolveResourceAction("UpdateQuota").action); + assertEquals(Action.UPDATE, ProxyAdminAuthInterceptor.resolveResourceAction("ResetGroupOffset").action); + assertEquals(Action.UPDATE, ProxyAdminAuthInterceptor.resolveResourceAction("ChangeLogLevel").action); + assertEquals(Action.DELETE, ProxyAdminAuthInterceptor.resolveResourceAction("DeleteSubscription").action); + assertEquals(Action.PUB, ProxyAdminAuthInterceptor.resolveResourceAction("AdminSendMessage").action); + } + + @Test + public void readOnlyOperationsMapToReadActions() { + assertEquals(Action.LIST, ProxyAdminAuthInterceptor.resolveResourceAction("ListClients").action); + assertEquals(Action.GET, ProxyAdminAuthInterceptor.resolveResourceAction("DescribeClient").action); + assertEquals(Action.GET, ProxyAdminAuthInterceptor.resolveResourceAction("DescribeProxyConfig").action); + assertEquals(Action.GET, ProxyAdminAuthInterceptor.resolveResourceAction("DescribeQuota").action); + assertEquals(Action.LIST, ProxyAdminAuthInterceptor.resolveResourceAction("SubscribeRouteEvents").action); + assertEquals(Action.GET, ProxyAdminAuthInterceptor.resolveResourceAction("DescribeRouteTopology").action); + } + + @Test + public void resourcesAreScopedPerModule() { + assertEquals(ProxyAdminAuthInterceptor.RESOURCE_CLIENT, + ProxyAdminAuthInterceptor.resolveResourceAction("ListClients").resource); + assertEquals(ProxyAdminAuthInterceptor.RESOURCE_CONFIG, + ProxyAdminAuthInterceptor.resolveResourceAction("UpdateProxyConfig").resource); + assertEquals(ProxyAdminAuthInterceptor.RESOURCE_CONNECTION, + ProxyAdminAuthInterceptor.resolveResourceAction("KickClient").resource); + assertEquals(ProxyAdminAuthInterceptor.RESOURCE_QUOTA, + ProxyAdminAuthInterceptor.resolveResourceAction("UpdateQuota").resource); + assertEquals(ProxyAdminAuthInterceptor.RESOURCE_ROUTE, + ProxyAdminAuthInterceptor.resolveResourceAction("DescribeRouteTopology").resource); + assertEquals(ProxyAdminAuthInterceptor.RESOURCE_OPS, + ProxyAdminAuthInterceptor.resolveResourceAction("ResetGroupOffset").resource); + + Set distinct = new HashSet<>(); + distinct.add(ProxyAdminAuthInterceptor.RESOURCE_CLIENT); + distinct.add(ProxyAdminAuthInterceptor.RESOURCE_CONFIG); + distinct.add(ProxyAdminAuthInterceptor.RESOURCE_CONNECTION); + distinct.add(ProxyAdminAuthInterceptor.RESOURCE_QUOTA); + distinct.add(ProxyAdminAuthInterceptor.RESOURCE_ROUTE); + distinct.add(ProxyAdminAuthInterceptor.RESOURCE_OPS); + assertEquals(6, distinct.size()); + for (String resource : distinct) { + assertTrue(resource.startsWith("proxy.admin.")); + } + } + + // --------------------------------------------------------------------- + // behavior modes + // --------------------------------------------------------------------- + + @Test + @SuppressWarnings("unchecked") + public void openModePassesThroughWhenClusterAuthDisabled() { + AuthConfig authConfig = new AuthConfig(); + authConfig.setAuthenticationEnabled(false); + authConfig.setAuthorizationEnabled(false); + ConfigurationManager.getProxyConfig().setProxyAdminRequireAuth(false); + + ProxyAdminAuthInterceptor interceptor = new ProxyAdminAuthInterceptor(authConfig, messagingProcessor); + ServerCall call = serverCall("ListClients"); + ServerCallHandler next = mock(ServerCallHandler.class); + + interceptor.interceptCall(call, new Metadata(), next); + verify(next).startCall(any(), any()); + verify(call, never()).close(any(), any()); + } + + @Test + @SuppressWarnings("unchecked") + public void failClosedRejectsWhenRequireAuthButClusterAuthDisabled() { + AuthConfig authConfig = new AuthConfig(); + authConfig.setAuthenticationEnabled(false); + authConfig.setAuthorizationEnabled(false); + ConfigurationManager.getProxyConfig().setProxyAdminRequireAuth(true); + try { + ProxyAdminAuthInterceptor interceptor = new ProxyAdminAuthInterceptor(authConfig, messagingProcessor); + ServerCall call = serverCall("ListClients"); + ServerCallHandler next = mock(ServerCallHandler.class); + + interceptor.interceptCall(call, new Metadata(), next); + verify(next, never()).startCall(any(), any()); + org.mockito.ArgumentCaptor statusCaptor = org.mockito.ArgumentCaptor.forClass(Status.class); + verify(call).close(statusCaptor.capture(), any(Metadata.class)); + assertEquals(Status.Code.UNAUTHENTICATED, statusCaptor.getValue().getCode()); + assertTrue(statusCaptor.getValue().getDescription().contains("proxyAdminRequireAuth")); + } finally { + ConfigurationManager.getProxyConfig().setProxyAdminRequireAuth(false); + } + } + + @Test + @SuppressWarnings("unchecked") + public void failClosedRejectsAnonymousWhenRequireAuth() { + AuthConfig authConfig = new AuthConfig(); + authConfig.setAuthenticationEnabled(true); + authConfig.setAuthorizationEnabled(true); + ConfigurationManager.getProxyConfig().setProxyAdminRequireAuth(true); + try { + ProxyAdminAuthInterceptor interceptor = new ProxyAdminAuthInterceptor(authConfig, messagingProcessor); + ServerCall call = serverCall("KickClient"); + ServerCallHandler next = mock(ServerCallHandler.class); + + // empty metadata: no credentials at all + ServerCall.Listener listener = interceptor.interceptCall(call, new Metadata(), next); + assertNotNull(listener); + verify(next, never()).startCall(any(), any()); + org.mockito.ArgumentCaptor statusCaptor = org.mockito.ArgumentCaptor.forClass(Status.class); + verify(call).close(statusCaptor.capture(), any(Metadata.class)); + assertEquals(Status.Code.UNAUTHENTICATED, statusCaptor.getValue().getCode()); + } finally { + ConfigurationManager.getProxyConfig().setProxyAdminRequireAuth(false); + } + } + + @Test + public void resourceActionModelIsImmutablePerMethod() { + // sanity: repeated resolution yields the same mapping (no stateful drift) + ProxyAdminAuthInterceptor.ResourceAction first = + ProxyAdminAuthInterceptor.resolveResourceAction("DescribeClient"); + ProxyAdminAuthInterceptor.ResourceAction second = + ProxyAdminAuthInterceptor.resolveResourceAction("DescribeClient"); + assertEquals(first.resource, second.resource); + assertEquals(first.action, second.action); + assertFalse(first.resource.isEmpty()); + } +} diff --git a/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminConfigSupportTest.java b/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminConfigSupportTest.java new file mode 100644 index 00000000000..d4a9fb910d4 --- /dev/null +++ b/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminConfigSupportTest.java @@ -0,0 +1,177 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.rocketmq.proxy.grpc.admin; + +import apache.rocketmq.v2.Code; +import apache.rocketmq.v2.DescribeProxyConfigRequest; +import apache.rocketmq.v2.DescribeProxyConfigResponse; +import apache.rocketmq.v2.DescribeQuotaRequest; +import apache.rocketmq.v2.DescribeQuotaResponse; +import apache.rocketmq.v2.ProxyRuntimeConfig; +import apache.rocketmq.v2.QuotaDimension; +import apache.rocketmq.v2.QuotaPolicy; +import apache.rocketmq.v2.Resource; +import apache.rocketmq.v2.UpdateProxyConfigRequest; +import apache.rocketmq.v2.UpdateProxyConfigResponse; +import apache.rocketmq.v2.UpdateQuotaRequest; +import apache.rocketmq.v2.UpdateQuotaResponse; +import com.google.protobuf.Duration; +import org.apache.rocketmq.proxy.config.ConfigurationManager; +import org.apache.rocketmq.proxy.config.InitConfigTest; +import org.apache.rocketmq.proxy.config.ProxyConfig; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +public class ProxyAdminConfigSupportTest extends InitConfigTest { + + private ProxyAdminConfigSupport support; + private int originalMaxMessageSize; + private long originalDefaultInvisibleTime; + private boolean originalTlsTestModeEnable; + private boolean originalTraceOn; + private boolean originalProxyAdminEnabled; + + @Before + public void setUp() { + support = new ProxyAdminConfigSupport(); + ProxyConfig config = ConfigurationManager.getProxyConfig(); + originalMaxMessageSize = config.getMaxMessageSize(); + originalDefaultInvisibleTime = config.getDefaultInvisibleTimeMills(); + originalTlsTestModeEnable = config.isTlsTestModeEnable(); + originalTraceOn = config.isTraceOn(); + originalProxyAdminEnabled = config.isProxyAdminEnabled(); + } + + @After + public void tearDown() { + ProxyConfig config = ConfigurationManager.getProxyConfig(); + config.setMaxMessageSize(originalMaxMessageSize); + config.setDefaultInvisibleTimeMills(originalDefaultInvisibleTime); + config.setTlsTestModeEnable(originalTlsTestModeEnable); + config.setTraceOn(originalTraceOn); + config.setProxyAdminEnabled(originalProxyAdminEnabled); + } + + private static apache.rocketmq.v2.Status ok() { + return apache.rocketmq.v2.Status.newBuilder().setCode(Code.OK).build(); + } + + @Test + public void describeProxyConfigReflectsLiveConfig() { + DescribeProxyConfigResponse response = support.describeProxyConfig( + DescribeProxyConfigRequest.newBuilder().build(), ok()); + assertEquals(Code.OK, response.getStatus().getCode()); + ProxyRuntimeConfig config = response.getConfig(); + assertNotNull(config); + assertEquals(ConfigurationManager.getProxyConfig().getMaxMessageSize(), config.getMaxMessageSize()); + assertEquals(ConfigurationManager.getProxyConfig().getGrpcThreadPoolNums(), config.getGrpcThreadPoolNums()); + assertTrue(config.getDefaultInvisibleTime().getSeconds() > 0); + } + + @Test + public void updateProxyConfigAppliesChangedFields() { + UpdateProxyConfigResponse response = support.updateProxyConfig(UpdateProxyConfigRequest.newBuilder() + .setConfig(ProxyRuntimeConfig.newBuilder() + .setMaxMessageSize(1234) + .setDefaultInvisibleTime(Duration.newBuilder().setSeconds(120).build()) + .build()) + .build(), ok()); + assertEquals(Code.OK, response.getStatus().getCode()); + assertTrue(response.getChangedFieldsList().contains("max_message_size")); + assertTrue(response.getChangedFieldsList().contains("default_invisible_time")); + assertEquals(1234, ConfigurationManager.getProxyConfig().getMaxMessageSize()); + assertEquals(120_000L, ConfigurationManager.getProxyConfig().getDefaultInvisibleTimeMills()); + // response carries the refreshed view + assertEquals(1234, response.getConfig().getMaxMessageSize()); + } + + @Test + public void updateProxyConfigReportsNoChangeForSameValues() { + ProxyConfig live = ConfigurationManager.getProxyConfig(); + // proto3 caveat: booleans cannot express "absent", so mirror the live values + UpdateProxyConfigResponse response = support.updateProxyConfig(UpdateProxyConfigRequest.newBuilder() + .setConfig(ProxyRuntimeConfig.newBuilder() + .setMaxMessageSize(live.getMaxMessageSize()) + .setTlsTestModeEnable(live.isTlsTestModeEnable()) + .setTraceOn(live.isTraceOn()) + .setProxyAdminEnabled(live.isProxyAdminEnabled()) + .build()) + .build(), ok()); + assertEquals(Code.OK, response.getStatus().getCode()); + assertTrue(response.getChangedFieldsList().isEmpty()); + } + + @Test + public void describeQuotaSeedsProxyLevelPolicies() { + DescribeQuotaResponse response = support.describeQuota( + DescribeQuotaRequest.newBuilder().build(), ok()); + assertEquals(Code.OK, response.getStatus().getCode()); + assertTrue(response.getPoliciesCount() >= 3); + boolean hasMaxMessageSize = false; + for (QuotaPolicy policy : response.getPoliciesList()) { + if (ProxyAdminConfigSupport.METRIC_MAX_MESSAGE_SIZE.equals(policy.getMetric())) { + hasMaxMessageSize = true; + assertEquals(ConfigurationManager.getProxyConfig().getMaxMessageSize(), policy.getLimit()); + assertTrue(policy.getLimit() > 0); + assertTrue(policy.hasWindow()); + } + } + assertTrue(hasMaxMessageSize); + } + + @Test + public void describeQuotaFiltersByDimensionAndResource() { + DescribeQuotaResponse byDimension = support.describeQuota(DescribeQuotaRequest.newBuilder() + .setDimension(QuotaDimension.QUOTA_DIMENSION_TOPIC) + .build(), ok()); + for (QuotaPolicy policy : byDimension.getPoliciesList()) { + assertEquals(QuotaDimension.QUOTA_DIMENSION_TOPIC, policy.getDimension()); + } + } + + @Test + public void updateQuotaAppliesMappedKnobImmediately() { + UpdateQuotaResponse response = support.updateQuota(UpdateQuotaRequest.newBuilder() + .setPolicy(QuotaPolicy.newBuilder() + .setDimension(QuotaDimension.QUOTA_DIMENSION_TOPIC) + .setResource(Resource.newBuilder().setName("*").build()) + .setMetric(ProxyAdminConfigSupport.METRIC_MAX_MESSAGE_SIZE) + .setLimit(4096) + .build()) + .build(), ok(), apache.rocketmq.v2.Status.newBuilder().setCode(Code.BAD_REQUEST).build()); + assertEquals(Code.OK, response.getStatus().getCode()); + assertEquals(4096, response.getPolicy().getLimit()); + assertEquals(4096, ConfigurationManager.getProxyConfig().getMaxMessageSize()); + } + + @Test + public void updateQuotaRejectsInvalidPolicy() { + UpdateQuotaResponse response = support.updateQuota(UpdateQuotaRequest.newBuilder() + .setPolicy(QuotaPolicy.newBuilder() + .setDimension(QuotaDimension.QUOTA_DIMENSION_TOPIC) + .setMetric("") + .setLimit(0) + .build()) + .build(), ok(), apache.rocketmq.v2.Status.newBuilder().setCode(Code.BAD_REQUEST).build()); + assertEquals(Code.BAD_REQUEST, response.getStatus().getCode()); + } +} diff --git a/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminGrpcServiceTest.java b/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminGrpcServiceTest.java new file mode 100644 index 00000000000..ae6dcdcb481 --- /dev/null +++ b/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminGrpcServiceTest.java @@ -0,0 +1,484 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.rocketmq.proxy.grpc.admin; + +import apache.rocketmq.v2.AdminSendMessageRequest; +import apache.rocketmq.v2.AdminSendMessageResponse; +import apache.rocketmq.v2.Broker; +import apache.rocketmq.v2.ChangeLogLevelRequest; +import apache.rocketmq.v2.ChangeLogLevelResponse; +import apache.rocketmq.v2.ClientType; +import apache.rocketmq.v2.Code; +import apache.rocketmq.v2.DescribeGroupAccumulationRequest; +import apache.rocketmq.v2.DescribeGroupAccumulationResponse; +import apache.rocketmq.v2.DescribeSubscriptionRequest; +import apache.rocketmq.v2.DescribeSubscriptionResponse; +import apache.rocketmq.v2.DescribeTopicStatusRequest; +import apache.rocketmq.v2.DescribeTopicStatusResponse; +import apache.rocketmq.v2.DeleteSubscriptionRequest; +import apache.rocketmq.v2.DeleteSubscriptionResponse; +import apache.rocketmq.v2.FilterExpression; +import apache.rocketmq.v2.FilterType; +import apache.rocketmq.v2.GetConsumerRunningInfoRequest; +import apache.rocketmq.v2.GetConsumerRunningInfoResponse; +import apache.rocketmq.v2.GetProxyRuntimeStatsRequest; +import apache.rocketmq.v2.GetProxyRuntimeStatsResponse; +import apache.rocketmq.v2.GetTopicRouteRequest; +import apache.rocketmq.v2.GetTopicRouteResponse; +import apache.rocketmq.v2.Language; +import apache.rocketmq.v2.ListConsumerConnectionRequest; +import apache.rocketmq.v2.ListConsumerConnectionResponse; +import apache.rocketmq.v2.ListMessageRequest; +import apache.rocketmq.v2.ListMessageResponse; +import apache.rocketmq.v2.ListSubscriptionRequest; +import apache.rocketmq.v2.ListSubscriptionResponse; +import apache.rocketmq.v2.PrintThreadStackTraceRequest; +import apache.rocketmq.v2.PrintThreadStackTraceResponse; +import apache.rocketmq.v2.QueryTimeSpanRequest; +import apache.rocketmq.v2.QueryTimeSpanResponse; +import apache.rocketmq.v2.ResetGroupOffsetRequest; +import apache.rocketmq.v2.ResetGroupOffsetResponse; +import apache.rocketmq.v2.Resource; +import apache.rocketmq.v2.Settings; +import apache.rocketmq.v2.Subscription; +import apache.rocketmq.v2.SubscriptionEntry; +import apache.rocketmq.v2.UA; +import apache.rocketmq.v2.VerifyMessageRequest; +import apache.rocketmq.v2.VerifyMessageResponse; +import io.grpc.stub.StreamObserver; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import java.util.concurrent.CompletableFuture; +import org.apache.rocketmq.client.producer.SendResult; +import org.apache.rocketmq.common.message.MessageExt; +import org.apache.rocketmq.common.message.MessageQueue; +import org.apache.rocketmq.proxy.grpc.v2.channel.GrpcChannelManager; +import org.apache.rocketmq.proxy.grpc.v2.channel.GrpcClientChannel; +import org.apache.rocketmq.proxy.grpc.v2.common.GrpcClientSettingsManager; +import org.apache.rocketmq.proxy.processor.MessagingProcessor; +import org.apache.rocketmq.proxy.service.ServiceManager; +import org.apache.rocketmq.proxy.service.admin.AdminService; +import org.apache.rocketmq.proxy.service.route.AddressableMessageQueue; +import org.apache.rocketmq.proxy.service.route.MessageQueueSelector; +import org.apache.rocketmq.proxy.service.route.MessageQueueView; +import org.apache.rocketmq.proxy.service.route.TopicRouteService; +import org.apache.rocketmq.remoting.protocol.admin.ConsumeStats; +import org.apache.rocketmq.remoting.protocol.admin.OffsetWrapper; +import org.apache.rocketmq.remoting.protocol.statictopic.TopicConfigAndQueueMapping; +import org.apache.rocketmq.remoting.protocol.route.TopicRouteData; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.Mock; +import org.mockito.junit.MockitoJUnitRunner; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyInt; +import static org.mockito.ArgumentMatchers.anyList; +import static org.mockito.ArgumentMatchers.anyLong; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +@RunWith(MockitoJUnitRunner.class) +public class ProxyAdminGrpcServiceTest { + + @Mock + private ServiceManager serviceManager; + @Mock + private MessagingProcessor messagingProcessor; + @Mock + private GrpcChannelManager grpcChannelManager; + @Mock + private GrpcClientSettingsManager grpcClientSettingsManager; + @Mock + private AdminService adminService; + @Mock + private TopicRouteService topicRouteService; + @Mock + private GrpcClientChannel channel; + + private ProxyAdminGrpcService service; + + @Before + public void setUp() { + when(serviceManager.getAdminService()).thenReturn(adminService); + when(serviceManager.getTopicRouteService()).thenReturn(topicRouteService); + service = new ProxyAdminGrpcService(serviceManager, messagingProcessor, grpcChannelManager, + grpcClientSettingsManager); + } + + // ------------------------------------------------------------------ helpers + + private static class SimpleObserver implements StreamObserver { + T value; + Throwable error; + + @Override + public void onNext(T value) { + this.value = value; + } + + @Override + public void onError(Throwable t) { + this.error = t; + } + + @Override + public void onCompleted() { + } + } + + private Settings subscriptionSettings(ClientType clientType, String group, String topic, String expression) { + return Settings.newBuilder() + .setClientType(clientType) + .setUserAgent(UA.newBuilder().setVersion("4.9.0").setLanguage(Language.JAVA).setHostname("host").build()) + .setSubscription(Subscription.newBuilder() + .setGroup(Resource.newBuilder().setName(group).build()) + .addSubscriptions(SubscriptionEntry.newBuilder() + .setTopic(Resource.newBuilder().setName(topic).build()) + .setExpression(FilterExpression.newBuilder() + .setType(FilterType.TAG).setExpression(expression).build()) + .build()) + .build()) + .build(); + } + + private void stubBrokerRoute() throws Exception { + AddressableMessageQueue mq = new AddressableMessageQueue(new MessageQueue("t", "broker-a", 0), + "127.0.0.1:10911"); + MessageQueueSelector selector = mock(MessageQueueSelector.class); + when(selector.getQueues()).thenReturn(Collections.singletonList(mq)); + MessageQueueView mqv = mock(MessageQueueView.class); + when(mqv.getReadSelector()).thenReturn(selector); + when(topicRouteService.getAllMessageQueueView(any(), anyString())).thenReturn(mqv); + } + + private void stubOnlineClient(String clientId, Settings settings) { + when(channel.getClientId()).thenReturn(clientId); + when(channel.getRemoteAddress()).thenReturn("1.2.3.4:8888"); + when(grpcChannelManager.getClientChannels()).thenReturn(Collections.singletonList(channel)); + when(grpcChannelManager.getChannel(clientId)).thenReturn(channel); + when(grpcClientSettingsManager.getRawClientSettings(clientId)).thenReturn(settings); + } + + // ------------------------------------------------------------------ tests + + @Test + public void changeLogLevelSucceeds() { + SimpleObserver obs = new SimpleObserver<>(); + service.changeLogLevel(ChangeLogLevelRequest.newBuilder().setLevel(ChangeLogLevelRequest.Level.DEBUG).build(), + obs); + assertNotNull(obs.value); + assertTrue(obs.value.getRemark().toLowerCase().contains("log level changed")); + } + + @Test + public void getProxyRuntimeStatsCountsClients() { + GrpcClientChannel producer = mock(GrpcClientChannel.class); + when(producer.getClientId()).thenReturn("p1"); + when(grpcChannelManager.getClientChannels()).thenReturn(java.util.Arrays.asList(channel, producer)); + when(channel.getClientId()).thenReturn("c1"); + when(grpcClientSettingsManager.getRawClientSettings("c1")) + .thenReturn(subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "*")); + when(grpcClientSettingsManager.getRawClientSettings("p1")) + .thenReturn(subscriptionSettings(ClientType.PRODUCER, "g", "t", "*")); + + SimpleObserver obs = new SimpleObserver<>(); + service.getProxyRuntimeStats(GetProxyRuntimeStatsRequest.newBuilder().build(), obs); + assertNotNull(obs.value); + assertEquals(1, obs.value.getConsumers()); + assertEquals(1, obs.value.getProducers()); + assertEquals(2, obs.value.getConnections()); + } + + @Test + public void getTopicRouteReturnsJson() throws Exception { + when(adminService.getTopicRouteData("t")).thenReturn(new TopicRouteData()); + SimpleObserver obs = new SimpleObserver<>(); + service.getTopicRoute(GetTopicRouteRequest.newBuilder().setTopic(Resource.newBuilder().setName("t")).build(), + obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + assertNotNull(obs.value.getTopicRouteData()); + } + + @Test + public void describeTopicStatusReturnsConfig() throws Exception { + stubBrokerRoute(); + TopicConfigAndQueueMapping cfg = new TopicConfigAndQueueMapping(); + cfg.setTopicName("t"); + cfg.setReadQueueNums(4); + cfg.setWriteQueueNums(4); + cfg.setPerm(6); + when(adminService.getTopicConfig(anyString(), eq("t"), anyLong())).thenReturn(cfg); + + SimpleObserver obs = new SimpleObserver<>(); + service.describeTopicStatus( + DescribeTopicStatusRequest.newBuilder().setTopic(Resource.newBuilder().setName("t")).build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + assertTrue(obs.value.getDescription().contains("readQueues=4")); + } + + @Test + public void listSubscriptionReturnsSubscriptions() { + stubOnlineClient("c1", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "tagA")); + SimpleObserver obs = new SimpleObserver<>(); + service.listSubscription(ListSubscriptionRequest.newBuilder().build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + assertTrue(obs.value.getSubscriptionInfoCount() >= 1); + assertEquals("t", obs.value.getSubscriptionInfo(0).getTopic().getName()); + } + + @Test + public void describeSubscriptionReturnsClientSubscriptions() { + stubOnlineClient("c1", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "tagA")); + SimpleObserver obs = new SimpleObserver<>(); + service.describeSubscription(DescribeSubscriptionRequest.newBuilder().build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + assertTrue(obs.value.getClientSubscriptionInfoCount() >= 1); + } + + @Test + public void listSubscriptionFiltersByGroup() { + stubOnlineClient("c1", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "tagA")); + SimpleObserver obs = new SimpleObserver<>(); + service.listSubscription( + ListSubscriptionRequest.newBuilder().setGroup(Resource.newBuilder().setName("other")).build(), obs); + assertNotNull(obs.value); + assertEquals(0, obs.value.getSubscriptionInfoCount()); + } + + @Test + public void deleteSubscriptionSucceeds() throws Exception { + stubBrokerRoute(); + SimpleObserver obs = new SimpleObserver<>(); + service.deleteSubscription( + DeleteSubscriptionRequest.newBuilder() + .setTopic(Resource.newBuilder().setName("t")) + .setGroup(Resource.newBuilder().setName("g")) + .build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + } + + @Test + public void listConsumerConnectionReturnsOnlineConsumers() { + stubOnlineClient("c1", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "*")); + SimpleObserver obs = new SimpleObserver<>(); + service.listConsumerConnection( + ListConsumerConnectionRequest.newBuilder().setGroup(Resource.newBuilder().setName("g")).build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + assertEquals(1, obs.value.getClientInfoCount()); + assertEquals("c1", obs.value.getClientInfo(0).getClientId()); + assertEquals("1.2.3.4:8888", obs.value.getClientInfo(0).getEgressIp()); + } + + @Test + public void describeGroupAccumulationSumsDiff() throws Exception { + stubBrokerRoute(); + MessageQueue mq = new MessageQueue("t", "broker-a", 0); + OffsetWrapper wrapper = new OffsetWrapper(); + wrapper.setBrokerOffset(100); + wrapper.setConsumerOffset(60); + Map offsetTable = new HashMap<>(); + offsetTable.put(mq, wrapper); + ConsumeStats consumeStats = new ConsumeStats(); + consumeStats.setOffsetTable(offsetTable); + when(adminService.fetchConsumeStats(anyString(), eq("g"), eq("t"), anyLong())).thenReturn(consumeStats); + + SimpleObserver obs = new SimpleObserver<>(); + service.describeGroupAccumulation(DescribeGroupAccumulationRequest.newBuilder() + .setGroup(Resource.newBuilder().setName("g")) + .addTopics(Resource.newBuilder().setName("t")) + .build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + assertEquals(40, obs.value.getAccumulation().getAccumulation()); + } + + @Test + public void resetGroupOffsetSucceeds() throws Exception { + stubBrokerRoute(); + when(adminService.resetOffset(anyString(), eq("t"), eq("g"), anyLong(), eq(true), anyLong())) + .thenReturn(Collections.emptyMap()); + SimpleObserver obs = new SimpleObserver<>(); + service.resetGroupOffset(ResetGroupOffsetRequest.newBuilder() + .setGroup(Resource.newBuilder().setName("g")) + .setTopic(Resource.newBuilder().setName("t")) + .setResetTimestamp(com.google.protobuf.Timestamp.newBuilder().setSeconds(1000).build()) + .build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + } + + @Test + public void queryMessageByMessageIdReturnsMessage() throws Exception { + stubBrokerRoute(); + MessageExt ext = new MessageExt(); + ext.setMsgId("MSG-1"); + ext.setTopic("t"); + ext.setTags("tag"); + when(adminService.viewMessage(anyString(), eq("t"), anyLong(), anyLong())).thenReturn(ext); + + SimpleObserver obs = new SimpleObserver<>(); + service.queryMessage(ListMessageRequest.newBuilder() + .setTopic(Resource.newBuilder().setName("t")) + .setMessageId("010000000000000000000000000000000000000000000000") + .build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + assertEquals(1, obs.value.getMessagesCount()); + assertEquals("MSG-1", obs.value.getMessages(0).getSystemProperties().getMessageId()); + } + + @Test + public void queryMessageByKeyReturnsMessages() throws Exception { + stubBrokerRoute(); + MessageExt ext = new MessageExt(); + ext.setMsgId("MSG-2"); + ext.setTopic("t"); + when(adminService.queryMessage(anyString(), eq("t"), eq("key"), anyInt(), anyLong(), anyLong(), anyLong())) + .thenReturn(Collections.singletonList(ext)); + + SimpleObserver obs = new SimpleObserver<>(); + service.queryMessage(ListMessageRequest.newBuilder() + .setTopic(Resource.newBuilder().setName("t")) + .setMessageKey("key") + .build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + assertEquals(1, obs.value.getMessagesCount()); + } + + @Test + public void queryMessageRequiresIdOrKey() { + SimpleObserver obs = new SimpleObserver<>(); + service.queryMessage(ListMessageRequest.newBuilder() + .setTopic(Resource.newBuilder().setName("t")).build(), obs); + assertNotNull(obs.value); + assertEquals(Code.BAD_REQUEST, obs.value.getStatus().getCode()); + } + + @Test + public void printThreadStackTraceDispatchesWhenConnected() { + stubOnlineClient("c1", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "*")); + SimpleObserver obs = new SimpleObserver<>(); + service.printThreadStackTrace( + PrintThreadStackTraceRequest.newBuilder().setClientId("c1").build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + } + + @Test + public void printThreadStackTraceNotFoundWhenDisconnected() { + when(grpcChannelManager.getChannel("missing")).thenReturn(null); + SimpleObserver obs = new SimpleObserver<>(); + service.printThreadStackTrace( + PrintThreadStackTraceRequest.newBuilder().setClientId("missing").build(), obs); + assertNotNull(obs.value); + assertEquals(Code.NOT_FOUND, obs.value.getStatus().getCode()); + } + + @Test + public void verifyMessageDispatchesWhenConnected() { + stubOnlineClient("c1", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "*")); + SimpleObserver obs = new SimpleObserver<>(); + service.verifyMessage(VerifyMessageRequest.newBuilder() + .setClientId("c1") + .setTopic(Resource.newBuilder().setName("t")) + .setMessageId("MSG-1") + .build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + } + + @Test + public void adminSendMessageReturnsId() { + SendResult sendResult = new SendResult(); + sendResult.setMsgId("SENT-1"); + when(messagingProcessor.sendMessage(any(), any(), anyString(), anyInt(), anyList(), anyLong())) + .thenReturn(CompletableFuture.completedFuture(Collections.singletonList(sendResult))); + + SimpleObserver obs = new SimpleObserver<>(); + service.adminSendMessage(AdminSendMessageRequest.newBuilder() + .setTopic(Resource.newBuilder().setName("t")) + .setBody(com.google.protobuf.ByteString.copyFromUtf8("hello")) + .build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + assertEquals("SENT-1", obs.value.getMessageId()); + } + + @Test + public void getConsumerRunningInfoReturnsSubscriptions() { + stubOnlineClient("c1", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "tagA")); + SimpleObserver obs = new SimpleObserver<>(); + service.getConsumerRunningInfo( + GetConsumerRunningInfoRequest.newBuilder().setClientId("c1").build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + assertTrue(obs.value.getConsumerRunningInfo().getSubscriptionsMap().containsKey("t")); + } + + @Test + public void getConsumerRunningInfoNotFoundWhenDisconnected() { + when(grpcChannelManager.getChannel("missing")).thenReturn(null); + SimpleObserver obs = new SimpleObserver<>(); + service.getConsumerRunningInfo( + GetConsumerRunningInfoRequest.newBuilder().setClientId("missing").build(), obs); + assertNotNull(obs.value); + assertEquals(Code.NOT_FOUND, obs.value.getStatus().getCode()); + } + + @Test + public void queryTimeSpanReturnsPerQueueSpan() throws Exception { + stubBrokerRoute(); + MessageQueue mq = new MessageQueue("t", "broker-a", 0); + OffsetWrapper wrapper = new OffsetWrapper(); + wrapper.setBrokerOffset(100); + wrapper.setConsumerOffset(60); + Map offsetTable = new HashMap<>(); + offsetTable.put(mq, wrapper); + ConsumeStats consumeStats = new ConsumeStats(); + consumeStats.setOffsetTable(offsetTable); + when(adminService.fetchConsumeStats(anyString(), eq("g"), eq("t"), anyLong())).thenReturn(consumeStats); + when(adminService.getEarliestMsgStoretime(anyString(), any(MessageQueue.class), anyLong())) + .thenReturn(12345L); + + SimpleObserver obs = new SimpleObserver<>(); + service.queryTimeSpan(QueryTimeSpanRequest.newBuilder() + .setGroup(Resource.newBuilder().setName("g")) + .addTopics(Resource.newBuilder().setName("t")) + .build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + assertTrue(obs.value.getQueueTimeSpanListCount() >= 1); + assertEquals(12345L, obs.value.getQueueTimeSpanList(0).getMinTimestamp()); + Broker broker = obs.value.getQueueTimeSpanList(0).getMessageQueue().getBroker(); + assertEquals("broker-a", broker.getName()); + } +} diff --git a/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminServiceGrpcServiceTest.java b/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminServiceGrpcServiceTest.java new file mode 100644 index 00000000000..d82407d2f47 --- /dev/null +++ b/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/ProxyAdminServiceGrpcServiceTest.java @@ -0,0 +1,366 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.rocketmq.proxy.grpc.admin; + +import apache.rocketmq.v2.ClientFilter; +import apache.rocketmq.v2.ClientInstance; +import apache.rocketmq.v2.ClientProtocol; +import apache.rocketmq.v2.ClientRole; +import apache.rocketmq.v2.ClientType; +import apache.rocketmq.v2.Code; +import apache.rocketmq.v2.DescribeClientRequest; +import apache.rocketmq.v2.DescribeClientResponse; +import apache.rocketmq.v2.FilterExpression; +import apache.rocketmq.v2.FilterType; +import apache.rocketmq.v2.Language; +import apache.rocketmq.v2.ListClientsByGroupRequest; +import apache.rocketmq.v2.ListClientsByGroupResponse; +import apache.rocketmq.v2.ListClientsByTopicRequest; +import apache.rocketmq.v2.ListClientsByTopicResponse; +import apache.rocketmq.v2.ListClientsRequest; +import apache.rocketmq.v2.ListClientsResponse; +import apache.rocketmq.v2.Publishing; +import apache.rocketmq.v2.Resource; +import apache.rocketmq.v2.Settings; +import apache.rocketmq.v2.Subscription; +import apache.rocketmq.v2.SubscriptionEntry; +import apache.rocketmq.v2.UA; +import io.grpc.stub.StreamObserver; +import java.util.ArrayList; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import org.apache.rocketmq.proxy.grpc.v2.channel.GrpcChannelManager; +import org.apache.rocketmq.proxy.grpc.v2.channel.GrpcClientChannel; +import org.apache.rocketmq.proxy.grpc.v2.common.GrpcClientSettingsManager; +import org.apache.rocketmq.proxy.processor.DefaultMessagingProcessor; +import org.apache.rocketmq.proxy.service.ServiceManager; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.mockito.Mock; +import org.mockito.junit.MockitoJUnitRunner; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +@RunWith(MockitoJUnitRunner.class) +public class ProxyAdminServiceGrpcServiceTest { + + @Mock + private ServiceManager serviceManager; + @Mock + private DefaultMessagingProcessor messagingProcessor; + @Mock + private GrpcChannelManager grpcChannelManager; + @Mock + private GrpcClientSettingsManager grpcClientSettingsManager; + + private ProxyAdminServiceGrpcService service; + + @Before + public void setUp() { + service = new ProxyAdminServiceGrpcService(serviceManager, messagingProcessor, grpcChannelManager, + grpcClientSettingsManager, new ProxyAdminPeerClient(), new RouteChangeNotifier()); + } + + // ------------------------------------------------------------------ helpers + + private static class SimpleObserver implements StreamObserver { + T value; + Throwable error; + + @Override + public void onNext(T value) { + this.value = value; + } + + @Override + public void onError(Throwable t) { + this.error = t; + } + + @Override + public void onCompleted() { + } + } + + private static Settings subscriptionSettings(ClientType clientType, String group, String topic, String expression) { + return Settings.newBuilder() + .setClientType(clientType) + .setUserAgent(UA.newBuilder().setVersion("5.0.0").setLanguage(Language.JAVA).setHostname("host").build()) + .setSubscription(Subscription.newBuilder() + .setGroup(Resource.newBuilder().setName(group).build()) + .addSubscriptions(SubscriptionEntry.newBuilder() + .setTopic(Resource.newBuilder().setName(topic).build()) + .setExpression(FilterExpression.newBuilder() + .setType(FilterType.TAG).setExpression(expression).build()) + .build()) + .build()) + .build(); + } + + private static Settings publishingSettings(ClientType clientType, String topic) { + return Settings.newBuilder() + .setClientType(clientType) + .setUserAgent(UA.newBuilder().setVersion("5.0.0").setLanguage(Language.GOLANG).setHostname("host").build()) + .setPublishing(Publishing.newBuilder() + .addTopics(Resource.newBuilder().setName(topic).build()) + .build()) + .build(); + } + + private GrpcClientChannel mockChannel(String clientId) { + GrpcClientChannel channel = mock(GrpcClientChannel.class); + when(channel.getClientId()).thenReturn(clientId); + when(channel.getRemoteAddress()).thenReturn("1.2.3.4:8888"); + when(channel.getLocalAddress()).thenReturn("9.9.9.9:8081"); + when(channel.getConnectTimeMillis()).thenReturn(1000L); + when(channel.getLastActiveTimeMillis()).thenReturn(2000L); + when(channel.getRecentHeartbeats()).thenReturn(new ArrayList<>()); + return channel; + } + + private void stubClients(Map clients) { + List channels = new ArrayList<>(); + for (Map.Entry entry : clients.entrySet()) { + GrpcClientChannel channel = mockChannel(entry.getKey()); + channels.add(channel); + when(grpcChannelManager.getChannel(entry.getKey())).thenReturn(channel); + when(grpcClientSettingsManager.getRawClientSettings(entry.getKey())).thenReturn(entry.getValue()); + } + when(grpcChannelManager.getClientChannels()).thenReturn(channels); + } + + private static ClientInstance find(List list, String clientId) { + for (ClientInstance ci : list) { + if (ci.getClientId().equals(clientId)) { + return ci; + } + } + return null; + } + + // ------------------------------------------------------------------ tests + + @Test + public void listClientsReturnsConnectedClientsWithRoles() { + Map clients = new LinkedHashMap<>(); + clients.put("consumer-1", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g1", "t1", "*")); + clients.put("producer-1", publishingSettings(ClientType.PRODUCER, "t2")); + clients.put("push-1", subscriptionSettings(ClientType.PUSH_CONSUMER, "g2", "t3", "*")); + stubClients(clients); + + SimpleObserver obs = new SimpleObserver<>(); + service.listClients(ListClientsRequest.newBuilder().build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + assertEquals(3, obs.value.getClientsCount()); + assertFalse(obs.value.getProxyEndpoint().isEmpty()); + assertTrue(obs.value.getEpoch() > 0); + + ClientInstance consumer = find(obs.value.getClientsList(), "consumer-1"); + assertNotNull(consumer); + assertEquals(ClientRole.CLIENT_ROLE_SIMPLE_CONSUMER, consumer.getRole()); + assertEquals(ClientProtocol.CLIENT_PROTOCOL_GRPC, consumer.getProtocol()); + assertEquals(Language.JAVA, consumer.getLanguage()); + assertEquals("5.0.0", consumer.getClientVersion()); + assertTrue(consumer.getGroupsList().contains("g1")); + assertTrue(consumer.getTopicsList().contains("t1")); + + ClientInstance producer = find(obs.value.getClientsList(), "producer-1"); + assertNotNull(producer); + assertEquals(ClientRole.CLIENT_ROLE_PRODUCER, producer.getRole()); + assertTrue(producer.getGroupsList().isEmpty()); + assertTrue(producer.getTopicsList().contains("t2")); + + ClientInstance push = find(obs.value.getClientsList(), "push-1"); + assertNotNull(push); + assertEquals(ClientRole.CLIENT_ROLE_PUSH_CONSUMER, push.getRole()); + } + + @Test + public void listClientsHonoursGroupFilter() { + Map clients = new LinkedHashMap<>(); + clients.put("consumer-1", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g1", "t1", "*")); + clients.put("producer-1", publishingSettings(ClientType.PRODUCER, "t2")); + stubClients(clients); + + SimpleObserver obs = new SimpleObserver<>(); + service.listClients(ListClientsRequest.newBuilder() + .setFilter(ClientFilter.newBuilder().setGroup(Resource.newBuilder().setName("g1"))) + .build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + assertEquals(1, obs.value.getClientsCount()); + assertEquals("consumer-1", obs.value.getClientsList().get(0).getClientId()); + } + + @Test + public void listClientsByGroupReturnsMatchingClients() { + Map clients = new LinkedHashMap<>(); + clients.put("consumer-1", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g1", "t1", "*")); + clients.put("consumer-2", subscriptionSettings(ClientType.PUSH_CONSUMER, "g2", "t3", "*")); + stubClients(clients); + + SimpleObserver obs = new SimpleObserver<>(); + service.listClientsByGroup(ListClientsByGroupRequest.newBuilder() + .setGroup(Resource.newBuilder().setName("g1")) + .build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + assertEquals(1, obs.value.getClientsCount()); + assertEquals("consumer-1", obs.value.getClientsList().get(0).getClientId()); + + SimpleObserver none = new SimpleObserver<>(); + service.listClientsByGroup(ListClientsByGroupRequest.newBuilder() + .setGroup(Resource.newBuilder().setName("absent")) + .build(), none); + assertEquals(0, none.value.getClientsCount()); + } + + @Test + public void listClientsByTopicReturnsMatchingClients() { + Map clients = new LinkedHashMap<>(); + clients.put("consumer-1", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g1", "t1", "*")); + clients.put("producer-1", publishingSettings(ClientType.PRODUCER, "t2")); + stubClients(clients); + + SimpleObserver bySub = new SimpleObserver<>(); + service.listClientsByTopic(ListClientsByTopicRequest.newBuilder() + .setTopic(Resource.newBuilder().setName("t1")) + .build(), bySub); + assertEquals(1, bySub.value.getClientsCount()); + assertEquals("consumer-1", bySub.value.getClientsList().get(0).getClientId()); + + SimpleObserver byPub = new SimpleObserver<>(); + service.listClientsByTopic(ListClientsByTopicRequest.newBuilder() + .setTopic(Resource.newBuilder().setName("t2")) + .build(), byPub); + assertEquals(1, byPub.value.getClientsCount()); + assertEquals("producer-1", byPub.value.getClientsList().get(0).getClientId()); + + SimpleObserver missing = new SimpleObserver<>(); + service.listClientsByTopic(ListClientsByTopicRequest.newBuilder() + .setTopic(Resource.newBuilder().setName("absent")) + .build(), missing); + assertEquals(0, missing.value.getClientsCount()); + } + + @Test + public void describeClientReturnsDetailForConnectedClient() { + Map clients = new LinkedHashMap<>(); + clients.put("consumer-1", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g1", "t1", "*")); + stubClients(clients); + + SimpleObserver obs = new SimpleObserver<>(); + service.describeClient(DescribeClientRequest.newBuilder().setClientId("consumer-1").build(), obs); + assertNotNull(obs.value); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + assertNotNull(obs.value.getClientDetail()); + assertEquals("consumer-1", obs.value.getClientDetail().getInstance().getClientId()); + assertNotNull(obs.value.getClientDetail().getSettings()); + assertEquals(1, obs.value.getClientDetail().getSubscriptionsCount()); + assertEquals("t1", obs.value.getClientDetail().getSubscriptions(0).getTopic().getName()); + assertEquals("1.2.3.4:8888", obs.value.getClientDetail().getNetworkInfo().getRemoteAddress()); + } + + @Test + public void describeClientReturnsNotFoundForUnknownClient() { + stubClients(Collections.emptyMap()); + + SimpleObserver obs = new SimpleObserver<>(); + service.describeClient(DescribeClientRequest.newBuilder().setClientId("ghost").build(), obs); + assertNotNull(obs.value); + assertEquals(Code.NOT_FOUND, obs.value.getStatus().getCode()); + } + + @Test + public void listClientsPaginatesWithCursor() { + Map clients = new LinkedHashMap<>(); + clients.put("c1", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "*")); + clients.put("c2", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "*")); + clients.put("c3", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "*")); + stubClients(clients); + + SimpleObserver first = new SimpleObserver<>(); + service.listClients(ListClientsRequest.newBuilder().setPageSize(2).build(), first); + assertEquals(2, first.value.getClientsCount()); + assertFalse(first.value.getNextToken().isEmpty()); + + SimpleObserver second = new SimpleObserver<>(); + service.listClients(ListClientsRequest.newBuilder() + .setPageSize(2) + .setNextToken(first.value.getNextToken()) + .build(), second); + assertEquals(1, second.value.getClientsCount()); + assertTrue(second.value.getNextToken().isEmpty()); + } + + @Test + public void listClientsCursorIsStableUnderMembershipChurn() { + // sorted order: c1 < c2 < c3 + Map clients = new LinkedHashMap<>(); + clients.put("c1", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "*")); + clients.put("c2", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "*")); + clients.put("c3", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "*")); + stubClients(clients); + + SimpleObserver first = new SimpleObserver<>(); + service.listClients(ListClientsRequest.newBuilder().setPageSize(2).build(), first); + assertEquals(2, first.value.getClientsCount()); + String token = first.value.getNextToken(); + assertFalse(token.isEmpty()); + + // a new client connects between the two page fetches; it sorts before the cursor + // position, so the continuation page must neither duplicate nor shift entries. + Map grown = new LinkedHashMap<>(clients); + grown.put("c10", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "*")); + stubClients(grown); + + SimpleObserver second = new SimpleObserver<>(); + service.listClients(ListClientsRequest.newBuilder() + .setPageSize(2) + .setNextToken(token) + .build(), second); + assertEquals(1, second.value.getClientsCount()); + assertEquals("c3", second.value.getClientsList().get(0).getClientId()); + assertTrue(second.value.getNextToken().isEmpty()); + } + + @Test + public void listClientsRejectsTamperedCursorGracefully() { + Map clients = new LinkedHashMap<>(); + clients.put("c1", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "*")); + clients.put("c2", subscriptionSettings(ClientType.SIMPLE_CONSUMER, "g", "t", "*")); + stubClients(clients); + + SimpleObserver obs = new SimpleObserver<>(); + service.listClients(ListClientsRequest.newBuilder() + .setPageSize(10) + .setNextToken("c1:not-base64-!!!") + .build(), obs); + assertEquals(Code.OK, obs.value.getStatus().getCode()); + // invalid cursor falls back to the first page + assertEquals(2, obs.value.getClientsCount()); + } +} diff --git a/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/RouteChangeNotifierTest.java b/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/RouteChangeNotifierTest.java new file mode 100644 index 00000000000..e0ff034d16c --- /dev/null +++ b/proxy/src/test/java/org/apache/rocketmq/proxy/grpc/admin/RouteChangeNotifierTest.java @@ -0,0 +1,197 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.rocketmq.proxy.grpc.admin; + +import apache.rocketmq.v2.RouteChangeEvent; +import apache.rocketmq.v2.RouteChangeEventType; +import apache.rocketmq.v2.SubscribeRouteEventsRequest; +import apache.rocketmq.v2.SubscribeRouteEventsResponse; +import io.grpc.stub.StreamObserver; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import org.apache.rocketmq.proxy.service.route.MessageQueueView; +import org.apache.rocketmq.remoting.protocol.route.BrokerData; +import org.apache.rocketmq.remoting.protocol.route.QueueData; +import org.apache.rocketmq.remoting.protocol.route.TopicRouteData; +import org.junit.Before; +import org.junit.Test; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +public class RouteChangeNotifierTest { + + private RouteChangeNotifier notifier; + private RecordingObserver observer; + + private static class RecordingObserver implements StreamObserver { + final List events = Collections.synchronizedList(new ArrayList<>()); + volatile boolean completed; + + @Override + public void onNext(SubscribeRouteEventsResponse value) { + events.add(value.getEvent()); + } + + @Override + public void onError(Throwable t) { + } + + @Override + public void onCompleted() { + completed = true; + } + } + + @Before + public void setUp() { + notifier = new RouteChangeNotifier(); + observer = new RecordingObserver(); + } + + private static MessageQueueView view(String topic, String brokerName, String brokerAddr, int readQueues, + int writeQueues) { + TopicRouteData data = new TopicRouteData(); + BrokerData brokerData = new BrokerData(); + brokerData.setCluster("DefaultCluster"); + brokerData.setBrokerName(brokerName); + HashMap addrs = new HashMap<>(); + addrs.put(0L, brokerAddr); + brokerData.setBrokerAddrs(addrs); + data.setBrokerDatas(Collections.singletonList(brokerData)); + QueueData queueData = new QueueData(); + queueData.setBrokerName(brokerName); + queueData.setReadQueueNums(readQueues); + queueData.setWriteQueueNums(writeQueues); + queueData.setPerm(6); + data.setQueueDatas(Collections.singletonList(queueData)); + return new MessageQueueView(topic, data, null); + } + + private static MessageQueueView multiBrokerView(String topic) { + TopicRouteData data = new TopicRouteData(); + List brokers = new ArrayList<>(); + List queues = new ArrayList<>(); + String[] names = {"broker-a", "broker-b"}; + for (int i = 0; i < names.length; i++) { + BrokerData brokerData = new BrokerData(); + brokerData.setCluster("DefaultCluster"); + brokerData.setBrokerName(names[i]); + HashMap addrs = new HashMap<>(); + addrs.put(0L, "127.0.0.1:1091" + i); + brokerData.setBrokerAddrs(addrs); + brokers.add(brokerData); + QueueData queueData = new QueueData(); + queueData.setBrokerName(names[i]); + queueData.setReadQueueNums(i == 0 ? 8 : 4); + queueData.setWriteQueueNums(4); + queueData.setPerm(6); + queues.add(queueData); + } + data.setBrokerDatas(brokers); + data.setQueueDatas(queues); + return new MessageQueueView(topic, data, null); + } + + @Test + public void subscribeReplaysSnapshotAndDetectsChanges() { + MessageQueueView initial = view("TopicTest", "broker-a", "127.0.0.1:10911", 4, 4); + notifier.onRouteLoaded("TopicTest", initial); + + notifier.subscribe(SubscribeRouteEventsRequest.newBuilder().build(), observer, + Collections.singletonMap("TopicTest", initial)); + assertEquals(1, observer.events.size()); + assertEquals(RouteChangeEventType.ROUTE_SNAPSHOT, observer.events.get(0).getEventType()); + assertEquals("TopicTest", observer.events.get(0).getTopic()); + assertEquals("broker-a", observer.events.get(0).getRouteSnapshot().getBrokers(0).getBrokerName()); + + // scale queues and add a second broker + notifier.onRouteRefreshed("TopicTest", view("TopicTest", "broker-a", "127.0.0.1:10911", 4, 4), + multiBrokerView("TopicTest")); + + List types = new ArrayList<>(); + for (int i = 1; i < observer.events.size(); i++) { + types.add(observer.events.get(i).getEventType()); + } + assertTrue("expected BROKER_ONLINE in " + types, types.contains(RouteChangeEventType.BROKER_ONLINE)); + assertTrue("expected QUEUE_SCALE in " + types, types.contains(RouteChangeEventType.QUEUE_SCALE)); + } + + @Test + public void topicDeleteDetectedWhenRouteDisappears() { + notifier.onRouteLoaded("TopicGone", view("TopicGone", "broker-a", "127.0.0.1:10911", 4, 4)); + notifier.subscribe(SubscribeRouteEventsRequest.newBuilder().build(), observer, + Collections.emptyMap()); + + notifier.onRouteRefreshed("TopicGone", view("TopicGone", "broker-a", "127.0.0.1:10911", 4, 4), + MessageQueueView.WRAPPED_EMPTY_QUEUE); + + boolean deleted = false; + for (RouteChangeEvent event : observer.events) { + if (event.getEventType() == RouteChangeEventType.TOPIC_DELETE) { + deleted = true; + } + } + assertTrue(deleted); + } + + @Test + public void topicFilterAppliesToSubscribers() { + notifier.subscribe(SubscribeRouteEventsRequest.newBuilder().addTopics("Other").build(), observer, + Collections.emptyMap()); + + notifier.onRouteLoaded("TopicTest", view("TopicTest", "broker-a", "127.0.0.1:10911", 4, 4)); + assertEquals(0, observer.events.size()); + + notifier.onRouteLoaded("Other", view("Other", "broker-a", "127.0.0.1:10911", 4, 4)); + assertEquals(1, observer.events.size()); + assertEquals("Other", observer.events.get(0).getTopic()); + } + + @Test + public void unsubscribeStopsDelivery() { + MessageQueueView initial = view("TopicTest", "broker-a", "127.0.0.1:10911", 4, 4); + notifier.onRouteLoaded("TopicTest", initial); + RouteChangeNotifier.Subscription subscription = notifier.subscribe( + SubscribeRouteEventsRequest.newBuilder().build(), observer, + Collections.singletonMap("TopicTest", initial)); + assertEquals(1, observer.events.size()); + assertEquals(1, notifier.getSubscriptionCount()); + + notifier.unsubscribe(subscription); + assertEquals(0, notifier.getSubscriptionCount()); + + notifier.onRouteRefreshed("TopicTest", view("TopicTest", "broker-a", "127.0.0.1:10911", 4, 4), + view("TopicTest", "broker-a", "127.0.0.1:10911", 16, 16)); + assertEquals(1, observer.events.size()); + } + + @Test + public void shutdownCompletesSubscribers() { + notifier.onRouteLoaded("TopicTest", view("TopicTest", "broker-a", "127.0.0.1:10911", 4, 4)); + notifier.subscribe(SubscribeRouteEventsRequest.newBuilder().build(), observer, Collections.emptyMap()); + try { + notifier.shutdown(); + } catch (Exception e) { + throw new AssertionError(e); + } + assertTrue(observer.completed); + assertEquals(0, notifier.getSubscriptionCount()); + } +} diff --git a/proxy/src/test/java/org/apache/rocketmq/proxy/service/admin/DefaultAdminServiceTest.java b/proxy/src/test/java/org/apache/rocketmq/proxy/service/admin/DefaultAdminServiceTest.java index cdfc7f7fc23..770453926e9 100644 --- a/proxy/src/test/java/org/apache/rocketmq/proxy/service/admin/DefaultAdminServiceTest.java +++ b/proxy/src/test/java/org/apache/rocketmq/proxy/service/admin/DefaultAdminServiceTest.java @@ -14,90 +14,209 @@ * See the License for the specific language governing permissions and * limitations under the License. */ - package org.apache.rocketmq.proxy.service.admin; import java.util.HashMap; -import java.util.HashSet; -import java.util.Set; -import org.apache.rocketmq.client.exception.MQClientException; -import org.apache.rocketmq.common.TopicConfig; -import org.apache.rocketmq.remoting.protocol.ResponseCode; -import org.apache.rocketmq.remoting.protocol.route.BrokerData; -import org.apache.rocketmq.remoting.protocol.route.TopicRouteData; +import java.util.List; +import java.util.Map; import org.apache.rocketmq.client.impl.mqclient.MQClientAPIExt; import org.apache.rocketmq.client.impl.mqclient.MQClientAPIFactory; +import org.apache.rocketmq.common.message.MessageDecoder; +import org.apache.rocketmq.common.message.MessageExt; +import org.apache.rocketmq.common.message.MessageQueue; +import org.apache.rocketmq.remoting.InvokeCallback; +import org.apache.rocketmq.remoting.netty.ResponseFuture; +import org.apache.rocketmq.remoting.protocol.RemotingCommand; +import org.apache.rocketmq.remoting.protocol.ResponseCode; +import org.apache.rocketmq.remoting.protocol.admin.ConsumeStats; +import org.apache.rocketmq.remoting.protocol.admin.OffsetWrapper; +import org.apache.rocketmq.remoting.protocol.header.QueryMessageRequestHeader; +import org.apache.rocketmq.remoting.protocol.route.TopicRouteData; +import org.apache.rocketmq.remoting.protocol.statictopic.TopicConfigAndQueueMapping; import org.junit.Before; import org.junit.Test; import org.junit.runner.RunWith; -import org.mockito.ArgumentCaptor; import org.mockito.Mock; import org.mockito.junit.MockitoJUnitRunner; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyBoolean; import static org.mockito.ArgumentMatchers.anyLong; import static org.mockito.ArgumentMatchers.anyString; -import static org.mockito.ArgumentMatchers.eq; -import static org.mockito.Mockito.doNothing; +import static org.mockito.Mockito.doAnswer; +import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; @RunWith(MockitoJUnitRunner.class) public class DefaultAdminServiceTest { + @Mock private MQClientAPIFactory mqClientAPIFactory; @Mock private MQClientAPIExt mqClientAPIExt; - private DefaultAdminService defaultAdminService; + private DefaultAdminService adminService; + + private static final String ADDR = "127.0.0.1:10911"; + private static final String TOPIC = "topicA"; + private static final String GROUP = "groupA"; + private static final long TIMEOUT = 3000L; @Before - public void before() { + public void setUp() { when(mqClientAPIFactory.getClient()).thenReturn(mqClientAPIExt); - defaultAdminService = new DefaultAdminService(mqClientAPIFactory); + adminService = new DefaultAdminService(mqClientAPIFactory); + } + + @Test + public void getMaxOffsetDelegatesToClient() throws Exception { + MessageQueue mq = new MessageQueue(TOPIC, "broker-a", 0); + when(mqClientAPIExt.getMaxOffset(anyString(), any(MessageQueue.class), anyLong())).thenReturn(100L); + + assertEquals(100L, adminService.getMaxOffset(ADDR, mq, TIMEOUT)); + } + + @Test + public void getMinOffsetDelegatesToClient() throws Exception { + MessageQueue mq = new MessageQueue(TOPIC, "broker-a", 0); + when(mqClientAPIExt.getMinOffset(anyString(), any(MessageQueue.class), anyLong())).thenReturn(5L); + + assertEquals(5L, adminService.getMinOffset(ADDR, mq, TIMEOUT)); + } + + @Test + public void getEarliestMsgStoretimeDelegatesToClient() throws Exception { + MessageQueue mq = new MessageQueue(TOPIC, "broker-a", 0); + when(mqClientAPIExt.getEarliestMsgStoretime(anyString(), any(MessageQueue.class), anyLong())).thenReturn(12345L); + + assertEquals(12345L, adminService.getEarliestMsgStoretime(ADDR, mq, TIMEOUT)); + } + + @Test + public void fetchConsumeStatsDelegatesToClient() throws Exception { + ConsumeStats consumeStats = new ConsumeStats(); + Map table = new HashMap<>(); + OffsetWrapper wrapper = new OffsetWrapper(); + wrapper.setBrokerOffset(200); + wrapper.setConsumerOffset(150); + table.put(new MessageQueue(TOPIC, "broker-a", 0), wrapper); + consumeStats.setOffsetTable(table); + when(mqClientAPIExt.getConsumeStats(anyString(), anyString(), anyString(), anyLong())).thenReturn(consumeStats); + + ConsumeStats result = adminService.fetchConsumeStats(ADDR, GROUP, TOPIC, TIMEOUT); + assertNotNull(result); + assertEquals(200L, result.getOffsetTable().get(new MessageQueue(TOPIC, "broker-a", 0)).getBrokerOffset()); } @Test - public void testCreateTopic() throws Exception { - when(mqClientAPIExt.getTopicRouteInfoFromNameServer(eq("createTopic"), anyLong())) - .thenThrow(new MQClientException(ResponseCode.TOPIC_NOT_EXIST, "")) - .thenReturn(createTopicRouteData(1)); - when(mqClientAPIExt.getTopicRouteInfoFromNameServer(eq("sampleTopic"), anyLong())) - .thenReturn(createTopicRouteData(2)); - - ArgumentCaptor addrArgumentCaptor = ArgumentCaptor.forClass(String.class); - ArgumentCaptor topicConfigArgumentCaptor = ArgumentCaptor.forClass(TopicConfig.class); - doNothing().when(mqClientAPIExt).createTopic(addrArgumentCaptor.capture(), anyString(), topicConfigArgumentCaptor.capture(), anyLong()); - - assertTrue(defaultAdminService.createTopicOnTopicBrokerIfNotExist( - "createTopic", - "sampleTopic", - 7, - 8, - true, - 1 - )); - - assertEquals(2, addrArgumentCaptor.getAllValues().size()); - Set createAddr = new HashSet<>(addrArgumentCaptor.getAllValues()); - assertTrue(createAddr.contains("127.0.0.1:10911")); - assertTrue(createAddr.contains("127.0.0.2:10911")); - assertEquals("createTopic", topicConfigArgumentCaptor.getValue().getTopicName()); - assertEquals(7, topicConfigArgumentCaptor.getValue().getWriteQueueNums()); - assertEquals(8, topicConfigArgumentCaptor.getValue().getReadQueueNums()); + public void resetOffsetDelegatesToClient() throws Exception { + Map map = new HashMap<>(); + map.put(new MessageQueue(TOPIC, "broker-a", 0), 42L); + when(mqClientAPIExt.invokeBrokerToResetOffset(anyString(), anyString(), anyString(), anyLong(), anyBoolean(), anyLong())) + .thenReturn(map); + + Map result = adminService.resetOffset(ADDR, TOPIC, GROUP, System.currentTimeMillis(), true, TIMEOUT); + assertEquals(42L, result.get(new MessageQueue(TOPIC, "broker-a", 0)).longValue()); } - private TopicRouteData createTopicRouteData(int brokerNum) { + @Test + public void viewMessageDelegatesToClient() throws Exception { + MessageExt ext = new MessageExt(); + ext.setTopic(TOPIC); + ext.setMsgId("m1"); + when(mqClientAPIExt.viewMessage(anyString(), anyString(), anyLong(), anyLong())).thenReturn(ext); + + MessageExt result = adminService.viewMessage(ADDR, TOPIC, 12345L, TIMEOUT); + assertNotNull(result); + assertEquals("m1", result.getMsgId()); + } + + @Test + public void getTopicConfigDelegatesToClient() throws Exception { + TopicConfigAndQueueMapping topicConfig = new TopicConfigAndQueueMapping(); + topicConfig.setTopicName(TOPIC); + topicConfig.setReadQueueNums(8); + when(mqClientAPIExt.getTopicConfig(anyString(), anyString(), anyLong())).thenReturn(topicConfig); + + TopicConfigAndQueueMapping result = adminService.getTopicConfig(ADDR, TOPIC, TIMEOUT); + assertNotNull(result); + assertEquals(TOPIC, result.getTopicName()); + } + + @Test + public void getTopicRouteDataDelegatesToClient() throws Exception { TopicRouteData topicRouteData = new TopicRouteData(); - for (int i = 0; i < brokerNum; i++) { - BrokerData brokerData = new BrokerData(); - HashMap addrMap = new HashMap<>(); - addrMap.put(0L, "127.0.0." + (i + 1) + ":10911"); - brokerData.setBrokerAddrs(addrMap); - brokerData.setBrokerName("broker-" + i); - brokerData.setCluster("cluster"); - topicRouteData.getBrokerDatas().add(brokerData); + topicRouteData.setOrderTopicConf("orderConf"); + when(mqClientAPIExt.getTopicRouteInfoFromNameServer(anyString(), anyLong())).thenReturn(topicRouteData); + + TopicRouteData result = adminService.getTopicRouteData(TOPIC); + assertNotNull(result); + assertEquals("orderConf", result.getOrderTopicConf()); + } + + @Test + public void queryMessageReturnsDecodedMessages() throws Exception { + MessageExt ext = new MessageExt(); + ext.setTopic(TOPIC); + ext.setMsgId("m1"); + ext.setBody("payload".getBytes()); + ext.setBornHost(new java.net.InetSocketAddress("127.0.0.1", 10909)); + ext.setStoreHost(new java.net.InetSocketAddress("127.0.0.1", 10911)); + ext.setBornTimestamp(System.currentTimeMillis()); + ext.setStoreTimestamp(System.currentTimeMillis()); + final byte[] body = MessageDecoder.encode(ext, false); + + doAnswer(invocation -> { + InvokeCallback callback = invocation.getArgument(3); + ResponseFuture responseFuture = mock(ResponseFuture.class); + RemotingCommand response = RemotingCommand.createResponseCommand(ResponseCode.SUCCESS, "ok"); + response.setBody(body); + when(responseFuture.getResponseCommand()).thenReturn(response); + callback.operationComplete(responseFuture); + return null; + }).when(mqClientAPIExt).queryMessage(anyString(), any(QueryMessageRequestHeader.class), anyLong(), any(InvokeCallback.class), anyBoolean()); + + List result = adminService.queryMessage(ADDR, TOPIC, "key", 10, 0L, System.currentTimeMillis(), TIMEOUT); + assertNotNull(result); + assertEquals(1, result.size()); + assertEquals(TOPIC, result.get(0).getTopic()); + assertEquals("payload", new String(result.get(0).getBody())); + } + + @Test + public void queryMessageFailsWhenCallbackFails() throws Exception { + doAnswer(invocation -> { + InvokeCallback callback = invocation.getArgument(3); + callback.operationFail(new RuntimeException("boom")); + return null; + }).when(mqClientAPIExt).queryMessage(anyString(), any(QueryMessageRequestHeader.class), anyLong(), any(InvokeCallback.class), anyBoolean()); + + try { + adminService.queryMessage(ADDR, TOPIC, "key", 10, 0L, System.currentTimeMillis(), TIMEOUT); + fail("expected queryMessage to throw"); + } catch (Exception e) { + assertTrue(e.getCause() instanceof RuntimeException); + assertEquals("boom", e.getCause().getMessage()); } - return topicRouteData; } -} \ No newline at end of file + + @Test + public void queryMessageReturnsEmptyWhenResponseNotSuccess() throws Exception { + doAnswer(invocation -> { + InvokeCallback callback = invocation.getArgument(3); + ResponseFuture responseFuture = mock(ResponseFuture.class); + RemotingCommand response = RemotingCommand.createResponseCommand(ResponseCode.SYSTEM_ERROR, "err"); + when(responseFuture.getResponseCommand()).thenReturn(response); + callback.operationComplete(responseFuture); + return null; + }).when(mqClientAPIExt).queryMessage(anyString(), any(QueryMessageRequestHeader.class), anyLong(), any(InvokeCallback.class), anyBoolean()); + + List result = adminService.queryMessage(ADDR, TOPIC, "key", 10, 0L, System.currentTimeMillis(), TIMEOUT); + assertNotNull(result); + assertTrue(result.isEmpty()); + } +}