Add whole-internet Web Search platform via server-side GraphQL

README.md

@@ -24,6 +24,12 @@ interactive terminal UI.
   Normal (fast) to Max (full comment graph), persisted across sessions.
 - **Selectable research platforms**: `/serching-platforms` lets you toggle source
   groups on/off — skip Reddit, use only calendars, or go all-in.
+- **Whole-internet Web Search**: a `Web Search` platform (in the same
+  `/serching-platforms` pop-up) layers a Google web search — modelled on
+  [web-agent-master/google-search](https://github.com/web-agent-master/google-search)
+  and scraped with Playwright, with a DuckDuckGo HTML fallback — *on top of* the
+  curated sources, so research is no longer limited to the built-in feeds. The
+  search runs **server-side** through the GraphQL `webSearch` query.
 - **Reddit-specific extraction**: specialized parser follows comment-thread links
   to other Reddit posts and external articles.
 - **13 strategies** computed from textbook technical indicators (SMA, EMA, RSI,
@@ -72,7 +78,8 @@ interactive terminal UI.
   facade onto GraphQL DTOs, and the frontend `GraphqlTradefastRepository`
   (`src/cli/graphql/`) is the only place the CLI talks to the API. Each GraphQL
   class lives in its own file on both sides. The headless `status`, `strategies`
-  and `clear` commands make their requests through this path.
+  and `clear` commands make their requests through this path, as does the
+  whole-internet `webSearch` query so web search executes server-side.
 - **Dockerised**: `docker compose up` brings up PostgreSQL and the CLI.
 
 ---
@@ -129,7 +136,7 @@ TRADEFAST_MARKET_SOURCE=synthetic TRADEFAST_DATA_DIR=:memory: node dist/index.js
 | `/operating-mode [name]` | Open the trading-style selector pop-up, or switch directly by name (`long-term`, `medium-term`, `scalping`). Applies that horizon's timeframe. |
 | `/operating-mode-time [tf]` | Open the timeframe selector, or set it directly (`1m`–`1d`) to fine-tune within the current mode. |
 | `/serching-level [level]` | Set crawl depth/resolution: `normal` (fast, depth 2), `high` (deep, depth 4), or `max` (full graph, depth 8 with comment traversal). Opens a pop-up without argument. |
-| `/serching-platforms` | Toggle source groups on/off: economic calendars, news portals, crypto news, Reddit communities, crypto communities, exchange communities. Opens a multi-select pop-up. |
+| `/serching-platforms` | Toggle research platforms on/off: economic calendars, news portals, crypto news, Reddit communities, crypto communities, exchange communities, plus **Web Search** (whole-internet Google/DuckDuckGo). Opens a multi-select pop-up. |
 | `/currency [symbol]` | Run a full forecast for a single symbol with news sentiment and price chart. |
 | `/exchange [name]` | Switch the venue/data source: `binance`, `okx`, `bybit`, `mexc` (crypto spot) or `pocketoption` (forex binary options). Pocket Option swaps in forex majors and renders an expiry **Time** column instead of TP/SL. Opens a pop-up without an argument. |
 | `/ratings` | Show source credibility ratings. Subcommands: `correct`, `incorrect`, `loud-claim`, or a numeric grade (`/ratings "Хабр" -1`). |
@@ -294,7 +301,7 @@ All configuration is environment-driven (see `.env.example`):
 | `TRADEFAST_AI_MODEL`       | `claude-4.7-opus`             | Model for per-symbol advice and cross-symbol correction.        |
 | `TRADEFAST_SKIP_AI_VALIDATION` | `0`                        | Set to `1` to skip the cross-symbol AI correction step.        |
 | `TRADEFAST_SEARCHING_LEVEL`| _(unset → pop-up)_           | Preset depth: `normal`, `high`, or `max`. Skips the pop-up.     |
-| `TRADEFAST_SEARCHING_PLATFORMS` | _(unset → pop-up)_      | Comma-separated source groups to enable (e.g. `news-portals,reddit-communities`). Skips the pop-up. |
+| `TRADEFAST_SEARCHING_PLATFORMS` | _(unset → pop-up)_      | Comma-separated platforms to enable (e.g. `news-portals,reddit-communities,web-search`). Add `web-search` for whole-internet search. Skips the pop-up. |
 
 The market source falls back gracefully: `resilient` uses live Binance data and
 transparently switches to deterministic synthetic candles if the network is
@@ -344,6 +351,13 @@ limit. Sources are grouped into 6 platform groups:
 | `crypto-communities`   | 6 crypto subreddits (Bitcoin, ethereum, CryptoMarkets, defi, Altcoin, CryptoTechnology) — kind: `reddit`. |
 | `exchange-communities` | Binance, Bybit, OKX, MEXC blogs + their subreddits.                     |
 
+Beyond these curated groups, the **`web-search`** platform adds a whole-internet
+Google web search (DuckDuckGo HTML fallback) that runs server-side via the
+GraphQL `webSearch` query, so `/start`, `/update` and `/currency` can pull in
+results from outside the configured feeds. It is **not** a news group — it has no
+curated `sources.json` entry — so it appears in `/serching-platforms` as its own
+toggle and contributes results to the general search table alongside the crawl.
+
 Sources with `kind: "reddit"` use a specialized extractor that reads Reddit
 threads, extracts the self-text/comments, and follows comment-thread links to
 other Reddit posts and external articles. This produces rich cross-referenced
@@ -425,9 +439,10 @@ exposes these tools to the AI:
 - **`run_serching_level`** — sets crawl depth/resolution. The AI can request
   `normal` (fast, shallow), `high` (deep), or `max` (exhaustive) depending on
   how much context it needs.
-- **`run_serching_platforms`** — enables/disables source groups. The AI can
+- **`run_serching_platforms`** — enables/disables research platforms. The AI can
   toggle specific groups (e.g. enable only `economic-calendars` for
-  fundamentals) to tailor research scope.
+  fundamentals) or the whole-internet `web-search` platform to tailor research
+  scope.
 - **`run_ratings_adjust`** — modifies a source's credibility rating. Parameters:
   `source` (title or ID) and `grade` (integer, e.g. `-1` = −1%). Triggered by
   user feedback like *"Хабр -1"* or *"понизь рейтинг Хабру"*.

package.json

@@ -1,6 +1,6 @@
 {
   "name": "Tradefast",
-  "version": "0.6.0",
+  "version": "0.7.0",
   "description": "TRADEFΛST — a disciplined crypto market-research CLI: strategies, analytics, risk and AI-assisted research, in the style of Gemini CLI.",
   "type": "module",
   "license": "MIT",

src/app/tradefast.ts

@@ -21,6 +21,9 @@ import {
 import { ALL_STRATEGIES } from '../strategies/registry.js';
 import { computeCrowdConsensus, type InstrumentConsensus } from '../services/news-consensus.js';
 import { SourceRatingService } from '../services/source-ratings.js';
+import { isWebSearchEnabled } from '../cli/sources.js';
+import { createWebSearchProvider } from '../services/web-search.js';
+import type { SearchResult } from '../services/search.js';
 
 export interface StatusReport {
   driver: string;
@@ -107,6 +110,19 @@ export class Tradefast {
     (this.config as any).interval = interval;
   }
 
+  /** Updates the enabled research platforms (used by the `/serching-platforms`
+   *  selector). The "Web Search" toggle here drives whether subsequent /start,
+   *  /update and /currency runs search the whole Internet in addition to the
+   *  curated knowledge base. */
+  setSearchingPlatforms(platforms: readonly string[]): void {
+    (this.config as any).searchingPlatforms = [...platforms];
+  }
+
+  /** Whether whole-internet web search is currently enabled in the config. */
+  private get webSearchEnabled(): boolean {
+    return isWebSearchEnabled(this.config.searchingPlatforms ?? []);
+  }
+
   /** Records the active operating mode (trading style). The associated timeframe
    *  is applied separately via {@link setInterval} so analysis actually shifts
    *  to the chosen horizon. */
@@ -129,6 +145,7 @@ export class Tradefast {
         limit: this.config.candleLimit,
         accountBalance: this.config.accountBalance,
         exchange: this.config.exchange,
+        webSearch: this.webSearchEnabled,
         ...extra,
       },
       onProgress,
@@ -145,6 +162,7 @@ export class Tradefast {
         limit: this.config.candleLimit,
         accountBalance: this.config.accountBalance,
         exchange: this.config.exchange,
+        webSearch: this.webSearchEnabled,
         ...extra,
       },
       onProgress,
@@ -194,6 +212,7 @@ export class Tradefast {
         limit: this.config.candleLimit,
         accountBalance: this.config.accountBalance,
         exchange: this.config.exchange,
+        webSearch: this.webSearchEnabled,
         ...extra,
       },
       onProgress,
@@ -215,6 +234,18 @@ export class Tradefast {
     return { symbol, report, price, newsConsensus: currencyNews, candles };
   }
 
+  /**
+   * Whole-internet web search exposed for the backend `webSearch` GraphQL query.
+   * Runs server-side through the Playwright/HTTP {@link WebSearchProvider} so the
+   * CLI can search the entire Internet, not just the curated sources. Returns an
+   * empty list on failure rather than throwing.
+   */
+  async search(query: string, limit = 10): Promise<SearchResult[]> {
+    const provider = createWebSearchProvider({ limit });
+    const results = await provider.search(query);
+    return results.slice(0, limit);
+  }
+
   /** Read back persisted candles for chart rendering. */
   getCandles(symbol: string, interval: string): Promise<Candle[]> {
     return this.store.getCandles(symbol, interval);

src/backend/graphql/dto/search-result.dto.ts

@@ -0,0 +1,20 @@
+import { Field, Float, ObjectType } from '@nestjs/graphql';
+
+/** A single whole-internet web search hit, returned by the `webSearch` query. */
+@ObjectType()
+export class SearchResultDto {
+  @Field(() => String)
+  source!: string;
+
+  @Field(() => String)
+  title!: string;
+
+  @Field(() => String, { nullable: true })
+  url!: string | null;
+
+  @Field(() => String, { nullable: true })
+  snippet!: string | null;
+
+  @Field(() => Float)
+  score!: number;
+}

src/backend/graphql/facade.ts

@@ -1,5 +1,6 @@
 import type { StatusReport } from '../../app/tradefast.js';
 import type { RunReport } from '../../pipeline/collector.js';
+import type { SearchResult } from '../../services/search.js';
 
 /** DI token under which the application facade is provided to the resolver. */
 export const TRADEFAST_FACADE = Symbol('TRADEFAST_FACADE');
@@ -16,4 +17,6 @@ export interface TradefastApiFacade {
   start(): Promise<RunReport>;
   update(): Promise<RunReport>;
   clear(): Promise<number>;
+  /** Whole-internet web search executed server-side. */
+  search(query: string, limit?: number): Promise<SearchResult[]>;
 }

src/backend/graphql/index.ts

@@ -2,6 +2,7 @@
 export { TRADEFAST_FACADE, type TradefastApiFacade } from './facade.js';
 export { AnalyticsDto } from './dto/analytics.dto.js';
 export { RunReportDto } from './dto/run-report.dto.js';
+export { SearchResultDto } from './dto/search-result.dto.js';
 export { StatusDto } from './dto/status.dto.js';
 export { StrategyDto } from './dto/strategy.dto.js';
 export { SymbolRunDto } from './dto/symbol-run.dto.js';

src/backend/graphql/repository.ts

@@ -2,6 +2,7 @@ import { Inject, Injectable } from '@nestjs/common';
 
 import type { RunReport } from '../../pipeline/collector.js';
 import { RunReportDto } from './dto/run-report.dto.js';
+import { SearchResultDto } from './dto/search-result.dto.js';
 import { StatusDto } from './dto/status.dto.js';
 import { StrategyDto } from './dto/strategy.dto.js';
 import { TRADEFAST_FACADE, type TradefastApiFacade } from './facade.js';
@@ -52,6 +53,17 @@ export class TradefastRepository {
   clear(): Promise<number> {
     return this.tradefast.clear();
   }
+
+  async webSearch(query: string, limit?: number): Promise<SearchResultDto[]> {
+    const results = await this.tradefast.search(query, limit);
+    return results.map((result) => ({
+      source: result.source,
+      title: result.title,
+      url: result.url ?? null,
+      snippet: result.snippet ?? null,
+      score: result.score,
+    }));
+  }
 }
 
 /** Map a pipeline {@link RunReport} into its GraphQL DTO. */

src/backend/graphql/tradefast.resolver.ts

@@ -1,7 +1,9 @@
+import 'reflect-metadata';
 import { Inject } from '@nestjs/common';
-import { Int, Mutation, Query, Resolver } from '@nestjs/graphql';
+import { Args, Int, Mutation, Query, Resolver } from '@nestjs/graphql';
 
 import { RunReportDto } from './dto/run-report.dto.js';
+import { SearchResultDto } from './dto/search-result.dto.js';
 import { StatusDto } from './dto/status.dto.js';
 import { StrategyDto } from './dto/strategy.dto.js';
 import { TradefastRepository } from './repository.js';
@@ -24,6 +26,15 @@ export class TradefastResolver {
     return this.repository.strategies();
   }
 
+  /** Whole-internet web search ("Web Search" platform), executed server-side. */
+  @Query(() => [SearchResultDto])
+  webSearch(
+    @Args('query', { type: () => String }) query: string,
+    @Args('limit', { type: () => Int, nullable: true }) limit?: number,
+  ): Promise<SearchResultDto[]> {
+    return this.repository.webSearch(query, limit ?? undefined);
+  }
+
   @Mutation(() => RunReportDto)
   start(): Promise<RunReportDto> {
     return this.repository.start();
@@ -39,3 +50,12 @@ export class TradefastResolver {
     return this.repository.clear();
   }
 }
+
+// The build pipeline (esbuild via tsup/vitest) cannot emit `design:paramtypes`,
+// which `@Args()` reflects at schema-build time. Every other field passes an
+// explicit `@Field(() => …)` thunk and so never needs it, but `@Args()` reads
+// the reflected parameter list *before* honouring its explicit `type` option and
+// would otherwise crash on `undefined[index]`. Declaring the metadata by hand —
+// exactly what `emitDecoratorMetadata` would generate — keeps the schema build
+// reflection-free while preserving the explicit `() => String` / `() => Int` types.
+Reflect.defineMetadata('design:paramtypes', [String, Number], TradefastResolver.prototype, 'webSearch');

src/cli/App.tsx

@@ -22,7 +22,13 @@ import { getExchange, exchangeNames, isKnownExchange, isBinaryOptions, type Exch
 import { getInterval, intervalNames, type IntervalName } from './intervals.js';
 import { getMode, modeNames, type ModeName } from './modes.js';
 import { searchLevelNames, getSearchLevel, type SearchLevelName } from './search-level.js';
-import { sourceGroupIds, getSourceGroup, resolveSourceIds, DEFAULT_ENABLED_GROUPS, type SourceGroupId } from './sources.js';
+import {
+  resolveSourceIds,
+  selectablePlatformIds,
+  getPlatformLabel,
+  DEFAULT_ENABLED_PLATFORMS,
+  type ResearchPlatformId,
+} from './sources.js';
 
 export interface AppProps {
   app: Tradefast;
@@ -65,14 +71,15 @@ export function App({ app, version, apiUrl, promptOperatingMode }: AppProps): Re
   const [searchingLevel, setSearchingLevel] = useState<SearchLevelName>(() => getSearchLevel(app.config.searchingLevel).name as SearchLevelName);
   const [platformSelectorOpen, setPlatformSelectorOpen] = useState(false);
   const [platformCursorIndex, setPlatformCursorIndex] = useState(0);
-  const [enabledPlatforms, setEnabledPlatforms] = useState<SourceGroupId[]>(() => {
+  const [enabledPlatforms, setEnabledPlatforms] = useState<ResearchPlatformId[]>(() => {
     if (Array.isArray(app.config.searchingPlatforms) && app.config.searchingPlatforms.length > 0) {
-      const valid = app.config.searchingPlatforms.filter((g): g is SourceGroupId =>
-        (sourceGroupIds() as string[]).includes(g),
+      const selectable = selectablePlatformIds() as string[];
+      const valid = app.config.searchingPlatforms.filter((g): g is ResearchPlatformId =>
+        selectable.includes(g),
       );
-      return valid.length > 0 ? valid : [...DEFAULT_ENABLED_GROUPS];
+      return valid.length > 0 ? valid : [...DEFAULT_ENABLED_PLATFORMS];
     }
-    return [...DEFAULT_ENABLED_GROUPS];
+    return [...DEFAULT_ENABLED_PLATFORMS];
   });
   const [busy, setBusy] = useState(false);
   const newsCrawlOptions = useCallback(
@@ -214,7 +221,7 @@ export function App({ app, version, apiUrl, promptOperatingMode }: AppProps): Re
   );
 
   const togglePlatform = useCallback(
-    (id: SourceGroupId) => {
+    (id: ResearchPlatformId) => {
       setEnabledPlatforms((prev) =>
         prev.includes(id) ? prev.filter((g) => g !== id) : [...prev, id],
       );
@@ -226,11 +233,12 @@ export function App({ app, version, apiUrl, promptOperatingMode }: AppProps): Re
     () => {
       setPlatformSelectorOpen(false);
       void saveSearchingPlatforms(enabledPlatforms);
+      app.setSearchingPlatforms(enabledPlatforms);
       const count = resolveSourceIds(enabledPlatforms).length;
-      const groups = enabledPlatforms.map((id) => getSourceGroup(id)?.label ?? id).join(', ');
+      const groups = enabledPlatforms.map((id) => getPlatformLabel(id)).join(', ');
       push({ kind: 'text', text: `Research platforms: ${groups} (${count} sources)`, color: theme.colors.info });
     },
-    [enabledPlatforms, push, theme],
+    [app, enabledPlatforms, push, theme],
   );
 
   const applyCurrency = useCallback(
@@ -355,13 +363,13 @@ export function App({ app, version, apiUrl, promptOperatingMode }: AppProps): Re
       if (key.escape) {
         setPlatformSelectorOpen(false);
       } else if (key.upArrow) {
-        setPlatformCursorIndex((index) => (index - 1 + sourceGroupIds().length) % sourceGroupIds().length);
+        setPlatformCursorIndex((index) => (index - 1 + selectablePlatformIds().length) % selectablePlatformIds().length);
       } else if (key.downArrow) {
-        setPlatformCursorIndex((index) => (index + 1) % sourceGroupIds().length);
+        setPlatformCursorIndex((index) => (index + 1) % selectablePlatformIds().length);
       } else if (key.return) {
         applyPlatforms();
       } else if (_input === ' ') {
-        togglePlatform(sourceGroupIds()[platformCursorIndex]);
+        togglePlatform(selectablePlatformIds()[platformCursorIndex]);
       }
       return;
     }
@@ -505,7 +513,7 @@ export function App({ app, version, apiUrl, promptOperatingMode }: AppProps): Re
           return;
         }
         if (args[0] === 'check') {
-          const groups = enabledPlatforms.map((id) => getSourceGroup(id)?.label ?? id).join(', ');
+          const groups = enabledPlatforms.map((id) => getPlatformLabel(id)).join(', ');
           const count = resolveSourceIds(enabledPlatforms).length;
           push({ kind: 'text', text: `Research platforms: ${groups} (${count} sources)`, color: theme.colors.info });
           return;
@@ -747,12 +755,14 @@ export function App({ app, version, apiUrl, promptOperatingMode }: AppProps): Re
               case 'run_serching_platforms': {
                 const groups = toolArgs.groups as string[] | undefined;
                 if (groups && Array.isArray(groups) && groups.length > 0) {
-                  const valid = groups.filter((g) => sourceGroupIds().includes(g as SourceGroupId));
+                  const selectable = selectablePlatformIds() as string[];
+                  const valid = groups.filter((g) => selectable.includes(g)) as ResearchPlatformId[];
                   if (valid.length > 0) {
-                    setEnabledPlatforms(valid as SourceGroupId[]);
-                    void saveSearchingPlatforms(valid as SourceGroupId[]);
-                    const count = resolveSourceIds(valid as SourceGroupId[]).length;
-                    return `Research platforms set to ${valid.length} group(s) (${count} sources).`;
+                    setEnabledPlatforms(valid);
+                    void saveSearchingPlatforms(valid);
+                    app.setSearchingPlatforms(valid);
+                    const count = resolveSourceIds(valid).length;
+                    return `Research platforms set to ${valid.length} platform(s) (${count} sources).`;
                   }
                 }
                 setPlatformCursorIndex(0);