diff --git a/ROADMAP.md b/ROADMAP.md index 549509c..48511b8 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,6 +1,6 @@ # Roadmap & Next Steps -Status: **v0.1.0 — alpha, release-ready for local/self-hosted use.** +Status: **v0.3.1 — alpha.** CI live at [`.github/workflows/ci.yml`](.github/workflows/ci.yml). Public VPS+gVisor path: [`docs/DEPLOY_GVISOR.md`](docs/DEPLOY_GVISOR.md). ## 🟢 Ready to use right now @@ -22,21 +22,12 @@ Status: **v0.1.0 — alpha, release-ready for local/self-hosted use.** ``` Or Docker: `docker compose up -d`. -2. **Enable GitHub Actions CI** - The CI workflow is staged at [`.github-pending/ci.yml`](.github-pending/ci.yml) - because the local `gh` token is missing the `workflow` scope. To enable: - ```bash - gh auth refresh -s workflow - mkdir -p .github/workflows - mv .github-pending/ci.yml .github/workflows/ci.yml - rmdir .github-pending - git add .github && git commit -m "ci: enable GitHub Actions" - git push - ``` +2. **GitHub Actions CI** + CI is live at [`.github/workflows/ci.yml`](.github/workflows/ci.yml). 3. **Publish to PyPI** - Reserve the name: - - Create a release: `git tag v0.1.0 && git push --tags` + - Create a release: `git tag v0.3.1 && git push --tags` - Add a `release.yml` workflow that runs on tags and publishes via [trusted publishing](https://docs.pypi.org/trusted-publishers/). diff --git a/deploy/docker-compose.gvisor.yml b/deploy/docker-compose.gvisor.yml new file mode 100644 index 0000000..56a4ee5 --- /dev/null +++ b/deploy/docker-compose.gvisor.yml @@ -0,0 +1,67 @@ +# gVisor-hardened openfindata deploy. +# Never attach this service to hermes/wealthuman networks. +# Code mode must stay off (do not set FINDATA_MCP_CODE_MODE). +# Requires Docker runtime runsc (gVisor). +services: + openfindata: + build: + context: .. + dockerfile: Dockerfile + image: openfindata:latest + container_name: openfindata + runtime: runsc + restart: unless-stopped + ports: + - "127.0.0.1:8000:8000" + # gVisor does not reliably use Docker's 127.0.0.11 stub DNS. + # Mount a resolv.conf that points straight at public resolvers. + dns: + - 1.1.1.1 + - 8.8.8.8 + volumes: + - ./resolv.gvisor.conf:/etc/resolv.conf:ro + read_only: true + tmpfs: + - /tmp:mode=1777 + cap_drop: + - ALL + security_opt: + - no-new-privileges:true + mem_limit: 512m + cpus: 1.0 + user: "65534:65534" + pids_limit: 256 + environment: + FINDATA_RATE_LIMIT_ENABLED: "true" + FINDATA_RATE_LIMIT_DEFAULT: "60/minute;1000/day" + # Explicitly off — do not override via .env on public VPS. + FINDATA_MCP_CODE_MODE: "0" + healthcheck: + test: + - "CMD" + - "python" + - "-c" + - "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8000/health', timeout=2).status==200 else 1)" + interval: 30s + timeout: 3s + retries: 3 + start_period: 5s + logging: + driver: json-file + options: + max-size: "10m" + max-file: "5" + networks: + - openfindata_net + labels: + - traefik.enable=true + - traefik.docker.network=deploy_openfindata_net + # Production: export OPENFINDATA_HOST=api.seudominio.com before up. + - traefik.http.routers.openfindata.rule=Host(`${OPENFINDATA_HOST:-findata.localhost}`) + - traefik.http.routers.openfindata.entrypoints=websecure + - traefik.http.routers.openfindata.tls.certresolver=letsencrypt + - traefik.http.services.openfindata.loadbalancer.server.port=8000 + +networks: + openfindata_net: + driver: bridge diff --git a/deploy/resolv.gvisor.conf b/deploy/resolv.gvisor.conf new file mode 100644 index 0000000..072d82e --- /dev/null +++ b/deploy/resolv.gvisor.conf @@ -0,0 +1,5 @@ +# Used by deploy/docker-compose.gvisor.yml — gVisor does not reliably +# follow Docker's 127.0.0.11 stub resolver. +nameserver 1.1.1.1 +nameserver 8.8.8.8 +options edns0 diff --git a/docs/DEPLOY_GVISOR.md b/docs/DEPLOY_GVISOR.md new file mode 100644 index 0000000..2090923 --- /dev/null +++ b/docs/DEPLOY_GVISOR.md @@ -0,0 +1,115 @@ +# Deploy público com gVisor (VPS) + +Guia prático para subir o **Dados Financeiros Abertos** em VPS com runtime +**runsc (gVisor)**, Traefik em host mode e rede isolada. + +> Esta VPS **não tem KVM aninhado**. O gVisor em modo **systrap** é a camada de +> sandbox do processo do container, **não** uma segunda VM. + +## Pré-requisitos + +- Docker Engine com runtime **runsc** instalado +- Traefik já em host mode na monvanti-vps (entrypoints `websecure`, certresolver + `letsencrypt`) +- Domínio apontando para a VPS + +### Instalar gVisor / runsc (snippet) + +```bash +# Exemplo baseado em release oficial do gVisor (ajuste a arquitetura se preciso) +ARCH=$(uname -m) +URL=https://storage.googleapis.com/gvisor/releases/release/latest/${ARCH} +curl -fsSL "${URL}/runsc" -o /tmp/runsc +curl -fsSL "${URL}/runsc.sha512" -o /tmp/runsc.sha512 +(cd /tmp && sha512sum -c runsc.sha512) +sudo mv /tmp/runsc /usr/local/bin/runsc +sudo chmod 755 /usr/local/bin/runsc + +# Registrar o runtime sem sobrescrever o daemon.json existente +sudo /usr/local/bin/runsc install +sudo systemctl reload docker || sudo systemctl restart docker +``` + +Verifique: + +```bash +docker info | grep -i runsc +docker run --rm --runtime=runsc hello-world +``` + +## Clone / update em `/opt/openfindata` + +```bash +sudo mkdir -p /opt +sudo git clone https://github.com/robertoecf/openfindata.git /opt/openfindata +# ou, se já existir: +cd /opt/openfindata && sudo git pull --ff-only +cd /opt/openfindata +``` + +## Variável de host + +```bash +export OPENFINDATA_HOST=seu.dominio +# opcional: persistir em deploy/.env ao lado do compose +``` + +## Subir o serviço + +```bash +cd /opt/openfindata +docker compose -f deploy/docker-compose.gvisor.yml up -d --build +``` + +O compose publica só em `127.0.0.1:8000` e usa labels Traefik. **Não** anexe esta +rede a stacks hermes/wealthuman. **Não** habilite code mode +(`FINDATA_MCP_CODE_MODE` deve permanecer ausente). + +## Smoke checks + +```bash +curl -sS http://127.0.0.1:8000/health +curl -sS http://127.0.0.1:8000/stats +curl -sS 'http://127.0.0.1:8000/bcb/series/name/selic?n=3' +# MCP HTTP transport: +curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/mcp +``` + +Pelo domínio (via Traefik): + +```bash +curl -sS "https://${OPENFINDATA_HOST}/health" +curl -sS "https://${OPENFINDATA_HOST}/stats" +curl -sS "https://${OPENFINDATA_HOST}/bcb/series/name/selic?n=3" +``` + +## Checklist de segurança + +- [ ] `runtime: runsc` ativo no container +- [ ] publish apenas em loopback (`127.0.0.1:8000`) +- [ ] sem mount de `docker.sock` +- [ ] sem `network_mode: host` +- [ ] code mode desligado (sem `FINDATA_MCP_CODE_MODE`) +- [ ] rede isolada `openfindata_net` (não compartilhada com hermes/wealthuman) +- [ ] limite de memória (`mem_limit: 512m`) e CPU (`cpus: 1.0`) +- [ ] `read_only: true`, `cap_drop: [ALL]`, `no-new-privileges:true` +- [ ] DNS via `deploy/resolv.gvisor.conf` (gVisor + `127.0.0.11` falha) + +## Troubleshooting + +```bash +# runtime registrado? +docker info | grep -i runsc + +# runtime executa? +docker run --rm --runtime=runsc hello-world + +# container e health +docker compose -f deploy/docker-compose.gvisor.yml ps +docker inspect --format '{{.HostConfig.Runtime}}' openfindata +curl -v http://127.0.0.1:8000/health +``` + +Se o Traefik não rotear, confira `OPENFINDATA_HOST`, se o Traefik enxerga a rede +do container e se o entrypoint `websecure` + `letsencrypt` já estão válidos na +monvanti-vps. diff --git a/docs/DEPLOY_PUBLIC.md b/docs/DEPLOY_PUBLIC.md index 5d28e8a..a3cd284 100644 --- a/docs/DEPLOY_PUBLIC.md +++ b/docs/DEPLOY_PUBLIC.md @@ -1,5 +1,7 @@ # Deploy público do Dados Financeiros Abertos no seu PC (WSL + Cloudflare Tunnel) +> **VPS + gVisor:** para deploy em VPS com runtime runsc, veja [`docs/DEPLOY_GVISOR.md`](DEPLOY_GVISOR.md). + > **Meta:** expor o **Dados Financeiros Abertos** como **servidor MCP público** — acessível via > HTTPS, com TLS, rate limit, e _sem_ abrir porta no roteador nem pagar nada. > @@ -71,7 +73,7 @@ Pronto. Em ~30s: ```bash curl https://findata.seudominio.com.br/health -# {"status":"ok","version":"0.1.0"} +# {"status":"ok","version":"0.3.1"} curl https://findata.seudominio.com.br/stats # { ... uptime, cache, rate limits ... } diff --git a/docs/MCP_SURFACE.md b/docs/MCP_SURFACE.md index 31c9961..0499b57 100644 --- a/docs/MCP_SURFACE.md +++ b/docs/MCP_SURFACE.md @@ -1,6 +1,6 @@ # MCP surface: curated tools over the REST API -> Status: prototype / design proposal (alpha 0.3.x). Non-breaking: the REST API +> Status: implemented (alpha curated catalog). Non-breaking: the REST API > is untouched. Implemented in [`src/findata/api/mcp_app.py`](../src/findata/api/mcp_app.py). ## Problem @@ -65,7 +65,7 @@ b3_quote b3_cotahist b3_index (B3: 9 → 3) tesouro_bonds tesouro_siconfi (Tesouro: 6 → 2) ibge_indicator ibge_ipca_breakdown (IBGE: 4 → 2) ipea_series ipea_search (IPEA: 4 → 2) -anbima (ANBIMA: 3 → 1) +anbima (ANBIMA: ima|ettj|debentures|tpf) openfinance_directory (Open Finance: 15 → 1) basedosdados_search basedosdados_sql (BdD: 7 → 2) receita_arrecadacao aneel_leiloes susep_empresas @@ -85,6 +85,7 @@ findata_run_code (code mode, opt-in) | `b3_index` | index portfolio + monthly + list | `dataset`, omit `symbol` to list | | `tesouro_bonds` | bonds list/search/history | `dataset` | | `tesouro_siconfi` | `rreo`, `rgf`, `entes` | `report` | +| `anbima` | ima, ettj, debentures, tpf | `dataset=ima\|ettj\|debentures\|tpf` | | `openfinance_directory` | participants/endpoints/resources/roles | `dataset` | ## Tradeoffs diff --git a/docs/SOURCES_AND_ENDPOINTS.md b/docs/SOURCES_AND_ENDPOINTS.md index 1823d67..ad31b5c 100644 --- a/docs/SOURCES_AND_ENDPOINTS.md +++ b/docs/SOURCES_AND_ENDPOINTS.md @@ -20,7 +20,7 @@ Para testar interativamente, rode `findata serve` e abra `/api/docs` ou `/redoc` | Open Finance Brasil | Diretório público, participantes, recursos, JWKS e Portal de Dados | `/openfinance/resources`, `/openfinance/participants`, `/openfinance/endpoints`, `/openfinance/directory/api-resources`, `/openfinance/portal/datasets` | Não para dados públicos | | B3 | Cotações, COTAHIST oficial, composição teórica e evolução mensal de índices | `/b3/quote/{ticker}`, `/b3/history/{ticker}`, `/b3/quotes`, `/b3/cotahist/year/{year}`, `/b3/indices`, `/b3/indices/{symbol}`, `/b3/indices/{symbol}/monthly` | Não | | Yahoo Finance | Endpoint experimental de gráfico de preços | `/yahoo/chart/{symbol}` | Não; fonte não oficial | -| ANBIMA | IMA, ETTJ e debêntures via arquivos públicos | `/anbima/ima`, `/anbima/ettj`, `/anbima/debentures` | Não para os arquivos usados | +| ANBIMA | IMA, ETTJ, debêntures e TPF via arquivos públicos | `/anbima/ima`, `/anbima/ettj`, `/anbima/debentures`, `/anbima/tpf` | Não para os arquivos usados | | Receita Federal | Arrecadação por período, UF e tributo | `/receita/arrecadacao`, `/receita/tributos` | Não | | ANEEL | Leilões de geração e transmissão | `/aneel/leiloes/geracao`, `/aneel/leiloes/transmissao` | Não | | SUSEP | Entidades supervisionadas | `/susep/empresas`, `/susep/empresas/search` | Não | @@ -101,6 +101,8 @@ projeto. ### ANBIMA +No MCP, a tool `anbima` inclui o seletor `dataset=tpf` além de `ima`, `ettj` e `debentures`. + O módulo atual usa arquivos públicos em `www.anbima.com.br/informacoes/*` (XLS/CSV/TXT), não a API comercial autenticada Sensedia. Produtos autenticados futuros devem seguir o padrão de `docs/SOURCES_WITH_AUTH.md` e nunca embutir diff --git a/docs/snapshots/obm_day0_robots_sitemap.json b/docs/snapshots/obm_day0_robots_sitemap.json new file mode 100644 index 0000000..f8f1cd1 --- /dev/null +++ b/docs/snapshots/obm_day0_robots_sitemap.json @@ -0,0 +1,42 @@ +{ + "base_url": "https://obm.com.br", + "robots": { + "status": 200, + "disallow": [ + "/api/", + "/ops" + ], + "body_preview": "# As a condition of accessing this website, you agree to abide by the following\n# content signals:\n\n# (a) If a Content-Signal = yes, you may collect content for the corresponding\n# use.\n# (b) If a Content-Signal = no, you may not collect content for the\n# corresponding use.\n# (c) If the website operator does not include a Content-Signal for a\n# corresponding use, the website operator neither grants nor restricts\n# permission via Content-Signal with respect to the corresponding use.\n\n# The content signals and their meanings are:\n\n# search: building a search index and providing search results (e.g., returning\n# hyperlinks and short excerpts from your website's contents). Search does not\n# include providing AI-generated search summaries.\n# ai-input" + }, + "sitemap": { + "url": "https://obm.com.br/sitemap.xml", + "status": 200, + "url_count": 21, + "by_section": { + "/sitemap": 21 + }, + "sample": [ + "https://obm.com.br/sitemap/static.xml", + "https://obm.com.br/sitemap/funds.xml", + "https://obm.com.br/sitemap/equities.xml", + "https://obm.com.br/sitemap/bdrs.xml", + "https://obm.com.br/sitemap/etfs.xml", + "https://obm.com.br/sitemap/fiis.xml", + "https://obm.com.br/sitemap/treasuries.xml", + "https://obm.com.br/sitemap/debentures.xml", + "https://obm.com.br/sitemap/letras-financeiras.xml", + "https://obm.com.br/sitemap/indices.xml", + "https://obm.com.br/sitemap/crypto.xml", + "https://obm.com.br/sitemap/blog.xml", + "https://obm.com.br/sitemap/glossario.xml", + "https://obm.com.br/sitemap/fidc.xml", + "https://obm.com.br/sitemap/fip.xml", + "https://obm.com.br/sitemap/fiagro.xml", + "https://obm.com.br/sitemap/fi-infra.xml", + "https://obm.com.br/sitemap/cri.xml", + "https://obm.com.br/sitemap/cra.xml", + "https://obm.com.br/sitemap/empresas.xml", + "https://obm.com.br/sitemap/entidades.xml" + ] + } +} diff --git a/scripts/smoke_public_surface.py b/scripts/smoke_public_surface.py new file mode 100644 index 0000000..ac91311 --- /dev/null +++ b/scripts/smoke_public_surface.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +"""Smoke test for a live deployment or the offline ASGI meta surface.""" + +from __future__ import annotations + +import argparse +from collections.abc import Callable +from dataclasses import dataclass + +import httpx + +META_PATHS = ("/health", "/stats", "/meta", "/", "/docs", "/charts", "/openapi.json") +LIVE_PATHS = (*META_PATHS, "/bcb/series/name/selic?n=1", "/mcp") +_HTTP_OK_LT = 300 +_HTTP_SERVER_ERR_LT = 500 + + +@dataclass(frozen=True) +class Result: + path: str + status: int | None + color: str + detail: str + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--base-url", + default="http://127.0.0.1:8000", + help="Live deployment base URL (default: http://127.0.0.1:8000)", + ) + parser.add_argument( + "--asgi", + action="store_true", + help="Run only offline-safe meta routes against findata.api.app:app", + ) + return parser.parse_args() + + +def classify(path: str, status: int | None, error: str | None = None) -> Result: + if error is not None: + return Result(path, status, "vermelho", error) + assert status is not None + if status < _HTTP_OK_LT: + color = "verde" + elif status < _HTTP_SERVER_ERR_LT: + color = "amarelo" + else: + color = "vermelho" + return Result(path, status, color, "ok" if color == "verde" else "verificar") + + +def run_requests(paths: tuple[str, ...], get: Callable[[str], httpx.Response]) -> list[Result]: + results: list[Result] = [] + for path in paths: + try: + response = get(path) + except httpx.HTTPError as exc: + results.append(classify(path, None, str(exc))) + else: + results.append(classify(path, response.status_code)) + return results + + +def run_asgi() -> list[Result]: + from fastapi.testclient import TestClient + + from findata.api.app import app + + with TestClient(app) as client: + return run_requests(META_PATHS, client.get) + + +def run_live(base_url: str) -> list[Result]: + with httpx.Client(base_url=base_url.rstrip("/"), timeout=15, follow_redirects=False) as client: + return run_requests(LIVE_PATHS, client.get) + + +def print_table(results: list[Result]) -> None: + print("| rota | status | resultado | detalhe |") + print("|---|---:|---|---|") + for result in results: + status = result.status if result.status is not None else "erro" + detail = result.detail.replace("|", "\\|").replace("\n", " ") + print(f"| `{result.path}` | {status} | {result.color} | {detail} |") + + +def main() -> int: + args = parse_args() + results = run_asgi() if args.asgi else run_live(args.base_url) + print_table(results) + scoped = [result for result in results if result.path in META_PATHS] if args.asgi else results + return int(any(result.color == "vermelho" for result in scoped)) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/findata/api/mcp_app.py b/src/findata/api/mcp_app.py index abef0e4..b3d6813 100644 --- a/src/findata/api/mcp_app.py +++ b/src/findata/api/mcp_app.py @@ -659,25 +659,33 @@ async def ipea_search( "/anbima", operation_id="anbima", response_model=None, - summary="ANBIMA public data, IMA index family, ETTJ yield curve, or debenture quotes", + summary="ANBIMA public data: IMA, ETTJ, debentures, or TPF secondary-market quotes", ) async def anbima_tool( - dataset: Literal["ima", "ettj", "debentures"] = Query("ima"), + dataset: Literal["ima", "ettj", "debentures", "tpf"] = Query("ima"), family: str | None = Query( None, description="ima: filter to one IMA family, e.g. IRF-M, IMA-B" ), data: date | None = Query(None, description="Reference date (ettj/debentures; default latest)"), emissor: str | None = Query(None, description="debentures: issuer-name substring filter"), - limit: int = Query(500, ge=1, le=5000), + titulo: str | None = Query(None, description="tpf: bond-type filter, e.g. LTN, NTN-B, LFT"), + limit: int = Query(1000, ge=1, le=5000), ) -> Any: """``ima`` returns the latest IMA snapshot (optionally one ``family``); ``ettj`` returns the zero-coupon yield curve for ``data``; ``debentures`` returns daily - secondary-market quotes (optionally filtered by ``emissor``). + secondary-market quotes (optionally filtered by ``emissor``); ``tpf`` returns + federal-bond secondary-market quotes (optionally filtered by ``titulo``). """ if dataset == "ettj": return await anbima_src.get_ettj(data) if dataset == "debentures": return (await anbima_src.get_debentures(data, emissor=emissor))[:limit] + if dataset == "tpf": + rows = await anbima_src.get_tpf(data) + if titulo: + needle = titulo.casefold() + rows = [row for row in rows if needle in row.titulo.casefold()] + return rows[:limit] fam = anbima_src.IMAFamily(family) if family else None return (await anbima_src.get_ima(fam))[:limit] diff --git a/src/findata/web/templates/index.html b/src/findata/web/templates/index.html index 4e32e13..88bded3 100644 --- a/src/findata/web/templates/index.html +++ b/src/findata/web/templates/index.html @@ -39,6 +39,7 @@

reutilizável: API REST, biblioteca Python, CLI e servidor MCP para analistas, devs, pesquisadores e agentes.

+

Alpha self-hosted — use por sua conta; rate limit ativo em deploys públicos.

Explorar API @@ -93,7 +94,7 @@

{{ source_count }}fontes públicas
-
0chaves de API
+
semchave na maioria
4interfaces
v{{ version }}alpha auditável