0.5.0 stabilization & docs polish

CONTRIBUTING.md

@@ -33,9 +33,14 @@ It opens and closes a minimum-size BTCUSD position on the **demo** account.
 
 ## Project layout
 
-- `capital_cli/core/` — API client, session, risk engine, models, streaming.
-  All safety rules live here.
-- `capital_cli/cli/` — Typer commands. Thin: parse → call core → render.
+- `capital_cli/core/` — low-level primitives: API client, session, rate limiter,
+  risk engine, models, streaming, state. All safety rules live here (private).
+- `capital_cli/services/` — presentation-free domain services (markets, accounts,
+  watchlists, trading, streaming, confirmations) composing `core`.
+- `capital_cli/sdk/` — the experimental public facade (`CapitalComApp`,
+  `CapitalComConfig`, `RiskPolicy`).
+- `capital_cli/cli/` — Typer commands. Thin: parse → call a service → render; the
+  `cli/` layer never calls the broker API directly.
 - `tests/` — offline unit tests (mocked HTTP/WS); `tests/e2e/` — opt-in live suite.
 
 ## Conventions

README.md

@@ -41,9 +41,11 @@ See [practical use cases](docs/use-cases.md) for worked, copy-pasteable scenario
 tested broker engine is also embeddable as an **experimental** Python SDK — see
 [Use as a library](#use-as-a-library-experimental) and [docs/sdk.md](docs/sdk.md).
 The `capital_cli.core.*` internals remain private and may change between releases.
-The CLI itself is short-lived: each command runs as its own process and may create
-its own API session. `capctl session login` is mainly a connectivity/account check;
-session tokens are not persisted between separate invocations.
+The CLI itself is short-lived: each command runs as its own process. By default
+(`CAP_PERSIST_SESSION=true`) the short-lived session tokens are cached so
+back-to-back commands reuse one login instead of re-authenticating each time
+(which can trip Capital.com's login-rate limit and return HTTP 429); `capctl
+session login` is mainly a connectivity/account check.
 
 ## Features
 
@@ -54,6 +56,8 @@ session tokens are not persisted between separate invocations.
 - **Real-time streaming** — live price tables, price-level alerts, and portfolio snapshots over WebSocket
 - **Demo and live environments** — defaults to demo; live requires explicit opt-in
 - **Built-in rate limiting** — client-side token buckets respect Capital.com's 10 req/s global, 1 req/s session, and trading-burst limits
+- **Session-token caching** — back-to-back commands reuse one login instead of re-authenticating (avoids HTTP 429); on by default, opt out with `CAP_PERSIST_SESSION=false`
+- **Embeddable SDK (experimental)** — the same broker engine ships as an async Python library; see [Use as a library](#use-as-a-library-experimental)
 
 ## Installation
 
@@ -144,6 +148,7 @@ Main settings and their defaults:
 | `CAP_REQUIRE_EXPLICIT_CONFIRM` | `true` | Mutations need `--yes` |
 | `CAP_DRY_RUN` | `false` | Block all executions regardless of other flags |
 | `CAP_DEFAULT_ACCOUNT_ID` | (none) | Account selected after login |
+| `CAP_PERSIST_SESSION` | `true` | Cache short-lived session tokens in the state file so back-to-back commands reuse one login (avoids HTTP 429); set `false` to keep tokens in-process only |
 | `CAP_HTTP_TIMEOUT_S` | `15` | HTTP timeout |
 | `CAP_LOG_LEVEL` | `WARNING` | `DEBUG` … `CRITICAL` |
 | `CAP_WS_ENABLED` | `false` | Required for `capctl stream …` |
@@ -243,9 +248,9 @@ capctl account topup 1000 --yes                 # demo environment only
 Read-only:
 
 ```bash
-capctl trade positions
+capctl trade positions [--limit 10]              # -n/--limit caps rows shown
 capctl trade position DEAL_ID
-capctl trade orders
+capctl trade orders [--limit 10]                 # -n/--limit caps rows shown
 capctl trade confirm DEAL_REFERENCE [--wait --timeout 30]
 ```
 

docs/troubleshooting.md

@@ -58,7 +58,11 @@ confirmation message from the broker (shown in the output) says why.
 ### HTTP 429: error.too-many.requests on login
 Capital.com limits how often you can create sessions. Wait a couple of
 minutes and try again. Scripts that fire many capctl commands in a tight
-loop should pause ~1 second between them.
+loop should pause ~1 second between them. By default capctl caches the
+short-lived session token (`CAP_PERSIST_SESSION=true`) so back-to-back
+commands reuse one login instead of re-authenticating each time, which is the
+main mitigation for this error; keep it enabled (set `CAP_PERSIST_SESSION=false`
+only if you need in-process-only tokens).
 
 ### `WebSocket streaming is disabled`
 Set `CAP_WS_ENABLED=true` in `.env`. Streaming also requires valid login
@@ -69,10 +73,11 @@ Quiet markets tick rarely outside their trading hours. Try `BTCUSD`, which
 trades around the clock.
 
 ### Where is my state stored?
-Trade previews and the daily order counter persist in
-`~/.config/capital-cli/state.json` (override with `CAPCTL_STATE_FILE`) so
-they survive between commands. Deleting the file is safe (you'll just lose
-unexecuted previews).
+Trade previews, the daily order counter, and (by default) the cached
+short-lived session token persist in `~/.config/capital-cli/state.json`
+(mode `0600`, override with `CAPCTL_STATE_FILE`) so they survive between
+commands. Deleting the file is safe (you'll just lose unexecuted previews
+and have to log in again). Your API key/password are never written there.
 
 ## Still stuck?
 

tests/e2e/conftest.py

@@ -0,0 +1,28 @@
+"""E2E tests hit the real demo API. The root conftest's autouse `_credentials`
+fixture points config at a dummy env file (for offline isolation); for in-process
+e2e (the SDK facade tests) we must instead read the real repo-root .env so the
+real credentials and trading flags apply. This fixture runs after the root one
+(deeper conftest) and re-points CAP_ENV_FILE accordingly. The CLI e2e
+(test_demo_e2e.py) is unaffected — it passes CAP_ENV_FILE to its subprocesses
+explicitly."""
+
+from pathlib import Path
+
+import pytest
+
+REPO = Path(__file__).resolve().parents[2]
+ENV_FILE = REPO / ".env"
+
+
+@pytest.fixture(autouse=True)
+def _use_real_env(monkeypatch):
+    if ENV_FILE.exists():
+        monkeypatch.setenv("CAP_ENV_FILE", str(ENV_FILE))
+        from capital_cli.core.config import reset_config
+
+        reset_config()
+    yield
+    if ENV_FILE.exists():
+        from capital_cli.core.config import reset_config
+
+        reset_config()

tests/e2e/test_demo_e2e.py

@@ -87,6 +87,20 @@ def capctl(*args: str, expect_code: int = 0) -> dict:
     return json.loads(proc.stdout) if proc.stdout.strip() else {}
 
 
+def capctl_ndjson(*args: str, expect_code: int = 0) -> list[dict]:
+    """Run `capctl --json <args>` and parse NDJSON stdout (one object per line)."""
+    env = dict(os.environ, CAP_ENV_FILE=str(ENV_FILE), CAPCTL_STATE_FILE=str(STATE_FILE))
+    proc = subprocess.run(
+        [sys.executable, "-m", "capital_cli", "--json", *args],
+        capture_output=True, text=True, timeout=180, env=env, cwd=REPO,
+    )
+    assert proc.returncode == expect_code, (
+        f"capctl {' '.join(args)} -> exit {proc.returncode}\nstderr: {proc.stderr}\nstdout: {proc.stdout[:500]}"
+    )
+    time.sleep(1.2)
+    return [json.loads(line) for line in proc.stdout.splitlines() if line.strip()]
+
+
 def _market_tradeable() -> bool:
     """True if EPIC is currently tradeable on the demo venue.
 
@@ -265,10 +279,15 @@ def test_15_position_gone():
 
 
 def test_16_stream_prices():
-    """15-second live stream; BTCUSD ticks frequently enough to collect >=1."""
-    data = capctl("stream", "prices", EPIC, "--duration", "15", "--interval", "1")
-    assert data.get("ticks_received", 0) >= 1, data
-    assert data.get("ticks"), data
+    """Live NDJSON price stream; each line is a PriceTick object."""
+    if not _market_tradeable():
+        pytest.skip(f"{EPIC} not TRADEABLE right now")
+    ticks = capctl_ndjson("stream", "prices", EPIC, "--duration", "15", "--interval", "1")
+    if not ticks:
+        pytest.skip("no ticks in the streaming window (quiet market)")
+    assert all(t.get("epic") == EPIC for t in ticks), ticks[:2]
+    # each tick carries a bid/offer price
+    assert all(("bid" in t and "offer" in t) for t in ticks), ticks[:2]
 
 
 def test_17_history_activity():
@@ -304,13 +323,18 @@ def test_22_sentiment_batch():
 
 
 def test_23_stream_candles():
-    """20-second live OHLC stream; the forming BTCUSD candle updates frequently."""
-    data = capctl(
+    """Live NDJSON OHLC stream; each line is an OHLCBar object."""
+    if not _market_tradeable():
+        pytest.skip(f"{EPIC} not TRADEABLE right now")
+    bars = capctl_ndjson(
         "stream", "candles", EPIC, "--resolution", "MINUTE", "--duration", "20", "--interval", "0"
     )
-    assert data.get("bars_received", 0) >= 1, data
-    assert data.get("bars"), data
-    assert data["bars"][0]["resolution"] == "MINUTE"
+    if not bars:
+        pytest.skip("no candle updates in the streaming window (quiet market)")
+    assert all(b.get("resolution") == "MINUTE" for b in bars), bars[:2]
+    assert all(b.get("epic") == EPIC for b in bars), bars[:2]
+    # each bar carries OHLC fields
+    assert all(all(k in b for k in ("open", "high", "low", "close")) for b in bars), bars[:2]
 
 
 @requires_trading

tests/e2e/test_sdk_e2e.py

@@ -43,3 +43,125 @@ async def _run():
             assert "positions" in positions
 
     asyncio.run(_run())
+
+
+def test_sdk_risk_policy_reflects_config():
+    from capital_cli.sdk import CapitalComApp
+
+    async def _run():
+        async with CapitalComApp() as app:
+            rp = app.risk_policy
+            assert rp.allow_trading == app.config.cap_allow_trading
+            assert rp.max_position_size == app.config.cap_max_position_size
+            assert isinstance(rp.allowed_epics, list)
+
+    asyncio.run(_run())
+
+
+def test_sdk_markets_and_prices():
+    from capital_cli.sdk import CapitalComApp
+
+    async def _run():
+        async with CapitalComApp() as app:
+            search = await app.markets.search("gold", limit=3)
+            assert "markets" in search
+            prices = await app.markets.prices("GOLD", resolution="HOUR", max_candles=5)
+            assert prices.get("prices")
+
+    asyncio.run(_run())
+
+
+def test_sdk_trading_preview_only():
+    from capital_cli.core.models import Direction, PreviewPositionRequest
+    from capital_cli.sdk import CapitalComApp
+
+    async def _run():
+        async with CapitalComApp() as app:
+            req = PreviewPositionRequest(epic="GOLD", direction=Direction.BUY, size=0.1)
+            preview = await app.trading.preview_position(req)
+            assert preview.preview_id
+            assert isinstance(preview.all_checks_passed, bool)
+
+    asyncio.run(_run())
+
+
+def test_sdk_watchlist_lifecycle():
+    from capital_cli.sdk import CapitalComApp
+
+    async def _run():
+        async with CapitalComApp() as app:
+            created = await app.watchlists.create("capctl-sdk-e2e", confirm=True)
+            wid = str(created.get("watchlistId") or created.get("id"))
+            assert wid and wid != "None", created
+            try:
+                await app.watchlists.add_market(wid, "GOLD", confirm=True)
+                got = await app.watchlists.get(wid)
+                epics = [m.get("epic") for m in got.get("markets", [])]
+                assert "GOLD" in epics, got
+                await app.watchlists.remove_market(wid, "GOLD", confirm=True)
+            finally:
+                await app.watchlists.delete(wid, confirm=True)
+
+    asyncio.run(_run())
+
+
+def test_sdk_stream_prices_short():
+    # CAP_WS_ENABLED is read from the .env file by config, not exported to the
+    # process env — gate on the loaded config so the test runs when streaming is
+    # enabled in .env (matching how the CLI streaming e2e is driven).
+    from capital_cli.core.models import PriceTick
+    from capital_cli.sdk import CapitalComApp
+
+    if not CapitalComApp().config.cap_ws_enabled:
+        pytest.skip("streaming disabled (CAP_WS_ENABLED not set in the environment/.env)")
+
+    async def _run():
+        async with CapitalComApp() as app:
+            seen = 0
+            async for tick in app.stream.prices(["BTCUSD"], duration=5):
+                assert isinstance(tick, PriceTick)
+                seen += 1
+                if seen >= 1:
+                    break
+
+    asyncio.run(_run())
+
+
+requires_sdk_trading = pytest.mark.skipif(
+    os.environ.get("CAPCTL_E2E_TRADING") != "I_UNDERSTAND",
+    reason="set CAPCTL_E2E_TRADING=I_UNDERSTAND to run the SDK trade lifecycle",
+)
+
+
+@requires_sdk_trading
+def test_sdk_trade_lifecycle_leaves_account_flat():
+    from capital_cli.core.models import Direction, PreviewPositionRequest
+    from capital_cli.sdk import CapitalComApp
+
+    async def _run():
+        async with CapitalComApp() as app:
+            mkt = await app.markets.get("BTCUSD")
+            if mkt.get("snapshot", {}).get("marketStatus") != "TRADEABLE":
+                pytest.skip("BTCUSD not TRADEABLE right now")
+            req = PreviewPositionRequest(epic="BTCUSD", direction=Direction.BUY, size=0.001)
+            preview = await app.trading.preview_position(req)
+            assert preview.all_checks_passed, preview.checks
+            result = await app.trading.execute_position(
+                preview.preview_id, confirm=True, timeout_s=30
+            )
+            conf = result.get("confirmation") or {}
+            assert conf.get("status") == "ACCEPTED", conf
+            affected = conf.get("affectedDeals") or []
+            deal_id = (affected[0].get("dealId") if affected else None) or conf.get("dealId")
+            assert deal_id, conf
+            try:
+                close = await app.trading.close_position(deal_id, confirm=True, timeout_s=30)
+                assert (close.get("confirmation") or {}).get("status") == "ACCEPTED", close
+            finally:
+                positions = await app.trading.list_positions()
+                open_ids = [
+                    p.get("position", {}).get("dealId") for p in positions.get("positions", [])
+                ]
+                assert deal_id not in open_ids, "SDK trade lifecycle left a position open"
+
+    asyncio.run(_run())

tests/sdk/test_facade.py

@@ -117,3 +117,56 @@ async def _run():
     assert calls["login"] == 1
     assert calls["logout"] == 0   # __aexit__ must NOT logout (preserve cached session #10)
     assert calls["close"] == 1
+
+
+def test_sdk_watchlist_create_requires_confirm(monkeypatch):
+    # An SDK consumer (MCP server / dashboard) calling a mutation service method
+    # without confirm=True must hit the REAL mutation guard and get a confirm
+    # error BEFORE any HTTP — never a silent write. This exercises the real
+    # RiskEngine + real default config (cap_require_explicit_confirm defaults to
+    # True, cap_dry_run defaults to False), not an injected fake risk engine.
+    monkeypatch.setenv("CAP_ENV", "demo")
+    monkeypatch.setenv("CAP_API_KEY", "k")
+    monkeypatch.setenv("CAP_IDENTIFIER", "id@example.com")
+    monkeypatch.setenv("CAP_API_PASSWORD", "pw")
+    from capital_cli.core.config import reset_config
+    from capital_cli.core.risk import reset_risk_engine
+
+    reset_config()
+    reset_risk_engine()
+
+    import asyncio
+
+    from capital_cli.core.errors import ConfirmRequiredError
+    from capital_cli.services.watchlists import WatchlistService
+
+    calls = []
+
+    class _FakeClient:
+        async def post(self, *a, **k):
+            calls.append(("post", a, k))
+
+    class _FakeSession:
+        async def ensure_logged_in(self):
+            return None
+
+    monkeypatch.setattr("capital_cli.services.watchlists.get_client", lambda: _FakeClient())
+    monkeypatch.setattr(
+        "capital_cli.services.watchlists.get_session_manager", lambda: _FakeSession()
+    )
+
+    async def _run():
+        try:
+            await WatchlistService().create("x", confirm=False)
+            raise AssertionError("expected a confirm-required error")
+        except ConfirmRequiredError:
+            pass
+        assert calls == []  # no HTTP attempted
+
+    try:
+        asyncio.run(_run())
+    finally:
+        # The real RiskEngine binds to this test's config; reset so later tests
+        # rebuild from their own env (test isolation).
+        reset_risk_engine()
+        reset_config()