Update docs for relationship indexing and activity tracking

CHANGELOG.md

@@ -4,6 +4,17 @@ All notable changes to this project will be documented in this file.
 
 This project follows [Keep a Changelog](https://keepachangelog.com/).
 
+## [0.2.0] - 2026-03-26
+
+### Added
+- Schema evolution: hydrate-on-read migration, `validate_schema_change()` guard, `add_field` MCP tool, required-without-default warning
+- Relationship indexing: write-time reverse index at `_index/relations.json`, auto-rebuild from entity files, atomic writes
+- Graph traversal methods on UpjackApp: `query_by_relationship`, `get_related`, `get_composite`
+- Activity tracking: opt-in via `"activities": true` in manifest, `log_activity` and `get_activities` methods
+- Per-entity MCP tools: `query_{plural}_by_relationship`, `get_related_{name}`, `get_{name}_composite`
+- Global MCP tools: `rebuild_index`, `log_activity`, `get_activities` (when activities enabled)
+- CRUD hooks: `on_relationships_changed` callback for automatic index maintenance
+
 ## [0.1.0] - 2026-02-24
 
 ### Added

CLAUDE.md

@@ -17,6 +17,45 @@ website/          Documentation site (Astro/Starlight) — upjack.dev
 workspace/        Runtime entity data (gitignored, created at runtime)
 ```
 
+## Key Modules (Python)
+
+| Module | Purpose |
+|--------|---------|
+| `upjack.relations` | Write-time reverse relationship index — `query_reverse()`, `rebuild_index()` |
+| `upjack.activity` | Activity entity tracking — `log_activity()`, `get_activities()` |
+
+## Graph Traversal (UpjackApp)
+
+Methods for navigating entity relationships:
+
+- `query_by_relationship(entity_type, rel, target_id, filter?, limit?)` — reverse index lookup (e.g., find all deals linked to a contact)
+- `get_related(entity_id, rel?, direction?)` — follow edges forward or reverse, returns resolved entities
+- `get_composite(entity_type, entity_id, depth?)` — load entity + all related in one call; `_related` key contains forward (`rel`) and reverse (`~rel`) edges
+
+CRUD hooks: `on_relationships_changed` callback fires on `create_entity`, `update_entity`, `delete_entity` — UpjackApp auto-wires this to maintain the reverse index.
+
+## Activity Tracking
+
+Opt-in via `"activities": true` in manifest `_meta["ai.nimblebrain/upjack"]` extension.
+
+- `log_activity(subject_id, action, detail?)` — creates an activity entity with a subject relationship
+- `get_activities(subject_id, action?, limit?)` — reverse index query for a subject's activities
+- Activity schema: `action` (string, required), `detail` (object, optional)
+
+## MCP Tools (auto-registered)
+
+Per entity type (in addition to existing CRUD tools):
+
+- `query_{plural}_by_relationship(rel, target_id, filter?, limit?)`
+- `get_related_{name}(entity_id, rel?, direction?)`
+- `get_{name}_composite(entity_id, depth?)`
+
+Global tools:
+
+- `rebuild_index()` — force rebuild reverse index from entity files
+- `log_activity(subject_id, action, detail?)` — when activities enabled
+- `get_activities(subject_id, action?, limit?)` — when activities enabled
+
 ## Verification
 
 **Always run before considering work done:**
@@ -91,6 +130,8 @@ git push origin main --tags
 - **Storage**: JSON files at `{namespace}/data/{plural}/{id}.json`
 - **FastMCP is optional** — Python core works without it; install `upjack[mcp]` for server support
 - **@modelcontextprotocol/sdk is optional** — TypeScript core works without it; import `upjack/server` for server support
+- **File-based reverse index for relationships** — write-time updated at `{namespace}/data/_index/relations.json`, auto-rebuilt from entity files if missing or corrupt
+- **Activities are entities, not a separate mechanism** — opt-in via `"activities": true` in manifest, stored as regular entities (prefix `act`, plural `activities`)
 
 ## Tooling
 

examples/crm/context.md

@@ -41,6 +41,15 @@ Activities track interactions. Common types:
 
 Always create an activity when you interact with a contact or progress a deal.
 
+## Querying Relationships
+
+Use relationship tools instead of listing all entities and filtering manually.
+
+- **Find entities by relationship**: `query_deals_by_relationship(rel="primary_contact", target_id="ct_...")` returns all deals for a contact. Works for any entity type — pass the relationship name and target ID.
+- **Follow edges**: `get_related_contact(entity_id="ct_...", direction="forward")` returns entities this contact points to. Use `direction="reverse"` to find entities that point to this contact.
+- **Load full context in one call**: `get_contact_composite(entity_id="ct_...")` returns the contact plus all related entities nested under `_related`. Forward relationships keyed by name (`works_at`), reverse keyed with tilde (`~primary_contact`). Use this before summarizing an entity.
+- **Stale results?** Run `rebuild_index()` to force-rebuild the relationship index from entity files.
+
 ## Follow-up Rules
 
 - New leads should be contacted within 24 hours

examples/crm/validate.py

@@ -78,6 +78,46 @@ def main() -> int:
     assert len(deal["relationships"]) == 2
     print(f"  Created: {deal['id']}")
 
+    # Validate graph traversal — get_related forward
+    print("\nValidating graph traversal...")
+    print("  get_related (forward)...")
+    related_fwd = app.get_related(deal["id"], direction="forward")
+    related_fwd_ids = {r["id"] for r in related_fwd}
+    if contact["id"] not in related_fwd_ids:
+        errors.append(f"get_related forward missing contact {contact['id']}")
+    if company["id"] not in related_fwd_ids:
+        errors.append(f"get_related forward missing company {company['id']}")
+    print(f"    Found {len(related_fwd)} related entities")
+
+    # Validate graph traversal — query_by_relationship
+    print("  query_by_relationship (deal->company)...")
+    deals_for_company = app.query_by_relationship("deal", "company", company["id"])
+    deals_for_company_ids = {d["id"] for d in deals_for_company}
+    if deal["id"] not in deals_for_company_ids:
+        errors.append(f"query_by_relationship did not find deal {deal['id']} for company {company['id']}")
+    print(f"    Found {len(deals_for_company)} deal(s) for company")
+
+    # Validate graph traversal — get_composite
+    print("  get_composite (deal)...")
+    composite = app.get_composite("deal", deal["id"])
+    composite_related = composite.get("_related", {})
+    composite_contact_ids = {r["id"] for r in composite_related.get("primary_contact", [])}
+    composite_company_ids = {r["id"] for r in composite_related.get("company", [])}
+    if contact["id"] not in composite_contact_ids:
+        errors.append(f"get_composite _related missing contact under primary_contact")
+    if company["id"] not in composite_company_ids:
+        errors.append(f"get_composite _related missing company under company")
+    print(f"    _related keys: {sorted(composite_related.keys())}")
+
+    # Validate rebuild_index
+    print("  rebuild_index...")
+    from upjack.relations import rebuild_index
+    index = rebuild_index(app.root, app.namespace, app._entity_defs_list())
+    index_entry_count = sum(len(v) for v in index.values())
+    if index_entry_count < 2:
+        errors.append(f"rebuild_index returned only {index_entry_count} entries, expected at least 2")
+    print(f"    Index entries: {index_entry_count}")
+
     # Create a pipeline (singleton)
     print("Creating pipeline...")
     pipeline_data = json.loads((EXAMPLE_DIR / "seed" / "default-pipeline.json").read_text())

examples/todo/context.md

@@ -26,6 +26,17 @@ Use the Eisenhower approach:
 - **Low**: Nice to have — do if time permits
 - **None**: Unclassified — needs triage
 
+## Querying by Relationship
+
+Use relationship tools instead of listing and filtering manually:
+
+- **Find all tasks in a project**: `query_tasks_by_relationship(rel="belongs_to", target_id="<project_id>")` — returns tasks linked to that project via the reverse index
+- **Find all tasks with a label**: `query_tasks_by_relationship(rel="labeled", target_id="<label_id>")`
+- **Load a task with its project and labels**: `get_task_composite(entity_id="<task_id>")` — returns the task plus all related entities in one call (forward edges under rel name, reverse under `~rel`)
+- **Follow one relationship**: `get_related_task(entity_id="<task_id>", rel="belongs_to")` — resolves the linked project directly
+
+Prefer `query_tasks_by_relationship` over `list_tasks` + client-side filtering. It uses the reverse index and supports `filter` and `limit` parameters.
+
 ## Rules
 
 - Keep task titles short and actionable (start with a verb)

website/src/content/docs/concepts/architecture.md

@@ -22,7 +22,7 @@ Every entity conforms to a JSON Schema (draft 2020-12). Schemas are composed via
 
 ### Git-Friendly
 
-All entity data lives as JSON files, one file per entity in a structured directory layout. The layout is optimized for git-backed workflows: the hosting platform (such as NimbleBrain) commits each write, providing a complete audit trail, branch-based workflows, and natural conflict resolution. The `upjack` library handles file I/O; git integration is a platform-level concern.
+All entity data lives as JSON files, one file per entity in a structured directory layout. The library also maintains a reverse index at `{namespace}/data/_index/relations.json`, updated at write time to support graph traversal queries. This index is a runtime artifact — it is rebuilt automatically from entity files if missing or corrupt. The layout is optimized for git-backed workflows: the hosting platform (such as NimbleBrain) commits each write, providing a complete audit trail, branch-based workflows, and natural conflict resolution. The `upjack` library handles file I/O; git integration is a platform-level concern.
 
 ### Skills Over Code
 
@@ -78,13 +78,15 @@ manifest.json
 [upjack library]
     |-- reads manifest and entity definitions
     |-- loads and composes JSON Schemas (base + app via allOf)
-    |-- provides UpjackApp with entity CRUD operations
+    |-- provides UpjackApp with entity CRUD, graph traversal, and activity tracking
     |
     v
 [MCP Server (FastMCP / MCP SDK)]
     create_server(manifest) / createServer(manifest) generates:
         create_{entity}, get_{entity}, update_{entity},
-        list_{plural}, search_{plural}, delete_{entity}
+        list_{plural}, search_{plural}, delete_{entity},
+        query_{plural}_by_relationship, get_related_{entity},
+        get_{entity}_composite, log_activity, get_activities
     + context and skill resources
     |
     v
@@ -105,14 +107,17 @@ The `upjack` library provides the core entity engine. Manifest fields like hooks
 | Schema validation (JSON Schema 2020-12) | Yes | Yes |
 | Full-text search and structured filters | Yes | Yes |
 | MCP server generation (`create_server()`) | Yes | Yes |
+| Relationship indexing (write-time reverse index) | Yes | Yes |
+| Graph traversal (`query_by_relationship`, `get_related`, `get_composite`) | Yes | Yes |
+| Activity tracking (opt-in entity-based event recording) | Yes | Yes |
 | Git commits on entity writes | No (file I/O only) | Yes |
 | Hook dispatch (react to entity events) | No (manifest metadata) | Yes |
 | Schedule execution (cron triggers) | No (manifest metadata) | Yes |
 | View materialization (named queries) | No (manifest metadata) | Yes |
 | Bundle dependency resolution | No (manifest metadata) | Yes |
 | App lifecycle (install, update, uninstall) | No | Yes |
 
-When using `upjack` standalone, hooks, schedules, views, and bundle declarations are stored in the manifest and available for inspection, but they have no effect unless a runtime interprets them.
+Relationship indexing, graph traversal, and activity tracking are handled entirely by the library and work in both standalone and hosted modes. Hooks, schedules, views, and bundle declarations are stored in the manifest and available for inspection, but they have no effect unless a runtime interprets them.
 
 ## Relationship to Existing Systems
 

website/src/content/docs/libraries/python.md

@@ -56,6 +56,96 @@ mcp.run()
 
 This generates CRUD tools for every entity type in the manifest, exposes context and skills as MCP resources, and handles validation automatically. See the [CRM example](https://github.com/NimbleBrainInc/upjack/tree/main/examples/crm/server.py) for a complete example.
 
+## Relationship Queries
+
+Query the relationship graph to traverse connections between entities. These methods use a reverse index that Upjack maintains automatically at write time.
+
+```python
+from upjack import UpjackApp
+
+app = UpjackApp.from_manifest("manifest.json")
+
+# Find all contacts at a company (reverse lookup)
+contacts = app.query_by_relationship("contact", "company", "co_01JQ3KM...")
+
+# With filtering and limit
+senior = app.query_by_relationship(
+    "contact", "company", "co_01JQ3KM...",
+    filter={"role": "VP"},
+    limit=10,
+)
+
+# Follow edges forward: get entities a contact links to
+related = app.get_related("ct_01JQ3KM...", direction="forward")
+# → [{"id": "co_01JQ3KM...", "name": "Acme Corp", ...}, ...]
+
+# Follow edges in reverse: get entities that link to a company
+referrers = app.get_related("co_01JQ3KM...", direction="reverse")
+# → [{"id": "ct_01JQ3KM...", "first_name": "Sarah", ...}, ...]
+
+# Filter by relationship name
+deals = app.get_related("co_01JQ3KM...", rel="company", direction="reverse")
+
+# Load an entity with all related entities in one call
+composite = app.get_composite("contact", "ct_01JQ3KM...", depth=1)
+# {
+#   "id": "ct_01JQ3KM...",
+#   "first_name": "Sarah",
+#   "company": "co_01JQ3KM...",
+#   "_related": {
+#     "company": [{"id": "co_01JQ3KM...", "name": "Acme Corp", ...}],
+#     "~company": [{"id": "dl_01JQ3KM...", "title": "Enterprise Plan", ...}]
+#   }
+# }
+```
+
+In `_related`, forward relationships use the relationship name (e.g. `"company"`) and reverse relationships use a `~` prefix (e.g. `"~company"` for deals that reference this contact via their `company` field).
+
+## Activity Tracking
+
+Log timestamped activities against any entity. Requires `"activities": true` in the manifest's upjack extension:
+
+```json
+{
+  "_meta": {
+    "ai.nimblebrain/upjack": {
+      "activities": true
+    }
+  }
+}
+```
+
+Once enabled, Upjack registers a built-in `activity` entity type (prefix `act`) and exposes two methods:
+
+```python
+from upjack import UpjackApp
+
+app = UpjackApp.from_manifest("manifest.json")
+
+# Log an activity against a contact
+app.log_activity("ct_01JQ3KM...", "email_sent", detail={
+    "subject": "Follow-up on proposal",
+    "channel": "email",
+})
+
+# Log a deal stage change
+app.log_activity("dl_01JQ3KM...", "stage_changed", detail={
+    "from": "qualification",
+    "to": "proposal",
+})
+
+# Get all activities for an entity
+activities = app.get_activities("ct_01JQ3KM...")
+
+# Filter by action
+emails = app.get_activities("ct_01JQ3KM...", action="email_sent")
+
+# Limit results
+recent = app.get_activities("ct_01JQ3KM...", limit=5)
+```
+
+Activities are stored as regular entities and linked to their subject via the relationship index, so they also appear in `get_composite` results.
+
 ## Requirements
 
 - Python >= 3.13