diff --git a/docs/DOCS_GUIDELINE.md b/docs/DOCS_GUIDELINE.md new file mode 100644 index 00000000..fb72d653 --- /dev/null +++ b/docs/DOCS_GUIDELINE.md @@ -0,0 +1,209 @@ +# Docusaurus Setup & Deployment Guide + +This guide provides a standardized approach to setting up, configuring, and deploying Docusaurus documentation sites to GitHub Pages, based on best practices and lessons learned from the OpenGIN project. + +## 1. Prerequisites & Environment +* **Node.js Version**: Docusaurus 3.x requires Node.js **>= 18.0**. We recommend enforcing **Node.js 20 (LTS)** or higher in the project to avoid version mismatches. +* **Package Manager**: `npm` is standard, key dependencies must be locked. + +### Recommended Configuration +Add or update the `engines` field in your `package.json` to enforce the Node version: + +```json +"engines": { + "node": ">=20.0" +} +``` + +Create a `.nvmrc` file in the root of your docs directory: +```text +20 +``` + +## 2. Configuration (`docusaurus.config.js`) + +### Essential GitHub Pages Settings +Configuring `url` and `baseUrl` correctly is critical for assets to load on GitHub Pages. + +```javascript +// docusaurus.config.js +const config = { + // ... + url: 'https://.github.io', // Your GitHub Pages domain + baseUrl: '//', // The name of your repository with slashes + + // GitHub Deployment Config + organizationName: '', + projectName: '', + + // Handling Broken Links + onBrokenLinks: 'throw', // Recommended: Break build on broken links + onBrokenMarkdownLinks: 'warn', + + // ... +}; +``` + +### Dynamic Base URL (Optional but Recommended) +For PR previews or local testing where the path might differ, you can make `baseUrl` dynamic: + +```javascript +baseUrl: process.env.BASE_URL || '//', +``` + +## 3. GitHub Actions Workflows + +We use two primary workflows: one for **production deployment** (on push to main) and one for **PR previews**. + +### A. Production Deployment (`.github/workflows/deploy-docs.yml`) + +
+Click to see template + +```yaml +name: Deploy to GitHub Pages + +on: + push: + branches: + - main + paths: + - 'docs/**' # specific directory trigger + workflow_dispatch: + +permissions: + contents: write # REQUIRED for pushing to gh-pages branch + +jobs: + deploy: + runs-on: ubuntu-latest + defaults: + run: + working-directory: docs # Set if docs are in a subdir + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-node@v4 + with: + node-version: 20 + cache: npm + cache-dependency-path: docs/package-lock.json + + - name: Install dependencies + run: npm ci + + - name: Build website + run: npm run build + + - name: Deploy to GitHub Pages + uses: peaceiris/actions-gh-pages@v3 + with: + github_token: ${{ secrets.GITHUB_TOKEN }} + publish_dir: ./docs/build +``` +
+ +### B. PR Preview (`.github/workflows/preview-docs.yml`) + +
+Click to see template + +```yaml +name: Deploy PR Preview + +on: + pull_request_target: + types: [opened, reopened, synchronize, closed] + paths: ['docs/**'] + +permissions: + contents: write + pull-requests: write + issues: write + +jobs: + deploy-preview: + runs-on: ubuntu-latest + defaults: + run: + working-directory: docs + steps: + - uses: actions/checkout@v4 + with: + ref: ${{ github.event.pull_request.head.sha }} # Checkout PR code + + - uses: actions/setup-node@v4 + with: + node-version: 20 + cache: npm + cache-dependency-path: docs/package-lock.json + + - name: Install dependencies + run: npm ci + + - name: Deploy preview + uses: rossjrw/pr-preview-action@v1 + with: + source_dir: docs/build + preview_branch: gh-pages + umbrella_dir: pr-preview + action: auto + build_script: npm run build + env: + # Crucial for assets to load in the preview sub-path + BASE_URL: //pr-preview/${{ github.event.number }}/ +``` +
+ +## 4. Common Pitfalls & Solutions + +### "Remote Rejected ... Repository Rule Violations" +**Cause:** The `gh-pages` branch (used for deployment) has "Branch Protection Rules" enabled that prevent the GitHub Actions bot from pushing. +**Fix:** +1. Go to Repo Settings -> Rules -> Rulesets (or Branches). +2. Add a bypass for the `github-actions` app on the `gh-pages` branch. +3. Alternatively, disable "Require pull request before merging" for `gh-pages`. + +### "Exit Code 1" during Build (Node Version) +**Cause:** Docusaurus 3.x failing on Node 16 or 18. +**Fix:** Ensure `actions/setup-node` uses `node-version: 20` and `package.json` engines are set. + +## 5. Master Prompt for AI Agents +Use the following prompt to instruct an AI assistant to set this up for a new project. + +--- + +**Copy & Paste this Prompt:** + +> I need to set up Docusaurus documentation for this project with automatic deployment to GitHub Pages. Please follow these specific guidelines based on our organization's standards: +> +> 1. **Project Structure**: +> * Initialize a new Docusaurus project in a `docs/` subdirectory if one doesn't exist. +> * Use `npm` for package management. +> * Ensure `package.json` enforces `engines: { "node": ">=20.0" }`. +> * Create an `.nvmrc` file with `20`. +> +> 2. **Configuration (`docusaurus.config.js`)**: +> * Set `url` to `https://.github.io`. +> * Set `baseUrl` to `//` but allow it to be overridden by `process.env.BASE_URL` (for PR previews). +> * Set `onBrokenLinks` to `'throw'` and `onBrokenMarkdownLinks` to `'warn'`. +> * Ensure `organizationName` and `projectName` are set correctly. +> +> 3. **CI/CD Workflows (GitHub Actions)**: +> * Create `.github/workflows/deploy-docs.yml`: +> * Trigger on `push` to `main` (filtered to `docs/` path). +> * Use `permissions: contents: write`. +> * Use `actions/setup-node@v4` with version `20`. +> * Use `peaceiris/actions-gh-pages@v3` for deployment. +> * Create `.github/workflows/preview-docs.yml` (Optional): +> * Trigger on `pull_request` to `main`. +> * Use `rossjrw/pr-preview-action@v1`. +> * Pass the correct `BASE_URL` environment variable to the build script to ensure assets load in the preview sub-path. +> +> 4. **Verification**: +> * Remind me to check "Branch Protection Rules" for the `gh-pages` branch if the push fails. +> * Verify that the build command (`npm run build`) succeeds locally with Node 20. +> +> Please analyze the current repository structure and generate the necessary files and changes. + +--- diff --git a/docs_deprecated/appendix/data-type-detection-patterns.md b/docs_deprecated/appendix/data-type-detection-patterns.md deleted file mode 100644 index 128b867d..00000000 --- a/docs_deprecated/appendix/data-type-detection-patterns.md +++ /dev/null @@ -1,72 +0,0 @@ -# Core Storage Type Detection Patterns - -This document describes the JSON patterns used by the OpenGIN system to automatically detect and classify the three core storage types when attributes are fed into the system. - -## Overview - -The system uses a hierarchical detection approach with the following precedence order for the three core storage types: -1. **Tabular Data** (highest priority) -2. **Graph Data** -3. **Document/Map Data** (lowest priority) - -## Detection Patterns - -### 1. Tabular Data Pattern - -**Detection Criteria**: Structure contains both `columns` and `rows` fields where: -- `columns` is an array of strings representing column names -- `rows` is an array of arrays representing data rows - -**Example**: -```json -{ - "columns": ["id", "name", "department", "salary"], - "rows": [ - [1, "John Doe", "Engineering", 75000], - [2, "Jane Smith", "Marketing", 65000] - ] -} -``` - -### 2. Graph Data Pattern - -**Detection Criteria**: Structure contains both `nodes` and `edges` fields where: -- `nodes` is an array of node objects with properties -- `edges` is an array of edge objects with source and target references - -**Example**: -```json -{ - "nodes": [ - {"id": "user1", "type": "user", "properties": {"name": "Alice", "age": 30}}, - {"id": "user2", "type": "user", "properties": {"name": "Bob", "age": 25}}, - {"id": "post1", "type": "post", "properties": {"title": "Hello", "content": "World"}} - ], - "edges": [ - {"source": "user1", "target": "user2", "type": "follows", "properties": {"since": "2024-01-01"}}, - {"source": "user1", "target": "post1", "type": "created", "properties": {"timestamp": "2024-03-20T10:00:00Z"}} - ] -} -``` - -### 3. Document/Map Data Pattern - -**Detection Criteria**: Object with key-value pairs that doesn't match tabular or graph patterns. - -**Example**: -```json -{ - "user": { - "name": "John", - "age": 30, - "address": { - "city": "New York", - "zip": "10001" - } - }, - "settings": { - "theme": "dark", - "notifications": true - } -} -``` diff --git a/docs_deprecated/appendix/datatype.md b/docs_deprecated/appendix/datatype.md deleted file mode 100644 index 922deabd..00000000 --- a/docs_deprecated/appendix/datatype.md +++ /dev/null @@ -1,159 +0,0 @@ -# Type Inference System - -The type inference system is a core component of the LDF Architecture, responsible for automatically determining the appropriate data types for values in the data structure. This document explains how the system works, what types are supported, and how type inference rules are applied. - -## Supported Data Types - -### Primitive Types - -1. **Integer (`int`)** - - Whole numbers without decimal points - - Examples: `42`, `-1`, `0` - - Used for counting, indexing, and whole number quantities - -2. **Float (`float`)** - - Numbers with decimal points or scientific notation - - Examples: `3.14`, `-0.001`, `1.0e-10` - - Used for measurements, percentages, and precise calculations - -3. **String (`string`)** - - Text data of any length - - Examples: `"hello"`, `"user123"`, `""` - - Used for names, descriptions, and general text - -4. **Boolean (`bool`)** - - True/false values - - Examples: `true`, `false` - - Used for flags, conditions, and binary states - -5. **Null (`null`)** - - Represents absence of a value - - Example: `null` - - Used when a value is not provided or not applicable - -### Special Types - -1. **Date (`date`)** - - Calendar dates without time information - - Supported formats: - - `YYYY-MM-DD` (e.g., "2024-03-20") - - `DD/MM/YYYY` (e.g., "20/03/2024") - - `MM/DD/YYYY` (e.g., "03/20/2024") - - `YYYY.MM.DD` (e.g., "2024.03.20") - - `DD-MM-YYYY` (e.g., "20-03-2024") - - `MM-DD-YYYY` (e.g., "03-20-2024") - - `YYYY/MM/DD` (e.g., "2024/03/20") - -2. **Time (`time`)** - - Time of day without date information - - Supported formats: - - `HH:MM:SS` (e.g., "14:30:00") - - `HH:MM` (e.g., "14:30") - - `h:MM AM/PM` (e.g., "2:30 PM") - - `HH:MM:SS.mmm` (e.g., "14:30:00.000") - - `HH:MM:SS±HH:MM` (e.g., "14:30:00-07:00") - - `HH:MM:SSZ` (e.g., "14:30:00Z") - -3. **DateTime (`datetime`)** - - Combined date and time information - - Supported formats: - - RFC3339 (e.g., "2024-03-20T14:30:00Z07:00") - - `YYYY-MM-DD HH:MM:SS` (e.g., "2024-03-20 14:30:00") - - `YYYY-MM-DDTHH:MM:SS` (e.g., "2024-03-20T14:30:00") - - `DD/MM/YYYY HH:MM:SS` (e.g., "20/03/2024 14:30:00") - - `MM/DD/YYYY HH:MM:SS` (e.g., "03/20/2024 14:30:00") - - `YYYY.MM.DD HH:MM:SS` (e.g., "2024.03.20 14:30:00") - - `YYYY-MM-DD HH:MM:SS.mmm` (e.g., "2024-03-20 14:30:00.000") - -## Type Inference Rules - -The system follows these rules when inferring types: - -1. **Number Type Resolution** - - If a number has a decimal point or is in scientific notation, it's inferred as `float` - - If a number is a whole number (no decimal), it's inferred as `int` - - Special case: zero (0) is checked against its string representation to determine if it was originally a float - -2. **String Type Resolution** - - All text values are first checked against date/time patterns - - If the text matches a date pattern, it's inferred as `date` - - If the text matches a time pattern, it's inferred as `time` - - If the text matches a datetime pattern, it's inferred as `datetime` - - Otherwise, it's inferred as `string` - -3. **Null Handling** - - Explicit null values are inferred as `null` type - - The `null` type is always nullable - - Missing fields are treated as `null` - -4. **Array Type Resolution** - - Arrays are marked with `IsArray: true` - - The element type is determined from the first non-null element - - Empty arrays default to `string` element type - - Array type information includes: - ```json - { - "type": "string", - "is_array": true, - "array_type": { - "type": "int" - } - } - ``` - -## Examples - -### Basic Types -```json -{ - "integer_value": 42, - "float_value": 3.14, - "string_value": "hello", - "boolean_value": true, - "null_value": null -} -``` - -### Special Types -```json -{ - "date_value": "2024-03-20", - "time_value": "14:30:00", - "datetime_value": "2024-03-20T14:30:00Z" -} -``` - -### Array Types -```json -{ - "int_array": [1, 2, 3], - "mixed_array": ["a", 1, true], - "empty_array": [] -} -``` - -## Type Information Structure - -The type information is structured to define the fundamental data type, including flags for nullability and array structures. For nested types, it recursively specifies element types for arrays and property maps for objects, enabling comprehensive type definitions for complex data structures. - -## Best Practices - -1. **Date and Time Formatting** - - Use ISO 8601 / RFC 3339 formats when possible - - Include timezone information for datetime values - - Be consistent with format choice within your application - -2. **Number Handling** - - Use integers for counting and indexing - - Use floats for measurements and calculations - - Be explicit about decimal points when floating-point precision is required - -3. **Null Values** - - Use explicit null values rather than empty strings or zero values - - Document which fields are nullable in your schema - - Consider using nullable types in strongly-typed languages - -4. **Array Types** - - Keep array elements consistent in type - - Provide type information for empty arrays - - Consider using single-type arrays for better type safety diff --git a/docs_deprecated/appendix/limitations.md b/docs_deprecated/appendix/limitations.md deleted file mode 100644 index 86793ee8..00000000 --- a/docs_deprecated/appendix/limitations.md +++ /dev/null @@ -1,12 +0,0 @@ -# Limitations - -## Read API - -1. Read for data like tables or documents (metadata, unstructured documents) doesn't include filters for querying parameters inside tables. -2. Join, aggregations and advanced data processing queries are not yet supported. -3. Sub-graph insertion as an attribute is not yet supported. -4. Scalar value insertion as an attribute is not yet supported. - -## OpenAPI Contract and Ballerina Service generation - -1. Time is being infered as `string` field. Must implement a time data type in the protobuf specification. diff --git a/docs_deprecated/appendix/operations/backup_integration.md b/docs_deprecated/appendix/operations/backup_integration.md deleted file mode 100644 index cdbc89d1..00000000 --- a/docs_deprecated/appendix/operations/backup_integration.md +++ /dev/null @@ -1,328 +0,0 @@ -# Backup Integration Guide - -This guide covers both local backup management and GitHub-based backup restoration using our `init.sh` script. There are two main workflows depending on your needs. - -## Two Main Workflows - -### 1. **Local Backup Workflow** -- Create backups locally using `init.sh` -- Store backups in local directories -- Restore from local backup files -- **Use case**: Development, testing, local data management - -### 2. **GitHub Release Workflow** -- Download and restore from [LDFLK/data-backups](https://github.com/LDFLK/data-backups) repository -- Use pre-built backup releases -- **Use case**: Production deployments, team collaboration, version management - ---- - -## Local Backup Workflow - -### Create Local Backups - -```bash -# Create individual database backups -./deployment/development/init.sh backup_mongodb -./deployment/development/init.sh backup_postgres -./deployment/development/init.sh backup_neo4j - -# List available local backups -./deployment/development/init.sh list_mongodb_backups -./deployment/development/init.sh list_postgres_backups -./deployment/development/init.sh list_neo4j_backups -``` - -### Restore from Local Backups - -```bash -# Restore from local backup directories -./deployment/development/init.sh restore_mongodb -./deployment/development/init.sh restore_postgres -./deployment/development/init.sh restore_neo4j -``` - -### Local Backup Structure - -``` -backups/ -├── mongodb/ -│ └── opengin.tar.gz -├── postgres/ -│ └── opengin.tar.gz -└── neo4j/ - └── neo4j.dump -``` - -### Environment Configuration - -Set backup directories in `configs/backup.env`: - -```bash -# MongoDB backup directory -MONGODB_BACKUP_DIR=/path/to/mongodb/backups - -# PostgreSQL backup directory -POSTGRES_BACKUP_DIR=/path/to/postgres/backups - -# Neo4j backup directory -NEO4J_BACKUP_DIR=/path/to/neo4j/backups -``` - ---- - -## GitHub Release Workflow - -### Restore from GitHub Releases - -```bash -# Restore latest version from GitHub -./deployment/development/init.sh restore_from_github - -# Restore specific version -./deployment/development/init.sh restore_from_github 0.0.1 - -# List available versions -./deployment/development/init.sh list_github_versions - -# Get latest version info -./deployment/development/init.sh get_latest_github_version -``` - -### Docker Compose Integration - -```bash -# Start all services (backup-manager auto-restores from GitHub) -docker-compose up -d - -# Check backup-manager logs -docker logs backup-manager - -# Run commands in backup-manager container -docker exec backup-manager /init.sh list_github_versions -docker exec backup-manager /init.sh restore_from_github 0.0.1 -``` - -### GitHub Repository Structure - -The system expects this structure in [LDFLK/data-backups](https://github.com/LDFLK/data-backups): - -``` -data-backups-0.0.1/ -└── opengin - └── development - ├── mongodb - │ └── mongodb.tar.gz - ├── postgres - │ └── postgres.tar.gz - └── neo4j - └── neo4j.dump -``` - -### GitHub Archive URLs - -The system uses GitHub's built-in archive feature: - -- **Version 0.0.1**: https://github.com/LDFLK/data-backups/archive/refs/tags/0.0.1.zip -- **Version 0.0.2**: https://github.com/LDFLK/data-backups/archive/refs/tags/0.0.2.zip -- **Any version**: https://github.com/LDFLK/data-backups/archive/refs/tags/{version}.zip - ---- - -## Complete Workflow Examples - -### Scenario 1: Development Setup - -```bash -# 1. Start your services -docker-compose up -d - -# 2. Create local backups -./deployment/development/init.sh backup_mongodb -./deployment/development/init.sh backup_postgres -./deployment/development/init.sh backup_neo4j - -# 3. Test your application -# ... do development work ... - -# 4. Restore from local backups if needed -./deployment/development/init.sh restore_mongodb -./deployment/development/init.sh restore_postgres -./deployment/development/init.sh restore_neo4j -``` - -### Scenario 2: Production Deployment - -```bash -# 1. Deploy with GitHub backup restoration -docker-compose up -d - -# 2. The backup-manager automatically restores from GitHub -# Check logs to confirm -docker logs backup-manager - -# 3. Your application is ready with production data -``` - -### Scenario 3: Team Collaboration - -```bash -# 1. Create backups locally -./deployment/development/init.sh backup_mongodb -./deployment/development/init.sh backup_postgres -./deployment/development/init.sh backup_neo4j - -# 2. Upload to GitHub repository (manual process) -# - Create release in LDFLK/data-backups -# - Upload backup files to correct directory structure - -# 3. Team members restore from GitHub -./deployment/development/init.sh restore_from_github 0.0.1 -``` - ---- - -## Configuration - -### Environment Variables - -| Variable | Default | Description | -|----------|---------|-------------| -| `ENVIRONMENT` | `development` | Environment (dev/staging/prod) | -| `MONGODB_BACKUP_DIR` | `./backups/mongodb` | Local MongoDB backup directory | -| `POSTGRES_BACKUP_DIR` | `./backups/postgres` | Local PostgreSQL backup directory | -| `NEO4J_BACKUP_DIR` | `./backups/neo4j` | Local Neo4j backup directory | - -### Database Credentials - -Configure in `configs/backup.env`: - -```bash -# MongoDB -MONGODB_USERNAME=admin -MONGODB_PASSWORD=admin123 -MONGODB_DATABASE=opengin - -# PostgreSQL -POSTGRES_USER=postgres -POSTGRES_PASSWORD=postgres -POSTGRES_DATABASE=opengin -``` - ---- - -## Troubleshooting - -### Local Backup Issues - -```bash -# Check if backup directories exist -ls -la ./backups/mongodb/ -ls -la ./backups/postgres/ -ls -la ./backups/neo4j/ - -# Check database connectivity -docker exec mongodb mongo --eval "db.adminCommand('ping')" -docker exec postgres pg_isready -U postgres -docker exec neo4j cypher-shell -u neo4j -p neo4j123 "RETURN 1" -``` - -### GitHub Backup Issues - -```bash -# Test GitHub archive download -wget -O test.zip "https://github.com/LDFLK/data-backups/archive/refs/tags/0.0.1.zip" -unzip -l test.zip - -# Check archive structure -unzip -l test.zip | grep opengin - -# Run with debug output -bash -x ./deployment/development/init.sh restore_from_github 0.0.1 -``` - -### Container State Issues - -```bash -# Check container status -docker-compose ps - -# Check Neo4j container state (smart handling) -docker-compose ps neo4j - -# View logs -docker logs mongodb -docker logs postgres -docker logs neo4j -docker logs backup-manager -``` - ---- - -## Key Features - -### Smart Container Handling -- **Neo4j functions** intelligently detect container state -- Only stop/start containers when necessary -- Faster operations when containers are already in desired state - -### No Authentication Required -- **GitHub workflow** uses public archive URLs -- No API tokens or rate limits -- Direct file downloads with `wget` - -### Flexible Workflows -- **Local backups** for development and testing -- **GitHub releases** for production and team collaboration -- Both workflows use the same `init.sh` commands - -### Environment Support -- Development, staging, production environments -- Configurable backup directories -- Environment-specific GitHub releases - ---- - -## Command Reference - -### Local Backup Commands -```bash -# Backup -./deployment/development/init.sh backup_mongodb -./deployment/development/init.sh backup_postgres -./deployment/development/init.sh backup_neo4j - -# Restore -./deployment/development/init.sh restore_mongodb -./deployment/development/init.sh restore_postgres -./deployment/development/init.sh restore_neo4j - -# List -./deployment/development/init.sh list_mongodb_backups -./deployment/development/init.sh list_postgres_backups -./deployment/development/init.sh list_neo4j_backups -``` - -### GitHub Commands -```bash -# Restore -./deployment/development/init.sh restore_from_github -./deployment/development/init.sh restore_from_github 0.0.1 - -# Version management -./deployment/development/init.sh list_github_versions -./deployment/development/init.sh get_latest_github_version -``` - -### Service Management -```bash -# Neo4j service -./deployment/development/init.sh setup_neo4j -./deployment/development/init.sh run_neo4j - -# General -./deployment/development/init.sh setup -./deployment/development/init.sh help -``` - -This covers both local backup management and GitHub-based restoration workflows! 🎯 diff --git a/docs_deprecated/appendix/operations/mongodb.md b/docs_deprecated/appendix/operations/mongodb.md deleted file mode 100644 index 9d9d486f..00000000 --- a/docs_deprecated/appendix/operations/mongodb.md +++ /dev/null @@ -1,379 +0,0 @@ -# MongoDB Backup and Restore Guide - -This guide provides comprehensive instructions for backing up and restoring MongoDB databases in Docker containers. - -## Data Backups Repository Structure - -Backups are stored in a structured repository following this hierarchy: - -``` -data-backups/ -├── README.md -└── opengin - ├── development - │ ├── mongodb - │ │ └── opengin.tar.gz - │ ├── neo4j - │ │ └── neo4j.dump - │ └── postgres - │ └── opengin.tar.gz - ├── production - │ ├── mongodb - │ └── neo4j - └── staging - ├── mongodb - └── neo4j -``` - -This structure allows for: -- **Environment separation**: development, staging, production -- **Database type organization**: mongodb, neo4j, postgres -- **Version management**: 0.0.1, 0.0.2, etc. -- **Consistent naming**: All backups follow the same pattern - -## Prerequisites - -- Docker installed and running -- MongoDB container running (using docker-compose) -- Access to MongoDB container -- MongoDB credentials (replace `` and `` in commands below) - -## Backup and Restore Commands - -### Method 1: Direct Docker Commands (Recommended) - -#### 1.1 Create MongoDB Backup - -```bash -# Create backup directory -mkdir -p ./backups/mongodb - -# Create MongoDB dump -docker exec mongodb mongodump \ - --host=localhost:27017 \ - --username= \ - --password= \ - --authenticationDatabase=admin \ - --db=opengin \ - --out=/data/backup/mongodb_backup_$(date +%Y%m%d_%H%M%S) - -# Copy backup from container to host -docker cp mongodb:/data/backup/mongodb_backup_$(date +%Y%m%d_%H%M%S) ./backups/mongodb/ - -# Create compressed archive -cd ./backups/mongodb -tar -czf mongodb_backup_$(date +%Y%m%d_%H%M%S).tar.gz mongodb_backup_* -rm -rf mongodb_backup_* - -# Clean up container backup -docker exec mongodb rm -rf /data/backup/mongodb_backup_* -``` - -#### 1.2 Restore MongoDB from Backup - -```bash -# Extract backup file -tar -xzf mongodb_backup_20241215_143022.tar.gz - -# Copy backup to container -docker cp mongodb_backup_20241215_143022 mongodb:/data/backup/ - -# Restore database -docker exec mongodb mongorestore \ - --host=localhost:27017 \ - --username= \ - --password= \ - --authenticationDatabase=admin \ - --db=opengin \ - --drop \ - /data/backup/mongodb_backup_20241215_143022/opengin - -# Clean up -docker exec mongodb rm -rf /data/backup/mongodb_backup_20241215_143022 -rm -rf mongodb_backup_20241215_143022 -``` - -### Method 2: Using Docker Volume Mounts - -#### 2.1 Create Backup with Volume Mount - -```bash -# Create backup directory -mkdir -p ./backups/mongodb - -# Run mongodump with volume mount -docker run --rm \ - --network=ldf-network \ - --volume=mongodb_data:/data/db \ - --volume=$(pwd)/backups/mongodb:/backups \ - mongo:4.4 \ - mongodump \ - --host=mongodb:27017 \ - --username= \ - --password= \ - --authenticationDatabase=admin \ - --db=opengin \ - --out=/backups/mongodb_backup_$(date +%Y%m%d_%H%M%S) - -# Create compressed archive -cd ./backups/mongodb -tar -czf mongodb_backup_$(date +%Y%m%d_%H%M%S).tar.gz mongodb_backup_* -rm -rf mongodb_backup_* -``` - -#### 2.2 Restore with Volume Mount - -```bash -# Extract backup file -tar -xzf mongodb_backup_20241215_143022.tar.gz - -# Run mongorestore with volume mount -docker run --rm \ - --network=ldf-network \ - --volume=mongodb_data:/data/db \ - --volume=$(pwd)/backups/mongodb:/backups \ - mongo:4.4 \ - mongorestore \ - --host=mongodb:27017 \ - --username= \ - --password= \ - --authenticationDatabase=admin \ - --db=opengin \ - --drop \ - /backups/mongodb_backup_20241215_143022/opengin - -# Clean up -rm -rf mongodb_backup_20241215_143022 -``` - -## Configuration - -### Environment Variables - -Configure your backup settings in `configs/backup.env`: - -```bash -# MongoDB Backup Configuration -MONGODB_BACKUP_DIR=/path/to/your/backups/mongodb - -# MongoDB Credentials -MONGO_USER=admin -MONGO_PASSWORD=admin123 -MONGO_DATABASE=opengin -``` - -### Docker Compose Volumes - -The MongoDB service uses the following volumes: - -```yaml -volumes: - - mongodb_data:/data/db # Database data - - mongodb_config:/data/configdb # Configuration - - mongodb_backup:/data/backup # Backup storage -``` - -## Backup Strategies - -### 1. Full Database Backup - -```bash -# Backup entire database -docker exec mongodb mongodump \ - --host=localhost:27017 \ - --username= \ - --password= \ - --authenticationDatabase=admin \ - --db=opengin \ - --out=/data/backup/mongodb_backup_$(date +%Y%m%d_%H%M%S) -``` - -### 2. Specific Collection Backup - -```bash -# Backup specific collection -docker exec mongodb mongodump \ - --host=localhost:27017 \ - --username= \ - --password= \ - --authenticationDatabase=admin \ - --db=opengin \ - --collection=your_collection \ - --out=/data/backup/collection_backup -``` - -### 3. Compressed Backup - -```bash -# Create compressed backup -docker exec mongodb mongodump \ - --host=localhost:27017 \ - --username= \ - --password= \ - --authenticationDatabase=admin \ - --db=opengin \ - --archive=/data/backup/mongodb_backup.gz \ - --gzip -``` - -## Restore Strategies - -### 1. Full Database Restore - -```bash -# Restore entire database -docker exec mongodb mongorestore \ - --host=localhost:27017 \ - --username= \ - --password= \ - --authenticationDatabase=admin \ - --db=opengin \ - --drop \ - /data/backup/mongodb_backup_20241215_143022/opengin -``` - -### 2. Specific Collection Restore - -```bash -# Restore specific collection -docker exec mongodb mongorestore \ - --host=localhost:27017 \ - --username= \ - --password= \ - --authenticationDatabase=admin \ - --db=opengin \ - --collection=your_collection \ - /data/backup/collection_backup/opengin/your_collection.bson -``` - -### 3. Compressed Restore - -```bash -# Restore from compressed backup -docker exec mongodb mongorestore \ - --host=localhost:27017 \ - --username= \ - --password= \ - --authenticationDatabase=admin \ - --db=opengin \ - --archive=/data/backup/mongodb_backup.gz \ - --gzip -``` - -## Troubleshooting - -### Common Issues - -#### 1. Permission Denied - -```bash -# Check container permissions -docker exec mongodb ls -la /data/backup - -# Fix permissions if needed -docker exec mongodb chown -R mongodb:mongodb /data/backup -``` - -#### 2. Authentication Failed - -```bash -# Verify MongoDB is running -docker exec mongodb mongo --eval "db.adminCommand('ping')" - -# Check credentials -docker exec mongodb mongo -u admin -p admin123 --authenticationDatabase=admin -``` - -#### 3. Backup Directory Not Found - -```bash -# Create backup directory -docker exec mongodb mkdir -p /data/backup - -# Set proper permissions -docker exec mongodb chown -R mongodb:mongodb /data/backup -``` - -### Verification Commands - -#### Check Backup Integrity - -```bash -# List backup contents -tar -tzf mongodb_backup_20241215_143022.tar.gz - -# Verify database structure -docker exec mongodb mongo --eval "db.adminCommand('listCollections')" -``` - -#### Check Restore Success - -```bash -# Verify data was restored -docker exec mongodb mongo --eval "db.stats()" - -# Check specific collections -docker exec mongodb mongo --eval "db.your_collection.count()" -``` - -## Restore to Mongodb Atlas - -Note that this dump must be taken from OpenGIN dump program. - -```bash -mongorestore --uri="" --db=opengin --drop "" -``` - -## Best Practices - -### 1. Regular Backups - -- Schedule daily backups using cron -- Keep multiple backup versions -- Test restore procedures regularly - -### 2. Backup Storage - -- Store backups in multiple locations -- Use compression to save space -- Encrypt sensitive backup data - -### 3. Monitoring - -- Monitor backup success/failure -- Set up alerts for backup failures -- Log backup activities - -### 4. Security - -- Use strong authentication credentials -- Limit backup access permissions -- Encrypt backup files - -### Backup Script Example - -```bash -#!/bin/bash -# Daily MongoDB backup script - -BACKUP_DIR="/path/to/backups/mongodb" -LOG_FILE="/var/log/mongodb_backup.log" - -# Create backup -/path/to/deployment/development/init.sh backup_mongodb >> $LOG_FILE 2>&1 - -# Clean up old backups (keep last 7 days) -find $BACKUP_DIR -name "*.tar.gz" -mtime +7 -delete - -# Send notification -echo "MongoDB backup completed on $(date)" | mail -s "MongoDB Backup" admin@example.com -``` - -## Support - -For issues or questions regarding MongoDB backup and restore: - -1. Check the troubleshooting section above -2. Review MongoDB documentation -3. Check container logs: `docker logs mongodb` -4. Verify network connectivity: `docker network ls` diff --git a/docs_deprecated/appendix/operations/neo4j.md b/docs_deprecated/appendix/operations/neo4j.md deleted file mode 100644 index d3063cd0..00000000 --- a/docs_deprecated/appendix/operations/neo4j.md +++ /dev/null @@ -1,155 +0,0 @@ -# Neo4j Migration Guide: Docker Container to Aura - -This guide provides step-by-step instructions for migrating a Neo4j database from a Docker container to Neo4j Aura. - -## Data Backups Repository Structure - -Backups are stored in a structured repository following this hierarchy: - -``` -data-backups/ -├── README.md -└── opengin - ├── development - │ ├── mongodb - │ │ └── opengin.tar.gz - │ ├── neo4j - │ │ └── neo4j.dump - │ └── postgres - │ └── opengin.tar.gz - ├── production - │ ├── mongodb - │ └── neo4j - └── staging - ├── mongodb - └── neo4j -``` - -This structure allows for: -- **Environment separation**: development, staging, production -- **Database type organization**: mongodb, neo4j, postgres -- **Version management**: 0.0.1, 0.0.2, etc. -- **Consistent naming**: All backups follow the same pattern - -## Prerequisites - -- Docker installed and running -- Access to your Neo4j Docker container -- Neo4j Aura account and database instance -- Neo4j Aura connection URI, username, and password - -## Migration Steps - -### 1. Stop the Neo4j Container - -Make sure the neo4j container is stopped before creating the dump. - -### 2. Identify where the data is stored in the container - -Find the directory path that's mapped to /data in your neo4j container, this is where the data is stored. - -Run `docker inspect ` and look for the `Source` path in the Mounts section. - -This will look something like the following: - -If you are using an standalone Docker container built on top of an exisitng neo4j image with volumes named -`/var/lib/docker/volumes/neo4j_data/_data` - -Or if you are using the `docker-compose.yml` to build you will see the volume as `ldfarchitecture_neo4j_data`. - -Set this as `NEO4J_CONTAINER_DATA_VOLUME` - -```bash -export NEO4J_CONTAINER_DATA_VOLUME= -``` - -Also we need to figure out the path where this volume is mounted in the container. -That is defined in the `Dockerfile` as `NEO4J_dbms_directories_data`. - -```bash -export NEO4J_DBMS_DIRECTORIES_DATA= -``` - -So you can actually see if the data has been written to this volume via a docker container since this file system is not accessible outside a docker environment and you may have to mount it manually to an image and check. - -```bash -docker run --rm -it \ - --volume ${NEO4J_CONTAINER_DATA_VOLUME}:/${NEO4J_DBMS_DIRECTORIES_DATA} \ - alpine:latest \ - sh -``` - -Within this you can check for the data - -### 3. Create a Local Dump Folder - -Create a folder on your local machine to store the dump file. - -### 4. Create a Database Dump - -Run the following command to create a dump from your Neo4j Docker container: - -What happen below is we are going to run a temporary container (that's why there is `rm`) -and mount the volume which has neo4j data which is `NEO4J_CONTAINER_DATA_VOLUME` volume and it -is mounted to the `/data` folder of this temporary container. - -And the following command will create a dump on the `/backups` folder. - -```bash -neo4j-admin database dump neo4j --to-path=/backups -``` - -And since we `--volume=/Users/your_username/Documents/neo4j_dump:/backups ` since that the -host system can also access the backup file. - -```bash -docker run --rm \ ---volume=${NEO4J_CONTAINER_DATA_VOLUME}:/data \ ---volume=${NEO4J_BACKUP_DIR}:/backups \ -neo4j/neo4j-admin:latest \ -neo4j-admin database dump neo4j --to-path=/backups -``` - -### 4. Verify the Database Dump was created - -Navigate to the local folder specified previously and check that a dump file has been created inside. - -### 5. Upload the Dump to the Neo4j Docker (Local) - -Stop the container - -```bash -docker compose down neo4j -``` - -```bash -docker run --interactive --tty --rm \ - --volume=${NEO4J_CONTAINER_DATA_VOLUME}:/data \ - --volume=${NEO4J_BACKUP_DIR}:/backups \ - neo4j/neo4j-admin \ -neo4j-admin database load neo4j --from-path=/backups --overwrite-destination=true -``` - -### 6. Upload the Dump to Neo4j Aura - -Run the following command to upload the dump to your Neo4j Aura instance: - -```bash -docker run --rm \ ---volume=/Users/your_username/Documents/neo4j_dump:/dump \ -neo4j/neo4j-admin:latest \ -neo4j-admin database upload neo4j \ ---from-path=/dump \ ---to-uri= \ ---to-user= \ ---to-password= \ ---overwrite-destination=true -``` - -**Important:** Replace the following placeholders with your actual values: -- `/Users/your_username/Documents/neo4j_dump` with the correct path to your local folder with the dump file. -- ``,``,`` with your actual aura db credentials - -### 5. Verify the Migration - -Connect to your Neo4j Aura instance and verify that the new data has been inserted. diff --git a/docs_deprecated/appendix/operations/postgres.md b/docs_deprecated/appendix/operations/postgres.md deleted file mode 100644 index d5ce7629..00000000 --- a/docs_deprecated/appendix/operations/postgres.md +++ /dev/null @@ -1,297 +0,0 @@ -# PostgreSQL Backup and Restore Guide - -This guide provides comprehensive instructions for backing up and restoring PostgreSQL databases in Docker containers. - -## Data Backups Repository Structure - -Backups are stored in a structured repository following this hierarchy: - -``` -data-backups/ -├── README.md -└── opengin - ├── development - │ ├── mongodb - │ │ └── opengin.tar.gz - │ ├── neo4j - │ │ └── neo4j.dump - │ └── postgres - │ └── opengin.tar.gz - ├── production - │ ├── mongodb - │ └── neo4j - └── staging - ├── mongodb - └── neo4j -``` - -This structure allows for: -- **Environment separation**: development, staging, production -- **Database type organization**: mongodb, neo4j, postgres -- **Version management**: 0.0.1, 0.0.2, etc. -- **Consistent naming**: All backups follow the same pattern - -## Prerequisites - -- Docker installed and running -- PostgreSQL container running (using docker-compose) -- Access to PostgreSQL container -- PostgreSQL credentials (replace `` and `` in commands below) - -## Backup and Restore Commands - -### Method 1: Direct Docker Commands (Recommended) - -#### 1.1 Create PostgreSQL Backup - -```bash -# Create backup directory -mkdir -p ./backups/postgres - -# Create PostgreSQL dump -docker exec postgres pg_dump -U -h localhost -d -f /var/lib/postgresql/backup/opengin.sql - -# Copy backup from container to host -docker cp postgres:/var/lib/postgresql/backup/opengin.sql ./backups/postgres/ - -# Create compressed archive -cd ./backups/postgres -tar -czf opengin.tar.gz opengin.sql -rm -rf opengin.sql - -# Clean up container backup -docker exec postgres rm -rf /var/lib/postgresql/backup/opengin.sql -``` - -#### 1.2 Restore PostgreSQL from Backup - -```bash -# Extract backup file -tar -xzf opengin.tar.gz - -# Copy backup to container -docker cp opengin.sql postgres:/var/lib/postgresql/backup/ - -# Restore database -docker exec postgres psql -U -d -f /var/lib/postgresql/backup/opengin.sql - -# Clean up -docker exec postgres rm -rf /var/lib/postgresql/backup/opengin.sql -rm -rf opengin.sql -``` - -### Method 2: Using Docker Volume Mounts - -#### 2.1 Create Backup with Volume Mount - -```bash -# Create backup directory -mkdir -p ./backups/postgres - -# Run pg_dump with volume mount -docker run --rm \ - --network=ldf-network \ - --volume=postgres_data:/var/lib/postgresql/data \ - --volume=$(pwd)/backups/postgres:/backups \ - postgres:16 \ - pg_dump -U -h postgres -d -f /backups/opengin.sql - -# Create compressed archive -cd ./backups/postgres -tar -czf opengin.tar.gz opengin.sql -rm -rf opengin.sql -``` - -#### 2.2 Restore with Volume Mount - -```bash -# Extract backup file -tar -xzf opengin.tar.gz - -# Run psql with volume mount -docker run --rm \ - --network=ldf-network \ - --volume=postgres_data:/var/lib/postgresql/data \ - --volume=$(pwd)/backups/postgres:/backups \ - postgres:16 \ - psql -U -h postgres -d -f /backups/opengin.sql - -# Clean up -rm -rf opengin.sql -``` - -## Configuration - -### Environment Variables - -The backup process uses the following environment variables from `configs/backup.env`: - -```bash -# PostgreSQL Backup Configuration -POSTGRES_BACKUP_DIR=/path/to/backup/directory -POSTGRES_USER=postgres -POSTGRES_PASSWORD=postgres -POSTGRES_DATABASE=opengin -``` - -### Docker Compose Volumes - -The PostgreSQL service uses the following volumes: - -```yaml -volumes: - - postgres_data:/var/lib/postgresql/data # Database data - - postgres_backup:/var/lib/postgresql/backup # Backup storage -``` - -## Backup Strategies - -### 1. Full Database Backup - -```bash -# Backup entire database -docker exec postgres pg_dump -U -h localhost -d -f /var/lib/postgresql/backup/opengin.sql -``` - -### 2. Specific Schema Backup - -```bash -# Backup specific schema -docker exec postgres pg_dump -U -h localhost -d -n -f /var/lib/postgresql/backup/schema_backup.sql -``` - -### 3. Compressed Backup - -```bash -# Create compressed backup -docker exec postgres pg_dump -U -h localhost -d -Z 9 -f /var/lib/postgresql/backup/opengin.sql.gz -``` - -### 4. Custom Format Backup - -```bash -# Create custom format backup (binary format) -docker exec postgres pg_dump -U -h localhost -d -Fc -f /var/lib/postgresql/backup/opengin.dump -``` - -## Restore Strategies - -### 1. Full Database Restore - -```bash -# Restore entire database -docker exec postgres psql -U -d -f /var/lib/postgresql/backup/opengin.sql -``` - -### 2. Custom Format Restore - -```bash -# Restore from custom format backup -docker exec postgres pg_restore -U -h localhost -d /var/lib/postgresql/backup/opengin.dump -``` - -### 3. Schema-only Restore - -```bash -# Restore schema only (no data) -docker exec postgres psql -U -d -f /var/lib/postgresql/backup/schema_backup.sql -``` - -## Troubleshooting - -### Common Issues - -#### 1. Connection Refused - -```bash -# Check if PostgreSQL container is running -docker ps | grep postgres - -# Check container logs -docker logs postgres - -# Test connection -docker exec postgres pg_isready -U postgres -``` - -#### 2. Permission Denied - -```bash -# Check file permissions in container -docker exec postgres ls -la /var/lib/postgresql/backup/ - -# Fix permissions if needed -docker exec postgres chown postgres:postgres /var/lib/postgresql/backup/ -``` - -#### 3. Database Not Found - -```bash -# List available databases -docker exec postgres psql -U postgres -c "\l" - -# Create database if needed -docker exec postgres createdb -U postgres -``` - -#### 4. Backup File Not Found - -```bash -# Check if backup directory exists -docker exec postgres ls -la /var/lib/postgresql/ - -# Create backup directory -docker exec postgres mkdir -p /var/lib/postgresql/backup -``` - -### Advanced Troubleshooting - -#### 1. Check PostgreSQL Version - -```bash -docker exec postgres psql -U postgres -c "SELECT version();" -``` - -#### 2. Check Database Size - -```bash -docker exec postgres psql -U postgres -c "SELECT pg_size_pretty(pg_database_size(''));" -``` - -#### 3. Check Active Connections - -```bash -docker exec postgres psql -U postgres -c "SELECT * FROM pg_stat_activity;" -``` - -## Restoring in Neon - -```bash -psql "" -f /opengin.sql -``` - -## Best Practices - -### 1. Regular Backups - -- Schedule automated backups using cron jobs -- Keep multiple backup versions -- Test restore procedures regularly - -### 2. Backup Storage - -- Store backups in multiple locations -- Use compression to save space -- Encrypt sensitive backup data - -### 3. Monitoring - -- Monitor backup success/failure -- Set up alerts for backup failures -- Log backup activities - -### 4. Security - -- Use strong passwords -- Limit backup file permissions -- Secure backup storage locations diff --git a/docs_deprecated/appendix/release_life_cycle.md b/docs_deprecated/appendix/release_life_cycle.md deleted file mode 100644 index c9a87bb0..00000000 --- a/docs_deprecated/appendix/release_life_cycle.md +++ /dev/null @@ -1,139 +0,0 @@ -# OpenGIN Release Lifecycle - -This document defines the **release stages**, **versioning scheme**, and **naming conventions** used in the OpenGIN platform. -It ensures consistency, clarity, and traceability across all releases and environments. - ---- - -## Versioning Scheme - -OpenGIN follows **Semantic Versioning (SemVer)** with optional pre-release identifiers and calendar tags. - -``` -MAJOR.MINOR.PATCH[-STAGE.NUMBER] -``` - -**Examples:** -- `1.0.0-alpha.1` — First internal alpha build -- `1.0.0-beta.2` — Second beta release -- `1.0.0-RC.1` — First release candidate -- `1.0.0` — General Availability (Stable) - ---- - -## Alpha Release - -**Definition:** -> Early internal version used for architecture validation, core integration, and early-stage testing. - -**Purpose:** -- Validate service-to-service communication (e.g., Database layer, backing up, integration tests, etc.). -- Test architecture, schema design, and API functionality. -- Identify critical issues before public testing. - -**Stability:** ❌ Not stable -**Audience:** Internal developers and system engineers -**Tag Example:** `1.0.0-alpha.1` - ---- - -## Beta Release - -**Definition:** -> Feature-complete version intended for broader testing and validation from selected external users. - -**Purpose:** -- Ensure all modules (Graph Engine, Read API, Extractors, Frontend Studio) function correctly together. -- Gather usability and performance feedback. -- Identify and fix known issues before production readiness. - -**Stability:** ⚙️ Moderate -**Audience:** Internal + selected external testers -**Tag Example:** `2.0.0-beta.1` - ---- - -## Release Candidate (RC) - -**Definition:** -> A near-final version that could become the official release if no significant issues are found. - -**Purpose:** -- Verify stability under production-like conditions. -- Validate end-to-end integrations and ensure all bugs are resolved. -- Prepare deployment artifacts (Docker images, Helm charts, documentation). - -**Stability:** 🧩 High -**Audience:** QA and staging environments -**Tag Example:** `2.0.0-RC.1` - ---- - -## General Availability (GA) - -**Definition:** -> The official, production-ready release validated through testing, review, and documentation. - -**Purpose:** -- Fully tested and validated for production deployment. -- Represents a stable baseline for users and partners. -- Supported under maintenance and patch cycles. - -**Stability:** ✅ Stable -**Audience:** All users, partners, and production environments -**Tag Example:** `2.0.0` - ---- - -## Optional Calendar Tagging - -Each major or minor release can optionally include a **calendar tag** to align with milestones. - -**Examples:** -- `OpenGIN 1.0.0 (2025.09)` — September 2025 major release -- `OpenGIN 1.1.0 (2026.Q1)` — First quarter 2026 update - -This helps track releases alongside development cycles or roadmap milestones. - ---- - -## Branch Naming Convention - -To maintain a consistent workflow across teams and CI/CD pipelines: - -| Branch Type | Format | Description | -|--------------|--------|-------------| -| **Main** | `main` | Always points to the latest stable (GA) release | -| **Development** | `dev` | Used for integration and pre-release work | -| **Feature** | `feat-` | New functionality under development | -| **Release** | `release-` | Prepares an upcoming release (alpha, beta, rc) | -| **Hotfix** | `hotfix-` | Urgent fixes for production issues | - -**Examples:** -- `release-2.0.0-beta.1` -- `feature-graph-link-expansion` -- `hotfix-v1.3.2` - ---- - -## Summary Table - -| Stage | Tag Example | Audience | Stability | Purpose | -|--------|--------------|-----------|------------|----------| -| **Alpha** | `2.0.0-alpha.1` | Internal | 🚧 Low | Architecture validation | -| **Beta** | `2.0.0-beta.1` | Testers | ⚙️ Medium | Feature validation | -| **RC** | `2.0.0-RC.1` | QA | 🧩 High | Final verification | -| **GA** | `2.0.0` | Public | ✅ Stable | Production deployment | - ---- - -## Notes - -- Increment numbers (`.1`, `.2`, `.3`, etc.) indicate iterations within a stage. -- Only **GA** releases are considered official and stable for production. -- Patch versions (e.g., `2.0.1`, `2.0.2`) are reserved for minor fixes. -- Each release should include changelogs summarizing new features, improvements, and bug fixes. - ---- - -_This standard ensures OpenGIN releases remain consistent, transparent, and traceable across all development and deployment environments._ diff --git a/docs_deprecated/appendix/storage.md b/docs_deprecated/appendix/storage.md deleted file mode 100644 index ffab335e..00000000 --- a/docs_deprecated/appendix/storage.md +++ /dev/null @@ -1,269 +0,0 @@ -# Storage Types Documentation - -This document describes the different storage types supported by the system and their JSON representations. - -## Overview - -The system supports five main storage types: -- Tabular Data -- Graph Data -- List Data -- Map Data -- Scalar Data - -Each type has a specific JSON structure that helps in identifying and processing the data correctly. - -## Storage Types - -### 1. Tabular Data - -Tabular data represents structured data in a table format with columns and rows. - -```json -{ - "attributes": { - "columns": ["id", "name", "age"], - "rows": [ - [1, "John", 30], - [2, "Jane", 25], - [3, "Bob", 35] - ] - } -} -``` - -#### Features: -- Must have both `columns` and `rows` fields -- `columns` is an array of strings representing column names -- `rows` is an array of arrays, where each inner array represents a row -- Each row must have the same number of elements as there are columns -- Column types should be consistent within each column - -### 2. Graph Data - -Graph data represents a network of nodes and their relationships. - -> **⚠️ Important Note** -> -> Graph insertion as an attribute is not supported yet. However, the Entity level API can be used to create graphs. - -```json -{ - "attributes": { - "nodes": [ - { - "id": "node1", - "type": "user", - "properties": { - "name": "John", - "age": 30 - } - } - ], - "edges": [ - { - "source": "node1", - "target": "node2", - "type": "follows", - "properties": { - "since": "2024-01-01" - } - } - ] - } -} -``` - -#### Features: -- Must have both `nodes` and `edges` fields -- `nodes` is an array of objects with: - - `id`: Unique identifier - - `type`: Node type - - `properties`: Additional node attributes -- `edges` is an array of objects with: - - `source`: Source node ID - - `target`: Target node ID - - `type`: Relationship type - - `properties`: Additional edge attributes - -### 3. List Data - -List data represents an ordered collection of items. - -```json -{ - "attributes": { - "items": [1, 2, 3, 4, 5] - } -} -``` - -> **⚠️ Important Note** -> -> List insertion as an attribute is not supported yet. However, the record can be saved as a table with one row or one column. - - -#### Features: -- Must have an `items` field -- `items` is an array that can contain: - - Simple values (numbers, strings, booleans) - - Complex objects - - Nested arrays - - Mixed types - -### 4. Map Data - -Map data represents a collection of key-value pairs. - -```json -{ - "attributes": { - "key1": "value1", - "key2": 42, - "key3": { - "nested": "value" - } - } -} -``` - -> **⚠️ Important Note** -> -> Document insertion as an attribute is not supported yet. However, the Entity level API can be used to create metadata which is stored as a document. - - -#### Features: -- Can have any number of key-value pairs -- Keys must be strings -- Values can be: - - Simple values (numbers, strings, booleans) - - Objects - - Arrays - - Nested structures - -### 5. Scalar Data - -Scalar data represents a single value. - -```json -{ - "attributes": { - "value": 42 - } -} -``` - -> **⚠️ Important Note** -> -> Scalar insertion as an attribute is not supported yet. However, the Entity level API can be used to save as a one row one column table. - - -#### Features: -- Must have exactly one field -- The value can be: - - Number - - String - - Boolean - - Null - -## Type Inference Rules - -The system follows these rules to determine the storage type: - -1. If the structure has both `columns` and `rows` fields, it's classified as Tabular Data -2. If the structure has both `nodes` and `edges` fields, it's classified as Graph Data -3. If the structure has an `items` field containing an array, it's classified as List Data -4. If the structure has a single field with a scalar value, it's classified as Scalar Data -5. If none of the above conditions are met, it's classified as Map Data - -## Best Practices - -1. **Consistency**: Maintain consistent data types within columns for tabular data -2. **Unique Identifiers**: Use unique IDs for nodes in graph data -3. **Type Safety**: Ensure proper type checking when working with mixed-type lists -4. **Nesting**: Be mindful of nesting depth in map and graph structures -5. **Validation**: Validate data structures before processing - -## Examples - -### Complex Tabular Data -```json -{ - "attributes": { - "columns": ["id", "name", "age", "address"], - "rows": [ - [1, "John", 30, {"city": "NY", "zip": "10001"}], - [2, "Jane", 25, {"city": "SF", "zip": "94105"}] - ], - "metadata": { - "total_rows": 2, - "last_updated": "2024-03-20" - } - } -} -``` - -### Complex Graph Data -```json -{ - "attributes": { - "nodes": [ - { - "id": "user1", - "type": "user", - "properties": { - "name": "Alice", - "age": 30, - "location": "NY" - } - } - ], - "edges": [ - { - "source": "user1", - "target": "post1", - "type": "created", - "properties": { - "timestamp": "2024-03-20T10:00:00Z" - } - } - ] - } -} -``` - -### Complex List Data -```json -{ - "attributes": { - "items": [ - [1, 2, 3], - {"id": 1, "name": "item1"}, - "string item", - true - ] - } -} -``` - -### Complex Map Data -```json -{ - "attributes": { - "user": { - "profile": { - "name": "John", - "age": 30, - "address": { - "city": "New York", - "zip": "10001" - } - }, - "settings": { - "theme": "dark", - "notifications": true - } - } - } -} -``` diff --git a/docs_deprecated/assets/css/style.css b/docs_deprecated/assets/css/style.css deleted file mode 100644 index 26f9e673..00000000 --- a/docs_deprecated/assets/css/style.css +++ /dev/null @@ -1,573 +0,0 @@ -/* Color Variables */ -:root { - --blue: #3b82f6; - --dark-blue: #1e3a8a; - --orange: #f97316; - --sidebar-width: 280px; - --sidebar-bg: var(--dark-blue); - --sidebar-text: #ffffff; - --content-bg: #ffffff; - --text-primary: #1f2937; - --text-secondary: #6b7280; - --border-color: #e5e7eb; - --link-color: var(--blue); - --link-hover: var(--orange); - --code-bg: #f3f4f6; - --code-border: #d1d5db; -} - -/* Reset and Base Styles */ -* { - margin: 0; - padding: 0; - box-sizing: border-box; - scroll-behavior: smooth; -} - -body { - font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, sans-serif; - line-height: 1.6; - color: var(--text-primary); - background-color: var(--content-bg); - overflow-x: hidden; -} - -/* Sidebar Toggle Button */ -.sidebar-toggle { - position: fixed; - top: 20px; - left: 20px; - z-index: 1001; - background: var(--sidebar-bg); - border: none; - padding: 12px; - cursor: pointer; - border-radius: 4px; - display: flex; - flex-direction: column; - gap: 4px; - box-shadow: 0 2px 8px rgba(0, 0, 0, 0.15); - transition: all 0.3s ease; -} - -.sidebar-toggle:hover { - background: var(--blue); -} - -.sidebar-toggle span { - width: 24px; - height: 3px; - background: var(--sidebar-text); - border-radius: 2px; - transition: all 0.3s ease; -} - -.sidebar.open ~ .content .sidebar-toggle { - left: calc(var(--sidebar-width) + 20px); -} - -/* Sidebar Styles */ -.sidebar { - position: fixed; - left: 0; - top: 0; - width: var(--sidebar-width); - height: 100vh; - background-color: var(--sidebar-bg); - color: var(--sidebar-text); - transform: translateX(-100%); - transition: transform 0.3s ease; - z-index: 1000; - overflow-y: auto; - box-shadow: 2px 0 8px rgba(0, 0, 0, 0.1); -} - -.sidebar.open { - transform: translateX(0); -} - -.sidebar-header { - padding: 24px 20px; - border-bottom: 1px solid rgba(255, 255, 255, 0.1); - background-color: rgba(0, 0, 0, 0.1); - display: flex; - align-items: center; - justify-content: flex-start; -} - -.sidebar-header a { - display: inline-flex; - align-items: center; - gap: 12px; - text-decoration: none; - transition: transform 0.3s ease, color 0.3s ease; -} - -.sidebar-header a:hover { - transform: translateX(8px); -} - -.sidebar-header .logo { - width: 44px; - height: auto; - display: block; - filter: drop-shadow(0 2px 6px rgba(0, 0, 0, 0.25)); -} - - -.sidebar-header h1 { - font-size: 24px; - font-weight: 700; - color: var(--sidebar-text); - margin: 0; - transition: color 0.3s ease; -} - -.sidebar-header a:hover h1 { - color: var(--link-hover); -} - -/* Sidebar Navigation */ -.sidebar-nav { - padding: 16px 0; -} - -.sidebar-nav ul { - list-style: none; -} - -.sidebar-nav > ul > li { - margin-bottom: 4px; -} - -.sidebar-nav .nav-link { - display: block; - padding: 12px 20px; - color: rgba(255, 255, 255, 0.9); - text-decoration: none; - transition: all 0.2s ease; - border-left: 3px solid transparent; -} - -.sidebar-nav .nav-link:hover { - background-color: rgba(255, 255, 255, 0.1); - color: var(--sidebar-text); - border-left-color: var(--orange); -} - -.sidebar-nav .nav-link.active { - background-color: rgba(59, 130, 246, 0.2); - color: var(--sidebar-text); - border-left-color: var(--blue); - font-weight: 600; -} - -/* Expandable Sections */ -.nav-section { - margin-bottom: 4px; -} - -.nav-section-header { - display: flex; - justify-content: space-between; - align-items: center; - padding: 12px 20px; - cursor: pointer; - user-select: none; - color: rgba(255, 255, 255, 0.95); - font-weight: 600; - transition: all 0.2s ease; - border-left: 3px solid transparent; -} - -.nav-section-header:hover { - background-color: rgba(255, 255, 255, 0.1); - border-left-color: var(--orange); -} - -.nav-section-header.expanded .expand-icon { - transform: rotate(180deg); -} - -.expand-icon { - transition: transform 0.3s ease; - font-size: 12px; - color: rgba(255, 255, 255, 0.7); -} - -.nav-subsection { - max-height: 0; - overflow: hidden; - transition: max-height 0.3s ease; - background-color: rgba(0, 0, 0, 0.1); -} - -.nav-subsection.expanded { - max-height: 1000px; -} - -.nav-subsection li { - padding-left: 20px; -} - -.nav-subsection .nav-link { - padding: 10px 20px; - font-size: 14px; - color: rgba(255, 255, 255, 0.85); -} - -.nav-subsection .nav-link:hover { - padding-left: 24px; -} - -/* Main Content */ -.content { - margin-left: 0; - min-height: 100vh; - padding: 80px 40px 40px; - transition: margin-left 0.3s ease; -} - -.sidebar.open ~ .content { - margin-left: var(--sidebar-width); -} - -.content-wrapper { - max-width: 1200px; - margin: 0 auto; -} - -.loading { - text-align: center; - padding: 60px 20px; - color: var(--text-secondary); - font-size: 18px; -} - -/* Markdown Content Styling */ -.markdown-body { - font-size: 16px; - line-height: 1.8; - color: var(--text-primary); -} - -.markdown-body h1 { - font-size: 2.5em; - margin-top: 0; - margin-bottom: 24px; - padding-bottom: 12px; - border-bottom: 2px solid var(--border-color); - color: var(--dark-blue); -} - -.markdown-body h2 { - font-size: 2em; - margin-top: 32px; - margin-bottom: 16px; - color: var(--dark-blue); - border-bottom: 1px solid var(--border-color); - padding-bottom: 8px; -} - -.markdown-body h3 { - font-size: 1.5em; - margin-top: 24px; - margin-bottom: 12px; - color: var(--blue); -} - -.markdown-body h4 { - font-size: 1.25em; - margin-top: 20px; - margin-bottom: 10px; - color: var(--text-primary); -} - -.markdown-body p { - margin-bottom: 16px; -} - -.markdown-body a { - color: var(--link-color); - text-decoration: none; - border-bottom: 1px solid transparent; - transition: all 0.2s ease; -} - -.markdown-body a:hover { - color: var(--link-hover); - border-bottom-color: var(--link-hover); -} - -.markdown-body ul, -.markdown-body ol { - margin-bottom: 16px; - padding-left: 30px; -} - -.markdown-body li { - margin-bottom: 8px; -} - -.markdown-body blockquote { - border-left: 4px solid var(--blue); - padding-left: 16px; - margin: 16px 0; - color: var(--text-secondary); - font-style: italic; -} - -.markdown-body code { - background-color: var(--code-bg); - border: 1px solid var(--code-border); - border-radius: 4px; - padding: 2px 6px; - font-size: 0.9em; - font-family: 'Monaco', 'Menlo', 'Ubuntu Mono', monospace; - color: #e83e8c; -} - -.markdown-body pre { - background-color: #1e293b; - border-radius: 8px; - padding: 16px; - overflow-x: auto; - margin: 20px 0; -} - -.markdown-body pre code { - background: transparent; - border: none; - padding: 0; - color: #e2e8f0; - font-size: 0.9em; -} - -.markdown-body table { - width: 100%; - border-collapse: collapse; - margin: 24px 0; - font-size: 0.95em; -} - -.markdown-body table th, -.markdown-body table td { - padding: 12px; - text-align: left; - border: 1px solid var(--border-color); -} - -.markdown-body table th { - background-color: var(--dark-blue); - color: white; - font-weight: 600; -} - -.markdown-body table tr:nth-child(even) { - background-color: #f9fafb; -} - -.markdown-body table tr:hover { - background-color: #f3f4f6; -} - -.markdown-body hr { - border: none; - border-top: 2px solid var(--border-color); - margin: 32px 0; -} - -.markdown-body img { - max-width: 100%; - height: auto; - border-radius: 8px; - margin: 20px 0; -} - -/* Responsive Design */ -@media (max-width: 768px) { - :root { - --sidebar-width: 260px; - } - - .content { - padding: 80px 20px 40px; - } - - .sidebar.open ~ .content { - margin-left: 0; - } - - .sidebar.open ~ .content .sidebar-toggle { - left: calc(var(--sidebar-width) - 40px); - } - - .footer { - margin-left: 0 !important; - } - - .markdown-body { - font-size: 15px; - } - - .markdown-body h1 { - font-size: 2em; - } - - .markdown-body h2 { - font-size: 1.75em; - } -} - -@media (min-width: 1024px) { - .sidebar { - transform: translateX(0); - } - - .content { - margin-left: var(--sidebar-width); - } - - .footer { - margin-left: var(--sidebar-width) !important; - } - - .sidebar-toggle { - display: none; - } -} - -/* Scrollbar Styling */ -.sidebar::-webkit-scrollbar { - width: 8px; -} - -.sidebar::-webkit-scrollbar-track { - background: rgba(0, 0, 0, 0.1); -} - -.sidebar::-webkit-scrollbar-thumb { - background: rgba(255, 255, 255, 0.2); - border-radius: 4px; -} - -.sidebar::-webkit-scrollbar-thumb:hover { - background: rgba(255, 255, 255, 0.3); -} - -/* Overlay for mobile */ -.sidebar-overlay { - display: none; - position: fixed; - top: 0; - left: 0; - width: 100%; - height: 100%; - background: rgba(0, 0, 0, 0.5); - z-index: 999; - opacity: 0; - transition: opacity 0.3s ease; -} - -.sidebar-overlay.active { - display: block; - opacity: 1; -} - -@media (max-width: 1023px) { - .sidebar-overlay.active { - display: block; - } -} - -/* Footer Styles */ -.footer { - position: relative; - z-index: 10; - border-top: 1px solid var(--border-color); - margin-top: 48px; - background-color: var(--content-bg); - margin-left: 0; - transition: margin-left 0.3s ease; -} - -.footer-container { - width: 100%; - max-width: 1200px; - margin: 0 auto; - padding: 24px 40px; -} - -.footer-content { - display: flex; - align-items: center; - justify-content: space-between; - flex-wrap: wrap; - gap: 16px; -} - -.footer-link { - text-decoration: none; - color: inherit; -} - -.footer-text { - color: var(--text-secondary); - font-size: 14px; - margin: 0; -} - -.footer-brand { - color: var(--text-secondary); - transition: color 0.2s ease; -} - -.footer-link:hover .footer-brand { - color: var(--link-hover); -} - -.footer-social { - display: flex; - align-items: center; - gap: 16px; -} - -.footer-social-link { - display: flex; - align-items: center; - justify-content: center; - padding: 8px; - color: var(--text-secondary); - text-decoration: none; - transition: all 0.2s ease; - border-radius: 4px; -} - -.footer-social-link:hover { - color: var(--link-hover); - transform: scale(1.1); - background-color: rgba(249, 115, 22, 0.1); -} - -.footer-social-link svg { - width: 20px; - height: 20px; -} - -/* Responsive Footer */ -@media (max-width: 768px) { - .footer-container { - padding: 20px; - } - - .footer-content { - flex-direction: column; - align-items: center; - text-align: center; - } - - .footer-social { - justify-content: center; - } -} - diff --git a/docs_deprecated/assets/images/data_flow_sequence_create.png b/docs_deprecated/assets/images/data_flow_sequence_create.png deleted file mode 100644 index 4ad7e3fa..00000000 Binary files a/docs_deprecated/assets/images/data_flow_sequence_create.png and /dev/null differ diff --git a/docs_deprecated/assets/images/data_flow_sequence_read.png b/docs_deprecated/assets/images/data_flow_sequence_read.png deleted file mode 100644 index bec0a484..00000000 Binary files a/docs_deprecated/assets/images/data_flow_sequence_read.png and /dev/null differ diff --git a/docs_deprecated/assets/images/opengin-architecture-diagram.png b/docs_deprecated/assets/images/opengin-architecture-diagram.png deleted file mode 100644 index 0a7d9ad8..00000000 Binary files a/docs_deprecated/assets/images/opengin-architecture-diagram.png and /dev/null differ diff --git a/docs_deprecated/assets/images/opengin_white.png b/docs_deprecated/assets/images/opengin_white.png deleted file mode 100644 index bc90c7f7..00000000 Binary files a/docs_deprecated/assets/images/opengin_white.png and /dev/null differ diff --git a/docs_deprecated/assets/images/storage_distribution_opengin.png b/docs_deprecated/assets/images/storage_distribution_opengin.png deleted file mode 100644 index 6c4a4ac3..00000000 Binary files a/docs_deprecated/assets/images/storage_distribution_opengin.png and /dev/null differ diff --git a/docs_deprecated/assets/js/app.js b/docs_deprecated/assets/js/app.js deleted file mode 100644 index 9cc1e5a3..00000000 --- a/docs_deprecated/assets/js/app.js +++ /dev/null @@ -1,446 +0,0 @@ -// App State -const appState = { - sidebarOpen: window.innerWidth >= 1024, // Open by default on desktop - expandedSections: new Set(['overview']), // Default expanded - currentPath: null -}; - -// Initialize the application -document.addEventListener('DOMContentLoaded', () => { - initializeApp(); -}); - -function initializeApp() { - // Initialize sidebar state (open on desktop) - const sidebar = document.getElementById('sidebar'); - if (appState.sidebarOpen) { - sidebar.classList.add('open'); - } - - // Setup sidebar toggle - const sidebarToggle = document.getElementById('sidebarToggle'); - - sidebarToggle.addEventListener('click', () => { - toggleSidebar(); - }); - - // Setup navigation section headers (expand/collapse) - setupNavigationSections(); - - // Setup navigation links - setupNavigationLinks(); - - // Setup routing - setupRouting(); - - // Handle initial route - handleRoute(); - - // Setup overlay for mobile - setupOverlay(); - - // Handle window resize - window.addEventListener('resize', () => { - const shouldBeOpen = window.innerWidth >= 1024; - if (shouldBeOpen !== appState.sidebarOpen) { - appState.sidebarOpen = shouldBeOpen; - sidebar.classList.toggle('open', shouldBeOpen); - } - }); -} - -function toggleSidebar() { - const sidebar = document.getElementById('sidebar'); - const overlay = document.querySelector('.sidebar-overlay'); - - appState.sidebarOpen = !appState.sidebarOpen; - sidebar.classList.toggle('open', appState.sidebarOpen); - - if (overlay) { - overlay.classList.toggle('active', appState.sidebarOpen); - } - - // Close on mobile when clicking outside - if (window.innerWidth <= 1023 && appState.sidebarOpen) { - overlay.addEventListener('click', () => { - toggleSidebar(); - }, { once: true }); - } -} - -function setupOverlay() { - const overlay = document.createElement('div'); - overlay.className = 'sidebar-overlay'; - document.body.appendChild(overlay); -} - -function setupNavigationSections() { - const sectionHeaders = document.querySelectorAll('.nav-section-header'); - - sectionHeaders.forEach(header => { - const section = header.getAttribute('data-section'); - const subsection = document.querySelector(`.nav-subsection[data-subsection="${section}"]`); - - // Set initial expanded state - if (appState.expandedSections.has(section)) { - header.classList.add('expanded'); - subsection.classList.add('expanded'); - } - - header.addEventListener('click', (e) => { - e.preventDefault(); - toggleSection(section); - }); - }); -} - -function toggleSection(section) { - const header = document.querySelector(`.nav-section-header[data-section="${section}"]`); - const subsection = document.querySelector(`.nav-subsection[data-subsection="${section}"]`); - - const isExpanded = header.classList.contains('expanded'); - - if (isExpanded) { - header.classList.remove('expanded'); - subsection.classList.remove('expanded'); - appState.expandedSections.delete(section); - } else { - header.classList.add('expanded'); - subsection.classList.add('expanded'); - appState.expandedSections.add(section); - } -} - -function setupNavigationLinks() { - const navLinks = document.querySelectorAll('.nav-link'); - - navLinks.forEach(link => { - link.addEventListener('click', (e) => { - e.preventDefault(); - const path = link.getAttribute('data-path'); - navigateTo(path); - }); - }); -} - -function setupRouting() { - // Handle hash changes - window.addEventListener('hashchange', () => { - handleRoute(); - }); - - // Handle browser back/forward - window.addEventListener('popstate', () => { - handleRoute(); - }); -} - -function handleRoute() { - const hash = window.location.hash || '#/'; - const path = hash.substring(2); // Remove '#/' - - if (!path) { - navigateTo('index.md'); - return; - } - - // Map route to markdown file - const routeMap = { - '': 'index.md', - - // Overview - 'overview/what_is_opengin': 'overview/what_is_opengin.md', - 'overview/architecture/index': 'overview/architecture/index.md', - 'overview/architecture/data_flow': 'overview/architecture/data_flow.md', - 'overview/architecture/getting-started': 'overview/architecture/getting-started.md', - 'overview/architecture/api-layer-details': 'overview/architecture/api-layer-details.md', - 'overview/architecture/core-api': 'overview/architecture/core-api.md', - - 'overview/architecture/database-schemas': 'overview/architecture/database-schemas.md', - - // Getting Started - 'getting_started/quick_start': 'getting_started/quick_start.md', - 'getting_started/installation': 'getting_started/installation.md', - - // Tutorial - 'tutorial/simple_app': 'tutorial/simple_app.md', - - // Reference - 'reference/datatype': 'reference/datatype.md', - 'reference/data-type-detection-patterns': 'reference/data-type-detection-patterns.md', - 'reference/storage': 'reference/storage.md', - 'reference/limitations': 'reference/limitations.md', - 'reference/release_life_cycle': 'reference/release_life_cycle.md', - - // Operations - 'reference/operations/backup_integration': 'reference/operations/backup_integration.md', - 'reference/operations/mongodb': 'reference/operations/mongodb.md', - 'reference/operations/neo4j': 'reference/operations/neo4j.md', - 'reference/operations/postgres': 'reference/operations/postgres.md', - - // FAQ - 'faq': 'faq.md' - }; - - const filePath = routeMap[path] || routeMap['']; - navigateTo(filePath); -} - -function navigateTo(filePath) { - appState.currentPath = filePath; - - // Update active link - updateActiveLink(filePath); - - // Auto-expand parent section if viewing a child page - autoExpandParentSection(filePath); - - // Load and render markdown - loadMarkdown(filePath); -} - -function updateActiveLink(filePath) { - const navLinks = document.querySelectorAll('.nav-link'); - navLinks.forEach(link => { - const linkPath = link.getAttribute('data-path'); - if (linkPath === filePath) { - link.classList.add('active'); - } else { - link.classList.remove('active'); - } - }); -} - -function autoExpandParentSection(filePath) { - // Auto-expand overview section - if (filePath.startsWith('overview/')) { - if (!appState.expandedSections.has('overview')) { - toggleSection('overview'); - } - } - - // Auto-expand getting_started section - if (filePath.startsWith('getting_started/')) { - if (!appState.expandedSections.has('getting_started')) { - toggleSection('getting_started'); - } - } - - // Auto-expand tutorial section - if (filePath.startsWith('tutorial/')) { - if (!appState.expandedSections.has('tutorial')) { - toggleSection('tutorial'); - } - } - - // Auto-expand reference section - if (filePath.startsWith('reference/')) { - if (!appState.expandedSections.has('reference')) { - toggleSection('reference'); - } - } -} - -async function loadMarkdown(filePath) { - const contentArea = document.getElementById('markdownContent'); - contentArea.innerHTML = '
Loading documentation...
'; - - try { - const response = await fetch(filePath); - if (!response.ok) { - throw new Error(`Failed to load ${filePath}`); - } - - const markdown = await response.text(); - renderMarkdown(markdown, filePath); - } catch (error) { - console.error('Error loading markdown:', error); - - let errorMessage = error.message; - let helpfulHint = ''; - - // Check for likely CORS/file protocol errors - if (window.location.protocol === 'file:' && (error instanceof TypeError || error.message.includes('fetch'))) { - errorMessage = 'Cannot load external files directly from the file system due to browser security restrictions (CORS).'; - helpfulHint = ` -
-

Solution: Please serve these files using a local web server.

-

Run one of the following commands in the docs/ directory:

- 
# Python 3
-python3 -m http.server 8000
-
-# Node.js
-npx http-server
-

Then open http://localhost:8000

-
- `; - } - - contentArea.innerHTML = ` -
-

Error Loading Document

-

${errorMessage}

- ${helpfulHint} -

Return to Home

-
- `; - } -} - -function renderMarkdown(markdown, filePath) { - const contentArea = document.getElementById('markdownContent'); - - // Configure marked options - marked.setOptions({ - breaks: true, - gfm: true, - highlight: function (code, lang) { - if (lang && hljs.getLanguage(lang)) { - try { - return hljs.highlight(code, { language: lang }).value; - } catch (err) { - console.error('Error highlighting code:', err); - } - } - return hljs.highlightAuto(code).value; - } - }); - - // Convert markdown to HTML - let html = marked.parse(markdown); - - // Process relative links in markdown - html = processMarkdownLinks(html, filePath); - - // Remove frontmatter if present - html = html.replace(/^---[\s\S]*?---\n/, ''); - - contentArea.innerHTML = html; - - // Highlight code blocks - hljs.highlightAll(); - - // Scroll to top - window.scrollTo(0, 0); -} - -function processMarkdownLinks(html, currentFilePath) { - // Create a temporary DOM element to parse HTML - const tempDiv = document.createElement('div'); - tempDiv.innerHTML = html; - - const links = tempDiv.querySelectorAll('a'); - links.forEach(link => { - const href = link.getAttribute('href'); - - if (!href) return; - - // Skip external links - if (href.startsWith('http://') || href.startsWith('https://') || href.startsWith('mailto:')) { - return; - } - - // Skip hash-only links - if (href.startsWith('#')) { - return; - } - - // Convert relative markdown links to hash routes - if (href.endsWith('.md') || (!href.includes('://') && !href.startsWith('/'))) { - let route = href; - - // Handle relative paths - if (href.startsWith('./')) { - const currentDir = currentFilePath.substring(0, currentFilePath.lastIndexOf('/') + 1); - route = currentDir + href.substring(2); - } else if (href.startsWith('../')) { - // Handle parent directory references - let currentDir = currentFilePath.substring(0, currentFilePath.lastIndexOf('/')); - const parts = href.split('../'); - for (let i = 0; i < parts.length - 1; i++) { - currentDir = currentDir.substring(0, currentDir.lastIndexOf('/')); - } - route = currentDir + '/' + parts[parts.length - 1]; - } else if (!href.startsWith('/')) { - // Relative to current directory - if (currentFilePath.includes('/')) { - const currentDir = currentFilePath.substring(0, currentFilePath.lastIndexOf('/') + 1); - route = currentDir + href; - } else { - route = href; - } - } else { - route = href.substring(1); - } - - // Ensure .md extension - if (!route.endsWith('.md') && !route.includes('.')) { - route += '.md'; - } - - // Normalize the route - route = route.replace(/\/+/g, '/').replace(/^\.\//, ''); - - // Map to hash route - const routeToHash = (filePath) => { - const cleanPath = filePath.replace(/\.md$/, ''); - const routeMap = { - 'index': '', - - // Overview - 'overview/what_is_opengin': 'overview/what_is_opengin', - 'overview/architecture/index': 'overview/architecture/index', - 'overview/architecture/data_flow': 'overview/architecture/data_flow', - 'overview/architecture/getting-started': 'overview/architecture/getting-started', - 'overview/architecture/api-layer-details': 'overview/architecture/api-layer-details', - 'overview/architecture/core-api': 'overview/architecture/core-api', - - 'overview/architecture/database-schemas': 'overview/architecture/database-schemas', - - // Getting Started - 'getting_started/quick_start': 'getting_started/quick_start', - 'getting_started/installation': 'getting_started/installation', - - // Tutorial - 'tutorial/simple_app': 'tutorial/simple_app', - - // Reference - 'reference/datatype': 'reference/datatype', - 'reference/data-type-detection-patterns': 'reference/data-type-detection-patterns', - 'reference/storage': 'reference/storage', - 'reference/limitations': 'reference/limitations', - 'reference/release_life_cycle': 'reference/release_life_cycle', - - // Operations - 'reference/operations/backup_integration': 'reference/operations/backup_integration', - 'reference/operations/mongodb': 'reference/operations/mongodb', - 'reference/operations/neo4j': 'reference/operations/neo4j', - 'reference/operations/postgres': 'reference/operations/postgres', - - // FAQ - 'faq': 'faq' - }; - return routeMap[cleanPath] !== undefined ? routeMap[cleanPath] : cleanPath; - }; - - const hashRoute = routeToHash(route); - link.setAttribute('href', `#/${hashRoute}`); - - // Remove existing listeners and add new one - const newLink = link.cloneNode(true); - link.parentNode.replaceChild(newLink, link); - newLink.addEventListener('click', (e) => { - e.preventDefault(); - navigateTo(route); - }); - } - }); - - return tempDiv.innerHTML; -} - -// Export functions for potential external use -window.docsApp = { - navigateTo, - toggleSidebar -}; diff --git a/docs_deprecated/faq.md b/docs_deprecated/faq.md deleted file mode 100644 index 8430b4be..00000000 --- a/docs_deprecated/faq.md +++ /dev/null @@ -1,27 +0,0 @@ -# Frequently Asked Questions - -## General - -### What is OpenGIN? -OpenGIN is a platform for building time-aware digital twins using a polyglot database architecture. - -### Why does it use three databases? -OpenGIN leverages the strengths of multiple databases: -- **MongoDB**: For flexible, schema-less metadata. -- **Neo4j**: For handling complex relationships and graph traversals. -- **PostgreSQL**: For storing time-series attributes and ensuring ACID compliance for structured data. - -## Technical - -### How do I reset the databases? -You can stop the containers and remove the volumes: -```bash -docker-compose down -v -``` -Then start them up again: -```bash -docker-compose up -d -``` - -### Can I add my own Entity types? -Yes, Entities are flexible. You define the `Kind` (Major/Minor) and the system adapts. You don't need to pre-define schemas in the database. diff --git a/docs_deprecated/getting_started/installation.md b/docs_deprecated/getting_started/installation.md deleted file mode 100644 index f5d842a0..00000000 --- a/docs_deprecated/getting_started/installation.md +++ /dev/null @@ -1,31 +0,0 @@ -# Installation & Setup - -## Quick Development Setup - -### Prerequisites -- Docker and Docker Compose -- Go 1.19+ (for CORE service) -- Ballerina (for APIs) - -### Start the System - -1. **Start databases** - ```bash - docker-compose up -d mongodb neo4j postgres - ``` - -2. **Start CORE service** - ```bash - docker-compose up -d core - ``` -3. **Start APIs** - ```bash - docker-compose up -d ingestion read - ``` - -### Test the System - -**Run E2E tests** -```bash -cd opengin/tests/e2e && ./run_e2e.sh -``` diff --git a/docs_deprecated/getting_started/quick_start.md b/docs_deprecated/getting_started/quick_start.md deleted file mode 100644 index 771b89f9..00000000 --- a/docs_deprecated/getting_started/quick_start.md +++ /dev/null @@ -1,83 +0,0 @@ -# Quick Start Guide - -This guide will help you interact with OpenGIN using cURL commands. Ensure that the system is running locally before proceeding (see [Installation](./installation.md)). - -## APIs - -* **Ingestion API**: `http://localhost:8080` (Write operations) -* **Read API**: `http://localhost:8081` (Read operations) - -## Run a sample query with CURL - -### Ingestion API (Write) - -**Create** - -```bash -curl -X POST http://localhost:8080/entities \ --H "Content-Type: application/json" \ --d '{ - "id": "12345", - "kind": { - "major": "example", - "minor": "test" - }, - "created": "2024-03-17T10:00:00Z", - "terminated": "", - "name": { - "startTime": "2024-03-17T10:00:00Z", - "endTime": "", - "value": { - "typeUrl": "type.googleapis.com/google.protobuf.StringValue", - "value": "entity-name" - } - }, - "metadata": [ - {"key": "owner", "value": "test-user"}, - {"key": "version", "value": "1.0"}, - {"key": "developer", "value": "V8A"} - ], - "attributes": [], - "relationships": [] -}' -``` - -**Read (via Ingestion API - usually for verification)** - -```bash -curl -X GET http://localhost:8080/entities/12345 -``` - -**Update** - -> **Note**: The update functionality is currently being refined to ensure it updates existing entities correctly. - -```bash -curl -X PUT http://localhost:8080/entities/12345 \ - -H "Content-Type: application/json" \ - -d '{ - "id": "12345", - "created": "2024-03-18T00:00:00Z", - "name": { - "startTime": "2024-03-18T00:00:00Z", - "value": "entity-name" - }, - "metadata": [ - {"key": "version", "value": "5.0"} - ] - }' -``` - -**Delete** - -```bash -curl -X DELETE http://localhost:8080/entities/12345 -``` - -### Read API (Read-Only) - -**Retrieve Metadata** - -```bash -curl -X GET "http://localhost:8081/v1/entities/12345/metadata" -``` diff --git a/docs_deprecated/index.html b/docs_deprecated/index.html deleted file mode 100644 index 1745d1dd..00000000 --- a/docs_deprecated/index.html +++ /dev/null @@ -1,182 +0,0 @@ - - - - - - - OpenGIN Documentation - - - - - - - - - - - - - -
-
-
-
Loading documentation...
-
-
-
- - - - - - - - - - - - \ No newline at end of file diff --git a/docs_deprecated/index.md b/docs_deprecated/index.md deleted file mode 100644 index bdda439c..00000000 --- a/docs_deprecated/index.md +++ /dev/null @@ -1,39 +0,0 @@ -# OpenGIN Documentation - -Welcome to the **OpenGIN (Open General Information Network)** documentation. This is your central hub for understanding, implementing, and contributing to the OpenGIN platform. - -## 📚 Overview -Understand the core concepts and architecture of OpenGIN. -- **[What is OpenGIN?](./overview/what_is_opengin.md)**: High-level introduction, data model, and key features. -- **[Architecture](./overview/architecture/index.md)**: Deep dive into the system design components. -- **[Data Flow](./overview/architecture/data_flow.md)**: End-to-end explanation of how data moves through the system. - -## 🚀 Getting Started -Get up and running quickly. -- **[Quick Start Guide](./getting_started/quick_start.md)**: Run your first queries using cURL. -- **[Installation](./getting_started/installation.md)**: Prerequisites and setup instructions. - -## 🎓 Tutorial -Hands-on guides to building with OpenGIN. -- **[Simple Application](./tutorial/simple_app.md)**: Learn how to model and create an "Employee" entity. - -## 📖 Appendix -Detailed documentation for specific components and operations. -- **[Data Types](./appendix/datatype.md)**: Supported data types and inference. -- **[Data Type Detection Patterns](./appendix/data-type-detection-patterns.md)**: Logic for inferring data types. -- **[Storage Types](./appendix/storage.md)**: How attributes are stored. -- **[Limitations](./appendix/limitations.md)**: Known constraints. -- **[Release Lifecycle](./appendix/release_life_cycle.md)**: Versioning and release process. - -## Operations - -- **[Operations Guide](./appendix/operations/backup_integration.md)**: - - **[Backup Integration](./appendix/operations/backup_integration.md)** - - **[MongoDB Backup](./appendix/operations/mongodb.md)** - - **[Neo4j Backup](./appendix/operations/neo4j.md)** - - **[PostgreSQL Backup](./appendix/operations/postgres.md)** - -## ❓ FAQ -- **[Frequently Asked Questions](./faq.md)** - ---- diff --git a/docs_deprecated/overview/architecture/api-layer-details.md b/docs_deprecated/overview/architecture/api-layer-details.md deleted file mode 100644 index 2fd779ac..00000000 --- a/docs_deprecated/overview/architecture/api-layer-details.md +++ /dev/null @@ -1,344 +0,0 @@ -# API Layer - Detailed Architecture - -This document provides comprehensive details about the Ingestion API and Read API layers of the OpenGIN system. - ---- - -## Overview - -The API Layer consists of two Ballerina-based REST services that provide external access to the OpenGIN system: -- **Ingestion API**: Handles entity mutations (CREATE, UPDATE, DELETE) -- **Read API**: Handles entity queries and retrieval - -Both APIs act as translation layers between external HTTP/JSON clients and the internal gRPC/Protobuf CORE service. - ---- - -## Ingestion API - -### Overview - -### Overview - -The Ingestion API serves as the primary gateway for data entering the system. Implemented as a Ballerina REST service, it allows external clients to create, update, and delete entities using standard HTTP methods. The service validates incoming JSON payloads against defined contracts before converting them for internal processing. - -### Request/Response Flow - -#### CREATE Entity - -**Request**: -```bash -POST /entities -Content-Type: application/json - -{ - "id": "entity123", - "kind": { - "major": "Person", - "minor": "Employee" - }, - "created": "2024-01-01T00:00:00Z", - "name": { - "startTime": "2024-01-01T00:00:00Z", - "endTime": "", - "value": "John Doe" - }, - "metadata": [ - {"key": "department", "value": "Engineering"}, - {"key": "role", "value": "Engineer"} - ], - "attributes": [ - { - "key": "salary", - "value": { - "values": [ - { - "startTime": "2024-01-01T00:00:00Z", - "endTime": "", - "value": 100000 - } - ] - } - } - ], - "relationships": [ - { - "key": "reports_to", - "value": { - "id": "rel123", - "relatedEntityId": "manager456", - "name": "reports_to", - "startTime": "2024-01-01T00:00:00Z", - "endTime": "", - "direction": "outgoing" - } - } - ] -} -``` - -**Response**: -```json -{ - "id": "entity123", - "kind": { - "major": "Person", - "minor": "Employee" - }, - "created": "2024-01-01T00:00:00Z", - "name": { - "startTime": "2024-01-01T00:00:00Z", - "value": "John Doe" - } -} -``` - -#### READ Entity - -**Request**: -```bash -GET /entities/entity123 -``` - -**Response**: -```json -{ - "id": "entity123", - "kind": { - "major": "Person", - "minor": "Employee" - }, - "created": "2024-01-01T00:00:00Z", - "name": { - "startTime": "2024-01-01T00:00:00Z", - "value": "John Doe" - }, - "metadata": [ - {"key": "department", "value": "Engineering"} - ], - "attributes": [], - "relationships": [] -} -``` - -#### UPDATE Entity - -**Request**: -```bash -PUT /entities/entity123 -Content-Type: application/json - -{ - "id": "entity123", - "kind": { - "major": "Person", - "minor": "Employee" - }, - "metadata": [ - {"key": "department", "value": "Sales"} - ] -} -``` - -**Response**: -```json -{ - "id": "entity123", - "kind": { - "major": "Person", - "minor": "Employee" - }, - "metadata": [ - {"key": "department", "value": "Sales"} - ] -} -``` - -#### DELETE Entity - -**Request**: -```bash -DELETE /entities/entity123 -``` - -**Response**: -``` -204 No Content -``` - ---- - -## Read API - -### Overview - -### Overview - -The Read API is dedicated to serving data retrieval requests. Also built with Ballerina, it provides a RESTful interface for querying entities, their metadata, relationships, and attributes. It supports complex queries including time-travel (temporal) lookups and selective field retrieval to optimize performance and payload size. - -### Read Operations - -#### Get Metadata - -**Request**: -```bash -GET /v1/entities/entity123/metadata -``` - -**Response**: -```json -{ - "metadata": { - "department": "Engineering", - "role": "Engineer", - "employeeId": "EMP-123" - } -} -``` - -**Use Case**: Retrieve flexible key-value metadata for an entity - -#### Get Relationships - -**Request**: -```bash -GET /v1/entities/entity123/relationships?name=reports_to&direction=outgoing -``` - -**Query Parameters**: -- `name`: Filter by relationship type -- `direction`: `outgoing` or `incoming` -- `relatedEntityId`: Filter by related entity -- `activeAt`: Temporal query (ISO 8601 timestamp) - -**Response**: -```json -{ - "relationships": [ - { - "id": "rel123", - "name": "reports_to", - "relatedEntityId": "manager456", - "startTime": "2024-01-01T00:00:00Z", - "endTime": null, - "direction": "outgoing" - } - ] -} -``` - -**Use Case**: Graph traversal, finding related entities - -#### Get Attributes - -**Request**: -```bash -GET /v1/entities/entity123/attributes?name=salary&activeAt=2024-06-01T00:00:00Z -``` - -**Query Parameters**: -- `name`: Filter by attribute name -- `activeAt`: Get attribute value at specific time - -**Response**: -```json -{ - "attributes": { - "salary": { - "values": [ - { - "startTime": "2024-01-01T00:00:00Z", - "endTime": "2024-06-30T23:59:59Z", - "value": 100000 - }, - { - "startTime": "2024-07-01T00:00:00Z", - "endTime": null, - "value": 110000 - } - ] - } - } -} -``` - -**Use Case**: Time-series data, historical attribute values - -#### Get Entity with Selective Fields - -**Request**: -```bash -GET /v1/entities/entity123?output=metadata,relationships -``` - -**Query Parameters**: -- `output`: Comma-separated list of fields to retrieve - - Options: `metadata`, `relationships`, `attributes` - - If omitted: returns only basic entity info - -**Response**: -```json -{ - "id": "entity123", - "kind": { - "major": "Person", - "minor": "Employee" - }, - "name": { - "value": "John Doe", - "startTime": "2024-01-01T00:00:00Z" - }, - "created": "2024-01-01T00:00:00Z", - "metadata": { - "department": "Engineering" - }, - "relationships": [ - { - "name": "reports_to", - "relatedEntityId": "manager456" - } - ] -} -``` - -**Use Case**: Optimized queries, reduce payload size - -### Temporal Queries - -The Read API supports temporal queries using the `activeAt` parameter: - -**Example**: Get employee's salary on specific date -```bash -GET /v1/entities/entity123/attributes?name=salary&activeAt=2024-03-15T00:00:00Z -``` - -This returns only the attribute value that was active on March 15, 2024. - -### Performance Optimization - -**Selective Field Retrieval**: -- Only fetch requested fields from CORE service -- Reduces database load -- Reduces network bandwidth -- Faster response times - -**Example**: -``` -Query: GET /v1/entities/entity123?output=metadata - -Instead of: - ├─ MongoDB (metadata) ✓ Retrieved - ├─ Neo4j (entity info) ✓ Retrieved - ├─ Neo4j (relationships) ✗ Skipped - └─ PostgreSQL (attributes) ✗ Skipped -``` - ---- - -## Related Documentation - -- [Main Architecture Overview](./index.md) -- [Service APIs](./api-layer-details.md) -- [Core API](./core-api-details.md) - ---- diff --git a/docs_deprecated/overview/architecture/core-api.md b/docs_deprecated/overview/architecture/core-api.md deleted file mode 100644 index 52f40274..00000000 --- a/docs_deprecated/overview/architecture/core-api.md +++ /dev/null @@ -1,200 +0,0 @@ -# Core API Documentation - -The Core API is the central orchestration service in the OpenGIN platform, responsible for coordinating data operations across multiple databases and providing a unified interface for entity management. - -## Overview - -The Core API serves as the business logic layer that: -- Receives protobuf messages from the Ingestion and Read APIs -- Orchestrates data operations across MongoDB, Neo4j, and PostgreSQL -- Implements type and storage inference algorithms -- Manages temporal data and relationships -- Returns processed data back to the API layer - ---- - -## Architecture - -### Service Components - -``` -┌─────────────────────────────────────────────────────────────────┐ -│ Core API (Go) │ -│ Port: 50051 │ -│ │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ gRPC Server │ │ -│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ -│ │ │CreateEntity │ │ ReadEntity │ │UpdateEntity │ │ │ -│ │ │ │ │ │ │ │ │ │ -│ │ │DeleteEntity │ │ QueryEntity │ │ │ │ │ -│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ -│ └─────────────────────────────────────────────────────────┘ │ -│ │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ Engine Layer │ │ -│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ -│ │ │AttributeProc│ │TypeInference│ │StorageInfer │ │ │ -│ │ │essor │ │ │ │ence │ │ │ -│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ -│ │ ┌─────────────────────────────────────────────────┐ │ │ -│ │ │ GraphMetadataManager │ │ │ -│ │ └─────────────────────────────────────────────────┘ │ │ -│ └─────────────────────────────────────────────────────────┘ │ -│ │ -│ ┌─────────────────────────────────────────────────────────┐ │ -│ │ Repository Layer │ │ -│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ -│ │ │ MongoDB │ │ Neo4j │ │ PostgreSQL │ │ │ -│ │ │ Repository │ │ Repository │ │ Repository │ │ │ -│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ -│ └─────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────┘ -``` - ---- - -## gRPC Service Methods - -### 1. CreateEntity - -Creates a new entity with metadata, attributes, and relationships. - -**Request Flow:** -1. Validate entity data and structure -2. Process metadata → MongoDB -3. Create entity node → Neo4j -4. Process attributes → PostgreSQL (with type/storage inference) -5. Create relationships → Neo4j -6. Return created entity - -**Key Features:** -- Automatic type inference for attributes -- Dynamic storage strategy determination -- Temporal relationship support -- Atomic operations across databases - -### 2. ReadEntity - -Retrieves entity data from multiple databases based on requested output parameters. - -**Request Flow:** -1. Always fetch core entity info from Neo4j -2. Conditionally fetch metadata from MongoDB -3. Conditionally fetch attributes from PostgreSQL -4. Conditionally fetch relationships from Neo4j -5. Assemble complete entity response - -**Output Parameters:** -- `metadata` - Include entity metadata -- `attributes` - Include entity attributes -- `relationships` - Include entity relationships -- `all` - Include everything - -### 3. UpdateEntity - -Updates existing entity data while maintaining temporal consistency. - -**Request Flow:** -1. Validate entity exists -2. Update metadata in MongoDB (if provided) -3. Update entity properties in Neo4j (if provided) -4. Update attributes in PostgreSQL (if provided) -5. Update relationships in Neo4j (if provided) -6. Return updated entity - -### 4. DeleteEntity - -Removes entity and all associated data from all databases. - -**Request Flow:** -1. Delete metadata from MongoDB -2. Delete entity node and relationships from Neo4j -3. Delete attributes from PostgreSQL -4. Return deletion confirmation - -### 5. QueryEntity - -Performs complex queries across multiple databases. - -**Capabilities:** -- Cross-database joins -- Temporal queries -- Relationship traversal -- Attribute filtering -- Metadata search - ---- - -## Engine Layer Components - -### AttributeProcessor - -**Purpose:** Processes and validates entity attributes before storage. - -**Key Functions:** -- Validates attribute data types -- Handles time-based attribute values -- Manages attribute schema evolution -- Coordinates with type and storage inference - -**Process Flow:** -``` -Input Attribute → Type Validation → Storage Determination → Database Storage -``` - -### TypeInference - -**Purpose:** Automatically determines data types for attribute values. - -**Supported Types:** -- `int` - Integer numbers -- `float` - Decimal numbers -- `string` - Text data -- `bool` - Boolean values -- `date` - Date values -- `time` - Time values -- `datetime` - Date and time values - -**Inference Rules:** -1. **Integer Detection:** Whole numbers without decimals -2. **Float Detection:** Numbers with decimal points or scientific notation -3. **Boolean Detection:** `true`, `false`, `1`, `0` -4. **Date Detection:** ISO date format patterns -5. **Time Detection:** Time format patterns -6. **String Fallback:** Any non-matching data - -### StorageInference - -**Purpose:** Determines optimal storage strategy for attributes. - -**Storage Types:** -- `SCALAR` - Single values (numbers, strings, booleans) -- `LIST` - Arrays of values -- `MAP` - Key-value pairs -- `TABULAR` - Table-like data with rows and columns -- `GRAPH` - Network data (not fully supported) - -**Inference Logic:** -``` -Data Structure Analysis → Storage Type Determination → Database Assignment -``` - -### GraphMetadataManager - -**Purpose:** Manages graph-specific metadata and relationships. Acts as a lookup -graph for entity and its data. - -**Key Functions:** -- Entity node creation and updates -- Relationship management -- Graph traversal optimization -- Temporal relationship handling - -## Related Documentation - -- [Architecture Overview](./index.md) - System architecture -- [Service APIs](./api-layer-details.md) - API documentation -- [Database Schemas](./database-schemas.md) - Database structures -- [How It Works](data_flow.md) - End-to-end data flow -- [Storage Types](../../appendix/storage.md) - Storage inference details diff --git a/docs_deprecated/overview/architecture/data_flow.md b/docs_deprecated/overview/architecture/data_flow.md deleted file mode 100644 index 112f0f8f..00000000 --- a/docs_deprecated/overview/architecture/data_flow.md +++ /dev/null @@ -1,306 +0,0 @@ -# How It Works: End-to-End Data Flow - -This document describes the complete workflow of how data flows through the system, from initial JSON input to final storage in the databases. - -## 1. Data Entry Point (Ingestion API) - -The system receives data through a REST API built with Ballerina. The API accepts JSON payloads for entity creation and updates. - -### Example JSON Input -```json -{ - "id": "entity123", - "kind": { - "major": "Person", - "minor": "Employee" - }, - "name": { - "startTime": "2024-01-01T00:00:00Z", - "endTime": "", - "value": "John Doe" - }, - "metadata": { - "department": "Engineering", - "role": "Software Engineer" - }, - "attributes": { - "expenses": { - "columns": ["type", "amount", "date", "category"], - "rows": [ - ["Travel", 500, "2024-01-15", "Business"], - ["Meals", 120, "2024-01-16", "Entertainment"], - ["Equipment", 300, "2024-01-17", "Office"] - ] - } - }, - "relationships": { - "reports_to": { - "relatedEntityId": "manager123", - "startTime": "2024-01-01T00:00:00Z", - "endTime": "", - "id": "rel123", - "name": "reports_to" - } - } -} -``` - -## 2. Data Transformation (Ingestion API → Core API) - -### 2.1 JSON to Protobuf Conversion -The Ingestion API converts the JSON payload into a protobuf Entity message. This conversion happens in the `convertJsonToEntity` function: - -#### Conversion Process Flowchart -``` -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ JSON TO PROTOBUF CONVERSION FLOW │ -└─────────────────────────────────────────────────────────────────────────────────┘ - - ┌──────────────────────────────────────────────────┐ - │ 📥 JSON Input │ - └─────────────────────┬────────────────────────────┘ - │ - ▼ - ┌─────────────────────────────────────────────────┐ - │ 🔍 Parse JSON │ - │ Payload │ - └─────────────────────┬───────────────────────────┘ - │ - ▼ - ┌─────────────────────────────────────────────────┐ - │ 📋 Extract Components │ - └────┬───────────┬────────────┬──────────┬────────┘ - │ │ │ │ - ▼ ▼ ▼ ▼ - ┌─────────┐ ┌──────────┐ ┌─────────┐ ┌─────────┐ - │ 🏷️ │ │ 📊 │ │ 🔗 │ │ 👤 │ - │ Metadata│ │Attributes│ │Relations│ │ Entity │ - │ │ │ │ │ │ │ Info │ - └─────┬───┘ └─────┬────┘ └─────┬───┘ └─────┬───┘ - │ │ │ │ - ▼ ▼ ▼ ▼ -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ PROCESSING │ -└─────────────────────────────────────────────────────────────────────────────────┘ - -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ 🏷️ METADATA PROCESSING │ -│ ┌─────────────────────────────────────────────────────────────────────────────┐ │ -│ │ Create metadataArray[] │ │ -│ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ -│ │ │ For each metadata entry: │ │ │ -│ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ -│ │ │ │ 1. Extract key-value pair │ │ │ │ -│ │ │ │ 2. Pack value with pbAny │ │ │ │ -│ │ │ │ 3. Push {key, packedValue} to metadataArray │ │ │ │ -│ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ -│ │ └─────────────────────────────────────────────────────────────────────────┘ │ │ -│ └─────────────────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────────────┘ - -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ 📊 ATTRIBUTES PROCESSING │ -│ ┌─────────────────────────────────────────────────────────────────────────────┐ │ -│ │ Create attributesArray[] │ │ -│ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ -│ │ │ For each attribute entry: │ │ │ -│ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ -│ │ │ │ Create TimeBasedValue[] array │ │ │ │ -│ │ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ │ -│ │ │ │ │ For each time-based value: │ │ │ │ │ -│ │ │ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ │ │ -│ │ │ │ │ │ 1. Create TimeBasedValue object │ │ │ │ │ │ -│ │ │ │ │ │ 2. Set startTime, endTime │ │ │ │ │ │ -│ │ │ │ │ │ 3. Pack value with pbAny │ │ │ │ │ │ -│ │ │ │ │ │ 4. Push to timeBasedValues array │ │ │ │ │ │ -│ │ │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ │ -│ │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ -│ │ │ │ Push {key, timeBasedValues} to attributesArray │ │ │ │ -│ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ -│ │ └─────────────────────────────────────────────────────────────────────────┘ │ │ -│ └─────────────────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────────────┘ - -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ 🔗 RELATIONSHIPS PROCESSING │ -│ ┌─────────────────────────────────────────────────────────────────────────────┐ │ -│ │ Create relationshipsArray[] │ │ -│ │ ┌─────────────────────────────────────────────────────────────────────────┐ │ │ -│ │ │ For each relationship entry: │ │ │ -│ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ -│ │ │ │ 1. Create Relationship object │ │ │ │ -│ │ │ │ 2. Set relatedEntityId, startTime, endTime, id, name │ │ │ │ -│ │ │ │ 3. Push {key, relationship} to relationshipsArray │ │ │ │ -│ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ -│ │ └─────────────────────────────────────────────────────────────────────────┘ │ │ -│ └─────────────────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────────────┘ - -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ 👤 ENTITY INFO PROCESSING │ -│ ┌─────────────────────────────────────────────────────────────────────────────┐ │ -│ │ 1. Extract id, kind (major, minor), name from jsonPayload │ │ -│ │ 2. Pack name value with pbAny │ │ -│ │ 3. Prepare startTimeValue, endTimeValue, namePackedValue │ │ -│ └─────────────────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────────────┘ - -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ COMBINE ALL COMPONENTS │ -└─────────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ 🏗️ CREATE FINAL ENTITY │ -│ ┌─────────────────────────────────────────────────────────────────────────────┐ │ -│ │ Entity { │ │ -│ │ id: jsonPayload.id │ │ -│ │ kind: {major: kindJson.major, minor: kindJson.minor} │ │ -│ │ name: {startTime: startTimeValue, endTime: endTimeValue, │ │ -│ │ value: namePackedValue} │ │ -│ │ metadata: metadataArray │ │ -│ │ attributes: attributesArray │ │ -│ │ relationships: relationshipsArray │ │ -│ │ } │ │ -│ └─────────────────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────────────┘ - │ - ▼ -┌─────────────────────────────────────────────────────────────────────────────────┐ -│ 📤 Return Entity|error │ -└─────────────────────────────────────────────────────────────────────────────────┘ -``` - -#### Conversion Details - -The conversion process involves several key transformations: - -1. **Metadata Processing**: Each metadata key-value pair is converted to a protobuf `Any` type using `pbAny:pack()` -2. **Attributes Processing**: Each attribute is converted to a `TimeBasedValueList` containing time-based values with `startTime`, `endTime`, and packed values -3. **Relationships Processing**: Each relationship is converted to a `Relationship` object with all required fields -4. **Entity Assembly**: All processed components are combined into the final `Entity` protobuf message - -The flowchart above shows the complete data transformation pipeline from JSON input to protobuf output. - -### 2.2 gRPC Communication -The converted protobuf message is sent to the CORE service via gRPC. The communication happens on port 50051. - -## 3. Core API - -The Core API receives the protobuf message and processes it through multiple steps: - -### 3.1 Create Entity Flow - -The Core API processes entity creation through a systematic three-step approach: - -**Step 1: Metadata Storage (MongoDB)** -- The system first saves the entity's metadata to MongoDB -- This includes all key-value pairs from the `metadata` field in the JSON payload -- MongoDB's flexible document structure allows for schema-less storage of metadata -- The entity ID serves as the document identifier for quick retrieval - -**Step 2: Entity Creation (Neo4j)** -- The core entity information is stored in Neo4j as a graph node -- This includes the entity ID, kind (major/minor), name, and timestamps -- Neo4j's graph structure enables efficient relationship traversal and queries -- The entity becomes a node in the graph database with the `Entity` label - -**Step 3: Relationship Management (Neo4j)** -- All relationships defined in the JSON payload are created in Neo4j -- Each relationship becomes a directed edge between entity nodes -- Relationship properties include start/end times and relationship metadata -- This enables complex graph queries and relationship traversal - -### 3.2 Data Storage - -#### MongoDB Storage (Metadata) - -The metadata is stored in MongoDB as a document: - -```json -{ - "_id": "entity123", - "metadata": { - "department": "Engineering", - "role": "Software Engineer" - } -} -``` - -#### Neo4j Storage (Entity and Relationships) - -The entity and its relationships are stored in Neo4j using a graph-based approach: - -**Entity Node Creation** -- Each entity becomes a node in the Neo4j graph with the `Entity` label -- The node contains core entity properties including: - - Unique entity identifier (`id`) - - Entity classification (`kind_major` and `kind_minor`) - - Entity name and display information - - Temporal information (`created` and `terminated` timestamps) -- This structure enables efficient entity lookup and management - -**Relationship Creation** -- Relationships between entities are represented as directed edges in the graph -- Each relationship has a specific type (e.g., `REPORTS_TO`, `MANAGES`, `WORKS_FOR`) -- Relationship properties include: - - Unique relationship identifier - - Temporal validity (`startTime` and `endTime`) - - Additional relationship metadata -- The directed nature allows for complex relationship traversal and queries - -**Graph Query Capabilities** -- The graph structure enables powerful relationship queries -- Users can traverse relationships in any direction -- Complex multi-hop queries are efficiently supported -- Temporal queries can find relationships active at specific time periods - -## 4. Data Retrieval Flow (Read API) - -### 4.1 Read Entity Flow - -The Core API retrieves entity data through a systematic three-step aggregation process: - -**Step 1: Metadata Retrieval (MongoDB)** -- The system first queries MongoDB to retrieve the entity's metadata -- This includes all key-value pairs associated with the entity ID -- MongoDB's document-based storage enables fast metadata lookup -- The metadata provides additional context and properties for the entity - -**Step 2: Entity Information Retrieval (Neo4j)** -- Core entity information is fetched from Neo4j using the entity ID -- This includes fundamental entity properties such as: - - Entity classification (kind major and minor) - - Entity name and display information - - Creation and termination timestamps -- Neo4j's graph structure enables efficient entity node lookup - -**Step 3: Attribute Retrieval (PostgreSQL)** -- Entity attributes are retrieved from PostgreSQL using the entity ID -- This includes all time-based attribute values stored in dynamic tables -- Attribute data includes: - - Attribute names and their corresponding values - - Temporal information (start and end times for each value) - - Data type information for proper value interpretation -- PostgreSQL's relational structure enables efficient attribute querying and time-based filtering - -**Step 4: Relationship Retrieval (Neo4j)** -- All relationships connected to the entity are retrieved from Neo4j -- This includes both incoming and outgoing relationships -- Relationship data includes: - - Related entity identifiers - - Relationship types and properties - - Temporal validity information -- The graph structure enables efficient relationship traversal - -**Data Aggregation and Response** -- All retrieved data is combined into a single entity response -- The system creates a comprehensive entity object containing: - - Core entity information from Neo4j - - Metadata from MongoDB - - All time-based attributes from PostgreSQL - - All associated relationships from Neo4j -- The aggregated data is returned as a complete entity representation with full temporal support - -### 4.2 Data Transformation (Core API → Ingestion API) -The retrieved data is converted back to JSON in the Ingestion API before being sent to the client. diff --git a/docs_deprecated/overview/architecture/database-schemas.md b/docs_deprecated/overview/architecture/database-schemas.md deleted file mode 100644 index 8a51729e..00000000 --- a/docs_deprecated/overview/architecture/database-schemas.md +++ /dev/null @@ -1,187 +0,0 @@ -# Database Schemas - Detailed Documentation - -This document provides comprehensive details about the database schemas used in OpenGIN across MongoDB, Neo4j, and PostgreSQL. - ---- - -## Overview - -OpenGIN uses a multi-database architecture where each database is optimized for specific data types: - -| Database | Purpose | Data Stored | -|----------|---------|-------------| -| MongoDB | Flexible metadata | Key-value metadata pairs | -| Neo4j | Graph relationships | Entity nodes and relationship edges | -| PostgreSQL | Structured attributes | Time-series attribute data with schemas | - ---- - -## MongoDB Schema - -### Database Information - -### Database Information - -The Metadata Store is implemented using a document-oriented database to provide flexibility for storing unstructured or semi-structured data. This allows for dynamic metadata fields without rigid schema constraints. - -### Collections - -#### 1. metadata - -**Purpose**: Store entity metadata as flexible key-value pairs - -**Schema** (document structure): -```javascript -{ - "_id": "entity123", // Entity ID (Primary Key) - "metadata": { // Metadata object - "key1": "value1", - "key2": "value2", - "key3": 123, - "key4": true, - "nested": { - "subkey": "subvalue" - } - }, - "created_at": ISODate("2024-01-01T00:00:00Z"), // Optional timestamp - "updated_at": ISODate("2024-01-01T00:00:00Z") // Optional timestamp -} -``` - - - -#### 2. metadata_test - -**Purpose**: Test collection for metadata (same schema as `metadata`) - -Used during testing to isolate test data from production data. - ---- - -## Neo4j Schema - -### Database Information - -The Graph Store is utilized to manage entities and their relationships. It supports both binary and HTTP protocols for interaction, enabling efficient graph traversals and complex relationship queries. - -### Node Types - -#### Entity Node - -**Label**: `:Entity` - -**Properties**: -```cypher -{ - id: String, // Unique entity identifier (REQUIRED) - kind_major: String, // Major entity classification (REQUIRED) - kind_minor: String, // Minor entity classification (optional) - name: String, // Entity name (REQUIRED) - created: String, // ISO 8601 timestamp (REQUIRED) - terminated: String // ISO 8601 timestamp (optional, null = active) -} -``` - -### Relationship Types - -**Dynamic Relationship System**: OpenGIN uses a completely generic relationship model where relationship types are not predefined. Users can create any relationship type they need by simply providing a `name` field in the relationship data. - -**How it works**: -1. User provides relationship with `name` field (e.g., "reports_to", "depends_on", "manages") -2. System dynamically creates Neo4j relationship with that type -3. Neo4j relationship type becomes the uppercased version or exact value of the `name` field -4. No schema validation or predefined list of relationship types - -#### Relationship Structure - -All relationships in Neo4j store the following properties: - -**Neo4j Properties** (what's actually stored in the graph): -```cypher -{ - Id: String, // Relationship identifier (uppercase I) - Created: DateTime, // When relationship started (Neo4j datetime type) - Terminated: DateTime // When relationship ended (Neo4j datetime type, null = active) -} -``` - -**Important**: The `name` field from the API/Protobuf becomes the **relationship TYPE** in Neo4j, not a property. It appears in the Cypher syntax as `[:relationshipType]`. - -**Note**: The `direction` field is not stored in Neo4j - it's determined by the direction of the arrow in the graph (→ for outgoing, ← for incoming). - -**Relationship Types**: -Relationship types are **completely dynamic and user-defined**. The system does not enforce any predefined relationship types. When creating a relationship, the `name` field from the `Relationship` protobuf message becomes the Neo4j relationship type. - -Examples from tests and usage: -- `reports_to`: Organizational hierarchy (from E2E tests) -- `depends_on`: Package dependencies (from unit tests) -- Any other name: Users can define any relationship type they need - ---- - -## PostgreSQL Schema - -### Database Information - -### Database Information - -The Attribute Store is built on a relational database system to manage structured, time-series attribute data. It ensures data integrity and supports complex querying capabilities through defined schemas. - -### Core Tables - -#### 1. Attribute Schema - -This table defines the structure of attributes for different entity kinds. It acts as a registry, specifying properties such as data types and storage strategies (e.g., scalar, list, map), ensuring consistent data handling for specific entity classifications. - -#### 2. Entity Attributes - -This table serves as a mapping layer, linking unique entity identifiers to the specific attribute definitions stored here. It allows the system to associate concrete data values with the defined structure for any given entity. - -### Dynamic Attribute Tables - -To optimize performance and organize data efficiently, the system automatically creates dedicated tables for attributes based on their entity classification. This approach ensures that attribute data is stored in a structured manner, allowing for faster retrieval and better data management compared to a single monolithic table. - - -### Data Integrity - -**No Distributed Transactions**: Currently, OpenGIN doesn't use distributed transactions. Each database operation is independent. - -**Eventual Consistency**: System relies on application-level consistency: -- Entity ID is the common key across all databases -- Core API orchestrates all operations -- Errors are logged but don't rollback previous successful operations - -**Future Enhancement**: Implement distributed transactions. - ---- - -## Backup and Restore - -### Metadata Store Backup - -```bash -mongodump --uri="mongodb://admin:@mongodb:27017/opengin?authSource=admin" \ - --out=/backup/mongodb/ -``` - -### Neo4j Backup - -```bash -neo4j-admin dump --database=neo4j --to=/backup/neo4j/neo4j.dump -``` - -### PostgreSQL Backup - -```bash -pg_dump -h postgres -U postgres -d opengin -F tar -f /backup/postgres/opengin.tar -``` - -See [Backup Integration Guide](../../appendix/operations/backup_integration.md) for complete backup/restore workflow. - -### Reference - -- [Data Models](../what_is_opengin.md#data-models) -- [Data Types](../../appendix/datatype.md) -- [Storage Types](../../appendix/storage.md) - ---- diff --git a/docs_deprecated/overview/architecture/getting-started.md b/docs_deprecated/overview/architecture/getting-started.md deleted file mode 100644 index e10887ac..00000000 --- a/docs_deprecated/overview/architecture/getting-started.md +++ /dev/null @@ -1,370 +0,0 @@ -# OpenGIN Architecture Documentation - -Welcome to the OpenGIN architecture documentation. This folder contains comprehensive documentation about the system's architecture, components, data flows, and design decisions. - ---- - -## Documentation Structure - -### 1. [Overview](./index.md) - **START HERE** -Complete system architecture overview including: -- High-level architecture diagram -- Layer-by-layer breakdown (API, Service, Database) -- Data model and storage strategy -- Data flow sequences -- Type and storage inference systems -- Technology stack -- Key features - -**Recommended for**: Everyone - developers, architects, stakeholders - ---- - -### 2. [Core API](./core-api.md) -In-depth documentation of the CRUD Service: -- Directory structure -- gRPC server implementation -- All service methods (CreateEntity, ReadEntity, UpdateEntity, DeleteEntity) -- Repository layer (MongoDB, Neo4j, PostgreSQL) -- Engine layer (AttributeProcessor, TypeInference, StorageInference) -- Configuration and environment variables -- Testing strategies - -**Recommended for**: Backend developers, service implementers - ---- - -### 3. [Service APIs](./api-layer-details.md) -Complete API layer documentation: -- Ingestion API (CREATE, UPDATE, DELETE operations) -- Read API (READ operations) -- Request/response formats -- JSON to Protobuf conversion -- OpenAPI contracts -- Swagger UI documentation -- Query parameters and filtering -- Temporal queries (activeAt) -- Best practices - -**Recommended for**: API consumers, frontend developers, integration engineers - ---- - -### 4. [Database Schemas](./database-schemas.md) -Detailed database schema documentation: -- **MongoDB**: Collections, document structures, indexes -- **Neo4j**: Node types, relationship types, Cypher queries -- **PostgreSQL**: Core tables, dynamic attribute tables, type mapping -- Cross-database consistency -- Schema evolution strategies (Not Implemented) -- Backup and restore procedures -- Performance optimization - -**Recommended for**: Database administrators, backend developers, data engineers - ---- - -## Quick Navigation - -### By Role - -**I'm a new developer** → Start with [Overview](./index.md), then [Diagrams](./diagrams.md) - -**I'm working on APIs** → Read [Service APIs](./api-layer-details.md) - -**I'm working on backend** → Read [Core API](./core-api.md) - -**I'm working on databases** → Read [Database Schemas](./database-schemas.md) - -**I'm presenting the architecture** → Use [Diagrams](./diagrams.md) and [Overview](./index.md) - -### By Task - -**Understanding data flow** → [Overview](./index.md) + [Diagrams](./diagrams.md) - -**Adding new endpoint** → [Service APIs](./api-layer-details.md) - -**Adding new entity type** → [Database Schemas](./database-schemas.md) + [Core API](./core-api.md) - -**Debugging data storage** → [Database Schemas](./database-schemas.md) - -**Performance tuning** → [Core API](./core-api.md) + [Database Schemas](./database-schemas.md) - -**Understanding types** → [Overview](./index.md) + [Core API](./core-api.md) - ---- - -## Key Concepts - -### Polyglot Database Strategy - -OpenGIN uses three databases, each optimized for specific data types: - -| Database | Purpose | Reason | -|----------|---------|--------| -| **MongoDB** | Metadata | Schema-less, flexible key-value storage | -| **Neo4j** | Entities & Relationships | Optimized graph traversal, Cypher queries | -| **PostgreSQL** | Attributes | ACID compliance, time-series data, strong typing | - -### Layered Architecture - -``` -Client Layer (HTTP/JSON) - ↓ -API Layer (Ingestion API, Read API) - ↓ gRPC/Protobuf -Service Layer (Core API) - ↓ Native Protocols -Database Layer (MongoDB, Neo4j, PostgreSQL) -``` - -### Time-Based Data - -All attributes and relationships support temporal tracking: -- `startTime`: When value/relationship became effective -- `endTime`: When value/relationship ended (NULL = current) -- `activeAt` queries: Retrieve data as it existed at specific point in time - -### Type and Storage Inference - -The system automatically: -1. **Infers data types**: int, float, string, bool, date, time, datetime -2. **Determines storage strategy**: SCALAR, LIST, MAP, TABULAR, GRAPH -3. **Creates schemas dynamically**: No manual schema definition needed - ---- - -## Architecture Principles - -### 1. Separation of Concerns -- APIs handle HTTP/JSON ↔ gRPC/Protobuf conversion -- Core API orchestrates business logic -- Repositories handle database-specific operations - -### 2. Database Specialization -- Each database does what it does best -- No single database bottleneck -- Independent scaling of each database - -### 3. Contract-First API Design -- OpenAPI specifications define APIs -- Code generated from contracts -- Swagger UI for documentation - -### 4. Type Safety -- Protobuf for internal communication -- Strong typing in Go (Core API) -- Type inference for flexibility - -### 5. Temporal Support -- All data versioned by time -- Historical queries supported -- Audit trail built-in - ---- - -## Common Patterns - -### Entity Creation Flow -``` -Client → Ingestion API → Core API → [MongoDB, Neo4j, PostgreSQL] → Response -``` - -### Entity Query Flow -``` -Client → Read API → Core API → Fetch from DBs based on output param → Response -``` - -### Selective Retrieval -``` -GET /v1/entities/{id}?output=metadata,relationships -``` -Only fetches requested fields, reducing load and improving performance. - -### Temporal Query -``` -GET /v1/entities/{id}/attributes?name=salary&activeAt=2024-03-15T00:00:00Z -``` -Returns attribute value as it was on specific date. - ---- - -## Technology Stack Summary - -| Layer | Technology | Language | -|-------|-----------|----------| -| Ingestion API | Ballerina | Ballerina | -| Read API | Ballerina | Ballerina | -| Core API | Go + gRPC | Go | -| MongoDB | MongoDB 5.0+ | - | -| Neo4j | Neo4j 5.x | Cypher | -| PostgreSQL | PostgreSQL 14+ | SQL | -| Messaging | Protobuf | IDL | -| Container | Docker + Compose | YAML | -| Testing | Various | Go, Ballerina, Python | - ---- - -## Architecture Diagrams at a Glance - -### System Architecture -``` -┌─────────────┐ -│ Clients │ -└──────┬──────┘ - │ HTTP/JSON -┌──────┴──────────────┐ -│ API Layer │ -│ Ingestion | Read │ -└──────┬──────────────┘ - │ gRPC/Protobuf -┌──────┴──────────────┐ -│ Core API │ -│ (Orchestration) │ -└──────┬──────────────┘ - │ Native Protocols -┌──────┴──────────────────────────┐ -│ MongoDB | Neo4j | PostgreSQL │ -│ Metadata| Graph | Attributes │ -└─────────────────────────────────┘ -``` - ---- - -## Development Workflow - -### 1. Understanding the System -- Read [Overview](./index.md) -- Understand data flow - -### 2. Setting Up Development Environment -- Clone repository -- Start databases: `docker-compose up -d mongodb neo4j postgres` -- Start CORE service: `cd opengin/core-api && ./core-service` -- Start APIs: Ingestion API (port 8080), Query API (port 8081) - -### 3. Making Changes - -**Adding new API endpoint**: -1. Update OpenAPI contract in `opengin/contracts/rest/` -2. Regenerate service code -3. Implement endpoint logic -4. Update [Service APIs](./api-layer-details.md) - -**Adding Core API feature**: -1. Implement in appropriate layer (server, engine, repository) -2. Add tests -3. Update [Core API Details](./core-api.md) - -**Modifying database schema**: -1. Consider impact across all databases -2. Update schema migration scripts -3. Update [Database Schemas](./database-schemas.md) - -### 4. Testing -- Unit tests: `go test ./...` or `bal test` -- Integration tests: E2E tests in `opengin/tests/e2e/` -- Database tests: Ensure all databases are running - -### 5. Documentation -- Update relevant architecture docs -- Add examples to appropriate sections -- Keep diagrams in sync with changes - ---- - -## Security Considerations - -### Current State -- Development mode: No authentication -- All endpoints publicly accessible -- No encryption in transit (within Docker network) - -### Planned Enhancements -- JWT authentication -- Role-based access control (RBAC) -- TLS/SSL for external communication -- API rate limiting -- Field-level access control - ---- - -## Monitoring and Observability - -### Logging -- Structured logging in all services -- Log aggregation (planned) -- Error tracking - -### Metrics (Planned) -- Request rates -- Response times -- Database query performance -- Error rates - -### Tracing (Planned) -- Distributed tracing with OpenTelemetry -- End-to-end request tracking - ---- - -## Contributing to Documentation - -### When to Update Documentation - -Update architecture docs when: -- Adding new services or components -- Changing data flow or structure -- Modifying database schemas -- Adding new features -- Changing APIs or contracts - -### Documentation Style - -- Clear, concise language -- Include code examples -- Add diagrams where helpful -- Cross-reference related docs -- Keep examples up to date - ---- - -## Related Documentation - -### In This Repository - -- [Main README](../../README.md) - Project overview and quick start -- [How It Works](data_flow.md) - Detailed data flow -- [Data Types](../../appendix/datatype.md) - Type inference system -- [Storage Types](../../appendix/storage.md) - Storage inference system -- [Deployment Guide](../../appendix/operations/backup_integration.md) - Backup and restore - -### External Resources - -- [Ballerina Documentation](https://ballerina.io/learn/) -- [gRPC Documentation](https://grpc.io/docs/) -- [Protocol Buffers](https://developers.google.com/protocol-buffers) -- [MongoDB Documentation](https://docs.mongodb.com/) -- [Neo4j Documentation](https://neo4j.com/docs/) -- [PostgreSQL Documentation](https://www.postgresql.org/docs/) -- [Docker Documentation](https://docs.docker.com/) - ---- - -## Version History - -| Version | Date | Changes | -|---------|------|---------| -| 1.0.0 - alpha | October 2024 | Initial comprehensive architecture documentation | - ---- - -## Contact and Support - -For questions about the architecture: -- Review this documentation first -- Check related service READMEs -- Review code comments and tests -- Consult the development team - ---- diff --git a/docs_deprecated/overview/architecture/index.md b/docs_deprecated/overview/architecture/index.md deleted file mode 100644 index 905bd42d..00000000 --- a/docs_deprecated/overview/architecture/index.md +++ /dev/null @@ -1,260 +0,0 @@ -# OpenGIN Architecture Overview - -## System Overview - -**OpenGIN** is a data orchestration and networking framework. It is based on a polyglot database and a microservices-based design that handles entities with metadata, attributes, and relationships. The architecture follows a layered approach with REST/gRPC communication protocols. - ---- - -## High-Level Architecture Diagram - - -![High-Level Architecture Diagram](assets/images/opengin-architecture-diagram.png) - - ---- - -## Architecture Layers - -### 1. API Layer (Client-Facing Services) - -#### Ingestion API - -The Ingestion API is responsible for handling entity mutations, including creation, updates, and deletions. Implemented as a Ballerina REST service, it accepts JSON payloads from clients, validates their structure, and converts them into Protobuf Entity messages. It then communicates with the Core API via gRPC and handles the conversion of Protobuf responses back to JSON for the client. - -#### Read API - -The Read API manages entity queries and retrieval. Also built as a Ballerina REST service, it accepts read requests and supports selective field retrieval, filtering, and search capabilities. It interfaces with the Core API using gRPC to fetch data and returns formatted JSON responses to the client. - -### 2. Service Layer - -#### Core API - -The Core API acts as the central orchestration service, managing data networking and interactions across the distributed database system. It exposes a gRPC server to handle entity operations such as creation, reading, updating, and deletion. - -Internally, the service utilizes an Engine Layer to process attributes, manage graph metadata, and perform type and storage inference. The Repository Layer abstracts the interactions with underlying databases, coordinating metadata storage in MongoDB, graph management in Neo4j, and attribute storage in PostgreSQL. - -### 3. Database Layer - -#### MongoDB -MongoDB is used for flexible metadata storage. Its document-based, schema-less structure allows for efficient handling of the dynamic metadata associated with entities. - -#### Neo4j -Neo4j serves as the specialized storage for entities and their relationships. By representing entities as nodes and relationships as directed edges, it optimizes the system for complex graph traversals and relationship-based queries. - -#### PostgreSQL -PostgreSQL provides robust storage for time-based attributes. It ensures ACID compliance and supports complex queries, making it ideal for managing time-series data and the evolution of attribute schemas. - -### 4. Supporting Services - -#### Cleanup Service -The Cleanup Service is a utility designed for database maintenance and testing. It provides a mechanism to clear PostgreSQL tables, drop MongoDB collections, and remove Neo4j nodes and relationships, facilitating a clean state for development and testing environments. - -#### Backup/Restore Service -The Backup and Restore Service ensures data persistence and version management. It handles the creation of local backups for all databases, stores them with versioning on GitHub, and supports automated restoration from specific backup releases. - ---- - -## Data Model - -### Entity Structure (Protobuf) - -```protobuf -message Entity { - string id = 1; // Unique identifier - Kind kind = 2; // major/minor classification - string created = 3; // Creation timestamp (ISO 8601) - string terminated = 4; // Optional termination timestamp - TimeBasedValue name = 5; // Entity name with temporal tracking - map metadata = 6; // Flexible metadata - map attributes = 7; // Time-based attributes - map relationships = 8; // Entity relationships -} - -message Kind { - string major = 1; // Primary classification - string minor = 2; // Secondary classification -} - -message TimeBasedValue { - string startTime = 1; // Value valid from - string endTime = 2; // Value valid until (empty = current) - google.protobuf.Any value = 3; // Actual value (any type) -} - -message Relationship { - string id = 1; // Relationship identifier - string relatedEntityId = 2; // Target entity - string name = 3; // Relationship type - string startTime = 4; // Relationship valid from - string endTime = 5; // Relationship valid until - string direction = 6; // Relationship direction -} -``` - -### Storage Distribution Strategy - -The entity data is strategically distributed across three databases: - -**Example Entity:** -```json -{ - "id": "entity123", - "kind": {"major": "Person", "minor": "Employee"}, - "name": "John Doe", - "created": "2024-01-01T00:00:00Z", - "metadata": {"department": "Engineering", "role": "Engineer"}, - "attributes": { - "expenses": { - "columns": ["type", "amount", "date", "category"], - "rows": [ - ["Travel", 500, "2024-01-15", "Business"], - ["Meals", 120, "2024-01-16", "Entertainment"], - ["Equipment", 300, "2024-01-17", "Office"] - ] - } - }, - "relationships": {"reports_to": "manager123"} -} -``` - -**Storage Distribution:** - -![Storage-Distribution-of-openGIN](assets/images/storage_distribution_opengin.png) - ---- - -## Data Flow Sequences - -### Create Entity Flow - -![Data-Flow-Sequence-Create](assets/images/data_flow_sequence_create.png) - -### Read Entity Flow - -![Data-Flow-Sequence-Read](assets/images/data_flow_sequence_read.png) - ---- - -## Type System - -### Type Inference System - -The Type Inference System automatically detects data types to eliminate the need for manual specification. It identifies primitive types such as integers, floats, strings, and booleans, as well as special temporal types like dates, times, and datetimes, assigning the appropriate type based on the input value's format. - -### Storage Type Inference - -The Storage Type Inference mechanism determines the optimal storage structure for data. It categorizes input into formats such as tabular (for data with rows and columns), graph (for nodes and edges), list, scalar, or map, ensuring efficient storage and retrieval. - ---- - -## Communication Protocols - -| Layer | Protocol | Format | Port | -|-------|----------|--------|------| -| Client ↔ Ingestion API | HTTP/REST | JSON | 8080 | -| Client ↔ Read API | HTTP/REST | JSON | 8081 | -| APIs ↔ Core API | gRPC | Protobuf | 50051 | -| Core API ↔ MongoDB | MongoDB Wire Protocol | BSON | 27017 | -| Core API ↔ Neo4j | Bolt Protocol | Cypher | 7687 | -| Core API ↔ PostgreSQL | PostgreSQL Wire Protocol | SQL | 5432 | - ---- - -## Network Architecture - -**Docker Network**: `ldf-network` (bridge network) -All services run within the same Docker network: -- Container-based service discovery -- Internal communication via container names -- Health checks ensure proper startup sequencing -- Volume persistence for data storage - -**Exposed Ports:** -- `8080` - Ingestion API (external access) -- `8081` - Read API (external access) -- `50051` - Core API (can be internal only) -- `27017` - MongoDB (development access) -- `7474/7687` - Neo4j (development access) -- `5432` - PostgreSQL (development access) - ---- - -## Deployment - -### Containerization -The system leverages Docker and Docker Compose for containerization, running all services within a shared bridge network. This setup ensures consistent environments and manages the persistence of database storage through volumes. - -### Health Checks -Integrated health checks are configured for all services to ensure proper startup sequencing. The Core API waits for databases to be ready, while the Ingestion and Read APIs wait for the Core API, ensuring a stable initialization process. - -### Service Orchestration -Services use dependency management to start in the correct order: databases initialize first, followed by the Core API, and finally the Ingestion and Read APIs. A default Docker Compose profile runs all core services, while a separate 'cleanup' profile is available to trigger the database cleanup service. - ---- - -## Technology Stack - -| Component | Technology | Language | Purpose | -|-----------|-----------|----------|---------| -| Ingestion API | Ballerina | Ballerina | REST API for mutations | -| Read API | Ballerina | Ballerina | REST API for queries | -| Core API | Go + gRPC | Go | Business logic orchestration | -| MongoDB | MongoDB 5.0+ | - | Metadata storage | -| Neo4j | Neo4j 5.x | - | Graph storage | -| PostgreSQL | PostgreSQL 14+ | - | Attribute storage | -| Protobuf | Protocol Buffers | - | Service communication | -| Docker | Docker + Compose | - | Containerization | -| Testing | Go test, Bal test, Python | Multiple | Unit & E2E tests | - ---- - -## Key Features - -### 1. Polyglot Database Strategy -- **Optimized Storage**: Each database serves its best use case -- **Data Separation**: Clear boundaries between metadata, entities, and attributes -- **Scalability**: Independent scaling of each database - -### 2. Time-Based Data Support -- **Temporal Attributes**: Track attribute values over time -- **Temporal Relationships**: Time-bound entity relationships -- **Historical Queries**: Query data at specific points in time (activeAt parameter) - -### 3. Type Inference -- **Automatic Detection**: No manual type specification required -- **Rich Type System**: Supports primitives and special types -- **Storage Optimization**: Determines optimal storage based on data structure - -### 4. Schema Evolution (Not Fully Supported) -- **Dynamic Schemas**: PostgreSQL tables created on-demand -- **Attribute Flexibility**: New attributes don't require migrations -- **Kind-Based Organization**: Attributes organized by entity kind - -### 5. Graph Relationships -- **Native Graph Storage**: Neo4j for optimal relationship queries -- **Bi-directional Support**: Forward and reverse relationship traversal -- **Relationship Properties**: Rich metadata on relationships - -### 6. Backup & Restore -- **Polyglot Database Backup**: Coordinated backups across all databases -- **Version Management**: GitHub-based version control -- **One-Command Restore**: Simple restoration from any version - -### 7. API Contract-First -- **OpenAPI Specifications**: APIs defined before implementation -- **Code Generation**: Service scaffolding from contracts -- **Documentation**: Swagger UI for interactive API docs - ---- - -## Related Documentation - -- [How It Works](data_flow.md) - Detailed data flow documentation -- [Data Types](../../appendix/datatype.md) - Type inference system details -- [Storage Types](../../appendix/storage.md) - Storage type inference details -- [Backup Integration](../../appendix/operations/backup_integration.md) - Backup and restore guide -- [Core API](../architecture/core-api.md) - Core API documentation -- [Service APIs](./api-layer-details.md) - Service APIs documentation - ---- diff --git a/docs_deprecated/overview/what_is_opengin.md b/docs_deprecated/overview/what_is_opengin.md deleted file mode 100644 index fdbe47b3..00000000 --- a/docs_deprecated/overview/what_is_opengin.md +++ /dev/null @@ -1,108 +0,0 @@ -# What is OpenGIN? - -Open General Information Network, hereafter referred to as **OpenGIN**, is an open-source platform designed to build a time-aware digital twin of an ecosystem by defining its entities, relationships, and data according to a specification. OpenGIN core supports a wide variety of data formats to provide efficient querying to simulate the digital twin. Underneath, OpenGIN uses a polyglot database definition which supports representing the changes of an ecosystem through time-traveling. - -## Data Model - -* **Temporal Data Model**: The use of `TimeBasedValue` enables temporal data management, allowing the system to track both when data is valid (business time) and when it was recorded (system time). -* **Flexible Schema**: The `metadata` field uses `google.protobuf.Any` to support arbitrary key-value pairs without schema constraints. -* **Immutable Core Fields**: Fields like `id`, `kind`, and `created` are read-only, ensuring data integrity. -* **Graph-Ready Structure**: The `relationships` field enables graph-based data modeling and traversal. -* **Storage Awareness**: The `attributes` of each entity can be stored in the most suitable storage format via a Polyglot database. - -## Entity Definition - -An **Entity** is defined by a set of core parameters: **Metadata**, **Relationships**, and **Attributes**. The core parameters are defined such that they are common to any data stored through the system. - -* The **Metadata** contains any unstructured data associated with an Entity. -* The **Relationships** refer to how an Entity is connected with other entities. -* The **Attributes** can be data in any format such as tabular, unstructured, graph, or blob. These can also be interpreted as the datasets owned by an entity. - -The format of the **Entity** is as follows: - -* `Id`: Unique Read-only identifier for the entity -* `Kind`: Read-only entity type classification -* `Created Time`: Read-only timestamp indicating entity creation time -* `Terminated Time`: Nullable timestamp indicating when the entity was terminated -* `Name` (**TimeBasedValue**): TimeBased value representing the entity's name -* `Metadata` (`map`): Flexible key-value map to store arbitrary metadata -* `Attributes` (`map>`): TimeBased attributes stored as lists -* `Relationships` (`map`): Relationships to other entities - -The core attributes of an **Entity** are *Id*, *Kind*, *Created Time*, *Terminated Time*, and *Name*. - -**Metadata** is very useful when we have to store unstructured values that can be subject to change from one entity to the other. **Attributes** are defined in such a way that they have the generic capability to store data of any storage type. - -**Relationships** are defined as a map in which the key is a unique identity and the value is a relationship. Note that the relationship contains the type (referred to as name) where one entity can have many relationships of the same type. This resembles the connections an entity has with other entities. The general idea of this specification is to provide a generic model to represent a workflow, an event, or static content in the real world. - -## Kind - -**OpenGIN** introduces a type system to interpret and represent the entities in a generalized manner. The type system defined in OpenGIN follows the definition of MIME (Multipurpose Internet Mail Extensions) and is named as **Kind**. - -**Kind** refers to a classification of various entities based on the nature of existence. It is defined by following the MIME type definition, where a major and a minor component together define a Kind. - -* `Major`: Base category of the type -* `Minor`: Sub-category of the type - -For instance, when we define an entity like a "Department of Education," the major of the Kind could be *Organization*, and the minor of the *Kind* could be *Department*. This information needs to be determined before creating a dataset for insertion. Once the major and the minor are selected for an entity, they cannot be changed once it is inserted into the system. - -## TimeBasedValue - -Any value except for immutable types is defined as a `TimeBasedValue`. This value has a start and an end time. One of the major purposes of OpenGIN is to record with time sensitivity. Also, the value of a record is defined as *Any* (protobuf) in order to support all data types and custom data types as decided by the user. - -It enables temporal versioning of values with the following fields: - -* `Start Time`: Timestamp when the value becomes active -* `End Time`: Timestamp when the value becomes inactive (nullable) -* `Value` (`Any`): Value to be stored of any type - -> **Example:** A `TimeBasedValue` would be: *Start Time=2025-01-10*, *End Time=N/A*, *Value=Facebook Handle of a user*. - -## Metadata - -**Metadata** in OpenGIN provides a flexible mechanism to store unstructured, key-value data associated with entities. Unlike **Attributes** which are time-based and stored in PostgreSQL, metadata is schema-less and stored in MongoDB, making it ideal for storing arbitrary information that doesn't require temporal tracking or complex querying. - -Metadata is defined as a `map` where: -* **Key**: A string identifier for the metadata field -* **Value**: Any protobuf `Any` type, allowing for maximum flexibility - -## Relationship - -**Relationship** defines the connection between two entities. Any parameter that changes with time is easy to process with OpenGIN. Likewise, a Relationship also contains the temporal values in the definition along with a direction. - -A Relationship can be defined from one entity to another only in one direction, but it could be queried as an incoming or an outgoing relationship. "Incoming" refers to a relationship originated from another entity towards the referred entity, and "outgoing" refers to that of the opposite. - -* `Id`: Unique identifier for the relationship -* `Related Entity Id`: ID of the related entity -* `Name`: Name or type of the relationship -* `Start Time`: Timestamp when the relationship begins -* `End Time`: Timestamp when the relationship ends (nullable) -* `Direction`: Direction of the relationship (Incoming or Outgoing) - -**Example:** A Relationship definition could include a scenario where we need to define a relationship between an organization and its employees. The entities here are *Employee* and *Organization*. The Relationship is *HIRED_AS* at a given time. The start time refers to the moment this employee gets into the organization. When that employee no longer continues to work, this relationship comes to an end, and the end time is updated. - -## Attribute - -OpenGIN considers that an **Entity** has a sense of belonging to data that originated through it or which are part of its core definition. To represent this, OpenGIN supports various storage types since various data can take various formats. Thus, one of the main objectives of OpenGIN is to provide a variety of storage formats. - -This is motivated by two main reasons: -1. **Storage Representation:** Representing entities and their connections in a traditional primary-key and foreign-key approach through a tabular data storage format may not be practical when those connections get denser (Vicknair et al., 2010). At scale, this implies the usage of a high-performance graph database. -2. **Efficient Data Ownership:** There is a necessity to efficiently store various datasets owned by each entity. These attributes can be in various forms, such as structured, unstructured, graph, or blob data. - -From the aforementioned cases, the necessity of a **polyglot database** is justified. - -OpenGIN automatically detects and classifies the core storage types when attributes are entered into the system. The system uses a hierarchical detection approach with the following precedence order for the four core storage types: - -1. Graph -2. Tabular -3. Document -4. Blob[^1] - -[^1]: *Blob storage format has not yet been released.* - -## Key Features - -- **Temporal Data**: Native support for time-based values (startTime, endTime) for attributes and relationships. -- **Graph Capabilities**: Powerful relationship traversal and querying. -- **Scalability**: Microservices architecture allows independent scaling of components. -- **Strict Contracts**: Uses Protobuf for internal communication and OpenAPI for external REST APIs. diff --git a/docs_deprecated/tutorial/simple_app.md b/docs_deprecated/tutorial/simple_app.md deleted file mode 100644 index a30d31ef..00000000 --- a/docs_deprecated/tutorial/simple_app.md +++ /dev/null @@ -1,85 +0,0 @@ -# Building a Personal Life Ecology Application - -In this tutorial, we will build a Python application that models a "Personal Life Ecology". Instead of a single entity, we will create a rich graph of connected entities representing a person, their relationships, and their financial activities. - -This tutorial uses the code and data provided in the `tutorials/` directory of the OpenGIN repository. - -## Scenario - -We are modeling the life of **Alex**, a software engineer. -- **Central Entity**: Alex (Person/Individual) -- **Relationships**: - - Works at **TechCorp** - - Lives at **Home** - - Exercises at **FitGym** - - Frequents **Espresso Corner** (Cafe) - - Shops at **FreshMart** (Supermarket) - - Subscribed to **FiberNet** (ISP) -- **Tabular Data**: - - **Expenses**: Monthly spending record linked to entities. - - **Income**: Monthly income record linked to employers/sources. - -## Project Structure - -Navigate to the `tutorials/` folder to see the implementation: - -- **`my_first_opengin_app.py`**: The Python script that performs the ingestion. -- **`data/`**: Contains the source data. - - `entities.csv`: Definitions of Alex and related organizations/places. - - `relationships.csv`: Graph edges connecting the entities. - - `expenses.csv` & `income.csv`: Tabular financial data. - - `metadata.json`: Additional key-value pairs for entities. - -## Implementation Details - -The application script (`my_first_opengin_app.py`) demonstrates best practices for ingesting complex data into OpenGIN. - -### 1. Two-Phase Ingestion Strategy -To satisfy referential integrity (you can't link to an entity that doesn't exist yet), the script uses a **2-pass strategy**: - -1. **Phase 1 (Creation)**: Iterate through `entities.csv` and create all entities using `POST /entities`. At this stage, we ingest Metadata and Attributes (like Expenses tables), but *exclude* relationships. -2. **Phase 2 (Linking)**: Iterate again and use `PUT /entities/{id}` to update the entities, adding their **Relationships**. We strictly exclude immutable fields like `Kind` during this update. - -### 2. Handling Tabular Attributes -The script reads `expenses.csv` using pandas and converts it into OpenGIN's tabular attribute format: -```json -"attributes": { - "expenses": { - "values": [ - { - "startTime": "...", - "value": { - "columns": ["Month", "Category", "Amount"], - "rows": [["Jan", "Internet", "50"], ...] - } - } - ] - } -} -``` - -## Running the Tutorial - -### Prerequisites -- OpenGIN running locally (Ingestion API at port 8080, Read API at port 8081). -- Python 3.8+ with `pandas` and `requests` installed (`pip install pandas requests`). - -### Commands - -The script provides a CLI for easy interaction. - -**1. Create Data (Ingest)** -```bash -python tutorials/my_first_opengin_app.py create -``` -*Output: You will see "Creating Entity..." logs for Phase 1 and "Linking Entity..." logs for Phase 2.* - -**2. Verify Data** -```bash -python tutorials/my_first_opengin_app.py verify -``` -*Output: This queries the Read API to confirm "Person" entities were created successfully.* - -## Next Steps -- Explore the `tutorials/data/` CSV files to see how the data is structured. -- Modify `my_first_opengin_app.py` to add a new entity (e.g., a "Friend") and link them to Alex.