A full-stack project for exploring and visualizing entity-relationship graphs from Wikidata using a custom force-directed layout. The project consists of a Python backend for crawling, cleaning, and serving Wikidata graphs, and a TypeScript/Vite frontend for interactive visualization.
Stable 60fps up to 250 entities
Deployed github page: https://kieroxide.github.io/Wikidata-Entity-Graph-Visualiser/
- SPARQL and Wikidata API: Gained hands-on experience with SPARQL queries, batching, and handling the quirks of the Wikidata endpoint.
- Graph Data Cleaning: Learned the importance of cleaning and validating graph data, including handling missing labels, disconnected nodes, and invalid references.
- Efficient Optional Caching: Implemented caching to avoid redundant API calls and speed up repeated queries. However this is turned off in production as to not waste space on the server.
- RESTful API Design: Built a robust Flask API with clear endpoints and flexible parameters for graph exploration.
- Raw Canvas Rendering: Built a force-directed graph visualizer from scratch using only TypeScript and the HTML5 Canvas API—no d3, no gl-matrix, just custom logic and a Rust/WASM module for physics.
- UI/UX for Graphs: Designed intuitive controls for panning, zooming, searching, and expanding graphs, making large and complex data feel interactive and approachable.
- Frontend-Backend Integration: Learned to coordinate async data fetching, error handling, and real-time updates between the frontend and a Flask backend.
- Performance Optimization: Managed animation frame queues, optimized rendering, and handled large graphs smoothly in the browser.
- Deployment: Automated deployment to Railway using Gunicorn, and made the server auto-detect its environment for seamless local and cloud use.
- Testing and Debugging: Developed backend tests to check for data consistency, connectivity, and filtering correctness.
- Vite & Modern Tooling: Used Vite for fast development and deployment, and set up proxying for seamless local development.
- Wikidata Rate Limiting: Had to tune worker counts and batch sizes to avoid being throttled or blocked by the Wikidata SPARQL endpoint.
- Data Inconsistencies: Encountered missing or malformed labels, orphaned relations, and other real-world data issues that required robust cleaning logic.
- Graph Expansion Control: It was difficult to precisely control the number of entities/relations due to the unpredictable nature of graph crawling and filtering.
- Deployment Path Issues: Ensured that config and data files were always found regardless of local or cloud deployment paths.
- Maintaining Consistent Data Formats: Needed to keep entity and property data formats consistent between cache, API, and frontend expectations.
- Debugging in Production: Diagnosed and fixed issues that only appeared after deployment, such as environment variable handling and file path mismatches.
- No Visualization Libraries: Everything from graph layout to rendering and interaction was implemented manually, which was both challenging and rewarding.
- WASM Integration: Integrated a Rust/WASM module for efficient force calculations, learning about cross-language module loading and performance tuning.
- Responsive UI: Ensured the app works well on different screen sizes and handles window resizing gracefully.
- Error Feedback: Built user feedback for loading, errors, and invalid input to make the app robust and user-friendly.
- Frontend: TypeScript + Vite + HTML5 Canvas + Rust/WASM (for physics)
- Backend: Python (Flask) for crawling, cleaning, and serving Wikidata entity graphs
- Purpose: Educational exploration of how ideas and entities are connected in Wikidata, with a focus on clean, interactive, and performant graph visualization
- Search for any Wikidata entity and visualize its connections
- Interactive force-directed graph layout (pan, zoom, expand nodes)
- Adjustable graph depth and relation limits
- Real-time stats and user feedback
- Data cleaning and filtering for high-quality graphs
- REST API for programmatic access to graph data
- WASM-accelerated physics for smooth, large-graph rendering
- Easy deployment (local or cloud)
- Offscreen sprite caching for vertices: vertex visuals (labels, thumbnails) are pre-rendered to offscreen canvases so the main paint loop issues far fewer expensive text/shape draws. This dramatically reduces "painting" time in DevTools and keeps animation smooth at high node counts.
- Batched edge rendering: edges are grouped (by stroke style/color) and drawn in batches to minimize context state changes and stroke() calls.
- Hover-only labels and LOD: labels and rich text are rendered only for hovered or nearby nodes, while a lighter "simple mode" is used at low zoom levels to keep the scene readable and fast.
- Offscreen edge cache: when the graph and camera are static, edge geometry is cached to an offscreen canvas and re-used across frames to avoid re-stroking thousands of paths.
- Dynamic borders: animated vertex borders are rendered separately from cached sprites to avoid recreating sprites each frame while preserving smooth animations.
- WASM for physics: the Rust/WASM module performs force calculations efficiently so the browser can dedicate more cycles to rendering and interaction.
These changes were targeted at reducing the browser "painting" phase (the most expensive step observed in profiling) and have been validated during development — stable 60 FPS is achievable for moderate graphs (~250 nodes) and the app scales gracefully when many nodes are off-screen.
- Added offscreen sprite caching and batching to reduce paint time in the browser.
- Implemented hover-only labels and a robust simple/detailed LOD rendering mode.
- Improved backend SPARQL reliability with per-entity limits and exponential backoff with jitter.
FDG/backend/— Python backend (Flask API, data crawling, cleaning, caching)FDG/frontend/— TypeScript/Vite frontend (UI, rendering, WASM integration)docs/— Static build output for GitHub PagesREADME.md— This fileLICENSE— MIT License
- Python 3.10+
- Node.js (v18+ recommended)
- Rust (for building WASM, optional if using prebuilt)
cd FDG/backend
pip install -r requirements.txt
python WikiGraphServer.pyThe API will be available at http://localhost:5000.
cd FDG/frontend
npm install
npm run devThe app will be available at http://localhost:5173.
npm run start:both- Backend crawls Wikidata, cleans and caches entity/property/relation data, and serves it via a REST API.
- Frontend fetches graph data, renders it as a force-directed graph, and provides interactive controls for exploration.
- WASM module (Rust) accelerates force calculations for smooth, real-time graph layouts.
- Python (Flask, requests, SPARQL)
- TypeScript, Vite
- HTML5 Canvas
- Rust/WASM (for physics)
- GitHub Pages (for static frontend hosting)
- Backend:
FDG/backend/config.jsonfor crawling/filtering options - Frontend:
FDG/frontend/vite.config.tsand.envfor API endpoints
MIT License (see LICENSE file)
Created by Kieran B, 2025. See individual README.md files in FDG/backend and FDG/frontend for more details.