Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
98e0357
Initial commit with task details for issue #18
konard Sep 10, 2025
98a2819
Remove CLAUDE.md - PR created successfully
konard Sep 10, 2025
56003b7
feat: Add comprehensive Abstract Wikipedia analysis and lessons learned
konard Sep 10, 2025
e13c141
docs: Add fresh 2025-2026 Abstract Wikipedia research and feature-gap…
konard Jun 9, 2026
2dadfaa
Merge remote-tracking branch 'origin/main' into issue-18-ef8b353c
konard Jun 10, 2026
5d434f3
feat(wikidata): add batch getLabels and searchLexemes to the API client
konard Jun 10, 2026
bfd32cc
feat(generation): add Q/P -> text reverse renderer (Abstract Wikipedia)
konard Jun 10, 2026
2403352
feat(transformer): emit typed constructors with negation and tense
konard Jun 10, 2026
99fe6f3
feat(app): add Generation SPA mode and redirect shell
konard Jun 10, 2026
32deb09
feat(pkg): export generation surface and bump to 0.2.0
konard Jun 10, 2026
cb038ee
test: cover generation renderer (offline unit + e2e) and update mode …
konard Jun 10, 2026
6a0e896
docs: add issue-18 case study, mark closed gaps, document Generation …
konard Jun 10, 2026
8e258cf
docs: link Generation demo to deployed app.html#mode=generation (avoi…
konard Jun 10, 2026
0c0287e
feat(generation): inflect tense on the copula where grammatical
konard Jun 10, 2026
adfe86d
test(generation): cover the apiClient.getLabels render path offline
konard Jun 10, 2026
641603f
fix(generation): agree es/fr indefinite article with grammatical gender
konard Jun 10, 2026
5a736a5
feat(transformation): structurally handle questions, quantities, repe…
konard Jun 10, 2026
2136624
feat(generation): add quantity (measurement) constructor across UN6
konard Jun 10, 2026
9629d1d
test(limitations): credit structural handling, 12 → 4 problems
konard Jun 10, 2026
2d128d1
feat(app): expose gender agreement and the quantity constructor in Ge…
konard Jun 10, 2026
a7b46ef
docs: document the quantity constructor, Romance gender, and analysis…
konard Jun 10, 2026
34ebeb0
fix(generation): inflect the Russian located_in object to the preposi…
konard Jun 10, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 27 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,9 @@ The whole project is published to GitHub Pages as a single unified single-page a
| [Entities](https://link-assistant.github.io/human-language/app.html#mode=entity) | Browse any Wikidata Q-id, with IPA toggle and an inline test runner. | [`js/src/app/modes/entity.jsx`](js/src/app/modes/entity.jsx) |
| [Properties](https://link-assistant.github.io/human-language/app.html#mode=property) | Same lens as Entities but for P-ids. | [`js/src/app/modes/property.jsx`](js/src/app/modes/property.jsx) |
| [Text → Q/P Transformer](https://link-assistant.github.io/human-language/app.html#mode=transformer) | Turn English text into a sequence of Wikidata entities (Q) and properties (P), with n-gram support. | [`js/src/app/modes/transformer.jsx`](js/src/app/modes/transformer.jsx) |
| [Q/P → Text Generation](https://link-assistant.github.io/human-language/app.html#mode=generation) | The reverse direction: render a typed constructor (subject · predicate · object, or a subject · value · unit measurement, with negation, tense, Romance gender agreement and Russian prepositional case) into sentences across the six official UN languages. | [`js/src/app/modes/generation.jsx`](js/src/app/modes/generation.jsx) |

The legacy URLs (`entities.html`, `properties.html`, `transformation/index.html`, and bare hashes like `entities.html#Q35120`) still work — they now redirect into the unified SPA preserving any parameters.
The legacy URLs (`entities.html`, `properties.html`, `transformation/index.html`, `generation/index.html`, and bare hashes like `entities.html#Q35120`) still work — they now redirect into the unified SPA preserving any parameters.

### Internal pages (for contributors)

Expand Down Expand Up @@ -62,28 +63,36 @@ This project will fundamentally transform how we store, access, and verify human
- **Real-time transformation**: Interactive web demo at `transformation/index.html`
- [Learn more →](transformation/README.md)

### 2. Entity & Property Viewer
### 2. Q/P-to-Text Generation (reverse renderer)
- **Closes the round-trip**: the reverse of the transformer — typed meaning → natural-language text
- **Abstract-Wikipedia-style constructors**: typed containers (`instance_of`, `located_in`, `relation`, `quantity`) with named roles — including a `quantity` measurement constructor (`subject` · `value` · `unit`) mirroring a Wikidata quantity claim
- **Multi-language rendering**: one templatic renderer per constructor-per-language across the six official UN languages (en, ar, es, fr, ru, zh)
- **Grammatical features**: negation, tense, English `a`/`an` indefinite-article phonotactics, Romance gender agreement (es `un`/`una`, fr `un`/`une`), and Russian prepositional-case inflection for the locative object (Германия → Германии)
- **Offline-friendly**: role values may be Wikidata ids (resolved to labels) or plain text (rendered verbatim, no network)
- Interactive web demo at `app.html#mode=generation`; library entry point [`human-language/generate`](js/src/generation/qp-to-text.js)

### 3. Entity & Property Viewer
- **Beautiful UI**: Modern, responsive interface with dark/light themes
- **Multi-language support**: Automatic language detection and switching
- **Rich statements display**: View all properties and relationships
- **Direct Wikidata links**: Seamlessly navigate to source data
- View entities at `entities.html` and properties at `properties.html`

### 3. Advanced Search & Disambiguation
### 4. Advanced Search & Disambiguation
- **Exact & fuzzy matching**: Find entities even with typos
- **Context-aware ranking**: Domain and type preferences
- **Batch searching**: Efficient parallel searches
- **Multi-language search**: Search in any supported language
- [API Documentation →](SEARCH_README.md)

### 4. Intelligent Caching System
### 5. Intelligent Caching System
- **Multi-tier caching**: File system (Node.js) and IndexedDB (browser)
- **Automatic fallback**: Seamless switching between cache types
- **Performance optimized**: Reduces API calls and improves response times
- **Cross-platform**: Works in both Node.js and browser environments
- **Persistent storage**: Cached data survives across sessions

### 5. Comprehensive Language Support
### 6. Comprehensive Language Support
- **100+ languages**: Full support for all major Wikidata languages
- **Locale-specific quotes**: Proper quotation marks for each language
- **Flag emojis**: Visual language indicators for better UX
Expand Down Expand Up @@ -179,18 +188,25 @@ Based on our [GitHub issues](https://github.com/link-assistant/human-language/is
- N-gram generation and matching
- Parallel search execution
- Priority-based result merging
- Typed-constructor output (`transformToConstructor`) with negation, tense, question (`detectQuestion`) and quantity (`extractQuantities`) detection, plus adjacent-duplicate collapsing (`dedupeSequence`)

3. **Q/P → Text Renderer** (`js/src/generation/qp-to-text.js`, `js/src/generation/constructors.js`)
- Typed constructors with named roles (Abstract-Wikipedia-style), including a `quantity` measurement constructor
- One templatic renderer per constructor-per-language across the UN 6 languages
- Negation, tense, English `a`/`an` phonotactics, Romance gender agreement (`un`/`una`, `un`/`une`) and Russian prepositional-case inflection on the locative object (`russianPrepositional`)
- Batch label resolution via the Wikidata client (`getLabels`), pluggable for offline use

3. **Search Utilities** (`js/src/wikidata-api.js`)
4. **Search Utilities** (`js/src/wikidata-api.js`)
- Exact and fuzzy search algorithms
- Context-aware ranking system
- Multi-language support

4. **Caching System** (`js/src/unified-cache.js`)
5. **Caching System** (`js/src/unified-cache.js`)
- Factory pattern for cache creation
- File system cache for Node.js
- IndexedDB cache for browsers

5. **UI Components** (`js/src/statements.jsx`, `js/src/loading.jsx`)
6. **UI Components** (`js/src/statements.jsx`, `js/src/loading.jsx`)
- React 19 components with JSX
- No build step required (Babel in-browser)
- Responsive and theme-aware design
Expand Down Expand Up @@ -259,6 +275,7 @@ See the [🎬 Demos](#-demos) section above for the full table — every demo is
- **Entity Viewer** — [`entities.html`](https://link-assistant.github.io/human-language/entities.html)
- **Property Viewer** — [`properties.html`](https://link-assistant.github.io/human-language/properties.html)
- **Text Transformer** — [`transformation/index.html`](https://link-assistant.github.io/human-language/transformation/index.html)
- **Q/P → Text Generation** — [`app.html#mode=generation`](https://link-assistant.github.io/human-language/app.html#mode=generation)
- **N-gram Test** — [`transformation/test-ngram.html`](https://link-assistant.github.io/human-language/transformation/test-ngram.html)
- **Search Demo** — [`search-demo.html`](https://link-assistant.github.io/human-language/search-demo.html)
- **Caching Demo** — [`cache-demo.html`](https://link-assistant.github.io/human-language/cache-demo.html)
Expand All @@ -269,9 +286,9 @@ See the [🎬 Demos](#-demos) section above for the full table — every demo is

The text transformation system currently has some limitations:

1. **Negation handling**: Phrases with "not" aren't properly processed
1. **Negation handling**: The raw Q/P sequence does not encode negation — but `transformToConstructor` now detects it and the generation renderer expresses it ("X is not a Y")
2. **Question parsing**: Direct questions (who, what, when) aren't supported
3. **Verb tenses**: Past/future tenses may not be accurately captured
3. **Verb tenses**: The raw sequence drops tense — `transformToConstructor` detects past/present/future and the renderer inflects the copula where grammatical (e.g. English "X was a Y", "X will be a Y", and the Spanish/French/Russian/Arabic equivalents). Languages that need noun-case morphology for a given tense (e.g. Russian/Arabic *instance_of* past) fall back to the present form, pending the Wikidata Lexeme integration tracked in `research/`
4. **Pronoun resolution**: Cannot resolve pronouns like "he", "she", "it"
5. **Complex sentences**: Struggles with subordinate clauses

Expand Down
10 changes: 10 additions & 0 deletions app.html
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@
import * as ipa from './js/src/app/ipa.js';
import { TextToQPTransformer } from './js/src/transformation/text-to-qp-transformer.js';
import { TextTransformerTest, demonstrateTransformer } from './js/src/transformation/text-transformer-test.js';
import { QPRenderer } from './js/src/generation/qp-to-text.js';
import { CONSTRUCTORS, UN6_LANGUAGES, LANGUAGE_NAMES, buildConstructor, validateConstructor }
from './js/src/generation/constructors.js';

// Shared globals expected by the unified shell and legacy components
// (statements.jsx, loading.jsx) that we re-mount inside the SPA.
Expand Down Expand Up @@ -70,6 +73,12 @@
TextToQPTransformer,
TextTransformerTest,
demonstrateTransformer,
QPRenderer,
CONSTRUCTORS,
UN6_LANGUAGES,
LANGUAGE_NAMES,
buildConstructor,
validateConstructor,
modes: {},
};
</script>
Expand All @@ -88,6 +97,7 @@
<script type="text/babel" src="./js/src/app/modes/entity.jsx"></script>
<script type="text/babel" src="./js/src/app/modes/property.jsx"></script>
<script type="text/babel" src="./js/src/app/modes/transformer.jsx"></script>
<script type="text/babel" src="./js/src/app/modes/generation.jsx"></script>

<!-- Mount the SPA. We wait until the next macrotask so every text/babel
script above has been transformed and executed before App reads
Expand Down
81 changes: 81 additions & 0 deletions docs/case-studies/issue-18/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Case Study — Issue #18

> **Issue:** [link-assistant/human-language#18](https://github.com/link-assistant/human-language/issues/18)
> "Learn from Abstract Wikipedia development"
> **Pull request:** [#19](https://github.com/link-assistant/human-language/pull/19)
> **Branch:** `issue-18-ef8b353c`

## Summary

Issue #18 asked us to learn from the [Abstract Wikipedia /
Wikifunctions](https://meta.wikimedia.org/wiki/Abstract_Wikipedia)
project. The first half of the work (see
[`research/`](../../../research)) gathered fresh 2025–2026 research and a
feature-gap analysis. This half acts on that analysis: it **implements**
the most valuable missing capability the research surfaced — the reverse
of our existing Text → Q/P transformer.

Until now the project could only do *analysis* (English text → a sequence
of Wikidata entities and properties). Abstract Wikipedia's core idea is
the other direction: hold meaning in a language-independent, typed form
and *render* it into natural language in many languages. That direction —
**generation** — is what closes the round-trip the README vision depends
on ("Language of Meaning", "Zero-Cost Translation").

## What shipped

A new **Generation** service (Q/P → text), built in the project's
established style — pure data layer, an injectable renderer, offline-first
unit tests, package subpath exports, hand-written type declarations, a
unified-SPA mode, a legacy redirect shell, and e2e coverage.

| Layer | File |
| --- | --- |
| Typed constructors + multi-language templates (pure, dependency-free) | [`js/src/generation/constructors.js`](../../../js/src/generation/constructors.js) |
| The renderer (`QPRenderer`) | [`js/src/generation/qp-to-text.js`](../../../js/src/generation/qp-to-text.js) |
| Batch labels + lexeme search on the Wikidata client | [`js/src/wikidata-api.js`](../../../js/src/wikidata-api.js), [`js/src/wikidata-api-browser.js`](../../../js/src/wikidata-api-browser.js) |
| Typed-constructor output from the transformer (negation/tense) | [`js/src/transformation/text-to-qp-transformer.js`](../../../js/src/transformation/text-to-qp-transformer.js) |
| SPA mode | [`js/src/app/modes/generation.jsx`](../../../js/src/app/modes/generation.jsx) |
| Legacy redirect shell | [`generation/index.html`](../../../generation/index.html) |
| Unit tests (offline) | [`js/tests/unit/qp-to-text.test.mjs`](../../../js/tests/unit/qp-to-text.test.mjs) |
| e2e tests | [`js/tests/e2e/app.spec.mjs`](../../../js/tests/e2e/app.spec.mjs) |

## Documents in this case study

| File | Purpose |
| --- | --- |
| [`requirements.md`](./requirements.md) | Every requirement extracted from the issue and its PR comments, with status. |
| [`architecture.md`](./architecture.md) | How the generation service is structured and which Abstract-Wikipedia concepts it maps to. |

## The Abstract-Wikipedia concepts we adopted

* **Constructors** — typed containers with named argument *roles*
(`instance_of`, `located_in`, `relation`, and the `quantity`
measurement constructor `subject` · `value` · `unit`). Our
`CONSTRUCTORS` catalogue mirrors this.
* **Templatic renderers** — one template per constructor *per language*.
We ship the six official UN languages (en, ar, es, fr, ru, zh), the
same demonstration set Abstract Wikipedia uses for its first
natural-language-generation "semantic fragments" (e.g. Z26039 "Berlin
is a city").
* **Grow the grammar incrementally** — we deliberately start with a
handful of high-value constructors plus English `a`/`an` phonotactics,
Romance gender agreement (es `un`/`una`, fr `un`/`une`, driven by
the object noun's grammatical gender — Wikidata P5185) and Russian
prepositional-case inflection on the locative object (Германия →
Германии) as the first grammatical features, exactly the way Abstract
Wikipedia grew its grammar.

## Round-trip in one example

```
"Berlin is a city" (English text)
→ transformToConstructor → { type: 'instance_of', subject: 'Berlin', object: 'city' }
→ QPRenderer.renderAll →
en: Berlin is a city es: Berlin es un city
fr: Berlin est un city ru: Berlin — city
zh: Berlin是city ar: Berlin city
```

(With Wikidata ids in the roles instead of plain text, each label is
resolved in the target language before rendering.)
Loading
Loading