Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,16 +29,16 @@ jobs:

- name: Lint with ruff
run: |
ruff check tableqa tests
ruff check statqa tests

- name: Type check with mypy
run: |
mypy tableqa --ignore-missing-imports
mypy statqa --ignore-missing-imports
continue-on-error: true

- name: Run tests with pytest
run: |
pytest --cov=tableqa --cov-report=xml --cov-report=term-missing
pytest --cov=statqa --cov-report=xml --cov-report=term-missing

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/python-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:

environment:
name: pypi
url: https://pypi.org/p/tableqa
url: https://pypi.org/p/statqa

steps:
- name: Download distribution packages
Expand Down
86 changes: 43 additions & 43 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# TableQA
# StatQA

[![CI](https://github.com/gojiplus/tableqa/actions/workflows/ci.yml/badge.svg)](https://github.com/gojiplus/tableqa/actions/workflows/ci.yml)
[![Documentation](https://github.com/gojiplus/tableqa/actions/workflows/docs.yml/badge.svg)](https://gojiplus.github.io/tableqa)
[![PyPI version](https://badge.fury.io/py/tableqa.svg)](https://pypi.org/project/tableqa/)
[![CI](https://github.com/gojiplus/statqa/actions/workflows/ci.yml/badge.svg)](https://github.com/gojiplus/statqa/actions/workflows/ci.yml)
[![Documentation](https://github.com/gojiplus/statqa/actions/workflows/docs.yml/badge.svg)](https://gojiplus.github.io/statqa)
[![PyPI version](https://badge.fury.io/py/statqa.svg)](https://pypi.org/project/statqa/)
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

**TableQA** is a modern Python framework for automatically extracting structured facts, statistical insights, and Q/A pairs from tabular datasets. It converts raw columns and values into clear, human-readable statements, enabling rapid knowledge discovery, RAG corpus construction, and LLM training.
**StatQA** is a modern Python framework for automatically extracting structured facts, statistical insights, and Q/A pairs from tabular datasets. It converts raw columns and values into clear, human-readable statements, enabling rapid knowledge discovery, RAG corpus construction, and LLM training.

## 🎯 Key Features

Expand All @@ -29,30 +29,30 @@
### Basic Installation

```bash
pip install tableqa
pip install statqa
```

### With Optional Features

```bash
# Include LLM support (OpenAI/Anthropic)
pip install tableqa[llm]
pip install statqa[llm]

# Include PDF parsing
pip install tableqa[pdf]
pip install statqa[pdf]

# Development installation
pip install tableqa[dev]
pip install statqa[dev]

# Complete installation
pip install tableqa[all]
pip install statqa[all]
```

### From Source

```bash
git clone https://github.com/gojiplus/tableqa.git
cd tableqa
git clone https://github.com/gojiplus/statqa.git
cd statqa
pip install -e ".[dev]"
```

Expand All @@ -61,7 +61,7 @@ pip install -e ".[dev]"
### 1. Create a Codebook

```python
from tableqa.metadata.parsers import TextParser
from statqa.metadata.parsers import TextParser

codebook_text = """
# Variable: age
Expand Down Expand Up @@ -90,7 +90,7 @@ codebook = parser.parse(codebook_text)

```python
import pandas as pd
from tableqa.analysis import UnivariateAnalyzer, BivariateAnalyzer
from statqa.analysis import UnivariateAnalyzer, BivariateAnalyzer

# Load your data
data = pd.read_csv("survey_data.csv")
Expand All @@ -114,7 +114,7 @@ result = biv_analyzer.analyze(
### 3. Generate Natural Language Insights

```python
from tableqa.interpretation import InsightFormatter
from statqa.interpretation import InsightFormatter

formatter = InsightFormatter()
insight = formatter.format_univariate(result)
Expand All @@ -126,7 +126,7 @@ print(insight)
### 4. Create Q/A Pairs for LLM Training

```python
from tableqa.qa import QAGenerator
from statqa.qa import QAGenerator

qa_gen = QAGenerator(use_llm=False) # Template-based
qa_pairs = qa_gen.generate_qa_pairs(result, insight)
Expand All @@ -139,19 +139,19 @@ for qa in qa_pairs:

Each Q/A pair includes **provenance metadata** tracking:
- **When** the answer was generated (timestamp)
- **What tool** was used (tableqa version)
- **What tool** was used (statqa version)
- **What compute** was performed (analysis type, analyzer)
- **How** it was generated (template vs. LLM paraphrase)
- **Which LLM** was used (if applicable)

## 🎨 Complete Pipeline Example

```python
from tableqa import Codebook, UnivariateAnalyzer
from tableqa.metadata.parsers import CSVParser
from tableqa.interpretation import InsightFormatter
from tableqa.qa import QAGenerator
from tableqa.utils.io import load_data, save_json
from statqa import Codebook, UnivariateAnalyzer
from statqa.metadata.parsers import CSVParser
from statqa.interpretation import InsightFormatter
from statqa.qa import QAGenerator
from statqa.utils.io import load_data, save_json

# 1. Parse codebook
parser = CSVParser()
Expand Down Expand Up @@ -184,7 +184,7 @@ with open("training_data.jsonl", "w") as f:

## 📝 Q/A Provenance Tracking

Every Q/A pair generated by TableQA includes detailed **provenance metadata** to ensure reproducibility and traceability:
Every Q/A pair generated by StatQA includes detailed **provenance metadata** to ensure reproducibility and traceability:

```json
{
Expand All @@ -193,7 +193,7 @@ Every Q/A pair generated by TableQA includes detailed **provenance metadata** to
"type": "descriptive",
"provenance": {
"generated_at": "2025-11-19T10:30:45.123456+00:00",
"tool": "tableqa",
"tool": "statqa",
"tool_version": "0.1.0",
"generation_method": "template",
"analysis_type": "univariate",
Expand All @@ -207,8 +207,8 @@ Every Q/A pair generated by TableQA includes detailed **provenance metadata** to
| Field | Description | Example Values |
|-------|-------------|----------------|
| `generated_at` | ISO 8601 timestamp (UTC) | `2025-11-19T10:30:45+00:00` |
| `tool` | Software used for generation | `tableqa` |
| `tool_version` | Version of tableqa | `0.1.0` |
| `tool` | Software used for generation | `statqa` |
| `tool_version` | Version of statqa | `0.1.0` |
| `generation_method` | How the Q/A was created | `template`, `llm_paraphrase` |
| `analysis_type` | Statistical analysis performed | `univariate`, `bivariate`, `temporal`, `causal` |
| `analyzer` | Specific analyzer class used | `UnivariateAnalyzer`, `BivariateAnalyzer` |
Expand All @@ -222,20 +222,20 @@ This provenance tracking enables:

## 🖥️ Command-Line Interface

TableQA provides a powerful CLI for common workflows:
StatQA provides a powerful CLI for common workflows:

```bash
# Parse a codebook
tableqa parse-codebook codebook.csv --output codebook.json --enrich
statqa parse-codebook codebook.csv --output codebook.json --enrich

# Run full analysis pipeline
tableqa analyze data.csv codebook.json --output-dir results/ --plots
statqa analyze data.csv codebook.json --output-dir results/ --plots

# Generate Q/A pairs
tableqa generate-qa results/all_insights.json --output qa_pairs.jsonl --llm
statqa generate-qa results/all_insights.json --output qa_pairs.jsonl --llm

# Complete pipeline
tableqa pipeline data.csv codebook.csv --output-dir output/ --enrich --qa
statqa pipeline data.csv codebook.csv --output-dir output/ --enrich --qa
```

## 📊 Supported Analyses
Expand Down Expand Up @@ -268,7 +268,7 @@ tableqa pipeline data.csv codebook.csv --output-dir output/ --enrich --qa
### LLM-Powered Metadata Enrichment

```python
from tableqa.metadata import MetadataEnricher
from statqa.metadata import MetadataEnricher

enricher = MetadataEnricher(provider="openai", api_key="your-key")
enriched_codebook = enricher.enrich_codebook(codebook)
Expand All @@ -279,7 +279,7 @@ enriched_codebook = enricher.enrich_codebook(codebook)
### Multiple Testing Correction

```python
from tableqa.utils.stats import correct_multiple_testing
from statqa.utils.stats import correct_multiple_testing

p_values = [0.03, 0.01, 0.15, 0.002]
reject, corrected_p = correct_multiple_testing(p_values, method="fdr_bh")
Expand All @@ -288,37 +288,37 @@ reject, corrected_p = correct_multiple_testing(p_values, method="fdr_bh")
### Custom Visualizations

```python
from tableqa.visualization import PlotFactory
from statqa.visualization import PlotFactory

plotter = PlotFactory(style="publication", figsize=(10, 6))
fig = plotter.plot_bivariate(data, var1, var2, output_path="plot.png")
```

## 📚 Documentation

- **Full Documentation**: [https://gojiplus.github.io/tableqa](https://gojiplus.github.io/tableqa)
- **API Reference**: [API Docs](https://gojiplus.github.io/tableqa/api/)
- **Full Documentation**: [https://gojiplus.github.io/statqa](https://gojiplus.github.io/statqa)
- **API Reference**: [API Docs](https://gojiplus.github.io/statqa/api/)
- **Examples**: See [examples/](examples/) directory

## 🧪 Development

### Running Tests

```bash
pytest --cov=tableqa --cov-report=html
pytest --cov=statqa --cov-report=html
```

### Code Quality

```bash
# Linting
ruff check tableqa tests
ruff check statqa tests

# Type checking
mypy tableqa
mypy statqa

# Formatting
black tableqa tests
black statqa tests
```

### Building Documentation
Expand Down Expand Up @@ -354,9 +354,9 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file

## 📬 Contact & Support

- **Issues**: [GitHub Issues](https://github.com/gojiplus/tableqa/issues)
- **Discussions**: [GitHub Discussions](https://github.com/gojiplus/tableqa/discussions)
- **Email**: maintainers@tableqa.org
- **Issues**: [GitHub Issues](https://github.com/gojiplus/statqa/issues)
- **Discussions**: [GitHub Discussions](https://github.com/gojiplus/statqa/discussions)
- **Email**: maintainers@statqa.org

## 🗺️ Roadmap

Expand Down
22 changes: 11 additions & 11 deletions docs/QA_DATASET_GENERATION_PIPELINE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Overview

This document describes TableQA's pipeline for generating Q/A-style datasets from tabular data with metadata. The pipeline converts raw tables into structured question-answer pairs suitable for training or fine-tuning LLMs on data analysis tasks.
This document describes StatQA's pipeline for generating Q/A-style datasets from tabular data with metadata. The pipeline converts raw tables into structured question-answer pairs suitable for training or fine-tuning LLMs on data analysis tasks.

## Pipeline Architecture

Expand Down Expand Up @@ -52,7 +52,7 @@ Output: Q/A Dataset for LLM Training

**Example**:
```python
from tableqa.metadata.parsers import CSVParser
from statqa.metadata.parsers import CSVParser

parser = CSVParser()
codebook = parser.parse('codebook.csv')
Expand Down Expand Up @@ -84,7 +84,7 @@ codebook = parser.parse('codebook.csv')

**Example**:
```python
from tableqa.analysis import UnivariateAnalyzer, BivariateAnalyzer
from statqa.analysis import UnivariateAnalyzer, BivariateAnalyzer

# Univariate
analyzer = UnivariateAnalyzer()
Expand Down Expand Up @@ -116,7 +116,7 @@ range=[25,000.00, 133,843.00]. N=500 [non-normal distribution].

**Example**:
```python
from tableqa.interpretation import InsightFormatter
from statqa.interpretation import InsightFormatter

formatter = InsightFormatter()
insight = formatter.format_univariate(result)
Expand All @@ -143,7 +143,7 @@ insight = formatter.format_univariate(result)

**Example**:
```python
from tableqa.qa import QAGenerator
from statqa.qa import QAGenerator

# Template-based only
qa_gen = QAGenerator(use_llm=False)
Expand Down Expand Up @@ -325,10 +325,10 @@ A: **Survived**: most common category is '0' (60.2%), N=400.

```python
from pathlib import Path
from tableqa.metadata.parsers import CSVParser
from tableqa.analysis import UnivariateAnalyzer, BivariateAnalyzer
from tableqa.interpretation import InsightFormatter
from tableqa.qa import QAGenerator
from statqa.metadata.parsers import CSVParser
from statqa.analysis import UnivariateAnalyzer, BivariateAnalyzer
from statqa.interpretation import InsightFormatter
from statqa.qa import QAGenerator

# 1. Parse codebook
parser = CSVParser()
Expand Down Expand Up @@ -366,13 +366,13 @@ with open('qa_dataset.jsonl', 'w') as f:

```bash
# Complete pipeline
tableqa pipeline data.csv codebook.csv \
statqa pipeline data.csv codebook.csv \
--output-dir output/ \
--qa \
--enrich

# Just Q/A generation from existing insights
tableqa generate-qa insights.json \
statqa generate-qa insights.json \
--output qa_pairs.jsonl \
--format openai \
--llm
Expand Down
10 changes: 5 additions & 5 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
"""Sphinx configuration for tableqa documentation."""
"""Sphinx configuration for statqa documentation."""

import sys
from pathlib import Path

from tableqa import __version__
from statqa import __version__


# Add project root to path
sys.path.insert(0, str(Path(__file__).parents[2]))

# Project information
project = "tableqa"
copyright = "2025, TableQA Contributors"
author = "TableQA Contributors"
project = "statqa"
copyright = "2025, StatQA Contributors"
author = "StatQA Contributors"

# Version
version = __version__
Expand Down
Loading
Loading