Skip to content

Add persistent CSV/JSON uploads and improve upload workflow#20

Merged
abhinavsingh9714 merged 6 commits into
mainfrom
feature/implement-data-layer
Apr 6, 2026
Merged

Add persistent CSV/JSON uploads and improve upload workflow#20
abhinavsingh9714 merged 6 commits into
mainfrom
feature/implement-data-layer

Conversation

@saranshkr

@saranshkr saranshkr commented Apr 5, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR adds CSV and JSON upload support, persists uploaded data per user, and improves the upload workflow across the app.

What changed

  • added support for uploading tabular CSV/TSV data and nested JSON data
  • normalized uploaded files into a registry-backed structure for planner/executor use
  • persisted upload metadata in SQLite and linked each upload to the signed-in user
  • stored raw uploaded files on the local filesystem
  • kept DuckDB as the normalized query layer for uploaded sources
  • added authenticated upload listing so files reappear after refresh and after logging out and back in
  • restricted upload visibility and analysis access to the owning user
  • made chat use saved uploads even when files were uploaded from the Uploads page
  • added delete actions for uploaded files
  • removed the Recent uploads sidebar block to avoid layout issues and keep the sidebar stable

Testing

  • ran .venv/bin/pytest tests/test_auth.py tests/test_api.py tests/test_source_registry.py tests/test_planner_schema.py tests/test_tools.py
  • verified CSV and JSON uploads are ingested correctly
  • verified uploads persist across refresh/login
  • verified users only see and analyze their own uploads
  • verified uploaded files can be deleted cleanly

@saranshkr saranshkr self-assigned this Apr 5, 2026
@saranshkr saranshkr changed the title normalize uploaded CSV and JSON sources in the data registry Add persistent CSV/JSON uploads and improve upload workflow Apr 5, 2026

@abhinavsingh9714 abhinavsingh9714 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@saranshkr saranshkr force-pushed the feature/implement-data-layer branch from 270b34a to a693b45 Compare April 5, 2026 20:21
@saranshkr saranshkr linked an issue Apr 6, 2026 that may be closed by this pull request
@abhinavsingh9714 abhinavsingh9714 merged commit 79fe299 into main Apr 6, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement data layer

2 participants