Awesum

This is the open-source repository for 2024 VIS submission: Towards Dataset-scale and Feature-oriented Evaluation of Large Language Model Prompts: https://doi.org/10.1109/TVCG.2024.3456398.

Feature-oriented Workflow

We introduce a feature-oriented workflow for dataset-scale prompt evaluation.

System Overview

Awesum is an implementation of the workflow on text summarization.

Launching the application locally

Awesum has a Flask back-end and a vite front-end. They need to be launched separately.

Backend

Navigate to server/ and run:

pip install -r requirements.txt

These packages need extra care:

OpenAI:

pip install --upgrade openai 

Spacy:

pip install spacy
python -m spacy download en_core_web_sm

Or if you're using M1:

pip install 'spacy[apple]' 
python -m spacy download en_core_web_sm

Finally,

# directory: server/ 
flask run

The backend server should then be listening to localhost:5000 by default.

Front-end

npm i
npm run dev

Credits to the developers of these packages/repositories

Backend: Flask

Frontend: Svelte, MeltUI, TailwindCss, Lucide, D3, Vite

Dataset: BBC

Computational Linguistics: Spacy, VADER, Textstat, Lexical Diversity, scikit-learn

Intelligent Agents: OpenAI

Source Code paths

Frontend

src/*

Backend

1. Flask server: server/app.py
2. Feature metrics: server/metrics/stylistic.py
3. Prompts: server/AnalysisUtils/gpt.py
4. Clustering: server/AnalysisUtils/clusters.py
5. Dimensionality Reduction: server/AnalysisUtils/dr.py
6. Data: server/data/prod/*