Skip to content

en-trophy/entrophy-front

BranchesTags

Repository files navigation

Equal Sign πŸ‘‹

AI-Powered Sign Language Learning Platform

An AI tutor that provides real-time feedback to students learning sign language using Azure AI and OpenAI. Built for Microsoft Imagine Cup.

🎯 Key Features

Learning Features

  • πŸŽ₯ Real-time Webcam Learning: Practice sign language in front of your camera
  • 🎨 Visual Feedback: Color-coded accuracy indicators (Green/Yellow/Red)
  • πŸ“Š Scoring System: Real-time accuracy scoring
  • πŸ€– AI-Based Tracking: Precise tracking of 21 hand joint points using MediaPipe
  • πŸ’¬ Personalized Feedback: Custom learning feedback powered by Azure OpenAI
  • πŸ“š Structured Learning: Categorized word/phrase learning flow
  • πŸ” Smart Search: Quick lesson search functionality
  • 🎯 Practice Today: Daily review simulation based on learned lessons

User Features

  • πŸ” User Authentication: Secure login and signup system
  • ⏰ Session Management: 60-minute session timeout for security
  • πŸ“ˆ Learning History: Track your progress over time
  • πŸ‘€ User Profile: View your learning statistics and achievements
  • πŸ“… Daily Practice: Review lessons you've learned today

Technical Features

  • πŸ“Έ Multi-frame Capture: Dynamic sign language recognition with multiple frames
  • 🎬 Video/Image Support: Flexible media types for different lesson modes
  • 🌐 RESTful API Integration: Seamless backend communication
  • 🎨 Responsive Design: Works on desktop and mobile devices

πŸ“Έ Screenshots

Home Screen

Home Screen

Search Feature

Search Feature

Practice Mode

Practice Mode

Practice Today (Simulation)

Practice Today

Learning History

Learning History

Result Screen

Result Screen

πŸ›  Tech Stack

Frontend

  • React 18 - UI framework
  • TypeScript - Type safety
  • Vite - Build tool
  • React Router - Navigation
  • MediaPipe Holistic - Hand/pose tracking
  • WebRTC - Webcam access
  • HTML5 Canvas - Skeleton rendering

Backend APIs

  • Azure AI - Computer vision and pose analysis
  • Azure OpenAI - Feedback generation
  • RESTful API - Backend integration
  • JWT Authentication - Secure user sessions

Deployment

  • Azure Static Web Apps - Frontend hosting
  • Azure App Service - Backend API hosting

πŸ“ Project Structure

src/
β”œβ”€β”€ pages/
β”‚   β”œβ”€β”€ HomePage.tsx              # Home (category selection + search)
β”‚   β”œβ”€β”€ CategoryDetailPage.tsx    # Level selection (word/phrase)
β”‚   β”œβ”€β”€ ItemListPage.tsx          # Lesson list
β”‚   β”œβ”€β”€ LessonDetailPage.tsx      # Lesson details
β”‚   β”œβ”€β”€ PracticePage.tsx          # Real-time learning
β”‚   β”œβ”€β”€ SimulationPage.tsx        # Practice Today mode
β”‚   β”œβ”€β”€ ResultPage.tsx            # Learning results
β”‚   β”œβ”€β”€ ProfilePage.tsx           # User profile & history
β”‚   └── LoginPage.tsx             # Login & signup
β”œβ”€β”€ components/
β”‚   β”œβ”€β”€ Camera.tsx                # Webcam + MediaPipe integration
β”‚   β”œβ”€β”€ ScoreBoard.tsx            # Score display
β”‚   β”œβ”€β”€ Header.tsx                # Header component
β”‚   β”œβ”€β”€ SidebarNav.tsx            # Navigation sidebar
β”‚   └── SessionTimeoutChecker.tsx # Session timeout handler
β”œβ”€β”€ services/
β”‚   β”œβ”€β”€ api.ts                    # API client
β”‚   └── authService.ts            # Authentication service
β”œβ”€β”€ types/
β”‚   └── index.ts                  # TypeScript type definitions
└── App.tsx                       # Router configuration

πŸš€ Getting Started

Prerequisites

  • Node.js 18+
  • npm or yarn
  • Webcam access

1. Install Dependencies

npm install

2. Environment Setup

Create a .env.development file:

VITE_BACKEND_API_URL=your_backend_api_url
VITE_AI_API_URL=your_ai_api_url

3. Run Development Server

npm run dev

Open http://localhost:5173 in your browser

4. Build for Production

npm run build

πŸ’‘ How to Use

Basic Learning Flow

  1. Sign Up / Login - Create an account or log in
  2. Browse Categories - Select from 8 different categories (Greetings, Emotions, Daily Life, etc.)
  3. Choose Level - Pick between Word or Phrase learning
  4. Select Lesson - Browse and select a lesson to learn
  5. View Lesson Details - Study the tutorial video and instructions
  6. Start Practice - Click "Start Learning" button
  7. Allow Webcam - Grant camera permission
  8. Practice Sign Language - Follow the on-screen guidance
  9. Get Real-time Feedback - See color-coded feedback:
    • 🟒 Green: Accurate movement
    • 🟑 Yellow: Moderate accuracy
    • πŸ”΄ Red: Needs correction
  10. Complete Learning - Click "Complete Learning" to see your results

Practice Today Mode

  1. Click "Practice Today" in the sidebar
  2. System creates scenarios based on lessons you learned today
  3. Practice in realistic conversation contexts
  4. Get AI-powered feedback on your performance

Search Feature

  1. Use the search bar on the home page
  2. Type keywords (e.g., "hello", "thank")
  3. Press Enter or click the search icon
  4. Browse search results with video previews

Profile & History

  1. Click "Learning History" or "Profile" in the sidebar
  2. View your learning statistics
  3. Track daily progress
  4. See total lessons completed and average scores

πŸ” Authentication

Session Management

  • Session Duration: 60 minutes
  • Auto Logout: Automatic logout after session expires
  • Session Timeout Modal: Notification when session expires
  • Secure Token Storage: JWT tokens in localStorage

Features

  • User registration with validation
  • Secure login with password confirmation
  • Protected routes requiring authentication
  • Persistent login across page refreshes

🎨 Design System

Colors

  • Primary: Azure Blue (#0078D4)
  • Secondary: #005A9E
  • Success: #16C60C
  • Warning: #FFB800
  • Error: #D13438
  • Background: White with subtle gradients
  • Text: #323130 (Primary), #605E5C (Secondary)

Typography

  • Font Family: System UI fonts (Segoe UI, San Francisco, etc.)
  • Headings: Bold, 24-48px
  • Body: Regular, 14-18px

πŸ“š Learning Categories

  • πŸ‘‹ Greetings: Basic greetings for meetings and farewells
  • 😊 Emotions: Express joy, sadness, anger, etc.
  • 🏠 Daily Life: Common expressions in daily life
  • πŸ‘ͺ Family: Family relationships and titles
  • 🏫 School: School-related expressions
  • 🍽️ Reactions: Expressions like "thank you", "sorry"
  • βœ‹ Alphabet: Fingerspelling alphabet
  • πŸ”’ Numbers: Finger counting and numbers

πŸ€– AI Integration

MediaPipe Holistic

  • Real-time hand tracking (21 landmarks per hand)
  • Pose tracking (33 landmarks)
  • Face tracking for comprehensive analysis

Azure OpenAI

  • Personalized feedback generation
  • Context-aware coaching messages
  • Performance analysis

Computer Vision API

  • Multi-frame capture for dynamic signs
  • Pose comparison and scoring
  • Accurate gesture recognition

🌟 Imagine Cup Highlights

  • β™Ώ Social Impact: Improving education accessibility for deaf communities
  • ☁️ Azure Technology: AI Vision + OpenAI + App Service + Static Web Apps
  • 🎨 Visual Impact: Real-time feedback creates powerful demonstrations
  • 🌍 Scalability: Can support sign languages from different countries
  • πŸ“± User Experience: Intuitive learning flow
  • πŸ”¬ Innovation: AI-powered personalized learning at scale

πŸ”„ API Endpoints

Backend API

GET  /api/categories                     # Get all categories
GET  /api/lessons                        # Get all lessons
GET  /api/lessons/:id                    # Get specific lesson
GET  /api/lessons/category/:categoryId   # Get lessons by category
GET  /api/lessons/:id/answer-frames/count # Get frame count for lesson
POST /api/auth/signup                    # User registration
POST /api/auth/login                     # User login
GET  /api/learning-histories             # Get learning history
POST /api/learning-histories             # Save learning record

AI API

POST /api/lessons/:id/feedback/image     # Single frame analysis
POST /api/lessons/:id/feedback/images    # Multi-frame analysis
POST /api/simulation                     # Create practice scenario

πŸ“ Roadmap

Completed βœ…

  • User authentication system
  • Session management with timeout
  • Search functionality
  • Learning history tracking
  • Practice Today simulation mode
  • Multi-frame capture for dynamic signs
  • Real-time AI feedback
  • Profile page with statistics
  • MediaPipe integration

Upcoming πŸš€

  • Achievement system (badges, levels)
  • Social features (share progress)
  • More sign language lessons
  • Multi-language UI support
  • Mobile app version
  • Offline mode support
  • Video recording and playback
  • Leaderboard and competitions

🀝 Contributing

This project was built for Microsoft Imagine Cup. Contributions are welcome!

πŸ“„ License

MIT License

Made with ❀️ for Microsoft Imagine Cup 2026

Team: en-trophy

About

No description, website, or topics provided.

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages