FWC scraper

Author: myshkin42

AUSTRALIAN_LEGAL_DATABASES_ROADMAP.md

@@ -0,0 +1,396 @@
+# Australian Legal Databases - Complete Roadmap
+
+**Strategic Focus:** Legal evidence for worker advocacy
+**Timeline:** 4 weeks to complete priority databases
+**Current Status:** 2/7 databases complete (FWC + WorkSafe VIC in progress)
+
+---
+
+## 🎯 Priority Order (Based on Client Value)
+
+### **TIER 1: CRITICAL (Weeks 1-2)** ⭐⭐⭐⭐⭐
+
+#### 1. Fair Work Commission (FWC) ✅ **COMPLETE**
+- **Status:** Working, tested, integrated
+- **Coverage:** Employment law cases Australia-wide
+- **Value:** Unfair dismissal, discrimination, wage disputes
+- **Tool:** `tools/fwc.py`
+
+#### 2. WorkSafe Victoria 🔄 **IN PROGRESS**
+- **Status:** Scraper built, needs testing
+- **Coverage:** Victoria safety prosecutions (2012+)
+- **Value:** Safety violations, penalties, incidents
+- **Tool:** `tools/worksafe_vic.py`
+- **Next:** Test and refine
+
+#### 3. SafeWork NSW ⏳ **NEXT**
+- **URL:** https://www.safework.nsw.gov.au/resource-library/prosecutions
+- **Coverage:** NSW safety prosecutions (largest state)
+- **Value:** Same as WorkSafe VIC
+- **Approach:** Replicate WorkSafe VIC pattern
+- **Timeline:** 2-3 days after VIC complete
+
+---
+
+### **TIER 2: HIGH VALUE (Weeks 2-3)** ⭐⭐⭐⭐⭐
+
+#### 4. AustLII (Australasian Legal Information Institute)
+- **URL:** https://www.austlii.edu.au/
+- **Coverage:** ALL Australian court cases (free database)
+- **Value:** Federal + State courts, comprehensive
+- **Unique:** One search covers all jurisdictions
+- **Search Types:**
+  - Company name
+  - Case type (employment, civil, commercial)
+  - Date range
+  - Court level (Federal, Supreme, etc.)
+
+**Why CRITICAL:**
+- **Free comprehensive legal database**
+- **Covers gaps between FWC and courts**
+- **Includes appeals and major decisions**
+- **Historical coverage back decades**
+
+**Implementation:**
+```python
+async def search_austlii(
+    company_name: str,
+    case_types: List[str] = ["employment", "workplace"],
+    courts: List[str] = ["federal", "supreme"],
+    years: int = 10
+) -> Dict:
+    """
+    Search AustLII for court cases involving company
+    """
+    # AustLII has search API
+    # Returns: Case citations, summaries, full text links
+```
+
+#### 5. Fair Work Ombudsman (FWO)
+- **URL:** https://www.fairwork.gov.au/about-us/our-role/enforcing-the-legislation
+- **Coverage:** Wage recovery actions, compliance notices
+- **Value:** Direct wage theft evidence
+- **Data:**
+  - Enforcement outcomes
+  - Penalties imposed
+  - Amounts recovered
+  - Court-enforceable undertakings
+
+**Why HIGH VALUE:**
+- **Wage theft is common** - many investigations involve this
+- **Complements FWC** - Shows enforcement actions
+- **Public register** - Searchable database
+
+#### 6. Federal Court of Australia
+- **URL:** https://www.fedcourt.gov.au/
+- **Coverage:** Federal litigation (employment law precedents)
+- **Value:** Major cases, important decisions
+- **Search:**
+  - Case search by party name
+  - Full text decisions
+  - Judgment summaries
+
+---
+
+### **TIER 3: VALUABLE (Week 3-4)** ⭐⭐⭐⭐
+
+#### 7. ASIC (Australian Securities & Investments Commission)
+- **Director Ban Register:** Disqualified directors
+- **Insolvency Notices:** Company liquidations
+- **Why:** Red flags for dodgy operators
+
+**Searches:**
+```python
+# Director ban search
+asic_director_ban(director_name) → Is this person banned?
+
+# Insolvency search  
+asic_insolvency(company_name) → Has company been liquidated?
+
+# Officer history
+asic_officers(company_name) → Who are/were the directors?
+```
+
+#### 8. Additional SafeWork States
+- **WorkSafe Queensland**
+- **WorkSafe WA**
+- **SafeWork SA**
+
+**Why Lower Priority:**
+- Smaller populations than VIC/NSW
+- Same data structure as VIC/NSW
+- Can add incrementally
+
+---
+
+## 📊 Coverage Analysis
+
+### By Database Type:
+
+| Type | Databases | Coverage | Status |
+|------|-----------|----------|--------|
+| Employment Law | FWC | Australia-wide | ✅ Complete |
+| Safety | WorkSafe VIC, NSW, QLD, WA, SA | State-by-state | 🔄 1/5 |
+| Courts | AustLII, Federal Court | All jurisdictions | ⏳ Planned |
+| Wage Enforcement | Fair Work Ombudsman | Australia-wide | ⏳ Planned |
+| Corporate | ASIC | Australia-wide | ⏳ Planned |
+
+### By Worker Coverage:
+
+| Database | Workers Covered | % of Australia |
+|----------|----------------|----------------|
+| FWC | All Australian workers | 100% |
+| WorkSafe VIC | ~3M workers | 25% |
+| SafeWork NSW | ~3.8M workers | 32% |
+| **VIC + NSW** | **~6.8M workers** | **57%** |
+| All SafeWork | ~12M workers | 100% |
+
+**Strategy:** VIC + NSW gives us 57% coverage with just 2 databases!
+
+---
+
+## 🛠️ Implementation Pattern
+
+### Reusable Code Structure:
+
+```python
+# Pattern for all safety databases
+async def search_safety_database(
+    database_name: str,
+    database_url: str,
+    company_name: str,
+    years: int = 10
+) -> Dict:
+    """Generic safety database scraper"""
+    
+    # 1. Navigate to database
+    # 2. Find search interface
+    # 3. Submit search
+    # 4. Extract results
+    # 5. Parse prosecution details
+    
+    return {
+        "database": database_name,
+        "prosecutions": [...],
+        "total_found": N
+    }
+
+# Customize selectors per database
+SELECTORS = {
+    "worksafe_vic": {
+        "search_box": "input[name='search']",
+        "results": "article.prosecution"
+    },
+    "safework_nsw": {
+        "search_box": "input#company",
+        "results": "div.result-card"
+    }
+}
+```
+
+---
+
+## 📅 4-Week Timeline
+
+### **Week 1: Safety Databases (VIC + NSW)**
+
+**Days 1-3: WorkSafe Victoria**
+- [x] Build scraper
+- [ ] Test and refine
+- [ ] Validate on 10 companies
+- [ ] Integration
+
+**Days 4-7: SafeWork NSW**
+- [ ] Adapt VIC code for NSW
+- [ ] Test and validate
+- [ ] Integration
+
+**Deliverable:** 57% of Australian workers covered for safety data
+
+---
+
+### **Week 2: Comprehensive Courts (AustLII + FWO)**
+
+**Days 1-4: AustLII**
+- [ ] Research API/search interface
+- [ ] Build case search
+- [ ] Filter for employment cases
+- [ ] Test on 20 companies
+
+**Days 5-7: Fair Work Ombudsman**
+- [ ] Research enforcement database
+- [ ] Build scraper
+- [ ] Test and validate
+
+**Deliverable:** Court cases + wage theft enforcement
+
+---
+
+### **Week 3: Federal Court + ASIC**
+
+**Days 1-3: Federal Court**
+- [ ] Build case search
+- [ ] Parse decisions
+- [ ] Test and validate
+
+**Days 4-7: ASIC Integration**
+- [ ] Director ban register
+- [ ] Insolvency notices
+- [ ] Officer history
+
+**Deliverable:** Corporate red flags
+
+---
+
+### **Week 4: Additional States + Polish**
+
+**Days 1-3: More SafeWork States**
+- [ ] Queensland
+- [ ] Western Australia
+- [ ] South Australia
+
+**Days 4-7: Testing & Documentation**
+- [ ] End-to-end testing
+- [ ] Performance optimization
+- [ ] Documentation
+- [ ] Sample investigations
+
+**Deliverable:** Complete Australian legal evidence platform
+
+---
+
+## 🎯 Success Metrics
+
+### Functionality:
+- [ ] Searches 7 databases in <2 minutes
+- [ ] Finds cases/prosecutions other tools miss
+- [ ] Accurate data extraction (>95%)
+- [ ] Handles "no results" gracefully
+- [ ] Error recovery works
+
+### Client Value:
+- [ ] Answers "Has this employer been prosecuted?"
+- [ ] Shows safety violation patterns
+- [ ] Provides court case history
+- [ ] Identifies director red flags
+- [ ] Generates credible evidence for organizing
+
+### Coverage:
+- [ ] 100% of Australian workers (employment law)
+- [ ] 100% of Australian workers (safety)
+- [ ] Federal + state court coverage
+- [ ] Wage theft enforcement
+- [ ] Corporate compliance
+
+---
+
+## 💰 Client Value Proposition
+
+### What We Can Tell Clients:
+
+**Before (just FWC):**
+> "We found 1 employment law case against this company."
+
+**After (7 databases):**
+> "We searched 7 legal databases and found:
+> - 3 Fair Work Commission cases (unfair dismissal)
+> - 2 SafeWork prosecutions ($150K in fines)
+> - 1 Federal Court case (pending)
+> - 1 Fair Work Ombudsman wage recovery ($85K)
+> - Clean ASIC record (no banned directors)
+> 
+> Total penalties: $235,000
+> Pattern: Repeat safety violations (2021, 2023)
+> Risk Assessment: HIGH"
+
+---
+
+## 🔍 Investigation Workflow
+
+### Complete Australian Investigation:
+
+```python
+async def investigate_company_australia(company_name: str, abn: str):
+    """Complete legal investigation - Australia"""
+    
+    results = {}
+    
+    # PHASE 1: Employment Law
+    results["fwc"] = await search_fwc(company_name)
+    results["fwo"] = await search_fwo(company_name)
+    
+    # PHASE 2: Safety
+    results["worksafe_vic"] = await search_worksafe_vic(company_name)
+    results["safework_nsw"] = await search_safework_nsw(company_name)
+    
+    # PHASE 3: Courts
+    results["austlii"] = await search_austlii(company_name)
+    results["federal_court"] = await search_federal_court(company_name)
+    
+    # PHASE 4: Corporate
+    results["asic"] = await search_asic(company_name)
+    
+    # Generate risk assessment
+    risk = assess_compliance_risk(results)
+    
+    return {
+        "legal_findings": results,
+        "risk_assessment": risk,
+        "summary": generate_summary(results)
+    }
+```
+
+---
+
+## 🚀 Current Status
+
+**Completed:** 1/7 databases (14%)  
+**In Progress:** 1/7 databases (WorkSafe VIC)  
+**Next Up:** SafeWork NSW  
+
+**Timeline to Complete:** 4 weeks  
+**Current Week:** Week 1, Day 1
+
+---
+
+## 📝 Notes for Implementation
+
+### Technical Decisions:
+
+1. **Playwright vs Requests**
+   - Using Playwright for all (consistency)
+   - Handles JavaScript-heavy sites
+   - Screenshot debugging built in
+
+2. **Error Handling**
+   - Continue on individual database failures
+   - Return partial results
+   - Log all errors for debugging
+
+3. **Rate Limiting**
+   - 2-3 second delays between requests
+   - Respect robots.txt
+   - Random user agents
+
+4. **Data Storage**
+   - JSON format for now
+   - Consider SQLite if volume grows
+   - Evidence preservation format
+
+### Testing Strategy:
+
+1. **Unit Tests** - Each database individually
+2. **Integration Tests** - Full investigation workflow
+3. **Known Cases** - Validate against public records
+4. **Edge Cases** - No results, name variations, etc.
+
+---
+
+**Next Session Start:** Test WorkSafe VIC scraper!
+
+```bash
+cd C:\Projects\osint-mcp-server
+python tools/worksafe_vic.py
+```
+
+🎯 **Let's build this!**