Add Extractor API for IoC extraction

- Add Extractor, ExtractedMatch, ExtractFlags, ItemType classes
- Add JNA bindings for extractor C API (MatchyLibrary, NativeStructs)
- Add ExtractorTest with comprehensive test coverage
- Update submodule to include new extractor C API
- Update WARP.md and README.md documentation
- Clean up .gitignore (remove vestigial entry)

Author: sethhall

.gitignore

@@ -29,8 +29,7 @@ hs_err_pid*
 # Java
 *.log
 
-# Native libraries (bundled in CI, not committed)
-java/src/main/resources/native/
+# Native library files (bundled at build time into target/classes)
 *.dylib
 *.so
 *.dll

README.md

@@ -2,24 +2,162 @@
 
 Java wrapper for [matchy](https://github.com/matchylabs/matchy) - fast IoC matching for threat intelligence.
 
-## Status
+## Installation
+
+Download the JAR from [Releases](https://github.com/matchylabs/matchy-java/releases) and add it to your classpath.
+
+## Quick Start
+
+### Querying a Database
+
+```java
+import com.matchylabs.matchy.*;
+import java.nio.file.Paths;
+
+try (Database db = Database.open(Paths.get("threats.mxy"))) {
+    // Query an IP address
+    QueryResult result = db.query("192.168.1.1");
+    
+    if (result.isMatch()) {
+        System.out.println("Match found!");
+        System.out.println("Data: " + result.getData());
+        System.out.println("Prefix length: " + result.getPrefixLength());
+    }
+    
+    // Query a domain
+    QueryResult domainResult = db.query("evil.example.com");
+}
+```
+
+### Building a Database
+
+```java
+import com.matchylabs.matchy.*;
+import java.nio.file.Paths;
+import java.util.Map;
+
+try (DatabaseBuilder builder = new DatabaseBuilder()) {
+    // Add IP addresses and CIDRs
+    builder.add("1.2.3.4", Map.of("threat", "malware", "confidence", 95));
+    builder.add("10.0.0.0/8", Map.of("type", "internal"));
+    
+    // Add patterns (glob syntax)
+    builder.add("*.evil.com", Map.of("category", "phishing"));
+    builder.add("malware-*.example.org", Map.of("category", "c2"));
+    
+    // Set metadata
+    builder.setDescription("Threat database v1.0");
+    
+    // Save to file
+    builder.save(Paths.get("threats.mxy"));
+    
+    // Or build in-memory
+    try (Database db = builder.build()) {
+        QueryResult result = db.query("1.2.3.4");
+    }
+}
+```
+
+### Configuration Options
+
+```java
+import com.matchylabs.matchy.*;
+import java.nio.file.Paths;
+
+// Custom cache size
+OpenOptions options = OpenOptions.defaults()
+    .cacheCapacity(100_000);  // LRU cache for query results
+
+try (Database db = Database.open(Paths.get("threats.mxy"), options)) {
+    // Queries are cached for faster repeated lookups
+}
+
+// Auto-reload on file changes
+OpenOptions watchOptions = OpenOptions.defaults()
+    .autoReload(true);  // Automatically reload when file changes
+
+// Disable caching entirely
+OpenOptions noCacheOptions = OpenOptions.defaults()
+    .noCache();
+```
+
+### Error Handling
+
+```java
+import com.matchylabs.matchy.*;
+import java.nio.file.Paths;
 
-✅ **Core functionality complete** - Ready for use. Download fat JARs from [Releases](https://github.com/matchylabs/matchy-java/releases).
+try (Database db = Database.open(Paths.get("threats.mxy"))) {
+    QueryResult result = db.query("192.168.1.1");
+    // ...
+} catch (MatchyException e) {
+    // Handle database errors (file not found, corrupt data, etc.)
+    System.err.println("Matchy error: " + e.getMessage());
+}
+```
 
-## Completed
+## API Reference
+
+### Database
+
+Main class for querying matchy databases.
+
+| Method | Description |
+|--------|-------------|
+| `Database.open(Path)` | Open a database file |
+| `Database.open(Path, OpenOptions)` | Open with custom options |
+| `Database.fromBuffer(byte[])` | Open from memory |
+| `query(String)` | Query IP address or pattern |
+| `getStats()` | Get query statistics |
+| `clearCache()` | Clear the LRU cache |
+| `getMetadata()` | Get database metadata as JSON |
+| `hasIpData()` / `hasStringData()` | Check data types |
+| `close()` | Free resources (use try-with-resources) |
+
+### DatabaseBuilder
+
+Create databases programmatically.
+
+| Method | Description |
+|--------|-------------|
+| `add(String, Map)` | Add entry with data |
+| `addJson(String, String)` | Add entry with JSON string |
+| `setDescription(String)` | Set database description |
+| `save(Path)` | Save to file |
+| `build()` | Build in-memory Database |
+| `toBytes()` | Build as byte array |
+
+### QueryResult
+
+| Method | Description |
+|--------|-------------|
+| `isMatch()` | Whether query matched |
+| `getData()` | Match data as JsonObject |
+| `getDataAsJson()` | Match data as JSON string |
+| `getPrefixLength()` | Network prefix (for IP matches) |
+
+## Requirements
+
+- **Java**: 11+
+- **Platforms**: Linux (x86_64, aarch64), macOS (x86_64, aarch64), Windows (x86_64)
+
+## Thread Safety
+
+- `Database` instances are thread-safe for queries
+- `DatabaseBuilder` is NOT thread-safe (use from a single thread)
+
+## More Information
+
+- [matchy documentation](https://matchylabs.github.io/matchy/) - concepts, file format, CLI
+- [matchy repository](https://github.com/matchylabs/matchy) - native library
+
+## Status
 
-- ✅ JNA bindings (NativeLoader, MatchyLibrary, NativeStructs)
-- ✅ Core wrapper classes (QueryResult, DatabaseStats, OpenOptions)
-- ✅ Database class (open, query, close, stats)
-- ✅ DatabaseBuilder class (create databases programmatically)
-- ✅ Exception handling (MatchyException)
-- ✅ Unit tests
-- ✅ GitHub Actions CI/CD (multi-platform, Java 11)
-- ✅ Fat JAR releases with bundled native libraries
+✅ **Core functionality complete** - Ready for use.
 
 ## TODO
 
-- [ ] Documentation and examples
+- [ ] Extractor API (extract IPs, domains, hashes from text)
 - [ ] Maven Central deployment
 
 ## License

WARP.md

@@ -6,11 +6,12 @@ This file provides guidance to WARP (warp.dev) when working with code in this re
 
 **matchy-java** is a Java wrapper for [matchy](https://github.com/matchylabs/matchy), providing JNA bindings to the native matchy library for fast IoC (Indicator of Compromise) matching.
 
-**Status**: 🚧 Work in Progress
+**Status**: ✅ Core Features Complete
 - JNA bindings implemented (NativeLoader, MatchyLibrary, NativeStructs)
 - Core wrapper classes implemented (QueryResult, DatabaseStats, OpenOptions, MatchyException)
-- Database and DatabaseBuilder classes pending
-- No tests or CI/CD yet
+- Database and DatabaseBuilder classes implemented
+- Extractor API implemented (Extractor, ExtractedMatch, ExtractFlags, ItemType)
+- Unit tests for Database, DatabaseBuilder, and Extractor
 
 ### Architecture
 
@@ -23,8 +24,12 @@ matchy-java/
 │       │   ├── NativeLoader.java  # Platform detection & native library loading
 │       │   ├── MatchyLibrary.java # JNA interface to matchy C API
 │       │   └── NativeStructs.java # JNA structure mappings (MatchyResult, etc.)
-│       ├── Database.java          # Main public API (TODO)
-│       ├── DatabaseBuilder.java   # Database builder API (TODO)
+│       ├── Database.java          # Main public API for querying
+│       ├── DatabaseBuilder.java   # Database builder API
+│       ├── Extractor.java         # IoC extraction from text
+│       ├── ExtractedMatch.java    # Single extracted match
+│       ├── ExtractFlags.java      # Extraction type flags
+│       ├── ItemType.java          # Enum of extractable item types
 │       ├── QueryResult.java       # Query result wrapper
 │       ├── DatabaseStats.java     # Database statistics
 │       ├── OpenOptions.java       # Database open configuration
@@ -222,59 +227,42 @@ OpenOptions options = OpenOptions.defaults()
 Database db = Database.open("threats.mxy", options);
 ```
 
-## Key TODOs
-
-Based on README.md, these are the next implementation priorities:
-
-### 1. Database Class (High Priority)
-
-Implement the main Database API:
-- `static Database open(String path)` and `open(String path, OpenOptions options)`
-- `static Database fromBuffer(byte[] buffer)`
-- `QueryResult query(String text)` - main query method
-- `void clearCache()` - clear LRU cache
-- `DatabaseStats getStats()` - get query statistics
-- `String getMetadata()` - database metadata
-- `boolean hasIpData()`, `hasStringData()`, `hasGlobData()`, `hasLiteralData()`
-- `String getFormat()` - get database format version
-- `void close()` - free native resources
-- Implement `AutoCloseable` for try-with-resources
-
-Reference the C API in MatchyLibrary for all available functions.
-
-### 2. DatabaseBuilder Class (High Priority)
-
-Builder for creating databases programmatically:
-- `DatabaseBuilder()` constructor
-- `DatabaseBuilder add(String key, JsonObject data)` - add entry with metadata
-- `DatabaseBuilder add(String key, String jsonData)` - add entry with JSON string
-- `DatabaseBuilder setDescription(String description)` - set database description
-- `Database build()` - build in-memory database
-- `void save(String path)` - save to file
-- `byte[] toBytes()` - serialize to byte array
+### Extractor Pattern
 
-### 3. Unit Tests (High Priority)
+Use the Extractor to find IoCs (Indicators of Compromise) in text:
 
-Create test files in `src/test/java/com/matchylabs/matchy/`:
-- `DatabaseTest.java` - test open, query, close
-- `DatabaseBuilderTest.java` - test database creation
-- `QueryResultTest.java` - test result parsing
-- `OpenOptionsTest.java` - test configuration
-- `ExceptionTest.java` - test error handling
+```java
+// Extract all supported types
+try (Extractor extractor = Extractor.create(ExtractFlags.ALL)) {
+    List<ExtractedMatch> matches = extractor.extract(
+        "Contact user@example.com at 192.168.1.1 about evil.com");
+    
+    for (ExtractedMatch match : matches) {
+        System.out.println(match.getItemType() + ": " + match.getValue());
+    }
+}
 
-Use JUnit 5 (already in pom.xml dependencies).
+// Extract only specific types
+int flags = ExtractFlags.DOMAINS | ExtractFlags.IPV4 | ExtractFlags.IPV6;
+try (Extractor extractor = Extractor.create(flags)) {
+    List<ExtractedMatch> matches = extractor.extract(text);
+    // Only domains and IPs are extracted
+}
+```
 
-### 4. Processing API (Medium Priority)
+Supported extraction types (see ExtractFlags):
+- DOMAINS - domain names (e.g., "example.com")
+- EMAILS - email addresses
+- IPV4 / IPV6 - IP addresses
+- HASHES - file hashes (MD5, SHA1, SHA256, SHA384, SHA512)
+- BITCOIN / ETHEREUM / MONERO - cryptocurrency addresses
+- ALL - extract everything
 
-Implement batch processing utilities (matching Rust processing module):
-- `Worker.java` - batch processing with extractor + multiple databases
-- `FileReader.java` - streaming file I/O with gzip support
-- `MatchResult.java` - match results without file context
-- `LineMatch.java` - match results with line numbers
+## Key TODOs
 
-See `native/matchy/WARP.md` "Processing Module API" section for design.
+Based on README.md, these are the next implementation priorities:
 
-### 5. CI/CD (Medium Priority)
+### 1. CI/CD (Medium Priority)
 
 Create `.github/workflows/ci.yml`:
 - Build native library (Rust) for Linux/macOS/Windows
@@ -283,7 +271,7 @@ Create `.github/workflows/ci.yml`:
 - Generate Javadoc
 - Create release artifacts with platform-specific native libraries
 
-### 6. Examples and Documentation (Low Priority)
+### 2. Examples and Documentation (Low Priority)
 
 Create example programs in `examples/`:
 - `BasicQuery.java` - simple query example

java/src/main/java/com/matchylabs/matchy/ExtractFlags.java

@@ -0,0 +1,46 @@
+package com.matchylabs.matchy;
+
+import com.matchylabs.matchy.jna.NativeStructs;
+
+/**
+ * Flags to configure what types of items to extract.
+ * 
+ * <p>Combine multiple flags with bitwise OR:
+ * <pre>{@code
+ * int flags = ExtractFlags.DOMAINS | ExtractFlags.IPV4 | ExtractFlags.IPV6;
+ * Extractor extractor = Extractor.create(flags);
+ * }</pre>
+ */
+public final class ExtractFlags {
+    
+    private ExtractFlags() {
+        // Non-instantiable
+    }
+    
+    /** Extract domain names (e.g., "example.com") */
+    public static final int DOMAINS = NativeStructs.MATCHY_EXTRACT_DOMAINS;
+    
+    /** Extract email addresses (e.g., "user@example.com") */
+    public static final int EMAILS = NativeStructs.MATCHY_EXTRACT_EMAILS;
+    
+    /** Extract IPv4 addresses */
+    public static final int IPV4 = NativeStructs.MATCHY_EXTRACT_IPV4;
+    
+    /** Extract IPv6 addresses */
+    public static final int IPV6 = NativeStructs.MATCHY_EXTRACT_IPV6;
+    
+    /** Extract file hashes (MD5, SHA1, SHA256, SHA384, SHA512) */
+    public static final int HASHES = NativeStructs.MATCHY_EXTRACT_HASHES;
+    
+    /** Extract Bitcoin addresses */
+    public static final int BITCOIN = NativeStructs.MATCHY_EXTRACT_BITCOIN;
+    
+    /** Extract Ethereum addresses */
+    public static final int ETHEREUM = NativeStructs.MATCHY_EXTRACT_ETHEREUM;
+    
+    /** Extract Monero addresses */
+    public static final int MONERO = NativeStructs.MATCHY_EXTRACT_MONERO;
+    
+    /** Extract all supported types */
+    public static final int ALL = NativeStructs.MATCHY_EXTRACT_ALL;
+}