JLogShip - Java Log Shipper

JLogShip is a lightweight, high-performance log shipping engine written in Java, inspired by Filebeat. It efficiently tails log files, processes events, and ships them to various outputs like Elasticsearch and Kafka with at-least-once delivery guarantees.

🚀 Features

• File Tailing: Efficiently monitors and tails log files in real-time
• Multiple Outputs: Support for Elasticsearch and Kafka destinations
• State Persistence: Registry-based offset tracking for reliable resumption
• Configurable: YAML-based configuration for inputs and outputs
• Multithreaded: Concurrent processing with thread pools for optimal performance
• Docker Ready: Pre-configured Docker Compose setups for ELK stack and Kafka
• At-Least-Once Delivery: Ensures no log events are lost during shipping

🏗️ Architecture

JLogShip follows a modular architecture with the following key components:

• Config Loader: Loads runtime configuration from YAML files
• Prospector: Discovers and monitors log files using glob patterns
• Harvester: Reads individual files line-by-line, handling log rotation
• Registry: Persists file offsets and metadata for state management
• Event Queue: Batches log events for efficient processing
• Output Workers: Ships batched events to configured destinations

Config Loader → Prospector → Harvester → Event Queue → Output Worker → Elasticsearch/Kafka
     ↓            ↓            ↓            ↓            ↓
   YAML        File Scan    Line Read    Batching    Shipping

📋 Prerequisites

• Java 21 or higher
• Maven 3.8+
• Docker and Docker Compose (for infrastructure setup)

🛠️ Installation

1. Clone the repository:

   git clone https://github.com/yourusername/JLogShip.git
   cd JLogShip/logshipengine

2. Build the project:

   mvn clean compile

⚙️ Configuration

JLogShip uses YAML configuration files. The main configuration is in src/main/resources/application.yml.

Example Configuration

inputs:
  - type: log
    paths:
      - "/var/log/*.log"
      - "/app/logs/**/*.log"

output:
  type: elasticsearch  # or kafka
  hosts: ["http://localhost:9200"]
  index: "logship-events"

# For Kafka output:
# output:
#   type: kafka
#   hosts: ["localhost:9092"]
#   topic: "logship-events"

registry:
  path: "./registry/registry.json"

Configuration Options

• inputs: Array of input configurations
  • type: Input type (currently supports "log")
  • paths: Glob patterns for log file paths
• output: Output destination configuration
  • type: Output type ("elasticsearch" or "kafka")
  • hosts: Array of host URLs
  • index/topic: Destination index/topic name
• registry: State persistence configuration
  • path: Path to registry JSON file

🚀 Usage

Running the Application

1. Configure your settings in application.yml
2. Run the application:

   mvn exec:java -Dexec.mainClass="org.jlogship.App"

Docker Infrastructure Setup

JLogShip includes Docker Compose files for setting up infrastructure:

ELK Stack (Elasticsearch + Kibana)

cd infra/util
docker-compose -f docker-setup-elk.yaml up -d

This starts:

• Elasticsearch on http://localhost:9200
• Kibana on http://localhost:5601

Kafka Stack

cd infra/util
docker-compose -f docker-setup-kafka.yaml up -d

This starts:

• Zookeeper
• Kafka broker on localhost:9092
• Kafka UI on http://localhost:8080

🏃‍♂️ Running Tests

mvn test

📁 Project Structure

JLogShip/
├── concepts.md                 # Core concepts documentation
├── design-doc.md              # Detailed design documentation
├── logshipengine/             # Main Maven project
│   ├── pom.xml               # Maven configuration
│   ├── infra/
│   │   └── util/             # Docker Compose files
│   ├── src/
│   │   ├── main/
│   │   │   ├── java/org/jlogship/
│   │   │   │   ├── App.java              # Main application entry point
│   │   │   │   ├── config/               # Configuration loading
│   │   │   │   ├── harvester/            # File reading components
│   │   │   │   ├── model/                # Data models
│   │   │   │   ├── output/               # Output plugins
│   │   │   │   ├── pipeline/             # Event processing pipeline
│   │   │   │   ├── prospector/           # File discovery
│   │   │   │   ├── registry/             # State persistence
│   │   │   │   └── util/                 # Utilities
│   │   │   └── resources/                # Configuration files
│   │   └── test/                         # Unit tests
│   └── logs/                             # Sample log files
└── README.md               # This file

🔧 Development

Building from Source

mvn clean install

Key Components

• Harvester: Handles individual file tailing and line reading
• Prospector: Manages file discovery and harvester lifecycle
• Output Plugins: Pluggable architecture for different destinations
• Registry: JSON-based state store for offset tracking

Adding New Outputs

Implement the Output interface and register via ServiceLoader:

public interface Output {
    void send(LogEvent event) throws Exception;
    void close() throws Exception;
}

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📝 Known Issues

1. File Discovery: Currently only detects files at startup. Post-startup file creation/deletion requires restart.

   • Solution: Implement periodic re-scan and change detection.

2. Backpressure: No backpressure handling for output publishing.

   • Solution: Add backpressure mechanisms in the event queue.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• Inspired by Filebeat architecture
• Built with modern Java features and best practices
• Uses industry-standard libraries for reliability

---

Happy Logging! 🚀