🚀 Redis + ELK Stack + Spring Boot 설정 가이드

[Tae4an]

1. 📋 사전 준비

1.1 시스템 요구사항

• 🐳 Docker와 Docker Compose 설치
• ☕ JDK 17 이상
• 💾 최소 4GB RAM (권장 8GB 이상)
• 💽 최소 20GB 여유 디스크 공간

1.2 필요한 포트

• 6379: Redis
• 9200: Elasticsearch
• 5601: Kibana
• 5044: Logstash
• 8080: Spring Boot Application

2. 🛠️ ELK + Redis 스택 설정

2.1 프로젝트 구조 생성

mkdir -p elk-stack && cd elk-stack

# 서비스별 디렉토리 생성
mkdir -p redis/config \
        elasticsearch/setup \
        logstash/{config,pipeline} \
        kibana/config

# 설정 파일 생성
touch docker-compose.yml .env

2.2 환경 변수 설정 (.env)

ELASTIC_PASSWORD=[YOUR_PASSWORD]
KIBANA_SYSTEM_PASSWORD=[YOUR_PASSWORD]
REDIS_PASSWORD=[YOUR_PASSWORD]

2.3 Docker Compose 설정 (docker-compose.yml)

version: '3.8'

services:
  elasticsearch:
    image: elasticsearch:8.12.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - elasticsearch_data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    networks:
      - elk_network
    restart: unless-stopped

  kibana:
    image: kibana:8.12.0
    container_name: kibana
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_SYSTEM_PASSWORD}
      - XPACK_ENCRYPTEDSILVEROBJECTS_ENCRYPTIONKEY=something_at_least_32_characters_long
      - XPACK_SECURITY_ENCRYPTIONKEY=something_at_least_32_characters_long
      - XPACK_REPORTING_ENCRYPTIONKEY=something_at_least_32_characters_long
    ports:
      - "5601:5601"
    networks:
      - elk_network
    depends_on:
      - elasticsearch
    restart: unless-stopped

  redis:
    image: redis:7.2
    container_name: redis
    command: redis-server --requirepass ${REDIS_PASSWORD}
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    networks:
      - elk_network
    restart: unless-stopped

  logstash:
    image: logstash:8.12.0
    container_name: logstash
    environment:
      - REDIS_PASSWORD=${REDIS_PASSWORD}
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
    ports:
      - "5044:5044"
    networks:
      - elk_network
    depends_on:
      - elasticsearch
      - redis
    restart: unless-stopped

networks:
  elk_network:
    driver: bridge

volumes:
  redis_data:
  elasticsearch_data:

2.4 Logstash 설정 (logstash/pipeline/logstash.conf)

input {
  redis {
    host => "redis"
    port => 6379
    password => "${REDIS_PASSWORD}"
    data_type => "list"
    key => "logstash"
    codec => json
  }
}

filter {
  if [message] =~ /^{.*}$/ {
    json {
      source => "message"
    }
  }

  date {
    match => [ "timestamp", "ISO8601" ]
    target => "@timestamp"
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    user => "elastic"
    password => "${ELASTIC_PASSWORD}"
    index => "logstash-%{+YYYY.MM.dd}"
  }

  # 디버깅용 콘솔 출력
  stdout { codec => rubydebug }
}

3. 🌱 Spring Boot 설정

3.1 프로젝트 생성

• ⚡ Spring Initializr 사용
• 🔧 Java 17 선택
• 📦 Spring Boot 3.x 선택

3.2 의존성 추가 (build.gradle)

dependencies {
    developmentOnly 'org.springframework.boot:spring-boot-devtools'
    testImplementation 'org.springframework.boot:spring-boot-starter-test'
    implementation 'org.springframework.boot:spring-boot-starter-web'
    compileOnly 'org.projectlombok:lombok'
    annotationProcessor 'org.projectlombok:lombok'

    // Spring Data Redis
    implementation 'org.springframework.boot:spring-boot-starter-data-redis'

    // Logback Redis Appender
    implementation 'com.cwbase:logback-redis-appender:1.1.6'

    // Jedis
    implementation 'redis.clients:jedis:3.1.0'

    // Logback Classic
    implementation 'ch.qos.logback:logback-classic'

    // Logstash Logback Encoder
    implementation 'net.logstash.logback:logstash-logback-encoder:7.4'
}

3.3 application.yml 설정

spring:
  data:
    redis:
      host: ${SERVER_URL}
      port: 6379
      password: ${REDIS_PASSWORD}

logging:
  config: classpath:logback-spring.xml
  level:
    root: INFO
    com.example: DEBUG

3.4 logback-spring.xml 설정

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <springProperty scope="context" name="SERVER_URL" source="SERVER_URL"/>

    <!-- 콘솔 출력 설정 -->
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>

    <!-- Redis Appender -->
    <appender name="REDIS" class="com.cwbase.logback.RedisAppender">
        <host>${SERVER_URL}</host>
        <port>6379</port>
        <password>${REDIS_PASSWORD}</password>
        <key>logstash</key>
        <type>redis</type>
        <layout class="net.logstash.logback.layout.LogstashLayout">
            <customFields>{"application":"test"}</customFields>
        </layout>
    </appender>

    <!-- 비동기 처리를 위한 Async Appender -->
    <appender name="ASYNC_REDIS" class="ch.qos.logback.classic.AsyncAppender">
        <appender-ref ref="REDIS" />
        <queueSize>512</queueSize>
        <discardingThreshold>0</discardingThreshold>
    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE" />
        <appender-ref ref="ASYNC_REDIS" />
    </root>

    <logger name="com.example" level="DEBUG" additivity="false">
        <appender-ref ref="CONSOLE" />
        <appender-ref ref="ASYNC_REDIS" />
    </logger>
</configuration>

3.5 테스트 컨트롤러 작성

@RestController
@RequestMapping("/api")
@Slf4j
public class TestController {

    @GetMapping("/log-test")
    public String test() {
        log.info("=== Test Start ===");
        log.debug("Debug level test message");
        log.info("Info level test message with data: {}", new Date());
        log.warn("Warning level test message");
        return "test";
    }
}

4. ▶️ 실행 및 테스트

4.1 ELK 스택 실행

docker-compose up -d

4.2 Spring Boot 실행

./gradlew bootRun

4.3 로그 생성 테스트

curl http://localhost:8080/api/log-test

4.4 로그 확인 방법

Redis 로그 확인

docker exec -it redis redis-cli -a ${REDIS_PASSWORD}
LLEN logstash
LRANGE logstash -1 -1

Kibana 로그 확인

1. 🌐 http://localhost:5601 접속
2. 📊 Kibana > Discover 메뉴 이동
3. 🔍 인덱스 패턴: logstash-* 설정
4. ⏰ 시간 범위 설정 후 조회

5. 🔄 로그 수집 프로세스

graph LR
    A[Spring Boot] -->|로그 생성| B[Redis LIST]
    B -->|수집| C[Logstash]
    C -->|저장| D[Elasticsearch]
    D -->|시각화| E[Kibana]

Loading

6. 💡 참고사항

• 📝 모든 비밀번호는 반드시 안전하게 관리
• 🔒 프로덕션 환경에서는 보안 설정 추가 필요
• 🔄 메모리 설정은 시스템 환경에 맞게 조정
• 📊 Kibana에서 대시보드 구성으로 모니터링 강화 가능