loki-stack broken in 2.3.3 on AWS EKS

[Obsecurus]

EKS Version: v1.22.13-eks-15b7512
loki-stack config:

loki-stack:
  fluentbit:
    enabled: false
  grafana:
    enabled: false
  loki:
    persistence:
      enabled: true
      accessModes:
      - ReadWriteOnce
      size: 20Gi
      storageClassName: gp3
      annotations: {}

When running v2.3.3+ pachyderm-loki-0 crashes with the following error (I haven't tested 2.4):

level=info ts=2022-10-14T20:31:48.599123205Z caller=worker.go:112 msg="Starting querier worker using query-scheduler and scheduler ring for addresses"
level=info ts=2022-10-14T20:31:48.601677809Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2022-10-14T20:31:48.601888115Z caller=module_service.go:82 msg=initialising module=query-frontend-tripperware
level=info ts=2022-10-14T20:31:48.601966606Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=error ts=2022-10-14T20:31:48.602039492Z caller=loki.go:386 msg="module failed" module=memberlist-kv error="invalid service state: Failed, expected: Running, failure: service &{0xc0009ce0a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true  7946  false [pachyderm-loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 {[] 7946 5000000000 5000000000 false 0xc000100a50 loki false {    false}} 0xc000100a50 loki [{ringDesc 0xe43cc0} {}]} 0xc0008c9090 0xc000100a50 0xc0000e16c0 {{} [0 1 0]} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc00005d500 0xc0009bff80 0xc0009ca060 0xc0009ca120 0xc0009ca1e0 0xc0009ca420 0xc0009ca2a0 0xc0009ca4e0 0xc0009ca360 0xc00092bec0 0xc00092bf00 0xc0009ca660 0xc0009ca720 0xc0009ca8a0 0xc0009ca7e0 0xc00000fdf0 0xc0002967e0 0xc00000fde0 0xc00000fde8 0xc00092bf40 0xc00092bf80 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2022-10-14T20:31:48.602086242Z caller=loki.go:386 msg="module failed" module=querier error="failed to start querier, because it depends on module store, which has failed: context canceled"
level=error ts=2022-10-14T20:31:48.60211813Z caller=loki.go:386 msg="module failed" module=ingester error="failed to start ingester, because it depends on module store, which has failed: context canceled"
level=error ts=2022-10-14T20:31:48.602142945Z caller=loki.go:386 msg="module failed" module=query-frontend error="failed to start query-frontend, because it depends on module usage-report, which has failed: context canceled"
level=error ts=2022-10-14T20:31:48.602174682Z caller=loki.go:386 msg="module failed" module=query-scheduler error="failed to start query-scheduler, because it depends on module usage-report, which has failed: context canceled"
level=info ts=2022-10-14T20:31:48.602200254Z caller=module_service.go:114 msg="module stopped" module=query-frontend-tripperware
level=error ts=2022-10-14T20:31:48.602220776Z caller=loki.go:386 msg="module failed" module=usage-report error="failed to start usage-report, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0009ce0a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true  7946  false [pachyderm-loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 {[] 7946 5000000000 5000000000 false 0xc000100a50 loki false {    false}} 0xc000100a50 loki [{ringDesc 0xe43cc0} {}]} 0xc0008c9090 0xc000100a50 0xc0000e16c0 {{} [0 1 0]} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc00005d500 0xc0009bff80 0xc0009ca060 0xc0009ca120 0xc0009ca1e0 0xc0009ca420 0xc0009ca2a0 0xc0009ca4e0 0xc0009ca360 0xc00092bec0 0xc00092bf00 0xc0009ca660 0xc0009ca720 0xc0009ca8a0 0xc0009ca7e0 0xc00000fdf0 0xc0002967e0 0xc00000fde0 0xc00000fde8 0xc00092bf40 0xc00092bf80 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2022-10-14T20:31:48.60226342Z caller=loki.go:386 msg="module failed" module=compactor error="failed to start compactor, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0009ce0a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true  7946  false [pachyderm-loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 {[] 7946 5000000000 5000000000 false 0xc000100a50 loki false {    false}} 0xc000100a50 loki [{ringDesc 0xe43cc0} {}]} 0xc0008c9090 0xc000100a50 0xc0000e16c0 {{} [0 1 0]} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc00005d500 0xc0009bff80 0xc0009ca060 0xc0009ca120 0xc0009ca1e0 0xc0009ca420 0xc0009ca2a0 0xc0009ca4e0 0xc0009ca360 0xc00092bec0 0xc00092bf00 0xc0009ca660 0xc0009ca720 0xc0009ca8a0 0xc0009ca7e0 0xc00000fdf0 0xc0002967e0 0xc00000fde0 0xc00000fde8 0xc00092bf40 0xc00092bf80 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2022-10-14T20:31:48.602282571Z caller=loki.go:386 msg="module failed" module=ingester-querier error="failed to start ingester-querier, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0009ce0a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true  7946  false [pachyderm-loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 {[] 7946 5000000000 5000000000 false 0xc000100a50 loki false {    false}} 0xc000100a50 loki [{ringDesc 0xe43cc0} {}]} 0xc0008c9090 0xc000100a50 0xc0000e16c0 {{} [0 1 0]} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc00005d500 0xc0009bff80 0xc0009ca060 0xc0009ca120 0xc0009ca1e0 0xc0009ca420 0xc0009ca2a0 0xc0009ca4e0 0xc0009ca360 0xc00092bec0 0xc00092bf00 0xc0009ca660 0xc0009ca720 0xc0009ca8a0 0xc0009ca7e0 0xc00000fdf0 0xc0002967e0 0xc00000fde0 0xc00000fde8 0xc00092bf40 0xc00092bf80 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2022-10-14T20:31:48.60234779Z caller=loki.go:386 msg="module failed" module=store error="failed to start store, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0009ce0a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true  7946  false [pachyderm-loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 {[] 7946 5000000000 5000000000 false 0xc000100a50 loki false {    false}} 0xc000100a50 loki [{ringDesc 0xe43cc0} {}]} 0xc0008c9090 0xc000100a50 0xc0000e16c0 {{} [0 1 0]} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc00005d500 0xc0009bff80 0xc0009ca060 0xc0009ca120 0xc0009ca1e0 0xc0009ca420 0xc0009ca2a0 0xc0009ca4e0 0xc0009ca360 0xc00092bec0 0xc00092bf00 0xc0009ca660 0xc0009ca720 0xc0009ca8a0 0xc0009ca7e0 0xc00000fdf0 0xc0002967e0 0xc00000fde0 0xc00000fde8 0xc00092bf40 0xc00092bf80 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2022-10-14T20:31:48.602389231Z caller=loki.go:386 msg="module failed" module=ring error="failed to start ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0009ce0a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true  7946  false [pachyderm-loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 {[] 7946 5000000000 5000000000 false 0xc000100a50 loki false {    false}} 0xc000100a50 loki [{ringDesc 0xe43cc0} {}]} 0xc0008c9090 0xc000100a50 0xc0000e16c0 {{} [0 1 0]} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc00005d500 0xc0009bff80 0xc0009ca060 0xc0009ca120 0xc0009ca1e0 0xc0009ca420 0xc0009ca2a0 0xc0009ca4e0 0xc0009ca360 0xc00092bec0 0xc00092bf00 0xc0009ca660 0xc0009ca720 0xc0009ca8a0 0xc0009ca7e0 0xc00000fdf0 0xc0002967e0 0xc00000fde0 0xc00000fde8 0xc00092bf40 0xc00092bf80 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2022-10-14T20:31:48.602605703Z caller=loki.go:386 msg="module failed" module=distributor error="failed to start distributor, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0009ce0a0 { true 10000000000 4 30000000000 200000000 3 30000000000 0 true  7946  false [pachyderm-loki-memberlist] 1000000000 60000000000 10 false 0 300000000000 5000000000 0 {[] 7946 5000000000 5000000000 false 0xc000100a50 loki false {    false}} 0xc000100a50 loki [{ringDesc 0xe43cc0} {}]} 0xc0008c9090 0xc000100a50 0xc0000e16c0 {{} [0 1 0]} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xe43cc0} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc00005d500 0xc0009bff80 0xc0009ca060 0xc0009ca120 0xc0009ca1e0 0xc0009ca420 0xc0009ca2a0 0xc0009ca4e0 0xc0009ca360 0xc00092bec0 0xc00092bf00 0xc0009ca660 0xc0009ca720 0xc0009ca8a0 0xc0009ca7e0 0xc00000fdf0 0xc0002967e0 0xc00000fde0 0xc00000fde8 0xc00092bf40 0xc00092bf80 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=info ts=2022-10-14T20:31:48.603695878Z caller=modules.go:1090 msg="server stopped"
level=info ts=2022-10-14T20:31:48.603788105Z caller=module_service.go:114 msg="module stopped" module=server
level=info ts=2022-10-14T20:31:48.603873044Z caller=loki.go:375 msg="Loki stopped"
failed services
github.com/grafana/loki/pkg/loki.(*Loki).Run
	/src/loki/pkg/loki/loki.go:421
main.main
	/src/loki/cmd/loki/main.go:105
runtime.main
	/usr/local/go/src/runtime/proc.go:255
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1581
level=error ts=2022-10-14T20:31:48.6040592Z caller=log.go:103 msg="error running loki" err="failed services\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:421\nmain.main\n\t/src/loki/cmd/loki/main.go:105\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"

[Obsecurus]

pinning to 2.3.2 fixes this for me:

  - name: pachyderm
    namespace: default
    chart: pach/pachyderm
    version: 2.3.2
    values:
      - ./values/pachyderm.yaml

[Obsecurus]

I also noticed this which may be related: grafana/helm-charts#1584

[pradeepacuver]

We are also facing the same issue with Loki 2.8 on EKS 1.25 (K8s). A fix or a sample values.yaml with IP address providing would help.