Recover logs from crashed and terminated pods#179
Open
sebasnallar wants to merge 1 commit into
Open
Conversation
Two layered fixes for the case where a pod's container has died but the log workflow returns nothing: 1. kube-logger-go now fetches the previous container instance's logs (PodLogOptions.Previous) on the first page for any pod whose application container has restarted or has a terminated last state. The kubelet retains exactly one prior instance per container, so this surfaces the crash output that `kubectl logs` alone hides while the replacement container is starting. Previous-container fetching is gated on the absence of a per-pod pagination cursor, so paginated requests stay unchanged: no extra round-trips, no duplicated lines (the existing timestamp-based dedup in processor handles overlap), and no token format change. 2. Deployment template flips terminationMessagePolicy from File to FallbackToLogsOnError on the application and traffic containers. When the file at /dev/termination-log is empty and the container exits non-zero, the kubelet auto-populates status.containerStatuses[].lastState.terminated.message with the tail (~80 kB) of stdout/stderr, which survives container restarts inside the Pod object itself. Apps that already write to the termination file are unaffected. Together these cover CrashLoopBackOff and recently-terminated containers without any new in-cluster tooling. Pods that have been fully garbage-collected still require an external log aggregator.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two layered fixes for the case where a pod's container has died but the log workflow returns nothing:
kube-logger-go now fetches the previous container instance's logs (PodLogOptions.Previous) on the first page for any pod whose application container has restarted or has a terminated last state. The kubelet retains exactly one prior instance per container, so this surfaces the crash output that
kubectl logsalone hides while the replacement container is starting. Previous-container fetching is gated on the absence of a per-pod pagination cursor, so paginated requests stay unchanged: no extra round-trips, no duplicated lines (the existing timestamp-based dedup in processor handles overlap), and no token format change.Deployment template flips terminationMessagePolicy from File to FallbackToLogsOnError on the application and traffic containers. When the file at /dev/termination-log is empty and the container exits non-zero, the kubelet auto-populates status.containerStatuses[].lastState.terminated.message with the tail (~80 kB) of stdout/stderr, which survives container restarts inside the Pod object itself. Apps that already write to the termination file are unaffected.
Together these cover CrashLoopBackOff and recently-terminated containers without any new in-cluster tooling. Pods that have been fully garbage-collected still require an external log aggregator.