Sebastian Gomez
Troubleshooting and analyzing problems in Kubernetes with kubectl
As a Kubernetes administrator, I have run into many problems and challenges while deploying and managing applications in clusters. In this article I want to share my experiences and knowledge using kubectl to troubleshoot and effectively analyze your Kubernetes applications. Throughout the post I will show you how to use essential kubectl commands, interpret the different pod phases, and run an efficient diagnosis of your deployments. I will also include a diagram to illustrate the pod lifecycle, a five point summary at the end, conclusions, and practical exercises so you can apply what you have learned.
Essential kubectl commands for troubleshooting
To get started, it is essential to get familiar with the basic kubectl commands that help us identify and fix problems in our Kubernetes clusters. Some of the commands I use frequently include:
`kubectl get`: this command lets me get information about the objects in my cluster, such as pods, services, and other components. For example, I can list all pods and check their current status:
kubectl get pods`kubectl describe`: when I need more details about a specific object, I turn to kubectl describe. For example, to get detailed information about the pod named my-pod-name:
kubectl describe pod my-pod-name`kubectl exec`: this command lets me run commands and applications inside a running pod. It is important to include the -- separator, which marks where the kubectl options end and the command you want to run inside the container begins:
# Run a one off command inside the pod
kubectl exec my-pod-name -- ls /app`kubectl logs`: when I need to see what is happening inside a pod, kubectl logs is a powerful tool for analyzing logs and debugging messages from the applications running inside the pods:
kubectl logs my-pod-namePod phases and what they mean
Pod phases in Kubernetes give me valuable information about the lifecycle and performance of my applications. The pod phase (the status.phase field) can take one of these five values:
- Pending: the pod has been accepted by Kubernetes but has not been scheduled yet, or its images are still being pulled. The containers are not running yet.
- Running: the pod has been assigned to a node and all of its containers have been created successfully; at least one is running.
- Succeeded: all containers in the pod have finished running successfully and will not be restarted.
- Failed: one or more containers in the pod terminated with errors and will not be restarted.
- Unknown: the pod's state cannot be obtained, most likely due to a communication error between the control plane and the kubelet.
Here is the lifecycle represented as a diagram:
stateDiagram-v2
[*] --> Pending
Pending --> Running
Running --> Succeeded
Running --> Failed
Pending --> Failed
Pending --> Unknown
Running --> Unknown
Succeeded --> [*]
Failed --> [*]Container waiting states: what you will actually see when debugging
The pod phase gives you a high level view, but in practice the most common problems show up in the waiting states of the individual containers, which kubectl get pods displays in the STATUS column. These are the ones you will run into most:
- `CrashLoopBackOff`: the container starts, fails, and Kubernetes restarts it over and over with a growing wait (backoff) between attempts. It usually points to a wrong start command, a missing environment variable, or an exception at application startup. Check the logs with
kubectl logsto find the cause. - `ImagePullBackOff` and `ErrImagePull`: Kubernetes could not pull the container image. Typical causes: the image name or tag is misspelled, the registry is private and the credentials are missing (an
imagePullSecrets), or the image does not exist. - Stuck in `Pending`: if a pod stays in
Pending, it is usually because the scheduler cannot find a node with enough resources, there is ataintwithout itstoleration, or aPersistentVolumeClaimcannot be satisfied. Thekubectl describe podcommand will show you the exact event at the end of the output.
Interpreting information about pods and their containers
To better understand what is happening inside my pods and their containers, I use the kubectl describe command to get additional details, such as labels, resource requirements, and volumes. It also gives me information about the container and pod state, like the number of restarts and the restart policy.
It is worth understanding the restart policy (restartPolicy) well, because it is easy to misread. With restartPolicy: Always, the container is restarted every time it terminates, regardless of whether it exited successfully or with an error; it does not mean that a healthy, running container restarts continuously. The three available options are:
- `Always` (default): restarts the container whenever it terminates, whatever its exit code. This is the usual choice for long running services.
- `OnFailure`: restarts the container only if it terminates with a non zero exit code. Useful for batch processes or jobs.
- `Never`: never restarts the container, no matter how it terminates.
When you see a container restarting endlessly (CrashLoopBackOff), it is not that the policy is "wrong"; it is that the container is failing to start and Always keeps trying again and again. I can also review the events at the end of the command output to identify any problem or change in the state of my containers and pods.
Running commands and working inside containers
When I need to troubleshoot an application, I can run individual commands in the container using kubectl exec with the -- separator. If I need to do more, like inspecting the file system or testing connectivity, I can launch an interactive shell using the -i and -t options (shortened as -it), which connect my terminal to the container so I can work inside it:
# Open an interactive shell inside the container
kubectl exec -it my-pod-name -- /bin/shThat said, it is important to remember that installing software directly into a container is not a good practice: those changes are lost as soon as the container restarts. Instead, you should build container images that contain exactly the software you need and deploy them accordingly.
Reviewing pod and container logs
I use the kubectl logs command to view a pod's logs. If a pod has several containers, I can use the -c option to show the logs of a specific container within the pod:
# Logs of a specific container in a multi container pod
kubectl logs my-pod-name -c my-container-nameThe logs contain both the standard output and the error messages from the applications running inside the container. If the container has already restarted and you want to see the logs from the previous instance (very useful with CrashLoopBackOff), add the --previous option:
kubectl logs my-pod-name --previousFive point summary
- Get familiar with the essential
kubectlcommands, such asget,describe,exec, andlogs. - Learn to interpret the different pod phases and, above all, the container waiting states (
CrashLoopBackOff,ImagePullBackOff), which are the real signals of trouble. - Use
kubectl describeto get detailed information about your pods and containers, including events and the restart policy. - Run commands and work inside containers using
kubectl execwith--, and open a shell with-it. - Review your pod and container logs with
kubectl logs, using-cfor multiple containers and--previousfor the previous instance.
Conclusions
By using kubectl effectively, you can troubleshoot and analyze your Kubernetes applications efficiently. Understanding pod phases, container waiting states, and how to interpret the information the kubectl commands return is essential to keeping your deployments running smoothly.
Practical exercises
- Use
kubectl get podsto see the current status of all pods in your cluster. - Pick a specific pod and use
kubectl describe podto get more details about it, paying attention to the events section. - Run a simple command inside a running container using
kubectl exec my-pod-name -- <command>. - Launch an interactive shell inside a container with
kubectl exec -it my-pod-name -- /bin/sh. - Review the logs of a specific pod using
kubectl logs, and try the--previousoption on a pod that has restarted.
That is all. I hope this post is useful to you and that you can apply it to a project you have in mind. Leave me a comment if it helped, if you want to add an opinion, or if you have any questions. And remember, if you liked it, you can also share it using the social links below.
Sebastian Gomez
Creador de contenido principalmente acerca de tecnología.