Debugging Production Pods

👋 Hi! I’m Bibin Wilson. In each edition, I share practical tips, guides, and the latest trends in DevOps and MLOps to make your day-to-day DevOps tasks more efficient. If someone forwarded this email to you, you can subscribe here to never miss out!

In this edition we will look at,

1. Ephemeral containers

2. Debugging a Pod using Kubectl debug

3. Isolating Production Pods for Debugging

Use Case

When running containers in production clusters, many projects prefer distroless or minimal base images to improve security and reduce resource usage.

However, there is a problem with these images. They usually don't have a shell or other utilities needed for troubleshooting. Because of this, you can't run kubectl exec to debug issues.

So how can you troubleshoot an application when something goes wrong?

For example:

• Checking network connectivity

• Troubleshooting DNS issues

• Inspecting files or configurations

This is where ephemeral containers come into the picture.

Ephemeral container With Kubectl debug

Ephemeral containers are temporary containers that run in the same pod namespace as your application. You can add them even after the pod has started.

They're ideal for troubleshooting.

Imagine your pod runs on a minimal base image containing only essential application binaries and dependencies. If something goes wrong, you might need to debug quickly.

With ephemeral containers, you can add a debug container directly into the running pod. This debug container can have all the necessary utilities like a shell, curl, or other custom networking tools for troubleshooting your application.

Here’s what happens when you create a debug container.

1. k8s adds a temporary (ephemeral) container directly inside the existing pod.

2. It shares the same pod's network namespace (can directly access localhost services).

3. It shares volumes, storage, and resources with existing containers in the pod.

Let's look at this with practical examples.

Debugging a Pod

First, let's look at how to debug a pod.

Suppose you have an application running inside a pod that's unable to connect to a database pod.

You've already checked configurations, endpoints, and done general troubleshooting. Now, you want to use the nc (netcat) command to verify network connectivity from your application pod to the database.

Or maybe you need to run strace to identify issues with a slow process.

In these cases, you can use the kubectl debug command to add an ephemeral debug container to your existing pod.

For example,

The following command shows how to add an ephemeral debug container (nicolaka/netshoot image) to nginx-pod and test the database endpoint using nc

$ kubectl debug -it backend-pod --image=nicolaka/netshoot -- sh

~# nc -zv 10.107.59.123 3306

Connection to 10.107.59.123 3306 port [tcp/mysql] succeeded! 

Although the above command adds a debug container inside the pod, Kubernetes won't show the ephemeral container when listing pods.

For example, after deploying the debug container, the nginx-pod still shows (1/1) containers running. This means it only counts the original application container, not the ephemeral one.

$ kubectl get po

NAME          READY   STATUS    RESTARTS   AGE
mariadb-pod   1/1     Running   0          4m19s
nginx-pod     1/1     Running   0          2m46s

However, if you describe the nginx-pod, you will see he ephmeric conaeirn in the output. (I have truncated the output)

Isolating Production Pods for Debugging

Lets say you have a live application running in production, and you're facing performance issues.

You want to troubleshoot or experiment without affecting the live pod.

For example, you might want to change parameters or use a new image version with the current production configuration.

In such cases, you can copy the production pod using kubectl debug.

This helps you:

• Reproduce and isolate issues outside the production pod.

• Run diagnostic tools (such as strace, tcpdump, or other debugging utilities) without risking or adding overhead to the original pod.

To copy a pod, use the --copy-to parameter as shown below.

$ kubectl debug mariadb-pod \
        --copy-to=mariadb-test-pod \
        -it --container=mariadb -- bash

The previous command just copies the pod and opens an exec session.

But you can also copy a pod and attach a debug container at the same time.

For example, if you want to copy mariadb-pod and add a debug image (nicolaka/netshoot) for troubleshooting, you can use:

$ kubectl debug mariadb-pod \
     --copy-to=mariadb-debug-pod \
     --image=nicolaka/netshoot -it -- sh

The above command creates a copy of the mariadb-pod with an additional debug container inside it.

$ kubectl get po

NAME                READY   STATUS             RESTARTS         AGE
mariadb-debug-pod   2/2     Running            0                7s
mariadb-pod         1/1     Running            0                25m

If you check the output, you'll see two containers inside the debug pod. This is different from regular debugging, where the ephemeral container doesn't appear in the pod listing.

This happens because the --copy-to flag creates a completely new pod. It copies your original container (mariadb) and adds the new debug container (netshoot).

Tomorrow…

In tomorrow's edition, we'll explore how to debug Kubernetes cluster nodes using kubectl debug, along with practical examples.