Kubernetes Node Debugging

👋 Hi! I’m Bibin Wilson. In each edition, I share practical tips, guides, and the latest trends in DevOps and MLOps to make your day-to-day DevOps tasks more efficient. If someone forwarded this email to you, you can subscribe here to never miss out!

In yesterday's edition, we looked at debugging pods.

In today's edition, we will look at how to debug nodes using kubectl debug

Use Case

The following are some troubleshooting tasks you might want to perform on a Kubernetes node.

• Check the filesystem quickly

• Run tools like tcpdump, nc, ping, etc.

For these use cases, you don't need to SSH into the specific node.

Instead, you can use kubectl debug.

Kubectl debug node

When you use kubectl debug, it launches a container with specific debugging utilities on a specific node.

For example, if your node name is node01, you can launch a debug container using the following command.

kubectl debug node/node01 -it --image=nicolaka/netshoot

The above command creates an ephemeral debug container inside the specified node (node01).

You can run the required troubleshooting commands on that node, such as nc, curl, etc.

Shared Host Filesystem

When you deploy the debug container on a node, the host's filesystem is shared in the /host folder of the container.

This allows you to check the host filesystem and logs.

node01~: /host: ls

bin                 lost+found          snap
bin.usr-is-merged   media               srv
boot                mnt                 swap.img
cdrom               opt                 sys
dev                 proc                tmp
etc                 root                usr
home                run                 vagrant
lib                 sbin                var
lib.usr-is-merged   sbin.usr-is-merged

You can find all the pod log entries in the relevant log files on the node.

node01~: cd /host/var/log/pods

Wrapping Up

Hope I was able to help you learn something new this week.

Happy weekend!