Isolating Kubernetes pods for debugging

This article introduces a technique that helps you with debugging running Pods in production.

By changing labels, you can detach Pods from the Service (no traffic), and you troubleshoot them live.GIF

Here’s how it works.

Imagine you have a Deployment with three replicas.

Each Pod has an app=hello label.

A Service routes the traffic to your Pods using the selector app=hello

If you want to isolate a Pod you can overwrite the existing label with: kubectl label pod <pod-name> app=debug --overwrite

Two things happen next.

First, the Service stops routing traffic to the Pod because the Service’s selector doesn’t match the label.

Then, the ReplicaSet notices that there are only two replicas, but you asked for 3.

The ReplicaSet creates a new Pod.

You end up with 4 pods:

• 3 Pods running production traffic.
• A single isolated pod that used to receive traffic, but doesn’t anymore.

The pod is still running and you can inspect its state.

What tool should you use?

For simple tasks, you could use:

• kubectl exec, to attach to the container.
• kubectl port-forward, to tunnel the traffic.
• kubectl debug to run a sidecar container alongside the existing one.

But there are more options.

Inspector gadget is a tool designed to introspect and debug Kubernetes applications using eBPF.

If you use VS Code, you might like the official Kubernetes extension.

In this tutorial, you’ll learn how to use it to attach the debugger to the application in the Pod.

If you feel at home in the command line, you should check out netshoot — a Docker + Kubernetes network trouble-shooting swiss-army container.

Which, of course, you can combine it with kubectl debug.

If you need more help to troubleshoot your deployments in Kubernetes, you should check out my flowchart: