How to remove stuck K8s namespaces

Chances are if you work with Kubernetes you have seen a namespace that is stuck in the “terminating” stage. I’m not a Kubernetes expert (yet) but this process works great for me whenever it happens, and doesn’t take long at all.

Step 1 – Identify which namespace you want to remove

Step 2 – Assign environment variable to the namespace to delete

The first namespace that I want to kill is “blog”, don’t worry though my blog was not harmed in the making of this post.  At a bash or sh prompt on a machine with kubectl, type this:

NAMESPACE=blog

Now, when you copy and paste the other commands into your terminal you won’t need to modify them.

Step 3 – Dump namespace info to JSON tmp file

kubectl get namespace ${NAMESPACE} -o json > tmp.json

Step 4 – Edit tmp.json to remove “kubernetes” line

I use nano for this, which makes it super easy, simply arrow down to the line you want to cut then press Control+k and it will remove it, then press Control+o to write the file and finally Control+x to exit nano.

nano tmp.json

When you are done it should look like this

Step 5 – Next start the kubernetes api proxy

This sets up a local port on your kubectl machine that is attached to the Kubernetes APIs. no need to authenticate as it takes care of all that for you from your .kube config file.

kubectl proxy --port=8080 &

Press enter a couple times to get back down to a prompt, since we backgrounded the task.

Step 6 – Issue POST api request to k8s

This next command sends the updated tmp.json to the k8s API. This will tell k8s its fine to let go 

curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8080/api/v1/namespaces/${NAMESPACE}/finalize

You will see a bunch of json output, but then be returned to a prompt. Here we can issue the command to get namespaces again, this time the namespace in question should be gone.

kubectl get namespaces

Nice! it’s gone! Now I just need to change my NAMESPACE variable to the next one in the list and reissue the commands (no need to start kubectl proxy again though, it’s still running in the background)

Hope this helps!